Commit Graph

334 Commits

Author SHA1 Message Date
Kevin Hoxha f3431fe1e7 metrics: Add MetricsLogger loop and more knobs 2022-12-08 10:07:11 -08:00
FoundationDB CI 86d6106dc1
format source code after switch to clang 15 2022-12-08 17:26:45 +00:00
sfc-gh-tclinkenbeard ec615181ce Mark several more methods const 2022-11-15 14:57:32 -08:00
Junhyun Shim 4600184a4d Fix circular reference in SSLConnection 2022-10-27 00:29:02 +02:00
Marian Dvorsky c6c449d047
Extract TaskQueue out of Net2 and reuse it in sim2 (#8330)
* Extract TaskQueue out of Net2 and reuse it in sim2

* empty commit

* Address review comments

* Introduce MAX_RUNLOOP_SLEEP_DELAY

* Apply clang-format
2022-10-10 12:46:06 -07:00
Markus Pilman 23edfd0d59 Fix formatting 2022-10-04 18:33:30 -06:00
Markus Pilman 550488b020 Merge remote-tracking branch 'origin/main' into bugfixes/open-for-ide
# Conflicts:
#	bindings/c/CMakeLists.txt
#	fdbclient/include/fdbclient/GetEncryptCipherKeys.actor.h
#	fdbserver/BackupWorker.actor.cpp
#	fdbserver/BlobWorker.actor.cpp
#	fdbserver/CommitProxyServer.actor.cpp
#	fdbserver/KeyValueStoreMemory.actor.cpp
#	fdbserver/StorageCache.actor.cpp
#	fdbserver/include/fdbserver/GetEncryptCipherKeys.actor.h
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/PhysicalShardMove.actor.cpp
#	flow/CMakeLists.txt
2022-10-04 18:27:48 -06:00
A.J. Beamon 4fd64630e8 Convert literal string ref instances to use _sr suffix 2022-09-19 11:35:58 -07:00
sfc-gh-tclinkenbeard 39c6989673 Remove some simulation unit tests.
These tests should not be run in simulation, because they either run too
long or break determinism.
2022-09-11 00:36:18 -07:00
Andrew Noyes fbf5830bb2 Rollback #7374
Valgrind is complaining about use of uninitialized memory in
absl::GetStackTrace, and the cases where it complains the backtraces are
incomplete. Note: this means that jemalloc heap profiling no longer
works out of the box. Advanced users who want to enable jemalloc heap
profiling will now have to revert this change and build from source.
2022-09-06 16:55:18 -07:00
Junhyun Shim 964dec9b91 Fix incorrect clang format 2022-08-23 19:48:02 +02:00
Junhyun Shim 8706833355 Further clang-format Net2.actor.cpp 2022-08-23 18:24:11 +02:00
Junhyun Shim de2e51877f Clang format Net2.actor.cpp 2022-08-23 18:00:24 +02:00
Junhyun Shim a312002333 Merge remote-tracking branch 'origin/main' into authz-general-tls-and-integration-test 2022-08-17 17:32:00 +02:00
Marian Dvorsky 8e6009427e Fix Net2 unittest flakiness 2022-08-13 18:51:30 +02:00
Markus Pilman c9865809e0
Merge pull request #7786 from sfc-gh-mdvorsky/mdvorsky/net2_unittest
Add Net2 unittests for ThreadSafeQueue and Net2::onMainThread
2022-08-08 11:23:31 -06:00
Vaidas Gasiunas 79571dd2b4
Testing upgrades to a future version of FDB (#7780)
* Enable configuring the next future protocol version as the current protocol version in FDB client, fdbserver, and fdbcli

* Auto format python files used in upgrade tests

* Add a test for upgrading to a future FDB version

* Emphasize that the options for using future protocol version are intended for test purposes only

* Make the global variable for current protocol version visible only locally

* Refactirng to avoid using currentProtocolVersion() in static intialization

* Update go bindings
2022-08-08 17:29:49 +02:00
Marian Dvorsky 6b790ef4b8 Disable the new onMainThread test in simulation 2022-08-05 17:06:34 +02:00
Marian Dvorsky 7c00c57253 Small fixes 2022-08-05 16:30:05 +02:00
Marian Dvorsky 9e4af40b0f Run clang-format 2022-08-04 16:20:42 +02:00
Marian Dvorsky 70748f51e4 Add Net2 unittests for ThreadSafeQueue and onMainThread 2022-08-04 16:03:59 +02:00
Junhyun Shim 65b441bd1b Watch public key file at CLI-arg path
Argument name: authorization-public-key-file
2022-07-28 20:03:19 +02:00
Junhyun Shim 1dbb13a899 Accept non-mTLS clients as untrusted 2022-07-28 20:03:07 +02:00
Renxuan Wang dc9599f2e9
Get network interfaces ready for https proxy. (#7556)
* Move HTTP from fdbclient/ to fdbrpc/.

* Move md5 and libb64 to contrib/.

* Get network interfaces ready for https proxy.

* Rebase
2022-07-25 17:08:32 -07:00
Markus Pilman 10e478dfc3 Flow is compiling 2022-06-23 16:35:19 -06:00
Lukas Joswiak 88557d9169 Simplify function call when transaction is null 2022-06-22 14:50:17 -07:00
Andrew Noyes 83aceb216c
Use absl::GetStackTrace for slow task profiler (#7374)
* Make SlowTask workload runnable in joshua

* Remove SignalSafeUnwind, and use absl::GetStackTrace for slow task profiler
2022-06-15 14:53:52 -07:00
Renxuan Wang 839af5701e
Fix bug in resolveTCPEndpoint() when hostname resolving fails. (#7375)
* Close trace file when error happens in runNetwork().

* Improve the bestCount algorithm in getLeader().

In the current implementation, if the nominees are [0,1], the chosen leader will be 1, which is an exception to other cases and our expectation that if 2 nominees have the same frequency, the one with lower id will be the leader.

* Remove unnecessary new statement.

stream will never be a nullptr.

* Move self->dnsCache out of lambda capture.

Member variables are not capture by default, thus, `host` and `service` are not captured. This somehow successfully compile, but throws std::bad_alloc or basic_string::_S_create exceptions when we call `host+":"+service` in dnsCache.remove().

* Revert unintended change.

* Address comments.
2022-06-13 20:24:30 -07:00
Steve Atherton 697aa6cc32 Performance bug fix: reactor.react() is called too often. 2022-06-07 17:53:40 -07:00
Renxuan Wang cd2a575e02
Move the resolve of coordinator hostname from getCoordinatorProtocol() to getClusterProtocolImpl(). (#7245)
* Move the resolve of coordinator hostname from getCoordinatorProtocol() to getClusterProtocolImpl().

* Guard DNS Cache behind a knob.
2022-05-26 09:45:54 -07:00
Renxuan Wang df4e0deb4d
coordinatorsKey should not always store IP addresses. (#7204)
* coordinatorsKey should not storing IP addresses.

Currently, when we do a commit of coordinator change, we are always converting hostnames to IP addresses and store the converted results in coordinatorsKey (\xff/coordinators). This result in ForwardRequest also sending IP addresses, and receivers will update their cluster files with IPs, then we lose the dynamic IP feature.

* Remove the legacy coordinators() function.

* Update async_resolve().

ip::basic_resolver::async_resolve(const query & q, ResolveHandler && handler) is deprecated.

* Clean code format.

* Fix typo.

* Remove SpecifiedQuorumChange and NoQuorumChange.
2022-05-23 11:42:56 -07:00
sfc-gh-tclinkenbeard 225146176d Apply clang-format to fdbcli.actor.cpp and Net2.actor.cpp 2022-05-03 12:13:09 -07:00
sfc-gh-tclinkenbeard 7f05221cfe Removed TLS_DISABLED macro 2022-05-02 22:15:27 -07:00
Ray Jenkins dc9e782ccc
OpenTelemetry Tracing Perf Fixes (#6990) 2022-05-02 14:56:51 -05:00
Sam Gwydir 5403a29ecb
add WolfSSL support (#6682)
remove extraneous include
2022-04-28 16:53:38 -07:00
Renxuan Wang e548c0d604 Add DNS cache. 2022-04-04 15:08:17 -07:00
Renxuan Wang ebe928e7e1 Throw lookup_failed() when hostname resolving fails. 2022-04-04 15:08:17 -07:00
Chaoguang Lin 7d365bd1bb
Remote ikvs debugging (#6465)
* initial structure for remote IKVS server

* moved struct to .h file, added new files to CMakeList

* happy path implementation, connection error when testing

* saved minor local change

* changed tracing to debug

* fixed onClosed and getError being called before init is finished

* fix spawn process bug, now use absolute path

* added server knob to set ikvs process port number

* added server knob for remote/local kv store

* implement simulator remote process spawning

* fixed bug for simulator timeout

* commit all changes

* removed print lines in trace

* added FlowProcess implementation by Markus

* initial debug of FlowProcess, stuck at parent sending OpenKVStoreRequest to child

* temporary fix for process factory throwing segfault on create

* specify public address in command

* change remote kv store knob to false for jenkins build

* made port 0 open random unused port

* change remote store knob to true for benchmark

* set listening port to randomly opened port

* added print lines for jenkins run open kv store timeout debug

* removed most tracing and print lines

* removed tutorial changes

* update handleIOErrors error handling to handle remote-ikvs cases

* Push all debugging changes

* A version where worker bug exists

* A version where restarting tests fail

* Use both the name and the port to determine the child process

* Remove unnecessary update on local address

* Disable remote-kvs for DiskFailureCycle test

* A version where restarting stuck

* A version where most restarting tests green

* Reset connection with child process explicitly

* Remove change on unnecessary files

* Unify flags from _ to -

* fix merging unexpected changes

* fix trac.error to .errorUnsuppressed

* Add license header

* Remove unnecessary header in FlowProcess.actor.cpp

* Fix Windows build

* Fix Windows build, add missing ;

* Fix a stupid bug caused by code dropped by code merging

* Disable remote kvs by default

* Pass the conn_file path to the flow process, though not needed, but the buildNetwork is difficult to tune

* serialization change on readrange

* Update traces

* Refactor the RemoteIKVS interface

* Format files

* Update sim2 interface to not clog connections between parent and child processes in simulation

* Update comments; remove debugging symbols; Add error handling for remote_kvs_cancelled

* Add comments, format files

* Change method name from isBuggifyDisabled to isStableConnection; Decrease(0.1x) latency for stable connections

* Commit the IConnection interface change, forgot in previous commit

* Fix the issue that onClosed request is cancelled by ActorCollection

* Enable the remote kv store knob

* Remove FlowProcess.actor.cpp and move functions to RemoteIKeyValueStore.actor.cpp; Add remote kv store delay to avoid race; Bind the child process to die with parent process

* Fix the bug where one process starts storage server more than once

* Add a please_reboot_remote_kv_store error to restart the storage server worker if remote kvs died abnormally

* Remove unreachable code path and add comments

* Clang format the code

* Fix a simple wait error

* Clang format after merging the main branch

* Testing mixed mode in simulation if remote_kvs knob is enabled, setting the default to false

* Disable remote kvs for PhysicalShardMove which is for RocksDB

* Cleanup #include orders, remove debugging traces

* Revert the reorder in fdbserver.actor.cpp, which fails the gcc build

Co-authored-by: “Lincoln <“lincoln.xiao@snowflake.com”>
2022-03-31 17:08:59 -07:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
sfc-gh-tclinkenbeard a32547cbbe Merge remote-tracking branch 'origin/main' into flow-enhancements 2022-02-15 15:56:42 -08:00
Ray Jenkins dd45805312
Merge branch 'apple:main' into threadname-issue-6064 2022-02-01 17:40:07 -06:00
Renxuan Wang 2ea4146e1f Add resolveTCPEndpointBlocking() to resolve hostnames where async resolving is impossible. 2022-01-28 12:20:41 -08:00
Ray Jenkins ff49bfaeb5 Add thread name for SSLHandshakerThread. 2022-01-24 20:00:00 -06:00
Renxuan Wang 28832a99d6 Address comment. 2022-01-18 14:34:18 -08:00
Renxuan Wang b8bab06e16 Add the functions to set and get mock DNS.
These functions will be used in restarting tests, where mock DNS needs to be saved to and read from files.
2022-01-18 14:34:18 -08:00
sfc-gh-tclinkenbeard 8076806e76 Add ITERATIONS_PER_REACTOR_CHECK 2022-01-01 20:14:57 -08:00
sfc-gh-tclinkenbeard 2ad8955dd3 Cleanup for Flow Enhancements PR 2022-01-01 20:04:00 -08:00
Jon Fu 476aaa73c6 modify benchmark to avoid threadsleep and change runloop to process IO properly 2021-12-16 16:20:22 -05:00
Jon Fu 169838cec3 stub changes for IO queue and attempt at new benchmark 2021-12-16 13:28:38 -05:00
Andrew Noyes def41697bf
Merge pull request #6083 from sfc-gh-tclinkenbeard/remove-temporaries
Avoid creating unnecessary temporary objects
2021-12-06 13:24:56 -08:00