Commit Graph

2292 Commits

Author SHA1 Message Date
Lukas Joswiak cdf98f987d
Revert "Fix global config not triggering changes on server processes" 2021-05-10 12:18:28 -07:00
Lukas Joswiak 71ba6f4501 Switch std::function to AsyncTrigger 2021-05-07 16:05:03 -07:00
Chaoguang Lin 73fdb55f4b Add headers in new files 2021-05-06 16:43:56 -07:00
Chaoguang Lin 24247057ff Add two safeThreadFuture unit tests into ctest 2021-05-06 16:34:13 -07:00
Chaoguang Lin 7da5004690 Merge branch 'master' of github.com:apple/foundationdb into refactor-fdbcli 2021-05-06 15:44:26 -07:00
Steve Atherton 16f9f6e75c Added a test of Arena dependencies and destruction when using the new aligned buffer feature. 2021-05-06 02:05:09 -07:00
Steve Atherton f8a8bf315b Added Arena::allocateAlignedBuffer() to get an aligned memory block owned by an Arena, and ArenaPage uses this. 2021-05-05 15:00:12 -07:00
sfc-gh-tclinkenbeard 3e229457a7 Use default initializer for TraceEvent::errorKind 2021-05-05 14:21:21 -07:00
Andrew Noyes 6bffbdf7e3
Revert "Actually close files in simulation" 2021-05-04 15:38:24 -07:00
sfc-gh-tclinkenbeard e5d6c5ed17 Merge remote-tracking branch 'origin/master' into encrypt-backup-files 2021-05-03 14:46:19 -07:00
Jingyu Zhou 1c92588cca
Merge pull request #4562 from sfc-gh-mpilman/bugfixes/simulator-close-files
Actually close files in simulation
2021-05-03 13:47:44 -07:00
sfc-gh-tclinkenbeard 2bcfbd6816 Move TraceEvent::setErrorKind implementation to cpp file 2021-05-03 11:10:43 -07:00
sfc-gh-tclinkenbeard 13a6c4cf06 Add ErrorKind field to SevError trace events 2021-05-03 11:10:38 -07:00
Lukas Joswiak 8dcd779fc4
Merge branch 'master' into features/actor-lineage 2021-05-02 14:11:42 -07:00
Lukas Joswiak cf4218dfd1 Fixes simulation failures
Fixes the following issues:

1. Use the right index when initializing the WriteOnlySet's vector of
   atomics. Also switch to std::atomic_init to initialize each atomic in
   the vector (cannot default construct the atomics in the vector
   because std::atomic does not have a copy constructor).
2. Add failure check for when items cannot be inserted into the
   WriteOnlySet due to capacity constraints. This situation occurs when
   `copy` is not called on the WriteOnlySet, such as when sampling is
   disabled. The `copy` function is what clears the WriteOnlySet.
3. Remove a global config feature I added to update the ClientDBInfo
   object used by the global config listener function. This needs more
   investigation, but the effect of this change could be that global
   config changes are not correctly recognized on fdbserver processes.
4. Add various ASSERTs to verify data in WriteOnlySet.
2021-05-01 15:26:28 -07:00
sfc-gh-tclinkenbeard 0d9eaa5d6d Add ISingleThreadTransaction.h 2021-04-29 22:31:16 -07:00
sfc-gh-tclinkenbeard 4d3a7ceb44 Added simple DynamicKnobsWorkload 2021-04-29 16:32:30 -07:00
Markus Pilman 5c1279ceb7 make actorlineage inherit from threadsaferefcounted 2021-04-29 15:23:46 -06:00
Markus Pilman e616545949 use std::atomic for threadsaferefcounted 2021-04-29 15:23:33 -06:00
Chaoguang Lin 283b774c2f Merge branch 'master' of github.com:apple/foundationdb into refactor-fdbcli 2021-04-29 12:04:54 -07:00
Markus Pilman 7ab6fedb5b fix refcounting in WriteOnlySet 2021-04-29 11:34:42 -06:00
Russell Sears ccee778753
Merge pull request #4683 from Mythra/mythra/4588/stderr
fdbcli prints error on TLS File not found
2021-04-28 17:13:03 -07:00
Andrew Noyes 904a39e473
Merge pull request #4667 from sfc-gh-ajbeamon/feature-mvc-monitor-protocol-version
Use fewer connections in the multi-version client
2021-04-28 14:13:17 -07:00
A.J. Beamon 8eb78fd317
Merge pull request #4714 from sfc-gh-anoyes/anoyes/more-thread-safety-ctest
Add /fdbclient/multiversionclient/ to ctest, and fix thread safety
2021-04-28 08:54:16 -07:00
Markus Pilman 4fab2ecd30 Merge remote-tracking branch 'origin/master' into features/actor-lineage 2021-04-28 09:20:54 -06:00
A.J. Beamon 16dfb2b2f2 Keep connections older than 6.2 open indefinitely to avoid weird bugs around quickly closing the database. 2021-04-27 15:00:56 -07:00
Markus Pilman 340f012e1a
Merge pull request #4695 from sfc-gh-etschannen/fix-rewrite-bme
rewrote tlog recruitment logic so that it is deterministic
2021-04-27 10:19:25 -06:00
Andrew Noyes 656c9a6c47 Add benchmark and document entities touched 2021-04-26 17:46:35 +00:00
Evan Tschannen 9ca2c3b6c4 instead of increasing the timeout for the lowLatency test, reduce the amount of time a commit takes because of long commit times 2021-04-26 09:50:19 -07:00
sfc-gh-tclinkenbeard 4709c2da5b Added version_already_compacted error 2021-04-23 17:49:57 -07:00
Andrew Noyes 168cba83ef Address review comments 2021-04-23 23:59:51 +00:00
A.J. Beamon a794fca932 Support 5.0 (and earlier) client versions by adding GRV probing for old versions. Update the C bindings implementation of get_server_protocol to convert the ProtocolVersion object into a uint64_t. Rename a misleading protocol version alias. 2021-04-23 15:00:21 -07:00
Andrew Noyes 6fc59379d8 Add /fdbclient/multiversionclient/ to ctest, and fix thread safety 2021-04-23 21:17:41 +00:00
Josh Slocum 4b5bca6761 Minor Redwood comparison optimizations 2021-04-23 18:49:43 +00:00
Chaoguang Lin a05ca1ba7b Disable the two unit tests in simulation as std::thread is not supported in simulation 2021-04-23 10:47:56 -07:00
sfc-gh-tclinkenbeard e3be3bd90c Fix versionedMutationKey ordering bug and strengthen default ConfigurationDatabaseWorkload 2021-04-22 23:00:42 -07:00
Chaoguang Lin 26569273b0 Update safeThreadFutureToFuture and remove the unnecessary wrapper 2021-04-22 19:34:49 -07:00
Chaoguang Lin f0a236c544 Merge branch 'master' of github.com:apple/foundationdb into refactor-fdbcli 2021-04-21 15:36:11 -07:00
Chaoguang Lin eef144bd38 Fix comment typo 2021-04-21 14:13:00 -07:00
A.J. Beamon eaaae2e16d Merge branch master into 'feature-mvc-monitor-protocol-version' 2021-04-20 15:07:02 -07:00
Cynthia f8054b82de fdbcli prints error on TLS File not found 2021-04-19 22:25:19 -06:00
Markus Pilman 51ce278d2d
Merge branch 'features/actor-lineage' into features/actor-lineage-collector 2021-04-19 12:55:29 -06:00
Markus Pilman 09ddcb3bae remove old sample thread 2021-04-19 11:55:35 -06:00
Markus Pilman 7307750e5e Merge remote-tracking branch 'origin/master' into features/actor-lineage 2021-04-19 11:29:52 -06:00
Evan Tschannen a43897c74c
Merge pull request #4653 from sfc-gh-satherton/correctness-fix
Correctness fix:  Prevent writes from being too slow in simulation
2021-04-16 21:57:03 -07:00
Steve Atherton db610355cf Keep simulated disk write delay high until speedUp is set. 2021-04-16 14:19:37 -07:00
Lukas Joswiak 551268b0f2 Add well known endpoint for worker communication 2021-04-15 13:50:50 -07:00
A.J. Beamon b2d6930103 The multi-version client monitors the cluster's protocol version and only activates the client library that can connect. 2021-04-15 11:45:14 -07:00
Steve Atherton 8616575c21 Merge commit 'd51f94f16ee903aa249ddaac0047c88cafac1e89' into unit-test-params 2021-04-14 18:23:00 -07:00
Lukas Joswiak f1415412f1 Add global configuration framework implementation 2021-04-14 10:56:32 -07:00
Steve Atherton 9475b6a5dd Correctness fix, prevent AsyncFileNonDurable from always making file writes take up to 5 seconds. 2021-04-13 20:15:19 -07:00
Chaoguang Lin 3e6215702f Add comments for refactoring 2021-04-13 13:42:19 -07:00
Chaoguang Lin 4fdfca0dc0 Update the thread-safe Future-to-ThreadFuture method, added two TSAN unit tests 2021-04-13 13:21:04 -07:00
Markus Pilman 13e00e8408 made ActorLineage thread safe 2021-04-12 09:43:45 -06:00
Markus Pilman 2efcf8efec Merge remote-tracking branch 'sfc/features/actor-lineage' into features/actor-serialization 2021-04-12 09:34:03 -06:00
Markus Pilman 20d98421af fix compiler errors 2021-04-09 15:16:07 -06:00
Markus Pilman 2064903705 collect and serialize 2021-04-09 14:25:11 -06:00
Steve Atherton 434f41a093 Renamed members of UnitTestParameters to look cleaner. Added getDouble(). Updated more Redwood unit test parameters to be initialized from params. 2021-04-07 18:14:44 -07:00
Lukas Joswiak 83cf965875 Add global variable to fetch each type of sample 2021-04-07 15:38:01 -07:00
Lukas Joswiak d6c4aa67d7 Sample actors waiting on network 2021-04-07 13:27:31 -07:00
Steve Atherton b4e42476b7 Unit test parameters are no longer global, they are accessible via a parameter to the unit test and initialized from otherwise unconsumed test options for the UnitTests workload in the test spec or from the fdbserver command line when using the unittests role. 2021-04-06 02:36:10 -07:00
Markus Pilman 9bcde529f8
Merge pull request #4 from sfc-gh-ljoswiak/features/current-actor
Sample running actor
2021-04-05 11:36:48 -06:00
Steve Atherton eec119e0d0 Added an fdbserver role to run unit tests directly without a cluster or test spec file, and added a unit test parameters concept for passing options into unit tests.
Updated p2p network test to use unit test parameters instead of the environment.
2021-04-04 23:27:34 -07:00
Lukas Joswiak c90be2003f Profile running actor 2021-04-01 10:34:59 -07:00
Markus Pilman 1987682e1e Merge remote-tracking branch 'origin/master' into bugfixes/simulator-close-files 2021-04-01 11:14:28 -06:00
Markus Pilman 7c74686883 Merge remote-tracking branch 'sfc/features/actor-lineage' into features/actor-lineage 2021-04-01 11:02:18 -06:00
Markus Pilman dc35af3760 Merge remote-tracking branch 'origin/master' into features/actor-lineage 2021-04-01 11:01:31 -06:00
Chaoguang Lin 107f66e4e1 Merge branch 'master' of github.com:apple/foundationdb into refactor-fdbcli 2021-03-30 16:35:41 -07:00
Chaoguang Lin 60747a52a5 Add a thread-safe function that converts ThreadFuture into Future 2021-03-30 14:52:56 -07:00
Lukas Joswiak f2d3687110 Print stack 2021-03-29 16:06:26 -07:00
Markus Pilman 41faa51c7a re-add debugRandom 2021-03-29 13:31:16 -06:00
sfc-gh-tclinkenbeard d4191899d9 Add comments for AsyncFileEncrypted changes 2021-03-28 22:14:37 -07:00
sfc-gh-tclinkenbeard 82420e5572 Merge remote-tracking branch 'origin/master' into encrypt-backup-files 2021-03-27 21:02:19 -07:00
Markus Pilman b51e4aa590 handle file renames properly 2021-03-24 19:57:24 -06:00
Lukas Joswiak 2dfd420882 Add sampling profiler thread 2021-03-24 14:52:42 -07:00
A.J. Beamon ed811008ca
Merge pull request #4504 from sfc-gh-nwijetunga/network_busyness
Monitor Network Thread Busyness
2021-03-24 09:44:09 -07:00
Nim Wijetunga ea16d2cccd address pr comments 2021-03-23 23:09:08 +00:00
Nim Wijetunga e174ea0dfd fix comment typo 2021-03-23 22:04:32 +00:00
Nim Wijetunga de1c354b12 address pr comments 2021-03-23 20:56:37 +00:00
Lukas Joswiak 0ec7340a6f Create reference 2021-03-22 10:55:52 -07:00
Lukas Joswiak 5bd79de881 Fix build 2021-03-22 10:52:51 -07:00
Markus Pilman 301daf3269 address review comments 2021-03-22 11:46:16 -06:00
Markus Pilman 61352b9124 use push_back where emplace_back is unnecessary 2021-03-22 11:41:45 -06:00
sfc-gh-tclinkenbeard a0c49234b2 Merge remote-tracking branch 'origin/master' into encrypt-backup-files 2021-03-19 20:47:53 -07:00
Markus Pilman 99ac47e96c documentation 2021-03-19 18:08:09 -06:00
Markus Pilman 8e87627198 Merge remote-tracking branch 'origin/master' into features/actor-lineage 2021-03-19 17:19:02 -06:00
Markus Pilman 995ae34b1e Bugfxies & hack to allow new unit test to run 2021-03-19 17:11:17 -06:00
Markus Pilman bfbd91d4b8 Merge remote-tracking branch 'origin/master' into features/alpset 2021-03-19 10:32:22 -06:00
Markus Pilman 5c1b674815 implemented test 2021-03-19 10:31:58 -06:00
Andrew Noyes 7f4bde74ef Add some documentation 2021-03-18 23:34:44 +00:00
Andrew Noyes 1a7302ff6f Fix a data race
TSAN output:

3: WARNING: ThreadSanitizer: data race (pid=63)
3:   Read of size 1 at 0x7b6800000970 by thread T1:
3:     #0 N2::Net2::onMainThread(Promise<Void>&&, TaskPriority) /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:1734:6 (libfdb_c.so+0xca3d9e)
3:     #1 void onMainThreadVoid<ThreadSingleAssignmentVarBase::cancel()::'lambda'()>(ThreadSingleAssignmentVarBase::cancel()::'lambda'(), Error*, TaskPriority) /home/anoyes/workspace/foundationdb/flow/ThreadHelper.actor.h:47:13 (libfdb_c.so+0x3fe73a)
3:     #2 ThreadSingleAssignmentVarBase::cancel() /home/anoyes/workspace/foundationdb/flow/ThreadHelper.actor.h:317:3 (libfdb_c.so+0x3fe290)
3:     #3 fdb_future_cancel /home/anoyes/workspace/foundationdb/bindings/c/fdb_c.cpp:141:96 (libfdb_c.so+0x3f7896)
3:     #4 DLThreadSingleAssignmentVar<long>::cancel() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionAssignmentVars.h:175:4 (libfdb_c.so+0x45bd4d)
3:     #5 MapSingleAssignmentVar<long, Void>::cancel() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionAssignmentVars.h:241:20 (libfdb_c.so+0x475ba7)
3:     #6 FlatMapSingleAssignmentVar<Void, Void>::cancel() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionAssignmentVars.h:301:27 (libfdb_c.so+0x4742ce)
3:     #7 ThreadFuture<Void>::cancel() /home/anoyes/workspace/foundationdb/flow/ThreadHelper.actor.h:444:32 (libfdb_c.so+0x441237)
3:     #8 MultiVersionDatabase::Connector::cancel() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:969:20 (libfdb_c.so+0x441237)
3:     #9 MultiVersionDatabase::DatabaseState::cancelConnections()::$_22::operator()() const /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1082:11 (libfdb_c.so+0x441237)
3:     #10 (anonymous namespace)::DoOnMainThreadVoidActorState<MultiVersionDatabase::DatabaseState::cancelConnections()::$_22, (anonymous namespace)::DoOnMainThreadVoidActor<MultiVersionDatabase::DatabaseState::cancelConnections()::$_22> >::a_body1cont1(Void const&, int) /home/anoyes/workspace/foundationdb/flow/ThreadHelper.actor.h:559:2 (libfdb_c.so+0x441237)
3:     #11 (anonymous namespace)::DoOnMainThreadVoidActorState<MultiVersionDatabase::DatabaseState::cancelConnections()::$_22, (anonymous namespace)::DoOnMainThreadVoidActor<MultiVersionDatabase::DatabaseState::cancelConnections()::$_22> >::a_body1when1(Void const&, int) /home/anoyes/build/foundationdb-tsan/flow/ThreadHelper.actor.g.h:992:15 (libfdb_c.so+0x441237)
3:     #12 (anonymous namespace)::DoOnMainThreadVoidActorState<MultiVersionDatabase::DatabaseState::cancelConnections()::$_22, (anonymous namespace)::DoOnMainThreadVoidActor<MultiVersionDatabase::DatabaseState::cancelConnections()::$_22> >::a_callback_fire(ActorCallback<(anonymous namespace)::DoOnMainThreadVoidActor<MultiVersionDatabase::DatabaseState::cancelConnections()::$_22>, 0, Void>*, Void const&) /home/anoyes/build/foundationdb-tsan/flow/ThreadHelper.actor.g.h:1013:4 (libfdb_c.so+0x440dff)
3:     #13 ActorCallback<(anonymous namespace)::DoOnMainThreadVoidActor<MultiVersionDatabase::DatabaseState::cancelConnections()::$_22>, 0, Void>::fire(Void const&) /home/anoyes/workspace/foundationdb/flow/flow.h:1016:78 (libfdb_c.so+0x440dff)
3:     #14 void SAV<Void>::send<Void>(Void&&) /home/anoyes/workspace/foundationdb/flow/flow.h:465:23 (libfdb_c.so+0x4086b2)
3:     #15 void Promise<Void>::send<Void>(Void&&) const /home/anoyes/workspace/foundationdb/flow/flow.h:782:8 (libfdb_c.so+0xcc079e)
3:     #16 N2::PromiseTask::operator()() /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:1157:11 (libfdb_c.so+0xcc079e)
3:     #17 N2::Net2::run() /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:1493:5 (libfdb_c.so+0xca1640)
3:     #18 runNetwork() /home/anoyes/workspace/foundationdb/fdbclient/NativeAPI.actor.cpp:1774:13 (libfdb_c.so+0x499eb4)
3:     #19 ThreadSafeApi::runNetwork() /home/anoyes/workspace/foundationdb/fdbclient/ThreadSafeTransaction.cpp:429:3 (libfdb_c.so+0xb666cc)
3:     #20 MultiVersionApi::runNetwork() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1521:20 (libfdb_c.so+0x41e9dd)
3:     #21 fdb_run_network /home/anoyes/workspace/foundationdb/bindings/c/fdb_c.cpp:129:45 (libfdb_c.so+0x3f7645)
3:     #22 decltype(std::__1::forward<int (*)()>(fp)()) std::__1::__invoke<int (*)()>(int (*&&)()) /usr/local/bin/../include/c++/v1/type_traits:3539:173 (fdb_c_unit_tests+0x384856)
3:     #23 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/thread:273:5 (fdb_c_unit_tests+0x384856)
3:     #24 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()> >(void*) /usr/local/bin/../include/c++/v1/thread:284:5 (fdb_c_unit_tests+0x384856)
3:
3:   Previous write of size 1 at 0x7b6800000970 by thread T2:
3:     #0 N2::Net2::stopImmediately() /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:253:11 (libfdb_c.so+0xca936b)
3:     #1 N2::Net2::stop()::'lambda'()::operator()() const /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:171:36 (libfdb_c.so+0xca936b)
3:     #2 (anonymous namespace)::DoOnMainThreadVoidActorState<N2::Net2::stop()::'lambda'(), (anonymous namespace)::DoOnMainThreadVoidActor<N2::Net2::stop()::'lambda'()> >::a_body1cont1(Void const&, int) /home/anoyes/workspace/foundationdb/flow/ThreadHelper.actor.h:559:2 (libfdb_c.so+0xca936b)
3:     #3 (anonymous namespace)::DoOnMainThreadVoidActorState<N2::Net2::stop()::'lambda'(), (anonymous namespace)::DoOnMainThreadVoidActor<N2::Net2::stop()::'lambda'()> >::a_body1when1(Void const&, int) /home/anoyes/build/foundationdb-tsan/flow/ThreadHelper.actor.g.h:992:15 (libfdb_c.so+0xca936b)
3:     #4 (anonymous namespace)::DoOnMainThreadVoidActorState<N2::Net2::stop()::'lambda'(), (anonymous namespace)::DoOnMainThreadVoidActor<N2::Net2::stop()::'lambda'()> >::a_callback_fire(ActorCallback<(anonymous namespace)::DoOnMainThreadVoidActor<N2::Net2::stop()::'lambda'()>, 0, Void>*, Void const&) /home/anoyes/build/foundationdb-tsan/flow/ThreadHelper.actor.g.h:1013:4 (libfdb_c.so+0xca9128)
3:     #5 ActorCallback<(anonymous namespace)::DoOnMainThreadVoidActor<N2::Net2::stop()::'lambda'()>, 0, Void>::fire(Void const&) /home/anoyes/workspace/foundationdb/flow/flow.h:1016:78 (libfdb_c.so+0xca9128)
3:     #6 void SAV<Void>::send<Void>(Void&&) /home/anoyes/workspace/foundationdb/flow/flow.h:465:23 (libfdb_c.so+0x4086b2)
3:     #7 void Promise<Void>::send<Void>(Void&&) const /home/anoyes/workspace/foundationdb/flow/flow.h:782:8 (libfdb_c.so+0xcc079e)
3:     #8 N2::PromiseTask::operator()() /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:1157:11 (libfdb_c.so+0xcc079e)
3:     #9 N2::Net2::run() /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:1493:5 (libfdb_c.so+0xca1640)
3:     #10 runNetwork() /home/anoyes/workspace/foundationdb/fdbclient/NativeAPI.actor.cpp:1774:13 (libfdb_c.so+0x499eb4)
3:     #11 ThreadSafeApi::runNetwork() /home/anoyes/workspace/foundationdb/fdbclient/ThreadSafeTransaction.cpp:429:3 (libfdb_c.so+0xb666cc)
3:     #12 MultiVersionApi::runNetwork() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1521:20 (libfdb_c.so+0x41e9dd)
3:     #13 fdb_run_network /home/anoyes/workspace/foundationdb/bindings/c/fdb_c.cpp:129:45 (libfdb_c.so+0x3f7645)
3:     #14 DLApi::runNetwork() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:492:11 (libfdb_c.so+0x40dd18)
3:     #15 runNetworkThread(void*) /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1495:30 (libfdb_c.so+0x41e895)
3:
3:   Location is heap block of size 1304 at 0x7b6800000600 allocated by main thread:
3:     #0 operator new(unsigned long) <null> (fdb_c_unit_tests+0x31694e)
3:     #1 newNet2(TLSConfig const&, bool, bool) /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:1969:16 (libfdb_c.so+0xca607d)
3:     #2 setupNetwork(unsigned long, bool) /home/anoyes/workspace/foundationdb/fdbclient/NativeAPI.actor.cpp:1754:14 (libfdb_c.so+0x499c53)
3:     #3 ThreadSafeApi::setupNetwork() /home/anoyes/workspace/foundationdb/fdbclient/ThreadSafeTransaction.cpp:423:2 (libfdb_c.so+0xb66675)
3:     #4 MultiVersionApi::setupNetwork() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1462:21 (libfdb_c.so+0x41dc42)
3:     #5 fdb_setup_network_impl() /home/anoyes/workspace/foundationdb/bindings/c/fdb_c.cpp:116:45 (libfdb_c.so+0x3f7525)
3:     #6 DLApi::setupNetwork() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:488:15 (libfdb_c.so+0x40dc88)
3:     #7 MultiVersionApi::setupNetwork()::$_27::operator()(Reference<ClientInfo>) const /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1481:17 (libfdb_c.so+0x442cd4)
3:     #8 decltype(std::__1::forward<MultiVersionApi::setupNetwork()::$_27&>(fp)(std::__1::forward<Reference<ClientInfo> >(fp0))) std::__1::__invoke<MultiVersionApi::setupNetwork()::$_27&, Reference<ClientInfo> >(MultiVersionApi::setupNetwork()::$_27&, Reference<ClientInfo>&&) /usr/local/bin/../include/c++/v1/type_traits:3539:173 (libfdb_c.so+0x442cd4)
3:     #9 void std::__1::__invoke_void_return_wrapper<void>::__call<MultiVersionApi::setupNetwork()::$_27&, Reference<ClientInfo> >(MultiVersionApi::setupNetwork()::$_27&, Reference<ClientInfo>&&) /usr/local/bin/../include/c++/v1/__functional_base:348:9 (libfdb_c.so+0x442cd4)
3:     #10 std::__1::__function::__alloc_func<MultiVersionApi::setupNetwork()::$_27, std::__1::allocator<MultiVersionApi::setupNetwork()::$_27>, void (Reference<ClientInfo>)>::operator()(Reference<ClientInfo>&&) /usr/local/bin/../include/c++/v1/functional:1540:16 (libfdb_c.so+0x442cd4)
3:     #11 std::__1::__function::__func<MultiVersionApi::setupNetwork()::$_27, std::__1::allocator<MultiVersionApi::setupNetwork()::$_27>, void (Reference<ClientInfo>)>::operator()(Reference<ClientInfo>&&) /usr/local/bin/../include/c++/v1/functional:1714:12 (libfdb_c.so+0x442cd4)
3:     #12 std::__1::__function::__value_func<void (Reference<ClientInfo>)>::operator()(Reference<ClientInfo>&&) const /usr/local/bin/../include/c++/v1/functional:1867:16 (libfdb_c.so+0x4153ed)
3:     #13 std::__1::function<void (Reference<ClientInfo>)>::operator()(Reference<ClientInfo>) const /usr/local/bin/../include/c++/v1/functional:2473:12 (libfdb_c.so+0x4153ed)
3:     #14 MultiVersionApi::runOnExternalClients(int, std::__1::function<void (Reference<ClientInfo>)>, bool) /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1117:5 (libfdb_c.so+0x4153ed)
3:     #15 MultiVersionApi::runOnExternalClientsAllThreads(std::__1::function<void (Reference<ClientInfo>)>, bool) /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1102:3 (libfdb_c.so+0x41992c)
3:     #16 MultiVersionApi::setupNetwork() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1475:3 (libfdb_c.so+0x41dd2a)
3:     #17 fdb_setup_network_impl() /home/anoyes/workspace/foundationdb/bindings/c/fdb_c.cpp:116:45 (libfdb_c.so+0x3f7525)
3:     #18 main /home/anoyes/workspace/foundationdb/bindings/c/test/unit/unit_tests.cpp:2147:12 (fdb_c_unit_tests+0x35f6b2)
3:
3:   Thread T1 (tid=65, running) created by main thread at:
3:     #0 pthread_create <null> (fdb_c_unit_tests+0x29df32)
3:     #1 std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /usr/local/bin/../include/c++/v1/__threading_support:394:10 (fdb_c_unit_tests+0x372da6)
3:     #2 std::__1:🧵:thread<int (*)(), void>(int (*&&)()) /usr/local/bin/../include/c++/v1/thread:300:16 (fdb_c_unit_tests+0x372da6)
3:     #3 main /home/anoyes/workspace/foundationdb/bindings/c/test/unit/unit_tests.cpp:2148:14 (fdb_c_unit_tests+0x35f6dd)
3:
3:   Thread T2 (tid=66, finished) created by thread T1 at:
3:     #0 pthread_create <null> (fdb_c_unit_tests+0x29df32)
3:     #1 startThread(void* (*)(void*), void*, int, char const*) /home/anoyes/workspace/foundationdb/flow/Platform.actor.cpp:2745:2 (libfdb_c.so+0xd08d74)
3:     #2 N2::Net2::startThread(void* (*)(void*), void*) /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:1749:9 (libfdb_c.so+0xca42a4)
3:     #3 MultiVersionApi::runNetwork()::$_28::operator()(Reference<ClientInfo>) const /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1516:34 (libfdb_c.so+0x443106)
3:     #4 decltype(std::__1::forward<MultiVersionApi::runNetwork()::$_28&>(fp)(std::__1::forward<Reference<ClientInfo> >(fp0))) std::__1::__invoke<MultiVersionApi::runNetwork()::$_28&, Reference<ClientInfo> >(MultiVersionApi::runNetwork()::$_28&, Reference<ClientInfo>&&) /usr/local/bin/../include/c++/v1/type_traits:3539:173 (libfdb_c.so+0x443106)
3:     #5 void std::__1::__invoke_void_return_wrapper<void>::__call<MultiVersionApi::runNetwork()::$_28&, Reference<ClientInfo> >(MultiVersionApi::runNetwork()::$_28&, Reference<ClientInfo>&&) /usr/local/bin/../include/c++/v1/__functional_base:348:9 (libfdb_c.so+0x443106)
3:     #6 std::__1::__function::__alloc_func<MultiVersionApi::runNetwork()::$_28, std::__1::allocator<MultiVersionApi::runNetwork()::$_28>, void (Reference<ClientInfo>)>::operator()(Reference<ClientInfo>&&) /usr/local/bin/../include/c++/v1/functional:1540:16 (libfdb_c.so+0x443106)
3:     #7 std::__1::__function::__func<MultiVersionApi::runNetwork()::$_28, std::__1::allocator<MultiVersionApi::runNetwork()::$_28>, void (Reference<ClientInfo>)>::operator()(Reference<ClientInfo>&&) /usr/local/bin/../include/c++/v1/functional:1714:12 (libfdb_c.so+0x443106)
3:     #8 std::__1::__function::__value_func<void (Reference<ClientInfo>)>::operator()(Reference<ClientInfo>&&) const /usr/local/bin/../include/c++/v1/functional:1867:16 (libfdb_c.so+0x4153ed)
3:     #9 std::__1::function<void (Reference<ClientInfo>)>::operator()(Reference<ClientInfo>) const /usr/local/bin/../include/c++/v1/functional:2473:12 (libfdb_c.so+0x4153ed)
3:     #10 MultiVersionApi::runOnExternalClients(int, std::__1::function<void (Reference<ClientInfo>)>, bool) /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1117:5 (libfdb_c.so+0x4153ed)
3:     #11 MultiVersionApi::runOnExternalClientsAllThreads(std::__1::function<void (Reference<ClientInfo>)>, bool) /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1102:3 (libfdb_c.so+0x41992c)
3:     #12 MultiVersionApi::runNetwork() /home/anoyes/workspace/foundationdb/fdbclient/MultiVersionTransaction.actor.cpp:1514:3 (libfdb_c.so+0x41ea53)
3:     #13 fdb_run_network /home/anoyes/workspace/foundationdb/bindings/c/fdb_c.cpp:129:45 (libfdb_c.so+0x3f7645)
3:     #14 decltype(std::__1::forward<int (*)()>(fp)()) std::__1::__invoke<int (*)()>(int (*&&)()) /usr/local/bin/../include/c++/v1/type_traits:3539:173 (fdb_c_unit_tests+0x384856)
3:     #15 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/thread:273:5 (fdb_c_unit_tests+0x384856)
3:     #16 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, int (*)()> >(void*) /usr/local/bin/../include/c++/v1/thread:284:5 (fdb_c_unit_tests+0x384856)
3:
3: SUMMARY: ThreadSanitizer: data race /home/anoyes/workspace/foundationdb/flow/Net2.actor.cpp:1734:6 in N2::Net2::onMainThread(Promise<Void>&&, TaskPriority)
3: ==================
3: ThreadSanitizer: reported 1 warnings
2021-03-18 23:23:50 +00:00
Evan Tschannen f2c82f4fc6 too much run loop batching slows down the speed of transactions enough that if cannot commit transactions fast enough when either TRANSACTION_LIFE_VERSIONS is buggified to be small 2021-03-18 12:06:28 -07:00
Nim Wijetunga 73571ad52e Merge branch 'master' of github.com:sfc-gh-nwijetunga/foundationdb into network_busyness
* 'master' of github.com:sfc-gh-nwijetunga/foundationdb: (55 commits)
  Port lost changes from #4004
  Fix markdown
  Factor our refreshTransaction gadget
  Align FastAllocator memory to 4096 for size >= 4096
  Fix heap-use-after-free's
  Disable lsan in joshua
  Manually apply changes made to BackupContainer.actor.cpp on release-6.3 to the new files where that code is located.
  add release branch instructions
  Fix PR number in release note link
  fixed a simulation bug where a process on an unreliable machine would be considered reliable by the simulator
  Add tini as PID 1 to docker image (#4363)
  Fix typo
  Fix broken_promise bug
  Explain FastAllocator<4096> special case
  Inform LSAN that we have unaligned pointers
  apply clang-format to flow/Platform.h
  Fix typos
  Address review comments
  Change the macro that forbids exit() calls to be a static assertion
  Add versionstamp serialization information
  ...
2021-03-18 18:58:43 +00:00
Markus Pilman 4f1b807e1f assert object alignment 2021-03-17 16:01:23 -06:00
Markus Pilman f6c7aa6ac7 fixed typo 2021-03-17 15:50:29 -06:00
Markus Pilman 9812a49058 use consume_all to clean up after copy 2021-03-17 15:40:19 -06:00
Markus Pilman a8c7a798f2 First prototype of actorlineageset 2021-03-17 15:34:20 -06:00
Nim Wijetunga ea922aa648 address pr comments 2021-03-17 18:16:18 +00:00
Markus Pilman eb036b7b02 Merge remote-tracking branch 'origin/master' into features/actor-lineage 2021-03-17 11:59:54 -06:00
Evan Tschannen 8f94167bc9 lowered the buggified time batching 2021-03-17 09:57:11 -07:00
Evan Tschannen a3c48772e1 Merge branch 'master' into feature-sim-time-batching 2021-03-17 09:55:07 -07:00
Evan Tschannen 412609cd62 increased max batching time 2021-03-16 19:09:12 -07:00
Nim Wijetunga 822cbf895d address pr comments 2021-03-16 23:29:02 +00:00
Nim Wijetunga bf5f83d2bf address pr comments 2021-03-16 21:49:36 +00:00
Nim Wijetunga 5f079904d4 add unit test 2021-03-16 18:30:39 +00:00
Nim Wijetunga 17f9da6962 remove traces 2021-03-16 16:47:50 +00:00
Trevor Clinkenbeard ffb8e27f43
Merge pull request #4461 from sfc-gh-anoyes/anoyes/fix-thread-leak
Fix several leaks
2021-03-15 22:03:45 -07:00
Nim Wijetunga cc6e395664 atomic network busyness 2021-03-16 02:16:43 +00:00
Andrew Noyes 5f64a0942f Align FastAllocator memory to 4096 for size >= 4096 2021-03-16 00:44:35 +00:00
Nim Wijetunga f0d79b3d86 Inital implementation of network busyness 2021-03-15 23:23:56 +00:00
Evan Tschannen 13242d8b35 the sim2 runloop now updates time in batches so that multiple tasks can execute with the same now() 2021-03-15 12:33:43 -07:00
Trevor Clinkenbeard a82d6cf7d4
Merge pull request #4486 from johscheuer/fix-typo
Fix typo
2021-03-14 15:38:48 -07:00
Johannes M. Scheuermann 4aa98f5035 Fix typo 2021-03-14 19:30:35 +00:00
Andrew Noyes d09603b6b3 Explain FastAllocator<4096> special case 2021-03-13 00:36:29 +00:00
Andrew Noyes a63dc3cfd6 Inform LSAN that we have unaligned pointers 2021-03-13 00:33:47 +00:00
FDB Formatster 7867cd454e apply clang-format to flow/Platform.h 2021-03-12 15:16:33 -08:00
A.J. Beamon 74f427d317 Change the macro that forbids exit() calls to be a static assertion 2021-03-12 14:47:19 -08:00
Markus Pilman 4bee976b37 completely ignore messages to unknown endpoint 2021-03-12 09:35:53 -07:00
Andrew Noyes a76c7b9754 Fix several memory leaks and a thread leak 2021-03-11 23:58:32 +00:00
Vishesh Yadav d7252da951 clang-format: Fix the TEST() macros which require comments in line 2021-03-10 16:50:53 -08:00
Vishesh Yadav 2cd3f45fd6 Merge remote-tracking branch 'apple/release-6.3' into master-format-final
release-6.3 was recently merged, and there were two PRs which were
merged in between and got those changes in here. Hence, since all the
changes were in, discarded the incoming changes and accepted all
current.
2021-03-10 16:50:44 -08:00
FDB Formatster df90cc89de apply clang-format to *.c, *.cpp, *.h, *.hpp files 2021-03-10 10:18:07 -08:00
Vishesh Yadav 2bb4f2e59f Merge branch 'release-6.3-pre-format' into master-format
This merges release-6.3 branch right before it was fully formatted.
There were quite a few conflicts that are resolved here. CoroFlow had
a check for OOM errors introduced in 6.3, but didn't seem applicable in
the new implmentation which seems to use boost.
2021-03-10 09:37:41 -08:00
Russell Sears 36bf68ed2e
Merge branch 'release-6.3' into clang_format_the_world/6.3 2021-03-08 14:40:52 -08:00
Andrew Noyes a7ccb6669d Change Net2::connect to only call initTLS for tls 2021-03-08 16:27:04 +00:00
FDB Formatster 8a8c488ede apply clang-format to *.c, *.cpp, *.h, *.hpp files 2021-03-05 18:13:38 -06:00
Andrew Noyes 23b85256ae Change default to 64, and only start threads if using TLS 2021-03-04 20:52:46 +00:00
Andrew Noyes cc7fed6ed4 Change default for TLS_SERVER_HANDSHAKE_THREADS to 0 2021-03-04 20:52:44 +00:00
Andrew Noyes 79cec09255 Apply clang-tidy's performance-inefficient-vector-operation fix
I ran this command in my build directory after compiling with
OPEN_FOR_IDE. It took a few small tweaks to get it to compile, which is
outside the scope of this commit.

    $ python run-clang-tidy.py -j $(nproc) -checks='-*,performance-inefficient-vector-operation' -fix
2021-03-04 03:58:25 +00:00
Markus Pilman 4806299821
Merge pull request #4382 from sfc-gh-nwijetunga/optimize-watches-server
Optimize Watches Server
2021-03-02 14:25:21 -07:00
Markus Pilman a03b5f5426
Merge pull request #4349 from sfc-gh-ljoswiak/fixes/log-file-tracing
Stop running log file tracer in simulation
2021-03-02 14:00:00 -07:00
Lukas Joswiak d676cb32e1 Separate tracers that will run in simulation from those that won't 2021-03-02 10:35:37 -08:00
Daniel Smith 4adce4eb83 Limit named thread support to Linux and add a comment documenting that. 2021-03-01 20:59:25 +00:00
Daniel Smith 179dea5a1b Name the RocksDB background threads 2021-03-01 20:35:55 +00:00
Markus Pilman 2d84e5e1f1
Merge pull request #4254 from sfc-gh-tclinkenbeard/capture-forked-process-output
Capture forked process output in traces
2021-02-24 14:34:24 -07:00
Nim Wijetunga 6c65b0e8e0 inital commit 2021-02-22 20:58:59 -05:00
Andrew Noyes 4a6ec52e0d
Merge pull request #4242 from sfc-gh-mpilman/features/coroutine2
Replace libcoro with boost::coroutine2
2021-02-17 15:45:19 -08:00
Lukas Joswiak 7a40190500 Don't test log file tracing in simulation 2021-02-17 09:50:46 -08:00
Markus Pilman f514194aec Merge remote-tracking branch 'origin/master' into features/coroutine2 2021-02-17 09:03:07 -07:00
Lukas Joswiak 4657db6705 Add comment warning against relying on TracerType order 2021-02-16 19:09:24 -08:00
Lukas Joswiak f9c8674bc9 Use auto 2021-02-16 16:48:51 -08:00
Lukas Joswiak 30d2d9ff82 Remove async tracer 2021-02-16 16:36:33 -08:00
Steve Atherton 76a4293d1d Merge branch 'release-6.3'
# Conflicts:
#	fdbserver/fdbserver.actor.cpp
2021-02-15 02:13:06 -08:00
Steve Atherton f4c9b88908 Merge branch 'release-6.2' into release-6.3
# Conflicts:
#	cmake/CompileBoost.cmake
#	fdbserver/DataDistribution.actor.cpp
#	fdbserver/fdbserver.actor.cpp
#	flow/Error.cpp
2021-02-15 02:05:03 -08:00
Andrew Noyes dd0e692f2a Log std::exception::what when creating an unknown_error
unknown_error usually gets created when an exception of some type other
than `Error` is thrown. If the current exception is a std::exception
(and it usually is), we can log more information about it - which is
(incredibly!) helpful for figuring out what the heck happened.
2021-02-13 17:42:11 +00:00
Andrew Noyes 4e184fe236 Fix memory errors 2021-02-11 02:58:21 +00:00
Andrew Noyes 53739c9da5
Merge pull request #4241 from sfc-gh-clin/add-c-function-snapshot
Add c function for snapshot
2021-02-09 15:49:20 -08:00
Russell Sears a56184c0b7 log TLSVerifyFailure as a warning (severity 20) since it generally indicates a miconfiguration or important network problem 2021-02-08 22:01:48 +00:00
A.J. Beamon 601a548cfe Revert "Move new config to config.h.cmake"
This reverts commit 2951ebb4e7.
2021-02-08 14:01:02 -08:00
A.J. Beamon 2951ebb4e7 Move new config to config.h.cmake 2021-02-08 13:34:17 -08:00
A.J. Beamon aaf0a9aa7b Merge branch 'release-6.3' into merge-release-6.3-into-master
# Conflicts:
#	build/docker-compose.yaml
#	cmake/ConfigureCompiler.cmake
#	fdbclient/FileBackupAgent.actor.cpp
#	fdbrpc/AsyncFileCached.actor.h
#	fdbrpc/IAsyncFile.h
#	fdbrpc/IRateControl.h
#	fdbrpc/simulator.h
#	fdbserver/KeyValueStoreSQLite.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbservice/ServiceBase.cpp
2021-02-08 12:58:34 -08:00
A.J. Beamon 449e4b98b4 Attempt to fix a build warning that's now blocking the compile due to werror. 2021-02-08 12:06:51 -08:00
A.J. Beamon 67e783acf8 Merge branch 'release-6.2' into merge-release-6.2-into-release-6.3
# Conflicts:
#	cmake/CompileBoost.cmake
#	cmake/FDBComponents.cmake
#	fdbrpc/AsyncFileCached.actor.h
#	fdbrpc/simulator.h
#	fdbserver/KeyValueStoreSQLite.actor.cpp
#	fdbserver/Knobs.cpp
#	fdbserver/Knobs.h
#	fdbserver/storageserver.actor.cpp
#	flow/Knobs.h
#	flow/network.h
2021-02-08 09:20:28 -08:00
Chaoguang Lin a85458dd1e Merge branch 'master' of github.com:apple/foundationdb into add-c-function-snapshot 2021-02-05 14:14:52 -08:00
Chaoguang Lin 97925de9d4 Update fdb_database_create_snapshot to take a UID as input 2021-02-05 14:14:38 -08:00
A.J. Beamon 5e80e16802
Revert "CDs reject requests from external cluster or requests need to be forwarded based on an old forward request." 2021-02-05 13:05:33 -08:00
sfc-gh-tclinkenbeard dea735a322 Fix Windows build 2021-02-05 00:08:29 -08:00
sfc-gh-tclinkenbeard b601a73a25 Clean up all cipher contexts in crashHandler 2021-02-04 23:47:34 -08:00
Evan Tschannen 8091d8179d
Merge pull request #4252 from dongxinEric/feature/4251/cds-reject-requests-from-other-cluster
CDs reject requests from external cluster or requests need to be forwarded based on an old forward request.
2021-02-04 17:33:07 -08:00
Andrew Noyes db82da5bc2 Only link to jemalloc in fdbserver
Including jemalloc in libfdb_c seems to be problematic, since it's
apparently not recommended to dlopen jemalloc:
https://github.com/jemalloc/jemalloc/issues/937#issuecomment-311242450
2021-02-05 01:30:31 +00:00
Xin Dong c017b51d66 Fix typo 2021-02-04 13:35:40 -08:00
Andrew Noyes e4a55908ff
Merge pull request #4222 from sfc-gh-mpilman/features/jemalloc
Replace standard malloc with jemalloc
2021-02-04 12:47:29 -08:00
Markus Pilman 7b489da13b introduce USE_JEMALLOC and some refactoring 2021-02-04 11:53:28 -07:00
sfc-gh-tclinkenbeard 8470a326a2 Clean up StreamCipher::Key::globalKey in crashHandler 2021-02-03 18:49:51 -08:00
sfc-gh-tclinkenbeard b5ed7dcdf8 Make StreamCipher::Key constructor effectively private 2021-02-03 18:21:04 -08:00
sfc-gh-tclinkenbeard 7c0e331e07 Disable copying StreamCipher::Key 2021-02-03 18:13:39 -08:00
Meng Xu 1db4b73215
Merge pull request #4243 from sfc-gh-etschannen/feature-storage-io-timeout
throw an io_timeout if it takes more than 2 minutes to commit on the storage server
2021-02-03 18:10:16 -08:00
Meng Xu 675e567305
Merge pull request #4218 from sfc-gh-etschannen/feature-low-priority-reads-6.2
Reduce read priority when durability lag is high
2021-02-03 18:09:39 -08:00
sfc-gh-tclinkenbeard 5be4df6f84 Move global key into StreamCipher 2021-02-03 17:39:59 -08:00
Evan Tschannen 068647ba1b updated naming and comments 2021-02-03 14:24:39 -08:00
Evan Tschannen c4b622e7b8 added a comment and updated a variable name 2021-02-03 13:58:04 -08:00
Evan Tschannen a0a71ff7f9 simplified the algorithm based on testing 2021-02-02 17:53:07 -08:00
Steve Atherton 6f0e2dcdd0
Merge pull request #4268 from sfc-gh-clin/add-knob-for-IO-size
Add option to define I/O size
2021-02-02 16:22:24 -08:00
Chaoguang Lin 593165679a Remove unnecessary header 2021-02-02 12:42:46 -08:00
Chaoguang Lin 17671604c0 Add option to define I/O size 2021-02-02 12:39:05 -08:00
Jon Fu 090eee33e7 rearrange include ordering 2021-02-02 14:58:51 -05:00
Jon Fu d10b452758 include <errno.h> instead of <sys/errno.h> 2021-02-02 14:55:02 -05:00
Jon Fu 761d9991a3 change to SevWarn instead of SevError 2021-02-02 14:47:19 -05:00
Jon Fu 5a28d783b0 add readable description of error to trace event 2021-02-02 14:34:56 -05:00
Jon Fu 5fa7c73810 add tracing to fileopen failure before throwing error for more info 2021-02-02 14:30:12 -05:00
Markus Pilman 534fa45291 Merge branch 'master' of github.com:apple/foundationdb into features/coroutine2 2021-02-01 16:31:16 -07:00
Markus Pilman c25a3f11c3
Merge pull request #4259 from sfc-gh-satherton/ratelimit-6.2
Backport AFCache write rate control to 6.2
2021-01-31 20:55:16 -07:00
sfc-gh-tclinkenbeard 04b06c0c3c Merge remote-tracking branch 'origin/master' into capture-forked-process-output 2021-01-31 16:14:38 -08:00
Markus Pilman e4b953ab38
Merge pull request #4229 from sfc-gh-satherton/afcache-write-limit
AsyncFileCached RateControl support and bug fixes
2021-01-31 15:13:03 -07:00
Steve Atherton 97781bde76 Removed stray knob added unintentionally in a merge conflict. 2021-01-30 18:35:16 -08:00
Steve Atherton b0fd6bfbcf More bug fixes in SQLite file handle lifetimes after storage server restart (including simulated process reboot) when a write rate limit is used. Added several debug trace events, renamed some VFSAsync* events to be more consistent. 2021-01-30 18:18:16 -08:00
Steve Atherton 52c2695f59 Bug fixes involving use of AFCached rate control in simulation. 2021-01-30 18:13:44 -08:00
Steve Atherton e56fe02a25 Merge branch 'add-throttling-on-AsyncfileCached' of github.com:sfc-gh-clin/foundationdb into afcache-write-limit 2021-01-30 18:11:54 -08:00
Steve Atherton 14af5857ed Another refactor of SQLite injected error handling to address failures of the previous attempt. Some code cleanup. Added documentation of injected error handling works and why. 2021-01-30 18:06:05 -08:00
Steve Atherton e72d486215 Refactored how injected fault state is managed to make debugging injected fault handling easier. Moved injected error tracking for open() into an INetwork global since it should be unique per simulated process. Fixed bug where injected error in WAL file was not recognized as injected because it occurred before SQLiteDB's vfsWAL pointer was initialized. Simplified logic in SQLiteDB. Added comments. 2021-01-30 18:06:04 -08:00
Christophe Chevalier ff0667b28d winbuild: define BOOST_USE_WINDOWS_H, WIN32_LEAN_AND_MEAN and NOMINMAX to fix various macro expansions issues
- NOMINMAX to fix confusion between std::max(..) and max(..) macro redefinition
- BOOST_USE_WINDOWS_H to prevent boost from redefining win32 APIs
- WIN32_LEAN_AND_MEAN to fix include ordering issues with winsock.h
2021-01-30 15:15:02 +01:00
Xin Dong 9f837a3b84 Coordinators will not serve following requests after this change:
- A request that needs to be forwarded after the forward was set for a long(configured via Knobs) time.
- A request that has a different connection string than the CD's local connection string.

Simulation has been made to work with this change.
2021-01-29 13:24:45 -08:00
Evan Tschannen 4f3da7c93b also kill tlogs if a commit takes too long 2021-01-29 12:18:06 -08:00
sfc-gh-tclinkenbeard acac02587d Trace output from forked processes 2021-01-29 01:31:26 -08:00
Markus Pilman 301e25334d fixed asan build 2021-01-28 09:01:40 -07:00
Steve Atherton 7ce0cca687 Merge branch 'release-6.3' of https://github.com/apple/foundationdb into afcache-write-limit 2021-01-28 07:27:32 -08:00
Markus Pilman 7c64e3943b compile boost for sanitizers 2021-01-27 19:13:02 -07:00
Evan Tschannen 75b24da109 fixed cast 2021-01-27 14:12:18 -08:00
Evan Tschannen 3e2a0a5264 throw an io_timeout if it takes more than 2 minutes to commit on the storage server 2021-01-27 11:50:42 -08:00
sfc-gh-tclinkenbeard 511ec43e62 Merge remote-tracking branch 'origin/master' into encrypt-backup-files 2021-01-27 11:17:58 -08:00