Commit Graph

2910 Commits

Author SHA1 Message Date
Andrew Noyes 8a35f03a18 Use allocateFast4kAligned instead of duplicating its logic 2022-03-22 11:01:33 -07:00
Josh Slocum f27475e2f4 Merge branch 'main' into blob_integration 2022-03-22 11:41:58 -05:00
Steve Atherton 9f205217a4 Merge branch 'main' of https://github.com/apple/foundationdb into redwood-rare-bugs-6574 2022-03-21 21:57:57 -07:00
Trevor Clinkenbeard 6390d93efd
Merge pull request #6646 from sfc-gh-tclinkenbeard/fix-copyright-headers
Update copyright header dates
2022-03-21 16:49:20 -07:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Steve Atherton 074698cdb1
Added 16k magazine size to memory stats. (#6639) 2022-03-21 13:30:27 -07:00
Jingyu Zhou fb046c0665
Merge pull request #6615 from sfc-gh-tclinkenbeard/add-missing-copyright-headers
Add missing copyright headers
2022-03-21 13:17:15 -07:00
Markus Pilman 35f7843d84 Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-03-21 12:33:53 +01:00
Markus Pilman 1fbeca8038 fix memory issue 2022-03-19 11:03:32 +01:00
Steve Atherton 5798674e86
Merge pull request #6628 from sfc-gh-satherton/redwood-small-mem-reduction
Redwood small memory usage reductions and a small CPU reduction
2022-03-18 16:22:17 -07:00
sfc-gh-tclinkenbeard 2c52d28ae2 Merge remote-tracking branch 'origin/main' into arena-get-size-boolean-param 2022-03-17 23:12:54 -07:00
Steve Atherton e0990fd712 Added generic detach() actor to allow large actor footprints to be freed once finished, used it to avoid storing writePhysicalPage actors in the page cache. 2022-03-17 19:58:20 -07:00
Xiaoge Su 99b030c2f6 Allow the TOML file assign knobs during test
In this patch, for a given test, it is possible to override the knob
values, e.g.

[[test]]

    [[test.knobs]]
    watch_timeout = 999

will set the client knob WATCH_TIMEOUT to 999 during the test. The
original value will be recovered after the test is over.
2022-03-17 16:53:10 -07:00
Josh Slocum 37e7c80f26 Merge branch 'main' into blob_integration 2022-03-17 18:45:42 -05:00
Andrew Noyes 68c03a7e32
Jemalloc integration fixes (#6626)
* Set default for USE_JEMALLOC initially in ConfigureCompiler

Instead of trying to change the value later on. This fixes the valgrind
build, which was previously incorrectly getting jemalloc involved.

* Check aligned_alloc result for null

And OOM if so - don't assert

* Check that we can allocate magazines with no internal fragmentation

We may want to do this so that the jemalloc heap profiler has some
knowledge of FastAlloc

* Populate TestFile field for noSim tests in TestHarness

* Remove handling for nonexistent "ActualRun"
2022-03-17 15:17:27 -07:00
Markus Pilman 158da462f7 fix memory bug 2022-03-17 14:37:05 +01:00
Markus Pilman 118b53b7cf Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-03-17 12:06:44 +01:00
sfc-gh-tclinkenbeard 44b0e05538 Merge remote-tracking branch 'origin/main' into arena-get-size-boolean-param 2022-03-16 22:53:30 -07:00
Steve Atherton 6cef85e268 Change assertions to not rely on the first argument being evaluated before the second. 2022-03-16 19:00:26 -07:00
sfc-gh-tclinkenbeard 0ae9ba5fd0 Add FastInaccurateEstimate boolean parameter for Arena::getSize 2022-03-16 16:54:27 -07:00
Andrew Noyes ccc11cbdb2
Apply jemalloc fix, and always build jemalloc (#6610)
* Apply jemalloc fix, and always build jemalloc

See https://github.com/jemalloc/jemalloc/pull/1924

* Move USE_JEMALLOC to flow/config.h

* Add unit test to validate fix
2022-03-16 13:19:28 -07:00
sfc-gh-tclinkenbeard 1aabbb07d2 Add missing copyright headers 2022-03-16 11:25:02 -07:00
Steve Atherton 59762fc784
Merge pull request #6423 from sfc-gh-satherton/redwood-bug-fixes-and-memory-leak
Redwood memory usage and small memory leak fixes
2022-03-15 16:52:28 -07:00
He Liu c3a68d661e
Physical Shard Move (#6264)
Physical Shard Move part I: Checkpoint creation, transfer and restore.
2022-03-15 13:03:23 -07:00
A.J. Beamon e8077b65e1
Merge pull request #6559 from sfc-gh-ajbeamon/fdb-tenant-client
Add client support for tenants
2022-03-15 12:40:17 -07:00
Jingyu Zhou e89ee7d5a0
Merge pull request #6589 from sfc-gh-tclinkenbeard/fix-typos
Fix typos
2022-03-15 10:10:23 -07:00
A.J. Beamon ecccfd0868 Add cache invalidation to tenant cache. Send tenant ID along with tenant name in requests to validate that the tenant hasn't changed. Fix a few bugs. 2022-03-15 09:23:30 -07:00
Markus Pilman 117ee637db Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-03-15 17:17:47 +01:00
Markus Pilman bed799220a Addressed review comments, added test 2022-03-15 16:57:26 +01:00
Jingyu Zhou 800478ad5e Add a new client error code stale_version_vector 2022-03-14 20:50:12 -07:00
Trevor Clinkenbeard 6a28bddd35
Fix file names in copyright headers (#6578) 2022-03-14 13:24:25 -07:00
sfc-gh-tclinkenbeard 8dcac2f76d Fix typos 2022-03-13 10:02:11 -03:00
Jingyu Zhou e371854103 Fix the reserve() of flat_map traits
By actually reserving with the given size for PR #6575.
2022-03-11 17:25:04 -08:00
Mohamed Oulmahdi 9e81945301 Enable encryption for Windows 2022-03-11 10:06:07 +01:00
Jingyu Zhou 5450b740ac Use boost::flat_map for better performance
bench_vv_getdelta/512/1024     8768606 ns      8768555 ns           80 Tags=512 getDeltaTimes=1024 items_per_second=116.781k/s
2022-03-10 21:36:03 -08:00
Steve Atherton 8128f9f0b5 Merge branch 'main' of https://github.com/apple/foundationdb into redwood-rare-bugs 2022-03-10 11:28:09 -08:00
Tao Lin e2c7c30faf
GetMappedRange support serializable & check RYW & continuation (#6181) 2022-03-10 10:05:44 -08:00
Yi Wu e6950abae6 Use jemalloc for SQLite/Redwood page cache allocation 2022-03-09 11:48:08 -08:00
Josh Slocum e71b3533f9 Merge branch 'main' into blob_integration 2022-03-09 08:59:56 -06:00
Markus Pilman 8fac0081a8 Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-03-09 11:00:00 +01:00
A.J. Beamon 402fa4dd9e
Merge pull request #6428 from sfc-gh-ajbeamon/fdb-tenant-add-interfaces
Add server-side and protocol support for tenants
2022-03-08 15:33:18 -08:00
Steve Atherton 9f690d5bd5 Added StringRef::same() which checks data pointers and lengths for match. Fixed a false negative (but not a bug) in boundariesNormal() and used same() to avoid string comparisons. 2022-03-07 22:44:24 -08:00
Andrew Noyes a827e1cbd5 Flush all output streams in flushAndExit
Also detect reading past the end of the infile for debug_determinism tracing
2022-03-07 13:25:52 -08:00
A.J. Beamon 3ae98189af Server processes send unknown_tenant responses back to clients, which is meant to be retried after updating the tenant cache. Fix bug where key selectors could be truncated after applying the tenant prefix. 2022-03-06 21:54:21 -08:00
A.J. Beamon 72a34945ce Add the ability to disable tenants. Server processes verify the ID of tenants being read or written. 2022-03-06 21:54:21 -08:00
A.J. Beamon c99c7cf55f Apply tenant prefixes on the storage server. The commit proxy can serve tenant location requests. Fix some bugs. 2022-03-06 21:54:21 -08:00
A.J. Beamon 5fa9d3e1b7 Add a tenant parameter to read and commit requests. Store a map of all tenants on commit proxy and storage servers. Add an option to require tenant mode. 2022-03-06 21:54:21 -08:00
Trevor Clinkenbeard b84d3af533
Merge pull request #6499 from sfc-gh-ajbeamon/fix-trace-overflow-bug
Fix possible infinite loop when a trace event is too long
2022-03-04 16:11:37 -08:00
Jingyu Zhou 1a5bf25b5c Update code base to use fmt 8.1.1 2022-03-04 15:52:06 -08:00
A.J. Beamon d45bae11b7 Fix substring call when reporting details about a trace event that was longer than the legal limit. 2022-03-04 13:59:49 -08:00
Andrew Noyes 2bd2ddd779
Fix FastAlloc thread cleanup on OSX (#6485)
* Use ThreadData constructor and destructor

Instead of pthread api

* Make threadData a function static thread local variable

* Fix linux build
2022-03-04 12:49:40 -08:00
A.J. Beamon 1ab963eb5b
Merge pull request #6477 from sfc-gh-ajbeamon/trace-overflow-debug-line
Add some extra print debugging if we overflow the trace buffer
2022-03-01 10:12:09 -08:00
A.J. Beamon beac368ba9 Add a mechanism to enable error backtrace logging for select errors 2022-03-01 09:11:21 -08:00
A.J. Beamon f6b24cd644 Add some extra print debugging if we overflow the trace buffer. 2022-03-01 09:04:15 -08:00
Steve Atherton 73a81a86da Added Redwood metrics printing during set unit test using new generic repeatEvery() actor. 2022-02-26 23:37:30 -08:00
Steve Atherton 6f16702042 Merge branch 'main' of https://github.com/apple/foundationdb into redwood-bug-fixes-and-memory-leak 2022-02-26 18:00:18 -08:00
Renxuan Wang 233c918ffb Replace printf() and fprintf() with fmt::print(). 2022-02-25 19:06:57 -08:00
Renxuan Wang f7eb66441d Try eliminating warnings in macOS and Windows CI builds.
MacOS warnings are format warnings, e.g., `format specifies type 'long' but the argument has type 'Version' (aka 'long long')`.
Windows warnings are `ACTOR does not contain a wait() statement`.
2022-02-25 19:06:57 -08:00
Markus Pilman a61dd41b45 Add an empty test to make sure noSim tests succeed 2022-02-25 15:38:16 -08:00
Andrew Noyes 276cf3a402
Fix minor bugs turned up by _GLIBCXX_DEBUG (#6375)
* Fix a benign bug turned up by _GLIBCXX_DEBUG

Just calling std::vector::operator[] with an out of bounds index at all
is technically undefined behavior.

* Fix compilation issue with _GLIBCXX_DEBUG

For some reason std::max with an initializer list isn't constexpr with
_GLIBCXX_DEBUG set
2022-02-25 13:50:41 -08:00
Andrew Noyes 7a9217a392
Add contrib/debug_determinism (#6389)
* Add contrib/debug_determinism

Add an instrumentation-based technique for debugging unseen mismatches. Also guard a few existing sources of nondeterminism that don't affect unseen with the DEBUG_DETERMINISM macro.

Also change the simulated run loop to not run as the only task inside the real run loop, since that was a source of nondeterminism.

Also fix nondeterminism from calling timer_int

* Add StorageMetadataType::currentTime

Basically a deterministic-in-simulation version of timer_int that we can
use instead of timer_int for StorageMetadataType::createdTime
2022-02-25 12:54:31 -08:00
Mohamed Oulmahdi 5b6098b5be Fix Windows build broken by #6449 2022-02-25 16:25:45 +01:00
Renxuan Wang 06b1d06d38 Support hostname in coordinators commands. 2022-02-24 23:02:29 -08:00
A.J. Beamon d3be393246 Add comments to BaseTraceEvent and TraceEvent. Also make BaseTraceEvent contstructor protected. 2022-02-24 13:29:44 -08:00
A.J. Beamon 250a88e682 Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement. 2022-02-24 12:25:52 -08:00
Markus Pilman f1189e5947
Merge pull request #6384 from vikasgupta8/ppc64le_support
added support for ppc64le
2022-02-24 14:31:31 +01:00
Dan Lambright 8cc9a5af1a Rebase 02/23 2022-02-23 14:23:28 -05:00
Markus Pilman 20bf3e1599 Address review coomments 2022-02-23 19:02:29 +01:00
Markus Pilman cf31e14904 Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-02-23 10:29:32 +01:00
Ata E Husain Bohra 344a14b010
Update StreamCipher ctx/cipher management to respect determinism (#6427)
* Update StreamCipher ctx/cipher management to respect determinism

StreamCipher keeps record of CipherKeys created
(including globalCipherKey) to ensure the sensitive data gets
ZERO-OUT and not recorded as part of FDB process dump. However,
in current code it is maintained as an unordered_set indexed
by the object itself. Approach adds non determinism given
object pointer based indexing scheme.

Patch addresses the concern by updating the recording to use
a map indexed by UID.
2022-02-22 17:34:29 -08:00
Markus Pilman dc973fb67e Allow List and first test 2022-02-22 11:15:16 +01:00
Steve Atherton f995b4a502 Merge branch 'main' of https://github.com/apple/foundationdb into redwood-bug-fixes-and-memory-leak 2022-02-18 18:44:28 -08:00
Steve Atherton 735c5697d0 Added fast size estimates to Arena which are O(1) and are usually accurate. When inaccurate, a full size scan will update all estimates in the tree. 2022-02-18 03:25:28 -08:00
Josh Slocum 38a75a8b89 Merge branch 'main' into blob_integration 2022-02-17 17:47:38 -06:00
sfc-gh-tclinkenbeard d01f035848 Increase ITERATIONS_PER_REACTOR_CHECK default 2022-02-17 13:45:17 -08:00
Markus Pilman c7899b9d39 Implemented public endpoints, started allow list 2022-02-16 18:12:03 +01:00
sfc-gh-tclinkenbeard a32547cbbe Merge remote-tracking branch 'origin/main' into flow-enhancements 2022-02-15 15:56:42 -08:00
vikasgupta8 595f50ce26 indentation corrected 2022-02-15 13:03:12 +00:00
root 9bf11ac2f3 Removed condition for arch64 2022-02-11 13:45:17 +00:00
root 459dd83583 resolved format error 2022-02-11 12:11:50 +00:00
root d61519c53d resolved format error 2022-02-11 10:24:56 +00:00
root 15159a5deb resolved format error 2022-02-11 09:57:24 +00:00
vikasgupta8 edfff755bf added support for ppc64le 2022-02-11 06:17:15 +00:00
Sreenath Bodagala c82180501a Merge remote-tracking branch 'apple-upstream/main' into version-vector-prototype 2022-02-10 22:22:34 +00:00
Sreenath Bodagala 72e06369a4 Merge remote-tracking branch 'apple-upstream/main' into version-vector-prototype 2022-02-08 17:47:57 +00:00
sfc-gh-tclinkenbeard acb3e840ac Merge remote-tracking branch 'origin/main' into dd-refactor 2022-02-08 00:33:51 -08:00
sfc-gh-tclinkenbeard 04a1347df2 Merge remote-tracking branch 'origin/main' into dd-refactor 2022-02-08 00:33:27 -08:00
Ray Jenkins e1ae0e03b9 Update flow/Knobs.cpp
Co-authored-by: Lukas Joswiak <lukas.joswiak@snowflake.com>
2022-02-07 15:49:35 -08:00
Ray Jenkins 250b57cc8d Force loopback address when in simulation mode. 2022-02-07 15:49:35 -08:00
Ray Jenkins a5681c52bf Force loopback addr when in simulation mode. 2022-02-07 15:49:35 -08:00
Ray Jenkins 226236600e Preserving original LISTENER_PORT knob and adding new Knob TRACING_UDB_LISTENER_ADDR 2022-02-07 15:49:35 -08:00
Ray Jenkins b588089fe7 formatting 2022-02-07 15:49:35 -08:00
Ray Jenkins 1bd084810c Refactor TRACING_UDP_LISTENER_PORT to TRACING_UDP_LISTENER_ADDR. 2022-02-07 15:49:35 -08:00
Xiaoxi Wang 6dc5921575
createdTime based storage wiggler (#6219)
* add storagemetadata

* add StorageWiggler;

* fix serverMetadataKey bug

* add metadata tracker in storage tracker

* finish StorageWiggler

* update next storage ID

* change pid to server id

* write metadata when seed SS

* add status json fields

* remove pid based ppw iteration

* fix time expression

* fix tss metadata nonexistence; fix transaction retry when retrieving metadata

* fix checkMetadata bug when store type is wrong

* fix remove storage status json

* format code

* refactor updateNextWigglingStoragePID

* seperate storage metadata tracker and store type tracker

* rename pid

* wiggler stats

* fix completion between waitServerListChange and storageRecruiter

* solve review comments

* rename system key

* fix database lock timeout by adding lock_aware

* format code

* status json

* resolve code format/naming comments

* delete expireNow; change PerpetualStorageWiggleID's value to KeyBackedObjectMap<UID, StorageWiggleValue>

* fix omit start rount

* format code

* status json reset

* solve status json format

* improve status json latency; replace binarywriter/reader to objectwriter/reader; refactor storagewigglerstats transactions

* status timestamp
2022-02-04 15:04:30 -08:00
sfc-gh-tclinkenbeard ca7fb0b171 Fix stringop-overflow warning from gcc in StringRef::compare 2022-02-04 14:58:03 -08:00
Markus Pilman 4414df1914
Merge pull request #6279 from sfc-gh-rjenkins/threadname-issue-6064
Make better use of thread names.
2022-02-04 08:46:23 -07:00
A.J. Beamon 1666211252
Merge pull request #6283 from sfc-gh-ajbeamon/adjust-protocol-version
Unset the last bit of the protocol version
2022-02-03 12:26:28 -08:00
Steven Li 36c6e23700
Add TraceEvent severity knob (#6326)
* add knob for trace event severity

* add knob for TraceEvent severity

* fix format

* fix switch format

* moved intToSeverity call inside __test initialization

* updated knob name

* fix line length format

* fix format

* git clang-format
2022-02-02 11:36:17 -08:00
A.J. Beamon 31adbce2f3 Clarify the expectations for the current protocol version and min invalid protocol version in comments. Add static asserts to validate those expectations, where possible. 2022-02-02 10:02:01 -08:00
Ray Jenkins dd45805312
Merge branch 'apple:main' into threadname-issue-6064 2022-02-01 17:40:07 -06:00
Ata E Husain Bohra 591ef57857
Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor (#6314)
* Upgrade AES 128 GCM -> AES 256, StreamCipher code refactor

Major changes proposed are:
1. Refactor StreamCipher code to enable instantiation of
   multiple encryption keys. However, code still retains
   a globalEncryption key semantics used in Backup file
   encryption usecase.
2. Enhance StreamCipher to provide HMAC signature digest
   generation. Further, the class implements HMAC encryption
   key derivation function.
3. Upgrade StreamCipher to use AES 256 GCM mode from currently
   supported AES 128 GCM mode.
   Note: The code changes the encryption key size, however, the
         feature is NOT currently in use, hence, should be OK.
3. Add EncryptionOps validation and benchmark toml supported
   workload, it does the following:
   a. Allow user to configure encrypt-decrypt of a fixed size
      buffer or variable size buffer [100, 512K]
   b. Allow user to configure number of interactions of the runs,
      in each iteration: generate random data, derive an encryption
      key using HMAC SHA256 method, encrypt data and
      then decrypt data. It collects following metrics:
    i) time taken to derive encryption key.
    ii) time taken to encrypt the buffer.
    iii) time taken to decrypt the buffer.
    iv) total bytes encrypted and/or decrypted
   c. Along with stats it basic basic validations on the encrypted
      and decrypted buffer
   d. On completion for test, records the above mentioned metrics
      in trace files.
2022-01-31 19:52:44 -06:00
Trevor Clinkenbeard e6b02532a6
Merge pull request #6192 from sfc-gh-tclinkenbeard/improve-commitbatcher-performance
Make `VectorRefPreserializer` move constructor noexcept
2022-01-31 09:52:51 -08:00
Ray Jenkins 6a1fe2489a format fixes 2022-01-28 18:21:42 -06:00
A.J. Beamon 39a7dc7a2d Add two unit tests for the new hashing functionality 2022-01-28 15:56:58 -08:00
A.J. Beamon 0dabd85e3b Add a hash function for Optional. Add a default boost hash function for any type that is not boost hashable but can be hashed using std::hash. 2022-01-28 15:56:58 -08:00
Ray Jenkins ef4e61ee33 Use GetLastError to include errno data in trace logs. 2022-01-28 17:27:41 -06:00
Ray Jenkins f9f0fb0781 Better error handling for pthread_setname_np.
In unit and simulation testing calls to pthread_setname_np may return errors,
as the threads may complete before calls to setname can be executed. This change
adds better error handling for cases where ENOENT or ESRCH is returned during testing.
Previously the ASSERT_EQ would cause tests to fail if a non-zero return value was encountered.

This change will trace log with a SevWarn when ENOENT or ESRCH is encountered. Otherwise
it will trace with SevError and throw a platform_error.
2022-01-28 16:18:22 -06:00
Andrew Noyes 96cbfe668c
Fix flaky ctest tests (#6310)
* Use localhost cluster for trace_partial_file_suffix_test

This way we get a predictable 127.0.0.1 in the trace file name

* Skip suspend test of pidof is not available

* Avoid writing to closed trace log

calling fdb_network_stop sends a "close" message to the trace thread,
but the network thread might can still be running and sending "flush"
messages to the network thread. This change basically ignores any
flushes that come after a close.

* Ensure unique ports for multi-process tests
2022-01-28 13:16:44 -08:00
Renxuan Wang 2ea4146e1f Add resolveTCPEndpointBlocking() to resolve hostnames where async resolving is impossible. 2022-01-28 12:20:41 -08:00
Josh Slocum ac1fd056dd Added change feed popped read guard for blob workers 2022-01-28 10:45:33 -06:00
Renxuan Wang 4a8e2a80e6 Improve/fix disk metrics.
1. Introduce processDiskReadSeconds and processDiskWriteSeconds, which stands for disk read/write times `since the last logging`. They can only be obtained on Linux and macOS, and will be 0 on Windows and FreeBSD;
2. Rename `busyTicks` to `IOMilliSecs`;
3. On FreeBSD, the metrics should be collected among all devices.
2022-01-27 14:40:32 -08:00
Ray Jenkins 783cbb0aea Merge branch 'main' into threadname-issue-6064 2022-01-27 09:57:11 -06:00
A.J. Beamon 1800259d7c
Merge pull request #6136 from sfc-gh-ajbeamon/client-description-trace-field
Add ClientDescription field for client processes to identify the client logging an event.
2022-01-26 11:37:31 -08:00
Ray Jenkins 41dab5f932 Merge branch 'main' into threadname-issue-6064 2022-01-26 08:53:36 -06:00
Yao Xiao c605226a56
Add error handling in RocksDB KVS. (#6277) 2022-01-25 13:59:25 -08:00
A.J. Beamon 1f61daa235 Unset the last bit of the protocol version. Reorder protocol version feature declarations to match version order. 2022-01-25 12:41:10 -08:00
Ray Jenkins 0d34ec0880 Add RunLoopProfiler thread name. 2022-01-25 13:22:22 -06:00
Ray Jenkins ff49bfaeb5 Add thread name for SSLHandshakerThread. 2022-01-24 20:00:00 -06:00
Renxuan Wang 1d62bc9437 Hostname needs a default constructor. 2022-01-20 19:18:15 -08:00
Renxuan Wang a6c482ee91 Add the support of using hostname in ConnectionString.
This PR comes without simulation tests except some unit tests. The simulation tests will be in the PR that uses hostname in code logic.
2022-01-20 19:18:15 -08:00
Josh Slocum 62acbcfe19 Added explicit error for old blob manager instead of attaching it to response objects 2022-01-20 14:43:08 -06:00
Dan Lambright 9544379cdf rebase 2022-01-20 11:12:33 -05:00
He Liu fd2f553d78
Merge pull request #6228 from liquid-helium/thread-pool-priority
Enabled setting thread poll priorities.
2022-01-19 11:00:45 -08:00
Renxuan Wang 28832a99d6 Address comment. 2022-01-18 14:34:18 -08:00
Renxuan Wang b8bab06e16 Add the functions to set and get mock DNS.
These functions will be used in restarting tests, where mock DNS needs to be saved to and read from files.
2022-01-18 14:34:18 -08:00
He Liu 2fb5c59440 Removed deprioritizeThread(). 2022-01-18 11:10:56 -08:00
sfc-gh-tclinkenbeard bbe0e1210e Merge remote-tracking branch 'origin/main' into flow-enhancements 2022-01-18 00:05:29 -08:00
Andrew Noyes cbadccaad5 Reject preprocessor directives in ACTORs 2022-01-14 11:37:29 -08:00
Markus Pilman 8b77c8d79a Remove unused OpenSSL code 2022-01-11 10:34:37 -08:00
He Liu 2c0c51dd6d Enabled setting thread poll priorities. 2022-01-10 17:50:56 -08:00
Josh Slocum 95e2d0993f Fixed destroy race in NotifiedQueue 2022-01-10 12:27:52 -06:00
Steve Atherton 722465f7cd Improvements to encrypted related APIs and error handling. 2022-01-09 05:17:32 -08:00
Steve Atherton 2742a22dc3 Merge branch 'master' of https://github.com/apple/foundationdb into redwood-header-changes 2022-01-06 23:16:35 -08:00
Steve Atherton 8efd22f9f2 Remove ReferenceCounted::construct() in favor of makeReference(). 2022-01-06 22:58:50 -08:00
Steve Atherton 7a08d59c22 Addressing PR discussions. Many renames. Changed encryption API to use new EncryptionKeyRef struct. Added a tree offset byte to BTreePage to allow for adding an additional field in the future. Added dummy definition for valgrind defined check. Added check that BTree pages are encrypted if encryption is enabled to prevent attack by replacing encrypted page with unencrypted page. 2022-01-06 22:27:22 -08:00
Ata E Husain Bohra 936bf5336a
Revert "Revert "Refactor: ClusterController driving cluster-recovery state machine" (#6191)
* Revert "Revert "Refactor: ClusterController driving cluster-recovery state machine""

Major changes includes:
1. Re-revert Sequencer refactor commits listed below (in listed order):
1.a. This reverts commit bb17e194d9.
1.b. This reverts commit d174bb2e06.
1.c. This reverts commit 30b05b469c.

2. Update Status.actor to track ClusterController interface to track
   recovery status.
3. Introduce a ServerKnob to define "cluster recovery trace event"
   prefix; for now keeping it as "Master", however, it should allow
   smooth transition to "Cluster" prefix as it seems more appropriate.
2022-01-06 12:15:51 -08:00
Steve Atherton dd90b7661d Merge branch 'master' of https://github.com/apple/foundationdb into redwood-header-changes 2022-01-06 04:43:00 -08:00
Steve Atherton d4bb27c05a Added static template ::construct(args...) on both ReferenceCounter<> classes as a more convenient way to construct references. 2022-01-06 03:01:29 -08:00
Andrew Noyes a3f37df94a
Merge pull request #6175 from sfc-gh-anoyes/anoyes/delete-non-virtual-destructor
Enable -Wdelete-non-virtual-dtor for clang build
2022-01-05 15:41:59 -08:00
Josh Slocum 807ec37e84 BG request retry on failed blob worker properly 2022-01-05 17:38:49 -06:00
Josh Slocum a96163d9d3 Fixed ASAN issues 2022-01-05 13:12:49 -06:00
A.J. Beamon b9581fb768 Convert variable to const& 2022-01-04 09:16:26 -08:00
sfc-gh-tclinkenbeard 8076806e76 Add ITERATIONS_PER_REACTOR_CHECK 2022-01-01 20:14:57 -08:00
sfc-gh-tclinkenbeard 2ad8955dd3 Cleanup for Flow Enhancements PR 2022-01-01 20:04:00 -08:00
sfc-gh-tclinkenbeard 648609d764 Make VectorRefPreserializer move constructor noexcept 2021-12-30 22:46:16 -08:00
Steve Atherton 7896c2979f Finished Redwood page API refactor and implemented dummy page encryption scheme with XOR, used randomly in simulation, to prove that pluggable encryption schemes fit in the design. 2021-12-25 00:22:36 -08:00
Aaron Molitor 30b05b469c Revert "Refactor: ClusterController driving cluster-recovery state machine"
This reverts commit dfe9d184ff.
2021-12-24 11:25:51 -08:00
Andrew Noyes 28971c5181 Fix memory leak. Closes #4482 2021-12-22 15:04:00 -08:00
Ata E Husain Bohra dfe9d184ff Refactor: ClusterController driving cluster-recovery state machine
At present, cluster recovery process consists of following steps:
1. ClusterController clusterWatchDatabase actor recruits
   master/sequencer process.
2. Sequencer process implements the cluster recovery state machine,
   responsible to recruit all other processes as well restore the
   cluster state.

Patch proposes a scheme where the cluster recovery state machine
is implemented and driven by the ClusterController process instead
of the Sequencer process.

Advantages of the scheme could be:
1. Simplified design where ClusterController recruits "sequencer"
   process like other worker processes compared to current scheme
   where "sequencer" process gets special treatment. In newer scheme
   sequencer is responsible for maintaining/providing
   "committed version" (as expected).
2. ClusterController is responsible for worker processes recruitment,
   the sequencer though orchestrating the recovery state machine, it
   need to reachout to the ClusterController for recruiting worker
   processes etc.

NOTE:
Patch has moved the recovery state machine code from
'sequencer' -> 'cluster-controller' process, however, necessary
updates were done for both functionality as well as performance
improvement reasons.

Next Steps:
Cluster recovery documentation will be updated in near future.
2021-12-22 14:06:27 -08:00
Josh Slocum b0aea91895 Broadening explicit disconnect handling to explicit error handling of all types 2021-12-21 14:12:09 -06:00
Andrew Noyes fd33d31ff5 Enable -Wdelete-non-virtual-dtor for clang build
We had been disabling -Wdelete-non-virtual-dtor, because this seems to be done intentionally in the generated code of the actor compiler. I spent some time trying to rewrite it in a way that doesn't literally delete/destroy through a pointer to a base class without a virtual destructor, but I was unable to come up with something that passes correctness. My best guess is that we do this so that we can destroy actor state classes, call callbacks registered on the actor SAV, and then destroy the SAV.

Anyway now we'll detect new usages of deleting through a pointer to a base class without a virtual destructor.
2021-12-20 16:19:31 -08:00
Josh Slocum 9f69715fec Fixing blob worker committed tracking and ReplyPromiseStream::onEmpty 2021-12-20 11:33:44 -06:00
Jon Fu 476aaa73c6 modify benchmark to avoid threadsleep and change runloop to process IO properly 2021-12-16 16:20:22 -05:00
Jon Fu 169838cec3 stub changes for IO queue and attempt at new benchmark 2021-12-16 13:28:38 -05:00
A.J. Beamon ff1cb58174 Convert hyphens to underscores for all prefix-based arguments (e.g. --knob-, --locality-) 2021-12-14 12:01:44 -08:00
A.J. Beamon f29f487823
Unify flags (#25)
* Unify flags implementation and change help text in backup.actor.cpp
Description

Testing

* Keep LOG_GROUP unchanged

Description

Testing

* Transfer the hyphens to underscores for internal options and user's input, EXCEPT leading hyphens

Description

Testing

* Use a deep copy of the user's input flag to do the match

Description

Testing

* Convert the _ to - in Option arrays of backup.actor.cpp

Description

Testing

* Transter _ to - for files:
        TLSConfig.actor.h, fdbcli.actor.cpp, fdbserver.actor.cpp, FileConverter.h, FileConverter.cpp

Description

Testing

* Change another way to unify flag: using SO_O_ICASE_HYPHEN_AND_UNDERSCORE to determine whether we do the conversion in function IsEqual

Description

Testing

* Change the config command's name from SO_O_ICASE_HYPHEN_AND_UNDERSCORE to SO_O_HYPHEN_TO_UNDERSCORE

Description

Testing

* Update the comment for the SO_O_HYPHEN_TO_UNDERSCORE

Description

Testing

* Fix left underscore in SOption arrays

Description

Testing

* Convert _ to - in several files for commands

Description

Testing

* Make the FDBService and fdbmonitor backward compatible

Description

Testing

* Fix bugs about pointers

Description

Testing

* Check underscore and hyphen at the same time for --knob_, --localily_ and --test_
And fix bugs in fdbmonitor and FDBService
Description

Testing

* Simplify the function in fdbmonitor and FDBService about retrieving arguments.
And fix some documents in masterserver.actor.cpp

Description

Testing

* Convert _ to - for knob in the setKnob functions

Description

Testing

* Convert - to _ in the setKnob functions

Description
Since key in the knob related maps only contain _

Testing

* Rename varialbe name in the fdbmonitor and FDBService for clarification

Description

Testing

Co-authored-by: Chang Liu <chang.liu@snowflake.com>
2021-12-14 08:44:39 -08:00
Steve Atherton 702c26d8ab Merge branch 'master' of https://github.com/apple/foundationdb into redwood-header-changes 2021-12-11 19:41:51 -08:00
Steve Atherton 592006a537 Several bug fixes in ArenaPage pre and post read/write initialization. Several bug fixes in Queue and Page usage of ArenaPage. Bug fixes in page splitting logic involving incorrect space calculations within the new available page space. 2021-12-11 19:41:14 -08:00
A.J. Beamon af931ad989 Add support for universal trace fields that apply to all events in a process. Add a universal field on clients called ClientDescription that identifies the client logging the event. 2021-12-10 15:03:14 -08:00
Josh Slocum 3afe9fb6e0 MVC bug fixes 2021-12-10 12:47:53 -06:00
Josh Slocum 1ee0b16bfa Fixed bug in merge cursor whenAtLeast 2021-12-09 14:19:00 -06:00
Steve Atherton fc86ee147b Merge branch 'page-api-bug-fixes' into redwood-header-changes 2021-12-09 02:22:26 -08:00
Steve Atherton 4bc0cfe4af Many bug fixes in new page API and usages of it. 2021-12-09 02:21:50 -08:00
Andrew Noyes def41697bf
Merge pull request #6083 from sfc-gh-tclinkenbeard/remove-temporaries
Avoid creating unnecessary temporary objects
2021-12-06 13:24:56 -08:00
Steve Atherton d1ede0225b Initial commit of Redwood page header format and API to enable different page encodings to support encryption or different checksum types. 2021-12-05 13:39:35 -08:00
Tao Lin 9b0a9c4503
Return error when getRangeAndFlatMap has more & Improve simulation tests (#6029) 2021-12-03 12:50:07 -08:00
sfc-gh-tclinkenbeard 3d36dfe5e9 Fix compilation error in resolveTCPEndpoint_impl 2021-12-02 12:51:00 -08:00
Evan Tschannen b11ae4dae8
Merge pull request #5910 from sfc-gh-jslocum/bg_bindings
Blob Granule C bindings
2021-12-02 11:40:26 -08:00
sfc-gh-tclinkenbeard 70c8f98eb9 Apply clang-format to Net2.actor.cpp 2021-12-02 10:22:22 -08:00
sfc-gh-tclinkenbeard 464d9488ef Merge remote-tracking branch 'origin/master' into fix-unused-warnings 2021-12-01 23:52:09 -08:00
sfc-gh-tclinkenbeard 6b45ef98ca Merge remote-tracking branch 'origin/master' into remove-temporaries 2021-12-01 23:50:29 -08:00
sfc-gh-tclinkenbeard d01a363e29 Avoid creating unnecessary temporary objects 2021-12-01 23:48:34 -08:00
FoundationDB CI e02dbe1a52
apply formatting with clang 13
Signed-off-by: FoundationDB CI <foundationdb_ci@apple.com>
2021-12-02 05:52:21 +00:00
FoundationDB CI ca5d5ac942
apply formatting with clang 13
Signed-off-by: FoundationDB CI <foundationdb_ci@apple.com>
2021-12-02 05:13:59 +00:00
sfc-gh-tclinkenbeard 90ced244eb Fix -Wunused-but-set-variable warnings 2021-12-01 18:15:53 -08:00
Josh Slocum 9cb6fb5114 fixing unrelated code formatting 2021-12-01 17:20:11 -06:00
Josh Slocum a82845af43 Merge branch 'master' into bg_bindings 2021-12-01 16:55:28 -06:00
Josh Slocum 0f2f5bc0b6 Cleanup of ThreadResult 2021-12-01 16:24:28 -06:00
Josh Slocum 7f4fcc8c2c Added FDBResult and made readBlobGranules use it 2021-12-01 16:22:05 -06:00
sfc-gh-tclinkenbeard ec64890ac1 Remove some usages of PRId64 by using fmt library 2021-11-30 23:35:36 -08:00
A.J. Beamon c47535245b
Merge pull request #6033 from sfc-gh-ajbeamon/improved-client-db-logging
Client logging improvements
2021-11-29 13:23:10 -08:00
Trevor Clinkenbeard 6429b82796
Merge pull request #6053 from RenxuanW/fromHostname
Change member variable fromHostname to type bool.
2021-11-29 13:13:21 -08:00
A.J. Beamon b8bd89f88d Shorten the name of external client threads. Add a thread name for trace logging threads. 2021-11-29 09:57:10 -08:00
A.J. Beamon 264c75b9a6 Add some extra client logging details:
1. Add a trace event when a database is created and move the cluster file / connection string from ClientStart to the new trace event
2. Add a detail for the path to the image being loaded
3. Add a detail for whether a client library is primary or not
4. Set a thread name for each external client thread that includes the release version
2021-11-29 09:57:10 -08:00
Renxuan Wang 09fedc429a Remove unnecessary boost/bind.hpp.
Complement of #6026.
2021-11-24 16:33:05 -08:00
Ata E Husain Bohra 0962fcb243 Override commit/grv proxies_count if mutation supplied new value is -1
Patch improves on handling scenarios where either commit or grv proxies
value is update to -1 OR `proxies_count` is being reset.
The code splits the proxies between two proxies by ensuring for invalid
input configuration, the min (read as 1) proxies gets provisioned, otherwise,
the split is done based on input values

Patch handles the scenario where mutation supplied values to update grv_proxies
and/or commit_proxies is -1, however, the total proxy count > 1,
uses DEFAULT_COMMIT_GRV_PROXIES_RATIO to split proxies between
grv_proxies & commit_proxies.
2021-11-24 12:52:31 -08:00
Renxuan Wang 46d17d748f Change member variable fromHostname to type bool.
So that it can be serialized.
2021-11-23 14:25:02 -10:00
Renxuan Wang 22e34bd6b9 Replace <boost/bind.hpp> with <boost/bind/bind.hpp>.
This eliminates many useless warnings when compiling.
`#pragma message: The practice of declaring the Bind placeholders (_1, _2, ...) in the global namespace is deprecated. Please use <boost/bind/bind.hpp> + using namespace boost::placeholders, or define BOOST_BIND_GLOBAL_PLACEHOLDERS to retain the current behavior.`
2021-11-18 14:00:13 -08:00
Lukas Joswiak 18243351e7 Fix possible data race
Transactions (created on a separate thread) can read the `globals` field
at the same time as `setGlobal` is called on the main thread, causing a
potential race. TSAN surfaced this issue.
2021-11-18 10:16:20 -08:00
negoyal 8b0938c7b3 Added a TODO 2021-11-17 09:45:50 -08:00
Steve Atherton 3caca74ac2 Merge commit 'fd707c6d7ee80de6d9fda5796da2d0add10abd79' into bit-flipping-workload 2021-11-16 21:54:27 -08:00
Markus Pilman b2019cd4f2
Merge pull request #5992 from sfc-gh-mpilman/features/fmt
added fmt dependency to flow
2021-11-16 18:58:55 -07:00
Steve Atherton 035e0d6e52
Merge branch 'master' into bit-flipping-workload 2021-11-16 14:42:22 -08:00
Markus Pilman b1633b90f1 Added fmt to flow 2021-11-16 12:03:49 -07:00
Steve Atherton 867999a41a Rename wrong_format_version to unsupported_format_version. 2021-11-16 03:25:54 -08:00
Renxuan Wang 4630b0ccea Move DNS mock from SimExternalConnection to Sim2.
This is a revise PR of #5934. In simulation, we don't have direct access to SimExternalConnection.
2021-11-15 17:02:51 -08:00
Evan Tschannen 964d0209ca
Merge pull request #5637 from sfc-gh-ljoswiak/features/data-loss-prevention
Data loss protection when joining new cluster
2021-11-15 15:26:32 -08:00
Jingyu Zhou 02d0c43bc2
Merge pull request #5982 from sfc-gh-tclinkenbeard/improve-error-descriptions
Make snapshot errors more descriptive
2021-11-15 13:18:19 -08:00
Evan Tschannen a546fb63ea
Merge pull request #5985 from sfc-gh-etschannen/feature-changefeed-empty-versions
Added a whenAtLeast function to change feeds to efficiently learn about empty versions
2021-11-15 10:51:28 -08:00
Markus Pilman daf6dc22d4
Merge pull request #5959 from mpilman/features/apple-silicon-3
FDB compiles on Apple Sillicon
2021-11-15 11:21:28 -07:00