903f7adbc4
more cleanup
2022-03-10 13:53:06 -06:00
8128f9f0b5
Merge branch 'main' of https://github.com/apple/foundationdb into redwood-rare-bugs
2022-03-10 11:28:09 -08:00
c35e020da7
SS and BM test macros
2022-03-10 12:38:23 -06:00
ba35e7908a
off by one
2022-03-10 12:11:44 -06:00
e2c7c30faf
GetMappedRange support serializable & check RYW & continuation ( #6181 )
2022-03-10 10:05:44 -08:00
d7367dad99
Simplifying CF gap validation to ignore SS CF fetches
2022-03-10 11:56:22 -06:00
4b254d259c
Ensuring BM split retry is idempotent
2022-03-10 11:54:57 -06:00
c8c97e0256
Blob Worker focused cleanup
2022-03-10 09:55:23 -06:00
bade9a3ec3
Added toString() methods for DeltaTree::DecodeCache.
2022-03-10 00:07:22 -08:00
6cb5f86994
Added a boundary sample to the boundary verifier, which clear operations in the Redwood unit test will randomly make use of. Added cold start limit in BTree unit test to prevent test running too long, and a few other parameter changes.
2022-03-09 23:17:55 -08:00
c76ba149b2
Removed unnecessary line as it does nothing, updated comment.
2022-03-09 20:28:14 -08:00
81d1e70435
Bug fix: Remap cleanup must delay freeing of unmapped destination pages until the oldest readable version passes the current latest readable version to prevent a very slow reader from reading a page after it has been recycled and reused. This is a very rare bug, seen in about 1 in 1 million runs of the Redwood unit test.
2022-03-09 19:08:00 -08:00
2ec15965ba
No logic changes, just debugging output changes and variable renames for clarity.
2022-03-09 18:45:03 -08:00
1f964ac085
BM focused cleanup
2022-03-09 15:04:17 -06:00
9dbb6d216e
SS-focused cleanup
2022-03-09 10:39:04 -06:00
b21d0943b9
client-focused cleanup
2022-03-09 10:01:25 -06:00
4d909abbd1
re-enabling all storage engine types
2022-03-09 09:21:45 -06:00
f230ac0e41
Fixing more merge issues
2022-03-09 09:13:03 -06:00
e71b3533f9
Merge branch 'main' into blob_integration
2022-03-09 08:59:56 -06:00
402fa4dd9e
Merge pull request #6428 from sfc-gh-ajbeamon/fdb-tenant-add-interfaces
...
Add server-side and protocol support for tenants
2022-03-08 15:33:18 -08:00
63b8b838a2
Merge pull request #6558 from sfc-gh-ajbeamon/special-keys-limits
...
Add support for a limits hint in special keys range reads
2022-03-08 15:32:59 -08:00
6f7adb04a5
Send a version along with GetKeyServerLocationsRequests to allow the proxy to reject missing tenants more quickly. Change the commit proxy tenant map to be unversioned.
2022-03-08 13:33:56 -08:00
24d0c158b4
Apply stricter condition for special keys check
2022-03-08 16:22:09 -05:00
5003a0d662
Remove unnecessary parameter from clampRangeToTenant
2022-03-08 12:01:51 -08:00
f39a53787f
Ignore changes that happen in the system key-space when evaluating whether a transaction should have conflicted in the conflict range test.
2022-03-08 07:46:07 -06:00
e96dc76aad
Bug fix: When a BTree node is updated and is to the left of a completely removed sibling subtree the placeholder record providing its upper decode boundary could be lost because expectedUpperBound was not being set.
2022-03-08 03:56:29 -08:00
d034d0c30f
Bug fix in boundary verifier which caused false failures after a process restart when a commit was in progress because the verification map would contain changes that were rolled back.
2022-03-08 03:33:22 -08:00
77f06eedfd
Use printable() on page boundary debug output.
2022-03-07 23:21:07 -08:00
8f844437da
Avoid unused variable warning when debug output is not on.
2022-03-07 23:12:32 -08:00
9f690d5bd5
Added StringRef::same() which checks data pointers and lengths for match. Fixed a false negative (but not a bug) in boundariesNormal() and used same() to avoid string comparisons.
2022-03-07 22:44:24 -08:00
35df00eba2
Re-enable simulation-only boundary verification.
2022-03-07 22:31:29 -08:00
55c98a4287
Large refactor of redwood debug output to improve context and readability.
2022-03-07 22:30:20 -08:00
212deb05e9
Merge pull request #6501 from neethuhaneesha/rocksdb-CompBytesLimiter
...
Rocksdb knobs for compaction, storageserver canCommit() waiting if rocksdb overloaded
2022-03-07 14:59:34 -08:00
36edfb8820
Update error type for timeouts. ( #6503 )
2022-03-07 13:14:20 -08:00
8f8f95931b
In the SQLite storage engine, destroy and create new cursors after SQLITE_CURSOR_MAX_LIFETIME_BYTES of KV read usage because cursor usage increases page cache memory usage in SQLite (internal page cache, not AsyncFileCached) by pinning pages which are not freed until the cursor is destroyed.
2022-03-07 11:20:59 -08:00
07169b1f11
Add unknown_tenant error to documentation. Avoid doing an extra tenant lookup in the clampRangeToTenant function.
2022-03-07 10:37:22 -08:00
3ae98189af
Server processes send unknown_tenant responses back to clients, which is meant to be retried after updating the tenant cache. Fix bug where key selectors could be truncated after applying the tenant prefix.
2022-03-06 21:54:21 -08:00
72a34945ce
Add the ability to disable tenants. Server processes verify the ID of tenants being read or written.
2022-03-06 21:54:21 -08:00
b4feddea8d
Fix a couple bugs. Don't store old versions in tenant map indefinitely.
2022-03-06 21:54:21 -08:00
e898e59d18
Fix merge issue that caused tenant serialization problems. Fix case where arena was passed by value rather than reference. Watches should check the tenant map at the latest version rather than the watch version.
2022-03-06 21:54:21 -08:00
c99c7cf55f
Apply tenant prefixes on the storage server. The commit proxy can serve tenant location requests. Fix some bugs.
2022-03-06 21:54:21 -08:00
bd3920c23d
Wait to initialize the SS tenant map until after the storage server has been added. Add some extra logging around inserting and erasing tenants.
2022-03-06 21:54:21 -08:00
b8192828d9
Add support for the global tenant prefix to tenant map entries
2022-03-06 21:54:21 -08:00
9c4773164e
Add ID to requests
2022-03-06 21:54:21 -08:00
5fa9d3e1b7
Add a tenant parameter to read and commit requests. Store a map of all tenants on commit proxy and storage servers. Add an option to require tenant mode.
2022-03-06 21:54:21 -08:00
36435af4f5
Add support for a limits hint in special keys. Fix a typo and resolve some comment formatting weirdness.
2022-03-06 21:52:39 -08:00
11f25bc08f
Initialize decode boundary verifier to nullptr for restart tests, also disable check for now as the feature is not finished.
2022-03-04 17:20:31 -08:00
f03c0b8c3c
Added ISimulated::restarted for detecting a restarted simulation test.
2022-03-04 17:19:46 -08:00
ddb42f33e5
Merge pull request #6475 from sfc-gh-ajbeamon/fix-trace-double-escaping
...
Fix several instances where trace events were double-escaping strings
2022-03-04 16:10:49 -08:00
1a5bf25b5c
Update code base to use fmt 8.1.1
2022-03-04 15:52:06 -08:00
ad792630aa
Fix formatting
2022-03-04 14:06:16 -08:00
41d1e4b5d5
Fix several instances where we were converting keys, ranges, and key selectors into strings before passing them into trace events. This had the effect of double escaping certain characters.
2022-03-04 13:58:36 -08:00
45649c543d
Merge pull request #6500 from sfc-gh-ajbeamon/fix-merge-error
...
Fix conflict between two recent merges by using the new errorUnsuppressed function
2022-03-04 13:56:52 -08:00
f4291af1ab
Merge pull request #6476 from sfc-gh-ajbeamon/fix-clang-formatting-issues
...
Fix formatting issues caused by clang-format in WriteDuringRead workload
2022-03-04 12:46:12 -08:00
11fbd1564e
Fix two more cases of error(e, true)
2022-03-04 12:44:52 -08:00
8796a763a5
Rocksdb knobs for compaction, storageserver canCommit() waiting if rocksdb overloaded.
2022-03-04 12:41:17 -08:00
cdebda35ab
Merge pull request #5725 from sfc-gh-jfu/jfu-grv-cache
...
Add transaction option for clients to use cached read versions
2022-03-04 09:17:27 -08:00
49bd0c34cc
Merge pull request #6493 from neethuhaneesha/rocks-compaction
...
RocksDB increasing background threads to speedup compaction.
2022-03-04 07:24:19 -08:00
6024bf0109
Merge pull request #6482 from sfc-gh-xwang/ppw-status
...
Print the IP:Port address of wiggling server in status
2022-03-03 15:50:58 -08:00
83e0368eaa
RocksDB increasing background threads to speedup compaction.
2022-03-03 15:14:37 -08:00
976375466f
fixed more pruning bugs
2022-03-02 17:03:37 -08:00
e7a82b68f2
Merge pull request #6486 from sfc-gh-tclinkenbeard/fix-commitlatencybands-sample
...
Fix name of `ProxyCommitData::commitLatencyBands` sample
2022-03-02 16:24:19 -08:00
8b2b74980e
Fix name of ProxyCommitData::commitLatencyBands sample
2022-03-02 14:57:42 -08:00
887d5a25cb
add storage wiggle to status
2022-03-02 11:31:55 -08:00
e73c0a31e6
add wiggle_server_ids and wiggle_server_addresses in status json
2022-03-02 10:03:23 -08:00
9f1d4580e4
Added simulation-only verification of DeltaTree initialization boundaries between writes and reads over time.
2022-03-01 23:46:38 -08:00
a14f3ff067
Do some verify tests with no pruning
2022-03-01 12:52:26 -06:00
e0578f24d2
Fixes for range streaming BG snapshot
2022-03-01 12:45:58 -06:00
6f1fb71949
Fixed a number of different pruning problems
2022-03-01 10:13:20 -08:00
8f8987ad67
Merge branch 'blob_integration' of https://github.com/apple/foundationdb into blob_integration
2022-03-01 10:12:36 -08:00
06f58f848d
Improving change feed validation, and another popped while reading race fixed
2022-03-01 12:05:55 -06:00
92a888910f
Fix many repeated recruitment bug
2022-03-01 12:00:53 -06:00
ce176e846e
Switching initial granule snapshot to use range stream
2022-03-01 11:20:44 -06:00
97aec45ce8
Fix formatting issues caused by clang-format
2022-03-01 08:51:50 -08:00
3090fb174c
Tweaks to granule initial snapshot
2022-03-01 08:52:38 -06:00
c27204d401
more BM split race fixes
2022-03-01 07:18:43 -06:00
59794ffcb2
Add randomlyRenameZoneId test parameter
2022-03-01 00:42:24 -08:00
839edbf5ea
moved granule split evaluation to not be specific to any individual blob worker, to simplify it and handle additional cases
2022-02-28 16:49:58 -06:00
642aab2b8b
blob manager range assignment race fixes
2022-02-28 16:12:27 -06:00
7acb633db8
Clean up SaveAndKillWorkload
2022-02-28 13:12:31 -08:00
db8def68db
Use std::unique_ptr for ISimulator::extraDB
2022-02-28 13:12:31 -08:00
fe957deef8
Merge pull request #6399 from sfc-gh-bvr/fdb#4271
...
Introduce a new server knob and use it to test if storage servers are…
2022-02-28 13:02:23 -08:00
f277662419
Fixing some change feed pop durability inconsistencies
2022-02-28 12:50:43 -06:00
a556582041
Fixes for new BM reecovery
2022-02-28 10:36:28 -06:00
921be1dedf
fixed revoking when same worker has granule
2022-02-28 08:44:12 -06:00
ee5030d06c
Fixed memory "leak" where child node update tracker entries for freed internal pages were not removed so the map would just grow.
2022-02-28 00:16:10 -08:00
398374c7d7
Rename cacheSizeBytes to pageCacheBytes to match other tests.
2022-02-27 00:36:23 -08:00
73a81a86da
Added Redwood metrics printing during set unit test using new generic repeatEvery() actor.
2022-02-26 23:37:30 -08:00
3a666610ad
Tweaked Redwood unit test ops limits and added total record read count to prevent test instances that run too long.
2022-02-26 23:36:37 -08:00
6f16702042
Merge branch 'main' of https://github.com/apple/foundationdb into redwood-bug-fixes-and-memory-leak
2022-02-26 18:00:18 -08:00
c122cf1ce0
Bug fix: Avoid having a local temporary Cursor destructed after the Reference<Page> containing its DecodeCache has been dropped by using the BTreeCursor's leaf cursor by reference.
2022-02-26 02:03:43 -08:00
a47f481bae
Bug fix, the extent cache used during recovery is not allowed to evict anything so it now uses a private Evictor with no size limit.
2022-02-25 19:21:11 -08:00
233c918ffb
Replace printf() and fprintf() with fmt::print().
2022-02-25 19:06:57 -08:00
f7eb66441d
Try eliminating warnings in macOS and Windows CI builds.
...
MacOS warnings are format warnings, e.g., `format specifies type 'long' but the argument has type 'Version' (aka 'long long')`.
Windows warnings are `ACTOR does not contain a wait() statement`.
2022-02-25 19:06:57 -08:00
276cf3a402
Fix minor bugs turned up by _GLIBCXX_DEBUG ( #6375 )
...
* Fix a benign bug turned up by _GLIBCXX_DEBUG
Just calling std::vector::operator[] with an out of bounds index at all
is technically undefined behavior.
* Fix compilation issue with _GLIBCXX_DEBUG
For some reason std::max with an initializer list isn't constexpr with
_GLIBCXX_DEBUG set
2022-02-25 13:50:41 -08:00
7a9217a392
Add contrib/debug_determinism ( #6389 )
...
* Add contrib/debug_determinism
Add an instrumentation-based technique for debugging unseen mismatches. Also guard a few existing sources of nondeterminism that don't affect unseen with the DEBUG_DETERMINISM macro.
Also change the simulated run loop to not run as the only task inside the real run loop, since that was a source of nondeterminism.
Also fix nondeterminism from calling timer_int
* Add StorageMetadataType::currentTime
Basically a deterministic-in-simulation version of timer_int that we can
use instead of timer_int for StorageMetadataType::createdTime
2022-02-25 12:54:31 -08:00
ed9f95b299
simplifying blob manager recovery by always treating the DB granule mapping as the source of truth, and persisting all boundary changes there BEFORE updating workers
2022-02-25 14:46:14 -06:00
940f26cb0c
Merge branch 'blob_integration' of https://github.com/apple/foundationdb into blob_integration
2022-02-25 11:00:13 -08:00
ee94f3ed23
Merge branch 'blob_integration' of github.com:apple/foundationdb into blob_integration
2022-02-25 11:46:25 -06:00
bc7cc984b0
Fixing BGVerifyBalance test killing issues
2022-02-25 11:30:21 -06:00
d41b18d132
more change feed validation fixes
2022-02-25 09:08:26 -06:00
623db663dc
don't reset watch config transaction
2022-02-25 08:48:52 -06:00
eac1cd6366
Fixing another case in change feed validation
2022-02-25 08:16:03 -06:00
a32c3a2891
Redwood now uses one global page cache eviction order and size limit for all page sizes, for real or virtual processes, and DecodeCache memory is counted against the global page cache budget.
2022-02-25 03:20:53 -08:00
06b1d06d38
Support hostname in coordinators commands.
2022-02-24 23:02:29 -08:00
3a3b44efe2
Fix macOS CI broken by #6449 .
...
foundationdb/fdbserver/CoroFlow.actor.cpp:262:8: error: no member named 'errorUnsuppressed' in 'BaseTraceEvent'
2022-02-24 20:26:13 -08:00
f3f4764916
Fixing change feed validation to handle special case
2022-02-24 17:12:05 -06:00
cee9f42099
More fixes and debugging improvements found by validation
2022-02-24 16:39:24 -06:00
e5b4fb3d80
Adding explicit popVersion to change feed reply, like tlogs have
2022-02-24 15:20:18 -06:00
e7d2abd764
Fix formatting issue
2022-02-24 12:28:13 -08:00
250a88e682
Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement.
2022-02-24 12:25:52 -08:00
43543b0d0e
Removing incorrect CF validation asserts
2022-02-24 12:39:15 -06:00
b6d02478cd
Adding stream uid tracking for ss feed debugging
2022-02-24 12:19:25 -06:00
86236b766d
Added print debugging for CF memory vs disk validation
2022-02-24 12:11:06 -06:00
b67930cb93
Fixing bug in new change feed fetch logic
2022-02-24 11:31:36 -06:00
1de4fb1b9e
Adding explicit stopVersion instead of stopped status to change feed durable data, several other minor fixes
2022-02-24 11:02:15 -06:00
f14e08a991
addRocksDBPerfContextMetrics
2022-02-23 22:29:07 -05:00
2701e44564
Fixing error handling and race for blob worker status stream to manager
2022-02-23 15:55:16 -06:00
8cb554a006
Fix formatting issues in DDTeamCollection.actor.cpp
2022-02-23 10:21:34 -08:00
102dcb30c6
Merge branch 'main' into fdb#4271
2022-02-23 10:04:19 -08:00
ce1d71472d
Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache
2022-02-23 12:35:25 -05:00
0714b7a250
Fixed blob manager split and reassign race
2022-02-23 08:48:45 -06:00
3c1394578b
Address comments.
2022-02-22 16:29:59 -08:00
ea8b2bd314
Resolve hostnames synchronously when calling actor is impossible.
2022-02-22 16:29:59 -08:00
622d89b552
Rebase on main.
...
Since we changed ClusterConnectionString's status flag from boolean to enum in #6422 , we need to update this PR correspondingly.
2022-02-22 16:29:59 -08:00
8eb7a10404
Address comments.
2022-02-22 16:29:59 -08:00
481587a8c6
Turn on hostname logic.
2022-02-22 16:29:59 -08:00
803fc86e61
Fix compaction issue when rolling nodes back
2022-02-22 16:21:33 -08:00
6739d506c4
More fixes for handling change feed empty version >= ss version
2022-02-22 16:23:32 -06:00
ae071a7211
clean up debug trace lines
2022-02-22 17:16:11 -05:00
1990216e89
use global knowledge inside simulation to determine when it should be possible to get a too_old error
2022-02-22 13:03:57 -08:00
330b2b48ec
improved file cleanup execution and testing
2022-02-22 12:00:09 -08:00
f7aeae5a5c
Merge branch 'apple:main' into dd-utestclass
2022-02-22 11:59:11 -08:00
d89c6bb3de
Fixing initial change feed stream reply to be in try/catch
2022-02-22 13:55:40 -06:00
5dca00a21d
Cleaning up change feed on move triggers when stream is cancelled
2022-02-22 13:48:57 -06:00
a8828db58e
Load balance dynamic knob requests
...
This commit also removes an attempt to read the latest configuration
snapshot when a rollforward timeout occurs. The normal retry loop will
eventually fetch an up to date snapshot and the rollforward will be
retried.
2022-02-22 10:53:48 -08:00
2e2c8bf88c
Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache
2022-02-22 12:43:55 -05:00
e75ab5f3a0
not advancing popped version of change feed sideband from another server
2022-02-22 11:22:35 -06:00
f995b4a502
Merge branch 'main' of https://github.com/apple/foundationdb into redwood-bug-fixes-and-memory-leak
2022-02-18 18:44:28 -08:00
82bbfa8aee
Merge pull request #6395 from sfc-gh-tclinkenbeard/global-tag-throttling
...
Create `TagThrottler` class
2022-02-18 13:17:28 -08:00
a45c9ccce1
Revert "Added Redwood knob to reserve part of configured page cache size for Redwood page decode cache state."
...
This reverts commit 072bc86bb1
2022-02-18 11:14:40 -08:00
17fe322bc4
Added near real-time tracking of Redwood page DecodeCache memory.
2022-02-18 03:26:05 -08:00
c1b0b336ce
Fix merge conflicts in DDTC.actor.cpp - change serverMetrics to setServerMetrics and setTeamHealthy to markTeamUnhealthy
2022-02-17 20:41:38 -08:00
49d2a56776
Merge branch 'main' into dd-utestclass
2022-02-17 20:22:07 -08:00
f9f676abf9
Also re-persisting change feed metadata on fetch after cleanup
2022-02-17 19:21:40 -06:00
072bc86bb1
Added Redwood knob to reserve part of configured page cache size for Redwood page decode cache state.
2022-02-17 16:15:01 -08:00
ef25e4fdfb
Sending empty version back immediately to establish stream connection and acknowledgement queue for change feed stream requests
2022-02-17 18:10:17 -06:00
4dffb61f78
Merge pull request #6403 from sfc-gh-xwang/ppw-restarting
...
move perpetual wiggle upgrade & downgrade test to 7.1
2022-02-17 15:57:24 -08:00
38a75a8b89
Merge branch 'main' into blob_integration
2022-02-17 17:47:38 -06:00
c87d868dd2
Fix SS restart and change feed pop races
2022-02-17 15:55:24 -06:00
f60420a8e6
Correctly updating change feed storage/durable versions after fetch
2022-02-17 10:36:32 -06:00
c568f9884d
Make a few more DDCollection methods private
2022-02-17 08:25:20 -08:00
662618a5bc
Blob Manager lock fixes
2022-02-17 10:01:20 -06:00
7a17dbeacc
Create DD unit test class. Start to make test-specific methods in DDTeamCollection private
2022-02-16 19:18:24 -08:00
f2b59af9b0
format code
2022-02-16 16:32:06 -08:00
27855bc5b5
Merge branch 'apple:main' into fdb#4271
2022-02-16 15:38:36 -08:00
e07ae6fdb9
Address comments
2022-02-16 15:28:56 -08:00
9da735c38e
Batch empty peek reply
2022-02-16 15:28:56 -08:00
e0d3b0a488
format times in trace event
2022-02-16 16:55:14 -05:00
6199faadc3
fix bug which constantly overwrote start time of last throttled queue
2022-02-16 16:52:38 -05:00
5846dda410
temporary changes and extra traces for debugging
2022-02-16 15:05:25 -05:00
949f1f1c3e
Switch to testing MIN_AVAILABLE_SPACE
2022-02-16 11:33:07 -08:00
3dfdc6943c
move downgrade/upgrade test to XXX_7.1.0 because in 7.0 the workload doesn't have the fix
2022-02-16 10:48:31 -08:00
3fe6a952f1
Merge with upstream tcinfo refactor and move the server knob init to be adjacent to related knobs
2022-02-16 10:28:55 -08:00
cb5f44c76d
Merge branch 'apple:main' into fdb#4271
2022-02-16 10:14:50 -08:00
bfb1902a7a
Fixing logic for when granule split is valid
2022-02-16 11:34:22 -06:00
fe03e6f822
Introduce a new server knob and use it to test if storage servers are near the min bar for available space
2022-02-15 22:43:06 -08:00
32bbabefd7
Merge remote-tracking branch 'origin/main' into dd-refactor
2022-02-15 14:11:30 -08:00
26480c4705
change feed popping fixes
2022-02-15 15:12:23 -06:00
53573e7648
Don't reset CF fetch version on pop
2022-02-15 15:12:05 -06:00
d399daebed
Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache
2022-02-15 15:09:40 -05:00
6f1c3d50bb
add debug traces for testing
2022-02-15 15:08:53 -05:00
42136bf808
Fixed memory leak (and minor performance bug) where RedwoodRecordRef::updateCache() would create the key copy to be placed in the cache but not actually place it into the real cache entry because the cache parameter was accepted by value.
2022-02-14 22:47:30 -08:00
47d5492459
Make many fields and methods of Ratekeeper private
2022-02-14 19:28:19 -08:00
0f4c808f37
Remove TagThrottler::ratekeeper field
2022-02-14 19:08:23 -08:00
39e7e4cdaa
Merge branch 'apple:main' into dd-utest
2022-02-14 19:01:28 -08:00
182fb8de3d
Change type of TagThrottlerImpl::ratekeeper
2022-02-14 18:57:16 -08:00
58669717f1
Move tryAutoThrottleTag method to TagThrottler
2022-02-14 18:54:12 -08:00
b80f89204a
Use PImpl for TagThrottler class
2022-02-14 18:17:55 -08:00
796249e99d
Move lastBusiestCommitTagPick field back to Ratekeeper
2022-02-14 16:15:51 -08:00
00f12687c6
Add TagThrottler class
2022-02-14 16:03:37 -08:00
092b5cee4b
MVC2.0: Rollback added code
2022-02-14 13:50:42 -08:00
c5810b2127
Reducing possibility for large write rate in BGCorrectness
2022-02-14 15:45:46 -06:00
49ba96e0bb
Clean up ratekeeper includes
2022-02-14 13:02:09 -08:00
8074630530
Rename class RatekeeperData to Ratekeeper
2022-02-14 12:42:25 -08:00
d6c5239080
Fix copyright headers
2022-02-14 12:36:44 -08:00
d4b4479399
Rename RatekeeperData.actor.cpp to Ratekeeper.actor.cpp
2022-02-14 12:35:50 -08:00
687df447ce
Move all code from Ratekeeper.actor.cpp into RatekeeperData.actor.cpp
2022-02-14 12:32:34 -08:00
9beae3fb64
Add RatekeeperData class with refactored implementation
2022-02-14 11:49:45 -08:00
abdbc5aafb
Fixing granule opening after not deleting granule locks
2022-02-14 13:20:54 -06:00
7492b755d8
Merge branch 'main' of github.com:apple/foundationdb into jfu-grv-cache
2022-02-14 14:06:49 -05:00
8129c4e21c
simplify sidebandsingle workload and be stricter with batch throttling on rk
2022-02-14 13:58:56 -05:00
eaef2a373e
Remove superfluous wait statements in DD unit tests. They wait on the getTeam future
2022-02-14 10:11:00 -08:00
b5ff2006b2
Removing incorrect assert
2022-02-14 11:36:54 -06:00
7f3f77fd77
Not cleaning up granule locks immediately after split
2022-02-14 11:29:21 -06:00
a14377641c
Renamed a couple of test-only collection methods and make the test case waits more succinct
2022-02-13 23:53:12 -08:00
5445dd3106
add option to fix consistency check
2022-02-13 23:30:14 -08:00
e77b70c508
Encapsulate TCMachineTeamInfo fields
2022-02-12 15:50:51 -08:00
51afe403d1
Encapsulate TCMachineTeamInfo::id field
2022-02-11 22:56:16 -08:00
a958fd2fbc
Encapsulate TCServerInfo::serverMetrics
2022-02-11 22:05:57 -08:00
62dbd29ddc
Remove unnecessary whitespace
2022-02-11 21:47:36 -08:00
10f40294a3
Encapsulate TCServerInfo::teams field
2022-02-11 21:44:28 -08:00
45f68c7db7
Add TCServerInfo::removeTeamsContainingServer method
2022-02-11 21:29:52 -08:00
5bfb2669a2
Add TCServerInfo::removeTeam method
2022-02-11 21:05:45 -08:00
3959ee6c27
Encapsulate TCServerInfo::dataInFlightToServer
2022-02-11 19:41:04 -08:00
0fe4dcf469
Make several more TCServerInfo fields private
2022-02-11 19:41:04 -08:00
4aa5a329ac
Encapsulate TCServerInfo::collection and TCServerInfo::tracker
2022-02-11 19:41:04 -08:00
711462dfd7
Encapsulate TCServerInfo::inDesiredDC
2022-02-11 19:41:04 -08:00
9175ba354a
Adding a separate phase for the end of change feed fetching to handle changes in fetchKeys available durability
2022-02-11 17:25:56 -06:00
0db843bc56
more CF movement and cleanup races
2022-02-11 16:04:28 -06:00
c3bb248d61
Adapt to members of DDCollection class that were accessed directly in test cases becoming private
2022-02-11 13:56:00 -08:00
a63d218e9d
simplify test workload and adjust ratekeeper throttling strategy
2022-02-11 16:41:14 -05:00
36c5d3a1e6
Merge branch 'main' into dd-utest
2022-02-11 12:25:31 -08:00
b8d8eafdec
Merge pull request #6379 from sfc-gh-tclinkenbeard/dd-refactor
...
Shrink public interface of `DDTeamCollection`
2022-02-11 11:44:15 -08:00
e6d68a0fdb
more variability in write workload for BGCorrectness and making tiny values drive slightly less hard
2022-02-11 13:35:39 -06:00
9158564bfc
Fix formatting
2022-02-11 10:27:41 -08:00
3bbddce668
Fixing shard move away and back race
2022-02-11 10:14:26 -06:00
8a56349ea3
Fixing shard move away then back then away race
2022-02-11 10:12:49 -06:00
08505325d3
Fixing change feed mapping in cleanup refactor
2022-02-11 08:41:01 -06:00
f9c16ed937
misc change feed improvements
2022-02-11 08:14:00 -06:00
3f8c1e2e69
Encapsulate TCServerInfo::id
2022-02-10 23:04:25 -08:00
faf74b2563
Removed some trace statements that were accidentally left behind in the code
2022-02-10 22:23:03 -08:00
ef68e6fe0d
Merge pull request #6353 from sfc-gh-ljoswiak/fixes/dynamic-knobs
...
Fix dynamic knobs correctness issues
2022-02-10 22:13:02 -08:00
41bd39a82a
Fix code formatting
2022-02-10 22:10:50 -08:00
b306288c62
Merge branch 'main' into dd-utest
2022-02-10 22:00:52 -08:00
df2f5c5709
Make UT parameters sensitive to server-knobs
2022-02-10 21:52:30 -08:00
83b7f1fcbf
Add test descriptions for DD unit tests
2022-02-10 19:29:34 -08:00
6e87c01404
Move implementations of TC*Info methods to TCInfo.actor.cpp
2022-02-10 19:19:50 -08:00
7790d57c06
Mask the use of the delay in getTeam and the buildTeams path in the DD unit test
...
delay doesn't work as expected in the UT envionment
2022-02-10 18:52:40 -08:00
2165635478
Make printSnapshotTeamsInfo a static function of DDTeamCollection
2022-02-10 18:45:52 -08:00
9bc38ae73e
Make DDTeamCollection::distributorId private
2022-02-10 18:26:06 -08:00
14c8483e9d
Mark DDTeamCollection::primary private
2022-02-10 18:16:57 -08:00
8e7963172b
Mark DDTeamCollection::excludedServers private
2022-02-10 17:58:59 -08:00
7ba250a991
Fix -Wreorder-ctor warnings
2022-02-10 17:54:57 -08:00
1f64477938
Make more DDTeamCollection fields private
2022-02-10 17:44:49 -08:00
36742bcd7e
Fixing durability model of change feed cleanup
2022-02-10 18:52:17 -06:00
6e60e26730
Make some DDTeamCollection fields private
2022-02-10 16:48:26 -08:00
641a38bd0b
Make more DDTeamCollection methods private.
...
The methods only used by DDTeamCollection::run can now be made private.
2022-02-10 16:19:32 -08:00
c4508330d2
Make dataDistributionTeamCollection a static function of DDTeamCollection
2022-02-10 16:19:32 -08:00
5477012ad8
Change DDTeamCollection method signatures to accept references.
...
Passing nullptr to these methods is invalid, but previously the
signature didn't indicate this. We previously needed to pass pointers
due to actor compiler restrictions, but these restrictions no longer
apply.
2022-02-10 16:19:32 -08:00
b51944b262
Pass constant reference to DDTeamCollection::isCorrectDC
2022-02-10 16:19:32 -08:00
c024fbe845
Mark DDTeamCollection::waitUntilHealthy const
2022-02-10 16:19:32 -08:00
f7118ad406
Merge pull request #6371 from sfc-gh-tclinkenbeard/dd-refactor
...
Move `DDTeamCollection` method implementations into `.cpp` file
2022-02-10 15:03:01 -08:00
2e63ac6963
code review changes: additional test checks, cleanup extra code, mark internal option hidden
2022-02-10 17:01:09 -05:00
d684508540
Add RatekeeperLimitReasonDetails traceevent for RK
2022-02-10 13:59:47 -08:00
0dce252c92
Make many DDTeamCollection methods private
2022-02-10 13:44:39 -08:00
458e708272
addressed code review comments: renamed variables, small functional changes, style changes
2022-02-10 16:17:54 -05:00
36700791a1
Use structured bindings in some for loops in DDTeamCollection.actor.cpp
2022-02-10 12:10:45 -08:00
3141698c41
Use special ASSERT_* macros for numeric comparison in data distribution
...
code.
This helps debugging by printing the exact input values when an
assertion fails.
2022-02-10 11:59:19 -08:00
975b9f3b32
Remove get helper function from DataDistribution.actor.cpp
2022-02-10 11:32:33 -08:00
f16aa56d4c
Fix comment grammar
2022-02-10 11:26:39 -08:00
b5cefdbcc2
Mark DDTeamCollection methods const
2022-02-10 11:26:04 -08:00
3c00a66f36
Remove public DDTeamCollection::zeroServerLeftLoggerActor method
2022-02-10 11:19:34 -08:00
b96c2477a3
Mark TSSPairState methods const
2022-02-10 11:12:40 -08:00
c37be7a898
Move DDTeamCollection method implementations into cpp file
2022-02-10 11:09:31 -08:00
631cf0b085
minor memory optimizations for SS change feeds
2022-02-10 12:21:22 -06:00
f997b2dab4
Smarter change feed cancelling when data moved away
2022-02-10 12:21:22 -06:00
017f9b2308
ApiWorkload ctor failed to perform null check on g_simulator
2022-02-10 09:51:37 -08:00
031a11116b
Fixing race between range mover and range split reassign
2022-02-10 11:50:51 -06:00
4af1b24c74
Using higher transferred version when fetching change feeds
2022-02-10 08:00:52 -06:00
a023d49947
Bug fix in DeltaTree - Insert() of record outside of lower/upper boundaries used for key decoding would calculate incorrect record delta.
2022-02-10 01:36:24 -08:00
6d46b03651
Add some unit tests for DD team selection
2022-02-09 22:22:56 -08:00
1b3aba018c
Mark DDTeamCollection::isCorrectDC const
2022-02-09 15:51:52 -08:00
3f0e2ae62e
Merge remote-tracking branch 'origin/main' into dd-refactor
2022-02-09 14:29:16 -08:00
7a76b86b53
Blob worker request timeout race
2022-02-09 16:22:56 -06:00
f300cec6ed
Fast-track ConfigNode registration with Simple DB
...
When using the `ConfigDBType::Simple` configuration database, allow
nodes to immediately register with the broadcaster without having to
wait for a quorum.
2022-02-09 14:18:48 -08:00
c8cd8c0622
Adding request timeout for blob worker
2022-02-09 15:49:33 -06:00
e8354d82bd
Fix timeout issue when using >3 coordinators
...
The calculation to determine how many non-timeout replies had been
received was incorrect, causing rollback/rollforward requests to not be
sent, causing the dynamic knob subsystem to get stuck.
2022-02-09 13:43:33 -08:00
7fc4f0d649
Reuse existing quorum timeout error code
2022-02-09 13:43:33 -08:00
ce67d1937f
Add explanation of out-or-order messages from broadcaster
2022-02-09 13:43:33 -08:00
b5a3312a26
Factor out known replica update step
2022-02-09 13:43:33 -08:00
1d496b7b30
Remove timeout
...
There are a few occasional failures after removing this timeout. They
mostly seem to be caused from contention among the ConfigIncrement
actors which have similar backoffs, causing an infinite loop of attempts
to get their generation accepted on a majority of ConfigNodes.
2022-02-09 13:43:33 -08:00
c86712b4c0
Move single interface serve functions to testing interface
2022-02-09 13:43:33 -08:00
1d15aa5580
Fix internal function name
2022-02-09 13:43:32 -08:00
d5a562e6b8
Fix dynamic knobs correctness issues
2022-02-09 13:43:32 -08:00
9189ea0448
Fixed SS restart and mutation log race
2022-02-09 15:37:09 -06:00
dcbbee56a0
Update copyright dates
2022-02-09 13:28:40 -08:00
96dd86ebf8
update RocskDB and Boost
...
add Finduring, and include into fdbserver
add BOOST asio/uring settings to fdbserver compile
move portable rocks, liburing up to be configurable at build time.
2022-02-09 10:48:18 -06:00
f4c235f134
propagating change_feed_popped correctly
2022-02-09 09:34:13 -06:00
0fb512489e
Removing incorrect epoch check
2022-02-09 09:33:57 -06:00
4d77932def
fixed reordering
2022-02-09 08:05:20 -06:00
3d3223d13c
Remove uses of pid2server_info
2022-02-08 15:58:20 -08:00
9277296102
Set doneRecovering for first blob manager
2022-02-08 17:47:08 -06:00
f05a589669
Fixing blob manager recovery/blob worker status stream races
2022-02-08 17:17:32 -06:00
b385db68a4
capped time and total data for BGVerifySmall tests to avoid OOM
2022-02-08 16:47:44 -06:00
ebf940b710
Remove pid2server_info field
2022-02-08 14:25:16 -08:00
d7955a9017
Much better mechanism for change feed popped checking
2022-02-08 14:23:04 -06:00
713c05bdcd
Improved availability checking at end of BG workload
2022-02-08 14:21:11 -06:00
6adf119e3d
Fixing change feed durableFetchVersion not getting updated
2022-02-08 13:59:06 -06:00
90a870962d
Unblocking bounded change feed request in changeFeedClientVersions once it hit end
2022-02-08 13:22:17 -06:00
acb3e840ac
Merge remote-tracking branch 'origin/main' into dd-refactor
2022-02-08 00:33:51 -08:00
04a1347df2
Merge remote-tracking branch 'origin/main' into dd-refactor
2022-02-08 00:33:27 -08:00
9aaee145b2
Fixing range assignment bug
2022-02-07 17:48:56 -06:00
9885d2cc90
More change feed fetch and read race fixes
2022-02-07 16:21:38 -06:00
7d17e00003
clean up extra trace messages in SidebandSingle workload
2022-02-07 16:58:29 -05:00
607b2a0184
update comment
2022-02-07 13:32:52 -08:00
b32c843522
Revert "use DiskQueueVersion::V2 for KeyValueStoreMemory"
...
This reverts commit ba83d73acd
2022-02-07 13:32:52 -08:00
eae3dab04d
use DiskQueueVersion::V2 for KeyValueStoreMemory
2022-02-07 13:32:52 -08:00
8c92c330c3
fix hash size
2022-02-07 13:32:52 -08:00
0bd4618b79
minor fix
2022-02-07 13:32:52 -08:00
Josh Slocum