Commit Graph

141 Commits

Author SHA1 Message Date
Markus Pilman 550488b020 Merge remote-tracking branch 'origin/main' into bugfixes/open-for-ide
# Conflicts:
#	bindings/c/CMakeLists.txt
#	fdbclient/include/fdbclient/GetEncryptCipherKeys.actor.h
#	fdbserver/BackupWorker.actor.cpp
#	fdbserver/BlobWorker.actor.cpp
#	fdbserver/CommitProxyServer.actor.cpp
#	fdbserver/KeyValueStoreMemory.actor.cpp
#	fdbserver/StorageCache.actor.cpp
#	fdbserver/include/fdbserver/GetEncryptCipherKeys.actor.h
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/PhysicalShardMove.actor.cpp
#	flow/CMakeLists.txt
2022-10-04 18:27:48 -06:00
Markus Pilman 97dfc6823f fixed build with OPEN_FOR_IDE 2022-10-04 17:01:02 -06:00
Xiaoxi Wang 1f1a66be39 merge upstream/main 2022-09-14 12:32:28 -07:00
Xiaoxi Wang 674da807ad add readMoveKeysLock method and catch movekeys_conflict() in the test workload 2022-09-14 12:17:47 -07:00
Xiaoxi Wang 949b1c1af9 Add MoveKeysParams struct; use txnProcessor->moveKeys() 2022-09-12 15:40:18 -07:00
He Liu fa418fd784
Change SHARD_ENCODE_LOCATION_METADATA to a server knob. (#7770)
Co-authored-by: He Liu <heliu@apple.com>
2022-08-03 13:51:40 -07:00
He Liu 35a4cb91d5
Disable ShardedRocksDB in simulation when shard_encode_location_metadata is disabled (#7726)
* Disabled tests for ShardedRocks.

Cleaned up ShardedRocks TraceEvent.

Added assertion in ShardManager::validate().

* Added test trace.

* Make sure TraceEvent contains `ShardedRocks`.

* Exclude ShardedRocksDB when SHARD_ENCODE_LOCATION_METADATA is disabled.

Co-authored-by: He Liu <heliu@apple.com>
2022-07-28 13:54:29 -07:00
A.J. Beamon fee131f24d
Merge pull request #7591 from sfc-gh-ajbeamon/key-backed-types-range-read-more
Add support for returning a more flag from key backed types' range reads
2022-07-20 15:57:50 -07:00
Markus Pilman 1de37afd52
Make TEST macros C++ only (#7558)
* proof of concept

* use code-probe instead of test

* code probe working on gcc

* code probe implemented

* renamed TestProbe to CodeProbe

* fixed refactoring typo

* support filtered output

* print probes at end of simulation

* fix missed probes print

* fix deduplication

* Fix refactoring issues

* revert bad refactor

* make sure file paths are relative

* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
A.J. Beamon c4cd72bea1 Fix a few compile warnings about unused variables 2022-07-19 10:01:41 -07:00
A.J. Beamon 7bca503528 Use narrow includes in key backed types; remove some unnecessary return statements. 2022-07-14 13:26:43 -07:00
He Liu bc5bfaffda
Shard based move (#6981)
* Shard based move.

* Clean up.

* Clear results on retry in getInitialDataDistribution.

* Remove assertion on SHARD_ENCODE_LOCATION_METADATA for compatibility.

* Resolved comments.

Co-authored-by: He Liu <heliu@apple.com>
2022-07-07 20:49:16 -07:00
A.J. Beamon 62683b4140 Update key backed types to support a templated Codec. This required updating the existing Codec to convert from objects into byte strings through tuples. 2022-07-01 10:11:02 -07:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
A.J. Beamon 1d44ef1c8e Specify system key access for a few move keys transactions 2022-03-15 09:23:30 -07:00
A.J. Beamon fb73e1857a Some get key requests should not happen within a tenant. Don't set read_system_keys in watch logic unless necessary. Set access_system_keys in a couple movekeys functions. 2022-03-15 09:23:30 -07:00
Andrew Noyes 7a9217a392
Add contrib/debug_determinism (#6389)
* Add contrib/debug_determinism

Add an instrumentation-based technique for debugging unseen mismatches. Also guard a few existing sources of nondeterminism that don't affect unseen with the DEBUG_DETERMINISM macro.

Also change the simulated run loop to not run as the only task inside the real run loop, since that was a source of nondeterminism.

Also fix nondeterminism from calling timer_int

* Add StorageMetadataType::currentTime

Basically a deterministic-in-simulation version of timer_int that we can
use instead of timer_int for StorageMetadataType::createdTime
2022-02-25 12:54:31 -08:00
A.J. Beamon 250a88e682 Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement. 2022-02-24 12:25:52 -08:00
Xiaoxi Wang 6dc5921575
createdTime based storage wiggler (#6219)
* add storagemetadata

* add StorageWiggler;

* fix serverMetadataKey bug

* add metadata tracker in storage tracker

* finish StorageWiggler

* update next storage ID

* change pid to server id

* write metadata when seed SS

* add status json fields

* remove pid based ppw iteration

* fix time expression

* fix tss metadata nonexistence; fix transaction retry when retrieving metadata

* fix checkMetadata bug when store type is wrong

* fix remove storage status json

* format code

* refactor updateNextWigglingStoragePID

* seperate storage metadata tracker and store type tracker

* rename pid

* wiggler stats

* fix completion between waitServerListChange and storageRecruiter

* solve review comments

* rename system key

* fix database lock timeout by adding lock_aware

* format code

* status json

* resolve code format/naming comments

* delete expireNow; change PerpetualStorageWiggleID's value to KeyBackedObjectMap<UID, StorageWiggleValue>

* fix omit start rount

* format code

* status json reset

* solve status json format

* improve status json latency; replace binarywriter/reader to objectwriter/reader; refactor storagewigglerstats transactions

* status timestamp
2022-02-04 15:04:30 -08:00
A.J. Beamon d8e161f89e Refactor NativeAPI transactions to create and pass around a reference counted state object. Watches no longer use the tranasction info object but instead use their own state. 2021-12-17 11:57:39 -08:00
He Liu 9c835a8e17 merge fix 2021-09-22 16:55:49 -07:00
He Liu a1f6dcc2d5 merge from master 2021-09-22 13:07:38 -07:00
He Liu 95f64f97de resolved some fmt issues 2021-09-20 11:58:30 -07:00
He Liu b2afa6e2f5 throw an error is no team can be found for a dropped range 2021-09-20 11:53:27 -07:00
He Liu 4d5bf08da8 address comments 2021-09-19 17:03:06 -07:00
Xiaoge Su abf73047ca Enforce std:: specifier rather than using namespace 2021-09-16 19:40:28 -07:00
He Liu ef7fdc0781 fmt 2021-09-15 10:32:09 -07:00
He Liu c8a3413820 exclude to-be-dropped server from the random team 2021-09-15 09:07:50 -07:00
helium e838d63011 exclude to-be-removed server from target team 2021-09-14 20:57:58 -07:00
helium fd6d088945 choose team before removing server 2021-09-14 19:24:59 -07:00
helium 7e53f8662d added comments 2021-09-13 13:28:55 -07:00
helium 65e6f26f05 bug fix on unwaited actor 2021-09-09 20:43:29 -07:00
helium 2ea055371b Introduced serverKeysTrueEmptyRange to indicate the SS that the new
shard is empty.
2021-09-09 14:15:07 -07:00
helium 8e0b572a18 Added comments. 2021-09-02 22:29:07 -07:00
helium 7e7f9372de Replaced clear() with krmSetRangeCoalescing(). 2021-09-02 21:58:05 -07:00
helium a8a6f6b8a3 Remove Transaction::set() when clearing the keyrange. 2021-09-01 15:35:40 -07:00
helium 6612cc00b6 Check if the src server list will be empty before removing a failed server." 2021-09-01 14:52:07 -07:00
Evan Tschannen 0a78c84a3e
Merge pull request #5183 from sfc-gh-xwang/tlog_dev
TLog Streaming Peek
2021-08-17 11:51:34 -07:00
Josh Slocum e444d3781c Various TSS improvements from snowblower testing 2021-08-13 10:24:15 -05:00
Xiaoxi Wang 2a88033800 clean 100k simulation test. revert changes of fdbrpc.h 2021-07-31 16:46:14 -07:00
Steve Atherton 507c1f11e3 Add .log() to bare TraceEvent() invocations without any .detail()s to avoid clang-tidy warning about immediate destruction of object without use. 2021-07-26 19:55:10 -07:00
Steve Atherton f596a81073 Rename ::TRUE and ::FALSE in BooleanParams to ::True and ::False so as to not conflict with the TRUE and FALSE macros provided by the Windows and MacOS SDKs. 2021-07-17 00:11:40 -07:00
sfc-gh-tclinkenbeard 8cc40e3a2b Expand use of BOOLEAN_PARAM 2021-07-02 21:41:50 -07:00
Neethu Haneesha Bingi decfb610ff Minor review comments. 2021-06-25 15:04:49 -07:00
Neethu Haneesha Bingi 66f2518405 exclude to work with any locality data match. 2021-06-23 18:03:27 -07:00
Neethu Haneesha Bingi 4ad5926a25 Snake naming of keys and added comments to all new functions. 2021-06-23 18:03:27 -07:00
Neethu Haneesha Bingi 73752f441b exclude locality:clang-format, ranged loops, documentation, tracking addStoragesever for exclusion. 2021-06-23 18:03:27 -07:00
Josh Slocum db04b65ae1 Adding comments and a quarantine case I forgot 2021-06-11 10:42:33 -05:00
Josh Slocum ac209b32fd Addressing review comments 2021-06-03 15:31:16 +00:00
Josh Slocum b3e4f182ef TSS Mapping Change 2021-06-02 17:30:09 +00:00