a78d800e8a
Clean up data move tombstone when DD init ( #9901 )
...
* init
* move cleanup out of getInitDD
* fmt
* address comments
* fixes
2023-04-06 13:48:18 -07:00
8102ac9a41
Serialize concurrent datamoves and their cleanups ( #9421 )
...
* retry when concurrent dm cleanups happens
* cleanup
* fix conflict data move and cleanup
* clean
* make code looks better
* use background cleanup
* fmt
* unset StartMoveShardsFoundConflictingDataMove error
* address comments
2023-04-04 11:38:24 -07:00
64b6a5d257
Allow boolean parameters to be nested inside of namespaces or classes
2023-03-30 15:09:59 -07:00
ac54962533
code cleanup
2023-03-16 09:47:21 -07:00
aaf7b9b32b
Added the ability to manually create a shard and also increase its replication factor
2023-03-15 11:26:15 -07:00
00203c8732
Validate Storage part II ( #8471 )
...
* Implemented AuditUtils.actor.cpp
Moved AuditUtils to fdbserver/
* Persist AuditStorageState.
* Passed persisted AuditStorageState test.
* Added audit_storage_error to indicate a corruption is caught.
Throw/Send audit_storage_error when there is a data corruption.
Added doAuditStorage() for resuming Audit.
* Load and resume AuditStorage when DD restarts.
* Generate audit id monotonically.
* Fixed minor issue AuditId/Type was not set.
* Adding getLatestAuditStates.
* Improved persisted errors and added AuditStorageCommand.actor.cpp for
fdbcli.
* Added `audit_storage` fdbcli command.
* fmt.
* Fixed null shared_ptr issue.
* Improve audit data.
* Change DDAuditFailed to SevWarn.
* Sev.
* set SERVE_AUDIT_STORAGE_PARALLELISM to 1.
* Moved AuditUtils* to fdbclient/.
* Added getAuditStatus fdbcli command.
* Refactor audit storage fdb cli commands.
* Added auditStorage in sim.
* Cleanup.
* Resolved comments.
* Resolved comments.
* Test disabling audit for sims.
* Cleanup.
Co-authored-by: He Liu <heliu@apple.com>
2023-01-15 21:46:14 -08:00
5f51129834
Fix other call site using new MoveKeysParams
2022-11-29 13:29:02 -08:00
e3410268b5
Make finishMoveShards and DDMockTxnProcessor work with list of key ranges for data movement
2022-11-29 13:29:02 -08:00
8971b5907c
add comments; mark some methods const
2022-11-16 13:18:40 -08:00
f997e73758
rename variable and solve some light comments
2022-11-14 13:11:27 -08:00
b79268326a
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/dataApi
2022-11-11 08:29:22 -08:00
7c9334121a
mark MoveKeysParams const&
2022-11-10 12:51:22 -08:00
62b88a0772
wait on lock take
2022-11-09 11:34:47 -08:00
fd425db1cf
fix rawStartMovement bugs for merge shard; change the test toml file name
2022-11-08 23:37:58 -08:00
96cf3f855b
add rawMoveShard function
2022-11-07 16:47:14 -08:00
9332a155d5
Merge pull request #8678 from sfc-gh-xwang/fix/dd/priority
...
Add immediate priority and read_lock options to DD transactions
2022-11-07 10:08:11 -08:00
56001de2d4
More nit changes around DD
2022-11-07 09:11:16 -08:00
3a02f919b9
Add some comments in DD and fix some nit
2022-11-07 09:11:16 -08:00
cc2225c18e
fix update txn options
2022-11-04 11:34:09 -07:00
aded047db1
add priority and read_lock options
2022-11-03 15:08:12 -07:00
a0489330d0
fix rawStartMovement defineShard bug
2022-11-01 14:56:55 -07:00
0cbd1dfcca
add comment and MockDDTest base class file
2022-11-01 14:56:55 -07:00
334fced572
add data api implementations; add more realistic fetchKey implementation; finish randomKeyBetween implementation
2022-11-01 14:56:55 -07:00
7bb823edbe
Replace KeyRange with std::vector<KeyRange> in DataMoveMetaData and ( #8591 )
...
* Replace KeyRange with std::vector<KeyRange> in DataMoveMetaData and
CheckpointMetaData.
* Checked if ranges.empty().
* fmt.
* Resolved some comments.
Co-authored-by: He Liu <heliu@apple.com>
2022-10-28 15:22:55 -07:00
9d3c3b1efe
Remove cluster ID logic from individual roles
...
The logic to determine the validity of a process joining a cluster now
belongs on the worker and the cluster controller. It is no longer
restricted to tlogs and storages, but instead applies to all processes
(even stateless ones).
2022-10-27 13:56:13 -07:00
36d9de9072
change UNREACHABLE to ASSERT(false); change function name
2022-10-25 15:43:24 -07:00
e07a50573a
splitStorageMetrics finish implementation (no unit test yet but 100k
...
test pass)
2022-10-24 09:58:41 -07:00
5d90703dc8
finish getKeysLocations etc, and unit test pass.
2022-10-24 09:58:41 -07:00
70eb9aef28
simple MGS IKeyLocationService implementation
2022-10-24 09:58:41 -07:00
a8391caf23
Revert "Data loss protection v2"
2022-10-20 18:09:58 -05:00
72bc89cf39
Remove cluster ID logic from individual roles
...
The logic to determine the validity of a process joining a cluster now
belongs on the worker and the cluster controller. It is no longer
restricted to tlogs and storages, but instead applies to all processes
(even stateless ones).
2022-10-18 21:37:42 -07:00
08b006efba
solve compile warning
2022-10-18 09:49:07 -07:00
6630ec8c1d
remove virtual on raw* Methods
2022-10-17 16:43:32 -07:00
9c28dd46be
add assert to parallelism lock take
2022-10-17 14:42:04 -07:00
5d57429949
modify boolean definition; add comments
2022-10-17 10:12:00 -07:00
313c76232c
skip shard size validity
2022-10-15 22:52:43 -07:00
83685e6a61
add status verification; add skipDDModeCheck in getInitialDataDistribution for workload test
2022-10-15 22:52:42 -07:00
4fa8d3e8ef
finish mock MoveKeys implementation (not tested yet)
2022-10-15 22:52:42 -07:00
00c19ec9ed
expose rawStartMovement and rawFinishMovement
2022-10-15 22:52:42 -07:00
e959bdef8d
extract startMoveKeysTransaction method
2022-10-15 22:52:42 -07:00
b52edd8658
Merge branch 'main' of https://github.com/apple/foundationdb into validate-data-consistency
2022-10-10 11:00:05 -07:00
1931e3266f
Resolve comments.
2022-10-10 10:28:19 -07:00
21b2e11bc4
getWorkers from IDDTxnProcessor
2022-10-04 14:57:04 -07:00
4cf4ccc089
correct getServerListAndProcessClasses implementation (100k pass)
2022-10-03 22:24:35 -07:00
76f2dc8ce0
merge upstream/main
2022-10-02 22:07:42 -07:00
1cc46a98f8
solve merge conflicts
2022-09-28 14:34:42 -07:00
3578832c69
merge upstream/main
2022-09-28 14:01:50 -07:00
e476e814b2
merge upstream/main
2022-09-27 10:15:26 -07:00
14d73193d5
waitDDTeamInfoPrintSignal, getClusterId, tryUpdateReplicasKeyForDc in IDDTxnProcessor
2022-09-26 23:00:31 -07:00
d9cc7ff44e
merge upstream/main
2022-09-23 14:49:46 -07:00
Zhe Wang