Bug behavior:
When DD has zero healthy machine teams but more unhealthy machine teams
than the max machine teams DD plans to build, DD will stop building
new machine teams. Due to zero healthy machine team (and zero healthy
server team), DD cannot find a healthy destination team to relocate data.
When data relocation stops, exclusion stops progressing and stuck.
Bug happens when we *shrink* a k-host cluster by
first adding k/2 new host;
then quickly excluding all old hosts.
Fix:
Let DD build temporary extra teams to relocate data.
The extra teams will be cleaned up later by DD's remove extra teams logic.
Simulation test:
There is no simulation test to cover cluster expansion scnenario.
To most closely simulate this behavior, we intentionally overbuild all possible
machine teams to trigger the condition that unhealthy teams is larger than
the maximum teams DD wants to build later.
If a log directory is specified in the "db_log_dir" option, RocksDB will add the whole file path in the log file name, which can exceed the 255 byte limit of Linux file system, causing storage server failures.
See
c3f720c60d/include/rocksdb/options.h (L674-L679)
* Allow multiple keyranges in CheckpointRequest.
Include DataMove ID in CheckpointMetaData.
* Use UID dataMoveId instead of Optional<UID>.
* Implemented ShardedRocks::checkpoint().
* Implementing createCheckpoint().
* Attempted to change getCheckpointMetaData*() for a single keyrange.
* Added getCheckpointMetaDataForRange.
* Minor fixes for NativeAPI.actor.cpp.
* Replace UID CheckpointMetaData::ssId with std::vector<UID>
CheckpointMetaData::src;
* Implemented getCheckpointMetaData() and completed checkpoint creation
and fetch in test.
* Refactoring CheckpointRequest and CheckpointMetaData
rename `dataMoveId` as `actionId` and make it Optional.
* Fixed ctor of CheckpointMetaData.
* Implemented ShardedRocksDB::restore().
* Tested checkpoint restore, and added range check for restore, so that
the target ranges can be a subset of the checkpoint ranges.
* Added test to partially restore a checkpoint.
* Refactor: added checkpointRestore().
* Sort ranges for comparison.
* Cleanups.
* Check restore ranges are empty; Add ranges in main thread.
* Resolved comments.
* Fixed GetCheckpointMetaData range check issue.
* Refactor CheckpointReader for CF checkpoint.
* Added CheckpointAsKeyValues as a parameter for newCheckpointReader.
* PhysicalShard::restoreKvs().
* Added `ranges` in fetchCheckpoint.
* Added RocksDBCheckpointKeyValues::ranges.
* Added ICheckpointIterator and implemented for RocksDBCheckpointReader.
* Refactored OpenAction for CheckpointReader, handled failure cases.
* Use RocksDBCheckpointIterator::end() in readRange.
* Set CheckpointReader timout and other Rocks read options.
* Implementing fetchCheckpointRange().
* Added more CheckpointReader tests.
* Cleanup.
* More cleanup.
* Resolved comments.
Co-authored-by: He Liu <heliu@apple.com>
* Add cleanIdempotencyIds
Delete zero or more idempotency ids older than minAgeSeconds
* Automatically clean idempotency ids from first proxy
* Add test for cleaner
* Fix formatting
* Address review comments
Xiaoxi Wang