Commit Graph

879 Commits

Author SHA1 Message Date
He Liu 7bb823edbe
Replace KeyRange with std::vector<KeyRange> in DataMoveMetaData and (#8591)
* Replace KeyRange with std::vector<KeyRange> in DataMoveMetaData and
CheckpointMetaData.

* Checked if ranges.empty().

* fmt.

* Resolved some comments.

Co-authored-by: He Liu <heliu@apple.com>
2022-10-28 15:22:55 -07:00
Xiaoxi Wang bb0236433c
Merge pull request #8540 from sfc-gh-xwang/feature/main/storageMetrics
Make MockStorageServer serve StorageMetrics related request
2022-10-25 17:29:21 -07:00
Xiaoxi Wang 5a8adca1f7 solve review comments: mark const; add comments; template abbreviation 2022-10-25 10:56:24 -07:00
Ankita Kejriwal 7647cea4e5 Improve the storage quota monitor code + add a knob for refresh interval 2022-10-21 15:44:13 -07:00
Ankita Kejriwal a8b1154e45 Move the storage quota monitor from DataDistributor to TenantCache 2022-10-21 15:17:41 -07:00
Ankita Kejriwal fb014a4834 Merge branch 'main' of github.com:apple/foundationdb into monitorusage 2022-10-18 13:14:47 -07:00
Xiaoxi Wang 5d57429949 modify boolean definition; add comments 2022-10-17 10:12:00 -07:00
Xiaoxi Wang 83685e6a61 add status verification; add skipDDModeCheck in getInitialDataDistribution for workload test 2022-10-15 22:52:42 -07:00
He Liu 88c7304e04 Resolved comments. 2022-10-11 10:24:31 -07:00
He Liu b52edd8658 Merge branch 'main' of https://github.com/apple/foundationdb into validate-data-consistency 2022-10-10 11:00:05 -07:00
He Liu 1931e3266f Resolve comments. 2022-10-10 10:28:19 -07:00
Jingyu Zhou 63f5705560
Merge pull request #8414 from sfc-gh-xwang/feature/main/txnProcessor_team
Replace self->cx with self->dbContext() method and trivial renaming
2022-10-10 10:09:36 -07:00
Markus Pilman ea1325a552
Merge pull request #8319 from sfc-gh-tclinkenbeard/add-rare-code-probe-annotation
Add `rare` code probe decoration
2022-10-07 09:39:00 -06:00
Xiaoxi Wang 2ad4f29539 dbContext() replace self->cx, remove cx member 2022-10-04 16:39:22 -07:00
Xiaoxi Wang df9b21169d change shared_ptr to Reference 2022-09-27 11:22:47 -07:00
sfc-gh-tclinkenbeard 985958c260 Add rare code probe decoration 2022-09-25 15:28:32 -07:00
Xiaoxi Wang d9cc7ff44e merge upstream/main 2022-09-23 14:49:46 -07:00
Xiaoxi Wang f9e0230b86 DDQueue constructor with ITxnProcessor 2022-09-21 10:56:22 -07:00
Xiaoxi Wang 97fd5878d9 change DDTeamCollection constructor 2022-09-20 13:00:28 -07:00
Xiaoxi Wang 08f4c06450 remove unnecessary creation of transactions; call all cx and transaction methods in DDShardTracker through txnProcessor 2022-09-19 14:41:34 -07:00
He Liu 0e69b19a6f Added AuditUtils.actor.h 2022-09-19 12:15:49 -07:00
A.J. Beamon 4fd64630e8 Convert literal string ref instances to use _sr suffix 2022-09-19 11:35:58 -07:00
He Liu 28e5a70dbe Clean up SS validate storage. 2022-09-14 10:53:54 -07:00
He Liu 958b28497e Merge branch 'main' of https://github.com/apple/foundationdb into validate-data-consistency 2022-09-13 13:55:01 -07:00
He Liu 18cdef43be Cleanup. 2022-09-13 13:52:08 -07:00
He Liu 092aee8bf2 Introduced `DDAuditStorage` to encapsulate DDAudit. 2022-09-13 13:25:33 -07:00
He Liu ebabe916f1 Cleanup. 2022-09-13 09:28:41 -07:00
He Liu e6df139204 Get sources servers by DC in DDAuditRange. 2022-09-12 11:04:23 -07:00
He Liu e6846e1ed5 Handle auditStorage API errors. 2022-09-09 15:04:34 -07:00
He Liu 071ed41caa ManagementAPI auditStorage passed test. 2022-09-07 22:34:14 -07:00
He Liu ad11c6e82d Cleanup. 2022-09-06 19:00:09 -07:00
Bharadwaj V.R beb00b4b7c
Split shards along tenant keyspace boundaries (disabled by default) (#7432)
* Boilerplate code for tenant-aware DD

* Add an DD tenant-cache-assembly actor

* Add basic tenant list monitoring for tenant cache. Tenant cache is not yet a DD-scoped structure

* Create a DDTenantCache class

* Move DDTenantCache to new files

* Tweak some traces; fix placement of build and monitor actor invocations

* Update DD tenant cache refresh to be more efficient and unit-testable

* Fix incorrect use of prefixToID in the DDTenantCache code; it is not correct when the tenant is within a tenant subspace

* Fix a faulty assertion in a DDTenantCache unit test, and add a server knob to switch of DDTenantCache code for the moment

* Fix a DDTenantCache UT bug where count of tenants could be 0, causing divide-by-0 exceptions in the test code; Fix accidental movement of atrace statement; rename some DD tenant-cache related variables

* Make some member DDTenantCache member functions const, and change the ddtc abbreviation to ddTenantCache, for ddtc could refer to DDTeamCollection

* Remove unnecessary assertion that erase-ing an element from a Map really removes it

* Remove the DD prefix in the tenant cache class name (and associated impl and UT class names); there is nothing specific to DD in it; DD uses it; other modules may use it in the future

* Disable DD tenant awareness by default

* Split shards along tenant keyspace boundaries

* Update some naming, switch to using prefixRange and fix a bug in tenantShardSplit

* Reorganize tenant shard splitter

* Moved new header file to new include dir under fdbserver

* Make shardEvaluator trigger shard merger only if adjacent shards belong to the same tenant, if tenant-keyspace boundaries are being honored

* Fix issues in the way shard merge decisions are made based on tenant boundaries

* Minor reorg and bug fix in shard merger decision-making

* Resolve merge issues

* Fix bug in ShardEvaluator that made it run in a loop rather than waiting for metrics changes

* Fix tenant splitter for when start and end are in different shards and only one of them needs to be split

* Remove some uses of printable

* Remove uses of printable

* Fix code formatting

* Add a TENANT_SPLIT RelocateReason and fix errors from incorrect merge of upstream code

* Move tenantCreationSignal ownership to TenantCache

* Update fdbserver/DataDistributionTracker.actor.cpp

Co-authored-by: Xiaoxi Wang <xiaoxi.wang@snowflake.com>

* Fix a couple of bugs and spelling errors in tenantShardSplitter

* Update fdbserver/DataDistributionTracker.actor.cpp

Co-authored-by: Xiaoxi Wang <xiaoxi.wang@snowflake.com>

* Refactor split-merge feasibility check methods

* Merge upstream changes

* Break down tenantShardSplitter to remove dependency on DataDistributionTracker and make the splitter unit-testable

* Add unit tests for tenant shard splitting

* Factor out some repeated code when finding tenant boundaries within a shard

* Extract fault line verification steps used in UTs into a separate function

Co-authored-by: Xiaoxi Wang <xiaoxi.wang@snowflake.com>
2022-09-06 10:53:02 -07:00
He Liu 6f4d09f947 Replace Reference<> with std::shared_ptr<>. 2022-09-05 12:45:14 -07:00
He Liu 68ea326738 Read local data with getKeyValuesQ(). 2022-09-02 12:44:06 -07:00
He Liu 033741daab Audit should always complete, any failures are retried. 2022-09-02 09:11:19 -07:00
He Liu cade0baf7e Added Interfaces in ClusterInterface and DataDistributorInterface for
audit.
2022-09-01 11:03:34 -07:00
Xiaoxi Wang 0b6ef10f3a
Merge pull request #7926 from sfc-gh-xwang/refactor/main/portEnableState
[DD Testability] use txnProcessor to call removeKeysFromFailedServer and removeStorageServer
2022-08-19 13:27:57 -07:00
Zhe Wang 2ceaae4219
dd physical shard core (#7703) 2022-08-19 14:47:00 -04:00
Xiaoxi Wang 26ba294fe5 solve merge conflict with upstream/main 2022-08-19 09:22:06 -07:00
Xiaoxi Wang 1aeb7b2953 call removeStorageServer and removeKeysFromFailedServer from txnProcessor 2022-08-18 11:44:27 -07:00
Ankita Kejriwal ff6fe48909
Merge branch 'main' into tenantquota 2022-08-18 10:59:00 -07:00
Xiaoxi Wang 52e7825179 move DataDistributionEnabled to DDTxnProcessor 2022-08-18 10:23:33 -07:00
Xiaoxi Wang 98096349e3 use ddEnabledState in DDSharedContext 2022-08-18 09:50:32 -07:00
Ankita Kejriwal e3ecba105e Monitor storage bytes used by tenants 2022-08-17 14:58:09 -07:00
Xiaoxi Wang e2e5abbc9d split DD related headers 2022-08-16 14:32:55 -07:00
Xiaoxi Wang 1a7f6090b9
Merge branch 'main' into refactor/main/enableState 2022-08-15 10:55:20 -07:00
Chaoguang Lin a27d27c5ee
Add traces for snapshot related updates (#7862)
* Add logging; fix typos in comments;

* format files
2022-08-13 03:10:20 -04:00
Xiaoxi Wang b131dc9692 make getNextWigglingServerID() consider TSS recruitment;add unittest 2022-08-12 13:44:29 -07:00
Xiaoxi Wang 7a11818d6d merge upstream/main 2022-08-09 14:00:33 -07:00
Xiaoxi Wang 85916e26f4 remove pqCanCheck because it's unnecessary 2022-08-09 10:28:02 -07:00