Commit Graph

26 Commits

Author SHA1 Message Date
Xiaoxi Wang 0d4b4d05e2 implement MSS as IStorageMetricsService and pass the unit test 2022-10-24 09:58:41 -07:00
Zhe Wu ada51ff12b Add a counter to track physical shard creation throuogh moves 2022-10-21 10:22:42 -07:00
Markus Pilman ea1325a552
Merge pull request #8319 from sfc-gh-tclinkenbeard/add-rare-code-probe-annotation
Add `rare` code probe decoration
2022-10-07 09:39:00 -06:00
Xiaoxi Wang 3578832c69 merge upstream/main 2022-09-28 14:01:50 -07:00
Xiaoxi Wang f2dea33d08 add global var transactionRetryableErrors 2022-09-27 18:14:34 -07:00
Xiaoxi Wang df9b21169d change shared_ptr to Reference 2022-09-27 11:22:47 -07:00
sfc-gh-tclinkenbeard 985958c260 Add rare code probe decoration 2022-09-25 15:28:32 -07:00
Xiaoxi Wang 1194774d54 rename dbProcessor to db; rename getDb() to context() 2022-09-23 15:35:39 -07:00
Xiaoxi Wang 08f4c06450 remove unnecessary creation of transactions; call all cx and transaction methods in DDShardTracker through txnProcessor 2022-09-19 14:41:34 -07:00
Xiaoxi Wang 7dd1d7a029 solve merge problem; enable an empty top k unit tests 2022-09-14 14:13:54 -07:00
Xiaoxi Wang 642f8f3a9f format code 2022-09-14 12:46:27 -07:00
Xiaoxi Wang 85f119c13b fix destination team issue 2022-09-14 00:17:00 -07:00
Xiaoxi Wang f50be2309f change FetchTopK unit test 2022-09-12 15:16:11 -07:00
Xiaoxi Wang 806d0a8599 merge upstream/main; solve conflicts 2022-09-12 12:34:35 -07:00
Bharadwaj V.R a30c1e2b9c add code probe for when shard stats are unavailable to inform split 2022-09-08 11:58:44 -07:00
Bharadwaj V.R 46ce06a876 Add code probes for tenant-based shard splitting 2022-09-07 10:53:16 -07:00
Bharadwaj V.R beb00b4b7c
Split shards along tenant keyspace boundaries (disabled by default) (#7432)
* Boilerplate code for tenant-aware DD

* Add an DD tenant-cache-assembly actor

* Add basic tenant list monitoring for tenant cache. Tenant cache is not yet a DD-scoped structure

* Create a DDTenantCache class

* Move DDTenantCache to new files

* Tweak some traces; fix placement of build and monitor actor invocations

* Update DD tenant cache refresh to be more efficient and unit-testable

* Fix incorrect use of prefixToID in the DDTenantCache code; it is not correct when the tenant is within a tenant subspace

* Fix a faulty assertion in a DDTenantCache unit test, and add a server knob to switch of DDTenantCache code for the moment

* Fix a DDTenantCache UT bug where count of tenants could be 0, causing divide-by-0 exceptions in the test code; Fix accidental movement of atrace statement; rename some DD tenant-cache related variables

* Make some member DDTenantCache member functions const, and change the ddtc abbreviation to ddTenantCache, for ddtc could refer to DDTeamCollection

* Remove unnecessary assertion that erase-ing an element from a Map really removes it

* Remove the DD prefix in the tenant cache class name (and associated impl and UT class names); there is nothing specific to DD in it; DD uses it; other modules may use it in the future

* Disable DD tenant awareness by default

* Split shards along tenant keyspace boundaries

* Update some naming, switch to using prefixRange and fix a bug in tenantShardSplit

* Reorganize tenant shard splitter

* Moved new header file to new include dir under fdbserver

* Make shardEvaluator trigger shard merger only if adjacent shards belong to the same tenant, if tenant-keyspace boundaries are being honored

* Fix issues in the way shard merge decisions are made based on tenant boundaries

* Minor reorg and bug fix in shard merger decision-making

* Resolve merge issues

* Fix bug in ShardEvaluator that made it run in a loop rather than waiting for metrics changes

* Fix tenant splitter for when start and end are in different shards and only one of them needs to be split

* Remove some uses of printable

* Remove uses of printable

* Fix code formatting

* Add a TENANT_SPLIT RelocateReason and fix errors from incorrect merge of upstream code

* Move tenantCreationSignal ownership to TenantCache

* Update fdbserver/DataDistributionTracker.actor.cpp

Co-authored-by: Xiaoxi Wang <xiaoxi.wang@snowflake.com>

* Fix a couple of bugs and spelling errors in tenantShardSplitter

* Update fdbserver/DataDistributionTracker.actor.cpp

Co-authored-by: Xiaoxi Wang <xiaoxi.wang@snowflake.com>

* Refactor split-merge feasibility check methods

* Merge upstream changes

* Break down tenantShardSplitter to remove dependency on DataDistributionTracker and make the splitter unit-testable

* Add unit tests for tenant shard splitting

* Factor out some repeated code when finding tenant boundaries within a shard

* Extract fault line verification steps used in UTs into a separate function

Co-authored-by: Xiaoxi Wang <xiaoxi.wang@snowflake.com>
2022-09-06 10:53:02 -07:00
Xiaoxi Wang e7623faf25
Merge pull request #8087 from sfc-gh-xwang/main
Prevent GetTopKMetricsRequest.topK < 1
2022-09-02 13:46:29 -07:00
Xiaoxi Wang 7932840eaf Prevent GetTopKMetricsRequest.topK < 1 2022-09-02 12:20:24 -07:00
Bharadwaj V.R d3d1ff6023
Test maxShardSize being exceeded when terminating shard merge search (#8045)
* Test maxShardSize being exceeded when terminating shard merge

* Fix formatting in DDShardTracker.actor.cpp
2022-09-01 15:44:49 -07:00
Xiaoxi Wang 8c6af958f0 change the DDTracker parameters back 2022-08-29 14:58:55 -07:00
Xiaoxi Wang 5c2b846fbd solve conflicts in upstream/main 2022-08-19 15:36:17 -07:00
Zhe Wang 2ceaae4219
dd physical shard core (#7703) 2022-08-19 14:47:00 -04:00
Xiaoxi Wang f83493355e format code 2022-08-16 23:37:55 -07:00
Xiaoxi Wang 0a15f18fa4 provide default destructor; fix other compilation problem 2022-08-16 18:28:36 -07:00
Xiaoxi Wang e4e6cfc59f rename file 2022-08-16 14:35:19 -07:00