Commit Graph

22048 Commits

Author SHA1 Message Date
A.J. Beamon 0c7f6a8e0c Add a restoring state for data clusters and a ID -> tenant name index to allow ID lookups. Also add a couple missing tenant state conversions and a missing encrypted check in TenantMapEntry. 2022-08-13 12:49:54 -07:00
Marian Dvorsky 7046218f35 Fix uninitialized bool values in GlobalTagThrottler 2022-08-13 18:57:43 +02:00
Marian Dvorsky 8e6009427e Fix Net2 unittest flakiness 2022-08-13 18:51:30 +02:00
Chaoguang Lin a27d27c5ee
Add traces for snapshot related updates (#7862)
* Add logging; fix typos in comments;

* format files
2022-08-13 03:10:20 -04:00
Xiaoxi Wang f103a10c37
Merge pull request #7871 from sfc-gh-xwang/feature/main/wiggler-tss
Make storage wiggler wiggle actively when we need more TSS pair to be recruited
2022-08-12 16:39:20 -07:00
Xiaoxi Wang dab8bcd109 Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/wiggler-tss 2022-08-12 15:27:50 -07:00
Josh Slocum bbbaa80e52
assorted bug fixes for blob granules (#7866)
* assorted bug fixes for blob granules

* Fixing transaction used after commit in blob manager recovery

* fixing race with granule merging across hard boundaries because it hadn't loaded them yet
2022-08-12 17:26:43 -05:00
Xiaoxi Wang 9133d4e16d
Merge pull request #7803 from sfc-gh-xwang/feature/main/ddvisibility
Add server selection counter in DDQueue
2022-08-12 15:10:25 -07:00
Xiaoxi Wang b131dc9692 make getNextWigglingServerID() consider TSS recruitment;add unittest 2022-08-12 13:44:29 -07:00
Xiaoxi Wang e554cc1443 format code 2022-08-12 13:42:31 -07:00
Xiaoxi Wang 17ddd452aa
Update fdbclient/include/fdbclient/ServerKnobs.h
Co-authored-by: Bharadwaj V.R <bharadwaj.vr@snowflake.com>
2022-08-12 13:36:59 -07:00
Xiaoxi Wang 2d94128140
Apply suggestions from code review
Co-authored-by: Bharadwaj V.R <bharadwaj.vr@snowflake.com>
2022-08-12 13:33:12 -07:00
Jingyu Zhou 92cae85fa8 Increase waitForLowInFlight timeout
Seeing a CheckRelocation test failed with 300s timeout.
2022-08-12 13:19:56 -07:00
Evan Tschannen a9d3c9f9b3
Added throttling when a blob worker falls behind (#7751)
* throttle the cluster when blob workers fall behind

* do not throttle on blob workers if they are not enabled

* remove an unnecessary actor

* fixed a compile error

* fetch blob worker metrics at the same interval as the rate is updated, avoid fetching the complete blob worker list too frequently

* fixed another compilation bug

* added a 5 second delay before bw throttling to prevent false positives caused by the 100e6 version jump during recovery. Lower the throttling thresholds to react much quicker to bw lag.

* fixed a number of problems

* changed the minBlobVersionRequest to look at storage server versions since this will be a lot more efficient

* fix: do not let desired go backwards

* fix: track the version of notAtLatest changefeeds for throttling

* ratekeeper now throttled blob workers by estimating the transaction per second throughput of the blob workers

* added metrics for blob worker change feeds

* added a knob to disable bw throttling

* fixed the transaction options in blob manager
2022-08-12 13:15:56 -07:00
Jingyu Zhou 8fb6d59e94 Use error_code_grv_proxy_memory_limit_exceeded
instead of error_code_proxy_memory_limit_exceeded
2022-08-12 11:36:17 -07:00
Jingyu Zhou f20936c90d Fix attribute_not_found error in bulkSetup() 2022-08-12 11:13:32 -07:00
Jingyu Zhou ed1b7ef173 Add a client knob GRV_ERROR_RETRY_DELAY 2022-08-12 11:13:32 -07:00
Jingyu Zhou f912a74ec1 Fix TenantManagementWorkload.actor.cpp 2022-08-12 11:13:32 -07:00
Jingyu Zhou 6122cb6acd Fix LocalRatekeeper workload 2022-08-12 11:13:32 -07:00
Jingyu Zhou 0b735b6b11 Fix SpecialKeySpaceCorrectness workload 2022-08-12 11:13:32 -07:00
Jingyu Zhou 4497fe4ccf Fix FuzzApiCorrectness workload 2022-08-12 11:13:32 -07:00
Jingyu Zhou 609e29f4c0 Fix backup worker to handle GRV errors 2022-08-12 11:13:32 -07:00
Jingyu Zhou e5da35d7bf Fix TenantManagementWorkload with GRV errors 2022-08-12 11:13:32 -07:00
Jingyu Zhou 317866fbbf Add retry in waitForCommittedVersion for GRV error response 2022-08-12 11:13:32 -07:00
Jingyu Zhou 970105f318 Client needs to handle proxy_memory_limit_exceeded error 2022-08-12 11:13:32 -07:00
Jingyu Zhou a32822a51d Add fault injection to GRV proxy to return error response
This turns out to fail a lot of tests that we need to fix.
2022-08-12 11:13:32 -07:00
Xiaoxi Wang 8b9684ae40 Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/wiggler-tss 2022-08-12 10:48:37 -07:00
Xiaoxi Wang 860ffbc51e getTargetTSSInDC() method 2022-08-12 10:47:39 -07:00
Marian Dvorsky 35a3a33d1c
Resolve TSAN-related issues in ctests (#7864) 2022-08-12 19:23:32 +02:00
Jingyu Zhou aa29de8d4c
Merge pull request #7838 from xis19/proxyerror
Split the proxy_memory_limit_exceeded to commit/grv version
2022-08-12 10:16:06 -07:00
Markus Pilman bca5ec780a
Merge pull request #7867 from sfc-gh-jslocum/bg_authz
Authorization for blob granule file request
2022-08-12 09:41:10 -06:00
A.J. Beamon 67646aa4bb
Merge pull request #7682 from sfc-gh-jshim/fix-skewed-u32-out-of-range
Fix incorrect range in randomSkewedUInt32()
2022-08-12 06:10:48 -07:00
Josh Slocum e470e64324 authz for blob granule file request 2022-08-12 08:03:58 -05:00
Junhyun Shim 3cf0914ce0 Fix incorrect parenthesis 2022-08-12 13:52:50 +02:00
Junhyun Shim 312d3da040 Merge remote-tracking branch 'origin/main' into fix-skewed-u32-out-of-range 2022-08-12 13:01:43 +02:00
Junhyun Shim e57d3a6f9b Address review comments 2022-08-12 10:24:23 +02:00
Xiaoge Su 0326d53965 Split proxy_memory_limit_exceeded to commit/grv specific exceptions
Currently GRV is reporting proxy_memory_limit_exceeded error which has
error message claiming Commit proxy failing. This split should remove
such confusion.
2022-08-12 00:45:57 -07:00
Xiaoxi Wang 4a552f1fb9
Merge pull request #7829 from sfc-gh-xwang/fix/main/AsanOOM
Fix heap-use-after-free in storage wiggler unit test
2022-08-11 23:19:48 -07:00
Jingyu Zhou 5929ac1d65
Merge pull request #7847 from xis19/knobCheck
Cleanup the knobs that are not being used
2022-08-11 21:52:52 -07:00
Josh Slocum 7c155f4521
Granule force purging (#7846)
* Granule purge cannot delete history entry for fully deleting granule until all children are completely done splitting

* Several purging fixes related to granule history

* Fixed typo in refactor

* fixing memory model for purgeRange

* formatting

* weakening granule purge test for now

* cleanup

* First version of force purging granules

* fixing issue in BW range assignment reporting

* Fixing incorrect assert with force purging

* Error handling when checking force purged state

* fixed force purging and recover/reassign range races and check

* Handling force purge + boundary change race

* more places to check for force purged status

* fixed manager restart in the middle of force purge bug

* fixing same-BM purge and assignment races in all cases

* weakening orphaned granule history check a bit because of difficult to solve races

* fixing txn options on retry

* loading force purged ranges at start to avoid resuming a merge that is being force purged

* cleanup

* Enabling purging in granule tests, and adding check for leaked change feeds in force purge

* formatting

* missed parameter in merge conflicts

* Fixing leaked change feed race with merge and force purge

* adding change feed cleanup when new blob manager recovers in-progress merge that raced with force purge

* added forcepurge fdbcli command
2022-08-11 15:22:32 -07:00
Xiaoge Su 114c266b04 fixup! Recover the KMS knob which is useful 2022-08-11 15:15:14 -07:00
Yao Xiao 599e4b86d5 Add more knobs 2022-08-11 15:13:42 -07:00
Josh Slocum 44f8bdd258
Blob Worker memory limit (#7858)
* Simulation version of blob_worker_full

* tracking blocked BM assignments

* actual memory estimation implementation
2022-08-11 15:07:08 -07:00
Trevor Clinkenbeard 583021c2d9
Merge pull request #7772 from sfc-gh-tclinkenbeard/global-tag-throttling6
Add status section for global tag throttler
2022-08-11 17:38:31 -03:00
Xiaoxi Wang 29fa83735a format code 2022-08-11 12:33:04 -07:00
Xiaoxi Wang 4819707af3 solve invalid detail name 2022-08-11 12:28:18 -07:00
Xiaoxi Wang fe45b4f0b2 update document 2022-08-11 12:14:24 -07:00
Xiaoxi Wang d724fc1be2 add enum count; filter out zero reason item 2022-08-11 11:59:46 -07:00
Yao Xiao fcfe6a8c29 Fix metrics and add knobs 2022-08-11 11:46:21 -07:00
Xiaoxi Wang 7689992113 shouldSplit = sizeSplit || writeSplit; 2022-08-11 11:39:53 -07:00