Commit Graph

276 Commits

Author SHA1 Message Date
sfc-gh-tclinkenbeard 9df990e375 Remove global_tag_throttler status section 2022-08-29 23:17:20 -07:00
A.J. Beamon 2907d2d4dd
Merge pull request #8004 from sfc-gh-ajbeamon/fix-ub
Fix some undefined bevavior in RK and a unit test
2022-08-29 09:16:11 -07:00
Evan Tschannen 8314e80371
Fixed a few bugs which caused ratekeeper to unnecessarily throttle a cluster (#8006)
* do not count recently created change feeds for throttling

* fix: blocked assignments were not decremented when force purging

* fix: created needs to be updated when the changefeed is reset

* added asserts to detect if ratekeeper is throttled on blob workers
2022-08-26 15:38:31 -07:00
A.J. Beamon 0e782412a8 Fix some undefined bevavior: 1) a unit test was not initializing members of the WorkloadContext it was using, and 2) very large ratekeeper limits for batch priority were overflowing the types used to log them 2022-08-26 14:17:01 -07:00
Evan Tschannen 493771b6a8
Throttle the cluster if the blob manager cannot assign ranges (#7900)
* Throttle the cluster if the blob manager cannot assign ranges

* fixed a number of different bugs which caused ratekeeper to throttle to zero because of blob worker lag

* fix: do not mark an assignment as block if it is cancelled

* remove asserts to merge bug fixes

* fix formatting

* restored old control flow to storage updater

* storage updater did not throw errors

* disable buggify to see if it fixes CI
2022-08-23 13:33:46 -05:00
Evan Tschannen a9d3c9f9b3
Added throttling when a blob worker falls behind (#7751)
* throttle the cluster when blob workers fall behind

* do not throttle on blob workers if they are not enabled

* remove an unnecessary actor

* fixed a compile error

* fetch blob worker metrics at the same interval as the rate is updated, avoid fetching the complete blob worker list too frequently

* fixed another compilation bug

* added a 5 second delay before bw throttling to prevent false positives caused by the 100e6 version jump during recovery. Lower the throttling thresholds to react much quicker to bw lag.

* fixed a number of problems

* changed the minBlobVersionRequest to look at storage server versions since this will be a lot more efficient

* fix: do not let desired go backwards

* fix: track the version of notAtLatest changefeeds for throttling

* ratekeeper now throttled blob workers by estimating the transaction per second throughput of the blob workers

* added metrics for blob worker change feeds

* added a knob to disable bw throttling

* fixed the transaction options in blob manager
2022-08-12 13:15:56 -07:00
Trevor Clinkenbeard 583021c2d9
Merge pull request #7772 from sfc-gh-tclinkenbeard/global-tag-throttling6
Add status section for global tag throttler
2022-08-11 17:38:31 -03:00
sfc-gh-tclinkenbeard 66373f1e74 Addressed review comments 2022-08-10 21:44:12 -03:00
Jingyu Zhou eba77d78f4 Add knobs for min/max Ratekeeper limit
The default has no effects.
2022-08-08 15:27:21 -07:00
sfc-gh-tclinkenbeard 1bd47a07b2 Add ENFORCE_TAG_THROTTLING_ON_PROXIES knob 2022-08-05 00:40:10 -07:00
Jingyu Zhou 84d483605b
Merge pull request #7431 from xis19/main
Let the storage server reports busiest write tag
2022-08-04 10:23:31 -07:00
sfc-gh-tclinkenbeard 2699439282 Add global_tag_throttler section to status 2022-08-02 16:53:03 -07:00
Xiaoge Su fd3c3f0774 fixup! Reformat source 2022-08-01 18:56:50 -07:00
Xiaoge Su 195890dd7b Add ratekeeper ID for storage server busiest write tag report 2022-08-01 18:56:50 -07:00
Xiaoge Su aa69f5f36e fixup! Update per code review 2022-08-01 18:56:50 -07:00
Xiaoge Su 90b887f394 fixup! Update per comments 2022-08-01 18:56:50 -07:00
Xiaoge Su ec40c6bfec fixup! Add a wrapper of ResourceWeakRef for better support of self pointer 2022-08-01 18:56:50 -07:00
Xiaoge Su cf04afe925 fixup! Non-owning reference to an object
See documents in flow/OwningResource.h
2022-08-01 18:56:50 -07:00
Xiaoge Su 542b5e61cf Let the storage server reports busiest write tag
Issue #7258

The ratekeeper is recording the busiest write tag for *all* storage
servers, which throttles the traceevent. Distribute the busiest write
tag to corresponding storage servers should reduces this throttling
issue.
2022-08-01 18:56:50 -07:00
sfc-gh-tclinkenbeard 20ac60fb11 Set throttling ratio in GlobalTagThrottler::tryUpdateAutoThrottling 2022-07-19 17:04:04 -07:00
sfc-gh-tclinkenbeard b49c36f0b0 Add StorageQueueInfo::getWriteQueueSizeLimitRatio method 2022-07-19 16:28:27 -07:00
Markus Pilman 1de37afd52
Make TEST macros C++ only (#7558)
* proof of concept

* use code-probe instead of test

* code probe working on gcc

* code probe implemented

* renamed TestProbe to CodeProbe

* fixed refactoring typo

* support filtered output

* print probes at end of simulation

* fix missed probes print

* fix deduplication

* Fix refactoring issues

* revert bad refactor

* make sure file paths are relative

* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
sfc-gh-tclinkenbeard 086e4bff06 Merge remote-tracking branch 'origin/main' into global-tag-throttling3 2022-06-28 10:18:13 -07:00
Xiaoxi Wang a5054b2beb move getServerListAndProcessClasses to NativeAPI 2022-06-23 15:28:45 -07:00
sfc-gh-tclinkenbeard 44e367830a Remove unnecessary indirection in Ratekeeper::monitorThrottlingChanges implementation 2022-06-14 11:24:26 -07:00
sfc-gh-tclinkenbeard 5a1de67757 Add GLOBAL_TAG_THROTTLING knob 2022-05-07 15:58:04 -07:00
Bharadwaj V.R 726cb3a18f merge commits from main 2022-03-28 22:49:03 -07:00
Bharadwaj V.R 961e4ae7fd ratekeeper and ser-des fixes 2022-03-24 17:25:07 -07:00
sfc-gh-tclinkenbeard 30651bf2c6 Fix order of TagInfo constructor arguments 2022-03-22 17:06:33 -07:00
Bharadwaj V.R 56613bcde5 Create a boolean state indicating whether an SSI is open for traffic 2022-03-17 15:59:41 -07:00
sfc-gh-tclinkenbeard 71976e51c5 Merge remote-tracking branch 'origin/main' into global-tag-throttling2 2022-03-16 13:53:46 -07:00
sfc-gh-tclinkenbeard a13b408793 Add StorageQueueInfo::addCommitCost method 2022-03-12 14:28:50 -04:00
sfc-gh-tclinkenbeard cdc099b192 Add StorageQueueInfo::update method 2022-03-12 14:17:44 -04:00
sfc-gh-tclinkenbeard 1ef0102974 Add TLogQueueInfo::update method 2022-03-12 14:04:10 -04:00
sfc-gh-tclinkenbeard 5799b6cbd7 Add comments and modify ITagThrottler::tryUpdateAutoThrottling method 2022-03-11 12:13:34 -04:00
sfc-gh-tclinkenbeard 5c053c6c37 Move expiredTagThrottleCleanup into TagThrottler class 2022-03-11 12:09:15 -04:00
sfc-gh-tclinkenbeard eb5a556c98 Move constructor implementations out of Ratekeeper.h 2022-03-11 12:09:15 -04:00
sfc-gh-tclinkenbeard 229f0cca8b Add StorageQueueInfo::refreshCommitCost method 2022-03-11 12:09:15 -04:00
sfc-gh-tclinkenbeard 455b75abca Use structured bindings for for loop in Ratekeeper::refreshStorageServerCommitCost 2022-03-11 12:09:15 -04:00
sfc-gh-tclinkenbeard cad106f9eb Support sending multiple busy tags from storage server to ratekeeper 2022-03-11 12:09:15 -04:00
sfc-gh-tclinkenbeard e00820cdd7 Reduce mutable access to *QueueInfo objects in Ratekeeper.actor.cpp 2022-03-11 01:04:13 -04:00
A.J. Beamon 250a88e682 Enforce that trace event suppression calls happen first when using trace event call chaining. Fix various instances where we weren't following this requirement. 2022-02-24 12:25:52 -08:00
sfc-gh-tclinkenbeard 0f4c808f37 Remove TagThrottler::ratekeeper field 2022-02-14 19:08:23 -08:00
sfc-gh-tclinkenbeard 58669717f1 Move tryAutoThrottleTag method to TagThrottler 2022-02-14 18:54:12 -08:00
sfc-gh-tclinkenbeard 796249e99d Move lastBusiestCommitTagPick field back to Ratekeeper 2022-02-14 16:15:51 -08:00
sfc-gh-tclinkenbeard 00f12687c6 Add TagThrottler class 2022-02-14 16:03:37 -08:00
sfc-gh-tclinkenbeard 49ba96e0bb Clean up ratekeeper includes 2022-02-14 13:02:09 -08:00
sfc-gh-tclinkenbeard 8074630530 Rename class RatekeeperData to Ratekeeper 2022-02-14 12:42:25 -08:00
sfc-gh-tclinkenbeard d6c5239080 Fix copyright headers 2022-02-14 12:36:44 -08:00
sfc-gh-tclinkenbeard d4b4479399 Rename RatekeeperData.actor.cpp to Ratekeeper.actor.cpp 2022-02-14 12:35:50 -08:00