209ebcc595
Revert "Add limiting health metrics"
2020-11-13 17:08:46 -08:00
6c4493166f
Add limiting storage queue and durability lag to health metrics
2020-11-12 20:14:41 -08:00
12b07d024f
Merge pull request #3775 from sfc-gh-xwang/fdbtest
...
add knob MIN_TAG_WRITE_PAGES_RATE for write tag-throttling
2020-09-18 10:40:28 -07:00
ea27e4bd9c
Merge pull request #3772 from yliucode/rename
...
Rename master proxy as commit proxy
2020-09-16 20:06:50 -07:00
cc5bc16bd8
Rename more places from proxy to commit proxy
2020-09-15 22:29:49 -07:00
3a68c89281
Remove initialisms
...
Co-authored-by: A.J. Beamon <ajbeamon@users.noreply.github.com>
2020-09-15 16:32:24 -07:00
52bd86ad42
update knob
2020-09-11 20:50:19 +00:00
d5fba9a69b
add write-tag throttling
2020-09-11 19:10:35 +00:00
195c55df4b
add comment
2020-09-11 16:48:21 +00:00
084c69b2ad
fix reason serialization bug
2020-09-11 07:21:57 +00:00
35bef73a1c
Rename proxy to commit proxy
2020-09-10 17:44:15 -07:00
7660fb3beb
report busiest tags in status json
2020-09-04 16:33:59 +00:00
ec56e1f286
merge with master
2020-09-04 16:11:42 +00:00
341feb2b6f
fix compare bug
2020-09-04 07:17:35 +00:00
87693cae81
merge master branch and resolve conflicts
2020-09-02 13:44:33 -07:00
47893c847b
fix rate calculation typo
2020-09-01 07:19:46 +00:00
e87327b33b
Merge master branch and keep master proxy reporting txn cost estimation to ratekeeper
2020-08-29 12:47:35 -07:00
b1c206b62a
change rate calculation
2020-08-25 18:47:13 +00:00
63b3612ad5
Merge master branch and resolve conflicts
2020-08-24 16:42:31 -07:00
4e4fa0fded
merge with master
2020-08-24 21:04:53 +00:00
1f134d1534
format
2020-08-21 05:06:13 +00:00
9398a78a3a
add busy-read count and busy-write count to status json
2020-08-21 04:50:56 +00:00
88cb352e2e
rename variables
2020-08-21 00:03:06 +00:00
9c709b33d6
scale cost
2020-08-19 19:47:41 +00:00
8a1b9394b8
do not send auto-throttled tags to proxy is autoThrottleEnabled == false
2020-08-19 16:23:11 +00:00
599675cba8
modify some details to get better performance
2020-08-19 04:23:23 +00:00
69914d4909
defer write throttling until found proper statistics for write saturation
2020-08-18 15:46:02 +00:00
ffa1973242
move OPERATION_COST_BYTE_FACTOR to CLIENT_KNOB; skip the first call of refreshBusiestTag.
2020-08-14 22:49:20 +00:00
548ba06510
change map to deque; code style problem
2020-08-14 18:30:58 +00:00
475d4cca72
solve some cr comments
2020-08-14 16:40:16 +00:00
f3ecf14601
change midShardSize type and other details
2020-08-12 17:49:12 +00:00
16ef2bd3bd
Pending commit
2020-08-12 10:34:07 -07:00
c0b7ac0b7d
try to throttle write tag and read tag seperatedly
2020-08-10 23:43:58 +00:00
cae6f04070
sample on cost instead of bytes
2020-08-10 22:30:06 +00:00
ba66b1f668
trivial changes
2020-08-06 14:53:26 +00:00
13307679c5
use median shard size"
2020-08-05 03:57:25 +00:00
484a75dd7b
compare read and write tag together
2020-08-04 00:02:41 +00:00
52073738c0
fix bugs; change some KNOB values
2020-08-03 22:18:34 +00:00
6db7040f17
update DDMetrics periodically
2020-08-03 05:52:38 +00:00
d1cc87452c
merge with master; solve conflicts; solve initialization;
2020-08-02 22:44:07 +00:00
0352e8ee0b
pick busiest commit tag periodically
2020-08-02 18:38:56 +00:00
92c1112c74
consider clear single key
2020-08-01 18:20:13 +00:00
4f7dab4951
sample clear op on client
2020-08-01 06:14:52 +00:00
c3a629588f
add client transaction tag sample
2020-07-31 19:08:42 +00:00
30ea639666
Remove debug traces
2020-07-29 07:55:05 -07:00
f7b76a92af
pass joshua
2020-07-29 07:26:55 -07:00
41a3e6c853
add write throttling
2020-07-28 03:49:47 +00:00
48a0fb5154
ask DD for shard info
2020-07-25 04:08:12 +00:00
b2a3b4fd83
Merge branch 'master' into mengxu/merge-6.3-PR
2020-07-20 11:34:18 -07:00
0df2a8d014
better code style
2020-07-18 01:48:58 +00:00
9d0d189cc8
better serialize; TransactionOption::clear patch
2020-07-15 22:39:21 +00:00
27a21e23bd
Add number comment to limitReason_t entries
2020-07-15 10:57:50 -07:00
eb44ae0e86
finish local shard estimation
2020-07-15 16:08:00 +00:00
a310faf9d1
solve some code reviews
2020-07-14 17:19:55 +00:00
c56daf3be7
merge with master
2020-07-14 01:08:56 +00:00
d512170cd8
add clear cost estimation
2020-07-14 00:18:52 +00:00
11b136c745
Various fixes to tag throttling:
...
* Master proxy reports transaction counts to ratekeeper for throttled tags only
* The ramp up behavior at the end of an auto-throttle was broken
* Fixed some issues with computing the initial transaction rate for auto-throttles
2020-06-30 16:24:41 -07:00
ced65cd30b
finished explicitly versioning everything stored in the database
2020-05-22 17:14:21 -07:00
cc4874918a
Merge branch 'release-6.3' into tag-throttling-by-priority
...
# Conflicts:
# fdbserver/Ratekeeper.actor.cpp
2020-05-20 14:26:35 -07:00
14b23c146f
Support throttling and unthrottling tags by priority and their auto/manual state in fdbcli.
2020-05-15 12:47:55 -07:00
d3f465fd56
Merge pull request #3102 from mpilman/features/trace-roles
...
Emit traces regularly about role assignment
2020-05-15 08:12:25 -07:00
3ee4912312
Merge pull request #3152 from ajbeamon/tag-throttling-status-improvements
...
Add and fix tag throttling status fields
2020-05-14 16:08:05 -07:00
2cdcab5aa7
address review comments
2020-05-14 14:54:38 -07:00
5230668a76
Pass Ratekeeper ID to all RK traces
2020-05-14 14:17:43 -07:00
c2bc75516f
Merge branch 'release-6.3' of github.com:apple/foundationdb into features/trace-roles
2020-05-14 10:34:53 -07:00
634c988059
Tag throttles reacted poorly to the rate being set to max
2020-05-12 18:09:43 -07:00
acf1244317
Fix: ratekeeper was logging the auto throttle count for the manual throttle count
2020-05-12 14:10:40 -07:00
5f9b127e56
Emit traces regularly about role assignment
...
We are currently emitting Role transition traces when a role starts and
when it ends. While this is useful for debugging, it doesn't work well
with tools that inject data and might potentially miss some trace lines.
We do decorate each trace lines with the roles assigned to that
particular process, however, this is not sufficient for tools that can
make use of the UID -> Role mapping
2020-05-08 16:27:57 -07:00
fbf436f45f
Various cleanup and knob adjustments.
2020-05-07 09:15:33 -07:00
35d382811f
Fix compilation error
2020-05-07 08:59:47 -07:00
66b4920fc3
Fix off-by-one error in auto-throttle limit. Allow updating existing auto-throttle when the limit is reached.
2020-05-05 09:58:00 -07:00
bb3d4b6b89
Add a bunch of TEST macros and some other little things
2020-05-04 10:11:36 -07:00
31cef6075a
Do the auto-throttle ramp up in a better place. Only commit manual throttle limit once. Add some asserts.
2020-05-03 19:15:29 -07:00
decf3e82b0
Fix various bugs and make sure to cleanup throttles from the database when they expire
2020-05-01 21:36:28 -07:00
3a7a026aae
Fix a couple bugs
2020-04-30 22:24:17 -07:00
b80225dde0
Initial support for ramping load back up. Fix some logging. Update auto-throttles less frequently.
2020-04-28 15:50:45 -07:00
0ed70accfa
Reorganization of throttle storage in ratekeeper to support various auto-throttling related actions
2020-04-28 14:30:37 -07:00
a65e97209a
Fix bug in flag management. Pass priority into ratekeeper's updateRate and use it when setting autothrottles.
2020-04-27 11:34:12 -07:00
239876351b
Add some initial auto-throttling. Move the definition of the priority enum to a more global place and use it for all transaction priorites (except in ClientLogEvents, because of serialization incompatibilites).
2020-04-24 11:31:16 -07:00
7343c1b333
Logging for throttle changes was moved
2020-04-23 20:51:53 -07:00
35c18ac60a
Improvements to expiration. Encode throttles with auto/manual and priority in the key to support throttling the same tag with different values in these parameters.
2020-04-23 20:50:40 -07:00
18f860d9d8
Minor cleanup around expiration on ratekeeper.
2020-04-23 10:42:16 -07:00
d2504c08c3
Send throttles at all priorities from RK->MP
2020-04-22 13:47:16 -07:00
434704fbd9
Various bug fixes
2020-04-22 12:28:51 -07:00
f1dd0ee298
Protect against a ratekeeper starting up with a clock set in the past (compared to old ratekeeper) extending the duration of throttles excessively.
2020-04-21 16:35:25 -07:00
d5fb4d26fe
Send tag throttle updates from ratekeeper to proxy only when they change
2020-04-21 16:33:56 -07:00
b2c14611f3
Ratekeeper maintains its throttle data a bit differently, and now it aggregates the effects of multiple priorities before sending results to the proxy
2020-04-21 11:58:59 -07:00
dfec896438
Enforce a throttle limit. Don't count transaction tags on RK if the proxy has updated us in a while.
2020-04-17 11:48:02 -07:00
6619a1a36a
Rename transaction tag map.
2020-04-17 09:06:45 -07:00
2b66dcd24a
Some more refactoring. Reduce what is sent from RK->MP->clients
2020-04-17 08:07:01 -07:00
0fba8c47be
Checkpoint: Ratekeeper sets absolute limits for tag throttles and enforces them by distributing requests to proxies, who distribute them to clients.
...
A few refactorings.
2020-04-16 14:43:22 -07:00
7f3fa00897
Reorganize a bit of code.
2020-04-10 13:53:23 -07:00
29b2c2f3aa
Add hash to StringRef. Use unordered maps for storing tags. Create some helpful typedefs.
2020-04-10 12:54:59 -07:00
55a0d00ad4
Encoding of tags in the database now supports multiple tags per throttle. Remove throttle prefix search.
2020-04-10 10:12:26 -07:00
ebeca10bce
Change the serialization of tags sent in some messages. Add communication of the sampling rate from cluster to clients.
2020-04-09 16:55:56 -07:00
a1d8623e5f
Various changes to the throttling scheme, including most notably that clients now enforce the throttles with info they receive from the proxy.
2020-04-07 16:28:09 -07:00
2336f073ad
Checkpointing a bunch of work on throttles. Rudimentary implementation of auto-throttling. Support for manual throttling via fdbcli. Throttles are stored in the system keyspace.
2020-04-03 15:24:14 -07:00
303df197cf
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/NativeAPI.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogRouter.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/fdbserver.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/flow.vcxproj
# flow/flow.vcxproj.filters
# versions.target
2020-03-06 18:22:46 -08:00
1128666840
added additional logging on the log router
2020-03-05 18:17:06 -08:00
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
Trevor Clinkenbeard