Commit Graph

152 Commits

Author SHA1 Message Date
sfc-gh-tclinkenbeard 64733d2e39 s/GrvProxyTransactionTagThrottler/GrvProxyTagThrottler 2022-11-02 07:35:05 -07:00
sfc-gh-tclinkenbeard 36731efe1d Apply clang-format 2022-11-01 12:53:15 -07:00
sfc-gh-tclinkenbeard 047578b3d9 Merge remote-tracking branch 'origin/main' into debug 2022-11-01 12:05:45 -07:00
sfc-gh-tclinkenbeard 5fd8d05810 Make PROXY_MAX_TAG_THROTTLE_DURATION a server knob 2022-11-01 11:00:45 -07:00
sfc-gh-tclinkenbeard 736cf4a22f Add GrvProxyTransactionTagThrottler::maxThrottleDuration field 2022-11-01 10:44:32 -07:00
sfc-gh-tclinkenbeard 6ae0aac153 Merge remote-tracking branch 'origin/main' into add-tag-throttling-latency-bands 2022-10-27 14:07:51 -07:00
sfc-gh-tclinkenbeard 08251329a9 Update tag throttling latency bands in GrvProxyData::updateLatencyBandConfig 2022-10-26 10:48:51 -07:00
Marian Dvorsky 3c5d3f7a94
Fix SpanContext for GP:getLiveCommittedVersion (#8565)
* Fix SpanContext for GP:getLiveCommittedVersion
2022-10-26 16:29:28 +02:00
sfc-gh-tclinkenbeard 8766809cee Add count parameter to LatencyBands::addMeasurement 2022-10-25 17:10:59 -07:00
sfc-gh-tclinkenbeard 74212eeacf Encapsulate CounterCollection 2022-10-25 10:17:15 -07:00
sfc-gh-tclinkenbeard b90722baa6 Add comment for GrvProxyStats latency bands and samples 2022-10-14 15:58:15 -07:00
sfc-gh-tclinkenbeard 6f01d97006 Subtract proxyTagThrottleDuration from measured GRV latencies 2022-10-12 14:41:32 -07:00
sfc-gh-tclinkenbeard f1cb4e40f5 Avoid adding untagged requests to GrvProxyTransactionTagThrottler 2022-10-10 22:10:08 -07:00
sfc-gh-tclinkenbeard 24a0dd9f17 Change GrvProxy tag throttling algorithm.
The new algorithm assumes there is only one tag per request,
so queues are partitioned by tag. This is a more efficient approach than
the old algorithm.
2022-10-10 22:10:08 -07:00
sfc-gh-tclinkenbeard bbf69b2d0a Rename TagQueue::runEpoch to TagQueue::releaseTransactions 2022-10-10 22:10:08 -07:00
sfc-gh-tclinkenbeard 31e8fb0490 Rename epoch to release window in GrvTransactionRateInfo
The term "epoch" was used in too many places
2022-10-10 22:10:08 -07:00
sfc-gh-tclinkenbeard 34857bd2bf Remove outSystemPriority parameter from TagQueue::runEpoch 2022-10-10 22:10:08 -07:00
sfc-gh-tclinkenbeard e313edbb85 Add proxyTagThrottledDuration field to GetReadVersion[Request|Reply] 2022-10-10 22:10:08 -07:00
sfc-gh-tclinkenbeard 2684464cc6 Use TagQueue in GRV proxy when ENFORCE_TAG_THROTTLING_ON_PROXIES is true 2022-10-10 22:10:08 -07:00
sfc-gh-tclinkenbeard 4c973c11ad Modularize and comment GrvTransactionRateInfo 2022-10-10 22:10:05 -07:00
A.J. Beamon 4fd64630e8 Convert literal string ref instances to use _sr suffix 2022-09-19 11:35:58 -07:00
sfc-gh-tclinkenbeard 82adc1e856 Make g_simulator a pointer 2022-09-15 09:00:33 -07:00
Markus Pilman eb16947860
Merge pull request #7844 from jzhou77/grv-error
Buggify GRV proxy to return errors and fix bugs found
2022-08-15 13:59:21 -06:00
Evan Tschannen a9d3c9f9b3
Added throttling when a blob worker falls behind (#7751)
* throttle the cluster when blob workers fall behind

* do not throttle on blob workers if they are not enabled

* remove an unnecessary actor

* fixed a compile error

* fetch blob worker metrics at the same interval as the rate is updated, avoid fetching the complete blob worker list too frequently

* fixed another compilation bug

* added a 5 second delay before bw throttling to prevent false positives caused by the 100e6 version jump during recovery. Lower the throttling thresholds to react much quicker to bw lag.

* fixed a number of problems

* changed the minBlobVersionRequest to look at storage server versions since this will be a lot more efficient

* fix: do not let desired go backwards

* fix: track the version of notAtLatest changefeeds for throttling

* ratekeeper now throttled blob workers by estimating the transaction per second throughput of the blob workers

* added metrics for blob worker change feeds

* added a knob to disable bw throttling

* fixed the transaction options in blob manager
2022-08-12 13:15:56 -07:00
Jingyu Zhou a32822a51d Add fault injection to GRV proxy to return error response
This turns out to fail a lot of tests that we need to fix.
2022-08-12 11:13:32 -07:00
Xiaoge Su 0326d53965 Split proxy_memory_limit_exceeded to commit/grv specific exceptions
Currently GRV is reporting proxy_memory_limit_exceeded error which has
error message claiming Commit proxy failing. This split should remove
such confusion.
2022-08-12 00:45:57 -07:00
sfc-gh-tclinkenbeard 1bd47a07b2 Add ENFORCE_TAG_THROTTLING_ON_PROXIES knob 2022-08-05 00:40:10 -07:00
Lukas Joswiak 7fb427f4ec Update fdbserver/GrvProxyServer.actor.cpp
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-07-22 10:37:29 -07:00
Lukas Joswiak a26344b877 Update fdbserver/GrvProxyServer.actor.cpp
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-07-22 10:37:29 -07:00
Lukas Joswiak 8395c6cf3f Fix Tuple error 2022-07-22 10:37:29 -07:00
Lukas Joswiak 5fa8e3f7d0 Simplify refresh condition 2022-07-22 10:37:29 -07:00
Lukas Joswiak 703aa1d279 Mess with timeout values 2022-07-22 10:37:29 -07:00
Lukas Joswiak 40d403ed5f Reduce global configuration system key reads from proxy
Clients now poll the proxy for the latest global config for a specific
version. The proxy now periodically requests the latest global
configuration data and stores it in memory, enabling it to respond
immediately to clients with the appropriate version.
2022-07-22 10:37:29 -07:00
Lukas Joswiak 56dfdbda83 Add migration timeout 2022-07-22 10:37:29 -07:00
Lukas Joswiak e024db85a0 Update fdbserver/GrvProxyServer.actor.cpp
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-07-22 10:37:29 -07:00
Lukas Joswiak 7a48a53778 Perform migration automatically on proxy boot 2022-07-22 10:37:29 -07:00
Lukas Joswiak 2e99d5f6cc Batch global config refresh requests 2022-07-22 10:37:29 -07:00
Lukas Joswiak c33e44b0f4 Proxy GlobalConfig reads through GRV proxies
Clients should avoid reading system keys unless authorized. Under global
config, each client reads from the system keyspace to check for new
global config keys. This commit moves these reads to a server role (the
GRV proxies) and sends the results back to GlobalConfig for an in-memory
update.
2022-07-22 10:37:29 -07:00
Markus Pilman 1de37afd52
Make TEST macros C++ only (#7558)
* proof of concept

* use code-probe instead of test

* code probe working on gcc

* code probe implemented

* renamed TestProbe to CodeProbe

* fixed refactoring typo

* support filtered output

* print probes at end of simulation

* fix missed probes print

* fix deduplication

* Fix refactoring issues

* revert bad refactor

* make sure file paths are relative

* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
Ray Jenkins dc9e782ccc
OpenTelemetry Tracing Perf Fixes (#6990) 2022-05-02 14:56:51 -05:00
Ray Jenkins 1c5bf135d5
Revert "Migrate to OpenTelemetry tracing. (#6855)" (#6941)
This reverts commit 5df3bac110.
2022-04-25 09:29:56 -05:00
Ray Jenkins 5df3bac110
Migrate to OpenTelemetry tracing. (#6855) 2022-04-20 09:26:37 -05:00
Sreenath Bodagala f038f37513 - Do not invoke version vector related code on the sequencer and
GRVs when version vector feature is disabled.
2022-04-12 20:05:32 +00:00
Markus Pilman 16467262f0 Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-04-10 14:12:37 -06:00
Markus Pilman bf956f5630 Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-04-07 13:29:27 -06:00
Jingyu Zhou cfcf0f152c Merge branch 'main-4a085fc84' into vv
Fix Conflicts:
	fdbclient/NativeAPI.actor.cpp
	fdbserver/ClusterRecovery.actor.cpp
	fdbserver/MasterInterface.h
	fdbserver/masterserver.actor.cpp
	flow/error_definitions.h
2022-03-30 22:28:06 -07:00
Jingyu Zhou 00b57d4cce Merge branch 'main-67eba5ec7' into vv
Fix Conflicts:
	fdbclient/CommitProxyInterface.h
	fdbclient/NativeAPI.actor.cpp
	fdbclient/StorageServerInterface.h
	fdbserver/CommitProxyServer.actor.cpp
	fdbserver/storageserver.actor.cpp
2022-03-30 20:05:55 -07:00
Jingyu Zhou e9659b5dd4 Merge branch 'master-PR-6500' into vv
Fix Conflicts:
	fdbclient/CommitProxyInterface.h
	fdbclient/NativeAPI.actor.cpp
	fdbserver/masterserver.actor.cpp
2022-03-30 14:53:49 -07:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
Markus Pilman 118b53b7cf Merge remote-tracking branch 'origin/main' into features/private-request-streams 2022-03-17 12:06:44 +01:00