Commit Graph

116 Commits

Author SHA1 Message Date
Dan Lambright 1231ffbbe4 knob name change/rebase 2022-10-12 20:26:44 -04:00
Dan Lambright ef3efbc641 Merge remote-tracking branch 'origin/main' into tracelib 2022-10-07 21:32:26 -04:00
Dan Lambright 2402b813e9 do not unlink fdb_c in tmp dir if knob set 2022-10-07 16:32:28 -04:00
sfc-gh-tclinkenbeard 7fc5c196c4 Make read and write quotas fungible 2022-09-25 21:00:11 -07:00
Yi Wu d831c87d14
Add encryption metrics (#8070)
Adding the following metrics:
* BlobCipherKeyCache hit/miss
* EKP: KMS requests latencies
* For each component that using encryption, they now need to pass a UsageType enum to the encryption helper methods (GetEncryptCipherKeys/GetLatestEncryptCipherKey/encrypt/decrypt) and those methods will help to log get cipher key latency samples and encryption/decryption cpu times accordingly.
2022-09-09 18:43:09 -07:00
Nim Wijetunga 72ccc681be move tenant entry cache to client 2022-09-01 10:10:47 -07:00
Evan Tschannen 8314e80371
Fixed a few bugs which caused ratekeeper to unnecessarily throttle a cluster (#8006)
* do not count recently created change feeds for throttling

* fix: blocked assignments were not decremented when force purging

* fix: created needs to be updated when the changefeed is reset

* added asserts to detect if ratekeeper is throttled on blob workers
2022-08-26 15:38:31 -07:00
Josh Slocum 46b02cab49
Blob granule summary implementation (in native client) (#7981)
* implemented blob granule summary call in native client

* clean up prints
2022-08-26 14:04:59 -05:00
Jingyu Zhou 8fb6d59e94 Use error_code_grv_proxy_memory_limit_exceeded
instead of error_code_proxy_memory_limit_exceeded
2022-08-12 11:36:17 -07:00
Jingyu Zhou ed1b7ef173 Add a client knob GRV_ERROR_RETRY_DELAY 2022-08-12 11:13:32 -07:00
Jingyu Zhou 5929ac1d65
Merge pull request #7847 from xis19/knobCheck
Cleanup the knobs that are not being used
2022-08-11 21:52:52 -07:00
Xiaoge Su 85836cbec9 Cleanup the knobs that are not being used 2022-08-10 16:51:01 -07:00
A.J. Beamon b33a9da7ce Don't buggify max tenants per cluster globally; instead buggify it in specific tests 2022-08-06 06:10:45 -07:00
Josh Slocum ddbf32d35e
adding knob and fixing blobrange cli check bug with too many granules (#7793) 2022-08-05 11:19:17 -05:00
A.J. Beamon ff23d5994e
Merge pull request #7729 from sfc-gh-ajbeamon/feature-metacluster
Metacluster
2022-08-04 15:29:44 -07:00
A.J. Beamon a2e66c9695 Don't buggify the assignment availability timeout too low or it will become impossible to make an assignment 2022-08-03 19:39:20 -07:00
He Liu fa418fd784
Change SHARD_ENCODE_LOCATION_METADATA to a server knob. (#7770)
Co-authored-by: He Liu <heliu@apple.com>
2022-08-03 13:51:40 -07:00
A.J. Beamon a69164d160 Merge branch 'main' into feature-metacluster 2022-07-29 11:53:06 -07:00
Josh Slocum cfc13e7018
Adding more blobrange cli commands and a couple other tweaks (#7727) 2022-07-29 08:20:45 -07:00
A.J. Beamon 7f53b340f5 Merge branch 'main' into feature-metacluster
# Conflicts:
#	bindings/python/tests/fdbcli_tests.py
#	fdbcli/TenantCommands.actor.cpp
#	fdbclient/ClientKnobs.cpp
#	fdbclient/include/fdbclient/ClientKnobs.h
#	fdbclient/include/fdbclient/Tenant.h
#	fdbserver/BlobManager.actor.cpp
#	fdbserver/BlobWorker.actor.cpp
#	fdbserver/TenantCache.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/TenantManagementWorkload.actor.cpp
#	flow/include/flow/error_definitions.h
2022-07-28 11:43:21 -07:00
A.J. Beamon 7c6b3fb0b8 Merge branch 'main' into feature-metacluster 2022-07-27 08:55:10 -07:00
A.J. Beamon d39c0b773a Add a limit to the number of tenants that can be created in a cluster 2022-07-27 08:21:03 -07:00
Renxuan Wang dc9599f2e9
Get network interfaces ready for https proxy. (#7556)
* Move HTTP from fdbclient/ to fdbrpc/.

* Move md5 and libb64 to contrib/.

* Get network interfaces ready for https proxy.

* Rebase
2022-07-25 17:08:32 -07:00
A.J. Beamon 077003c4f5 Cleanup tenant tombstones periodically 2022-07-24 18:24:19 -07:00
A.J. Beamon 41c1bef3bf Merge branch 'feature-tenant-groups' into feature-metacluster 2022-07-22 14:26:02 -07:00
Lukas Joswiak 703aa1d279 Mess with timeout values 2022-07-22 10:37:29 -07:00
Lukas Joswiak 40d403ed5f Reduce global configuration system key reads from proxy
Clients now poll the proxy for the latest global config for a specific
version. The proxy now periodically requests the latest global
configuration data and stores it in memory, enabling it to respond
immediately to clients with the appropriate version.
2022-07-22 10:37:29 -07:00
A.J. Beamon 279296c29f Merge branch 'tenant-metadata-change' into feature-metacluster
# Conflicts:
#	fdbclient/SystemData.cpp
#	fdbclient/Tenant.cpp
#	fdbclient/include/fdbclient/SystemData.h
#	fdbclient/include/fdbclient/Tenant.h
#	fdbclient/include/fdbclient/TenantManagement.actor.h
#	fdbserver/TenantCache.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/TenantManagementWorkload.actor.cpp
2022-07-20 09:18:27 -07:00
A.J. Beamon 537ceff8ac Remove the ability to configure a tenant subspace. Rename the prefixes used for tenant metadata. 2022-07-19 14:32:05 -07:00
A.J. Beamon 70bc67ce34 Update tenant assignment to check data cluster availability. Cleanup errors when performing an invalid operation on a non-metacluster and remove redundant function. 2022-07-18 16:15:33 -07:00
A.J. Beamon cb499fa5db Merge branch 'main' into feature-metacluster 2022-07-13 15:28:34 -07:00
He Liu bc5bfaffda
Shard based move (#6981)
* Shard based move.

* Clean up.

* Clear results on retry in getInitialDataDistribution.

* Remove assertion on SHARD_ENCODE_LOCATION_METADATA for compatibility.

* Resolved comments.

Co-authored-by: He Liu <heliu@apple.com>
2022-07-07 20:49:16 -07:00
A.J. Beamon e1a93988ef Merge branch 'main' into feature-metacluster 2022-06-28 14:58:07 -07:00
A.J. Beamon 4b69723a2f Cleanup metadata associated with a cluster when force removing it. Move some metacluster metadata into the normal key-space. Cleanup some todos. 2022-06-24 14:55:57 -07:00
A.J. Beamon 96fad987eb Add new command to get total metacluster capacity. Fix bug where removing a data cluster could make it impossible to create new tenants on it. Add knobs for max data clusters and max tenants. 2022-06-17 10:42:03 -07:00
Xiaoge Su e493f1c3cd fixup! Add a retry mechanism in changeQuorumChecker and changeQuorum
This is to fix an issue when recovery and change coordinator key happens
together. The issue will occur when:

1. Recovery starts
2. Coordinator key change transaction started
3. During the recovery the coordinator key is read from cluster file and
   stored in the storage server
4. The cluster controller received `ChangeCoordiatorsRequest`, and
   updated the cluster name with the new value.

at this stage, the value related to coordinator key in storage server and
the worker is inconsistent.

5. changeQuorumChecker is called, which will verify such consistency.
   Since they are different, the call is returning failure and the
   caller, which could be a TEST_CASE, fails.

This is a rare race issue, and it is also noticed that when the
recovery/coordinator key change process is done, the database is in a
proper state which allows changeQuorumChecker behave properly. In this
case, a retry mechanism should be sufficiently fix corresponding test
failures.
2022-06-14 10:43:13 -07:00
Zhanwei Wang da346d718f Fix compatibility issue of s3 backup
1. For v4 signature, http request path should be encoded, currently only parameters are encoded.
2. For v2 signature, http request date header should be in form of either the http date header or x-amz-date, currently in form of ISO 8601
3. Empty response header value cause http_bad_response error.
4. Skip verifying MD5 for range get request and if http response code 206 is returned.
2022-05-25 09:48:54 -05:00
Josh Slocum 674e6a8fdc Merge branch 'main' into min_shard_bytes_main 2022-04-29 16:00:27 -05:00
A.J. Beamon 79063d5fc4
Better accounting for tenant prefixes when working with key size limits (#6973)
* Consider tenant prefix sizes when doing key size limit checks

* Fix formatting
2022-04-27 14:38:59 -07:00
Josh Slocum ca68fefca2 Decreasing MIN_SHARD_BYTES knob 2022-04-20 18:46:11 -05:00
Dan Lambright 5bdc525353
Merge branch 'main' into vv 2022-04-08 13:16:04 -04:00
Dan Lambright 60c55e0785 Merge remote-tracking branch 'origin/version-vector-prototype' into vv 2022-04-05 11:17:39 -04:00
Renxuan Wang 465ff712b6
Move Hostname to its own files. (#6759)
* Change DNS cache to use std::map.

Revert commit 90c259d84e, because if we use unordered_map, toString() can be inconsistent.

* Move ClientKnob::COORDINATOR_HOSTNAME_RESOLVE_DELAY to FlowKnob::HOSTNAME_RESOLVE_DELAY.

* Move Hostname to its own files.

Also, add resolve-related variables and functions in Hostname.
2022-04-04 19:04:51 -07:00
Josh Slocum 268caa5ac8 fixing shard size knobs outside of simulation 2022-04-04 11:38:18 -05:00
Jingyu Zhou 4fd414a8ed Merge branch 'main' into vv
Fix Conflicts:
	fdbclient/NativeAPI.actor.cpp
2022-03-31 09:38:36 -07:00
Jingyu Zhou cfcf0f152c Merge branch 'main-4a085fc84' into vv
Fix Conflicts:
	fdbclient/NativeAPI.actor.cpp
	fdbserver/ClusterRecovery.actor.cpp
	fdbserver/MasterInterface.h
	fdbserver/masterserver.actor.cpp
	flow/error_definitions.h
2022-03-30 22:28:06 -07:00
Jingyu Zhou b34f4052cd Merge branch 'main-f28dfc12b' into vv
Fix Conflicts:
	fdbclient/MultiVersionTransaction.actor.cpp
	fdbclient/NativeAPI.actor.cpp
	fdbclient/NativeAPI.actor.h
	fdbserver/storageserver.actor.cpp
2022-03-30 21:01:25 -07:00
Jingyu Zhou 00b57d4cce Merge branch 'main-67eba5ec7' into vv
Fix Conflicts:
	fdbclient/CommitProxyInterface.h
	fdbclient/NativeAPI.actor.cpp
	fdbclient/StorageServerInterface.h
	fdbserver/CommitProxyServer.actor.cpp
	fdbserver/storageserver.actor.cpp
2022-03-30 20:05:55 -07:00
Jingyu Zhou e9659b5dd4 Merge branch 'master-PR-6500' into vv
Fix Conflicts:
	fdbclient/CommitProxyInterface.h
	fdbclient/NativeAPI.actor.cpp
	fdbserver/masterserver.actor.cpp
2022-03-30 14:53:49 -07:00
Evan Tschannen 90983990df
Merge pull request #6680 from sfc-gh-jslocum/bg_client_bindings_improvements
granule parallelism, metrics, and other improvements to bg client bindings
2022-03-25 13:44:43 -07:00