Commit Graph

852 Commits

Author SHA1 Message Date
Jon Fu 0487aa5445 Merge branch 'main' of github.com:apple/foundationdb into jfu-metacluster-rename 2022-08-10 14:07:43 -07:00
A.J. Beamon cd82f65ef5 Change the data cluster entry to use a state field rather than a locked boolean 2022-08-09 11:53:57 -07:00
Jon Fu 74df84f686 Merge branch 'main' of github.com:apple/foundationdb into jfu-metacluster-rename 2022-08-08 17:49:15 -07:00
Vaidas Gasiunas 79571dd2b4
Testing upgrades to a future version of FDB (#7780)
* Enable configuring the next future protocol version as the current protocol version in FDB client, fdbserver, and fdbcli

* Auto format python files used in upgrade tests

* Add a test for upgrading to a future FDB version

* Emphasize that the options for using future protocol version are intended for test purposes only

* Make the global variable for current protocol version visible only locally

* Refactirng to avoid using currentProtocolVersion() in static intialization

* Update go bindings
2022-08-08 17:29:49 +02:00
Jon Fu cc85973957 Merge branch 'main' of github.com:apple/foundationdb into jfu-metacluster-rename 2022-08-05 12:38:43 -07:00
Hui Liu 29ad2c0654
fdbcli: show status details about # works and # key ranges if blob granules enabled (#7792)
* fdbcli: show status if blob granules is enabled

* fdbcli: show status details for blob granules for # works and # key ranges
2022-08-05 12:33:57 -05:00
Josh Slocum ddbf32d35e
adding knob and fixing blobrange cli check bug with too many granules (#7793) 2022-08-05 11:19:17 -05:00
Jon Fu 7a45aba74d Merge branch 'main' of github.com:apple/foundationdb into jfu-metacluster-rename 2022-08-04 16:36:09 -07:00
A.J. Beamon ff23d5994e
Merge pull request #7729 from sfc-gh-ajbeamon/feature-metacluster
Metacluster
2022-08-04 15:29:44 -07:00
Hui Liu 4f75f01882
fdbcli: show status if blob granules is enabled (#7784) 2022-08-03 19:15:34 -05:00
Josh Slocum 1cda8a2fc1
More blob granule operational stuff (#7783)
* More blob manager metrics

* making blobrange check command work for large ranges
2022-08-03 18:11:25 -05:00
Jon Fu bceb44f5a1 address code review comments 2022-08-03 12:03:44 -07:00
Jon Fu 5dc8e930a9 remove whitespace 2022-08-01 11:06:04 -07:00
Jon Fu d9dc1999c6 metacluster rename before using metacluster operation ctx 2022-07-29 13:43:47 -07:00
A.J. Beamon a69164d160 Merge branch 'main' into feature-metacluster 2022-07-29 11:53:06 -07:00
Josh Slocum cfc13e7018
Adding more blobrange cli commands and a couple other tweaks (#7727) 2022-07-29 08:20:45 -07:00
A.J. Beamon e5bf838c83 Fix metacluster register command hints, fix bug in cluster registration on retries, extend the length of the tenant management concurrency test 2022-07-28 21:02:12 -07:00
A.J. Beamon 65cfc839a6 Some various cleanup 2022-07-28 13:35:24 -07:00
A.J. Beamon 7f53b340f5 Merge branch 'main' into feature-metacluster
# Conflicts:
#	bindings/python/tests/fdbcli_tests.py
#	fdbcli/TenantCommands.actor.cpp
#	fdbclient/ClientKnobs.cpp
#	fdbclient/include/fdbclient/ClientKnobs.h
#	fdbclient/include/fdbclient/Tenant.h
#	fdbserver/BlobManager.actor.cpp
#	fdbserver/BlobWorker.actor.cpp
#	fdbserver/TenantCache.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/TenantManagementWorkload.actor.cpp
#	flow/include/flow/error_definitions.h
2022-07-28 11:43:21 -07:00
A.J. Beamon 7c6b3fb0b8 Merge branch 'main' into feature-metacluster 2022-07-27 08:55:10 -07:00
Jon Fu 6dbaf46ec8 restructure test workload impl and map 2022-07-26 13:53:04 -07:00
Jon Fu 21309c2af8 Merge branch 'feature-tenant-groups' of github.com:sfc-gh-ajbeamon/foundationdb into jfu-tenant-rename-special-keys 2022-07-26 11:15:56 -07:00
A.J. Beamon 59ccdc35ab Disallow configuring the same tenant parameter more than once at a time. Made some slight changes to serialization code for tenants. 2022-07-26 09:04:29 -07:00
A.J. Beamon f2a8af9155 Better API version handling for fdbcli tenant commands 2022-07-26 09:04:29 -07:00
A.J. Beamon a64693518a Add support for tenant groups 2022-07-26 09:04:29 -07:00
A.J. Beamon 2990294f6a Make the metacluster experimental by updating the name of the command used to configure one. 2022-07-24 19:11:25 -07:00
Jon Fu 84269bac3e Merge branch 'feature-tenant-groups' of github.com:sfc-gh-ajbeamon/foundationdb into jfu-tenant-rename-special-keys 2022-07-22 16:42:27 -07:00
A.J. Beamon 2b6f349649 Better API version handling for fdbcli tenant commands 2022-07-22 15:37:48 -07:00
Jon Fu 43c7146d86 Merge branch 'feature-tenant-groups' of github.com:sfc-gh-ajbeamon/foundationdb into jfu-tenant-rename-special-keys 2022-07-22 15:21:13 -07:00
A.J. Beamon b308aa6604 Merge branch 'feature-tenant-groups' into feature-metacluster 2022-07-22 11:38:23 -07:00
A.J. Beamon 237081905d Add support for tenant groups 2022-07-22 09:27:31 -07:00
A.J. Beamon 17146c484b Use key-backed types for tenants. Add a tenant state field that will be used in upcoming work. Some other tenant related refactoring. 2022-07-21 20:33:28 -07:00
A.J. Beamon 3e1763e7ac Merge branch 'main' into feature-metacluster 2022-07-21 14:48:53 -07:00
A.J. Beamon 410f27412b
Merge pull request #7620 from sfc-gh-ajbeamon/make-tuple
Add a Tuple::makeTuple function to easily construct a tuple
2022-07-20 17:09:10 -07:00
A.J. Beamon 279296c29f Merge branch 'tenant-metadata-change' into feature-metacluster
# Conflicts:
#	fdbclient/SystemData.cpp
#	fdbclient/Tenant.cpp
#	fdbclient/include/fdbclient/SystemData.h
#	fdbclient/include/fdbclient/Tenant.h
#	fdbclient/include/fdbclient/TenantManagement.actor.h
#	fdbserver/TenantCache.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/TenantManagementWorkload.actor.cpp
2022-07-20 09:18:27 -07:00
A.J. Beamon 190ad8c7e9 Convert existing tuple usages to use Tuple::makeTuple() 2022-07-19 13:45:59 -07:00
Markus Pilman 1de37afd52
Make TEST macros C++ only (#7558)
* proof of concept

* use code-probe instead of test

* code probe working on gcc

* code probe implemented

* renamed TestProbe to CodeProbe

* fixed refactoring typo

* support filtered output

* print probes at end of simulation

* fix missed probes print

* fix deduplication

* Fix refactoring issues

* revert bad refactor

* make sure file paths are relative

* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
A.J. Beamon 1b81e72604 Add a Tuple::makeTuple function to easily construct a tuple. Update Tuple to allow all types to be passed via .append() so they can be used with makeTuple. 2022-07-19 11:50:58 -07:00
A.J. Beamon c7c7bd5f17 Undo change to mark tenant modes non-experimental. Other minor cleanup. 2022-07-19 09:18:53 -07:00
A.J. Beamon 8f77048e95 Add the ability to configure tenant groups in a metacluster 2022-07-18 11:24:49 -07:00
A.J. Beamon 860d3843cc Merge remote-tracking branch 'origin/feature-tenant-groups' into feature-metacluster 2022-07-16 19:33:26 -07:00
A.J. Beamon c08592368f Add support for tenant groups 2022-07-15 21:18:41 -07:00
A.J. Beamon 0fe6f6ad36 Merge remote-tracking branch 'origin/update-prefix-encoding-in-tenant-metadata-json' into feature-metacluster
# Conflicts:
#	bindings/python/tests/fdbcli_tests.py
#	fdbcli/TenantCommands.actor.cpp
#	fdbclient/include/fdbclient/TenantManagement.actor.h
#	fdbclient/include/fdbclient/TenantSpecialKeys.actor.h
#	fdbserver/workloads/TenantManagementWorkload.actor.cpp
2022-07-14 14:32:29 -07:00
Jon Fu 7f3b51f31a initial commit to introduce tenant renaming to special keys 2022-07-13 15:59:52 -07:00
A.J. Beamon cb499fa5db Merge branch 'main' into feature-metacluster 2022-07-13 15:28:34 -07:00
A.J. Beamon 91ccdbcb03 Use a more standard encoding in the JSON metadata reported for tenants. 2022-07-13 15:22:19 -07:00
A.J. Beamon a3f970cf3a Parse integer options in fdbcli rather than passing the bytes unparsed. This allows specifing the string "1" instead of "\x01\x00\x00\x00\x00\x00\x00\x00". 2022-07-11 16:39:59 -07:00
A.J. Beamon 712f40b727 Add support for tenant groups 2022-07-08 15:59:25 -07:00
A.J. Beamon b9cc5389b1 Undo some changes that aren't needed 2022-07-07 14:48:36 -07:00
A.J. Beamon a32cc5218c Fix some merge issues 2022-07-07 11:55:31 -07:00
A.J. Beamon 1519f24f77 Merge branch 'main' into feature-metacluster 2022-07-07 09:35:40 -07:00
Jon Fu df1f108b78 Merge branch 'main' of github.com:apple/foundationdb into jfu-tenant-rename 2022-07-06 14:10:22 -07:00
A.J. Beamon aea4d802c6 Merge branch 'main' into feature-metacluster 2022-06-30 15:08:09 -07:00
A.J. Beamon 2f67328a0c Update the tenant special keys submodule to support multiple sub-ranges. This will enable future work that allows configuring tenants at the same time as creating them. 2022-06-30 15:03:37 -07:00
Jon Fu 5e7bb0aa21 Merge branch 'main' of github.com:apple/foundationdb into jfu-tenant-rename 2022-06-29 13:20:19 -07:00
Trevor Clinkenbeard db769667ae
Merge pull request #7112 from sfc-gh-tclinkenbeard/global-tag-throttling3
Create Global Tag Throttler
2022-06-29 10:06:00 -07:00
sfc-gh-tclinkenbeard 2134fa9a0c Merge remote-tracking branch 'origin/main' into global-tag-throttling3 2022-06-28 19:51:41 -07:00
A.J. Beamon df90d87903 Add missing return statement 2022-06-28 16:03:08 -07:00
A.J. Beamon e1a93988ef Merge branch 'main' into feature-metacluster 2022-06-28 14:58:07 -07:00
Jon Fu 5fee5e1a73 add renameTenant back to TenantAPI after rebase 2022-06-28 11:54:03 -07:00
Jon Fu 869c6fcac5 reintroduce renameTenantCommandActor as a pass-through function 2022-06-28 11:35:40 -07:00
Jon Fu 904ac358ce call directly into management api and remove extra function 2022-06-28 11:35:38 -07:00
Jon Fu 5f4f95db4d fix unreachable code 2022-06-28 11:34:54 -07:00
Jon Fu ddfdbdcb09 initial commit introducing renameTenant function 2022-06-28 11:34:44 -07:00
sfc-gh-tclinkenbeard 086e4bff06 Merge remote-tracking branch 'origin/main' into global-tag-throttling3 2022-06-28 10:18:13 -07:00
Markus Pilman fc2f695b15 Merge remote-tracking branch 'origin/main' into features/cmake-refactor-dirs 2022-06-28 10:41:07 -06:00
A.J. Beamon fcd91f676a
Merge pull request #7436 from sfc-gh-ajbeamon/tenant-reorganization
Tenant code refactoring
2022-06-28 09:16:19 -07:00
Markus Pilman 1bdbd89839 Merge remote-tracking branch 'origin/main' into features/cmake-refactor-dirs 2022-06-28 08:52:03 -06:00
Markus Pilman 4ac236eeba fdbcli compiling 2022-06-27 19:20:18 -06:00
Markus Pilman 03d913a1de Flow compiling 2022-06-27 17:05:55 -06:00
A.J. Beamon f990fba44d Add support for getting tenant metadata as a JSON document. 2022-06-27 12:34:40 -07:00
A.J. Beamon 4bafe77889 Some refactoring of tenant code:
* extract tenant management into its own file and namespace
* rename the tenant management workload source file
* extract tenant special keys functions to a separate file
* extract some helper functions to GenericTransactionHelper.h
* convert StringRef -> TenantNameRef
* move some TenantMapEntry implementation into the cpp file
* add some helper functions to decode/encode a tenant mode
2022-06-27 12:32:49 -07:00
A.J. Beamon eccf244ba6 Add an index that allows us to allocate tenants from the non-full cluster with the most tenants already assigned to it. 2022-06-27 12:28:55 -07:00
A.J. Beamon 4b69723a2f Cleanup metadata associated with a cluster when force removing it. Move some metacluster metadata into the normal key-space. Cleanup some todos. 2022-06-24 14:55:57 -07:00
Markus Pilman d35445a868 enforce include modularization in cmake 2022-06-23 14:37:35 -06:00
A.J. Beamon 9f3819752f Change the command to create a metacluster from using 'configure tenant_mode=management' to 'metacluster create <NAME>'. Distribute this name to all processes in a metacluster. Eliminate the tenant mode entirely from metacluster clusters, instead relying on a metacluster registration key. 2022-06-22 12:15:43 -07:00
A.J. Beamon 96fad987eb Add new command to get total metacluster capacity. Fix bug where removing a data cluster could make it impossible to create new tenants on it. Add knobs for max data clusters and max tenants. 2022-06-17 10:42:03 -07:00
sfc-gh-tclinkenbeard 308e0113a8 Update fdbcli quota usage output 2022-06-16 16:26:45 -07:00
sfc-gh-tclinkenbeard 99d243197e Add fdbcli quota command 2022-06-16 14:07:16 -07:00
A.J. Beamon 2907aafa60 Merge branch 'main' into feature-metacluster 2022-06-02 12:18:42 -07:00
Jingyu Zhou db5508520c
Merge pull request #7249 from johscheuer/fix-fdbcli-exclude-check
Check if all specific processes are excluded
2022-05-26 15:01:01 -07:00
Johannes M. Scheuermann 8277965158 Correct variable for for loop to iterate over inprogress exclusions 2022-05-25 17:21:15 +01:00
Johannes M. Scheuermann 0bc8e2cea6 Check if all specific processes are excluded 2022-05-25 17:08:02 +01:00
A.J. Beamon 9773261a03 Expand support for tenant groups. Track where each tenant group is assigned and used groups to help assign tenants. Fix tenant state tracking bugs. Add help hints for metacluster remove FORCE. 2022-05-24 11:58:40 -07:00
Xiaoxi Wang fd35fde481 Merge branch 'main' of https://github.com/apple/foundationdb into readaware 2022-05-23 15:09:03 -07:00
Lukas Joswiak 475b157310 Add `get` command to versionepoch help text 2022-05-23 11:45:18 -07:00
Xiaoxi Wang 73624bcd2a Merge remote-tracking branch 'upstream/main' into readaware 2022-05-23 11:17:38 -07:00
A.J. Beamon d784173f7f Add first implementation of tenant creation and deletion in a metacluster 2022-05-20 15:21:21 -07:00
Chaoguang Lin 5e8c3130e6 Extend the reboot interface to accept comma-delimited string;
Only fetch requested worker interfaces in rebootWorkerActor
Add killall test to multiprocess_fdbcli_tests
Make the rebootWorker only sends requests if all the requested workers are valid
Add comments; Organize print texts; Fix the bug clear the map every time run kill/suspend/expensive_data_check
2022-05-20 12:06:33 -07:00
A.J. Beamon 69261f9f10 Merge branch 'main' into feature-metacluster
# Conflicts:
#	fdbclient/MultiVersionTransaction.actor.cpp
#	fdbclient/MultiVersionTransaction.h
#	fdbserver/tester.actor.cpp
2022-05-19 09:11:30 -07:00
A.J. Beamon 367e59dc33 Add support for getting tenant and cluster metadata in fdbcli in a JSON format 2022-05-18 14:53:25 -07:00
A.J. Beamon 98c3813431 Refactor the register logic so that only two write transactions take place. The first is to the data cluster, the second to the management cluster. 2022-05-18 11:21:39 -07:00
A.J. Beamon e8e26c9f7c Remove the internal special key-space mechanism (at least for now), instead relying on the local logic to implement the metacluster functions. Add ability to force remove a cluster. Improve idempotency of register operation. 2022-05-18 08:18:25 -07:00
Xiaoxi Wang 382f0fc4a2 merge upstream/main 2022-05-17 10:20:51 -07:00
Evan Tschannen 583120514f fixed compilation errors 2022-05-12 17:19:25 -07:00
Evan Tschannen c51ad847d6 revert "kill" command to previous 7.0 behavior because the current implementation is killing processes one at a time 2022-05-12 13:12:23 -07:00
A.J. Beamon 85019612d4 Remove subordinate mode and instead use a key in the system key-space to track metacluster membership. Use this key to determine if a cluster is already part of a metacluster or can be configured away from required mode. Disallow configuring to or from a management cluster that has data. 2022-05-11 17:10:55 -07:00
Vishesh Yadav f14baf2af8 clang-format changes 2022-05-09 14:54:51 -07:00
Vishesh Yadav 9173e2e19b Move GlobalConfig to DatabaseContext 2022-05-09 14:54:51 -07:00
Vishesh Yadav 7578d5ebc7 Create GlobalConfig object for each database instance
Currently, GlobalConfig is a singleton that means for each process there is only
one GlobalConfig object. This is bug from clients perspective as a client can
keep connections to several databases. This patch tracks GlobalConfig for each
database using an unordered_map in flowGlobals.

We discovered this bug while testing multi-version client, where the client got
stuck. This was lucky, as normally it'd just write down config to the wrong
database.
2022-05-09 14:54:51 -07:00
A.J. Beamon 80a3c8dd50 All metacluster operations to the management cluster go through the management cluster special key space 2022-05-09 14:50:07 -07:00
A.J. Beamon 7c3b99f5bd Add support for connection to clusters via a connection string. Start work on multi-cluster operations with data cluster registration. 2022-05-06 09:53:53 -07:00
Xiaoxi Wang 992ff59bfb Merge branch 'main' of https://github.com/apple/foundationdb into readaware 2022-05-05 09:57:22 -07:00
A.J. Beamon 0c41225d76 Add support for setting and storing the tenant group for a tenant 2022-05-05 08:57:01 -07:00
Trevor Clinkenbeard 0069ce51a5
Merge pull request #7054 from sfc-gh-tclinkenbeard/remove-non-tls-support
Remove `DISABLE_TLS` CMake option
2022-05-03 22:13:07 -07:00
Xiaoxi Wang 7c37d172b9 solve some comments 2022-05-03 17:21:08 -07:00
Xiaoxi Wang 269d85daa8 Merge branch 'main' of https://github.com/apple/foundationdb into readaware 2022-05-03 13:37:56 -07:00
A.J. Beamon 7e860dc92c Some initial metacluster implementation 2022-05-03 12:53:35 -07:00
sfc-gh-tclinkenbeard 225146176d Apply clang-format to fdbcli.actor.cpp and Net2.actor.cpp 2022-05-03 12:13:09 -07:00
sfc-gh-tclinkenbeard 06825775db Fix formatting of lines with TLS_OPTION_FLAGS 2022-05-02 22:56:06 -07:00
sfc-gh-tclinkenbeard 7f05221cfe Removed TLS_DISABLED macro 2022-05-02 22:15:27 -07:00
Xiaoxi Wang 69985ba251 Merge branch 'main' of https://github.com/apple/foundationdb into readaware 2022-05-02 10:53:22 -07:00
A.J. Beamon 43c2ca35a5 Move fdbcli command and hint generators into the files implementing the command. 2022-05-02 08:39:59 -07:00
Renxuan Wang c69a07a858
Check in the new Hostname logic. (#6926)
* Revert #6655.

20220407-031010-renxuan-c101052c21da8346           compressed=True data_size=31004844 duration=4310801 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=1:04:15 sanity=False started=100047 stopped=20220407-041425 submitted=20220407-031010 timeout=5400 username=renxuan

* Revert #6271.

20220407-051532-renxuan-470f0fe6aac1c217           compressed=True data_size=30982370 duration=3491067 ended=100002 fail_fast=10 max_runs=100000 pass=100002 priority=100 remaining=0 runtime=0:59:57 sanity=False started=100141 stopped=20220407-061529 submitted=20220407-051532 timeout=5400 username=renxuan

* Revert #6266.

Remove resolving-related functionalities in connection string. Connection string will be used for storing purpose only, and non-mutable.

20220407-175119-renxuan-55d30ee1a4b42c2f           compressed=True data_size=30970443 duration=5437659 ended=100000 fail_fast=10 max_runs=100000 pass=100000 priority=100 remaining=0 runtime=0:59:31 sanity=False started=100154 stopped=20220407-185050 submitted=20220407-175119 timeout=5400 username=renxuan

* Add hostname to coordinator interfaces.

* Turn on the new hostname logic.

* Add the corresponding change in config txns.

The most notable change is before calling basicLoadBalance(), we need to call tryInitializeRequestStream() to initialize request streams first.

Passed correctness tests.

* Return error when hostnames cannot be resolved in coordinators command.

* Minor fixes.
2022-04-27 21:54:13 -07:00
Xiaoxi Wang 0639810b66 merge upstream/main 2022-04-22 11:09:15 -07:00
Zhe Wang 6c9ff6ee5e
Add sharded rocksdb type (#6862)
* add-sharded-rocksdb-type

* address comments

Co-authored-by: Zhe Wang <zhewang@Zhes-MacBook-Pro.local>
2022-04-21 22:53:14 -04:00
Xiaoxi Wang d17b36acf8 Merge branch 'main' of https://github.com/apple/foundationdb into readaware 2022-04-15 11:10:09 -07:00
Andrew Noyes 29cf5f1fbf
Fix an ASSERT when an fdbcli command times out (#6857)
* Re-throw operation_cancelled

There's a few places in fdbcli where we don't rethrow operation
cancelled but wait on a future. It's very unusual that you don't want to
rethrow operation_cancelled.

* Update ASSERT

It's possible to get error_code_broken_promise here if the network has
already shutdown.
2022-04-14 12:09:25 -07:00
Xiaoxi Wang 8d3f851495 merge upstream/mainA 2022-04-12 17:03:09 -07:00
Xiaoxi Wang ed97a35dc0 Merge branch 'main' into readaware 2022-04-12 16:47:15 -07:00
Xiaoxi Wang 61a1f7683b fix dd command line read special key space error 2022-04-11 22:49:21 -07:00
Aaron Molitor cbaef8f03b update version to 7.2.0 2022-04-11 23:23:27 -05:00
Xiaoxi Wang 6c841cd32b merge readaware-better 2022-04-11 17:09:39 -07:00
Xiaoxi Wang 82e5859e03 allow safeThreadFutureToFuture 2022-04-11 14:35:12 -07:00
Lukas Joswiak 73a7c32982
Add fdbcli command to read/write version epoch (#6480)
* Initialize cluster version at wall-clock time

Previously, new clusters would begin at version 0. After this change,
clusters will initialize at a version matching wall-clock time. Instead
of using the Unix epoch (or Windows epoch), FDB clusters will use a new
epoch, defaulting to January 1, 2010, 01:00:00+00:00. In the future,
this base epoch will be modifiable through fdbcli, allowing
administrators to advance the cluster version.

Basing the version off of time allows different FDB clusters to share
data without running into version issues.

* Send version epoch to master

* Cleanup

* Update fdbserver/storageserver.actor.cpp

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>

* Jump directly to expected version if possible

* Fix initial version issue on storage servers

* Add random recovery offset to start version in simulation

* Type fixes

* Disable reference time by default

Enable on a cluster using the fdbcli command `versionepoch add 0`.

* Use correct recoveryTransactionVersion when recovering

* Allow version epoch to be adjusted forwards (to decrease the version)

* Set version epoch in simulation

* Add quiet database check to ensure small version offset

* Fix initial version issue on storage servers

* Disable reference time by default

Enable on a cluster using the fdbcli command `versionepoch add 0`.

* Add fdbcli command to read/write version epoch

* Cause recovery when version epoch is set

* Handle optional version epoch key

* Add ability to clear the version epoch

This causes version advancement to revert to the old methodology whereas
versions attempt to advance by about a million versions per second,
instead of trying to match the clock.

* Update transaction access

* Modify version epoch to use microseconds instead of seconds

* Modify fdbcli version target API

Move commands from `versionepoch` to `targetversion` top level command.

* Add fdbcli tests for

* Temporarily disable targetversion cli tests

* Fix version epoch fetch issue

* Fix Arena issue

* Reduce max version jump in simulation to 1,000,000

* Rework fdbcli API

It now requires two commands to fully switch a cluster to using the
version epoch. First, enable the version epoch with `versionepoch
enable` or `versionepoch set <versionepoch>`. At this point, versions
will be given out at a faster or slower rate in an attempt to reach the
expected version. Then, run `versionepoch commit` to perform a one time
jump to the expected version. This is essentially irreversible.

* Temporarily disable old targetversion tests

* Cleanup

* Move version epoch buggify to sequencer

This will cause some issues with the QuietDatabase check for the version
offset - namely, it won't do anything, since the version epoch is not
being written to the txnStateStore in simulation. This will get fixed in
the future.

Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-04-08 12:33:19 -07:00
Xiaoxi Wang 5113277e9a Merge branch 'main' into readaware 2022-04-08 11:50:01 -07:00
Xiaoxi Wang 411550580e merge 2022-04-08 11:49:33 -07:00
neethuhaneesha 5311d93c72
Merge pull request #6783 from neethuhaneesha/rocksdbEngineType
Adding warning on configuring rocksdb storage until production tested.
2022-04-07 13:29:12 -07:00
Xiaoxi Wang 150b4318aa refactor datadistribution command; try dual-mode code 2022-04-06 22:10:23 -07:00
Yi Wu 994b8c92f8
Add option to limit resident memory and remove default memory limit (#6719)
Changing `memory` option to limit resident memory instead of virtual memory, in config file and fdbserver/fdbbackup/fdbcli command-line argument. Since `rlimit` doesn't support limiting virtual memory, the current implementation have both of fdbmonitor and the fdbserver/fdbbackup process checking process RSS periodically and kill and restart the process if the limit is exceeded.

Adding a new `memory_vsize` option to limit virtual memory, if backward-compatible behavior is desired.

closes #6671, closes #6672
2022-04-06 20:06:24 -07:00
Neethu Haneesha Bingi 5d3981821e Adding warning on configuring rocksdb storage until production tested. 2022-04-06 13:24:57 -07:00
Xiaoxi Wang aba9d85560 merge main 2022-04-06 09:57:52 -07:00
Jingyu Zhou f68fd28d73 Refactor duplicated code into IKnobCollection::setupKnobs() 2022-04-05 02:06:38 -07:00
Xiaoxi Wang d6d4596c35
Merge pull request #6709 from sfc-gh-ajbeamon/fix-fdbcli-bugs
Fix issues with command completion for exclude and storage_migration_type
2022-03-30 19:53:49 -07:00
Andrew Noyes d727e7648e Fix a few memory issues found by ASAN 2022-03-29 12:39:12 -07:00
A.J. Beamon 23fcd8c076 Fix issues with command completion for exclude and storage_migration_type. Add missing documentation for tenant_mode in one spot. 2022-03-29 09:07:33 -07:00
Xiaoxi Wang d93b57dd88 conflict solving 2022-03-24 20:45:51 -07:00
Xiaoxi Wang 1b631a9263 solve conflict with main 2022-03-24 16:29:11 -07:00
Josh Slocum f27475e2f4 Merge branch 'main' into blob_integration 2022-03-22 11:41:58 -05:00
sfc-gh-tclinkenbeard a71099471b Update copyright header dates 2022-03-21 13:36:23 -07:00
A.J. Beamon fd81ef99c0
Merge pull request #6561 from sfc-gh-ajbeamon/fdb-tenant-fdbcli
Add fdbcli support for tenants
2022-03-21 11:26:10 -07:00
sfc-gh-tclinkenbeard be875e040d Prevent infinite loop on invalid input to 'throttle off' command 2022-03-18 20:22:49 -07:00
sfc-gh-tclinkenbeard 004e73b2f1 Add TagSet::toString method 2022-03-18 20:10:56 -07:00
sfc-gh-tclinkenbeard fdab73d7b3 Rename TagSet variables in throttleCommandActor.
Previously these were named "tags", conflicting with a state variable,
and creating confusing issues when they go out of scope after a wait
statement.
2022-03-18 19:20:25 -07:00
sfc-gh-tclinkenbeard 6eaa20078f Add ContainsRecommended boolean parameter 2022-03-18 18:12:50 -07:00
sfc-gh-tclinkenbeard 63ec6d5f5b Add static defaultThrottleListLimit in ThrottleCommand.actor.cpp 2022-03-18 17:14:34 -07:00
Josh Slocum 37e7c80f26 Merge branch 'main' into blob_integration 2022-03-17 18:45:42 -05:00
A.J. Beamon a23add6bc4 Add fdbcli test for tenants. Add documentation for new fdbcli tenant commands. Various output cleanup. Fix limit parsing bug in listtenants command. Update gettenant output format. 2022-03-17 12:10:39 -07:00
A.J. Beamon bd64781ad9 Use special keys to manage tenants 2022-03-17 12:10:39 -07:00
A.J. Beamon 4aa053d342 Update tenant mode options in fdbcli 2022-03-17 12:10:39 -07:00