Commit Graph

772 Commits

Author SHA1 Message Date
Dennis Zhou e6222be6c6 fdbcli: use ApiVersion::LATEST_VERSION instead of hardcoding 2022-09-12 17:08:57 -07:00
Xiaoxi Wang 79ccc453aa Merge branch 'main' of https://github.com/apple/foundationdb into fix/main/ppwmessage 2022-09-08 09:24:04 -07:00
Andrew Noyes 984150bf74
Link fdbcli with -rdynamic for ubsan builds (#8033)
Closes #7957
2022-09-08 08:27:57 -07:00
A.J. Beamon 726d5215a0
Remove API 720 guards for tenants (experimental feature) and the cluster ID special keys (#8108)
* Remove API 720 guards for tenants (experimental feature) and the cluster ID special keys (no need to guard)

* Enable the relaxed special key access in transactions that need to use special key-space APIs introduced in 7.2
2022-09-08 17:22:36 +02:00
Dennis Zhou 7fc4a92d09
Merge pull request #8089 from sfc-gh-jslocum/unblobbify_idempotent
Blob range idempotency test and fixes
2022-09-07 17:31:05 -07:00
Xiaoxi Wang 50167b8c13 Merge branch 'main' of https://github.com/apple/foundationdb into fix/main/ppwmessage 2022-09-07 09:48:25 -07:00
Xiaoxi Wang a12f2dbdce make configure perpetual_storage_wiggle=0 a true warning when the storage_migration_type=gradual 2022-09-06 09:43:38 -07:00
Junhyun Shim f2e68d0566
Merge pull request #8102 from sfc-gh-jshim/fdbcli-tls-connect
Make FDBCLI check coordinator :tls suffix
2022-09-06 16:17:49 +02:00
Josh Slocum ca22a47107 fixing fdbcli 2022-09-02 17:41:31 -05:00
Ata E Husain Bohra 28e608e717
Encryption data at-rest db-config (#7929)
* Encryption data at-rest db-config

Description

 diff-1: Handle 'force' updates to encryption_at_rest db-config

Major changes proposed:
1. Introduce 'encryption_data_at_rest_mode" 'configure new'
option to enable Encryption data at-rest. The feature is disabled
by default.
2. The configuration is meant to be set at the time of database
creation, addition checks will be done to avoid updating the config
in subsequent PR.
3. DatabaseConfiguration validity check to account for "tenant_mode"
set to `required` if Encryption data at-rest is selected given
EncryptionDomain matches Tenant boundaries.

Testing

devCorrectness - 100K
2022-09-02 14:11:38 -07:00
Josh Slocum 94e0c83214 Blob range idempotency test and fixes 2022-09-02 15:53:06 -05:00
Junhyun Shim bcf0861a0a Make fdbcli check for TLSConfig and coordinator address mismatch
At least one of the coordinator addresses in cluster file must contain ":tls" suffix
if fdbcli's resolved TLS client configuration holds any of the TLS elements (key, cert, or CA)

Conversely, if none of the TLS elements are configured,
at least one of coordinator addresses must be without ":tls" suffix
2022-09-01 15:50:11 +02:00
Jon Fu 2836852a7f Merge branch 'main' of github.com:apple/foundationdb into jfu-metacluster-rename 2022-08-16 16:14:28 -07:00
Dennis Zhou 3d400cff64 blob: verifyBlobRange() c api 2022-08-16 13:29:23 -07:00
Dennis Zhou 1c2109dcbd blob: add rangeLimit to getBlobGranuleRanges() 2022-08-16 13:29:23 -07:00
Dennis Zhou 5085c21d0d blob: check if blob range exists and fail accordingly 2022-08-15 16:25:36 -07:00
Dennis Zhou 736ef4f2c9 blob: move blobrange command implementation to native api 2022-08-15 16:25:36 -07:00
Chaoguang Lin 3fed0456ca
Add an verify option for \xff\xff/worker_interfaces special keys (#7873)
* Add the verify option for \xff\xff/worker_interfaces

* Remove unused code

* update documentations

* update documentations

* solve comments from review

* update some of the comments to be more clear
2022-08-15 14:05:07 -04:00
Jon Fu 0c85efee43 Merge branch 'main' of github.com:apple/foundationdb into jfu-metacluster-rename 2022-08-14 11:34:16 -07:00
Josh Slocum 7c155f4521
Granule force purging (#7846)
* Granule purge cannot delete history entry for fully deleting granule until all children are completely done splitting

* Several purging fixes related to granule history

* Fixed typo in refactor

* fixing memory model for purgeRange

* formatting

* weakening granule purge test for now

* cleanup

* First version of force purging granules

* fixing issue in BW range assignment reporting

* Fixing incorrect assert with force purging

* Error handling when checking force purged state

* fixed force purging and recover/reassign range races and check

* Handling force purge + boundary change race

* more places to check for force purged status

* fixed manager restart in the middle of force purge bug

* fixing same-BM purge and assignment races in all cases

* weakening orphaned granule history check a bit because of difficult to solve races

* fixing txn options on retry

* loading force purged ranges at start to avoid resuming a merge that is being force purged

* cleanup

* Enabling purging in granule tests, and adding check for leaked change feeds in force purge

* formatting

* missed parameter in merge conflicts

* Fixing leaked change feed race with merge and force purge

* adding change feed cleanup when new blob manager recovers in-progress merge that raced with force purge

* added forcepurge fdbcli command
2022-08-11 15:22:32 -07:00
Jon Fu 0487aa5445 Merge branch 'main' of github.com:apple/foundationdb into jfu-metacluster-rename 2022-08-10 14:07:43 -07:00
A.J. Beamon cd82f65ef5 Change the data cluster entry to use a state field rather than a locked boolean 2022-08-09 11:53:57 -07:00
Jon Fu 74df84f686 Merge branch 'main' of github.com:apple/foundationdb into jfu-metacluster-rename 2022-08-08 17:49:15 -07:00
Vaidas Gasiunas 79571dd2b4
Testing upgrades to a future version of FDB (#7780)
* Enable configuring the next future protocol version as the current protocol version in FDB client, fdbserver, and fdbcli

* Auto format python files used in upgrade tests

* Add a test for upgrading to a future FDB version

* Emphasize that the options for using future protocol version are intended for test purposes only

* Make the global variable for current protocol version visible only locally

* Refactirng to avoid using currentProtocolVersion() in static intialization

* Update go bindings
2022-08-08 17:29:49 +02:00
Jon Fu cc85973957 Merge branch 'main' of github.com:apple/foundationdb into jfu-metacluster-rename 2022-08-05 12:38:43 -07:00
Hui Liu 29ad2c0654
fdbcli: show status details about # works and # key ranges if blob granules enabled (#7792)
* fdbcli: show status if blob granules is enabled

* fdbcli: show status details for blob granules for # works and # key ranges
2022-08-05 12:33:57 -05:00
Josh Slocum ddbf32d35e
adding knob and fixing blobrange cli check bug with too many granules (#7793) 2022-08-05 11:19:17 -05:00
Jon Fu 7a45aba74d Merge branch 'main' of github.com:apple/foundationdb into jfu-metacluster-rename 2022-08-04 16:36:09 -07:00
A.J. Beamon ff23d5994e
Merge pull request #7729 from sfc-gh-ajbeamon/feature-metacluster
Metacluster
2022-08-04 15:29:44 -07:00
Hui Liu 4f75f01882
fdbcli: show status if blob granules is enabled (#7784) 2022-08-03 19:15:34 -05:00
Josh Slocum 1cda8a2fc1
More blob granule operational stuff (#7783)
* More blob manager metrics

* making blobrange check command work for large ranges
2022-08-03 18:11:25 -05:00
Jon Fu bceb44f5a1 address code review comments 2022-08-03 12:03:44 -07:00
Jon Fu 5dc8e930a9 remove whitespace 2022-08-01 11:06:04 -07:00
Jon Fu d9dc1999c6 metacluster rename before using metacluster operation ctx 2022-07-29 13:43:47 -07:00
A.J. Beamon a69164d160 Merge branch 'main' into feature-metacluster 2022-07-29 11:53:06 -07:00
Josh Slocum cfc13e7018
Adding more blobrange cli commands and a couple other tweaks (#7727) 2022-07-29 08:20:45 -07:00
A.J. Beamon e5bf838c83 Fix metacluster register command hints, fix bug in cluster registration on retries, extend the length of the tenant management concurrency test 2022-07-28 21:02:12 -07:00
A.J. Beamon 65cfc839a6 Some various cleanup 2022-07-28 13:35:24 -07:00
A.J. Beamon 7f53b340f5 Merge branch 'main' into feature-metacluster
# Conflicts:
#	bindings/python/tests/fdbcli_tests.py
#	fdbcli/TenantCommands.actor.cpp
#	fdbclient/ClientKnobs.cpp
#	fdbclient/include/fdbclient/ClientKnobs.h
#	fdbclient/include/fdbclient/Tenant.h
#	fdbserver/BlobManager.actor.cpp
#	fdbserver/BlobWorker.actor.cpp
#	fdbserver/TenantCache.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/TenantManagementWorkload.actor.cpp
#	flow/include/flow/error_definitions.h
2022-07-28 11:43:21 -07:00
A.J. Beamon 7c6b3fb0b8 Merge branch 'main' into feature-metacluster 2022-07-27 08:55:10 -07:00
Jon Fu 6dbaf46ec8 restructure test workload impl and map 2022-07-26 13:53:04 -07:00
Jon Fu 21309c2af8 Merge branch 'feature-tenant-groups' of github.com:sfc-gh-ajbeamon/foundationdb into jfu-tenant-rename-special-keys 2022-07-26 11:15:56 -07:00
A.J. Beamon 59ccdc35ab Disallow configuring the same tenant parameter more than once at a time. Made some slight changes to serialization code for tenants. 2022-07-26 09:04:29 -07:00
A.J. Beamon f2a8af9155 Better API version handling for fdbcli tenant commands 2022-07-26 09:04:29 -07:00
A.J. Beamon a64693518a Add support for tenant groups 2022-07-26 09:04:29 -07:00
A.J. Beamon 2990294f6a Make the metacluster experimental by updating the name of the command used to configure one. 2022-07-24 19:11:25 -07:00
Jon Fu 84269bac3e Merge branch 'feature-tenant-groups' of github.com:sfc-gh-ajbeamon/foundationdb into jfu-tenant-rename-special-keys 2022-07-22 16:42:27 -07:00
A.J. Beamon 2b6f349649 Better API version handling for fdbcli tenant commands 2022-07-22 15:37:48 -07:00
Jon Fu 43c7146d86 Merge branch 'feature-tenant-groups' of github.com:sfc-gh-ajbeamon/foundationdb into jfu-tenant-rename-special-keys 2022-07-22 15:21:13 -07:00
A.J. Beamon b308aa6604 Merge branch 'feature-tenant-groups' into feature-metacluster 2022-07-22 11:38:23 -07:00