Commit Graph

761 Commits

Author SHA1 Message Date
Josh Slocum 94e0c83214 Blob range idempotency test and fixes 2022-09-02 15:53:06 -05:00
Jon Fu 2836852a7f Merge branch 'main' of github.com:apple/foundationdb into jfu-metacluster-rename 2022-08-16 16:14:28 -07:00
Dennis Zhou 3d400cff64 blob: verifyBlobRange() c api 2022-08-16 13:29:23 -07:00
Dennis Zhou 1c2109dcbd blob: add rangeLimit to getBlobGranuleRanges() 2022-08-16 13:29:23 -07:00
Dennis Zhou 5085c21d0d blob: check if blob range exists and fail accordingly 2022-08-15 16:25:36 -07:00
Dennis Zhou 736ef4f2c9 blob: move blobrange command implementation to native api 2022-08-15 16:25:36 -07:00
Chaoguang Lin 3fed0456ca
Add an verify option for \xff\xff/worker_interfaces special keys (#7873)
* Add the verify option for \xff\xff/worker_interfaces

* Remove unused code

* update documentations

* update documentations

* solve comments from review

* update some of the comments to be more clear
2022-08-15 14:05:07 -04:00
Jon Fu 0c85efee43 Merge branch 'main' of github.com:apple/foundationdb into jfu-metacluster-rename 2022-08-14 11:34:16 -07:00
Josh Slocum 7c155f4521
Granule force purging (#7846)
* Granule purge cannot delete history entry for fully deleting granule until all children are completely done splitting

* Several purging fixes related to granule history

* Fixed typo in refactor

* fixing memory model for purgeRange

* formatting

* weakening granule purge test for now

* cleanup

* First version of force purging granules

* fixing issue in BW range assignment reporting

* Fixing incorrect assert with force purging

* Error handling when checking force purged state

* fixed force purging and recover/reassign range races and check

* Handling force purge + boundary change race

* more places to check for force purged status

* fixed manager restart in the middle of force purge bug

* fixing same-BM purge and assignment races in all cases

* weakening orphaned granule history check a bit because of difficult to solve races

* fixing txn options on retry

* loading force purged ranges at start to avoid resuming a merge that is being force purged

* cleanup

* Enabling purging in granule tests, and adding check for leaked change feeds in force purge

* formatting

* missed parameter in merge conflicts

* Fixing leaked change feed race with merge and force purge

* adding change feed cleanup when new blob manager recovers in-progress merge that raced with force purge

* added forcepurge fdbcli command
2022-08-11 15:22:32 -07:00
Jon Fu 0487aa5445 Merge branch 'main' of github.com:apple/foundationdb into jfu-metacluster-rename 2022-08-10 14:07:43 -07:00
A.J. Beamon cd82f65ef5 Change the data cluster entry to use a state field rather than a locked boolean 2022-08-09 11:53:57 -07:00
Jon Fu 74df84f686 Merge branch 'main' of github.com:apple/foundationdb into jfu-metacluster-rename 2022-08-08 17:49:15 -07:00
Vaidas Gasiunas 79571dd2b4
Testing upgrades to a future version of FDB (#7780)
* Enable configuring the next future protocol version as the current protocol version in FDB client, fdbserver, and fdbcli

* Auto format python files used in upgrade tests

* Add a test for upgrading to a future FDB version

* Emphasize that the options for using future protocol version are intended for test purposes only

* Make the global variable for current protocol version visible only locally

* Refactirng to avoid using currentProtocolVersion() in static intialization

* Update go bindings
2022-08-08 17:29:49 +02:00
Jon Fu cc85973957 Merge branch 'main' of github.com:apple/foundationdb into jfu-metacluster-rename 2022-08-05 12:38:43 -07:00
Hui Liu 29ad2c0654
fdbcli: show status details about # works and # key ranges if blob granules enabled (#7792)
* fdbcli: show status if blob granules is enabled

* fdbcli: show status details for blob granules for # works and # key ranges
2022-08-05 12:33:57 -05:00
Josh Slocum ddbf32d35e
adding knob and fixing blobrange cli check bug with too many granules (#7793) 2022-08-05 11:19:17 -05:00
Jon Fu 7a45aba74d Merge branch 'main' of github.com:apple/foundationdb into jfu-metacluster-rename 2022-08-04 16:36:09 -07:00
A.J. Beamon ff23d5994e
Merge pull request #7729 from sfc-gh-ajbeamon/feature-metacluster
Metacluster
2022-08-04 15:29:44 -07:00
Hui Liu 4f75f01882
fdbcli: show status if blob granules is enabled (#7784) 2022-08-03 19:15:34 -05:00
Josh Slocum 1cda8a2fc1
More blob granule operational stuff (#7783)
* More blob manager metrics

* making blobrange check command work for large ranges
2022-08-03 18:11:25 -05:00
Jon Fu bceb44f5a1 address code review comments 2022-08-03 12:03:44 -07:00
Jon Fu 5dc8e930a9 remove whitespace 2022-08-01 11:06:04 -07:00
Jon Fu d9dc1999c6 metacluster rename before using metacluster operation ctx 2022-07-29 13:43:47 -07:00
A.J. Beamon a69164d160 Merge branch 'main' into feature-metacluster 2022-07-29 11:53:06 -07:00
Josh Slocum cfc13e7018
Adding more blobrange cli commands and a couple other tweaks (#7727) 2022-07-29 08:20:45 -07:00
A.J. Beamon e5bf838c83 Fix metacluster register command hints, fix bug in cluster registration on retries, extend the length of the tenant management concurrency test 2022-07-28 21:02:12 -07:00
A.J. Beamon 65cfc839a6 Some various cleanup 2022-07-28 13:35:24 -07:00
A.J. Beamon 7f53b340f5 Merge branch 'main' into feature-metacluster
# Conflicts:
#	bindings/python/tests/fdbcli_tests.py
#	fdbcli/TenantCommands.actor.cpp
#	fdbclient/ClientKnobs.cpp
#	fdbclient/include/fdbclient/ClientKnobs.h
#	fdbclient/include/fdbclient/Tenant.h
#	fdbserver/BlobManager.actor.cpp
#	fdbserver/BlobWorker.actor.cpp
#	fdbserver/TenantCache.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/TenantManagementWorkload.actor.cpp
#	flow/include/flow/error_definitions.h
2022-07-28 11:43:21 -07:00
A.J. Beamon 7c6b3fb0b8 Merge branch 'main' into feature-metacluster 2022-07-27 08:55:10 -07:00
Jon Fu 6dbaf46ec8 restructure test workload impl and map 2022-07-26 13:53:04 -07:00
Jon Fu 21309c2af8 Merge branch 'feature-tenant-groups' of github.com:sfc-gh-ajbeamon/foundationdb into jfu-tenant-rename-special-keys 2022-07-26 11:15:56 -07:00
A.J. Beamon 59ccdc35ab Disallow configuring the same tenant parameter more than once at a time. Made some slight changes to serialization code for tenants. 2022-07-26 09:04:29 -07:00
A.J. Beamon f2a8af9155 Better API version handling for fdbcli tenant commands 2022-07-26 09:04:29 -07:00
A.J. Beamon a64693518a Add support for tenant groups 2022-07-26 09:04:29 -07:00
A.J. Beamon 2990294f6a Make the metacluster experimental by updating the name of the command used to configure one. 2022-07-24 19:11:25 -07:00
Jon Fu 84269bac3e Merge branch 'feature-tenant-groups' of github.com:sfc-gh-ajbeamon/foundationdb into jfu-tenant-rename-special-keys 2022-07-22 16:42:27 -07:00
A.J. Beamon 2b6f349649 Better API version handling for fdbcli tenant commands 2022-07-22 15:37:48 -07:00
Jon Fu 43c7146d86 Merge branch 'feature-tenant-groups' of github.com:sfc-gh-ajbeamon/foundationdb into jfu-tenant-rename-special-keys 2022-07-22 15:21:13 -07:00
A.J. Beamon b308aa6604 Merge branch 'feature-tenant-groups' into feature-metacluster 2022-07-22 11:38:23 -07:00
A.J. Beamon 237081905d Add support for tenant groups 2022-07-22 09:27:31 -07:00
A.J. Beamon 17146c484b Use key-backed types for tenants. Add a tenant state field that will be used in upcoming work. Some other tenant related refactoring. 2022-07-21 20:33:28 -07:00
A.J. Beamon 3e1763e7ac Merge branch 'main' into feature-metacluster 2022-07-21 14:48:53 -07:00
A.J. Beamon 410f27412b
Merge pull request #7620 from sfc-gh-ajbeamon/make-tuple
Add a Tuple::makeTuple function to easily construct a tuple
2022-07-20 17:09:10 -07:00
A.J. Beamon 279296c29f Merge branch 'tenant-metadata-change' into feature-metacluster
# Conflicts:
#	fdbclient/SystemData.cpp
#	fdbclient/Tenant.cpp
#	fdbclient/include/fdbclient/SystemData.h
#	fdbclient/include/fdbclient/Tenant.h
#	fdbclient/include/fdbclient/TenantManagement.actor.h
#	fdbserver/TenantCache.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/TenantManagementWorkload.actor.cpp
2022-07-20 09:18:27 -07:00
A.J. Beamon 190ad8c7e9 Convert existing tuple usages to use Tuple::makeTuple() 2022-07-19 13:45:59 -07:00
Markus Pilman 1de37afd52
Make TEST macros C++ only (#7558)
* proof of concept

* use code-probe instead of test

* code probe working on gcc

* code probe implemented

* renamed TestProbe to CodeProbe

* fixed refactoring typo

* support filtered output

* print probes at end of simulation

* fix missed probes print

* fix deduplication

* Fix refactoring issues

* revert bad refactor

* make sure file paths are relative

* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
A.J. Beamon 1b81e72604 Add a Tuple::makeTuple function to easily construct a tuple. Update Tuple to allow all types to be passed via .append() so they can be used with makeTuple. 2022-07-19 11:50:58 -07:00
A.J. Beamon c7c7bd5f17 Undo change to mark tenant modes non-experimental. Other minor cleanup. 2022-07-19 09:18:53 -07:00
A.J. Beamon 8f77048e95 Add the ability to configure tenant groups in a metacluster 2022-07-18 11:24:49 -07:00
A.J. Beamon 860d3843cc Merge remote-tracking branch 'origin/feature-tenant-groups' into feature-metacluster 2022-07-16 19:33:26 -07:00