Commit Graph

12377 Commits

Author SHA1 Message Date
Josh Slocum 716a9c3817 addressing review comments and fixmes in bg commit proxy code 2023-02-27 10:51:47 -06:00
Josh Slocum 4fa528f3fa added unit test for bg tenant deletion 2023-02-27 10:21:26 -06:00
Josh Slocum 1e7368aa21 removing double counted counter 2023-02-27 10:18:50 -06:00
Josh Slocum faf8ec6281 adding authz blob management tests to ensure tenant-scoped clients cannot do blob management 2023-02-27 08:06:36 -06:00
Steve Atherton cad7596e2d Apply clang-format. 2023-02-27 01:07:35 -08:00
Steve Atherton 674c105050
Merge pull request #9473 from sfc-gh-etschannen/feature-change-feed-lock
Replace fetchKeysParallelismFullLock to speed up fetch keys in idle clusters
2023-02-26 23:18:43 -08:00
Steve Atherton 01f89d1245 Merge commit '97c355085c0e22e727a5c3e35ace926a58b16f5d' into redwood-first-commit-record 2023-02-26 21:19:17 -08:00
Steve Atherton 92bfa1d578 Throw an error if reopened Redwood file is recovered but to a point prior to when the BTree was created if the Encryption Mode of the cluster is not known. 2023-02-26 21:15:48 -08:00
Steve Atherton 29444252fc Prevent Redwood Pager from recovering to a point before the user commit record (ie the BTree CommitRecord) was initialized as this isn't a useful recovery point. 2023-02-26 01:38:24 -08:00
A.J. Beamon b5b7d51213 Fix maybe committed issue in tenant lock test 2023-02-25 20:24:16 -08:00
A.J. Beamon 846b3c731d Update metacluster consistency check to handle tenants that are in the process of being renamed correctly 2023-02-25 19:52:07 -08:00
A.J. Beamon 040d44927b Store the rename destination for tenant movements in the crluster tenant index with an ID of -1. Use this to filter out tenant aliases when modifying the tenant count during a tenant purge. 2023-02-25 19:51:21 -08:00
A.J. Beamon fddec2b573 Add restore support to the metacluster management concurrency workload 2023-02-25 19:27:04 -08:00
A.J. Beamon 8c3ee768a2 Add an option to allow exceeding the tenant group capacity limit when changing tenant configuration 2023-02-24 21:01:36 -08:00
A.J. Beamon 1c71056e26
Merge pull request #9479 from sfc-gh-nwijetunga/nim/enforce-metacluster-tenant-mode
Enforce Disabled Tenant Mode in Metacluster
2023-02-24 19:27:57 -08:00
Jingyu Zhou 6b121de6a6
Merge pull request #9464 from jzhou77/fix
Add exclude to fdbcli's configure command
2023-02-24 16:31:02 -08:00
Nim Wijetunga c8b7cff10c fix api test 2023-02-24 15:26:59 -08:00
Josh Slocum 6187811f71
Reworking getBlobGranuleRanges to also use commit proxy rpc for authz, and adding test (#9470) 2023-02-24 17:15:32 -06:00
Jingyu Zhou 204b31bc87
Merge pull request #9472 from sfc-gh-ajbeamon/fix-invalid-iterator-use
Fix memory error from using invalidated iterator
2023-02-24 14:55:28 -08:00
Nim Wijetunga eca98afcb0 metacluster check tenant mode 2023-02-24 13:59:54 -08:00
Evan Tschannen 8872e5a462
Merge pull request #9347 from sfc-gh-etschannen/feature-change-feed-cache
added a disk to blob workers
2023-02-24 13:59:03 -08:00
Josh Slocum 17bcbaeb03
add tenant reopen to ensure no races with tenant opening (#9476) 2023-02-24 15:34:57 -06:00
Dan Adkins e3a61b9b22
Add metrics to understand tail commit latency (#9435)
* Add server-side latency metrics for Resolver requests.

* Add separate resolver latency metrics for queue wait and compute time.

* Add histogram for queue depth observed on resolver (during metrics interval).

* Fix tlog latency measurement to use timer() instead of now().
2023-02-24 14:13:12 -05:00
Evan Tschannen f3673d808b Replaced the fetchKeysParallelismFullLock with a lock specifically for change feeds to avoid blocking fetches on idle clusters 2023-02-24 10:59:35 -08:00
A.J. Beamon 2d936acbdd Fix memory error from using invalidated iterator 2023-02-24 10:25:38 -08:00
A.J. Beamon 03fbc59bb1
Merge pull request #9461 from sfc-gh-ajbeamon/metacluster-concurrent-restore-testing
Metacluster concurrent restore testing
2023-02-24 09:13:51 -08:00
Josh Slocum 910965a5a6
Adding additional blob granule authz tests (#9443)
* added granule location authz tests

* added authz test for blob worker endpoint

* addressing comments

* fixing ide build
2023-02-24 09:32:05 -06:00
A.J. Beamon 344f6977c9 Restores can now throw a cluster_already_exists error in the metacluster management workload if we timeout a restore and have to retry 2023-02-23 20:32:18 -08:00
Nim Wijetunga 29819b0645
Change Feed Bug Fix + Encryption Asserts (#9457)
* add encryption asserts

* modify function name

* address pr comments

* address pr comments

* Trigger Build
2023-02-23 19:33:25 -08:00
Xiaoxi Wang 7e14324cc1 remove dataDistributionQueue actor and replace it with class DDQueue 2023-02-23 17:15:33 -08:00
A.J. Beamon 2b25cfef8b Merge branch 'main' into metacluster-concurrent-restore-testing 2023-02-23 16:06:47 -08:00
Jingyu Zhou 0b2e02c402 Fix rare test failures
Unclog after DB is recovered, otherwise another recovery may become stuck again.
2023-02-23 15:42:33 -08:00
Xiaoxi Wang 9cda7ebe5a encapsulate DDQueue constructor params into DDQueueInitParams 2023-02-23 15:38:48 -08:00
Jon Fu 33f8e90f9f
Split tenant group metadata (#9446)
* initial commit to split tenant group metadata

* attempt to fix merge errors

* fix compile errors and adjust existing tests

* fix infinite loop and extra ACTOR tag

* direct assignment instead of store

* direct assign instead of store (missed a few)
2023-02-23 18:11:49 -05:00
A.J. Beamon 3ac7e17b79 Fix create tenant usage in tenant management workload 2023-02-23 15:08:52 -08:00
Jingyu Zhou 65443b6541 Fix compiling errors 2023-02-23 15:02:44 -08:00
Jingyu Zhou ecae81882c Change to only clog once for a particular tlog
If we repeat clogging, different tlogs may be excluded, which can cause the
recovery to stuck.
2023-02-23 14:31:39 -08:00
Jingyu Zhou 6055f752c2 Exclude failed tlog if recovery stuck more than 30s
Because the tlog is clogged, recovery can stuck in initializing_transaction_servers.
This exclude allows the recovery to complete.
2023-02-23 14:31:32 -08:00
Jingyu Zhou c4773b7cc8 Update clogTlog workload to be single region 2023-02-23 14:31:24 -08:00
Jingyu Zhou 955826f2fe Add ClogTlog workload 2023-02-23 14:31:12 -08:00
Jingyu Zhou 792950dbdc
Merge pull request #9434 from sfc-gh-huliu/splitmetrics
Implement SplitMetric pagination in blob migrator
2023-02-23 14:10:27 -08:00
A.J. Beamon b828f3f257 Add missing change to explicit TenantMapEntry conversion 2023-02-23 13:38:04 -08:00
A.J. Beamon 06fe00544a Remove TenantMapEntry <-> MetaclusterTenantMapEntry conversion constructors and use named functions instead 2023-02-23 13:28:10 -08:00
A.J. Beamon dcae48cbbd Add concurrent restore testing to the metacluster restore workload 2023-02-23 13:27:20 -08:00
Xiaoxi Wang cdfaf7d4e0 move DDRelocationQueue to header file 2023-02-23 13:12:38 -08:00
A.J. Beamon e151a2d363
Merge pull request #9451 from sfc-gh-ajbeamon/metacluster-management-workload-restore-support
Improve restore support in the metacluster management workload
2023-02-23 13:10:31 -08:00
Markus Pilman efc5bf9ee8
Merge pull request #9456 from sfc-gh-ajbeamon/smaller-tenant-in-txn-state-store
Store a smaller tenant object in the txn state store
2023-02-23 14:00:12 -07:00
A.J. Beamon 9e9a31c0f1 Use error variable consistently 2023-02-23 11:27:53 -08:00
Xiaoxi Wang 021f7c1b43 remove unused headers and outdated comments 2023-02-23 11:00:21 -08:00
Xiaoxi Wang a6feafc371 remove dataDistributionTracker actor and use DataDistributionTracker class instead 2023-02-23 10:18:28 -08:00
Evan Tschannen cf3a4e6161 Merge branch 'main' into feature-change-feed-cache 2023-02-23 10:16:13 -08:00
Evan Tschannen a581a55452 ensure a worker cannot run multiple blob worker roles 2023-02-23 09:51:26 -08:00
A.J. Beamon dd650215d4 Store a smaller tenant object in the txn state store 2023-02-23 09:29:33 -08:00
Ata E Husain Bohra 7d079690d4 Merge branch 'main' into ahusain-misc-fixes 2023-02-22 18:11:11 -08:00
A.J. Beamon a76af5c696 Improve restore support in the metacluster management workload 2023-02-22 17:42:36 -08:00
Ata E Husain Bohra 1f7ee9437f EaR: RESTClient and EKP changes to handle unreachable external KMS
Description

Two major changes proposed are:

I)
Used following setup for testing:
1. Run `fdbserver` locally.
2. Run a mock python based HTTP server (encryption endpoints not implemented)

Expectation was RESTClient code should go in loop trying to establish connection
to the desired encryption endpoint. However, observation was the code loops for
one cycle and followup cycle SEGV while printing a log using RESTUrl object which
is obtained as a 'pointer' from the caller. Update the code to use RESTUrl object
instead of the pointer.

II) In above setup, KMSConnector would throw 'encrypt_key_fetch_failed' error
which wasn't handled by EKProxy, hence, causing the service to terminate. Add
code to re-throw the error to the caller.

Testing
2023-02-22 17:15:34 -08:00
A.J. Beamon 9b906d9b3d
Merge pull request #9447 from sfc-gh-ajbeamon/metacluster-restore-fixes
Metacluster restore fixes
2023-02-22 17:07:19 -08:00
Hui Liu 0fba65a3cd Implement SplitMetric pagination in blob migrator 2023-02-22 16:00:49 -08:00
Xiaoxi Wang 35a3d2f15f Merge branch 'main' of https://github.com/apple/foundationdb into fix/main/anyExisted 2023-02-22 15:36:37 -08:00
Xiaoxi Wang 16404c5bdc make DataDistributionTracker constructor use DataDistributionTrackerInitParams 2023-02-22 15:36:15 -08:00
A.J. Beamon 87cb21be06
Merge pull request #9310 from sfc-gh-mpilman/features/tenant-lock2
Tenant Lock
2023-02-22 15:18:14 -08:00
A.J. Beamon 33431f062d Add some trace events, use a more appropriate error, and improve a check of allocated tenant groups 2023-02-22 14:39:51 -08:00
Markus Pilman 4e31cd7582 Fix compilation error due to TenantLockState being moved to a different namespace 2023-02-22 15:21:47 -07:00
A.J. Beamon a5a8c57a38 Fix some merge issues and missing updates to use new boolean parameters 2023-02-22 12:52:58 -08:00
A.J. Beamon f06ed85044 Update the tenant and metacluster consistency checks to account for the possibility of renamed tenants or partially registered clusters 2023-02-22 12:29:32 -08:00
A.J. Beamon ec79ecce73 Add a boolean parameter for ForceRemove; rename ForceJoinNewMetacluster to ForceJoin 2023-02-22 12:26:24 -08:00
Markus Pilman 8695fc15fc Merge remote-tracking branch 'origin/main' into features/tenant-lock2 2023-02-22 13:12:23 -07:00
A.J. Beamon 006a2ead6f Merge branch 'main' into check-metacluster-restore-dryrun 2023-02-22 11:16:45 -08:00
A.J. Beamon 935cd70600 Use MetaclusterData and TenantData in the MetaclusterConsistencyCheck and TenantConsistencyCheck 2023-02-22 11:04:42 -08:00
Josh Slocum 9cd0f32e87
Fixing several metrics issues in blob workers (#9426)
* fixing int vs int64 data type, and fixing cause of incorrect request counter

* fixing incorrect count of mutation bytes buffered on granule cancel
2023-02-22 11:08:12 -06:00
Steve Atherton 23df46773d
Merge pull request #9422 from sfc-gh-satherton/client-read-options
Add transaction option definitions for read priority and read cache
2023-02-22 09:00:25 -08:00
Josh Slocum 354bfef099
Added blob granules authz test, and fixed a bug it found (#9433)
* Adding blob granules authz test

* Fixing tenant race found by authz test
2023-02-22 10:53:45 -06:00
A.J. Beamon cb57541c98 Add testing that the metacluster restore dry-run mode doesn't change anything 2023-02-22 08:41:16 -08:00
Steve Atherton a21b2fe9f9 Simplified read priority option to three separate options for normal/low/high priority. 2023-02-21 22:48:38 -08:00
Steve Atherton 5969616af8 Merge commit '6de85e7cd8e9dd74a571de9e04679e669bcbb5b6' into client-read-options 2023-02-21 20:46:20 -08:00
Yao Xiao 36d79e0d2d resolve comments 2023-02-21 15:40:44 -08:00
Jon Fu ff7174065f Merge branch 'main' of github.com:apple/foundationdb into split-tenant-metadata 2023-02-21 14:16:13 -08:00
Steve Atherton bb4fb3d81d
Merge pull request #9419 from sfc-gh-satherton/page-rebuild-fix
Optimize/fix node rebuild vs update trigger in Redwood
2023-02-21 13:49:14 -08:00
Evan Tschannen 9047edd14f added a comment 2023-02-21 13:24:17 -08:00
Josh Slocum 33c0b35ee6
No RK throttling on blob workers if no blob ranges (#9425) 2023-02-21 15:23:40 -06:00
Markus Pilman 15d8548c0e Merge remote-tracking branch 'origin/main' into features/tenant-lock2
# Conflicts:
#	fdbserver/ApplyMetadataMutation.cpp
#	fdbserver/storageserver.actor.cpp
2023-02-21 13:39:35 -07:00
Evan Tschannen 8129381689 merge in main 2023-02-21 12:06:35 -08:00
Jon Fu 2d74d10a91 use MetaclusterAPI namespace over TenantAPI for TenantState 2023-02-21 11:33:36 -08:00
Josh Slocum 958f3b531b
Plumbing blob worker mapping through commit proxy like storage server (#9401)
* Plumbing blob worker mapping through commit proxy like storage server mapping

* review comments

* formatting
2023-02-21 13:21:44 -06:00
Jon Fu c0b332501b update code to match style 2023-02-21 11:17:25 -08:00
Hui Liu 2eeb29132c
Merge pull request #9413 from sfc-gh-huliu/counters
Add stats counters for manifest dump
2023-02-21 11:13:36 -08:00
Jon Fu f27c3e24a8 fix code formatting 2023-02-21 11:09:14 -08:00
Jingyu Zhou 81f8c360db
Merge pull request #8811 from sfc-gh-tclinkenbeard/expose-tag-throttled-duration 2023-02-21 10:47:35 -08:00
Jon Fu 9e01cffef0 fix some merge conflicts and address review comments 2023-02-21 10:29:36 -08:00
Jon Fu 428eb07766 Merge branch 'main' of github.com:apple/foundationdb into split-tenant-metadata 2023-02-21 10:11:35 -08:00
Junhyun Shim 2497aa5701
Clamp GetKeyServerLocations result to tenant prefix (#9424) 2023-02-21 18:55:24 +01:00
Steve Atherton 246fd1dd4e Remove auto cache option since there is no meaningful implementation of this yet. Change places using trState in a native Transaction to set cache mode or Low/Normal/High priority to use the new transaction options instead. 2023-02-21 02:50:30 -08:00
Ata E Husain Bohra fa60f1b4fa
RESTClient: Initialize RESTClient connection pool instance (#9414)
Description

Patch fixes an issue where new connection for a corresponding
'connectKey' isn't getting added to the connectionPoolMap.

Testing

Standlone fdbserver triggering RESTClient connection path
2023-02-20 19:32:10 -08:00
sfc-gh-tclinkenbeard 398079db3a Merge remote-tracking branch 'origin/main' into expose-tag-throttled-duration 2023-02-20 17:54:06 -08:00
Evan Tschannen 4f9e86b0a4 fixed two bugs that prevented the blob manager from properly loading worker affinity 2023-02-20 16:47:26 -08:00
Steve Atherton a5238eaeb3
Allow fetch keys to do more work when mutation stream is infrequent and low throughput. (#9398) 2023-02-20 13:38:04 -08:00
Steve Atherton e169e65021 Fix to BTree node rebuild logic - rebuild when imbalance hits a limit controlled by a new knob. 2023-02-19 16:40:28 -08:00
A.J. Beamon 487b444da9
Merge pull request #9416 from sfc-gh-ajbeamon/fix-tester-tenant-delete-bug
When clearing tenants at the end of a simulation test, reset the transaction when looping
2023-02-18 20:25:15 -08:00
Yi Wu eac757d186
EaR: cleanup encryption knobs (#9386)
Changes:
* Cleanup all encryption knobs 
* Update simulated cluster to randomly enable encryption with higher probability
2023-02-18 13:18:20 -08:00
A.J. Beamon 3163201201 Restore ID fixes: we weren't generating a restore ID; we weren't setting the restore ID on the management cluster in some restore modes; it is possible in some test scenarios to encounter a restore conflict, in which case we need to retry. 2023-02-17 21:20:06 -08:00
A.J. Beamon 95156198b5 When clearing tenants at the end of a simulation test, reset the transaction when looping 2023-02-17 21:13:53 -08:00
sfc-gh-tclinkenbeard 1aef6cb5f7 Merge remote-tracking branch 'origin/main' into expose-tag-throttled-duration 2023-02-17 20:41:59 -08:00
Hui Liu 323a257f1f Add stats counters for manifest dump 2023-02-17 16:22:45 -08:00
Jon Fu 8e6800663b add missing txn reset 2023-02-17 15:22:04 -08:00
Jon Fu 762cbcdc5d unconditionally set restore id 2023-02-17 14:41:41 -08:00
Jon Fu edb7a51b7e Revert "let client supply restore id"
This reverts commit 5fe32b8503.
2023-02-17 14:37:22 -08:00
Jon Fu 5fe32b8503 let client supply restore id 2023-02-17 14:01:58 -08:00
Jingyu Zhou 5d12b05090
Merge pull request #9369 from sfc-gh-ahusain/ahusain-fdbcore-3951 2023-02-17 13:11:00 -08:00
Hui Liu bdba85a86f
Merge pull request #9303 from sfc-gh-huliu/logtruncation
Truncate mutation logs after flushing blob granules
2023-02-17 12:52:45 -08:00
Ata E Husain Bohra b3d3889328 EaR: Fix ASAN OOM failure - EncryptionOps
Description

Fix ASAN build OOM failure while running OOM error.
Patch addresses the issue by avoiding using a global arena shortening
the lifecycle of allocation per iteration. Recently test was extended
to support 'configurable encryption' codepath coverage, hence, per
iteration the memory allocation was doubled.

Testing

EncryptionOps - ASAN compatible
2023-02-17 11:49:16 -08:00
Ankita Kejriwal 253934bb82
Merge pull request #9395 from sfc-gh-akejriwal/tenantmap
Assert that `checkTenantEntry()` on storage server is reading tenantMap at a legal version
2023-02-17 11:42:37 -08:00
Yi Wu 653a5eee28
EaR: Configurable encryption support for Redwood (#9359)
Supporting configurable encryption for Redwood, which supports switching to different encryption algorithm and having variable size encryption header.

Currently to support both old (non-configurable) and new (configurable) encryption header, the PR assume we have a fixed size encryption header (104 bytes) which is large enough to fit both kind of encryption header. Moving forward the plan is to update the IPager interface to support variable size encoding header, and when Redwood tries to in-place update a page but the reserved encoding header buffer is not large enough, rebuild the page instead.
2023-02-17 10:58:48 -08:00
Jon Fu 0d7b6d626b update restoreCluster test to account for conflicting_restore 2023-02-17 10:36:28 -08:00
Josh Slocum 6c2fb13173
adding wait parameter to blobbify api (#9360)
* adding wait parameter to blobbify api

* formatting

* fixing comment style

* fixing bug and adding debugging

* adding blob ranges unit test

* testing both blobbify cases in cancel

* formatting

* switch to explicit blocking api instead of boolean flag

* remove comments

* format
2023-02-17 12:20:53 -06:00
Hui Liu aa1d983132 Truncate logs after force-flushing cold blob granules 2023-02-17 10:17:04 -08:00
Steve Atherton aba2188491
Merge pull request #9399 from sfc-gh-satherton/read-after-shutdown-protection
Shut down PriorityMultiLocks more gracefully
2023-02-17 10:04:51 -08:00
Steve Atherton bca42ed232 Merge commit '99b23ac04d302b1edc6db04f1488a20f8f772ae1' into redwood-used-bytes-update 2023-02-16 22:07:44 -08:00
Ata E Husain Bohra 99b23ac04d
EaR: Configurable encryption support for Tlog mutations (#9394)
* EaR: Configurable encryption support for TLog mutations

Description

  diff-1 : Address review comments

Major changes includes:
1. Update the code involved in ensuring Tlog mutation encryption to be
compliant with "configurable encryption" feature.
2. Update ENABLE_CONFIGURABLE_ENCRYPTION flag to be 'true' by default
and BUGGIFY it.

Testing

devRunCorrectness - 100K
2023-02-16 19:01:59 -08:00
Ankita Kejriwal da7d3c1129 Merge branch 'main' of github.com:apple/foundationdb into tenantmap 2023-02-16 16:06:10 -08:00
Nim Wijetunga fd231e3f14
Configurable Encryption Support for TxnStateStore (#9387)
Configurable encryption for Transaction State Store
2023-02-16 15:20:14 -08:00
Josh Slocum c26831ec04
adding version metadata to blob granule file pointers (#9392) 2023-02-16 17:11:11 -06:00
Nim Wijetunga e03eca778c
Configurable Encryption Support for Backup (#9375)
Snapshot backup configurable encryption support
2023-02-16 15:03:27 -08:00
Jon Fu 53fc43a3a6 Merge branch 'main' of github.com:apple/foundationdb into split-tenant-metadata 2023-02-16 14:46:03 -08:00
Yanqin Jin c3d6ae0213
Throw invalid_tenant_configuration when changing assigned cluster (#9350)
Since we currently do not support tenant movement, we should as well explicitly disallow changing the assigned
cluster of a tenant during configuration by throwing `invalid_tenant_configuration` for now.

Test plan:
- add coverage for changing assigned cluster during tenant configuration
- fdbcli
- simulation tests
2023-02-16 14:20:59 -08:00
Yao Xiao a3b2324816 add validation 2023-02-16 13:52:27 -08:00
Jon Fu 036078234e fix some leftover issues from branch merge 2023-02-16 12:54:45 -08:00
Ankita Kejriwal b92aeca869 Merge branch 'main' of github.com:apple/foundationdb into tenantmap 2023-02-16 12:00:34 -08:00
Jon Fu ab15478ef9 Merge branch 'main' of github.com:apple/foundationdb into split-tenant-metadata 2023-02-16 10:04:07 -08:00
Jingyu Zhou 437cdb37ee
Merge pull request #9388 from jzhou77/fix
Fix exclusion in repairDeadDatacenter to be remote only
2023-02-16 09:42:42 -08:00
Jingyu Zhou 6b5d4fcb36
Merge pull request #9390 from sfc-gh-ajbeamon/clear-tenants-after-test
When clearing the database after a test, delete all tenants except the default tenant
2023-02-16 09:40:30 -08:00
Junhyun Shim d9c126a2d9
Introduce WipedString for Arena block holding AuthZ tokens (#9381)
* Enable secure allocation mode in Arena

This mode allows zeroing out blocks holding sensitive data after use

* Introduce WipedString to all token-holding memory

Also introduce a option flag "sensitive"

* Make pointer equivalency a hard requirement for non-ASAN builds

So that we can detect when Arena/malloc/memory-wipe behavior changes
2023-02-16 10:44:32 +01:00
Steve Atherton a1c804bc87 Added PriorityMultiLock::halt() and used that instead of kill() which avoids broken_promise errors if the lock user does not stop all of its lock-taking actors before destructing the lock and itself. 2023-02-15 23:32:17 -08:00
Steve Atherton 1d01444383 Update StorageBytes::used reported by Redwood to be the physical file size to match the behavior of other storage engines. 2023-02-15 19:02:56 -08:00
Ankita Kejriwal 56bdffa9dc Assert that storage server is reading tenantMap at a legal version 2023-02-15 18:47:17 -08:00
Jon Fu 8f4aec7d7f address review comments and fix some test errors 2023-02-15 15:58:19 -08:00
Jon Fu 1af6d8ff92 fix various correctness issues and listTenant test 2023-02-15 15:01:17 -08:00
Jingyu Zhou 99a8bfda11 Reorder trace events 2023-02-15 12:27:17 -08:00
A.J. Beamon 84142ab48d When clearing the database after a test, delete all tenants except the default tenant 2023-02-15 11:57:25 -08:00
Jingyu Zhou 02cdd0e1db Fix exclusion in repairDeadDatacenter to be remote only
If primary is excluded, the recovery will become stuck because no servers can
be recruited in the next time.
2023-02-15 11:15:18 -08:00
Jingyu Zhou 63814c9145
Merge pull request #9380 from sfc-gh-ajbeamon/fix-assigned-cluster-in-test
Try again to fix the logic in choosing the assigned cluster in the metacluster management test
2023-02-15 11:05:10 -08:00
Hui Liu fa9c80e17f
Merge pull request #9373 from sfc-gh-huliu/fixurl
Fix manifest file list issue on s3
2023-02-15 10:16:38 -08:00
Josh Slocum eefc889389
Add tenant and encryption support to new bg file apis (#9315)
* Add tenant and encryption support to new bg file apis

* formatting

* fixing comment style for linter
2023-02-15 11:48:40 -06:00
Ata E Husain Bohra 8c94b340ce
EaR: Update encryption methods to make 'cipherHeaderKey' optional (#9378)
* EaR: Update encryption methods to make 'cipherHeaderKey' optional

Description

 diff-1: Address review comments

Major changes includes:
1. Update BlobCipher Encrypt/Decrypt classes to make 'headerCipher' optional
2. Update GetEncryptionCipherKeys actor methods to make 'headerCipherKey' optional
3. Update the usage across all encryption participant methods

Testing

BlobCipherUnitTest
EnryptedBackupCorrecctness
BlobGranuleCorrectness*

devRunCorrectness - 100K
2023-02-15 08:56:11 -08:00
A.J. Beamon 545e08e1af Try again to fix the logic in choosing the assigned cluster in the metacluster management test 2023-02-14 21:39:42 -08:00
Jingyu Zhou b97ee87eba
Merge pull request #9367 from jzhou77/fix
Enable RocksDB restarting tests
2023-02-14 20:47:31 -08:00
Jingyu Zhou a5ca96ddf6
Merge pull request #9342 from sfc-gh-ajbeamon/metacluster-mgmt-restore
Metacluster restore support
2023-02-14 20:41:20 -08:00
Yi Wu 3d882a99c5
EaR: Refactor encryption header std::variant serializer and versioning (#9345)
Changes:
1. Make binary serializer natively support `std::variant`. Serialize size is 1 byte (the type index, i.e. `std::variant::index()`), plus the serialize size of the actual type stored in the `std::variant`. Update `BlobCipherEncryptHeaderRef` to use the `std::variant` binary serializer
3. Remove `flagsVersion` and `algoHeaderVersion` from `BlobCipherEncryptHeaderRef`. The former is replaced by `flags.index() + 1`, and the latter is moved into each of the algorithm-specific sub-headers. Each sub-header types will have nesting version-specific subtypes to handle serialization of that specific version (e.g. for `AesCtrNoAuth` it has a `AesCtrNoAuthV1` subtype).
2023-02-14 20:19:27 -08:00
Nim Wijetunga bf85c9f8af
Backup Mutation Log Separates Tenant Map Modifications During Restore (#9292)
mutation log separates tenant map modifications
2023-02-14 16:46:09 -08:00
A.J. Beamon adc4e932c4
Merge pull request #9364 from sfc-gh-ajbeamon/fix-get-mapped-range-assertion
Fix get mapped range test assertion
2023-02-14 15:56:41 -08:00
Hui Liu d9adbdcc66 Fix manifest file list issue on s3 2023-02-14 14:56:09 -08:00
Jingyu Zhou 4184239e72
Merge pull request #9372 from sfc-gh-xwang/fix/main/anyExisted
fix anyExisted when beginTenant==endTenant
2023-02-14 14:48:47 -08:00
Yi Wu fe18c87ac6
EaR: commit proxy fetch additional cipher keys post-resolution (#9308)
Commit proxy needs to fetch additional cipher keys post-resolution, since tenant ids for raw access requests and cross-tenant clear ranges are calculated after resolution.
2023-02-14 13:05:51 -08:00
A.J. Beamon 7284e691fb Fix a few minor restore bugs and add a dry-run mode. Some improvements to the fdbcli output. 2023-02-14 12:28:55 -08:00
Evan Tschannen c0597cc614 the blob worker uses affinity when assigning ranges on startup or after a failure 2023-02-14 11:16:59 -08:00
Jingyu Zhou d28f253182 Save shard_encode_location_metadata knob value for restarting tests
This is needed so that sharded rocks use consistent knob values.
2023-02-14 09:57:08 -08:00
Xiaoxi Wang 53f105eec5 fix anyExisted when beginTenant==endTenant 2023-02-14 09:07:09 -08:00
Ata E Husain Bohra 401b9c8918
EaR: Helper routines to support configurable encryption (#9368)
* EaR: Helper routines to support configurable encryption

Description

Add helper methods to BlobCipherEncryptHeaderRef enabling:
1. Extract 'IV' abstracting out underlying algorithm header
1. Extract 'cipherDetails' abstracting out underlying algorithm header

Testing

BlobCipherUnitTest & EncryptionOps are updated - 100K loop

* EaR: Helper routines to support configurable encryption

Description

Add helper methods to BlobCipherEncryptHeaderRef enabling:
1. Extract 'IV' abstracting out underlying algorithm header
1. Extract 'cipherDetails' abstracting out underlying algorithm header

Testing

BlobCipherUnitTest & EncryptionOps are updated - 100K loop
2023-02-14 08:34:41 -08:00
Jingyu Zhou a4d3035f64 Enable RocksDB restarting tests
Disable sharded rocks storage for downgrade tests where we
Need to keep knob "shard_encode_location_metadata" so that downgrade tests
can pass the second phase.
2023-02-13 20:12:19 -08:00
Evan Tschannen a1576a890c fix formatting 2023-02-13 15:57:16 -08:00
A.J. Beamon f3b58a063f Fix some merge issues and review comments 2023-02-13 15:32:44 -08:00
Evan Tschannen a646d15326 use Redwood as the storage engine for the blob worker cache 2023-02-13 15:28:41 -08:00
Jingyu Zhou 05e73b7836
Merge pull request #9361 from apple/sfc-gh-dadkins/disk-failure-attrition2
Disable machine attrition in DiskFailure workload.
2023-02-13 14:07:24 -08:00
Jon Fu ad210072c0 address code review 2023-02-13 13:29:19 -08:00
A.J. Beamon 958ff862e0 Fix some merge issues 2023-02-13 12:59:48 -08:00
Evan Tschannen 20bc868ee0 merge in main 2023-02-13 12:41:31 -08:00
A.J. Beamon 98407809d9 Merge branch 'main' into metacluster-mgmt-restore
# Conflicts:
#	fdbcli/MetaclusterCommands.actor.cpp
#	fdbclient/Metacluster.cpp
#	fdbclient/include/fdbclient/MetaclusterManagement.actor.h
#	fdbserver/workloads/MetaclusterManagementWorkload.actor.cpp
#	tests/CMakeLists.txt
2023-02-13 12:30:33 -08:00
A.J. Beamon 0127dd4b5a
Merge pull request #9356 from sfc-gh-ajbeamon/metacluster-concurrency-testing
Add metacluster concurrency test and fix various bugs that it found
2023-02-13 11:57:47 -08:00
A.J. Beamon 473dd33a1f Fix get mapped range test assertion to account for the possibility of a range terminating early when it reaches the end of a shard 2023-02-13 11:53:47 -08:00
A.J. Beamon 93b1e04aa9
Merge pull request #9355 from sfc-gh-ajbeamon/fix-assigned-cluster-test-retry-logic
Fix logic in MetaclusterManagementWorkload when retrying a tenant creation with an invalid assigned cluster
2023-02-13 10:43:09 -08:00
Markus Pilman 0172ca9615 Address review comments 2023-02-13 18:27:18 +01:00
Markus Pilman 8a47a484c9
Update fdbserver/storageserver.actor.cpp
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2023-02-13 18:25:03 +01:00
Jingyu Zhou fa7f15e46a
Merge pull request #9353 from sfc-gh-yiwu/redwood_restart
Redwood: fix restart test failure with XOR encoding
2023-02-13 09:24:57 -08:00
Markus Pilman 3b4bd1692a
Remove unused function
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2023-02-13 18:22:25 +01:00
Dan Adkins 5ede2d439c Disable machine attrition in DiskFailure workload.
The machine attrition logic doesn't take into account the possibility
that a disk corruption could an unrecoverable failure in the cluster.

Before disabling attrition during the DiskFailure workload, the failure
rate was >10/100,000 in the DiskFailureCycle test. Afterwards, there
were no failures in 100,000 runs.
2023-02-13 08:53:58 -08:00
Markus Pilman 3017e15448 Merge remote-tracking branch 'origin/main' into features/tenant-lock2
# Conflicts:
#	fdbclient/include/fdbclient/TenantManagement.actor.h
#	fdbserver/storageserver.actor.cpp
#	fdbserver/workloads/TenantManagementWorkload.actor.cpp
2023-02-13 11:43:07 +01:00
Steve Atherton 41fa3eada9
Merge branch 'main' into add-redwood-slack-knob 2023-02-12 19:31:20 -08:00
Evan Tschannen add21bbc3c fixed formatting 2023-02-12 10:50:17 -08:00
Evan Tschannen 73691467ee register previous affinity from the blob worker 2023-02-12 10:45:30 -08:00
Evan Tschannen bad8b2fad4 blob workers reboot with a different ID and register in the database their previous ID 2023-02-12 10:44:53 -08:00
A.J. Beamon a261c1d94c Run tenant management concurrency alongside metacluster management concurrency. Fix a few issues where performing tenant operations returned undesirable errors when the associated cluster was removed. 2023-02-11 19:46:47 -08:00
Xiaoxi Wang 93f892c085
Merge pull request #9340 from sfc-gh-xwang/fix/main/tenantList
fix the way verifyListFilter detect tenant state change
2023-02-11 17:20:46 -08:00
A.J. Beamon e6021f8326 Add Jon's metacluster concurrency test and fix various bugs that it found 2023-02-11 15:15:32 -08:00
Xiaoxi Wang 21a2378de5
Merge pull request #9298 from sfc-gh-xwang/feature/main/clearRange
Split raw clear ranges across tenants in required mode
2023-02-11 14:29:46 -08:00
Xiaoxi Wang a9c7632c83 Merge branch 'main' of https://github.com/apple/foundationdb into fix/main/tenantList 2023-02-11 13:54:27 -08:00
Xiaoxi Wang ac1ddc81b0 remove debug trace; change function comment 2023-02-11 13:17:59 -08:00
A.J. Beamon b4f45a0a87 Fix logic in MetaclusterManagementWorkload when retrying a tenant creation with an invalid assigned cluster 2023-02-11 12:09:17 -08:00
A.J. Beamon ee1b48323d
Merge pull request #9346 from sfc-gh-nwijetunga/nim/global-tenant-ids
Support for Two Byte Prefix for Tenant IDs
2023-02-11 11:31:24 -08:00
A.J. Beamon 4579a4319d Merge branch 'main' into storage-quota-in-tenant-metadata-space 2023-02-11 09:04:15 -08:00
Xiaoxi Wang a0f7943fc3 simplify implementation of lowerBoundTenantId and withinSingleTenant 2023-02-10 22:14:59 -08:00
Yi Wu a37d8f757c Redwood: fix restart test failure with xor encoding 2023-02-10 21:01:52 -08:00
Nim Wijetunga 640f1afd77 address pr comments 2023-02-10 16:39:06 -08:00
Nim Wijetunga 9e5c61e127 address pr comments 2023-02-10 15:56:41 -08:00
Nim Wijetunga de9eef72ff address pr comments 2023-02-10 13:49:15 -08:00
Jon Fu 5c68c95a60 fix assertion 2023-02-10 13:12:24 -08:00
Xiaoxi Wang ffc5733e9c add comments 2023-02-10 12:51:13 -08:00
Evan Tschannen 487189738f change reassignment after reboot to be the blob manager's responsibility 2023-02-10 12:47:03 -08:00
Xiaoxi Wang bb8d96c026 Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/clearRange 2023-02-10 12:30:16 -08:00
Xiaoxi Wang ffadea08cb change isSingleTenant check; add unit tests 2023-02-10 12:29:38 -08:00
Jon Fu bf508f5642 adjust tenantconsistency workload to account for lastTenantId on data clusters 2023-02-10 12:25:29 -08:00
A.J. Beamon a6b47c1da4 Fix merge issue 2023-02-10 11:12:36 -08:00
Jingyu Zhou 622520bd2d Return the source team if remote DC is dead
Also refactor the code with findTeamFromServers().
2023-02-10 11:11:07 -08:00
Jingyu Zhou 9aa15b459c Clean up trace events 2023-02-10 11:11:07 -08:00
Jingyu Zhou 6c4a9b5f23 Fix DD stuck when remote DC is dead
When remote DC is down, the remote team collection of DD can initializing
waiting for the remote to recover (all_tlog_recruited state). However, the
getTeam request can already be served by the remote team collection. So, for
a RelocateShard (data movement such as split, move), it will get a team for
the remote DC. But the data movement can't make progress on the remote team
because the remote DC hasn't recovered yet. Because of the stuck of data
movement, the primary cannot reach the "storage_recovered" state and stay in
accepting_commit state.

The specifc test failure: slow/ApiCorrectness.toml -s 339026305 -b on
at commit:  0edd899d65

In this test, primary DC has 1 SS killed, remote DC has 2 TLog and 2 SS killed.
So the remote is dead, the remaining 2 SSes can't make progress because of the
loss of 2 TLogs. The repairDeadDatacenter() can't reach the "storage_recovered"
state due to DD's failure of moving shards away from the killed SS in the
primary.

The fix is to exclude all remote in repairDeadDatacenter() so that tells DD to
mark all SSes in the remote as unhealthy. Another fix is to return empty
results for getTeam request if the remote team collection is not ready. This
will allow the data movement to continue, essentially remote team is not changed
for the data movement.
2023-02-10 11:11:07 -08:00
Jon Fu 1c0e784a7a attempt to fix broken test workloads 2023-02-10 11:01:59 -08:00
A.J. Beamon 13eee09ce8 Merge branch 'main' into metacluster-mgmt-restore 2023-02-10 10:58:01 -08:00
A.J. Beamon 4b13c9c211 Make a few minor fixes, refactor some code for clarity, and improve throughput of repopulating a management cluster 2023-02-10 10:41:55 -08:00
Evan Tschannen 18bc099ca6 added a disk to blob workers, and allowed blob workers to rejoin the blob manager after a reboot 2023-02-10 10:29:01 -08:00
Nim Wijetunga 8a3f3ea674 clean up code 2023-02-10 01:01:16 -08:00
Nim Wijetunga dd57e85064 remove traces 2023-02-09 23:05:43 -08:00
Xiaoxi Wang 09da7efdc0 handle clear range when tenantMap.size() == 0 2023-02-09 22:28:18 -08:00
Nim Wijetunga fed650894d working version 2023-02-09 21:10:40 -08:00
Xiaoxi Wang 53923c77cb Merge branch 'main' of https://github.com/apple/foundationdb into fix/main/tenantList 2023-02-09 17:27:54 -08:00
Yi Wu f17024e615
Redwood: fix btree unit test reopen memory only pager (#9334)
The Redwood btree test are not suppose to reopen the pager if it is memory only, which will open an empty pager.
2023-02-09 17:12:50 -08:00
A.J. Beamon 788095536b Merge branch 'main' into storage-quota-in-tenant-metadata-space 2023-02-09 16:52:44 -08:00
A.J. Beamon f9a68056ac Add support for modifying a data cluster that is being restored so that we can manage conflicts 2023-02-09 15:33:40 -08:00
Nim Wijetunga b7ef50d1f8 inital commit 2023-02-09 14:32:29 -08:00
Xiaoxi Wang 76d8c1555f accept review comments 2023-02-09 14:02:57 -08:00
Hui Liu cb9d4d8bb5
Merge pull request #9276 from sfc-gh-huliu/manifest
Split blob manifest as segments when writting
2023-02-09 13:54:02 -08:00
Xiaoxi Wang 47ac2a8fb2 fix the way verifyListFilter detect tenant state change 2023-02-09 11:34:14 -08:00
Hui Liu 6b6959d35f Split blob manifest as segments when writting 2023-02-09 11:26:19 -08:00
A.J. Beamon 2d59c5681d Bug fixes and test improvements for management cluster restoration 2023-02-09 08:42:23 -08:00
A.J. Beamon 1815254b76
Merge pull request #9333 from sfc-gh-xwang/fix/main/tenantList
Fix list tenant name bug
2023-02-09 08:33:23 -08:00
Ata E Husain Bohra 9c649d7880
EaR: Configurable encryption framework (#9271)
* EaR: Configurable encryption framework

Description

EaR implementation only supports fixed size on-disk encryption header format.
One drawback of the scheme is, introducing a newer encryption scheme as well
as updating header format in future may incur data migration restrictions.
Major changes proposed in the patch includes:
1. Flexible Encryption header format allowing the following:
 1.1. Header flags (metadata) can evolve separately from the encryption algorithm
 1.2. Specific encryption algorithm header to allow future extensions.
2. Update the BlobCipher encryption/decryption util classes to work with newer
encryption header format.
3. Continue supporting multiple encryption authentication schemes such as:
HMAC-SHA and AES-CMAC; also, supports no encryption-authentication schemes.
4. Refactor BlobCipher unit test to enable testing of new format.
5. Configuration knobs to control encryption header flags and algorithm
versions.

Note: 
The on-disk header storage footprint savings due to the newer scheme is as follows:
1. No encryption authentication: 54% smaller compared to existing implementation.
3. AES-CMAC: 16% smaller compared to existing implementation.
3. HMAC-SHA encryption authentication: almost same size.


Testing

BlobCipherTest
EncryptionOpsTest
2023-02-08 22:51:05 -08:00
Xiaoxi Wang e5d3fd2e96 add debug events; fix list tenant name bug 2023-02-08 20:37:27 -08:00
Hao Fu 2790693862
Fix getMappedRange metrics (#9330) 2023-02-08 16:48:36 -08:00
Xiaoxi Wang a77bf236c7 fix unit test bug 2023-02-08 16:09:32 -08:00
Ankita Kejriwal 05dce05f52
Merge pull request #9329 from sfc-gh-akejriwal/debug2
Remove the version bump in `checkTenantEntry` on storage server
2023-02-08 14:53:14 -08:00
Xiaoxi Wang e75f38a6fc extract pushToBackupMutations method and add single tenant range validation 2023-02-08 14:35:35 -08:00
sfc-gh-tclinkenbeard 9a3cdb9a3b Add /GrvProxyTagThrottler/MultiTag unit test 2023-02-08 14:28:21 -08:00
sfc-gh-tclinkenbeard ed20a30c11 Remove debugging trace event 2023-02-08 14:28:21 -08:00
Xiaoxi Wang 1ff7865ee2 Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/clearRange 2023-02-08 13:38:55 -08:00
Ankita Kejriwal 0789ab35e9 Merge branch 'main' of github.com:apple/foundationdb into debug2 2023-02-08 13:26:39 -08:00
Xiaoxi Wang e28fc664d0 extract shouldBackup method 2023-02-08 13:25:46 -08:00
Ankita Kejriwal cc3ae02961 Remove the version bump in `checkTenantEntry` on storage server
This was mistakenly added in PR #9291.
2023-02-08 12:08:04 -08:00
Ankita Kejriwal b9630e57d9
Merge pull request #9291 from sfc-gh-akejriwal/debug2
Update storage metrics functions to use the version at which the tenant was read
2023-02-08 11:33:33 -08:00
sfc-gh-tclinkenbeard 09ad864eb5 Merge remote-tracking branch 'origin/main' into expose-tag-throttled-duration 2023-02-08 11:25:14 -08:00
Junhyun Shim d7d0b658a3
Merge pull request #9314 from sfc-gh-jshim/authz-tenant-name-to-tenant-id
Make Authz use tenant ids instead of tenant names
2023-02-08 20:13:29 +01:00
Jon Fu d981d5e9c2 initial commit to separate metadata 2023-02-08 10:44:26 -08:00
Ankita Kejriwal 187f2a42e8 Merge branch 'main' of github.com:apple/foundationdb into debug2 2023-02-08 10:13:39 -08:00
Junhyun Shim ea0083cda2 Remove redundant calls to setAuthToken 2023-02-08 18:56:10 +01:00
sfc-gh-tclinkenbeard dd2ba18d45 Create GLOBAL_TAG_THROTTLING_MIN_TPS knob 2023-02-08 09:48:33 -08:00
sfc-gh-tclinkenbeard 7f9ec344db GlobalTagThrottlerImpl::getAverageTransactionCost doesn't compute cost when transaction rate is <1 2023-02-08 09:48:33 -08:00
Junhyun Shim d32d3dd085 Make token option NOT survive transaction hard resets 2023-02-08 18:12:22 +01:00
Ankita Kejriwal a5257144c6 Merge branch 'main' of github.com:apple/foundationdb into debug2 2023-02-08 09:04:17 -08:00
Markus Pilman 1c712df709 Tenant lock workload succeeding 2023-02-08 15:51:09 +01:00
Ankita Kejriwal 6077419900 Update the waitMetrics version if it is too old on the storage server. 2023-02-07 19:33:58 -08:00
Nim Wijetunga 1cd0922ed1
Snapshot Backup Bug Fix (#9319)
snapshot restore bug fix
2023-02-07 18:44:03 -08:00
A.J. Beamon 42eb82cd30 Merge branch 'main' into storage-quota-in-tenant-metadata-space 2023-02-07 15:11:29 -08:00
A.J. Beamon 0410f7d9a1 Add intial support for restoring a management cluster in a metacluster 2023-02-07 15:08:46 -08:00
Jon Fu c8e12a7921
Add tenant watches to storage server (#9284)
* add tenant watches to storage server

* combine watches and throw different error after first loop

* try to change SS tenantmap to avoid tenantname altogether

* update code style to be consistent

* add new operation in tenant management test

* address code review changes

* add more to tenant workload and fix id parse bug

* account for TenantRef in TenantData when renaming

* code review comments

* change test assertion
2023-02-07 17:08:51 -05:00
Xiaoxi Wang 145b81170c add unit test for replaceRawClearRanges 2023-02-07 13:21:18 -08:00
Josh Slocum e6c7927619
improving bg restart with higher workload and chaos for cycle, and new large restart workload (#9313) 2023-02-07 07:47:53 -06:00
Ankita Kejriwal d8bc494859 Merge branch 'main' of github.com:apple/foundationdb into debug2 2023-02-06 16:44:47 -08:00
Xiaoxi Wang 46027aa4aa
Merge pull request #9272 from sfc-gh-xwang/feature/main/listTenant
Change listTenants API to only list tenant Ids
2023-02-06 15:22:31 -08:00
Xiaoxi Wang 6df5f1fa56 Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/clearRange 2023-02-06 14:28:35 -08:00
Junhyun Shim be225acd2a Merge remote-tracking branch 'origin/main' into authz-tenant-name-to-tenant-id 2023-02-06 23:13:43 +01:00
Josh Slocum 181970b160
adding metadata-only update for empty delta files to avoid blob storage writes (#9312) 2023-02-06 16:11:35 -06:00
Yi Wu d3bc2afc8e
EaR: storage server uses encryption DB config (#9115)
The PR is updating storage server and Redwood to enable encryption based on the encryption mode in DB config, which was previously controlled by a knob. High level changes are
1. Passing encryption mode in DB config to storage server
    1.1 If it is a new storage server, pass the encryption mode through `InitializeStorageRequest`. the encryption mode is pass to Redwood for initialization
    1.2 If it is an existing storage server, on restart the storage server will send `GetStorageServerRejoinInfoRequest` to commit proxy, and commit proxy will return the current encryption mode, which it get from DB config on its own initialization. Storage server will compare the DB config encryption mode to the local storage encryption mode, and fail if they don't match
2. Adding a new `encryptionMode()` method to `IKeyValueStore`, which return a future of local encryption mode of the KV store instance. A KV store supporting encryption would need to persist its own encryption mode, and return the mode via the API.
3. Redwood accepts encryption mode from its constructor. For a new Redwood instance, caller has to specific the encryption mode, which will be stored in Redwood per-instance file header. For existing instance, caller is supposed to not passing the encryption mode, and let Redwood find it out from its own header.
4. Refactoring in Redwood to accommodate the above changes.
2023-02-06 14:02:31 -08:00
Josh Slocum 0a19db8ce9
Blob granule restarting tests (#9259)
* Adding BlobGranuleRestartCycle tests

* test harness fix to not copy blob symlinks for restarting test
2023-02-06 14:54:20 -06:00
Xiaoxi Wang 18a3de3594 Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/listTenant 2023-02-06 12:15:40 -08:00
Ankita Kejriwal 3b418b4431 Merge branch 'main' of github.com:apple/foundationdb into debug2 2023-02-06 11:07:28 -08:00
Xiaoxi Wang 8cfc3f28e1
Merge pull request #9307 from sfc-gh-xwang/fix/main/tenantModeUpdate
fix temporary tenantMode=0 by only updating db info when recovery cou…
2023-02-06 10:25:32 -08:00
A.J. Beamon 99bc126667 Merge branch 'main' into storage-quota-in-tenant-metadata-space 2023-02-06 09:32:24 -08:00
Markus Pilman f321933bc9 Minor bug fix 2023-02-06 17:34:46 +01:00
Junhyun Shim 0d1801ed88 Expose private endpoints used in MVC mode and make MVT token sticky 2023-02-06 17:23:26 +01:00
Josh Slocum 4101379295
Assumption that caller would be cancelled after delay(0) after triggering simInjectFailure was false because of granuleMetadata.clearAsync. (#9286) 2023-02-06 09:05:36 -06:00
Markus Pilman 1fb3c33b3f Merge remote-tracking branch 'origin/main' into features/tenant-lock2
# Conflicts:
#	fdbserver/CommitProxyServer.actor.cpp
2023-02-06 11:58:49 +01:00
Markus Pilman 280e50d42d fixed compilation issues 2023-02-06 11:56:39 +01:00
Xiaoxi Wang 3a84fc5375 fix temporary tenantMode=0 by only updating db info when recovery count increasing 2023-02-05 18:37:46 -08:00
A.J. Beamon 6991b62106 Improve comments on runRYWTransaction*, use MAX_TENANTS_PER_CLUSTER instead of TOO_MANY knob in tenant cache. 2023-02-03 15:35:47 -08:00
Xiaoxi Wang f1dd3aea92 Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/listTenant 2023-02-03 15:27:21 -08:00
Xiaoxi Wang 7190fa0c08 Merge branch 'main' of https://github.com/apple/foundationdb into fix/main/testTimeout 2023-02-03 13:48:54 -08:00
Xiaoxi Wang b757e8914a fix BOOST_SYSTEM_NO_LIB redefinition in CI 2023-02-03 13:47:50 -08:00
Xiaoxi Wang f1a7841fb8 Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/listTenant 2023-02-03 13:02:29 -08:00
Xiaoxi Wang 8371883e3b remove log; disabled clear range temporarily 2023-02-03 12:49:23 -08:00
Junhyun Shim ce652fa284 Replace AuthZ's use of tenant names in token with tenant ID
Also, to minimize audit log loss, handle token usage audit logging at each usage.
This has a side-effect of making the token use log less bursty.
This also subtly changes the dedup cache policy.
Dedup time window used to be 5 seconds (default) since the start of batch-logging.
Now it's 5 seconds from the first usage since the closing of the previous dedup window
2023-02-03 21:46:31 +01:00
Xiaoxi Wang b0b8f21dcc Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/clearRange 2023-02-03 12:27:03 -08:00
Xiaoxi Wang b0902a515a handle prefix convert to negative id 2023-02-03 12:26:25 -08:00
Ankita Kejriwal 1a240a0eab Merge branch 'main' of github.com:apple/foundationdb into debug2 2023-02-03 12:26:19 -08:00
Xiaoxi Wang 0e242226bc add trace log; fix size bug 2023-02-03 10:36:57 -08:00
Hui Liu cafff2c4d0
Merge pull request #9290 from sfc-gh-huliu/splitpoints
Support pagination for StorageServer splitMetrics API
2023-02-03 09:43:09 -08:00
Ankita Kejriwal 913bc9114d Handle errors that can't be sent in reply; fix formatting 2023-02-02 19:05:05 -08:00
Xiaoxi Wang ae6b57d5c8 handle system keys 2023-02-02 16:25:41 -08:00
Xiaoxi Wang 3e1fee8ed4 finalize splitClearRangeByTenant unit test and implement replaceRawClearRanges 2023-02-02 15:19:21 -08:00
Hui Liu 774446d3a0 Support pagination for StorageServer splitMetrics API 2023-02-02 14:21:38 -08:00
Hui Liu d5ff370e4e Add metadataVersion to manifest backup 2023-02-02 11:39:01 -08:00
Xiaoxi Wang 347b3d4ca2 Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/listTenant 2023-02-02 09:24:14 -08:00
Vaidas Gasiunas f8b1da8bc6
An option to initialize client tracing in setupNetwork (#9209)
* client_config_tester: use a generic mechanism to set specific network options

* trace_initialize_on_setup option to initialize client traces on network setup without local IP address

* trace_initialize_on_setup: Addressing review comments

* Restore correct formatting

* trace_initialize_on_setup: Update go bindings

* Include PID for identification into trace file names by default

* Use the same naming pattern for trace files in all configurations

* Empty commit
2023-02-02 10:00:51 +01:00
Xiaoxi Wang a8013bad5c add splitClearRangeByTenant and unit tests 2023-02-01 20:25:56 -08:00
Ankita Kejriwal 9529f360ee Update storage metrics functions to use the version at which tenant was read 2023-02-01 19:44:31 -08:00
Nim Wijetunga 86f3665514
Handle EKP Tenant Not Found Errors (#9261)
handle EKP tenant not found errors
2023-02-01 19:15:38 -08:00
Xiaoxi Wang 128bfccfa7 Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/listTenant 2023-02-01 13:16:28 -08:00
Josh Slocum 6dbb96312b
move feed cleanup check to after data is guaranteed to be available for granule (#9283) 2023-02-01 13:19:08 -06:00
Xiaoxi Wang c37042c81d Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/listTenant 2023-02-01 10:08:32 -08:00
Jingyu Zhou 982ce105e5 Reduce logging level for verbose events
From one of nightly failure due to too many log lines, these are top 3:

  60100 FastRestoreLoaderDispatchRequests
  79655 FastRestoreGetVersionSize
  93888 FastRestoreSplitMutation
2023-02-01 09:51:16 -08:00
Jingyu Zhou 9c5ce68d98 Merge branch 'main' of https://github.com/apple/foundationdb into fix 2023-01-31 17:40:13 -08:00
Xiaoxi Wang a447ec98a0 Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/listTenant 2023-01-31 16:25:07 -08:00
A.J. Beamon 6665eb6654 Update cluster recovery for tenant ID changes 2023-01-31 16:06:45 -08:00
He Liu eb4839d046
Fix audit_storage issues (#9265)
* Erase audit from DD after completion.

* Use CLIENT_KNOBS->TOO_MANY as the default number of rows for
get_audit_status.
2023-01-31 15:39:01 -08:00
Josh Slocum 77875d7fee
Fixing SkewedReadWrite to load its metadata in a transactionally consistent way (#9274) 2023-01-31 14:52:36 -06:00