Josh Slocum
716a9c3817
addressing review comments and fixmes in bg commit proxy code
2023-02-27 10:51:47 -06:00
Josh Slocum
4fa528f3fa
added unit test for bg tenant deletion
2023-02-27 10:21:26 -06:00
Josh Slocum
1e7368aa21
removing double counted counter
2023-02-27 10:18:50 -06:00
Josh Slocum
faf8ec6281
adding authz blob management tests to ensure tenant-scoped clients cannot do blob management
2023-02-27 08:06:36 -06:00
Steve Atherton
cad7596e2d
Apply clang-format.
2023-02-27 01:07:35 -08:00
Steve Atherton
674c105050
Merge pull request #9473 from sfc-gh-etschannen/feature-change-feed-lock
...
Replace fetchKeysParallelismFullLock to speed up fetch keys in idle clusters
2023-02-26 23:18:43 -08:00
Steve Atherton
01f89d1245
Merge commit '97c355085c0e22e727a5c3e35ace926a58b16f5d' into redwood-first-commit-record
2023-02-26 21:19:17 -08:00
Steve Atherton
92bfa1d578
Throw an error if reopened Redwood file is recovered but to a point prior to when the BTree was created if the Encryption Mode of the cluster is not known.
2023-02-26 21:15:48 -08:00
Steve Atherton
29444252fc
Prevent Redwood Pager from recovering to a point before the user commit record (ie the BTree CommitRecord) was initialized as this isn't a useful recovery point.
2023-02-26 01:38:24 -08:00
A.J. Beamon
b5b7d51213
Fix maybe committed issue in tenant lock test
2023-02-25 20:24:16 -08:00
A.J. Beamon
846b3c731d
Update metacluster consistency check to handle tenants that are in the process of being renamed correctly
2023-02-25 19:52:07 -08:00
A.J. Beamon
040d44927b
Store the rename destination for tenant movements in the crluster tenant index with an ID of -1. Use this to filter out tenant aliases when modifying the tenant count during a tenant purge.
2023-02-25 19:51:21 -08:00
A.J. Beamon
fddec2b573
Add restore support to the metacluster management concurrency workload
2023-02-25 19:27:04 -08:00
A.J. Beamon
8c3ee768a2
Add an option to allow exceeding the tenant group capacity limit when changing tenant configuration
2023-02-24 21:01:36 -08:00
A.J. Beamon
1c71056e26
Merge pull request #9479 from sfc-gh-nwijetunga/nim/enforce-metacluster-tenant-mode
...
Enforce Disabled Tenant Mode in Metacluster
2023-02-24 19:27:57 -08:00
Jingyu Zhou
6b121de6a6
Merge pull request #9464 from jzhou77/fix
...
Add exclude to fdbcli's configure command
2023-02-24 16:31:02 -08:00
Nim Wijetunga
c8b7cff10c
fix api test
2023-02-24 15:26:59 -08:00
Josh Slocum
6187811f71
Reworking getBlobGranuleRanges to also use commit proxy rpc for authz, and adding test ( #9470 )
2023-02-24 17:15:32 -06:00
Jingyu Zhou
204b31bc87
Merge pull request #9472 from sfc-gh-ajbeamon/fix-invalid-iterator-use
...
Fix memory error from using invalidated iterator
2023-02-24 14:55:28 -08:00
Nim Wijetunga
eca98afcb0
metacluster check tenant mode
2023-02-24 13:59:54 -08:00
Evan Tschannen
8872e5a462
Merge pull request #9347 from sfc-gh-etschannen/feature-change-feed-cache
...
added a disk to blob workers
2023-02-24 13:59:03 -08:00
Josh Slocum
17bcbaeb03
add tenant reopen to ensure no races with tenant opening ( #9476 )
2023-02-24 15:34:57 -06:00
Dan Adkins
e3a61b9b22
Add metrics to understand tail commit latency ( #9435 )
...
* Add server-side latency metrics for Resolver requests.
* Add separate resolver latency metrics for queue wait and compute time.
* Add histogram for queue depth observed on resolver (during metrics interval).
* Fix tlog latency measurement to use timer() instead of now().
2023-02-24 14:13:12 -05:00
Evan Tschannen
f3673d808b
Replaced the fetchKeysParallelismFullLock with a lock specifically for change feeds to avoid blocking fetches on idle clusters
2023-02-24 10:59:35 -08:00
A.J. Beamon
2d936acbdd
Fix memory error from using invalidated iterator
2023-02-24 10:25:38 -08:00
A.J. Beamon
03fbc59bb1
Merge pull request #9461 from sfc-gh-ajbeamon/metacluster-concurrent-restore-testing
...
Metacluster concurrent restore testing
2023-02-24 09:13:51 -08:00
Josh Slocum
910965a5a6
Adding additional blob granule authz tests ( #9443 )
...
* added granule location authz tests
* added authz test for blob worker endpoint
* addressing comments
* fixing ide build
2023-02-24 09:32:05 -06:00
A.J. Beamon
344f6977c9
Restores can now throw a cluster_already_exists error in the metacluster management workload if we timeout a restore and have to retry
2023-02-23 20:32:18 -08:00
Nim Wijetunga
29819b0645
Change Feed Bug Fix + Encryption Asserts ( #9457 )
...
* add encryption asserts
* modify function name
* address pr comments
* address pr comments
* Trigger Build
2023-02-23 19:33:25 -08:00
Xiaoxi Wang
7e14324cc1
remove dataDistributionQueue actor and replace it with class DDQueue
2023-02-23 17:15:33 -08:00
A.J. Beamon
2b25cfef8b
Merge branch 'main' into metacluster-concurrent-restore-testing
2023-02-23 16:06:47 -08:00
Jingyu Zhou
0b2e02c402
Fix rare test failures
...
Unclog after DB is recovered, otherwise another recovery may become stuck again.
2023-02-23 15:42:33 -08:00
Xiaoxi Wang
9cda7ebe5a
encapsulate DDQueue constructor params into DDQueueInitParams
2023-02-23 15:38:48 -08:00
Jon Fu
33f8e90f9f
Split tenant group metadata ( #9446 )
...
* initial commit to split tenant group metadata
* attempt to fix merge errors
* fix compile errors and adjust existing tests
* fix infinite loop and extra ACTOR tag
* direct assignment instead of store
* direct assign instead of store (missed a few)
2023-02-23 18:11:49 -05:00
A.J. Beamon
3ac7e17b79
Fix create tenant usage in tenant management workload
2023-02-23 15:08:52 -08:00
Jingyu Zhou
65443b6541
Fix compiling errors
2023-02-23 15:02:44 -08:00
Jingyu Zhou
ecae81882c
Change to only clog once for a particular tlog
...
If we repeat clogging, different tlogs may be excluded, which can cause the
recovery to stuck.
2023-02-23 14:31:39 -08:00
Jingyu Zhou
6055f752c2
Exclude failed tlog if recovery stuck more than 30s
...
Because the tlog is clogged, recovery can stuck in initializing_transaction_servers.
This exclude allows the recovery to complete.
2023-02-23 14:31:32 -08:00
Jingyu Zhou
c4773b7cc8
Update clogTlog workload to be single region
2023-02-23 14:31:24 -08:00
Jingyu Zhou
955826f2fe
Add ClogTlog workload
2023-02-23 14:31:12 -08:00
Jingyu Zhou
792950dbdc
Merge pull request #9434 from sfc-gh-huliu/splitmetrics
...
Implement SplitMetric pagination in blob migrator
2023-02-23 14:10:27 -08:00
A.J. Beamon
b828f3f257
Add missing change to explicit TenantMapEntry conversion
2023-02-23 13:38:04 -08:00
A.J. Beamon
06fe00544a
Remove TenantMapEntry <-> MetaclusterTenantMapEntry conversion constructors and use named functions instead
2023-02-23 13:28:10 -08:00
A.J. Beamon
dcae48cbbd
Add concurrent restore testing to the metacluster restore workload
2023-02-23 13:27:20 -08:00
Xiaoxi Wang
cdfaf7d4e0
move DDRelocationQueue to header file
2023-02-23 13:12:38 -08:00
A.J. Beamon
e151a2d363
Merge pull request #9451 from sfc-gh-ajbeamon/metacluster-management-workload-restore-support
...
Improve restore support in the metacluster management workload
2023-02-23 13:10:31 -08:00
Markus Pilman
efc5bf9ee8
Merge pull request #9456 from sfc-gh-ajbeamon/smaller-tenant-in-txn-state-store
...
Store a smaller tenant object in the txn state store
2023-02-23 14:00:12 -07:00
A.J. Beamon
9e9a31c0f1
Use error variable consistently
2023-02-23 11:27:53 -08:00
Xiaoxi Wang
021f7c1b43
remove unused headers and outdated comments
2023-02-23 11:00:21 -08:00
Xiaoxi Wang
a6feafc371
remove dataDistributionTracker actor and use DataDistributionTracker class instead
2023-02-23 10:18:28 -08:00
Evan Tschannen
cf3a4e6161
Merge branch 'main' into feature-change-feed-cache
2023-02-23 10:16:13 -08:00
Evan Tschannen
a581a55452
ensure a worker cannot run multiple blob worker roles
2023-02-23 09:51:26 -08:00
A.J. Beamon
dd650215d4
Store a smaller tenant object in the txn state store
2023-02-23 09:29:33 -08:00
Ata E Husain Bohra
7d079690d4
Merge branch 'main' into ahusain-misc-fixes
2023-02-22 18:11:11 -08:00
A.J. Beamon
a76af5c696
Improve restore support in the metacluster management workload
2023-02-22 17:42:36 -08:00
Ata E Husain Bohra
1f7ee9437f
EaR: RESTClient and EKP changes to handle unreachable external KMS
...
Description
Two major changes proposed are:
I)
Used following setup for testing:
1. Run `fdbserver` locally.
2. Run a mock python based HTTP server (encryption endpoints not implemented)
Expectation was RESTClient code should go in loop trying to establish connection
to the desired encryption endpoint. However, observation was the code loops for
one cycle and followup cycle SEGV while printing a log using RESTUrl object which
is obtained as a 'pointer' from the caller. Update the code to use RESTUrl object
instead of the pointer.
II) In above setup, KMSConnector would throw 'encrypt_key_fetch_failed' error
which wasn't handled by EKProxy, hence, causing the service to terminate. Add
code to re-throw the error to the caller.
Testing
2023-02-22 17:15:34 -08:00
A.J. Beamon
9b906d9b3d
Merge pull request #9447 from sfc-gh-ajbeamon/metacluster-restore-fixes
...
Metacluster restore fixes
2023-02-22 17:07:19 -08:00
Hui Liu
0fba65a3cd
Implement SplitMetric pagination in blob migrator
2023-02-22 16:00:49 -08:00
Xiaoxi Wang
35a3d2f15f
Merge branch 'main' of https://github.com/apple/foundationdb into fix/main/anyExisted
2023-02-22 15:36:37 -08:00
Xiaoxi Wang
16404c5bdc
make DataDistributionTracker constructor use DataDistributionTrackerInitParams
2023-02-22 15:36:15 -08:00
A.J. Beamon
87cb21be06
Merge pull request #9310 from sfc-gh-mpilman/features/tenant-lock2
...
Tenant Lock
2023-02-22 15:18:14 -08:00
A.J. Beamon
33431f062d
Add some trace events, use a more appropriate error, and improve a check of allocated tenant groups
2023-02-22 14:39:51 -08:00
Markus Pilman
4e31cd7582
Fix compilation error due to TenantLockState being moved to a different namespace
2023-02-22 15:21:47 -07:00
A.J. Beamon
a5a8c57a38
Fix some merge issues and missing updates to use new boolean parameters
2023-02-22 12:52:58 -08:00
A.J. Beamon
f06ed85044
Update the tenant and metacluster consistency checks to account for the possibility of renamed tenants or partially registered clusters
2023-02-22 12:29:32 -08:00
A.J. Beamon
ec79ecce73
Add a boolean parameter for ForceRemove; rename ForceJoinNewMetacluster to ForceJoin
2023-02-22 12:26:24 -08:00
Markus Pilman
8695fc15fc
Merge remote-tracking branch 'origin/main' into features/tenant-lock2
2023-02-22 13:12:23 -07:00
A.J. Beamon
006a2ead6f
Merge branch 'main' into check-metacluster-restore-dryrun
2023-02-22 11:16:45 -08:00
A.J. Beamon
935cd70600
Use MetaclusterData and TenantData in the MetaclusterConsistencyCheck and TenantConsistencyCheck
2023-02-22 11:04:42 -08:00
Josh Slocum
9cd0f32e87
Fixing several metrics issues in blob workers ( #9426 )
...
* fixing int vs int64 data type, and fixing cause of incorrect request counter
* fixing incorrect count of mutation bytes buffered on granule cancel
2023-02-22 11:08:12 -06:00
Steve Atherton
23df46773d
Merge pull request #9422 from sfc-gh-satherton/client-read-options
...
Add transaction option definitions for read priority and read cache
2023-02-22 09:00:25 -08:00
Josh Slocum
354bfef099
Added blob granules authz test, and fixed a bug it found ( #9433 )
...
* Adding blob granules authz test
* Fixing tenant race found by authz test
2023-02-22 10:53:45 -06:00
A.J. Beamon
cb57541c98
Add testing that the metacluster restore dry-run mode doesn't change anything
2023-02-22 08:41:16 -08:00
Steve Atherton
a21b2fe9f9
Simplified read priority option to three separate options for normal/low/high priority.
2023-02-21 22:48:38 -08:00
Steve Atherton
5969616af8
Merge commit '6de85e7cd8e9dd74a571de9e04679e669bcbb5b6' into client-read-options
2023-02-21 20:46:20 -08:00
Yao Xiao
36d79e0d2d
resolve comments
2023-02-21 15:40:44 -08:00
Jon Fu
ff7174065f
Merge branch 'main' of github.com:apple/foundationdb into split-tenant-metadata
2023-02-21 14:16:13 -08:00
Steve Atherton
bb4fb3d81d
Merge pull request #9419 from sfc-gh-satherton/page-rebuild-fix
...
Optimize/fix node rebuild vs update trigger in Redwood
2023-02-21 13:49:14 -08:00
Evan Tschannen
9047edd14f
added a comment
2023-02-21 13:24:17 -08:00
Josh Slocum
33c0b35ee6
No RK throttling on blob workers if no blob ranges ( #9425 )
2023-02-21 15:23:40 -06:00
Markus Pilman
15d8548c0e
Merge remote-tracking branch 'origin/main' into features/tenant-lock2
...
# Conflicts:
# fdbserver/ApplyMetadataMutation.cpp
# fdbserver/storageserver.actor.cpp
2023-02-21 13:39:35 -07:00
Evan Tschannen
8129381689
merge in main
2023-02-21 12:06:35 -08:00
Jon Fu
2d74d10a91
use MetaclusterAPI namespace over TenantAPI for TenantState
2023-02-21 11:33:36 -08:00
Josh Slocum
958f3b531b
Plumbing blob worker mapping through commit proxy like storage server ( #9401 )
...
* Plumbing blob worker mapping through commit proxy like storage server mapping
* review comments
* formatting
2023-02-21 13:21:44 -06:00
Jon Fu
c0b332501b
update code to match style
2023-02-21 11:17:25 -08:00
Hui Liu
2eeb29132c
Merge pull request #9413 from sfc-gh-huliu/counters
...
Add stats counters for manifest dump
2023-02-21 11:13:36 -08:00
Jon Fu
f27c3e24a8
fix code formatting
2023-02-21 11:09:14 -08:00
Jingyu Zhou
81f8c360db
Merge pull request #8811 from sfc-gh-tclinkenbeard/expose-tag-throttled-duration
2023-02-21 10:47:35 -08:00
Jon Fu
9e01cffef0
fix some merge conflicts and address review comments
2023-02-21 10:29:36 -08:00
Jon Fu
428eb07766
Merge branch 'main' of github.com:apple/foundationdb into split-tenant-metadata
2023-02-21 10:11:35 -08:00
Junhyun Shim
2497aa5701
Clamp GetKeyServerLocations result to tenant prefix ( #9424 )
2023-02-21 18:55:24 +01:00
Steve Atherton
246fd1dd4e
Remove auto cache option since there is no meaningful implementation of this yet. Change places using trState in a native Transaction to set cache mode or Low/Normal/High priority to use the new transaction options instead.
2023-02-21 02:50:30 -08:00
Ata E Husain Bohra
fa60f1b4fa
RESTClient: Initialize RESTClient connection pool instance ( #9414 )
...
Description
Patch fixes an issue where new connection for a corresponding
'connectKey' isn't getting added to the connectionPoolMap.
Testing
Standlone fdbserver triggering RESTClient connection path
2023-02-20 19:32:10 -08:00
sfc-gh-tclinkenbeard
398079db3a
Merge remote-tracking branch 'origin/main' into expose-tag-throttled-duration
2023-02-20 17:54:06 -08:00
Evan Tschannen
4f9e86b0a4
fixed two bugs that prevented the blob manager from properly loading worker affinity
2023-02-20 16:47:26 -08:00
Steve Atherton
a5238eaeb3
Allow fetch keys to do more work when mutation stream is infrequent and low throughput. ( #9398 )
2023-02-20 13:38:04 -08:00
Steve Atherton
e169e65021
Fix to BTree node rebuild logic - rebuild when imbalance hits a limit controlled by a new knob.
2023-02-19 16:40:28 -08:00
A.J. Beamon
487b444da9
Merge pull request #9416 from sfc-gh-ajbeamon/fix-tester-tenant-delete-bug
...
When clearing tenants at the end of a simulation test, reset the transaction when looping
2023-02-18 20:25:15 -08:00
Yi Wu
eac757d186
EaR: cleanup encryption knobs ( #9386 )
...
Changes:
* Cleanup all encryption knobs
* Update simulated cluster to randomly enable encryption with higher probability
2023-02-18 13:18:20 -08:00
A.J. Beamon
3163201201
Restore ID fixes: we weren't generating a restore ID; we weren't setting the restore ID on the management cluster in some restore modes; it is possible in some test scenarios to encounter a restore conflict, in which case we need to retry.
2023-02-17 21:20:06 -08:00
A.J. Beamon
95156198b5
When clearing tenants at the end of a simulation test, reset the transaction when looping
2023-02-17 21:13:53 -08:00
sfc-gh-tclinkenbeard
1aef6cb5f7
Merge remote-tracking branch 'origin/main' into expose-tag-throttled-duration
2023-02-17 20:41:59 -08:00
Hui Liu
323a257f1f
Add stats counters for manifest dump
2023-02-17 16:22:45 -08:00
Jon Fu
8e6800663b
add missing txn reset
2023-02-17 15:22:04 -08:00
Jon Fu
762cbcdc5d
unconditionally set restore id
2023-02-17 14:41:41 -08:00
Jon Fu
edb7a51b7e
Revert "let client supply restore id"
...
This reverts commit 5fe32b8503
.
2023-02-17 14:37:22 -08:00
Jon Fu
5fe32b8503
let client supply restore id
2023-02-17 14:01:58 -08:00
Jingyu Zhou
5d12b05090
Merge pull request #9369 from sfc-gh-ahusain/ahusain-fdbcore-3951
2023-02-17 13:11:00 -08:00
Hui Liu
bdba85a86f
Merge pull request #9303 from sfc-gh-huliu/logtruncation
...
Truncate mutation logs after flushing blob granules
2023-02-17 12:52:45 -08:00
Ata E Husain Bohra
b3d3889328
EaR: Fix ASAN OOM failure - EncryptionOps
...
Description
Fix ASAN build OOM failure while running OOM error.
Patch addresses the issue by avoiding using a global arena shortening
the lifecycle of allocation per iteration. Recently test was extended
to support 'configurable encryption' codepath coverage, hence, per
iteration the memory allocation was doubled.
Testing
EncryptionOps - ASAN compatible
2023-02-17 11:49:16 -08:00
Ankita Kejriwal
253934bb82
Merge pull request #9395 from sfc-gh-akejriwal/tenantmap
...
Assert that `checkTenantEntry()` on storage server is reading tenantMap at a legal version
2023-02-17 11:42:37 -08:00
Yi Wu
653a5eee28
EaR: Configurable encryption support for Redwood ( #9359 )
...
Supporting configurable encryption for Redwood, which supports switching to different encryption algorithm and having variable size encryption header.
Currently to support both old (non-configurable) and new (configurable) encryption header, the PR assume we have a fixed size encryption header (104 bytes) which is large enough to fit both kind of encryption header. Moving forward the plan is to update the IPager interface to support variable size encoding header, and when Redwood tries to in-place update a page but the reserved encoding header buffer is not large enough, rebuild the page instead.
2023-02-17 10:58:48 -08:00
Jon Fu
0d7b6d626b
update restoreCluster test to account for conflicting_restore
2023-02-17 10:36:28 -08:00
Josh Slocum
6c2fb13173
adding wait parameter to blobbify api ( #9360 )
...
* adding wait parameter to blobbify api
* formatting
* fixing comment style
* fixing bug and adding debugging
* adding blob ranges unit test
* testing both blobbify cases in cancel
* formatting
* switch to explicit blocking api instead of boolean flag
* remove comments
* format
2023-02-17 12:20:53 -06:00
Hui Liu
aa1d983132
Truncate logs after force-flushing cold blob granules
2023-02-17 10:17:04 -08:00
Steve Atherton
aba2188491
Merge pull request #9399 from sfc-gh-satherton/read-after-shutdown-protection
...
Shut down PriorityMultiLocks more gracefully
2023-02-17 10:04:51 -08:00
Steve Atherton
bca42ed232
Merge commit '99b23ac04d302b1edc6db04f1488a20f8f772ae1' into redwood-used-bytes-update
2023-02-16 22:07:44 -08:00
Ata E Husain Bohra
99b23ac04d
EaR: Configurable encryption support for Tlog mutations ( #9394 )
...
* EaR: Configurable encryption support for TLog mutations
Description
diff-1 : Address review comments
Major changes includes:
1. Update the code involved in ensuring Tlog mutation encryption to be
compliant with "configurable encryption" feature.
2. Update ENABLE_CONFIGURABLE_ENCRYPTION flag to be 'true' by default
and BUGGIFY it.
Testing
devRunCorrectness - 100K
2023-02-16 19:01:59 -08:00
Ankita Kejriwal
da7d3c1129
Merge branch 'main' of github.com:apple/foundationdb into tenantmap
2023-02-16 16:06:10 -08:00
Nim Wijetunga
fd231e3f14
Configurable Encryption Support for TxnStateStore ( #9387 )
...
Configurable encryption for Transaction State Store
2023-02-16 15:20:14 -08:00
Josh Slocum
c26831ec04
adding version metadata to blob granule file pointers ( #9392 )
2023-02-16 17:11:11 -06:00
Nim Wijetunga
e03eca778c
Configurable Encryption Support for Backup ( #9375 )
...
Snapshot backup configurable encryption support
2023-02-16 15:03:27 -08:00
Jon Fu
53fc43a3a6
Merge branch 'main' of github.com:apple/foundationdb into split-tenant-metadata
2023-02-16 14:46:03 -08:00
Yanqin Jin
c3d6ae0213
Throw invalid_tenant_configuration when changing assigned cluster ( #9350 )
...
Since we currently do not support tenant movement, we should as well explicitly disallow changing the assigned
cluster of a tenant during configuration by throwing `invalid_tenant_configuration` for now.
Test plan:
- add coverage for changing assigned cluster during tenant configuration
- fdbcli
- simulation tests
2023-02-16 14:20:59 -08:00
Yao Xiao
a3b2324816
add validation
2023-02-16 13:52:27 -08:00
Jon Fu
036078234e
fix some leftover issues from branch merge
2023-02-16 12:54:45 -08:00
Ankita Kejriwal
b92aeca869
Merge branch 'main' of github.com:apple/foundationdb into tenantmap
2023-02-16 12:00:34 -08:00
Jon Fu
ab15478ef9
Merge branch 'main' of github.com:apple/foundationdb into split-tenant-metadata
2023-02-16 10:04:07 -08:00
Jingyu Zhou
437cdb37ee
Merge pull request #9388 from jzhou77/fix
...
Fix exclusion in repairDeadDatacenter to be remote only
2023-02-16 09:42:42 -08:00
Jingyu Zhou
6b5d4fcb36
Merge pull request #9390 from sfc-gh-ajbeamon/clear-tenants-after-test
...
When clearing the database after a test, delete all tenants except the default tenant
2023-02-16 09:40:30 -08:00
Junhyun Shim
d9c126a2d9
Introduce WipedString for Arena block holding AuthZ tokens ( #9381 )
...
* Enable secure allocation mode in Arena
This mode allows zeroing out blocks holding sensitive data after use
* Introduce WipedString to all token-holding memory
Also introduce a option flag "sensitive"
* Make pointer equivalency a hard requirement for non-ASAN builds
So that we can detect when Arena/malloc/memory-wipe behavior changes
2023-02-16 10:44:32 +01:00
Steve Atherton
a1c804bc87
Added PriorityMultiLock::halt() and used that instead of kill() which avoids broken_promise errors if the lock user does not stop all of its lock-taking actors before destructing the lock and itself.
2023-02-15 23:32:17 -08:00
Steve Atherton
1d01444383
Update StorageBytes::used reported by Redwood to be the physical file size to match the behavior of other storage engines.
2023-02-15 19:02:56 -08:00
Ankita Kejriwal
56bdffa9dc
Assert that storage server is reading tenantMap at a legal version
2023-02-15 18:47:17 -08:00
Jon Fu
8f4aec7d7f
address review comments and fix some test errors
2023-02-15 15:58:19 -08:00
Jon Fu
1af6d8ff92
fix various correctness issues and listTenant test
2023-02-15 15:01:17 -08:00
Jingyu Zhou
99a8bfda11
Reorder trace events
2023-02-15 12:27:17 -08:00
A.J. Beamon
84142ab48d
When clearing the database after a test, delete all tenants except the default tenant
2023-02-15 11:57:25 -08:00
Jingyu Zhou
02cdd0e1db
Fix exclusion in repairDeadDatacenter to be remote only
...
If primary is excluded, the recovery will become stuck because no servers can
be recruited in the next time.
2023-02-15 11:15:18 -08:00
Jingyu Zhou
63814c9145
Merge pull request #9380 from sfc-gh-ajbeamon/fix-assigned-cluster-in-test
...
Try again to fix the logic in choosing the assigned cluster in the metacluster management test
2023-02-15 11:05:10 -08:00
Hui Liu
fa9c80e17f
Merge pull request #9373 from sfc-gh-huliu/fixurl
...
Fix manifest file list issue on s3
2023-02-15 10:16:38 -08:00
Josh Slocum
eefc889389
Add tenant and encryption support to new bg file apis ( #9315 )
...
* Add tenant and encryption support to new bg file apis
* formatting
* fixing comment style for linter
2023-02-15 11:48:40 -06:00
Ata E Husain Bohra
8c94b340ce
EaR: Update encryption methods to make 'cipherHeaderKey' optional ( #9378 )
...
* EaR: Update encryption methods to make 'cipherHeaderKey' optional
Description
diff-1: Address review comments
Major changes includes:
1. Update BlobCipher Encrypt/Decrypt classes to make 'headerCipher' optional
2. Update GetEncryptionCipherKeys actor methods to make 'headerCipherKey' optional
3. Update the usage across all encryption participant methods
Testing
BlobCipherUnitTest
EnryptedBackupCorrecctness
BlobGranuleCorrectness*
devRunCorrectness - 100K
2023-02-15 08:56:11 -08:00
A.J. Beamon
545e08e1af
Try again to fix the logic in choosing the assigned cluster in the metacluster management test
2023-02-14 21:39:42 -08:00
Jingyu Zhou
b97ee87eba
Merge pull request #9367 from jzhou77/fix
...
Enable RocksDB restarting tests
2023-02-14 20:47:31 -08:00
Jingyu Zhou
a5ca96ddf6
Merge pull request #9342 from sfc-gh-ajbeamon/metacluster-mgmt-restore
...
Metacluster restore support
2023-02-14 20:41:20 -08:00
Yi Wu
3d882a99c5
EaR: Refactor encryption header std::variant serializer and versioning ( #9345 )
...
Changes:
1. Make binary serializer natively support `std::variant`. Serialize size is 1 byte (the type index, i.e. `std::variant::index()`), plus the serialize size of the actual type stored in the `std::variant`. Update `BlobCipherEncryptHeaderRef` to use the `std::variant` binary serializer
3. Remove `flagsVersion` and `algoHeaderVersion` from `BlobCipherEncryptHeaderRef`. The former is replaced by `flags.index() + 1`, and the latter is moved into each of the algorithm-specific sub-headers. Each sub-header types will have nesting version-specific subtypes to handle serialization of that specific version (e.g. for `AesCtrNoAuth` it has a `AesCtrNoAuthV1` subtype).
2023-02-14 20:19:27 -08:00
Nim Wijetunga
bf85c9f8af
Backup Mutation Log Separates Tenant Map Modifications During Restore ( #9292 )
...
mutation log separates tenant map modifications
2023-02-14 16:46:09 -08:00
A.J. Beamon
adc4e932c4
Merge pull request #9364 from sfc-gh-ajbeamon/fix-get-mapped-range-assertion
...
Fix get mapped range test assertion
2023-02-14 15:56:41 -08:00
Hui Liu
d9adbdcc66
Fix manifest file list issue on s3
2023-02-14 14:56:09 -08:00
Jingyu Zhou
4184239e72
Merge pull request #9372 from sfc-gh-xwang/fix/main/anyExisted
...
fix anyExisted when beginTenant==endTenant
2023-02-14 14:48:47 -08:00
Yi Wu
fe18c87ac6
EaR: commit proxy fetch additional cipher keys post-resolution ( #9308 )
...
Commit proxy needs to fetch additional cipher keys post-resolution, since tenant ids for raw access requests and cross-tenant clear ranges are calculated after resolution.
2023-02-14 13:05:51 -08:00
A.J. Beamon
7284e691fb
Fix a few minor restore bugs and add a dry-run mode. Some improvements to the fdbcli output.
2023-02-14 12:28:55 -08:00
Evan Tschannen
c0597cc614
the blob worker uses affinity when assigning ranges on startup or after a failure
2023-02-14 11:16:59 -08:00
Jingyu Zhou
d28f253182
Save shard_encode_location_metadata knob value for restarting tests
...
This is needed so that sharded rocks use consistent knob values.
2023-02-14 09:57:08 -08:00
Xiaoxi Wang
53f105eec5
fix anyExisted when beginTenant==endTenant
2023-02-14 09:07:09 -08:00
Ata E Husain Bohra
401b9c8918
EaR: Helper routines to support configurable encryption ( #9368 )
...
* EaR: Helper routines to support configurable encryption
Description
Add helper methods to BlobCipherEncryptHeaderRef enabling:
1. Extract 'IV' abstracting out underlying algorithm header
1. Extract 'cipherDetails' abstracting out underlying algorithm header
Testing
BlobCipherUnitTest & EncryptionOps are updated - 100K loop
* EaR: Helper routines to support configurable encryption
Description
Add helper methods to BlobCipherEncryptHeaderRef enabling:
1. Extract 'IV' abstracting out underlying algorithm header
1. Extract 'cipherDetails' abstracting out underlying algorithm header
Testing
BlobCipherUnitTest & EncryptionOps are updated - 100K loop
2023-02-14 08:34:41 -08:00
Jingyu Zhou
a4d3035f64
Enable RocksDB restarting tests
...
Disable sharded rocks storage for downgrade tests where we
Need to keep knob "shard_encode_location_metadata" so that downgrade tests
can pass the second phase.
2023-02-13 20:12:19 -08:00
Evan Tschannen
a1576a890c
fix formatting
2023-02-13 15:57:16 -08:00
A.J. Beamon
f3b58a063f
Fix some merge issues and review comments
2023-02-13 15:32:44 -08:00
Evan Tschannen
a646d15326
use Redwood as the storage engine for the blob worker cache
2023-02-13 15:28:41 -08:00
Jingyu Zhou
05e73b7836
Merge pull request #9361 from apple/sfc-gh-dadkins/disk-failure-attrition2
...
Disable machine attrition in DiskFailure workload.
2023-02-13 14:07:24 -08:00
Jon Fu
ad210072c0
address code review
2023-02-13 13:29:19 -08:00
A.J. Beamon
958ff862e0
Fix some merge issues
2023-02-13 12:59:48 -08:00
Evan Tschannen
20bc868ee0
merge in main
2023-02-13 12:41:31 -08:00
A.J. Beamon
98407809d9
Merge branch 'main' into metacluster-mgmt-restore
...
# Conflicts:
# fdbcli/MetaclusterCommands.actor.cpp
# fdbclient/Metacluster.cpp
# fdbclient/include/fdbclient/MetaclusterManagement.actor.h
# fdbserver/workloads/MetaclusterManagementWorkload.actor.cpp
# tests/CMakeLists.txt
2023-02-13 12:30:33 -08:00
A.J. Beamon
0127dd4b5a
Merge pull request #9356 from sfc-gh-ajbeamon/metacluster-concurrency-testing
...
Add metacluster concurrency test and fix various bugs that it found
2023-02-13 11:57:47 -08:00
A.J. Beamon
473dd33a1f
Fix get mapped range test assertion to account for the possibility of a range terminating early when it reaches the end of a shard
2023-02-13 11:53:47 -08:00
A.J. Beamon
93b1e04aa9
Merge pull request #9355 from sfc-gh-ajbeamon/fix-assigned-cluster-test-retry-logic
...
Fix logic in MetaclusterManagementWorkload when retrying a tenant creation with an invalid assigned cluster
2023-02-13 10:43:09 -08:00
Markus Pilman
0172ca9615
Address review comments
2023-02-13 18:27:18 +01:00
Markus Pilman
8a47a484c9
Update fdbserver/storageserver.actor.cpp
...
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2023-02-13 18:25:03 +01:00
Jingyu Zhou
fa7f15e46a
Merge pull request #9353 from sfc-gh-yiwu/redwood_restart
...
Redwood: fix restart test failure with XOR encoding
2023-02-13 09:24:57 -08:00
Markus Pilman
3b4bd1692a
Remove unused function
...
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2023-02-13 18:22:25 +01:00
Dan Adkins
5ede2d439c
Disable machine attrition in DiskFailure workload.
...
The machine attrition logic doesn't take into account the possibility
that a disk corruption could an unrecoverable failure in the cluster.
Before disabling attrition during the DiskFailure workload, the failure
rate was >10/100,000 in the DiskFailureCycle test. Afterwards, there
were no failures in 100,000 runs.
2023-02-13 08:53:58 -08:00
Markus Pilman
3017e15448
Merge remote-tracking branch 'origin/main' into features/tenant-lock2
...
# Conflicts:
# fdbclient/include/fdbclient/TenantManagement.actor.h
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/TenantManagementWorkload.actor.cpp
2023-02-13 11:43:07 +01:00
Steve Atherton
41fa3eada9
Merge branch 'main' into add-redwood-slack-knob
2023-02-12 19:31:20 -08:00
Evan Tschannen
add21bbc3c
fixed formatting
2023-02-12 10:50:17 -08:00
Evan Tschannen
73691467ee
register previous affinity from the blob worker
2023-02-12 10:45:30 -08:00
Evan Tschannen
bad8b2fad4
blob workers reboot with a different ID and register in the database their previous ID
2023-02-12 10:44:53 -08:00
A.J. Beamon
a261c1d94c
Run tenant management concurrency alongside metacluster management concurrency. Fix a few issues where performing tenant operations returned undesirable errors when the associated cluster was removed.
2023-02-11 19:46:47 -08:00
Xiaoxi Wang
93f892c085
Merge pull request #9340 from sfc-gh-xwang/fix/main/tenantList
...
fix the way verifyListFilter detect tenant state change
2023-02-11 17:20:46 -08:00
A.J. Beamon
e6021f8326
Add Jon's metacluster concurrency test and fix various bugs that it found
2023-02-11 15:15:32 -08:00
Xiaoxi Wang
21a2378de5
Merge pull request #9298 from sfc-gh-xwang/feature/main/clearRange
...
Split raw clear ranges across tenants in required mode
2023-02-11 14:29:46 -08:00
Xiaoxi Wang
a9c7632c83
Merge branch 'main' of https://github.com/apple/foundationdb into fix/main/tenantList
2023-02-11 13:54:27 -08:00
Xiaoxi Wang
ac1ddc81b0
remove debug trace; change function comment
2023-02-11 13:17:59 -08:00
A.J. Beamon
b4f45a0a87
Fix logic in MetaclusterManagementWorkload when retrying a tenant creation with an invalid assigned cluster
2023-02-11 12:09:17 -08:00
A.J. Beamon
ee1b48323d
Merge pull request #9346 from sfc-gh-nwijetunga/nim/global-tenant-ids
...
Support for Two Byte Prefix for Tenant IDs
2023-02-11 11:31:24 -08:00
A.J. Beamon
4579a4319d
Merge branch 'main' into storage-quota-in-tenant-metadata-space
2023-02-11 09:04:15 -08:00
Xiaoxi Wang
a0f7943fc3
simplify implementation of lowerBoundTenantId and withinSingleTenant
2023-02-10 22:14:59 -08:00
Yi Wu
a37d8f757c
Redwood: fix restart test failure with xor encoding
2023-02-10 21:01:52 -08:00
Nim Wijetunga
640f1afd77
address pr comments
2023-02-10 16:39:06 -08:00
Nim Wijetunga
9e5c61e127
address pr comments
2023-02-10 15:56:41 -08:00
Nim Wijetunga
de9eef72ff
address pr comments
2023-02-10 13:49:15 -08:00
Jon Fu
5c68c95a60
fix assertion
2023-02-10 13:12:24 -08:00
Xiaoxi Wang
ffc5733e9c
add comments
2023-02-10 12:51:13 -08:00
Evan Tschannen
487189738f
change reassignment after reboot to be the blob manager's responsibility
2023-02-10 12:47:03 -08:00
Xiaoxi Wang
bb8d96c026
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/clearRange
2023-02-10 12:30:16 -08:00
Xiaoxi Wang
ffadea08cb
change isSingleTenant check; add unit tests
2023-02-10 12:29:38 -08:00
Jon Fu
bf508f5642
adjust tenantconsistency workload to account for lastTenantId on data clusters
2023-02-10 12:25:29 -08:00
A.J. Beamon
a6b47c1da4
Fix merge issue
2023-02-10 11:12:36 -08:00
Jingyu Zhou
622520bd2d
Return the source team if remote DC is dead
...
Also refactor the code with findTeamFromServers().
2023-02-10 11:11:07 -08:00
Jingyu Zhou
9aa15b459c
Clean up trace events
2023-02-10 11:11:07 -08:00
Jingyu Zhou
6c4a9b5f23
Fix DD stuck when remote DC is dead
...
When remote DC is down, the remote team collection of DD can initializing
waiting for the remote to recover (all_tlog_recruited state). However, the
getTeam request can already be served by the remote team collection. So, for
a RelocateShard (data movement such as split, move), it will get a team for
the remote DC. But the data movement can't make progress on the remote team
because the remote DC hasn't recovered yet. Because of the stuck of data
movement, the primary cannot reach the "storage_recovered" state and stay in
accepting_commit state.
The specifc test failure: slow/ApiCorrectness.toml -s 339026305 -b on
at commit: 0edd899d65
In this test, primary DC has 1 SS killed, remote DC has 2 TLog and 2 SS killed.
So the remote is dead, the remaining 2 SSes can't make progress because of the
loss of 2 TLogs. The repairDeadDatacenter() can't reach the "storage_recovered"
state due to DD's failure of moving shards away from the killed SS in the
primary.
The fix is to exclude all remote in repairDeadDatacenter() so that tells DD to
mark all SSes in the remote as unhealthy. Another fix is to return empty
results for getTeam request if the remote team collection is not ready. This
will allow the data movement to continue, essentially remote team is not changed
for the data movement.
2023-02-10 11:11:07 -08:00
Jon Fu
1c0e784a7a
attempt to fix broken test workloads
2023-02-10 11:01:59 -08:00
A.J. Beamon
13eee09ce8
Merge branch 'main' into metacluster-mgmt-restore
2023-02-10 10:58:01 -08:00
A.J. Beamon
4b13c9c211
Make a few minor fixes, refactor some code for clarity, and improve throughput of repopulating a management cluster
2023-02-10 10:41:55 -08:00
Evan Tschannen
18bc099ca6
added a disk to blob workers, and allowed blob workers to rejoin the blob manager after a reboot
2023-02-10 10:29:01 -08:00
Nim Wijetunga
8a3f3ea674
clean up code
2023-02-10 01:01:16 -08:00
Nim Wijetunga
dd57e85064
remove traces
2023-02-09 23:05:43 -08:00
Xiaoxi Wang
09da7efdc0
handle clear range when tenantMap.size() == 0
2023-02-09 22:28:18 -08:00
Nim Wijetunga
fed650894d
working version
2023-02-09 21:10:40 -08:00
Xiaoxi Wang
53923c77cb
Merge branch 'main' of https://github.com/apple/foundationdb into fix/main/tenantList
2023-02-09 17:27:54 -08:00
Yi Wu
f17024e615
Redwood: fix btree unit test reopen memory only pager ( #9334 )
...
The Redwood btree test are not suppose to reopen the pager if it is memory only, which will open an empty pager.
2023-02-09 17:12:50 -08:00
A.J. Beamon
788095536b
Merge branch 'main' into storage-quota-in-tenant-metadata-space
2023-02-09 16:52:44 -08:00
A.J. Beamon
f9a68056ac
Add support for modifying a data cluster that is being restored so that we can manage conflicts
2023-02-09 15:33:40 -08:00
Nim Wijetunga
b7ef50d1f8
inital commit
2023-02-09 14:32:29 -08:00
Xiaoxi Wang
76d8c1555f
accept review comments
2023-02-09 14:02:57 -08:00
Hui Liu
cb9d4d8bb5
Merge pull request #9276 from sfc-gh-huliu/manifest
...
Split blob manifest as segments when writting
2023-02-09 13:54:02 -08:00
Xiaoxi Wang
47ac2a8fb2
fix the way verifyListFilter detect tenant state change
2023-02-09 11:34:14 -08:00
Hui Liu
6b6959d35f
Split blob manifest as segments when writting
2023-02-09 11:26:19 -08:00
A.J. Beamon
2d59c5681d
Bug fixes and test improvements for management cluster restoration
2023-02-09 08:42:23 -08:00
A.J. Beamon
1815254b76
Merge pull request #9333 from sfc-gh-xwang/fix/main/tenantList
...
Fix list tenant name bug
2023-02-09 08:33:23 -08:00
Ata E Husain Bohra
9c649d7880
EaR: Configurable encryption framework ( #9271 )
...
* EaR: Configurable encryption framework
Description
EaR implementation only supports fixed size on-disk encryption header format.
One drawback of the scheme is, introducing a newer encryption scheme as well
as updating header format in future may incur data migration restrictions.
Major changes proposed in the patch includes:
1. Flexible Encryption header format allowing the following:
1.1. Header flags (metadata) can evolve separately from the encryption algorithm
1.2. Specific encryption algorithm header to allow future extensions.
2. Update the BlobCipher encryption/decryption util classes to work with newer
encryption header format.
3. Continue supporting multiple encryption authentication schemes such as:
HMAC-SHA and AES-CMAC; also, supports no encryption-authentication schemes.
4. Refactor BlobCipher unit test to enable testing of new format.
5. Configuration knobs to control encryption header flags and algorithm
versions.
Note:
The on-disk header storage footprint savings due to the newer scheme is as follows:
1. No encryption authentication: 54% smaller compared to existing implementation.
3. AES-CMAC: 16% smaller compared to existing implementation.
3. HMAC-SHA encryption authentication: almost same size.
Testing
BlobCipherTest
EncryptionOpsTest
2023-02-08 22:51:05 -08:00
Xiaoxi Wang
e5d3fd2e96
add debug events; fix list tenant name bug
2023-02-08 20:37:27 -08:00
Hao Fu
2790693862
Fix getMappedRange metrics ( #9330 )
2023-02-08 16:48:36 -08:00
Xiaoxi Wang
a77bf236c7
fix unit test bug
2023-02-08 16:09:32 -08:00
Ankita Kejriwal
05dce05f52
Merge pull request #9329 from sfc-gh-akejriwal/debug2
...
Remove the version bump in `checkTenantEntry` on storage server
2023-02-08 14:53:14 -08:00
Xiaoxi Wang
e75f38a6fc
extract pushToBackupMutations method and add single tenant range validation
2023-02-08 14:35:35 -08:00
sfc-gh-tclinkenbeard
9a3cdb9a3b
Add /GrvProxyTagThrottler/MultiTag unit test
2023-02-08 14:28:21 -08:00
sfc-gh-tclinkenbeard
ed20a30c11
Remove debugging trace event
2023-02-08 14:28:21 -08:00
Xiaoxi Wang
1ff7865ee2
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/clearRange
2023-02-08 13:38:55 -08:00
Ankita Kejriwal
0789ab35e9
Merge branch 'main' of github.com:apple/foundationdb into debug2
2023-02-08 13:26:39 -08:00
Xiaoxi Wang
e28fc664d0
extract shouldBackup method
2023-02-08 13:25:46 -08:00
Ankita Kejriwal
cc3ae02961
Remove the version bump in `checkTenantEntry` on storage server
...
This was mistakenly added in PR #9291 .
2023-02-08 12:08:04 -08:00
Ankita Kejriwal
b9630e57d9
Merge pull request #9291 from sfc-gh-akejriwal/debug2
...
Update storage metrics functions to use the version at which the tenant was read
2023-02-08 11:33:33 -08:00
sfc-gh-tclinkenbeard
09ad864eb5
Merge remote-tracking branch 'origin/main' into expose-tag-throttled-duration
2023-02-08 11:25:14 -08:00
Junhyun Shim
d7d0b658a3
Merge pull request #9314 from sfc-gh-jshim/authz-tenant-name-to-tenant-id
...
Make Authz use tenant ids instead of tenant names
2023-02-08 20:13:29 +01:00
Jon Fu
d981d5e9c2
initial commit to separate metadata
2023-02-08 10:44:26 -08:00
Ankita Kejriwal
187f2a42e8
Merge branch 'main' of github.com:apple/foundationdb into debug2
2023-02-08 10:13:39 -08:00
Junhyun Shim
ea0083cda2
Remove redundant calls to setAuthToken
2023-02-08 18:56:10 +01:00
sfc-gh-tclinkenbeard
dd2ba18d45
Create GLOBAL_TAG_THROTTLING_MIN_TPS knob
2023-02-08 09:48:33 -08:00
sfc-gh-tclinkenbeard
7f9ec344db
GlobalTagThrottlerImpl::getAverageTransactionCost doesn't compute cost when transaction rate is <1
2023-02-08 09:48:33 -08:00
Junhyun Shim
d32d3dd085
Make token option NOT survive transaction hard resets
2023-02-08 18:12:22 +01:00
Ankita Kejriwal
a5257144c6
Merge branch 'main' of github.com:apple/foundationdb into debug2
2023-02-08 09:04:17 -08:00
Markus Pilman
1c712df709
Tenant lock workload succeeding
2023-02-08 15:51:09 +01:00
Ankita Kejriwal
6077419900
Update the waitMetrics version if it is too old on the storage server.
2023-02-07 19:33:58 -08:00
Nim Wijetunga
1cd0922ed1
Snapshot Backup Bug Fix ( #9319 )
...
snapshot restore bug fix
2023-02-07 18:44:03 -08:00
A.J. Beamon
42eb82cd30
Merge branch 'main' into storage-quota-in-tenant-metadata-space
2023-02-07 15:11:29 -08:00
A.J. Beamon
0410f7d9a1
Add intial support for restoring a management cluster in a metacluster
2023-02-07 15:08:46 -08:00
Jon Fu
c8e12a7921
Add tenant watches to storage server ( #9284 )
...
* add tenant watches to storage server
* combine watches and throw different error after first loop
* try to change SS tenantmap to avoid tenantname altogether
* update code style to be consistent
* add new operation in tenant management test
* address code review changes
* add more to tenant workload and fix id parse bug
* account for TenantRef in TenantData when renaming
* code review comments
* change test assertion
2023-02-07 17:08:51 -05:00
Xiaoxi Wang
145b81170c
add unit test for replaceRawClearRanges
2023-02-07 13:21:18 -08:00
Josh Slocum
e6c7927619
improving bg restart with higher workload and chaos for cycle, and new large restart workload ( #9313 )
2023-02-07 07:47:53 -06:00
Ankita Kejriwal
d8bc494859
Merge branch 'main' of github.com:apple/foundationdb into debug2
2023-02-06 16:44:47 -08:00
Xiaoxi Wang
46027aa4aa
Merge pull request #9272 from sfc-gh-xwang/feature/main/listTenant
...
Change listTenants API to only list tenant Ids
2023-02-06 15:22:31 -08:00
Xiaoxi Wang
6df5f1fa56
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/clearRange
2023-02-06 14:28:35 -08:00
Junhyun Shim
be225acd2a
Merge remote-tracking branch 'origin/main' into authz-tenant-name-to-tenant-id
2023-02-06 23:13:43 +01:00
Josh Slocum
181970b160
adding metadata-only update for empty delta files to avoid blob storage writes ( #9312 )
2023-02-06 16:11:35 -06:00
Yi Wu
d3bc2afc8e
EaR: storage server uses encryption DB config ( #9115 )
...
The PR is updating storage server and Redwood to enable encryption based on the encryption mode in DB config, which was previously controlled by a knob. High level changes are
1. Passing encryption mode in DB config to storage server
1.1 If it is a new storage server, pass the encryption mode through `InitializeStorageRequest`. the encryption mode is pass to Redwood for initialization
1.2 If it is an existing storage server, on restart the storage server will send `GetStorageServerRejoinInfoRequest` to commit proxy, and commit proxy will return the current encryption mode, which it get from DB config on its own initialization. Storage server will compare the DB config encryption mode to the local storage encryption mode, and fail if they don't match
2. Adding a new `encryptionMode()` method to `IKeyValueStore`, which return a future of local encryption mode of the KV store instance. A KV store supporting encryption would need to persist its own encryption mode, and return the mode via the API.
3. Redwood accepts encryption mode from its constructor. For a new Redwood instance, caller has to specific the encryption mode, which will be stored in Redwood per-instance file header. For existing instance, caller is supposed to not passing the encryption mode, and let Redwood find it out from its own header.
4. Refactoring in Redwood to accommodate the above changes.
2023-02-06 14:02:31 -08:00
Josh Slocum
0a19db8ce9
Blob granule restarting tests ( #9259 )
...
* Adding BlobGranuleRestartCycle tests
* test harness fix to not copy blob symlinks for restarting test
2023-02-06 14:54:20 -06:00
Xiaoxi Wang
18a3de3594
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/listTenant
2023-02-06 12:15:40 -08:00
Ankita Kejriwal
3b418b4431
Merge branch 'main' of github.com:apple/foundationdb into debug2
2023-02-06 11:07:28 -08:00
Xiaoxi Wang
8cfc3f28e1
Merge pull request #9307 from sfc-gh-xwang/fix/main/tenantModeUpdate
...
fix temporary tenantMode=0 by only updating db info when recovery cou…
2023-02-06 10:25:32 -08:00
A.J. Beamon
99bc126667
Merge branch 'main' into storage-quota-in-tenant-metadata-space
2023-02-06 09:32:24 -08:00
Markus Pilman
f321933bc9
Minor bug fix
2023-02-06 17:34:46 +01:00
Junhyun Shim
0d1801ed88
Expose private endpoints used in MVC mode and make MVT token sticky
2023-02-06 17:23:26 +01:00
Josh Slocum
4101379295
Assumption that caller would be cancelled after delay(0) after triggering simInjectFailure was false because of granuleMetadata.clearAsync. ( #9286 )
2023-02-06 09:05:36 -06:00
Markus Pilman
1fb3c33b3f
Merge remote-tracking branch 'origin/main' into features/tenant-lock2
...
# Conflicts:
# fdbserver/CommitProxyServer.actor.cpp
2023-02-06 11:58:49 +01:00
Markus Pilman
280e50d42d
fixed compilation issues
2023-02-06 11:56:39 +01:00
Xiaoxi Wang
3a84fc5375
fix temporary tenantMode=0 by only updating db info when recovery count increasing
2023-02-05 18:37:46 -08:00
A.J. Beamon
6991b62106
Improve comments on runRYWTransaction*, use MAX_TENANTS_PER_CLUSTER instead of TOO_MANY knob in tenant cache.
2023-02-03 15:35:47 -08:00
Xiaoxi Wang
f1dd3aea92
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/listTenant
2023-02-03 15:27:21 -08:00
Xiaoxi Wang
7190fa0c08
Merge branch 'main' of https://github.com/apple/foundationdb into fix/main/testTimeout
2023-02-03 13:48:54 -08:00
Xiaoxi Wang
b757e8914a
fix BOOST_SYSTEM_NO_LIB redefinition in CI
2023-02-03 13:47:50 -08:00
Xiaoxi Wang
f1a7841fb8
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/listTenant
2023-02-03 13:02:29 -08:00
Xiaoxi Wang
8371883e3b
remove log; disabled clear range temporarily
2023-02-03 12:49:23 -08:00
Junhyun Shim
ce652fa284
Replace AuthZ's use of tenant names in token with tenant ID
...
Also, to minimize audit log loss, handle token usage audit logging at each usage.
This has a side-effect of making the token use log less bursty.
This also subtly changes the dedup cache policy.
Dedup time window used to be 5 seconds (default) since the start of batch-logging.
Now it's 5 seconds from the first usage since the closing of the previous dedup window
2023-02-03 21:46:31 +01:00
Xiaoxi Wang
b0b8f21dcc
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/clearRange
2023-02-03 12:27:03 -08:00
Xiaoxi Wang
b0902a515a
handle prefix convert to negative id
2023-02-03 12:26:25 -08:00
Ankita Kejriwal
1a240a0eab
Merge branch 'main' of github.com:apple/foundationdb into debug2
2023-02-03 12:26:19 -08:00
Xiaoxi Wang
0e242226bc
add trace log; fix size bug
2023-02-03 10:36:57 -08:00
Hui Liu
cafff2c4d0
Merge pull request #9290 from sfc-gh-huliu/splitpoints
...
Support pagination for StorageServer splitMetrics API
2023-02-03 09:43:09 -08:00
Ankita Kejriwal
913bc9114d
Handle errors that can't be sent in reply; fix formatting
2023-02-02 19:05:05 -08:00
Xiaoxi Wang
ae6b57d5c8
handle system keys
2023-02-02 16:25:41 -08:00
Xiaoxi Wang
3e1fee8ed4
finalize splitClearRangeByTenant unit test and implement replaceRawClearRanges
2023-02-02 15:19:21 -08:00
Hui Liu
774446d3a0
Support pagination for StorageServer splitMetrics API
2023-02-02 14:21:38 -08:00
Hui Liu
d5ff370e4e
Add metadataVersion to manifest backup
2023-02-02 11:39:01 -08:00
Xiaoxi Wang
347b3d4ca2
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/listTenant
2023-02-02 09:24:14 -08:00
Vaidas Gasiunas
f8b1da8bc6
An option to initialize client tracing in setupNetwork ( #9209 )
...
* client_config_tester: use a generic mechanism to set specific network options
* trace_initialize_on_setup option to initialize client traces on network setup without local IP address
* trace_initialize_on_setup: Addressing review comments
* Restore correct formatting
* trace_initialize_on_setup: Update go bindings
* Include PID for identification into trace file names by default
* Use the same naming pattern for trace files in all configurations
* Empty commit
2023-02-02 10:00:51 +01:00
Xiaoxi Wang
a8013bad5c
add splitClearRangeByTenant and unit tests
2023-02-01 20:25:56 -08:00
Ankita Kejriwal
9529f360ee
Update storage metrics functions to use the version at which tenant was read
2023-02-01 19:44:31 -08:00
Nim Wijetunga
86f3665514
Handle EKP Tenant Not Found Errors ( #9261 )
...
handle EKP tenant not found errors
2023-02-01 19:15:38 -08:00
Xiaoxi Wang
128bfccfa7
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/listTenant
2023-02-01 13:16:28 -08:00
Josh Slocum
6dbb96312b
move feed cleanup check to after data is guaranteed to be available for granule ( #9283 )
2023-02-01 13:19:08 -06:00
Xiaoxi Wang
c37042c81d
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/listTenant
2023-02-01 10:08:32 -08:00
Jingyu Zhou
982ce105e5
Reduce logging level for verbose events
...
From one of nightly failure due to too many log lines, these are top 3:
60100 FastRestoreLoaderDispatchRequests
79655 FastRestoreGetVersionSize
93888 FastRestoreSplitMutation
2023-02-01 09:51:16 -08:00
Jingyu Zhou
9c5ce68d98
Merge branch 'main' of https://github.com/apple/foundationdb into fix
2023-01-31 17:40:13 -08:00
Xiaoxi Wang
a447ec98a0
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/listTenant
2023-01-31 16:25:07 -08:00
A.J. Beamon
6665eb6654
Update cluster recovery for tenant ID changes
2023-01-31 16:06:45 -08:00
He Liu
eb4839d046
Fix audit_storage issues ( #9265 )
...
* Erase audit from DD after completion.
* Use CLIENT_KNOBS->TOO_MANY as the default number of rows for
get_audit_status.
2023-01-31 15:39:01 -08:00
Josh Slocum
77875d7fee
Fixing SkewedReadWrite to load its metadata in a transactionally consistent way ( #9274 )
2023-01-31 14:52:36 -06:00