7d7ca0bb34
Apply suggestions from code review
...
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2023-02-22 16:28:53 -07:00
b79885d5b9
Merge remote-tracking branch 'origin/main' into features/tenant-lock-fdbcli
...
# Conflicts:
# fdbserver/ApplyMetadataMutation.cpp
2023-02-15 18:37:18 +01:00
afc25035f8
Client status report API in Java and python bindings ( #9366 )
...
* get_client_status: add to Java bindings
* Add get_client_status in python binding
* Make python unit tests available as ctest
* Fix file name in the copyright header
* Fix library path name for mac
* get_client_status: Minor changes in Java & Python bindings addressing review comments
* Rename fdb_python_unit_tests to unit_tests
2023-02-15 18:00:53 +01:00
8c94b340ce
EaR: Update encryption methods to make 'cipherHeaderKey' optional ( #9378 )
...
* EaR: Update encryption methods to make 'cipherHeaderKey' optional
Description
diff-1: Address review comments
Major changes includes:
1. Update BlobCipher Encrypt/Decrypt classes to make 'headerCipher' optional
2. Update GetEncryptionCipherKeys actor methods to make 'headerCipherKey' optional
3. Update the usage across all encryption participant methods
Testing
BlobCipherUnitTest
EnryptedBackupCorrecctness
BlobGranuleCorrectness*
devRunCorrectness - 100K
2023-02-15 08:56:11 -08:00
b97ee87eba
Merge pull request #9367 from jzhou77/fix
...
Enable RocksDB restarting tests
2023-02-14 20:47:31 -08:00
a5ca96ddf6
Merge pull request #9342 from sfc-gh-ajbeamon/metacluster-mgmt-restore
...
Metacluster restore support
2023-02-14 20:41:20 -08:00
3d882a99c5
EaR: Refactor encryption header std::variant serializer and versioning ( #9345 )
...
Changes:
1. Make binary serializer natively support `std::variant`. Serialize size is 1 byte (the type index, i.e. `std::variant::index()`), plus the serialize size of the actual type stored in the `std::variant`. Update `BlobCipherEncryptHeaderRef` to use the `std::variant` binary serializer
3. Remove `flagsVersion` and `algoHeaderVersion` from `BlobCipherEncryptHeaderRef`. The former is replaced by `flags.index() + 1`, and the latter is moved into each of the algorithm-specific sub-headers. Each sub-header types will have nesting version-specific subtypes to handle serialization of that specific version (e.g. for `AesCtrNoAuth` it has a `AesCtrNoAuthV1` subtype).
2023-02-14 20:19:27 -08:00
c27cdcd2df
Fix arm nightly tests by skipping "until" restarting tests if no old binaries ( #9362 )
...
* Fix the joshua bug for restarting tests with until keyword
* Remove unnecessary changes
* Remove unnecessary changes
* Remove unnecessary changes
* Add comments, remove debugging symbols
2023-02-14 18:27:50 -08:00
bf85c9f8af
Backup Mutation Log Separates Tenant Map Modifications During Restore ( #9292 )
...
mutation log separates tenant map modifications
2023-02-14 16:46:09 -08:00
2fe32610d6
Merge pull request #9376 from sfc-gh-yiwu/encrypt_build
...
EaR: fix BlobCipher.h build failure
2023-02-14 16:16:18 -08:00
adc4e932c4
Merge pull request #9364 from sfc-gh-ajbeamon/fix-get-mapped-range-assertion
...
Fix get mapped range test assertion
2023-02-14 15:56:41 -08:00
e29fe81b14
Merge branch 'main' into encrypt_build
2023-02-14 14:55:20 -08:00
d99fb2ee02
Remove storageEngineExcludeTypes from blob tests
...
I.e., enable rocksDB for these tests.
2023-02-14 14:53:14 -08:00
4184239e72
Merge pull request #9372 from sfc-gh-xwang/fix/main/anyExisted
...
fix anyExisted when beginTenant==endTenant
2023-02-14 14:48:47 -08:00
cc3536e48b
EaR: fix BlobCipher.h build failure
2023-02-14 13:43:41 -08:00
fe18c87ac6
EaR: commit proxy fetch additional cipher keys post-resolution ( #9308 )
...
Commit proxy needs to fetch additional cipher keys post-resolution, since tenant ids for raw access requests and cross-tenant clear ranges are calculated after resolution.
2023-02-14 13:05:51 -08:00
87a71049df
Fix one more test toml spec
2023-02-14 12:52:39 -08:00
a93407303f
Include missing tenants in the restore output if their state is already an error state
2023-02-14 12:39:56 -08:00
7284e691fb
Fix a few minor restore bugs and add a dry-run mode. Some improvements to the fdbcli output.
2023-02-14 12:28:55 -08:00
99e8b95bf0
Fix restarting to 7.2 tests for sharded rocks
2023-02-14 10:36:29 -08:00
d28f253182
Save shard_encode_location_metadata knob value for restarting tests
...
This is needed so that sharded rocks use consistent knob values.
2023-02-14 09:57:08 -08:00
53f105eec5
fix anyExisted when beginTenant==endTenant
2023-02-14 09:07:09 -08:00
401b9c8918
EaR: Helper routines to support configurable encryption ( #9368 )
...
* EaR: Helper routines to support configurable encryption
Description
Add helper methods to BlobCipherEncryptHeaderRef enabling:
1. Extract 'IV' abstracting out underlying algorithm header
1. Extract 'cipherDetails' abstracting out underlying algorithm header
Testing
BlobCipherUnitTest & EncryptionOps are updated - 100K loop
* EaR: Helper routines to support configurable encryption
Description
Add helper methods to BlobCipherEncryptHeaderRef enabling:
1. Extract 'IV' abstracting out underlying algorithm header
1. Extract 'cipherDetails' abstracting out underlying algorithm header
Testing
BlobCipherUnitTest & EncryptionOps are updated - 100K loop
2023-02-14 08:34:41 -08:00
32684c802b
Add tenant lock commands and fdcli tests
2023-02-14 16:05:21 +01:00
a2ec8632b2
Merge branch 'features/tenant-lock2' into features/tenant-lock-fdbcli
2023-02-14 10:57:07 +01:00
a4d3035f64
Enable RocksDB restarting tests
...
Disable sharded rocks storage for downgrade tests where we
Need to keep knob "shard_encode_location_metadata" so that downgrade tests
can pass the second phase.
2023-02-13 20:12:19 -08:00
f3b58a063f
Fix some merge issues and review comments
2023-02-13 15:32:44 -08:00
05e73b7836
Merge pull request #9361 from apple/sfc-gh-dadkins/disk-failure-attrition2
...
Disable machine attrition in DiskFailure workload.
2023-02-13 14:07:24 -08:00
958ff862e0
Fix some merge issues
2023-02-13 12:59:48 -08:00
98407809d9
Merge branch 'main' into metacluster-mgmt-restore
...
# Conflicts:
# fdbcli/MetaclusterCommands.actor.cpp
# fdbclient/Metacluster.cpp
# fdbclient/include/fdbclient/MetaclusterManagement.actor.h
# fdbserver/workloads/MetaclusterManagementWorkload.actor.cpp
# tests/CMakeLists.txt
2023-02-13 12:30:33 -08:00
0127dd4b5a
Merge pull request #9356 from sfc-gh-ajbeamon/metacluster-concurrency-testing
...
Add metacluster concurrency test and fix various bugs that it found
2023-02-13 11:57:47 -08:00
473dd33a1f
Fix get mapped range test assertion to account for the possibility of a range terminating early when it reaches the end of a shard
2023-02-13 11:53:47 -08:00
93b1e04aa9
Merge pull request #9355 from sfc-gh-ajbeamon/fix-assigned-cluster-test-retry-logic
...
Fix logic in MetaclusterManagementWorkload when retrying a tenant creation with an invalid assigned cluster
2023-02-13 10:43:09 -08:00
303a20c323
fix compilation issue
2023-02-13 19:08:38 +01:00
0172ca9615
Address review comments
2023-02-13 18:27:18 +01:00
8a47a484c9
Update fdbserver/storageserver.actor.cpp
...
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2023-02-13 18:25:03 +01:00
fa7f15e46a
Merge pull request #9353 from sfc-gh-yiwu/redwood_restart
...
Redwood: fix restart test failure with XOR encoding
2023-02-13 09:24:57 -08:00
3b4bd1692a
Remove unused function
...
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2023-02-13 18:22:25 +01:00
bba782cc56
Add command
2023-02-13 18:21:55 +01:00
5ede2d439c
Disable machine attrition in DiskFailure workload.
...
The machine attrition logic doesn't take into account the possibility
that a disk corruption could an unrecoverable failure in the cluster.
Before disabling attrition during the DiskFailure workload, the failure
rate was >10/100,000 in the DiskFailureCycle test. Afterwards, there
were no failures in 100,000 runs.
2023-02-13 08:53:58 -08:00
40f571fddc
fix merge mistake
2023-02-13 12:26:22 +01:00
3017e15448
Merge remote-tracking branch 'origin/main' into features/tenant-lock2
...
# Conflicts:
# fdbclient/include/fdbclient/TenantManagement.actor.h
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/TenantManagementWorkload.actor.cpp
2023-02-13 11:43:07 +01:00
844890bf93
Merge pull request #9042 from imperatorx/add-redwood-slack-knob
...
Introduced a new knob to control Redwood slack balance when splitting a page
2023-02-12 20:37:08 -08:00
41fa3eada9
Merge branch 'main' into add-redwood-slack-knob
2023-02-12 19:31:20 -08:00
a261c1d94c
Run tenant management concurrency alongside metacluster management concurrency. Fix a few issues where performing tenant operations returned undesirable errors when the associated cluster was removed.
2023-02-11 19:46:47 -08:00
93f892c085
Merge pull request #9340 from sfc-gh-xwang/fix/main/tenantList
...
fix the way verifyListFilter detect tenant state change
2023-02-11 17:20:46 -08:00
e6021f8326
Add Jon's metacluster concurrency test and fix various bugs that it found
2023-02-11 15:15:32 -08:00
21a2378de5
Merge pull request #9298 from sfc-gh-xwang/feature/main/clearRange
...
Split raw clear ranges across tenants in required mode
2023-02-11 14:29:46 -08:00
a9c7632c83
Merge branch 'main' of https://github.com/apple/foundationdb into fix/main/tenantList
2023-02-11 13:54:27 -08:00
ac1ddc81b0
remove debug trace; change function comment
2023-02-11 13:17:59 -08:00
Markus Pilman