Commit Graph

24814 Commits

Author SHA1 Message Date
Xiaoxi Wang 1ff7865ee2 Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/clearRange 2023-02-08 13:38:55 -08:00
Xiaoxi Wang e28fc664d0 extract shouldBackup method 2023-02-08 13:25:46 -08:00
Ankita Kejriwal b9630e57d9
Merge pull request #9291 from sfc-gh-akejriwal/debug2
Update storage metrics functions to use the version at which the tenant was read
2023-02-08 11:33:33 -08:00
Junhyun Shim d7d0b658a3
Merge pull request #9314 from sfc-gh-jshim/authz-tenant-name-to-tenant-id
Make Authz use tenant ids instead of tenant names
2023-02-08 20:13:29 +01:00
Josh Slocum ebbd221fe7
formatting files (#9322)
* formatting files

* another file
2023-02-08 13:01:24 -06:00
Ankita Kejriwal 187f2a42e8 Merge branch 'main' of github.com:apple/foundationdb into debug2 2023-02-08 10:13:39 -08:00
Josh Slocum 2f8f7515e8
adding java blob granules unit test (#9317)
* adding java blob granules unit test

* reduce sleep and organize imports
2023-02-08 12:02:14 -06:00
Junhyun Shim ea0083cda2 Remove redundant calls to setAuthToken 2023-02-08 18:56:10 +01:00
sfc-gh-tclinkenbeard dd2ba18d45 Create GLOBAL_TAG_THROTTLING_MIN_TPS knob 2023-02-08 09:48:33 -08:00
sfc-gh-tclinkenbeard 7f9ec344db GlobalTagThrottlerImpl::getAverageTransactionCost doesn't compute cost when transaction rate is <1 2023-02-08 09:48:33 -08:00
Junhyun Shim d32d3dd085 Make token option NOT survive transaction hard resets 2023-02-08 18:12:22 +01:00
Ankita Kejriwal a5257144c6 Merge branch 'main' of github.com:apple/foundationdb into debug2 2023-02-08 09:04:17 -08:00
Ankita Kejriwal 6077419900 Update the waitMetrics version if it is too old on the storage server. 2023-02-07 19:33:58 -08:00
Nim Wijetunga 1cd0922ed1
Snapshot Backup Bug Fix (#9319)
snapshot restore bug fix
2023-02-07 18:44:03 -08:00
Jon Fu c8e12a7921
Add tenant watches to storage server (#9284)
* add tenant watches to storage server

* combine watches and throw different error after first loop

* try to change SS tenantmap to avoid tenantname altogether

* update code style to be consistent

* add new operation in tenant management test

* address code review changes

* add more to tenant workload and fix id parse bug

* account for TenantRef in TenantData when renaming

* code review comments

* change test assertion
2023-02-07 17:08:51 -05:00
Xiaoxi Wang 145b81170c add unit test for replaceRawClearRanges 2023-02-07 13:21:18 -08:00
Ankita Kejriwal 5f00ee681f
Merge pull request #9277 from sfc-gh-akejriwal/main
Add a high level guide for the storage quota feature
2023-02-07 13:10:34 -08:00
Ankita Kejriwal 354ea1cb7e Merge branch 'main' of github.com:sfc-gh-akejriwal/foundationdb 2023-02-07 11:53:24 -08:00
Ankita Kejriwal 22b4bef9f9 Merge branch 'main' of github.com:apple/foundationdb 2023-02-07 11:52:44 -08:00
Ankita Kejriwal 58a4c4a16e
Update design/storage-quota.md as per review comment
Co-authored-by: Trevor Clinkenbeard <trevor.clinkenbeard@snowflake.com>
2023-02-07 11:34:20 -08:00
Vaidas Gasiunas e1ec0d1ae2
Fix UBSAN build for C shim library (#9311) 2023-02-07 20:17:07 +01:00
Junhyun Shim 21651bdd6d Log failed security verifications from server-side
Log request types related to private endpoint access and failed security verifications
2023-02-07 18:51:25 +01:00
Junhyun Shim c4703b5940 Mako: lower tenant id fetch error to debug
With MVC mode, it always fails at least once for each tenant and gets spammy
2023-02-07 18:44:43 +01:00
Josh Slocum e6c7927619
improving bg restart with higher workload and chaos for cycle, and new large restart workload (#9313) 2023-02-07 07:47:53 -06:00
Junhyun Shim ff61e2990b Fix clang build error and relax authz test tx timeout 2023-02-07 09:59:39 +01:00
Ankita Kejriwal d8bc494859 Merge branch 'main' of github.com:apple/foundationdb into debug2 2023-02-06 16:44:47 -08:00
Ankita Kejriwal 68f3e29a47 Merge branch 'main' of github.com:apple/foundationdb 2023-02-06 16:24:47 -08:00
Xiaoxi Wang 46027aa4aa
Merge pull request #9272 from sfc-gh-xwang/feature/main/listTenant
Change listTenants API to only list tenant Ids
2023-02-06 15:22:31 -08:00
Xiaoxi Wang 6df5f1fa56 Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/clearRange 2023-02-06 14:28:35 -08:00
Junhyun Shim 180ddba4ec Fix clang format 2023-02-06 23:14:39 +01:00
Junhyun Shim be225acd2a Merge remote-tracking branch 'origin/main' into authz-tenant-name-to-tenant-id 2023-02-06 23:13:43 +01:00
Josh Slocum 181970b160
adding metadata-only update for empty delta files to avoid blob storage writes (#9312) 2023-02-06 16:11:35 -06:00
Yi Wu d3bc2afc8e
EaR: storage server uses encryption DB config (#9115)
The PR is updating storage server and Redwood to enable encryption based on the encryption mode in DB config, which was previously controlled by a knob. High level changes are
1. Passing encryption mode in DB config to storage server
    1.1 If it is a new storage server, pass the encryption mode through `InitializeStorageRequest`. the encryption mode is pass to Redwood for initialization
    1.2 If it is an existing storage server, on restart the storage server will send `GetStorageServerRejoinInfoRequest` to commit proxy, and commit proxy will return the current encryption mode, which it get from DB config on its own initialization. Storage server will compare the DB config encryption mode to the local storage encryption mode, and fail if they don't match
2. Adding a new `encryptionMode()` method to `IKeyValueStore`, which return a future of local encryption mode of the KV store instance. A KV store supporting encryption would need to persist its own encryption mode, and return the mode via the API.
3. Redwood accepts encryption mode from its constructor. For a new Redwood instance, caller has to specific the encryption mode, which will be stored in Redwood per-instance file header. For existing instance, caller is supposed to not passing the encryption mode, and let Redwood find it out from its own header.
4. Refactoring in Redwood to accommodate the above changes.
2023-02-06 14:02:31 -08:00
Junhyun Shim 0d85425964 Add cross-tenant raw access testcase 2023-02-06 22:48:07 +01:00
Josh Slocum 0a19db8ce9
Blob granule restarting tests (#9259)
* Adding BlobGranuleRestartCycle tests

* test harness fix to not copy blob symlinks for restarting test
2023-02-06 14:54:20 -06:00
Junhyun Shim 1afd63d7e3 Minimize the risk of TracedTooManyLines in simulation
- Disable audit logging for simulation
- Relax the max_trace_lines knob limit to reduce false positives
2023-02-06 21:50:39 +01:00
Xiaoxi Wang 18a3de3594 Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/listTenant 2023-02-06 12:15:40 -08:00
Josh Slocum 73c9735c21
Bg api tenant tests (#9301)
* fixing tenant creation + setup order

* removing seenReadSuccess since verifyBlobRange is called before test start

* fixing bg_desc skip tenant check
2023-02-06 13:18:27 -06:00
Ankita Kejriwal 3b418b4431 Merge branch 'main' of github.com:apple/foundationdb into debug2 2023-02-06 11:07:28 -08:00
Jingyu Zhou 4809b22f6c
Merge pull request #9254 from sfc-gh-yiwu/restart_fix
Fix malform from_7.0.0/UpgradeAndBackupRestore test spec
2023-02-06 10:34:18 -08:00
Xiaoxi Wang 8cfc3f28e1
Merge pull request #9307 from sfc-gh-xwang/fix/main/tenantModeUpdate
fix temporary tenantMode=0 by only updating db info when recovery cou…
2023-02-06 10:25:32 -08:00
Junhyun Shim 0d1801ed88 Expose private endpoints used in MVC mode and make MVT token sticky 2023-02-06 17:23:26 +01:00
Josh Slocum 9eac2b5f8b
un-buggifying PEEK_TRACKER_EXPIRATION_TIME to invalid value (#9275) 2023-02-06 09:06:16 -06:00
Josh Slocum 4101379295
Assumption that caller would be cancelled after delay(0) after triggering simInjectFailure was false because of granuleMetadata.clearAsync. (#9286) 2023-02-06 09:05:36 -06:00
Josh Slocum 6b5500a930
auto-formatting test harness file (#9302) 2023-02-06 09:05:09 -06:00
Junhyun Shim 6993e391f2 Extend authz test to include with/without GRV caching and forced MVC 2023-02-06 13:18:10 +01:00
Yi Wu 680c9d49dc Fix malform from_7.0.0/UpgradeAndBackupRestore test spec 2023-02-05 19:57:59 -08:00
Xiaoxi Wang 3a84fc5375 fix temporary tenantMode=0 by only updating db info when recovery count increasing 2023-02-05 18:37:46 -08:00
Xiaoxi Wang 241f32262e Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/listTenant 2023-02-05 11:42:04 -08:00
Yanqin Jin 5087c08637
BUGGIFY `safeThreadFutureToFuture()` (#9263)
The `safeThreadFutureToFuture` function converts a `ThreadFuture` to a Future. There are a few cases where we have a
 result of type `ThreadFuture<Standalone<T>>` or `ThreadFuture<Optional<Standalone<T>>` where the memory is not
actually owned by the Standalone. Rather it is owned by the ThreadFuture.

Eventually we should fix this so that the memory is properly owned by the `Standalone`, which is beyond the scope of
this PR. Until then, we should update the implementation of `safeThreadFutureToFuture` to take away ownership from the
`Standalone`/`Optional<Standalone>` so we can detect this problem in simulation.

Test plan:
- correctness test with and without valgrind
2023-02-05 08:22:52 -08:00