Commit Graph

24972 Commits

Author SHA1 Message Date
Junhyun Shim 499a7de718 Mako: add missing token reassignment after transaction reset
Also remove redundant calls to tx.reset() in step functions after OP_COMMIT
(The step function caller is already resetting the object after OP_COMMIT is run successfully)
2023-02-09 13:35:21 +01:00
Ata E Husain Bohra 9c649d7880
EaR: Configurable encryption framework (#9271)
* EaR: Configurable encryption framework

Description

EaR implementation only supports fixed size on-disk encryption header format.
One drawback of the scheme is, introducing a newer encryption scheme as well
as updating header format in future may incur data migration restrictions.
Major changes proposed in the patch includes:
1. Flexible Encryption header format allowing the following:
 1.1. Header flags (metadata) can evolve separately from the encryption algorithm
 1.2. Specific encryption algorithm header to allow future extensions.
2. Update the BlobCipher encryption/decryption util classes to work with newer
encryption header format.
3. Continue supporting multiple encryption authentication schemes such as:
HMAC-SHA and AES-CMAC; also, supports no encryption-authentication schemes.
4. Refactor BlobCipher unit test to enable testing of new format.
5. Configuration knobs to control encryption header flags and algorithm
versions.

Note: 
The on-disk header storage footprint savings due to the newer scheme is as follows:
1. No encryption authentication: 54% smaller compared to existing implementation.
3. AES-CMAC: 16% smaller compared to existing implementation.
3. HMAC-SHA encryption authentication: almost same size.


Testing

BlobCipherTest
EncryptionOpsTest
2023-02-08 22:51:05 -08:00
Xiaoxi Wang fcebcbcf72 reset TenantManagement listTenantMetadataTransaction 2023-02-08 21:11:43 -08:00
Xiaoxi Wang e5d3fd2e96 add debug events; fix list tenant name bug 2023-02-08 20:37:27 -08:00
Dan Adkins 07c4924645 Fix typo in format string. 2023-02-08 17:15:07 -08:00
Hao Fu 2790693862
Fix getMappedRange metrics (#9330) 2023-02-08 16:48:36 -08:00
Xiaoxi Wang a77bf236c7 fix unit test bug 2023-02-08 16:09:32 -08:00
Ankita Kejriwal 05dce05f52
Merge pull request #9329 from sfc-gh-akejriwal/debug2
Remove the version bump in `checkTenantEntry` on storage server
2023-02-08 14:53:14 -08:00
Xiaoxi Wang e75f38a6fc extract pushToBackupMutations method and add single tenant range validation 2023-02-08 14:35:35 -08:00
sfc-gh-tclinkenbeard 9a3cdb9a3b Add /GrvProxyTagThrottler/MultiTag unit test 2023-02-08 14:28:21 -08:00
sfc-gh-tclinkenbeard ed20a30c11 Remove debugging trace event 2023-02-08 14:28:21 -08:00
Jingyu Zhou 1a9aed795f
Merge pull request #9327 from sfc-gh-tclinkenbeard/enable-gtt-by-default
Enable `GLOBAL_TAG_THROTTLING` by default
2023-02-08 14:03:39 -08:00
Josh Slocum 1e5bac6238
fixing fault injection stalling change feed fetch (#9316) 2023-02-08 15:49:56 -06:00
Dan Adkins f36c20248e Add timer_monotonic() to CommitDebug trace events.
CommitDebug trace events are useful for measuring, in detail, the time spent in
the various parts of a single transaction. Like all log events, they have a time
associated with them. This time comes for now(), which in a real fdb system is
only updated in the run loop. This renders the timestamps inaccurate in certain
CPU bound sections which don't have a wait, e.g. in the resolver.

The desired solution is to directly call ::timer_monotonic() instead of now()
for a timestamp, since ::timer_monotonic() is what now() ultimately uses in
the run loop, but is updated between waits as well.

In order to not disrupt any current uses of the logs which might depend on the
current behavior of the Time field, we introduced the `MonotonicTime` field.

This change only affects CommitDebug trace events, not trace events in general.
2023-02-08 13:39:56 -08:00
Xiaoxi Wang 1ff7865ee2 Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/clearRange 2023-02-08 13:38:55 -08:00
Ankita Kejriwal 0789ab35e9 Merge branch 'main' of github.com:apple/foundationdb into debug2 2023-02-08 13:26:39 -08:00
Xiaoxi Wang e28fc664d0 extract shouldBackup method 2023-02-08 13:25:46 -08:00
Ankita Kejriwal cc3ae02961 Remove the version bump in `checkTenantEntry` on storage server
This was mistakenly added in PR #9291.
2023-02-08 12:08:04 -08:00
sfc-gh-tclinkenbeard 484dc4f74c Enable GLOBAL_TAG_THROTTLING by default 2023-02-08 11:34:31 -08:00
Ankita Kejriwal b9630e57d9
Merge pull request #9291 from sfc-gh-akejriwal/debug2
Update storage metrics functions to use the version at which the tenant was read
2023-02-08 11:33:33 -08:00
Junhyun Shim d7d0b658a3
Merge pull request #9314 from sfc-gh-jshim/authz-tenant-name-to-tenant-id
Make Authz use tenant ids instead of tenant names
2023-02-08 20:13:29 +01:00
Josh Slocum ebbd221fe7
formatting files (#9322)
* formatting files

* another file
2023-02-08 13:01:24 -06:00
Ankita Kejriwal 187f2a42e8 Merge branch 'main' of github.com:apple/foundationdb into debug2 2023-02-08 10:13:39 -08:00
A.J. Beamon e210c7a933 Fix storage quota metadata to use the tenant metadata subspace as a prefix rather than a suffix 2023-02-08 10:12:17 -08:00
Josh Slocum 2f8f7515e8
adding java blob granules unit test (#9317)
* adding java blob granules unit test

* reduce sleep and organize imports
2023-02-08 12:02:14 -06:00
Junhyun Shim ea0083cda2 Remove redundant calls to setAuthToken 2023-02-08 18:56:10 +01:00
sfc-gh-tclinkenbeard dd2ba18d45 Create GLOBAL_TAG_THROTTLING_MIN_TPS knob 2023-02-08 09:48:33 -08:00
sfc-gh-tclinkenbeard 7f9ec344db GlobalTagThrottlerImpl::getAverageTransactionCost doesn't compute cost when transaction rate is <1 2023-02-08 09:48:33 -08:00
Junhyun Shim d32d3dd085 Make token option NOT survive transaction hard resets 2023-02-08 18:12:22 +01:00
Ankita Kejriwal a5257144c6 Merge branch 'main' of github.com:apple/foundationdb into debug2 2023-02-08 09:04:17 -08:00
Markus Pilman d9bd49fdaa Make lock id a map 2023-02-08 16:26:34 +01:00
Markus Pilman 1c712df709 Tenant lock workload succeeding 2023-02-08 15:51:09 +01:00
Markus Pilman ccdf579648
make lockId spelling consistent in xff keyspace
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2023-02-08 11:16:23 +01:00
Ankita Kejriwal 6077419900 Update the waitMetrics version if it is too old on the storage server. 2023-02-07 19:33:58 -08:00
Nim Wijetunga 1cd0922ed1
Snapshot Backup Bug Fix (#9319)
snapshot restore bug fix
2023-02-07 18:44:03 -08:00
A.J. Beamon 42eb82cd30 Merge branch 'main' into storage-quota-in-tenant-metadata-space 2023-02-07 15:11:29 -08:00
A.J. Beamon 0410f7d9a1 Add intial support for restoring a management cluster in a metacluster 2023-02-07 15:08:46 -08:00
Jon Fu c8e12a7921
Add tenant watches to storage server (#9284)
* add tenant watches to storage server

* combine watches and throw different error after first loop

* try to change SS tenantmap to avoid tenantname altogether

* update code style to be consistent

* add new operation in tenant management test

* address code review changes

* add more to tenant workload and fix id parse bug

* account for TenantRef in TenantData when renaming

* code review comments

* change test assertion
2023-02-07 17:08:51 -05:00
Xiaoxi Wang 145b81170c add unit test for replaceRawClearRanges 2023-02-07 13:21:18 -08:00
Ankita Kejriwal 5f00ee681f
Merge pull request #9277 from sfc-gh-akejriwal/main
Add a high level guide for the storage quota feature
2023-02-07 13:10:34 -08:00
Ankita Kejriwal 354ea1cb7e Merge branch 'main' of github.com:sfc-gh-akejriwal/foundationdb 2023-02-07 11:53:24 -08:00
Ankita Kejriwal 22b4bef9f9 Merge branch 'main' of github.com:apple/foundationdb 2023-02-07 11:52:44 -08:00
Ankita Kejriwal 58a4c4a16e
Update design/storage-quota.md as per review comment
Co-authored-by: Trevor Clinkenbeard <trevor.clinkenbeard@snowflake.com>
2023-02-07 11:34:20 -08:00
Vaidas Gasiunas e1ec0d1ae2
Fix UBSAN build for C shim library (#9311) 2023-02-07 20:17:07 +01:00
Junhyun Shim 21651bdd6d Log failed security verifications from server-side
Log request types related to private endpoint access and failed security verifications
2023-02-07 18:51:25 +01:00
Junhyun Shim c4703b5940 Mako: lower tenant id fetch error to debug
With MVC mode, it always fails at least once for each tenant and gets spammy
2023-02-07 18:44:43 +01:00
Josh Slocum e6c7927619
improving bg restart with higher workload and chaos for cycle, and new large restart workload (#9313) 2023-02-07 07:47:53 -06:00
Junhyun Shim ff61e2990b Fix clang build error and relax authz test tx timeout 2023-02-07 09:59:39 +01:00
Ankita Kejriwal d8bc494859 Merge branch 'main' of github.com:apple/foundationdb into debug2 2023-02-06 16:44:47 -08:00
Ankita Kejriwal 68f3e29a47 Merge branch 'main' of github.com:apple/foundationdb 2023-02-06 16:24:47 -08:00