Commit Graph

20366 Commits

Author SHA1 Message Date
Markus Pilman d4ee7be1d7 Reduce excessive tracing and fail after 1M traces 2022-04-19 21:11:51 -06:00
Junhyun Shim 278009e1c8 Let OP_TRANSACTION to measure one iteration of '-x' 2022-04-20 04:45:29 +02:00
Evan Tschannen 442d2b34c7
fix: pops which were ignored during a snapshot would not be replayed on the proper tlogs within a shared tlog (#6892) 2022-04-19 16:57:41 -07:00
Chaoguang Lin 6bfa4f6ac4 Fix typos; Polish documentations according to comments 2022-04-19 13:57:11 -07:00
Andrew Noyes 297d831192
Put guard pages next to fast alloc memory (#6885)
* Put guard pages next to fast alloc memory

I verified that we can now detect #6753 without creating tons of
threads.

* Use pageSize instead of 4096

* Don't include mmapInternal for windows
2022-04-19 11:22:35 -07:00
Bharadwaj V.R 51ef860612
Merge branch 'apple:main' into block-down 2022-04-19 10:16:56 -07:00
Vaidas Gasiunas eedac50bb5
Merge pull request #6888 from sfc-gh-vgasiunas/vgasiunas-upgrade-test-72
Regression tests for upgrades to 7.2
2022-04-19 17:42:01 +02:00
Junhyun Shim 0cf1349c37 Optimizations and restructuring 2022-04-19 16:34:11 +02:00
Vaidas Gasiunas 3372b3a6eb Upgrade Tests: Check logs for warnings and dump them 2022-04-19 13:37:21 +02:00
Vaidas Gasiunas 79b2ebeb4b Upgrade Tests: Test upgrade from 7.0 to 7.2 2022-04-19 11:53:33 +02:00
Vaidas Gasiunas 43b3ac23f8
Merge pull request #6876 from sfc-gh-ljoswiak/fixes/upgrade-tests-arm-support
Disable end-to-end upgrade tests for non-x86_64 platforms
2022-04-19 10:55:35 +02:00
Ata E Husain Bohra a38318a6ac
Update 'salt' details for EncryptHeader AuthToken details (#6881)
* Update 'salt' details for EncryptHeader AuthToken details

Description

Major changes:
1. Add 'salt' to BlobCipherEncryptHeader::cipherHeaderDetails.
2. During decryption it is possible that BlobKeyCacheId doesn't
    contain required baseCipherDetails. Add API to KeyCache to
    allowing re-populating of CipherDetails with a given 'salt'
3. Update BaseCipherKeyIdCache indexing using {BaseCipherKeyId, salt}
    tuple. FDB processes leverage BlobCipherKeyCache to implement
    in-memory caching of cipherKeys, given EncryptKeyProxy supplies
    BaseCipher details, each encryption participant service would
    generate its derived key by using different 'salt'. Further,
    it is possible to cache multiple {baseCipherKeyId, salt} tuples;
    for instance: CP encrypted mutations being deciphered by
    StorageServer etc.

Testing

1. Update EncyrptionOps simulation test to simulate KeyCache miss
2. Update BlobCipher unit tests to validate above mentioned changes
2022-04-18 22:01:56 -07:00
Jingyu Zhou 8003f4207f
Merge pull request #6843 from sbodagala/main
Do not invoke version vector related code when the feature is disabled
2022-04-18 21:35:23 -07:00
Xiaoxi Wang 7c643369c3 Merge branch 'main' of https://github.com/apple/foundationdb 2022-04-18 21:24:19 -07:00
Chaoguang Lin 36c876b769 Fix a typo; Fix a format error in special-keys.rst 2022-04-18 18:10:17 -07:00
Chaoguang Lin 4af36fa9ea Update documentations 2022-04-18 17:59:28 -07:00
Chaoguang Lin c0264a8522 Remove the client profiling special keys and update related documentations 2022-04-18 17:54:50 -07:00
Ray Jenkins d45d6f9f1f
Merge pull request #6882 from sfc-gh-rjenkins/typo-client-testing
Fix typo, expose a function...
2022-04-18 16:43:42 -05:00
Sreenath Bodagala abd3d5a3d7 Merge remote-tracking branch 'apple-upstream/main' 2022-04-18 20:55:16 +00:00
Russell Sears 8819534a5e
Improve YCSB image logging; remove logic to support partial restarts (#6830) 2022-04-18 13:32:04 -07:00
Ray Jenkins 865b5f1ff8 Substitute other with author. 2022-04-18 14:50:30 -05:00
Ray Jenkins 28ee6fd7b2 Fix typo, expose a function... 2022-04-18 14:45:03 -05:00
Lukas Joswiak 01807502ca Remove Arm support 2022-04-18 12:39:39 -07:00
Lukas Joswiak 07d11ec2e1 Fix failing upgrades due to non-persisted initial cluster version 2022-04-18 10:59:17 -07:00
Markus Pilman 3cbba4bea4
Don't test requests that don't initialize properly (#6880)
* Don't test requests that don't initialize properly

Some request objects don't initialize their members
properly when being constructed using the default
constructor. This makes valgrind unhappy. Don't test
these endpoints for now.

* fixed code formatting
2022-04-18 10:44:56 -07:00
Jingyu Zhou 17dc1a61f3 ClientDBInfo may be unintentionally not set
The ClientDBInfo's comparison is through an internal UID and shrinkProxyList()
can change proxies inside ClientDBInfo. Since the UID is not changed by that
function, subsequent set can be unintentionally skipped.

This was not a big issue before. However, VV introduces a change that the
client side compares the returned proxy ID with its known set of GRV proxies
and will retry GRV if the returned proxy ID is not in the set. Due the above
bug, GRV returned by a proxy is not within the client set, and results in
indefinite retrying GRVs.
2022-04-18 09:09:14 -07:00
Bharadwaj V.R b10ba334de
Merge branch 'apple:main' into block-down 2022-04-18 08:56:04 -07:00
Markus Pilman 1f26943099
Merge pull request #6859 from sfc-gh-ajbeamon/check-tenant-clear-range
When clearing the database between tests, check that the normal key-space is empty
2022-04-16 11:24:41 -06:00
Lukas Joswiak 7b737e91a1 Disable upgrade tests an non-x86_64 architectures 2022-04-15 16:28:11 -07:00
Lukas Joswiak 86a4e778d6 Enable Arm architecture in end-to-end upgrade tests 2022-04-15 16:24:49 -07:00
Bharadwaj V.R 47a1a282e9
Merge branch 'apple:main' into dbcorever 2022-04-15 15:47:14 -07:00
A.J. Beamon 6151f9c858
Merge pull request #6873 from sfc-gh-ajbeamon/tenant-test-fix
The tenant deletion test now deletes multiple tenants concurrently rather than serially
2022-04-15 14:29:47 -07:00
Jingyu Zhou 0a03b190da Fix multiple PeekStream requests to log routers
There is a bug in how a log router handles streaming read:
* Log router has a `logRouterPeekStream` actor A running.
* Remote tlog detects some problem and starts another streaming connection (maybe just reuse the connection?)
* Log router now has a new `logRouterPeekStream` actor B running.
* B runs and found that popped version > reqBegin, so `LogRouterPeekPopped` . This is because A is still running and changed the popped version.
* A ends with `TLogPeekStreamEnd operation_obsolete`
* B become stuck at `wait(req.reply.onReady() && store(reply.rep, future)`, because the future was sent `Never()`.

As a result, the remote tlog can no longer retrieve data from this log router.

Fix by killing the `logRouterPeekStream` B.
2022-04-15 14:11:52 -07:00
Bharadwaj V.R 11f2827c82 Make names even across change and add lowest-compatible-version to status json 2022-04-15 13:57:58 -07:00
Vaidas Gasiunas eb436ec14b
Merge pull request #6868 from sfc-gh-vgasiunas/vgasiunas-disable-upgrade-tests-sanitizer
Disable upgrade tests in sanitizer builds
2022-04-15 22:55:15 +02:00
A.J. Beamon e2222355dc The tenant deletion test now deletes multiple tenants concurrently rather than serially. Fix some variable shadowing in the delete test. 2022-04-15 13:17:19 -07:00
neethuhaneesha 5ffd9e0172
Merge pull request #6867 from neethuhaneesha/rocksdb-aggrProperty
RocksDb using aggr property metrics for pendingCompactionBytes.
2022-04-15 11:02:29 -07:00
Bharadwaj V.R ce5bcc0261 Remove system key tracking newest server version 2022-04-15 10:57:11 -07:00
Bharadwaj V.R 576957d335 Fix version status reporting 2022-04-15 09:52:42 -07:00
Bharadwaj V.R 3787ddae89 Add trace-based status reporting for server version 2022-04-15 09:04:52 -07:00
Vaidas Gasiunas 5abdb1e655 Upgrade Tests: Disable upgrade tests in sanitizer builds 2022-04-15 16:33:48 +02:00
Junhyun Shim e74a676e26 Perf-optimize synchronous execution path
- Wrap timer ops into if blocks
- Pull key/value buffer out to upper level function
2022-04-15 07:50:12 +02:00
Neethu Haneesha Bingi 6543bce8ae RocksDb using aggr property metrics for pendingCompactionBytes. 2022-04-14 18:08:42 -07:00
Xiaoxi Wang 0166b8ce69 Merge branch 'main' of https://github.com/apple/foundationdb 2022-04-14 14:33:57 -07:00
Xiaoxi Wang aa96af47ad fix doc compilation 2022-04-14 12:14:42 -07:00
Andrew Noyes 29cf5f1fbf
Fix an ASSERT when an fdbcli command times out (#6857)
* Re-throw operation_cancelled

There's a few places in fdbcli where we don't rethrow operation
cancelled but wait on a future. It's very unusual that you don't want to
rethrow operation_cancelled.

* Update ASSERT

It's possible to get error_code_broken_promise here if the network has
already shutdown.
2022-04-14 12:09:25 -07:00
A.J. Beamon cf5d3c83a1 Fix formatting issues. 2022-04-14 12:03:39 -07:00
A.J. Beamon 19d78cf2a3 When clearing the database between tests, check that clearing the tenant left the entire normal key-space empty. Update the configuration of some tests. Disable a special key-space test that is invoking broken behavior. 2022-04-14 11:39:02 -07:00
Bharadwaj V.R adfc4ca379 Revert accidental commit of a ProtocolVersion.h 2022-04-14 11:12:26 -07:00
Bharadwaj V.R 78c4771f9d Fix ProtocolVersion.h formatting diffs 2022-04-14 11:10:39 -07:00