853458cd75
Merge remote-tracking branch 'origin/main' into subtract-proxy-tag-throttle-duration
2022-10-15 10:27:39 -07:00
a556f21ed2
Merge remote-tracking branch 'origin/main' into limit-gtt-size
2022-10-15 10:26:16 -07:00
a0539bd63b
Addressed review comments
2022-10-15 10:26:13 -07:00
0f9da9d1ad
Merge branch 'main' of github.com:apple/foundationdb into getsizetenant
2022-10-14 16:50:51 -07:00
4ab947a9b3
Merge remote-tracking branch 'origin/main' into limit-gtt-size
2022-10-14 15:59:57 -07:00
b90722baa6
Add comment for GrvProxyStats latency bands and samples
2022-10-14 15:58:15 -07:00
914dfd7438
REST Blob metadata kms connector ( #8474 )
...
* Changing RESTKMSConnector request handling to not be synchronous, as that would be a huge perf bottleneck
* Implementation of blob kms fetch
* cleanup
* review comments
2022-10-14 17:49:00 -05:00
a24ab2d4fa
Incorporate code review comments + workload code cleanup
2022-10-14 13:59:36 -07:00
f5f33c5883
Merge pull request #8473 from sfc-gh-mpilman/features/enforce-workload-name
...
Force name and description of workloads to be the same
2022-10-14 12:42:03 -06:00
da5b5a354e
Fix failure injection naming
2022-10-14 11:28:03 -06:00
180238076c
Fix HighContentionPrefixAllocatorWorkload and make untrusted argument to workload factory type safe
2022-10-14 09:51:49 -06:00
fb1390ef3d
reformat code
2022-10-14 08:47:56 -06:00
7cec0a5249
Blob metadata refresh ( #8456 )
...
* Adding EKP refresh of blob metadata
* Adding re-fetching blob metadata from BGTenantMap
* adding buggifies from code review comments
2022-10-14 08:17:50 -05:00
49f0cf5ab0
Force name and description of workloads to be the same
2022-10-13 20:53:48 -06:00
3609fc8aef
Merge pull request #8470 from sfc-gh-jslocum/blob_kms_valgrind_fix
...
Fixing memory ownership in T -> TRef conversion
2022-10-13 17:57:22 -07:00
854212fe94
Incorportate code review suggestions
2022-10-13 17:41:31 -07:00
05df8cd64c
Fixing memory ownership in T -> TRef conversion
2022-10-13 17:19:54 -05:00
169c341f79
Merge pull request #8386 from sfc-gh-huliu/blobmigrator
...
Add blob migrator to assist data copy from blob to storage server
2022-10-13 14:46:04 -07:00
be64b12ecf
Clean up TraceEvents, TODOs, and simplify workload.
2022-10-13 14:31:42 -07:00
8d650390fc
Fixing race between feed cleanup and persisting memory mutations ( #8331 )
2022-10-13 16:11:48 -05:00
59686fa2e5
Increase the IO timeout to avoid flakiness in simulation tests.
2022-10-13 13:40:40 -07:00
049df622f1
add a blob migrator
2022-10-13 13:21:45 -07:00
df5825ff65
Merge pull request #8398 from sfc-gh-anoyes/anoyes/idempotency-id2
...
Initial work for automatic idempotency
2022-10-13 13:07:14 -07:00
b21b04da10
Merge pull request #8466 from sfc-gh-tclinkenbeard/add-fifo-unit-test
...
Add `/GrvProxyTransactionTagThrottler/Fifo` unit test
2022-10-13 12:20:16 -07:00
27b11797ff
Make pqOfQueues a min heap in the proper way
2022-10-13 10:08:02 -07:00
e4752309a1
Merge pull request #8075 from liquid-helium/validate-data-consistency
...
Validate data consistency
2022-10-13 09:20:01 -07:00
79f0e3fcaa
Improve mockFifoClient by waiting on all replies concurrently
2022-10-13 08:37:48 -07:00
63c018ccb3
Add /GrvProxyTransactionTagThrottler/Fifo unit test
2022-10-12 19:47:42 -07:00
7db6090766
Merge pull request #8445 from neethuhaneesha/rocksdb-hist
...
RocksDB commit latency sample corrections.
2022-10-12 18:33:45 -07:00
1bfe00ca61
Merge pull request #8449 from sfc-gh-ahusain/ahusain-auth-token-cleanup
...
Remove MultiToken Encryption auth-token generation
2022-10-12 17:51:42 -07:00
395ac11416
Merge pull request #8463 from sfc-gh-mpilman/bugfixes/different-cluster-same-rv-race
...
Fix DifferentClustersSameRV workload
2022-10-12 17:36:33 -06:00
3db0c786e5
RocksDB commit latency sample corrections.
2022-10-12 16:19:49 -07:00
97acc94a7f
Fixed out-of-scope variable issue.
2022-10-12 16:13:57 -07:00
2caa4290c6
Fixed clang variable scope issue.
2022-10-12 15:43:33 -07:00
9997ebba64
Fix DifferentClustersSameRV workload
2022-10-12 16:14:38 -06:00
999c5e3f49
Fixed clang compile errors.
2022-10-12 14:58:15 -07:00
6f01d97006
Subtract proxyTagThrottleDuration from measured GRV latencies
2022-10-12 14:41:32 -07:00
a2a372d083
Fix Redwood xor encoding incompatibility with 7.1
2022-10-12 14:19:42 -07:00
ac6aaf3785
encryption: fix some data not being encrypted ( #8403 )
...
Changes:
1. Change `isEncryptionOpSupported` to not check against `clientDBInfo.isEncryptionEnabled`, but instead against ENABLE_ENCRYPTION server knob. The problem with clientDBInfo is before its being broadcast to the workers, its content is uninitialized, during which some data (e.g. item 2) is not getting encrypted when they should.
2. Fix CommitProxy not encrypting metadata mutations which are recovered from txnStateStore
3. Fix KeyValueStoreMemory (thus TxnStateStore) partial transaction coming from recovery is not encrypted
4. new CODE_PROBE for the above fixes
5. Logging changes
2022-10-12 14:18:56 -07:00
a43e424d8a
Merge branch 'main' of https://github.com/apple/foundationdb into validate-data-consistency
2022-10-12 13:01:07 -07:00
ee0027ba76
remove tenantModeRequired flag and disable defaulttenant for cyclerestart test
2022-10-12 12:32:43 -07:00
a3d42c6073
Merge branch 'main' of github.com:apple/foundationdb into tenant-restarting-tests
2022-10-12 11:59:27 -07:00
056f938991
add tenantmode to testconfig and change precedence rules for some tenant options
2022-10-12 11:51:50 -07:00
cf48ae206f
Benchmark adding idempotency id mutations for commit batch
2022-10-12 11:29:09 -07:00
8ca6397e97
Enable Backup Tests for Encrypted Snapshots ( #8419 )
...
* enable encrypted snapshot backup tests
* address pr comments
* remove tenant cache when decoding
* remove client knob
2022-10-12 10:45:30 -07:00
96574bacaa
Change Feed TSS Support ( #8384 )
...
* Change Feed TSS Support
* bug fixing for rare mismatch cases
* Adding rollback handling to tss change feed comparison
2022-10-12 10:23:51 -05:00
858d562fce
Remove MultiToken Encryption auth-token generation
...
Description
The current code supports two modes for authentication encryption:
1. Single auth-token mode
2. Multi auth-token mode
The code currently uses SingleAuthToken mode only, the multi-auth
token mode was added to support large file (such as backup files)
encryption assisting encryption authentication by allowing reading
header independently from encrypted payload itself. However, the
backup files are organized as 'chunks' and every chunk is encrypted,
hence, MultiToken mode isn't used in the code.
Removing the usage saves 32 bytes per encryption header, which might
translate to decent storage saving, further, allows lesser encryption
header overhead when encrypting small Key-Value mutations.
Testing
BlobCipher unittests
EncryptionOps.toml
2022-10-11 22:49:53 -07:00
3223208f0c
Explicitly pass `arena` to the `withPrefix` function.
...
Also update some TraceEvents to be more succinct.
2022-10-11 18:35:01 -07:00
fec05e5bc6
Merge pull request #8450 from sfc-gh-tclinkenbeard/proxy-tag-enforcement
...
Add `GrvProxyTransactionTagThrottler` class
2022-10-11 16:54:49 -07:00
cc09bd3ca2
Assert that transactionsReleased vector is not resized multiple times in GrvProxyTransactionTagThrottler::releaseTransactions
2022-10-11 15:33:51 -07:00
6c62ff3cf8
configdb: Fix ConfigIncrementWorkload ConfigDatabaseUnitTests to use tuple instead of binary reader
2022-10-11 15:32:01 -07:00
ff1b2df8f6
fdbcli: Add options for knob management
...
- setknob <knob_name> <knob_value> [config_class]
- getknob <knob_name> [config_class]
- Added new option to begin to specify if it's a configuration txn. Syntax is begin [config-txn]
- Added utility function for converting tuples to string
- Added knobmanagment test in fdbcli_tests.py
2022-10-11 15:32:01 -07:00
2a791a6212
Addressed review comments
2022-10-11 15:11:26 -07:00
7abe785669
Merge branch 'main' of github.com:apple/foundationdb into getsizetenant
2022-10-11 15:03:42 -07:00
bd55bcd4ff
Update GrvTransactionRateInfo::rate initialization
...
Co-authored-by: Markus Pilman <markus.pilman@snowflake.com>
2022-10-11 14:44:52 -07:00
9e7630869e
Test ids > 16 bytes in workload
2022-10-11 13:46:40 -07:00
e51d2a765c
Make sure ids are unique
2022-10-11 13:46:40 -07:00
aad3899cb7
Enforce the lifetime of a write tx is actually <= MAX_WRITE_TRANSACTION_LIFE_VERSIONS
2022-10-11 13:46:39 -07:00
43a72ad406
Use versionstamped keys instead of atomic add in test
...
This makes it easier to debug if something goes wrong
2022-10-11 13:46:39 -07:00
6ef8d7ec95
Add AutomaticIdempotencyWorkload
2022-10-11 13:46:39 -07:00
711b6a29d3
Add CommitTransactionRequest::idempotencyId
2022-10-11 13:46:39 -07:00
329a2724ff
Add IdempotencyId struct
2022-10-11 13:46:39 -07:00
11658c7135
Merge branch 'main' of github.com:apple/foundationdb into getsizetenant
2022-10-11 13:09:47 -07:00
0466eff56b
Fix potential memory error in GrvTransactionTagThrottler::releaseTransactions
2022-10-11 13:09:21 -07:00
7e2764243b
Fix /GrvTransactionRateInfo/Simple unit test
2022-10-11 13:04:29 -07:00
2c0f936740
Merge branch 'main' of github.com:apple/foundationdb into tenant-restarting-tests
2022-10-11 12:41:03 -07:00
3c7d17e2bb
work in progress commit for defaulttenant in restarting tests
2022-10-11 12:40:36 -07:00
06d9ebd620
Adding domain name to blob metadata requests ( #8415 )
2022-10-11 14:31:14 -05:00
88c7304e04
Resolved comments.
2022-10-11 10:24:31 -07:00
f16ba3991f
add comment
2022-10-11 09:27:05 -07:00
3cc7932953
Merge remote-tracking branch 'origin/main' into proxy-tag-enforcement
2022-10-11 08:51:07 -07:00
b76268544f
disallow workload injection in IDDTxnProcessorApiCorrectness
2022-10-10 22:41:31 -07:00
a4e9e525bd
Apply clang-format to GrvProxyTransactionTagThrottler.actor.cpp
2022-10-10 22:31:09 -07:00
671c4fe0a2
Make GrvProxyTransactionTagThrottler a FIFO queue for unthrottled transactions, even for transactions with different tags
2022-10-10 22:10:08 -07:00
f1cb4e40f5
Avoid adding untagged requests to GrvProxyTransactionTagThrottler
2022-10-10 22:10:08 -07:00
128e8c43da
Include unactorcompiler.h in actor header files
2022-10-10 22:10:08 -07:00
ebe956b228
Add copyright headers to GrvProxyTransactionTagThrottler files
2022-10-10 22:10:08 -07:00
93fe133784
Clean up tags in GrvProxyTransactionTagThrottler::updateRates
2022-10-10 22:10:08 -07:00
24a0dd9f17
Change GrvProxy tag throttling algorithm.
...
The new algorithm assumes there is only one tag per request,
so queues are partitioned by tag. This is a more efficient approach than
the old algorithm.
2022-10-10 22:10:08 -07:00
bbf69b2d0a
Rename TagQueue::runEpoch to TagQueue::releaseTransactions
2022-10-10 22:10:08 -07:00
f74b724c03
Addressed review comments
2022-10-10 22:10:08 -07:00
31e8fb0490
Rename epoch to release window in GrvTransactionRateInfo
...
The term "epoch" was used in too many places
2022-10-10 22:10:08 -07:00
8ff66d6fad
Add comments to TagQueue class
2022-10-10 22:10:08 -07:00
69b3c09cf7
Run 10 clients in /TagQueue/MultiClient
2022-10-10 22:10:08 -07:00
34857bd2bf
Remove outSystemPriority parameter from TagQueue::runEpoch
2022-10-10 22:10:08 -07:00
e313edbb85
Add proxyTagThrottledDuration field to GetReadVersion[Request|Reply]
2022-10-10 22:10:08 -07:00
2684464cc6
Use TagQueue in GRV proxy when ENFORCE_TAG_THROTTLING_ON_PROXIES is true
2022-10-10 22:10:08 -07:00
8405d42aa9
Add /TagQueue/Batch unit test
2022-10-10 22:10:08 -07:00
bffc41fa45
Add /TagQueue/MultiClient unit test
2022-10-10 22:10:08 -07:00
66dc0396dd
Add /TagQueue/MultiTag unit test
2022-10-10 22:10:08 -07:00
a79419788f
Add /TagQueue/Immediate unit test
2022-10-10 22:10:08 -07:00
c29213eec2
Reorder process of queues in TagQueue::runEpoch
2022-10-10 22:10:08 -07:00
0203c93a44
Add TagQueue class
2022-10-10 22:10:08 -07:00
4c973c11ad
Modularize and comment GrvTransactionRateInfo
2022-10-10 22:10:05 -07:00
d74c072382
Merge pull request #8448 from sfc-gh-yiwu/encrypt_disable
...
Add a knob to disable Redwood tenant page split
2022-10-10 21:17:43 -07:00
12aaabb283
Fix bugs in finding correct shards for tenant aware storage metrics.
...
* Account for tenant in during cache invalidation in NativeAPI actor.
* Account for tenant prefix while checking whether a shard is readable
on storage server.
This commit also adds many debug TraceEvents that will later be
removed.
2022-10-10 18:52:03 -07:00
8ed2b85427
Reduce workload flakiness.
...
* Scale up the workload so that the size for the tenant is large enough
so that the estimated size will not be rounded down to 0.
* Retry getting the size in the workload (with a delay, upto a maximum
time) to allow for the size estimate to catch up.
2022-10-10 18:40:31 -07:00
5dc14accfd
Add a knob to disable Redwood tenant page split
2022-10-10 17:02:37 -07:00
595520537a
Extend DBCoreState to store EncryptionAtRest status
...
Description
diff-1: Avoid (de)serialize 'encryptionAtRest' for version < tagLocality
Patch updates DBCoreState to incude `encryptionAtRestMode` stored
in co-ordinators. The change only extends DBCoreState, however,
the value wouldn't get serialzied as the 'CoordinateState' writer
(ReusableCoordiantedState) protocol version is left at DBCoreState.
Approach assists in playing nice with downgrades, the subsequent
release would bump up writer protocol version to support
'EncryptionAtRest' protcol version compatibility.
Testing
devRunCorrectness - 100K
2022-10-10 13:17:30 -07:00
9c937a8e6e
Extend DBCoreState to store EncryptionAtRest status
...
Description
Patch updates DBCoreState to incude `encryptionAtRestMode` stored
in co-ordinators. The change only extends DBCoreState, however,
the value wouldn't get serialzied as the 'CoordinateState' writer
(ReusableCoordiantedState) protocol version is left at DBCoreState.
Approach assists in playing nice with downgrades, the subsequent
release would bump up writer protocol version to support
'EncryptionAtRest' protcol version compatibility.
Testing
devRunCorrectness - 100K
2022-10-10 13:17:30 -07:00
a730e32164
Merge branch 'main' of https://github.com/apple/foundationdb into validate-data-consistency
2022-10-10 13:17:17 -07:00
bc72f9d5fd
Storage server readrange and update latency. ( #8380 )
2022-10-10 12:52:07 -07:00
b930a8a380
Fix bug in GlobalTagThrottlerImpl::getQuotaRatio
2022-10-10 12:51:27 -07:00
4b238e3985
RocksDB disable WAL for experimentation. ( #8443 )
2022-10-10 12:32:05 -07:00
b887b1e85c
Limit number of tags tracked by GlobalTagThrottler
2022-10-10 12:28:26 -07:00
b52edd8658
Merge branch 'main' of https://github.com/apple/foundationdb into validate-data-consistency
2022-10-10 11:00:05 -07:00
1931e3266f
Resolve comments.
2022-10-10 10:28:19 -07:00
63f5705560
Merge pull request #8414 from sfc-gh-xwang/feature/main/txnProcessor_team
...
Replace self->cx with self->dbContext() method and trivial renaming
2022-10-10 10:09:36 -07:00
20864e67de
Merge pull request #8224 from liquid-helium/thread-priority
...
Added knobs for RocksDB reader/writer thread priorities.
2022-10-10 09:38:25 -07:00
6f4e155623
Include unactorcompiler.h in actor header files
2022-10-08 11:59:07 -07:00
8ccdc91b5e
Merge commit '7c89cd705faee52d5d78e6c77665cb7cc4502f58' into redwood-commit-overlap
2022-10-07 15:59:45 -07:00
7c89cd705f
Merge pull request #8402 from halfprice:zhewu/add-satellite-info
...
Add info in HealthMonitorDetectDegradedPeer to indicate whether a peer is in satellite DC
2022-10-07 11:25:55 -07:00
ea1325a552
Merge pull request #8319 from sfc-gh-tclinkenbeard/add-rare-code-probe-annotation
...
Add `rare` code probe decoration
2022-10-07 09:39:00 -06:00
3a15f4cf55
Merge branch 'main' of https://github.com/apple/foundationdb into thread-priority
2022-10-06 20:48:21 -07:00
f5381ca2a8
Format clean
2022-10-06 14:09:00 -07:00
43cd419078
Comments cleanup
2022-10-06 14:09:00 -07:00
c4c71c7806
Add tests
2022-10-06 14:09:00 -07:00
f41d88cd45
Ignore CC_DEGRADED_PEER_DEGREE_TO_EXCLUDE for disconnected peers. Core logic
2022-10-06 14:09:00 -07:00
c1b2519b9c
Remove additional calls to KVS init in TLogServer in favor of a single call at startup.
2022-10-06 13:14:24 -07:00
1030f6c536
Merge commit '14107b315d84b78daf6df6b9541c2168cae10f99' into redwood-commit-overlap
2022-10-06 13:07:32 -07:00
f2dbbcce40
Allow overlapping commits in Redwood in which the caller drops the commit futures. Call IKVS::init() in TLogServer.
2022-10-06 12:48:06 -07:00
d3e5fa6789
Fix heap-use-after-free in TenantAwareEncryptionKeyProvider
2022-10-06 11:00:11 -07:00
f8186988bc
Merge pull request #8378 from sfc-gh-ajbeamon/tenant-support-for-database-switching
...
Add tenant support for database switching
2022-10-05 14:57:33 -07:00
88c37c81d8
Merge branch 'main' of https://github.com/apple/foundationdb into thread-priority
2022-10-05 14:20:53 -07:00
d64d405af7
Validate all storage servers.
2022-10-05 14:19:33 -07:00
e83229ed8b
- Version vector specific: Propagate the latest commit version, as ( #8393 )
...
part of the read request, when reading from a storage server.
2022-10-05 16:23:56 -04:00
354e36fb41
Merge pull request #8400 from sfc-gh-satherton/sqlite-sim-fix-main
...
Sim-only bug fix where SQLite can take too long to shut down
2022-10-05 13:10:45 -07:00
299b1b32e2
create new tracking key for metacluster traces
2022-10-05 12:08:36 -07:00
30902da0d2
trackLatest on recovery event
2022-10-05 11:57:41 -07:00
2fe1d19e95
Merge branch 'main' of github.com:apple/foundationdb into metacluster-status
2022-10-05 11:48:04 -07:00
dc917453c1
Targeted blob granules fault injection ( #8231 )
2022-10-05 13:44:38 -05:00
8592f7e253
Added test check to verify change feeds get cleaned up at the end of blob granule tests ( #8322 )
...
* implemented check, but it doesn't always work
* cleanup
2022-10-05 12:58:05 -05:00
9799329b99
Merge pull request #8390 from sfc-gh-huliu/actor
...
Misc changes for blob manifest dumper
2022-10-05 10:30:56 -07:00
45a31884d5
Address review comments
2022-10-05 09:30:25 -06:00
0361cf74e5
fix RocksDB compilation errors
2022-10-05 09:21:15 -06:00
67076aae4c
Fix compilation errors
2022-10-05 07:52:54 -06:00
1f568fe9e9
Fix compilation errors
2022-10-05 07:48:34 -06:00
a09d6e8b08
Add info in HealthMonitorDetectDegradedPeer to indicate whether a peer is in satellite DC
2022-10-04 23:09:30 -07:00
9a934cf98b
Sim-only bug fix where SQLite can take too long to shut down so its files remain open when the next virtual process starts.
2022-10-04 19:29:38 -07:00
23edfd0d59
Fix formatting
2022-10-04 18:33:30 -06:00
550488b020
Merge remote-tracking branch 'origin/main' into bugfixes/open-for-ide
...
# Conflicts:
# bindings/c/CMakeLists.txt
# fdbclient/include/fdbclient/GetEncryptCipherKeys.actor.h
# fdbserver/BackupWorker.actor.cpp
# fdbserver/BlobWorker.actor.cpp
# fdbserver/CommitProxyServer.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/StorageCache.actor.cpp
# fdbserver/include/fdbserver/GetEncryptCipherKeys.actor.h
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/PhysicalShardMove.actor.cpp
# flow/CMakeLists.txt
2022-10-04 18:27:48 -06:00
16290ca439
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/txnProcessor_team
2022-10-04 16:39:38 -07:00
2ad4f29539
dbContext() replace self->cx, remove cx member
2022-10-04 16:39:22 -07:00
15666851fc
Merge pull request #8335 from sfc-gh-xwang/feature/main/txnProcessor_team
...
Move part of methods in DDTeamCollection to IDDTxnProcessor
2022-10-04 16:02:29 -07:00
97dfc6823f
fixed build with OPEN_FOR_IDE
2022-10-04 17:01:02 -06:00
a565863189
RocksDB compaction knobs and stats. ( #8392 )
2022-10-04 15:13:50 -07:00
21b2e11bc4
getWorkers from IDDTxnProcessor
2022-10-04 14:57:04 -07:00
28e170ca69
remove unnecessary Database cx parameters
2022-10-04 14:29:45 -07:00
5c549601d2
Split Redwood page by tenant boundary ( #7979 )
...
Redwood encrypt with page granularity. To do per-tenant encryption (i.e. each tenant are encrypted with different set of cipher keys), we need to split Redwood pages by tenant boundary. Moreover, it also needs to handle different tenant modes:
* tenantMode = disabled: do not split page by tenant, all data encrypt using default encryption domain.
* tenantMode = required: look at the prefix of the keys and split by tenant accordingly, and encrypt in tenant specific encryption domain accordingly.
* tenantMode = optional: some key ranges may not map to a tenant. In additional to looking at the key prefix, the key provider also query the tenant prefix index. For prefixes not found in the tenant prefix index, corresponding key should be encrypted using the default encryption domain.
The change also enforce data for each tenant forms a subtree, and key of the link to the subtree is exactly the tenant prefix.
This PR is building on top of #8172 and use the IPageEncryptionKeyProvider interface added there. Changes:
* In `writePages` and `splitPages`, query the key provider to split page accordingly.
* In `commitSubtree`, when doing in-place update (to both of leaf or internal page), check if the entry being insert belong to the same encryption domain as existing data of the page. If not, fallback to full page rebuild, where `writePages` will handle the page split.
* When updating the root, check if it is encrypted using non-default (i.e. tenant specific) domain. If so, add a redundant root node which will be encrypted with default encryption domain.
Tested with 100K run of `Lredwood/correctness/btree` unit test, where it uses `RandomEncryptionKeyProvider`, which is updated to support and generate random encryption domain with 4 byte domain prefixes.
2022-10-04 12:53:55 -07:00
50b3a8d3ba
misc changes for manifest dumper
...
1) move dumper to a dedicated actor
2) include blob manager epoch into the manifest name
3) keep multiple manifest files(up to 5)
2022-10-04 10:33:43 -07:00
55b880432e
Merge pull request #8379 from sfc-gh-ajbeamon/literal-string-ref-removal
...
LiteralStringRef removal
2022-10-04 09:41:42 -07:00
4cf4ccc089
correct getServerListAndProcessClasses implementation (100k pass)
2022-10-03 22:24:35 -07:00
53593c77ac
When updating watches after a database switch, lookup the tenant again to get the ID on the new cluster. Update the switching test to set up tenants properly.
2022-10-03 09:24:13 -07:00
0a1d03c3d7
Formatting fixes
2022-10-03 09:23:17 -07:00
76f2dc8ce0
merge upstream/main
2022-10-02 22:07:42 -07:00
fec791be62
Merge remote-tracking branch 'origin/main' into split-failure-injection-workloads
2022-09-30 18:00:48 -07:00
111bf9e34d
Add macros for literal string ref versions of __FUNCTION__ and __FILE__
2022-09-30 16:05:47 -07:00
e1fe28b78b
Switch some usages of LiteralStringRef to use the _sr suffix
2022-09-30 16:04:16 -07:00
442ed0de10
Update TenantEntryCache workload refresh to handle fault injection ( #8369 )
...
* Update TenantEntryCache workload refresh to handle fault injection
Description
diff-1: Add trace event to log wait
Patch update TenantEntryCache::testCacheRefresh to better handle fault
injection causing longer cluster recovery/availability delays. The
test is modified to loop for a threshold time waiting for next round
of tenantCache refresh cycle to happen. Otherwise throws 'timed_out'
error.
Testing
devRunCorrectness - 100K
2022-09-30 15:48:40 -07:00
957ce81a7d
Add per CF stats. ( #8354 )
2022-09-30 15:25:29 -07:00
13f022160c
add trace detail
2022-09-30 15:02:10 -07:00
201eac77cf
Init TLogPersistent storage for a sharedTLog ( #8363 )
...
* Init TLogPersistent storage for a sharedTLog
Description
diff-1: Address review comments
Patch udpates the code to intialize TLogPersistent storage for a
shared TLog independent of intializing persistentState for a
versioned Tlog data. Appraoch allows initializing Tlog persistent
storage as well as writing 'persistFormat' key for a shared TLog
earlier in the TLog creation lifecycle.
Testing
devRunCorrectness - 100K
2022-09-30 14:06:53 -07:00
90ce4053f4
Repair Dead Data Center Outside Quiet Database ( #8359 )
...
* fix snap test
* repair dead data center
2022-09-30 13:59:02 -07:00
1f87286424
Merge pull request #8366 from sfc-gh-huliu/fix7
...
fix killMachine - make sure we have at least 1 blob worker in a dc
2022-09-30 13:56:32 -07:00
9db48eb10c
fix killMachine - make sure we have at least 1 blob worker in a dc
2022-09-30 11:18:27 -07:00
88be74732f
Merge pull request #8365 from sfc-gh-huliu/fix6
...
injected faults should be retryable errors in blob worker
2022-09-30 09:59:06 -07:00
020e28a63f
Disable RocksDB storage engine in unit test TestConfig
2022-09-30 08:59:47 -07:00
5f975623fb
Merge branch 'main' of https://github.com/apple/foundationdb into thread-priority
2022-09-30 08:57:53 -07:00
999809e79b
Increase the data in the workload and add ASSERTs for expected values.
2022-09-29 19:21:44 -07:00
5a3cd51255
Remove temporary TraceEvents and comments from previous commits
2022-09-29 18:53:11 -07:00
c2b6b288b7
Fixes in the GetEstimatedRangeSize workload
...
* Run the workload on a trusted client rather than untrusted
clients. This allows the workload to be substantially simplified
as well as enables testing for the case where no tenant is
present.
* Explicitly pass tenant to BulkSetup so that the setup phase
can be run in parallel for multiple tenants without causing a race.
2022-09-29 18:34:47 -07:00
0264262f78
Explicitly pass tenant to BulkSetup for use in Transactions
...
With this change, BulkSetup accepts Optional<TenantName> as a parameter.
It then uses this while constructing transactions. If a TenantName is
passed to a transaction this way, the transaction can use this tenant
rather than relying on the DatabaseContext. This can be used by
workloads to run the BulkSetup in parallel for multiple tenants without
causing a race.
2022-09-29 18:34:35 -07:00
de90e1aab1
Improve the code in the actors and server, simplify the workload
2022-09-29 18:34:35 -07:00
f63934117d
Make the storage metrics function tenant aware [WIP]
...
This change makes the getEstimatedRangeSizeBytes function tenant aware.
Previously, this function would return the size of the requested
keyspace even if the tenant in the Transaction or DatabaseContext did
not match the tenant corresponding to the keyspace.
Also make some improvements to the new workload.
2022-09-29 18:32:04 -07:00
85ba47f4c8
Add a workload to test getEstimatedRangeSizeBytes funtionality
2022-09-29 18:28:39 -07:00
1b333734a2
set updaterDelay on every loop
2022-09-29 16:53:37 -07:00
01d7668fd1
injected faults should be retryable errors in blob worker
2022-09-29 16:47:31 -07:00
461e42bfe1
restructure updater code and add capacity check in metacluster management workload
2022-09-29 16:24:02 -07:00
03f1d13be3
Enable encryption authentication configurability ( #8312 )
...
* Enable encryption authentication configurability
Description
diff-1: Remove memcpy due to auth-token computation
Address review comments
Patch proposes major changes:
1. Enable FDB to choose encryption authentication as a configurable
parameter. Fix issues choosing ENCRYPT_HEADER_AUTH_TOKEN_NONE mode.
2. Introduce AES_CMAC as supported encryption authentication scheme.
Patch allows cluster to govern: if encryption authentication needs to
enabled, if yes, then choose from two supported schemes:
1. HMAC_SHA_256
2. AES_256_CMAC
Testing
devRunCorrectness - 100K
BlobCipher unittests
EncryptionOps.toml
BlobGranuleCorrectness/BlobGranuleCorrectnessClean
2022-09-29 16:18:55 -07:00
63b8d775a3
Knob for RocksDb behaviors ( #8360 )
...
* Added knob for rocksdb suggest compact.
* Added ROCKSDB_LEVEL_COMPACTION_DYNAMIC_LEVEL_BYTES knob.
Co-authored-by: He Liu <heliu@apple.com>
2022-09-29 15:40:29 -07:00
232e23c87e
Snapshot Backup Encryption ( #8095 )
...
* inital commit
* add encryption support
* refactor passing encryption keys
* disable encryption codepath
* pass enable encryption cli arg for fdbbackup
* change encryption key passing
* only enable encryption for backup if tenant mode is required
* revert server knobs
* ignore json trace files
* --amend
* integrate ekp
* add comments
* comments
* comments
* revert knobs
* fix bug
* modify encrypted backup logic
* use common prefix
* check boundaries between begin and end key
* return system encrypt domain
* modify
* add tenant entry cache to backup
* slight revert
* format
* Trigger Build
* format
* use tenant name to fetch keys
* Trigger Build
* Trigger Build
* merge
* fix getting ekp
* fix bug
* address pr comments
* Trigger Build
* Trigger Build
* remove faulty check
2022-09-29 14:45:47 -07:00
2434f18c5c
Enable failure injection for all simulation tests
2022-09-29 14:14:24 -07:00
a160fc697e
Address comments by passing currentTime by value
2022-09-29 13:55:16 -07:00
fcbd65c2ea
Add unit test
2022-09-29 13:55:16 -07:00
219dde9f41
differentiate degraded peers and disconnected peers in gray failure
2022-09-29 13:55:16 -07:00
93d08b55ba
Merge pull request #8350 from sfc-gh-ajbeamon/fix-metacluster-workload-heap-use-after-free
...
Fix heap use after free in metacluster management workload
2022-09-29 13:38:32 -07:00
5d075d3aff
Fix heap use after free
2022-09-29 12:34:36 -07:00
e26be7a099
Merge branch 'main' of github.com:apple/foundationdb into metacluster-status
2022-09-29 11:37:50 -07:00
257dbff430
recreate transaction every loop in background updater
2022-09-29 11:34:36 -07:00
6357ad1750
pass info through cc data to populate in status
2022-09-28 16:18:44 -07:00
1cc46a98f8
solve merge conflicts
2022-09-28 14:34:42 -07:00
3578832c69
merge upstream/main
2022-09-28 14:01:50 -07:00
c8bb15e8ee
Merge pull request #8216 from sfc-gh-ajbeamon/backup-support-for-tenants
...
Include tenant and metacluster metadata in backups
2022-09-28 12:00:24 -07:00
ef13985feb
Merge pull request #8320 from sfc-gh-tclinkenbeard/read-write-fungible
...
Make read and write quotas fungible
2022-09-28 10:59:37 -07:00
2a640f927c
Merge pull request #8232 from sfc-gh-xwang/feature/main/txnProcessor_tracker
...
Port the DB methods calling in DDShardTracker through DDTxnProcessor interface
2022-09-28 10:33:28 -07:00
251afa3610
Set g_simulator fields for restart tests ( #8318 )
...
* fix tests
* fix tenant concurrency
* Set g_simulator variables for restart tests
* fix machine attrition for restarting
2022-09-28 10:09:26 -07:00
ffe1529f51
Apply clang-format
2022-09-27 19:18:18 -07:00
f2dea33d08
add global var transactionRetryableErrors
2022-09-27 18:14:34 -07:00
3353103d9d
Fix filtering of potential backup mutations in commit proxy and backup worker; add code probe to ensure we are testing default backup sharing and add some tests to hit it
2022-09-27 15:25:30 -07:00
07dae1620c
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/txnProcessor_team
2022-09-27 11:56:02 -07:00
sfc-gh-tclinkenbeard