6251e8be32
Track transaction cost for range reads
2022-10-17 10:08:49 -07:00
6752da5e65
blob/test: check tenant_not_found exception
2022-10-17 11:42:27 -05:00
46ec2f416e
blob/test: convert BlobGranuleRangesWorkload to use a tenant on coin flip
2022-10-17 11:42:27 -05:00
17c24c91aa
Refactor TransactionCostWorkload
2022-10-17 07:06:32 -07:00
d0f09d26e1
Add readLargeValue test to TransactionCostWorkload
2022-10-16 21:58:08 -07:00
445e871cf2
Use fungibility ratio in getWriteOperationCost
2022-10-16 21:58:08 -07:00
84aa815026
Implemented first version of Transaction::getTotalCost, along with workload
2022-10-16 21:58:08 -07:00
e8e6f47e09
format code
2022-10-15 22:52:43 -07:00
eb4d872127
finish MoveKeys test (not run yet)
2022-10-15 22:52:43 -07:00
313c76232c
skip shard size validity
2022-10-15 22:52:43 -07:00
8f5e9255f1
finish testRawMovementApi (not run test yet)
2022-10-15 22:52:43 -07:00
dd9294ac66
add shard status transition validation
2022-10-15 22:52:43 -07:00
83685e6a61
add status verification; add skipDDModeCheck in getInitialDataDistribution for workload test
2022-10-15 22:52:42 -07:00
0f9da9d1ad
Merge branch 'main' of github.com:apple/foundationdb into getsizetenant
2022-10-14 16:50:51 -07:00
a24ab2d4fa
Incorporate code review comments + workload code cleanup
2022-10-14 13:59:36 -07:00
f5f33c5883
Merge pull request #8473 from sfc-gh-mpilman/features/enforce-workload-name
...
Force name and description of workloads to be the same
2022-10-14 12:42:03 -06:00
da5b5a354e
Fix failure injection naming
2022-10-14 11:28:03 -06:00
180238076c
Fix HighContentionPrefixAllocatorWorkload and make untrusted argument to workload factory type safe
2022-10-14 09:51:49 -06:00
fb1390ef3d
reformat code
2022-10-14 08:47:56 -06:00
7cec0a5249
Blob metadata refresh ( #8456 )
...
* Adding EKP refresh of blob metadata
* Adding re-fetching blob metadata from BGTenantMap
* adding buggifies from code review comments
2022-10-14 08:17:50 -05:00
49f0cf5ab0
Force name and description of workloads to be the same
2022-10-13 20:53:48 -06:00
854212fe94
Incorportate code review suggestions
2022-10-13 17:41:31 -07:00
169c341f79
Merge pull request #8386 from sfc-gh-huliu/blobmigrator
...
Add blob migrator to assist data copy from blob to storage server
2022-10-13 14:46:04 -07:00
be64b12ecf
Clean up TraceEvents, TODOs, and simplify workload.
2022-10-13 14:31:42 -07:00
049df622f1
add a blob migrator
2022-10-13 13:21:45 -07:00
df5825ff65
Merge pull request #8398 from sfc-gh-anoyes/anoyes/idempotency-id2
...
Initial work for automatic idempotency
2022-10-13 13:07:14 -07:00
e4752309a1
Merge pull request #8075 from liquid-helium/validate-data-consistency
...
Validate data consistency
2022-10-13 09:20:01 -07:00
1bfe00ca61
Merge pull request #8449 from sfc-gh-ahusain/ahusain-auth-token-cleanup
...
Remove MultiToken Encryption auth-token generation
2022-10-12 17:51:42 -07:00
395ac11416
Merge pull request #8463 from sfc-gh-mpilman/bugfixes/different-cluster-same-rv-race
...
Fix DifferentClustersSameRV workload
2022-10-12 17:36:33 -06:00
97acc94a7f
Fixed out-of-scope variable issue.
2022-10-12 16:13:57 -07:00
9997ebba64
Fix DifferentClustersSameRV workload
2022-10-12 16:14:38 -06:00
a43e424d8a
Merge branch 'main' of https://github.com/apple/foundationdb into validate-data-consistency
2022-10-12 13:01:07 -07:00
a3d42c6073
Merge branch 'main' of github.com:apple/foundationdb into tenant-restarting-tests
2022-10-12 11:59:27 -07:00
056f938991
add tenantmode to testconfig and change precedence rules for some tenant options
2022-10-12 11:51:50 -07:00
8ca6397e97
Enable Backup Tests for Encrypted Snapshots ( #8419 )
...
* enable encrypted snapshot backup tests
* address pr comments
* remove tenant cache when decoding
* remove client knob
2022-10-12 10:45:30 -07:00
858d562fce
Remove MultiToken Encryption auth-token generation
...
Description
The current code supports two modes for authentication encryption:
1. Single auth-token mode
2. Multi auth-token mode
The code currently uses SingleAuthToken mode only, the multi-auth
token mode was added to support large file (such as backup files)
encryption assisting encryption authentication by allowing reading
header independently from encrypted payload itself. However, the
backup files are organized as 'chunks' and every chunk is encrypted,
hence, MultiToken mode isn't used in the code.
Removing the usage saves 32 bytes per encryption header, which might
translate to decent storage saving, further, allows lesser encryption
header overhead when encrypting small Key-Value mutations.
Testing
BlobCipher unittests
EncryptionOps.toml
2022-10-11 22:49:53 -07:00
6c62ff3cf8
configdb: Fix ConfigIncrementWorkload ConfigDatabaseUnitTests to use tuple instead of binary reader
2022-10-11 15:32:01 -07:00
7abe785669
Merge branch 'main' of github.com:apple/foundationdb into getsizetenant
2022-10-11 15:03:42 -07:00
9e7630869e
Test ids > 16 bytes in workload
2022-10-11 13:46:40 -07:00
e51d2a765c
Make sure ids are unique
2022-10-11 13:46:40 -07:00
43a72ad406
Use versionstamped keys instead of atomic add in test
...
This makes it easier to debug if something goes wrong
2022-10-11 13:46:39 -07:00
6ef8d7ec95
Add AutomaticIdempotencyWorkload
2022-10-11 13:46:39 -07:00
329a2724ff
Add IdempotencyId struct
2022-10-11 13:46:39 -07:00
11658c7135
Merge branch 'main' of github.com:apple/foundationdb into getsizetenant
2022-10-11 13:09:47 -07:00
2c0f936740
Merge branch 'main' of github.com:apple/foundationdb into tenant-restarting-tests
2022-10-11 12:41:03 -07:00
3c7d17e2bb
work in progress commit for defaulttenant in restarting tests
2022-10-11 12:40:36 -07:00
06d9ebd620
Adding domain name to blob metadata requests ( #8415 )
2022-10-11 14:31:14 -05:00
f16ba3991f
add comment
2022-10-11 09:27:05 -07:00
b76268544f
disallow workload injection in IDDTxnProcessorApiCorrectness
2022-10-10 22:41:31 -07:00
8ed2b85427
Reduce workload flakiness.
...
* Scale up the workload so that the size for the tenant is large enough
so that the estimated size will not be rounded down to 0.
* Retry getting the size in the workload (with a delay, upto a maximum
time) to allow for the size estimate to catch up.
2022-10-10 18:40:31 -07:00
b52edd8658
Merge branch 'main' of https://github.com/apple/foundationdb into validate-data-consistency
2022-10-10 11:00:05 -07:00
1931e3266f
Resolve comments.
2022-10-10 10:28:19 -07:00
b7ce834d28
Merge commit '7c89cd705faee52d5d78e6c77665cb7cc4502f58' into storageserver-pml
2022-10-07 11:39:42 -07:00
ea1325a552
Merge pull request #8319 from sfc-gh-tclinkenbeard/add-rare-code-probe-annotation
...
Add `rare` code probe decoration
2022-10-07 09:39:00 -06:00
3228afefd3
Unrevert #7578 - storage server PriorityMultiLock and PML rewrite.
2022-10-06 23:41:28 -07:00
f8186988bc
Merge pull request #8378 from sfc-gh-ajbeamon/tenant-support-for-database-switching
...
Add tenant support for database switching
2022-10-05 14:57:33 -07:00
e83229ed8b
- Version vector specific: Propagate the latest commit version, as ( #8393 )
...
part of the read request, when reading from a storage server.
2022-10-05 16:23:56 -04:00
2fe1d19e95
Merge branch 'main' of github.com:apple/foundationdb into metacluster-status
2022-10-05 11:48:04 -07:00
8592f7e253
Added test check to verify change feeds get cleaned up at the end of blob granule tests ( #8322 )
...
* implemented check, but it doesn't always work
* cleanup
2022-10-05 12:58:05 -05:00
45a31884d5
Address review comments
2022-10-05 09:30:25 -06:00
67076aae4c
Fix compilation errors
2022-10-05 07:52:54 -06:00
550488b020
Merge remote-tracking branch 'origin/main' into bugfixes/open-for-ide
...
# Conflicts:
# bindings/c/CMakeLists.txt
# fdbclient/include/fdbclient/GetEncryptCipherKeys.actor.h
# fdbserver/BackupWorker.actor.cpp
# fdbserver/BlobWorker.actor.cpp
# fdbserver/CommitProxyServer.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/StorageCache.actor.cpp
# fdbserver/include/fdbserver/GetEncryptCipherKeys.actor.h
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/PhysicalShardMove.actor.cpp
# flow/CMakeLists.txt
2022-10-04 18:27:48 -06:00
97dfc6823f
fixed build with OPEN_FOR_IDE
2022-10-04 17:01:02 -06:00
55b880432e
Merge pull request #8379 from sfc-gh-ajbeamon/literal-string-ref-removal
...
LiteralStringRef removal
2022-10-04 09:41:42 -07:00
53593c77ac
When updating watches after a database switch, lookup the tenant again to get the ID on the new cluster. Update the switching test to set up tenants properly.
2022-10-03 09:24:13 -07:00
fec791be62
Merge remote-tracking branch 'origin/main' into split-failure-injection-workloads
2022-09-30 18:00:48 -07:00
e1fe28b78b
Switch some usages of LiteralStringRef to use the _sr suffix
2022-09-30 16:04:16 -07:00
442ed0de10
Update TenantEntryCache workload refresh to handle fault injection ( #8369 )
...
* Update TenantEntryCache workload refresh to handle fault injection
Description
diff-1: Add trace event to log wait
Patch update TenantEntryCache::testCacheRefresh to better handle fault
injection causing longer cluster recovery/availability delays. The
test is modified to loop for a threshold time waiting for next round
of tenantCache refresh cycle to happen. Otherwise throws 'timed_out'
error.
Testing
devRunCorrectness - 100K
2022-09-30 15:48:40 -07:00
90ce4053f4
Repair Dead Data Center Outside Quiet Database ( #8359 )
...
* fix snap test
* repair dead data center
2022-09-30 13:59:02 -07:00
999809e79b
Increase the data in the workload and add ASSERTs for expected values.
2022-09-29 19:21:44 -07:00
5a3cd51255
Remove temporary TraceEvents and comments from previous commits
2022-09-29 18:53:11 -07:00
c2b6b288b7
Fixes in the GetEstimatedRangeSize workload
...
* Run the workload on a trusted client rather than untrusted
clients. This allows the workload to be substantially simplified
as well as enables testing for the case where no tenant is
present.
* Explicitly pass tenant to BulkSetup so that the setup phase
can be run in parallel for multiple tenants without causing a race.
2022-09-29 18:34:47 -07:00
de90e1aab1
Improve the code in the actors and server, simplify the workload
2022-09-29 18:34:35 -07:00
f63934117d
Make the storage metrics function tenant aware [WIP]
...
This change makes the getEstimatedRangeSizeBytes function tenant aware.
Previously, this function would return the size of the requested
keyspace even if the tenant in the Transaction or DatabaseContext did
not match the tenant corresponding to the keyspace.
Also make some improvements to the new workload.
2022-09-29 18:32:04 -07:00
85ba47f4c8
Add a workload to test getEstimatedRangeSizeBytes funtionality
2022-09-29 18:28:39 -07:00
461e42bfe1
restructure updater code and add capacity check in metacluster management workload
2022-09-29 16:24:02 -07:00
03f1d13be3
Enable encryption authentication configurability ( #8312 )
...
* Enable encryption authentication configurability
Description
diff-1: Remove memcpy due to auth-token computation
Address review comments
Patch proposes major changes:
1. Enable FDB to choose encryption authentication as a configurable
parameter. Fix issues choosing ENCRYPT_HEADER_AUTH_TOKEN_NONE mode.
2. Introduce AES_CMAC as supported encryption authentication scheme.
Patch allows cluster to govern: if encryption authentication needs to
enabled, if yes, then choose from two supported schemes:
1. HMAC_SHA_256
2. AES_256_CMAC
Testing
devRunCorrectness - 100K
BlobCipher unittests
EncryptionOps.toml
BlobGranuleCorrectness/BlobGranuleCorrectnessClean
2022-09-29 16:18:55 -07:00
232e23c87e
Snapshot Backup Encryption ( #8095 )
...
* inital commit
* add encryption support
* refactor passing encryption keys
* disable encryption codepath
* pass enable encryption cli arg for fdbbackup
* change encryption key passing
* only enable encryption for backup if tenant mode is required
* revert server knobs
* ignore json trace files
* --amend
* integrate ekp
* add comments
* comments
* comments
* revert knobs
* fix bug
* modify encrypted backup logic
* use common prefix
* check boundaries between begin and end key
* return system encrypt domain
* modify
* add tenant entry cache to backup
* slight revert
* format
* Trigger Build
* format
* use tenant name to fetch keys
* Trigger Build
* Trigger Build
* merge
* fix getting ekp
* fix bug
* address pr comments
* Trigger Build
* Trigger Build
* remove faulty check
2022-09-29 14:45:47 -07:00
2434f18c5c
Enable failure injection for all simulation tests
2022-09-29 14:14:24 -07:00
93d08b55ba
Merge pull request #8350 from sfc-gh-ajbeamon/fix-metacluster-workload-heap-use-after-free
...
Fix heap use after free in metacluster management workload
2022-09-29 13:38:32 -07:00
5d075d3aff
Fix heap use after free
2022-09-29 12:34:36 -07:00
3578832c69
merge upstream/main
2022-09-28 14:01:50 -07:00
c8bb15e8ee
Merge pull request #8216 from sfc-gh-ajbeamon/backup-support-for-tenants
...
Include tenant and metacluster metadata in backups
2022-09-28 12:00:24 -07:00
ef13985feb
Merge pull request #8320 from sfc-gh-tclinkenbeard/read-write-fungible
...
Make read and write quotas fungible
2022-09-28 10:59:37 -07:00
251afa3610
Set g_simulator fields for restart tests ( #8318 )
...
* fix tests
* fix tenant concurrency
* Set g_simulator variables for restart tests
* fix machine attrition for restarting
2022-09-28 10:09:26 -07:00
3353103d9d
Fix filtering of potential backup mutations in commit proxy and backup worker; add code probe to ensure we are testing default backup sharing and add some tests to hit it
2022-09-27 15:25:30 -07:00
3428fe467a
Merge branch 'main' of https://github.com/apple/foundationdb into feature/dd-refactor-simple
2022-09-27 11:55:35 -07:00
e476e814b2
merge upstream/main
2022-09-27 10:15:26 -07:00
ba8fbc9573
Merge remote-tracking branch 'origin/main' into read-write-fungible
2022-09-26 23:26:14 -07:00
1b8512e949
Merge pull request #8314 from sfc-gh-ajbeamon/metacluster-management-workload-expansion
...
Improve Metacluster Management Workload
2022-09-26 15:51:28 -07:00
b0c3534463
Handle invalid tenant configuration when a group and assigned cluster are specified for a tenant and the assigned cluster doesn't match the group's cluster
2022-09-26 14:32:26 -07:00
248c82eb3f
Merge pull request #8229 from sfc-gh-tclinkenbeard/guard-gsimulator
...
Guard a few uses of gsimulator.
2022-09-26 13:52:42 -07:00
05615107da
Minor refactoring to remove throw internal_error lines
2022-09-26 13:14:24 -07:00
093db25d11
Add tenant group support to the tenant management workload and test configure and rename tenant operations. Fix a few error types picked up by the test.
2022-09-26 13:14:24 -07:00
1d00ca3261
address code review comments
2022-09-26 10:37:58 -07:00
ce0ad38c94
Merge remote-tracking branch 'origin/main' into guard-gsimulator
2022-09-26 10:18:43 -07:00
76543cbd8a
remove debug trace
2022-09-26 10:02:16 -07:00
2d1a6b4434
Merge branch 'main' of github.com:apple/foundationdb into metacluster-assigned-cluster
2022-09-26 09:59:20 -07:00
6b2c932e75
modify tests to work with error case of changing preferred cluster on retries
2022-09-26 09:59:06 -07:00
86c27754fb
Fixed a bug where rollbackVersion >= data->storageVersion() ( #8290 )
...
* fix: the log router could see popped() versions between recoveredAt and the recovery txn version which cause remote tlogs to be stopped at a version in this range, which causes storage servers to read at versions in this range
* fix: a spurious correctness bug resulted from the consistency check running during a recovery
2022-09-26 09:19:53 -07:00
sfc-gh-tclinkenbeard