6c62ff3cf8
configdb: Fix ConfigIncrementWorkload ConfigDatabaseUnitTests to use tuple instead of binary reader
2022-10-11 15:32:01 -07:00
ff1b2df8f6
fdbcli: Add options for knob management
...
- setknob <knob_name> <knob_value> [config_class]
- getknob <knob_name> [config_class]
- Added new option to begin to specify if it's a configuration txn. Syntax is begin [config-txn]
- Added utility function for converting tuples to string
- Added knobmanagment test in fdbcli_tests.py
2022-10-11 15:32:01 -07:00
2a791a6212
Addressed review comments
2022-10-11 15:11:26 -07:00
7abe785669
Merge branch 'main' of github.com:apple/foundationdb into getsizetenant
2022-10-11 15:03:42 -07:00
bd55bcd4ff
Update GrvTransactionRateInfo::rate initialization
...
Co-authored-by: Markus Pilman <markus.pilman@snowflake.com>
2022-10-11 14:44:52 -07:00
9e7630869e
Test ids > 16 bytes in workload
2022-10-11 13:46:40 -07:00
e51d2a765c
Make sure ids are unique
2022-10-11 13:46:40 -07:00
aad3899cb7
Enforce the lifetime of a write tx is actually <= MAX_WRITE_TRANSACTION_LIFE_VERSIONS
2022-10-11 13:46:39 -07:00
43a72ad406
Use versionstamped keys instead of atomic add in test
...
This makes it easier to debug if something goes wrong
2022-10-11 13:46:39 -07:00
6ef8d7ec95
Add AutomaticIdempotencyWorkload
2022-10-11 13:46:39 -07:00
711b6a29d3
Add CommitTransactionRequest::idempotencyId
2022-10-11 13:46:39 -07:00
329a2724ff
Add IdempotencyId struct
2022-10-11 13:46:39 -07:00
11658c7135
Merge branch 'main' of github.com:apple/foundationdb into getsizetenant
2022-10-11 13:09:47 -07:00
0466eff56b
Fix potential memory error in GrvTransactionTagThrottler::releaseTransactions
2022-10-11 13:09:21 -07:00
7e2764243b
Fix /GrvTransactionRateInfo/Simple unit test
2022-10-11 13:04:29 -07:00
2c0f936740
Merge branch 'main' of github.com:apple/foundationdb into tenant-restarting-tests
2022-10-11 12:41:03 -07:00
3c7d17e2bb
work in progress commit for defaulttenant in restarting tests
2022-10-11 12:40:36 -07:00
06d9ebd620
Adding domain name to blob metadata requests ( #8415 )
2022-10-11 14:31:14 -05:00
88c7304e04
Resolved comments.
2022-10-11 10:24:31 -07:00
f16ba3991f
add comment
2022-10-11 09:27:05 -07:00
3cc7932953
Merge remote-tracking branch 'origin/main' into proxy-tag-enforcement
2022-10-11 08:51:07 -07:00
b76268544f
disallow workload injection in IDDTxnProcessorApiCorrectness
2022-10-10 22:41:31 -07:00
a4e9e525bd
Apply clang-format to GrvProxyTransactionTagThrottler.actor.cpp
2022-10-10 22:31:09 -07:00
671c4fe0a2
Make GrvProxyTransactionTagThrottler a FIFO queue for unthrottled transactions, even for transactions with different tags
2022-10-10 22:10:08 -07:00
f1cb4e40f5
Avoid adding untagged requests to GrvProxyTransactionTagThrottler
2022-10-10 22:10:08 -07:00
128e8c43da
Include unactorcompiler.h in actor header files
2022-10-10 22:10:08 -07:00
ebe956b228
Add copyright headers to GrvProxyTransactionTagThrottler files
2022-10-10 22:10:08 -07:00
93fe133784
Clean up tags in GrvProxyTransactionTagThrottler::updateRates
2022-10-10 22:10:08 -07:00
24a0dd9f17
Change GrvProxy tag throttling algorithm.
...
The new algorithm assumes there is only one tag per request,
so queues are partitioned by tag. This is a more efficient approach than
the old algorithm.
2022-10-10 22:10:08 -07:00
bbf69b2d0a
Rename TagQueue::runEpoch to TagQueue::releaseTransactions
2022-10-10 22:10:08 -07:00
f74b724c03
Addressed review comments
2022-10-10 22:10:08 -07:00
31e8fb0490
Rename epoch to release window in GrvTransactionRateInfo
...
The term "epoch" was used in too many places
2022-10-10 22:10:08 -07:00
8ff66d6fad
Add comments to TagQueue class
2022-10-10 22:10:08 -07:00
69b3c09cf7
Run 10 clients in /TagQueue/MultiClient
2022-10-10 22:10:08 -07:00
34857bd2bf
Remove outSystemPriority parameter from TagQueue::runEpoch
2022-10-10 22:10:08 -07:00
e313edbb85
Add proxyTagThrottledDuration field to GetReadVersion[Request|Reply]
2022-10-10 22:10:08 -07:00
2684464cc6
Use TagQueue in GRV proxy when ENFORCE_TAG_THROTTLING_ON_PROXIES is true
2022-10-10 22:10:08 -07:00
8405d42aa9
Add /TagQueue/Batch unit test
2022-10-10 22:10:08 -07:00
bffc41fa45
Add /TagQueue/MultiClient unit test
2022-10-10 22:10:08 -07:00
66dc0396dd
Add /TagQueue/MultiTag unit test
2022-10-10 22:10:08 -07:00
a79419788f
Add /TagQueue/Immediate unit test
2022-10-10 22:10:08 -07:00
c29213eec2
Reorder process of queues in TagQueue::runEpoch
2022-10-10 22:10:08 -07:00
0203c93a44
Add TagQueue class
2022-10-10 22:10:08 -07:00
4c973c11ad
Modularize and comment GrvTransactionRateInfo
2022-10-10 22:10:05 -07:00
d74c072382
Merge pull request #8448 from sfc-gh-yiwu/encrypt_disable
...
Add a knob to disable Redwood tenant page split
2022-10-10 21:17:43 -07:00
12aaabb283
Fix bugs in finding correct shards for tenant aware storage metrics.
...
* Account for tenant in during cache invalidation in NativeAPI actor.
* Account for tenant prefix while checking whether a shard is readable
on storage server.
This commit also adds many debug TraceEvents that will later be
removed.
2022-10-10 18:52:03 -07:00
8ed2b85427
Reduce workload flakiness.
...
* Scale up the workload so that the size for the tenant is large enough
so that the estimated size will not be rounded down to 0.
* Retry getting the size in the workload (with a delay, upto a maximum
time) to allow for the size estimate to catch up.
2022-10-10 18:40:31 -07:00
5dc14accfd
Add a knob to disable Redwood tenant page split
2022-10-10 17:02:37 -07:00
595520537a
Extend DBCoreState to store EncryptionAtRest status
...
Description
diff-1: Avoid (de)serialize 'encryptionAtRest' for version < tagLocality
Patch updates DBCoreState to incude `encryptionAtRestMode` stored
in co-ordinators. The change only extends DBCoreState, however,
the value wouldn't get serialzied as the 'CoordinateState' writer
(ReusableCoordiantedState) protocol version is left at DBCoreState.
Approach assists in playing nice with downgrades, the subsequent
release would bump up writer protocol version to support
'EncryptionAtRest' protcol version compatibility.
Testing
devRunCorrectness - 100K
2022-10-10 13:17:30 -07:00
9c937a8e6e
Extend DBCoreState to store EncryptionAtRest status
...
Description
Patch updates DBCoreState to incude `encryptionAtRestMode` stored
in co-ordinators. The change only extends DBCoreState, however,
the value wouldn't get serialzied as the 'CoordinateState' writer
(ReusableCoordiantedState) protocol version is left at DBCoreState.
Approach assists in playing nice with downgrades, the subsequent
release would bump up writer protocol version to support
'EncryptionAtRest' protcol version compatibility.
Testing
devRunCorrectness - 100K
2022-10-10 13:17:30 -07:00
a730e32164
Merge branch 'main' of https://github.com/apple/foundationdb into validate-data-consistency
2022-10-10 13:17:17 -07:00
bc72f9d5fd
Storage server readrange and update latency. ( #8380 )
2022-10-10 12:52:07 -07:00
b930a8a380
Fix bug in GlobalTagThrottlerImpl::getQuotaRatio
2022-10-10 12:51:27 -07:00
4b238e3985
RocksDB disable WAL for experimentation. ( #8443 )
2022-10-10 12:32:05 -07:00
b887b1e85c
Limit number of tags tracked by GlobalTagThrottler
2022-10-10 12:28:26 -07:00
b52edd8658
Merge branch 'main' of https://github.com/apple/foundationdb into validate-data-consistency
2022-10-10 11:00:05 -07:00
1931e3266f
Resolve comments.
2022-10-10 10:28:19 -07:00
63f5705560
Merge pull request #8414 from sfc-gh-xwang/feature/main/txnProcessor_team
...
Replace self->cx with self->dbContext() method and trivial renaming
2022-10-10 10:09:36 -07:00
20864e67de
Merge pull request #8224 from liquid-helium/thread-priority
...
Added knobs for RocksDB reader/writer thread priorities.
2022-10-10 09:38:25 -07:00
6f4e155623
Include unactorcompiler.h in actor header files
2022-10-08 11:59:07 -07:00
8ccdc91b5e
Merge commit '7c89cd705faee52d5d78e6c77665cb7cc4502f58' into redwood-commit-overlap
2022-10-07 15:59:45 -07:00
7c89cd705f
Merge pull request #8402 from halfprice:zhewu/add-satellite-info
...
Add info in HealthMonitorDetectDegradedPeer to indicate whether a peer is in satellite DC
2022-10-07 11:25:55 -07:00
ea1325a552
Merge pull request #8319 from sfc-gh-tclinkenbeard/add-rare-code-probe-annotation
...
Add `rare` code probe decoration
2022-10-07 09:39:00 -06:00
3a15f4cf55
Merge branch 'main' of https://github.com/apple/foundationdb into thread-priority
2022-10-06 20:48:21 -07:00
f5381ca2a8
Format clean
2022-10-06 14:09:00 -07:00
43cd419078
Comments cleanup
2022-10-06 14:09:00 -07:00
c4c71c7806
Add tests
2022-10-06 14:09:00 -07:00
f41d88cd45
Ignore CC_DEGRADED_PEER_DEGREE_TO_EXCLUDE for disconnected peers. Core logic
2022-10-06 14:09:00 -07:00
c1b2519b9c
Remove additional calls to KVS init in TLogServer in favor of a single call at startup.
2022-10-06 13:14:24 -07:00
1030f6c536
Merge commit '14107b315d84b78daf6df6b9541c2168cae10f99' into redwood-commit-overlap
2022-10-06 13:07:32 -07:00
f2dbbcce40
Allow overlapping commits in Redwood in which the caller drops the commit futures. Call IKVS::init() in TLogServer.
2022-10-06 12:48:06 -07:00
d3e5fa6789
Fix heap-use-after-free in TenantAwareEncryptionKeyProvider
2022-10-06 11:00:11 -07:00
f8186988bc
Merge pull request #8378 from sfc-gh-ajbeamon/tenant-support-for-database-switching
...
Add tenant support for database switching
2022-10-05 14:57:33 -07:00
88c37c81d8
Merge branch 'main' of https://github.com/apple/foundationdb into thread-priority
2022-10-05 14:20:53 -07:00
d64d405af7
Validate all storage servers.
2022-10-05 14:19:33 -07:00
e83229ed8b
- Version vector specific: Propagate the latest commit version, as ( #8393 )
...
part of the read request, when reading from a storage server.
2022-10-05 16:23:56 -04:00
354e36fb41
Merge pull request #8400 from sfc-gh-satherton/sqlite-sim-fix-main
...
Sim-only bug fix where SQLite can take too long to shut down
2022-10-05 13:10:45 -07:00
299b1b32e2
create new tracking key for metacluster traces
2022-10-05 12:08:36 -07:00
30902da0d2
trackLatest on recovery event
2022-10-05 11:57:41 -07:00
2fe1d19e95
Merge branch 'main' of github.com:apple/foundationdb into metacluster-status
2022-10-05 11:48:04 -07:00
dc917453c1
Targeted blob granules fault injection ( #8231 )
2022-10-05 13:44:38 -05:00
8592f7e253
Added test check to verify change feeds get cleaned up at the end of blob granule tests ( #8322 )
...
* implemented check, but it doesn't always work
* cleanup
2022-10-05 12:58:05 -05:00
9799329b99
Merge pull request #8390 from sfc-gh-huliu/actor
...
Misc changes for blob manifest dumper
2022-10-05 10:30:56 -07:00
45a31884d5
Address review comments
2022-10-05 09:30:25 -06:00
0361cf74e5
fix RocksDB compilation errors
2022-10-05 09:21:15 -06:00
67076aae4c
Fix compilation errors
2022-10-05 07:52:54 -06:00
1f568fe9e9
Fix compilation errors
2022-10-05 07:48:34 -06:00
a09d6e8b08
Add info in HealthMonitorDetectDegradedPeer to indicate whether a peer is in satellite DC
2022-10-04 23:09:30 -07:00
9a934cf98b
Sim-only bug fix where SQLite can take too long to shut down so its files remain open when the next virtual process starts.
2022-10-04 19:29:38 -07:00
23edfd0d59
Fix formatting
2022-10-04 18:33:30 -06:00
550488b020
Merge remote-tracking branch 'origin/main' into bugfixes/open-for-ide
...
# Conflicts:
# bindings/c/CMakeLists.txt
# fdbclient/include/fdbclient/GetEncryptCipherKeys.actor.h
# fdbserver/BackupWorker.actor.cpp
# fdbserver/BlobWorker.actor.cpp
# fdbserver/CommitProxyServer.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/StorageCache.actor.cpp
# fdbserver/include/fdbserver/GetEncryptCipherKeys.actor.h
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/PhysicalShardMove.actor.cpp
# flow/CMakeLists.txt
2022-10-04 18:27:48 -06:00
16290ca439
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/txnProcessor_team
2022-10-04 16:39:38 -07:00
2ad4f29539
dbContext() replace self->cx, remove cx member
2022-10-04 16:39:22 -07:00
15666851fc
Merge pull request #8335 from sfc-gh-xwang/feature/main/txnProcessor_team
...
Move part of methods in DDTeamCollection to IDDTxnProcessor
2022-10-04 16:02:29 -07:00
97dfc6823f
fixed build with OPEN_FOR_IDE
2022-10-04 17:01:02 -06:00
a565863189
RocksDB compaction knobs and stats. ( #8392 )
2022-10-04 15:13:50 -07:00
21b2e11bc4
getWorkers from IDDTxnProcessor
2022-10-04 14:57:04 -07:00
28e170ca69
remove unnecessary Database cx parameters
2022-10-04 14:29:45 -07:00
5c549601d2
Split Redwood page by tenant boundary ( #7979 )
...
Redwood encrypt with page granularity. To do per-tenant encryption (i.e. each tenant are encrypted with different set of cipher keys), we need to split Redwood pages by tenant boundary. Moreover, it also needs to handle different tenant modes:
* tenantMode = disabled: do not split page by tenant, all data encrypt using default encryption domain.
* tenantMode = required: look at the prefix of the keys and split by tenant accordingly, and encrypt in tenant specific encryption domain accordingly.
* tenantMode = optional: some key ranges may not map to a tenant. In additional to looking at the key prefix, the key provider also query the tenant prefix index. For prefixes not found in the tenant prefix index, corresponding key should be encrypted using the default encryption domain.
The change also enforce data for each tenant forms a subtree, and key of the link to the subtree is exactly the tenant prefix.
This PR is building on top of #8172 and use the IPageEncryptionKeyProvider interface added there. Changes:
* In `writePages` and `splitPages`, query the key provider to split page accordingly.
* In `commitSubtree`, when doing in-place update (to both of leaf or internal page), check if the entry being insert belong to the same encryption domain as existing data of the page. If not, fallback to full page rebuild, where `writePages` will handle the page split.
* When updating the root, check if it is encrypted using non-default (i.e. tenant specific) domain. If so, add a redundant root node which will be encrypted with default encryption domain.
Tested with 100K run of `Lredwood/correctness/btree` unit test, where it uses `RandomEncryptionKeyProvider`, which is updated to support and generate random encryption domain with 4 byte domain prefixes.
2022-10-04 12:53:55 -07:00
50b3a8d3ba
misc changes for manifest dumper
...
1) move dumper to a dedicated actor
2) include blob manager epoch into the manifest name
3) keep multiple manifest files(up to 5)
2022-10-04 10:33:43 -07:00
55b880432e
Merge pull request #8379 from sfc-gh-ajbeamon/literal-string-ref-removal
...
LiteralStringRef removal
2022-10-04 09:41:42 -07:00
4cf4ccc089
correct getServerListAndProcessClasses implementation (100k pass)
2022-10-03 22:24:35 -07:00
53593c77ac
When updating watches after a database switch, lookup the tenant again to get the ID on the new cluster. Update the switching test to set up tenants properly.
2022-10-03 09:24:13 -07:00
0a1d03c3d7
Formatting fixes
2022-10-03 09:23:17 -07:00
76f2dc8ce0
merge upstream/main
2022-10-02 22:07:42 -07:00
fec791be62
Merge remote-tracking branch 'origin/main' into split-failure-injection-workloads
2022-09-30 18:00:48 -07:00
111bf9e34d
Add macros for literal string ref versions of __FUNCTION__ and __FILE__
2022-09-30 16:05:47 -07:00
e1fe28b78b
Switch some usages of LiteralStringRef to use the _sr suffix
2022-09-30 16:04:16 -07:00
442ed0de10
Update TenantEntryCache workload refresh to handle fault injection ( #8369 )
...
* Update TenantEntryCache workload refresh to handle fault injection
Description
diff-1: Add trace event to log wait
Patch update TenantEntryCache::testCacheRefresh to better handle fault
injection causing longer cluster recovery/availability delays. The
test is modified to loop for a threshold time waiting for next round
of tenantCache refresh cycle to happen. Otherwise throws 'timed_out'
error.
Testing
devRunCorrectness - 100K
2022-09-30 15:48:40 -07:00
957ce81a7d
Add per CF stats. ( #8354 )
2022-09-30 15:25:29 -07:00
13f022160c
add trace detail
2022-09-30 15:02:10 -07:00
201eac77cf
Init TLogPersistent storage for a sharedTLog ( #8363 )
...
* Init TLogPersistent storage for a sharedTLog
Description
diff-1: Address review comments
Patch udpates the code to intialize TLogPersistent storage for a
shared TLog independent of intializing persistentState for a
versioned Tlog data. Appraoch allows initializing Tlog persistent
storage as well as writing 'persistFormat' key for a shared TLog
earlier in the TLog creation lifecycle.
Testing
devRunCorrectness - 100K
2022-09-30 14:06:53 -07:00
90ce4053f4
Repair Dead Data Center Outside Quiet Database ( #8359 )
...
* fix snap test
* repair dead data center
2022-09-30 13:59:02 -07:00
1f87286424
Merge pull request #8366 from sfc-gh-huliu/fix7
...
fix killMachine - make sure we have at least 1 blob worker in a dc
2022-09-30 13:56:32 -07:00
9db48eb10c
fix killMachine - make sure we have at least 1 blob worker in a dc
2022-09-30 11:18:27 -07:00
88be74732f
Merge pull request #8365 from sfc-gh-huliu/fix6
...
injected faults should be retryable errors in blob worker
2022-09-30 09:59:06 -07:00
020e28a63f
Disable RocksDB storage engine in unit test TestConfig
2022-09-30 08:59:47 -07:00
5f975623fb
Merge branch 'main' of https://github.com/apple/foundationdb into thread-priority
2022-09-30 08:57:53 -07:00
999809e79b
Increase the data in the workload and add ASSERTs for expected values.
2022-09-29 19:21:44 -07:00
5a3cd51255
Remove temporary TraceEvents and comments from previous commits
2022-09-29 18:53:11 -07:00
c2b6b288b7
Fixes in the GetEstimatedRangeSize workload
...
* Run the workload on a trusted client rather than untrusted
clients. This allows the workload to be substantially simplified
as well as enables testing for the case where no tenant is
present.
* Explicitly pass tenant to BulkSetup so that the setup phase
can be run in parallel for multiple tenants without causing a race.
2022-09-29 18:34:47 -07:00
0264262f78
Explicitly pass tenant to BulkSetup for use in Transactions
...
With this change, BulkSetup accepts Optional<TenantName> as a parameter.
It then uses this while constructing transactions. If a TenantName is
passed to a transaction this way, the transaction can use this tenant
rather than relying on the DatabaseContext. This can be used by
workloads to run the BulkSetup in parallel for multiple tenants without
causing a race.
2022-09-29 18:34:35 -07:00
de90e1aab1
Improve the code in the actors and server, simplify the workload
2022-09-29 18:34:35 -07:00
f63934117d
Make the storage metrics function tenant aware [WIP]
...
This change makes the getEstimatedRangeSizeBytes function tenant aware.
Previously, this function would return the size of the requested
keyspace even if the tenant in the Transaction or DatabaseContext did
not match the tenant corresponding to the keyspace.
Also make some improvements to the new workload.
2022-09-29 18:32:04 -07:00
85ba47f4c8
Add a workload to test getEstimatedRangeSizeBytes funtionality
2022-09-29 18:28:39 -07:00
1b333734a2
set updaterDelay on every loop
2022-09-29 16:53:37 -07:00
01d7668fd1
injected faults should be retryable errors in blob worker
2022-09-29 16:47:31 -07:00
461e42bfe1
restructure updater code and add capacity check in metacluster management workload
2022-09-29 16:24:02 -07:00
03f1d13be3
Enable encryption authentication configurability ( #8312 )
...
* Enable encryption authentication configurability
Description
diff-1: Remove memcpy due to auth-token computation
Address review comments
Patch proposes major changes:
1. Enable FDB to choose encryption authentication as a configurable
parameter. Fix issues choosing ENCRYPT_HEADER_AUTH_TOKEN_NONE mode.
2. Introduce AES_CMAC as supported encryption authentication scheme.
Patch allows cluster to govern: if encryption authentication needs to
enabled, if yes, then choose from two supported schemes:
1. HMAC_SHA_256
2. AES_256_CMAC
Testing
devRunCorrectness - 100K
BlobCipher unittests
EncryptionOps.toml
BlobGranuleCorrectness/BlobGranuleCorrectnessClean
2022-09-29 16:18:55 -07:00
63b8d775a3
Knob for RocksDb behaviors ( #8360 )
...
* Added knob for rocksdb suggest compact.
* Added ROCKSDB_LEVEL_COMPACTION_DYNAMIC_LEVEL_BYTES knob.
Co-authored-by: He Liu <heliu@apple.com>
2022-09-29 15:40:29 -07:00
232e23c87e
Snapshot Backup Encryption ( #8095 )
...
* inital commit
* add encryption support
* refactor passing encryption keys
* disable encryption codepath
* pass enable encryption cli arg for fdbbackup
* change encryption key passing
* only enable encryption for backup if tenant mode is required
* revert server knobs
* ignore json trace files
* --amend
* integrate ekp
* add comments
* comments
* comments
* revert knobs
* fix bug
* modify encrypted backup logic
* use common prefix
* check boundaries between begin and end key
* return system encrypt domain
* modify
* add tenant entry cache to backup
* slight revert
* format
* Trigger Build
* format
* use tenant name to fetch keys
* Trigger Build
* Trigger Build
* merge
* fix getting ekp
* fix bug
* address pr comments
* Trigger Build
* Trigger Build
* remove faulty check
2022-09-29 14:45:47 -07:00
2434f18c5c
Enable failure injection for all simulation tests
2022-09-29 14:14:24 -07:00
a160fc697e
Address comments by passing currentTime by value
2022-09-29 13:55:16 -07:00
fcbd65c2ea
Add unit test
2022-09-29 13:55:16 -07:00
219dde9f41
differentiate degraded peers and disconnected peers in gray failure
2022-09-29 13:55:16 -07:00
93d08b55ba
Merge pull request #8350 from sfc-gh-ajbeamon/fix-metacluster-workload-heap-use-after-free
...
Fix heap use after free in metacluster management workload
2022-09-29 13:38:32 -07:00
5d075d3aff
Fix heap use after free
2022-09-29 12:34:36 -07:00
e26be7a099
Merge branch 'main' of github.com:apple/foundationdb into metacluster-status
2022-09-29 11:37:50 -07:00
257dbff430
recreate transaction every loop in background updater
2022-09-29 11:34:36 -07:00
6357ad1750
pass info through cc data to populate in status
2022-09-28 16:18:44 -07:00
1cc46a98f8
solve merge conflicts
2022-09-28 14:34:42 -07:00
3578832c69
merge upstream/main
2022-09-28 14:01:50 -07:00
c8bb15e8ee
Merge pull request #8216 from sfc-gh-ajbeamon/backup-support-for-tenants
...
Include tenant and metacluster metadata in backups
2022-09-28 12:00:24 -07:00
ef13985feb
Merge pull request #8320 from sfc-gh-tclinkenbeard/read-write-fungible
...
Make read and write quotas fungible
2022-09-28 10:59:37 -07:00
2a640f927c
Merge pull request #8232 from sfc-gh-xwang/feature/main/txnProcessor_tracker
...
Port the DB methods calling in DDShardTracker through DDTxnProcessor interface
2022-09-28 10:33:28 -07:00
251afa3610
Set g_simulator fields for restart tests ( #8318 )
...
* fix tests
* fix tenant concurrency
* Set g_simulator variables for restart tests
* fix machine attrition for restarting
2022-09-28 10:09:26 -07:00
ffe1529f51
Apply clang-format
2022-09-27 19:18:18 -07:00
f2dea33d08
add global var transactionRetryableErrors
2022-09-27 18:14:34 -07:00
3353103d9d
Fix filtering of potential backup mutations in commit proxy and backup worker; add code probe to ensure we are testing default backup sharing and add some tests to hit it
2022-09-27 15:25:30 -07:00
07dae1620c
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/txnProcessor_team
2022-09-27 11:56:02 -07:00
3428fe467a
Merge branch 'main' of https://github.com/apple/foundationdb into feature/dd-refactor-simple
2022-09-27 11:55:35 -07:00
91d2757a51
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/txnProcessor_tracker
2022-09-27 11:22:59 -07:00
df9b21169d
change shared_ptr to Reference
2022-09-27 11:22:47 -07:00
e476e814b2
merge upstream/main
2022-09-27 10:15:26 -07:00
4d9d0afe01
Update fdbserver/TransactionTagCounter.cpp
...
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-09-27 09:45:12 -07:00
ba8fbc9573
Merge remote-tracking branch 'origin/main' into read-write-fungible
2022-09-26 23:26:14 -07:00
6b094c122f
Make throttling cost functions round up to nearest page size
2022-09-26 23:24:32 -07:00
14d73193d5
waitDDTeamInfoPrintSignal, getClusterId, tryUpdateReplicasKeyForDc in IDDTxnProcessor
2022-09-26 23:00:31 -07:00
b13331c85d
Only pass SSD_ROCKSDB_EXPERIMENTAL if RocksDB is actually enabled
2022-09-26 18:38:54 -06:00
0fa462fca9
initial commit to trace metacluster metrics
2022-09-26 15:56:45 -07:00
1b8512e949
Merge pull request #8314 from sfc-gh-ajbeamon/metacluster-management-workload-expansion
...
Improve Metacluster Management Workload
2022-09-26 15:51:28 -07:00
215f23f897
Merge pull request #8308 from sfc-gh-tclinkenbeard/remove-increment-busy-tag-count-probe
...
Replace code probe with UnsetTagThrottledReason trace event
2022-09-26 15:32:19 -07:00
437efc60f0
Merge pull request #8199 from sfc-gh-tclinkenbeard/improve-code-coverage
...
Add `probe::assert::RocksDB` code probe annotation
2022-09-26 16:09:08 -06:00
b0c3534463
Handle invalid tenant configuration when a group and assigned cluster are specified for a tenant and the assigned cluster doesn't match the group's cluster
2022-09-26 14:32:26 -07:00
c503b2dc53
fix probe::assert::RocksDB
2022-09-26 15:03:35 -06:00
248c82eb3f
Merge pull request #8229 from sfc-gh-tclinkenbeard/guard-gsimulator
...
Guard a few uses of gsimulator.
2022-09-26 13:52:42 -07:00
f17742f14c
Update fdbserver/GlobalTagThrottler.actor.cpp
...
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
2022-09-26 13:24:54 -07:00
05615107da
Minor refactoring to remove throw internal_error lines
2022-09-26 13:14:24 -07:00
093db25d11
Add tenant group support to the tenant management workload and test configure and rename tenant operations. Fix a few error types picked up by the test.
2022-09-26 13:14:24 -07:00
1d00ca3261
address code review comments
2022-09-26 10:37:58 -07:00
ce0ad38c94
Merge remote-tracking branch 'origin/main' into guard-gsimulator
2022-09-26 10:18:43 -07:00
8fbbd32a91
Merge remote-tracking branch 'origin/main' into remove-increment-busy-tag-count-probe
2022-09-26 10:17:17 -07:00
76543cbd8a
remove debug trace
2022-09-26 10:02:16 -07:00
2d1a6b4434
Merge branch 'main' of github.com:apple/foundationdb into metacluster-assigned-cluster
2022-09-26 09:59:20 -07:00
6b2c932e75
modify tests to work with error case of changing preferred cluster on retries
2022-09-26 09:59:06 -07:00
0d6fc30d03
Merge pull request #8097 from dwagin/dwagin/freebsd
...
FreeBSD build
2022-09-26 10:39:24 -06:00
cc5ed55263
Change constructor initialization order to silence -Wreoder-ctor. ( #7984 )
2022-09-26 09:38:03 -07:00
86c27754fb
Fixed a bug where rollbackVersion >= data->storageVersion() ( #8290 )
...
* fix: the log router could see popped() versions between recoveredAt and the recovery txn version which cause remote tlogs to be stopped at a version in this range, which causes storage servers to read at versions in this range
* fix: a spurious correctness bug resulted from the consistency check running during a recovery
2022-09-26 09:19:53 -07:00
080acd811f
tuple: user defined type support ( #8300 )
...
* tuple: add support for usertype as last part of tuple
The expectation is user defined types come at the end as there is no
delimiter standard. In that case, if we encounter a user defined type,
we append it to the end of the tuple and allow users to transform that
type by parsing raw string returned.
* blob: use Tuple::unpackUserType() for key alignment
We want blob granule key alignment to be able to understand user data
types. This upgrades support and allows user data types to be the last
key in a blob granule and for us to be able to parse that properly.
2022-09-26 09:18:37 -07:00
7fc5c196c4
Make read and write quotas fungible
2022-09-25 21:00:11 -07:00
985958c260
Add rare code probe decoration
2022-09-25 15:28:32 -07:00
df2c1374cb
Correct a couple of comments in simulator and simulated workloads. ( #8310 )
2022-09-24 22:52:13 -07:00
45ef302af6
Merge pull request #8306 from apple/revert-7578-RedwoodIOLaunchLimit
...
Revert "[DRAFT] Redwood PriorityMultiLock enable different launch limits to be specified based on different priority level."
2022-09-23 16:56:32 -06:00
1194774d54
rename dbProcessor to db; rename getDb() to context()
2022-09-23 15:35:39 -07:00
d9cc7ff44e
merge upstream/main
2022-09-23 14:49:46 -07:00
d643313793
Merge pull request #8278 from sfc-gh-xwang/feature/main/txnProcessor_queue
...
Replace BgDDMountainChopper and BgDDValleyFiller with BgDDLoadRebalance and make DDQueue use txnProcessor
2022-09-23 13:01:39 -07:00
03554cfa0d
Replace code probe with UnsetTagThrottledReason trace event
2022-09-23 11:29:29 -07:00
5774249e5b
Revert "[DRAFT] Redwood PriorityMultiLock enable different launch limits to be specified based on different priority level."
2022-09-23 12:22:47 -06:00
82d8c17d00
Merge pull request #8272 from sfc-gh-dadkins/sfc-gh-dadkins/bugs/reboot-not-kill
...
Downgrade kill to reboot in Rollback workload.
2022-09-23 11:32:04 -06:00
6081f40cfc
Merge pull request #8269 from sfc-gh-dadkins/sfc-gh-dadkins/bugs/kill-process
...
sim: don't allow calls to killProcess or killInterface to kill protected processes.
2022-09-23 11:31:53 -06:00
fe4c33fabb
Merge pull request #8302 from sfc-gh-huliu/testfix2
...
disable MoveKeysWorkload for a few tests that need to manipulate dd
2022-09-23 11:02:04 -06:00
9fd16bd38e
disable MoveKeysWorkload for a few tests that need to manipulate dd
2022-09-23 08:56:27 -07:00
11a6cba2c6
rename dbProcessor to db; readability improvement
2022-09-22 17:11:07 -07:00
48e1b06d29
Add comments about the use of protectedAddresses.
2022-09-22 17:01:01 -07:00
c9b4fc5761
disallow MoveKeysWorkload running in parallel
2022-09-22 17:00:01 -07:00
430f6e9670
Fix purge at latest racing with other checking threads at the end of BlobGranuleVerify ( #8281 )
2022-09-22 16:32:33 -07:00
9122e6e5fe
fixing off by one ( #8279 )
2022-09-22 16:31:55 -07:00
339183228d
allowing ss feed fetches to go over hard limit ( #8243 )
2022-09-22 16:31:06 -07:00
28f3cacfd7
Merge pull request #8285 from sfc-gh-jslocum/bg_history_fdbcli
...
Adding blobkey history fdbcli debug command
2022-09-22 16:56:59 -05:00
2eea93f170
move cluster assignment into loop to test error case
2022-09-22 14:43:08 -07:00
e342a9db43
Merge branch 'main' of github.com:apple/foundationdb into metacluster-assigned-cluster
2022-09-22 14:39:27 -07:00
8a6e68cf63
adjust check if existing entry cluster does not match
2022-09-22 14:39:03 -07:00
bd006526d6
Merge pull request #8251 from sfc-gh-ajbeamon/fdbcli-tenant-group-metadata
...
Fdbcli command to get tenant group metadata
2022-09-22 14:17:08 -07:00
36df430b14
Merge branch 'main' into bg_history_fdbcli
2022-09-22 15:49:07 -05:00
97a325adab
Add an fdbcli command to get tenant group metadata
2022-09-22 13:24:21 -07:00
ccba4dba12
Merge branch 'feature/dd-refactor-simple' of https://github.com/sfc-gh-xwang/foundationdb into feature/dd-refactor-simple
2022-09-22 13:12:05 -07:00
050bf22e1e
add comments to test toml and moveKeys to non-const method
2022-09-22 13:11:53 -07:00
be1cc6c111
Update fdbserver/workloads/IDDTxnProcessorApiCorrectness.actor.cpp
...
Co-authored-by: Bharadwaj V.R <bharadwaj.vr@snowflake.com>
2022-09-22 13:06:21 -07:00
61b1798634
Merge pull request #8249 from jzhou77/fix-tuple
...
Fix rare/CheckRelocation setup failure
2022-09-22 12:59:57 -07:00
7831f6b2f1
Merge pull request #7578 from sfc-gh-fzhao/RedwoodIOLaunchLimit
...
[DRAFT] Redwood PriorityMultiLock enable different launch limits to be specified based on different priority level.
2022-09-22 12:29:17 -07:00
a7c200b16c
Add support for sharing mutation streams between a default backup and DR
2022-09-22 11:36:46 -07:00
37b93a6232
fix tests ( #8286 )
2022-09-22 11:34:06 -07:00
b559be1184
assign cluster outside of retry loop
2022-09-22 11:22:41 -07:00
e91afa15b6
Merge branch 'main' of github.com:apple/foundationdb into metacluster-assigned-cluster
2022-09-22 11:14:27 -07:00
21c627ba97
Adding blobkey history fdbcli debug command
2022-09-22 13:03:53 -05:00
fda0d7223d
Update backup to include system key ranges needed for tenants. Run simulated backup tests with tenants.
2022-09-22 10:00:13 -07:00
04b4960786
Merge branch 'main' into fzhao/RedwoodIOLaunchLimit
...
# Conflicts:
# fdbserver/VersionedBTree.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/ReadWrite.actor.cpp
2022-09-22 00:39:51 -07:00
a6fa8cd345
Rework event detail names to sort differently and remove stats for ReadType::Eager because it is only used internally within the storage server.
2022-09-22 00:20:17 -07:00
e7a280ec03
format code
2022-09-21 20:49:39 -07:00
aaed0c8b7e
Actor to DDQueue methods
2022-09-21 17:57:40 -07:00
7a09b701cc
Merge pull request #8141 from sfc-gh-jfu/network-disable-bypass
...
Introduce network option for disabling mvc bypass
2022-09-21 17:33:48 -07:00
bf943bfd57
Merge pull request #8273 from sfc-gh-huliu/testfix
...
disable MoveKeysWorkload when running SpecialKeySpaceCorrectness test
2022-09-21 17:32:44 -07:00
937fb38dcc
address review comments
2022-09-21 16:23:12 -07:00
f115599a67
disable MoveKeysWorkload when running SpecialKeySpaceCorrectness test
2022-09-21 16:12:20 -07:00
141bd99133
Merge branch 'main' of https://github.com/apple/foundationdb into feature/main/txnProcessor_team
2022-09-21 15:35:26 -07:00
a8d3282250
Merge pull request #8137 from xis19/main
...
Reads and reports cpu.stat
2022-09-21 15:33:57 -07:00
4794ebabcc
Merge pull request #8262 from sfc-gh-dzhou/purge-fixes
...
blob: java api fixes (purge + verify)
2022-09-21 17:25:54 -05:00
5aa9a36559
Downgrade kill to reboot in Rollback workload.
...
Indiscriminately killing machines can lead to unrecoverable clusters,
even if we avoid killing the protected coordinators.
2022-09-21 15:22:31 -07:00
5500ec8126
replace BgDDMountainChopper and BgDDValleyFiller with BgDDLoadRebalance
2022-09-21 15:11:04 -07:00
9358aea097
fix busy loop with correct error handling in valley filler
2022-09-21 14:58:34 -07:00
c4f59ba654
Merge pull request #8261 from neethuhaneesha/rocksdb-cachesize
...
RocksDB block cache size increase.
2022-09-21 14:50:31 -07:00
4ea4546cb6
blob/java: verifyBlobRange() with latestVersion
2022-09-21 14:07:16 -07:00
e353169a50
blob: teach purge about latestVersion
...
This teaches purgeBlobGranules about latestVersion and rejects
versions <= 0.
2022-09-21 14:04:58 -07:00
b7be356a98
removing overhead of local change feed stream copy ( #8205 )
2022-09-21 12:44:51 -07:00
6270016bed
Seq insert perf fixes main ( #8264 )
...
* Force flushing granules post-split to guarantee parent feeds get cleaned up
* fixing bug and cleaning up split finalize code
2022-09-21 12:36:02 -07:00
bc9e806f1c
RocksDB block cache size increase.
2022-09-21 11:41:27 -07:00
0343ca9c53
Merge branch 'main' of github.com:apple/foundationdb into network-disable-bypass
2022-09-21 11:33:27 -07:00
f9e0230b86
DDQueue constructor with ITxnProcessor
2022-09-21 10:56:22 -07:00
1f015b7e1a
Lookup encryption cipher key while writing RAW_ACCESS mutation ( #8256 )
...
* Lookup encryption cipher key while writing RAW_ACCESS mutation
Description
Patch update writeMutation handling of RAW_ACCESS mutations by
looking up Encryption keys if not already part of 'cipherKeys'
map.
Testing
FuzzApiCorrectnessClean.toml -s 3194376849 - issue was found
devCorrectnesRun - 100k
2022-09-21 09:59:09 -07:00
0cc3e6875f
Merge branch 'apple:main' into dwagin/freebsd
2022-09-21 17:16:28 +03:00
7002150188
add storage server read concurrency knob
2022-09-20 21:23:54 -07:00
715b58686d
Fix CompoundWorkload::check method
2022-09-20 20:55:36 -07:00
bad797bd96
Fix rare/CheckRelocation setup failure
...
To ignore timed_out error from getDataInFlight() calls.
2022-09-20 20:35:00 -07:00
970463223c
Merge branch 'main' into main
2022-09-20 16:56:56 -07:00
c237186ec3
Merge pull request #8252 from sfc-gh-huliu/fixtest
...
Fix correctness test: skip lock check in MoveKeysWorkload
2022-09-20 16:34:22 -07:00
3b5117ca92
Merge pull request #8233 from sfc-gh-mpilman/features/disable-failure-injection-for-workload
...
All workloads to disable some failures
2022-09-20 16:17:35 -07:00
8b0f6e2d24
Encryption: Remove memcpy due to computeToken ( #8211 )
...
Description
Major changes include:
1. Eliminate memcpy due to authToken
2. Update TraceEvent message format
3. Remove redundant code.
Testing
flow/BlobCipher unit test
EncryptionOps.toml
devRunCorrectness - 100k
2022-09-20 15:58:53 -07:00
eadb769cfa
Encrypt Backup Mutation Log ( #8159 )
...
* encrypt backup mutation log
* format
* address pr comments
* format
* fix bug
* revert knobs
* address pr comments
2022-09-20 15:43:39 -07:00
9628561235
Add DataDistributionMetrics workload into correctness packages, ( #8237 )
...
which makes the code probe hit in nightly tests.
2022-09-20 15:33:15 -07:00
52f1bef8ec
Fix correctness test: skip lock check in MoveKeysWorkload
2022-09-20 15:16:56 -07:00
Kevin Hoxha