12b6f386cb
Refactoring granule flush to retry properly on granule rollback
2022-07-19 19:49:20 -05:00
c357a90b30
Ensure change coordinators request came from same generation proxy
...
There was a rare but possible issue where a change coordinators request
could be in flight, but reading `\xff/coordinators` would return the old
set of coordinators, even though a previous attempt to write it returned
`commit_unknown_result`. This happened if there was an unrelated
recovery between setting `\xff/coordinators` and the updated
coordinators reaching each machine. The `commit_unknown_result` caused
by this recovery meant the change coordinators retry loop would read
`\xff/coordinators`, but since the request was ongoing it would read the
old set of coordinators. The fix is to have the cluster controller
ignore any change coordinators request from an old generation, meaning
when the retry loop reads the old coordiantors from `\xff/coordinators`,
it is guaranteed that the in progress change coordinators message will
be rejected when arriving at the cluster controller.
2022-07-19 17:44:48 -07:00
8a18fa23cf
Reenable GlobalTagThrottler_GotClientRate trace events
2022-07-19 17:04:04 -07:00
f197314cb5
Add /GlobalTagThrottler/MultiTagActiveThrottling3 unit test
2022-07-19 17:04:04 -07:00
118eb4d93b
Add /GlobalTagThrottler/MultiTagActiveThrottling2 unit test and fix bug uncovered in GlobalTagThrottlerImpl::getLimitingTps
2022-07-19 17:04:04 -07:00
b4e711f1c3
Make parameter more readable in GlobalTagThrottlerTesting::runClient
2022-07-19 17:04:04 -07:00
20ac60fb11
Set throttling ratio in GlobalTagThrottler::tryUpdateAutoThrottling
2022-07-19 17:04:04 -07:00
f41515e66b
Implement GlobalTagThrottler::busy*TagCount methods
2022-07-19 17:04:01 -07:00
166de4b704
Fixing merge convergence bugs
2022-07-19 18:44:40 -05:00
72e72f6d87
Add getMappedRange latency metrics
2022-07-19 16:31:24 -07:00
afba51b7f8
Add tracing to GlobalTagThrottler
2022-07-19 16:29:11 -07:00
fe05cc5c72
Update busy read tag reporting in status json
2022-07-19 16:29:11 -07:00
145d10e3a4
Compute cluster-wide average transaction cost for each tag only once in getClientRates
2022-07-19 16:29:11 -07:00
080518d014
Use getQuota method in GlobalTagThrottlerImpl::getDesiredTps
2022-07-19 16:29:11 -07:00
d0851a67e2
Update TransactionTagCounter::startNewInterval tracing
2022-07-19 16:29:11 -07:00
2306757e90
Fix GlobalTagThrottler::autoThrottleCount
2022-07-19 16:29:11 -07:00
4977805d8e
Fix failing global tag throttling unit tests
2022-07-19 16:29:11 -07:00
c86ae6cee5
Fix /GlobalTagThrottler/Simple unit test
2022-07-19 16:29:11 -07:00
e128c517d3
New GlobalTagThrottler implementation tracking more per-storage
...
server metrics
2022-07-19 16:29:11 -07:00
6a63db08c8
Added /GlobalTagThrottler/ReservedWriteQuota unit test
2022-07-19 16:29:11 -07:00
5b0eb135e0
Guarantee reserved quota in GlobalTagThrottler
2022-07-19 16:29:11 -07:00
8e08033974
Add /GlobalTagThrottler/MultiTagActiveThrottling unit test
2022-07-19 16:29:11 -07:00
bac583ef0c
Add new active throttling unit test for global tag throttler
2022-07-19 16:29:11 -07:00
34a78a82f1
Add MockStorageServer class for global tag throttler testing
2022-07-19 16:29:11 -07:00
88ca6376ce
Add LimitType and OpType enum classes for global tag throttler
2022-07-19 16:29:11 -07:00
fb6540ab43
Add throttling ratio to GlobalTagThrottler
2022-07-19 16:29:10 -07:00
ab2110142a
Add /GlobalTagThrottler/RemoveQuota unit test
2022-07-19 16:29:10 -07:00
019900be67
Add GlobalTagThrottlerImpl::removeUnseenTags
2022-07-19 16:29:10 -07:00
3161bcfa52
Remove outdated TODO and FIXME comments from GlobalTagThrottlerImpl
2022-07-19 16:29:06 -07:00
b49c36f0b0
Add StorageQueueInfo::getWriteQueueSizeLimitRatio method
2022-07-19 16:28:27 -07:00
fde5aeeb62
Remove unused "readQueueSizeMetric"
2022-07-19 14:38:10 -07:00
537ceff8ac
Remove the ability to configure a tenant subspace. Rename the prefixes used for tenant metadata.
2022-07-19 14:32:05 -07:00
a5fba6031f
resolve conflict upstream/main
2022-07-19 13:55:51 -07:00
43099bda0d
resolve conflict upstream/main
2022-07-19 13:48:39 -07:00
190ad8c7e9
Convert existing tuple usages to use Tuple::makeTuple()
2022-07-19 13:45:59 -07:00
7865ead54d
Merge branch 'main' into readaware
2022-07-19 13:35:33 -07:00
1de37afd52
Make TEST macros C++ only ( #7558 )
...
* proof of concept
* use code-probe instead of test
* code probe working on gcc
* code probe implemented
* renamed TestProbe to CodeProbe
* fixed refactoring typo
* support filtered output
* print probes at end of simulation
* fix missed probes print
* fix deduplication
* Fix refactoring issues
* revert bad refactor
* make sure file paths are relative
* fix more wrong refactor changes
2022-07-19 13:15:51 -07:00
19893b4b7a
Persisting change feed metadata version to fix fetch/destroy race across restarts
2022-07-19 14:20:14 -05:00
b9317bebca
Reducing ss injection rates
2022-07-19 14:10:47 -05:00
acc5aee89e
Fixed change feed validation issue
2022-07-19 14:10:10 -05:00
1c543e3b4a
add unittest
2022-07-19 11:58:24 -07:00
1b81e72604
Add a Tuple::makeTuple function to easily construct a tuple. Update Tuple to allow all types to be passed via .append() so they can be used with makeTuple.
2022-07-19 11:50:58 -07:00
c4cd72bea1
Fix a few compile warnings about unused variables
2022-07-19 10:01:41 -07:00
491938fdf5
Merge branch 'main' of https://github.com/apple/foundationdb into feature/dd-refactor-incremental
2022-07-19 09:51:44 -07:00
0e03fbd9e7
Merge pull request #7605 from sfc-gh-ajbeamon/fix-trace-log-detail-formatting
...
Remove some instances of using printable or toString in a trace event detail
2022-07-19 08:24:58 -07:00
78d4d85f3b
Adding non-tss delay injection to SS as well
2022-07-19 09:59:14 -05:00
0d9bb9f4a5
Added targeted storage server restarts at critical metadata points
2022-07-19 08:33:43 -05:00
2573c0018c
cleanup after passing tests
2022-07-19 08:00:25 -05:00
679bb05d34
change feed move/destroy race fixes
2022-07-19 07:19:45 -05:00
1eec6f993d
Merge pull request #7572 from sfc-gh-akejriwal/tenantquota
...
Introduce storage quotas per tenant
2022-07-18 20:00:22 -07:00
0a4aef16d0
Several bugs with new change feed metadata
2022-07-18 19:32:48 -05:00
38855fd786
Remove some instances of using printable or toString in a trace event detail.
2022-07-18 16:24:33 -07:00
bfab550435
cleanup
2022-07-18 16:26:03 -05:00
2909614b0c
smaller function
2022-07-18 14:21:50 -07:00
340a01763b
disabling merge convergence check for now
2022-07-18 16:09:14 -05:00
18a2c3f8a8
Merge pull request #7489 from sfc-gh-tclinkenbeard/cstate-pimpl
...
Mark `CoordinatedState::getConflict` const
2022-07-18 13:22:18 -07:00
4540e2879c
Change Feed Metadata refactor to fix races, simplify model, and clean up code
2022-07-18 12:46:11 -05:00
ac21cbe957
Testing data is not readable after pop
2022-07-18 08:22:20 -05:00
150346af64
Explicitly avoiding large keys problem in single value
2022-07-15 21:08:46 -05:00
08186f9245
More efficient merge intent and granule history serialization
2022-07-15 20:38:06 -05:00
882df774c0
Storage quota: use transaction options and minor fixes in the test
2022-07-15 16:34:04 -07:00
8a31fa87a8
add other operations to test workload
2022-07-15 16:22:37 -07:00
02ab3b05ab
Merge pull request #7590 from sfc-gh-ajbeamon/update-prefix-encoding-in-tenant-metadata-json
...
Use a more standard encoding in the JSON metadata reported for tenants
2022-07-15 14:32:53 -07:00
fb4fc8af9d
Merge pull request #7587 from sfc-gh-ahusain/ahusain-hybrid_encryption_active
...
BlobGranule encryption and compression support
2022-07-15 15:57:51 -05:00
306610bfcb
batch periodic merging in blob manager
2022-07-15 15:52:10 -05:00
4dc742493e
Add cleanUpChangeFeeds(). ( #7598 )
...
Co-authored-by: He Liu <heliu@apple.com>
2022-07-15 10:29:09 -07:00
1da88d5d8c
BlobFile Encryption and compression support
...
Fix formatting issues
Description
Testing
2022-07-14 22:46:14 -07:00
866dda5763
Adding check for merge convergence
2022-07-14 20:49:43 -05:00
f288abebc2
BlobFile Encryption and compression support
...
Fix formatting issues and rename KNOB
Description
Testing
2022-07-14 17:22:00 -07:00
6f130b4a69
Change `quota` type to `unit64_t` to avoid macOS compilation issue
2022-07-14 17:09:03 -07:00
f6f117592d
BlobFile Encryption and compression support
...
- Limit verbose logging under DEBUG_MACRO
- Update/Add code documentation
Description
Testing
2022-07-14 17:04:14 -07:00
3a3811d6cc
BlobFile Encryption and compression support
...
Few fixes
- Update BlobGranuleCipherKeysMeta to use 'std::string' to
persist 'Initialization Vector'
- Update WriteSnapshot to persist 'BlobGranuleCipherKeysMeta' as part
of BlobFileIndex
Description
Testing
2022-07-14 17:04:14 -07:00
24b2de8de8
BlobFile Encryption and compression support
...
Description
Testing
2022-07-14 17:04:14 -07:00
bb05321d24
Introduce storage quotas per tenant.
...
This change adds:
* ability to store the mapping from tenants to quota in the system keyspace,
* a setter and getter function
* a new workload to test this functionality
FDBCORE-2437
2022-07-14 16:35:12 -07:00
4cbe5dd683
Merge pull request #7595 from sfc-gh-xwang/feature/dd-refactor-incremental
...
[DD testability] add resumeRelocation method
2022-07-14 15:24:56 -07:00
7d7ce0909f
Restart tests carry forward encryption knobs value ( #7497 )
...
Previously to get around the issue that EKP is not present when restart test switching encryption from on to off and read encrypted data, EKP was made to start in simulation regardless of encryption knob. This PR revert that change, and instead force restart test not to change encryption knob, by passing previous encryption knob through restartInfo.ini file. Also since we don't allow downgrading an encrypted cluster to previous version, disable encryption in downgrade tests.
Also adding an assert to allow reading encrypted mutations only if encryption knob is on. We may reconsider allowing switching encryption on/off for existing cluster, but for now we don't allow it.
2022-07-14 14:45:17 -07:00
7bca503528
Use narrow includes in key backed types; remove some unnecessary return statements.
2022-07-14 13:26:43 -07:00
598457b872
Merge pull request #7551 from neethuhaneesha/rocksCompressionStats
...
Adding rocksdb compression and data size stats.
2022-07-14 12:58:10 -07:00
f53c108e4f
Update fdbserver/DataDistribution.actor.cpp
2022-07-14 12:17:34 -07:00
42bef3b265
Update fdbserver/DataDistribution.actor.cpp
2022-07-14 12:17:29 -07:00
7ee091d1da
add test namespace
2022-07-14 12:13:23 -07:00
015026f146
add comment; add resumeRelocation
2022-07-14 11:32:27 -07:00
3cbb51de21
Adding rocksdb compression and data size stats.
2022-07-14 11:28:20 -07:00
83ae15012d
Added popping to change feed operations workload
2022-07-14 11:43:52 -05:00
87f833469b
Merge branch 'main' of https://github.com/apple/foundationdb into feature/dd-refactor-incremental
2022-07-14 09:15:51 -07:00
9c20b15f5a
Use enum variables to invoke Priority checking ( #7514 )
...
* Use enum variables to invoke Priority checking
* add an explicit isDataMovementForReadBalancing function
2022-07-14 09:06:56 -07:00
407300bfa6
Disable testing of the remote key value store in simulation
2022-07-13 18:32:50 -07:00
7e3e79c53e
Merge pull request #7585 from sfc-gh-xwang/feature/dd-refactor-incremental
...
[DD Testability] create DataDistributor::init() method
2022-07-13 17:03:12 -07:00
91949439d2
Add support for returning whether a key-backed type range read has more results or not.
2022-07-13 15:46:10 -07:00
b85fbaef52
Merge pull request #7395 from sfc-gh-jslocum/bg_file_chunking
...
Chunked Snapshot Files
2022-07-13 17:22:34 -05:00
91ccdbcb03
Use a more standard encoding in the JSON metadata reported for tenants.
2022-07-13 15:22:19 -07:00
c3aca3f28c
add comment
2022-07-13 15:08:02 -07:00
e32e726b9f
Merge pull request #7583 from sfc-gh-jslocum/granule_merging_improvements
...
Granule merging improvements
2022-07-13 16:04:22 -05:00
0e488ee16b
Merge branch 'main' of https://github.com/apple/foundationdb into feature/dd-refactor-incremental
2022-07-13 12:53:21 -07:00
1592f6ad34
create DataDistributor::init() method
2022-07-13 12:53:04 -07:00
a2062df220
Merge pull request #7573 from sfc-gh-xwang/feature/dd-refactor-incremental
...
move getInitialDataDistribution to DDTxnProcessor
2022-07-13 10:00:38 -07:00
e0984091f3
Merge pull request #7494 from sfc-gh-xwang/bug/cycle-issue7423
...
Extend transaction window in Cycle workload if in simulation the transaction_too_old keep appearing
2022-07-13 09:25:16 -07:00
142ed7713e
Initial version of ChangeFeedOperations workload
2022-07-13 11:16:53 -05:00
b9437e9a8d
BW re-sending merge candidate if manager changes
2022-07-13 10:39:55 -05:00
0720b358ff
comments and bug fix
2022-07-13 08:01:26 -05:00
b2a96b64e6
refactoring one function into two
2022-07-13 07:49:19 -05:00
03c39dc33b
Merge pull request #7579 from sfc-gh-jslocum/granule_merging
...
Granule merging
2022-07-13 07:25:26 -05:00
e367a5a160
Add KVS physical shard metrics. ( #7581 )
2022-07-13 01:31:05 -07:00
a9e9772b98
fix argument bug
2022-07-12 16:49:09 -07:00
f707626213
Cleanup
2022-07-12 17:43:44 -05:00
d3686143a1
merge upstream/main
2022-07-12 14:28:49 -07:00
236630855b
Merge branch 'main' into granule_merging
2022-07-12 15:24:26 -05:00
a4929f3f6f
Merge pull request #7576 from sfc-gh-xwang/feature/dd-refactor-simple
...
[DD Testability] move waitDataDistributorEnabled to txnProcessor
2022-07-12 13:23:29 -07:00
bf98dc2a2b
move waitDataDistributorEnabled to txnProcessor
2022-07-12 11:25:59 -07:00
7d8131a39c
Worker register with cluster controller before disk file recovery ( #7510 )
...
Previously worker servers only register with cluster controller after local file recovery. This PR change to let worker servers register itself before local file recovery, but indicate that although it can server stateless roles, it cannot become storage or TLog server. After local file recovery, the worker will register again to indicate it can now become storage and Tlog.
The change fixes the deadlock issue with encryption, where when there's only stateful workers that can become master and EKP, those worker servers won't be able to register with cluster controller since no EKP is available to provide cipher keys to decrypt encrypted storage.
For now we have the change only when encryption is on, and will make it the default behavior once we think it is stable.
2022-07-12 10:19:32 -07:00
110f3bbde9
remove reset speedUpSimulation
2022-07-12 09:13:33 -07:00
878adad43f
reset some changes
2022-07-12 09:13:33 -07:00
5e8f55768a
use speedUp to extend transaction window
2022-07-12 09:13:32 -07:00
b4b4a4a2f9
an failed fix
2022-07-12 09:13:32 -07:00
55aed68a45
detach debug commit
2022-07-12 09:13:32 -07:00
ac5436a090
Catch up on changes to main
2022-07-12 16:48:09 +02:00
0b0ac16a4c
Merge branch 'main' into granule_merging
2022-07-12 09:09:30 -05:00
2edbcf2c65
Merge pull request #44 from apple/main
...
Merge main
2022-07-12 07:51:22 -06:00
c6700fe62f
Merge branch 'main' into bg_file_chunking
2022-07-12 08:28:06 -05:00
7e067ee4de
move getInitialDataDistribution to DDTxnProcessor
2022-07-11 21:54:47 -07:00
cc185c51c6
Merge pull request #7563 from sfc-gh-xwang/feature/dd-refactor-incremental
...
[DD Testability] move updateReplicaKey to txnProcessor
2022-07-11 17:11:55 -07:00
4f4095b6ae
Merge pull request #7385 from sfc-gh-etschannen/fix-multiple-shared-tlogs
...
Avoid recruiting multiple shared tlogs
2022-07-11 17:28:35 -06:00
4ece33a0a8
Merge pull request #7445 from sfc-gh-anoyes/anoyes/fix-ubsan
...
Fix UBSAN build when statically linking libcxx
2022-07-11 17:27:37 -06:00
3aeb2f17cd
Redwood remove duplicate header files
2022-07-11 12:44:44 -07:00
094999439d
Fixing another change feed fetch race by making refreshing idempotent
2022-07-11 14:25:01 -05:00
f42b8687c2
add comment
2022-07-11 11:26:46 -07:00
447eaa1d81
move updateReplicaKey to txnProcessor
2022-07-11 11:16:17 -07:00
33fcdc4764
Change Feed and Blob Worker operational improvements
2022-07-11 12:29:51 -05:00
dbf8f98481
Apply Clang format
2022-07-11 17:10:45 +02:00
545a9a8043
Fix bugs and add token timeout-in-cache test
2022-07-11 16:58:04 +02:00
61f3d14ec6
Add TokenCache test
...
Add function authz::jwt::TokenRef::toStringRef() for token tracing
Add StringRef::toStringView()
2022-07-11 13:42:41 +02:00
901d988de9
Add a knob SNAPSHOT_ALL_STATEFUL_PROCESSES to snapshot all processes with stateful class type(storage, log, transaction) even if they are not recruited ( #7554 )
2022-07-08 20:53:49 -07:00
5ce1d7342d
Merge branch 'feature/dd-refactor-incremental' of https://github.com/sfc-gh-xwang/foundationdb into feature/dd-refactor-incremental
2022-07-08 16:35:31 -07:00
9fe76ecf21
format
2022-07-08 16:34:55 -07:00
a941a28de7
Merge branch 'apple:main' into feature/dd-refactor-incremental
2022-07-08 16:07:16 -07:00
b48b1e93f9
remove header dependency; move getDatabaseConfiguration to txnProcessor
2022-07-08 16:01:23 -07:00
712f40b727
Add support for tenant groups
2022-07-08 15:59:25 -07:00
6be20d08ae
Merge pull request #7555 from sfc-gh-xwang/feature/dd-refactor-incremental
...
[DD Testability] move takeMoveKeysLock to DDTxnProcessor
2022-07-08 15:40:14 -07:00
aa43956fbd
Sharded RocksDB ( #7540 )
...
* Some fixes.
* Enabled ShardedRocksDB in IKeyValueStore.h
* Added unit test for ShardedRocks for \xff\xff key space read/write.
* Resolved comments.
* Return empty read results if the physical shards are not initialized.
Co-authored-by: He Liu <heliu@apple.com>
2022-07-08 15:17:38 -07:00
aca44c8644
Increase frequency of RelocateShardStartSplit traces for better observability of splits ( #7503 )
2022-07-08 14:45:44 -07:00
9cead35911
add contract comments
2022-07-08 14:19:14 -07:00
f5a81154ef
move takeMoveKeysLock to DDTxnProcessor
2022-07-08 14:11:31 -07:00
4103b16793
Fix a performance bug in Resolver
...
The erased bytes from recent state transactions was accidently set to 0 and
cause anyPopped to be always false. Thus, when totalStateBytes will never
decrease and can cause high latency between "Resolver.resolveBatch.Before" and
"Resolver.resolveBatch.AfterQueueSizeCheck", in hundreds of milliseconds. This
will cause high server side commit latency.
2022-07-08 09:58:28 -07:00
f268265d96
Apply suggestions from code review
...
Co-authored-by: A.J. Beamon <aj.beamon@snowflake.com>
Co-authored-by: Junhyun Shim <junhyun.shim@snowflake.com>
2022-07-08 10:42:44 -06:00
bc5bfaffda
Shard based move ( #6981 )
...
* Shard based move.
* Clean up.
* Clear results on retry in getInitialDataDistribution.
* Remove assertion on SHARD_ENCODE_LOCATION_METADATA for compatibility.
* Resolved comments.
Co-authored-by: He Liu <heliu@apple.com>
2022-07-07 20:49:16 -07:00
e63cb431e1
Merge pull request #7534 from sfc-gh-satherton/storage-type-improvements
...
Add StorageEngine type to StorageMetrics trace events
2022-07-07 16:26:43 -07:00
a2ef5dc72a
set token properly in test
2022-07-07 12:00:32 -06:00
55548e4ac8
Avoid signed integer overflow
2022-07-07 10:19:20 -07:00
8c1813e377
Don't pass nullptr to memcpy
2022-07-07 10:19:20 -07:00
fa0c3249ca
Merge pull request #7529 from sbodagala/main
...
Make sequencer aware of the locality of the primary region
2022-07-07 14:00:10 +05:30
c4b0f6eaae
Add an internal C API to support connection to a cluster using a connection string ( #7438 )
...
* Add an internal C API to support memory connection records
* Track shared state in the client using a unique and immutable cluster ID from the cluster
* Add missing code to store the clusterId in the database state object
* Update some arguments to pass by const&
2022-07-07 10:12:49 +02:00
03df62e2fc
Added StorageEngine detail to StorageMetrics trace event.
2022-07-07 00:10:04 -07:00
56541167c2
send token with tenant info object
2022-07-06 20:58:17 -06:00
f98d5ae9e9
[ShardedRocksDB] 5/N Generate and persist metadata for range mapping. ( #7408 )
2022-07-06 15:57:46 -07:00
df1f108b78
Merge branch 'main' of github.com:apple/foundationdb into jfu-tenant-rename
2022-07-06 14:10:22 -07:00
bb99b59deb
Change VersionVectorTagUpdates to LatencySample ( #7531 )
2022-07-06 16:48:15 -04:00
1e8225a19e
Merge pull request #7511 from sfc-gh-ajbeamon/add-subrange-support-to-tenant-special-keys
...
Update the tenant special keys submodule to support multiple sub-ranges
2022-07-06 10:33:14 -07:00
a84112d30b
- Propagate the locality of the primary region from ClusterController
...
to the sequencer, rather than deriving it from "MasterInterface" on
the sequencer (which is not correct).
2022-07-06 11:57:06 +00:00
db449b80f7
Authz JWKS ( #7519 )
...
* Add JsonWebKeySet parser/stringifier
* Update header directory
* Make JWKS parser correctness clean for OpenSSL 1.x
Add RSA keygen support
* Make JWKS parser correctness clean for OpenSSL 3.x
+extend unique_ptr for scoped destruction of OpenSSL objects
* Use PKey::{sign|verify}() in TokenSign
* Apply AutoCPointer to MkCert
* Apply Clang format
* JWKS::toStringRef() returns StringRef > Optional<StringRef>
* Fix Mac/Windows build error
* Fix incorrect fix of Mac build
* Fix filename in license comment for AutoCPointer.h
* Refactor complex C macros into function templates
2022-07-05 10:13:09 +02:00
c417df83f3
Merge pull request #7515 from sfc-gh-ajbeamon/more-generic-key-backed-types
...
Update key backed types to support multiple transaction types and different codecs
2022-07-01 14:37:02 -07:00
08ca0a9883
Merge pull request #7509 from sfc-gh-jslocum/cf_memory_perf
...
Change Feed Reading CPU optimization
2022-07-01 14:06:39 -05:00
ea42e9f59c
Do not delay sending empty messages twice ( #7462 )
2022-07-01 14:22:53 -04:00
9e659b62ee
Fix idempotent spelling
2022-07-01 10:34:10 -07:00
62683b4140
Update key backed types to support a templated Codec. This required updating the existing Codec to convert from objects into byte strings through tuples.
2022-07-01 10:11:02 -07:00
1dc78a3b3f
Add support for different transaction types (Reference<ITransaction>, Reference<ReadYourWritesTransaction>) to key backed types
2022-07-01 10:02:39 -07:00
aff21f062f
Fix indentation and refactor prefix construction to be in forward order
2022-07-01 09:50:05 -07:00
9e64037b25
Merge branch 'main' into bg_file_chunking
2022-06-30 17:13:02 -05:00
2f67328a0c
Update the tenant special keys submodule to support multiple sub-ranges. This will enable future work that allows configuring tenants at the same time as creating them.
2022-06-30 15:03:37 -07:00
f943efb3b1
Change Feed Reading CPU optimization
2022-06-30 15:32:41 -05:00
48bcf6eaba
parameterizing min split size instead of hardcoding it to min_shard_bytes ( #7404 )
2022-06-30 12:28:22 -07:00
e85e7794c2
Merge pull request #7501 from sfc-gh-xwang/features/ppw-pause-state
...
Add timeout handler for storage wiggler status fetcher
2022-06-30 11:23:29 -07:00
d60cab788e
Merge pull request #7502 from jzhou77/main
...
Add pipelining for secondary queries in index prefetch
2022-06-30 10:27:29 -07:00
76a147a104
Merge pull request #7496 from sfc-gh-ajbeamon/refactor-tenant-map-encoding
...
Refactor how tenant map entries are encoded and decoded
2022-06-30 09:48:38 -07:00
4910f69f60
Update fdbserver/storageserver.actor.cpp
...
Co-authored-by: Hao Fu <77984096+hfu94@users.noreply.github.com>
2022-06-30 09:19:19 -07:00
98b18e3a18
Remove code obsoleted by commit c48d5690 ( #7499 )
2022-06-30 12:16:23 -04:00
467642ac7f
Remove semicolons
2022-06-30 09:12:58 -07:00
e69604be42
Merge branch 'main' of https://github.com/apple/foundationdb into features/ppw-pause-state
2022-06-30 09:01:29 -07:00
0beaf3bb93
Merge pull request #7504 from sfc-gh-tclinkenbeard/monitor-client-rates
...
Make GlobalTagThrottlerTesting::monitorClientRates more lenient
2022-06-30 09:00:22 -07:00
3d529fb145
Merge pull request #7055 from sfc-gh-tclinkenbeard/remove-tpcc
...
Remove TPC-C workloads
2022-06-30 09:49:09 -06:00
3e542b7ab1
Make GlobalTagThrottlerTesting::monitorClientRates less strict
2022-06-30 00:59:26 -07:00
b51e9430a5
fix timeout buggify
2022-06-29 22:46:50 -07:00
72f5335460
Remove TPCC workloads
2022-06-29 22:31:56 -07:00
842004f3b3
Code clean-up
2022-06-29 21:29:41 -07:00
ba5e2ed72d
Add batching for secondary queries in index prefetch
...
This is to reduce excessive load being issued concurrently, controlled by knob
MAX_PARALLEL_QUICK_GET_VALUE.
2022-06-29 21:17:11 -07:00
1b0780398f
Merge branch 'main' of https://github.com/apple/foundationdb into features/ppw-pause-state
2022-06-29 21:03:47 -07:00
da26a4c373
Issue secondary query in parallel for GetMappedKeyValues
...
In case the subquery is a local miss, it becomes a remote call. Then such an
optimization can significantly reduce latency.
2022-06-29 17:45:10 -07:00
570dfd6760
add timeout handler
2022-06-29 15:30:37 -07:00
364644673f
Support TLog encryption in commit proxy ( #6942 )
...
This PR add support for TLog encryption through commit proxy. The encryption is done on per-mutation basis. As CP writes mutations to TLog, it inserts encryption header alongside encrypted mutations. Storage server (and other consumers of TLog such as storage cache and backup worker) decrypts the mutations as they peek TLog.
2022-06-29 14:21:05 -07:00
5e7bb0aa21
Merge branch 'main' of github.com:apple/foundationdb into jfu-tenant-rename
2022-06-29 13:20:19 -07:00
c9366f8b01
Merge pull request #7491 from sfc-gh-xwang/bug/remotekv
...
Make g_simulator.datacenterDead() don't take spawn KV process into account
2022-06-29 11:43:27 -07:00
29f98f3654
Avoid duplicate snapshot on one process if it serves as multiple roles ( #7294 )
...
* Fix comments
* Add simulation value for SERVER_KNOBS->SNAP_CREATE_MAX_TIMEOUT
* A work version with correctness clean
* Remove unnecessay comments; debugging symbols
* Only check secondary address for coordinators, same as before
* Change the trace to SevError and remove the ASSERT(false)
* Remove TLogSnapRequest handling on TlogServer, which is changed to use WorkerSnapRequest
* Add retry for network failures
* Add retry limit for network failures; still allow duplicate snapshots on processes are both tlog and storage to avoid race
* Add retry limit as a knob and make backoff exponentail
* Add getDatabaseConfiguration(Transaction* tr)
* revert back to send request for each role once
* update some comments
2022-06-29 11:23:07 -07:00
26b35c07cd
Refactor how tenant map entries are encoded and decoded. Add a specific version to the encoding that matches the version used when this feature was introduced (and the only version in which it was used).
2022-06-29 10:58:58 -07:00
db769667ae
Merge pull request #7112 from sfc-gh-tclinkenbeard/global-tag-throttling3
...
Create Global Tag Throttler
2022-06-29 10:06:00 -07:00
0054b8a754
Merge branch 'main' of https://github.com/apple/foundationdb into bug/remotekv
2022-06-28 23:41:25 -07:00
c6ff556f06
fix bug by exclude the spawned kv process info when counting dead process; add method
2022-06-28 23:39:22 -07:00
2134fa9a0c
Merge remote-tracking branch 'origin/main' into global-tag-throttling3
2022-06-28 19:51:41 -07:00
b4193f701e
Mark CoordinatedState::getConflict const
2022-06-28 19:45:58 -07:00
be9a2002c3
Merge remote-tracking branch 'origin/main' into fix-knob-typo
2022-06-28 15:58:18 -07:00
cc93dfd289
Add getStorageBytes. ( #7485 )
2022-06-28 14:37:26 -07:00
ec9c3cf05c
Remove unused variables
2022-06-28 16:41:58 -04:00
Josh Slocum