34da5e25a2
Allow multiple keyranges in CheckpointRequest. ( #8743 )
...
* Allow multiple keyranges in CheckpointRequest.
Include DataMove ID in CheckpointMetaData.
* Use UID dataMoveId instead of Optional<UID>.
Co-authored-by: He Liu <heliu@apple.com>
2022-11-09 11:18:58 -08:00
ca33fe1f1e
Rocksdb stats level knob. ( #8711 )
2022-11-07 22:32:48 -08:00
d28187ca90
Do not set timeout for RocksDB reads in simulation. ( #8716 )
...
* Do not set timeout for RocksDB reads in simulation.
* Cleanup.
* Ignore validateStorage test.
2022-11-07 15:59:52 -08:00
1daa346cb4
Adding boundaries to rocksdb read iterator pool. ( #8584 )
2022-11-03 10:26:47 -07:00
f1eca6d672
fixup! Fix compile error
2022-11-01 10:36:25 -07:00
520a14cd72
fixup! update code per comments
2022-11-01 10:36:25 -07:00
74de1aaa8d
Redirect RocksDB Log data to FDB TraceEvent
...
RocksDB supports injecting a custom logger. By introducing
RocksDBLogForwarder, it is possible to retrieve RocksDB log lines and
log them as TraceEvents.
Note that RocksDB is multi-threaded while FDB is designed to be
single threaded, in the FDB event loop there are certain variables
defined as thread_local. This causes that only FDB main thread could
generate TraceEvent, since information like Machine are only available
in the main thread. The current design is to cache all log lines from
the other threads, and let the main thread report the cached log, when:
* The main thread received log
* Every some time (this should be a Knob but at this stage a magic
number is used for convenience.)
It is possible to parse the log lines into Key/Value pairs for
TraceEvent output, but at this stage only the format string is printed
out. Also, RocksDB may log some errors that are interpreted as SevError,
which will cause simulation failures. To prevent this, the maximum error
level is set to SevWarn, which should be reconsidered carefully.
The code is tested by 1) random simulations to ensure the log lines are
included in the XML files 2) force the simulator uses RocksDB and run a
10k correctness run. Both tests passed.
2022-11-01 10:36:25 -07:00
a1eb1d4a48
Rocksdb storage using single_key_deletes instead of deleterange on clearrange operation. ( #8452 )
2022-10-21 15:47:19 -07:00
3db0c786e5
RocksDB commit latency sample corrections.
2022-10-12 16:19:49 -07:00
4b238e3985
RocksDB disable WAL for experimentation. ( #8443 )
2022-10-10 12:32:05 -07:00
88c37c81d8
Merge branch 'main' of https://github.com/apple/foundationdb into thread-priority
2022-10-05 14:20:53 -07:00
0361cf74e5
fix RocksDB compilation errors
2022-10-05 09:21:15 -06:00
550488b020
Merge remote-tracking branch 'origin/main' into bugfixes/open-for-ide
...
# Conflicts:
# bindings/c/CMakeLists.txt
# fdbclient/include/fdbclient/GetEncryptCipherKeys.actor.h
# fdbserver/BackupWorker.actor.cpp
# fdbserver/BlobWorker.actor.cpp
# fdbserver/CommitProxyServer.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/StorageCache.actor.cpp
# fdbserver/include/fdbserver/GetEncryptCipherKeys.actor.h
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/PhysicalShardMove.actor.cpp
# flow/CMakeLists.txt
2022-10-04 18:27:48 -06:00
a565863189
RocksDB compaction knobs and stats. ( #8392 )
2022-10-04 15:13:50 -07:00
e1fe28b78b
Switch some usages of LiteralStringRef to use the _sr suffix
2022-09-30 16:04:16 -07:00
5f975623fb
Merge branch 'main' of https://github.com/apple/foundationdb into thread-priority
2022-09-30 08:57:53 -07:00
63b8d775a3
Knob for RocksDb behaviors ( #8360 )
...
* Added knob for rocksdb suggest compact.
* Added ROCKSDB_LEVEL_COMPACTION_DYNAMIC_LEVEL_BYTES knob.
Co-authored-by: He Liu <heliu@apple.com>
2022-09-29 15:40:29 -07:00
4fd64630e8
Convert literal string ref instances to use _sr suffix
2022-09-19 11:35:58 -07:00
2e2c198df7
Added knobs for RocksDB reader/writer thread priorities.
2022-09-19 11:20:17 -07:00
7c6dbaf3cf
update trState ReadOption to optional
2022-08-24 15:04:17 -07:00
4e748d6bed
use optional ReadOptions and RangeReadOptions in requests
2022-08-23 17:16:47 -07:00
78f4b4f739
add RangReadOptions which inherit from ReadOptions
2022-08-22 11:46:01 -07:00
d24a52952c
update readOption struct
2022-08-10 10:16:26 -07:00
1a5b8fbd30
resolve conflicts
2022-08-09 09:57:11 -07:00
e4fb565057
use readOptions to pass type and cacheResult
2022-08-08 15:30:56 -07:00
a2dc4e20e9
Store rocksdb::DBOptions and rocksdb::ColumnFamilyOptions to ( #7766 )
...
* Store rocksdb::DBOptions and rocksdb::ColumnFamilyOptions to
SharedRocksDBState.
* Make KeyValueStoreRocksDBTest.toml `UNIT`.
* Added TEST_STORAGE_ENGINE_TYPE
* Added readOptions.
2022-08-08 13:38:02 -07:00
118c2ae806
Fixed memory leak for RocksDb and CheckpointReader. ( #7665 )
...
* Fixed memory leak for RocksDb and CheckpointReader.
* Close kvStore at the end of StorageServerCheckpointRestoreTest.
2022-07-25 12:36:29 -07:00
3cbb51de21
Adding rocksdb compression and data size stats.
2022-07-14 11:28:20 -07:00
d80506fd21
Adding logId to trace events.
2022-06-27 10:41:05 -07:00
410ab6352f
Traceevent correction.
2022-06-01 15:21:15 -07:00
78dc7c5d77
Resolving comments.
2022-05-19 13:27:21 -07:00
54c566e57d
Drop RocksDB column family before create a new column family from
...
import.
2022-05-19 13:27:21 -07:00
05bbe174c3
Implemented checkpoint(), restore() for KeyValueStoreRocksDB, for the
...
new format.
2022-05-19 13:27:21 -07:00
93f9ea9a5b
Adding rocksdb throttling counters to trace event.
2022-05-05 17:42:26 -04:00
6c9ff6ee5e
Add sharded rocksdb type ( #6862 )
...
* add-sharded-rocksdb-type
* address comments
Co-authored-by: Zhe Wang <zhewang@Zhes-MacBook-Pro.local>
2022-04-21 22:53:14 -04:00
6543bce8ae
RocksDb using aggr property metrics for pendingCompactionBytes.
2022-04-14 18:08:42 -07:00
2f75a4bd78
Use actor collection for rocksdb histogram actors. ( #6805 )
...
Co-authored-by: Zhe Wang <zhewang@Zhes-MacBook-Pro.local>
2022-04-13 14:41:54 -07:00
b7096c410f
Merge pull request #6795 from neethuhaneesha/rocksdb-blocksize
...
Adding rocksdb block size option.
2022-04-08 14:20:54 -07:00
37054af7e2
Fix RocksDB Metrics ( #6803 )
...
* fix-metrics-in-rocksdb
* remain-GetIntProperty-for-checkRocksdbState
Co-authored-by: Zhe Wang <zhewang@Zhes-MacBook-Pro.local>
2022-04-08 17:07:22 -04:00
3d325940ad
Fix data race issue when multithreaded RocksDB store using histogram ( #6766 )
...
* using-ThreadReturnPromiseStream-to-make-histogram-thread-safe
* address-comment-and-verify-functionality-in-cluster
* remove-old-histogram-metrics
* fix-comment
Co-authored-by: Zhe Wang <zhewang@Zhes-MacBook-Pro.local>
2022-04-07 00:31:36 -04:00
0d05669c22
Adding rocksdb block size option.
2022-04-06 19:15:17 -07:00
37c7b3ff18
fix-rocksdb-blockcache-recreation-problem
2022-03-29 12:07:41 -04:00
383b0cf0e1
add-rocksdb-memory-usage-metrics
2022-03-23 15:59:32 -04:00
790ef9ff36
Adding rocksdb compaction readahead size option.
2022-03-22 13:29:35 -07:00
1aabbb07d2
Add missing copyright headers
2022-03-16 11:25:02 -07:00
c3a68d661e
Physical Shard Move ( #6264 )
...
Physical Shard Move part I: Checkpoint creation, transfer and restore.
2022-03-15 13:03:23 -07:00
212deb05e9
Merge pull request #6501 from neethuhaneesha/rocksdb-CompBytesLimiter
...
Rocksdb knobs for compaction, storageserver canCommit() waiting if rocksdb overloaded
2022-03-07 14:59:34 -08:00
36edfb8820
Update error type for timeouts. ( #6503 )
2022-03-07 13:14:20 -08:00
8796a763a5
Rocksdb knobs for compaction, storageserver canCommit() waiting if rocksdb overloaded.
2022-03-04 12:41:17 -08:00
83e0368eaa
RocksDB increasing background threads to speedup compaction.
2022-03-03 15:14:37 -08:00
f14e08a991
addRocksDBPerfContextMetrics
2022-02-23 22:29:07 -05:00
96dd86ebf8
update RocskDB and Boost
...
add Finduring, and include into fdbserver
add BOOST asio/uring settings to fdbserver compile
move portable rocks, liburing up to be configurable at build time.
2022-02-09 10:48:18 -06:00
c605226a56
Add error handling in RocksDB KVS. ( #6277 )
2022-01-25 13:59:25 -08:00
162bce7a58
Rocksdb write rate limiter.
2022-01-18 13:23:00 -08:00
ef4038fe8d
Rocksdb read range iterator pool to reuse iterators.
2022-01-18 02:05:21 -08:00
1c2c2783ea
Implement single deletion in RocksDB.
...
If the target deletion range contains a single key, convert it into a
single deletion, to avoid extra cost in RocksDB due to range deletion.
2022-01-04 09:59:40 -08:00
1f30368e71
KeyValueStoreRocksDB histograms to track latencies
2021-12-21 23:09:46 -08:00
d23b8645f8
Enabling rocksdb metrics logger in simulation.
2021-12-07 15:18:29 -08:00
9dccb0131e
Clean up RocksDB error logging
2021-11-12 12:14:12 -05:00
394b9dc619
Code review changes
2021-11-10 11:53:27 -05:00
66520eb1c1
Utilize read types to do selective throttling
2021-11-10 11:51:04 -05:00
8822c589de
Move table factory options to CFOptions
2021-11-09 17:29:06 -05:00
421dee532c
Add const qualifiers in KeyValueStoreRocksDB.actor.cpp
2021-10-18 13:40:47 -07:00
9713a14ef1
Reverse order of read type and debug ID args
2021-10-18 12:23:09 -04:00
df53cc9580
Add an enum to IKeyValueStore to indicate the source/priority of the read
2021-10-15 14:35:59 -04:00
3c35bc69d7
Decoupling the rocksdb version check to multiple static asserts.
2021-09-17 08:53:23 -07:00
c0e6b09fe3
Enforcing 6.22.1 rocksdb version and enabling readRange deadline option.
2021-09-17 08:53:23 -07:00
627ef2e739
Enable rocksdb storage in simulation with some simulation fixes.
2021-09-10 11:11:29 -07:00
f7842ff3db
Log SubCode and Severity for RocksDB erros ( #5515 )
2021-08-31 21:03:02 -07:00
b75fb0d105
Improve error to status conversion
2021-08-23 11:35:26 -04:00
473e2bbf5f
Fix error handling of reads.
2021-08-23 11:35:26 -04:00
0fbfcc582e
Increasing the read cancellation timeouts to 5 minutes in simulation.
2021-08-19 14:47:12 -07:00
6c97782378
Timedout reads cancellation check removed in simulation.
2021-08-19 14:11:28 -07:00
02b3ed3ff1
Adding deadline option to rocksdb calls.
2021-08-19 14:11:28 -07:00
fbb393f998
Added readrange timeoout check and rocksdb read deadline option.
2021-08-19 14:11:28 -07:00
24ac173c95
Adding ReadRangeAction timeout, returning error and using timer_monotonic changes.
2021-08-19 14:11:28 -07:00
01e85610ab
Cancelling the timedout reads with rocksdb storage.
2021-08-19 14:11:28 -07:00
b8e2575f92
Update log directory to server log directory. ( #5396 )
2021-08-18 13:07:56 -07:00
adc466acb6
Fix key name
2021-07-30 13:27:43 -04:00
a26dbba66f
Add RocksDB metrics
2021-07-30 13:27:43 -04:00
0f99d449fc
Revert "Merge pull request #5194 from apple/revert-5144-rocksdb-in-simulation"
...
This reverts commit 8ecff8c730c8e18f99f0
2021-07-16 12:01:01 -04:00
b7b845dab8
Revert "RocksDB in simulation"
2021-07-15 16:41:33 -07:00
ed35df9a00
Add comment explaining coro threads
2021-07-13 14:56:23 -04:00
0613ff892f
Enable RocksDB in simulation
2021-07-13 12:25:11 -04:00
1feedb700b
Don't log key not found as an error
2021-07-09 13:07:06 -04:00
efdda3cd0e
Revert "Revert "Add support for running non-simulation tests on Joshua""
2021-06-15 14:15:02 -04:00
549cf0512b
Revert "Add support for running non-simulation tests on Joshua"
2021-06-14 19:23:56 -04:00
8af0affeff
Add support for running non-simulation tests on Joshua
2021-06-14 15:00:45 -04:00
25c3468a08
Estimate live SST file size and not live data size
2021-06-11 16:53:10 -04:00
5c2d7b6080
Create RangeResult type alias
2021-05-03 13:14:16 -07:00
df90cc89de
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-10 10:18:07 -08:00
2bb4f2e59f
Merge branch 'release-6.3-pre-format' into master-format
...
This merges release-6.3 branch right before it was fully formatted.
There were quite a few conflicts that are resolved here. CoroFlow had
a check for OOM errors introduced in 6.3, but didn't seem applicable in
the new implmentation which seems to use boost.
2021-03-10 09:37:41 -08:00
d2d6fcccfa
Merge pull request #4009 from Daniel-B-Smith/rocksdb-named-threads
...
Name the RocksDB background threads
2021-03-03 20:34:32 -08:00
346a4e3ecd
Merge branch 'release-6.3'
...
# Conflicts:
# fdbcli/fdbcli.actor.cpp
# fdbrpc/LoadBalance.actor.h
# fdbrpc/MultiInterface.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/masterserver.actor.cpp
2021-03-01 18:52:06 -08:00
179dea5a1b
Name the RocksDB background threads
2021-03-01 20:35:55 +00:00
0e9e53a163
Make the RocksDB init method idempotent
2021-03-01 17:20:41 +00:00
a8d08a8792
Merge branch 'release-6.3' into merge-release-6.3-into-master
2021-02-09 10:18:00 -08:00
aaf0a9aa7b
Merge branch 'release-6.3' into merge-release-6.3-into-master
...
# Conflicts:
# build/docker-compose.yaml
# cmake/ConfigureCompiler.cmake
# fdbclient/FileBackupAgent.actor.cpp
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/IAsyncFile.h
# fdbrpc/IRateControl.h
# fdbrpc/simulator.h
# fdbserver/KeyValueStoreSQLite.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbservice/ServiceBase.cpp
2021-02-08 12:58:34 -08:00
4c5079e4f6
Add option to prevent synchronous file delets on reads.
2021-02-03 21:57:55 +00:00
de700b83f2
Review fixes
2021-02-01 21:58:39 +00:00
He Liu