df90cc89de
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-10 10:18:07 -08:00
862336de8f
Merge branch 'master' into mengxu/merge-to-master-PR
2020-09-24 17:06:00 -07:00
002b1bec4c
FastRestore:Control write traffic at each applier
...
Controller assigns each applier a write rate.
Applier keeps the write-rate worth of transactions outstanding to DB.
This is to avoid heavily overloading DB while still keep enough
traffic to DB to get a good write throughput.
2020-09-22 08:14:28 -07:00
12edadd059
Merge branch 'release-6.3'
...
# Conflicts:
# CMakeLists.txt
# fdbclient/Knobs.cpp
# fdbclient/MasterProxyInterface.h
# fdbrpc/simulator.h
# fdbserver/MasterProxyServer.actor.cpp
# tests/fast/CycleAndLock.txt
# tests/fast/TxnStateStoreCycleTest.txt
# tests/fast/VersionStamp.txt
# tests/slow/ParallelRestoreOldBackupApiCorrectnessAtomicRestore.txt
# tests/slow/ParallelRestoreOldBackupCorrectnessCycle.txt
# versions.target
2020-08-31 19:33:34 -07:00
bd7c07436b
FastRestore:Add batchIndex to RestoreAsset for better performance tracking
2020-08-25 09:34:18 -07:00
6729b4143b
FastRestore:Requests:Add const for toString function
2020-08-18 16:05:23 -07:00
7b7490efe7
FastRestore:Debug trace for seg fault
2020-08-17 20:34:33 -07:00
7e302b5910
FastRestore:Loader:Add scheduler to dispatch requests based on resource utilization and request priority
2020-08-15 21:21:51 -07:00
3f82c815f9
changed another file identifier
2020-08-11 22:53:34 -07:00
610e1e2525
change file identifier
2020-08-11 22:51:03 -07:00
2f52c5f79b
Merge branch 'release-6.3'
...
# Conflicts:
# fdbserver/RestoreLoader.actor.cpp
2020-08-11 22:45:43 -07:00
879998d01b
FR:fix compilation error
2020-08-04 23:04:38 -07:00
7992cef025
FR:Fix sample network pkg can be too big
2020-08-04 22:35:21 -07:00
e9adec421d
FR:Add controller interface and send to roles
2020-08-03 15:06:15 -07:00
abaae40453
Modified ComposedIdentifier to prevent file identifier conflicts at
...
compile-time
2020-07-20 18:52:44 -07:00
8cffd30951
FastRestore:Replace master word with controller
2020-07-16 16:37:13 -07:00
b09dddc07e
Merge branch 'release-6.2' into merge-release-6.2-into-release-6.3
...
# Conflicts:
# cmake/ConfigureCompiler.cmake
# documentation/sphinx/source/downloads.rst
# fdbrpc/FlowTransport.actor.cpp
# fdbrpc/fdbrpc.vcxproj
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogSystemPeekCursor.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/Status.actor.cpp
# fdbserver/storageserver.actor.cpp
# flow/flow.vcxproj
2020-07-10 15:06:34 -07:00
dd5c1ec416
FastRestore:Add debug trace
...
The bug behavior is:
In transformDatabaseContents(), we write all new kvs into the DB in 100 keys batch.
We immediately read these keys out.
There are 100 keys missing.
2020-06-22 19:47:21 -07:00
3d6f69c8e2
FastRestore:addPrefix:Transform must clear both orignal and transformed range
...
Otherwise, anything left in the range can interfer with the result.
2020-06-21 22:18:12 -07:00
2fcef90cbc
FastRestore:Add addPrefix and removePrefix feature
2020-06-18 11:58:42 -07:00
bbdd451fb6
FastRestore:Add priorities for loader and applier endpoints
...
Endpoint for later restore step should have higher priority for
better performance: later step will not be blocked by the
next version batch's earlier steps.
2020-06-16 22:48:32 -07:00
72ce997d22
explicitly versioned every key in systemData, so we only will update the associated protocolVersion when the serialization actually changes
2020-05-22 16:35:01 -07:00
7b5d43da9c
FastRestore:Remove unused field in RestoreRequest
2020-05-03 20:59:47 -07:00
9b2f6d5c13
FastRestore:Loader:Enable sending message out of order
2020-04-22 08:57:11 -07:00
d21da5065a
FastRestore:Loader:Merge MutationsVec and LogMessageVersionVec into VersionedMutationsVec
...
Remove the actor that sends one mutation message batch in the previous commit,
because that actor no longer reduces the code complexity.
2020-04-21 22:05:34 -07:00
d6c1baa784
FastRestore:Filter out log mutations whose version is smaller than range mutation version
2020-04-15 19:45:03 -07:00
dbc9c23193
FastRestore:Loader:Send mutations at different versions in the same message to appliers
...
This increases the bandwidth sent from loaders to appliers.
2020-04-12 10:46:58 -07:00
e5b2cd81d5
FastRestore:Cleanup debug code
2020-04-07 15:56:44 -07:00
432c99afd0
FastRestore:Applier:Keep incompleteStagingKeys content before values are applied to DB
...
To avoid the incompleteStagingKeys is cleared before getAndComputeStagingKeys() finish using it.
2020-04-04 22:38:04 -07:00
ab0b59b0c3
Add subsequence number to restore loader & applier
...
The subsequence number is needed so that mutations of the same commit version
number, but from different partitioned logs can be correctly reassembled in
order.
For old backup files, the sub number is always 0. For partitioned mutation
logs, the actual sub number is used. For range files, the sub number is always
0.
2020-03-20 20:13:38 -07:00
88ad28e576
Integrate parallel restore with partitioned logs
...
In parallel restore, use new getPartitionedRestoreSet() to get a set containing
partitioned mutation logs. The loader uses a new parser to extract mutations
from partitioned logs.
TODO: fix unable to restore errors.
2020-03-20 20:13:38 -07:00
ca726fc68e
FastRestore:Introduce OOM protection
...
An actor is schedulable to run if the current worker has enough resourc, i.e.,
the worker's memory usage is below the threshold;
Exception: If the actor is working on the current version batch, we have to schedule
the actor to run to avoid dead-lock.
Future: When we release the actors that are blocked by memory usage, we should release them
in increasing order of their version batch.
2020-02-26 14:09:18 -08:00
31a6ec34b7
Merge branch 'master' into mengxu/fast-restore-agent-PR
2020-02-18 16:17:59 -08:00
c603b20e7e
FastRestore:Resolve review comments
2020-02-18 14:08:27 -08:00
9c2046b11b
FastRestore:Minic fdbd to monitor coordintors
...
Before we start a fdb restore process.
2020-02-03 14:48:31 -08:00
cab9d51e06
Merge branch 'master' into mengxu/fast-restore-pipeline-PR
2020-01-27 18:16:26 -08:00
141609e80a
FastRestore:Improve code style and fix typos
2020-01-27 18:13:14 -08:00
009fcdeb16
FastRestore:Sanity check each restore asset is processed exactly once
2020-01-21 17:17:45 -08:00
4ac92d223b
Cleanup batch buffer for each restore request
2020-01-21 14:49:36 -08:00
d69bd2f661
FastRestore:Loader buffer data for multiple batches
2020-01-17 17:01:06 -08:00
bfbf2164c4
FastRestore:Applier buffer data for multiple batches
2020-01-17 17:01:01 -08:00
f436ea806e
FastRestore:Resolve review comment
...
1) Sort logfiles by endVersion
2) Exit program early when restore will not succeed
3) Do not increase nextVersion unncessarily when
calculate version batches.
4) Change assert condition that ensures progress in
calculating version batches.
2020-01-13 14:08:27 -08:00
a2b26906e8
FastRestore:Filter out empty files before distributing workload
...
and clean up unused code
2020-01-07 17:01:53 -08:00
c29e380076
FastRestore:Remove prevVersion from LoadingParam
2020-01-07 14:59:17 -08:00
9df02512ab
FastRestore:Apply clang-format
2020-01-07 11:50:32 -08:00
67e913c3d5
Change LoadingParam struct and endVersion definition
...
1) Remove endVersion field because it has been included in RestoreAsset;
2) Ensure endVersion in VersionBatch and RestoreAsset is always exclusive;
3) Revise ASSERT in laoder and applier in situations when the dummy commit version
is endVersion, to avoid false positive ASSERT failure.
2020-01-07 11:48:03 -08:00
c3f8f3b445
FastRestore:Build VersionBatch less than threshold size
2020-01-07 11:46:56 -08:00
8d6f511816
FastRestore:Resolve review comment
...
Filter out range mutations that do not overlap with the restore range.
Small changes on format.
2019-12-22 20:09:10 -08:00
61b29de3ce
FastRestore:Self code review
...
Clean up commented code;
Add sanity check.
2019-12-20 22:24:34 -08:00
ddcf3fdd80
FastRestore:Apply clang format
2019-12-20 22:00:36 -08:00
FDB Formatster