9f18146f4a
FastRestore:Loader:Free memory after send
2020-04-25 17:59:33 -07:00
8569f1a93f
FastRestore:Loader:Use shallow copy for single key mutations when send mutation messages to appliers
2020-04-25 17:30:01 -07:00
96855d9b47
FastRestore:Loader:Enable sending mutation messages out of order
2020-04-25 17:21:17 -07:00
70ae22e72f
FastRestore:Loader:Deep copy splitted mutation
...
otherwise, the mutation buffer will have an invalid address for MutationRef
when it was sent to applier.
2020-04-24 10:31:30 -07:00
f073049865
FastRestore:Revise trace events to be descriptive
...
Revert changes that send mutations to appliers out of order
2020-04-24 10:31:08 -07:00
38193a3866
Merge branch 'master' into mengxu/fr-code-improvement-PR
2020-04-22 10:51:33 -07:00
9b2f6d5c13
FastRestore:Loader:Enable sending message out of order
2020-04-22 08:57:11 -07:00
d85a39df69
FastRestore:Loader:Make requests no longer state varialbe
2020-04-21 22:21:59 -07:00
d21da5065a
FastRestore:Loader:Merge MutationsVec and LogMessageVersionVec into VersionedMutationsVec
...
Remove the actor that sends one mutation message batch in the previous commit,
because that actor no longer reduces the code complexity.
2020-04-21 22:05:34 -07:00
6909f0b8fc
Remove decodeRangeFileBlock from parallel restore
...
Reuse the one from fileBackup namespace.
2020-04-21 13:42:24 -07:00
9ade63a685
FastRestore:Loader:Add sendOneMutationBatchToAppliers actor
2020-04-21 11:36:21 -07:00
00188c3157
Merge branch 'master' into mengxu/fr-code-improvement-PR
2020-04-19 11:52:42 -07:00
d6021132a8
FastRestore:Merge mutationMap and mutationPartMap for efficiency
2020-04-19 11:52:17 -07:00
719eda9421
FastRestore:Add an assertion in handleRestoreSysInfoRequest
...
as suggested in code review.
2020-04-18 22:42:46 -07:00
10a6461d13
FastRestore:Change __inline__ to inline
...
__inline__ is compiler specific while inline is the standard keyword
2020-04-17 22:31:44 -07:00
916d361587
BackupAndParallelRestoreCorrectness:Remove unnecessary checking optional variable
2020-04-17 18:32:14 -07:00
c8d049d0bb
FastRestore:Loader:Add counter oldLogMutations
2020-04-17 15:21:59 -07:00
d6c1baa784
FastRestore:Filter out log mutations whose version is smaller than range mutation version
2020-04-15 19:45:03 -07:00
acdf6816b8
FastRestore:Loader:Fix:Last version in each RA is forgotten
2020-04-13 17:28:21 -07:00
061bcd2fb4
FastRestore:Replace typeString with safe getTypeString func
...
Also fix compilation error in previous commit
2020-04-13 15:15:54 -07:00
56cdd18224
FastRestore:Loader:Allow sending not all mutations at a version to appliers
2020-04-13 13:59:21 -07:00
68adbef5bf
FastRestore:Loader:Send multiple versions of mutations in a message to appliers
2020-04-12 11:18:41 -07:00
dbc9c23193
FastRestore:Loader:Send mutations at different versions in the same message to appliers
...
This increases the bandwidth sent from loaders to appliers.
2020-04-12 10:46:58 -07:00
2325ab209f
FastRestore:Applier:Avoid extra copy in getAndComputeStagingKeys
2020-04-08 12:22:08 -07:00
da7249ed1c
FastRestore:Minor revision based on review comments
2020-04-02 11:15:22 -07:00
6bce67ca75
FastRestore:Apply clang-format
2020-04-01 21:27:54 -07:00
33c4be9c42
Improve debug message for debug mutations
2020-03-31 16:00:51 -07:00
e286f316b9
Increase generated key length for splitMutation unit test
2020-03-31 14:27:07 -07:00
b7da76223c
Fix a tricky splitMutation bug
...
splitMutation result may include the end key of a clearrange mutation
2020-03-31 13:38:58 -07:00
ccbbdc4ba4
Unit test:Verify splitMutation by comparing with intersectingRanges result
2020-03-31 12:13:02 -07:00
8a30526336
FastRestore:Remove commented assertion
2020-03-28 13:11:32 -07:00
404a3e2619
FastRestore:Loader:Remove sanity chech for the order of sending log and range mutations
2020-03-27 23:36:13 -07:00
97f8e46388
Sanity check subversion for log mutations
2020-03-27 13:07:08 -07:00
99f4ef6e0c
Fix restore loader to handle mutation sub number
...
For old backup format, give them a sub sequence number starting from 0 for each
commit version.
2020-03-26 13:04:00 -07:00
4bdb32be14
Batch sending all mutations of a version from RestoreLoader
...
This optimization is to reduce the number of messages sent from loader to
applier, which was unintentionally done when introducing sub sequence numbers
for mutations.
2020-03-20 20:15:09 -07:00
799f0b4b0e
Small code refactor
2020-03-20 20:15:09 -07:00
e40f937d3a
Fix missing mutations in splitMutation
...
When a range mutation is larger than the last split point, this mutation can
become missing in the RestoreLoader, which is fixed in this commit.
2020-03-20 20:15:09 -07:00
fe51ba3d16
Give maximum subsequence number for snapshot mutations
...
This is needed so that mutations in partitioned logs are applied first and
snapshot mutations are applied later for the same commit version.
2020-03-20 20:15:09 -07:00
2eac17b553
StagingKey can add out-of-order mutations
...
For partitioned logs, mutations of the same version may be sent to applier
out-of-order. If one loader advances to the next version, an applier may
receive later version mutations for different loaders. So, dropping of early
mutations is wrong.
2020-03-20 20:13:38 -07:00
ab0b59b0c3
Add subsequence number to restore loader & applier
...
The subsequence number is needed so that mutations of the same commit version
number, but from different partitioned logs can be correctly reassembled in
order.
For old backup files, the sub number is always 0. For partitioned mutation
logs, the actual sub number is used. For range files, the sub number is always
0.
2020-03-20 20:13:38 -07:00
6b9b93314e
Check block padding is \0xff for new mutation logs
2020-03-20 20:13:38 -07:00
35aafefb89
Consolidate StringRefReader classes
...
Fix a compiler error of unused variable too.
2020-03-20 20:13:38 -07:00
88ad28e576
Integrate parallel restore with partitioned logs
...
In parallel restore, use new getPartitionedRestoreSet() to get a set containing
partitioned mutation logs. The loader uses a new parser to extract mutations
from partitioned logs.
TODO: fix unable to restore errors.
2020-03-20 20:13:38 -07:00
e15015ee6c
Add mutation log version names
...
I.e., BACKUP_AGENT_MLOG_VERSION for 2001 and PARTITIONED_MLOG_VERSION for 4110.
2020-03-20 20:13:38 -07:00
d3071409c5
FastRestore:Add comment for integrating with new backup format
2020-03-20 20:13:38 -07:00
2520e8d44c
FastRestore:Use more concise code as suggested in review
2020-03-01 22:32:36 -08:00
62b9043ff6
FastRestore:DB can be destroyed before master unlock it in simulation
...
Because retore roles run as workload in simulation,
they do not know when DB is destroyed by the backup and restore test workload.
So if DB is destroyed earlier than restore master unlocks DB, which is rare,
restore master should abort the unlocking DB step.
2020-02-28 14:25:58 -08:00
fbb6e8f39d
FastRestore:Create low memory situation in simulation on purpose
2020-02-26 14:54:38 -08:00
06495b90ae
FastRestore:Loader:Use isSchedulable to guard OOM
...
And trigger delayed actors that are blocked on memory to recheck memory.
2020-02-26 14:35:05 -08:00
ca726fc68e
FastRestore:Introduce OOM protection
...
An actor is schedulable to run if the current worker has enough resourc, i.e.,
the worker's memory usage is below the threshold;
Exception: If the actor is working on the current version batch, we have to schedule
the actor to run to avoid dead-lock.
Future: When we release the actors that are blocked by memory usage, we should release them
in increasing order of their version batch.
2020-02-26 14:09:18 -08:00
Meng Xu