df90cc89de
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-10 10:18:07 -08:00
684048f449
FastRestore:Fix trace event overflow in traceRoleVersionBatchProgress
2020-09-06 21:05:16 -07:00
3ca1359e89
FastRestore:Simulate cpu busy and adjust simulation of overused memory usage
...
Increase the chance of cpu busy and memory overused
2020-08-26 10:07:49 -07:00
7e302b5910
FastRestore:Loader:Add scheduler to dispatch requests based on resource utilization and request priority
2020-08-15 21:21:51 -07:00
ad915e462e
Add knob FASTRESTORE_NOT_WRITE_DB to skip writting to DB
2020-07-30 10:17:17 -07:00
8cffd30951
FastRestore:Replace master word with controller
2020-07-16 16:37:13 -07:00
8f5ee1f4aa
Refine trace events for better perf diagnosis
2020-06-29 10:22:07 -07:00
82dfb5ce3f
FastRestore:Update process metrics for restore master
2020-06-28 12:37:04 -07:00
ecd2d8b239
FastRestore:Add counters for applier and disable FlowLock on applyStagingKeysBatch
2020-06-27 00:20:54 -07:00
5860a5b4db
FastRestore:Suppress or mute spammy trace events
2020-06-24 22:10:54 -07:00
68bcecd7d4
FastRestoreLoader:Clean batchData when a version batch finishes
...
Add sanity check for each loader phase to ensure an older version batch
will not be executed once a version batch has been marked as finished.
2020-05-04 11:31:42 -07:00
0ba1551116
FastRestore:Trace memory usage periodically
2020-05-04 11:20:53 -07:00
32c791a9b0
FastRestore:Fix:memory threshold is set in MB not Bytes
2020-04-30 18:51:25 -07:00
a81ec332a9
FastRestore:Fix:Master cannot throttle on in progress version batches when it release batches out of order in simulation
2020-04-04 17:34:26 -07:00
3e507e0515
FastRestore:Reenable isSchedulable
2020-02-28 15:08:45 -08:00
7c003bfa70
FastRestore:Disable isSchedulable for test
2020-02-28 14:27:45 -08:00
1be1145bc6
FastRestore:Fix corrupted trace event in FastRestoreVersionBatchProgressState
2020-02-28 00:23:58 -08:00
89b121ae25
FastRestore:Fix duplicate type in traceRoleVersionBatchProgress
2020-02-28 00:08:01 -08:00
eaf340652f
FastRestore:ensure setVersionBatchState after version batch is initialized
2020-02-27 23:50:09 -08:00
22b34bc609
FastRestore:getVersionBatchState can be called before version batch is initialized
2020-02-27 23:45:48 -08:00
f4cd0ef74f
FastRestore:Apply clang-format
2020-02-27 20:59:56 -08:00
d77177367c
FastRestore:Track each ongoing version batch progress state for applier and loader roles
2020-02-27 19:47:22 -08:00
2657d41bb2
FastRestore:Add debug msg when memory is over threshold
2020-02-27 18:32:11 -08:00
fbb6e8f39d
FastRestore:Create low memory situation in simulation on purpose
2020-02-26 14:54:38 -08:00
ca726fc68e
FastRestore:Introduce OOM protection
...
An actor is schedulable to run if the current worker has enough resourc, i.e.,
the worker's memory usage is below the threshold;
Exception: If the actor is working on the current version batch, we have to schedule
the actor to run to avoid dead-lock.
Future: When we release the actors that are blocked by memory usage, we should release them
in increasing order of their version batch.
2020-02-26 14:09:18 -08:00
dbce1e9974
FastRestore:Applier:Add metrics counter and proc counter
2020-02-10 16:38:26 -08:00
141609e80a
FastRestore:Improve code style and fix typos
2020-01-27 18:13:14 -08:00
e011f39829
FastRestore:Add sanity check and trace events
2020-01-23 16:03:41 -08:00
022783b449
Start batches in reverse order for testings and code cleanup
2020-01-21 14:49:40 -08:00
4ac92d223b
Cleanup batch buffer for each restore request
2020-01-21 14:49:36 -08:00
bfbf2164c4
FastRestore:Applier buffer data for multiple batches
2020-01-17 17:01:01 -08:00
630c29d160
FastRestore:resolve review comments
...
1) wait on whenAtLeast;
2) Put BigEndian64 into the function call and the decoder to prevent
future people from making the same mistake.
2019-11-11 17:00:16 -08:00
63359bfc8b
FastRestore:handleInitVersionBatchRequest:Ensure exact once execution
...
Also increase the test workload for BackupAndParallelRestoreWithAtomicOp test
2019-11-03 17:26:13 -08:00
2383c29123
FastRestore:Use reference for handleInitVersionBatchRequest func
2019-10-24 13:54:44 -07:00
7903b47b82
FastRestore:Remove unnecessary return
2019-10-24 13:09:24 -07:00
c53f817c5e
FastRestore:Convert handleInitVersionBatchRequest to plain func
2019-10-24 13:06:50 -07:00
60d26ff5d7
FastRestore:Resolve review comments
2019-10-24 12:52:12 -07:00
bae0c907a6
FastRestore:Convert unnecessary actor function to plain function
2019-10-23 15:10:34 -07:00
e9a48cb63b
FastRestore:Fix bug in handleInitVersionBatchRequest
...
We should unconditionally resetPerVersionBatch()
2019-10-19 17:40:50 -07:00
3b54363780
FastRestore:Apply Clang-format
2019-08-01 18:09:12 -07:00
f1741aa90d
FastRestore: Resolve review comments
...
1) Do not keep restore role data (e.g., masterData) in restore worker;
2) Change function parameter list by only passing in the needed variables in role data;
3) Remove unneccessary files vector from masterData;
4) Change typos in comments and some functions name.
2019-07-24 17:51:53 -07:00
701676dbd2
FastRestore:Refactor code and add missing files
...
Add RestoreWorker.actor.cpp and RestoreWorkerInterface.actor.h back.
2019-06-18 09:54:27 -07:00
022b555b69
FastRestore:Fix bug in finish restore
...
RestoreMaster may not receive all acks. for the last command, i.e., finishRestore,
because RestoreLoaders and RestoreAppliers exit immediately after sending the ack.
If the ack is lost, it will not be resent.
This commit also removes some unneeded code.
This commit passes 50k random tests without errors.
2019-06-05 20:07:18 -07:00
3fcb6ec0a1
FastRestore:Refactor RestoreLoader and fix bugs
...
Refactor RestoreLoader code and
Fix a bug in notifying restore finish.
2019-06-04 21:53:31 -07:00
477fd152c0
FastRestore:Refactor code
...
1) Use the runRYWTransaction for simple DB access
2) Replace some printf with TraceEvent
3) Remove printf not used in debugging
4) Avoid wait inside the condition in loop-choose-when for
the core routine of restore worker, loader and applier.
5) Rename Restore.actor.cpp to RestoreWorker.actor.cpp since
the file only has functionalities related to restore worker.
Passed correctness test
2019-06-04 11:22:47 -07:00
9e1216af1c
FastRestore:Remove CMDUID
2019-05-29 13:48:04 -07:00
4f484a2a5d
FastRestore:Refactor out the use of cmdID and other non-must functions
2019-05-29 13:26:17 -07:00
3eadb31798
FastRestore:Resolve two major reveiw comments
...
1) Add sendBatchRequests and getBatchReplies
sendBatchRequests is a generic actor to send requests without
processing replies.
getBatchReplies is similar to sendBatchRequests expect that
it returns the reply to caller.
2) Share applier interface to loaders by using RequestStream,
instead of using DB.
Create RestoreSysInfo struct, similar purpose as DBInfo, for
the restore system information that are shared among restore workers.
2019-05-24 21:53:21 -07:00
12817af03f
FastRestore:Fix CMake compiling errors
2019-05-16 20:01:43 -07:00
32c030b7d6
FastRestore: Clear RestoreRole key in DB at finishRestore
...
This commit is the one that passes correctness tests after
refactoring the fast restore.
2019-05-11 22:25:36 -07:00
FDB Formatster