wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
15,395 Commits 34 Branches 304 Tags 265 MiB
Commit Graph

56 Commits

Author SHA1 Message Date
sfc-gh-tclinkenbeard 594e8944ae Move RestoreWorkerInterface into fdbserver 2021-05-30 11:51:47 -07:00
sfc-gh-tclinkenbeard eab75d4ee1 Make enums automatically binary serializable 2020-11-14 19:22:04 -08:00
Evan Tschannen 2f52c5f79b Merge branch 'release-6.3' 
# Conflicts:
#	fdbserver/RestoreLoader.actor.cpp
 2020-08-11 22:45:43 -07:00
Meng Xu 879998d01b FR:fix compilation error 2020-08-04 23:04:38 -07:00
Meng Xu 7992cef025 FR:Fix sample network pkg can be too big 2020-08-04 22:35:21 -07:00
sfc-gh-tclinkenbeard abaae40453 Modified ComposedIdentifier to prevent file identifier conflicts at 
compile-time
 2020-07-20 18:52:44 -07:00
Meng Xu 8cffd30951 FastRestore:Replace master word with controller 2020-07-16 16:37:13 -07:00
A.J. Beamon b09dddc07e Merge branch 'release-6.2' into merge-release-6.2-into-release-6.3 
# Conflicts:
#	cmake/ConfigureCompiler.cmake
#	documentation/sphinx/source/downloads.rst
#	fdbrpc/FlowTransport.actor.cpp
#	fdbrpc/fdbrpc.vcxproj
#	fdbserver/DataDistributionQueue.actor.cpp
#	fdbserver/Knobs.cpp
#	fdbserver/Knobs.h
#	fdbserver/LogSystemPeekCursor.actor.cpp
#	fdbserver/MasterProxyServer.actor.cpp
#	fdbserver/Status.actor.cpp
#	fdbserver/storageserver.actor.cpp
#	flow/flow.vcxproj
 2020-07-10 15:06:34 -07:00
Meng Xu 5860a5b4db FastRestore:Suppress or mute spammy trace events 2020-06-24 22:10:54 -07:00
Meng Xu 4e27fd34e5 Refactor transformDatabaseContents into RestoreCommon 
Prepare to enable addPrefix for atomicRestore
 2020-06-23 14:33:13 -07:00
Meng Xu 142fa5477b Add SevFRTestInfo trace level to control debug info in BackupAndParallelRestoreCorrectness 2020-06-23 10:49:05 -07:00
Meng Xu 9318ec033d FastRestore:FixTest:Remove unnecessary read 2020-06-20 20:07:18 -07:00
Meng Xu 8d59568dc7 FastRestore:Fix:Forgot to assign transformed key to mutations 
The earlier change in restore logic did not really take effect.
This commit fix it and confirm the addPrefix test does cause expected
failures if we do not transform the keys at the end of the test.
 2020-06-18 17:40:16 -07:00
Meng Xu b506ff3af9 Fix merge conflict on missing struct VersionedMutation 2020-04-29 22:35:54 -07:00
Meng Xu a0d67cac16 Merge branch 'master' into mengxu/fr-code-improvement-PR 2020-04-29 21:07:33 -07:00
Meng Xu f073049865 FastRestore:Revise trace events to be descriptive 
Revert changes that send mutations to appliers out of order
 2020-04-24 10:31:08 -07:00
Meng Xu 46ec766cab FastRestore:Disable debug trace 2020-04-22 16:11:46 -07:00
Meng Xu 38193a3866 Merge branch 'master' into mengxu/fr-code-improvement-PR 2020-04-22 10:51:33 -07:00
Meng Xu d21da5065a FastRestore:Loader:Merge MutationsVec and LogMessageVersionVec into VersionedMutationsVec 
Remove the actor that sends one mutation message batch in the previous commit,
because that actor no longer reduces the code complexity.
 2020-04-21 22:05:34 -07:00
Meng Xu dbc9c23193 FastRestore:Loader:Send mutations at different versions in the same message to appliers 
This increases the bandwidth sent from loaders to appliers.
 2020-04-12 10:46:58 -07:00
Meng Xu 5ebafdb94c FastRestore:Apply clang-format to changes 2020-04-07 15:57:03 -07:00
Meng Xu 536e65cd76 FastRestore:Introduce debugFRMutation for debug keys 2020-04-05 15:00:36 -07:00
Meng Xu a81ec332a9 FastRestore:Fix:Master cannot throttle on in progress version batches when it release batches out of order in simulation 2020-04-04 17:34:26 -07:00
Jingyu Zhou ab0b59b0c3 Add subsequence number to restore loader & applier 
The subsequence number is needed so that mutations of the same commit version
number, but from different partitioned logs can be correctly reassembled in
order.

For old backup files, the sub number is always 0. For partitioned mutation
logs, the actual sub number is used. For range files, the sub number is always
0.
 2020-03-20 20:13:38 -07:00
Meng Xu dc848f4297 FastRestore:Disable verbose trace for perf. measurement 2020-02-06 20:50:23 -08:00
Meng Xu b04e98771e FastRestore:Replace FastRestoreOpConfig with Knobs 
And randomize value for the rest of knobs
 2020-01-24 14:24:34 -08:00
Meng Xu 009fcdeb16 FastRestore:Sanity check each restore asset is processed exactly once 2020-01-21 17:17:45 -08:00
Meng Xu f436ea806e FastRestore:Resolve review comment 
1) Sort logfiles by endVersion

2) Exit program early when restore will not succeed

3) Do not increase nextVersion unncessarily when
calculate version batches.

4) Change assert condition that ensures progress in
calculating version batches.
 2020-01-13 14:08:27 -08:00
Meng Xu dba85d28fc FastRestore:Cosmetic revision 2020-01-08 10:53:53 -08:00
Meng Xu c3f8f3b445 FastRestore:Build VersionBatch less than threshold size 2020-01-07 11:46:56 -08:00
Meng Xu e98b2a0d1c FastRestore:Introduce RestoreAsset 2019-12-20 18:00:10 -08:00
Meng Xu 39a4f2372f Change FASTRESTORE_SAMPLING_PERCENT to 0 to 100 2019-12-04 21:26:27 -08:00
Meng Xu c6b36dbffb FastRestore:Sampling:Resolve review comments 2019-12-04 17:35:11 -08:00
Meng Xu 2b987d1945 FastRestore:typedef Standalone<VectorRef<MutationRef>> MutationsVec 2019-12-04 11:39:55 -08:00
Meng Xu 592f4c0fc4 FastRestore:Remove RestoreSetApplierKeyRangeVectorRequest 2019-11-12 17:59:11 -08:00
Meng Xu e7210fe842 Trace:Resolve review comments and add SevVerbose level 2019-11-05 09:42:29 -08:00
Meng Xu c4d1e6e1a9 Trace:Severity:Include SevNoInfo to mute trace 
Define SevFRMutationInfo to trace mutations in restore.
 2019-11-04 16:18:40 -08:00
Meng Xu d160810662 FastRestore:Resolve review comments 2019-09-04 16:48:43 -07:00
Meng Xu 3b54363780 FastRestore:Apply Clang-format 2019-08-01 18:09:12 -07:00
Meng Xu c2c2536de2 FastRestore:Resolve compile erroers due to conflict with master 2019-08-01 13:32:24 -07:00
Meng Xu 45083edf74 Merge branch 'master' into mengxu/performant-restore-PR 
Fix conflicts as well.
 2019-07-25 10:46:11 -07:00
Meng Xu 022b555b69 FastRestore:Fix bug in finish restore 
RestoreMaster may not receive all acks. for the last command, i.e., finishRestore,
because RestoreLoaders and RestoreAppliers exit immediately after sending the ack.
If the ack is lost, it will not be resent.

This commit also removes some unneeded code.
This commit passes 50k random tests without errors.
 2019-06-05 20:07:18 -07:00
Meng Xu 3fcb6ec0a1 FastRestore:Refactor RestoreLoader and fix bugs 
Refactor RestoreLoader code and
Fix a bug in notifying restore finish.
 2019-06-04 21:53:31 -07:00
Meng Xu 477fd152c0 FastRestore:Refactor code 
1) Use the runRYWTransaction for simple DB access
2) Replace some printf with TraceEvent
3) Remove printf not used in debugging
4) Avoid wait inside the condition in loop-choose-when for
   the core routine of restore worker, loader and applier.
5) Rename Restore.actor.cpp to RestoreWorker.actor.cpp since
   the file only has functionalities related to restore worker.

Passed correctness test
 2019-06-04 11:22:47 -07:00
Meng Xu 67f5c8b493 FastRestore:Remove performance status 
Remove the non-functional code to reduce the code review size.
 2019-05-30 20:24:40 -07:00
Meng Xu 9e1216af1c FastRestore:Remove CMDUID 2019-05-29 13:48:04 -07:00
Meng Xu 3eadb31798 FastRestore:Resolve two major reveiw comments 
1) Add sendBatchRequests and getBatchReplies

sendBatchRequests is a generic actor to send requests without
processing replies.
getBatchReplies is similar to sendBatchRequests expect that
it returns the reply to caller.

2) Share applier interface to loaders by using RequestStream,
instead of using DB.
   Create RestoreSysInfo struct, similar purpose as DBInfo, for
 the restore system information that are shared among restore workers.
 2019-05-24 21:53:21 -07:00
Meng Xu 9ea83e0f3c FastRestore:Remove dbprintf 2019-05-17 17:34:42 -07:00
Meng Xu a7f1b69804 FastRestore:Add dbprintf 2019-05-15 21:15:15 -07:00
Meng Xu d8658a581f FastRestore:Change parameter for performance test 2019-05-15 19:53:14 -07:00