wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Some formatting changes, e.g., remove extra spaces

This commit is contained in:
Jingyu Zhou 2019-12-16 15:55:51 -08:00
parent a98b466f98
commit 7debde6fda
1 changed files with 28 additions and 30 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
design/recovery-internals.md
View File

@ -72,7 +72,7 @@ This phase locks the coordinated state (cstate) to make sure there is only one m
Recall that `ServerDBInfo` has master's interface and is propogated by CC to every process in a cluster. The current running tLogs can use the master interface in its `ServerDBInfo` to send itself's interface to master.
Master simply waits on receiving the `TLogRejoinRequest` streams: for each tLog’s interface received, the master compares the interface ID with the tLog ID read from cstate. Once the master collects enough old tLog interfaces, it will use the interfaces to lock those tLogs.
The logic of collecting tLogs’ interfaces is implemented in `trackRejoins()` function.
The logic of locking the tLogs is implemented in `epochEnd()` function in TagPartitionedLogSystems.actor.cpp.
The logic of locking the tLogs is implemented in `epochEnd()` function in [TagPartitionedLogSystems.actor.cpp](https://github.com/apple/foundationdb/blob/master/fdbserver/TagPartitionedLogSystem.actor.cpp).
Once we lock the cstate, we bump the `recoveryCount` by 1 and write the `recoveryCount` to the cstate. Each tLog in a recovery attempt records the `recoveryCount` and monitors the change of the variable. If the `recoveryCount` increases, becoming larger than the recorded value, the tLog will terminate itself. This mechanism makes sure that when multiple recovery attempts happen concurrently, only tLogs in the most recent recovery will be running. tLogs in other recovery attempts can release their memory earlier, reducing the memory pressure during recovery. This is an important memory optimization before shared tLogs, which allows tLogs in different generations to share the same memory, is introduced.
@ -87,7 +87,7 @@ Master interface is stored in `serverDBInfo`. Once the CC recruits the master, i
* tLog monitors `serverDBInfo` change and sends its interface to the new master;
* Data distributor (DD) and ratekeeper rejoin themselves to CC because they are no longer a part of the recovery process;
* Data distributor (DD) and Ratekeeper rejoin themselves to CC because they are no longer a part of the recovery process (they are moved out of the master process since 6.2 release);
* Storage server (SS) does not rejoin. It waits for the tLogs to be ready and commit their interfaces into database with a special transaction.
@ -115,17 +115,15 @@ Two situations may invalidate the calculated knownCommittedVersion:
* Situation 1: Too many tLogs in the previous generation permanently died, say due to hardware failure. If force recovery is allowed by system administrator, the master can choose to force recovery, which can cause data loss; otherwise, to unblock the recovery, system administrator has to bring up those died tLogs, for example by copying their files onto new hardware.
* Situation 2: A tLog may die after it reports alive to the master in the RECRUITING phase. This may cause the knownCommittedVersion calculated by the master in this phase to no longer be valid in the next phases. When this happens, the master will detect it, terminate the current recovery, and start a new recovery.
* Situation 2: A tLog may die after it reports alive to the master in the RECRUITING phase. This may cause the `knownCommittedVersion` calculated by the master in this phase to no longer be valid in the next phases. When this happens, the master will detect it, terminate the current recovery, and start a new recovery.
Once we have a knownCommittedVersion, the master will reconstruct the transaction state store (txnStateStore) by peeking the txnStateTag in oldLogSystem.
Once we have a `knownCommittedVersion`, the master will reconstruct the transaction state store (txnStateStore) by peeking the txnStateTag in oldLogSystem.
Recall that the txnStateStore includes the transaction system’s configuration, such as the assignment of shards to SS and to tLogs and that the txnStateStore was durable on disk in the oldLogSystem.
Once we get the txnStateStore, we know the configuration of the transaction system, such as the number of proxies. The master then can ask the CC to recruit roles for the new generation in the `recruitEverything()` function. Those recruited roles includes proxies, tLogs and seed SSes, which are the storage servers created for an empty database in the first generation to host the first shard and serve as the starting point of the bootstrap process to recruit more SSes. Once all roles are recruited, the master starts a new epoch in `newEpoch()`.
At this point, we have recovered the txnStateStore, recruited new proxies and tLogs, and copied data from old tLogs to new tLogs. We have a working transaction system in the new generation now.
### Where can the recovery get stuck in this phase?
Recovery can get stuck at the following two steps:
@ -168,7 +166,7 @@ Coordinators store the transaction systems’ information. The master needs to w
The master only needs to write the new tLogs to a quorum of coordinators for a running cluster. The only time the master has to write all coordinators is when creating a brand new database.
Once the cstate is written, the master sets the cstateUpdated promise and moves to the ACCEPTING_COMMITS phase.
Once the cstate is written, the master sets the `cstateUpdated` promise and moves to the ACCEPTING_COMMITS phase.
The cstate update is done in `trackTlogRecovery()` actor.