319 lines
12 KiB
Plaintext
319 lines
12 KiB
Plaintext
The cluster MD is a shared-device RAID for a cluster.
|
|
|
|
|
|
1. On-disk format
|
|
|
|
Separate write-intent-bitmaps are used for each cluster node.
|
|
The bitmaps record all writes that may have been started on that node,
|
|
and may not yet have finished. The on-disk layout is:
|
|
|
|
0 4k 8k 12k
|
|
-------------------------------------------------------------------
|
|
| idle | md super | bm super [0] + bits |
|
|
| bm bits[0, contd] | bm super[1] + bits | bm bits[1, contd] |
|
|
| bm super[2] + bits | bm bits [2, contd] | bm super[3] + bits |
|
|
| bm bits [3, contd] | | |
|
|
|
|
During "normal" functioning we assume the filesystem ensures that only
|
|
one node writes to any given block at a time, so a write request will
|
|
|
|
- set the appropriate bit (if not already set)
|
|
- commit the write to all mirrors
|
|
- schedule the bit to be cleared after a timeout.
|
|
|
|
Reads are just handled normally. It is up to the filesystem to ensure
|
|
one node doesn't read from a location where another node (or the same
|
|
node) is writing.
|
|
|
|
|
|
2. DLM Locks for management
|
|
|
|
There are three groups of locks for managing the device:
|
|
|
|
2.1 Bitmap lock resource (bm_lockres)
|
|
|
|
The bm_lockres protects individual node bitmaps. They are named in
|
|
the form bitmap000 for node 1, bitmap001 for node 2 and so on. When a
|
|
node joins the cluster, it acquires the lock in PW mode and it stays
|
|
so during the lifetime the node is part of the cluster. The lock
|
|
resource number is based on the slot number returned by the DLM
|
|
subsystem. Since DLM starts node count from one and bitmap slots
|
|
start from zero, one is subtracted from the DLM slot number to arrive
|
|
at the bitmap slot number.
|
|
|
|
The LVB of the bitmap lock for a particular node records the range
|
|
of sectors that are being re-synced by that node. No other
|
|
node may write to those sectors. This is used when a new nodes
|
|
joins the cluster.
|
|
|
|
2.2 Message passing locks
|
|
|
|
Each node has to communicate with other nodes when starting or ending
|
|
resync, and for metadata superblock updates. This communication is
|
|
managed through three locks: "token", "message", and "ack", together
|
|
with the Lock Value Block (LVB) of one of the "message" lock.
|
|
|
|
2.3 new-device management
|
|
|
|
A single lock: "no-new-dev" is used to co-ordinate the addition of
|
|
new devices - this must be synchronized across the array.
|
|
Normally all nodes hold a concurrent-read lock on this device.
|
|
|
|
3. Communication
|
|
|
|
Messages can be broadcast to all nodes, and the sender waits for all
|
|
other nodes to acknowledge the message before proceeding. Only one
|
|
message can be processed at a time.
|
|
|
|
3.1 Message Types
|
|
|
|
There are six types of messages which are passed:
|
|
|
|
3.1.1 METADATA_UPDATED: informs other nodes that the metadata has
|
|
been updated, and the node must re-read the md superblock. This is
|
|
performed synchronously. It is primarily used to signal device
|
|
failure.
|
|
|
|
3.1.2 RESYNCING: informs other nodes that a resync is initiated or
|
|
ended so that each node may suspend or resume the region. Each
|
|
RESYNCING message identifies a range of the devices that the
|
|
sending node is about to resync. This over-rides any pervious
|
|
notification from that node: only one ranged can be resynced at a
|
|
time per-node.
|
|
|
|
3.1.3 NEWDISK: informs other nodes that a device is being added to
|
|
the array. Message contains an identifier for that device. See
|
|
below for further details.
|
|
|
|
3.1.4 REMOVE: A failed or spare device is being removed from the
|
|
array. The slot-number of the device is included in the message.
|
|
|
|
3.1.5 RE_ADD: A failed device is being re-activated - the assumption
|
|
is that it has been determined to be working again.
|
|
|
|
3.1.6 BITMAP_NEEDS_SYNC: if a node is stopped locally but the bitmap
|
|
isn't clean, then another node is informed to take the ownership of
|
|
resync.
|
|
|
|
3.2 Communication mechanism
|
|
|
|
The DLM LVB is used to communicate within nodes of the cluster. There
|
|
are three resources used for the purpose:
|
|
|
|
3.2.1 token: The resource which protects the entire communication
|
|
system. The node having the token resource is allowed to
|
|
communicate.
|
|
|
|
3.2.2 message: The lock resource which carries the data to
|
|
communicate.
|
|
|
|
3.2.3 ack: The resource, acquiring which means the message has been
|
|
acknowledged by all nodes in the cluster. The BAST of the resource
|
|
is used to inform the receiving node that a node wants to
|
|
communicate.
|
|
|
|
The algorithm is:
|
|
|
|
1. receive status - all nodes have concurrent-reader lock on "ack".
|
|
|
|
sender receiver receiver
|
|
"ack":CR "ack":CR "ack":CR
|
|
|
|
2. sender get EX on "token"
|
|
sender get EX on "message"
|
|
sender receiver receiver
|
|
"token":EX "ack":CR "ack":CR
|
|
"message":EX
|
|
"ack":CR
|
|
|
|
Sender checks that it still needs to send a message. Messages
|
|
received or other events that happened while waiting for the
|
|
"token" may have made this message inappropriate or redundant.
|
|
|
|
3. sender writes LVB.
|
|
sender down-convert "message" from EX to CW
|
|
sender try to get EX of "ack"
|
|
[ wait until all receivers have *processed* the "message" ]
|
|
|
|
[ triggered by bast of "ack" ]
|
|
receiver get CR on "message"
|
|
receiver read LVB
|
|
receiver processes the message
|
|
[ wait finish ]
|
|
receiver releases "ack"
|
|
receiver tries to get PR on "message"
|
|
|
|
sender receiver receiver
|
|
"token":EX "message":CR "message":CR
|
|
"message":CW
|
|
"ack":EX
|
|
|
|
4. triggered by grant of EX on "ack" (indicating all receivers
|
|
have processed message)
|
|
sender down-converts "ack" from EX to CR
|
|
sender releases "message"
|
|
sender releases "token"
|
|
receiver upconvert to PR on "message"
|
|
receiver get CR of "ack"
|
|
receiver release "message"
|
|
|
|
sender receiver receiver
|
|
"ack":CR "ack":CR "ack":CR
|
|
|
|
|
|
4. Handling Failures
|
|
|
|
4.1 Node Failure
|
|
|
|
When a node fails, the DLM informs the cluster with the slot
|
|
number. The node starts a cluster recovery thread. The cluster
|
|
recovery thread:
|
|
|
|
- acquires the bitmap<number> lock of the failed node
|
|
- opens the bitmap
|
|
- reads the bitmap of the failed node
|
|
- copies the set bitmap to local node
|
|
- cleans the bitmap of the failed node
|
|
- releases bitmap<number> lock of the failed node
|
|
- initiates resync of the bitmap on the current node
|
|
md_check_recovery is invoked within recover_bitmaps,
|
|
then md_check_recovery -> metadata_update_start/finish,
|
|
it will lock the communication by lock_comm.
|
|
Which means when one node is resyncing it blocks all
|
|
other nodes from writing anywhere on the array.
|
|
|
|
The resync process is the regular md resync. However, in a clustered
|
|
environment when a resync is performed, it needs to tell other nodes
|
|
of the areas which are suspended. Before a resync starts, the node
|
|
send out RESYNCING with the (lo,hi) range of the area which needs to
|
|
be suspended. Each node maintains a suspend_list, which contains the
|
|
list of ranges which are currently suspended. On receiving RESYNCING,
|
|
the node adds the range to the suspend_list. Similarly, when the node
|
|
performing resync finishes, it sends RESYNCING with an empty range to
|
|
other nodes and other nodes remove the corresponding entry from the
|
|
suspend_list.
|
|
|
|
A helper function, ->area_resyncing() can be used to check if a
|
|
particular I/O range should be suspended or not.
|
|
|
|
4.2 Device Failure
|
|
|
|
Device failures are handled and communicated with the metadata update
|
|
routine. When a node detects a device failure it does not allow
|
|
any further writes to that device until the failure has been
|
|
acknowledged by all other nodes.
|
|
|
|
5. Adding a new Device
|
|
|
|
For adding a new device, it is necessary that all nodes "see" the new
|
|
device to be added. For this, the following algorithm is used:
|
|
|
|
1. Node 1 issues mdadm --manage /dev/mdX --add /dev/sdYY which issues
|
|
ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CLUSTER_ADD)
|
|
2. Node 1 sends a NEWDISK message with uuid and slot number
|
|
3. Other nodes issue kobject_uevent_env with uuid and slot number
|
|
(Steps 4,5 could be a udev rule)
|
|
4. In userspace, the node searches for the disk, perhaps
|
|
using blkid -t SUB_UUID=""
|
|
5. Other nodes issue either of the following depending on whether
|
|
the disk was found:
|
|
ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CANDIDATE and
|
|
disc.number set to slot number)
|
|
ioctl(CLUSTERED_DISK_NACK)
|
|
6. Other nodes drop lock on "no-new-devs" (CR) if device is found
|
|
7. Node 1 attempts EX lock on "no-new-dev"
|
|
8. If node 1 gets the lock, it sends METADATA_UPDATED after
|
|
unmarking the disk as SpareLocal
|
|
9. If not (get "no-new-dev" lock), it fails the operation and sends
|
|
METADATA_UPDATED.
|
|
10. Other nodes get the information whether a disk is added or not
|
|
by the following METADATA_UPDATED.
|
|
|
|
6. Module interface.
|
|
|
|
There are 17 call-backs which the md core can make to the cluster
|
|
module. Understanding these can give a good overview of the whole
|
|
process.
|
|
|
|
6.1 join(nodes) and leave()
|
|
|
|
These are called when an array is started with a clustered bitmap,
|
|
and when the array is stopped. join() ensures the cluster is
|
|
available and initializes the various resources.
|
|
Only the first 'nodes' nodes in the cluster can use the array.
|
|
|
|
6.2 slot_number()
|
|
|
|
Reports the slot number advised by the cluster infrastructure.
|
|
Range is from 0 to nodes-1.
|
|
|
|
6.3 resync_info_update()
|
|
|
|
This updates the resync range that is stored in the bitmap lock.
|
|
The starting point is updated as the resync progresses. The
|
|
end point is always the end of the array.
|
|
It does *not* send a RESYNCING message.
|
|
|
|
6.4 resync_start(), resync_finish()
|
|
|
|
These are called when resync/recovery/reshape starts or stops.
|
|
They update the resyncing range in the bitmap lock and also
|
|
send a RESYNCING message. resync_start reports the whole
|
|
array as resyncing, resync_finish reports none of it.
|
|
|
|
resync_finish() also sends a BITMAP_NEEDS_SYNC message which
|
|
allows some other node to take over.
|
|
|
|
6.5 metadata_update_start(), metadata_update_finish(),
|
|
metadata_update_cancel().
|
|
|
|
metadata_update_start is used to get exclusive access to
|
|
the metadata. If a change is still needed once that access is
|
|
gained, metadata_update_finish() will send a METADATA_UPDATE
|
|
message to all other nodes, otherwise metadata_update_cancel()
|
|
can be used to release the lock.
|
|
|
|
6.6 area_resyncing()
|
|
|
|
This combines two elements of functionality.
|
|
|
|
Firstly, it will check if any node is currently resyncing
|
|
anything in a given range of sectors. If any resync is found,
|
|
then the caller will avoid writing or read-balancing in that
|
|
range.
|
|
|
|
Secondly, while node recovery is happening it reports that
|
|
all areas are resyncing for READ requests. This avoids races
|
|
between the cluster-filesystem and the cluster-RAID handling
|
|
a node failure.
|
|
|
|
6.7 add_new_disk_start(), add_new_disk_finish(), new_disk_ack()
|
|
|
|
These are used to manage the new-disk protocol described above.
|
|
When a new device is added, add_new_disk_start() is called before
|
|
it is bound to the array and, if that succeeds, add_new_disk_finish()
|
|
is called the device is fully added.
|
|
|
|
When a device is added in acknowledgement to a previous
|
|
request, or when the device is declared "unavailable",
|
|
new_disk_ack() is called.
|
|
|
|
6.8 remove_disk()
|
|
|
|
This is called when a spare or failed device is removed from
|
|
the array. It causes a REMOVE message to be send to other nodes.
|
|
|
|
6.9 gather_bitmaps()
|
|
|
|
This sends a RE_ADD message to all other nodes and then
|
|
gathers bitmap information from all bitmaps. This combined
|
|
bitmap is then used to recovery the re-added device.
|
|
|
|
6.10 lock_all_bitmaps() and unlock_all_bitmaps()
|
|
|
|
These are called when change bitmap to none. If a node plans
|
|
to clear the cluster raid's bitmap, it need to make sure no other
|
|
nodes are using the raid which is achieved by lock all bitmap
|
|
locks within the cluster, and also those locks are unlocked
|
|
accordingly.
|