From fb6d1a5062d42828cad36ee72efeaa0613dcf479 Mon Sep 17 00:00:00 2001 From: sramamoorthy Date: Tue, 3 Dec 2019 14:50:37 -0800 Subject: [PATCH 1/9] Disk snapshot backup and restore documentation --- .../sphinx/source/disk-snapshot-backup.rst | 90 +++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100644 documentation/sphinx/source/disk-snapshot-backup.rst diff --git a/documentation/sphinx/source/disk-snapshot-backup.rst b/documentation/sphinx/source/disk-snapshot-backup.rst new file mode 100644 index 0000000000..a7c39cccbb --- /dev/null +++ b/documentation/sphinx/source/disk-snapshot-backup.rst @@ -0,0 +1,90 @@ + +.. _disk-snapshot-backups: + +################################# +Disk Snapshot Backup and Restore +################################# + +This document covers disk snapshot based backup and restoration of a FoundationDB database. This tool leverages disk level snapshots and gets a point-in-time consistent copy of the database. The disk snapshot backup can be used to provide additional level of protection in case of hardware or software failures, test and development purposes or for compliance reasons. + +.. _disk-snapshot-backup-introduction: + +Introduction +============ + +FoundationDB's disk snapshot backup tool makes a consistent, point-in-time backup of FoundationDB database without downtime by taking crash consitent snapshot of all the disk stores that has persistent data. + +Crash consitent snapshot feature at file-system or disk level is a pre-requsiste for using this feature. + +Disk snapshot backup tool orchestrates the snapshotting of all the disk images and ensures that they are restorable in a point-in-time consistent basis. Externally, all the disk stores of a cluster could be snapshotted, but those disk snapshots will not be point-in-time consistent and hence not restorable. + +Resetore is achieved by copying or attaching the disk snapshot images to FoundationDB compute instances. Restore behaves as if the cluster was powered down and restarted. + +Backup vs Disk snapshot backup +============================== + +Both these tools provide a point-in-time consistent backup of FoundationDB database, they operate at different levels and there are differences in terms of performance, features and external dependency. + +Backup/fdbbackup operates at the key-value level, backup will involve copying of all the key-values from the source cluster and the restore will invovle applying all the key-values to the destination database. Performance will depend on the amount of data and the throughput at which the data can be read and written. This approach is agnostic to external dependency, there is no requirement for any snapshotting feature from disk system. Additionally, it has an option for continuous backup that enables a restorable point-in-time very close to now. This feature already exists in FoundationDB and is detailed here :ref:`backups`. + +Disk snapshot backup and restore are generally high performant because it deals at disk level and data is not read or written through FoundationDB stack. In environments where disk snapshot and restore are highly performant this approach can be very fast. Feature is dependent on crash consistent snapshot feature from disk system. In environments where disk snapshots and restore are fast, frequent backups could be done as a substitute to the continuos backup. + +Limitations +=========== + + * data encryption is dependent on the disk system. + * backup and resotre involves tooling which are deployment and environment specific to be developed by operators. + +Backup Steps +============= + +``snapshot`` + This command line tool is used to create the snapshot. It takes a full path to a binary and reports the status, optionally, can take additional arguments to be passed down to the binary. It returns a Unique Identifier which can be used to identify all the disk snapshots of a backup. + +In response to the snapshot request from the user, FoundationDB will run a user specificed binary on all processes which has persistent data in it, binary should call environment specific snapshot create API and gather some additional data for the restore. Please note that the binary may be invoked multiple times on a single process if it plays two roles say storage and TLog. + +Before using the ``snapshot`` command the following setup needs to be done + * Develop and install a binary on the FoundationDB instances that can take snapshot of the local disk store. Binary should take the arguments mentioned below and be able to create a snapshot of the local disk store and gather any additional data that is needed for restore. + * binary will be invoked with the following arguments: + * UID - 32 byte alpha-numeric UID, the same id will be passed to all the nodes for this snapshot create instance, unique way to identify the set of disk snapshots associated with this backup + * Version - version string of the FoundationDB binary + * Path - path of the FoundationDB disk store + * Role - tlog/storage/coordinator, identifies the role of the node on which the snapshot is being invoked + * Set a new config parameter ``whitelist_binpath`` for fdbserver section, whose value is the full-binary path. Running any snapshot command will validate that it is in the whitelist_binpath. This is a security mechanism to stop running a random/unsecure command on the cluster by a client using snapshot command. + * snap create binary could capture any additional data needed to restore the cluster, additional data could be stored as tags in cloud environments or it could be stored in a additional file/directory in the data repo and then snapshotted. + +The section below describes a recommended specification of the list of things that needs to be gathered as part of backup to aid with restore. + +Backup Specification +==================== + +================================ ======================================================== +Field Name Source of information +================================ ======================================================== +``UID`` ``snapshot`` commands output contains the UID, this + can be used to catalog the disk images. +``FDB Server Version`` command line argument to snap create binary +``CreationTime`` Obtained by calling the system time. +``FDB Cluster File`` Read from the location of the cluster file location + mentioned in the command line arguments. Command + line arguments of fdbserver can be accessed from + /proc/$PPID/cmdline +``FDB Server Config Parameters`` Available from command line arguments of fdbserver + or from foundationdb.conf +``IP Address + Port`` Available from command line arguments of fdbserver +``Machine-Id`` Available from command line arguments of fdbserver +``Name for the snapshot file`` cluster-name:ip-addr:port:snapshotUID +================================ ======================================================== + +Any machines that does not have any persistent data in it will not have their foundationdb.conf be available in any of the disk images, they need to be backed up externally and restored. + +Restore Steps +============== +Restore is the process of building up the cluster from the snapshotted disk images. Here is list of steps for the restore process: + * Identify the disk images associated with a particular backup + * Group disk images of a backup by IP address or any other machine identifier + * Bring up a new cluster similar to the source cluster with FoundationDB services stopped and either attach the snapshot disk images or copy the snapshot disk images to the cluster in the following manner: + * Map the old IP address to new IP address in a one to one fashion and use that mapping to guide the restoration of disk images + * compute the new fdb.cluster file based on where the new coordinators disk stores are placed and push it to the all the instances in the new cluster + * start the FoundationDB service on all the instances + * NOTE: if one process share two roles which has persistent data then they will have a shared disk and there will be two snapshots of the disk once for each role. In that case, snapshot disk image needs to be cleaned, If a snapshot image had files that belongs to other roles than they need to be deleted. From 2bb0f3c8596504e582318a36c9d35d93a173a085 Mon Sep 17 00:00:00 2001 From: sramamoorthy Date: Wed, 4 Dec 2019 11:16:27 -0800 Subject: [PATCH 2/9] add ref to disk-snapshot-backup in operation page --- documentation/sphinx/source/operations.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/documentation/sphinx/source/operations.rst b/documentation/sphinx/source/operations.rst index 7c5b3628ef..bfdca2b45c 100644 --- a/documentation/sphinx/source/operations.rst +++ b/documentation/sphinx/source/operations.rst @@ -20,6 +20,8 @@ Ready to operate an externally accessible FoundationDB cluster? You'll find what * :doc:`backups` covers the FoundationDB backup tool, which provides an additional level of protection by supporting recovery from disasters or unintentional modification of the database. +* :doc:`disk-snapshot-backup` covers disk snapshot based FoundationDB backup tool, which is an alternate backup solution. + * :doc:`platforms` describes issues on particular platforms that affect the operation of FoundationDB. .. toctree:: @@ -34,4 +36,5 @@ Ready to operate an externally accessible FoundationDB cluster? You'll find what mr-status tls backups + disk-snapshot-backup platforms From d9d10ce1ba813906823d69799bc86023f87b0e65 Mon Sep 17 00:00:00 2001 From: sramamoorthy Date: Thu, 5 Dec 2019 13:19:40 -0800 Subject: [PATCH 3/9] add few sections and more edits --- .../sphinx/source/disk-snapshot-backup.rst | 96 +++++++++++-------- 1 file changed, 55 insertions(+), 41 deletions(-) diff --git a/documentation/sphinx/source/disk-snapshot-backup.rst b/documentation/sphinx/source/disk-snapshot-backup.rst index a7c39cccbb..4ac25d857c 100644 --- a/documentation/sphinx/source/disk-snapshot-backup.rst +++ b/documentation/sphinx/source/disk-snapshot-backup.rst @@ -12,79 +12,93 @@ This document covers disk snapshot based backup and restoration of a FoundationD Introduction ============ -FoundationDB's disk snapshot backup tool makes a consistent, point-in-time backup of FoundationDB database without downtime by taking crash consitent snapshot of all the disk stores that has persistent data. +FoundationDB's disk snapshot backup tool makes a consistent, point-in-time backup of FoundationDB database without downtime by taking crash consistent snapshot of all the disk stores that have persistent data. -Crash consitent snapshot feature at file-system or disk level is a pre-requsiste for using this feature. +The prerequisite of this feature is to have crash consistent snapshot support on the filesystem (or the disks) in which FoundationDB is running on. -Disk snapshot backup tool orchestrates the snapshotting of all the disk images and ensures that they are restorable in a point-in-time consistent basis. Externally, all the disk stores of a cluster could be snapshotted, but those disk snapshots will not be point-in-time consistent and hence not restorable. +Disk snapshot backup tool orchestrates the snapshotting of all the disk images and ensures that they are restorable in a point-in-time consistent basis. -Resetore is achieved by copying or attaching the disk snapshot images to FoundationDB compute instances. Restore behaves as if the cluster was powered down and restarted. +Restore is achieved by copying or attaching the disk snapshot images to FoundationDB compute instances. Restore behaves as if the cluster was powered down and restarted. Backup vs Disk snapshot backup ============================== Both these tools provide a point-in-time consistent backup of FoundationDB database, they operate at different levels and there are differences in terms of performance, features and external dependency. -Backup/fdbbackup operates at the key-value level, backup will involve copying of all the key-values from the source cluster and the restore will invovle applying all the key-values to the destination database. Performance will depend on the amount of data and the throughput at which the data can be read and written. This approach is agnostic to external dependency, there is no requirement for any snapshotting feature from disk system. Additionally, it has an option for continuous backup that enables a restorable point-in-time very close to now. This feature already exists in FoundationDB and is detailed here :ref:`backups`. +Backup/fdbbackup operates at the key-value level, backup will involve copying of all the key-values from the source cluster and the restore will involve applying all the key-values to the destination database. Performance will depend on the amount of data and the throughput with which the data can be read and written. This approach is agnostic to external dependency, there is no requirement for any snapshotting feature from disk system. Additionally, it has an option for continuous backup that enables a restorable point-in-time very close to now. This feature already exists in FoundationDB and is detailed here :ref:`backups`. -Disk snapshot backup and restore are generally high performant because it deals at disk level and data is not read or written through FoundationDB stack. In environments where disk snapshot and restore are highly performant this approach can be very fast. Feature is dependent on crash consistent snapshot feature from disk system. In environments where disk snapshots and restore are fast, frequent backups could be done as a substitute to the continuos backup. +Disk snapshot backup and restore are generally high performant because it deals at disk level and data is not read or written through FoundationDB stack. In environments where disk snapshot and restore are highly performant this approach can be very fast. Feature is strictly dependent on crash consistent snapshot feature from disk system. Frequent backups could be done as a substitute to continuous backup if the backups are performant. Limitations =========== - * data encryption is dependent on the disk system. - * backup and resotre involves tooling which are deployment and environment specific to be developed by operators. +* No support for continuous backup +* Feature is not supported on Windows operating system +* Data encryption is dependent on the disk system +* Backup and restore involves tooling which are deployment and environment specific to be developed by operators Backup Steps ============= ``snapshot`` - This command line tool is used to create the snapshot. It takes a full path to a binary and reports the status, optionally, can take additional arguments to be passed down to the binary. It returns a Unique Identifier which can be used to identify all the disk snapshots of a backup. + This command line tool is used to create the snapshot. It takes a full path to a ``snapshot create binary`` and reports the status, optionally, can take additional arguments to be passed down to the ``snapshot create binary``. It returns a unique identifier which can be used to identify all the disk snapshots of a backup. Even in case of failures unique identifier is returned to identify and clear any partially create disk snapshots. -In response to the snapshot request from the user, FoundationDB will run a user specificed binary on all processes which has persistent data in it, binary should call environment specific snapshot create API and gather some additional data for the restore. Please note that the binary may be invoked multiple times on a single process if it plays two roles say storage and TLog. +In response to the snapshot request from the user, FoundationDB will run a user specified ``snapshot create binary`` on all processes which has persistent data in it, binary should call filesystem/disk system specific snapshot create API and gather some additional data for the restore. Before using the ``snapshot`` command the following setup needs to be done - * Develop and install a binary on the FoundationDB instances that can take snapshot of the local disk store. Binary should take the arguments mentioned below and be able to create a snapshot of the local disk store and gather any additional data that is needed for restore. - * binary will be invoked with the following arguments: - * UID - 32 byte alpha-numeric UID, the same id will be passed to all the nodes for this snapshot create instance, unique way to identify the set of disk snapshots associated with this backup + * Write a program that will snapshot the local disk store when invoked by the ``fdbserver`` with the following arguments: + * UID - 32 byte alpha-numeric unique identifier, the same identifier will be passed to all the nodes in the cluster, can be used to identify the set of disk snapshots associated with this backup * Version - version string of the FoundationDB binary - * Path - path of the FoundationDB disk store + * Path - path of the FoundationDB disk store to be snapshotted * Role - tlog/storage/coordinator, identifies the role of the node on which the snapshot is being invoked - * Set a new config parameter ``whitelist_binpath`` for fdbserver section, whose value is the full-binary path. Running any snapshot command will validate that it is in the whitelist_binpath. This is a security mechanism to stop running a random/unsecure command on the cluster by a client using snapshot command. - * snap create binary could capture any additional data needed to restore the cluster, additional data could be stored as tags in cloud environments or it could be stored in a additional file/directory in the data repo and then snapshotted. + * Install ``snapshot create binary`` on the FoundationDB instance in a secure path that can be invoked by the ``fdbserver`` + * Set a new config parameter ``whitelist_binpath`` for ``fdbserver`` section, whose value is the absolute ``snapshot create binary`` path. Running any ``snapshot`` command will validate that it is in the ``whitelist_binpath``. This is a security mechanism to stop running a random/unsecure command on the cluster by a client using ``snapshot`` command + * ``snapshot create program`` should capture any additional data needed to restore the cluster, additional data could be stored as tags in cloud environments or it could be stored in an additional file/directory in the data repository and then snapshotted. The section below describes a recommended specification of the list of things that can be gathered by the binary: + +``snapshot`` is a synchronous command and when it returns successfully backup is considered complete. The time it takes to finish a backup is a function of the time it takes to snapshot the disk store. For eg: if disk snapshot takes 1 second, time to finish backup should be less than < 10 seconds, this is a general guidance and in some cases it may take longer. If the command is aborted by the user then the disk snapshots should not be used for restore, because the state of backup is undefined. If the command fails or aborts, operator can issue the next backup by issuing another ``snapshot``. -The section below describes a recommended specification of the list of things that needs to be gathered as part of backup to aid with restore. Backup Specification -==================== +-------------------- -================================ ======================================================== -Field Name Source of information -================================ ======================================================== -``UID`` ``snapshot`` commands output contains the UID, this - can be used to catalog the disk images. -``FDB Server Version`` command line argument to snap create binary -``CreationTime`` Obtained by calling the system time. -``FDB Cluster File`` Read from the location of the cluster file location - mentioned in the command line arguments. Command - line arguments of fdbserver can be accessed from - /proc/$PPID/cmdline -``FDB Server Config Parameters`` Available from command line arguments of fdbserver - or from foundationdb.conf -``IP Address + Port`` Available from command line arguments of fdbserver -``Machine-Id`` Available from command line arguments of fdbserver -``Name for the snapshot file`` cluster-name:ip-addr:port:snapshotUID -================================ ======================================================== +================================ ======================================================== ======================================================== +Field Name Description Source of information +================================ ======================================================== ======================================================== +``UID`` unique identifier passed with all the ``snapshot`` CLI command output contains the UID + snapshot create binary invocations associated with + a backup. Disk snapshots could be tagged with this UID. +``FoundationDB Server Version`` software version of the ``fdbserver`` command line argument to snap create binary +``CreationTime`` current system date and time time obtained by calling the system time +``FoundationDB Cluster File`` cluster file which has cluster-name, magic and read from the location of the cluster file location + the list of coordinators. mentioned in the command line arguments. Command + line arguments of ``fdbserver`` can be accessed from + /proc/$PPID/cmdline +``Config Knobs`` command line arguments passed to ``fdbserver`` available from command line arguments of ``fdbserver`` + or from foundationdb.conf +``IP Address + Port`` host address and port information of the ``fdbserver`` available from command line arguments of ``fdbserver`` + that is invoking the snapshot +``LocalityData`` machine id, zone id or any other locality information available from command line arguments of ``fdbserver`` +``Name for the snapshot file`` Recommended name for the disk snapshot cluster-name:ip-addr:port:UID +================================ ======================================================== ======================================================== -Any machines that does not have any persistent data in it will not have their foundationdb.conf be available in any of the disk images, they need to be backed up externally and restored. +``snapshot create binary`` will not be invoked on processes which does not have any persistent data (for eg: Cluster Controller or Master or MasterProxy). Since these processes are completely stateless, there is no need for any state information from them. But, if there are specialized configuration knobs used for one of these stateless processes then they need to be backed up and restored externally. + +Management of disk snapshots +---------------------------- + +Deleting unused disk snapshots or disk snapshots that are part of failed backups have to deleted by the operator externally. Restore Steps ============== -Restore is the process of building up the cluster from the snapshotted disk images. Here is list of steps for the restore process: - * Identify the disk images associated with a particular backup - * Group disk images of a backup by IP address or any other machine identifier + +Restore is the process of building up the cluster from the snapshotted disk images. There is no option to specify a restore version because there is no support for continuous backup. Here is the list of steps for the restore process: + * Identify the snapshot disk images associated with the backup to be restored with the help of UID or creation time + * Group disk images of a backup by IP address and/or locality information * Bring up a new cluster similar to the source cluster with FoundationDB services stopped and either attach the snapshot disk images or copy the snapshot disk images to the cluster in the following manner: + * Map the old IP address to new IP address in a one to one fashion and use that mapping to guide the restoration of disk images - * compute the new fdb.cluster file based on where the new coordinators disk stores are placed and push it to the all the instances in the new cluster - * start the FoundationDB service on all the instances + * Compute the new fdb.cluster file based on where the new coordinators disk stores are placed and push it to the all the instances in the new cluster + * Start the FoundationDB service on all the instances * NOTE: if one process share two roles which has persistent data then they will have a shared disk and there will be two snapshots of the disk once for each role. In that case, snapshot disk image needs to be cleaned, If a snapshot image had files that belongs to other roles than they need to be deleted. + +Cluster will start and get to healthy state indicating the completion of restore. Applications can optionally do any additional validations and use the cluster. From eae0daabf818ea8c7e60b5c80bb8d2f015c446e3 Mon Sep 17 00:00:00 2001 From: sramamoorthy Date: Fri, 6 Dec 2019 17:10:50 -0800 Subject: [PATCH 4/9] indentation and minor tweak --- .../sphinx/source/disk-snapshot-backup.rst | 34 ++++++++++--------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/documentation/sphinx/source/disk-snapshot-backup.rst b/documentation/sphinx/source/disk-snapshot-backup.rst index 4ac25d857c..1701caaf88 100644 --- a/documentation/sphinx/source/disk-snapshot-backup.rst +++ b/documentation/sphinx/source/disk-snapshot-backup.rst @@ -46,16 +46,17 @@ Backup Steps In response to the snapshot request from the user, FoundationDB will run a user specified ``snapshot create binary`` on all processes which has persistent data in it, binary should call filesystem/disk system specific snapshot create API and gather some additional data for the restore. Before using the ``snapshot`` command the following setup needs to be done - * Write a program that will snapshot the local disk store when invoked by the ``fdbserver`` with the following arguments: - * UID - 32 byte alpha-numeric unique identifier, the same identifier will be passed to all the nodes in the cluster, can be used to identify the set of disk snapshots associated with this backup - * Version - version string of the FoundationDB binary - * Path - path of the FoundationDB disk store to be snapshotted - * Role - tlog/storage/coordinator, identifies the role of the node on which the snapshot is being invoked - * Install ``snapshot create binary`` on the FoundationDB instance in a secure path that can be invoked by the ``fdbserver`` - * Set a new config parameter ``whitelist_binpath`` for ``fdbserver`` section, whose value is the absolute ``snapshot create binary`` path. Running any ``snapshot`` command will validate that it is in the ``whitelist_binpath``. This is a security mechanism to stop running a random/unsecure command on the cluster by a client using ``snapshot`` command - * ``snapshot create program`` should capture any additional data needed to restore the cluster, additional data could be stored as tags in cloud environments or it could be stored in an additional file/directory in the data repository and then snapshotted. The section below describes a recommended specification of the list of things that can be gathered by the binary: -``snapshot`` is a synchronous command and when it returns successfully backup is considered complete. The time it takes to finish a backup is a function of the time it takes to snapshot the disk store. For eg: if disk snapshot takes 1 second, time to finish backup should be less than < 10 seconds, this is a general guidance and in some cases it may take longer. If the command is aborted by the user then the disk snapshots should not be used for restore, because the state of backup is undefined. If the command fails or aborts, operator can issue the next backup by issuing another ``snapshot``. +* Write a program that will snapshot the local disk store when invoked by the ``fdbserver`` with the following arguments: + * UID - 32 byte alpha-numeric unique identifier, the same identifier will be passed to all the nodes in the cluster, can be used to identify the set of disk snapshots associated with this backup + * Version - version string of the FoundationDB binary + * Path - path of the FoundationDB disk store to be snapshotted + * Role - tlog/storage/coordinator, identifies the role of the node on which the snapshot is being invoked +* Install ``snapshot create binary`` on the FoundationDB instance in a secure path that can be invoked by the ``fdbserver`` +* Set a new config parameter ``whitelist_binpath`` for ``fdbserver`` section, whose value is the absolute ``snapshot create binary`` path. Running any ``snapshot`` command will validate that it is in the ``whitelist_binpath``. This is a security mechanism to stop running a random/unsecure command on the cluster by a client using ``snapshot`` command +* ``snapshot create program`` should capture any additional data needed to restore the cluster, additional data could be stored as tags in cloud environments or it could be stored in an additional file/directory in the data repository and then snapshotted. The section below describes a recommended specification of the list of things that can be gathered by the binary: + +``snapshot`` is a synchronous command and when it returns successfully backup is considered complete. The time it takes to finish a backup is a function of the time it takes to snapshot the disk store. For eg: if disk snapshot takes 1 second, time to finish backup should be less than < 10 seconds, this is a general guidance and in some cases it may take longer. If the command is aborted by the user then the disk snapshots should not be used for restore, because the state of backup is undefined. If the command fails or aborts, operator can retry by issuing another ``snapshot`` command. Backup Specification @@ -92,13 +93,14 @@ Restore Steps ============== Restore is the process of building up the cluster from the snapshotted disk images. There is no option to specify a restore version because there is no support for continuous backup. Here is the list of steps for the restore process: - * Identify the snapshot disk images associated with the backup to be restored with the help of UID or creation time - * Group disk images of a backup by IP address and/or locality information - * Bring up a new cluster similar to the source cluster with FoundationDB services stopped and either attach the snapshot disk images or copy the snapshot disk images to the cluster in the following manner: - * Map the old IP address to new IP address in a one to one fashion and use that mapping to guide the restoration of disk images - * Compute the new fdb.cluster file based on where the new coordinators disk stores are placed and push it to the all the instances in the new cluster - * Start the FoundationDB service on all the instances - * NOTE: if one process share two roles which has persistent data then they will have a shared disk and there will be two snapshots of the disk once for each role. In that case, snapshot disk image needs to be cleaned, If a snapshot image had files that belongs to other roles than they need to be deleted. +* Identify the snapshot disk images associated with the backup to be restored with the help of UID or creation time +* Group disk images of a backup by IP address and/or locality information +* Bring up a new cluster similar to the source cluster with FoundationDB services stopped and either attach the snapshot disk images or copy the snapshot disk images to the cluster in the following manner: + + * Map the old IP address to new IP address in a one to one fashion and use that mapping to guide the restoration of disk images +* Compute the new fdb.cluster file based on where the new coordinators disk stores are placed and push it to the all the instances in the new cluster +* Start the FoundationDB service on all the instances +* NOTE: if one process share two roles which has persistent data then they will have a shared disk and there will be two snapshots of the disk once for each role. In that case, snapshot disk image needs to be cleaned, If a snapshot image had files that belongs to other roles than they need to be deleted. Cluster will start and get to healthy state indicating the completion of restore. Applications can optionally do any additional validations and use the cluster. From 71e93f4d04423b7af18001b61982e5563b9831a1 Mon Sep 17 00:00:00 2001 From: sramamoorthy Date: Wed, 11 Dec 2019 01:34:16 -0800 Subject: [PATCH 5/9] more edits and an example workflow --- .../sphinx/source/disk-snapshot-backup.rst | 218 +++++++++++++++--- 1 file changed, 192 insertions(+), 26 deletions(-) diff --git a/documentation/sphinx/source/disk-snapshot-backup.rst b/documentation/sphinx/source/disk-snapshot-backup.rst index 1701caaf88..2e24275644 100644 --- a/documentation/sphinx/source/disk-snapshot-backup.rst +++ b/documentation/sphinx/source/disk-snapshot-backup.rst @@ -2,10 +2,10 @@ .. _disk-snapshot-backups: ################################# -Disk Snapshot Backup and Restore +Disk snapshot backup and Restore ################################# -This document covers disk snapshot based backup and restoration of a FoundationDB database. This tool leverages disk level snapshots and gets a point-in-time consistent copy of the database. The disk snapshot backup can be used to provide additional level of protection in case of hardware or software failures, test and development purposes or for compliance reasons. +This document covers disk snapshot based backup and restoration of a FoundationDB database. This tool leverages disk level snapshots and gets a point-in-time consistent copy of the database. The disk snapshot backup can be used to provide an additional level of protection in case of hardware or software failures, test and development purposes or for compliance reasons. .. _disk-snapshot-backup-introduction: @@ -14,20 +14,21 @@ Introduction FoundationDB's disk snapshot backup tool makes a consistent, point-in-time backup of FoundationDB database without downtime by taking crash consistent snapshot of all the disk stores that have persistent data. -The prerequisite of this feature is to have crash consistent snapshot support on the filesystem (or the disks) in which FoundationDB is running on. +The prerequisite of this feature is to have crash consistent snapshot support on the filesystem (or the disks) on which FoundationDB is running. -Disk snapshot backup tool orchestrates the snapshotting of all the disk images and ensures that they are restorable in a point-in-time consistent basis. +The disk snapshot backup tool orchestrates the snapshotting of all the disk images and ensures that they are restorable to a consistent point in time. -Restore is achieved by copying or attaching the disk snapshot images to FoundationDB compute instances. Restore behaves as if the cluster was powered down and restarted. +Restore is achieved by copying or attaching the disk snapshot images to FoundationDB compute instances. Restore behaves as if the cluster were powered down and restarted. Backup vs Disk snapshot backup ============================== +Backup feature already exists in FoundationDB and is detailed here :ref:`backups`, any use of fdbbackup will refer to this feature. -Both these tools provide a point-in-time consistent backup of FoundationDB database, they operate at different levels and there are differences in terms of performance, features and external dependency. +Both fdbbackup and Disk snapshot backup tools provide a point-in-time consistent backup of FoundationDB database, but, they operate at different levels and there are differences in terms of performance, features and external dependency. -Backup/fdbbackup operates at the key-value level, backup will involve copying of all the key-values from the source cluster and the restore will involve applying all the key-values to the destination database. Performance will depend on the amount of data and the throughput with which the data can be read and written. This approach is agnostic to external dependency, there is no requirement for any snapshotting feature from disk system. Additionally, it has an option for continuous backup that enables a restorable point-in-time very close to now. This feature already exists in FoundationDB and is detailed here :ref:`backups`. +fdbbackup operates at the key-value level. Backup involves copying of all the key-value pairs from the source cluster and restore involves applying all the key-value pairs to the destination database. Performance depends on the amount of data and the throughput with which the data can be read and written. This approach has no external dependency, there is no requirement for any snapshotting feature from the disk system. Additionally, it has an option for continuous backup with the flexibility to pick a restore point. -Disk snapshot backup and restore are generally high performant because it deals at disk level and data is not read or written through FoundationDB stack. In environments where disk snapshot and restore are highly performant this approach can be very fast. Feature is strictly dependent on crash consistent snapshot feature from disk system. Frequent backups could be done as a substitute to continuous backup if the backups are performant. +Disk snapshot backup and restore are generally high performance because it operates at disk level and data is not read or written through the FoundationDB stack. In environments where disk snapshot and restore are highly performant this approach can be very fast. Frequent backups can be done as a substitute to continuous backup if the backups are performant. Limitations =========== @@ -36,31 +37,52 @@ Limitations * Feature is not supported on Windows operating system * Data encryption is dependent on the disk system * Backup and restore involves tooling which are deployment and environment specific to be developed by operators +* ``snapshot`` command is a hidden command in the current release and will be unhidden in a future patch release. -Backup Steps -============= +Disk snapshot backup steps +========================== ``snapshot`` - This command line tool is used to create the snapshot. It takes a full path to a ``snapshot create binary`` and reports the status, optionally, can take additional arguments to be passed down to the ``snapshot create binary``. It returns a unique identifier which can be used to identify all the disk snapshots of a backup. Even in case of failures unique identifier is returned to identify and clear any partially create disk snapshots. + This command line tool is used to create the snapshot. It takes a full path to a ``snapshot create binary`` and reports the status. Optionally, it can take additional arguments to be passed down to the ``snapshot create binary``. It returns a unique identifier which can be used to identify all the disk snapshots of a backup. Even in case of failures the unique identifier is returned to identify and clear any partially create disk snapshots. -In response to the snapshot request from the user, FoundationDB will run a user specified ``snapshot create binary`` on all processes which has persistent data in it, binary should call filesystem/disk system specific snapshot create API and gather some additional data for the restore. +In response to the snapshot request from the user, FoundationDB will run the user specified ``snapshot create binary`` on all processes which have persistent data, binary should call filesystem/disk system specific snapshot create API. Before using the ``snapshot`` command the following setup needs to be done * Write a program that will snapshot the local disk store when invoked by the ``fdbserver`` with the following arguments: - * UID - 32 byte alpha-numeric unique identifier, the same identifier will be passed to all the nodes in the cluster, can be used to identify the set of disk snapshots associated with this backup - * Version - version string of the FoundationDB binary - * Path - path of the FoundationDB disk store to be snapshotted - * Role - tlog/storage/coordinator, identifies the role of the node on which the snapshot is being invoked + + - UID - 32 byte alpha-numeric unique identifier, the same identifier will be passed to all the nodes in the cluster, can be used to identify the set of disk snapshots associated with this backup + - Version - version string of the FoundationDB binary + - Path - path of the FoundationDB ``datadir`` to be snapshotted, ``datadir`` specified in :ref:`foundationdb-conf-fdbserver` + - Role - tlog/storage/coordinator, identifies the role of the node on which the snapshot is being invoked + * Install ``snapshot create binary`` on the FoundationDB instance in a secure path that can be invoked by the ``fdbserver`` -* Set a new config parameter ``whitelist_binpath`` for ``fdbserver`` section, whose value is the absolute ``snapshot create binary`` path. Running any ``snapshot`` command will validate that it is in the ``whitelist_binpath``. This is a security mechanism to stop running a random/unsecure command on the cluster by a client using ``snapshot`` command -* ``snapshot create program`` should capture any additional data needed to restore the cluster, additional data could be stored as tags in cloud environments or it could be stored in an additional file/directory in the data repository and then snapshotted. The section below describes a recommended specification of the list of things that can be gathered by the binary: +* Set a new config parameter ``whitelist_binpath`` in :ref:`foundationdb-conf-fdbserver`, whose value is the ``snapshot create binary`` absolute path. Running any ``snapshot`` command will validate that it is in the ``whitelist_binpath``. This is a security mechanism to stop running a random/insecure command on the cluster by a client using the ``snapshot`` command. Example configuration entry will look like:: -``snapshot`` is a synchronous command and when it returns successfully backup is considered complete. The time it takes to finish a backup is a function of the time it takes to snapshot the disk store. For eg: if disk snapshot takes 1 second, time to finish backup should be less than < 10 seconds, this is a general guidance and in some cases it may take longer. If the command is aborted by the user then the disk snapshots should not be used for restore, because the state of backup is undefined. If the command fails or aborts, operator can retry by issuing another ``snapshot`` command. + whitelist_binpath = "/bin/snap_create.sh" + +* ``snapshot create program`` should capture any additional data needed to restore the cluster. Additional data can be stored as tags in cloud environments or it can be stored in an additional file/directory in the ``datadir`` and then snapshotted. The section :ref:`disk-snapshot-backup-specification` describes the recommended specification of the list of things that can be gathered by the binary. +* Program should return a non-zero status for any failures and zero for success +* If the ``snapshot create binary`` process takes longer than 5 minutes to return a status then it will be killed and ``snapshot`` command will fail. Timeout of 5 minutes is configurable and can be set with ``SNAP_CREATE_MAX_TIMEOUT`` config parameter in :ref:`foundationdb-conf-fdbserver`. Since the default value is large enough, there should not be a need to modify this configuration. + +``snapshot`` is a synchronous command and when it returns successfully backup is considered complete and restorable. The time it takes to finish a backup is a function of the time it takes to snapshot the disk store. For example, if disk snapshot takes 1 second, time to finish backup should be less than < 10 seconds, this is general guidance and in some cases it may take longer. If the command is aborted by the user then the disk snapshots should not be used for restore, because the state of backup is undefined. If the command fails or aborts, operator can retry by issuing another ``snapshot`` command. + +Example ``snapshot`` command usage:: + + fdbcli> snapshot /bin/snap_create.sh --key1 value1 --key2 value2 + Snapshot command succeeded with UID c50263df28be44ebb596f5c2a849adbb + +will invoke the ``snapshot create binary`` on all the roles with the following arguments:: + + --key1 value1 --key2 value2 --path /mnt/circus/data/4502 --version 6.2.6 --role tlog --uid c50263df28be44ebb596f5c2a849adbb -Backup Specification --------------------- +.. _disk-snapshot-backup-specification: + +Disk snapshot backup specification +---------------------------------- + +Details the list of artifacts the ``snapshot create binary`` should gather to aid the restore. ================================ ======================================================== ======================================================== Field Name Description Source of information @@ -82,15 +104,15 @@ Field Name Description ``Name for the snapshot file`` Recommended name for the disk snapshot cluster-name:ip-addr:port:UID ================================ ======================================================== ======================================================== -``snapshot create binary`` will not be invoked on processes which does not have any persistent data (for eg: Cluster Controller or Master or MasterProxy). Since these processes are completely stateless, there is no need for any state information from them. But, if there are specialized configuration knobs used for one of these stateless processes then they need to be backed up and restored externally. +``snapshot create binary`` will not be invoked on processes which does not have any persistent data (for example, Cluster Controller or Master or MasterProxy). Since these processes are stateless, there is no need for a snapshot. If there are specialized configuration knobs used for one of these stateless processes then they need to be copied and restored externally. Management of disk snapshots ---------------------------- -Deleting unused disk snapshots or disk snapshots that are part of failed backups have to deleted by the operator externally. +Unused disk snapshots or disk snapshots that are part of failed backups have to deleted by the operator externally. -Restore Steps -============== +Disk snapshot restore steps +=========================== Restore is the process of building up the cluster from the snapshotted disk images. There is no option to specify a restore version because there is no support for continuous backup. Here is the list of steps for the restore process: @@ -101,6 +123,150 @@ Restore is the process of building up the cluster from the snapshotted disk imag * Map the old IP address to new IP address in a one to one fashion and use that mapping to guide the restoration of disk images * Compute the new fdb.cluster file based on where the new coordinators disk stores are placed and push it to the all the instances in the new cluster * Start the FoundationDB service on all the instances -* NOTE: if one process share two roles which has persistent data then they will have a shared disk and there will be two snapshots of the disk once for each role. In that case, snapshot disk image needs to be cleaned, If a snapshot image had files that belongs to other roles than they need to be deleted. +* NOTE: Process can have multiple roles with persistent data which share the same ``datadir``. ``snapshot create binary`` will create multiple snapshots, one per role. In such case, snapshot disk images needs to go through additional processing before restore, if a snapshot image of a role has files that belongs to other roles then they need to be deleted. Cluster will start and get to healthy state indicating the completion of restore. Applications can optionally do any additional validations and use the cluster. + +Example backup and restore steps +================================ + +Here are the backup and restore steps on a over simplified setup with a single node cluster and ``cp`` command to create snapshots and restore. This is purely for illustration, real world backup and restore scripts needs to follow all the steps detailed above. + + +* Create a single node cluster by following the steps here :ref:`building-cluster` + +* Check the status of the cluster and write few sample keys:: + + fdbcli> status + + Using cluster file `/mnt/source/fdb.cluster'. + + Configuration: + Redundancy mode - single + Storage engine - ssd-2 + Coordinators - 1 + + Cluster: + FoundationDB processes - 1 + Zones - 1 + Machines - 1 + Memory availability - 30.6 GB per process on machine with least available + Fault Tolerance - 0 machines + Server time - 12/11/19 04:02:57 + + Data: + Replication health - Healthy + Moving data - 0.000 GB + Sum of key-value sizes - 0 MB + Disk space used - 210 MB + + Operating space: + Storage server - 72.6 GB free on most full server + Log server - 72.6 GB free on most full server + + Workload: + Read rate - 9 Hz + Write rate - 0 Hz + Transactions started - 5 Hz + Transactions committed - 0 Hz + Conflict rate - 0 Hz + + Backup and DR: + Running backups - 0 + Running DRs - 0 + + Client time: 12/11/19 04:02:57 + + fdbcli> writemode on + fdbcli> set key1 value1 + Committed (76339236) + fdbcli> set key2 value2 + Committed (80235963) + +* Write a ``snap create binary`` which copies the ``datadir`` to a user passed destination directory location:: + + #!/bin/sh + UID="" + PATH="" + ROLE="" + DESTDIR="" + PARSE_ARGS $@ // not detailed here + + mkdir -p $DESTDIR/$UID/$ROLE || exit "snapshot failed for $@" + cp $PATH/ $DESTDIR/$UID/$ROLE/* || exit "snapshot failed for $@" + +* Install the ``snap create binary`` as ``/bin/snap_create.sh``, add the entry for ``whitelist_binpath`` in :ref:`foundationdb-conf-fdbserver`, stop and start the foundationdb service for the configuration change to take effect +* Issue ``snapshot`` command as follows:: + + fdbcli> snapshot /bin/snap_create.sh --destdir /mnt/backup + Snapshot command succeeded with UID 69a5e0576621892f85f55b4ebfeb4312 + +* ``snapshot create binary`` gets invoked once for each role namely tlog, storage and coordinator in this process with the following arguments:: + + --path /mnt/source/datadir --version 6.2.6 --role storage --uid 69a5e0576621892f85f55b4ebfeb4312 --destdir /mnt/backup + --path /mnt/source/datadir --version 6.2.6 --role tlog --uid 69a5e0576621892f85f55b4ebfeb4312 --destdir /mnt/backup + --path /mnt/source/datadir --version 6.2.6 --role coord --uid 69a5e0576621892f85f55b4ebfeb4312 --destdir /mnt/backup + +* Snapshot is successful and all the snapshot images are in ``destdir`` specified by the user in the command line argument to ``snaphsot`` command, here is a sample directory listing of one of the coordinator backup directory:: + + $ ls /mnt/backup/69a5e0576621892f85f55b4ebfeb4312/coord/ + coordination-0.fdq log2-V_3_LS_2-b9990ae9bc00672f07264ad43d9d0792.sqlite-wal processId + coordination-1.fdq logqueue-V_3_LS_2-b9990ae9bc00672f07264ad43d9d0792-0.fdq storage-f0e72cdfed12a233e0e58291150ca597.sqlite + log2-V_3_LS_2-b9990ae9bc00672f07264ad43d9d0792.sqlite logqueue-V_3_LS_2-b9990ae9bc00672f07264ad43d9d0792-1.fdq storage-f0e72cdfed12a233e0e58291150ca597.sqlite-wal + +* To restore the coordinator backup image, setup a restore ``datadir`` and copy all the coordinator related files to it:: + + $ cp /mnt/backup/69a5e0576621892f85f55b4ebfeb4312/coord/coord* /mnt/restore/datadir/ + +* Repeat the above steps to restore storage and tlog backup images +* Prepare the ``fdb.cluster`` for the restore with new coordinator IP address, example:: + + znC1NC5b:iYHJLq7z@10.2.80.40:4500 -> znC1NC5b:iYHJLq7z@10.2.80.41:4500 +* ``foundationdb.conf`` can be exact same copy as the source cluster for this example +* Once all the backup images are restored, start a new fdbserver with the ``datadir`` pointing to ``/mnt/restore/datadir`` and the new ``fdb.cluster``. +* Verify the cluster is healthy and check the sample keys that we added are still there:: + + fdb> status + + Using cluster file `/mnt/restore/fdb.cluster'. + + Configuration: + Redundancy mode - single + Storage engine - ssd-2 + Coordinators - 1 + + Cluster: + FoundationDB processes - 1 + Zones - 1 + Machines - 1 + Memory availability - 30.5 GB per process on machine with least available + Fault Tolerance - 0 machines + Server time - 12/11/19 09:04:53 + + Data: + Replication health - Healthy + Moving data - 0.000 GB + Sum of key-value sizes - 0 MB + Disk space used - 210 MB + + Operating space: + Storage server - 72.5 GB free on most full server + Log server - 72.5 GB free on most full server + + Workload: + Read rate - 7 Hz + Write rate - 0 Hz + Transactions started - 3 Hz + Transactions committed - 0 Hz + Conflict rate - 0 Hz + + Backup and DR: + Running backups - 0 + Running DRs - 0 + + Client time: 12/11/19 09:04:53 + + fdb> get key1 + `key1' is `value1' + fdb> get key2 + `key2' is `value2' From 3ba9d79e883a8e527c6db4e0ce828375bfb6d6b9 Mon Sep 17 00:00:00 2001 From: sramamoorthy Date: Thu, 12 Dec 2019 14:20:52 -0800 Subject: [PATCH 6/9] Fix sample snap create script and add error codes --- .../sphinx/source/disk-snapshot-backup.rst | 71 ++++++++++++++++--- 1 file changed, 61 insertions(+), 10 deletions(-) diff --git a/documentation/sphinx/source/disk-snapshot-backup.rst b/documentation/sphinx/source/disk-snapshot-backup.rst index 2e24275644..0036c13443 100644 --- a/documentation/sphinx/source/disk-snapshot-backup.rst +++ b/documentation/sphinx/source/disk-snapshot-backup.rst @@ -104,13 +104,42 @@ Field Name Description ``Name for the snapshot file`` Recommended name for the disk snapshot cluster-name:ip-addr:port:UID ================================ ======================================================== ======================================================== -``snapshot create binary`` will not be invoked on processes which does not have any persistent data (for example, Cluster Controller or Master or MasterProxy). Since these processes are stateless, there is no need for a snapshot. If there are specialized configuration knobs used for one of these stateless processes then they need to be copied and restored externally. +``snapshot create binary`` will not be invoked on processes which does not have any persistent data (for example, Cluster Controller or Master or MasterProxy). Since these processes are stateless, there is no need for a snapshot. Any specialized configuration knobs used for one of these stateless processes need to be copied and restored externally. Management of disk snapshots ---------------------------- Unused disk snapshots or disk snapshots that are part of failed backups have to deleted by the operator externally. +Error codes +----------- + +Errors codes returned by ``snapshot`` command + +======================================= ============ ============================= ============================================================= +Name Code Description Comments +======================================= ============ ============================= ============================================================= +snap_path_not_whitelisted 2505 Snapshot create binary path Whitelist the ``snap create binary`` path and retry the + not whitelisted operation. +snap_not_fully_recovered_unsupported 2506 Unsupported when the cluster Wait for the cluster to finish recovery and then retry the + is not fully recovered operation +snap_log_anti_quorum_unsupported 2507 Unsupported when log anti Feature is not supported when log anti quorum is configured + quorum is configured +snap_with_recovery_unsupported 2508 Cluster recovery during Recovery happened while snapshot operation was in progress, + snapshot operation not retry the operation. + supported +snap_storage_failed 2501 Failed to snapshot storage Verify that the ``snap create binary`` is installed and + nodes can be executed by the user running ``fdbserver`` +snap_tlog_failed 2502 Failed to snapshot TLog ,, + nodes +snap_coord_failed 2503 Failed to snapshot ,, + coordinator nodes +unknown_error 4000 An unknown error occurred ,, +snap_disable_tlog_pop_failed 2500 Disk Snapshot error No operator action is needed, retry the operation +snap_enable_tlog_pop_failed 2504 Disk Snapshot error ,, +======================================= ============ ============================= ============================================================= + + Disk snapshot restore steps =========================== @@ -127,6 +156,7 @@ Restore is the process of building up the cluster from the snapshotted disk imag Cluster will start and get to healthy state indicating the completion of restore. Applications can optionally do any additional validations and use the cluster. + Example backup and restore steps ================================ @@ -185,15 +215,36 @@ Here are the backup and restore steps on a over simplified setup with a single n * Write a ``snap create binary`` which copies the ``datadir`` to a user passed destination directory location:: - #!/bin/sh - UID="" - PATH="" - ROLE="" - DESTDIR="" - PARSE_ARGS $@ // not detailed here + #!/bin/sh - mkdir -p $DESTDIR/$UID/$ROLE || exit "snapshot failed for $@" - cp $PATH/ $DESTDIR/$UID/$ROLE/* || exit "snapshot failed for $@" + while (( "$#" )); do + case "$1" in + --uid) + SNAPUID=$2 + shift 2 + ;; + --path) + DATADIR=$2 + shift 2 + ;; + --role) + ROLE=$2 + shift 2 + ;; + --destdir) + DESTDIR=$2 + shift 2 + ;; + *) + shift + ;; + esac + done + + mkdir -p "$DESTDIR/$SNAPUID/$ROLE" || exit 1 + cp "$DATADIR/"* "$DESTDIR/$SNAPUID/$ROLE/" || exit 1 + + exit 0 * Install the ``snap create binary`` as ``/bin/snap_create.sh``, add the entry for ``whitelist_binpath`` in :ref:`foundationdb-conf-fdbserver`, stop and start the foundationdb service for the configuration change to take effect * Issue ``snapshot`` command as follows:: @@ -224,7 +275,7 @@ Here are the backup and restore steps on a over simplified setup with a single n znC1NC5b:iYHJLq7z@10.2.80.40:4500 -> znC1NC5b:iYHJLq7z@10.2.80.41:4500 * ``foundationdb.conf`` can be exact same copy as the source cluster for this example * Once all the backup images are restored, start a new fdbserver with the ``datadir`` pointing to ``/mnt/restore/datadir`` and the new ``fdb.cluster``. -* Verify the cluster is healthy and check the sample keys that we added are still there:: +* Verify the cluster is healthy and check the sample keys that we added are there:: fdb> status From 67e724468bcb7f3875610dca9a0e4c08f7971a53 Mon Sep 17 00:00:00 2001 From: sramamoorthy Date: Thu, 12 Dec 2019 14:37:34 -0800 Subject: [PATCH 7/9] more minor tweaks --- documentation/sphinx/source/disk-snapshot-backup.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/documentation/sphinx/source/disk-snapshot-backup.rst b/documentation/sphinx/source/disk-snapshot-backup.rst index 0036c13443..ea7545f936 100644 --- a/documentation/sphinx/source/disk-snapshot-backup.rst +++ b/documentation/sphinx/source/disk-snapshot-backup.rst @@ -24,7 +24,7 @@ Backup vs Disk snapshot backup ============================== Backup feature already exists in FoundationDB and is detailed here :ref:`backups`, any use of fdbbackup will refer to this feature. -Both fdbbackup and Disk snapshot backup tools provide a point-in-time consistent backup of FoundationDB database, but, they operate at different levels and there are differences in terms of performance, features and external dependency. +Both fdbbackup and Disk snapshot backup tools provide a point-in-time consistent backup of FoundationDB database, but they operate at different levels and there are differences in terms of performance, features and external dependency. fdbbackup operates at the key-value level. Backup involves copying of all the key-value pairs from the source cluster and restore involves applying all the key-value pairs to the destination database. Performance depends on the amount of data and the throughput with which the data can be read and written. This approach has no external dependency, there is no requirement for any snapshotting feature from the disk system. Additionally, it has an option for continuous backup with the flexibility to pick a restore point. @@ -114,7 +114,7 @@ Unused disk snapshots or disk snapshots that are part of failed backups have to Error codes ----------- -Errors codes returned by ``snapshot`` command +Error codes returned by ``snapshot`` command ======================================= ============ ============================= ============================================================= Name Code Description Comments @@ -160,12 +160,12 @@ Cluster will start and get to healthy state indicating the completion of restore Example backup and restore steps ================================ -Here are the backup and restore steps on a over simplified setup with a single node cluster and ``cp`` command to create snapshots and restore. This is purely for illustration, real world backup and restore scripts needs to follow all the steps detailed above. +Here are the backup and restore steps on an over simplified setup with a single node cluster and ``cp`` command to create snapshots and restore. This is purely for illustration, real world backup and restore scripts needs to follow all the steps detailed above. * Create a single node cluster by following the steps here :ref:`building-cluster` -* Check the status of the cluster and write few sample keys:: +* Check the status of the cluster and write a few sample keys:: fdbcli> status From 92726dfd91a2bdb0e21a48e27fe642fd95bfcdb0 Mon Sep 17 00:00:00 2001 From: sramamoorthy Date: Thu, 12 Dec 2019 14:54:56 -0800 Subject: [PATCH 8/9] make fdb role names as inline clear text --- .../sphinx/source/disk-snapshot-backup.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/documentation/sphinx/source/disk-snapshot-backup.rst b/documentation/sphinx/source/disk-snapshot-backup.rst index ea7545f936..d9d65de995 100644 --- a/documentation/sphinx/source/disk-snapshot-backup.rst +++ b/documentation/sphinx/source/disk-snapshot-backup.rst @@ -54,7 +54,7 @@ Before using the ``snapshot`` command the following setup needs to be done - UID - 32 byte alpha-numeric unique identifier, the same identifier will be passed to all the nodes in the cluster, can be used to identify the set of disk snapshots associated with this backup - Version - version string of the FoundationDB binary - Path - path of the FoundationDB ``datadir`` to be snapshotted, ``datadir`` specified in :ref:`foundationdb-conf-fdbserver` - - Role - tlog/storage/coordinator, identifies the role of the node on which the snapshot is being invoked + - Role - ``tlog``/``storage``/``coordinator``, identifies the role of the node on which the snapshot is being invoked * Install ``snapshot create binary`` on the FoundationDB instance in a secure path that can be invoked by the ``fdbserver`` * Set a new config parameter ``whitelist_binpath`` in :ref:`foundationdb-conf-fdbserver`, whose value is the ``snapshot create binary`` absolute path. Running any ``snapshot`` command will validate that it is in the ``whitelist_binpath``. This is a security mechanism to stop running a random/insecure command on the cluster by a client using the ``snapshot`` command. Example configuration entry will look like:: @@ -72,7 +72,7 @@ Example ``snapshot`` command usage:: fdbcli> snapshot /bin/snap_create.sh --key1 value1 --key2 value2 Snapshot command succeeded with UID c50263df28be44ebb596f5c2a849adbb -will invoke the ``snapshot create binary`` on all the roles with the following arguments:: +will invoke the ``snapshot create binary`` on ``tlog`` role with the following arguments:: --key1 value1 --key2 value2 --path /mnt/circus/data/4502 --version 6.2.6 --role tlog --uid c50263df28be44ebb596f5c2a849adbb @@ -150,7 +150,7 @@ Restore is the process of building up the cluster from the snapshotted disk imag * Bring up a new cluster similar to the source cluster with FoundationDB services stopped and either attach the snapshot disk images or copy the snapshot disk images to the cluster in the following manner: * Map the old IP address to new IP address in a one to one fashion and use that mapping to guide the restoration of disk images -* Compute the new fdb.cluster file based on where the new coordinators disk stores are placed and push it to the all the instances in the new cluster +* Compute the new fdb.cluster file based on where the new ``coordinators`` disk stores are placed and push it to the all the instances in the new cluster * Start the FoundationDB service on all the instances * NOTE: Process can have multiple roles with persistent data which share the same ``datadir``. ``snapshot create binary`` will create multiple snapshots, one per role. In such case, snapshot disk images needs to go through additional processing before restore, if a snapshot image of a role has files that belongs to other roles then they need to be deleted. @@ -252,7 +252,7 @@ Here are the backup and restore steps on an over simplified setup with a single fdbcli> snapshot /bin/snap_create.sh --destdir /mnt/backup Snapshot command succeeded with UID 69a5e0576621892f85f55b4ebfeb4312 -* ``snapshot create binary`` gets invoked once for each role namely tlog, storage and coordinator in this process with the following arguments:: +* ``snapshot create binary`` gets invoked once for each role namely ``tlog``, ``storage`` and ``coordinator`` in this process with the following arguments:: --path /mnt/source/datadir --version 6.2.6 --role storage --uid 69a5e0576621892f85f55b4ebfeb4312 --destdir /mnt/backup --path /mnt/source/datadir --version 6.2.6 --role tlog --uid 69a5e0576621892f85f55b4ebfeb4312 --destdir /mnt/backup @@ -265,12 +265,12 @@ Here are the backup and restore steps on an over simplified setup with a single coordination-1.fdq logqueue-V_3_LS_2-b9990ae9bc00672f07264ad43d9d0792-0.fdq storage-f0e72cdfed12a233e0e58291150ca597.sqlite log2-V_3_LS_2-b9990ae9bc00672f07264ad43d9d0792.sqlite logqueue-V_3_LS_2-b9990ae9bc00672f07264ad43d9d0792-1.fdq storage-f0e72cdfed12a233e0e58291150ca597.sqlite-wal -* To restore the coordinator backup image, setup a restore ``datadir`` and copy all the coordinator related files to it:: +* To restore the ``coordinator`` backup image, setup a restore ``datadir`` and copy all the ``coordinator`` related files to it:: $ cp /mnt/backup/69a5e0576621892f85f55b4ebfeb4312/coord/coord* /mnt/restore/datadir/ -* Repeat the above steps to restore storage and tlog backup images -* Prepare the ``fdb.cluster`` for the restore with new coordinator IP address, example:: +* Repeat the above steps to restore ``storage`` and ``tlog`` backup images +* Prepare the ``fdb.cluster`` for the restore with new ``coordinator`` IP address, example:: znC1NC5b:iYHJLq7z@10.2.80.40:4500 -> znC1NC5b:iYHJLq7z@10.2.80.41:4500 * ``foundationdb.conf`` can be exact same copy as the source cluster for this example From 34eff8475638319dd8ac6cbc00e8cd317209da91 Mon Sep 17 00:00:00 2001 From: sramamoorthy Date: Mon, 16 Dec 2019 13:31:43 -0800 Subject: [PATCH 9/9] address review comments --- .../sphinx/source/disk-snapshot-backup.rst | 28 +++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/documentation/sphinx/source/disk-snapshot-backup.rst b/documentation/sphinx/source/disk-snapshot-backup.rst index d9d65de995..e5eccd8051 100644 --- a/documentation/sphinx/source/disk-snapshot-backup.rst +++ b/documentation/sphinx/source/disk-snapshot-backup.rst @@ -5,7 +5,7 @@ Disk snapshot backup and Restore ################################# -This document covers disk snapshot based backup and restoration of a FoundationDB database. This tool leverages disk level snapshots and gets a point-in-time consistent copy of the database. The disk snapshot backup can be used to provide an additional level of protection in case of hardware or software failures, test and development purposes or for compliance reasons. +This document covers disk snapshot based backup and restoration of a FoundationDB database. This tool leverages disk level snapshots and gets a point-in-time consistent copy of the database. The disk snapshot backup can be used for test and development purposes, for compliance reasons or to provide an additional level of protection in case of hardware or software failures. .. _disk-snapshot-backup-introduction: @@ -37,7 +37,7 @@ Limitations * Feature is not supported on Windows operating system * Data encryption is dependent on the disk system * Backup and restore involves tooling which are deployment and environment specific to be developed by operators -* ``snapshot`` command is a hidden command in the current release and will be unhidden in a future patch release. +* ``snapshot`` command is a hidden fdbcli command in the current release and will be unhidden in a future patch release. Disk snapshot backup steps ========================== @@ -61,7 +61,7 @@ Before using the ``snapshot`` command the following setup needs to be done whitelist_binpath = "/bin/snap_create.sh" -* ``snapshot create program`` should capture any additional data needed to restore the cluster. Additional data can be stored as tags in cloud environments or it can be stored in an additional file/directory in the ``datadir`` and then snapshotted. The section :ref:`disk-snapshot-backup-specification` describes the recommended specification of the list of things that can be gathered by the binary. +* ``snapshot create binary`` should capture any additional data needed to restore the cluster. Additional data can be stored as tags in cloud environments or it can be stored in an additional file/directory in the ``datadir`` and then snapshotted. The section :ref:`disk-snapshot-backup-specification` describes the recommended specification of the list of things that can be gathered by the binary. * Program should return a non-zero status for any failures and zero for success * If the ``snapshot create binary`` process takes longer than 5 minutes to return a status then it will be killed and ``snapshot`` command will fail. Timeout of 5 minutes is configurable and can be set with ``SNAP_CREATE_MAX_TIMEOUT`` config parameter in :ref:`foundationdb-conf-fdbserver`. Since the default value is large enough, there should not be a need to modify this configuration. @@ -69,12 +69,12 @@ Before using the ``snapshot`` command the following setup needs to be done Example ``snapshot`` command usage:: - fdbcli> snapshot /bin/snap_create.sh --key1 value1 --key2 value2 + fdb> snapshot /bin/snap_create.sh --param1 param1-value --param2 param2-value Snapshot command succeeded with UID c50263df28be44ebb596f5c2a849adbb will invoke the ``snapshot create binary`` on ``tlog`` role with the following arguments:: - --key1 value1 --key2 value2 --path /mnt/circus/data/4502 --version 6.2.6 --role tlog --uid c50263df28be44ebb596f5c2a849adbb + --param1 param1-value --param2 param2-value --path /mnt/circus/data/4502 --version 6.2.6 --role tlog --uid c50263df28be44ebb596f5c2a849adbb .. _disk-snapshot-backup-specification: @@ -93,15 +93,15 @@ Field Name Description ``FoundationDB Server Version`` software version of the ``fdbserver`` command line argument to snap create binary ``CreationTime`` current system date and time time obtained by calling the system time ``FoundationDB Cluster File`` cluster file which has cluster-name, magic and read from the location of the cluster file location - the list of coordinators. mentioned in the command line arguments. Command - line arguments of ``fdbserver`` can be accessed from + the list of coordinators, cluster file is detailed mentioned in the command line arguments. Command + here :ref:`foundationdb-cluster-file` line arguments of ``fdbserver`` can be accessed from /proc/$PPID/cmdline ``Config Knobs`` command line arguments passed to ``fdbserver`` available from command line arguments of ``fdbserver`` or from foundationdb.conf ``IP Address + Port`` host address and port information of the ``fdbserver`` available from command line arguments of ``fdbserver`` that is invoking the snapshot ``LocalityData`` machine id, zone id or any other locality information available from command line arguments of ``fdbserver`` -``Name for the snapshot file`` Recommended name for the disk snapshot cluster-name:ip-addr:port:UID +``Name for the snapshot file`` recommended name for the disk snapshot cluster-name:ip-addr:port:UID ================================ ======================================================== ======================================================== ``snapshot create binary`` will not be invoked on processes which does not have any persistent data (for example, Cluster Controller or Master or MasterProxy). Since these processes are stateless, there is no need for a snapshot. Any specialized configuration knobs used for one of these stateless processes need to be copied and restored externally. @@ -167,7 +167,7 @@ Here are the backup and restore steps on an over simplified setup with a single * Check the status of the cluster and write a few sample keys:: - fdbcli> status + fdb> status Using cluster file `/mnt/source/fdb.cluster'. @@ -207,10 +207,10 @@ Here are the backup and restore steps on an over simplified setup with a single Client time: 12/11/19 04:02:57 - fdbcli> writemode on - fdbcli> set key1 value1 + fdb> writemode on + fdb> set key1 value1 Committed (76339236) - fdbcli> set key2 value2 + fdb> set key2 value2 Committed (80235963) * Write a ``snap create binary`` which copies the ``datadir`` to a user passed destination directory location:: @@ -249,7 +249,7 @@ Here are the backup and restore steps on an over simplified setup with a single * Install the ``snap create binary`` as ``/bin/snap_create.sh``, add the entry for ``whitelist_binpath`` in :ref:`foundationdb-conf-fdbserver`, stop and start the foundationdb service for the configuration change to take effect * Issue ``snapshot`` command as follows:: - fdbcli> snapshot /bin/snap_create.sh --destdir /mnt/backup + fdb> snapshot /bin/snap_create.sh --destdir /mnt/backup Snapshot command succeeded with UID 69a5e0576621892f85f55b4ebfeb4312 * ``snapshot create binary`` gets invoked once for each role namely ``tlog``, ``storage`` and ``coordinator`` in this process with the following arguments:: @@ -258,7 +258,7 @@ Here are the backup and restore steps on an over simplified setup with a single --path /mnt/source/datadir --version 6.2.6 --role tlog --uid 69a5e0576621892f85f55b4ebfeb4312 --destdir /mnt/backup --path /mnt/source/datadir --version 6.2.6 --role coord --uid 69a5e0576621892f85f55b4ebfeb4312 --destdir /mnt/backup -* Snapshot is successful and all the snapshot images are in ``destdir`` specified by the user in the command line argument to ``snaphsot`` command, here is a sample directory listing of one of the coordinator backup directory:: +* Snapshot is successful and all the snapshot images are in ``destdir`` specified by the user in the command line argument to ``snapshot`` command, here is a sample directory listing of one of the coordinator backup directory:: $ ls /mnt/backup/69a5e0576621892f85f55b4ebfeb4312/coord/ coordination-0.fdq log2-V_3_LS_2-b9990ae9bc00672f07264ad43d9d0792.sqlite-wal processId