<liclass="toctree-l2"><aclass="reference internal"href="client-testing.html#testing-error-handling-with-buggify">Testing Error Handling with Buggify</a></li>
<liclass="toctree-l2"><aclass="reference internal"href="client-testing.html#simulation-and-cluster-workloads">Simulation and Cluster Workloads</a></li>
<spanid="backups"></span><h1>Backup, Restore, and Replication for Disaster Recovery</h1>
<p>This document covers backup and restoration of a FoundationDB database. While FoundationDB itself is fault tolerant, the backup tool provides an additional level of protection by supporting recovery from disasters or unintentional modification of the database.</p>
<p>FoundationDB’s backup tool makes a consistent, point-in-time backup of a FoundationDB database without downtime. Like FoundationDB itself, the backup/restore software is distributed, with multiple backup agents cooperating to perform a backup or restore faster than a single machine can send or receive data and to continue the backup process seamlessly even when some backup agents fail.</p>
<p>The FoundationDB database usually cannot maintain a consistent snapshot long enough to read the entire database, so full backup consists of an <em>inconsistent</em> copy of the data with a log of database changes that took place during the creation of that inconsistent copy. During a restore, the inconsistent copy and the log of changes are combined to reconstruct a consistent, point-in-time snapshot of the original database.</p>
<p>A FoundationDB Backup job can run continuously, pushing multiple inconsistent snapshots and logs of changes over time to maintain the backup’s restorable point-in-time very close to now.</p>
<p>Backing up one database to another is a special form of backup is called DR backup or just DR for short. DR stands for Disaster Recovery, as it can be used to keep two geographically separated databases in close synchronization to recover from a catastrophic disaster. Once a DR operation has reached ‘differential’ mode, the secondary database (the destination of the DR job) will always contain a <em>consistent</em> copy of the primary database (the source of the DR job) but it will be from some past point in time. If the primary database is lost and applications continue using the secondary database, the “ACI” in ACID is preserved but D (Durability) is lost for some amount of most recent changes. When DR is operating normally, the secondary database will lag behind the primary database by as little as a few seconds worth of database commits.</p>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">fdbbackup</span></code></dt><dd><p>This command line tool is used to control (but not execute) backup jobs and manage backup data. It can <codeclass="docutils literal notranslate"><spanclass="pre">start</span></code>, <codeclass="docutils literal notranslate"><spanclass="pre">modify</span></code> or <codeclass="docutils literal notranslate"><spanclass="pre">abort</span></code> a backup, <codeclass="docutils literal notranslate"><spanclass="pre">discontinue</span></code> a continuous backup, get the <codeclass="docutils literal notranslate"><spanclass="pre">status</span></code> of an ongoing backup, or <codeclass="docutils literal notranslate"><spanclass="pre">wait</span></code> for a backup to complete. It can also <codeclass="docutils literal notranslate"><spanclass="pre">describe</span></code>, <codeclass="docutils literal notranslate"><spanclass="pre">delete</span></code>, <codeclass="docutils literal notranslate"><spanclass="pre">expire</span></code> data in a backup, or <codeclass="docutils literal notranslate"><spanclass="pre">list</span></code> the backups at a destination folder URL.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">fdbrestore</span></code></dt><dd><p>This command line tool is used to control (but not execute) restore jobs. It can <codeclass="docutils literal notranslate"><spanclass="pre">start</span></code> or <codeclass="docutils literal notranslate"><spanclass="pre">abort</span></code> a restore, get the <codeclass="docutils literal notranslate"><spanclass="pre">status</span></code> of current and recent restore tasks, or <codeclass="docutils literal notranslate"><spanclass="pre">wait</span></code> for a restore task to complete while printing ongoing progress details.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">backup_agent</span></code></dt><dd><p>The backup agent is a daemon that actually executes the work of the backup and restore jobs. Any number of backup agents pointed at the same database will cooperate to perform backups and restores. The Backup URL specified for a backup or restore must be accessible by all <codeclass="docutils literal notranslate"><spanclass="pre">backup_agent</span></code> processes.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">fdbdr</span></code></dt><dd><p>This command line tool is used to control (but not execute) DR jobs - backups from one database to another. It can <codeclass="docutils literal notranslate"><spanclass="pre">start</span></code>, <codeclass="docutils literal notranslate"><spanclass="pre">abort</span></code> a DR job, or <codeclass="docutils literal notranslate"><spanclass="pre">switch</span></code> the DR direction. It can also get the <codeclass="docutils literal notranslate"><spanclass="pre">status</span></code> of a running DR job.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">dr_agent</span></code></dt><dd><p>The database backup agent is a daemon that actually executes the work of the DR jobs, writing snapshot and log data to the destination database. Any number of agents pointed at the same databases will cooperate to perform the backup.</p>
<p>By default, the FoundationDB packages are configured to start a single <codeclass="docutils literal notranslate"><spanclass="pre">backup_agent</span></code> process on each FoundationDB server. If you want to perform a backup to a network drive or blob store instance that is accessible to every server, you can immediately use the <codeclass="docutils literal notranslate"><spanclass="pre">fdbbackup</span><spanclass="pre">start</span></code> command from any machine with access to your cluster to start the backup</p>
<p>If instead you want to perform a backup to the local disk of a particular machine or machines which are not network accessible to the FoundationDB servers, then you should disable the backup agents on the FoundationDB servers. This is accomplished by commenting out all of the <codeclass="docutils literal notranslate"><spanclass="pre">[backup_agent.<ID>]</span></code> sections in <aclass="reference internal"href="configuration.html#foundationdb-conf"><spanclass="std std-ref">foundationdb.conf</span></a>. Do not comment out the global <codeclass="docutils literal notranslate"><spanclass="pre">[backup_agent]</span></code> section. Next, start backup agents on the destination machine or machines. Now, when you start a backup, you can specify the destination directory (as a Backup URL) using a local path on the destination machines. The backup agents will fetch data from the database and store it locally on the destination machines.</p>
<p>Backup and Restore locations are specified by Backup URLs. Currently there are two valid Backup URL formats.</p>
<p>Note that items in angle brackets (< and >) are just placeholders and must be replaced (including the brackets) with meaningful values. Items within square brackets ([ and ]) are optional.</p>
<p>For local directories, the Backup URL format is</p>
<p>An example would be <codeclass="docutils literal notranslate"><spanclass="pre">file:///home/backups</span></code> which would refer to the directory <codeclass="docutils literal notranslate"><spanclass="pre">/home/backups</span></code>.
Note that since paths must be absolute this will result in three slashes (/) in a row in the URL.</p>
<p>Note that for local directory URLs the actual backup files will not be written to <base_dir> directly but rather to a uniquely timestamped subdirectory. When starting a restore the path to the timestamped subdirectory must be specified.</p>
<p>For blob store backup locations, the Backup URL format is</p>
<name> - Name of the backup within the backup bucket. It can contain '/' characters in order to organize backups into a folder-like structure.
<bucket_name> - Name of the bucket to use for backup data.
<region_name> - If <hostname> is not in s3 compatible form (s3.region-name.example.com) and aws v4 signature is enabled, region name is required.
<p>If <secret> is not specified, it will be looked up in <aclass="reference internal"href="#blob-credential-files"><spanclass="std std-ref">blob credential sources</span></a>.</p>
<p>An example blob store Backup URL would be <codeclass="docutils literal notranslate"><spanclass="pre">blobstore://myKey:mySecret@something.domain.com:80/dec_1_2017_0400?bucket=backups</span></code>.</p>
<p>Blob store Backup URLs can have optional parameters at the end which set various limits or options used when communicating with the store. All values must be positive decimal integers unless otherwise specified. The speed related default values are not very restrictive. The most likely parameter a user would want to change is <codeclass="docutils literal notranslate"><spanclass="pre">max_send_bytes_per_second</span></code> (or <codeclass="docutils literal notranslate"><spanclass="pre">sbps</span></code> for short) which determines the upload speed to the blob service.</p>
<p><em>header</em> - Add an additional HTTP header to each blob store REST API request. Can be specified multiple times. Format is <em>header=<FieldName>:<FieldValue></em> where both strings are non-empty.</p>
<p><em>sdk_auth</em> (or <em>sa</em>) - Use the AWS SDK to do credentials and authentication. This supports all aws authentication types, including credential-less iam role-based authentication in aws. Experimental, and only works if FDB was compiled with BUILD_AWS_BACKUP=ON. When this parameter is set, all other credential parts of the backup url can be ignored.</p>
<div><p><strong>Example</strong>: The URL parameter <em>header=x-amz-storage-class:REDUCED_REDUNDANCY</em> would send the HTTP header required to use the reduced redundancy storage option in the S3 API.</p>
<p>AWS signature version 4 is the default signing protocol choice. This boolean knob <codeclass="docutils literal notranslate"><spanclass="pre">--knob_http_request_aws_v4_header</span></code> can be used to select between v4 style and v2 style signatures.
If the knob is set to <codeclass="docutils literal notranslate"><spanclass="pre">true</span></code> then v4 signature will be used and if set to <codeclass="docutils literal notranslate"><spanclass="pre">false</span></code> then v2 signature will be used.</p>
<p>In order to help safeguard blob store credentials, the <SECRET> can optionally be omitted from blobstore:// URLs on the command line. Omitted secrets will be resolved at connect time using 1 or more Blob Credential files.</p>
<p>Blob Credential files can be specified on the command line (via –blob-credentials <FILE>) or via the environment variable FDB_BLOB_CREDENTIALS which can be set to a colon-separated list of files. The command line takes priority over the environment variable however all files from both sources will be used.</p>
<p>At connect time, the specified files are read in order and the first matching account specification (<aclass="reference external"href="mailto:user%40host">user<span>@</span>host</a>)
will be used to obtain the secret key.</p>
<p>The Blob Credential File format is JSON with the following schema:</p>
<p>In-flight traffic for blob store or disaster recovery backups can be encrypted with the following environment variables. They are also offered as command-line flags or can be specified in <codeclass="docutils literal notranslate"><spanclass="pre">foundationdb.conf</span></code> for backup agents.</p>
<p>Blob store backups can be configured to use HTTPS/TLS by setting the <codeclass="docutils literal notranslate"><spanclass="pre">secure_connection</span></code> or <codeclass="docutils literal notranslate"><spanclass="pre">sc</span></code> backup URL option to <codeclass="docutils literal notranslate"><spanclass="pre">1</span></code>, which is the default. Disaster recovery backups are secured by using TLS for both the source and target clusters and setting the TLS options for the <codeclass="docutils literal notranslate"><spanclass="pre">fdbdr</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">dr_agent</span></code> commands.</p>
</section>
<sectionid="fdbbackup-command-line-tool">
<h2><codeclass="docutils literal notranslate"><spanclass="pre">fdbbackup</span></code> command line tool</h2>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">fdbbackup</span></code> command line tool is used to control backup jobs or to manage backup data.</p>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-C</span><spanclass="pre"><CLUSTER_FILE></span></code></dt><dd><p>Path to the cluster file that should be used to connect to the FoundationDB cluster you want to use. If not specified, a <aclass="reference internal"href="administration.html#default-cluster-file"><spanclass="std std-ref">default cluster file</span></a> will be used.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-d</span><spanclass="pre"><BACKUP_URL></span></code></dt><dd><p>The Backup URL which the subcommand should read, write, or modify. For <codeclass="docutils literal notranslate"><spanclass="pre">start</span></code> and <codeclass="docutils literal notranslate"><spanclass="pre">modify</span></code> operations, the Backup URL must be accessible by the <codeclass="docutils literal notranslate"><spanclass="pre">backup_agent</span></code> processes.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-t</span><spanclass="pre"><TAG></span></code></dt><dd><p>A “tag” is a named slot in which a backup task executes. Backups on different named tags make progress and are controlled independently, though their executions are handled by the same set of backup agent processes. Any number of unique backup tags can be active at once. It the tag is not specified, the default tag name “default” is used.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--blob-credentials</span><spanclass="pre"><FILE></span></code></dt><dd><p>Use FILE as a <aclass="reference internal"href="#blob-credential-files"><spanclass="std std-ref">Blob Credential File</span></a>. Can be used multiple times.</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">start</span></code> subcommand is used to start a backup. If there is already a backup in progress, the command will fail and the current backup will be unaffected. Otherwise, a backup is started. If the wait option is used, the command will wait for the backup to complete; otherwise, it returns immediately.</p>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-z</span></code></dt><dd><p>Perform the backup continuously rather than terminating once a restorable backup is achieved. Database mutations within the backup’s target key ranges will be continuously written to the backup as well as repeated inconsistent snapshots at the configured snapshot rate.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-s</span><spanclass="pre"><DURATION></span></code> or <codeclass="docutils literal notranslate"><spanclass="pre">--snapshot-interval</span><spanclass="pre"><DURATION></span></code></dt><dd><p>Specifies the duration, in seconds, of the inconsistent snapshots written to the backup in continuous mode. The default is 864000 which is 10 days.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--initial-snapshot-interval</span><spanclass="pre"><DURATION></span></code></dt><dd><p>Specifies the duration, in seconds, of the first inconsistent snapshot written to the backup. The default is 0, which means as fast as possible.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--partitioned-log-experimental</span></code></dt><dd><p>Specifies the backup uses the partitioned mutation logs generated by backup workers. Since FDB version 6.3, this option is experimental and requires using fast restore for restoring the database from the generated files. The default is to use non-partitioned mutation logs generated by backup agents.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-w</span></code></dt><dd><p>Wait for the backup to complete with behavior identical to that of the <aclass="reference internal"href="#backup-wait"><spanclass="std std-ref">wait command</span></a>.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-k</span><spanclass="pre">'<BEGIN>[</span><spanclass="pre"><END>]'</span></code></dt><dd><p>Specify a key range to be included in the backup. Can be used multiple times to specify multiple key ranges. The argument should be a single string containing either a BEGIN alone or both a BEGIN and END separated by a space. If only the BEGIN is specified, the END is assumed to be BEGIN + ‘xff’. If no key ranges are different, the default is all user keys (’’ to ‘xff’).</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">modify</span></code> subcommand is used to modify parameters of a running backup. All specified changes are made in a single transaction.</p>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-d</span><spanclass="pre"><BACKUP_URL></span></code></dt><dd><p>Sets a new Backup URL for the backup to write to. This is most likely to be used to change only URL parameters or account information. However, it can also be used to start writing to a new destination mid-backup. The new old location will cease gaining any additional restorability, while the new location will not be restorable until a new snapshot begins and completes. Full restorability would be regained, however, if the contents of the two destinations were to be combined by the user.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-s</span><spanclass="pre"><DURATION></span></code> or <codeclass="docutils literal notranslate"><spanclass="pre">--snapshot-interval</span><spanclass="pre"><DURATION></span></code></dt><dd><p>Sets a new duration for backup snapshots, in seconds.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--active-snapshot-interval</span><spanclass="pre"><DURATION></span></code></dt><dd><p>Sets new duration for the backup’s currently active snapshot, in seconds, relative to the start of the snapshot.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--verify-uid</span><spanclass="pre"><UID></span></code></dt><dd><p>Specifies a UID to verify against the BackupUID of the running backup. If provided, the UID is verified in the same transaction which sets the new backup parameters (if the UID matches).</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">abort</span></code> subcommand is used to abort a backup that is currently in progress. If there is no backup in progress, the command will return an error. The destination backup is NOT deleted automatically, and it may or may not be restorable depending on when the abort is done.</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">discontinue</span></code> subcommand is only available for backups that were started with the continuous (<codeclass="docutils literal notranslate"><spanclass="pre">-z</span></code>) option. Its effect is to discontinue the continuous backup. Note that the subcommand does <em>not</em> abort the backup; it simply allows the backup to complete as a noncontinuous backup would.</p>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-w</span></code></dt><dd><p>Wait for the backup to complete with behavior identical to that of the <aclass="reference internal"href="#backup-wait"><spanclass="std std-ref">wait command</span></a>.</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">wait</span></code> subcommand is used to wait for a backup to complete, which is useful for scripting purposes. If there is a backup in progress, it waits for it to complete or be aborted and returns a status based on the result of the backup. If there is no backup in progress, it returns immediately based on the result of the previous backup. The exit code is zero (success) if the backup was completed successfully and nonzero if it was aborted.</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">status</span></code> subcommand is used to get information on the current status of backup. It will show several backup metrics as well as recent errors which organized by whether or not they appear to be preventing backup progress.</p>
<divclass="highlight-default notranslate"><divclass="highlight"><pre><span></span>user@host$ fdbbackup status [-t <TAG>]
<p>If you cancel a delete operation while it is in progress the specified backup is in an unknown state and is likely no longer usable. Repeat the delete command to finish deleting the backup.</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">expire</span></code> subcommand will remove data from a backup prior to some point in time referred to as the ‘cutoff’.</p>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--expire-before-timestamp</span><spanclass="pre"><DATETIME></span></code></dt><dd><p>Specifies the expiration cutoff to DATETIME. Requires a cluster file and will use version/timestamp metadata in the database to convert DATETIME to a database commit version. DATETIME must be in the form “YYYY/MM/DD.HH:MI:SS+hhmm”, for example “2018/12/31.23:59:59-0800”.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--expire-before-version</span><spanclass="pre"><VERSION></span></code></dt><dd><p>Specifies the cutoff by a database commit version.</p>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--restorable-after-timestamp</span><spanclass="pre"><DATETIME></span></code></dt><dd><p>Specifies that the backup must be restorable to DATETIME and later. Requires a cluster file and will use version/timestamp metadata in the database to convert DATETIME to a database commit version. DATETIME must be in the form “YYYY/MM/DD.HH:MI:SS+hhmm”, for example “2018/12/31.23:59:59-0800”.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--restorable-after-version</span><spanclass="pre"><VERSION></span></code></dt><dd><p>Specifies that the backup must be restorable as of VERSION and later.</p>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-f</span></code> or <codeclass="docutils literal notranslate"><spanclass="pre">--force</span></code></dt><dd><p>If the designated cutoff will result in removal of data such that the backup’s restorability would be reduced to either unrestorable or less restorable than the optional restorability requirement then the –force option must be given or the result will be an error and no action will be taken.</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">describe</span></code> subcommand will analyze the given backup and print a summary of the snapshot and mutation data versions it contains as well as the version range of restorability the backup can currently provide.</p>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--version-timestamps</span></code></dt><dd><p>If the originating cluster is still available and is passed on the command line, this option can be specified in order for all versions in the output to also be converted to timestamps for better human readability.</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">list</span></code> subcommand will list the backups at a given ‘base’ or shortened Backup URL.</p>
<divclass="highlight-default notranslate"><divclass="highlight"><pre><span></span>user@host$ fdbbackup list -b <BASE_URL>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-b</span><spanclass="pre"><BASE_URL></span></code> or <codeclass="docutils literal notranslate"><spanclass="pre">--base-url</span><spanclass="pre"><BASE_URL></span></code></dt><dd><p>This a shortened Backup URL which looks just like a Backup URL but without the backup <name> so that the list command will discover and list all of the backups in the bucket.</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">tags</span></code> subcommand will list the tags of all backups on a source cluster.</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">cleanup</span></code> subcommand will list orphaned backups and DRs and optionally remove their mutations.</p>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--delete-data</span></code></dt><dd><p>This flag will cause <codeclass="docutils literal notranslate"><spanclass="pre">cleanup</span></code> to remove mutations for the most stale backup or DR.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--min-cleanup-seconds</span></code></dt><dd><p>Specifies the amount of time a backup or DR needs to be stale before <codeclass="docutils literal notranslate"><spanclass="pre">cleanup</span></code> will remove mutations for it. By default this is set to one hour.</p>
<h2><codeclass="docutils literal notranslate"><spanclass="pre">fdbrestore</span></code> command line tool</h2>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">fdbrestore</span></code> command line tool is used to control restore tasks. Note that a restore operation will not clear the target key ranges, for safety reasons, so you must manually clear the ranges to be restored prior to starting the restore.</p>
<p>It is your responsibility to ensure that no clients are accessing the database while it is being restored. During the restore process the database is in an inconsistent state, and writes that happen during the restore process might be partially or completely overwritten by restored data.</p>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-h</span></code></dt><dd><p>Get help on the <codeclass="docutils literal notranslate"><spanclass="pre">fdbrestore</span></code> command.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-t</span><spanclass="pre"><TAG></span></code></dt><dd><p>Specify the tag for the restore task. Multiple restore tasks can be in progress at once so long as each task uses a different tag. The default tag is “default”.</p>
<divclass="admonition warning">
<pclass="admonition-title">Warning</p>
<p>If multiple restore tasks are in progress they should be restoring to different prefixes or the result is undefined.</p>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--blob-credentials</span><spanclass="pre"><FILE></span></code></dt><dd><p>Use FILE as a <aclass="reference internal"href="#blob-credential-files"><spanclass="std std-ref">Blob Credential File</span></a>. Can be used multiple times.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--dest-cluster-file</span><spanclass="pre"><CONNFILE></span></code></dt><dd><p>Required. Path to the cluster file that should be used to connect to the FoundationDB cluster you are restoring to.</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">start</span></code> command will start a new restore on the specified (or default) tag. The command will fail if a tag is already in use by an active restore.</p>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-r</span><spanclass="pre"><BACKUP_URL></span></code></dt><dd><p>Required. Specifies the Backup URL for the source backup data to restore to the database. The source data must be accessible by the <codeclass="docutils literal notranslate"><spanclass="pre">backup_agent</span></code> processes for the cluster.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-w</span></code></dt><dd><p>Wait for the restore to reach a final state (such as complete) before exiting. Prints a progress update every few seconds. Behavior is identical to that of the wait command.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-k</span><spanclass="pre"><KEYS></span></code></dt><dd><p>Specify list of key ranges from the backup to restore to the database</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--remove-prefix</span><spanclass="pre"><PREFIX></span></code></dt><dd><p>Remove PREFIX from the keys read from the backup</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--add-prefix</span><spanclass="pre"><PREFIX></span></code></dt><dd><p>Add PREFIX to restored keys before writing them to the database</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-n</span></code></dt><dd><p>Perform a trial run without actually restoring any data.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-v</span><spanclass="pre"><VERSION></span></code></dt><dd><p>Instead of the latest version the backup can be restored to, restore to VERSION.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--timestamp</span><spanclass="pre"><DATETIME></span></code></dt><dd><p>Instead of the latest version the backup can be restored to, restore to a version from approximately the given timestamp. Requires orig_cluster_file to be specified. DATETIME must be in the form “YYYY/MM/DD.HH:MI:SS+hhmm”, for example “2018/12/31.23:59:59-0800”.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--orig-cluster-file</span><spanclass="pre"><CONNFILE></span></code></dt><dd><p>The cluster file for the original database from which the backup was created. The original database is only needed to convert a –timestamp argument to a database version.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--inconsistent-snapshot-only</span></code></dt><dd><p>Ignore mutation log files during the restore to speedup the process. Because only range files are restored, this option gives an inconsistent snapshot in most cases and is not recommended to use.</p>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--user-data</span></code></dt><dd><p>Restore only the user keyspace. This option should NOT be used alongside –system-metadata (below) and CANNOT be used alongside other specified key ranges.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--system-metadata</span></code></dt><dd><p>Restore only the relevant system keyspace. This option should NOT be used alongside –user-data (above) and CANNOT be used alongside other specified key ranges.</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">abort</span></code> command will stop an active backup on the specified (or default) tag. It will display the final state of the restore tag.</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">wait</span></code> command will wait for the restore on the specified (or default) tag to reach a final state (such as complete or abort) and then exit. While waiting it will prints a progress update every few seconds.</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">status</span></code> command will print a detailed status report on restore job progress. If a tag is specified, it will only show status for that specific tag, otherwise status for all tags will be shown.</p>
</section>
</section>
<sectionid="backup-agent-command-line-tool">
<h2><codeclass="docutils literal notranslate"><spanclass="pre">backup_agent</span></code> command line tool</h2>
<p><codeclass="docutils literal notranslate"><spanclass="pre">backup_agent</span></code> is started automatically on each server in the default configuration of FoundationDB, so you will not normally need to invoke it at the command line. One case in which you would need to do so would be to perform a backup to a destination which is not accessible via a shared filesystem.</p>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-h</span></code></dt><dd><p>Get help on the <codeclass="docutils literal notranslate"><spanclass="pre">backup_agent</span></code> command.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-v</span></code></dt><dd><p>Get the version of FoundationDB in use.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-C</span><spanclass="pre"><CLUSTER_FILE></span></code></dt><dd><p>Specify the path to the <codeclass="docutils literal notranslate"><spanclass="pre">fdb.cluster</span></code> file that should be used to connect to the FoundationDB cluster you want to back up.</p>
<p>If not specified, a <aclass="reference internal"href="administration.html#default-cluster-file"><spanclass="std std-ref">default cluster file</span></a> will be used.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">--blob-credentials</span><spanclass="pre"><FILE></span></code></dt><dd><p>Use FILE as a <aclass="reference internal"href="#blob-credential-files"><spanclass="std std-ref">Blob Credential File</span></a>. Can be used multiple times.</p>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-h</span></code></dt><dd><p>Get help on the <codeclass="docutils literal notranslate"><spanclass="pre">fdbdr</span></code> command.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-v</span></code></dt><dd><p>Get the version of FoundationDB in use.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-d</span><spanclass="pre"><CLUSTER_FILE></span></code></dt><dd><p>Specify the path to the <codeclass="docutils literal notranslate"><spanclass="pre">fdb.cluster</span></code> file for the destination cluster of the DR operation.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-s</span><spanclass="pre"><CLUSTER_FILE></span></code></dt><dd><p>Specify the path to the <codeclass="docutils literal notranslate"><spanclass="pre">fdb.cluster</span></code> file for the source cluster of the DR operation.</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">start</span></code> subcommand is used to start a DR backup. If there is already a DR backup in progress, the command will fail and the current DR backup will be unaffected.</p>
<dlclass="simple">
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-k</span><spanclass="pre">'<BEGIN>[</span><spanclass="pre"><END>]'</span></code></dt><dd><p>Specify a key range to be included in the DR. Can be used multiple times to specify multiple key ranges. The argument should be a single string containing either a BEGIN alone or both a BEGIN and END separated by a space. If only the BEGIN is specified, the END is assumed to be BEGIN + ‘xff’. If no key ranges are different, the default is all user keys (’’ to ‘xff’).</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">switch</span></code> subcommand is used to swap the source and destination database clusters of an active DR in differential mode. This means the destination will be unlocked and start streaming data into the source database, which will subsequently be locked.</p>
<p>This command requires both databases to be available. On the destination cluster, a <codeclass="docutils literal notranslate"><spanclass="pre">dr_agent</span></code> that points to the source cluster must be running. While the switch command is working, both databases will be locked for a few seconds.</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">abort</span></code> subcommand is used to abort a DR that is currently in progress. If there is no backup in progress, the command will return an error. If the DR had already reached differential status, the abort command will leave the destination database at consistent snapshot of the source database from sometime in the past.</p>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">status</span></code> subcommand is used to get information on the current status of DR backup. It tells whether or not there is a DR in progress and whether or not there are active DR agents. It will also report any errors that have been encountered by the DR agents.</p>
<dlclass="simple">
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-e</span><spanclass="pre"><LIMIT></span></code></dt><dd><p>Print the last (up to) <codeclass="docutils literal notranslate"><spanclass="pre"><LIMIT></span></code> errors that were logged into the database by backup agents. The default is 10.</p>
<h2><codeclass="docutils literal notranslate"><spanclass="pre">dr_agent</span></code> command line tool</h2>
<p>Unlike <codeclass="docutils literal notranslate"><spanclass="pre">backup_agent</span></code>, <codeclass="docutils literal notranslate"><spanclass="pre">dr_agent</span></code> is not started automatically in a default FoundationDB installation. A <codeclass="docutils literal notranslate"><spanclass="pre">dr_agent</span></code> needs the cluster files for both the source database and the destination database, and can only perform a backup in one direction (from source to destination) at a time.</p>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-h</span></code></dt><dd><p>Get help on the <codeclass="docutils literal notranslate"><spanclass="pre">fdbdr</span></code> command.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-v</span></code></dt><dd><p>Get the version of FoundationDB in use.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-d</span><spanclass="pre"><CLUSTER_FILE></span></code></dt><dd><p>Specify the path to the <codeclass="docutils literal notranslate"><spanclass="pre">fdb.cluster</span></code> file for the destination cluster of the DR operation.</p>
</dd>
<dt><codeclass="docutils literal notranslate"><spanclass="pre">-s</span><spanclass="pre"><CLUSTER_FILE></span></code></dt><dd><p>Specify the path to the <codeclass="docutils literal notranslate"><spanclass="pre">fdb.cluster</span></code> file for the source cluster of the DR operation.</p>
