wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
foundationdb/fdbbackup/backup.actor.cpp

3549 lines
124 KiB
C++
Raw Normal View History

Initial repository commit
2017-05-26 04:48:44 +08:00
/*
* backup.actor.cpp
*
* This source file is part of the FoundationDB open source project
*
* Copyright 2013-2018 Apple Inc. and the FoundationDB project authors
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* http://www.apache.org/licenses/LICENSE-2.0
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
Fix weird bug with boost interprocess Strangely, boost interprocess didn't compile with VS 2017. However, it does compile if it is included as the first thing. I don't quite know what is happening here, but for now this fix makes it that I am not blocked
2019-02-06 08:46:08 +08:00
#define BOOST_DATE_TIME_NO_LIB
#include <boost/interprocess/managed_shared_memory.hpp>
Remove including flow.h in actorcompiler.h, and fix resulting breakage. For files that required flow.h, and only got it through actorcompiler.h, their version of flow.h would have the actorcompiler #defines defined. Then, if it included a STL/boost file, the same breakage would result. This needs to not happen, so the include of flow.h in actorcompiler.h was removed.
2018-08-11 07:05:40 +08:00
#include "flow/flow.h"
Initial repository commit
2017-05-26 04:48:44 +08:00
#include "flow/FastAlloc.h"
#include "flow/serialize.h"
#include "flow/IRandom.h"
#include "flow/genericactors.actor.h"
fix: call initSignalSafeUnwind in fdbcli and backup to avoid hangs in slow task profiling.
2017-08-08 06:55:08 +08:00
#include "flow/SignalSafeUnwind.h"
Initial repository commit
2017-05-26 04:48:44 +08:00
#include "fdbclient/FDBTypes.h"
Proper fwd decl in BackupAgent Also BackupAgent.h -> BackupAgent.actor.h
2019-02-18 07:19:05 +08:00
#include "fdbclient/BackupAgent.actor.h"
Initial repository commit
2017-05-26 04:48:44 +08:00
#include "fdbclient/Status.h"
#include "fdbclient/BackupContainer.h"
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
#include "fdbclient/KeyBackedTypes.h"
Initial repository commit
2017-05-26 04:48:44 +08:00
#include "fdbclient/RunTransaction.actor.h"
Move AsyncFileBlobStore and related files to fdbclient.
2018-10-27 04:49:42 +08:00
#include "fdbclient/BlobStore.h"
Initial repository commit
2017-05-26 04:48:44 +08:00
#include "fdbclient/json_spirit/json_spirit_writer_template.h"
Move AsyncFileBlobStore and related files to fdbclient.
2018-10-27 04:49:42 +08:00
#include "fdbrpc/Platform.h"
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#include "fdbrpc/TLSConnection.h"
Move AsyncFileBlobStore and related files to fdbclient.
2018-10-27 04:49:42 +08:00
Initial repository commit
2017-05-26 04:48:44 +08:00
#include <stdarg.h>
#include <stdio.h>
fdbbackup: fix some print/scan format warnings Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-05 01:48:09 +08:00
#include <cinttypes>
Initial repository commit
2017-05-26 04:48:44 +08:00
#include <algorithm> // std::transform
#include <string>
#include <iostream>
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
#include <ctime>
Initial repository commit
2017-05-26 04:48:44 +08:00
using std::cout;
using std::endl;
#ifdef _WIN32
#define WIN32_LEAN_AND_MEAN
Fix weird bug with boost interprocess Strangely, boost interprocess didn't compile with VS 2017. However, it does compile if it is included as the first thing. I don't quite know what is happening here, but for now this fix makes it that I am not blocked
2019-02-06 08:46:08 +08:00
#define NOMINMAX
Initial repository commit
2017-05-26 04:48:44 +08:00
#include <Windows.h>
#endif
#include <time.h>
#ifdef __linux__
#include <execinfo.h>
#ifdef ALLOC_INSTRUMENTATION
#include <cxxabi.h>
#endif
#endif
Guard all versions.h to not break old WIN32 build
2019-02-09 04:24:32 +08:00
#if defined(CMAKE_BUILD) || !defined(WIN32)
Address comments from PR
2019-02-07 10:16:54 +08:00
#include "versions.h"
Guard all versions.h to not break old WIN32 build
2019-02-09 04:24:32 +08:00
#endif
Initial repository commit
2017-05-26 04:48:44 +08:00
#include "flow/SimpleOpt.h"
Rewrite all files to have #include actorcompiler.h as the last include.
2018-08-11 06:18:24 +08:00
#include "flow/actorcompiler.h" // This must be the last #include.
Initial repository commit
2017-05-26 04:48:44 +08:00
// Type of program being executed
enum enumProgramExe {
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
EXE_AGENT, EXE_BACKUP, EXE_RESTORE, EXE_DR_AGENT, EXE_DB_BACKUP, EXE_UNDEFINED
Initial repository commit
2017-05-26 04:48:44 +08:00
};
enum enumBackupType {
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
BACKUP_UNDEFINED=0, BACKUP_START, BACKUP_MODIFY, BACKUP_STATUS, BACKUP_ABORT, BACKUP_WAIT, BACKUP_DISCONTINUE, BACKUP_PAUSE, BACKUP_RESUME, BACKUP_EXPIRE, BACKUP_DELETE, BACKUP_DESCRIBE, BACKUP_LIST, BACKUP_DUMP
Initial repository commit
2017-05-26 04:48:44 +08:00
};
enum enumDBType {
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
DB_UNDEFINED=0, DB_START, DB_STATUS, DB_SWITCH, DB_ABORT, DB_PAUSE, DB_RESUME
Initial repository commit
2017-05-26 04:48:44 +08:00
};
enum enumRestoreType {
RESTORE_UNKNOWN, RESTORE_START, RESTORE_STATUS, RESTORE_ABORT, RESTORE_WAIT
};
//
enum {
// Backup constants
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
OPT_DESTCONTAINER, OPT_SNAPSHOTINTERVAL, OPT_ERRORLIMIT, OPT_NOSTOPWHENDONE,
Added fdbbackup expire options to calculate approximate version boundaries based on a number of days prior to the latest log file found in the backup container. This enables expiration operations based on time (with reasonable precision) without accessing the source cluster.
2018-12-19 10:55:44 +08:00
OPT_EXPIRE_BEFORE_VERSION, OPT_EXPIRE_BEFORE_DATETIME, OPT_EXPIRE_DELETE_BEFORE_DAYS,
OPT_EXPIRE_RESTORABLE_AFTER_VERSION, OPT_EXPIRE_RESTORABLE_AFTER_DATETIME, OPT_EXPIRE_MIN_RESTORABLE_DAYS,
Added fdbbackup describe --version_timestamps option which is now required for describe to print version timestamps. Improved help...slightly.
2018-01-23 07:50:28 +08:00
OPT_BASEURL, OPT_BLOB_CREDENTIALS, OPT_DESCRIBE_DEEP, OPT_DESCRIBE_TIMESTAMPS,
Added --json option to fdbbackup describe. Also added expired percentage indicator to snapshot details.
2019-03-07 06:14:06 +08:00
OPT_DUMP_BEGIN, OPT_DUMP_END, OPT_JSON,
Initial repository commit
2017-05-26 04:48:44 +08:00
// Backup and Restore constants
OPT_TAGNAME, OPT_BACKUPKEYS, OPT_WAITFORDONE,
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
// Backup Modify
OPT_MOD_ACTIVE_INTERVAL, OPT_MOD_VERIFY_UID,
Initial repository commit
2017-05-26 04:48:44 +08:00
// Restore constants
Restore now requires an explicit destination cluster file. Restore target database version can now be specified by a timestamp, which must be resolved using the cluster file for the cluster from which the backup originated.
2019-03-06 04:28:05 +08:00
OPT_RESTORECONTAINER, OPT_RESTORE_VERSION, OPT_RESTORE_TIMESTAMP, OPT_PREFIX_ADD, OPT_PREFIX_REMOVE, OPT_RESTORE_CLUSTERFILE_DEST, OPT_RESTORE_CLUSTERFILE_ORIG,
Initial repository commit
2017-05-26 04:48:44 +08:00
// Shared constants
OPT_CLUSTERFILE, OPT_QUIET, OPT_DRYRUN, OPT_FORCE,
OPT_HELP, OPT_DEVHELP, OPT_VERSION, OPT_PARENTPID, OPT_CRASHONERROR,
OPT_NOBUFSTDOUT, OPT_BUFSTDOUTERR, OPT_TRACE, OPT_TRACE_DIR,
Merge branch 'release-5.0' # Conflicts: # fdbbackup/backup.actor.cpp # fdbserver/ClusterController.actor.cpp # versions.target
2017-10-02 09:46:44 +08:00
OPT_KNOB, OPT_TRACE_LOG_GROUP, OPT_MEMLIMIT, OPT_LOCALITY,
Initial repository commit
2017-05-26 04:48:44 +08:00
//DB constants
OPT_SOURCE_CLUSTER,
OPT_DEST_CLUSTER,
Add --trace_format flag to fdbserver
2018-12-21 05:51:06 +08:00
OPT_CLEANUP,
Fix whitespace
2019-01-03 07:24:11 +08:00
OPT_TRACE_FORMAT
Initial repository commit
2017-05-26 04:48:44 +08:00
};
CSimpleOpt::SOption g_rgAgentOptions[] = {
#ifdef _WIN32
{ OPT_PARENTPID, "--parentpid", SO_REQ_SEP },
#endif
{ OPT_CLUSTERFILE, "-C", SO_REQ_SEP },
{ OPT_CLUSTERFILE, "--cluster_file", SO_REQ_SEP },
{ OPT_KNOB, "--knob_", SO_REQ_SEP },
{ OPT_VERSION, "--version", SO_NONE },
{ OPT_VERSION, "-v", SO_NONE },
{ OPT_QUIET, "-q", SO_NONE },
{ OPT_QUIET, "--quiet", SO_NONE },
{ OPT_TRACE, "--log", SO_NONE },
{ OPT_TRACE_DIR, "--logdir", SO_REQ_SEP },
s/--trace-format/--trace_format/
2018-12-21 07:58:26 +08:00
{ OPT_TRACE_FORMAT, "--trace_format", SO_REQ_SEP },
Added --loggroup option to all fdbbackup modes.
2019-01-04 16:15:51 +08:00
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_CRASHONERROR, "--crash", SO_NONE },
Add locality to backup agent and DR agent
2017-09-27 01:19:26 +08:00
{ OPT_LOCALITY, "--locality_", SO_REQ_SEP },
added configurable memory limits for backup and dr executables added a default memory limit of 8GB for fdbcli
2017-09-30 01:35:40 +08:00
{ OPT_MEMLIMIT, "-m", SO_REQ_SEP },
{ OPT_MEMLIMIT, "--memory", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_HELP, "-?", SO_NONE },
{ OPT_HELP, "-h", SO_NONE },
{ OPT_HELP, "--help", SO_NONE },
{ OPT_DEVHELP, "--dev-help", SO_NONE },
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
{ OPT_BLOB_CREDENTIALS, "--blob_credentials", SO_REQ_SEP },
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
TLS_OPTION_FLAGS
#endif
Initial repository commit
2017-05-26 04:48:44 +08:00
SO_END_OF_OPTIONS
};
CSimpleOpt::SOption g_rgBackupStartOptions[] = {
#ifdef _WIN32
{ OPT_PARENTPID, "--parentpid", SO_REQ_SEP },
#endif
{ OPT_CLUSTERFILE, "-C", SO_REQ_SEP },
{ OPT_CLUSTERFILE, "--cluster_file", SO_REQ_SEP },
{ OPT_WAITFORDONE, "-w", SO_NONE },
{ OPT_WAITFORDONE, "--waitfordone", SO_NONE },
{ OPT_NOSTOPWHENDONE, "-z", SO_NONE },
{ OPT_NOSTOPWHENDONE, "--no-stop-when-done",SO_NONE },
{ OPT_DESTCONTAINER, "-d", SO_REQ_SEP },
{ OPT_DESTCONTAINER, "--destcontainer", SO_REQ_SEP },
Added snapshot interval option to "fdbbackup start" which defaults to a new knob's value. Added snapshot info to backup status text. Improvements to fdbbackup help.
2017-12-20 16:49:08 +08:00
{ OPT_SNAPSHOTINTERVAL, "-s", SO_REQ_SEP },
{ OPT_SNAPSHOTINTERVAL, "--snapshot_interval", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_TAGNAME, "-t", SO_REQ_SEP },
{ OPT_TAGNAME, "--tagname", SO_REQ_SEP },
{ OPT_BACKUPKEYS, "-k", SO_REQ_SEP },
{ OPT_BACKUPKEYS, "--keys", SO_REQ_SEP },
{ OPT_DRYRUN, "-n", SO_NONE },
{ OPT_DRYRUN, "--dryrun", SO_NONE },
{ OPT_TRACE, "--log", SO_NONE },
{ OPT_TRACE_DIR, "--logdir", SO_REQ_SEP },
s/--trace-format/--trace_format/
2018-12-21 07:58:26 +08:00
{ OPT_TRACE_FORMAT, "--trace_format", SO_REQ_SEP },
Added --loggroup option to all fdbbackup modes.
2019-01-04 16:15:51 +08:00
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_QUIET, "-q", SO_NONE },
{ OPT_QUIET, "--quiet", SO_NONE },
{ OPT_VERSION, "--version", SO_NONE },
{ OPT_VERSION, "-v", SO_NONE },
{ OPT_CRASHONERROR, "--crash", SO_NONE },
added configurable memory limits for backup and dr executables added a default memory limit of 8GB for fdbcli
2017-09-30 01:35:40 +08:00
{ OPT_MEMLIMIT, "-m", SO_REQ_SEP },
{ OPT_MEMLIMIT, "--memory", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_HELP, "-?", SO_NONE },
{ OPT_HELP, "-h", SO_NONE },
{ OPT_HELP, "--help", SO_NONE },
{ OPT_DEVHELP, "--dev-help", SO_NONE },
{ OPT_KNOB, "--knob_", SO_REQ_SEP },
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
{ OPT_BLOB_CREDENTIALS, "--blob_credentials", SO_REQ_SEP },
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
TLS_OPTION_FLAGS
#endif
Initial repository commit
2017-05-26 04:48:44 +08:00
SO_END_OF_OPTIONS
};
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
CSimpleOpt::SOption g_rgBackupModifyOptions[] = {
#ifdef _WIN32
{ OPT_PARENTPID, "--parentpid", SO_REQ_SEP },
#endif
{ OPT_TRACE, "--log", SO_NONE },
{ OPT_TRACE_DIR, "--logdir", SO_REQ_SEP },
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
{ OPT_QUIET, "-q", SO_NONE },
{ OPT_QUIET, "--quiet", SO_NONE },
{ OPT_VERSION, "-v", SO_NONE },
{ OPT_VERSION, "--version", SO_NONE },
{ OPT_CRASHONERROR, "--crash", SO_NONE },
{ OPT_MEMLIMIT, "-m", SO_REQ_SEP },
{ OPT_MEMLIMIT, "--memory", SO_REQ_SEP },
{ OPT_HELP, "-?", SO_NONE },
{ OPT_HELP, "-h", SO_NONE },
{ OPT_HELP, "--help", SO_NONE },
{ OPT_DEVHELP, "--dev-help", SO_NONE },
{ OPT_BLOB_CREDENTIALS, "--blob_credentials", SO_REQ_SEP },
{ OPT_KNOB, "--knob_", SO_REQ_SEP },
{ OPT_CLUSTERFILE, "-C", SO_REQ_SEP },
{ OPT_CLUSTERFILE, "--cluster_file", SO_REQ_SEP },
{ OPT_TAGNAME, "-t", SO_REQ_SEP },
{ OPT_TAGNAME, "--tagname", SO_REQ_SEP },
{ OPT_MOD_VERIFY_UID, "--verify_uid", SO_REQ_SEP },
{ OPT_DESTCONTAINER, "-d", SO_REQ_SEP },
{ OPT_DESTCONTAINER, "--destcontainer", SO_REQ_SEP },
{ OPT_SNAPSHOTINTERVAL, "-s", SO_REQ_SEP },
{ OPT_SNAPSHOTINTERVAL, "--snapshot_interval", SO_REQ_SEP },
{ OPT_MOD_ACTIVE_INTERVAL, "--active_snapshot_interval", SO_REQ_SEP },
SO_END_OF_OPTIONS
};
Initial repository commit
2017-05-26 04:48:44 +08:00
CSimpleOpt::SOption g_rgBackupStatusOptions[] = {
#ifdef _WIN32
{ OPT_PARENTPID, "--parentpid", SO_REQ_SEP },
#endif
{ OPT_CLUSTERFILE, "-C", SO_REQ_SEP },
{ OPT_CLUSTERFILE, "--cluster_file", SO_REQ_SEP },
{ OPT_ERRORLIMIT, "-e", SO_REQ_SEP },
{ OPT_ERRORLIMIT, "--errorlimit", SO_REQ_SEP },
{ OPT_TAGNAME, "-t", SO_REQ_SEP },
{ OPT_TAGNAME, "--tagname", SO_REQ_SEP },
{ OPT_TRACE, "--log", SO_NONE },
{ OPT_TRACE_DIR, "--logdir", SO_REQ_SEP },
s/--trace-format/--trace_format/
2018-12-21 07:58:26 +08:00
{ OPT_TRACE_FORMAT, "--trace_format", SO_REQ_SEP },
Added --loggroup option to all fdbbackup modes.
2019-01-04 16:15:51 +08:00
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_VERSION, "--version", SO_NONE },
{ OPT_VERSION, "-v", SO_NONE },
{ OPT_QUIET, "-q", SO_NONE },
{ OPT_QUIET, "--quiet", SO_NONE },
{ OPT_CRASHONERROR, "--crash", SO_NONE },
added configurable memory limits for backup and dr executables added a default memory limit of 8GB for fdbcli
2017-09-30 01:35:40 +08:00
{ OPT_MEMLIMIT, "-m", SO_REQ_SEP },
{ OPT_MEMLIMIT, "--memory", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_HELP, "-?", SO_NONE },
{ OPT_HELP, "-h", SO_NONE },
{ OPT_HELP, "--help", SO_NONE },
{ OPT_DEVHELP, "--dev-help", SO_NONE },
Added --json option to fdbbackup status.
2019-03-07 13:32:46 +08:00
{ OPT_JSON, "--json", SO_NONE},
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
TLS_OPTION_FLAGS
#endif
Initial repository commit
2017-05-26 04:48:44 +08:00
SO_END_OF_OPTIONS
};
CSimpleOpt::SOption g_rgBackupAbortOptions[] = {
#ifdef _WIN32
{ OPT_PARENTPID, "--parentpid", SO_REQ_SEP },
#endif
{ OPT_CLUSTERFILE, "-C", SO_REQ_SEP },
{ OPT_CLUSTERFILE, "--cluster_file", SO_REQ_SEP },
{ OPT_TAGNAME, "-t", SO_REQ_SEP },
{ OPT_TAGNAME, "--tagname", SO_REQ_SEP },
{ OPT_TRACE, "--log", SO_NONE },
{ OPT_TRACE_DIR, "--logdir", SO_REQ_SEP },
s/--trace-format/--trace_format/
2018-12-21 07:58:26 +08:00
{ OPT_TRACE_FORMAT, "--trace_format", SO_REQ_SEP },
Added --loggroup option to all fdbbackup modes.
2019-01-04 16:15:51 +08:00
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_QUIET, "-q", SO_NONE },
{ OPT_QUIET, "--quiet", SO_NONE },
{ OPT_VERSION, "--version", SO_NONE },
{ OPT_VERSION, "-v", SO_NONE },
{ OPT_CRASHONERROR, "--crash", SO_NONE },
added configurable memory limits for backup and dr executables added a default memory limit of 8GB for fdbcli
2017-09-30 01:35:40 +08:00
{ OPT_MEMLIMIT, "-m", SO_REQ_SEP },
{ OPT_MEMLIMIT, "--memory", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_HELP, "-?", SO_NONE },
{ OPT_HELP, "-h", SO_NONE },
{ OPT_HELP, "--help", SO_NONE },
{ OPT_DEVHELP, "--dev-help", SO_NONE },
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
TLS_OPTION_FLAGS
#endif
Initial repository commit
2017-05-26 04:48:44 +08:00
SO_END_OF_OPTIONS
};
CSimpleOpt::SOption g_rgBackupDiscontinueOptions[] = {
#ifdef _WIN32
{ OPT_PARENTPID, "--parentpid", SO_REQ_SEP },
#endif
{ OPT_CLUSTERFILE, "-C", SO_REQ_SEP },
{ OPT_CLUSTERFILE, "--cluster_file", SO_REQ_SEP },
{ OPT_TAGNAME, "-t", SO_REQ_SEP },
{ OPT_TAGNAME, "--tagname", SO_REQ_SEP },
{ OPT_WAITFORDONE, "-w", SO_NONE },
{ OPT_WAITFORDONE, "--waitfordone", SO_NONE },
{ OPT_TRACE, "--log", SO_NONE },
{ OPT_TRACE_DIR, "--logdir", SO_REQ_SEP },
s/--trace-format/--trace_format/
2018-12-21 07:58:26 +08:00
{ OPT_TRACE_FORMAT, "--trace_format", SO_REQ_SEP },
Added --loggroup option to all fdbbackup modes.
2019-01-04 16:15:51 +08:00
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_QUIET, "-q", SO_NONE },
{ OPT_QUIET, "--quiet", SO_NONE },
{ OPT_VERSION, "--version", SO_NONE },
{ OPT_VERSION, "-v", SO_NONE },
{ OPT_CRASHONERROR, "--crash", SO_NONE },
added configurable memory limits for backup and dr executables added a default memory limit of 8GB for fdbcli
2017-09-30 01:35:40 +08:00
{ OPT_MEMLIMIT, "-m", SO_REQ_SEP },
{ OPT_MEMLIMIT, "--memory", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_HELP, "-?", SO_NONE },
{ OPT_HELP, "-h", SO_NONE },
{ OPT_HELP, "--help", SO_NONE },
{ OPT_DEVHELP, "--dev-help", SO_NONE },
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
TLS_OPTION_FLAGS
#endif
Initial repository commit
2017-05-26 04:48:44 +08:00
SO_END_OF_OPTIONS
};
CSimpleOpt::SOption g_rgBackupWaitOptions[] = {
#ifdef _WIN32
{ OPT_PARENTPID, "--parentpid", SO_REQ_SEP },
#endif
{ OPT_CLUSTERFILE, "-C", SO_REQ_SEP },
{ OPT_CLUSTERFILE, "--cluster_file", SO_REQ_SEP },
{ OPT_TAGNAME, "-t", SO_REQ_SEP },
{ OPT_TAGNAME, "--tagname", SO_REQ_SEP },
{ OPT_NOSTOPWHENDONE, "-z", SO_NONE },
{ OPT_NOSTOPWHENDONE, "--no-stop-when-done",SO_NONE },
{ OPT_TRACE, "--log", SO_NONE },
{ OPT_TRACE_DIR, "--logdir", SO_REQ_SEP },
s/--trace-format/--trace_format/
2018-12-21 07:58:26 +08:00
{ OPT_TRACE_FORMAT, "--trace_format", SO_REQ_SEP },
Added --loggroup option to all fdbbackup modes.
2019-01-04 16:15:51 +08:00
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_QUIET, "-q", SO_NONE },
{ OPT_QUIET, "--quiet", SO_NONE },
{ OPT_VERSION, "--version", SO_NONE },
{ OPT_VERSION, "-v", SO_NONE },
{ OPT_CRASHONERROR, "--crash", SO_NONE },
added configurable memory limits for backup and dr executables added a default memory limit of 8GB for fdbcli
2017-09-30 01:35:40 +08:00
{ OPT_MEMLIMIT, "-m", SO_REQ_SEP },
{ OPT_MEMLIMIT, "--memory", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_HELP, "-?", SO_NONE },
{ OPT_HELP, "-h", SO_NONE },
{ OPT_HELP, "--help", SO_NONE },
{ OPT_DEVHELP, "--dev-help", SO_NONE },
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
TLS_OPTION_FLAGS
#endif
Initial repository commit
2017-05-26 04:48:44 +08:00
SO_END_OF_OPTIONS
};
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
CSimpleOpt::SOption g_rgBackupPauseOptions[] = {
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
#ifdef _WIN32
{ OPT_PARENTPID, "--parentpid", SO_REQ_SEP },
#endif
{ OPT_CLUSTERFILE, "-C", SO_REQ_SEP },
{ OPT_CLUSTERFILE, "--cluster_file", SO_REQ_SEP },
{ OPT_TRACE, "--log", SO_NONE },
{ OPT_TRACE_DIR, "--logdir", SO_REQ_SEP },
s/--trace-format/--trace_format/
2018-12-21 07:58:26 +08:00
{ OPT_TRACE_FORMAT, "--trace_format", SO_REQ_SEP },
Added --loggroup option to all fdbbackup modes.
2019-01-04 16:15:51 +08:00
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
{ OPT_QUIET, "-q", SO_NONE },
{ OPT_QUIET, "--quiet", SO_NONE },
{ OPT_VERSION, "--version", SO_NONE },
{ OPT_VERSION, "-v", SO_NONE },
{ OPT_CRASHONERROR, "--crash", SO_NONE },
{ OPT_MEMLIMIT, "-m", SO_REQ_SEP },
{ OPT_MEMLIMIT, "--memory", SO_REQ_SEP },
{ OPT_HELP, "-?", SO_NONE },
{ OPT_HELP, "-h", SO_NONE },
{ OPT_HELP, "--help", SO_NONE },
{ OPT_DEVHELP, "--dev-help", SO_NONE },
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
TLS_OPTION_FLAGS
#endif
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
SO_END_OF_OPTIONS
};
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
CSimpleOpt::SOption g_rgBackupExpireOptions[] = {
#ifdef _WIN32
{ OPT_PARENTPID, "--parentpid", SO_REQ_SEP },
#endif
{ OPT_CLUSTERFILE, "-C", SO_REQ_SEP },
{ OPT_CLUSTERFILE, "--cluster_file", SO_REQ_SEP },
{ OPT_DESTCONTAINER, "-d", SO_REQ_SEP },
{ OPT_DESTCONTAINER, "--destcontainer", SO_REQ_SEP },
{ OPT_TRACE, "--log", SO_NONE },
{ OPT_TRACE_DIR, "--logdir", SO_REQ_SEP },
s/--trace-format/--trace_format/
2018-12-21 07:58:26 +08:00
{ OPT_TRACE_FORMAT, "--trace_format", SO_REQ_SEP },
Added --loggroup option to all fdbbackup modes.
2019-01-04 16:15:51 +08:00
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
{ OPT_QUIET, "-q", SO_NONE },
{ OPT_QUIET, "--quiet", SO_NONE },
{ OPT_VERSION, "-v", SO_NONE },
{ OPT_VERSION, "--version", SO_NONE },
{ OPT_CRASHONERROR, "--crash", SO_NONE },
{ OPT_MEMLIMIT, "-m", SO_REQ_SEP },
{ OPT_MEMLIMIT, "--memory", SO_REQ_SEP },
{ OPT_HELP, "-?", SO_NONE },
{ OPT_HELP, "-h", SO_NONE },
{ OPT_HELP, "--help", SO_NONE },
{ OPT_DEVHELP, "--dev-help", SO_NONE },
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
{ OPT_BLOB_CREDENTIALS, "--blob_credentials", SO_REQ_SEP },
{ OPT_KNOB, "--knob_", SO_REQ_SEP },
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
{ OPT_FORCE, "-f", SO_NONE },
{ OPT_FORCE, "--force", SO_NONE },
{ OPT_EXPIRE_RESTORABLE_AFTER_VERSION, "--restorable_after_version", SO_REQ_SEP },
{ OPT_EXPIRE_RESTORABLE_AFTER_DATETIME, "--restorable_after_timestamp", SO_REQ_SEP },
{ OPT_EXPIRE_BEFORE_VERSION, "--expire_before_version", SO_REQ_SEP },
{ OPT_EXPIRE_BEFORE_DATETIME, "--expire_before_timestamp", SO_REQ_SEP },
Added fdbbackup expire options to calculate approximate version boundaries based on a number of days prior to the latest log file found in the backup container. This enables expiration operations based on time (with reasonable precision) without accessing the source cluster.
2018-12-19 10:55:44 +08:00
{ OPT_EXPIRE_MIN_RESTORABLE_DAYS, "--min_restorable_days", SO_REQ_SEP },
{ OPT_EXPIRE_DELETE_BEFORE_DAYS, "--delete_before_days", SO_REQ_SEP },
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
TLS_OPTION_FLAGS
#endif
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
SO_END_OF_OPTIONS
};
CSimpleOpt::SOption g_rgBackupDeleteOptions[] = {
#ifdef _WIN32
{ OPT_PARENTPID, "--parentpid", SO_REQ_SEP },
#endif
{ OPT_DESTCONTAINER, "-d", SO_REQ_SEP },
{ OPT_DESTCONTAINER, "--destcontainer", SO_REQ_SEP },
{ OPT_TRACE, "--log", SO_NONE },
{ OPT_TRACE_DIR, "--logdir", SO_REQ_SEP },
s/--trace-format/--trace_format/
2018-12-21 07:58:26 +08:00
{ OPT_TRACE_FORMAT, "--trace_format", SO_REQ_SEP },
Added --loggroup option to all fdbbackup modes.
2019-01-04 16:15:51 +08:00
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
{ OPT_QUIET, "-q", SO_NONE },
{ OPT_QUIET, "--quiet", SO_NONE },
{ OPT_VERSION, "-v", SO_NONE },
{ OPT_VERSION, "--version", SO_NONE },
{ OPT_CRASHONERROR, "--crash", SO_NONE },
{ OPT_MEMLIMIT, "-m", SO_REQ_SEP },
{ OPT_MEMLIMIT, "--memory", SO_REQ_SEP },
{ OPT_HELP, "-?", SO_NONE },
{ OPT_HELP, "-h", SO_NONE },
{ OPT_HELP, "--help", SO_NONE },
{ OPT_DEVHELP, "--dev-help", SO_NONE },
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
{ OPT_BLOB_CREDENTIALS, "--blob_credentials", SO_REQ_SEP },
{ OPT_KNOB, "--knob_", SO_REQ_SEP },
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
TLS_OPTION_FLAGS
#endif
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
SO_END_OF_OPTIONS
};
CSimpleOpt::SOption g_rgBackupDescribeOptions[] = {
#ifdef _WIN32
{ OPT_PARENTPID, "--parentpid", SO_REQ_SEP },
#endif
Added fdbbackup describe --version_timestamps option which is now required for describe to print version timestamps. Improved help...slightly.
2018-01-23 07:50:28 +08:00
{ OPT_CLUSTERFILE, "-C", SO_REQ_SEP },
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
{ OPT_CLUSTERFILE, "--cluster_file", SO_REQ_SEP },
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
{ OPT_DESTCONTAINER, "-d", SO_REQ_SEP },
{ OPT_DESTCONTAINER, "--destcontainer", SO_REQ_SEP },
{ OPT_TRACE, "--log", SO_NONE },
{ OPT_TRACE_DIR, "--logdir", SO_REQ_SEP },
s/--trace-format/--trace_format/
2018-12-21 07:58:26 +08:00
{ OPT_TRACE_FORMAT, "--trace_format", SO_REQ_SEP },
Added --loggroup option to all fdbbackup modes.
2019-01-04 16:15:51 +08:00
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
{ OPT_QUIET, "-q", SO_NONE },
{ OPT_QUIET, "--quiet", SO_NONE },
{ OPT_VERSION, "-v", SO_NONE },
{ OPT_VERSION, "--version", SO_NONE },
{ OPT_CRASHONERROR, "--crash", SO_NONE },
{ OPT_MEMLIMIT, "-m", SO_REQ_SEP },
{ OPT_MEMLIMIT, "--memory", SO_REQ_SEP },
{ OPT_HELP, "-?", SO_NONE },
{ OPT_HELP, "-h", SO_NONE },
{ OPT_HELP, "--help", SO_NONE },
{ OPT_DEVHELP, "--dev-help", SO_NONE },
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
{ OPT_BLOB_CREDENTIALS, "--blob_credentials", SO_REQ_SEP },
{ OPT_KNOB, "--knob_", SO_REQ_SEP },
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
{ OPT_DESCRIBE_DEEP, "--deep", SO_NONE },
Added fdbbackup describe --version_timestamps option which is now required for describe to print version timestamps. Improved help...slightly.
2018-01-23 07:50:28 +08:00
{ OPT_DESCRIBE_TIMESTAMPS, "--version_timestamps", SO_NONE },
Added --json option to fdbbackup describe. Also added expired percentage indicator to snapshot details.
2019-03-07 06:14:06 +08:00
{ OPT_JSON, "--json", SO_NONE},
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
TLS_OPTION_FLAGS
#endif
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
SO_END_OF_OPTIONS
};
Added fdbbackup dump option to list files and sizes in a backup for a given version range.
2019-01-03 18:35:31 +08:00
CSimpleOpt::SOption g_rgBackupDumpOptions[] = {
#ifdef _WIN32
{ OPT_PARENTPID, "--parentpid", SO_REQ_SEP },
#endif
{ OPT_CLUSTERFILE, "-C", SO_REQ_SEP },
{ OPT_CLUSTERFILE, "--cluster_file", SO_REQ_SEP },
{ OPT_DESTCONTAINER, "-d", SO_REQ_SEP },
{ OPT_DESTCONTAINER, "--destcontainer", SO_REQ_SEP },
{ OPT_TRACE, "--log", SO_NONE },
{ OPT_TRACE_DIR, "--logdir", SO_REQ_SEP },
Added --loggroup option to all fdbbackup modes.
2019-01-04 16:15:51 +08:00
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
Added fdbbackup dump option to list files and sizes in a backup for a given version range.
2019-01-03 18:35:31 +08:00
{ OPT_QUIET, "-q", SO_NONE },
{ OPT_QUIET, "--quiet", SO_NONE },
{ OPT_VERSION, "-v", SO_NONE },
{ OPT_VERSION, "--version", SO_NONE },
{ OPT_CRASHONERROR, "--crash", SO_NONE },
{ OPT_MEMLIMIT, "-m", SO_REQ_SEP },
{ OPT_MEMLIMIT, "--memory", SO_REQ_SEP },
{ OPT_HELP, "-?", SO_NONE },
{ OPT_HELP, "-h", SO_NONE },
{ OPT_HELP, "--help", SO_NONE },
{ OPT_DEVHELP, "--dev-help", SO_NONE },
{ OPT_BLOB_CREDENTIALS, "--blob_credentials", SO_REQ_SEP },
{ OPT_KNOB, "--knob_", SO_REQ_SEP },
{ OPT_DUMP_BEGIN, "--begin", SO_REQ_SEP },
{ OPT_DUMP_END, "--end", SO_REQ_SEP },
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
TLS_OPTION_FLAGS
#endif
Added fdbbackup dump option to list files and sizes in a backup for a given version range.
2019-01-03 18:35:31 +08:00
SO_END_OF_OPTIONS
};
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
CSimpleOpt::SOption g_rgBackupListOptions[] = {
#ifdef _WIN32
{ OPT_PARENTPID, "--parentpid", SO_REQ_SEP },
#endif
{ OPT_BASEURL, "-b", SO_REQ_SEP },
{ OPT_BASEURL, "--base_url", SO_REQ_SEP },
{ OPT_TRACE, "--log", SO_NONE },
{ OPT_TRACE_DIR, "--logdir", SO_REQ_SEP },
s/--trace-format/--trace_format/
2018-12-21 07:58:26 +08:00
{ OPT_TRACE_FORMAT, "--trace_format", SO_REQ_SEP },
Added --loggroup option to all fdbbackup modes.
2019-01-04 16:15:51 +08:00
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
{ OPT_QUIET, "-q", SO_NONE },
{ OPT_QUIET, "--quiet", SO_NONE },
{ OPT_VERSION, "-v", SO_NONE },
{ OPT_VERSION, "--version", SO_NONE },
{ OPT_CRASHONERROR, "--crash", SO_NONE },
{ OPT_MEMLIMIT, "-m", SO_REQ_SEP },
{ OPT_MEMLIMIT, "--memory", SO_REQ_SEP },
{ OPT_HELP, "-?", SO_NONE },
{ OPT_HELP, "-h", SO_NONE },
{ OPT_HELP, "--help", SO_NONE },
{ OPT_DEVHELP, "--dev-help", SO_NONE },
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
{ OPT_BLOB_CREDENTIALS, "--blob_credentials", SO_REQ_SEP },
{ OPT_KNOB, "--knob_", SO_REQ_SEP },
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
TLS_OPTION_FLAGS
#endif
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
SO_END_OF_OPTIONS
};
Initial repository commit
2017-05-26 04:48:44 +08:00
CSimpleOpt::SOption g_rgRestoreOptions[] = {
#ifdef _WIN32
{ OPT_PARENTPID, "--parentpid", SO_REQ_SEP },
#endif
Restore now requires an explicit destination cluster file. Restore target database version can now be specified by a timestamp, which must be resolved using the cluster file for the cluster from which the backup originated.
2019-03-06 04:28:05 +08:00
{ OPT_RESTORE_CLUSTERFILE_DEST, "--dest_cluster_file", SO_REQ_SEP },
{ OPT_RESTORE_CLUSTERFILE_ORIG, "--orig_cluster_file", SO_REQ_SEP },
Fixed option name to match documentation.
2019-03-06 15:26:45 +08:00
{ OPT_RESTORE_TIMESTAMP, "--timestamp", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_KNOB, "--knob_", SO_REQ_SEP },
{ OPT_RESTORECONTAINER,"-r", SO_REQ_SEP },
{ OPT_PREFIX_ADD, "-add_prefix", SO_REQ_SEP },
{ OPT_PREFIX_REMOVE, "-remove_prefix", SO_REQ_SEP },
{ OPT_TAGNAME, "-t", SO_REQ_SEP },
{ OPT_TAGNAME, "--tagname", SO_REQ_SEP },
{ OPT_BACKUPKEYS, "-k", SO_REQ_SEP },
{ OPT_BACKUPKEYS, "--keys", SO_REQ_SEP },
Restore now requires an explicit destination cluster file. Restore target database version can now be specified by a timestamp, which must be resolved using the cluster file for the cluster from which the backup originated.
2019-03-06 04:28:05 +08:00
{ OPT_WAITFORDONE, "-w", SO_NONE },
{ OPT_WAITFORDONE, "--waitfordone", SO_NONE },
{ OPT_RESTORE_VERSION, "--version", SO_REQ_SEP },
{ OPT_RESTORE_VERSION, "-v", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_TRACE, "--log", SO_NONE },
{ OPT_TRACE_DIR, "--logdir", SO_REQ_SEP },
s/--trace-format/--trace_format/
2018-12-21 07:58:26 +08:00
{ OPT_TRACE_FORMAT, "--trace_format", SO_REQ_SEP },
Added --loggroup option to all fdbbackup modes.
2019-01-04 16:15:51 +08:00
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_QUIET, "-q", SO_NONE },
{ OPT_QUIET, "--quiet", SO_NONE },
{ OPT_DRYRUN, "-n", SO_NONE },
{ OPT_DRYRUN, "--dryrun", SO_NONE },
{ OPT_FORCE, "-f", SO_NONE },
{ OPT_CRASHONERROR, "--crash", SO_NONE },
added configurable memory limits for backup and dr executables added a default memory limit of 8GB for fdbcli
2017-09-30 01:35:40 +08:00
{ OPT_MEMLIMIT, "-m", SO_REQ_SEP },
{ OPT_MEMLIMIT, "--memory", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_HELP, "-?", SO_NONE },
{ OPT_HELP, "-h", SO_NONE },
{ OPT_HELP, "--help", SO_NONE },
{ OPT_DEVHELP, "--dev-help", SO_NONE },
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
{ OPT_BLOB_CREDENTIALS, "--blob_credentials", SO_REQ_SEP },
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
TLS_OPTION_FLAGS
#endif
Initial repository commit
2017-05-26 04:48:44 +08:00
SO_END_OF_OPTIONS
};
CSimpleOpt::SOption g_rgDBAgentOptions[] = {
#ifdef _WIN32
{ OPT_PARENTPID, "--parentpid", SO_REQ_SEP },
#endif
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
{ OPT_SOURCE_CLUSTER, "-s", SO_REQ_SEP },
{ OPT_SOURCE_CLUSTER, "--source", SO_REQ_SEP },
{ OPT_DEST_CLUSTER, "-d", SO_REQ_SEP },
{ OPT_DEST_CLUSTER, "--destination", SO_REQ_SEP },
{ OPT_KNOB, "--knob_", SO_REQ_SEP },
{ OPT_VERSION, "--version", SO_NONE },
{ OPT_VERSION, "-v", SO_NONE },
{ OPT_QUIET, "-q", SO_NONE },
{ OPT_QUIET, "--quiet", SO_NONE },
{ OPT_TRACE, "--log", SO_NONE },
{ OPT_TRACE_DIR, "--logdir", SO_REQ_SEP },
s/--trace-format/--trace_format/
2018-12-21 07:58:26 +08:00
{ OPT_TRACE_FORMAT, "--trace_format", SO_REQ_SEP },
Added --loggroup option to all fdbbackup modes.
2019-01-04 16:15:51 +08:00
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_CRASHONERROR, "--crash", SO_NONE },
Add locality to backup agent and DR agent
2017-09-27 01:19:26 +08:00
{ OPT_LOCALITY, "--locality_", SO_REQ_SEP },
added configurable memory limits for backup and dr executables added a default memory limit of 8GB for fdbcli
2017-09-30 01:35:40 +08:00
{ OPT_MEMLIMIT, "-m", SO_REQ_SEP },
{ OPT_MEMLIMIT, "--memory", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_HELP, "-?", SO_NONE },
{ OPT_HELP, "-h", SO_NONE },
{ OPT_HELP, "--help", SO_NONE },
{ OPT_DEVHELP, "--dev-help", SO_NONE },
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
TLS_OPTION_FLAGS
#endif
Initial repository commit
2017-05-26 04:48:44 +08:00
SO_END_OF_OPTIONS
};
CSimpleOpt::SOption g_rgDBStartOptions[] = {
#ifdef _WIN32
{ OPT_PARENTPID, "--parentpid", SO_REQ_SEP },
#endif
{ OPT_SOURCE_CLUSTER, "-s", SO_REQ_SEP },
{ OPT_SOURCE_CLUSTER, "--source", SO_REQ_SEP },
{ OPT_DEST_CLUSTER, "-d", SO_REQ_SEP },
{ OPT_DEST_CLUSTER, "--destination", SO_REQ_SEP },
{ OPT_TAGNAME, "-t", SO_REQ_SEP },
{ OPT_TAGNAME, "--tagname", SO_REQ_SEP },
{ OPT_BACKUPKEYS, "-k", SO_REQ_SEP },
{ OPT_BACKUPKEYS, "--keys", SO_REQ_SEP },
{ OPT_TRACE, "--log", SO_NONE },
{ OPT_TRACE_DIR, "--logdir", SO_REQ_SEP },
s/--trace-format/--trace_format/
2018-12-21 07:58:26 +08:00
{ OPT_TRACE_FORMAT, "--trace_format", SO_REQ_SEP },
Added --loggroup option to all fdbbackup modes.
2019-01-04 16:15:51 +08:00
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_QUIET, "-q", SO_NONE },
{ OPT_QUIET, "--quiet", SO_NONE },
{ OPT_VERSION, "--version", SO_NONE },
{ OPT_VERSION, "-v", SO_NONE },
{ OPT_CRASHONERROR, "--crash", SO_NONE },
added configurable memory limits for backup and dr executables added a default memory limit of 8GB for fdbcli
2017-09-30 01:35:40 +08:00
{ OPT_MEMLIMIT, "-m", SO_REQ_SEP },
{ OPT_MEMLIMIT, "--memory", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_HELP, "-?", SO_NONE },
{ OPT_HELP, "-h", SO_NONE },
{ OPT_HELP, "--help", SO_NONE },
{ OPT_DEVHELP, "--dev-help", SO_NONE },
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
TLS_OPTION_FLAGS
#endif
Initial repository commit
2017-05-26 04:48:44 +08:00
SO_END_OF_OPTIONS
};
CSimpleOpt::SOption g_rgDBStatusOptions[] = {
#ifdef _WIN32
{ OPT_PARENTPID, "--parentpid", SO_REQ_SEP },
#endif
{ OPT_SOURCE_CLUSTER, "-s", SO_REQ_SEP },
{ OPT_SOURCE_CLUSTER, "--source", SO_REQ_SEP },
{ OPT_DEST_CLUSTER, "-d", SO_REQ_SEP },
{ OPT_DEST_CLUSTER, "--destination", SO_REQ_SEP },
{ OPT_ERRORLIMIT, "-e", SO_REQ_SEP },
{ OPT_ERRORLIMIT, "--errorlimit", SO_REQ_SEP },
{ OPT_TAGNAME, "-t", SO_REQ_SEP },
{ OPT_TAGNAME, "--tagname", SO_REQ_SEP },
{ OPT_TRACE, "--log", SO_NONE },
{ OPT_TRACE_DIR, "--logdir", SO_REQ_SEP },
s/--trace-format/--trace_format/
2018-12-21 07:58:26 +08:00
{ OPT_TRACE_FORMAT, "--trace_format", SO_REQ_SEP },
Added --loggroup option to all fdbbackup modes.
2019-01-04 16:15:51 +08:00
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_VERSION, "--version", SO_NONE },
{ OPT_VERSION, "-v", SO_NONE },
{ OPT_QUIET, "-q", SO_NONE },
{ OPT_QUIET, "--quiet", SO_NONE },
{ OPT_CRASHONERROR, "--crash", SO_NONE },
added configurable memory limits for backup and dr executables added a default memory limit of 8GB for fdbcli
2017-09-30 01:35:40 +08:00
{ OPT_MEMLIMIT, "-m", SO_REQ_SEP },
{ OPT_MEMLIMIT, "--memory", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_HELP, "-?", SO_NONE },
{ OPT_HELP, "-h", SO_NONE },
{ OPT_HELP, "--help", SO_NONE },
{ OPT_DEVHELP, "--dev-help", SO_NONE },
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
TLS_OPTION_FLAGS
#endif
Initial repository commit
2017-05-26 04:48:44 +08:00
SO_END_OF_OPTIONS
};
CSimpleOpt::SOption g_rgDBSwitchOptions[] = {
#ifdef _WIN32
{ OPT_PARENTPID, "--parentpid", SO_REQ_SEP },
#endif
{ OPT_SOURCE_CLUSTER, "-s", SO_REQ_SEP },
{ OPT_SOURCE_CLUSTER, "--source", SO_REQ_SEP },
{ OPT_DEST_CLUSTER, "-d", SO_REQ_SEP },
{ OPT_DEST_CLUSTER, "--destination", SO_REQ_SEP },
{ OPT_TAGNAME, "-t", SO_REQ_SEP },
{ OPT_TAGNAME, "--tagname", SO_REQ_SEP },
{ OPT_TRACE, "--log", SO_NONE },
{ OPT_TRACE_DIR, "--logdir", SO_REQ_SEP },
s/--trace-format/--trace_format/
2018-12-21 07:58:26 +08:00
{ OPT_TRACE_FORMAT, "--trace_format", SO_REQ_SEP },
Added --loggroup option to all fdbbackup modes.
2019-01-04 16:15:51 +08:00
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_QUIET, "-q", SO_NONE },
{ OPT_QUIET, "--quiet", SO_NONE },
{ OPT_VERSION, "--version", SO_NONE },
{ OPT_VERSION, "-v", SO_NONE },
Add force option to 'fdbdr switch'.
2019-03-05 08:36:58 +08:00
{ OPT_FORCE, "-f", SO_NONE },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_CRASHONERROR, "--crash", SO_NONE },
added configurable memory limits for backup and dr executables added a default memory limit of 8GB for fdbcli
2017-09-30 01:35:40 +08:00
{ OPT_MEMLIMIT, "-m", SO_REQ_SEP },
{ OPT_MEMLIMIT, "--memory", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_HELP, "-?", SO_NONE },
{ OPT_HELP, "-h", SO_NONE },
{ OPT_HELP, "--help", SO_NONE },
{ OPT_DEVHELP, "--dev-help", SO_NONE },
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
TLS_OPTION_FLAGS
#endif
Initial repository commit
2017-05-26 04:48:44 +08:00
SO_END_OF_OPTIONS
};
CSimpleOpt::SOption g_rgDBAbortOptions[] = {
#ifdef _WIN32
{ OPT_PARENTPID, "--parentpid", SO_REQ_SEP },
#endif
{ OPT_SOURCE_CLUSTER, "-s", SO_REQ_SEP },
{ OPT_SOURCE_CLUSTER, "--source", SO_REQ_SEP },
{ OPT_DEST_CLUSTER, "-d", SO_REQ_SEP },
{ OPT_DEST_CLUSTER, "--destination", SO_REQ_SEP },
{ OPT_CLEANUP, "--cleanup", SO_NONE },
{ OPT_TAGNAME, "-t", SO_REQ_SEP },
{ OPT_TAGNAME, "--tagname", SO_REQ_SEP },
{ OPT_TRACE, "--log", SO_NONE },
{ OPT_TRACE_DIR, "--logdir", SO_REQ_SEP },
s/--trace-format/--trace_format/
2018-12-21 07:58:26 +08:00
{ OPT_TRACE_FORMAT, "--trace_format", SO_REQ_SEP },
Added --loggroup option to all fdbbackup modes.
2019-01-04 16:15:51 +08:00
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_QUIET, "-q", SO_NONE },
{ OPT_QUIET, "--quiet", SO_NONE },
{ OPT_VERSION, "--version", SO_NONE },
{ OPT_VERSION, "-v", SO_NONE },
{ OPT_CRASHONERROR, "--crash", SO_NONE },
added configurable memory limits for backup and dr executables added a default memory limit of 8GB for fdbcli
2017-09-30 01:35:40 +08:00
{ OPT_MEMLIMIT, "-m", SO_REQ_SEP },
{ OPT_MEMLIMIT, "--memory", SO_REQ_SEP },
Initial repository commit
2017-05-26 04:48:44 +08:00
{ OPT_HELP, "-?", SO_NONE },
{ OPT_HELP, "-h", SO_NONE },
{ OPT_HELP, "--help", SO_NONE },
{ OPT_DEVHELP, "--dev-help", SO_NONE },
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
TLS_OPTION_FLAGS
#endif
Initial repository commit
2017-05-26 04:48:44 +08:00
SO_END_OF_OPTIONS
};
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
CSimpleOpt::SOption g_rgDBPauseOptions[] = {
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
#ifdef _WIN32
{ OPT_PARENTPID, "--parentpid", SO_REQ_SEP },
#endif
{ OPT_SOURCE_CLUSTER, "-s", SO_REQ_SEP },
{ OPT_SOURCE_CLUSTER, "--source", SO_REQ_SEP },
{ OPT_DEST_CLUSTER, "-d", SO_REQ_SEP },
{ OPT_DEST_CLUSTER, "--destination", SO_REQ_SEP },
{ OPT_TRACE, "--log", SO_NONE },
{ OPT_TRACE_DIR, "--logdir", SO_REQ_SEP },
s/--trace-format/--trace_format/
2018-12-21 07:58:26 +08:00
{ OPT_TRACE_FORMAT, "--trace_format", SO_REQ_SEP },
Added --loggroup option to all fdbbackup modes.
2019-01-04 16:15:51 +08:00
{ OPT_TRACE_LOG_GROUP, "--loggroup", SO_REQ_SEP },
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
{ OPT_QUIET, "-q", SO_NONE },
{ OPT_QUIET, "--quiet", SO_NONE },
{ OPT_VERSION, "--version", SO_NONE },
{ OPT_VERSION, "-v", SO_NONE },
{ OPT_CRASHONERROR, "--crash", SO_NONE },
{ OPT_MEMLIMIT, "-m", SO_REQ_SEP },
{ OPT_MEMLIMIT, "--memory", SO_REQ_SEP },
{ OPT_HELP, "-?", SO_NONE },
{ OPT_HELP, "-h", SO_NONE },
{ OPT_HELP, "--help", SO_NONE },
{ OPT_DEVHELP, "--dev-help", SO_NONE },
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
TLS_OPTION_FLAGS
#endif
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
SO_END_OF_OPTIONS
};
Initial repository commit
2017-05-26 04:48:44 +08:00
const KeyRef exeAgent = LiteralStringRef("backup_agent");
const KeyRef exeBackup = LiteralStringRef("fdbbackup");
const KeyRef exeRestore = LiteralStringRef("fdbrestore");
const KeyRef exeDatabaseAgent = LiteralStringRef("dr_agent");
const KeyRef exeDatabaseBackup = LiteralStringRef("fdbdr");
extern const char* getHGVersion();
#ifdef _WIN32
void parentWatcher(void *parentHandle) {
HANDLE parent = (HANDLE)parentHandle;
int signal = WaitForSingleObject(parent, INFINITE);
CloseHandle(parentHandle);
if (signal == WAIT_OBJECT_0)
criticalError(FDB_EXIT_SUCCESS, "ParentProcessExited", "Parent process exited");
TraceEvent(SevError, "ParentProcessWaitFailed").detail("RetCode", signal).GetLastError();
}
#endif
static void printVersion() {
printf("FoundationDB " FDB_VT_PACKAGE_NAME " (v" FDB_VT_VERSION ")\n");
printf("source version %s\n", getHGVersion());
ProtocolVersion type - second try
2019-06-19 08:55:27 +08:00
printf("protocol %llx\n", (long long) currentProtocolVersion.version());
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Merge branch 'release-6.0' # Conflicts: # documentation/sphinx/source/release-notes.rst # fdbbackup/backup.actor.cpp # fdbclient/BackupContainer.actor.cpp # fdbclient/HTTP.actor.cpp # fdbserver/storageserver.actor.cpp # fdbserver/workloads/BackupCorrectness.actor.cpp # versions.target
2019-01-10 08:14:46 +08:00
const char *BlobCredentialInfo =
Added fdbbackup describe --version_timestamps option which is now required for describe to print version timestamps. Improved help...slightly.
2018-01-23 07:50:28 +08:00
" BLOB CREDENTIALS\n"
" Blob account secret keys can optionally be omitted from blobstore:// URLs, in which case they will be\n"
" loaded, if possible, from 1 or more blob credentials definition files.\n\n"
" These files can be specified with the --blob_credentials argument described above or via the environment variable\n"
" FDB_BLOB_CREDENTIALS, whose value is a colon-separated list of files. The command line takes priority over\n"
" over the environment but all files from both sources are used.\n\n"
" At connect time, the specified files are read in order and the first matching account specification (user@host)\n"
" will be used to obtain the secret key.\n\n"
" The JSON schema is:\n"
" { \"accounts\" : { \"user@host\" : { \"secret\" : \"SECRETKEY\" }, \"user2@host2\" : { \"secret\" : \"SECRET\" } } }\n";
Initial repository commit
2017-05-26 04:48:44 +08:00
static void printHelpTeaser( const char *name ) {
fprintf(stderr, "Try `%s --help' for more information.\n", name);
}
static void printAgentUsage(bool devhelp) {
printf("FoundationDB " FDB_VT_PACKAGE_NAME " (v" FDB_VT_VERSION ")\n");
printf("Usage: %s [OPTIONS]\n\n", exeAgent.toString().c_str());
printf(" -C CONNFILE The path of a file containing the connection string for the\n"
" FoundationDB cluster. The default is first the value of the\n"
" FDB_CLUSTER_FILE environment variable, then `./fdb.cluster',\n"
" then `%s'.\n", platform::getDefaultClusterFilePath().c_str());
printf(" --log Enables trace file logging for the CLI session.\n"
" --logdir PATH Specifes the output directory for trace files. If\n"
" unspecified, defaults to the current directory. Has\n"
" no effect unless --log is specified.\n");
Document --trace_format flag
2018-12-21 08:22:41 +08:00
printf(" --trace_format FORMAT\n"
Mention that xml is the default
2019-01-04 00:48:31 +08:00
" Select the format of the trace files. xml (the default) and json are supported.\n"
Fix whitespace
2019-01-03 07:24:11 +08:00
" Has no effect unless --log is specified.\n");
added help messages for setting a memory limit
2017-09-30 03:38:42 +08:00
printf(" -m SIZE, --memory SIZE\n"
" Memory limit. The default value is 8GiB. When specified\n"
" without a unit, MiB is assumed.\n");
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
printf(TLS_HELP);
#endif
Initial repository commit
2017-05-26 04:48:44 +08:00
printf(" -v, --version Print version information and exit.\n");
printf(" -h, --help Display this help and exit.\n");
Added fdbbackup describe --version_timestamps option which is now required for describe to print version timestamps. Improved help...slightly.
2018-01-23 07:50:28 +08:00
Initial repository commit
2017-05-26 04:48:44 +08:00
if (devhelp) {
#ifdef _WIN32
printf(" -n Create a new console.\n");
printf(" -q Disable error dialog on crash.\n");
printf(" --parentpid PID\n");
printf(" Specify a process after whose termination to exit.\n");
#endif
}
Added fdbbackup describe --version_timestamps option which is now required for describe to print version timestamps. Improved help...slightly.
2018-01-23 07:50:28 +08:00
printf("\n");
puts(BlobCredentialInfo);
Initial repository commit
2017-05-26 04:48:44 +08:00
return;
}
void printBackupContainerInfo() {
printf(" Backup URL forms:\n\n");
std::vector<std::string> formats = IBackupContainer::getURLFormats();
for(auto &f : formats)
printf(" %s\n", f.c_str());
printf("\n");
}
static void printBackupUsage(bool devhelp) {
printf("FoundationDB " FDB_VT_PACKAGE_NAME " (v" FDB_VT_VERSION ")\n");
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
printf("Usage: %s (start | status | abort | wait | discontinue | pause | resume | expire | delete | describe | list) [OPTIONS]\n\n", exeBackup.toString().c_str());
Initial repository commit
2017-05-26 04:48:44 +08:00
printf(" -C CONNFILE The path of a file containing the connection string for the\n"
" FoundationDB cluster. The default is first the value of the\n"
" FDB_CLUSTER_FILE environment variable, then `./fdb.cluster',\n"
" then `%s'.\n", platform::getDefaultClusterFilePath().c_str());
printf(" -d, --destcontainer URL\n"
Added help and documentation for fdbbackup modify. Add BackupUID and more parseable BackupURL to fdbbackup status output.
2019-03-05 21:13:48 +08:00
" The Backup container URL for start, modify, describe, expire, and delete operations.\n");
Initial repository commit
2017-05-26 04:48:44 +08:00
printBackupContainerInfo();
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
printf(" -b, --base_url BASEURL\n"
" Base backup URL for list operations. This looks like a Backup URL but without a backup name.\n");
printf(" --blob_credentials FILE\n"
Added blob credentials details to fdbbackup help.
2017-12-21 18:50:02 +08:00
" File containing blob credentials in JSON format. Can be specified multiple times for multiple files. See below for more details.\n");
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
printf(" --expire_before_timestamp DATETIME\n"
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
" Datetime cutoff for expire operations. Requires a cluster file and will use version/timestamp metadata\n"
Renamed backup state enums for clarity, added backup state names. Changed Epochs to EpochSeconds in backup JSON along with some other renaming/moving of fields, and added information about snapshot dispatch. Changed timestamp format for input/output in all backup/restore contexts to be a fully qualified time with timezone offset. Added information about the last snapshot dispatch to backup config and status (not yet populated).
2019-03-11 07:00:01 +08:00
" in the database to obtain a cutoff version very close to the timestamp given in %s.\n", BackupAgentBase::timeFormat().c_str());
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
printf(" --expire_before_version VERSION\n"
" Version cutoff for expire operations. Deletes data files containing no data at or after VERSION.\n");
Added fdbbackup expire options to calculate approximate version boundaries based on a number of days prior to the latest log file found in the backup container. This enables expiration operations based on time (with reasonable precision) without accessing the source cluster.
2018-12-19 10:55:44 +08:00
printf(" --delete_before_days NUM_DAYS\n"
" Another way to specify version cutoff for expire operations. Deletes data files containing no data at or after a\n"
" version approximately NUM_DAYS days worth of versions prior to the latest log version in the backup.\n");
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
printf(" --restorable_after_timestamp DATETIME\n"
" For expire operations, set minimum acceptable restorability to the version equivalent of DATETIME and later.\n");
Timekeeper lookup improvements, moved both function declartions to BackupContainer.h. VersionFromEpochs() now uses versions/sec to adjust the lookup result to improve accuracy. Conversions in both directions look for the latest record less than the target conversion value, but failing that they will now fall back on any available data point and adjust from there using versions/sec.
2018-01-23 15:57:01 +08:00
printf(" --restorable_after_version VERSION\n"
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
" For expire operations, set minimum acceptable restorability to the VERSION and later.\n");
Added fdbbackup expire options to calculate approximate version boundaries based on a number of days prior to the latest log file found in the backup container. This enables expiration operations based on time (with reasonable precision) without accessing the source cluster.
2018-12-19 10:55:44 +08:00
printf(" --min_restorable_days NUM_DAYS\n"
" For expire operations, set minimum acceptable restorability to approximately NUM_DAYS days worth of versions\n"
" prior to the latest log version in the backup.\n");
Added fdbbackup describe --version_timestamps option which is now required for describe to print version timestamps. Improved help...slightly.
2018-01-23 07:50:28 +08:00
printf(" --version_timestamps\n");
printf(" For describe operations, lookup versions in the database to obtain timestamps. A cluster file is required.\n");
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
printf(" -f, --force For expire operations, force expiration even if minimum restorability would be violated.\n");
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
printf(" -s, --snapshot_interval DURATION\n"
Added help and documentation for fdbbackup modify. Add BackupUID and more parseable BackupURL to fdbbackup status output.
2019-03-05 21:13:48 +08:00
" For start or modify operations, specifies the backup's default target snapshot interval as DURATION seconds. Defaults to %d for start operations.\n", CLIENT_KNOBS->BACKUP_DEFAULT_SNAPSHOT_INTERVAL_SEC);
Remove short option name for active snapshot interval
2019-03-11 16:48:51 +08:00
printf(" --active_snapshot_interval DURATION\n"
Added help and documentation for fdbbackup modify. Add BackupUID and more parseable BackupURL to fdbbackup status output.
2019-03-05 21:13:48 +08:00
" For modify operations, sets the desired interval for the backup's currently active snapshot, relative to the start of the snapshot.\n");
printf(" --verify_uid UID\n"
More documentation and release note changes for fdbbackup modify. Added release note for #1205 bug fix which was missed.
2019-03-06 02:44:14 +08:00
" Specifies a UID to verify against the BackupUID of the running backup. If provided, the UID is verified in the same transaction\n"
" which sets the new backup parameters (if the UID matches).\n");
Initial repository commit
2017-05-26 04:48:44 +08:00
printf(" -e ERRORLIMIT The maximum number of errors printed by status (default is 10).\n");
printf(" -k KEYS List of key ranges to backup.\n"
" If not specified, the entire database will be backed up.\n");
Restore start with dry run option no longer requires or uses a database. Removed duplicate and unnecessary code. Dry run execution is more clear, it does not pretend to have actually restored data. Fixed typo in help text.
2019-01-09 11:00:58 +08:00
printf(" -n, --dryrun For start or restore operations, performs a trial run with no actual changes made.\n");
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
printf(TLS_HELP);
#endif
Initial repository commit
2017-05-26 04:48:44 +08:00
printf(" -v, --version Print version information and exit.\n");
printf(" -w, --wait Wait for the backup to complete (allowed with `start' and `discontinue').\n");
printf(" -z, --no-stop-when-done\n"
" Do not stop backup when restorable.\n");
printf(" -h, --help Display this help and exit.\n");
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
if (devhelp) {
#ifdef _WIN32
printf(" -n Create a new console.\n");
printf(" -q Disable error dialog on crash.\n");
printf(" --parentpid PID\n");
printf(" Specify a process after whose termination to exit.\n");
#endif
printf(" --deep For describe operations, do not use cached metadata. Warning: Very slow\n");
Merge branch 'release-6.0' # Conflicts: # documentation/sphinx/source/release-notes.rst # fdbbackup/backup.actor.cpp # fdbclient/BackupContainer.actor.cpp # fdbclient/HTTP.actor.cpp # fdbserver/storageserver.actor.cpp # fdbserver/workloads/BackupCorrectness.actor.cpp # versions.target
2019-01-10 08:14:46 +08:00
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
printf("\n"
" KEYS FORMAT: \"<BEGINKEY> <ENDKEY>\" [...]\n");
Added fdbbackup describe --version_timestamps option which is now required for describe to print version timestamps. Improved help...slightly.
2018-01-23 07:50:28 +08:00
printf("\n");
puts(BlobCredentialInfo);
Initial repository commit
2017-05-26 04:48:44 +08:00
return;
}
static void printRestoreUsage(bool devhelp ) {
printf("FoundationDB " FDB_VT_PACKAGE_NAME " (v" FDB_VT_VERSION ")\n");
printf("Usage: %s (start | status | abort | wait) [OPTIONS]\n\n", exeRestore.toString().c_str());
//printf(" FOLDERS Paths to folders containing the backup files.\n");
printf("Options for all commands:\n\n");
Restore now requires an explicit destination cluster file. Restore target database version can now be specified by a timestamp, which must be resolved using the cluster file for the cluster from which the backup originated.
2019-03-06 04:28:05 +08:00
printf(" --dest_cluster_file CONNFILE\n");
printf(" The cluster file to restore data into.\n");
printf(" -t, --tagname TAGNAME\n");
printf(" The restore tag to act on. Default is 'default'\n");
printf("Options for start:\n\n");
Initial repository commit
2017-05-26 04:48:44 +08:00
printf(" -r URL The Backup URL for the restore to read from.\n");
printBackupContainerInfo();
Restore now requires an explicit destination cluster file. Restore target database version can now be specified by a timestamp, which must be resolved using the cluster file for the cluster from which the backup originated.
2019-03-06 04:28:05 +08:00
printf(" -w, --waitfordone\n");
printf(" Wait for the restore to complete before exiting. Prints progress updates.\n");
printf(" -k KEYS List of key ranges from the backup to restore.\n");
printf(" --remove_prefix PREFIX\n");
printf(" Prefix to remove from the restored keys.\n");
printf(" --add_prefix PREFIX\n");
printf(" Prefix to add to the restored keys\n");
printf(" -n, --dryrun Perform a trial run with no changes made.\n");
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
printf(TLS_HELP);
#endif
Initial repository commit
2017-05-26 04:48:44 +08:00
printf(" -v DBVERSION The version at which the database will be restored.\n");
Renamed backup state enums for clarity, added backup state names. Changed Epochs to EpochSeconds in backup JSON along with some other renaming/moving of fields, and added information about snapshot dispatch. Changed timestamp format for input/output in all backup/restore contexts to be a fully qualified time with timezone offset. Added information about the last snapshot dispatch to backup config and status (not yet populated).
2019-03-11 07:00:01 +08:00
printf(" --timestamp Instead of a numeric version, use this to specify a timestamp in %s\n", BackupAgentBase::timeFormat().c_str());
Restore now requires an explicit destination cluster file. Restore target database version can now be specified by a timestamp, which must be resolved using the cluster file for the cluster from which the backup originated.
2019-03-06 04:28:05 +08:00
printf(" and it will be converted to a version from that time using metadata in orig_cluster_file.\n");
printf(" --orig_cluster_file CONNFILE\n");
printf(" The cluster file for the original database from which the backup was created. The original database\n");
printf(" is only needed to convert a --timestamp argument to a database version.\n");
Initial repository commit
2017-05-26 04:48:44 +08:00
printf(" -h, --help Display this help and exit.\n");
if( devhelp ) {
#ifdef _WIN32
printf(" -q Disable error dialog on crash.\n");
printf(" --parentpid PID\n");
printf(" Specify a process after whose termination to exit.\n");
#endif
}
Added fdbbackup describe --version_timestamps option which is now required for describe to print version timestamps. Improved help...slightly.
2018-01-23 07:50:28 +08:00
printf("\n"
" KEYS FORMAT: \"<BEGINKEY> <ENDKEY>\" [...]\n");
printf("\n");
puts(BlobCredentialInfo);
Initial repository commit
2017-05-26 04:48:44 +08:00
return;
}
static void printDBAgentUsage(bool devhelp) {
printf("FoundationDB " FDB_VT_PACKAGE_NAME " (v" FDB_VT_VERSION ")\n");
printf("Usage: %s [OPTIONS]\n\n", exeDatabaseAgent.toString().c_str());
printf(" -d CONNFILE The path of a file containing the connection string for the\n"
" destination FoundationDB cluster.\n");
printf(" -s CONNFILE The path of a file containing the connection string for the\n"
" source FoundationDB cluster.\n");
printf(" --log Enables trace file logging for the CLI session.\n"
" --logdir PATH Specifes the output directory for trace files. If\n"
" unspecified, defaults to the current directory. Has\n"
" no effect unless --log is specified.\n");
Document --trace_format flag
2018-12-21 08:22:41 +08:00
printf(" --trace_format FORMAT\n"
Mention that xml is the default
2019-01-04 00:48:31 +08:00
" Select the format of the trace files. xml (the default) and json are supported.\n"
Fix whitespace
2019-01-03 07:24:11 +08:00
" Has no effect unless --log is specified.\n");
added help messages for setting a memory limit
2017-09-30 03:38:42 +08:00
printf(" -m SIZE, --memory SIZE\n"
" Memory limit. The default value is 8GiB. When specified\n"
" without a unit, MiB is assumed.\n");
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
printf(TLS_HELP);
#endif
Initial repository commit
2017-05-26 04:48:44 +08:00
printf(" -v, --version Print version information and exit.\n");
printf(" -h, --help Display this help and exit.\n");
if (devhelp) {
#ifdef _WIN32
printf(" -n Create a new console.\n");
printf(" -q Disable error dialog on crash.\n");
printf(" --parentpid PID\n");
printf(" Specify a process after whose termination to exit.\n");
#endif
}
return;
}
static void printDBBackupUsage(bool devhelp) {
printf("FoundationDB " FDB_VT_PACKAGE_NAME " (v" FDB_VT_VERSION ")\n");
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
printf("Usage: %s (start | status | switch | abort | pause | resume) [OPTIONS]\n\n", exeDatabaseBackup.toString().c_str());
Initial repository commit
2017-05-26 04:48:44 +08:00
printf(" -d, --destination CONNFILE\n"
" The path of a file containing the connection string for the\n");
printf(" destination FoundationDB cluster.\n");
printf(" -s, --source CONNFILE\n"
" The path of a file containing the connection string for the\n"
" source FoundationDB cluster.\n");
printf(" -e ERRORLIMIT The maximum number of errors printed by status (default is 10).\n");
printf(" -k KEYS List of key ranges to backup.\n"
" If not specified, the entire database will be backed up.\n");
printf(" --cleanup Abort will attempt to stop mutation logging on the source cluster.\n");
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
printf(TLS_HELP);
#endif
Initial repository commit
2017-05-26 04:48:44 +08:00
printf(" -v, --version Print version information and exit.\n");
printf(" -h, --help Display this help and exit.\n");
printf("\n"
" KEYS FORMAT: \"<BEGINKEY> <ENDKEY>\" [...]\n");
if (devhelp) {
#ifdef _WIN32
printf(" -n Create a new console.\n");
printf(" -q Disable error dialog on crash.\n");
printf(" --parentpid PID\n");
printf(" Specify a process after whose termination to exit.\n");
#endif
}
return;
}
static void printUsage(enumProgramExe programExe, bool devhelp)
{
switch (programExe)
{
case EXE_AGENT:
printAgentUsage(devhelp);
break;
case EXE_BACKUP:
printBackupUsage(devhelp);
break;
case EXE_RESTORE:
printRestoreUsage(devhelp);
break;
case EXE_DR_AGENT:
printDBAgentUsage(devhelp);
break;
case EXE_DB_BACKUP:
printDBBackupUsage(devhelp);
break;
case EXE_UNDEFINED:
default:
break;
}
return;
}
extern bool g_crashOnError;
// Return the type of program executable based on the name of executable file
enumProgramExe getProgramType(std::string programExe)
{
enumProgramExe enProgramExe = EXE_UNDEFINED;
// lowercase the string
std::transform(programExe.begin(), programExe.end(), programExe.begin(), ::tolower);
// Remove the extension, if Windows
#ifdef _WIN32
size_t lastDot = programExe.find_last_of(".");
if (lastDot != std::string::npos) {
size_t lastSlash = programExe.find_last_of("\\");
// Ensure last dot is after last slash, if present
if ((lastSlash == std::string::npos)||
(lastSlash < lastDot) )
{
programExe = programExe.substr(0, lastDot);
}
}
#endif
Fdbbackup binary will ignore .debug suffix when determining how it was invoked.
2019-03-06 13:17:56 +08:00
// For debugging convenience, remove .debug suffix if present.
if(StringRef(programExe).endsWith(LiteralStringRef(".debug")))
programExe = programExe.substr(0, programExe.size() - 6);
Initial repository commit
2017-05-26 04:48:44 +08:00
// Check if backup agent
if ((programExe.length() >= exeAgent.size()) &&
(programExe.compare(programExe.length()-exeAgent.size(), exeAgent.size(), (const char*) exeAgent.begin()) == 0) )
{
enProgramExe = EXE_AGENT;
}
// Check if backup
else if ((programExe.length() >= exeBackup.size()) &&
(programExe.compare(programExe.length() - exeBackup.size(), exeBackup.size(), (const char*)exeBackup.begin()) == 0))
{
enProgramExe = EXE_BACKUP;
}
// Check if restore
else if ((programExe.length() >= exeRestore.size()) &&
(programExe.compare(programExe.length() - exeRestore.size(), exeRestore.size(), (const char*)exeRestore.begin()) == 0))
{
enProgramExe = EXE_RESTORE;
}
// Check if db agent
else if ((programExe.length() >= exeDatabaseAgent.size()) &&
(programExe.compare(programExe.length() - exeDatabaseAgent.size(), exeDatabaseAgent.size(), (const char*)exeDatabaseAgent.begin()) == 0))
{
enProgramExe = EXE_DR_AGENT;
}
// Check if db backup
else if ((programExe.length() >= exeDatabaseBackup.size()) &&
(programExe.compare(programExe.length() - exeDatabaseBackup.size(), exeDatabaseBackup.size(), (const char*)exeDatabaseBackup.begin()) == 0))
{
enProgramExe = EXE_DB_BACKUP;
}
return enProgramExe;
}
enumBackupType getBackupType(std::string backupType)
{
enumBackupType enBackupType = BACKUP_UNDEFINED;
// lowercase the string
std::transform(backupType.begin(), backupType.end(), backupType.begin(), ::tolower);
static std::map<std::string, enumBackupType> values;
if(values.empty()) {
values["start"] = BACKUP_START;
values["status"] = BACKUP_STATUS;
values["abort"] = BACKUP_ABORT;
values["wait"] = BACKUP_WAIT;
values["discontinue"] = BACKUP_DISCONTINUE;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
values["pause"] = BACKUP_PAUSE;
values["resume"] = BACKUP_RESUME;
values["expire"] = BACKUP_EXPIRE;
values["delete"] = BACKUP_DELETE;
values["describe"] = BACKUP_DESCRIBE;
values["list"] = BACKUP_LIST;
Added fdbbackup dump option to list files and sizes in a backup for a given version range.
2019-01-03 18:35:31 +08:00
values["dump"] = BACKUP_DUMP;
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
values["modify"] = BACKUP_MODIFY;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
auto i = values.find(backupType);
if(i != values.end())
enBackupType = i->second;
return enBackupType;
}
enumRestoreType getRestoreType(std::string name) {
if(name == "start") return RESTORE_START;
if(name == "abort") return RESTORE_ABORT;
if(name == "status") return RESTORE_STATUS;
if(name == "wait") return RESTORE_WAIT;
return RESTORE_UNKNOWN;
}
enumDBType getDBType(std::string dbType)
{
enumDBType enBackupType = DB_UNDEFINED;
// lowercase the string
std::transform(dbType.begin(), dbType.end(), dbType.begin(), ::tolower);
static std::map<std::string, enumDBType> values;
if(values.empty()) {
values["start"] = DB_START;
values["status"] = DB_STATUS;
values["switch"] = DB_SWITCH;
values["abort"] = DB_ABORT;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
values["pause"] = DB_PAUSE;
values["resume"] = DB_RESUME;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
auto i = values.find(dbType);
if(i != values.end())
enBackupType = i->second;
return enBackupType;
}
ACTOR Future<std::string> getLayerStatus(Reference<ReadYourWritesTransaction> tr, std::string name, std::string id, enumProgramExe exe, Database dest) {
// This process will write a document that looks like this:
// { backup : { $expires : {<subdoc>}, version: <version from approximately 30 seconds from now> }
// so that the value under 'backup' will eventually expire to null and thus be ignored by
// readers of status. This is because if all agents die then they can no longer clean up old
// status docs from other dead agents.
state Version readVer = wait(tr->getReadVersion());
state json_spirit::mValue layersRootValue; // Will contain stuff that goes into the doc at the layers status root
JSONDoc layersRoot(layersRootValue); // Convenient mutator / accessor for the layers root
JSONDoc op = layersRoot.subDoc(name); // Operator object for the $expires operation
// Create the $expires key which is where the rest of the status output will go
state JSONDoc layerRoot = op.subDoc("$expires");
// Set the version argument in the $expires operator object.
op.create("version") = readVer + 120 * CLIENT_KNOBS->CORE_VERSIONSPERSECOND;
layerRoot.create("instances_running.$sum") = 1;
layerRoot.create("last_updated.$max") = now();
state JSONDoc o = layerRoot.subDoc("instances." + id);
o.create("version") = FDB_VT_VERSION;
o.create("id") = id;
o.create("last_updated") = now();
o.create("memory_usage") = (int64_t)getMemoryUsage();
o.create("resident_size") = (int64_t)getResidentMemoryUsage();
o.create("main_thread_cpu_seconds") = getProcessorTimeThread();
o.create("process_cpu_seconds") = getProcessorTimeProcess();
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
o.create("configured_workers") = CLIENT_KNOBS->BACKUP_TASKS_PER_AGENT;
Initial repository commit
2017-05-26 04:48:44 +08:00
if(exe == EXE_AGENT) {
static BlobStoreEndpoint::Stats last_stats;
static double last_ts = 0;
BlobStoreEndpoint::Stats current_stats = BlobStoreEndpoint::s_stats;
JSONDoc blobstats = o.create("blob_stats");
blobstats.create("total") = current_stats.getJSON();
BlobStoreEndpoint::Stats diff = current_stats - last_stats;
json_spirit::mObject diffObj = diff.getJSON();
if(last_ts > 0)
diffObj["bytes_per_second"] = double(current_stats.bytes_sent - last_stats.bytes_sent) / (now() - last_ts);
blobstats.create("recent") = diffObj;
last_stats = current_stats;
last_ts = now();
JSONDoc totalBlobStats = layerRoot.subDoc("blob_recent_io");
for(auto &p : diffObj)
totalBlobStats.create(p.first + ".$sum") = p.second;
state FileBackupAgent fba;
Using BackupConfig from backup.actor.cpp to reduce intermediate functions.
2017-09-07 11:06:32 +08:00
state std::vector<KeyBackedTag> backupTags = wait(getAllBackupTags(tr));
Initial repository commit
2017-05-26 04:48:44 +08:00
state std::vector<Future<Version>> tagLastRestorableVersions;
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
state std::vector<Future<EBackupState>> tagStates;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
state std::vector<Future<Reference<IBackupContainer>>> tagContainers;
Initial repository commit
2017-05-26 04:48:44 +08:00
state std::vector<Future<int64_t>> tagRangeBytes;
state std::vector<Future<int64_t>> tagLogBytes;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
state Future<Optional<Value>> fBackupPaused = tr->get(fba.taskBucket->getPauseKey());
Initial repository commit
2017-05-26 04:48:44 +08:00
Using BackupConfig from backup.actor.cpp to reduce intermediate functions.
2017-09-07 11:06:32 +08:00
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr->setOption(FDBTransactionOptions::LOCK_AWARE);
Fixing windows build. Fixing backup.actor.cpp
2017-10-14 08:40:00 +08:00
state std::vector<KeyBackedTag>::iterator tag;
Untested implementation of feature to log basic backup and DR info to fdbcli status.
2017-11-04 05:02:03 +08:00
state std::vector<UID> backupTagUids;
Fixing windows build. Fixing backup.actor.cpp
2017-10-14 08:40:00 +08:00
for (tag = backupTags.begin(); tag != backupTags.end(); tag++) {
UidAndAbortedFlagT uidAndAbortedFlag = wait(tag->getOrThrow(tr));
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
BackupConfig config(uidAndAbortedFlag.first);
Untested implementation of feature to log basic backup and DR info to fdbcli status.
2017-11-04 05:02:03 +08:00
backupTagUids.push_back(config.getUid());
Using BackupConfig from backup.actor.cpp to reduce intermediate functions.
2017-09-07 11:06:32 +08:00
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
tagStates.push_back(config.stateEnum().getOrThrow(tr));
Minor (and asymptomatic) bug fixes where ints or enumerations are being passed as the bool snapshot argument instead of as the default value which was intended.
2018-10-13 06:03:21 +08:00
tagRangeBytes.push_back(config.rangeBytesWritten().getD(tr, false, 0));
tagLogBytes.push_back(config.logBytesWritten().getD(tr, false, 0));
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
tagContainers.push_back(config.backupContainer().getOrThrow(tr));
Fixing windows build. Fixing backup.actor.cpp
2017-10-14 08:40:00 +08:00
tagLastRestorableVersions.push_back(fba.getLastRestorable(tr, StringRef(tag->tagName)));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( waitForAll(tagLastRestorableVersions) && waitForAll(tagStates) && waitForAll(tagContainers) && waitForAll(tagRangeBytes) && waitForAll(tagLogBytes) && success(fBackupPaused));
Initial repository commit
2017-05-26 04:48:44 +08:00
JSONDoc tagsRoot = layerRoot.subDoc("tags.$latest");
layerRoot.create("tags.timestamp") = now();
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
layerRoot.create("total_workers.$sum") = fBackupPaused.get().present() ? 0 : CLIENT_KNOBS->BACKUP_TASKS_PER_AGENT;
layerRoot.create("paused.$latest") = fBackupPaused.get().present();
Initial repository commit
2017-05-26 04:48:44 +08:00
Using BackupConfig from backup.actor.cpp to reduce intermediate functions.
2017-09-07 11:06:32 +08:00
int j = 0;
for (KeyBackedTag eachTag : backupTags) {
Initial repository commit
2017-05-26 04:48:44 +08:00
Version last_restorable_version = tagLastRestorableVersions[j].get();
double last_restorable_seconds_behind = ((double)readVer - last_restorable_version) / CLIENT_KNOBS->CORE_VERSIONSPERSECOND;
BackupAgentBase::enumState status = (BackupAgentBase::enumState)tagStates[j].get();
const char *statusText = fba.getStateText(status);
// The object for this backup tag inside this instance's subdocument
Using BackupConfig from backup.actor.cpp to reduce intermediate functions.
2017-09-07 11:06:32 +08:00
JSONDoc tagRoot = tagsRoot.subDoc(eachTag.tagName);
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
tagRoot.create("current_container") = tagContainers[j].get()->getURL();
Initial repository commit
2017-05-26 04:48:44 +08:00
tagRoot.create("current_status") = statusText;
tagRoot.create("last_restorable_version") = tagLastRestorableVersions[j].get();
tagRoot.create("last_restorable_seconds_behind") = last_restorable_seconds_behind;
Renamed backup state enums for clarity, added backup state names. Changed Epochs to EpochSeconds in backup JSON along with some other renaming/moving of fields, and added information about snapshot dispatch. Changed timestamp format for input/output in all backup/restore contexts to be a fully qualified time with timezone offset. Added information about the last snapshot dispatch to backup config and status (not yet populated).
2019-03-11 07:00:01 +08:00
tagRoot.create("running_backup") = (status == BackupAgentBase::STATE_RUNNING_DIFFERENTIAL || status == BackupAgentBase::STATE_RUNNING);
tagRoot.create("running_backup_is_restorable") = (status == BackupAgentBase::STATE_RUNNING_DIFFERENTIAL);
Initial repository commit
2017-05-26 04:48:44 +08:00
tagRoot.create("range_bytes_written") = tagRangeBytes[j].get();
tagRoot.create("mutation_log_bytes_written") = tagLogBytes[j].get();
Untested implementation of feature to log basic backup and DR info to fdbcli status.
2017-11-04 05:02:03 +08:00
tagRoot.create("mutation_stream_id") = backupTagUids[j].toString();
Using BackupConfig from backup.actor.cpp to reduce intermediate functions.
2017-09-07 11:06:32 +08:00
j++;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
else if(exe == EXE_DR_AGENT) {
state DatabaseBackupAgent dba;
state Reference<ReadYourWritesTransaction> tr2(new ReadYourWritesTransaction(dest));
tr2->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr2->setOption(FDBTransactionOptions::LOCK_AWARE);
state Standalone<RangeResultRef> tagNames = wait(tr2->getRange(dba.tagNames.range(), 10000));
state std::vector<Future<Optional<Key>>> backupVersion;
state std::vector<Future<int>> backupStatus;
state std::vector<Future<int64_t>> tagRangeBytesDR;
state std::vector<Future<int64_t>> tagLogBytesDR;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
state Future<Optional<Value>> fDRPaused = tr->get(dba.taskBucket->getPauseKey());
Initial repository commit
2017-05-26 04:48:44 +08:00
Untested implementation of feature to log basic backup and DR info to fdbcli status.
2017-11-04 05:02:03 +08:00
state std::vector<UID> drTagUids;
Initial repository commit
2017-05-26 04:48:44 +08:00
for(int i = 0; i < tagNames.size(); i++) {
backupVersion.push_back(tr2->get(tagNames[i].value.withPrefix(applyMutationsBeginRange.begin)));
UID tagUID = BinaryReader::fromStringRef<UID>(tagNames[i].value, Unversioned());
Untested implementation of feature to log basic backup and DR info to fdbcli status.
2017-11-04 05:02:03 +08:00
drTagUids.push_back(tagUID);
Initial repository commit
2017-05-26 04:48:44 +08:00
backupStatus.push_back(dba.getStateValue(tr2, tagUID));
tagRangeBytesDR.push_back(dba.getRangeBytesWritten(tr2, tagUID));
tagLogBytesDR.push_back(dba.getLogBytesWritten(tr2, tagUID));
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(waitForAll(backupStatus) && waitForAll(backupVersion) && waitForAll(tagRangeBytesDR) && waitForAll(tagLogBytesDR) && success(fDRPaused));
Initial repository commit
2017-05-26 04:48:44 +08:00
JSONDoc tagsRoot = layerRoot.subDoc("tags.$latest");
layerRoot.create("tags.timestamp") = now();
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
layerRoot.create("total_workers.$sum") = fDRPaused.get().present() ? 0 : CLIENT_KNOBS->BACKUP_TASKS_PER_AGENT;
layerRoot.create("paused.$latest") = fDRPaused.get().present();
Initial repository commit
2017-05-26 04:48:44 +08:00
for (int i = 0; i < tagNames.size(); i++) {
std::string tagName = dba.sourceTagNames.unpack(tagNames[i].key).getString(0).toString();
BackupAgentBase::enumState status = (BackupAgentBase::enumState)backupStatus[i].get();
JSONDoc tagRoot = tagsRoot.create(tagName);
Renamed backup state enums for clarity, added backup state names. Changed Epochs to EpochSeconds in backup JSON along with some other renaming/moving of fields, and added information about snapshot dispatch. Changed timestamp format for input/output in all backup/restore contexts to be a fully qualified time with timezone offset. Added information about the last snapshot dispatch to backup config and status (not yet populated).
2019-03-11 07:00:01 +08:00
tagRoot.create("running_backup") = (status == BackupAgentBase::STATE_RUNNING_DIFFERENTIAL || status == BackupAgentBase::STATE_RUNNING);
tagRoot.create("running_backup_is_restorable") = (status == BackupAgentBase::STATE_RUNNING_DIFFERENTIAL);
Initial repository commit
2017-05-26 04:48:44 +08:00
tagRoot.create("range_bytes_written") = tagRangeBytesDR[i].get();
tagRoot.create("mutation_log_bytes_written") = tagLogBytesDR[i].get();
Untested implementation of feature to log basic backup and DR info to fdbcli status.
2017-11-04 05:02:03 +08:00
tagRoot.create("mutation_stream_id") = drTagUids[i].toString();
Initial repository commit
2017-05-26 04:48:44 +08:00
if (backupVersion[i].get().present()) {
double seconds_behind = ((double)readVer - BinaryReader::fromStringRef<Version>(backupVersion[i].get().get(), Unversioned())) / CLIENT_KNOBS->CORE_VERSIONSPERSECOND;
tagRoot.create("seconds_behind") = seconds_behind;
Attempt to normalize trace events: * Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check. * Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase. * Use seconds instead of milliseconds in details. Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed. This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-09 02:11:08 +08:00
//TraceEvent("BackupMetrics").detail("SecondsBehind", seconds_behind);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
tagRoot.create("backup_state") = BackupAgentBase::getStateText(status);
}
}
std::string json = json_spirit::write_string(layersRootValue);
return json;
}
// Check for unparseable or expired statuses and delete them.
// First checks the first doc in the key range, and if it is valid, alive and not "me" then
// returns. Otherwise, checks the rest of the range as well.
ACTOR Future<Void> cleanupStatus(Reference<ReadYourWritesTransaction> tr, std::string rootKey, std::string name, std::string id, int limit = 1) {
state Standalone<RangeResultRef> docs = wait(tr->getRange(KeyRangeRef(rootKey, strinc(rootKey)), limit, true));
state bool readMore = false;
state int i;
for(i = 0; i < docs.size(); ++i) {
json_spirit::mValue docValue;
try {
json_spirit::read_string(docs[i].value.toString(), docValue);
JSONDoc doc(docValue);
// Update the reference version for $expires
JSONDoc::expires_reference_version = tr->getReadVersion().get();
// Evaluate the operators in the document, which will reduce to nothing if it is expired.
doc.cleanOps();
if(!doc.has(name + ".last_updated"))
throw Error();
// Alive and valid.
// If limit == 1 and id is present then read more
if(limit == 1 && doc.has(name + ".instances." + id))
readMore = true;
} catch(Error &e) {
// If doc can't be parsed or isn't alive, delete it.
Remove trace-calls to printable (in non-workloads)
2019-03-19 06:03:43 +08:00
TraceEvent(SevWarn, "RemovedDeadBackupLayerStatus").detail("Key", docs[i].key);
Initial repository commit
2017-05-26 04:48:44 +08:00
tr->clear(docs[i].key);
// If limit is 1 then read more.
if(limit == 1)
readMore = true;
}
if(readMore) {
limit = 10000;
Standalone<RangeResultRef> docs2 = wait(tr->getRange(KeyRangeRef(rootKey, strinc(rootKey)), limit, true));
docs = std::move(docs2);
readMore = false;
}
}
return Void();
}
// Get layer status document for just this layer
ACTOR Future<json_spirit::mObject> getLayerStatus(Database src, std::string rootKey) {
state Transaction tr(src);
loop {
try {
tr.setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr.setOption(FDBTransactionOptions::LOCK_AWARE);
state Standalone<RangeResultRef> kvPairs = wait(tr.getRange(KeyRangeRef(rootKey, strinc(rootKey)), CLIENT_KNOBS->ROW_LIMIT_UNLIMITED));
json_spirit::mObject statusDoc;
JSONDoc modifier(statusDoc);
for(auto &kv : kvPairs) {
json_spirit::mValue docValue;
json_spirit::read_string(kv.value.toString(), docValue);
modifier.absorb(docValue);
}
JSONDoc::expires_reference_version = (uint64_t)tr.getReadVersion().get();
modifier.cleanOps();
return statusDoc;
}
catch (Error& e) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr.onError(e));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
}
// Read layer status for this layer and get the total count of agent processes (instances) then adjust the poll delay based on that and BACKUP_AGGREGATE_POLL_RATE
ACTOR Future<Void> updateAgentPollRate(Database src, std::string rootKey, std::string name, double *pollDelay) {
loop {
try {
json_spirit::mObject status = wait(getLayerStatus(src, rootKey));
int64_t processes = 0;
// If instances count is present and greater than 0 then update pollDelay
if(JSONDoc(status).tryGet<int64_t>(name + ".instances_running", processes) && processes > 0) {
// The aggregate poll rate is the target poll rate for all agent processes in the cluster
// The poll rate (polls/sec) for a single processes is aggregate poll rate / processes, and pollDelay is the inverse of that
*pollDelay = (double)processes / CLIENT_KNOBS->BACKUP_AGGREGATE_POLL_RATE;
}
} catch(Error &e) {
TraceEvent(SevWarn, "BackupAgentPollRateUpdateError").error(e);
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(delay(CLIENT_KNOBS->BACKUP_AGGREGATE_POLL_RATE_UPDATE_INTERVAL));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
Add some basic checks before doing an atomic switchover.
2019-03-02 06:49:04 +08:00
ACTOR Future<Void> statusUpdateActor(Database statusUpdateDest, std::string name, enumProgramExe exe, double *pollDelay, Database taskDest = Database(),
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
std::string id = nondeterministicRandom()->randomUniqueID().toString()) {
Initial repository commit
2017-05-26 04:48:44 +08:00
state std::string metaKey = layerStatusMetaPrefixRange.begin.toString() + "json/" + name;
state std::string rootKey = backupStatusPrefixRange.begin.toString() + name + "/json";
state std::string instanceKey = rootKey + "/" + "agent-" + id;
state Reference<ReadYourWritesTransaction> tr(new ReadYourWritesTransaction(statusUpdateDest));
state Future<Void> pollRateUpdater;
// Register the existence of this layer in the meta key space
loop {
try {
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr->setOption(FDBTransactionOptions::LOCK_AWARE);
tr->set(metaKey, rootKey);
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr->commit());
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
}
catch (Error& e) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr->onError(e));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
// Write status periodically
loop {
tr->reset();
try {
loop{
try {
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr->setOption(FDBTransactionOptions::LOCK_AWARE);
state Future<std::string> futureStatusDoc = getLayerStatus(tr, name, id, exe, taskDest);
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(cleanupStatus(tr, rootKey, name, id));
Initial repository commit
2017-05-26 04:48:44 +08:00
std::string statusdoc = wait(futureStatusDoc);
tr->set(instanceKey, statusdoc);
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr->commit());
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
}
catch (Error& e) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr->onError(e));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
wait(delay(CLIENT_KNOBS->BACKUP_STATUS_DELAY * ( ( 1.0 - CLIENT_KNOBS->BACKUP_STATUS_JITTER ) + 2 * deterministicRandom()->random01() * CLIENT_KNOBS->BACKUP_STATUS_JITTER )));
Initial repository commit
2017-05-26 04:48:44 +08:00
// Now that status was written at least once by this process (and hopefully others), start the poll rate control updater if it wasn't started yet
if(!pollRateUpdater.isValid() && pollDelay != nullptr)
pollRateUpdater = updateAgentPollRate(statusUpdateDest, rootKey, name, pollDelay);
}
catch (Error& e) {
TraceEvent(SevWarnAlways, "UnableToWriteStatus").error(e);
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(delay(10.0));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
}
ACTOR Future<Void> runDBAgent(Database src, Database dest) {
state double pollDelay = 1.0 / CLIENT_KNOBS->BACKUP_AGGREGATE_POLL_RATE;
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
std::string id = nondeterministicRandom()->randomUniqueID().toString();
Add some basic checks before doing an atomic switchover.
2019-03-02 06:49:04 +08:00
state Future<Void> status = statusUpdateActor(src, "dr_backup", EXE_DR_AGENT, &pollDelay, dest, id);
state Future<Void> status_other = statusUpdateActor(dest, "dr_backup_dest", EXE_DR_AGENT, &pollDelay, dest, id);
Initial repository commit
2017-05-26 04:48:44 +08:00
state DatabaseBackupAgent backupAgent(src);
loop {
try {
Ban `Void _ = wait(...)` constructions, and require just `wait(...)`. There's never any reason to save the value of a Void return, and it's the easiest source of redefined variable bugs that will creep back in over time. So just `wait(...)`, it's cleaner that way.
2018-08-11 08:25:43 +08:00
wait(backupAgent.run(dest, &pollDelay, CLIENT_KNOBS->BACKUP_TASKS_PER_AGENT));
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
}
catch (Error& e) {
if (e.code() == error_code_operation_cancelled)
throw;
TraceEvent(SevError, "DA_runAgent").error(e);
fprintf(stderr, "ERROR: DR agent encountered fatal error `%s'\n", e.what());
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( delay(FLOW_KNOBS->PREVENT_FAST_SPIN_DELAY) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
return Void();
}
ACTOR Future<Void> runAgent(Database db) {
state double pollDelay = 1.0 / CLIENT_KNOBS->BACKUP_AGGREGATE_POLL_RATE;
state Future<Void> status = statusUpdateActor(db, "backup", EXE_AGENT, &pollDelay);
state FileBackupAgent backupAgent;
loop {
try {
Ban `Void _ = wait(...)` constructions, and require just `wait(...)`. There's never any reason to save the value of a Void return, and it's the easiest source of redefined variable bugs that will creep back in over time. So just `wait(...)`, it's cleaner that way.
2018-08-11 08:25:43 +08:00
wait(backupAgent.run(db, &pollDelay, CLIENT_KNOBS->BACKUP_TASKS_PER_AGENT));
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
}
catch (Error& e) {
if (e.code() == error_code_operation_cancelled)
throw;
TraceEvent(SevError, "BA_runAgent").error(e);
fprintf(stderr, "ERROR: backup agent encountered fatal error `%s'\n", e.what());
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( delay(FLOW_KNOBS->PREVENT_FAST_SPIN_DELAY) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
return Void();
}
ACTOR Future<Void> submitDBBackup(Database src, Database dest, Standalone<VectorRef<KeyRangeRef>> backupRanges, std::string tagName) {
try
{
state DatabaseBackupAgent backupAgent(src);
// Backup everything, if no ranges were specified
if (backupRanges.size() == 0) {
backupRanges.push_back_deep(backupRanges.arena(), normalKeys);
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(backupAgent.submitBackup(dest, KeyRef(tagName), backupRanges, false, StringRef(), StringRef(), true));
Initial repository commit
2017-05-26 04:48:44 +08:00
// Check if a backup agent is running
bool agentRunning = wait(backupAgent.checkActive(dest));
if (!agentRunning) {
printf("The DR on tag `%s' was successfully submitted but no DR agents are responding.\n", printable(StringRef(tagName)).c_str());
// Throw an error that will not display any additional information
throw actor_cancelled();
}
else {
printf("The DR on tag `%s' was successfully submitted.\n", printable(StringRef(tagName)).c_str());
}
}
catch (Error& e) {
if(e.code() == error_code_actor_cancelled)
throw;
switch (e.code())
{
case error_code_backup_error:
fprintf(stderr, "ERROR: An error was encountered during submission\n");
break;
case error_code_backup_duplicate:
fprintf(stderr, "ERROR: A DR is already running on tag `%s'\n", printable(StringRef(tagName)).c_str());
break;
default:
fprintf(stderr, "ERROR: %s\n", e.what());
break;
}
throw backup_error();
}
return Void();
}
Added snapshot interval option to "fdbbackup start" which defaults to a new knob's value. Added snapshot info to backup status text. Improvements to fdbbackup help.
2017-12-20 16:49:08 +08:00
ACTOR Future<Void> submitBackup(Database db, std::string url, int snapshotIntervalSeconds, Standalone<VectorRef<KeyRangeRef>> backupRanges, std::string tagName, bool dryRun, bool waitForCompletion, bool stopWhenDone) {
Initial repository commit
2017-05-26 04:48:44 +08:00
try
{
state FileBackupAgent backupAgent;
// Backup everything, if no ranges were specified
if (backupRanges.size() == 0) {
backupRanges.push_back_deep(backupRanges.arena(), normalKeys);
}
if (dryRun) {
More backup cleanup.
2017-09-08 00:04:10 +08:00
state KeyBackedTag tag = makeBackupTag(tagName);
Optional<UidAndAbortedFlagT> uidFlag = wait(tag.get(db));
Initial repository commit
2017-05-26 04:48:44 +08:00
More backup cleanup.
2017-09-08 00:04:10 +08:00
if (uidFlag.present()) {
BackupConfig config(uidFlag.get().first);
EBackupState backupStatus = wait(config.stateEnum().getOrThrow(db));
// Throw error if a backup is currently running until we support parallel backups
if (BackupAgentBase::isRunnable((BackupAgentBase::enumState)backupStatus)) {
throw backup_duplicate();
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
if (waitForCompletion) {
printf("Submitted and now waiting for the backup on tag `%s' to complete. (DRY RUN)\n", printable(StringRef(tagName)).c_str());
}
else {
// Check if a backup agent is running
bool agentRunning = wait(backupAgent.checkActive(db));
if (!agentRunning) {
printf("The backup on tag `%s' was successfully submitted but no backup agents are responding. (DRY RUN)\n", printable(StringRef(tagName)).c_str());
// Throw an error that will not display any additional information
throw actor_cancelled();
}
else {
printf("The backup on tag `%s' was successfully submitted. (DRY RUN)\n", printable(StringRef(tagName)).c_str());
}
}
}
else {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(backupAgent.submitBackup(db, KeyRef(url), snapshotIntervalSeconds, tagName, backupRanges, stopWhenDone));
Initial repository commit
2017-05-26 04:48:44 +08:00
// Wait for the backup to complete, if requested
if (waitForCompletion) {
printf("Submitted and now waiting for the backup on tag `%s' to complete.\n", printable(StringRef(tagName)).c_str());
Replace unused _ variables with wait(success(...))
2019-02-13 08:07:17 +08:00
wait(success(backupAgent.waitBackup(db, tagName)));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
else {
// Check if a backup agent is running
bool agentRunning = wait(backupAgent.checkActive(db));
if (!agentRunning) {
printf("The backup on tag `%s' was successfully submitted but no backup agents are responding.\n", printable(StringRef(tagName)).c_str());
// Throw an error that will not display any additional information
throw actor_cancelled();
}
else {
printf("The backup on tag `%s' was successfully submitted.\n", printable(StringRef(tagName)).c_str());
}
}
}
}
catch (Error& e) {
if(e.code() == error_code_actor_cancelled)
throw;
switch (e.code())
{
case error_code_backup_error:
fprintf(stderr, "ERROR: An error was encountered during submission\n");
break;
case error_code_backup_duplicate:
fprintf(stderr, "ERROR: A backup is already running on tag `%s'\n", printable(StringRef(tagName)).c_str());
break;
default:
fprintf(stderr, "ERROR: %s\n", e.what());
break;
}
throw backup_error();
}
return Void();
}
Skip switchover checks for force option.
2019-03-05 07:58:36 +08:00
ACTOR Future<Void> switchDBBackup(Database src, Database dest, Standalone<VectorRef<KeyRangeRef>> backupRanges, std::string tagName, bool forceAction) {
Initial repository commit
2017-05-26 04:48:44 +08:00
try
{
state DatabaseBackupAgent backupAgent(src);
// Backup everything, if no ranges were specified
if (backupRanges.size() == 0) {
backupRanges.push_back_deep(backupRanges.arena(), normalKeys);
}
Skip switchover checks for force option.
2019-03-05 07:58:36 +08:00
wait(backupAgent.atomicSwitchover(dest, KeyRef(tagName), backupRanges, StringRef(), StringRef(), forceAction));
Initial repository commit
2017-05-26 04:48:44 +08:00
printf("The DR on tag `%s' was successfully switched.\n", printable(StringRef(tagName)).c_str());
}
catch (Error& e) {
if(e.code() == error_code_actor_cancelled)
throw;
switch (e.code())
{
case error_code_backup_error:
fprintf(stderr, "ERROR: An error was encountered during submission\n");
break;
case error_code_backup_duplicate:
fprintf(stderr, "ERROR: A DR is already running on tag `%s'\n", printable(StringRef(tagName)).c_str());
break;
default:
fprintf(stderr, "ERROR: %s\n", e.what());
break;
}
throw backup_error();
}
return Void();
}
ACTOR Future<Void> statusDBBackup(Database src, Database dest, std::string tagName, int errorLimit) {
try
{
state DatabaseBackupAgent backupAgent(src);
std::string statusText = wait(backupAgent.getStatus(dest, errorLimit, StringRef(tagName)));
printf("%s\n", statusText.c_str());
}
catch (Error& e) {
if(e.code() == error_code_actor_cancelled)
throw;
fprintf(stderr, "ERROR: %s\n", e.what());
throw;
}
return Void();
}
Added --json option to fdbbackup status.
2019-03-07 13:32:46 +08:00
ACTOR Future<Void> statusBackup(Database db, std::string tagName, bool showErrors, bool json) {
Initial repository commit
2017-05-26 04:48:44 +08:00
try
{
state FileBackupAgent backupAgent;
Added --json option to fdbbackup status.
2019-03-07 13:32:46 +08:00
std::string statusText = wait(json ? backupAgent.getStatusJSON(db, tagName) : backupAgent.getStatus(db, showErrors, tagName));
Initial repository commit
2017-05-26 04:48:44 +08:00
printf("%s\n", statusText.c_str());
}
catch (Error& e) {
if(e.code() == error_code_actor_cancelled)
throw;
fprintf(stderr, "ERROR: %s\n", e.what());
throw;
}
return Void();
}
ACTOR Future<Void> abortDBBackup(Database src, Database dest, std::string tagName, bool partial) {
try
{
state DatabaseBackupAgent backupAgent(src);
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(backupAgent.abortBackup(dest, Key(tagName), partial));
wait(backupAgent.unlockBackup(dest, Key(tagName)));
Initial repository commit
2017-05-26 04:48:44 +08:00
printf("The DR on tag `%s' was successfully aborted.\n", printable(StringRef(tagName)).c_str());
}
catch (Error& e) {
if(e.code() == error_code_actor_cancelled)
throw;
switch (e.code())
{
case error_code_backup_error:
fprintf(stderr, "ERROR: An error was encountered during submission\n");
break;
case error_code_backup_unneeded:
fprintf(stderr, "ERROR: A DR was not running on tag `%s'\n", printable(StringRef(tagName)).c_str());
break;
default:
fprintf(stderr, "ERROR: %s\n", e.what());
break;
}
throw;
}
return Void();
}
ACTOR Future<Void> abortBackup(Database db, std::string tagName) {
try
{
state FileBackupAgent backupAgent;
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(backupAgent.abortBackup(db, tagName));
Initial repository commit
2017-05-26 04:48:44 +08:00
printf("The backup on tag `%s' was successfully aborted.\n", printable(StringRef(tagName)).c_str());
}
catch (Error& e) {
if(e.code() == error_code_actor_cancelled)
throw;
switch (e.code())
{
case error_code_backup_error:
fprintf(stderr, "ERROR: An error was encountered during submission\n");
break;
case error_code_backup_unneeded:
fprintf(stderr, "ERROR: A backup was not running on tag `%s'\n", printable(StringRef(tagName)).c_str());
break;
default:
fprintf(stderr, "ERROR: %s\n", e.what());
break;
}
throw;
}
return Void();
}
ACTOR Future<Void> waitBackup(Database db, std::string tagName, bool stopWhenDone) {
try
{
state FileBackupAgent backupAgent;
Using BackupConfig from backup.actor.cpp to reduce intermediate functions.
2017-09-07 11:06:32 +08:00
int status = wait(backupAgent.waitBackup(db, tagName, stopWhenDone));
Initial repository commit
2017-05-26 04:48:44 +08:00
printf("The backup on tag `%s' %s.\n", printable(StringRef(tagName)).c_str(),
BackupAgentBase::getStateText((BackupAgentBase::enumState) status));
}
catch (Error& e) {
if(e.code() == error_code_actor_cancelled)
throw;
fprintf(stderr, "ERROR: %s\n", e.what());
throw;
}
return Void();
}
ACTOR Future<Void> discontinueBackup(Database db, std::string tagName, bool waitForCompletion) {
try
{
state FileBackupAgent backupAgent;
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(backupAgent.discontinueBackup(db, StringRef(tagName)));
Initial repository commit
2017-05-26 04:48:44 +08:00
// Wait for the backup to complete, if requested
if (waitForCompletion) {
printf("Discontinued and now waiting for the backup on tag `%s' to complete.\n", printable(StringRef(tagName)).c_str());
Replace unused _ variables with wait(success(...))
2019-02-13 08:07:17 +08:00
wait(success(backupAgent.waitBackup(db, tagName)));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
else {
printf("The backup on tag `%s' was successfully discontinued.\n", printable(StringRef(tagName)).c_str());
}
}
catch (Error& e) {
if(e.code() == error_code_actor_cancelled)
throw;
switch (e.code())
{
case error_code_backup_error:
fprintf(stderr, "ERROR: An encounter was error during submission\n");
break;
case error_code_backup_unneeded:
fprintf(stderr, "ERROR: A backup in not running on tag `%s'\n", printable(StringRef(tagName)).c_str());
break;
case error_code_backup_duplicate:
fprintf(stderr, "ERROR: The backup on tag `%s' is already discontinued\n", printable(StringRef(tagName)).c_str());
break;
default:
fprintf(stderr, "ERROR: %s\n", e.what());
break;
}
throw;
}
return Void();
}
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
ACTOR Future<Void> changeBackupResumed(Database db, bool pause) {
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
try {
state FileBackupAgent backupAgent;
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(backupAgent.taskBucket->changePause(db, pause));
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
printf("All backup agents have been %s.\n", pause ? "paused" : "resumed");
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
}
catch (Error& e) {
if(e.code() == error_code_actor_cancelled)
throw;
fprintf(stderr, "ERROR: %s\n", e.what());
throw;
}
return Void();
}
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
ACTOR Future<Void> changeDBBackupResumed(Database src, Database dest, bool pause) {
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
try {
state DatabaseBackupAgent backupAgent(src);
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(backupAgent.taskBucket->changePause(dest, pause));
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
printf("All DR agents have been %s.\n", pause ? "paused" : "resumed");
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
}
catch (Error& e) {
if(e.code() == error_code_actor_cancelled)
throw;
fprintf(stderr, "ERROR: %s\n", e.what());
throw;
}
return Void();
}
Restore now requires an explicit destination cluster file. Restore target database version can now be specified by a timestamp, which must be resolved using the cluster file for the cluster from which the backup originated.
2019-03-06 04:28:05 +08:00
Reference<IBackupContainer> openBackupContainer(const char *name, std::string destinationContainer) {
// Error, if no dest container was specified
if (destinationContainer.empty()) {
fprintf(stderr, "ERROR: No backup destination was specified.\n");
printHelpTeaser(name);
throw backup_error();
}
Reference<IBackupContainer> c;
try {
c = IBackupContainer::openContainer(destinationContainer);
}
catch (Error& e) {
std::string msg = format("ERROR: '%s' on URL '%s'", e.what(), destinationContainer.c_str());
if(e.code() == error_code_backup_invalid_url && !IBackupContainer::lastOpenError.empty()) {
msg += format(": %s", IBackupContainer::lastOpenError.c_str());
}
fprintf(stderr, "%s\n", msg.c_str());
printHelpTeaser(name);
throw;
}
return c;
}
Bug fix: fdbrestore abort, wait, and status were not using the --dest_cluster_file argument.
2019-06-14 08:42:11 +08:00
ACTOR Future<Void> runRestore(Database db, std::string originalClusterFile, std::string tagName, std::string container, Standalone<VectorRef<KeyRangeRef>> ranges, Version targetVersion, std::string targetTimestamp, bool performRestore, bool verbose, bool waitForDone, std::string addPrefix, std::string removePrefix) {
Bug fix: fdbrestore was not using a default range set.
2019-03-06 17:57:03 +08:00
if(ranges.empty()) {
ranges.push_back_deep(ranges.arena(), normalKeys);
}
Restore now requires an explicit destination cluster file. Restore target database version can now be specified by a timestamp, which must be resolved using the cluster file for the cluster from which the backup originated.
2019-03-06 04:28:05 +08:00
if(targetVersion != invalidVersion && !targetTimestamp.empty()) {
fprintf(stderr, "Restore target version and target timestamp cannot both be specified\n");
throw restore_error();
}
Bug fix, restore was using the destination cluster to look up timestamps when printing the backup description instead of (optionally) the original cluster which generated the backup. Made missing cluster file errors more clear.
2019-03-06 18:45:55 +08:00
state Optional<Database> origDb;
Restore now requires an explicit destination cluster file. Restore target database version can now be specified by a timestamp, which must be resolved using the cluster file for the cluster from which the backup originated.
2019-03-06 04:28:05 +08:00
// Resolve targetTimestamp if given
if(!targetTimestamp.empty()) {
if(originalClusterFile.empty()) {
fprintf(stderr, "An original cluster file must be given in order to resolve restore target timestamp '%s'\n", targetTimestamp.c_str());
throw restore_error();
}
Bug fix, restore was using the destination cluster to look up timestamps when printing the backup description instead of (optionally) the original cluster which generated the backup. Made missing cluster file errors more clear.
2019-03-06 18:45:55 +08:00
if(!fileExists(originalClusterFile)) {
fprintf(stderr, "Original source database cluster file '%s' does not exist.\n", originalClusterFile.c_str());
throw restore_error();
}
origDb = Database::createDatabase(originalClusterFile, Database::API_VERSION_LATEST);
Version v = wait(timeKeeperVersionFromDatetime(targetTimestamp, origDb.get()));
fdbbackup: fix some print/scan format warnings Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-05 01:48:09 +08:00
printf("Timestamp '%s' resolves to version %" PRId64 "\n", targetTimestamp.c_str(), v);
Restore now requires an explicit destination cluster file. Restore target database version can now be specified by a timestamp, which must be resolved using the cluster file for the cluster from which the backup originated.
2019-03-06 04:28:05 +08:00
targetVersion = v;
}
try {
Initial repository commit
2017-05-26 04:48:44 +08:00
state FileBackupAgent backupAgent;
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
state Reference<IBackupContainer> bc = openBackupContainer(exeRestore.toString().c_str(), container);
Restore start with dry run option no longer requires or uses a database. Removed duplicate and unnecessary code. Dry run execution is more clear, it does not pretend to have actually restored data. Fixed typo in help text.
2019-01-09 11:00:58 +08:00
// If targetVersion is unset then use the maximum restorable version from the backup description
if(targetVersion == invalidVersion) {
if(verbose)
printf("No restore target version given, will use maximum restorable version from backup description.\n");
Fdbbackup restore will now default to the latest usable version in a backup if a specific restore version was not given. Expire will make sure a cluster is provided if either of its timestamp options are given.
2018-01-20 04:19:32 +08:00
Restore start with dry run option no longer requires or uses a database. Removed duplicate and unnecessary code. Dry run execution is more clear, it does not pretend to have actually restored data. Fixed typo in help text.
2019-01-09 11:00:58 +08:00
BackupDescription desc = wait(bc->describeBackup());
Fdbbackup restore will now default to the latest usable version in a backup if a specific restore version was not given. Expire will make sure a cluster is provided if either of its timestamp options are given.
2018-01-20 04:19:32 +08:00
Restore start with dry run option no longer requires or uses a database. Removed duplicate and unnecessary code. Dry run execution is more clear, it does not pretend to have actually restored data. Fixed typo in help text.
2019-01-09 11:00:58 +08:00
if(!desc.maxRestorableVersion.present()) {
fprintf(stderr, "The specified backup is not restorable to any version.\n");
throw restore_error();
Fdbbackup restore will now default to the latest usable version in a backup if a specific restore version was not given. Expire will make sure a cluster is provided if either of its timestamp options are given.
2018-01-20 04:19:32 +08:00
}
Merge branch 'release-6.0' # Conflicts: # documentation/sphinx/source/release-notes.rst # fdbbackup/backup.actor.cpp # fdbclient/BackupContainer.actor.cpp # fdbclient/HTTP.actor.cpp # fdbserver/storageserver.actor.cpp # fdbserver/workloads/BackupCorrectness.actor.cpp # versions.target
2019-01-10 08:14:46 +08:00
Restore start with dry run option no longer requires or uses a database. Removed duplicate and unnecessary code. Dry run execution is more clear, it does not pretend to have actually restored data. Fixed typo in help text.
2019-01-09 11:00:58 +08:00
targetVersion = desc.maxRestorableVersion.get();
if(verbose)
fdbbackup: fix some print/scan format warnings Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-05 01:48:09 +08:00
printf("Using target restore version %" PRId64 "\n", targetVersion);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Restore start with dry run option no longer requires or uses a database. Removed duplicate and unnecessary code. Dry run execution is more clear, it does not pretend to have actually restored data. Fixed typo in help text.
2019-01-09 11:00:58 +08:00
if (performRestore) {
Bug fix, restore was using the destination cluster to look up timestamps when printing the backup description instead of (optionally) the original cluster which generated the backup. Made missing cluster file errors more clear.
2019-03-06 18:45:55 +08:00
Version restoredVersion = wait(backupAgent.restore(db, origDb, KeyRef(tagName), KeyRef(container), ranges, waitForDone, targetVersion, verbose, KeyRef(addPrefix), KeyRef(removePrefix)));
Restore start with dry run option no longer requires or uses a database. Removed duplicate and unnecessary code. Dry run execution is more clear, it does not pretend to have actually restored data. Fixed typo in help text.
2019-01-09 11:00:58 +08:00
if(waitForDone && verbose) {
// If restore is now complete then report version restored
fdbbackup: fix some print/scan format warnings Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-05 01:48:09 +08:00
printf("Restored to version %" PRId64 "\n", restoredVersion);
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
}
Restore start with dry run option no longer requires or uses a database. Removed duplicate and unnecessary code. Dry run execution is more clear, it does not pretend to have actually restored data. Fixed typo in help text.
2019-01-09 11:00:58 +08:00
}
else {
state Optional<RestorableFileSet> rset = wait(bc->getRestoreSet(targetVersion));
Initial repository commit
2017-05-26 04:48:44 +08:00
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
if(!rset.present()) {
fdbbackup: fix some print/scan format warnings Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-05 01:48:09 +08:00
fprintf(stderr, "Insufficient data to restore to version %" PRId64 ". Describe backup for more information.\n", targetVersion);
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
throw restore_invalid_version();
}
Initial repository commit
2017-05-26 04:48:44 +08:00
fdbbackup: fix some print/scan format warnings Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-05 01:48:09 +08:00
printf("Backup can be used to restore to version %" PRId64 "\n", targetVersion);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
catch (Error& e) {
if(e.code() == error_code_actor_cancelled)
throw;
fprintf(stderr, "ERROR: %s\n", e.what());
throw;
}
return Void();
}
Added fdbbackup dump option to list files and sizes in a backup for a given version range.
2019-01-03 18:35:31 +08:00
ACTOR Future<Void> dumpBackupData(const char *name, std::string destinationContainer, Version beginVersion, Version endVersion) {
state Reference<IBackupContainer> c = openBackupContainer(name, destinationContainer);
if(beginVersion < 0 || endVersion < 0) {
BackupDescription desc = wait(c->describeBackup());
if(!desc.maxLogEnd.present()) {
fprintf(stderr, "ERROR: Backup must have log data in order to use relative begin/end versions.\n");
throw backup_invalid_info();
}
if(beginVersion < 0) {
beginVersion += desc.maxLogEnd.get();
}
if(endVersion < 0) {
endVersion += desc.maxLogEnd.get();
}
}
fdbbackup: fix some print/scan format warnings Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-05 01:48:09 +08:00
printf("Scanning version range %" PRId64 " to %" PRId64 "\n", beginVersion, endVersion);
Added fdbbackup dump option to list files and sizes in a backup for a given version range.
2019-01-03 18:35:31 +08:00
BackupFileList files = wait(c->dumpFileList(beginVersion, endVersion));
files.toStream(stdout);
return Void();
}
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
ACTOR Future<Void> expireBackupData(const char *name, std::string destinationContainer, Version endVersion, std::string endDatetime, Database db, bool force, Version restorableAfterVersion, std::string restorableAfterDatetime) {
if (!endDatetime.empty()) {
Corrected function name to timeKeeperVersionFromDatetime(). 'Fdbbackup expire' now allows an expire_before version of 0 if explicitly passed by version or by timestamp.
2018-01-23 16:19:51 +08:00
Version v = wait( timeKeeperVersionFromDatetime(endDatetime, db) );
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
endVersion = v;
}
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
if (!restorableAfterDatetime.empty()) {
Corrected function name to timeKeeperVersionFromDatetime(). 'Fdbbackup expire' now allows an expire_before version of 0 if explicitly passed by version or by timestamp.
2018-01-23 16:19:51 +08:00
Version v = wait( timeKeeperVersionFromDatetime(restorableAfterDatetime, db) );
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
restorableAfterVersion = v;
}
Corrected function name to timeKeeperVersionFromDatetime(). 'Fdbbackup expire' now allows an expire_before version of 0 if explicitly passed by version or by timestamp.
2018-01-23 16:19:51 +08:00
if (endVersion == invalidVersion) {
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
fprintf(stderr, "ERROR: No version or date/time is specified.\n");
printHelpTeaser(name);
throw backup_error();;
}
Initial repository commit
2017-05-26 04:48:44 +08:00
try {
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
Reference<IBackupContainer> c = openBackupContainer(name, destinationContainer);
Added progress reporting to backup expiration. Simplified backup delete progress.
2018-12-20 16:23:26 +08:00
state IBackupContainer::ExpireProgress progress;
state std::string lastProgress;
state Future<Void> expire = c->expireData(endVersion, force, &progress, restorableAfterVersion);
loop {
choose {
Merge branch 'release-6.0' # Conflicts: # documentation/sphinx/source/release-notes.rst # fdbbackup/backup.actor.cpp # fdbclient/BackupContainer.actor.cpp # fdbclient/HTTP.actor.cpp # fdbserver/storageserver.actor.cpp # fdbserver/workloads/BackupCorrectness.actor.cpp # versions.target
2019-01-10 08:14:46 +08:00
when(wait(delay(5))) {
Added progress reporting to backup expiration. Simplified backup delete progress.
2018-12-20 16:23:26 +08:00
std::string p = progress.toString();
if(p != lastProgress) {
int spaces = lastProgress.size() - p.size();
printf("\r%s%s", p.c_str(), (spaces > 0 ? std::string(spaces, ' ').c_str() : "") );
lastProgress = p;
}
}
Merge branch 'release-6.0' # Conflicts: # documentation/sphinx/source/release-notes.rst # fdbbackup/backup.actor.cpp # fdbclient/BackupContainer.actor.cpp # fdbclient/HTTP.actor.cpp # fdbserver/storageserver.actor.cpp # fdbserver/workloads/BackupCorrectness.actor.cpp # versions.target
2019-01-10 08:14:46 +08:00
when(wait(expire)) {
Added progress reporting to backup expiration. Simplified backup delete progress.
2018-12-20 16:23:26 +08:00
break;
}
}
}
std::string p = progress.toString();
int spaces = lastProgress.size() - p.size();
printf("\r%s%s\n", p.c_str(), (spaces > 0 ? std::string(spaces, ' ').c_str() : "") );
Added fdbbackup expire options to calculate approximate version boundaries based on a number of days prior to the latest log file found in the backup container. This enables expiration operations based on time (with reasonable precision) without accessing the source cluster.
2018-12-19 10:55:44 +08:00
if(endVersion < 0)
fdbbackup: fix some print/scan format warnings Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-05 01:48:09 +08:00
printf("All data before %" PRId64 " versions (%" PRId64 " days) prior to latest backup log has been deleted.\n", -endVersion, -endVersion / ((int64_t)24 * 3600 * CLIENT_KNOBS->CORE_VERSIONSPERSECOND));
Added fdbbackup expire options to calculate approximate version boundaries based on a number of days prior to the latest log file found in the backup container. This enables expiration operations based on time (with reasonable precision) without accessing the source cluster.
2018-12-19 10:55:44 +08:00
else
fdbbackup: fix some print/scan format warnings Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-05 01:48:09 +08:00
printf("All data before version %" PRId64 " has been deleted.\n", endVersion);
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
}
catch (Error& e) {
if(e.code() == error_code_actor_cancelled)
throw;
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
if(e.code() == error_code_backup_cannot_expire)
fprintf(stderr, "ERROR: Requested expiration would be unsafe. Backup would not meet minimum restorability. Use --force to delete data anyway.\n");
else
fprintf(stderr, "ERROR: %s\n", e.what());
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
throw;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
return Void();
}
Initial repository commit
2017-05-26 04:48:44 +08:00
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
ACTOR Future<Void> deleteBackupContainer(const char *name, std::string destinationContainer) {
Initial repository commit
2017-05-26 04:48:44 +08:00
try {
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
state Reference<IBackupContainer> c = openBackupContainer(name, destinationContainer);
Bug fixes. Fdbbackup delete was broken. Blobstore backup container deletion would do too much listing before deletions began due to list operations queueing up ahead of and starving the delete operations. Created new knob and blob endpoint limit for concurrent list operations to fix this. Increased blob request timeout default because some requests were taking longer. Crash fixes in blobstore doRequest() which wasn't checking that response object is valid before using it in error conditions. Filesystem-like backup container class (covering blobstore and local dirs) now ignores unrecognized filenames for describe() and expire() operations.
2018-01-06 15:06:39 +08:00
state int numDeleted = 0;
state Future<Void> done = c->deleteContainer(&numDeleted);
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
Added progress reporting to backup expiration. Simplified backup delete progress.
2018-12-20 16:23:26 +08:00
state int lastUpdate = -1;
Backup describe, expire, and delete will now clearly indicate when there is no backup at the given URL.
2018-12-21 10:05:23 +08:00
printf("Deleting %s...\n", destinationContainer.c_str());
Added progress reporting to backup expiration. Simplified backup delete progress.
2018-12-20 16:23:26 +08:00
Initial repository commit
2017-05-26 04:48:44 +08:00
loop {
choose {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when ( wait(done) ) {
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
}
Merge branch 'release-6.0' # Conflicts: # documentation/sphinx/source/release-notes.rst # fdbbackup/backup.actor.cpp # fdbclient/BackupContainer.actor.cpp # fdbclient/HTTP.actor.cpp # fdbserver/storageserver.actor.cpp # fdbserver/workloads/BackupCorrectness.actor.cpp # versions.target
2019-01-10 08:14:46 +08:00
when ( wait(delay(5)) ) {
Added progress reporting to backup expiration. Simplified backup delete progress.
2018-12-20 16:23:26 +08:00
if(numDeleted != lastUpdate) {
printf("\r%d...", numDeleted);
lastUpdate = numDeleted;
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
}
Print a final deleted object count at the end of backup deletion.
2018-12-20 17:24:31 +08:00
printf("\r%d objects deleted\n", numDeleted);
Backup describe, expire, and delete will now clearly indicate when there is no backup at the given URL.
2018-12-21 10:05:23 +08:00
printf("The entire container has been deleted.\n");
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
}
catch (Error& e) {
if(e.code() == error_code_actor_cancelled)
throw;
fprintf(stderr, "ERROR: %s\n", e.what());
throw;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
return Void();
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Added --json option to fdbbackup describe. Also added expired percentage indicator to snapshot details.
2019-03-07 06:14:06 +08:00
ACTOR Future<Void> describeBackup(const char *name, std::string destinationContainer, bool deep, Optional<Database> cx, bool json) {
Initial repository commit
2017-05-26 04:48:44 +08:00
try {
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
Reference<IBackupContainer> c = openBackupContainer(name, destinationContainer);
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
state BackupDescription desc = wait(c->describeBackup(deep));
if(cx.present())
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(desc.resolveVersionTimes(cx.get()));
Added --json option to fdbbackup describe. Also added expired percentage indicator to snapshot details.
2019-03-07 06:14:06 +08:00
printf("%s\n", (json ? desc.toJSON() : desc.toString()).c_str());
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
catch (Error& e) {
if(e.code() == error_code_actor_cancelled)
throw;
fprintf(stderr, "ERROR: %s\n", e.what());
throw;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
return Void();
}
Initial repository commit
2017-05-26 04:48:44 +08:00
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
ACTOR Future<Void> listBackup(std::string baseUrl) {
Initial repository commit
2017-05-26 04:48:44 +08:00
try {
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
std::vector<std::string> containers = wait(IBackupContainer::listContainers(baseUrl));
for (std::string container : containers) {
printf("%s\n", container.c_str());
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
}
catch (Error& e) {
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
std::string msg = format("ERROR: %s", e.what());
if(e.code() == error_code_backup_invalid_url && !IBackupContainer::lastOpenError.empty()) {
msg += format(": %s", IBackupContainer::lastOpenError.c_str());
}
fprintf(stderr, "%s\n", msg.c_str());
Initial repository commit
2017-05-26 04:48:44 +08:00
throw;
}
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
return Void();
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
struct BackupModifyOptions {
Optional<std::string> verifyUID;
Optional<std::string> destURL;
Bug fixes in fdbbackup modify command. Active snapshot interval was wrong. New backup destination containers were not being created. Renamed some variables for clarity.
2019-03-06 10:05:52 +08:00
Optional<int> snapshotIntervalSeconds;
Optional<int> activeSnapshotIntervalSeconds;
bool hasChanges() const {
return destURL.present() || snapshotIntervalSeconds.present() || activeSnapshotIntervalSeconds.present();
}
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
};
ACTOR Future<Void> modifyBackup(Database db, std::string tagName, BackupModifyOptions options) {
Bug fixes in fdbbackup modify command. Active snapshot interval was wrong. New backup destination containers were not being created. Renamed some variables for clarity.
2019-03-06 10:05:52 +08:00
if(!options.hasChanges()) {
fprintf(stderr, "No changes were specified, nothing to do!\n");
throw backup_error();
}
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
state KeyBackedTag tag = makeBackupTag(tagName);
Bug fixes in fdbbackup modify command. Active snapshot interval was wrong. New backup destination containers were not being created. Renamed some variables for clarity.
2019-03-06 10:05:52 +08:00
state Reference<IBackupContainer> bc;
if(options.destURL.present()) {
bc = openBackupContainer(exeBackup.toString().c_str(), options.destURL.get());
try {
Merge branch 'release-6.0' of https://github.com/apple/foundationdb # Conflicts: # documentation/sphinx/source/release-notes.rst # fdbbackup/backup.actor.cpp # fdbclient/BlobStore.actor.cpp
2019-03-12 19:49:12 +08:00
wait(timeoutError(bc->create(), 30));
Bug fixes in fdbbackup modify command. Active snapshot interval was wrong. New backup destination containers were not being created. Renamed some variables for clarity.
2019-03-06 10:05:52 +08:00
} catch(Error &e) {
if(e.code() == error_code_actor_cancelled)
throw;
fprintf(stderr, "ERROR: Could not create backup container at '%s': %s\n", options.destURL.get().c_str(), e.what());
throw backup_error();
}
}
state Reference<ReadYourWritesTransaction> tr(new ReadYourWritesTransaction(db));
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
loop {
try {
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr->setOption(FDBTransactionOptions::LOCK_AWARE);
state Optional<UidAndAbortedFlagT> uidFlag = wait(tag.get(db));
Bug fix: fdbbackup modify on a tag that has never been used before would result in an internal error.
2019-03-06 14:17:37 +08:00
if(!uidFlag.present()) {
fprintf(stderr, "No backup exists on tag '%s'\n", tagName.c_str());
throw backup_error();
}
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
if(uidFlag.get().second) {
fprintf(stderr, "Cannot modify aborted backup on tag '%s'\n", tagName.c_str());
throw backup_error();
}
state BackupConfig config(uidFlag.get().first);
EBackupState s = wait(config.stateEnum().getOrThrow(tr, false, backup_invalid_info()));
if(!FileBackupAgent::isRunnable(s)) {
fprintf(stderr, "Backup on tag '%s' is not runnable.\n", tagName.c_str());
throw backup_error();
}
if(options.verifyUID.present() && options.verifyUID.get() != uidFlag.get().first.toString()) {
fprintf(stderr, "UID verification failed, backup on tag '%s' is '%s' but '%s' was specified.\n", tagName.c_str(), uidFlag.get().first.toString().c_str(), options.verifyUID.get().c_str());
throw backup_error();
}
Bug fixes in fdbbackup modify command. Active snapshot interval was wrong. New backup destination containers were not being created. Renamed some variables for clarity.
2019-03-06 10:05:52 +08:00
if(options.snapshotIntervalSeconds.present()) {
config.snapshotIntervalSeconds().set(tr, options.snapshotIntervalSeconds.get());
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
}
Bug fixes in fdbbackup modify command. Active snapshot interval was wrong. New backup destination containers were not being created. Renamed some variables for clarity.
2019-03-06 10:05:52 +08:00
if(options.activeSnapshotIntervalSeconds.present()) {
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
Version begin = wait(config.snapshotBeginVersion().getOrThrow(tr, false, backup_error()));
Bug fixes in fdbbackup modify command. Active snapshot interval was wrong. New backup destination containers were not being created. Renamed some variables for clarity.
2019-03-06 10:05:52 +08:00
config.snapshotTargetEndVersion().set(tr, begin + ((int64_t)options.activeSnapshotIntervalSeconds.get() * CLIENT_KNOBS->CORE_VERSIONSPERSECOND));
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
}
if(options.destURL.present()) {
config.backupContainer().set(tr, bc);
}
Merge branch 'release-6.0' of https://github.com/apple/foundationdb # Conflicts: # documentation/sphinx/source/release-notes.rst # fdbbackup/backup.actor.cpp # fdbclient/BlobStore.actor.cpp
2019-03-12 19:49:12 +08:00
wait(tr->commit());
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
break;
}
catch (Error& e) {
Merge branch 'release-6.0' of https://github.com/apple/foundationdb # Conflicts: # documentation/sphinx/source/release-notes.rst # fdbbackup/backup.actor.cpp # fdbclient/BlobStore.actor.cpp
2019-03-12 19:49:12 +08:00
wait(tr->onError(e));
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
}
}
return Void();
}
Initial repository commit
2017-05-26 04:48:44 +08:00
static std::vector<std::vector<StringRef>> parseLine(std::string &line, bool& err, bool& partial)
{
err = false;
partial = false;
bool quoted = false;
std::vector<StringRef> buf;
std::vector<std::vector<StringRef>> ret;
size_t i = line.find_first_not_of(' ');
size_t offset = i;
bool forcetoken = false;
while (i <= line.length()) {
switch (line[i]) {
case ';':
if (!quoted) {
if (i > offset)
buf.push_back(StringRef((uint8_t*)(line.data() + offset), i - offset));
ret.push_back(std::move(buf));
offset = i = line.find_first_not_of(' ', i + 1);
}
else
i++;
break;
case '"':
quoted = !quoted;
line.erase(i, 1);
if (quoted)
forcetoken = true;
break;
case ' ':
if (!quoted) {
buf.push_back(StringRef((uint8_t *)(line.data() + offset),
i - offset));
offset = i = line.find_first_not_of(' ', i);
forcetoken = false;
}
else
i++;
break;
case '\\':
if (i + 2 > line.length()) {
err = true;
ret.push_back(std::move(buf));
return ret;
}
switch (line[i + 1]) {
char ent, save;
case '"':
case '\\':
case ' ':
case ';':
line.erase(i, 1);
break;
case 'x':
if (i + 4 > line.length()) {
err = true;
ret.push_back(std::move(buf));
return ret;
}
char *pEnd;
save = line[i + 4];
line[i + 4] = 0;
ent = char(strtoul(line.data() + i + 2, &pEnd, 16));
if (*pEnd) {
err = true;
ret.push_back(std::move(buf));
return ret;
}
line[i + 4] = save;
line.replace(i, 4, 1, ent);
break;
default:
err = true;
ret.push_back(std::move(buf));
return ret;
}
default:
i++;
}
}
i -= 1;
if (i > offset || forcetoken)
buf.push_back(StringRef((uint8_t*)(line.data() + offset), i - offset));
ret.push_back(std::move(buf));
if (quoted)
partial = true;
return ret;
}
static void addKeyRange(std::string optionValue, Standalone<VectorRef<KeyRangeRef>>& keyRanges)
{
bool err = false, partial = false;
int tokenArray = 0, tokenIndex = 0;
auto parsed = parseLine(optionValue, err, partial);
for (auto tokens : parsed)
{
tokenArray++;
tokenIndex = 0;
/*
for (auto token : tokens)
{
tokenIndex++;
printf("%4d token #%2d: %s\n", tokenArray, tokenIndex, printable(token).c_str());
}
*/
// Process the keys
// <begin> [end]
switch (tokens.size())
{
// empty
case 0:
break;
// single key range
case 1:
keyRanges.push_back_deep(keyRanges.arena(), KeyRangeRef(tokens.at(0), strinc(tokens.at(0))));
break;
// full key range
case 2:
try {
keyRanges.push_back_deep(keyRanges.arena(), KeyRangeRef(tokens.at(0), tokens.at(1)));
}
catch (Error& e) {
fprintf(stderr, "ERROR: Invalid key range `%s %s' reported error %s\n",
tokens.at(0).toString().c_str(), tokens.at(1).toString().c_str(), e.what());
throw invalid_option_value();
}
break;
// Too many keys
default:
fprintf(stderr, "ERROR: Invalid key range identified with %ld keys", tokens.size());
throw invalid_option_value();
break;
}
}
return;
}
Added fdbbackup dump option to list files and sizes in a backup for a given version range.
2019-01-03 18:35:31 +08:00
Version parseVersion(const char *str) {
StringRef s((const uint8_t *)str, strlen(str));
if(s.endsWith(LiteralStringRef("days")) || s.endsWith(LiteralStringRef("d"))) {
float days;
if(sscanf(str, "%f", &days) != 1) {
fprintf(stderr, "Could not parse version: %s\n", str);
flushAndExit(FDB_EXIT_ERROR);
}
return (double)CLIENT_KNOBS->CORE_VERSIONSPERSECOND * 24 * 3600 * -days;
}
Version ver;
fdbbackup: fix some print/scan format warnings Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-05 01:48:09 +08:00
if(sscanf(str, "%" SCNd64, &ver) != 1) {
Added fdbbackup dump option to list files and sizes in a backup for a given version range.
2019-01-03 18:35:31 +08:00
fprintf(stderr, "Could not parse version: %s\n", str);
flushAndExit(FDB_EXIT_ERROR);
}
return ver;
}
Initial repository commit
2017-05-26 04:48:44 +08:00
#ifdef ALLOC_INSTRUMENTATION
extern uint8_t *g_extra_memory;
#endif
int main(int argc, char* argv[]) {
platformInit();
fix: call initSignalSafeUnwind in fdbcli and backup to avoid hangs in slow task profiling.
2017-08-08 06:55:08 +08:00
initSignalSafeUnwind();
Initial repository commit
2017-05-26 04:48:44 +08:00
int status = FDB_EXIT_SUCCESS;
try {
#ifdef ALLOC_INSTRUMENTATION
g_extra_memory = new uint8_t[1000000];
#endif
registerCrashHandler();
// Set default of line buffering standard out and error
setvbuf(stdout, NULL, _IONBF, 0);
setvbuf(stderr, NULL, _IONBF, 0);
enumProgramExe programExe = getProgramType(argv[0]);
enumBackupType backupType = BACKUP_UNDEFINED;
enumRestoreType restoreType = RESTORE_UNKNOWN;
enumDBType dbType = DB_UNDEFINED;
CSimpleOpt* args = NULL;
switch (programExe)
{
case EXE_AGENT:
args = new CSimpleOpt(argc, argv, g_rgAgentOptions, SO_O_EXACT);
break;
case EXE_DR_AGENT:
args = new CSimpleOpt(argc, argv, g_rgDBAgentOptions, SO_O_EXACT);
break;
case EXE_BACKUP:
// Display backup help, if no arguments
if (argc < 2) {
printBackupUsage(false);
return FDB_EXIT_ERROR;
}
else {
// Get the backup type
backupType = getBackupType(argv[1]);
// Create the appropriate simple opt
switch (backupType)
{
case BACKUP_START:
args = new CSimpleOpt(argc-1, &argv[1], g_rgBackupStartOptions, SO_O_EXACT);
break;
case BACKUP_STATUS:
args = new CSimpleOpt(argc - 1, &argv[1], g_rgBackupStatusOptions, SO_O_EXACT);
break;
case BACKUP_ABORT:
args = new CSimpleOpt(argc - 1, &argv[1], g_rgBackupAbortOptions, SO_O_EXACT);
break;
case BACKUP_WAIT:
args = new CSimpleOpt(argc - 1, &argv[1], g_rgBackupWaitOptions, SO_O_EXACT);
break;
case BACKUP_DISCONTINUE:
args = new CSimpleOpt(argc - 1, &argv[1], g_rgBackupDiscontinueOptions, SO_O_EXACT);
break;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
case BACKUP_PAUSE:
args = new CSimpleOpt(argc - 1, &argv[1], g_rgBackupPauseOptions, SO_O_EXACT);
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
break;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
case BACKUP_RESUME:
args = new CSimpleOpt(argc - 1, &argv[1], g_rgBackupPauseOptions, SO_O_EXACT);
break;
case BACKUP_EXPIRE:
args = new CSimpleOpt(argc - 1, &argv[1], g_rgBackupExpireOptions, SO_O_EXACT);
break;
case BACKUP_DELETE:
args = new CSimpleOpt(argc - 1, &argv[1], g_rgBackupDeleteOptions, SO_O_EXACT);
break;
case BACKUP_DESCRIBE:
args = new CSimpleOpt(argc - 1, &argv[1], g_rgBackupDescribeOptions, SO_O_EXACT);
break;
Added fdbbackup dump option to list files and sizes in a backup for a given version range.
2019-01-03 18:35:31 +08:00
case BACKUP_DUMP:
args = new CSimpleOpt(argc - 1, &argv[1], g_rgBackupDumpOptions, SO_O_EXACT);
break;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
case BACKUP_LIST:
args = new CSimpleOpt(argc - 1, &argv[1], g_rgBackupListOptions, SO_O_EXACT);
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
break;
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
case BACKUP_MODIFY:
args = new CSimpleOpt(argc - 1, &argv[1], g_rgBackupModifyOptions, SO_O_EXACT);
break;
Initial repository commit
2017-05-26 04:48:44 +08:00
case BACKUP_UNDEFINED:
default:
// Display help, if requested
if ((strcmp(argv[1], "-h") == 0) ||
(strcmp(argv[1], "--help") == 0) )
{
printBackupUsage(false);
return FDB_EXIT_ERROR;
}
else {
fprintf(stderr, "ERROR: Unsupported backup action %s\n", argv[1]);
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
}
break;
}
}
break;
case EXE_DB_BACKUP:
// Display backup help, if no arguments
if (argc < 2) {
printDBBackupUsage(false);
return FDB_EXIT_ERROR;
}
else {
// Get the backup type
dbType = getDBType(argv[1]);
// Create the appropriate simple opt
switch (dbType)
{
case DB_START:
args = new CSimpleOpt(argc-1, &argv[1], g_rgDBStartOptions, SO_O_EXACT);
break;
case DB_STATUS:
args = new CSimpleOpt(argc - 1, &argv[1], g_rgDBStatusOptions, SO_O_EXACT);
break;
case DB_SWITCH:
args = new CSimpleOpt(argc - 1, &argv[1], g_rgDBSwitchOptions, SO_O_EXACT);
break;
case DB_ABORT:
args = new CSimpleOpt(argc - 1, &argv[1], g_rgDBAbortOptions, SO_O_EXACT);
break;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
case DB_PAUSE:
args = new CSimpleOpt(argc - 1, &argv[1], g_rgDBPauseOptions, SO_O_EXACT);
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
break;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
case DB_RESUME:
args = new CSimpleOpt(argc - 1, &argv[1], g_rgDBPauseOptions, SO_O_EXACT);
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
break;
Initial repository commit
2017-05-26 04:48:44 +08:00
case DB_UNDEFINED:
default:
// Display help, if requested
if ((strcmp(argv[1], "-h") == 0) ||
(strcmp(argv[1], "--help") == 0) )
{
printDBBackupUsage(false);
return FDB_EXIT_ERROR;
}
else {
fprintf(stderr, "ERROR: Unsupported dr action %s %d\n", argv[1], dbType);
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
}
break;
}
}
break;
case EXE_RESTORE:
if (argc < 2) {
printRestoreUsage(false);
return FDB_EXIT_ERROR;
}
// Get the restore operation type
restoreType = getRestoreType(argv[1]);
if(restoreType == RESTORE_UNKNOWN) {
// Display help, if requested
if ((strcmp(argv[1], "-h") == 0) ||
(strcmp(argv[1], "--help") == 0) )
{
printRestoreUsage(false);
return FDB_EXIT_ERROR;
}
else {
fprintf(stderr, "ERROR: Unsupported restore command: '%s'\n", argv[1]);
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
}
}
args = new CSimpleOpt(argc - 1, argv + 1, g_rgRestoreOptions, SO_O_EXACT);
break;
case EXE_UNDEFINED:
default:
fprintf(stderr, "FoundationDB " FDB_VT_PACKAGE_NAME " (v" FDB_VT_VERSION ")\n");
fprintf(stderr, "ERROR: Unable to determine program type based on executable `%s'\n", argv[0]);
return FDB_EXIT_ERROR;
break;
}
std::string destinationContainer;
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
bool describeDeep = false;
Added fdbbackup describe --version_timestamps option which is now required for describe to print version timestamps. Improved help...slightly.
2018-01-23 07:50:28 +08:00
bool describeTimestamps = false;
Added snapshot interval option to "fdbbackup start" which defaults to a new knob's value. Added snapshot info to backup status text. Improvements to fdbbackup help.
2017-12-20 16:49:08 +08:00
int snapshotIntervalSeconds = CLIENT_KNOBS->BACKUP_DEFAULT_SNAPSHOT_INTERVAL_SEC;
Initial repository commit
2017-05-26 04:48:44 +08:00
std::string clusterFile;
std::string sourceClusterFile;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
std::string baseUrl;
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
std::string expireDatetime;
Corrected function name to timeKeeperVersionFromDatetime(). 'Fdbbackup expire' now allows an expire_before version of 0 if explicitly passed by version or by timestamp.
2018-01-23 16:19:51 +08:00
Version expireVersion = invalidVersion;
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
std::string expireRestorableAfterDatetime;
Version expireRestorableAfterVersion = std::numeric_limits<Version>::max();
Initial repository commit
2017-05-26 04:48:44 +08:00
std::vector<std::pair<std::string, std::string>> knobs;
std::string tagName = BackupAgentBase::getDefaultTag().toString();
bool tagProvided = false;
std::string restoreContainer;
std::string addPrefix;
std::string removePrefix;
Standalone<VectorRef<KeyRangeRef>> backupKeys;
int maxErrors = 20;
Restore now requires an explicit destination cluster file. Restore target database version can now be specified by a timestamp, which must be resolved using the cluster file for the cluster from which the backup originated.
2019-03-06 04:28:05 +08:00
Version restoreVersion = invalidVersion;
std::string restoreTimestamp;
Initial repository commit
2017-05-26 04:48:44 +08:00
bool waitForDone = false;
bool stopWhenDone = true;
bool forceAction = false;
bool trace = false;
bool quietDisplay = false;
bool dryRun = false;
std::string traceDir = "";
Set TRACE_FORMAT network option in fdbbackup
2019-03-23 01:39:13 +08:00
std::string traceFormat = "";
Initial repository commit
2017-05-26 04:48:44 +08:00
std::string traceLogGroup;
Fdbbackup will now still write a trace file if --log is specified when doing operations that do not require a cluster. Previous behavior was to rely on createCluster()'s trace file initialization.
2018-02-08 14:36:01 +08:00
uint64_t traceRollSize = TRACE_DEFAULT_ROLL_SIZE;
uint64_t traceMaxLogsSize = TRACE_DEFAULT_MAX_LOGS_SIZE;
Initial repository commit
2017-05-26 04:48:44 +08:00
ESOError lastError;
bool partial = true;
Add locality to backup agent and DR agent
2017-09-27 01:19:26 +08:00
LocalityData localities;
added configurable memory limits for backup and dr executables added a default memory limit of 8GB for fdbcli
2017-09-30 01:35:40 +08:00
uint64_t memLimit = 8LL << 30;
Optional<uint64_t> ti;
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
std::vector<std::string> blobCredentials;
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
std::string tlsCertPath, tlsKeyPath, tlsCAPath, tlsPassword, tlsVerifyPeers;
Added fdbbackup dump option to list files and sizes in a backup for a given version range.
2019-01-03 18:35:31 +08:00
Version dumpBegin = 0;
Version dumpEnd = std::numeric_limits<Version>::max();
Restore now requires an explicit destination cluster file. Restore target database version can now be specified by a timestamp, which must be resolved using the cluster file for the cluster from which the backup originated.
2019-03-06 04:28:05 +08:00
std::string restoreClusterFileDest;
std::string restoreClusterFileOrig;
Added --json option to fdbbackup describe. Also added expired percentage indicator to snapshot details.
2019-03-07 06:14:06 +08:00
bool jsonOutput = false;
Initial repository commit
2017-05-26 04:48:44 +08:00
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
BackupModifyOptions modifyOptions;
Initial repository commit
2017-05-26 04:48:44 +08:00
if( argc == 1 ) {
printUsage(programExe, false);
Merge branch 'release-6.0' # Conflicts: # documentation/sphinx/source/release-notes.rst # fdbbackup/backup.actor.cpp # fdbclient/BackupContainer.actor.cpp # fdbclient/HTTP.actor.cpp # fdbserver/storageserver.actor.cpp # fdbserver/workloads/BackupCorrectness.actor.cpp # versions.target
2019-01-10 08:14:46 +08:00
return FDB_EXIT_ERROR;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
#ifdef _WIN32
// Windows needs a gentle nudge to format floats correctly
//_set_output_format(_TWO_DIGIT_EXPONENT);
#endif
while (args->Next()) {
lastError = args->LastError();
switch (lastError)
{
case SO_SUCCESS:
break;
case SO_ARG_INVALID_DATA:
fprintf(stderr, "ERROR: invalid argument to option `%s'\n", args->OptionText());
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
break;
case SO_ARG_INVALID:
fprintf(stderr, "ERROR: argument given for option `%s'\n", args->OptionText());
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
break;
case SO_ARG_MISSING:
fprintf(stderr, "ERROR: missing argument for option `%s'\n", args->OptionText());
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
case SO_OPT_INVALID:
fprintf(stderr, "ERROR: unknown option `%s'\n", args->OptionText());
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
break;
default:
fprintf(stderr, "ERROR: argument given for option `%s'\n", args->OptionText());
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
break;
}
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
int optId = args->OptionId();
switch (optId) {
Initial repository commit
2017-05-26 04:48:44 +08:00
case OPT_HELP:
printUsage(programExe, false);
return FDB_EXIT_SUCCESS;
break;
case OPT_DEVHELP:
printUsage(programExe, true);
return FDB_EXIT_SUCCESS;
break;
case OPT_VERSION:
printVersion();
return FDB_EXIT_SUCCESS;
break;
case OPT_NOBUFSTDOUT:
setvbuf(stdout, NULL, _IONBF, 0);
setvbuf(stderr, NULL, _IONBF, 0);
break;
case OPT_BUFSTDOUTERR:
setvbuf(stdout, NULL, _IOFBF, BUFSIZ);
setvbuf(stderr, NULL, _IOFBF, BUFSIZ);
break;
case OPT_QUIET:
quietDisplay = true;
break;
case OPT_DRYRUN:
dryRun = true;
break;
case OPT_FORCE:
forceAction = true;
break;
case OPT_TRACE:
trace = true;
break;
Fix whitespace
2019-01-03 07:24:11 +08:00
case OPT_TRACE_DIR:
Initial repository commit
2017-05-26 04:48:44 +08:00
trace = true;
traceDir = args->OptionArg();
break;
Fix whitespace
2019-01-03 07:24:11 +08:00
case OPT_TRACE_FORMAT:
Set TRACE_FORMAT network option in fdbbackup
2019-03-23 01:39:13 +08:00
if (!validateTraceFormat(args->OptionArg())) {
Fix whitespace
2019-01-04 01:12:03 +08:00
fprintf(stderr, "WARNING: Unrecognized trace format `%s'\n", args->OptionArg());
}
Set TRACE_FORMAT network option in fdbbackup
2019-03-23 01:39:13 +08:00
traceFormat = args->OptionArg();
Fix whitespace
2019-01-04 01:12:03 +08:00
break;
Initial repository commit
2017-05-26 04:48:44 +08:00
case OPT_TRACE_LOG_GROUP:
traceLogGroup = args->OptionArg();
break;
Add locality to backup agent and DR agent
2017-09-27 01:19:26 +08:00
case OPT_LOCALITY: {
std::string syn = args->OptionSyntax();
if (!StringRef(syn).startsWith(LiteralStringRef("--locality_"))) {
fprintf(stderr, "ERROR: unable to parse locality key '%s'\n", syn.c_str());
return FDB_EXIT_ERROR;
}
syn = syn.substr(11);
std::transform(syn.begin(), syn.end(), syn.begin(), ::tolower);
localities.set(Standalone<StringRef>(syn), Standalone<StringRef>(std::string(args->OptionArg())));
break;
}
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
case OPT_EXPIRE_BEFORE_DATETIME:
expireDatetime = args->OptionArg();
break;
case OPT_EXPIRE_RESTORABLE_AFTER_DATETIME:
expireRestorableAfterDatetime = args->OptionArg();
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
break;
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
case OPT_EXPIRE_BEFORE_VERSION:
case OPT_EXPIRE_RESTORABLE_AFTER_VERSION:
Added fdbbackup expire options to calculate approximate version boundaries based on a number of days prior to the latest log file found in the backup container. This enables expiration operations based on time (with reasonable precision) without accessing the source cluster.
2018-12-19 10:55:44 +08:00
case OPT_EXPIRE_MIN_RESTORABLE_DAYS:
case OPT_EXPIRE_DELETE_BEFORE_DAYS:
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
{
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
const char* a = args->OptionArg();
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
long long ver = 0;
if (!sscanf(a, "%lld", &ver)) {
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
fprintf(stderr, "ERROR: Could not parse expiration version `%s'\n", a);
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
}
Added fdbbackup expire options to calculate approximate version boundaries based on a number of days prior to the latest log file found in the backup container. This enables expiration operations based on time (with reasonable precision) without accessing the source cluster.
2018-12-19 10:55:44 +08:00
// Interpret the value as days worth of versions relative to now (negative)
if(optId == OPT_EXPIRE_MIN_RESTORABLE_DAYS || optId == OPT_EXPIRE_DELETE_BEFORE_DAYS) {
ver = -ver * 24 * 60 * 60 * CLIENT_KNOBS->CORE_VERSIONSPERSECOND;
}
if(optId == OPT_EXPIRE_BEFORE_VERSION || optId == OPT_EXPIRE_DELETE_BEFORE_DAYS)
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
expireVersion = ver;
else
expireRestorableAfterVersion = ver;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
break;
}
Restore now requires an explicit destination cluster file. Restore target database version can now be specified by a timestamp, which must be resolved using the cluster file for the cluster from which the backup originated.
2019-03-06 04:28:05 +08:00
case OPT_RESTORE_TIMESTAMP:
restoreTimestamp = args->OptionArg();
break;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
case OPT_BASEURL:
baseUrl = args->OptionArg();
break;
Restore now requires an explicit destination cluster file. Restore target database version can now be specified by a timestamp, which must be resolved using the cluster file for the cluster from which the backup originated.
2019-03-06 04:28:05 +08:00
case OPT_RESTORE_CLUSTERFILE_DEST:
restoreClusterFileDest = args->OptionArg();
break;
case OPT_RESTORE_CLUSTERFILE_ORIG:
restoreClusterFileOrig = args->OptionArg();
break;
Initial repository commit
2017-05-26 04:48:44 +08:00
case OPT_CLUSTERFILE:
clusterFile = args->OptionArg();
break;
case OPT_DEST_CLUSTER:
clusterFile = args->OptionArg();
break;
case OPT_SOURCE_CLUSTER:
sourceClusterFile = args->OptionArg();
break;
case OPT_CLEANUP:
partial = false;
break;
case OPT_KNOB: {
std::string syn = args->OptionSyntax();
if (!StringRef(syn).startsWith(LiteralStringRef("--knob_"))) {
fprintf(stderr, "ERROR: unable to parse knob option '%s'\n", syn.c_str());
return FDB_EXIT_ERROR;
}
syn = syn.substr(7);
knobs.push_back( std::make_pair( syn, args->OptionArg() ) );
break;
}
case OPT_BACKUPKEYS:
try {
addKeyRange(args->OptionArg(), backupKeys);
}
catch (Error &) {
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
}
break;
case OPT_DESTCONTAINER:
destinationContainer = args->OptionArg();
// If the url starts with '/' then prepend "file://" for backwards compatibility
if(StringRef(destinationContainer).startsWith(LiteralStringRef("/")))
destinationContainer = std::string("file://") + destinationContainer;
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
modifyOptions.destURL = destinationContainer;
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
case OPT_SNAPSHOTINTERVAL:
case OPT_MOD_ACTIVE_INTERVAL:
{
Added snapshot interval option to "fdbbackup start" which defaults to a new knob's value. Added snapshot info to backup status text. Improvements to fdbbackup help.
2017-12-20 16:49:08 +08:00
const char* a = args->OptionArg();
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
int seconds;
if (!sscanf(a, "%d", &seconds)) {
Added snapshot interval option to "fdbbackup start" which defaults to a new knob's value. Added snapshot info to backup status text. Improvements to fdbbackup help.
2017-12-20 16:49:08 +08:00
fprintf(stderr, "ERROR: Could not parse snapshot interval `%s'\n", a);
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
}
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
if(optId == OPT_SNAPSHOTINTERVAL) {
snapshotIntervalSeconds = seconds;
Bug fixes in fdbbackup modify command. Active snapshot interval was wrong. New backup destination containers were not being created. Renamed some variables for clarity.
2019-03-06 10:05:52 +08:00
modifyOptions.snapshotIntervalSeconds = seconds;
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
}
else if(optId == OPT_MOD_ACTIVE_INTERVAL) {
Bug fixes in fdbbackup modify command. Active snapshot interval was wrong. New backup destination containers were not being created. Renamed some variables for clarity.
2019-03-06 10:05:52 +08:00
modifyOptions.activeSnapshotIntervalSeconds = seconds;
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
}
Added snapshot interval option to "fdbbackup start" which defaults to a new knob's value. Added snapshot info to backup status text. Improvements to fdbbackup help.
2017-12-20 16:49:08 +08:00
break;
}
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
case OPT_MOD_VERIFY_UID:
modifyOptions.verifyUID = args->OptionArg();
break;
Initial repository commit
2017-05-26 04:48:44 +08:00
case OPT_WAITFORDONE:
waitForDone = true;
break;
case OPT_NOSTOPWHENDONE:
stopWhenDone = false;
break;
case OPT_RESTORECONTAINER:
restoreContainer = args->OptionArg();
// If the url starts with '/' then prepend "file://" for backwards compatibility
if(StringRef(restoreContainer).startsWith(LiteralStringRef("/")))
restoreContainer = std::string("file://") + restoreContainer;
break;
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
case OPT_DESCRIBE_DEEP:
describeDeep = true;
break;
Added fdbbackup describe --version_timestamps option which is now required for describe to print version timestamps. Improved help...slightly.
2018-01-23 07:50:28 +08:00
case OPT_DESCRIBE_TIMESTAMPS:
describeTimestamps = true;
break;
Initial repository commit
2017-05-26 04:48:44 +08:00
case OPT_PREFIX_ADD:
addPrefix = args->OptionArg();
break;
case OPT_PREFIX_REMOVE:
removePrefix = args->OptionArg();
break;
case OPT_ERRORLIMIT: {
const char* a = args->OptionArg();
if (!sscanf(a, "%d", &maxErrors)) {
fprintf(stderr, "ERROR: Could not parse max number of errors `%s'\n", a);
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
}
break;
}
Restore now requires an explicit destination cluster file. Restore target database version can now be specified by a timestamp, which must be resolved using the cluster file for the cluster from which the backup originated.
2019-03-06 04:28:05 +08:00
case OPT_RESTORE_VERSION: {
Initial repository commit
2017-05-26 04:48:44 +08:00
const char* a = args->OptionArg();
Restore now requires an explicit destination cluster file. Restore target database version can now be specified by a timestamp, which must be resolved using the cluster file for the cluster from which the backup originated.
2019-03-06 04:28:05 +08:00
long long ver = 0;
if (!sscanf(a, "%lld", &ver)) {
Initial repository commit
2017-05-26 04:48:44 +08:00
fprintf(stderr, "ERROR: Could not parse database version `%s'\n", a);
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
}
Restore now requires an explicit destination cluster file. Restore target database version can now be specified by a timestamp, which must be resolved using the cluster file for the cluster from which the backup originated.
2019-03-06 04:28:05 +08:00
restoreVersion = ver;
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
}
#ifdef _WIN32
case OPT_PARENTPID: {
auto pid_str = args->OptionArg();
int parent_pid = atoi(pid_str);
auto pHandle = OpenProcess( SYNCHRONIZE, FALSE, parent_pid );
if( !pHandle ) {
TraceEvent("ParentProcessOpenError").GetLastError();
fprintf(stderr, "Could not open parent process at pid %d (error %d)", parent_pid, GetLastError());
throw platform_error();
}
startThread(&parentWatcher, pHandle);
break;
}
#endif
case OPT_TAGNAME:
tagName = args->OptionArg();
tagProvided = true;
break;
case OPT_CRASHONERROR:
g_crashOnError = true;
break;
added configurable memory limits for backup and dr executables added a default memory limit of 8GB for fdbcli
2017-09-30 01:35:40 +08:00
case OPT_MEMLIMIT:
ti = parse_with_suffix(args->OptionArg(), "MiB");
if (!ti.present()) {
fprintf(stderr, "ERROR: Could not parse memory limit from `%s'\n", args->OptionArg());
printHelpTeaser(argv[0]);
flushAndExit(FDB_EXIT_ERROR);
}
memLimit = ti.get();
break;
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
case OPT_BLOB_CREDENTIALS:
blobCredentials.push_back(args->OptionArg());
break;
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
#ifndef TLS_DISABLED
case TLSOptions::OPT_TLS_PLUGIN:
args->OptionArg();
break;
case TLSOptions::OPT_TLS_CERTIFICATES:
tlsCertPath = args->OptionArg();
break;
case TLSOptions::OPT_TLS_PASSWORD:
tlsPassword = args->OptionArg();
break;
case TLSOptions::OPT_TLS_CA_FILE:
tlsCAPath = args->OptionArg();
break;
case TLSOptions::OPT_TLS_KEY:
tlsKeyPath = args->OptionArg();
break;
case TLSOptions::OPT_TLS_VERIFY_PEERS:
tlsVerifyPeers = args->OptionArg();
break;
#endif
Added fdbbackup dump option to list files and sizes in a backup for a given version range.
2019-01-03 18:35:31 +08:00
case OPT_DUMP_BEGIN:
dumpBegin = parseVersion(args->OptionArg());
break;
case OPT_DUMP_END:
dumpEnd = parseVersion(args->OptionArg());
break;
Added --json option to fdbbackup describe. Also added expired percentage indicator to snapshot details.
2019-03-07 06:14:06 +08:00
case OPT_JSON:
jsonOutput = true;
break;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
// Process the extra arguments
for (int argLoop = 0; argLoop < args->FileCount(); argLoop++)
{
switch (programExe)
{
case EXE_AGENT:
fprintf(stderr, "ERROR: Backup Agent does not support argument value `%s'\n", args->File(argLoop));
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
break;
// Add the backup key range
case EXE_BACKUP:
// Error, if the keys option was not specified
if (backupKeys.size() == 0) {
fprintf(stderr, "ERROR: Unknown backup option value `%s'\n", args->File(argLoop));
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
}
// Otherwise, assume the item is a key range
else {
try {
addKeyRange(args->File(argLoop), backupKeys);
}
Fix compiler warning about unreferenced exception variable
2019-03-27 00:58:54 +08:00
catch (Error& ) {
Initial repository commit
2017-05-26 04:48:44 +08:00
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
}
}
break;
case EXE_RESTORE:
fprintf(stderr, "ERROR: FDB Restore does not support argument value `%s'\n", args->File(argLoop));
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
break;
case EXE_DR_AGENT:
fprintf(stderr, "ERROR: DR Agent does not support argument value `%s'\n", args->File(argLoop));
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
break;
case EXE_DB_BACKUP:
// Error, if the keys option was not specified
if (backupKeys.size() == 0) {
fprintf(stderr, "ERROR: Unknown DR option value `%s'\n", args->File(argLoop));
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
}
// Otherwise, assume the item is a key range
else {
try {
addKeyRange(args->File(argLoop), backupKeys);
}
Fix compiler warning about unreferenced exception variable
2019-03-27 00:58:54 +08:00
catch (Error& ) {
Initial repository commit
2017-05-26 04:48:44 +08:00
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
}
}
break;
case EXE_UNDEFINED:
default:
return FDB_EXIT_ERROR;
}
}
// Delete the simple option object, if defined
if (args)
{
delete args;
args = NULL;
}
std::string commandLine;
for(int a=0; a<argc; a++) {
if (a) commandLine += ' ';
commandLine += argv[a];
}
Allow setting flow knobs in backup.
2018-08-23 01:38:07 +08:00
delete FLOW_KNOBS;
FlowKnobs* flowKnobs = new FlowKnobs(true);
FLOW_KNOBS = flowKnobs;
Initial repository commit
2017-05-26 04:48:44 +08:00
delete CLIENT_KNOBS;
ClientKnobs* clientKnobs = new ClientKnobs(true);
CLIENT_KNOBS = clientKnobs;
for(auto k=knobs.begin(); k!=knobs.end(); ++k) {
try {
Use the local flowKnobs and clientKnobs rather than const_casting the global ones
2018-08-25 05:16:57 +08:00
if (!flowKnobs->setKnob( k->first, k->second ) &&
Merge branch 'release-6.0' # Conflicts: # documentation/sphinx/source/release-notes.rst # fdbbackup/backup.actor.cpp # fdbclient/BackupContainer.actor.cpp # fdbclient/HTTP.actor.cpp # fdbserver/storageserver.actor.cpp # fdbserver/workloads/BackupCorrectness.actor.cpp # versions.target
2019-01-10 08:14:46 +08:00
!clientKnobs->setKnob( k->first, k->second ))
Allow setting flow knobs in backup.
2018-08-23 01:38:07 +08:00
{
Initial repository commit
2017-05-26 04:48:44 +08:00
fprintf(stderr, "Unrecognized knob option '%s'\n", k->first.c_str());
return FDB_EXIT_ERROR;
}
} catch (Error& e) {
if (e.code() == error_code_invalid_option_value) {
fprintf(stderr, "Invalid value '%s' for option '%s'\n", k->second.c_str(), k->first.c_str());
return FDB_EXIT_ERROR;
}
throw;
}
}
if (trace) {
if(!traceLogGroup.empty())
setNetworkOption(FDBNetworkOptions::TRACE_LOG_GROUP, StringRef(traceLogGroup));
if (traceDir.empty())
setNetworkOption(FDBNetworkOptions::TRACE_ENABLE);
else
setNetworkOption(FDBNetworkOptions::TRACE_ENABLE, StringRef(traceDir));
Set TRACE_FORMAT network option in fdbbackup
2019-03-23 01:39:13 +08:00
if (!traceFormat.empty()) {
setNetworkOption(FDBNetworkOptions::TRACE_FORMAT, StringRef(traceFormat));
}
Initial repository commit
2017-05-26 04:48:44 +08:00
setNetworkOption(FDBNetworkOptions::ENABLE_SLOW_TASK_PROFILING);
}
setNetworkOption(FDBNetworkOptions::DISABLE_CLIENT_STATISTICS_LOGGING);
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
// deferred TLS options
if (tlsCertPath.size()) {
try {
setNetworkOption(FDBNetworkOptions::TLS_CERT_PATH, tlsCertPath);
}
catch (Error& e) {
fprintf(stderr, "ERROR: cannot set TLS certificate path to `%s' (%s)\n", tlsCertPath.c_str(), e.what());
return 1;
}
}
if (tlsCAPath.size()) {
try {
setNetworkOption(FDBNetworkOptions::TLS_CA_PATH, tlsCAPath);
}
catch (Error& e) {
fprintf(stderr, "ERROR: cannot set TLS CA path to `%s' (%s)\n", tlsCAPath.c_str(), e.what());
return 1;
Corrected spacing issue.
2019-02-07 03:27:26 +08:00
}
Corrected spacing.
2019-02-07 03:35:50 +08:00
}
Support TLS command line options in backup
2019-01-31 09:36:51 +08:00
if (tlsKeyPath.size()) {
try {
if (tlsPassword.size())
setNetworkOption(FDBNetworkOptions::TLS_PASSWORD, tlsPassword);
setNetworkOption(FDBNetworkOptions::TLS_KEY_PATH, tlsKeyPath);
}
catch (Error& e) {
fprintf(stderr, "ERROR: cannot set TLS key path to `%s' (%s)\n", tlsKeyPath.c_str(), e.what());
return 1;
}
}
if (tlsVerifyPeers.size()) {
try {
setNetworkOption(FDBNetworkOptions::TLS_VERIFY_PEERS, tlsVerifyPeers);
}
catch (Error& e) {
fprintf(stderr, "ERROR: cannot set TLS peer verification to `%s' (%s)\n", tlsVerifyPeers.c_str(), e.what());
return 1;
}
}
Initial repository commit
2017-05-26 04:48:44 +08:00
Error::init();
std::set_new_handler( &platform::outOfMemory );
added configurable memory limits for backup and dr executables added a default memory limit of 8GB for fdbcli
2017-09-30 01:35:40 +08:00
setMemoryQuota( memLimit );
Initial repository commit
2017-05-26 04:48:44 +08:00
int total = 0;
for(auto i = Error::errorCounts().begin(); i != Error::errorCounts().end(); ++i)
total += i->second;
if (total)
printf("%d errors:\n", total);
for(auto i = Error::errorCounts().begin(); i != Error::errorCounts().end(); ++i)
if (i->second > 0)
printf(" %d: %d %s\n", i->second, i->first, Error::fromCode(i->first).what());
Reference<ClusterConnectionFile> ccf;
Database db;
This removes the idea of clusters from IClientApi.
2018-09-22 06:58:14 +08:00
Reference<ClusterConnectionFile> sourceCcf;
Database sourceDb;
Initial repository commit
2017-05-26 04:48:44 +08:00
FileBackupAgent ba;
Key tag;
Future<Optional<Void>> f;
Future<Optional<int>> fstatus;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
Reference<IBackupContainer> c;
Initial repository commit
2017-05-26 04:48:44 +08:00
try {
setupNetwork(0, true);
}
catch (Error& e) {
fprintf(stderr, "ERROR: %s\n", e.what());
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
return FDB_EXIT_ERROR;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
This removes the idea of clusters from IClientApi.
2018-09-22 06:58:14 +08:00
TraceEvent("ProgramStart")
.detail("SourceVersion", getHGVersion())
.detail("Version", FDB_VT_VERSION )
.detail("PackageName", FDB_VT_PACKAGE_NAME)
.detailf("ActualTime", "%lld", DEBUG_DETERMINISM ? 0 : time(NULL))
.detail("CommandLine", commandLine)
.detail("MemoryLimit", memLimit)
.trackLatest("ProgramStart");
Initial repository commit
2017-05-26 04:48:44 +08:00
// Ordinarily, this is done when the network is run. However, network thread should be set before TraceEvents are logged. This thread will eventually run the network, so call it now.
Merge branch 'release-6.0' # Conflicts: # documentation/sphinx/source/release-notes.rst # fdbbackup/backup.actor.cpp # fdbclient/BackupContainer.actor.cpp # fdbclient/HTTP.actor.cpp # fdbserver/storageserver.actor.cpp # fdbserver/workloads/BackupCorrectness.actor.cpp # versions.target
2019-01-10 08:14:46 +08:00
TraceEvent::setNetworkThread();
Initial repository commit
2017-05-26 04:48:44 +08:00
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
// Add blob credentials files from the environment to the list collected from the command line.
const char *blobCredsFromENV = getenv("FDB_BLOB_CREDENTIALS");
if(blobCredsFromENV != nullptr) {
StringRef t((uint8_t*)blobCredsFromENV, strlen(blobCredsFromENV));
do {
StringRef file = t.eat(":");
if(file.size() != 0)
blobCredentials.push_back(file.toString());
} while(t.size() != 0);
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
}
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
// Update the global blob credential files list
std::vector<std::string> *pFiles = (std::vector<std::string> *)g_network->global(INetwork::enBlobCredentialFiles);
if(pFiles != nullptr) {
for(auto &f : blobCredentials) {
pFiles->push_back(f);
}
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
}
Fdbbackup will now still write a trace file if --log is specified when doing operations that do not require a cluster. Previous behavior was to rely on createCluster()'s trace file initialization.
2018-02-08 14:36:01 +08:00
// Opens a trace file if trace is set (and if a trace file isn't already open)
// For most modes, initCluster() will open a trace file, but some fdbbackup operations do not require
// a cluster so they should use this instead.
auto initTraceFile = [&]() {
if(trace)
openTraceFile(NetworkAddress(), traceRollSize, traceMaxLogsSize, traceDir, "trace", traceLogGroup);
};
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
auto initCluster = [&](bool quiet = false) {
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
auto resolvedClusterFile = ClusterConnectionFile::lookupClusterFileName(clusterFile);
try {
ccf = Reference<ClusterConnectionFile>(new ClusterConnectionFile(resolvedClusterFile.first));
}
catch (Error& e) {
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
if(!quiet)
fprintf(stderr, "%s\n", ClusterConnectionFile::getErrorString(resolvedClusterFile, e).c_str());
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
return false;
}
try {
This removes the idea of clusters from IClientApi.
2018-09-22 06:58:14 +08:00
db = Database::createDatabase(ccf, -1, localities);
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
}
catch (Error& e) {
fprintf(stderr, "ERROR: %s\n", e.what());
fprintf(stderr, "ERROR: Unable to connect to cluster from `%s'\n", ccf->getFilename().c_str());
return false;
}
return true;
};
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
if(sourceClusterFile.size()) {
auto resolvedSourceClusterFile = ClusterConnectionFile::lookupClusterFileName(sourceClusterFile);
Initial repository commit
2017-05-26 04:48:44 +08:00
try {
This removes the idea of clusters from IClientApi.
2018-09-22 06:58:14 +08:00
sourceCcf = Reference<ClusterConnectionFile>(new ClusterConnectionFile(resolvedSourceClusterFile.first));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
catch (Error& e) {
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
fprintf(stderr, "%s\n", ClusterConnectionFile::getErrorString(resolvedSourceClusterFile, e).c_str());
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
return FDB_EXIT_ERROR;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
try {
Add some basic checks before doing an atomic switchover.
2019-03-02 06:49:04 +08:00
sourceDb = Database::createDatabase(sourceCcf, -1, localities);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
catch (Error& e) {
fprintf(stderr, "ERROR: %s\n", e.what());
This removes the idea of clusters from IClientApi.
2018-09-22 06:58:14 +08:00
fprintf(stderr, "ERROR: Unable to connect to cluster from `%s'\n", sourceCcf->getFilename().c_str());
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
return FDB_EXIT_ERROR;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
switch (programExe)
{
case EXE_AGENT:
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
if(!initCluster())
return FDB_EXIT_ERROR;
Initial repository commit
2017-05-26 04:48:44 +08:00
f = stopAfter(runAgent(db));
break;
case EXE_BACKUP:
switch (backupType)
{
case BACKUP_START:
{
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
if(!initCluster())
return FDB_EXIT_ERROR;
Initial repository commit
2017-05-26 04:48:44 +08:00
// Test out the backup url to make sure it parses. Doesn't test to make sure it's actually writeable.
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
openBackupContainer(argv[0], destinationContainer);
Added snapshot interval option to "fdbbackup start" which defaults to a new knob's value. Added snapshot info to backup status text. Improvements to fdbbackup help.
2017-12-20 16:49:08 +08:00
f = stopAfter( submitBackup(db, destinationContainer, snapshotIntervalSeconds, backupKeys, tagName, dryRun, waitForDone, stopWhenDone) );
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
}
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 20:00:11 +08:00
case BACKUP_MODIFY:
{
if(!initCluster())
return FDB_EXIT_ERROR;
f = stopAfter( modifyBackup(db, tagName, modifyOptions) );
break;
}
Initial repository commit
2017-05-26 04:48:44 +08:00
case BACKUP_STATUS:
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
if(!initCluster())
return FDB_EXIT_ERROR;
Added --json option to fdbbackup status.
2019-03-07 13:32:46 +08:00
f = stopAfter( statusBackup(db, tagName, true, jsonOutput) );
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
case BACKUP_ABORT:
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
if(!initCluster())
return FDB_EXIT_ERROR;
Initial repository commit
2017-05-26 04:48:44 +08:00
f = stopAfter( abortBackup(db, tagName) );
break;
case BACKUP_WAIT:
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
if(!initCluster())
return FDB_EXIT_ERROR;
Initial repository commit
2017-05-26 04:48:44 +08:00
f = stopAfter( waitBackup(db, tagName, stopWhenDone) );
break;
case BACKUP_DISCONTINUE:
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
if(!initCluster())
return FDB_EXIT_ERROR;
Initial repository commit
2017-05-26 04:48:44 +08:00
f = stopAfter( discontinueBackup(db, tagName, waitForDone) );
break;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
case BACKUP_PAUSE:
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
if(!initCluster())
return FDB_EXIT_ERROR;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
f = stopAfter( changeBackupResumed(db, true) );
break;
case BACKUP_RESUME:
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
if(!initCluster())
return FDB_EXIT_ERROR;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
f = stopAfter( changeBackupResumed(db, false) );
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
break;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
case BACKUP_EXPIRE:
Fdbbackup will now still write a trace file if --log is specified when doing operations that do not require a cluster. Previous behavior was to rely on createCluster()'s trace file initialization.
2018-02-08 14:36:01 +08:00
initTraceFile();
Fdbbackup restore will now default to the latest usable version in a backup if a specific restore version was not given. Expire will make sure a cluster is provided if either of its timestamp options are given.
2018-01-20 04:19:32 +08:00
// Must have a usable cluster if either expire DateTime options were used
Added fdbbackup describe --version_timestamps option which is now required for describe to print version timestamps. Improved help...slightly.
2018-01-23 07:50:28 +08:00
if(!expireDatetime.empty() || !expireRestorableAfterDatetime.empty()) {
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
if(!initCluster())
return FDB_EXIT_ERROR;
Added fdbbackup describe --version_timestamps option which is now required for describe to print version timestamps. Improved help...slightly.
2018-01-23 07:50:28 +08:00
}
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 20:09:43 +08:00
f = stopAfter( expireBackupData(argv[0], destinationContainer, expireVersion, expireDatetime, db, forceAction, expireRestorableAfterVersion, expireRestorableAfterDatetime) );
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
break;
case BACKUP_DELETE:
Fdbbackup will now still write a trace file if --log is specified when doing operations that do not require a cluster. Previous behavior was to rely on createCluster()'s trace file initialization.
2018-02-08 14:36:01 +08:00
initTraceFile();
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
f = stopAfter( deleteBackupContainer(argv[0], destinationContainer) );
break;
case BACKUP_DESCRIBE:
Fdbbackup will now still write a trace file if --log is specified when doing operations that do not require a cluster. Previous behavior was to rely on createCluster()'s trace file initialization.
2018-02-08 14:36:01 +08:00
initTraceFile();
Added fdbbackup describe --version_timestamps option which is now required for describe to print version timestamps. Improved help...slightly.
2018-01-23 07:50:28 +08:00
// If timestamp lookups are desired, require a cluster file
if(describeTimestamps && !initCluster())
return FDB_EXIT_ERROR;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
Added fdbbackup describe --version_timestamps option which is now required for describe to print version timestamps. Improved help...slightly.
2018-01-23 07:50:28 +08:00
// Only pass database optionDatabase Describe will lookup version timestamps if a cluster file was given, but quietly skip them if not.
Added --json option to fdbbackup describe. Also added expired percentage indicator to snapshot details.
2019-03-07 06:14:06 +08:00
f = stopAfter( describeBackup(argv[0], destinationContainer, describeDeep, describeTimestamps ? Optional<Database>(db) : Optional<Database>(), jsonOutput) );
Added fdbbackup describe --version_timestamps option which is now required for describe to print version timestamps. Improved help...slightly.
2018-01-23 07:50:28 +08:00
break;
Added fdbbackup dump option to list files and sizes in a backup for a given version range.
2019-01-03 18:35:31 +08:00
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
case BACKUP_LIST:
Fdbbackup will now still write a trace file if --log is specified when doing operations that do not require a cluster. Previous behavior was to rely on createCluster()'s trace file initialization.
2018-02-08 14:36:01 +08:00
initTraceFile();
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
f = stopAfter( listBackup(baseUrl) );
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
break;
Added fdbbackup dump option to list files and sizes in a backup for a given version range.
2019-01-03 18:35:31 +08:00
case BACKUP_DUMP:
initTraceFile();
f = stopAfter( dumpBackupData(argv[0], destinationContainer, dumpBegin, dumpEnd) );
break;
Initial repository commit
2017-05-26 04:48:44 +08:00
case BACKUP_UNDEFINED:
default:
fprintf(stderr, "ERROR: Unsupported backup action %s\n", argv[1]);
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
break;
}
break;
case EXE_RESTORE:
Bug fix, restore start dry run would not open a trace file.
2019-01-09 19:54:13 +08:00
if(dryRun) {
initTraceFile();
}
Restore now requires an explicit destination cluster file. Restore target database version can now be specified by a timestamp, which must be resolved using the cluster file for the cluster from which the backup originated.
2019-03-06 04:28:05 +08:00
else if(restoreType != RESTORE_START && !initCluster()) {
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
return FDB_EXIT_ERROR;
Bug fix, restore start dry run would not open a trace file.
2019-01-09 19:54:13 +08:00
}
Bug fix: fdbrestore abort, wait, and status were not using the --dest_cluster_file argument.
2019-06-14 08:42:11 +08:00
if(restoreClusterFileDest.empty()) {
fprintf(stderr, "Restore destination cluster file must be specified explicitly.\n");
return FDB_EXIT_ERROR;
}
if(!fileExists(restoreClusterFileDest)) {
fprintf(stderr, "Restore destination cluster file '%s' does not exist.\n", restoreClusterFileDest.c_str());
return FDB_EXIT_ERROR;
}
try {
db = Database::createDatabase(restoreClusterFileDest, Database::API_VERSION_LATEST);
} catch(Error &e) {
fprintf(stderr, "Restore destination cluster file '%s' invalid: %s\n", restoreClusterFileDest.c_str(), e.what());
return FDB_EXIT_ERROR;
}
Initial repository commit
2017-05-26 04:48:44 +08:00
switch(restoreType) {
case RESTORE_START:
Bug fix: fdbrestore abort, wait, and status were not using the --dest_cluster_file argument.
2019-06-14 08:42:11 +08:00
f = stopAfter( runRestore(db, restoreClusterFileOrig, tagName, restoreContainer, backupKeys, restoreVersion, restoreTimestamp, !dryRun, !quietDisplay, waitForDone, addPrefix, removePrefix) );
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
case RESTORE_WAIT:
f = stopAfter( success(ba.waitRestore(db, KeyRef(tagName), true)) );
break;
case RESTORE_ABORT:
f = stopAfter( map(ba.abortRestore(db, KeyRef(tagName)), [tagName](FileBackupAgent::ERestoreState s) -> Void {
printf("Tag: %s State: %s\n", tagName.c_str(), FileBackupAgent::restoreStateText(s).toString().c_str());
return Void();
}) );
break;
case RESTORE_STATUS:
// If no tag is specifically provided then print all tag status, don't just use "default"
if(tagProvided)
tag = tagName;
f = stopAfter( map(ba.restoreStatus(db, KeyRef(tag)), [](std::string s) -> Void {
printf("%s\n", s.c_str());
return Void();
}) );
break;
default:
throw restore_error();
}
break;
case EXE_DR_AGENT:
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
if(!initCluster())
return FDB_EXIT_ERROR;
This removes the idea of clusters from IClientApi.
2018-09-22 06:58:14 +08:00
f = stopAfter( runDBAgent(sourceDb, db) );
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
case EXE_DB_BACKUP:
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 17:58:15 +08:00
if(!initCluster())
return FDB_EXIT_ERROR;
Initial repository commit
2017-05-26 04:48:44 +08:00
switch (dbType)
{
case DB_START:
This removes the idea of clusters from IClientApi.
2018-09-22 06:58:14 +08:00
f = stopAfter( submitDBBackup(sourceDb, db, backupKeys, tagName) );
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
case DB_STATUS:
This removes the idea of clusters from IClientApi.
2018-09-22 06:58:14 +08:00
f = stopAfter( statusDBBackup(sourceDb, db, tagName, maxErrors) );
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
case DB_SWITCH:
Skip switchover checks for force option.
2019-03-05 07:58:36 +08:00
f = stopAfter( switchDBBackup(sourceDb, db, backupKeys, tagName, forceAction) );
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
case DB_ABORT:
This removes the idea of clusters from IClientApi.
2018-09-22 06:58:14 +08:00
f = stopAfter( abortDBBackup(sourceDb, db, tagName, partial) );
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
case DB_PAUSE:
This removes the idea of clusters from IClientApi.
2018-09-22 06:58:14 +08:00
f = stopAfter( changeDBBackupResumed(sourceDb, db, true) );
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
break;
Add fdbbackup interface
2017-12-15 05:54:01 +08:00
case DB_RESUME:
This removes the idea of clusters from IClientApi.
2018-09-22 06:58:14 +08:00
f = stopAfter( changeDBBackupResumed(sourceDb, db, false) );
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-31 03:35:00 +08:00
break;
Initial repository commit
2017-05-26 04:48:44 +08:00
case DB_UNDEFINED:
default:
fprintf(stderr, "ERROR: Unsupported DR action %s\n", argv[1]);
printHelpTeaser(argv[0]);
return FDB_EXIT_ERROR;
break;
}
break;
case EXE_UNDEFINED:
default:
return FDB_EXIT_ERROR;
}
runNetwork();
if(f.isValid() && f.isReady() && !f.isError() && !f.get().present()) {
status = FDB_EXIT_ERROR;
}
if(fstatus.isValid() && fstatus.isReady() && !fstatus.isError() && fstatus.get().present()) {
status = fstatus.get().get();
}
#ifdef ALLOC_INSTRUMENTATION
{
cout << "Page Counts: "
<< FastAllocator<16>::pageCount << " "
<< FastAllocator<32>::pageCount << " "
<< FastAllocator<64>::pageCount << " "
<< FastAllocator<128>::pageCount << " "
<< FastAllocator<256>::pageCount << " "
<< FastAllocator<512>::pageCount << " "
<< FastAllocator<1024>::pageCount << " "
<< FastAllocator<2048>::pageCount << " "
print correct pahes in alloc instrumentation
2019-03-09 04:37:04 +08:00
<< FastAllocator<4096>::pageCount << " "
<< FastAllocator<8192>::pageCount << endl;
Initial repository commit
2017-05-26 04:48:44 +08:00
vector< std::pair<std::string, const char*> > typeNames;
for( auto i = allocInstr.begin(); i != allocInstr.end(); ++i ) {
std::string s;
#ifdef __linux__
char *demangled = abi::__cxa_demangle(i->first, NULL, NULL, NULL);
if (demangled) {
s = demangled;
if (StringRef(s).startsWith(LiteralStringRef("(anonymous namespace)::")))
s = s.substr(LiteralStringRef("(anonymous namespace)::").size());
free(demangled);
} else
s = i->first;
#else
s = i->first;
if (StringRef(s).startsWith(LiteralStringRef("class `anonymous namespace'::")))
s = s.substr(LiteralStringRef("class `anonymous namespace'::").size());
else if (StringRef(s).startsWith(LiteralStringRef("class ")))
s = s.substr(LiteralStringRef("class ").size());
else if (StringRef(s).startsWith(LiteralStringRef("struct ")))
s = s.substr(LiteralStringRef("struct ").size());
#endif
typeNames.push_back( std::make_pair(s, i->first) );
}
std::sort(typeNames.begin(), typeNames.end());
for(int i=0; i<typeNames.size(); i++) {
const char* n = typeNames[i].second;
auto& f = allocInstr[n];
printf("%+d\t%+d\t%d\t%d\t%s\n", f.allocCount, -f.deallocCount, f.allocCount-f.deallocCount, f.maxAllocated, typeNames[i].first.c_str());
}
// We're about to exit and clean up data structures, this will wreak havoc on allocation recording
memSample_entered = true;
}
#endif
} catch (Error& e) {
TraceEvent(SevError, "MainError").error(e);
status = FDB_EXIT_MAIN_ERROR;
} catch (std::exception& e) {
Attempt to normalize trace events: * Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check. * Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase. * Use seconds instead of milliseconds in details. Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed. This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-09 02:11:08 +08:00
TraceEvent(SevError, "MainError").error(unknown_error()).detail("RootException", e.what());
Initial repository commit
2017-05-26 04:48:44 +08:00
status = FDB_EXIT_MAIN_EXCEPTION;
}
Static de-initialization order isn't well thought out so use flushAndExit() instead of returning from main after runNetwork() has been called.
2018-12-18 10:39:14 +08:00
flushAndExit(status);
Initial repository commit
2017-05-26 04:48:44 +08:00
}