b77276d2f0
First snapshot of a backup should go as fast as possible instead of using the configured snapshot interval.
2017-12-20 01:07:03 -08:00
7caa012fbf
Added snapshot interval option to "fdbbackup start" which defaults to a new knob's value. Added snapshot info to backup status text. Improvements to fdbbackup help.
2017-12-20 00:49:08 -08:00
d87aa521e9
Merge branch 'backup-container-refactor' into continuous-backup
2017-12-19 23:39:00 -08:00
e0d9cea008
Merge branch 'master' into continuous-backup
...
# Conflicts:
# fdbclient/FileBackupAgent.actor.cpp
# fdbrpc/BlobStore.actor.cpp
2017-12-19 23:02:14 -08:00
2cd1ff6aae
Bug fix, in restore dispatch the apply lag was being retrieved before updating the apply end version which would make it look like mutations were finished applying early.
2017-12-19 18:11:40 -08:00
61a043ebfa
Added tr->reset() to prevent initial transaction loop attempts from having a higher chance of expiring.
2017-12-19 17:33:45 -08:00
aa5169bd3c
Removed unnecessary trace event.
2017-12-19 15:29:22 -08:00
e28641886d
TraceEvent improvements. Minor bug fix, restore log writing tasks didn't have the log file endVersion but it's only for logging purposes.
2017-12-19 15:27:04 -08:00
a276985baf
Bug fix, if there are range files in a restore which begin at exactly the restore version they will be repeatedly dispatched forever.
2017-12-18 17:48:18 -08:00
005a4a0706
Restore status bug fix, during restore the apply lag would appear as a large negative number until the first restore batch is completed. Test improvement, snapshot dispatch now chooses a random number of tasks to dispatch per commit.
2017-12-18 15:56:57 -08:00
937fa75bec
Bug fix, if target snapshot end version is at or before the begin version then no progress would be made.
2017-12-18 00:13:25 -08:00
d32a770648
Bug fix, backup never went to differential mode once it was restorable which caused waitBackup to only return once the backup was discontinued.
2017-12-17 23:22:18 -08:00
2b92815e8c
Bug fix. The snapshot dispatch add task retry loop was incorrectly deciding that the second and further transaction of an execution was already committed and therefore skipping it, resulting in missing ranges in the snapshot.
2017-12-17 21:01:31 -08:00
afd2603576
Refactored backup task flow and config to support ongoing snapshots and allow stopping the backup cleanly between snapshots. The previously separate tasks for initial and differential mode log dispatching have been merged into BackupLogsDispatchTask.
2017-12-17 14:29:57 -08:00
18305ab326
Bug fixes. Added snapshotBatchSize to backupConfig to enable detecting if a transaction for adding a group of tasks to a batch had already completed. Changed KeyRangeMap usage so that each range value to be dispatched has a unique integer value, enabling more efficient range coalescing and avoiding some iterator invalidation bugs.
2017-12-15 01:39:50 -08:00
50c154fed4
Add fdbbackup interface
2017-12-14 13:54:01 -08:00
33f9f1a95c
Added SnapshotDispatch task for writing snapshots in random order over a specified period of time and adapting speed to a growing or shrinking database. TaskBucket now supports scheduling tasks. TaskFuture now correctly recognizes multiple tasks in its callback space. TaskBucket extendTimeout() now supports specifying the new timeout version. Submitting a backup now requires a snapshot duration.
2017-12-14 01:44:38 -08:00
872edd7540
Merge branch 'release-5.0'
...
# Conflicts:
# fdbclient/FileBackupAgent.actor.cpp
2017-12-06 16:27:04 -08:00
532de63a05
Changed log and range backup task error events to SevWarn from SevError.
2017-12-06 16:21:15 -08:00
ce6c49e173
Corrected a bunch of retry loops to not reset the backoff timer.
2017-12-06 14:11:40 -08:00
44f0f943e8
fix: an abort is not successful until a second dummy transaction in committed to ensure that apply mutations has stopped
2017-12-04 17:21:43 -08:00
86ae6c09c7
Bug fixes, take(1) is incorrect usage of FlowLock.
2017-12-04 10:20:50 -08:00
42c6f7db34
Taskbucket but fix, caused by accidental removal of task function lookup. Added extendMutex to Task for use around transaction loops that call extendTimeout() to reduce conflicts.
2017-12-03 20:52:09 -08:00
eadf93826d
Bug fixes with transaction options and exception handling that were causing internal errors.
2017-12-01 15:16:44 -08:00
aeebe711ce
TaskBucket’s saveAndExtend() is now accomplished through extendTimeout() with an option to save parameters. SaveAndExtendIncrementally() has been removed as it is no longer needed because TaskBucket’s normal execution loop calls extendTimeout() periodically as long as the TaskFunc’s execute() actor has not finished or thrown. If a TaskFunc wants to save changes to task parameters to checkpoint progress for task restarts to benefit from it can call extendTimeout() explicitly with the updateParams flag set to true.
2017-11-30 17:18:57 -08:00
39edda1804
Bug fix, and some code cleanup along the way. If a range backup task dies in finish() the re-run of the task will start at begin == end, which wasn’t being handled correctly.
2017-11-27 15:57:19 -08:00
1d3af8f4f0
Bug fix.
2017-11-25 21:13:56 -08:00
1b1c8e985a
Merge branch 'master' into backup-container-refactor
...
# Conflicts:
# fdbclient/FileBackupAgent.actor.cpp
2017-11-25 19:54:51 -08:00
6695c9e6a2
Bug fixes and improvements to error handling and trace events. The most serious bug was that restore would start at the wrong version, possibly skipping early log and range files.
2017-11-25 00:46:16 -08:00
3449bc4cdc
Bug fix, range end was wrong for final range file of backup range task.
2017-11-19 04:44:33 -08:00
a31216f3f7
Added toString() to Backup/Restore TaskFunc interface so tasks can provide a method to describe important task parameters for the default handleError() methods to use.
2017-11-19 04:39:18 -08:00
32903ffa77
Trace event improvements and severity changes.
2017-11-19 04:34:28 -08:00
5e596ea670
Adding TraceEvents for BackupRangeTask.
2017-11-16 19:11:31 -08:00
cc47d0e161
Bug fix in restore dispatch, begin file was not being incremented. Removed try/catch because the inherited handleError() is better.
2017-11-15 22:38:31 -08:00
ab0017f023
TaskBucket’s TaskFunc interface now has an optional handleError() which is called on any task that throws an error from execute() or finish(). Restore and Backup tasks use this to ensure that any errors that occur are placed in the backup or restore config’s lastError property. Bug fixes in log and range file encodings.
2017-11-15 13:33:09 -08:00
a77162b53d
Merge branch 'master' into backup-container-refactor
...
# Conflicts:
# fdbclient/BackupAgent.h
# fdbclient/FileBackupAgent.actor.cpp
# fdbclient/KeyBackedTypes.h
2017-11-15 08:14:47 -08:00
3dfaf13b67
IBackupContainer has been rewritten to be a logical interface for storing, reading, deleting, expiring, and querying backup data. The details of how the data is organized or stored is now hidden from users of the interface. Both the local and blobstore containers have been rewritten, the key changes being a multi level directory structure and no more use of temporary files or pseudo-symlinks in the blob store implementation. This refactor has a large impact radius as the previous backup container was just a thin wrapper that presented a single level list of files and offered no methods for managing or interpreting the file structure so all of that logic was spread around other places in the code base. This made moving to the new blob store schema very messy, and without this refactor further changes in the future would only be worse.
...
Several backup tasks have been cleaned up / simplified because they no longer need to manage the ‘raw’ structure of the backup. The addition of IBackupFile and its finish() method simplified the log and range writer tasks. Updated BlobStoreEndpoint to support now-required bucket creation and bucket listing prefix/delimiter options for finding common prefixes. Added KeyBackedSet<T> type. Moved JSONDoc to its own header. Added platform::findFilesRecursively().
Still to do: update command line tool to use new IBackupContainer interface, fix bugs in Restore startup.
2017-11-14 23:33:17 -08:00
986b73f458
Fixed an issue where an ACTOR outlives an object passed to it and then crashes while accessing it.
2017-11-14 13:51:23 -08:00
b1fe3d1b55
Delete spurious spaces
2017-11-06 12:59:00 -08:00
d46e240de2
Merge branch 'release-5.0'
...
# Conflicts:
# fdbclient/FileBackupAgent.actor.cpp
# versions.target
2017-11-02 10:42:30 -07:00
c2fd27f294
Previous commit was accidentally early and didn’t compile.
2017-11-01 01:22:08 -07:00
acf747c41f
Bug fix in backup file termination, errors during sync() were being logged but then not thrown. This can cause a backup to appear to have a log or keyrange file in it but the temp file pointed to by the log/keyrange symbolic link was not successfully uploaded.
2017-11-01 01:16:16 -07:00
196aac4cec
fix: tag needs to be assigned directly before using it
2017-10-30 16:23:26 -07:00
fb89ae9f85
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-30 12:35:00 -07:00
7c2185d5f7
fix: last restorable subspace was not initialized
2017-10-27 14:06:15 -07:00
3848f7808c
Fixing Backup wrong key range issue.
2017-10-24 15:26:55 -07:00
5718d32553
Bug fix, file backup on Windows has been broken for a long time because it was constructing a path using / and then checking that it matches a normalized absolute path which would have the / converted to \.
2017-10-19 13:33:12 -07:00
360b777b78
Fail with correct error code in case of abort or discontinue of
...
non-existing backups.
2017-10-18 23:17:48 -07:00
d1e9d28239
Backup log messages.
2017-10-12 16:12:42 -07:00
659e39103e
Missed file from merge of master into backup-refactor
2017-10-12 11:25:29 -07:00
13a79482d8
Added comments for clarity.
2017-10-01 16:03:12 -07:00
a95107417f
Improved behavior of slow writes during backup. KeyRange and Log backup tasks now use TaskBucket::saveAndExtend() to keep the task alive until flushing the file finishes or fails with an error (blob uploads fail after a limited number of retries). This prevents blob uploads from being retried too often if the destination is slow since a task abort and retry would start the backoff counters back at zero. Also removed a debugging behavior that was accidentally checked in.
2017-10-01 16:01:24 -07:00
10402e0c32
Removing add_task() code duplication in FileBackupAgent.actor.cpp
2017-09-11 11:14:30 -07:00
c36a30837d
Moving keyErrors in BackupConfig.
2017-09-08 16:09:18 -07:00
02cc8b43c4
More backup cleanup.
2017-09-07 09:04:10 -07:00
c7df951f7c
Using BackupConfig from backup.actor.cpp to reduce intermediate
...
functions.
2017-09-07 08:36:36 -07:00
9f8056754a
Moving KeyBackedTag and KeyBackedConfig into BackupAgent.h to use them from backup.actor.cpp
2017-09-06 09:46:27 -07:00
d917f9449f
Fixing Steve's review comments.
2017-09-05 14:06:55 -07:00
92d05f6fc3
backupContainer is a std::string not a Key.
2017-09-05 13:22:44 -07:00
83810edabc
Backup/Restore tag can be std::string instad of Key.
2017-09-05 11:38:40 -07:00
456ced2c65
Minor backup code cleanup
2017-09-05 09:42:14 -07:00
e1a7e11347
Minor backup code cleanup
2017-09-01 14:39:38 -07:00
d834ab9d4d
Moving from task->params to Params
2017-09-01 13:50:38 -07:00
c564aaae68
Moving keyConfigBackupRanges into BackupConfig::backupRanges().
2017-09-01 11:52:08 -07:00
871bac0f96
Cleanup submitCleanup()
2017-08-30 18:05:50 -07:00
b38f131a46
Move keyStateStop to BackupConfig::stopVersion()
2017-08-30 16:22:28 -07:00
e73b72cdb9
Moving keyConfigStopWhenDoneKey to BackupConfig::stopWhenDone()
2017-08-30 15:31:55 -07:00
1655547048
Removing keyConfigLogUid in preference to KeyBackedConfig::getUidAsKey().
2017-08-30 15:07:36 -07:00
c1b6f3fdf2
Moving keyBackupTag to BackupConfig.tag()
2017-08-30 14:34:44 -07:00
439193d17b
Moving keyBackupContainer to BackupConfig.backupContainer()
2017-08-30 12:48:28 -07:00
c766bcb797
Moving keyStateStatus to BackupConfig::stateEnum.
2017-08-30 10:38:06 -07:00
819566c166
keyFolderId is not used in File Backup anymore. We are instead using tag->uid based task validation.
2017-08-29 09:26:32 -07:00
df15dce000
Make BackupConfig subclass of KeyBackedConfig and remoe old way of Task
...
key validation.
2017-08-28 18:20:55 -07:00
2ece658e60
Don't reuse backup logUid.
2017-08-28 16:50:39 -07:00
32a690bce8
Generalize RestoreConfig class.
2017-08-28 16:48:26 -07:00
8ac750672b
Make RestoreTag and RestoreTags classes generic to be used with Backup.
2017-08-28 11:28:19 -07:00
a674cb4ef4
Initial repository commit
2017-05-25 13:48:44 -07:00
Stephen Atherton