6bddf8aefa
Upgrade DR from 5.1 to 5.2
2018-04-26 17:24:40 -07:00
ede5cab192
Merge pull request #89 from yichic/share-log-mutations-5.2
...
Share log mutations 5.2
2018-03-19 12:01:26 -07:00
ec02e54f64
Refactor EraseLogData()
2018-03-19 11:56:01 -07:00
1f2602d2b3
Fix all review comments
2018-03-19 11:33:33 -07:00
d6559b144f
Share log mutations between backups and DRs which have the same backup range
2018-03-19 11:32:50 -07:00
d136fc8eea
Improve fdbbackup status
...
Additional details added are
log bytes written
range bytes written
last snapshot version and its corresponding timestamp
last log version and its corresponding timestamp
If backup is supposed to stop at the next snapshot completion (stopWhenDone)
version/timestamp of snapshot start and expected end
Backup error reporting format changed to
Recent Errors (since latest restore point TIME ago)
<TIME> ago: <error description> on <task name>
Older Errors
<TIME> ago: <error description> on <task name>
TIME format:
<= 59 seconds ago
<= 59.99 minutes ago
<= 23.99 hours ago
N.NN days ago
2018-03-08 12:43:33 -08:00
dcf5b2e35d
All readCommitted() functions now use Transaction instead of ReadYourWritesTransaction to reduce memory consumption in Backup and DR. Also removed one readCommitted() variant as it is just a special case of another definition.
2018-03-07 13:56:34 -08:00
0bae9880f1
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-21 10:25:11 -08:00
54fc81b260
Improved backup error reporting in backup status. The most recent error for each error type is reported along with how long ago the error occurred, and errors are divided into two categories based on whether or not they occurred since the most recent backup progress.
2018-02-16 19:38:31 -08:00
93b34a945f
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 04:09:43 -08:00
69f7409c37
fix: latestRestorable was incorrect
2017-12-21 17:09:21 -08:00
5ed080721d
fix: atomic restore must wait for the restorable version is greater than the lock version
...
fix: latestRestorableVersion calculation was wrong
2017-12-21 15:45:10 -08:00
f70e3b9fe8
Add or change a bunch of comments to provide descriptions of function contracts.
...
This cleans up a bit of the VersionStamp DR work I did, and leaves hints and
advice for anyone who will be touching mutation applying code in the future.
2017-12-20 16:57:14 -08:00
e0d9cea008
Merge branch 'master' into continuous-backup
...
# Conflicts:
# fdbclient/FileBackupAgent.actor.cpp
# fdbrpc/BlobStore.actor.cpp
2017-12-19 23:02:14 -08:00
afd2603576
Refactored backup task flow and config to support ongoing snapshots and allow stopping the backup cleanly between snapshots. The previously separate tasks for initial and differential mode log dispatching have been merged into BackupLogsDispatchTask.
2017-12-17 14:29:57 -08:00
18305ab326
Bug fixes. Added snapshotBatchSize to backupConfig to enable detecting if a transaction for adding a group of tasks to a batch had already completed. Changed KeyRangeMap usage so that each range value to be dispatched has a unique integer value, enabling more efficient range coalescing and avoiding some iterator invalidation bugs.
2017-12-15 01:39:50 -08:00
33f9f1a95c
Added SnapshotDispatch task for writing snapshots in random order over a specified period of time and adapting speed to a growing or shrinking database. TaskBucket now supports scheduling tasks. TaskFuture now correctly recognizes multiple tasks in its callback space. TaskBucket extendTimeout() now supports specifying the new timeout version. Submitting a backup now requires a snapshot duration.
2017-12-14 01:44:38 -08:00
44f0f943e8
fix: an abort is not successful until a second dummy transaction in committed to ensure that apply mutations has stopped
2017-12-04 17:21:43 -08:00
42c6f7db34
Taskbucket but fix, caused by accidental removal of task function lookup. Added extendMutex to Task for use around transaction loops that call extendTimeout() to reduce conflicts.
2017-12-03 20:52:09 -08:00
d9c2f6d705
Bug fix. The terminator argument of readCommitted() previously did nothing, and end_of_stream() was always sent to the output stream. The parameter was fixed to enable changing this behavior but original the behavior was not being correctly preserved in at least one case.
2017-11-26 22:52:47 -08:00
32903ffa77
Trace event improvements and severity changes.
2017-11-19 04:34:28 -08:00
a77162b53d
Merge branch 'master' into backup-container-refactor
...
# Conflicts:
# fdbclient/BackupAgent.h
# fdbclient/FileBackupAgent.actor.cpp
# fdbclient/KeyBackedTypes.h
2017-11-15 08:14:47 -08:00
3dfaf13b67
IBackupContainer has been rewritten to be a logical interface for storing, reading, deleting, expiring, and querying backup data. The details of how the data is organized or stored is now hidden from users of the interface. Both the local and blobstore containers have been rewritten, the key changes being a multi level directory structure and no more use of temporary files or pseudo-symlinks in the blob store implementation. This refactor has a large impact radius as the previous backup container was just a thin wrapper that presented a single level list of files and offered no methods for managing or interpreting the file structure so all of that logic was spread around other places in the code base. This made moving to the new blob store schema very messy, and without this refactor further changes in the future would only be worse.
...
Several backup tasks have been cleaned up / simplified because they no longer need to manage the ‘raw’ structure of the backup. The addition of IBackupFile and its finish() method simplified the log and range writer tasks. Updated BlobStoreEndpoint to support now-required bucket creation and bucket listing prefix/delimiter options for finding common prefixes. Added KeyBackedSet<T> type. Moved JSONDoc to its own header. Added platform::findFilesRecursively().
Still to do: update command line tool to use new IBackupContainer interface, fix bugs in Restore startup.
2017-11-14 23:33:17 -08:00
7c2185d5f7
fix: last restorable subspace was not initialized
2017-10-27 14:06:15 -07:00
d1e9d28239
Backup log messages.
2017-10-12 16:12:42 -07:00
c36a30837d
Moving keyErrors in BackupConfig.
2017-09-08 16:09:18 -07:00
02cc8b43c4
More backup cleanup.
2017-09-07 09:04:10 -07:00
c7df951f7c
Using BackupConfig from backup.actor.cpp to reduce intermediate
...
functions.
2017-09-07 08:36:36 -07:00
9f8056754a
Moving KeyBackedTag and KeyBackedConfig into BackupAgent.h to use them from backup.actor.cpp
2017-09-06 09:46:27 -07:00
83810edabc
Backup/Restore tag can be std::string instad of Key.
2017-09-05 11:38:40 -07:00
d834ab9d4d
Moving from task->params to Params
2017-09-01 13:50:38 -07:00
871bac0f96
Cleanup submitCleanup()
2017-08-30 18:05:50 -07:00
660729839c
moved Notified.h from flow -> fdbclient ; flow bindings package does better job when excluding testers
2017-07-14 15:49:30 -07:00
a674cb4ef4
Initial repository commit
2017-05-25 13:48:44 -07:00
Yichi Chiang