Commit Graph

993 Commits

Author SHA1 Message Date
Evan Tschannen 79d94214a4 Merge commit 'f4ffc9752b5ec66ac47f5f684a5d8be06a7eae6e' into feature-remote-logs 2018-01-25 10:12:06 -08:00
Stephen Atherton afae892616 Merge branch 'release-5.1' 2018-01-24 19:45:18 -08:00
Stephen Atherton 9fd2a8df3d Tweaked a trace event suppression time. 2018-01-24 19:08:24 -08:00
Stephen Atherton 4dec5423f7 Optimization in backup snapshot dispatching. If the next dispatch version is not ahead of the recently known current version then do not set a scheduled time for new range tasks in order to avoid the overhead and delay of task timeout handling. Adjusted task timeout knobs to avoid large transaction warnings. Removed backtrace() from LargeTransaction trace event. Tweaked suppression in backup trace events. 2018-01-24 17:40:02 -08:00
A.J. Beamon 55f55b6750 Merge branch 'release-5.1' 2018-01-24 14:44:07 -08:00
A.J. Beamon 19ed388c0e Merge branch 'release-5.0' into release-5.1
# Conflicts:
#	documentation/sphinx/source/downloads.rst
#	documentation/sphinx/source/release-notes.rst
#	versions.target
2018-01-24 14:43:41 -08:00
A.J. Beamon 6c57ec3680 Update versions.target and FDBInstaller.wxs 2018-01-24 14:41:41 -08:00
Stephen Atherton 95d4e5520b Added TraceEvent. 2018-01-24 12:44:37 -08:00
A.J. Beamon 31dfedcd21 Release notes and other release prep 2018-01-24 12:32:33 -08:00
A.J. Beamon 7b19fe4d11 Remove non-existing file FdbJNI.h from visual studio project 2018-01-24 12:07:43 -08:00
Stephen Atherton aebbe1dcfd Changed core_versionspersecond knob to int64_t type to avoid integer overflow. Cleaned up backup TraceEvent suppression. Added backtrace to LargeTransaction TraceEvent to make it easier to find the source of large commits in applications using NativeAPI directly. 2018-01-24 11:59:37 -08:00
Stephen Atherton 83409fb067 Bug fix, versionFolderString() was not reducing the precision of the number in the output string. Not technically a 'bug' as the scheme will still work but produces an overly deep and sparse folder structure. 2018-01-24 10:29:37 -08:00
Stephen Atherton 7f18d59dfe Bug fix, the blob request attempt count is now incremented for all errors except response code 429. 2018-01-24 01:15:01 -08:00
Stephen Atherton a2481343ec Bug fix, HTTP error code 429 was not being considered retryable in blob client (this was previously fixed but apparently reintroduced). 2018-01-24 00:22:11 -08:00
Stephen Atherton 40d38880fe Changed version-based folder naming scheme to something simpler, a fixed width 0-padded 19 digit number (the longest a Version can be) with /'s inserted to limit the size of each folder level. Comparisons using these folder names ignore the /'s so any future change to the splitting scheme would still be compatible with the current listing/reading logic. 2018-01-23 15:02:15 -08:00
Stephen Atherton 7db7a51440 Changes to backup folder structure in BackupContainerBlobStore at the top level inside the backup bucket. All data files now live under data/<backup_name> and there is an 'index' at backups/<backup_name> which indicates what backups exist. Backup names can now contain '/' characters. 2018-01-23 11:46:16 -08:00
Stephen Atherton 7f0b7311b9 Corrected function name to timeKeeperVersionFromDatetime(). 'Fdbbackup expire' now allows an expire_before version of 0 if explicitly passed by version or by timestamp. 2018-01-23 00:19:51 -08:00
Stephen Atherton 51a1bd9327 Timekeeper lookup improvements, moved both function declartions to BackupContainer.h. VersionFromEpochs() now uses versions/sec to adjust the lookup result to improve accuracy. Conversions in both directions look for the latest record less than the target conversion value, but failing that they will now fall back on any available data point and adjust from there using versions/sec. 2018-01-22 23:57:01 -08:00
Stephen Atherton f086ba9d9d Improved version to timestamp lookup - if there are no older versioned records in the database then the next available record, if any, will be used to calculate a result. 2018-01-22 22:47:57 -08:00
Stephen Atherton 02197994b0 Added fdbbackup describe --version_timestamps option which is now required for describe to print version timestamps. Improved help...slightly. 2018-01-22 15:50:28 -08:00
Stephen Atherton 1f59f9ee5b Reduce restore dispatch transaction size. 2018-01-22 15:04:14 -08:00
Stephen Atherton 66de9d392b New error code, http_auth_failed, which is used when blob authentication fails instead of the previous generic http_request_failed. 2018-01-22 14:58:56 -08:00
A.J. Beamon 1e23d02c3c Merge branch 'release-5.1' 2018-01-22 11:58:52 -08:00
A.J. Beamon 2744646090 Merge branch 'release-5.0' into release-5.1 2018-01-22 11:57:58 -08:00
A.J. Beamon 188562ccbc fix: Status should create its DatabaseConfiguration using fromKeyValues(). This makes sure that various state is correctly set if not specified in the configuration. 2018-01-22 11:40:08 -08:00
Evan Tschannen 66b2218989 added tlog support for upgrading from 5.X clusters. Does not support upgrading from 4.X or earlier. Untested, storage servers still need the ability to change their tag. 2018-01-21 12:21:46 -08:00
Evan Tschannen 698ef4117e Merge branch 'master' into feature-remote-logs 2018-01-20 10:34:30 -08:00
Evan Tschannen 07cdb092e5 Merge branch 'release-5.1' 2018-01-20 10:34:01 -08:00
Evan Tschannen 89f0f9318a decodeServerTagValue decodes tags encoded pre-6.0 2018-01-20 10:33:13 -08:00
Evan Tschannen b5eba4f13a fix: do not check for desired data centers if they have not been set 2018-01-20 10:28:59 -08:00
A.J. Beamon 2451e11ff1 ProgramStart wasn't actually be logged in fdbbackup because it happened before the cluster object was created. 2018-01-19 16:57:10 -08:00
A.J. Beamon 16cd0c8f75 Change WaitStorageMetricsPenalty to not be SevWarnAlways unless it happens multiple times in a row (about 2.5 minutes with current knobs). 2018-01-19 16:41:15 -08:00
Evan Tschannen 0ca8b612c7 fix: do not set onDone until all addTaskFutures have had a chance to join that future 2018-01-19 16:29:42 -08:00
Stephen Atherton 9d5f7bd5ab Aborting an old incompatible backup did not work if the old backup was never actually initialized, started, or was already aborted. 2018-01-19 15:08:03 -08:00
Stephen Atherton 93a7eb7474 Fdbbackup restore will now default to the latest usable version in a backup if a specific restore version was not given. Expire will make sure a cluster is provided if either of its timestamp options are given. 2018-01-19 12:19:32 -08:00
Stephen Atherton 6e96d3c30c Bug fix, backup snapshots could take unexpectedly long if the desired snapshot interval is less than the configured snapshotDispatch interval. 2018-01-19 12:14:04 -08:00
Stephen Atherton da02099c4c Merge branch 'release-5.1' of github.com:apple/foundationdb into release-5.1 2018-01-19 11:02:48 -08:00
Stephen Atherton fc16bb94ab Discontinuing a backup that is already restorable now stops immediately and aborts (via validation key) any tasks scheduled to run later. 2018-01-19 11:02:43 -08:00
Evan Tschannen 570f72ba40 fix: nextDispatchVersion was being set too large if the snapshot interval was small 2018-01-19 10:53:58 -08:00
A.J. Beamon 35b91bfb55 Add back (in different form) some ratekeeper trace events when a storage server or log doesn't respond. Add actualTPS (named TPSBasis) to RkUpdate. 2018-01-18 14:51:38 -08:00
Evan Tschannen b78e0a362a fix: do not pause when running multiple backup tests simultaneously 2018-01-18 12:24:33 -08:00
Stephen Atherton b6dd06d945 Bug fix in version to timestamp conversion. 2018-01-18 02:54:12 -08:00
Stephen Atherton 307e04c0ad Updated backup container unit test to match new safer behavior of expireData(). Rewrote BackupContainerLocalDirectory::deleteContainer() to actually delete the whole directory but only if it appears to be a backup with either log or snapshot data. 2018-01-18 00:36:28 -08:00
Evan Tschannen 2e46ee3dba fix: getTeam works when there are no teams 2018-01-17 17:49:13 -08:00
Evan Tschannen 264dc44dfa fixed many more bugs associated with running without remote logs 2018-01-17 17:03:17 -08:00
Stephen Atherton cdd1e784dc Added yields to writing backup snapshot manifests to avoid slow tasks. 2018-01-17 13:28:56 -08:00
Stephen Atherton f6f0816bc1 Merge branch 'release-5.1' of github.com:apple/foundationdb into release-5.1 2018-01-17 12:12:12 -08:00
Stephen Atherton 8fece71662 Bug fix in backup metadata handling if logEnd becomes less than logBegin, which can happen if an expire is done without logEnd being updated. 2018-01-17 12:12:04 -08:00
Stephen Atherton d7f8fe218a Bug fix, resolving versions to timestamps for use in backup descriptions did not work on a locked database. Added some trace events to AtomicRestore to show progress. 2018-01-17 12:03:19 -08:00
A.J. Beamon 4bfbdbf454 Extract getLocalTime to platform.cpp 2018-01-17 11:35:34 -08:00