3728ed03dd
Resolve comments
2020-09-05 18:55:09 -07:00
e9d1f1c9c8
change formatting
2020-08-31 09:43:11 -07:00
b6c0299d09
Add help message in backup CLI for added options
2020-08-31 09:31:57 -07:00
30e27ba27b
Add support for keys in CLI
2020-08-30 00:44:17 -07:00
33aa10b461
Minor optimizations
2020-08-29 20:10:45 -07:00
fd7198d874
Extend backup container interface to support query restorable files set by key ranges
2020-08-29 19:58:07 -07:00
00c77ba2b4
Added beginVersion cmd line option and addressed code review comments
2020-08-28 14:29:22 -04:00
0270ee13c8
change targetversion from maxlogend to contiguouslogend. also adjusted test workload and specs
2020-08-28 12:46:34 -04:00
ae999aa118
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-incremental-backup-only
2020-08-19 16:36:47 -04:00
6e1d3f6cbf
comment out command-line arguments until atomics are supported
2020-08-19 14:36:11 -04:00
35aa1034cd
added new test spec and workload to verify correctness of incremental backup and restore
2020-08-19 13:45:51 -04:00
7dce3a9187
fixed issue with mutations not applying and allow backup to non-empty db
2020-08-11 15:39:21 -04:00
21635f8a28
update backup restore for local testing
2020-08-04 15:48:43 -04:00
a49cb41de7
Merge branch 'release-6.3'
...
# Conflicts:
# CMakeLists.txt
# cmake/ConfigureCompiler.cmake
# fdbserver/Knobs.cpp
# fdbserver/StorageCache.actor.cpp
# fdbserver/storageserver.actor.cpp
# flow/ThreadHelper.actor.h
# flow/serialize.h
# tests/CMakeLists.txt
2020-07-29 00:31:55 -07:00
9c569c6ecf
Merge pull request #3569 from xumengpanda/mengxu/fr-fix-frtool-PR
...
FastRestoreTool: Fix --dest_cluster_file option
2020-07-28 09:56:00 -07:00
2d09d2f68e
re-ordered parameters to fit existing test workloads
2020-07-28 12:39:35 -04:00
71a1b2cb0d
first attempt at making restore pipeline incremental only
2020-07-28 12:39:35 -04:00
207bfec085
added --incremental description to usage messages
2020-07-28 12:39:35 -04:00
3c62651ad6
initial commit to introduce incremental backup only (ignore base snapshot)
2020-07-28 12:39:35 -04:00
e232f805ee
FastRestoreTool: Consume restoreClusterFileDest as fdbrestore does
2020-07-27 22:16:35 -07:00
1a34f5c20e
FastRestoreTool: Fix option list
2020-07-27 21:40:20 -07:00
e0db748fb3
Merge pull request #3403 from satherton/tls-background-handshake
...
TLS handshaking in background threads
2020-07-27 10:55:00 -07:00
b2a3b4fd83
Merge branch 'master' into mengxu/merge-6.3-PR
2020-07-20 11:34:18 -07:00
38cade1146
Merged tag release-6.3.3 into tls-background-handshake.
2020-07-17 23:39:57 -07:00
49095160e3
Merge pull request #3267 from Nishikoh/rename
...
Rename fastRestore_agent to fastRestoreTool
2020-07-16 15:33:07 -07:00
90654628fb
Merge pull request #3490 from apple/release-6.3
...
Merge Release 6.3
2020-07-10 12:54:37 -07:00
25a61bc15d
Detach a Clone of DR Site without Affecting Replication to the Original DR Site
...
Added description of fdbdr --dstonly to the usage text
2020-07-10 15:14:27 +03:00
4c9df78076
dstonly
2020-07-10 15:13:42 +03:00
bcbca81dfe
Remove ROW_LIMIT_UNLIMITED and BYTE_LIMIT_UNLIMITED from CLIENT_KNOBS
2020-07-04 12:48:44 -07:00
ab3de8700e
Rename fastRestoreAgent to fastRestoreTool
2020-07-03 00:16:39 +09:00
22f7f804b8
Merge branch 'release-6.3' into mengxu/merge-6.3-PR
2020-06-28 11:19:39 -07:00
648c870fec
Merge branch 'release-6.3' of https://github.com/apple/foundationdb into tls-background-handshake
2020-06-26 00:21:57 -07:00
0da4c91ad5
Performance improvements in socket use and buffering which results in TLS network connections using around 3.5x less CPU.
...
PacketBuffers are 16k minimum sizes, using FastAlloc. All calling paths to PacketBuffer creation now pass a size hint if known. P2PNetworkTest improved in many ways, errors go to TraceEvents instead of stdout, error counts are reported in stats, range min/max parameters are combined into "min:max" strings, new options to "stutter" connections by delaying randomly before socket read or write, message size to send is now configured separately for incoming vs outgoing sessions.
2020-06-25 20:44:43 -07:00
c13ba7aeb4
FastRestore:Cosmetic code style improvement
...
Also mute some trace events
2020-06-24 21:25:37 -07:00
2fcef90cbc
FastRestore:Add addPrefix and removePrefix feature
2020-06-18 11:58:42 -07:00
8595813b7d
Merge pull request #3355 from apple/release-6.3
...
Merge Release 6.3 into master branch
2020-06-12 20:08:47 -07:00
dc70591859
Fix fdbbackup help message
2020-06-11 17:26:21 -07:00
d128252e90
Merge release-6.3 into master
2020-05-22 09:25:32 -07:00
3b9e9a9257
Merge pull request #3210 from ajbeamon/remove-include-version-h
...
Remove IncludeVersion.h
2020-05-20 14:12:23 -07:00
4b5e94cf27
Merge pull request #3208 from apple/release-6.2
...
Merge release-6.2 into release-6.3
2020-05-20 13:47:48 -07:00
b589c3c940
Remove IncludeVersions.h, which is no longer needed for cmake-only builds
2020-05-20 13:23:02 -07:00
54c97fe2bc
Backup's ProgramStart command line string would be incomplete if arguments were specified using '='
2020-05-20 09:38:33 -07:00
59e33968c6
Remove erroneous restore options.
2020-05-20 08:57:56 -07:00
3ff2fa9c2a
Removed uses of outdated cpuTicks and errorCounts
2020-05-13 23:27:44 -07:00
f17f00fdd5
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2020-05-10 22:33:38 -07:00
c237ca9708
Merge pull request #3081 from jzhou77/backup-cmd
...
Update fdbbackup documentation on new "-p" option
2020-05-04 14:17:34 -07:00
2c5da1c116
Remove "-p" option and use "--partitioned_log_experimental" option
2020-05-04 13:45:54 -07:00
78d644f673
Merge branch 'master' into bugfixes/cmake-includes
2020-05-04 02:11:46 -07:00
37a537c2a3
Fix MacOS compilation error
2020-04-30 18:30:15 -07:00
02aeab1edd
Re-add improperly named options in order to not break compatibility with anything depending on them in a patch release.
2020-04-30 16:23:30 -07:00
07a9a05683
FastRestore:Agent:Fix restore requests
2020-04-30 16:20:20 -07:00
697beb3ebb
Fix fdbrestore parameters with one leading hyphen instead of two, which did not match documentation.
2020-04-30 12:13:35 -07:00
8742cc0ab6
FastRestore:Fix windows build
2020-04-28 19:12:27 -07:00
f5e8345496
FastRestoreAgent:Use atomicParallelRestore to kick off restore
...
Replace the handcrafted version with atomicParallelRestore actor
which is simulation tested
2020-04-27 22:15:00 -07:00
52860043c9
Merge remote-tracking branch 'origin' into atomically-update-dependent-knobs
2020-04-08 12:26:21 -07:00
9fb3fb9d82
Add pause/resume for new backups
...
To pause/resume the backup workers, the fdbbackup command will write to the
backupPausedKey. Then backup workers noticed the value of the key has been
changed and stops/resumes pulling from TLog.
2020-04-06 14:29:46 -07:00
e4611e8ae4
fix versions.h stupidity
2020-04-06 10:28:55 -07:00
8b5780c36c
don't include source and binary dir
...
This forces users to use include paths from the sources root.
So `#include "Arena.h"` won't work anymore, only
`#include "flow/Arena.h"` will.
2020-04-06 10:13:49 -07:00
884e92bb49
Atomically update dependent knobs
2020-04-01 15:18:49 -07:00
906174e3e8
Add an option for fdbbackup to use new backup system
...
I.e., "-p", or "--partitioned_log" to enable it. By default, old backup system
is used.
2020-03-31 14:54:32 -07:00
a93f13cfd7
Remove redundant restoreRequestDone break in backup.actor
2020-03-25 15:19:46 -07:00
495afe2e0b
Improve how to wati for restore to finish
...
Remove default parameter for atomicRestore as suggested in review.
2020-03-25 13:54:21 -07:00
303df197cf
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/NativeAPI.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogRouter.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/fdbserver.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/flow.vcxproj
# flow/flow.vcxproj.filters
# versions.target
2020-03-06 18:22:46 -08:00
595dd77ed1
Merge remote-tracking branch 'upstream/release-6.2' into certificate-refresh
2020-03-04 20:25:42 -08:00
9b5ef3416e
Refactor TLSParams into TLSConfig + LoadedTLSConfig
...
The idea being that we keep around a TLSConfig that the configuration
that the user has provided, and then when we want to intialize an SSL
context, we ask the TLSConfig to load all certificates and return us a
LoadedTLSConfig that is a concrete set of certificate bytes in memory.
initTLS now just takes the in-memory bytes and applies them to the ssl
context.
This is a large refactor to lead up into certificate refeshing, where we
will periodically check for changes to the certificates, and then
re-load them and apply them to a new SSL context.
2020-03-04 20:14:47 -08:00
cdcb816866
Update fdbbackup/backup.actor.cpp
2020-03-04 16:08:45 -08:00
58e621eca1
Invalid knobs or knob values are treated as warnings rather than errors. Apply this change to backup as well.
2020-03-04 15:50:04 -08:00
c11c24b79d
removed the fdbrpc version of platform.h
2020-02-28 14:56:10 -08:00
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
927cff3317
Report errors on TLS misconfigurations ... or at least try to.
2020-02-20 16:57:29 -08:00
761da5a059
code cleanup
2020-02-19 17:59:45 -08:00
132f5aa9ba
FastRestore:Improve trace name and cosmetic change
2020-02-18 16:41:19 -08:00
72110de7e2
FastRestore:Add trace for quick perf. measurement
2020-02-06 19:48:26 -08:00
7628d04fb9
Merge branch 'release-6.2' of github.com:apple/foundationdb into release_6.2_merge
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2020-01-09 07:21:16 -08:00
4ff058e86b
Backup and DR layer status document generation now uses snapshot reads for all keys read to avoid unnecessary conflicts when read during a status update or cleanup transaction. Since many of the keys read use wrapper functions, all of the underlying functions in BackupAgentBase and its two implementations also required a snapshot mode argument. All snapshot arguments default to false to match the underlying FDB API get/getrange methods.
2019-12-19 00:29:35 -08:00
ed8d3f163c
Rename hgVersion to sourceVersion.
2019-11-15 12:26:51 -08:00
4de60fc437
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/TLogServer.actor.cpp
2019-11-01 15:48:04 -07:00
f175ed30b3
Cleanup the fdbbackup cleanup command output. Add cleanup to the usage output printed for fdbbackup.
2019-10-31 09:52:21 -07:00
d4de608bb6
Fix OPEN_FOR_IDE build
2019-10-25 10:42:22 -07:00
7599dd0093
Merge pull request #2205 from jzhou77/backup-fix
...
Fix format string with more portable code
2019-10-02 20:03:35 -07:00
ceb39c0279
Fix format string with more portable code
2019-10-02 15:25:14 -07:00
d0147e5e5d
Merge branch 'release-6.2' into mengxu/merge-release620-to-master-v3
...
Resolved Conflicts:
documentation/sphinx/source/release-notes.rst
fdbserver/DataDistribution.actor.cpp
versions.target
2019-10-02 13:22:56 -07:00
9463b91594
Update fdbbackup/backup.actor.cpp
...
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2019-09-30 13:16:55 -07:00
4d659662b8
made cleanup handle retries better
2019-09-30 12:44:20 -07:00
ef01ad2ed8
optimized log range clearing to clear everything for each possible hash (256 clears) if that would be more efficient than one clear per second that has elapsed
...
aborting a DR without the —cleanup flag will still attempt to cleanup for 30 seconds before giving up
added a cleanup command to fdbbackup which can remove mutations from orphaned DRs which were stopped without the —cleanup flag
2019-09-27 18:32:27 -07:00
9a64dc7f24
Merge pull request #1792 from ajbeamon/remove-dl-iterate-phdr-hack
...
Remove dl_iterate_phdr caching used by slow task profiler
2019-09-17 16:51:30 -07:00
b495cc697b
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-09-13 09:25:08 -07:00
bd7678e71b
Remove --object-serializer help text
2019-09-05 11:57:59 -07:00
c2355f721e
Merge branch 'master' into mengxu/performant-restore-PR
2019-09-04 17:11:42 -07:00
d160810662
FastRestore:Resolve review comments
2019-09-04 16:48:43 -07:00
e465cbef07
Merge branch 'master' into remove-dl-iterate-phdr-hack
...
# Conflicts:
# fdbserver/fdbserver.actor.cpp
2019-08-22 10:13:23 -07:00
f02799455e
Add --loggroup to fdbserver and fdbbackup help text.
2019-08-19 12:59:14 -07:00
7ff46e6772
Merge branch 'master' into mengxu/performant-restore-PR
2019-08-07 20:31:56 -07:00
370ba8b841
Remove --object-serializer flag from executables
2019-08-06 09:25:40 -07:00
3b54363780
FastRestore:Apply Clang-format
2019-08-01 18:09:12 -07:00
7ccaeddf05
Merge branch 'master' into mengxu/performant-restore-PR
2019-08-01 13:23:17 -07:00
b5d2234a13
Merge branch 'release-6.1' into merge-release-6.1-into-master
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbserver/MoveKeys.actor.cpp
# flow/FastAlloc.h
# versions.target
2019-07-30 16:23:42 -07:00
9e1042d903
Bug fix, restore with --dryrun option would still require the --dest_cluster_file option even though it does not connect to a cluster.
2019-07-29 16:21:12 -07:00
b644f15b87
Bug fix: fdbrestore commands other than "start" were using default cluster file argument handling (but without the -C flag) instead of using the --dest_cluster_file argument.
2019-07-29 13:19:28 -07:00
b0c31f28af
FastRestore:Fix bug that blocks restore
...
1) Should recruit only configured number of roles;
2) Should never register a restore master interface as a restore worker (loader or applier) interface.
2019-07-25 17:55:37 -07:00
Young Liu