074531309b
Merge pull request #3769 from sfc-gh-ljoswiak/binary-info
...
Add CLI argument to print build information
2021-01-26 10:29:57 -07:00
3e72163374
Fix compile issues
2021-01-21 10:22:48 -08:00
ad99bf0471
Merge remote-tracking branch 'origin' into misc-changes
2021-01-21 10:03:07 -08:00
f011af14bc
Add build flags option to fdbdecode binary
2021-01-19 11:21:21 -08:00
b7758cd6b4
Add build flags option to fdbconvert binary
2021-01-19 11:21:21 -08:00
39c6cedf10
Add build flags option to backup binaries
2021-01-19 11:21:21 -08:00
2c4b5acd28
Add help option back to actions
2021-01-19 10:33:55 -08:00
d82b61ee0f
Update help message for top level options
2021-01-15 17:05:11 -08:00
1177183819
Copy help option to top level
2021-01-15 17:05:11 -08:00
14b08e1103
Add top level binary CLI options
2021-01-15 17:05:10 -08:00
95eaa5e866
Merge remote-tracking branch 'origin/master' into misc-changes
2021-01-13 21:14:36 -08:00
8ff14878fe
Merge remote-tracking branch 'origin/master' into simplify-global-knobs
2021-01-13 14:39:35 -08:00
e29ed3bf99
Remove createGlobal*Knobs functions
2021-01-13 12:14:04 -08:00
2609c3d619
Merge pull request #4072 from sfc-gh-tclinkenbeard/improve-type-safety
...
Make enums automatically binary serializable
2021-01-12 10:31:34 -07:00
86c7c1e946
Fix IAsyncFileSystem method signatures
2020-12-28 01:57:42 -04:00
5b2e88b187
Use structured bindings in for loops
2020-12-27 01:46:20 -04:00
b7f4d69dc4
Added VersionedKVPart class to FileDecoder.actor.cpp
2020-12-26 21:53:14 -04:00
5bfa6cea98
Merge remote-tracking branch 'origin/master' into misc-changes
2020-12-26 20:47:00 -04:00
0740e35621
Use unique_ptr for backup args
2020-12-08 09:09:32 -08:00
c77d9e4abe
merge conflicts
2020-12-02 21:53:19 +00:00
89194a48b9
Enable --incremental usage for fdbbackup on command line
2020-12-01 15:53:20 -05:00
b1e3478267
Simplified global knobs
2020-11-21 13:27:48 -08:00
79146d2e31
Improve type-safety of backup.actor.cpp
2020-11-14 22:25:32 -08:00
4669f837fa
Add uses of makeReference
2020-11-07 22:10:18 -08:00
9c6c2704d0
Merge branch 'master' of https://github.com/apple/foundationdb into stable_interfaces
2020-10-30 23:59:32 +00:00
55e86c1336
Disambiguate between S3BlobStore and other blob stores
2020-10-29 20:42:23 -07:00
41843f07e6
add simulator support for different process versions and ProtocolVersion test
2020-10-12 18:19:31 +00:00
bbc7ce581e
Resolve conflicts merging from 6.3 to master
2020-09-25 16:08:31 -07:00
862336de8f
Merge branch 'master' into mengxu/merge-to-master-PR
2020-09-24 17:06:00 -07:00
d13a9d3c91
Make EBackupState an enum class
2020-09-23 15:14:09 -07:00
60ccaa2ae9
Support finding max restorable version for key ranges when keyspace's max restorable version is unavailable
2020-09-17 17:58:58 -07:00
d4e3e98957
Use new getRestoreSet in RestoreController
2020-09-17 17:56:36 -07:00
f9a5b727a6
Add comments questions and TODOs
2020-09-14 16:28:01 -07:00
3728ed03dd
Resolve comments
2020-09-05 18:55:09 -07:00
e9d1f1c9c8
change formatting
2020-08-31 09:43:11 -07:00
b6c0299d09
Add help message in backup CLI for added options
2020-08-31 09:31:57 -07:00
30e27ba27b
Add support for keys in CLI
2020-08-30 00:44:17 -07:00
33aa10b461
Minor optimizations
2020-08-29 20:10:45 -07:00
fd7198d874
Extend backup container interface to support query restorable files set by key ranges
2020-08-29 19:58:07 -07:00
00c77ba2b4
Added beginVersion cmd line option and addressed code review comments
2020-08-28 14:29:22 -04:00
0270ee13c8
change targetversion from maxlogend to contiguouslogend. also adjusted test workload and specs
2020-08-28 12:46:34 -04:00
ae999aa118
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-incremental-backup-only
2020-08-19 16:36:47 -04:00
6e1d3f6cbf
comment out command-line arguments until atomics are supported
2020-08-19 14:36:11 -04:00
35aa1034cd
added new test spec and workload to verify correctness of incremental backup and restore
2020-08-19 13:45:51 -04:00
7dce3a9187
fixed issue with mutations not applying and allow backup to non-empty db
2020-08-11 15:39:21 -04:00
21635f8a28
update backup restore for local testing
2020-08-04 15:48:43 -04:00
a49cb41de7
Merge branch 'release-6.3'
...
# Conflicts:
# CMakeLists.txt
# cmake/ConfigureCompiler.cmake
# fdbserver/Knobs.cpp
# fdbserver/StorageCache.actor.cpp
# fdbserver/storageserver.actor.cpp
# flow/ThreadHelper.actor.h
# flow/serialize.h
# tests/CMakeLists.txt
2020-07-29 00:31:55 -07:00
d2cf700bd4
Fix compiler warnings
2020-07-28 18:30:26 +00:00
9c569c6ecf
Merge pull request #3569 from xumengpanda/mengxu/fr-fix-frtool-PR
...
FastRestoreTool: Fix --dest_cluster_file option
2020-07-28 09:56:00 -07:00
2d09d2f68e
re-ordered parameters to fit existing test workloads
2020-07-28 12:39:35 -04:00
71a1b2cb0d
first attempt at making restore pipeline incremental only
2020-07-28 12:39:35 -04:00
207bfec085
added --incremental description to usage messages
2020-07-28 12:39:35 -04:00
3c62651ad6
initial commit to introduce incremental backup only (ignore base snapshot)
2020-07-28 12:39:35 -04:00
e232f805ee
FastRestoreTool: Consume restoreClusterFileDest as fdbrestore does
2020-07-27 22:16:35 -07:00
1a34f5c20e
FastRestoreTool: Fix option list
2020-07-27 21:40:20 -07:00
e0db748fb3
Merge pull request #3403 from satherton/tls-background-handshake
...
TLS handshaking in background threads
2020-07-27 10:55:00 -07:00
b2a3b4fd83
Merge branch 'master' into mengxu/merge-6.3-PR
2020-07-20 11:34:18 -07:00
38cade1146
Merged tag release-6.3.3 into tls-background-handshake.
2020-07-17 23:39:57 -07:00
49095160e3
Merge pull request #3267 from Nishikoh/rename
...
Rename fastRestore_agent to fastRestoreTool
2020-07-16 15:33:07 -07:00
90654628fb
Merge pull request #3490 from apple/release-6.3
...
Merge Release 6.3
2020-07-10 12:54:37 -07:00
25a61bc15d
Detach a Clone of DR Site without Affecting Replication to the Original DR Site
...
Added description of fdbdr --dstonly to the usage text
2020-07-10 15:14:27 +03:00
4c9df78076
dstonly
2020-07-10 15:13:42 +03:00
bcbca81dfe
Remove ROW_LIMIT_UNLIMITED and BYTE_LIMIT_UNLIMITED from CLIENT_KNOBS
2020-07-04 12:48:44 -07:00
ab3de8700e
Rename fastRestoreAgent to fastRestoreTool
2020-07-03 00:16:39 +09:00
22f7f804b8
Merge branch 'release-6.3' into mengxu/merge-6.3-PR
2020-06-28 11:19:39 -07:00
648c870fec
Merge branch 'release-6.3' of https://github.com/apple/foundationdb into tls-background-handshake
2020-06-26 00:21:57 -07:00
0da4c91ad5
Performance improvements in socket use and buffering which results in TLS network connections using around 3.5x less CPU.
...
PacketBuffers are 16k minimum sizes, using FastAlloc. All calling paths to PacketBuffer creation now pass a size hint if known. P2PNetworkTest improved in many ways, errors go to TraceEvents instead of stdout, error counts are reported in stats, range min/max parameters are combined into "min:max" strings, new options to "stutter" connections by delaying randomly before socket read or write, message size to send is now configured separately for incoming vs outgoing sessions.
2020-06-25 20:44:43 -07:00
c13ba7aeb4
FastRestore:Cosmetic code style improvement
...
Also mute some trace events
2020-06-24 21:25:37 -07:00
2fcef90cbc
FastRestore:Add addPrefix and removePrefix feature
2020-06-18 11:58:42 -07:00
8595813b7d
Merge pull request #3355 from apple/release-6.3
...
Merge Release 6.3 into master branch
2020-06-12 20:08:47 -07:00
dc70591859
Fix fdbbackup help message
2020-06-11 17:26:21 -07:00
57390bfbc5
Merge pull request #3263 from apple/release-6.2
...
Merge release-6.2 into release-6.3
2020-05-28 14:14:38 -07:00
74c11b4c9a
Merge pull request #3262 from apple/release-6.2
...
Merge release-6.2 into release-6.3
2020-05-28 13:43:46 -07:00
382666d21e
Permissions were wrong on installed files. A missing DESTINATION for fdbserver caused the binary to be missing.
2020-05-28 10:33:09 -07:00
135b88d85f
Merge pull request #3254 from apple/release-6.3
...
Merge release-6.3 into master
2020-05-27 20:03:19 -07:00
b3c269351a
Merge pull request #3252 from apple/release-6.2
...
Merge release-6.2 into release-6.3
2020-05-27 19:13:24 -07:00
96f396ec7f
Make sure to build the binary stripping targets when packaging stripped binaries
2020-05-27 16:40:07 -07:00
736a47837d
Merge pull request #3245 from apple/release-6.3
...
Merge release-6.3 into master
2020-05-27 13:27:47 -07:00
d6be580319
Merge branch 'release-6.2' into merge-release-6.2-into-release-6.3
...
# Conflicts:
# documentation/sphinx/source/command-line-interface.rst
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
2020-05-27 11:42:27 -07:00
e4b1489a0e
Merge pull request #3202 from ajbeamon/cmake-use-stripped-binaries-for-packages
...
cmake: use stripped binaries for packages unless generating debug packages
2020-05-27 11:36:38 -07:00
d128252e90
Merge release-6.3 into master
2020-05-22 09:25:32 -07:00
2f9cd383ea
Use stripped executables in packages when not building with debug symbols.
2020-05-21 13:39:55 -07:00
06676fc88f
Merge pull request #3166 from tclinken/6.3-avoid-ub
...
Eliminate some undefined behavior and const_cast usage
2020-05-20 14:36:30 -07:00
3b9e9a9257
Merge pull request #3210 from ajbeamon/remove-include-version-h
...
Remove IncludeVersion.h
2020-05-20 14:12:23 -07:00
4b5e94cf27
Merge pull request #3208 from apple/release-6.2
...
Merge release-6.2 into release-6.3
2020-05-20 13:47:48 -07:00
b589c3c940
Remove IncludeVersions.h, which is no longer needed for cmake-only builds
2020-05-20 13:23:02 -07:00
54c97fe2bc
Backup's ProgramStart command line string would be incomplete if arguments were specified using '='
2020-05-20 09:38:33 -07:00
59e33968c6
Remove erroneous restore options.
2020-05-20 08:57:56 -07:00
3ff2fa9c2a
Removed uses of outdated cpuTicks and errorCounts
2020-05-13 23:27:44 -07:00
667b224fc8
Several changes to reduce unnecessary moves/copies
2020-05-13 19:59:34 -07:00
f17f00fdd5
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2020-05-10 22:33:38 -07:00
c237ca9708
Merge pull request #3081 from jzhou77/backup-cmd
...
Update fdbbackup documentation on new "-p" option
2020-05-04 14:17:34 -07:00
2c5da1c116
Remove "-p" option and use "--partitioned_log_experimental" option
2020-05-04 13:45:54 -07:00
78d644f673
Merge branch 'master' into bugfixes/cmake-includes
2020-05-04 02:11:46 -07:00
37a537c2a3
Fix MacOS compilation error
2020-04-30 18:30:15 -07:00
02aeab1edd
Re-add improperly named options in order to not break compatibility with anything depending on them in a patch release.
2020-04-30 16:23:30 -07:00
07a9a05683
FastRestore:Agent:Fix restore requests
2020-04-30 16:20:20 -07:00
697beb3ebb
Fix fdbrestore parameters with one leading hyphen instead of two, which did not match documentation.
2020-04-30 12:13:35 -07:00
3f510d0653
Merge pull request #3036 from jzhou77/backup-cmd
...
Several bug fixes for new backups
2020-04-29 16:34:59 -07:00
e09394459a
Merge branch 'master' into mengxu/fr-circus
2020-04-28 22:08:29 -07:00
8742cc0ab6
FastRestore:Fix windows build
2020-04-28 19:12:27 -07:00
7d59e53349
Consolidate makePadding()
2020-04-28 15:39:23 -07:00
f5e8345496
FastRestoreAgent:Use atomicParallelRestore to kick off restore
...
Replace the handcrafted version with atomicParallelRestore actor
which is simulation tested
2020-04-27 22:15:00 -07:00
d4509090d4
Generate fastrestore_agent symlink in CMake
2020-04-23 07:55:44 -07:00
af4e0088ba
Merge pull request #2896 from tclinken/atomically-update-dependent-knobs
...
Atomically update dependent knobs
2020-04-08 15:00:49 -07:00
52860043c9
Merge remote-tracking branch 'origin' into atomically-update-dependent-knobs
2020-04-08 12:26:21 -07:00
7ba2cec9ef
Merge branch 'master' of github.com:apple/foundationdb into features/no-make
2020-04-08 10:23:46 -07:00
d4542dbb5a
Delete old build system
2020-04-07 11:03:45 -07:00
9fb3fb9d82
Add pause/resume for new backups
...
To pause/resume the backup workers, the fdbbackup command will write to the
backupPausedKey. Then backup workers noticed the value of the key has been
changed and stops/resumes pulling from TLog.
2020-04-06 14:29:46 -07:00
e4611e8ae4
fix versions.h stupidity
2020-04-06 10:28:55 -07:00
8b5780c36c
don't include source and binary dir
...
This forces users to use include paths from the sources root.
So `#include "Arena.h"` won't work anymore, only
`#include "flow/Arena.h"` will.
2020-04-06 10:13:49 -07:00
bbd2fe62cc
Merge branch 'master' of github.com:apple/foundationdb into features/boost70
2020-04-02 09:21:01 -07:00
4cf796e838
Merge pull request #2891 from jzhou77/backup-cmd
...
Add an option for fdbbackup to use new backup system
2020-04-01 16:12:10 -07:00
884e92bb49
Atomically update dependent knobs
2020-04-01 15:18:49 -07:00
906174e3e8
Add an option for fdbbackup to use new backup system
...
I.e., "-p", or "--partitioned_log" to enable it. By default, old backup system
is used.
2020-03-31 14:54:32 -07:00
3c76722504
Fix valgrind error of invalid memory access
2020-03-30 16:38:22 -07:00
3c32835cce
Fix decoder for unfinished version batch in a log
...
A mutation log's version batch data can be split into multiple blocks, and some
of the blocks can be spread across two mutation logs. Thus, the decoder needs
copy unfinished version batch data from previous file progress to the next one.
2020-03-30 11:34:51 -07:00
28cd38913d
Merge branch 'master' of github.com:apple/foundationdb into features/boost70
2020-03-27 13:44:10 -07:00
feedab02a0
Merge pull request #2855 from xumengpanda/mengxu/fr-api-atomicrestore-PR
...
Add ApiCorrectnessAtomicRestore workload for the new performant restore
2020-03-25 18:05:26 -07:00
a93f13cfd7
Remove redundant restoreRequestDone break in backup.actor
2020-03-25 15:19:46 -07:00
495afe2e0b
Improve how to wati for restore to finish
...
Remove default parameter for atomicRestore as suggested in review.
2020-03-25 13:54:21 -07:00
7831bec2b0
Address review comments on trace events
2020-03-24 10:54:12 -07:00
b18f192831
Fix decode bug of missing mutations
...
After reading a new block, all mutations are sorted by version again, which
can invalidate previously tuple. As a result, the decoded file will miss some
of the mutations.
2020-03-20 20:15:09 -07:00
ca1a4ef9fd
Ignore mutation logs of size 0 in converter
2020-03-20 20:15:08 -07:00
d0a24dd20d
Decode out of order mutations in old mutation logs
...
In the old mutation logs, a version's mutations are serialized as a buffer.
Then the buffer is split into smaller chunks, e.g., 10000 bytes each. When
writting chunks to the final mutation log file, these chunks can be flushed
out of order. For instance, the (version, chunck_part) can be in the order of
(3, 0), (4, 0), (3, 1). As a result, the decoder must read forward to find all
chunks of data for a version.
Another complication is that the files are organized into blocks, where (3, 1)
can be in a subsequent block. This change checks the value size for each
version, if the size is smaller than the right size, the decoder will look
for the missing chucks in the next block.
2020-03-20 20:15:08 -07:00
20df67ee6a
Filter partitioned logs with subset relationship
...
If a log file's progress is not saved, a new log file will be generated
with the same begin version. Then we can have a file that contains a subset
of contents in another log file. During restore, we should filter out files
that their contents are subset of other files.
2020-03-20 20:15:08 -07:00
e15015ee6c
Add mutation log version names
...
I.e., BACKUP_AGENT_MLOG_VERSION for 2001 and PARTITIONED_MLOG_VERSION for 4110.
2020-03-20 20:13:38 -07:00
303df197cf
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/NativeAPI.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogRouter.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/fdbserver.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/flow.vcxproj
# flow/flow.vcxproj.filters
# versions.target
2020-03-06 18:22:46 -08:00
595dd77ed1
Merge remote-tracking branch 'upstream/release-6.2' into certificate-refresh
2020-03-04 20:25:42 -08:00
9b5ef3416e
Refactor TLSParams into TLSConfig + LoadedTLSConfig
...
The idea being that we keep around a TLSConfig that the configuration
that the user has provided, and then when we want to intialize an SSL
context, we ask the TLSConfig to load all certificates and return us a
LoadedTLSConfig that is a concrete set of certificate bytes in memory.
initTLS now just takes the in-memory bytes and applies them to the ssl
context.
This is a large refactor to lead up into certificate refeshing, where we
will periodically check for changes to the certificates, and then
re-load them and apply them to a new SSL context.
2020-03-04 20:14:47 -08:00
cdcb816866
Update fdbbackup/backup.actor.cpp
2020-03-04 16:08:45 -08:00
58e621eca1
Invalid knobs or knob values are treated as warnings rather than errors. Apply this change to backup as well.
2020-03-04 15:50:04 -08:00
c11c24b79d
removed the fdbrpc version of platform.h
2020-02-28 14:56:10 -08:00
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
927cff3317
Report errors on TLS misconfigurations ... or at least try to.
2020-02-20 16:57:29 -08:00
761da5a059
code cleanup
2020-02-19 17:59:45 -08:00
132f5aa9ba
FastRestore:Improve trace name and cosmetic change
2020-02-18 16:41:19 -08:00
31a6ec34b7
Merge branch 'master' into mengxu/fast-restore-agent-PR
2020-02-18 16:17:59 -08:00
3a1e878a9b
Upgrade to boost 1.72
2020-02-14 18:10:13 -08:00
e3061ab713
Use ArenaReader to deserialize MutationRef
2020-02-13 10:28:08 -08:00
72110de7e2
FastRestore:Add trace for quick perf. measurement
2020-02-06 19:48:26 -08:00
09f3690f09
Fix OPEN_FOR_IDE build
2020-02-03 10:42:05 -08:00
690e93145e
Fix some comments
2020-01-22 19:38:46 -08:00
06fb45f32a
FileConverter skips mutation files without tag ID
...
Fileconverter doesn't know the format of old mutation logs.
2020-01-22 19:38:46 -08:00
3854d9a49a
Fix decoder to handle multi-part values
...
When a value (i.e., mutations for a version) is large, it will be split into
multiple key value pairs. This is not handled previously and fixing it also
consolidate the interface of DecodeProgress.
2020-01-22 19:38:46 -08:00
7d1b9fe6d3
Add mutation file decoder
2020-01-22 19:38:46 -08:00
568a8a8e77
Use big endian for mutation log files
...
For each mutation, its version, sub-version, and size are prefixed with big
endian representation. This is required, especially for the first version
variable, because we use 0xFF for padding purpose. A little endian version
number can easily collide with 0xFF, while big endian is guaranteed to have
0x00 as the first byte.
2020-01-22 19:38:46 -08:00
114e153bc8
Use block size encoded in file names
...
The log files have block size encoded in their names and the converter should
use these sizes.
2020-01-22 19:38:46 -08:00
1123157ae0
Ignore mutations large than the end version
2020-01-22 19:38:46 -08:00
b92363bc29
Remove duplicated log files before the conversion
...
Duplicates can happen because backup workers may store the log for
old epochs successfully, but do not update the progress before another
recovery happened. As a result, next epoch will retry and creates
duplicated log files.
2020-01-22 19:38:46 -08:00
Markus Pilman