2d09d2f68e
re-ordered parameters to fit existing test workloads
2020-07-28 12:39:35 -04:00
71a1b2cb0d
first attempt at making restore pipeline incremental only
2020-07-28 12:39:35 -04:00
207bfec085
added --incremental description to usage messages
2020-07-28 12:39:35 -04:00
3c62651ad6
initial commit to introduce incremental backup only (ignore base snapshot)
2020-07-28 12:39:35 -04:00
e232f805ee
FastRestoreTool: Consume restoreClusterFileDest as fdbrestore does
2020-07-27 22:16:35 -07:00
1a34f5c20e
FastRestoreTool: Fix option list
2020-07-27 21:40:20 -07:00
e0db748fb3
Merge pull request #3403 from satherton/tls-background-handshake
...
TLS handshaking in background threads
2020-07-27 10:55:00 -07:00
b2a3b4fd83
Merge branch 'master' into mengxu/merge-6.3-PR
2020-07-20 11:34:18 -07:00
38cade1146
Merged tag release-6.3.3 into tls-background-handshake.
2020-07-17 23:39:57 -07:00
49095160e3
Merge pull request #3267 from Nishikoh/rename
...
Rename fastRestore_agent to fastRestoreTool
2020-07-16 15:33:07 -07:00
90654628fb
Merge pull request #3490 from apple/release-6.3
...
Merge Release 6.3
2020-07-10 12:54:37 -07:00
25a61bc15d
Detach a Clone of DR Site without Affecting Replication to the Original DR Site
...
Added description of fdbdr --dstonly to the usage text
2020-07-10 15:14:27 +03:00
4c9df78076
dstonly
2020-07-10 15:13:42 +03:00
bcbca81dfe
Remove ROW_LIMIT_UNLIMITED and BYTE_LIMIT_UNLIMITED from CLIENT_KNOBS
2020-07-04 12:48:44 -07:00
ab3de8700e
Rename fastRestoreAgent to fastRestoreTool
2020-07-03 00:16:39 +09:00
22f7f804b8
Merge branch 'release-6.3' into mengxu/merge-6.3-PR
2020-06-28 11:19:39 -07:00
648c870fec
Merge branch 'release-6.3' of https://github.com/apple/foundationdb into tls-background-handshake
2020-06-26 00:21:57 -07:00
0da4c91ad5
Performance improvements in socket use and buffering which results in TLS network connections using around 3.5x less CPU.
...
PacketBuffers are 16k minimum sizes, using FastAlloc. All calling paths to PacketBuffer creation now pass a size hint if known. P2PNetworkTest improved in many ways, errors go to TraceEvents instead of stdout, error counts are reported in stats, range min/max parameters are combined into "min:max" strings, new options to "stutter" connections by delaying randomly before socket read or write, message size to send is now configured separately for incoming vs outgoing sessions.
2020-06-25 20:44:43 -07:00
c13ba7aeb4
FastRestore:Cosmetic code style improvement
...
Also mute some trace events
2020-06-24 21:25:37 -07:00
2fcef90cbc
FastRestore:Add addPrefix and removePrefix feature
2020-06-18 11:58:42 -07:00
8595813b7d
Merge pull request #3355 from apple/release-6.3
...
Merge Release 6.3 into master branch
2020-06-12 20:08:47 -07:00
dc70591859
Fix fdbbackup help message
2020-06-11 17:26:21 -07:00
57390bfbc5
Merge pull request #3263 from apple/release-6.2
...
Merge release-6.2 into release-6.3
2020-05-28 14:14:38 -07:00
74c11b4c9a
Merge pull request #3262 from apple/release-6.2
...
Merge release-6.2 into release-6.3
2020-05-28 13:43:46 -07:00
382666d21e
Permissions were wrong on installed files. A missing DESTINATION for fdbserver caused the binary to be missing.
2020-05-28 10:33:09 -07:00
135b88d85f
Merge pull request #3254 from apple/release-6.3
...
Merge release-6.3 into master
2020-05-27 20:03:19 -07:00
b3c269351a
Merge pull request #3252 from apple/release-6.2
...
Merge release-6.2 into release-6.3
2020-05-27 19:13:24 -07:00
96f396ec7f
Make sure to build the binary stripping targets when packaging stripped binaries
2020-05-27 16:40:07 -07:00
736a47837d
Merge pull request #3245 from apple/release-6.3
...
Merge release-6.3 into master
2020-05-27 13:27:47 -07:00
d6be580319
Merge branch 'release-6.2' into merge-release-6.2-into-release-6.3
...
# Conflicts:
# documentation/sphinx/source/command-line-interface.rst
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
2020-05-27 11:42:27 -07:00
e4b1489a0e
Merge pull request #3202 from ajbeamon/cmake-use-stripped-binaries-for-packages
...
cmake: use stripped binaries for packages unless generating debug packages
2020-05-27 11:36:38 -07:00
d128252e90
Merge release-6.3 into master
2020-05-22 09:25:32 -07:00
2f9cd383ea
Use stripped executables in packages when not building with debug symbols.
2020-05-21 13:39:55 -07:00
06676fc88f
Merge pull request #3166 from tclinken/6.3-avoid-ub
...
Eliminate some undefined behavior and const_cast usage
2020-05-20 14:36:30 -07:00
3b9e9a9257
Merge pull request #3210 from ajbeamon/remove-include-version-h
...
Remove IncludeVersion.h
2020-05-20 14:12:23 -07:00
4b5e94cf27
Merge pull request #3208 from apple/release-6.2
...
Merge release-6.2 into release-6.3
2020-05-20 13:47:48 -07:00
b589c3c940
Remove IncludeVersions.h, which is no longer needed for cmake-only builds
2020-05-20 13:23:02 -07:00
54c97fe2bc
Backup's ProgramStart command line string would be incomplete if arguments were specified using '='
2020-05-20 09:38:33 -07:00
59e33968c6
Remove erroneous restore options.
2020-05-20 08:57:56 -07:00
3ff2fa9c2a
Removed uses of outdated cpuTicks and errorCounts
2020-05-13 23:27:44 -07:00
667b224fc8
Several changes to reduce unnecessary moves/copies
2020-05-13 19:59:34 -07:00
f17f00fdd5
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2020-05-10 22:33:38 -07:00
c237ca9708
Merge pull request #3081 from jzhou77/backup-cmd
...
Update fdbbackup documentation on new "-p" option
2020-05-04 14:17:34 -07:00
2c5da1c116
Remove "-p" option and use "--partitioned_log_experimental" option
2020-05-04 13:45:54 -07:00
78d644f673
Merge branch 'master' into bugfixes/cmake-includes
2020-05-04 02:11:46 -07:00
37a537c2a3
Fix MacOS compilation error
2020-04-30 18:30:15 -07:00
02aeab1edd
Re-add improperly named options in order to not break compatibility with anything depending on them in a patch release.
2020-04-30 16:23:30 -07:00
07a9a05683
FastRestore:Agent:Fix restore requests
2020-04-30 16:20:20 -07:00
697beb3ebb
Fix fdbrestore parameters with one leading hyphen instead of two, which did not match documentation.
2020-04-30 12:13:35 -07:00
3f510d0653
Merge pull request #3036 from jzhou77/backup-cmd
...
Several bug fixes for new backups
2020-04-29 16:34:59 -07:00
e09394459a
Merge branch 'master' into mengxu/fr-circus
2020-04-28 22:08:29 -07:00
8742cc0ab6
FastRestore:Fix windows build
2020-04-28 19:12:27 -07:00
7d59e53349
Consolidate makePadding()
2020-04-28 15:39:23 -07:00
f5e8345496
FastRestoreAgent:Use atomicParallelRestore to kick off restore
...
Replace the handcrafted version with atomicParallelRestore actor
which is simulation tested
2020-04-27 22:15:00 -07:00
d4509090d4
Generate fastrestore_agent symlink in CMake
2020-04-23 07:55:44 -07:00
af4e0088ba
Merge pull request #2896 from tclinken/atomically-update-dependent-knobs
...
Atomically update dependent knobs
2020-04-08 15:00:49 -07:00
52860043c9
Merge remote-tracking branch 'origin' into atomically-update-dependent-knobs
2020-04-08 12:26:21 -07:00
7ba2cec9ef
Merge branch 'master' of github.com:apple/foundationdb into features/no-make
2020-04-08 10:23:46 -07:00
d4542dbb5a
Delete old build system
2020-04-07 11:03:45 -07:00
9fb3fb9d82
Add pause/resume for new backups
...
To pause/resume the backup workers, the fdbbackup command will write to the
backupPausedKey. Then backup workers noticed the value of the key has been
changed and stops/resumes pulling from TLog.
2020-04-06 14:29:46 -07:00
e4611e8ae4
fix versions.h stupidity
2020-04-06 10:28:55 -07:00
8b5780c36c
don't include source and binary dir
...
This forces users to use include paths from the sources root.
So `#include "Arena.h"` won't work anymore, only
`#include "flow/Arena.h"` will.
2020-04-06 10:13:49 -07:00
bbd2fe62cc
Merge branch 'master' of github.com:apple/foundationdb into features/boost70
2020-04-02 09:21:01 -07:00
4cf796e838
Merge pull request #2891 from jzhou77/backup-cmd
...
Add an option for fdbbackup to use new backup system
2020-04-01 16:12:10 -07:00
884e92bb49
Atomically update dependent knobs
2020-04-01 15:18:49 -07:00
906174e3e8
Add an option for fdbbackup to use new backup system
...
I.e., "-p", or "--partitioned_log" to enable it. By default, old backup system
is used.
2020-03-31 14:54:32 -07:00
3c76722504
Fix valgrind error of invalid memory access
2020-03-30 16:38:22 -07:00
3c32835cce
Fix decoder for unfinished version batch in a log
...
A mutation log's version batch data can be split into multiple blocks, and some
of the blocks can be spread across two mutation logs. Thus, the decoder needs
copy unfinished version batch data from previous file progress to the next one.
2020-03-30 11:34:51 -07:00
28cd38913d
Merge branch 'master' of github.com:apple/foundationdb into features/boost70
2020-03-27 13:44:10 -07:00
feedab02a0
Merge pull request #2855 from xumengpanda/mengxu/fr-api-atomicrestore-PR
...
Add ApiCorrectnessAtomicRestore workload for the new performant restore
2020-03-25 18:05:26 -07:00
a93f13cfd7
Remove redundant restoreRequestDone break in backup.actor
2020-03-25 15:19:46 -07:00
495afe2e0b
Improve how to wati for restore to finish
...
Remove default parameter for atomicRestore as suggested in review.
2020-03-25 13:54:21 -07:00
7831bec2b0
Address review comments on trace events
2020-03-24 10:54:12 -07:00
b18f192831
Fix decode bug of missing mutations
...
After reading a new block, all mutations are sorted by version again, which
can invalidate previously tuple. As a result, the decoded file will miss some
of the mutations.
2020-03-20 20:15:09 -07:00
ca1a4ef9fd
Ignore mutation logs of size 0 in converter
2020-03-20 20:15:08 -07:00
d0a24dd20d
Decode out of order mutations in old mutation logs
...
In the old mutation logs, a version's mutations are serialized as a buffer.
Then the buffer is split into smaller chunks, e.g., 10000 bytes each. When
writting chunks to the final mutation log file, these chunks can be flushed
out of order. For instance, the (version, chunck_part) can be in the order of
(3, 0), (4, 0), (3, 1). As a result, the decoder must read forward to find all
chunks of data for a version.
Another complication is that the files are organized into blocks, where (3, 1)
can be in a subsequent block. This change checks the value size for each
version, if the size is smaller than the right size, the decoder will look
for the missing chucks in the next block.
2020-03-20 20:15:08 -07:00
20df67ee6a
Filter partitioned logs with subset relationship
...
If a log file's progress is not saved, a new log file will be generated
with the same begin version. Then we can have a file that contains a subset
of contents in another log file. During restore, we should filter out files
that their contents are subset of other files.
2020-03-20 20:15:08 -07:00
e15015ee6c
Add mutation log version names
...
I.e., BACKUP_AGENT_MLOG_VERSION for 2001 and PARTITIONED_MLOG_VERSION for 4110.
2020-03-20 20:13:38 -07:00
303df197cf
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# bindings/c/test/mako/mako.c
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbclient/NativeAPI.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/LogRouter.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/fdbserver.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/flow.vcxproj
# flow/flow.vcxproj.filters
# versions.target
2020-03-06 18:22:46 -08:00
595dd77ed1
Merge remote-tracking branch 'upstream/release-6.2' into certificate-refresh
2020-03-04 20:25:42 -08:00
9b5ef3416e
Refactor TLSParams into TLSConfig + LoadedTLSConfig
...
The idea being that we keep around a TLSConfig that the configuration
that the user has provided, and then when we want to intialize an SSL
context, we ask the TLSConfig to load all certificates and return us a
LoadedTLSConfig that is a concrete set of certificate bytes in memory.
initTLS now just takes the in-memory bytes and applies them to the ssl
context.
This is a large refactor to lead up into certificate refeshing, where we
will periodically check for changes to the certificates, and then
re-load them and apply them to a new SSL context.
2020-03-04 20:14:47 -08:00
cdcb816866
Update fdbbackup/backup.actor.cpp
2020-03-04 16:08:45 -08:00
58e621eca1
Invalid knobs or knob values are treated as warnings rather than errors. Apply this change to backup as well.
2020-03-04 15:50:04 -08:00
c11c24b79d
removed the fdbrpc version of platform.h
2020-02-28 14:56:10 -08:00
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
927cff3317
Report errors on TLS misconfigurations ... or at least try to.
2020-02-20 16:57:29 -08:00
761da5a059
code cleanup
2020-02-19 17:59:45 -08:00
132f5aa9ba
FastRestore:Improve trace name and cosmetic change
2020-02-18 16:41:19 -08:00
31a6ec34b7
Merge branch 'master' into mengxu/fast-restore-agent-PR
2020-02-18 16:17:59 -08:00
3a1e878a9b
Upgrade to boost 1.72
2020-02-14 18:10:13 -08:00
e3061ab713
Use ArenaReader to deserialize MutationRef
2020-02-13 10:28:08 -08:00
72110de7e2
FastRestore:Add trace for quick perf. measurement
2020-02-06 19:48:26 -08:00
09f3690f09
Fix OPEN_FOR_IDE build
2020-02-03 10:42:05 -08:00
690e93145e
Fix some comments
2020-01-22 19:38:46 -08:00
06fb45f32a
FileConverter skips mutation files without tag ID
...
Fileconverter doesn't know the format of old mutation logs.
2020-01-22 19:38:46 -08:00
3854d9a49a
Fix decoder to handle multi-part values
...
When a value (i.e., mutations for a version) is large, it will be split into
multiple key value pairs. This is not handled previously and fixing it also
consolidate the interface of DecodeProgress.
2020-01-22 19:38:46 -08:00
7d1b9fe6d3
Add mutation file decoder
2020-01-22 19:38:46 -08:00
568a8a8e77
Use big endian for mutation log files
...
For each mutation, its version, sub-version, and size are prefixed with big
endian representation. This is required, especially for the first version
variable, because we use 0xFF for padding purpose. A little endian version
number can easily collide with 0xFF, while big endian is guaranteed to have
0x00 as the first byte.
2020-01-22 19:38:46 -08:00
114e153bc8
Use block size encoded in file names
...
The log files have block size encoded in their names and the converter should
use these sizes.
2020-01-22 19:38:46 -08:00
1123157ae0
Ignore mutations large than the end version
2020-01-22 19:38:46 -08:00
b92363bc29
Remove duplicated log files before the conversion
...
Duplicates can happen because backup workers may store the log for
old epochs successfully, but do not update the progress before another
recovery happened. As a result, next epoch will retry and creates
duplicated log files.
2020-01-22 19:38:46 -08:00
4327435601
Fix a data corruption bug
...
VersionedData used to include a MutationRef, which is made from BinaryReader.
Unfortunately, the StringRef inside MutationRef points a memory allocated from
the BinaryReader's arena, which is free'd after BinaryReader is destroyed.
Change to use a StringRef pointing to the serialized mutation solves this bug.
2020-01-22 19:38:46 -08:00
c1748c0460
Code refactoring
...
The BackupWorker produces files not in blocks, which should be fixed.
2020-01-22 19:38:46 -08:00
84a49cf389
Add merge sorting mutations from multiple files
...
This is implemented in MutationFilesReadProgress.
2020-01-22 19:38:46 -08:00
5ab9d0925c
Add namespace file_converter
2020-01-22 19:38:46 -08:00
7f7ec99170
Serialize and deserialize new backup files
...
The BackupWorker writes files that can be read by FileConverter. Move
StringRefReader to the header file for reuse in FileConverter.
2020-01-22 19:38:46 -08:00
5ac63ec526
Apply clang-format
2020-01-22 19:38:46 -08:00
674b468609
Add more parameter parsing
2020-01-22 19:38:46 -08:00
2707ab3eba
Add fdbconvert command line utility
...
fdbconvert is intended to convert new backup files which are tagged mutation
logs to old backup format. The actual conversion is not included in this commit
and will be added in future commits.
Note that the BackupContainer needs to be updated to support new backup files,
which is also not included in this commit.
2020-01-22 19:38:46 -08:00
7628d04fb9
Merge branch 'release-6.2' of github.com:apple/foundationdb into release_6.2_merge
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
2020-01-09 07:21:16 -08:00
4ff058e86b
Backup and DR layer status document generation now uses snapshot reads for all keys read to avoid unnecessary conflicts when read during a status update or cleanup transaction. Since many of the keys read use wrapper functions, all of the underlying functions in BackupAgentBase and its two implementations also required a snapshot mode argument. All snapshot arguments default to false to match the underlying FDB API get/getrange methods.
2019-12-19 00:29:35 -08:00
3bf971ba8b
Merge branch 'release-6.2' of github.com:apple/foundationdb into release_6.2_merge
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/storageserver.actor.cpp
2019-12-12 07:13:12 -08:00
363feafacd
Changed the name of a cmake functions
...
Added working directory for cmake command
2019-12-11 13:59:03 -08:00
aece864143
Added support for creating symlinks to fdbbackup within bin and packages/bin directory
2019-12-10 11:44:45 -08:00
ed8d3f163c
Rename hgVersion to sourceVersion.
2019-11-15 12:26:51 -08:00
4de60fc437
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/TLogServer.actor.cpp
2019-11-01 15:48:04 -07:00
f175ed30b3
Cleanup the fdbbackup cleanup command output. Add cleanup to the usage output printed for fdbbackup.
2019-10-31 09:52:21 -07:00
d4de608bb6
Fix OPEN_FOR_IDE build
2019-10-25 10:42:22 -07:00
7599dd0093
Merge pull request #2205 from jzhou77/backup-fix
...
Fix format string with more portable code
2019-10-02 20:03:35 -07:00
ceb39c0279
Fix format string with more portable code
2019-10-02 15:25:14 -07:00
d0147e5e5d
Merge branch 'release-6.2' into mengxu/merge-release620-to-master-v3
...
Resolved Conflicts:
documentation/sphinx/source/release-notes.rst
fdbserver/DataDistribution.actor.cpp
versions.target
2019-10-02 13:22:56 -07:00
9463b91594
Update fdbbackup/backup.actor.cpp
...
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2019-09-30 13:16:55 -07:00
4d659662b8
made cleanup handle retries better
2019-09-30 12:44:20 -07:00
ef01ad2ed8
optimized log range clearing to clear everything for each possible hash (256 clears) if that would be more efficient than one clear per second that has elapsed
...
aborting a DR without the —cleanup flag will still attempt to cleanup for 30 seconds before giving up
added a cleanup command to fdbbackup which can remove mutations from orphaned DRs which were stopped without the —cleanup flag
2019-09-27 18:32:27 -07:00
9a64dc7f24
Merge pull request #1792 from ajbeamon/remove-dl-iterate-phdr-hack
...
Remove dl_iterate_phdr caching used by slow task profiler
2019-09-17 16:51:30 -07:00
b495cc697b
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-09-13 09:25:08 -07:00
bd7678e71b
Remove --object-serializer help text
2019-09-05 11:57:59 -07:00
c2355f721e
Merge branch 'master' into mengxu/performant-restore-PR
2019-09-04 17:11:42 -07:00
d160810662
FastRestore:Resolve review comments
2019-09-04 16:48:43 -07:00
e465cbef07
Merge branch 'master' into remove-dl-iterate-phdr-hack
...
# Conflicts:
# fdbserver/fdbserver.actor.cpp
2019-08-22 10:13:23 -07:00
f02799455e
Add --loggroup to fdbserver and fdbbackup help text.
2019-08-19 12:59:14 -07:00
7ff46e6772
Merge branch 'master' into mengxu/performant-restore-PR
2019-08-07 20:31:56 -07:00
370ba8b841
Remove --object-serializer flag from executables
2019-08-06 09:25:40 -07:00
3b54363780
FastRestore:Apply Clang-format
2019-08-01 18:09:12 -07:00
7ccaeddf05
Merge branch 'master' into mengxu/performant-restore-PR
2019-08-01 13:23:17 -07:00
b5d2234a13
Merge branch 'release-6.1' into merge-release-6.1-into-master
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbserver/MoveKeys.actor.cpp
# flow/FastAlloc.h
# versions.target
2019-07-30 16:23:42 -07:00
9e1042d903
Bug fix, restore with --dryrun option would still require the --dest_cluster_file option even though it does not connect to a cluster.
2019-07-29 16:21:12 -07:00
b644f15b87
Bug fix: fdbrestore commands other than "start" were using default cluster file argument handling (but without the -C flag) instead of using the --dest_cluster_file argument.
2019-07-29 13:19:28 -07:00
b0c31f28af
FastRestore:Fix bug that blocks restore
...
1) Should recruit only configured number of roles;
2) Should never register a restore master interface as a restore worker (loader or applier) interface.
2019-07-25 17:55:37 -07:00
45083edf74
Merge branch 'master' into mengxu/performant-restore-PR
...
Fix conflicts as well.
2019-07-25 10:46:11 -07:00
1ac2d01b03
Merge remote-tracking branch 'upstream/master' into flatbuffers-fixes2
2019-07-18 09:50:08 -07:00
d5caf0c1b4
Merge branch 'flatbuffers-fixes2' of github.com:mpilman/foundationdb into flatbuffers-fixes2
2019-07-16 14:47:40 -07:00
d5051b08dd
Make trace event field lengths (and total event sizes) default knobified and configurable. Add a transaction option to control the field length of transaction debug logging. Make the program start command line field less likely to be truncated.
2019-07-12 16:12:35 -07:00
5be0c47409
Add some missing help text
2019-07-12 10:46:46 -07:00
269f077196
Add --object-serializer flag to fdbbackup etc
2019-07-12 10:46:45 -07:00
a5a6f8431c
Add a random UID to TransactionMetrics in case a client opens multiple connections and also a field to indicate whether the connection is internal. Convert some of the metrics to our Counter object instead of running totals.
2019-07-08 14:01:04 -07:00
6a899ddff3
Remove dl_iterate_phdr results caching that is used by slow task profiler, instead favoring disabling and reenabling profiling around the call. Add a mechanism to handle deferred profile requests.
2019-07-03 12:48:36 -07:00
37c1df2491
Merge pull request #1705 from bnamasivayam/suspend-process
...
Extend RebootRequest API to include time to suspend the process befor…
2019-06-20 17:36:25 -07:00
68ce9a5e75
ProtocolVersion type - second try
2019-06-18 17:55:27 -07:00
20e3edeb0a
Merge branch 'release-6.1'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/storageserver.actor.cpp
# versions.target
2019-06-14 12:42:59 -07:00
5eb833759e
Extend RebootRequest API to include time to suspend the process before reboot. This is intended to be used for testing purposes to simulate failures.
2019-06-14 11:35:38 -07:00
235d24ee26
Bug fix: fdbrestore abort, wait, and status were not using the --dest_cluster_file argument.
2019-06-13 17:42:11 -07:00
3fcb6ec0a1
FastRestore:Refactor RestoreLoader and fix bugs
...
Refactor RestoreLoader code and
Fix a bug in notifying restore finish.
2019-06-04 21:53:31 -07:00
477fd152c0
FastRestore:Refactor code
...
1) Use the runRYWTransaction for simple DB access
2) Replace some printf with TraceEvent
3) Remove printf not used in debugging
4) Avoid wait inside the condition in loop-choose-when for
the core routine of restore worker, loader and applier.
5) Rename Restore.actor.cpp to RestoreWorker.actor.cpp since
the file only has functionalities related to restore worker.
Passed correctness test
2019-06-04 11:22:47 -07:00
a3f61e6df7
FastRestore:Rafctor:Reduce code size
...
1) Use runRYWTransaction to replace the loop-try style;
2) Remove unnecessary printf
3) Do not mistakenly send reply twice.
2019-05-29 17:03:50 -07:00
603721e125
Merge branch 'master' into thread-safe-random-number-generation
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/AsyncFileCached.actor.h
# fdbrpc/genericactors.actor.cpp
# fdbrpc/sim2.actor.cpp
# fdbserver/DiskQueue.actor.cpp
# fdbserver/workloads/BulkSetup.actor.h
# flow/ActorCollection.actor.cpp
# flow/Net2.actor.cpp
# flow/Trace.cpp
# flow/flow.cpp
2019-05-23 08:35:47 -07:00
3acaa7343e
Enabled C++17 for all Windows projects
...
Set Visual Studio version to 2017 (first version to support C++17)
2019-05-16 17:44:13 -07:00
94aed513c7
Switched Windows tools within projects to 2017
2019-05-16 15:05:11 -07:00
5f55f3f613
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-10 14:01:52 -07:00
a43071339c
fdbbackup: fix some print/scan format warnings
...
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-06 13:35:29 -07:00
af5daed260
FastRestore: Lower case for fastrestore_agent
2019-04-23 16:38:00 -07:00
705b1d1538
FastRestore: Reduce the printf further
...
This help reduce the false positive in correctness.
2019-04-19 09:59:12 -07:00
529ce66b6c
Merge branch 'apple/master' into mengxu/performant-restore-PR
2019-04-18 18:02:45 -07:00
ef04471a66
Fix more unused-variable warnings
2019-04-17 16:04:10 -07:00
4c3ccebe8a
FastRestore: Cleanup code
...
Remove unused code and comments.
2019-04-12 13:49:55 -07:00
1c16f87a4e
Remove trace-calls to printable (in non-workloads)
2019-04-05 13:12:19 -07:00
c4a8a80d6f
Merge branch 'apple/master' into mengxu/performant-restore-PR
2019-04-04 22:51:00 -07:00
56a1128a9b
Enhance cmake's gperftools support
...
Add compiler flags and link flags for gperftools.
2019-04-02 17:34:29 -07:00
49fdc35e5e
Gperftools Profiling fix.
...
Fix a bug and update gperftools compiling flags
The added flags are recommended by gperftools here:
https://github.com/gperftools/gperftools
Verified that heap profiles are saved with the following command:
HEAPPROFILE=/tmp/fdbserver fdbserver [args...]
2019-04-01 14:42:18 -07:00
d68c9ec09e
FastRestore: Fix after merge with master
2019-03-31 22:07:37 -07:00
70d7c289f4
Merge branch 'master' into mengxu/restore/parallel-v7
2019-03-30 22:13:10 -07:00
589fb76826
FastRestore:Attempt to fix old restore
2019-03-30 15:19:30 -07:00
7c02ee6fdd
Fix compiler warning about unreferenced exception variable
2019-03-26 13:43:47 -07:00
56955deed9
Set TRACE_FORMAT network option in fdbbackup
2019-03-22 10:39:13 -07:00
8aab719c22
Merge branch 'master' into feature-backup-json
2019-03-12 18:23:16 -07:00
bc0b2aa040
Merge branch 'release-6.0' of https://github.com/apple/foundationdb
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/BlobStore.actor.cpp
2019-03-12 04:49:12 -07:00
f0eae0295f
Merge branch 'master' of https://github.com/apple/foundationdb into feature-backup-json
2019-03-12 03:35:03 -07:00
880e8643d1
Fix Windows link errors
2019-03-11 17:49:03 -07:00
041b261464
Remove short option name for active snapshot interval
2019-03-11 01:48:51 -07:00
023bbb566f
Renamed backup state enums for clarity, added backup state names. Changed Epochs to EpochSeconds in backup JSON along with some other renaming/moving of fields, and added information about snapshot dispatch. Changed timestamp format for input/output in all backup/restore contexts to be a fully qualified time with timezone offset. Added information about the last snapshot dispatch to backup config and status (not yet populated).
2019-03-10 16:00:01 -07:00
ebffe8c633
print correct pahes in alloc instrumentation
2019-03-08 15:03:17 -08:00
da72306642
several minor bug fixes
2019-03-07 16:49:29 -08:00
2537f26de6
First implementaion of more user-friendly cpack
...
Up unto here this code is only very rudiemantery tested.
This is a firest attempt of making cpack more user-friendly.
The basic idea is to generate a component for package type so
that we can have different paths depending on whether we build
an RPM, a DEB, a TGZ, or a MacOS installer. The cpack package
config file will then chose the correct components to use.
In a later point this should make it possible to build these
with `make packages` and the ugly iteration with calling cmake
between each package would be obsolete. While this solution is
a bit more bloated, it is also much more flexible and it will be
much easier to use.
Another benefit is, that this will get rid of all warnings during
a cpack run
2019-03-07 16:49:29 -08:00
1399aee532
Added --json option to fdbbackup status.
2019-03-06 21:32:46 -08:00
f3391ea413
Merge pull request #1240 from satherton/feature-restore-by-timestamp
...
Restore by timestamp
2019-03-06 16:21:06 -08:00
ca8bbad657
Added --json option to fdbbackup describe. Also added expired percentage indicator to snapshot details.
2019-03-06 14:14:06 -08:00
7778112f6a
Bug fix, restore was using the destination cluster to look up timestamps when printing the backup description instead of (optionally) the original cluster which generated the backup. Made missing cluster file errors more clear.
2019-03-06 02:45:55 -08:00
888c4764d3
Bug fix: fdbrestore was not using a default range set.
2019-03-06 01:57:03 -08:00
28e67aa041
Fixed option name to match documentation.
2019-03-05 23:26:45 -08:00
eed8dc816d
Bug fix: fdbbackup modify on a tag that has never been used before would result in an internal error.
2019-03-05 22:17:37 -08:00
af617d68e6
boost 1.52.0 -> 1.67.0 in all vcxproj files
2019-03-05 22:06:12 -08:00
7f37e25c75
Fdbbackup binary will ignore .debug suffix when determining how it was invoked.
2019-03-05 21:17:56 -08:00
a57ddd29f2
Bug fixes in fdbbackup modify command. Active snapshot interval was wrong. New backup destination containers were not being created. Renamed some variables for clarity.
2019-03-05 18:05:52 -08:00
337a5aa1fe
Updated to new Database API.
2019-03-05 12:44:28 -08:00
2f7d559442
Restore now requires an explicit destination cluster file. Restore target database version can now be specified by a timestamp, which must be resolved using the cluster file for the cluster from which the backup originated.
2019-03-05 12:28:05 -08:00
a5b53d3c0f
More documentation and release note changes for fdbbackup modify. Added release note for #1205 bug fix which was missed.
2019-03-05 10:44:14 -08:00
21f55e1878
Merge pull request #1190 from bnamasivayam/restore-multiple-ranges
...
Add support for restoring multiple ranges.
2019-03-05 10:15:55 -08:00
f6559bbb11
Added help and documentation for fdbbackup modify. Add BackupUID and more parseable BackupURL to fdbbackup status output.
2019-03-05 05:13:48 -08:00
d3377722d5
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 04:00:11 -08:00
627edd4724
Add force option to 'fdbdr switch'.
2019-03-04 16:36:58 -08:00
Jon Fu