94aed513c7
Switched Windows tools within projects to 2017
2019-05-16 15:05:11 -07:00
5f55f3f613
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-10 14:01:52 -07:00
a43071339c
fdbbackup: fix some print/scan format warnings
...
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-05-06 13:35:29 -07:00
af5daed260
FastRestore: Lower case for fastrestore_agent
2019-04-23 16:38:00 -07:00
705b1d1538
FastRestore: Reduce the printf further
...
This help reduce the false positive in correctness.
2019-04-19 09:59:12 -07:00
529ce66b6c
Merge branch 'apple/master' into mengxu/performant-restore-PR
2019-04-18 18:02:45 -07:00
ef04471a66
Fix more unused-variable warnings
2019-04-17 16:04:10 -07:00
4c3ccebe8a
FastRestore: Cleanup code
...
Remove unused code and comments.
2019-04-12 13:49:55 -07:00
1c16f87a4e
Remove trace-calls to printable (in non-workloads)
2019-04-05 13:12:19 -07:00
c4a8a80d6f
Merge branch 'apple/master' into mengxu/performant-restore-PR
2019-04-04 22:51:00 -07:00
56a1128a9b
Enhance cmake's gperftools support
...
Add compiler flags and link flags for gperftools.
2019-04-02 17:34:29 -07:00
49fdc35e5e
Gperftools Profiling fix.
...
Fix a bug and update gperftools compiling flags
The added flags are recommended by gperftools here:
https://github.com/gperftools/gperftools
Verified that heap profiles are saved with the following command:
HEAPPROFILE=/tmp/fdbserver fdbserver [args...]
2019-04-01 14:42:18 -07:00
d68c9ec09e
FastRestore: Fix after merge with master
2019-03-31 22:07:37 -07:00
70d7c289f4
Merge branch 'master' into mengxu/restore/parallel-v7
2019-03-30 22:13:10 -07:00
589fb76826
FastRestore:Attempt to fix old restore
2019-03-30 15:19:30 -07:00
7c02ee6fdd
Fix compiler warning about unreferenced exception variable
2019-03-26 13:43:47 -07:00
56955deed9
Set TRACE_FORMAT network option in fdbbackup
2019-03-22 10:39:13 -07:00
8aab719c22
Merge branch 'master' into feature-backup-json
2019-03-12 18:23:16 -07:00
bc0b2aa040
Merge branch 'release-6.0' of https://github.com/apple/foundationdb
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/BlobStore.actor.cpp
2019-03-12 04:49:12 -07:00
f0eae0295f
Merge branch 'master' of https://github.com/apple/foundationdb into feature-backup-json
2019-03-12 03:35:03 -07:00
880e8643d1
Fix Windows link errors
2019-03-11 17:49:03 -07:00
041b261464
Remove short option name for active snapshot interval
2019-03-11 01:48:51 -07:00
023bbb566f
Renamed backup state enums for clarity, added backup state names. Changed Epochs to EpochSeconds in backup JSON along with some other renaming/moving of fields, and added information about snapshot dispatch. Changed timestamp format for input/output in all backup/restore contexts to be a fully qualified time with timezone offset. Added information about the last snapshot dispatch to backup config and status (not yet populated).
2019-03-10 16:00:01 -07:00
ebffe8c633
print correct pahes in alloc instrumentation
2019-03-08 15:03:17 -08:00
da72306642
several minor bug fixes
2019-03-07 16:49:29 -08:00
2537f26de6
First implementaion of more user-friendly cpack
...
Up unto here this code is only very rudiemantery tested.
This is a firest attempt of making cpack more user-friendly.
The basic idea is to generate a component for package type so
that we can have different paths depending on whether we build
an RPM, a DEB, a TGZ, or a MacOS installer. The cpack package
config file will then chose the correct components to use.
In a later point this should make it possible to build these
with `make packages` and the ugly iteration with calling cmake
between each package would be obsolete. While this solution is
a bit more bloated, it is also much more flexible and it will be
much easier to use.
Another benefit is, that this will get rid of all warnings during
a cpack run
2019-03-07 16:49:29 -08:00
1399aee532
Added --json option to fdbbackup status.
2019-03-06 21:32:46 -08:00
f3391ea413
Merge pull request #1240 from satherton/feature-restore-by-timestamp
...
Restore by timestamp
2019-03-06 16:21:06 -08:00
ca8bbad657
Added --json option to fdbbackup describe. Also added expired percentage indicator to snapshot details.
2019-03-06 14:14:06 -08:00
7778112f6a
Bug fix, restore was using the destination cluster to look up timestamps when printing the backup description instead of (optionally) the original cluster which generated the backup. Made missing cluster file errors more clear.
2019-03-06 02:45:55 -08:00
888c4764d3
Bug fix: fdbrestore was not using a default range set.
2019-03-06 01:57:03 -08:00
28e67aa041
Fixed option name to match documentation.
2019-03-05 23:26:45 -08:00
eed8dc816d
Bug fix: fdbbackup modify on a tag that has never been used before would result in an internal error.
2019-03-05 22:17:37 -08:00
af617d68e6
boost 1.52.0 -> 1.67.0 in all vcxproj files
2019-03-05 22:06:12 -08:00
7f37e25c75
Fdbbackup binary will ignore .debug suffix when determining how it was invoked.
2019-03-05 21:17:56 -08:00
a57ddd29f2
Bug fixes in fdbbackup modify command. Active snapshot interval was wrong. New backup destination containers were not being created. Renamed some variables for clarity.
2019-03-05 18:05:52 -08:00
337a5aa1fe
Updated to new Database API.
2019-03-05 12:44:28 -08:00
2f7d559442
Restore now requires an explicit destination cluster file. Restore target database version can now be specified by a timestamp, which must be resolved using the cluster file for the cluster from which the backup originated.
2019-03-05 12:28:05 -08:00
a5b53d3c0f
More documentation and release note changes for fdbbackup modify. Added release note for #1205 bug fix which was missed.
2019-03-05 10:44:14 -08:00
21f55e1878
Merge pull request #1190 from bnamasivayam/restore-multiple-ranges
...
Add support for restoring multiple ranges.
2019-03-05 10:15:55 -08:00
f6559bbb11
Added help and documentation for fdbbackup modify. Add BackupUID and more parseable BackupURL to fdbbackup status output.
2019-03-05 05:13:48 -08:00
d3377722d5
Added blob store Backup URL parameter 'header' which enables addition of custom HTTP header fields to blob store HTTP requests. Added 'fdbbackup modify' command line tool for changing the backup URL and parameters, default snapshot interval, and/or current snapshot interval of a running backup.
2019-03-05 04:00:11 -08:00
627edd4724
Add force option to 'fdbdr switch'.
2019-03-04 16:36:58 -08:00
a258df32f6
Skip switchover checks for force option.
2019-03-04 15:58:36 -08:00
7cf71b0931
Add some basic checks before doing an atomic switchover.
2019-03-01 14:49:04 -08:00
d0c06ec8c0
Make old build system work on build/cmake/Dockerfile
2019-02-28 14:24:52 -08:00
7eba50b086
Add support for restoring multiple ranges.
2019-02-25 18:00:28 -08:00
9b14aeb156
Tell cmake not to link/install on ide build
2019-02-19 15:16:59 -08:00
78dd80ea8a
Proper fwd decl in BackupAgent
...
Also BackupAgent.h -> BackupAgent.actor.h
2019-02-19 15:16:59 -08:00
b06aa3b8ea
Several bug fixes - debian package working
...
Did some manual minor testing on the debian package
2019-02-15 00:01:42 -08:00
ddddda7328
docker-compose can now build rpm
...
weirdly it still generates a foundationd-unspecified
rpm. However, it is empty and can be ignored for now
2019-02-15 00:01:42 -08:00
75f692b931
simplify actorcompiler and target to compile coveragetool
2019-02-15 00:01:42 -08:00
067a445e06
Replace unused _ variables with wait(success(...))
2019-02-12 17:30:30 -08:00
6da5971e79
Guard all versions.h to not break old WIN32 build
2019-02-08 16:06:00 -08:00
7e26b4ef0d
Address comments from PR
2019-02-07 15:37:04 -08:00
5737349676
Fix weird bug with boost interprocess
...
Strangely, boost interprocess didn't compile with VS 2017.
However, it does compile if it is included as the first thing.
I don't quite know what is happening here, but for now this fix
makes it that I am not blocked
2019-02-07 15:37:04 -08:00
f0a58c6e31
Corrected spacing.
2019-02-06 11:35:50 -08:00
b1e3c9c22d
Corrected spacing issue.
2019-02-06 11:27:26 -08:00
d1145e5c04
Support TLS command line options in backup
2019-01-30 17:36:51 -08:00
550f2e2682
Merge with master to use the latest backup container
...
In fdb 6.0.15, backup container is changed on how to organize the backup data.
The backup made by fdb >6.0.15 has to be restored with fdb > 6.0.15.
Merge with master so that the fast restore uses fdb > 6.0.15
2019-01-30 12:05:15 -08:00
2e11b38f3f
Add print in fast restore agent about backup info
2019-01-30 11:18:11 -08:00
4e9dcb3c74
fastrestore_agent must use lower_case name
2019-01-25 14:01:09 -08:00
a41bcf79d8
Move fastRestore function from fdbserver into fdbbackup
...
fdbbackup cannot depend on fdbserver.
So we cannot use functions defined in fdbserver in fdbbackup or fdbclient.
We move the fastRestore function that initialize the restore request from fdbserver to fdbclient.
It is compilable now.
2019-01-24 19:25:14 -08:00
16f363b234
Add fdbserver dependency to fdbbackup
2019-01-24 18:00:18 -08:00
e721a6a99e
FastRestore:Add restore agent
...
Restore agent will by used by system operator to send out the restore requests to restore workers
Restore agent uses the same code as backup agent, except that its binary name is fastRestore_agent
2019-01-24 16:26:59 -08:00
684a22a52b
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbbackup/backup.actor.cpp
# fdbclient/BackupContainer.actor.cpp
# fdbclient/HTTP.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/BackupCorrectness.actor.cpp
# versions.target
2019-01-09 16:14:46 -08:00
7c5b2ab330
Merge pull request #976 from alexmiller-apple/jsonlogs
...
Allow trace logs to be output as JSON instead of XML
2019-01-09 17:04:50 -05:00
f116678f0d
Bug fix, restore start dry run would not open a trace file.
2019-01-09 03:54:13 -08:00
e7f800b019
Restore start with dry run option no longer requires or uses a database. Removed duplicate and unnecessary code. Dry run execution is more clear, it does not pretend to have actually restored data. Fixed typo in help text.
2019-01-08 19:00:58 -08:00
276ca82af4
Added --loggroup option to all fdbbackup modes.
2019-01-04 00:15:51 -08:00
71f9de2db6
Fix whitespace
2019-01-03 09:12:03 -08:00
7eb6765698
Mention that xml is the default
2019-01-03 08:48:31 -08:00
19c0ccd4c0
Added fdbbackup dump option to list files and sizes in a backup for a given version range.
2019-01-03 02:35:31 -08:00
bce5b03340
Fix whitespace
2019-01-02 15:24:11 -08:00
f64d5321e9
Backup describe, expire, and delete will now clearly indicate when there is no backup at the given URL.
2018-12-20 18:05:23 -08:00
1bca665b29
Document --trace_format flag
2018-12-20 16:22:41 -08:00
03b48fb452
s/--trace-format/--trace_format/
2018-12-20 15:58:26 -08:00
b8df5acc15
Add --trace_format flag to fdbserver
2018-12-20 15:02:01 -08:00
567a7bd58a
Print a final deleted object count at the end of backup deletion.
2018-12-20 01:24:31 -08:00
354abebf64
Added progress reporting to backup expiration. Simplified backup delete progress.
2018-12-20 00:23:26 -08:00
afa243bc97
Added fdbbackup expire options to calculate approximate version boundaries based on a number of days prior to the latest log file found in the backup container. This enables expiration operations based on time (with reasonable precision) without accessing the source cluster.
2018-12-18 18:55:44 -08:00
779a4b1e3b
Static de-initialization order isn't well thought out so use flushAndExit() instead of returning from main after runNetwork() has been called.
2018-12-17 18:39:14 -08:00
c654b1bb61
First version of working rpm packages
2018-12-13 15:55:31 -08:00
df0f491c29
Some more improvements to the build and preparations for packaging
2018-12-13 15:04:13 -08:00
c06bb33730
CMake compiles all binaries except for bindings
2018-12-13 14:15:44 -08:00
4b5d0b4e2c
Merge branch 'release-6.0'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/AsyncFileBlobStore.actor.cpp
# fdbclient/AsyncFileBlobStore.actor.h
# fdbclient/BlobStore.actor.cpp
# fdbclient/BlobStore.h
# fdbclient/HTTP.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbclient/NativeAPI.actor.cpp
# fdbrpc/LoadBalance.actor.h
# fdbrpc/batcher.actor.h
# fdbrpc/fdbrpc.vcxproj
# fdbrpc/sim2.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistributionTracker.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/masterserver.actor.cpp
2018-11-10 13:04:24 -08:00
a654183f63
Merge pull request #791 from ajbeamon/remove-cluster-from-iclientapi
...
Remove cluster from IClientApi (phase 2 of removing DB names)
2018-11-10 10:16:18 -08:00
776b289bfe
Move AsyncFileBlobStore and related files to fdbclient.
2018-10-26 13:49:42 -07:00
0217aed74c
Merge branch 'release-6.0'
...
# Conflicts:
# bindings/go/README.md
# documentation/sphinx/source/release-notes.rst
# fdbserver/MasterProxyServer.actor.cpp
# versions.target
2018-10-15 18:38:51 -07:00
770b4b858b
Minor (and asymptomatic) bug fixes where ints or enumerations are being passed as the bool snapshot argument instead of as the default value which was intended.
2018-10-12 15:03:21 -07:00
c831051474
This removes the idea of clusters from IClientApi.
2018-09-21 15:58:14 -07:00
90301f497f
Merge branch 'release-6.0'
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbrpc/TLSConnection.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/Status.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/StatusWorkload.actor.cpp
# versions.target
2018-09-05 16:06:33 -07:00
b08792ba60
Use the local flowKnobs and clientKnobs rather than const_casting the global ones
2018-08-24 14:16:57 -07:00
d6582b82d1
Allow setting flow knobs in backup.
2018-08-22 10:38:07 -07:00
2a97139d5d
This is the first step in eliminating the usage of database names in our code. The C API remains the same, but underneath that all usage of database names is eliminated.
2018-08-16 10:24:12 -07:00
63b1e85338
Ban `Void _ = wait(...)` constructions, and require just `wait(...)`.
...
There's never any reason to save the value of a Void return, and it's
the easiest source of redefined variable bugs that will creep back in
over time. So just `wait(...)`, it's cleaner that way.
2018-08-14 15:50:26 -07:00
7feb5d8209
Remove including flow.h in actorcompiler.h, and fix resulting breakage.
...
For files that required flow.h, and only got it through actorcompiler.h,
their version of flow.h would have the actorcompiler #defines defined.
Then, if it included a STL/boost file, the same breakage would result.
This needs to not happen, so the include of flow.h in actorcompiler.h
was removed.
2018-08-14 15:50:26 -07:00
fb31a6999f
Rewrite all files to have #include actorcompiler.h as the last include.
2018-08-14 15:50:26 -07:00
535b5701e5
Rewrite all `Void _ = wait(...)` -> `wait(...)`.
...
This takes advantage of the new actorcompiler functionality to avoid
having duplicate definitions of `Void _` when trying to feed the
un-actorompiled source through clang.
2018-08-14 15:50:26 -07:00
6c98aa8aac
Fix not depending on FDBLibTLS.a and default TLS_LIBDIR to a sensible place.
2018-07-05 13:23:20 -07:00
01659e34cc
Move TLS libs into STATIC_LIBS to avoid having a make dependency on them.
...
And fix STATIC_LIBS to be cross platform.
2018-07-04 00:29:53 -07:00
ab255b444f
Added support for specifying location of LibreSSL libraries via define TLS_LIBDIR
2018-07-03 09:01:01 -07:00
132e2d9267
Defined TLS build flags for projects
...
Updated TLS documentation
2018-07-01 22:49:39 -07:00
ef8de426d3
Changed the TLS_DISABLED macro
...
Disable TLS within Windows until working
2018-06-26 12:08:32 -07:00
f8ce1de601
Added support for compiling TLS into binaries
2018-06-20 09:21:23 -07:00
e5488419cc
Attempt to normalize trace events:
...
* Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check.
* Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase.
* Use seconds instead of milliseconds in details.
Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed.
This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-08 11:11:08 -07:00
0bae9880f1
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-21 10:25:11 -08:00
54fc81b260
Improved backup error reporting in backup status. The most recent error for each error type is reported along with how long ago the error occurred, and errors are divided into two categories based on whether or not they occurred since the most recent backup progress.
2018-02-16 19:38:31 -08:00
7bfaffe10d
Merge branch 'release-5.1' of github.com:apple/foundationdb into release-5.1
2018-02-13 16:37:10 -08:00
2d483e7a1c
Fdbbackup will now still write a trace file if --log is specified when doing operations that do not require a cluster. Previous behavior was to rely on createCluster()'s trace file initialization.
2018-02-07 22:36:01 -08:00
9a4d039495
Removed fdbblob (an alias for fdbbackup) generation.
2018-02-07 15:10:39 -08:00
7f0b7311b9
Corrected function name to timeKeeperVersionFromDatetime(). 'Fdbbackup expire' now allows an expire_before version of 0 if explicitly passed by version or by timestamp.
2018-01-23 00:19:51 -08:00
51a1bd9327
Timekeeper lookup improvements, moved both function declartions to BackupContainer.h. VersionFromEpochs() now uses versions/sec to adjust the lookup result to improve accuracy. Conversions in both directions look for the latest record less than the target conversion value, but failing that they will now fall back on any available data point and adjust from there using versions/sec.
2018-01-22 23:57:01 -08:00
02197994b0
Added fdbbackup describe --version_timestamps option which is now required for describe to print version timestamps. Improved help...slightly.
2018-01-22 15:50:28 -08:00
2451e11ff1
ProgramStart wasn't actually be logged in fdbbackup because it happened before the cluster object was created.
2018-01-19 16:57:10 -08:00
93a7eb7474
Fdbbackup restore will now default to the latest usable version in a backup if a specific restore version was not given. Expire will make sure a cluster is provided if either of its timestamp options are given.
2018-01-19 12:19:32 -08:00
93b34a945f
Major usability and performance improvements to backup management. Backup descriptions now calculate and display timestamps using TimeKeeper data (if given a cluster) and restorability of snapshots. Expire now requires a --force option to leave a backup unrestorable or unrestorable after a given point in time, specified by version or timestamp. BackupContainerFilesystem now maintains metadata on key version boundaries in order to avoid large list operations for describe and expire operations. Blob parallel recursive list operations can now take a path (aka prefix) filter function. New describe and expire options are available in fdbbackup.
2018-01-17 04:09:43 -08:00
96cb06cbc7
Bug fixes. Fdbbackup delete was broken. Blobstore backup container deletion would do too much listing before deletions began due to list operations queueing up ahead of and starving the delete operations. Created new knob and blob endpoint limit for concurrent list operations to fix this. Increased blob request timeout default because some requests were taking longer. Crash fixes in blobstore doRequest() which wasn't checking that response object is valid before using it in error conditions. Filesystem-like backup container class (covering blobstore and local dirs) now ignores unrecognized filenames for describe() and expire() operations.
2018-01-05 23:06:39 -08:00
3616035415
Add datetime conversion error message in backup
2017-12-21 14:45:05 -08:00
9dc952e3b2
Added blob credentials details to fdbbackup help.
2017-12-21 02:50:02 -08:00
e3aee45a74
Backup tools and agent now accept blob account credentials via files containing JSON which are specified using command line arguments and/or an environment variable. Improved fdbbackup help, clarifying which options are for which operations. Fdbbackup operations which do not need to use a database no longer require a cluster file parameter. Added eat() commands to StringRef for incrementally tokenizing strings using separator strings.
2017-12-21 01:58:15 -08:00
7caa012fbf
Added snapshot interval option to "fdbbackup start" which defaults to a new knob's value. Added snapshot info to backup status text. Improvements to fdbbackup help.
2017-12-20 00:49:08 -08:00
50c154fed4
Add fdbbackup interface
2017-12-14 13:54:01 -08:00
3ae3200cc5
Untested implementation of feature to log basic backup and DR info to fdbcli status.
2017-11-03 14:02:03 -07:00
fb89ae9f85
added the ability to enable and disable all backup and DR agents from fdbbackup and fdbdr.
2017-10-30 12:35:00 -07:00
36b15f029d
Fixing windows build. Fixing backup.actor.cpp
2017-10-13 17:41:17 -07:00
11517f7bfc
Merge branch 'master' into continuous-backup
...
# Conflicts:
# fdbclient/FileBackupAgent.actor.cpp
2017-10-12 11:03:23 -07:00
6ea9903c82
Merge branch 'release-5.0'
...
# Conflicts:
# fdbbackup/backup.actor.cpp
# fdbserver/ClusterController.actor.cpp
# versions.target
2017-10-01 18:46:44 -07:00
bff8bcc8a9
added help messages for setting a memory limit
2017-09-29 12:38:42 -07:00
e2b65e86ed
added configurable memory limits for backup and dr executables
...
added a default memory limit of 8GB for fdbcli
2017-09-29 10:35:40 -07:00
5e9c6d6b64
Add locality to backup agent and DR agent
2017-09-26 10:19:26 -07:00
02cc8b43c4
More backup cleanup.
2017-09-07 09:04:10 -07:00
c7df951f7c
Using BackupConfig from backup.actor.cpp to reduce intermediate
...
functions.
2017-09-07 08:36:36 -07:00
83810edabc
Backup/Restore tag can be std::string instad of Key.
2017-09-05 11:38:40 -07:00
c7402eb10c
fix: call initSignalSafeUnwind in fdbcli and backup to avoid hangs in slow task profiling.
2017-08-07 15:55:08 -07:00
a674cb4ef4
Initial repository commit
2017-05-25 13:48:44 -07:00
Alvin Moore