567a7bd58a
Print a final deleted object count at the end of backup deletion.
2018-12-20 01:24:31 -08:00
354abebf64
Added progress reporting to backup expiration. Simplified backup delete progress.
2018-12-20 00:23:26 -08:00
fcb34a8768
In backup describe, when not using the original cluster to resolve versions to date/time strings the versions will be converted to approximate day deltas from maxLogEndVersion (if available) using core_versionspersecond.
2018-12-19 16:53:39 -08:00
00568f4011
Error code was misleading, added comment.
2018-12-19 13:14:48 -08:00
fd4a62fbfd
Backup expire will fail earlier if force option is needed but not specified.
2018-12-19 10:36:25 -08:00
172c3f2021
Bug fix in backup describe, do not update log begin/end metadata in backup if describe was given a start version override as it can result in incorrect metadata.
2018-12-19 10:35:06 -08:00
afa243bc97
Added fdbbackup expire options to calculate approximate version boundaries based on a number of days prior to the latest log file found in the backup container. This enables expiration operations based on time (with reasonable precision) without accessing the source cluster.
2018-12-18 18:55:44 -08:00
69d847dbbd
Backup describe can now analyze a backup assuming log data starts at a given version. This is used by expire both as an optimization and because doing so enables expire to repair a bad metadata state that can result from a cancelled or failed expire operation from fdbbackup <= 6.0.17. ListLogFiles() and ListRangeFiles() no longer sort results as in most cases the sort is not required and the result set can be very large. Describe will now update minLogBegin metadata to the beginning of the log range known to be present when possible. Several serial log and range list pairings are now done in parallel.
2018-12-18 04:33:37 -08:00
f88d5f56ff
Fix copyright header and use c++11 for now.
2018-12-17 23:50:33 -08:00
779a4b1e3b
Static de-initialization order isn't well thought out so use flushAndExit() instead of returning from main after runNetwork() has been called.
2018-12-17 18:39:14 -08:00
bfab7c150a
Require PageHeader to be 36 bytes, and don't use magic numbers.
2018-12-17 13:37:44 -08:00
9ef9041fba
Bug fix, metadata read futures were moved to a vector but then not waited on.
2018-12-17 13:13:35 -08:00
dac1827d23
Backup describe's "deep scan" mode should only ignore log begin/end versions, not expire and unreliable end versions.
2018-12-16 00:41:38 -08:00
5951e9d577
Added backup URL and exceptions to trace events where applicable.
2018-12-16 00:33:30 -08:00
223b19f5ba
Rewrote backup metadata scheme to fix several design flaws involving cancelled or concurrent operations. Most importantly, before an expire deletes any data it marks the end of the range being deleted as 'unreliable' so further reads of the backup will treat versions before that point as having data holes.
2018-12-16 00:18:13 -08:00
b0f9aab1de
Merge pull request #979 from xumengpanda/mengxu/teamcollection-release-osx-build-fix
...
TeamCollection: Fix build in osX
2018-12-14 15:12:46 -08:00
b4b7f382a7
Fix issues that a newer compiler warned about.
2018-12-14 14:43:50 -08:00
486a7b04fa
TeamCollection: Fix build in osX
...
In osX, we cannot adding unsigned long to a string to append to the string.
2018-12-14 13:44:11 -08:00
0a6d423569
Improve three_data_hall / three_dc descriptions
...
This patch includes descriptions of the transaction log configurations
for these redundancy modes, and corrects what I believe is an incorrect
statement about the minimum number of machines required to make
progress. It also clarifies that commits require synchronous
replication across datacenters.
2018-12-14 16:31:23 -05:00
e18713450f
Remove outdated information on multi-dc config
...
The only published datacenter-aware mode is three_datacenter, which
replicates data 6x (not in triplicate) and does replicate each piece of
data to all three datacenters.
2018-12-14 16:04:44 -05:00
cdc7e83993
Update flow/Trace.cpp
...
Co-Authored-By: alexmiller-apple <35046903+alexmiller-apple@users.noreply.github.com>
2018-12-14 11:30:40 -08:00
4ae701d8a9
minor bugfix to look up correct filename in cache
...
(manually cherry-picked from flat-buffers branch)
2018-12-13 22:21:25 -08:00
0207831fd6
Use abspath when dealing with the simulator file-cache
...
The simulator uses a hash table to cache all open files to make sure
that several simulated processes don't open the file more than once.
This currently doesn't work properly and deleted files are often kept
open forever. As a result, we often ran out of file descriptors.
The problem is luckily quite simple: files are often opened with an
absolute path but later a relativ path is passed for deletion. This
is not working because the map that is used to store the file
descriptors is not aware of paths - so deleted files are often not
removed from this map. The fix that works for us is to just always
work with absolute paths when adding and removing files from this map.
2018-12-13 22:21:06 -08:00
a5401b834b
Integrate JsonTraceLogFormatter into build system + more.
...
Where more is:
* Changed the default format to xml, so that all traces don't suddenly
change as of this commit. Someone is going to need to start passing a
flag to all their servers, but let's worry about that later.
* Changed the copyright headers to be the correct filename.
* Include paths should be specified from the root.
2018-12-13 22:07:16 -08:00
2389cf65f3
fixed escaping bug
2018-12-13 21:50:01 -08:00
aa1eff8346
formatted json log formatter
2018-12-13 21:49:46 -08:00
40890e9dbe
Implemented a json log formatter
2018-12-13 21:46:02 -08:00
a982b9da72
Additional changes from a merge commit.
2018-12-13 17:13:41 -08:00
879cb31fa2
first version of debian build
2018-12-13 15:57:12 -08:00
1007708dfb
RPM package now building correctly
2018-12-13 15:56:45 -08:00
0b8af89826
Added missing dependency between boostProject and boost_target
2018-12-13 15:56:11 -08:00
f13ad0d50a
install fdb service with rpm and add dependencies
2018-12-13 15:55:53 -08:00
c654b1bb61
First version of working rpm packages
2018-12-13 15:55:31 -08:00
8e5f62011a
Compiler flags for Windows (generated via cmake-converter script)
...
I don't have access to a windows machine, so I am flying blind here.
This means that these options are completely untested
2018-12-13 15:54:55 -08:00
f7a87e52d9
Use system boost if installed
2018-12-13 15:53:08 -08:00
550daa05f8
Default configuration now builds.
2018-12-13 15:52:27 -08:00
df0f491c29
Some more improvements to the build and preparations for packaging
2018-12-13 15:04:13 -08:00
f8eac70700
Fixed C bindings DLLEXPORT and Java bindings
2018-12-13 14:59:36 -08:00
5fafeaf219
added bindings to cmake build
2018-12-13 14:54:38 -08:00
4833afea53
CMake compiles fdb c bindings
2018-12-13 14:54:13 -08:00
e70e59a895
Change some file locations.
2018-12-13 14:53:19 -08:00
c06bb33730
CMake compiles all binaries except for bindings
2018-12-13 14:15:44 -08:00
dce290909d
fdbserver now compiling
2018-12-13 14:13:47 -08:00
bd52f8cbd6
Merge pull request #959 from bluk/fix-go-bindings-warning
...
Fix Go bindings possible nested comment warnings
2018-12-13 17:09:40 -05:00
51beb8b48c
fdbrpc compiling with cmake
2018-12-13 14:02:16 -08:00
f5fcb666c5
flow compiling with cmake on OS X
2018-12-13 13:58:06 -08:00
2379552aea
Fix Go bindings possible nested comment warnings
...
When using the Go bindings in a project, you can get compilation
warnings for the `/*` in the comment blocks. These warnings seem
unnecessary.
Warning is:
\# github.com/apple/foundationdb/bindings/go/src/fdb
In file included from _cgo_export.c:4:
cgo-gcc-export-header-prolog:43:1: warning: '/*' within block comment [-Wcomment]
cgo-gcc-export-header-prolog:44:1: warning: '/*' within block comment [-Wcomment]
2018-12-13 15:41:01 -06:00
d9626895b1
Merge pull request #964 from xumengpanda/mengxu/teamcollection-release
...
TeamCollection: Use machine teams to create server teams to increase availability at scale when a machine has multiple servers
2018-12-13 13:18:54 -08:00
79d94f78f1
TeamCollection: Improve code efficiency
...
Further improve code efficiency by
1) Avoid rebuild machine locality map when machine locality is changed.
This may leave the global machine locality map stale.
This is ok as long as we do not use the global map to validate
the machine team follows the locality policy.
2) Use ASSERT_WE_THINK instead of ASSERT to avoid runtime overhead.
ASSERT_WE_THINK will only validate the condition in simulation mode.
Signed-off-by: Meng Xu <meng_xu@apple.com>
2018-12-12 22:38:38 -08:00
e197926c80
TeamCollection: Remove a duplicate function
...
Remove a duplicate function that has different signature.
No functionality change.
Signed-off-by: Meng Xu <meng_xu@apple.com>
2018-12-12 15:21:37 -08:00
Stephen Atherton