7feb10e5da
Merge pull request #2965 from tclinken/add-advanceversion-command
...
Added advanceversion command to fdbcli
2020-04-22 10:40:47 -07:00
9b2f6d5c13
FastRestore:Loader:Enable sending message out of order
2020-04-22 08:57:11 -07:00
d21da5065a
FastRestore:Loader:Merge MutationsVec and LogMessageVersionVec into VersionedMutationsVec
...
Remove the actor that sends one mutation message batch in the previous commit,
because that actor no longer reduces the code complexity.
2020-04-21 22:05:34 -07:00
9bfc5bbea8
Check RestorableFileSet's key ranges in simulation
...
Ranges written in the manifest file should match with actual file content.
2020-04-21 12:55:40 -07:00
0938e45c6a
Set continuous version to invalidVersion when snapshot version is the target version
2020-04-20 22:26:42 -07:00
930b175c4c
Add range files' key ranges to RestorableFileSet
...
Also add continuous logs' begin and end version in RestorableFileSet.
2020-04-20 22:26:42 -07:00
a2b867c6f9
Fix a unit test failure
2020-04-20 22:26:42 -07:00
3063611355
Write range files' begin & end keys to manifest file
...
This information can be very useful in knowing the content in these files,
especially for restores.
2020-04-20 22:26:42 -07:00
e51d0365cf
Cleanup: Use the shutdown callback for destroying TLS state.
2020-04-20 13:16:16 -07:00
da8e47ea25
Merge remote-tracking branch 'upstream/release-6.2' into tls-background-eio-thread
2020-04-20 13:15:05 -07:00
4c66c8c377
Fix backup progress calculation
...
The oldest epoch the master gets can assume its begin version is 1, which can
be wrong. In this case, we use the saved backup progress to "true-up" the real
begin version.
2020-04-20 11:06:46 -07:00
2ce539ef6d
Respect flow<->fdbrpc module boundaries.
...
Which fixes a compilation error due to a circular dependency between
flow.a and fdbrpc.a. However, this is now done at the cost of newNet2
users have to remember to add Net2FileSystem::stop() as a callback.
2020-04-20 02:53:07 -07:00
00188c3157
Merge branch 'master' into mengxu/fr-code-improvement-PR
2020-04-19 11:52:42 -07:00
3f3a728bb3
Merge pull request #2970 from xumengpanda/mengxu/fr-range-versions-PR
...
Fix:New backup and restore: Mutations before min range version may not be complete
2020-04-19 09:49:07 -07:00
94b4f78ea9
Fix clients crashing in TLS code on exit.
...
If client code initiates an FDB operation to a TLS cluster, and then
immediately exits the main thread, then OpenSSL's atexit handler would
potentially run while the network thread is attempting to do TLS
operations, and thus crash.
This commit removes the OpenSSL atexit hander, and instead relies on a
client intentionally ending the network thread to do TLS cleanup. If
the client code exits without stopping the network thread, then we'll
never free OpenSSL data structures, which is the safer thing to do.
2020-04-18 15:48:02 -07:00
82ae82c98f
Move MAX_VERSION to FDBTypes.h
2020-04-17 18:46:04 -07:00
ba3e2af473
Merge commit '5288033bcfe40c3ade97c8bf2d04cf31b3f16cb1' into feature-tree-broadcast
2020-04-17 15:17:37 -07:00
b011efa8df
Merge remote-tracking branch 'upstream/master' into private-keyspace-framework
2020-04-16 10:58:04 -07:00
93400d25c8
Added advanceversion command to fdbcli
2020-04-15 20:01:01 -07:00
d6c1baa784
FastRestore:Filter out log mutations whose version is smaller than range mutation version
2020-04-15 19:45:03 -07:00
456b794d24
Reformat the file
2020-04-15 12:24:44 -07:00
b5d7780293
Merge pull request #2849 from ajbeamon/remove-internal-tools-from-client-list
...
Don't include internal tools in the status client list
2020-04-14 16:49:30 -07:00
e104a2e3a6
Merge commit 'cf01233f28a2c42908656a39f458a4475c1d44a3' into run-loop-busy-profiler
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbclient/NativeAPI.actor.h
# fdbserver/fdbserver.actor.cpp
# flow/Net2.actor.cpp
2020-04-14 14:02:24 -07:00
ae1de060c2
Reformat SpecialKeyRangeBaseImpl constructor
2020-04-14 09:10:40 -07:00
061bcd2fb4
FastRestore:Replace typeString with safe getTypeString func
...
Also fix compilation error in previous commit
2020-04-13 15:15:54 -07:00
65620b00f3
Merge remote-tracking branch 'upstream/master' into private-keyspace-framework
2020-04-13 14:21:07 -07:00
c8ab69cce6
Fix incorrect if/ASSERT logic from last minute cleanups.
...
My test code serialized UIDs and Tags, so that I could compare and
verify I got the same thing, and I undid this right before committing,
but forgot to also change the `if` to not rely on UIDs being first also.
2020-04-12 19:38:51 -07:00
1439de37b5
Convert GetRangeLimits() -> TOO_MANY + ASSERT().
2020-04-12 18:23:14 -07:00
db071256db
Merge remote-tracking branch 'upstream/master' into reduce-txn-state-store-size
2020-04-12 18:06:10 -07:00
3be131af76
Merge pull request #2944 from xumengpanda/mengxu/fr-send-mutations-batch-PR
...
Performant restore [28/xx]: Improve bandwidth of sending mutations from loaders to appliers
2020-04-12 16:54:33 -07:00
dbc9c23193
FastRestore:Loader:Send mutations at different versions in the same message to appliers
...
This increases the bandwidth sent from loaders to appliers.
2020-04-12 10:46:58 -07:00
a272bf4e66
Merge pull request #2931 from jzhou77/backup-progress
...
Consolidate backup container APIs
2020-04-11 22:47:20 -07:00
e8d333733a
Merge branch 'master' into feature-tree-broadcast
2020-04-10 13:51:09 -07:00
ce4493f679
many bug fixes
2020-04-10 13:45:16 -07:00
db2cef844b
Write mutation log type as a backup property
...
This can solve the problem when listing log files returns empty results.
2020-04-09 09:29:24 -07:00
4d06e837dc
Remove getPartitionedRestoreSet() API
...
Use getRestoreSet() instead for both old and new partitioned logs.
2020-04-08 20:12:09 -07:00
fd9caa88a0
Remove isPartitionedBackup()
...
This is no longer needed, since describeBackup() figures this out.
2020-04-08 16:09:18 -07:00
0cf6013357
Refactor to remove describePartitionedBackup()
...
The backup container can figure out if partitioned logs are used by looking at
mutation logs, thus consolidating the API to a single describeBackup() as
before.
2020-04-08 15:59:37 -07:00
af4e0088ba
Merge pull request #2896 from tclinken/atomically-update-dependent-knobs
...
Atomically update dependent knobs
2020-04-08 15:00:49 -07:00
2a6de93b92
add copyright information
2020-04-08 14:50:55 -07:00
34cddd675c
clang-format
2020-04-08 14:33:41 -07:00
535efa0c4c
Add assertion to make sure new registered range is not overlapping with existing ones
2020-04-08 14:27:05 -07:00
4a9658d6b8
fix issues according to andrew's comments
2020-04-08 13:38:12 -07:00
20b2984841
fix issues according to andrew's comments
2020-04-08 12:43:25 -07:00
52860043c9
Merge remote-tracking branch 'origin' into atomically-update-dependent-knobs
2020-04-08 12:26:21 -07:00
5aee6880a8
fix issues according to andrew's comments
2020-04-08 11:33:07 -07:00
7ba2cec9ef
Merge branch 'master' of github.com:apple/foundationdb into features/no-make
2020-04-08 10:23:46 -07:00
f500353368
Merge pull request #2908 from xumengpanda/mengxu/fr-fix-apitest-PR
...
Performant restore [25/xx]: Fix rare bugs found by API correctness test
2020-04-07 22:04:01 -07:00
867f734d8f
Merge pull request #2901 from jzhou77/backup-cmd
...
Add pause/resume for new backups
2020-04-07 16:53:51 -07:00
60407bdee3
Use LiteralStringRef for backup paused key
2020-04-07 16:02:25 -07:00
e5b2cd81d5
FastRestore:Cleanup debug code
2020-04-07 15:56:44 -07:00
efca39a09a
make traces SevDebug to avoid too verbose
2020-04-07 12:49:46 -07:00
d4542dbb5a
Delete old build system
2020-04-07 11:03:45 -07:00
ff92336887
refine code
2020-04-07 10:25:26 -07:00
c11b9d0721
Including port in addresses option is deprecated in 630 rather than 700. Includes other latent generated go changes to the documentation.
2020-04-07 08:51:15 -07:00
2309e9f156
Consistently use timeout instead of timedout in status messages.
2020-04-07 08:43:23 -07:00
640a29252b
add comments
2020-04-07 00:24:01 -07:00
86c66a4fca
add a parameter indicating whether keys are prefixed with \xff\xff
2020-04-07 00:08:47 -07:00
e4d99b0197
remove snapshot argument in the interface
2020-04-06 22:09:17 -07:00
a51c92854a
Merge branch 'master' into feature-tree-broadcast
...
# Conflicts:
# fdbserver/WorkerInterface.actor.h
# fdbserver/worker.actor.cpp
2020-04-06 21:09:44 -07:00
2a1bd97120
fix compilation errors
2020-04-06 20:58:43 -07:00
9fb3fb9d82
Add pause/resume for new backups
...
To pause/resume the backup workers, the fdbbackup command will write to the
backupPausedKey. Then backup workers noticed the value of the key has been
changed and stops/resumes pulling from TLog.
2020-04-06 14:29:46 -07:00
9408915ab9
reformat code
2020-04-06 13:38:18 -07:00
33780fc16e
reformat code
2020-04-06 13:23:41 -07:00
03795cc6b8
delete commented code
2020-04-06 11:42:31 -07:00
cfa8c4c74f
Merge pull request #2910 from dongxinEric/bugfix/fix-missing-schema
...
Fix the missing schema field which caused a lot noise in nightly
2020-04-06 11:24:27 -07:00
587419f984
Fix the missing schema field which caused a lot noise in nightly
2020-04-06 10:18:58 -07:00
477d66b46d
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-05 23:09:36 -07:00
536e65cd76
FastRestore:Introduce debugFRMutation for debug keys
2020-04-05 15:00:36 -07:00
6078fd1b18
Convert UID to Tag in keyServers to reduce txnStateStore size
2020-04-05 14:30:09 -07:00
432c99afd0
FastRestore:Applier:Keep incompleteStagingKeys content before values are applied to DB
...
To avoid the incompleteStagingKeys is cleared before getAndComputeStagingKeys() finish using it.
2020-04-04 22:38:04 -07:00
aad8f850f3
refactor conflicting keys feature using special key space
2020-04-04 13:26:11 -07:00
6d570cd865
Change version from 7.0 to 6.3
2020-04-03 21:38:58 -07:00
73272fc72e
Version difference is now the diff between TLog versions and SS version.
2020-04-03 19:04:43 -07:00
5b6ae4da7f
A simple intergration to put conflictingKeys into special-key-space
2020-04-03 16:11:20 -07:00
b76baa9cc2
Update conflicting keys related constants
2020-04-03 16:08:57 -07:00
760dc68b7f
Merge pull request #2869 from dongxinEric/feature/1689/allow-custome-trace-log-file-identifier
...
Allow the user to provide a custome trace log file identifier that wi…
2020-04-03 14:08:17 -07:00
2ef8f1eb96
Fixed the indentation. Addressed review comments
2020-04-03 13:21:57 -07:00
3489b2be7d
Apply suggestions from code review
...
Co-Authored-By: A.J. Beamon <ajbeamon@users.noreply.github.com>
2020-04-03 13:19:13 -07:00
0816423d78
Merge pull request #2853 from Daniel-B-Smith/debug-transactions
...
Add ability to mark transactions to add debug logging
2020-04-03 11:43:45 -07:00
638369492f
Merge pull request #2863 from dongxinEric/feature/2290/expose-region-config-via-status-default-or-details
...
Expose region related configurations in status
2020-04-03 11:42:43 -07:00
f6a2a7cee9
Move log statement to correct section
2020-04-03 17:31:38 +00:00
5092f52042
Fix location of logging code.
2020-04-03 13:19:48 -04:00
e1ec6cd7ee
Update fdb.options
2020-04-03 13:18:00 -04:00
78f0cddb14
Merge pull request #2684 from mpilman/features/boost70
...
Upgrade to boost 1.72
2020-04-03 09:30:59 -07:00
36bf2ecf4c
add \xff\xff prefix to returned keys
2020-04-03 00:26:11 -07:00
1ec7b1ebe3
Add logging for server tracing
2020-04-02 22:16:58 +00:00
d0210fd54f
Rename option
2020-04-02 21:41:16 +00:00
bbd2fe62cc
Merge branch 'master' of github.com:apple/foundationdb into features/boost70
2020-04-02 09:21:01 -07:00
92b9fdf4c4
Added /fdbclient/knobs/initialize unit test
2020-04-01 22:05:20 -07:00
884e92bb49
Atomically update dependent knobs
2020-04-01 15:18:49 -07:00
e755583c07
Address review comments.
2020-04-01 15:13:04 -07:00
4226c713fb
MutationRef:typeString:Add MAX_ATOMIC_OP as padding
2020-04-01 10:49:51 -07:00
81dd97d27f
Change special keyspace to \xff\xff, \xff\xff\xff
2020-03-31 19:06:45 -07:00
debbe648e8
Use specialkeys
2020-03-31 19:05:47 -07:00
c8abd9c493
Reformat some code
2020-03-31 18:53:36 -07:00
863ebc2d6d
fix merge conflict
2020-03-31 18:49:59 -07:00
b1c3893d40
Fix some corner case bugs exposed by simulation.
...
In one case, when a SS joins the cluster and DD doesn't find any healthy server to form a team with the newly added server, then the SS does not get added to any team even when the other servers get healthy.
Another is an extreme case where a data center is down, and a SS in the active DC joins and then dies immediately but not before DD adds it to a destination team for a relocating shard which will result in DD waiting indefinitely for the dead data center to come back up for the cluster to be fully recovered.
2020-03-31 18:33:12 -07:00
ad1dd4fd9b
Mark the storage servers that are continually lagging as unhealthy and so this will give the Data Distributor the chance to move data out of this server.
2020-03-31 18:25:39 -07:00
6820167d77
Merge branch 'master' into feature/1689/allow-custome-trace-log-file-identifier
2020-03-31 16:50:46 -07:00
Evan Tschannen