9df02512ab
FastRestore:Apply clang-format
2020-01-07 11:50:32 -08:00
67e913c3d5
Change LoadingParam struct and endVersion definition
...
1) Remove endVersion field because it has been included in RestoreAsset;
2) Ensure endVersion in VersionBatch and RestoreAsset is always exclusive;
3) Revise ASSERT in laoder and applier in situations when the dummy commit version
is endVersion, to avoid false positive ASSERT failure.
2020-01-07 11:48:03 -08:00
c3f8f3b445
FastRestore:Build VersionBatch less than threshold size
2020-01-07 11:46:56 -08:00
9a3dfec7c5
open a connection with processes before attempting to kill them to improve the reliability of the kill process
...
secondaryAddresses are included in the list of processes which can be killed
2020-01-03 16:10:44 -08:00
8d6f511816
FastRestore:Resolve review comment
...
Filter out range mutations that do not overlap with the restore range.
Small changes on format.
2019-12-22 20:09:10 -08:00
61b29de3ce
FastRestore:Self code review
...
Clean up commented code;
Add sanity check.
2019-12-20 22:24:34 -08:00
ddcf3fdd80
FastRestore:Apply clang format
2019-12-20 22:00:36 -08:00
e98b2a0d1c
FastRestore:Introduce RestoreAsset
2019-12-20 18:00:10 -08:00
fca0e74655
Merge pull request #2418 from satherton/blobstore-api-enhancements
...
BlobStoreEndpoint enhancements
2019-12-19 11:23:37 -08:00
4ff058e86b
Backup and DR layer status document generation now uses snapshot reads for all keys read to avoid unnecessary conflicts when read during a status update or cleanup transaction. Since many of the keys read use wrapper functions, all of the underlying functions in BackupAgentBase and its two implementations also required a snapshot mode argument. All snapshot arguments default to false to match the underlying FDB API get/getrange methods.
2019-12-19 00:29:35 -08:00
cc4e57f72c
Use consistent "> 0" style for guarding memcpy
2019-12-16 08:40:35 -08:00
ded2a301e0
Merge pull request #2443 from xumengpanda/mengxu/fast-restore-fix-valgrind-PR
...
Performant restore [12/XX]: Code clean up
2019-12-13 14:35:20 -08:00
97030d9168
FastRestore:Revise and test SevFRMutationInfo
...
Enabled SevFRMutationInfo for valgrind test, no error found, and disable it again.
Revise debug trace message a bit.
2019-12-13 13:51:21 -08:00
3bf971ba8b
Merge branch 'release-6.2' of github.com:apple/foundationdb into release_6.2_merge
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/storageserver.actor.cpp
2019-12-12 07:13:12 -08:00
1371db4cdc
FastRestore:Self code review and cleanup
...
1. Review memory use cases and improve:
Ensure state varialble is initialized and
change unnecessary state variable to variable.
2. Remove debug code that is no longer useful;
3. Mute verbose debug.
2019-12-11 16:37:33 -08:00
9a031bfc47
Function was renamed.
2019-12-11 11:00:12 -08:00
9a6dabe47e
Merge branch 'mengxu/fastrestore-code-cleanup-PR' into mengxu/fast-restore-fix-valgrind-PR
2019-12-10 20:05:35 -08:00
feb2a8c70c
FastRestore Change RestoreSendMutationVectorVersionedRequest name
...
Change RestoreSendMutationVectorVersionedRequest to
RestoreSendVersionedMutationsRequest for better naming
2019-12-10 17:23:40 -08:00
20a19978f9
FastRestore:LoadingParam cleanup
2019-12-10 17:20:44 -08:00
290cbb7628
Fix an internal error that occurred when using client buggify and a successful transaction was failed with commit_unknown_result.
...
Move some buggify errors into a try catch block so that the normal error handling path can run.
Initialize startTime so that it's valid in the case of errors thrown early.
2019-12-09 09:44:48 -08:00
e8dfc1c187
Replace pop_front(size) with new empty standalone obj
2019-12-06 23:16:49 -08:00
09e8d804e8
Added BlobStoreEndpoint::listBuckets(), renamed listBucket() and several related functions with similar names to listObjects() to avoid confusion and closer match what it actually does. Added a bytesDeleted output statistic to BlobStoreEndpoint::deleteRecursively.
2019-12-06 00:14:13 -08:00
c6b36dbffb
FastRestore:Sampling:Resolve review comments
2019-12-04 17:35:11 -08:00
2b987d1945
FastRestore:typedef Standalone<VectorRef<MutationRef>> MutationsVec
2019-12-04 11:39:55 -08:00
c8e8d0c8ab
Fix UBSAN report
...
/home/anoyes/workspace/foundationdb/fdbclient/SnapshotCache.h:36:9: runtime error: null pointer passed as argument 2, which is declared to never be null
2019-12-04 09:42:35 -08:00
cc170fe8d8
Fix UBSAN report
...
/home/anoyes/workspace/foundationdb/fdbclient/SnapshotCache.h:44:10: runtime error: null pointer passed as argument 2, which is declared to never be null
2019-12-04 09:42:34 -08:00
a15320cca7
Merge branch 'master' into mengxu/fast-restore-sampling-PR
2019-12-03 21:42:01 -08:00
3c1ba46cbe
Allow any valid int to be loaded as an Option
...
We assume that this is kosher in FuzzApiCorrectness, so make it kosher
This fixes some UBSAN reports
2019-12-03 18:07:31 -08:00
3310f67e9e
Merge branch 'mengxu/fast-restore-fix-valgrind-PR' into mengxu/fast-restore-sampling-PR
2019-12-03 16:24:40 -08:00
46d10dc7dc
Fix "null passed as argument declared not null"
...
Fix several such reports from ubsan
E.g.
/Users/anoyes/workspace/foundationdb/flow/Arena.h:794:16: runtime error: null pointer passed as argument 1, which is declared to never be null
2019-12-03 14:46:53 -08:00
153b713b53
FastRestore:Add sampling on parsed mutations
2019-12-03 12:52:17 -08:00
2184f59875
Merge pull request #2405 from satherton/add-backup-comment
...
Added code comments in backup log range task.
2019-12-03 10:33:33 -08:00
a3ef3199c6
Merge pull request #2281 from fzhjon/include-failed-ss
...
Allow re-inclusion of servers marked as failed
2019-12-03 09:53:33 -08:00
f3ed3f462b
Added code comments in backup log range task.
2019-12-03 09:44:15 -08:00
07331ab5fd
Merge pull request #2362 from etschannen/master
...
Merge 6.2 into master
2019-12-02 15:04:27 -08:00
17ab2f8e00
Default initialize absent flatbuffers members
2019-11-26 10:58:29 -08:00
da5f48f344
Merge branch 'master' of https://github.com/apple/foundationdb into merge-attrition-workloads
2019-11-25 12:55:55 -08:00
1e5bff34dc
changed function parameter to pass by reference
2019-11-25 12:55:50 -08:00
92892d4168
Merge branch 'master' of https://github.com/apple/foundationdb into include-failed-ss
2019-11-25 11:05:35 -08:00
d529d82b77
Merge pull request #2349 from xumengpanda/mengxu/fastrestore-load-before-sampling-PR
...
Performant restore [9/XX]: Move loading data phase before sampling phase
2019-11-22 15:47:39 -08:00
ebcb2f79ed
Merge branch 'master' of github.com:apple/foundationdb
2019-11-22 15:34:49 -08:00
cb77c7dd47
Merge pull request #2355 from atn34/more-no-discard
...
Refactor Notified
2019-11-22 12:09:10 -08:00
dbfa3dc217
Merge pull request #2200 from negoyal/storage-cache-subfeature1
...
Storage cache subfeature1
2019-11-20 13:59:06 -08:00
ed8d3f163c
Rename hgVersion to sourceVersion.
2019-11-15 12:26:51 -08:00
d46c7ded59
Merge remote-tracking branch 'origin/master' into storage-cache-subfeature1
2019-11-14 17:52:22 -08:00
8d3ef89540
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# fdbclient/MutationList.h
# fdbserver/MasterProxyServer.actor.cpp
# versions.target
2019-11-14 15:49:56 -08:00
66344996a9
backup_agent crashed on startup
2019-11-13 20:54:10 -08:00
1f547eab23
Version -> ValueType
2019-11-13 15:32:52 -08:00
b4aa72303f
Add [[nodiscard]] for whenAtLeast, and make Notified generic
2019-11-13 13:30:34 -08:00
e3644d9c00
Add a little extra time to the timeout delay so that we don't end up in a situation where we get stuck in a loop due to floating point math.
2019-11-13 13:13:36 -08:00
9bb27ee6c0
Merge branch 'release-6.2' into limit-transaction-timeout-delay
2019-11-13 12:55:20 -08:00
11525f6922
added comments
2019-11-13 12:53:23 -08:00
5bd4bf357f
Limit length of delays in timeout, repeating them as necessary.
2019-11-13 12:49:07 -08:00
8f725db92e
serialization of logRangeMutation->second caused long slow tasks
2019-11-12 23:06:58 -08:00
9e36b897e6
FastRestore:Loaders must send to appliers log files data before range files
2019-11-12 21:43:12 -08:00
be303cad7a
Merge pull request #2339 from etschannen/feature-increase-reboot-priority
...
Increase the priority of reboot and recruitment requests
2019-11-12 20:42:22 -08:00
5fbd9f2ed5
added logging to TaskBucket
2019-11-12 19:15:56 -08:00
592f4c0fc4
FastRestore:Remove RestoreSetApplierKeyRangeVectorRequest
2019-11-12 17:59:11 -08:00
7e4c4ea98e
FastRestore:Load mutations before assign ranges to appliers
2019-11-12 17:14:17 -08:00
a4a0bf18f9
Merging with Master.
2019-11-12 13:01:29 -08:00
ae7e42face
Merge pull request #2313 from xumengpanda/mengxu/fastrestore-applyToDB-bugfix-PR
...
Performant restore [8/XX]: Fix bugs in applyToDB logic and add more tests
2019-11-12 08:50:23 -08:00
630c29d160
FastRestore:resolve review comments
...
1) wait on whenAtLeast;
2) Put BigEndian64 into the function call and the decoder to prevent
future people from making the same mistake.
2019-11-11 17:00:16 -08:00
1e5677b55a
increase the priority of reboot and recruitment requests
2019-11-11 15:17:11 -08:00
eb67886b75
FastRestore:Move comment to func definition
...
Resolve review comments.
2019-11-11 15:10:27 -08:00
cf2ec3418c
Merge pull request #2317 from xumengpanda/mengxu/fastrestore-extend-atomicOpTest-PR
...
AtomicOps Test: Add more detailed debug information when test fails with opType = AddValue
2019-11-11 15:03:10 -08:00
2147401a21
added function lambda and ability to specify zone kill
2019-11-08 15:05:18 -08:00
afc9713005
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# fdbclient/FDBTypes.h
# fdbserver/LogSystem.h
# fdbserver/LogSystemPeekCursor.actor.cpp
# fdbserver/OldTLogServer_6_0.actor.cpp
# fdbserver/TLogServer.actor.cpp
# versions.target
2019-11-06 13:45:37 -08:00
0ccded1929
AtomicOps:Resolve review comments
2019-11-05 19:27:49 -08:00
a8ca47beff
optimized memory allocations by using VectorRef<Tag> instead of std::vector<Tag>
2019-11-05 18:07:30 -08:00
e345c9061f
FastRestore:Refine debug messages
2019-11-04 11:47:38 -08:00
58aa6711e4
FastRestore:ApplyToDB:BugFix:Serialize integer as bigEndian to ensure lexico order
2019-11-03 17:26:07 -08:00
4de60fc437
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/TLogServer.actor.cpp
2019-11-01 15:48:04 -07:00
2ce57b6735
Merge pull request #2198 from ajbeamon/increase-location-cache-size
...
Increase the default client shard location cache size by a factor of 2.
2019-10-31 10:23:45 -07:00
f175ed30b3
Cleanup the fdbbackup cleanup command output. Add cleanup to the usage output printed for fdbbackup.
2019-10-31 09:52:21 -07:00
b7b5d2ead3
Remove several nonsensical const uses
...
These seem to be all the ones that clang's -Wignored-qualifiers
complains about
2019-10-26 14:30:34 -07:00
aed9dfd148
Fix flow header guard
2019-10-25 10:42:22 -07:00
de8921b660
Move RestoreWorkerInterface to fdbclient
2019-10-25 10:42:22 -07:00
d4de608bb6
Fix OPEN_FOR_IDE build
2019-10-25 10:42:22 -07:00
9db95bd976
initial commit to allow re-inclusion of servers marked as failed
2019-10-23 11:05:48 -07:00
e676348710
Merge pull request #1955 from fzhjon/mark-ss-failed
...
Add fdbcli and API command to mark storage servers as permanently failed
2019-10-22 23:36:30 -07:00
fca9aab17a
Merge pull request #2046 from dongxinEric/feature/hot-read-key-detection
...
Added metrics for read hot key detection
2019-10-21 14:31:48 -07:00
d2b6626d5c
Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed
2019-10-21 13:47:06 -07:00
688940b685
merge 6.2 into master
2019-10-21 11:43:46 -07:00
4d8cfc00f6
added comment explaining quorum
2019-10-18 15:00:00 -07:00
f38d52b82d
re-introduced coordinator quorum and added some comments
2019-10-18 14:52:07 -07:00
f67eb2f371
removed smartquorum for coordinators check
2019-10-18 09:44:23 -07:00
ef0890c23a
updated status schema
2019-10-16 22:37:57 -07:00
42b7acf7b7
Merge pull request #2202 from etschannen/feature-share-mutations
...
Backup and DR would not share mutations if started on different versions of FDB
2019-10-16 20:28:39 -07:00
1fd9411395
fix: eraseLogData did not cleanup destUidLookupPrefix
2019-10-16 20:18:39 -07:00
85bc5f6b8b
Update fdbclient/DatabaseConfiguration.h
...
Co-Authored-By: Markus Pilman <markus@pilman.ch>
2019-10-16 15:11:24 -07:00
1af44afad3
Update fdbclient/DatabaseConfiguration.h
...
Co-Authored-By: Markus Pilman <markus@pilman.ch>
2019-10-16 14:55:02 -07:00
8b09cd16b2
Merge branch 'release-6.2' of github.com:apple/foundationdb into feature-share-mutations
2019-10-16 14:50:37 -07:00
f4237ebfff
Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed
2019-10-16 11:32:16 -07:00
896701006f
addressed code review changes
2019-10-16 11:30:20 -07:00
e7b97c393d
added zipfian distribution to mako workload
2019-10-15 01:14:21 -07:00
35e816e9ad
added the ability to configure satellite_logs by satellite location, this will overwrite the region configure if both are present
2019-10-14 18:30:15 -07:00
d0e2da3c53
fixed naming in header file
2019-10-11 11:17:31 -07:00
71509a5157
FastRestore:Applier:applyToDB:Clang format
2019-10-10 17:36:38 -07:00
84b5a5525f
FastRestore:Add restoreApplierKeys
2019-10-10 17:18:34 -07:00
12293d5497
Added metrics for read hot key detection
2019-10-09 16:42:42 -07:00
a36caed3bd
Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed
2019-10-09 10:22:55 -07:00
36b1247054
Merge pull request #2136 from ajbeamon/dont-require-future-get-database
...
Don't require fdb_future_get_database from new client binaries.
2019-10-09 09:58:56 -07:00
d146f1d636
Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed
2019-10-07 11:27:15 -07:00
1b946d588f
Merge pull request #2208 from alexmiller-apple/faster-txstag-recovery
...
Recover Txs Faster [0/?]: Combine spill-by-value and spill-by-reference into one file/SharedTLog
2019-10-07 11:15:56 -07:00
28f6275f94
Use AssumeVersion instead of Unversioned
...
Which lets us revert the unversioned serilaization of TLogSpillType
2019-10-03 15:59:09 -07:00
d96a7b2c69
Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed
2019-10-03 09:47:45 -07:00
d38a96ab73
Make LogData aware of the spill type it was created to perform.
...
The spilling type is now pulled out of the request, and then stored on
LogData for later access, and persisted in the tlog metadata per tlog
generation.
It turns out that serializing types as Unversioned is a bit wonky.
2019-10-03 01:45:10 -07:00
24c46337e1
Advance TLogVersions for 7.0 while adding a V5 that is TLogServer
...
Advancing the MIN_RECRUITABLE and DEFAULT is just following the standard
progression for 7.0. It was convenient to do while adding the V5 so
that we can hook TLogServer back into being used.
2019-10-03 01:43:01 -07:00
628b4e0220
added a warning if multiple log ranges exist for the same range
2019-10-02 17:06:19 -07:00
d0147e5e5d
Merge branch 'release-6.2' into mengxu/merge-release620-to-master-v3
...
Resolved Conflicts:
documentation/sphinx/source/release-notes.rst
fdbserver/DataDistribution.actor.cpp
versions.target
2019-10-02 13:22:56 -07:00
9ec9f41d34
backup and DR would not share mutations if they were started on different versions of FDB
2019-10-01 18:52:07 -07:00
656eacc965
Increase the default client shard location cache size by a factor of 2.
2019-10-01 14:31:48 -07:00
1d63ba6980
Use immediate priority for coordinator changes
2019-10-01 08:36:37 -07:00
e5a6ebae7e
fixed compiler error
2019-09-30 12:46:30 -07:00
4d659662b8
made cleanup handle retries better
2019-09-30 12:44:20 -07:00
045175bd0e
added tracking for the size of the system keyspace
2019-09-27 22:39:19 -07:00
324d0bd3b0
Merge branch 'release-6.2' of github.com:apple/foundationdb into feature-cleanup-mutations
2019-09-27 19:15:14 -07:00
3cc5d484a5
the include and exclude commands do not need to set the moveKeysLockOwnerKey, which will kill the data distribution algorithm
2019-09-27 18:33:56 -07:00
ef01ad2ed8
optimized log range clearing to clear everything for each possible hash (256 clears) if that would be more efficient than one clear per second that has elapsed
...
aborting a DR without the —cleanup flag will still attempt to cleanup for 30 seconds before giving up
added a cleanup command to fdbbackup which can remove mutations from orphaned DRs which were stopped without the —cleanup flag
2019-09-27 18:32:27 -07:00
81ebc7bfc9
Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed
2019-09-27 13:14:13 -07:00
d41852abbc
Merge pull request #1988 from negoyal/negoyal_dequeue
...
Replace std::map with std::deque for versionedMap
2019-09-26 23:25:58 -07:00
0d06bf8a66
Update fdbclient/VersionedMap.h
...
Co-Authored-By: Steve Atherton <stevea@apple.com>
2019-09-26 21:58:04 -07:00
2526c1a0d2
Update fdbclient/VersionedMap.h
...
Co-Authored-By: Steve Atherton <stevea@apple.com>
2019-09-26 21:57:54 -07:00
d049105a69
Update fdbclient/VersionedMap.h
...
Co-Authored-By: Steve Atherton <stevea@apple.com>
2019-09-26 21:57:44 -07:00
f168c3fda9
Update fdbclient/VersionedMap.h
...
Renamed variable to be more specific
Co-Authored-By: Steve Atherton <stevea@apple.com>
2019-09-26 21:57:32 -07:00
381f3220b3
Update fdbclient/VersionedMap.h
...
Indentation Fix
Co-Authored-By: Steve Atherton <stevea@apple.com>
2019-09-26 21:57:11 -07:00
efbce26c78
Update fdbclient/VersionedMap.h
...
Indentation fix.
Co-Authored-By: Steve Atherton <stevea@apple.com>
2019-09-26 21:56:43 -07:00
efa2f5df6b
avoid double overwrite on failed servers and reordered removal of keys
2019-09-26 13:52:53 -07:00
450a09e117
Code Review Changes
2019-09-24 15:48:50 -07:00
a51ab0003d
Don't require fdb_future_get_database from new client binaries. The previous code, which technically incorrect, is believed to be asymptomatic because new versions of the client do have this symbol in order to support old API header versions.
2019-09-24 13:11:30 -07:00
40ad6f0931
created new reply types to work with flatbuffer serialization requirements
2019-09-18 13:40:18 -07:00
471e283128
Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed
2019-09-18 11:49:07 -07:00
a4d38f1158
Fix #2057 snapshot cli to print UID in failure too
2019-09-17 05:18:28 -07:00
b495cc697b
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# versions.target
2019-09-13 09:25:08 -07:00
d9fb199486
Remove debug messages from checkDatabaseLock()
2019-09-12 22:11:25 -07:00
9b4f7626bb
cache the serialization of clientDBInfo
2019-09-11 15:19:42 -07:00
6100d3274d
Merge pull request #2058 from tclinken/expose-lock-status
...
Added lockUID to status output if database is locked
2019-09-11 08:47:35 -07:00
41fd3d9467
Merge branch 'master' into remove-unused-ssi-get-version
...
# Conflicts:
# fdbclient/StorageServerInterface.h
# fdbserver/storageserver.actor.cpp
2019-09-10 08:11:43 -07:00
94668c6f1f
Merge pull request #2063 from jzhou77/clang
...
Refactor deserialization of on-wire buffer with TagsAndMessage
2019-09-09 16:34:56 -07:00
8fbd90e2f6
Merge pull request #1985 from xumengpanda/mengxu/storage-engine-switch-PR-v2
...
Graceful storage engine migration
2019-09-09 13:51:53 -07:00
8c31a839be
s/lockUID/lock_uid in status
2019-09-06 22:20:55 -07:00
2d216f7ae5
Added database_lock_state to statusSchema
2019-09-06 22:20:50 -07:00
2d5ebebb7b
Use TagsAndMessage for deserialization in TLogServer
2019-09-05 16:53:10 -07:00
34dedc9a62
Fix whitespace
2019-09-05 16:44:58 -07:00
c18c4c1b83
Use a transaction option to control includePort behavior
2019-09-05 14:58:39 -07:00
2723922f5f
Replace -1 as VERSION_HEADER constant for serialization
2019-09-05 12:45:39 -07:00
f9357c5ad8
Fix side effect of ArenaReader
...
ServerPeekCursor::nextMessage() should only consume the message header, because
the reader() directly inherits the current position. The previous commit
changes the positon to the begining of the next message, which breaks storage
server code.
2019-09-05 11:07:07 -07:00
11f6adf645
Treat \xff\xff prefix as 'includePort' for get_addresses_for_key
2019-09-04 17:47:40 -07:00
c2355f721e
Merge branch 'master' into mengxu/performant-restore-PR
2019-09-04 17:11:42 -07:00
d160810662
FastRestore:Resolve review comments
2019-09-04 16:48:43 -07:00
Meng Xu