c0f857cd64
attempt to have logAntiQuorum as an option preset prior to starting simulation and running tests
2020-10-06 14:47:13 -04:00
7e6c0c9d94
explicitly throw error when snapshot has failed rather than hitting hard-to-trace assertion
2020-10-06 14:31:41 -04:00
b610f01c77
Fine-grained interleaving
2020-10-06 09:32:22 -07:00
efe50b68e6
fix compile error
2020-10-05 14:16:52 -07:00
b4ad989252
use stack transaction instead of heap
2020-10-05 16:51:01 -04:00
a4cba5a162
Remove verifyOnly from Cycle workload
2020-10-05 13:19:12 -07:00
1ab1e12dcd
Use discontinueBackup in RestoreBackup.actor.cpp
2020-10-05 12:56:25 -07:00
7ba06a4434
fix: min and max compute estimate logging on the proxy was always zero
...
added comments and fixed formatting
2020-10-05 12:35:10 -07:00
c622a11c70
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-snapshot-record-version
2020-10-05 13:15:56 -04:00
a646df520d
add option to remove snaptest retry limit
2020-10-05 13:15:43 -04:00
b4ffc04e4e
Add UpgradeAndBackupRestore test
2020-10-04 23:27:10 -07:00
7f0d14c8e4
Modernize/refactor workloads directory
2020-10-04 22:29:07 -07:00
5807b1ec3d
changed the recent requests to be the per second amount; increased precision of cpu estimate
2020-10-04 19:31:40 -07:00
f546034366
do not prevent computePerOperation from being updated for small computeDurations. Added logging for the compute per operation. Protect against erroneously large compute estimates
2020-10-04 19:19:05 -07:00
da26b0411c
increased the proxy commit memory limit
2020-10-04 19:16:51 -07:00
52a6496a54
fix compiler errors
2020-10-04 16:50:54 -07:00
614c8bc895
Get read versions requests must be load balanced on the number of requests because ratekeeper gives out an equal budget to each proxy
2020-10-04 16:20:24 -07:00
0e96233015
Merge pull request #3842 from sfc-gh-tclinkenbeard/remove-unflushed-data-ratio-knob
...
Remove deprecated UNFLUSHED_DATA_RATIO knob
2020-10-02 12:32:39 -07:00
f757beaac1
Remove deprecated UNFLUSHED_DATA_RATIO knob
2020-10-02 10:36:21 -07:00
268ba0bddc
Merge remote-tracking branch 'origin/release-6.3' into merge-6.3
2020-10-01 14:14:06 -06:00
91a8367acb
Avoid slow task in ~DataDistributionTracker
2020-10-01 11:44:55 -07:00
b1180f8eb4
fixed naming and comments
2020-09-30 20:35:09 -07:00
b1570c740f
extraTlogEligileZones should consider the database available both during a failover and also if the cluster cannot recruit tlogs in the remote region
2020-09-30 18:10:04 -07:00
8c729ca8e6
only add additional fault tolerance for availability if automatic failover is enabled
2020-09-30 18:04:23 -07:00
9f61039858
more fixes
2020-09-30 16:52:58 -07:00
d7454ac7da
fixed compile error
2020-09-30 16:49:36 -07:00
2a279f64af
Merge branch 'release-6.3' into feature-fix-fault-tolerance
2020-09-30 16:42:18 -07:00
fe5c30e778
fault tolerance was not being properly increased when usable regions was 2 and satellites are configured.
2020-09-30 16:41:00 -07:00
8da8c6ff0d
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-snapshot-record-version
2020-09-30 14:25:29 -04:00
44cd3b0999
add stopBackup to incrementalBackup workload
2020-09-30 14:24:52 -04:00
3705f5a6fe
Merge pull request #3820 from sfc-gh-tclinkenbeard/improve-type-safety
...
Improve ConfigurationResult and CoordinatorsResult type safety
2020-09-30 10:46:59 -07:00
3aa92286aa
FastRestore:Fix segmentation fault
2020-09-29 22:28:52 -07:00
21071b6214
Merge pull request #3813 from xumengpanda/mengxu/merge-to-master-PR
...
Merge 6.3 to master and resolve conflicts
2020-09-28 15:24:18 -07:00
3ceb44f4df
add TEST macros in code paths
2020-09-28 16:40:38 -04:00
cfaa207f88
fix spacing and newlines at ends of files
2020-09-28 16:32:20 -04:00
3aa4070adb
remove trace events used for debugging
2020-09-28 16:27:30 -04:00
e02caac2d9
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-snapshot-record-version
2020-09-28 16:20:33 -04:00
57a80cfef2
added new SimpleAtomicAdd workload and adjusted SnapIncrementalRestore restarting test
2020-09-28 16:19:48 -04:00
6c726ba8dd
Improve ConfigurationResult and CoordinatorsResult type safety
2020-09-27 15:29:15 -07:00
c613fc6dee
Merge pull request #3761 from sfc-gh-tclinkenbeard/document-watchbytes-overhead
...
Add comments for WATCH_OVERHEAD_BYTES
2020-09-26 20:39:27 -07:00
480fc82779
Resolve review comments
2020-09-25 16:58:54 -07:00
3278e21b0b
Change description of GRV proxy and Commit proxy in docs and add backward compatibility support for configuring proxies in fdbcli
2020-09-25 10:50:48 -07:00
862336de8f
Merge branch 'master' into mengxu/merge-to-master-PR
2020-09-24 17:06:00 -07:00
58e043c7a5
Enable run loop profiler for test and multitest roles.
2020-09-24 14:14:55 -07:00
a78da8b3ce
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-snapshot-record-version
2020-09-24 14:40:01 -04:00
d1be303821
Merge pull request #3793 from sfc-gh-jfu/jfu-ibackup-test-timeout
...
Attempt to use version instead of test duration to have backup capture all changes
2020-09-24 11:17:49 -07:00
a96d6f85c5
Removed redundant field number_of_old_generations_of_tlogs from status json
2020-09-24 09:44:51 -07:00
ebad1cd9e6
change waitBackup result from int to EBackupState
2020-09-24 12:31:21 -04:00
5f7e784742
Merge pull request #3796 from xis19/master
...
Refactor fdbserver.actor.cpp for better readability
2020-09-24 09:12:47 -07:00
d0b8f7b1e3
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-ibackup-test-timeout
2020-09-24 12:10:15 -04:00
6ed330572e
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-snapshot-record-version
2020-09-24 12:09:17 -04:00
34d1d04904
Commit pipelining attempt #2
2020-09-23 20:03:04 -07:00
de5b0abb92
Merge pull request #3806 from xumengpanda/mengxu/fix-typo-PR
...
Fast Restore: Fix a typo in FastRestoreApplerPhaseApplyTxnStart event name
2020-09-23 17:11:59 -07:00
5214becaa8
FR:Fix typo for event FastRestoreApplerPhaseApplyTxnDone
2020-09-23 16:43:35 -07:00
feb3bda79e
Merge pull request #3797 from xumengpanda/mengxu/fr-write-traffic-control-PR
...
Fast Restore: Add write rate control
2020-09-23 15:50:08 -07:00
262307d557
FR:Change applierRemainMB map to unordered_map
2020-09-23 15:39:01 -07:00
aa683c0d26
FRApplier:Fix applyingDataBytes accounting at exception
...
When exception is thrown out after txnSize is calculated but before
it is accounted into applyingDataBytes, we will decrease applyingDataBytes in the
error handling block incorrectly.
2020-09-23 15:19:02 -07:00
d13a9d3c91
Make EBackupState an enum class
2020-09-23 15:14:09 -07:00
77048c3d0f
Handle possbile timeout when getting a read version. Updated documentation of the status json format
2020-09-23 13:50:59 -07:00
7553daba20
change waitVersion to waitForBackup
2020-09-23 16:13:30 -04:00
69580593dd
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-snapshot-record-version
2020-09-23 15:35:05 -04:00
581427c880
fixup! Remove "\n" and extra spaces in printUsage
2020-09-23 12:03:09 -07:00
89c334eb53
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-ibackup-test-timeout
2020-09-23 14:30:46 -04:00
285e2594ef
adjusted delay and timing of version gate
2020-09-23 14:28:47 -04:00
e2ea6bbdd0
fixup! Use currLine.size() for the first currLength in printOptionUsage
2020-09-23 11:19:02 -07:00
50f681cd32
Apply suggestions from code review
...
Co-authored-by: A.J. Beamon <ajbeamon@users.noreply.github.com>
2020-09-23 10:54:49 -07:00
a4aad591fd
FR:Remove applyStagingKeysBatchLock flow lock
2020-09-23 08:47:37 -07:00
850ec6aa43
fixed compile error
2020-09-22 20:27:10 -07:00
814df222b9
testRywLifetime fails below API version 630
2020-09-22 20:18:15 -07:00
6da7e56e03
FastRestore:Add more trace events
2020-09-22 19:57:19 -07:00
08e243a07e
Fix assert failure at addCallbackAndDelFutureRef
2020-09-22 15:41:07 -07:00
c16d76745d
FastRestore:small fix compilation error
2020-09-22 13:36:26 -07:00
f8c5d85b79
FastRestore:Catch controller error and fix unhandled error occurred in updateHeartbeatTime
2020-09-22 13:32:26 -07:00
17ece3d477
FastRestore:Fix FastRestoreApplierTransactionRateControl events
2020-09-22 13:25:43 -07:00
b4254473d7
FastRestore:Add transaction rate info tracer
2020-09-22 09:10:11 -07:00
002b1bec4c
FastRestore:Control write traffic at each applier
...
Controller assigns each applier a write rate.
Applier keeps the write-rate worth of transactions outstanding to DB.
This is to avoid heavily overloading DB while still keep enough
traffic to DB to get a good write throughput.
2020-09-22 08:14:28 -07:00
2bf4ebbd51
fixup! Add missing comma
2020-09-21 20:43:07 -07:00
3c505786ca
Refactor fdbserver.actor.cpp
2020-09-21 20:19:47 -07:00
7780c24cbe
Attempt to use version instead of test duration to have backup capture all changes
2020-09-21 14:45:29 -04:00
9b7d75b6dd
WIP
2020-09-21 13:41:35 -04:00
0814841827
Replace NULL with nullptr in fdbserver
2020-09-20 11:31:49 -07:00
a52ef0acb8
Merge pull request #3677 from sfc-gh-clin/add-setclass-to-special-key-space
...
Add setclass to special keys
2020-09-19 14:55:03 -07:00
dbeb8772f0
Fix isMetadataMutation
2020-09-19 09:40:40 -07:00
4d36ed7ba3
Rename traces and add comments when getting empty worker list
2020-09-18 19:28:32 -07:00
443385b052
Skip the case where the test client gets zero workers in setclass test
2020-09-18 18:20:47 -07:00
12b07d024f
Merge pull request #3775 from sfc-gh-xwang/fdbtest
...
add knob MIN_TAG_WRITE_PAGES_RATE for write tag-throttling
2020-09-18 10:40:28 -07:00
6032a21d30
Fix comment
2020-09-17 18:02:23 -07:00
60ccaa2ae9
Support finding max restorable version for key ranges when keyspace's max restorable version is unavailable
2020-09-17 17:58:58 -07:00
d4e3e98957
Use new getRestoreSet in RestoreController
2020-09-17 17:56:36 -07:00
71309a79f9
Merge branch 'release-6.3' into range
2020-09-17 17:54:33 -07:00
cf69f455a9
Merge pull request #3785 from apple/release-6.3
...
Merge Release 6.3 to master
2020-09-17 14:43:56 -07:00
3746d52fb8
apply symmetrical clear to txnStateStore for writeRecoveryKey
2020-09-17 15:38:09 -04:00
4df0f60729
Instead of using fully_recovered, use accepting_commits as a singal of DB turned available. Also add the number of old generations into status
2020-09-17 09:55:25 -07:00
0c98a145b9
Fix whitespace
2020-09-17 08:09:21 -07:00
303e1dc0c6
Fix fall through in switch statement
2020-09-17 08:07:53 -07:00
ea27e4bd9c
Merge pull request #3772 from yliucode/rename
...
Rename master proxy as commit proxy
2020-09-16 20:06:50 -07:00
1fe31aa9e8
Merge pull request #3706 from yliucode/tlogs
...
Report missing old tlogs in recovery before storage servers are fully recovered
2020-09-16 14:07:06 -07:00
cc5bc16bd8
Rename more places from proxy to commit proxy
2020-09-15 22:29:49 -07:00
3a68c89281
Remove initialisms
...
Co-authored-by: A.J. Beamon <ajbeamon@users.noreply.github.com>
2020-09-15 16:32:24 -07:00
37d77ecb64
WIP of adding tests
2020-09-15 13:31:06 -04:00
ed2d261266
Add MinV2 and AndV2 into AtomicOps test workload
2020-09-15 10:01:36 -07:00
8224e17a08
Integrate extended getRestoreSet API into fast restore
...
The extended getRestoreSet provides a much smaller set of backup files for small keyrange restore;
This commit integrate it into fast restore so that fast restore does not have to filter out unneeded backup files.
2020-09-15 09:38:43 -07:00
8afd041184
Merge pull request #3776 from sfc-gh-tclinkenbeard/update-checkunreadability
...
Remove unnecessary parameter from checkUnreadability
2020-09-14 13:55:30 -07:00
94e7e11664
Merge pull request #3779 from ajbeamon/fix-write-tag-test-memory-error
...
Fix memory issue in write tag throttling test
2020-09-14 10:59:57 -07:00
a3c861b59d
Merge pull request #3777 from apple/release-6.3
...
Merge Release 6.3 into master
2020-09-14 10:32:50 -07:00
268eb43cc2
Fix memory issue in test
2020-09-14 10:29:40 -07:00
09b3b7a325
Remove unnecessary parameter from checkUnreadability
2020-09-13 21:32:47 -07:00
3c7bd3549a
Fix compile errors
2020-09-11 14:23:27 -07:00
52bd86ad42
update knob
2020-09-11 20:50:19 +00:00
f2f3351560
Only report if the field FullyRecoveredAtVersion exists.
2020-09-11 13:44:17 -07:00
2619e4d3df
Use version clock to mitigate network clock skew.
2020-09-11 13:39:16 -07:00
62c81e03c7
changed incremental backup workload to have an option to check system keys for version
2020-09-11 16:02:13 -04:00
d5fba9a69b
add write-tag throttling
2020-09-11 19:10:35 +00:00
260c8d9568
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-snapshot-record-version
2020-09-11 15:05:58 -04:00
22996284c7
added changes to allow writing of last epoch end version to special keys when performing recovery due to snapshot
2020-09-11 15:00:11 -04:00
224f23b0f8
Rely on MasterRecoveryState message since we only care about the current generation.
2020-09-11 11:45:02 -07:00
195c55df4b
add comment
2020-09-11 16:48:21 +00:00
9947b416ee
Merge branch 'master' of https://github.com/apple/foundationdb into fdbtest
2020-09-11 15:27:35 +00:00
084c69b2ad
fix reason serialization bug
2020-09-11 07:21:57 +00:00
35bef73a1c
Rename proxy to commit proxy
2020-09-10 17:44:15 -07:00
732a457ba4
Merge pull request #3720 from sfc-gh-xwang/mako
...
update mako and WriteTagThrottling workload so as to trigger tag throttling event
2020-09-10 15:21:00 -07:00
1867ee1f5f
Change cli output format
2020-09-09 22:34:36 -07:00
d270b15c89
Merge remote-tracking branch 'upstream/master' into add-setclass-to-special-key-space
2020-09-09 16:21:19 -07:00
2c224de2f8
Update test of setclass
2020-09-09 16:19:55 -07:00
ce92f1a224
FR:Init StagingKey when created
...
The key field was used in various places, such as figuring out the conflict key range.
We should not leave it empty
2020-09-09 16:12:32 -07:00
5c5abd7afa
FastRestoreApplier:Calculate conflict range in applyStagingKeysBatch
2020-09-09 15:00:25 -07:00
2febbe74ce
FastRestoreApplier:Fix conflict range inverted due to invalid memory access
...
Rerpot error on loader and applier if not error_code_operation_cancelled error ever happens
2020-09-09 14:40:21 -07:00
60efc5d44b
Don't set SPECIAL_KEY_SPACE_ENABLE_WRITES randomly in Fuzz API test
2020-09-09 12:40:05 -07:00
f10e9ea679
FastRestoreApplier:Add write conflict range
2020-09-09 12:12:14 -07:00
1155d015c9
fetch current log generation as well
2020-09-09 11:54:58 -07:00
e69ee87f33
Refactor blob failure to simulateBlobFailure function
2020-09-09 10:54:21 -07:00
c9171e15b8
Change FastRestoreApplierPrecomputeResultSkipUnexpectedBackupMutation to SevError
2020-09-09 10:42:51 -07:00
2ce81a3c44
Make WATCH_OVERHEAD_BYTES constexpr
2020-09-09 10:35:42 -07:00
5880bd9fa7
BackupContainer:Fix getSnapshotFileKeyRange_impl on blob http error
2020-09-09 08:47:43 -07:00
83d1350d8d
FastRestore:Handle retriable blob error
2020-09-09 07:29:17 -07:00
6d58064964
Buggify _parsePartitionedLogFileOnLoader with blob http error
2020-09-09 06:46:08 -07:00
cffb54ba0b
Buggify decodeLogFileBlock with blob http error
2020-09-09 06:44:02 -07:00
f3986d201d
Buggify backup decodeRangeFileBlock with blob connection error
2020-09-09 06:40:41 -07:00
9eeed8144e
FastRestore:Fix bug in precomputeResult
2020-09-08 16:37:12 -07:00
916c4ef628
Merge branch 'master' of https://github.com/apple/foundationdb into fdbtest
2020-09-08 23:29:39 +00:00
62dd1f7234
Merge pull request #3696 from sfc-gh-xwang/tag-report
...
report busiest write tag of each storage server
2020-09-08 15:21:14 -07:00
90984467c8
Add comments for WATCH_OVERHEAD_BYTES
2020-09-08 15:19:31 -07:00
a60342a5ba
Merge pull request #3757 from yliucode/grv-metrics
...
Fix status fetcher for GrvProxyStats
2020-09-08 15:15:50 -07:00
2935d3d4f6
change workload; solve some comments
2020-09-08 21:47:49 +00:00
4363dd0f25
This resolves issue #3739 by exposing time since last full recovery.
2020-09-08 14:26:01 -07:00
23e1ff694c
Report missing old tlogs in recovery between accepting commits and storage recovered
2020-09-08 13:35:42 -07:00
223c419f35
Merge pull request #3676 from sfc-gh-jfu/jfu-incremental-backup-only
...
incremental backup and restore
2020-09-08 12:06:25 -07:00
cda9e93fe0
Clang-format
2020-09-08 11:37:27 -07:00
6c3d919295
Fix status fetcher for GrvProxyStats
2020-09-08 11:11:45 -07:00
43e3e320e3
Refactor getrange for read-write module and add a test to make sure we have consistent results
2020-09-08 11:08:48 -07:00
Jon Fu