a188f91fe6
Merge pull request #3738 from dongxinEric/misc/remove-unused-client-knob
...
The actual BYTE_LIMIT_UNLIMITED and ROW_LIMIT_UNLIMITED are defined i…
2020-09-08 10:46:27 -07:00
0eb7092faa
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-incremental-backup-only
2020-09-08 12:53:29 -04:00
c3e153b5d9
Merge branch 'master' of https://github.com/apple/foundationdb into tag-report
2020-09-08 15:35:22 +00:00
d9bfdd0196
Merge pull request #3755 from apple/release-6.3
...
Merge Release 6.3 into master
2020-09-07 18:05:46 -07:00
1ee40848df
Merge pull request #3753 from xumengpanda/mengxu/fr-fix-trace-PR
...
FastRestore:Fix trace event overflow in traceRoleVersionBatchProgress
2020-09-07 00:08:57 -07:00
b9e43bca5d
Merge pull request #3754 from xumengpanda/mengxu/fr-valgrind-asset-PR
...
FastRestore: Fix valgrind error
2020-09-07 00:06:06 -07:00
fc01b7558c
FastRestore:fix uninitialized variable in param.asset
2020-09-06 22:40:57 -07:00
684048f449
FastRestore:Fix trace event overflow in traceRoleVersionBatchProgress
2020-09-06 21:05:16 -07:00
1ad5e17458
add support for comparing original and current impls
2020-09-05 11:14:59 -07:00
efde86340a
Add knob to disable span serialization
2020-09-04 17:37:34 -07:00
783e6a170e
Add code coverage
2020-09-04 17:36:56 -07:00
7dc55fdffd
Revert state
2020-09-04 15:36:47 -07:00
b96dbc45cb
Update formatting
2020-09-04 15:36:47 -07:00
00d3aa3acc
Update formatting
2020-09-04 15:36:47 -07:00
53b7721d6c
Add additional trace information
2020-09-04 15:36:47 -07:00
9398025f6a
Remove test
2020-09-04 15:36:47 -07:00
f896c68996
Cleanup
2020-09-04 15:36:47 -07:00
2a58e775d2
Add original changes
2020-09-04 15:36:47 -07:00
1ca7fe1a05
Add span metadata message
2020-09-04 15:36:47 -07:00
a0204caccc
Merge remote-tracking branch 'myrepo/mako' into fdbtest
2020-09-04 19:07:52 +00:00
7a8cedd42b
format
2020-09-04 18:58:46 +00:00
ecf2c0109c
more concise status json
2020-09-04 18:40:45 +00:00
5b5087c566
format
2020-09-04 16:34:05 +00:00
7660fb3beb
report busiest tags in status json
2020-09-04 16:33:59 +00:00
f94a78ff03
update TagThrottleApiWorkload
2020-09-04 16:31:10 +00:00
ec56e1f286
merge with master
2020-09-04 16:11:42 +00:00
341feb2b6f
fix compare bug
2020-09-04 07:17:35 +00:00
ae7bf24353
Merge pull request #3549 from yliucode/grv-proxy
...
Separate out a new role GrvProxy to serve GRVs.
2020-09-03 19:03:45 -07:00
4e511295a6
Use the correct ROW_LIMIT_UNLIMITED value
2020-09-03 15:08:41 -07:00
42298647e3
change ProxyStats to GrvProxyStats
2020-09-02 17:09:09 -07:00
87693cae81
merge master branch and resolve conflicts
2020-09-02 13:44:33 -07:00
b4c96cadc7
Merge branch 'release-6.3' into merge-release-6.3-into-master
2020-09-02 12:45:57 -07:00
d334b6484e
attempt to write to system keys with snapshot
2020-09-02 15:17:54 -04:00
0c508660dc
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-incremental-backup-only
2020-09-02 13:45:44 -04:00
8ac3b21459
changed occurrences of getOrThrow to get
2020-09-02 13:45:15 -04:00
c6477bbc61
Merge branch 'master' into feature/range-split-points-based-on-size
2020-09-02 09:41:17 -07:00
834426e848
Merge pull request #3725 from sfc-gh-clin/add-latency-for-local-operations-to-mako-workload
...
Log latency for set, clear, clearrange in MakoWorkload
2020-09-01 18:29:10 -06:00
5078bc5728
clang-format MakoWorkload
2020-09-01 17:28:32 -07:00
83b09784c7
Change now() to timer()
2020-09-01 17:20:53 -07:00
34813f9e63
Change ms to us
2020-09-01 16:33:06 -07:00
ce7a35a4be
Log latency for set, clear, clearrange
2020-09-01 16:14:43 -07:00
0443ea7a9b
fix: prioritize marking a region as fully replicated over removing machine teams
2020-09-01 15:55:33 -07:00
de3dde5f51
Merge pull request #3716 from ajbeamon/fix-memory-error
...
Remove a line that logs whether confChange was non-null
2020-09-01 13:20:49 -07:00
b9baa996e3
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-incremental-backup-only
2020-09-01 14:38:02 -04:00
cf19c666f0
Merge pull request #3718 from etschannen/master
...
Merge release 6.3 into master
2020-09-01 09:00:34 -07:00
972b776a4d
Merge pull request #3719 from sfc-gh-xwang/master
...
fix rate calculation typo
2020-09-01 00:34:51 -07:00
47893c847b
fix rate calculation typo
2020-09-01 07:19:46 +00:00
4001f9052e
remove vcxproj files
2020-08-31 23:27:56 -07:00
4e0dcd8a50
removed orphaned request tracking in ProxyCommitData
2020-08-31 21:25:54 -07:00
12edadd059
Merge branch 'release-6.3'
...
# Conflicts:
# CMakeLists.txt
# fdbclient/Knobs.cpp
# fdbclient/MasterProxyInterface.h
# fdbrpc/simulator.h
# fdbserver/MasterProxyServer.actor.cpp
# tests/fast/CycleAndLock.txt
# tests/fast/TxnStateStoreCycleTest.txt
# tests/fast/VersionStamp.txt
# tests/slow/ParallelRestoreOldBackupApiCorrectnessAtomicRestore.txt
# tests/slow/ParallelRestoreOldBackupCorrectnessCycle.txt
# versions.target
2020-08-31 19:33:34 -07:00
fb477399f8
Remove a line that logs whether confChange was non-null. This was previously a pointer, and the logging indicated whether the caller expected a result. It is now a reference type, and we can't determine what the caller expects.
2020-08-31 16:25:39 -07:00
974c9248f8
The workload sometimes would wrongly report a passed tests as failed due to not return after the hotkey is found.
2020-08-31 14:09:01 -07:00
944c78ac54
update WriteTagThrottling workload
2020-08-31 18:36:10 +00:00
4c6fac058b
Merge pull request #3709 from etschannen/release-6.3
...
Disabled two features using knobs for performance reasons
2020-08-31 10:39:59 -07:00
5c97461d18
Merge pull request #3653 from etschannen/feature-proxy-busy-loadbalance
...
Changed proxy load balancing to balance on CPU usage
2020-08-31 10:39:08 -07:00
bbb178c430
Merge pull request #3701 from satherton/fix-redwood-correctness-timeout
...
Prevent Redwood unit test timeout from trying to do too much work
2020-08-31 10:38:20 -07:00
d42a6b6ea7
remove spammy trace event
2020-08-31 10:37:00 -07:00
8994719e46
Merge branch 'master' into grv-proxy
2020-08-31 10:21:32 -07:00
fa306e07fd
temporarily disable hot read detection on the storage servers because it is causing a 3% to 5% performance regression in read heavy workloads
2020-08-31 09:30:51 -07:00
53def68946
fix compiler warning
2020-08-31 01:23:47 -07:00
29eec30183
Merge branch 'release-6.2' into release-6.3
...
# Conflicts:
# CMakeLists.txt
# build/Dockerfile
# build/Dockerfile.devel
# documentation/sphinx/source/downloads.rst
# fdbserver/Knobs.cpp
# fdbserver/LogSystem.h
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/TagPartitionedLogSystem.actor.cpp
# fdbserver/WaitFailure.actor.cpp
# fdbserver/fdbserver.vcxproj
# fdbserver/fdbserver.vcxproj.filters
# packaging/msi/FDBInstaller.wxs
2020-08-31 01:10:29 -07:00
d750ed303a
fixed whitespace
2020-08-30 22:34:03 -07:00
331a49a62a
do not allow a proxy to reset a connection with the logs immediately upon starting up
2020-08-30 18:50:19 -07:00
89ae0aaf39
In Redwood unit test, if the verify actor falls far behind the latest commit version it will unnecessarily verify the latest version repeatedly which extends the test's run time needlessly.
2020-08-29 23:47:54 -07:00
167d3c5a8f
To avoid running too long, the Redwood correctness unit test now limits the size of the KV verification map. This map only grows as the test runs, as it contains a record for every key set or clear committed during the test.
2020-08-29 16:44:38 -07:00
e87327b33b
Merge master branch and keep master proxy reporting txn cost estimation to ratekeeper
2020-08-29 12:47:35 -07:00
00c77ba2b4
Added beginVersion cmd line option and addressed code review comments
2020-08-28 14:29:22 -04:00
4831f11e2a
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-incremental-backup-only
2020-08-28 12:47:19 -04:00
0270ee13c8
change targetversion from maxlogend to contiguouslogend. also adjusted test workload and specs
2020-08-28 12:46:34 -04:00
f6f9aea09e
fix: runLoopBusyness was always zero
2020-08-28 09:29:54 -07:00
ce1139e588
added missing dumpToken trace events
2020-08-27 17:17:27 -07:00
86cba16d7d
Added logging whenever a wait failure client triggers
2020-08-27 17:07:05 -07:00
ca9b1f5b34
Merge branch 'release-6.3' into mengxu/fr-sched-PR
...
Resolve conflict at BackupContainer.actor.cpp
2020-08-27 16:54:00 -07:00
cd86ca6850
Merge branch 'master' into feature/range-split-points-based-on-size
2020-08-27 10:06:02 -07:00
c3991262cf
Add nullptr check to traceAllInfo
2020-08-27 09:40:42 -07:00
ca486fdcd4
FastRestoreOldBackupTest:Revert workload time to 30s
2020-08-27 09:31:57 -07:00
d36d61e0ba
Merge pull request #3571 from sfc-gh-xwang/master
...
Consider write/clear operation in ratekeeper
2020-08-26 18:21:27 -07:00
7a29a3157f
FastRestore:Fix various bugs discovered by enhanced simulation
...
1. sendMutation request can be dispatched when the version batch has finished and its data has been deleted;
2. Request scheduler on loader may get into infinite loop because FASTRESTORE_SCHED_LOAD_REQ_BATCHSIZE knob can be set to 0 in simulation
2020-08-26 15:53:18 -07:00
6d0610c6b4
Merge pull request #3683 from sfc-gh-ljoswiak/setup-trace-batch
...
Add call to set up tracing on transaction commit path
2020-08-26 13:24:37 -06:00
74edf61745
Merge pull request #3674 from sfc-gh-anoyes/anoyes/read-conflict-dbLocked
...
Add read conflict on \xff/dbLocked
2020-08-26 12:18:42 -07:00
507c67c930
Added additional information to trace events
2020-08-26 11:42:23 -07:00
919c78d7ed
Add wait to onError
2020-08-26 11:02:22 -07:00
28cb5f242c
another fix
2020-08-26 11:01:40 -07:00
e81ccd2dc9
another compiler fix
2020-08-26 10:59:06 -07:00
e531046b53
fix compiler errors
2020-08-26 10:56:21 -07:00
fd1a4304fa
fix: made ConnectionResetInfo reference counted
2020-08-26 10:53:17 -07:00
f1bd2a18ed
Resovle review comments: No functional change
2020-08-26 10:30:31 -07:00
3d2b18b663
FastRestore:AtomicRestore:Add resetDBTimeout option
2020-08-26 10:21:17 -07:00
3ca1359e89
FastRestore:Simulate cpu busy and adjust simulation of overused memory usage
...
Increase the chance of cpu busy and memory overused
2020-08-26 10:07:49 -07:00
bc766ab7ef
FastRestoreKnob:Adjust knobs
2020-08-26 09:54:33 -07:00
a8bd628216
FastRestoreAtomicRestoreTest:Increase timeout from 2100 to 36000
2020-08-26 09:48:52 -07:00
6c981096ec
Add test for \xff\xff/configuration/process/class_source/
2020-08-25 19:46:13 -07:00
a07b9f234d
Add readonly range \xff\xff/configuration/process/class_source, and change \xff\xff/configuration/class/ to \xff\xff/configuration/process/class_type/
2020-08-25 18:18:32 -07:00
4e08aca32e
Change SevInfo to SevError
2020-08-25 14:58:47 -07:00
d8e73fddb6
FastRestore:Cancel actors when restore request finishes
2020-08-25 14:46:26 -07:00
b1c206b62a
change rate calculation
2020-08-25 18:47:13 +00:00
bd7c07436b
FastRestore:Add batchIndex to RestoreAsset for better performance tracking
2020-08-25 09:34:18 -07:00
23c8d0154b
FastRestoreOldBackupTest:Increase test duragion
...
Add comment on Rollback test as well
2020-08-25 09:14:37 -07:00
778daf20c0
FastRestore:Fix incorrect assert
2020-08-24 19:59:56 -07:00
996ba2374c
FastREstore:Fix:Incorrect condition in printing out FastRestoreLoaderSendMutationToApplierDoneTooLate
2020-08-24 17:27:40 -07:00
63b3612ad5
Merge master branch and resolve conflicts
2020-08-24 16:42:31 -07:00
e94261efe5
FastRestore:LoaderScheduler:Add validation on too old requests
2020-08-24 16:32:16 -07:00
4e4fa0fded
merge with master
2020-08-24 21:04:53 +00:00
e7662eecda
Merge pull request #3669 from sfc-gh-xwang/tag-report
...
Report recommended tx tag to be throttled to status json
2020-08-24 12:06:25 -07:00
6e3e36c8fc
FastRestore:RequestScheduler:Minor code style improvement
2020-08-24 10:45:46 -07:00
3afdb44c7a
merge master
2020-08-23 17:09:04 +00:00
b4cec6577a
Fix merge bugs
2020-08-22 15:04:42 -07:00
e1b7dd0c7d
Merge remote-tracking branch 'upstream/release-6.3' into dyoungworth/fixMerge1
2020-08-22 12:25:19 -07:00
7e094f217b
Merge pull request #3675 from ajbeamon/fix-proxy-latency-bands
...
Fix commit and GRV latency band and statistics publishing
2020-08-22 10:53:33 -07:00
a552181c48
Merge pull request #3652 from satherton/feature-redwood
...
Optimization to reduce page writes when multiple siblings of the same parent are updated with the page remap window
2020-08-21 17:14:31 -07:00
3b63d8b01b
remove FIXME; remote tagSet.reset(); trivial changes
2020-08-21 19:17:16 +00:00
f864606d8d
Don't block the data distributor when getting a GetDataDistributorMetricsRequest.
2020-08-21 18:16:07 +00:00
5cebba8e74
Add call to set up tracing
2020-08-21 11:06:31 -07:00
6380b92b10
Don't block the data distributor when getting a GetDataDistributorMetricsRequest.
2020-08-21 09:26:18 -07:00
951699af4f
Refine tests for management special keys
2020-08-21 00:57:04 -07:00
1f134d1534
format
2020-08-21 05:06:13 +00:00
9398a78a3a
add busy-read count and busy-write count to status json
2020-08-21 04:50:56 +00:00
88cb352e2e
rename variables
2020-08-21 00:03:06 +00:00
5aaf86206a
add counter for expensive clear estimation for audit and trace
2020-08-20 23:00:45 +00:00
802125e01a
clang-format
2020-08-20 14:50:30 -07:00
bff20faa42
Merge remote-tracking branch 'upstream/master' into add-setclass-to-special-key-space
2020-08-20 13:50:54 -07:00
707b88583a
Add default encode, decode methods. Add test for setclass special keys
2020-08-20 13:50:35 -07:00
e375761581
Merge branch 'master' of https://github.com/apple/foundationdb
2020-08-20 20:29:14 +00:00
9dd4589d61
Fix merge issue that resulted in commit and GV latency band info not being published.
2020-08-20 11:49:30 -07:00
f0cdfb8781
Fix memory error caused by referencing a value with limited lifetime.
2020-08-20 11:12:59 -07:00
4dc2975f5a
Only switch primary if extraDB != nullptr
2020-08-20 18:06:24 +00:00
30c086ed96
Improve comment
...
Also use push_back. The lifetime of databaseLockedKey and
databaseLockedKeyEnd will outlive trRequest.arena, since they're
literals.
2020-08-20 18:05:03 +00:00
9cf3041393
Add read conflict for metadata txs that aren't lock-aware
2020-08-20 02:52:37 +00:00
5660de9c09
Add \xff\xff/configuration/class to change process class
2020-08-19 17:54:38 -07:00
fbc97a12b8
Add LockDatabaseFrequently workload
2020-08-19 23:17:38 +00:00
ae999aa118
Merge branch 'master' of https://github.com/apple/foundationdb into jfu-incremental-backup-only
2020-08-19 16:36:47 -04:00
9c709b33d6
scale cost
2020-08-19 19:47:41 +00:00
bc6e42c634
add status json report for recommended throttled tags
2020-08-19 19:22:16 +00:00
0bc2e4f58d
revert changes made to BackupCorrectness.actor.cpp
2020-08-19 15:04:38 -04:00
422784d545
FastRestore:Send reply before assert fail
2020-08-19 11:25:08 -07:00
35aa1034cd
added new test spec and workload to verify correctness of incremental backup and restore
2020-08-19 13:45:51 -04:00
d9ea14ea6c
FastRestore:fix:loader can receive reply from vb that has been processed and deleted
2020-08-19 10:39:49 -07:00
8a1b9394b8
do not send auto-throttled tags to proxy is autoThrottleEnabled == false
2020-08-19 16:23:11 +00:00
22a2fac689
FastRestore:Fix segmentation fault when previous duplicate request is sent too late
...
This seg fault was not caught by simulation test;
It is only reproduced very easily in circus test.
Add an ASSERT to check if the scenario happens in simulation
2020-08-19 09:15:18 -07:00
88b94a3e59
FastRestore:Fix:Delayed request can be skipped from processing
...
When a request of sending mutations to appliers is delayed and the request for future version batches have been processed,
loader will increase its counter. The previous implementation will use the counter to decide if a request is too old and
have been processed. If the counter is larger than the batch index of the request, the request will be ignored although
it may not have been processed.
This causes correctness test failure for fast restore.
2020-08-18 23:48:06 -07:00
599675cba8
modify some details to get better performance
2020-08-19 04:23:23 +00:00
d53e6cf687
Bug fix in KV byte counts.
2020-08-18 17:58:57 -07:00
e1d150662f
FastRestore:small fix on compile error
2020-08-18 16:15:11 -07:00
6729b4143b
FastRestore:Requests:Add const for toString function
2020-08-18 16:05:23 -07:00
a32cf7d293
FastRestore:Loader:Add sanity check before remove batch data
2020-08-18 15:44:22 -07:00
9b2f667bbe
FastRestore:Fix uninitialized variable
2020-08-18 11:58:57 -07:00
69914d4909
defer write throttling until found proper statistics for write saturation
2020-08-18 15:46:02 +00:00
8ede143941
Track tlog push latencies and reset connections if they are above 500ms
2020-08-18 08:43:14 -07:00
046260b9d7
FastRestore:Applier:Assert to ensure batchData will not be used after deleted
2020-08-17 22:42:41 -07:00
4a0315483b
FastRestore:Safeguard when request of earlier vb may be sent after the vb has finished
2020-08-17 22:20:54 -07:00
7b7490efe7
FastRestore:Debug trace for seg fault
2020-08-17 20:34:33 -07:00
24dbb3961c
Knob adjustment
2020-08-17 22:38:22 +00:00
421ed80295
Recover the machine count to be 12
2020-08-16 15:10:36 -07:00
01255b7ead
FastRestoreLoader:Sched:Add trace field and misc code style improvement
2020-08-16 09:43:04 -07:00
be75fd4894
Workaround flow limitation:state variable must be initialized at the beginning of actors
2020-08-16 08:45:13 -07:00
c2dfc5a00a
Fix compile error
2020-08-16 08:13:28 -07:00
cb0e44ff42
FastRestoreLoader:Sched send mutations per loadingParam requests
2020-08-16 07:38:51 -07:00
98167a6d20
FastRestoreLoader:Fix:sched may stuck due to race condition
2020-08-16 06:29:09 -07:00
015397be74
FastRestore:LoaderScheduler:Control number of load request to release at once
2020-08-15 22:18:48 -07:00
1e571a5a1a
FastRestore:Loader:Kick off scheduler when loader starts to have new requests
2020-08-15 21:57:00 -07:00
a035e7a872
FastRestore:LoaderScheduler:Add knobs
2020-08-15 21:44:51 -07:00
7e302b5910
FastRestore:Loader:Add scheduler to dispatch requests based on resource utilization and request priority
2020-08-15 21:21:51 -07:00
090241f0df
Count BTreeCursor page reads that come from the cursor's cache of pages as cache hits in RedwoodMetrics. BTree Unit Test now verifies all data at the initial version after recovery from disk.
2020-08-15 14:30:49 -07:00
84a4d040c1
Merge branch 'release-6.3' of github.com:apple/foundationdb into feature-redwood
2020-08-15 14:09:03 -07:00
ffa1973242
move OPERATION_COST_BYTE_FACTOR to CLIENT_KNOB; skip the first call of refreshBusiestTag.
2020-08-14 22:49:20 +00:00
daead84fdd
Merge master
2020-08-14 14:07:08 -07:00
548ba06510
change map to deque; code style problem
2020-08-14 18:30:58 +00:00
94571786bf
Add RocksDB work
2020-08-14 17:16:20 +00:00
475d4cca72
solve some cr comments
2020-08-14 16:40:16 +00:00
e94de126de
merge with master
2020-08-14 01:59:09 +00:00
ce05bb2a26
Merge pull request #3617 from xumengpanda/tmp/trace-PR
...
Resolve throttling events
2020-08-13 16:31:55 -07:00
19df032aec
Change some formatting issues
2020-08-13 15:30:21 -07:00
4a30492186
Remove debug trace
2020-08-13 14:42:00 -07:00
20700f75f4
Merge master
2020-08-13 14:28:27 -07:00
6f98ed3d24
Merge branch 'release-6.3' into feature-proxy-busy-loadbalance
2020-08-13 11:30:28 -07:00
051ec7066a
BTree unit test duration now targets a number of page read/writes so it should time out less often.
2020-08-13 01:50:51 -07:00
ced4f8c49f
fix: getReadHotRanges should be const function
2020-08-12 22:18:28 -07:00
002cca5e58
Merge branch 'master' into tmp/merge-PR
...
Conflicts:
fdbclient/NativeAPI.actor.cpp
fdbserver/StorageMetrics.actor.h
2020-08-12 20:08:13 -07:00
539366156c
more fixes
2020-08-12 17:26:20 -07:00
7c84e04028
fixed compiler errors
2020-08-12 17:22:37 -07:00
c72068d6b5
clients load balance across proxies based on process busyness instead of number of requests
2020-08-12 17:17:21 -07:00
8953cc5f98
Merge master again
2020-08-12 16:59:07 -07:00
8f1deaa47e
Merge branch 'master' of https://github.com/apple/foundationdb
2020-08-12 23:32:23 +00:00
e13e7c5725
Update fdbserver/workloads/ReadHotDetection.actor.cpp
...
Co-authored-by: Jingyu Zhou <jingyuzhou@gmail.com>
2020-08-12 15:48:01 -07:00
b17667323e
Resolve review comments
2020-08-12 15:15:25 -07:00
3ac6996844
Increased some knobs to throttle the spammy read hot logging. Also added more details inside the read hot log to make it useful
2020-08-12 14:27:34 -07:00
215a0ad42c
Merge branch 'release-6.3' of github.com:apple/foundationdb into feature-redwood
2020-08-12 12:06:56 -07:00
f3ecf14601
change midShardSize type and other details
2020-08-12 17:49:12 +00:00
0d0efcfce7
FR:Fix incorrect ASSERT:batchIndex starts at 1 instead of 0
2020-08-12 10:36:31 -07:00
16ef2bd3bd
Pending commit
2020-08-12 10:34:07 -07:00
a817c9abbe
Redwood metrics was adding details with empty names.
2020-08-12 03:24:23 -07:00
729fce9f0e
Merge branch 'release-6.3' of github.com:apple/foundationdb into feature-redwood
2020-08-12 02:59:01 -07:00
c1ccbc3d27
Bug fixes. Internal page update would be missed if child page ID updates were expected but none were done. When rebuilding internal pages, child page ID updates were being written to the original page memory instead of the rebuild record set.
2020-08-12 01:33:07 -07:00
2f52c5f79b
Merge branch 'release-6.3'
...
# Conflicts:
# fdbserver/RestoreLoader.actor.cpp
2020-08-11 22:45:43 -07:00
79ce16650d
merge master branch
2020-08-11 19:22:10 -07:00
ba803a5ea3
Fixed formatting issues and removed GRV related code in MasterProxy
2020-08-11 18:54:54 -07:00
4268f3c900
fixup! Fix the bug introduced in commit a7cff484bc
2020-08-11 18:21:10 -07:00
0cceda9908
solve distributor present bug
2020-08-11 21:54:52 +00:00
06dae74730
Revert change to ReadHotRangeLog event
2020-08-11 11:04:57 -07:00
97e49f2f70
Resolve throttling events
2020-08-10 22:01:12 -07:00
7ee6ca342e
merge with master
2020-08-11 01:01:15 +00:00
c0b7ac0b7d
try to throttle write tag and read tag seperatedly
2020-08-10 23:43:58 +00:00
cae6f04070
sample on cost instead of bytes
2020-08-10 22:30:06 +00:00
c5c6906a3d
Merge pull request #3600 from xumengpanda/mengxu/fr-fix-large-pkg-PR
...
Fast Restore: Fix the sampling packet size at sampling phase
2020-08-10 13:53:24 -07:00
696e77c94e
query midShardSize from proxy
2020-08-10 20:13:44 +00:00
c18d16ad30
Multiple sibling page updates within the remap cleanup window can now avoid the second update step, copying updated contents over the original page, by instead updating the parent page to point to the new sibling locations.
2020-08-09 03:04:38 -07:00
c4d2d66de7
Merge master branch and resolve conflicts
2020-08-07 18:04:00 -07:00
c89a62b935
FR:SampleBackup:Assert req.batchIndex < controller batch size
2020-08-07 16:52:07 -07:00
c27542d4e9
FR:SampleBackup:Send reply when receive duplicate msg
2020-08-07 16:50:36 -07:00
df9149fea4
ignore transaction tag of immediate transactions
2020-08-07 23:36:17 +00:00
104bac3cbd
Add trace to debug
2020-08-07 13:02:41 -07:00
56cc15ee71
Add trace to debug
2020-08-07 01:02:07 -07:00
fccebd9b80
FR:Use ByteSampleInfo to accurately sample backup
2020-08-06 20:45:46 -07:00
a46e620baa
Merge branch 'master' into feature/range-split-points-based-on-size
2020-08-06 15:23:11 -07:00
b9f41f3975
Rename CommitBatch::Context to CommitBatch::CommitBatchContext
2020-08-06 11:40:40 -07:00
d8bb36c4c8
Refactor applyMetadataMutations to accept less args
2020-08-06 11:40:40 -07:00
ed1bbdb8a7
fixup! Update code per review
2020-08-06 11:40:40 -07:00
377f0695b0
Refactor postResolution
2020-08-06 11:40:40 -07:00
9f8244f2dd
Add a shortcut for applyMetadataMutations
2020-08-06 11:40:40 -07:00
376aff079c
Extract Step 4 and 5 from commitBatch
2020-08-06 11:40:40 -07:00
a7cff484bc
Extract Step 3 from commitBatch
2020-08-06 11:40:40 -07:00
ccb103caf1
Extract Step 1 and 2 from commitBatch
2020-08-06 11:40:40 -07:00
87bde08695
FR:fix:sending samples may have broken promise error
...
The error should not skip the ack reply from loader to controller, saying
the file has been loaded.
Otherwise, restore controller can get stuck at waiting for the ack reply.
2020-08-06 10:41:18 -07:00
ba66b1f668
trivial changes
2020-08-06 14:53:26 +00:00
2e41391690
Merge master branch
2020-08-06 00:11:00 -07:00
d6a23a4d6b
Resolve comments to make GRV proxy a separate process class
2020-08-06 00:01:57 -07:00
fe5902994c
Merge pull request #3605 from apple/release-6.3
...
Merge Release 6.3 to master
2020-08-05 23:37:44 -07:00
8defd8ff8d
reset transaction at the end of the test
2020-08-05 13:25:21 -07:00
b00b1bb30f
Add a test for management api options
2020-08-05 13:20:12 -07:00
ca5705b5ea
Merge pull request #3576 from Daniel-B-Smith/rocksdb-fixes
...
RocksDB fixes
2020-08-05 11:32:22 -07:00
859c3145e2
FR:Fix samples not send reply and false positive failures
2020-08-05 11:32:04 -07:00
9640ba42d3
merge master
2020-08-05 16:23:10 +00:00
309f383be3
FR:Report error if controller throws error
2020-08-04 23:08:41 -07:00
879998d01b
FR:fix compilation error
2020-08-04 23:04:38 -07:00
7992cef025
FR:Fix sample network pkg can be too big
2020-08-04 22:35:21 -07:00
13307679c5
use median shard size"
2020-08-05 03:57:25 +00:00
d21e9766fe
fix #3512 merge problem
2020-08-05 01:35:32 +00:00
21635f8a28
update backup restore for local testing
2020-08-04 15:48:43 -04:00
4997754cf8
fix format in monitorDDMetricsChanges
2020-08-04 04:14:11 +00:00
484a75dd7b
compare read and write tag together
2020-08-04 00:02:41 +00:00
52073738c0
fix bugs; change some KNOB values
2020-08-03 22:18:34 +00:00
e9adec421d
FR:Add controller interface and send to roles
2020-08-03 15:06:15 -07:00
f071d81ad0
Report warning on FastRestoreApplierClearRangeMutationsStart if delayTime is too large
2020-08-03 14:08:31 -07:00
b903e60cb7
fix monitorDDMetricsChanges bugs
2020-08-03 17:12:36 +00:00
6db7040f17
update DDMetrics periodically
2020-08-03 05:52:38 +00:00
d1cc87452c
merge with master; solve conflicts; solve initialization;
2020-08-02 22:44:07 +00:00
0352e8ee0b
pick busiest commit tag periodically
2020-08-02 18:38:56 +00:00
92c1112c74
consider clear single key
2020-08-01 18:20:13 +00:00
d9e5d65aaf
Merge remote-tracking branch 'upstream/master' into add-management-api-into-special-key-space
2020-08-01 00:31:26 -07:00
1f38c2f2a4
check clear sample on proxy
2020-08-01 06:48:40 +00:00
0c49646486
Fix:FastRestore:rangeVersion must be 0 when no range file is used
...
otherwise, all mutations will be ignored by loaders as old mutations
2020-07-31 23:23:17 -07:00
4f7dab4951
sample clear op on client
2020-08-01 06:14:52 +00:00
27a3d37594
RestoreApplier:Change receivedBytes from double to long
2020-07-31 22:10:18 -07:00
e22baa552b
FastRestoreApplier:Init stats variables
2020-07-31 21:56:03 -07:00
f36d5aa180
FR:Applier received bytes per batch
2020-07-31 17:48:55 -07:00
fbe36ced23
Rewrite all \xff\xff/reboot_worker call
2020-07-31 16:33:16 -07:00
0e3f4285ec
added missing paraetheses
2020-07-31 16:05:21 -07:00
fec3de9b7b
Fix RestoreLoader loadedLogBytes counter
2020-07-31 16:00:15 -07:00
5c7b4c7acf
Merge branch 'master' of github.com:apple/foundationdb
2020-07-31 15:58:13 -07:00
81cdc64c58
re-added nowAssigned
2020-07-31 15:46:46 -07:00
c18dcb500c
fixed compile error
2020-07-31 15:39:37 -07:00
47c35a7a69
FastRestore:Add stats to ApplierBatchData
2020-07-31 14:59:45 -07:00
e4d2ab72fe
DumpVersionBatch with log and file summary info
2020-07-31 14:21:33 -07:00
fc3dfc170f
Merge branch 'master' of https://github.com/apple/foundationdb
2020-07-31 19:09:33 +00:00
c3a629588f
add client transaction tag sample
2020-07-31 19:08:42 +00:00
2f5293fcc7
Introduce knob FASTRESTORE_USE_LOG_FILE and FASTRESTORE_USE_RANGE_FILE
2020-07-31 10:40:29 -07:00
37c3bd8615
FastRestore:Ensure FASTRESTORE_NOT_WRITE_DB only work in non simulation mode
2020-07-30 20:27:54 -07:00
f2662a9a74
Merge remote-tracking branch 'upstream/master' into add-management-api-into-special-key-space
2020-07-30 19:01:11 -07:00
10540a3c73
Change error name and fix a bug in ExclusionInProgressRangeImpl::getRange()
2020-07-30 18:58:09 -07:00
8cace30bb2
FastRestore:change TXN_BATCH_MAX_BYTES default to 1KB from 1MB
2020-07-30 16:43:35 -07:00
bfa4eb9ab2
Resolve comments
2020-07-30 14:45:03 -07:00
db30943fbf
Merge pull request #3399 from sfc-gh-tclinkenbeard/vectorref-emplace-back
...
Add VectorRef::emplace_back
2020-07-30 13:01:22 -07:00
d16db8e733
FastRestore:Fix segmentation fault
2020-07-30 12:10:32 -07:00
d71361245b
FastRestore:Short cut DB for get and clearange
2020-07-30 11:17:05 -07:00
ad915e462e
Add knob FASTRESTORE_NOT_WRITE_DB to skip writting to DB
2020-07-30 10:17:17 -07:00
1bd6ea51ac
Merge pull request #3572 from xumengpanda/mengxu/master-dd-failure-PR
...
Fix master branch failure on DDBalanceAndRemove.toml test
2020-07-30 10:07:41 -07:00
17056b2a24
Fix comment grammar typo
...
Co-authored-by: Xin Dong <jiangzian1987dx@gmail.com>
2020-07-30 10:02:11 -07:00
6efd5996c2
Calculate data stored by Rocks
2020-07-29 22:08:32 +00:00
871402f68d
Add File field in TestFailure event for cycleCheck and incrementCheck
...
In workloads, cycleCheck and incrementCheck, both might trigger a test
failure with reason "Rate below desired rate", and the other text are
the same. Adding a File field will help differentiate these two events.
2020-07-29 14:57:05 -07:00
fd973e5055
RocksDB fixes
2020-07-29 21:26:36 +00:00
b5dc6b715f
Add file comment for GrvProxyServer.actor.cpp
2020-07-29 12:10:53 -07:00
cb0808a170
Resolve comments
2020-07-29 11:19:29 -07:00
86148f1277
Merge branch 'master' into grv-proxy
2020-07-29 11:14:51 -07:00
fad18ad85a
Remove more debug traces
2020-07-29 08:06:15 -07:00
30ea639666
Remove debug traces
2020-07-29 07:55:05 -07:00
f7b76a92af
pass joshua
2020-07-29 07:26:55 -07:00
a49cb41de7
Merge branch 'release-6.3'
...
# Conflicts:
# CMakeLists.txt
# cmake/ConfigureCompiler.cmake
# fdbserver/Knobs.cpp
# fdbserver/StorageCache.actor.cpp
# fdbserver/storageserver.actor.cpp
# flow/ThreadHelper.actor.h
# flow/serialize.h
# tests/CMakeLists.txt
2020-07-29 00:31:55 -07:00
a2089b354a
RemoveServersSafely:Safety check toKill1 to avoid cluster getting stuck
...
toKill1 and toKill2 are a random subset of all processes. If simply kill all processes in toKill1 or toKill2,
we may kill too many processes to make the cluster unavailable and stuck.
Similar as what toKill2 were modified if it can cause cluster unavailable,
we should do the same thing for toKill1
2020-07-28 21:07:31 -07:00
602b0f9dea
Merge branch 'master' into vectorref-emplace-back
2020-07-28 17:17:11 -07:00
35ebdec9d2
Merge pull request #3558 from sfc-gh-anoyes/anoyes/remove-redundant-cast
...
Remove redundant cast
2020-07-28 17:01:48 -07:00
2b8aa296bb
Merge pull request #3482 from sfc-gh-tclinkenbeard/support-downgrades
...
Support and test downgrades
2020-07-28 17:00:54 -07:00
8a5fde1b85
change SPECIAL_KEY_SPACE_CHANGE_CONFIGURATION to SPECIAL_KEY_SPACE_ENABLE_WRITES
2020-07-28 12:03:00 -07:00
d2cf700bd4
Fix compiler warnings
2020-07-28 18:30:26 +00:00
819e3ab3e8
Merge branch 'ratekeeper'
2020-07-28 16:48:50 +00:00
f065cbbd2d
Merge pull request #3534 from sfc-gh-xwang/master
...
Throttling related workload improvements
2020-07-28 00:40:06 -07:00
41a3e6c853
add write throttling
2020-07-28 03:49:47 +00:00
454ac21daf
Add tests for errors introduced by writes module, and shema check for error message
2020-07-27 17:40:55 -07:00
c6a2043d53
Merge pull request #3450 from sfc-gh-tclinkenbeard/6.3-use-vector-mutations
...
Replace VersionUpdateRef with VerUpdateRef
2020-07-27 15:21:47 -06:00
a641429d63
Add an enum for implementation type and use for registeration
2020-07-27 12:29:03 -07:00
c3348be3a0
changed knobs to improve master recovery speeds
2020-07-27 10:59:06 -07:00
A.J. Beamon