a93c0053ca
add two asserts
2020-04-03 19:09:12 -07:00
b4bde7f8f1
Added TPC-C workload
2020-04-03 11:33:24 -07:00
78f0cddb14
Merge pull request #2684 from mpilman/features/boost70
...
Upgrade to boost 1.72
2020-04-03 09:30:59 -07:00
1e41fd3681
Attempt to fix Windows compilation error
2020-04-02 09:23:27 -07:00
393cbabd52
Rename class
2020-03-31 19:06:11 -07:00
321b1b6ee7
Add RunRestoreWorkerWorkload into Parallel Restore WriteDuringReadAtomicRestore tests
2020-03-31 18:48:29 -07:00
a5af31de23
Addressed simple review comments
2020-03-31 18:34:13 -07:00
b1c3893d40
Fix some corner case bugs exposed by simulation.
...
In one case, when a SS joins the cluster and DD doesn't find any healthy server to form a team with the newly added server, then the SS does not get added to any team even when the other servers get healthy.
Another is an extreme case where a data center is down, and a SS in the active DC joins and then dies immediately but not before DD adds it to a destination team for a relocating shard which will result in DD waiting indefinitely for the dead data center to come back up for the cluster to be fully recovered.
2020-03-31 18:33:12 -07:00
a63f8b839c
Merge remote-tracking branch 'upstream/master' into private-keyspace-framework
2020-03-31 12:05:42 -07:00
53a04740be
refine part of the code
2020-03-31 01:36:07 -07:00
2a0bb34005
delete code and format
2020-03-31 01:21:21 -07:00
79db3b2a57
add randomness
2020-03-31 01:14:35 -07:00
3640dfed99
add trace lines fore test failures
2020-03-31 00:09:38 -07:00
37ca0bf7e2
add code to handle the three flags in RangeResultRef
2020-03-30 23:27:09 -07:00
60f6edc3b5
Merge pull request #2860 from zjuLcg/report-conflicting-key-roll-forward
...
Report conflicting key roll forward
2020-03-30 17:33:56 -07:00
75fc9af5c8
Apply clang format
2020-03-27 16:55:52 -07:00
32b0ba1822
Merge branch 'master' into mengxu/parallel-range-log-file-loading-PR
2020-03-27 12:13:47 -07:00
113d0fb48b
Remove incorrect assertion
2020-03-27 12:13:30 -07:00
9a9af7d8a8
Add more trace event details on partitioned log
2020-03-26 13:57:31 -07:00
6be913a430
Add partitioned logs option to AtomicRestore workload
2020-03-26 13:04:00 -07:00
aca458cd96
Set 50% chance to restore old backup files for fast restore
2020-03-26 13:04:00 -07:00
772ab70aee
Add an option for fast restore to restore old backups
...
If "usePartitionedLogs" is set to false, then the workload uses old backups for
restore.
2020-03-26 13:04:00 -07:00
1052b23ee1
Merge pull request #2370 from atn34/test-watch-outliving-transaction
...
Test watch outliving transaction
2020-03-26 12:40:38 -07:00
cdb6bbfc85
Test watch outliving transaction
2020-03-26 10:09:03 -07:00
feedab02a0
Merge pull request #2855 from xumengpanda/mengxu/fr-api-atomicrestore-PR
...
Add ApiCorrectnessAtomicRestore workload for the new performant restore
2020-03-25 18:05:26 -07:00
1ba11dc74b
Apply clang format
2020-03-25 11:20:17 -07:00
120272f025
Change unlockDB from RestoreMaster to Agent
2020-03-25 11:04:49 -07:00
e0424a52f8
Merge branch 'master' into transaction-tagging
2020-03-25 08:23:11 -07:00
62627dd2ee
Fix a randomness bug and naming issue in TraceEvent
2020-03-25 00:55:40 -07:00
ca8966a28b
Move lockDB into submitRestore request from restore worker
...
AtomicRestore needs to lock DB before we start the restore worker.
So we cannot lock DB in restore worker with a different randomUID.
2020-03-24 23:39:35 -07:00
6a8d6ddb8e
Introduce ParallelRestoreApiCorrectnessAtomicRestore.txt test
...
This covers ApiCorrectnessTest as workload for parallel restore.
2020-03-24 22:30:51 -07:00
289487559d
Revert "Revert "Merge pull request #2257 from zjuLcg/report-conflicting-key""
...
This reverts commit 804fe1b22e
2020-03-24 18:11:15 -07:00
b173929316
Add atomicParallelRestore to AtomicRestore workload
2020-03-24 15:58:49 -07:00
81f7181c9e
Refactor submitParallelRestore function into FileBackupAgent
2020-03-24 14:44:55 -07:00
5584884c12
Refactor parallelRestoreFinish function into FileBackupAgent
2020-03-24 14:15:15 -07:00
8fbeefc2a2
refine code
2020-03-24 01:15:14 -07:00
c02fdd47dc
add randomness for bytes limit
2020-03-24 01:14:50 -07:00
4825e1bae7
clang-format
2020-03-24 01:00:04 -07:00
ea86089ee0
refine code
2020-03-24 00:59:37 -07:00
049888e70e
refine code
2020-03-24 00:29:00 -07:00
6df92a8b8d
A work version of strict workload test
2020-03-23 23:55:56 -07:00
1155304cd5
Remove a spurious assertion
...
It's possible that there is a gap between backup's contiguousLogEnd and snapshot
version.
2020-03-23 21:39:40 -07:00
dd90845277
Fix assert failure
...
Should be backup's contiguousLogEnd > maxRestorableVersion.
2020-03-23 14:49:05 -07:00
90b40e1d75
Merge branch 'mengxu/new-backup-format-PR-delta' of github.com:xumengpanda/foundationdb into backup-worker-bak
...
Resolve Conflicts:
fdbclient/BackupAgent.actor.h
fdbserver/BackupWorker.actor.cpp
fdbserver/RestoreMaster.actor.cpp
fdbserver/masterserver.actor.cpp
2020-03-23 13:35:33 -07:00
3f31ebf659
New backup:Revise event name and explain code
2020-03-23 10:55:44 -07:00
38def426f4
Add a flag to submitBackup for partitioned log
...
This is to distinguish with old workloads so that they can work in simulation.
2020-03-20 20:15:08 -07:00
1f95cba53e
Add describePartitionedBackup() for parallel restore
...
For partitioned logs, computing continuous log end version from min logs begin
version. Old backup test keeps using describeBackup() to be correctness clean.
Rename partitioned log file so that the last number is block size.
2020-03-20 20:13:38 -07:00
26b7e02d4c
Some initial work to support tagging transactions and passing them around.
2020-03-20 11:23:11 -07:00
804fe1b22e
Revert "Merge pull request #2257 from zjuLcg/report-conflicting-key"
...
This reverts commit 648dc4a933487d131b38
2020-03-19 21:34:28 -07:00
524b275a94
Add a flag to submitBackup for partitioned log
...
This is to distinguish with old workloads so that they can work in simulation.
2020-03-18 16:41:35 -07:00
f697ccd1b9
Add describePartitionedBackup() for parallel restore
...
For partitioned logs, computing continuous log end version from min logs begin
version. Old backup test keeps using describeBackup() to be correctness clean.
Rename partitioned log file so that the last number is block size.
2020-03-18 16:41:35 -07:00
a476127f5f
Merge pull request #2802 from xumengpanda/mengxu/debug-master-PR
...
Fix correctness failure on master branch
2020-03-18 16:07:36 -07:00
648dc4a933
Merge pull request #2257 from zjuLcg/report-conflicting-key
...
Report conflicting keys
2020-03-18 13:39:42 -07:00
16cd68d3d9
Delete unnecessary parameters
2020-03-17 13:41:14 -07:00
7f559bc712
Cleanup code and apply clang-format
...
Self code review
2020-03-16 15:08:32 -07:00
1513df22f3
AutoQuorumChange:Exclude unreliable node from coordinator in simulation
2020-03-16 14:39:25 -07:00
15c48b9e19
Add event for getDesired coordinators
2020-03-16 09:40:35 -07:00
9dc441c65a
clang-format
2020-03-13 15:43:01 -07:00
39a37531db
Fix issues according to Andrew's comments
2020-03-13 15:42:15 -07:00
c246f79d72
Update comments
2020-03-13 13:18:18 -07:00
a3b0dce3cd
Rename vars, update comments
2020-03-13 13:12:22 -07:00
8ee4fea3d3
clang-format
2020-03-13 12:54:12 -07:00
aef9b515de
Change the workload to a more controlled test like ConflictRange test
2020-03-13 12:42:28 -07:00
7118759dfa
Delete code to test resolvers' performance, simplify the workload to only test correctness
2020-03-13 01:48:55 -07:00
0ef09539a9
addressMap[normalizedAddress]->address may not equal to normalizedAddress
2020-03-12 13:01:25 -07:00
6f90228a0b
change to krmSetRangeCoalescing
2020-03-12 11:31:36 -07:00
1759d5c8c4
Apply clang-format
2020-03-12 10:18:53 -07:00
a9136f3f72
Add waitForUnreliableExtraStoreReboot to wait for extra store to reboot
2020-03-12 10:18:31 -07:00
c2f0c41c52
use krmSetRange
2020-03-11 23:12:38 -07:00
1a5b41157e
add test for native transaction object
2020-03-11 11:17:34 -07:00
d87ed92f78
checkForExtraDataStores:Fix compilation error
2020-03-11 09:59:11 -07:00
e0d2eca7a8
checkForExtraDataStores:Add coordinators into stateful process list
2020-03-10 23:38:30 -07:00
bd345f85db
ConsistencyCheck:Fix failue due to address inconsistency between process and worker
...
With TLS, a worker (or process) can have a TLS address and non-TLS address.
When a process is created in simulation, the primary address is TLS by default.
The non-TLS one is the TLS address port plus one.
In a connection between two workers, if their primary addresses do not enable
or disable TLS together, one worker will swap its primary address and secondary address
so that the TLS config of the two endpoints can match.
The swap can make the primary address no longer the TLS one that was created
when the process is created. And the swap only happens for worker instead of
process struct in simulation.
This swap can cause worker->address != process->address.
In checkForExtraDataStores actor, we use worker->address to check if a process
is killable and use the process->address to kill the process. The inconsistency
can cause simulation to kill a protected process that is not killable and leads
to simulation failure.
2020-03-10 21:07:16 -07:00
157d026ba2
debugging
2020-03-10 18:42:49 -07:00
a7f1efc543
add test workload
2020-03-09 10:46:47 -07:00
3a98c691b6
Update comments
2020-03-04 11:42:19 -08:00
7a76e9556d
Merge remote-tracking branch 'upstream/master' into report-conflicting-key
2020-03-04 11:24:39 -08:00
e79faae175
Add some more tail latencies to the latency trace events in the ReadWrite workload.
2020-02-24 11:06:13 -08:00
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
e422874758
fix: reboot does not work un unreliable processes
2020-02-21 14:29:42 -08:00
f04e311a1e
Merge commit 'b46d6e25e24993ab5a5f04091fd3235050b7cd09' into feature-boost-ssl
...
# Conflicts:
# fdbserver/SimulatedCluster.actor.cpp
# flow/Net2.actor.cpp
2020-02-20 17:36:38 -08:00
4f1301b2dd
Merge pull request #2583 from etschannen/feature-keep-status-connected
...
Clients should not disconnect from the CC after fetching status
2020-02-20 13:12:30 -08:00
fcbdcda490
Merge pull request #2650 from ajbeamon/fix-reverse-range-read-byte-limit-bug
...
Fix reverse range read performance bug
2020-02-20 12:47:17 -08:00
52ba093f23
BackupAndParallelRestoreTest:Apply Steve change in BackupCorrectness on 604ad062d5
...
The backup container behavior was changed at commit 604ad062d5
2020-02-20 08:07:03 -08:00
0507e9703c
FastRestore:Disable abort fast restore test
2020-02-20 06:54:14 -08:00
03f699f2f9
Merge branch 'master' into mengxu/fast-restore-applier-multi-applying-PR
2020-02-19 15:22:33 -08:00
a12a161fb3
Merge branch 'master' into mengxu/fast-restore-pipeline-PR
2020-02-18 14:49:52 -08:00
c603b20e7e
FastRestore:Resolve review comments
2020-02-18 14:08:27 -08:00
acf34319c1
FastRestore:Applier:Precompute mutations and apply in parallel
...
Precompute mutations received by an applier;
Only apply the final result to the destination DB;
Execute multiple txns in parallel to apply final results to the destination DB.
2020-02-12 22:47:48 -08:00
38a5511b96
additional simulation fixes
2020-02-11 15:52:06 -08:00
17660fb18d
Fix simulation test
2020-02-11 13:49:19 -08:00
1e1e75123f
Add simulation testing
2020-02-11 11:10:22 -08:00
dcbce3593e
fixed TLS in simulation
2020-02-10 14:00:21 -08:00
ff44bd2b33
Merge pull request #2639 from atn34/atn34/include-port-in-address-default
...
Enable include_port_in_address by default for api version 700
2020-02-10 09:50:59 -08:00
e71fe44ee3
Merge branch 'master' into features/icc
2020-02-08 21:33:02 -08:00
df2b0452b4
Step 3 of fixing storage server range reads: change return type of readRange from VectorRef<KeyValueRef> to RangeResultRef.
2020-02-06 13:19:24 -08:00
844c8511c4
Merge pull request #2588 from jzhou77/backup-worker
...
Integrate new backup worker with existing backup command
2020-02-05 14:14:43 -08:00
d09e07f1f5
Merge remote-tracking branch 'upstream/master' into features/icc
2020-02-04 10:26:18 -08:00
0db03f1d3c
Use backup_logging_enabled flag
...
The default is to enable new backup workers. Users can disable this flag to
turn off the backup worker feature.
2020-02-03 20:03:22 -08:00
3b57bf1781
Merge branch 'master' into mengxu/fast-restore-agent-PR
2020-02-03 17:23:54 -08:00
9400766be9
Remove include for Make build
...
IWYU would not be pleased :(
2020-02-03 15:39:06 -08:00
2ce887012c
Respect api version for include_port_in_address
2020-02-03 15:25:30 -08:00
f16c4cfa66
Fix second semantic error
2020-02-03 11:20:53 -08:00
f25e913db2
Fix semantic change
2020-02-03 11:15:28 -08:00
09f3690f09
Fix OPEN_FOR_IDE build
2020-02-03 10:42:05 -08:00
297f22726c
Add backup_type database configuration option
...
Update simulation tests to randomly set backup types to be one of: old backup
(default), new backup (tagged), or both (default+tagged).
2020-01-31 19:29:09 -08:00
ee6490c9d1
Merge pull request #2314 from mengranwo/memory-engine
...
New Radix-Tree based Memory Storage Engine
2020-01-30 16:20:13 -08:00
182dac7cd5
Convert the slow task profiler into a run loop profiler that also logs when the run loop is 100% busy for a knob-configurable duration.
2020-01-28 12:09:37 -08:00
754e2cb023
FastRestore:Fix:number of workers should be calculated
...
based on number of appliers and loaders.
2020-01-27 21:14:00 -08:00
cab9d51e06
Merge branch 'master' into mengxu/fast-restore-pipeline-PR
2020-01-27 18:16:26 -08:00
d03e49b4a1
Fixed the location of crc32c.h from fdbrpc to flow
2020-01-26 07:01:25 -08:00
6945a6ea01
Merge pull request #2345 from zjuLcg/add-consistency-verification-in-mako-workload
...
Add consistency verification in mako workload
2020-01-24 17:07:49 -08:00
499ffdb572
add memory-radixtree-beta into database configuration test
2020-01-24 16:40:11 -08:00
16f9ec45bd
Merge branch 'master' into mengxu/fast-restore-pipeline-PR
2020-01-23 20:15:21 -08:00
e011f39829
FastRestore:Add sanity check and trace events
2020-01-23 16:03:41 -08:00
8a065b9da4
Merge pull request #2557 from alexmiller-apple/reduce-versionstamp-conflictranges
...
Narrow the unreadable range of keys after a versionstamped key operation
2020-01-23 11:14:47 -08:00
acebfdc67b
Restore storage queue limit to 0 in consistency check
...
The storage queue is no longer going to be a problem failing tests. Now the
backup worker life cycle is tied with backup. So consistency check only happens
after the backup workload is done. Thus, we no longer need to save backup
progress when consistency check is running.
2020-01-22 19:43:40 -08:00
840e74d696
Allow storage server queue in consistency check
...
The backup worker needs to update its progress even during consistency check by
commit transactions to the database. Thus we can't really achieve zero storage
server queue. So add a limit of 10,000 to pass the consistency check.
2020-01-22 19:38:45 -08:00
73ad702d14
Clients which fetch status should not disconnect from the coordinators and cluster controller between each retrieval
2020-01-22 15:41:22 -08:00
bffb4e380d
AtomicOp:Debug
2020-01-22 15:04:25 -08:00
5386b4ecdf
Adding prefix to returned keys, keeping consistency with query range
2020-01-21 15:18:16 -08:00
b0a1af1288
Added the actual read hot detection algorithm and logging machanism.
...
- When a shard has a read bandwidth larger than a threshold value(configurable via knob), and it's read-bandwidth/byte-size ratio is also larger than a threshold(configurable via knob), the corresponding shard tracker will run the algorithm
- The algorithm will divide the shard into 10MB(configurable via knob) chunks and try to find the chunk(s) that has large aforementioned ratio
- Then those ranges will be logged into TraceEvents. This will later do more like actually cache them.
2020-01-21 11:19:52 -08:00
eb64eede8d
Make a smaller range inaccessable after writing a versionstamped key
...
A transaction's read version is the lower bound of what a transaction's
commit version could be. Thus, we can narrow the conflict range of a
versionstamped key, and thus reduce the amount of the keyspace that is
rendered inaccessable, by filling in the read version on the
versionstamped key and using that as the lower bound of the conflict
range.
This allows reads to still be done to versionstampled keys lower than
the read version of the current transaction.
2020-01-16 21:41:59 -08:00
227edd4248
change memory storage engine name from memory-radixtree to memory-radixtree-beta
2020-01-15 13:49:45 -08:00
6836e370a7
revert changes inside KVStoreTest, ready for code review
2020-01-15 13:49:45 -08:00
f597aa7e18
WIP : deployable/stable version since Nov 3. Start rebase to master branch
2020-01-15 13:49:45 -08:00
72d39a31f1
Change to send back index of read_conflict_range not keys, which speeds up but fail to give narrowed keyrange
2020-01-15 10:21:51 -08:00
1cf3b5b807
Change to general prefix '\xff\xff/transaction/conflicting_keys/'
2020-01-13 20:54:16 -08:00
9a736dde8d
delete wait in workload when checking, add assert to make sure the getRange for conflicting keys happens locally
2020-01-10 18:01:17 -08:00
57fb28af2c
Explicitly set read-version and clear the whole keyspace to make sure the getRange happens locally
2020-01-07 13:54:33 -08:00
10719200c3
A hack way to call API through getRange("\xff\xff/conflicting_keys\<start_key>", "\xff\xff/conflicting_keys\<end_key>").
2020-01-06 11:22:11 -08:00
ded2a301e0
Merge pull request #2443 from xumengpanda/mengxu/fast-restore-fix-valgrind-PR
...
Performant restore [12/XX]: Code clean up
2019-12-13 14:35:20 -08:00
3bf971ba8b
Merge branch 'release-6.2' of github.com:apple/foundationdb into release_6.2_merge
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbserver/storageserver.actor.cpp
2019-12-12 07:13:12 -08:00
9670d64fbd
FastRestore:Remove commented code
2019-12-11 16:48:40 -08:00
1371db4cdc
FastRestore:Self code review and cleanup
...
1. Review memory use cases and improve:
Ensure state varialble is initialized and
change unnecessary state variable to variable.
2. Remove debug code that is no longer useful;
3. Mute verbose debug.
2019-12-11 16:37:33 -08:00
a9ccefe58e
Add a checksum mechanism into MakoWorkload.
2019-12-11 14:23:02 -08:00
56f1ff7ff6
Test client-side buggify in simulation
2019-12-09 12:55:23 -08:00
4a66366a05
Use MutationsVec instead of VectorRef
2019-12-06 22:00:40 -08:00
9dc79488a8
narrow the returned conflicting ranges
2019-12-06 16:21:19 -08:00
b376b947d5
Encode conflicting keys using base64.
2019-12-06 16:21:19 -08:00
eacfdf6da3
Add a simple workload, ReportConflictingKeysWorkload, to test correctness of the API and performance overhead added to the resovler.
2019-12-06 16:21:03 -08:00
39a4f2372f
Change FASTRESTORE_SAMPLING_PERCENT to 0 to 100
2019-12-04 21:26:27 -08:00
c6b36dbffb
FastRestore:Sampling:Resolve review comments
2019-12-04 17:35:11 -08:00
e6678573db
Fix load of bool which is not 0 or 1
2019-12-04 09:42:35 -08:00
a15320cca7
Merge branch 'master' into mengxu/fast-restore-sampling-PR
2019-12-03 21:42:01 -08:00
3310f67e9e
Merge branch 'mengxu/fast-restore-fix-valgrind-PR' into mengxu/fast-restore-sampling-PR
2019-12-03 16:24:40 -08:00
2aeb9e0cbf
Fix UBSAN report
2019-12-03 16:20:39 -08:00
4d28793c76
Merge pull request #2365 from xumengpanda/mengxu/no-wait-actor-fix-PR
...
Fix compilation warning that actor that does not contain wait statement
2019-12-03 08:30:04 -08:00
07331ab5fd
Merge pull request #2362 from etschannen/master
...
Merge 6.2 into master
2019-12-02 15:04:27 -08:00
edf52e8c97
First version for reporting conflicting keys
2019-12-02 13:45:39 -08:00
chaoguang