2a848edb41
Merge remote-tracking branch 'upstream/master'
2020-07-09 18:10:38 +00:00
c40c4693b4
better correctness criteria
2020-07-09 18:10:12 +00:00
1dead7c199
Merge pull request #3440 from sfc-gh-xwang/master
...
add HealthMetricsWorkload and WriteTagThrottlingWorkload
2020-07-09 10:04:46 -07:00
a5af214861
Use KeyRangeMap for splitMutation
...
This simplifies code by reuse KeyRangeMap class.
2020-07-09 09:59:37 -07:00
d61206e126
Merge pull request #3453 from thisismiller/restrict-testspec
...
Make testspec more restrictive and correctly "scoped"
2020-07-08 11:43:37 -07:00
e78cc9ee4b
Use KeyRangeMap for splitMutation
...
This simplifies code by reuse KeyRangeMap class.
2020-07-08 09:53:34 -07:00
627075e0ef
1. add test parameters; 2. change ContinuousSample initialization 3. add&change metrics
2020-07-06 18:18:15 +00:00
17570b5b10
Make the testspec more restrictive in terms of what can be set where.
...
Testspec is currently very permissive in very misleading ways. In particular,
the tester parser itself will swallow K=V settings and apply them at the test
level, which breaks how a person would expect the scoping to work. Other
settings apply to the entire simulation run globally, but appear to be workload
specific. Even further, others affect simulation cluster creation or test
harness behavior, but can again be set anywhere in a testspec.
This changes testspec parsing to error if a setting that applies globally is
anywhere but the top of the file, or if a setting that applies test-wide is
applied to a workload instead of a test.
2020-07-06 02:03:30 -07:00
121465b051
change metrics
...
fix latency sum bug
2020-07-03 21:49:44 +00:00
98b3ef69aa
1.change metrics; 2.change the way to generate key (avoid to much confict)
2020-07-03 00:20:39 +00:00
c1face27f1
fix dangling pointer bugs
2020-07-02 18:35:39 +00:00
c76e0cfe04
Add a ReadAfterWrite workload, to measure TLog->SS propagation delay.
2020-07-01 02:17:43 -07:00
6ebd3c8083
1. add range clear operation to workload
...
2. add tag logic to WriteTagThrottling
2020-07-01 06:35:14 +00:00
ad6904413c
Merge branch 'master' of https://github.com/apple/foundationdb
2020-07-01 04:58:02 +00:00
dc568cad21
add HealthMetricsWorkload and WriteTagThrottlingWorkload to prepare for new throttling feature
2020-07-01 04:56:21 +00:00
22f7f804b8
Merge branch 'release-6.3' into mengxu/merge-6.3-PR
2020-06-28 11:19:39 -07:00
90861ead6a
Reenable all tests in BackupAndParallelRestoreWithAtomicOp
2020-06-23 08:29:31 -07:00
b122575dda
BackupAndParallelRestoreWithAtomicOp:Turn on debug options
2020-06-22 16:53:43 -07:00
6e18b6454a
Remove obsolete file
2020-06-16 10:13:57 -07:00
96206a8032
Merge pull request #3368 from apple/release-6.3
...
Merge Release 6.3 to master
2020-06-15 20:15:22 -07:00
39c50798b8
Merge pull request #3341 from sfc-gh-clin/fix-rpc-calls-in-special-key-space
...
Fix fetchShardMetricsList_impl and add read cache in special key space
2020-06-15 16:12:38 -07:00
acbfe2e4c9
Revert "Revert "Initial RocksDB""
2020-06-15 12:45:36 -04:00
980bee1d13
Fix fetchShardMetricsList_impl and add read cache in special key space
2020-06-12 10:12:44 -07:00
9cd1614c82
Revert "Initial RocksDB"
2020-06-11 15:29:46 -07:00
a4dbb5dd01
Merge branch 'trace-batch-thread-hostile' into rocksdb-6.3
2020-06-11 15:53:57 +00:00
693d9e8b89
Merge branch 'master' into fdb_cache_wo_allocator
2020-06-09 15:09:58 -07:00
cf1ae52751
Merge pull request #3281 from apple/release-6.3
...
Merge Release 6.3 into master
2020-06-03 10:55:37 -07:00
cf13e00a8f
Merge remote-tracking branch 'origin/release-6.3' into fdb_cache_wo_allocator
2020-06-01 17:38:31 -07:00
d128252e90
Merge release-6.3 into master
2020-05-22 09:25:32 -07:00
5d361fe532
Copy/paste rebase onto 6.3
2020-05-22 15:02:51 +00:00
56e14a25fe
Normal correctness package can be built when USE_VALGRIND is on. Correctness and valgrind now stage their files in separate directories. Add CMakeCache.txt to testing packages.
2020-05-21 13:35:54 -07:00
6ce574f5ad
Merge remote-tracking branch 'upstream/release-6.3' into add-data-distribution-metrics
2020-05-17 23:36:52 -07:00
5632af3b4d
Removed inaccurate warning
2020-05-16 20:20:11 -07:00
25390f0968
Remove SimpleExternalTest.txt from fdb_test_files even if UBSAN is enabled
2020-05-13 22:46:07 -07:00
02307ba7b6
Merge branch 'master' into transaction-tagging
...
# Conflicts:
# fdbclient/DatabaseContext.h
2020-05-09 07:50:29 -07:00
ef724bf939
Merge remote-tracking branch 'upstream/master' into add-data-distribution-metrics
2020-05-08 18:39:28 -07:00
749fcd13b0
Merge branch 'master' into fdb_cache_wo_allocator
2020-05-08 16:23:29 -07:00
13ff1b3b20
Revert "Exclude Snap related tests from correctness test temporily"
...
This reverts commit f95bbc0ffa
2020-05-07 14:39:13 -07:00
b1055a8501
Merge branch 'master' into transaction-tagging
2020-05-05 16:03:39 -07:00
f329164fb4
Merge pull request #2532 from dongxinEric/feature/hot-read-key-detection-part-2
...
Feature/hot read key detection part 2
2020-05-05 14:33:34 -07:00
378951360d
Merge pull request #3086 from alexmiller-apple/ctest-use-crash
...
Make ctest use --crash when running simulation tests.
2020-05-05 12:49:21 -07:00
f040694bf0
Make ctest use --crash when running simulation tests.
...
So that tests with ASSERTion failures don't cause CI to run slow and
produce massive logs. There's also rarely a reason to continue through
assertion failures.
2020-05-05 03:11:09 -07:00
dd033736ed
Merge branch 'master' into fdb_cache_subfeature2
2020-05-04 17:29:43 -07:00
80a235aa80
Add some correctness tests
2020-05-04 10:15:18 -07:00
1927f15898
Now the ReadHotDetection tests can be run reliably. Added it to nightly bundle.
2020-04-28 13:26:02 -07:00
50ad4a33fe
Added possibility to include and exclude tests
...
Description
Testing
2020-04-28 16:19:04 +00:00
7dd7406c59
Merge branch 'master' into feature/hot-read-key-detection-part-2
2020-04-16 14:54:05 -07:00
b85dc16c6d
Merge branch 'master' into fdb_cache_subfeature2
2020-04-14 17:07:41 -07:00
65620b00f3
Merge remote-tracking branch 'upstream/master' into private-keyspace-framework
2020-04-13 14:21:07 -07:00
f95bbc0ffa
Exclude Snap related tests from correctness test temporily
...
Those tests fail with high chance in nightly test.
2020-04-10 10:01:07 -07:00
2a856b23d3
Added test files and updated vsproj files
2020-04-06 09:53:18 -07:00
a0c8946f31
Merge branch 'master' into fdb_cache_subfeature2
2020-04-02 12:27:04 -07:00
38d7f16de0
FastRestore:Add new tests to ctests
2020-04-01 21:34:26 -07:00
16898408ae
FastRestore:Rename tests for new backup by adding NewBackup keyword
2020-04-01 21:31:54 -07:00
1eea388278
Add unit test for splitMutation test file
...
Comment out debug trace as well.
2020-04-01 21:27:18 -07:00
321b1b6ee7
Add RunRestoreWorkerWorkload into Parallel Restore WriteDuringReadAtomicRestore tests
2020-03-31 18:48:29 -07:00
a1d2779f04
Add ParallelRestoreWriteDuringReadAtomicRestore.txt
...
Test for fast restore with new backup.
2020-03-31 18:25:17 -07:00
0fa3bfa6f3
Increase timeout threshold for ParallelRestoreOldBackupWriteDuringReadAtomicRestore.txt
2020-03-31 18:21:36 -07:00
8471cb5351
Disable ParallelRestoreApiCorrectnessAtomicRestore.txt temporarily.
2020-03-31 15:52:07 -07:00
ccbbdc4ba4
Unit test:Verify splitMutation by comparing with intersectingRanges result
2020-03-31 12:13:02 -07:00
a63f8b839c
Merge remote-tracking branch 'upstream/master' into private-keyspace-framework
2020-03-31 12:05:42 -07:00
37ca0bf7e2
add code to handle the three flags in RangeResultRef
2020-03-30 23:27:09 -07:00
faab8a0d19
Add ParallelRestoreOldBackupApiCorrectnessAtomicRestore.txt
...
Add tests/ParallelRestoreApiCorrectnessAtomicRestore.txt back which
was mistakenly deleted in the previous commit.
2020-03-30 21:59:47 -07:00
73345a7ebd
Add new tests for parallel restore
...
Add two tests for parallel restore with old backup data
ParallelRestoreApiCorrectnessAtomicRestore.txt
ParallelRestoreOldBackupWriteDuringReadAtomicRestore.txt
2020-03-30 21:53:26 -07:00
60f6edc3b5
Merge pull request #2860 from zjuLcg/report-conflicting-key-roll-forward
...
Report conflicting key roll forward
2020-03-30 17:33:56 -07:00
cf7404a771
Merge pull request #2871 from xumengpanda/mengxu/parallel-range-log-file-loading-PR
...
Performant restore [24/xx]: Process range and log files in parallel for old backup format
2020-03-28 17:33:50 -07:00
92998d3df9
Add restore with old backup tests into CMakeLists
2020-03-28 16:08:22 -07:00
6e60c62d7b
Add ParallelRestoreCorrectnessAtomicOp.txt
2020-03-28 16:00:51 -07:00
137ea71203
Merge pull request #2865 from mpilman/features/build-test-harness
...
Build all joshua packages with cmake
2020-03-27 13:11:39 -07:00
f7233bade7
Rename ParallelRestoreCorrectnessAtomicOpTinyData.txt by removing TinyData
2020-03-27 13:08:59 -07:00
5c6752fb24
Add tests for new restore with old backup
2020-03-27 13:07:38 -07:00
97f8e46388
Sanity check subversion for log mutations
2020-03-27 13:07:08 -07:00
96a7b4b459
Disable ParallelRestoreApiCorrectnessAtomicRestore test
2020-03-26 14:59:50 -07:00
acaf91ac47
Merge branch 'master' into fdb_cache_subfeature2
2020-03-26 13:33:08 -07:00
cffcac92ad
Disable partitioned log for atomic restore workload
...
This will be reenabled after fixing test failures related to it.
2020-03-26 13:07:40 -07:00
7e2938b290
Merge branch 'master' of github.com:apple/foundationdb into features/build-test-harness
2020-03-26 12:47:55 -07:00
ce6c4cb8d8
Build and package TestHarness
2020-03-25 17:36:47 -07:00
f240d393f2
Add ParallelRestoreApiCorrectnessAtomicRestore.txt into ctests
2020-03-25 11:23:19 -07:00
6a8d6ddb8e
Introduce ParallelRestoreApiCorrectnessAtomicRestore.txt test
...
This covers ApiCorrectnessTest as workload for parallel restore.
2020-03-24 22:30:51 -07:00
289487559d
Revert "Revert "Merge pull request #2257 from zjuLcg/report-conflicting-key""
...
This reverts commit 804fe1b22e
2020-03-24 18:11:15 -07:00
d185359422
ApiCorrectnessAtomicRestore test: Increase timeout value
2020-03-24 17:36:17 -07:00
6df92a8b8d
A work version of strict workload test
2020-03-23 23:55:56 -07:00
247ab84323
Merge branch 'master' of https://github.com/apple/foundationdb into add-data-distribution-metrics
2020-03-23 17:01:17 -07:00
d7cf562ec4
Enable rollback workload for multi cycle restore test
2020-03-23 16:33:11 -07:00
85a9f6ab96
Merge branch 'master' into mengxu/fr-enable-rollback-and-abort-test-PR
2020-03-23 16:32:00 -07:00
14c641ce2b
Enable rollback workload
2020-03-23 16:29:01 -07:00
4756447b74
Enable abortAndRestartAfter option for restore cycle test
2020-03-23 15:15:14 -07:00
f936adedb0
Change tab to space for test workload file
2020-03-23 15:10:35 -07:00
d0bce1a105
Add ParallelRestoreCorrectnessCycle test into CMakeList
2020-03-23 13:56:33 -07:00
9c2c3b26d3
Merge branch 'master' into mengxu/fr-multi-cycle-test-PR
2020-03-23 13:54:20 -07:00
00ada7e086
Enable attrition workload for restore multi cycle tests
2020-03-23 13:50:05 -07:00
ace19eefe4
Introduce multi cycle tests for fast restore
2020-03-23 12:59:43 -07:00
5b2b2a8767
Enable assassination for restore cycle test
2020-03-23 12:28:04 -07:00
48db54424f
Add assassination workload to restore test workload
...
Add assert to ensure restore worker is reliable and not killed.
2020-03-23 11:11:13 -07:00
4f61973ede
Remove the ReportConflictingKeys.txt test from CMake
2020-03-19 10:49:54 -07:00
308d82245c
Remove ReportConflictingKeys.txt workload
...
The PR 2257 still has many correctness failures.
Remove the workload to avoid noise while we are working on the fixes.
2020-03-19 10:34:14 -07:00
aef9b515de
Change the workload to a more controlled test like ConflictRange test
2020-03-13 12:42:28 -07:00
7118759dfa
Delete code to test resolvers' performance, simplify the workload to only test correctness
2020-03-13 01:48:55 -07:00
357ca88b4c
test spec file
2020-03-10 18:51:45 -07:00
7a76e9556d
Merge remote-tracking branch 'upstream/master' into report-conflicting-key
2020-03-04 11:24:39 -08:00
087c6fa33d
Merge branch 'master' into feature-redwood
2020-02-26 12:25:04 -08:00
cd949eca71
Merge branch 'master' into fdb_cache_subfeature2
2020-02-26 11:22:08 -08:00
735e7c71f2
Add the cachetest back in.
2020-02-24 17:26:20 -08:00
be70b18c22
Remove Cachetest for merge.
2020-02-24 16:46:36 -08:00
96258b9809
Merge branch 'release-6.2'
...
# Conflicts:
# documentation/sphinx/source/release-notes.rst
# fdbcli/fdbcli.actor.cpp
# fdbclient/ManagementAPI.actor.cpp
# fdbrpc/FlowTransport.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/DataDistribution.actor.h
# fdbserver/DataDistributionQueue.actor.cpp
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/QuietDatabase.actor.cpp
# fdbserver/SkipList.cpp
# fdbserver/StorageMetrics.actor.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/fdbserver.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KVStoreTest.actor.cpp
# flow/CMakeLists.txt
# flow/Knobs.cpp
# flow/Knobs.h
# flow/genericactors.actor.cpp
# flow/serialize.h
2020-02-21 19:09:16 -08:00
f1ec780b31
Merge branch 'release-6.2' of github.com:apple/foundationdb into feature-redwood
2020-02-21 17:43:11 -08:00
e9fa7803a8
FastRestore:Test:Let simulation tests turn buggify on and off
2020-02-20 07:06:04 -08:00
0507e9703c
FastRestore:Disable abort fast restore test
2020-02-20 06:54:14 -08:00
e794003b16
FastRestore:AtomicOpTest:Enable buggify
2020-02-19 20:25:07 -08:00
28768715f2
Custom bookkeeping allocator for PTree. Simple Cache workload.
2020-02-14 11:42:47 -08:00
0c7c815396
Merge branch 'release-6.2' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# tests/CMakeLists.txt
2020-02-12 16:12:57 -08:00
8fb9722277
Fix newline at end of file
2020-02-11 13:52:56 -08:00
86089fdc1b
Merge branch 'release-6.2' into atn34/configure-locked
2020-02-11 13:51:41 -08:00
17660fb18d
Fix simulation test
2020-02-11 13:49:19 -08:00
1e1e75123f
Add simulation testing
2020-02-11 11:10:22 -08:00
5a9d420cb7
Merge remote-tracking branch 'upstream/release-6.2' into release-merges/20200210
2020-02-10 10:02:05 -08:00
5c294e2646
Several build improvements - enables parallel builds on Windows (again)
2020-02-09 08:34:14 -08:00
85cc35e81e
Merge branch 'master' into HEAD
2020-02-05 14:59:55 -08:00
058de86b88
Merge branch 'master' of https://github.com/apple/foundationdb into add-data-distribution-metrics
2020-02-05 10:07:36 -08:00
09f3690f09
Fix OPEN_FOR_IDE build
2020-02-03 10:42:05 -08:00
b0a1af1288
Added the actual read hot detection algorithm and logging machanism.
...
- When a shard has a read bandwidth larger than a threshold value(configurable via knob), and it's read-bandwidth/byte-size ratio is also larger than a threshold(configurable via knob), the corresponding shard tracker will run the algorithm
- The algorithm will divide the shard into 10MB(configurable via knob) chunks and try to find the chunk(s) that has large aforementioned ratio
- Then those ranges will be logged into TraceEvents. This will later do more like actually cache them.
2020-01-21 11:19:52 -08:00
7708034cc9
Added the function used to determin sub-ranges within a read hot shard that has a readSize-to-byteSize ratio higher than the knob value. Alos added unit tests for that function.
2020-01-21 11:15:52 -08:00
4de8f2fdfa
Move the test to basic correctness test
2020-01-16 14:42:14 -08:00
1d6ac716a1
Merge remote-tracking branch 'origin' into add-data-distribution-metrics
2020-01-15 13:20:04 -08:00
4fa619e782
Merge branch 'fdb_cache_subfeature2' into features/cache-initialization
2020-01-07 19:59:05 -08:00
d3d6016c90
Merge remote-tracking branch 'negoyal/fdb_cache_subfeature2' into features/cache-initialization
2020-01-07 19:53:09 -08:00
e8e5f2d118
Bug fixes in the cache role.
2020-01-07 16:51:40 -08:00
29b77863f0
Cache warmup and Consistency check workload changes.
2020-01-07 13:06:58 -08:00
10719200c3
A hack way to call API through getRange("\xff\xff/conflicting_keys\<start_key>", "\xff\xff/conflicting_keys\<end_key>").
2020-01-06 11:22:11 -08:00
414be7a0e4
Merge pull request #2479 from AlvinMooreSr/release_6.2_merge
...
Release 6.2 merge
2019-12-19 09:36:05 -08:00
21390c493a
Merge branch 'release-6.2' of github.com:apple/foundationdb into release_6.2_merge
...
Resolved merge by keeping new test file from master branch: SampleNoSimAttrition.txt adding new constraint from Release branch about existing test file: SimpleExternalTest.txt
# Conflicts:
# tests/CMakeLists.txt
2019-12-18 09:05:08 -08:00
4d027428df
FastRestore:Test:Increase txn load to 2500txn per second
...
More transactions per second can create more data in a backup version,
this can help test if fast restore handles large memory allocation correctly.
2019-12-16 12:13:29 -08:00
650be617f1
FastRestore:Add tests to CMakefile
2019-12-12 10:32:13 -08:00
c9994422a9
Ignore instead of omit
2019-12-12 09:12:34 -08:00
960e87ee8c
Disable SimpleExternalTest for UBSAN temporarily
2019-12-12 09:12:34 -08:00
64abaaf02d
FastRestore:Reenable tests for nightly test
2019-12-12 07:57:52 -08:00
a6cfad6c3f
FastRestore: Add ParallelRestoreCorrectnessAtomicOpTinyData test back
2019-12-11 16:39:17 -08:00
04230b59bf
Increase load a bit
2019-12-07 21:14:56 -08:00
d95180f853
Reduce parallel restore atomic op test workload
2019-12-06 21:23:08 -08:00
eacfdf6da3
Add a simple workload, ReportConflictingKeysWorkload, to test correctness of the API and performance overhead added to the resovler.
2019-12-06 16:21:03 -08:00
cf2563f1c7
Mix of various things, a lot of which will change.
2019-12-05 17:10:32 -08:00
edf52e8c97
First version for reporting conflicting keys
2019-12-02 13:45:39 -08:00
545a12533a
Added redwood sequential insert unit test.
2019-12-01 23:40:59 -08:00
923a22db66
Change CMakeLists for the parallel tests
2019-11-25 10:54:56 -08:00
78b8961891
Move parallel restore tests to tests folder
...
Valgrind found errors on these two parallel restore tests,
although correctness test confirms these two tests have no correctness error.
To prevent these two parallel restore tests from spamming valgrind test results,
we exclude these two tests from our nightly tests for now.
2019-11-25 10:53:10 -08:00
dbfa3dc217
Merge pull request #2200 from negoyal/storage-cache-subfeature1
...
Storage cache subfeature1
2019-11-20 13:59:06 -08:00
4e404e34e5
Added prefix size comparison test which generates records with a configurable prefix pattern and compares storage size between Redwood and the SQLite storage engine.
2019-11-17 17:53:42 -08:00
d46c7ded59
Merge remote-tracking branch 'origin/master' into storage-cache-subfeature1
2019-11-14 17:52:22 -08:00
5144e57e11
Reenable restart tests in from_5.*
2019-11-14 14:49:51 -08:00
d147071bfd
Merge branch 'release-6.2'
...
# Conflicts:
# fdbserver/MemoryPager.actor.cpp
2019-11-12 16:31:47 -08:00
Xiaoxi Wang