29b77863f0
Cache warmup and Consistency check workload changes.
2020-01-07 13:06:58 -08:00
10719200c3
A hack way to call API through getRange("\xff\xff/conflicting_keys\<start_key>", "\xff\xff/conflicting_keys\<end_key>").
2020-01-06 11:22:11 -08:00
414be7a0e4
Merge pull request #2479 from AlvinMooreSr/release_6.2_merge
...
Release 6.2 merge
2019-12-19 09:36:05 -08:00
21390c493a
Merge branch 'release-6.2' of github.com:apple/foundationdb into release_6.2_merge
...
Resolved merge by keeping new test file from master branch: SampleNoSimAttrition.txt adding new constraint from Release branch about existing test file: SimpleExternalTest.txt
# Conflicts:
# tests/CMakeLists.txt
2019-12-18 09:05:08 -08:00
4d027428df
FastRestore:Test:Increase txn load to 2500txn per second
...
More transactions per second can create more data in a backup version,
this can help test if fast restore handles large memory allocation correctly.
2019-12-16 12:13:29 -08:00
650be617f1
FastRestore:Add tests to CMakefile
2019-12-12 10:32:13 -08:00
c9994422a9
Ignore instead of omit
2019-12-12 09:12:34 -08:00
960e87ee8c
Disable SimpleExternalTest for UBSAN temporarily
2019-12-12 09:12:34 -08:00
64abaaf02d
FastRestore:Reenable tests for nightly test
2019-12-12 07:57:52 -08:00
a6cfad6c3f
FastRestore: Add ParallelRestoreCorrectnessAtomicOpTinyData test back
2019-12-11 16:39:17 -08:00
04230b59bf
Increase load a bit
2019-12-07 21:14:56 -08:00
d95180f853
Reduce parallel restore atomic op test workload
2019-12-06 21:23:08 -08:00
eacfdf6da3
Add a simple workload, ReportConflictingKeysWorkload, to test correctness of the API and performance overhead added to the resovler.
2019-12-06 16:21:03 -08:00
cf2563f1c7
Mix of various things, a lot of which will change.
2019-12-05 17:10:32 -08:00
edf52e8c97
First version for reporting conflicting keys
2019-12-02 13:45:39 -08:00
545a12533a
Added redwood sequential insert unit test.
2019-12-01 23:40:59 -08:00
923a22db66
Change CMakeLists for the parallel tests
2019-11-25 10:54:56 -08:00
78b8961891
Move parallel restore tests to tests folder
...
Valgrind found errors on these two parallel restore tests,
although correctness test confirms these two tests have no correctness error.
To prevent these two parallel restore tests from spamming valgrind test results,
we exclude these two tests from our nightly tests for now.
2019-11-25 10:53:10 -08:00
dbfa3dc217
Merge pull request #2200 from negoyal/storage-cache-subfeature1
...
Storage cache subfeature1
2019-11-20 13:59:06 -08:00
4e404e34e5
Added prefix size comparison test which generates records with a configurable prefix pattern and compares storage size between Redwood and the SQLite storage engine.
2019-11-17 17:53:42 -08:00
d46c7ded59
Merge remote-tracking branch 'origin/master' into storage-cache-subfeature1
2019-11-14 17:52:22 -08:00
5144e57e11
Reenable restart tests in from_5.*
2019-11-14 14:49:51 -08:00
d147071bfd
Merge branch 'release-6.2'
...
# Conflicts:
# fdbserver/MemoryPager.actor.cpp
2019-11-12 16:31:47 -08:00
a4a0bf18f9
Merging with Master.
2019-11-12 13:01:29 -08:00
ae7e42face
Merge pull request #2313 from xumengpanda/mengxu/fastrestore-applyToDB-bugfix-PR
...
Performant restore [8/XX]: Fix bugs in applyToDB logic and add more tests
2019-11-12 08:50:23 -08:00
9227de5c20
Redwood correctness unit test was using wallclock based time limit which breaks determinism.
2019-11-11 15:13:58 -08:00
120444ab4e
Merge pull request #2334 from mpilman/features/package-tests
...
Add cmake command to package tests
2019-11-08 19:52:57 -08:00
d0d036b3a7
Add cmake command to package tests
2019-11-08 21:39:21 +00:00
fe79742881
Merge pull request #2333 from apple/release-6.2
...
Merge release 6.2 to master
2019-11-08 11:01:07 -08:00
b50d26c4da
KVStoreTest now runs sqlite and redwood tests.
2019-11-05 13:46:04 -08:00
27c7ef09a3
FastRestore:Revise code in self review
...
When we read the txnId from decodeRestoreApplierKey func,
we should convert the integer to little endian.
2019-11-03 20:43:17 -08:00
7cf87e9ae3
FastRestore:Add ParallelRestoreCorrectnessCycle.txt test
2019-11-03 17:31:54 -08:00
63359bfc8b
FastRestore:handleInitVersionBatchRequest:Ensure exact once execution
...
Also increase the test workload for BackupAndParallelRestoreWithAtomicOp test
2019-11-03 17:26:13 -08:00
ef14f7a718
Merge pull request #2292 from etschannen/master
...
Merge 6.2 into master
2019-10-25 09:18:20 -07:00
a30e6ec147
Merge pull request #2277 from xumengpanda/mengxu/fastrestore-atomicOpTest-increaseLoadAndBugFix-PR
...
Performant restore [7/XX]: Add tests for transactionBatchSizeThreshold when apply mutations
2019-10-24 21:21:14 -07:00
3325980c03
Merge branch 'release-6.2'
...
# Conflicts:
# CMakeLists.txt
# documentation/sphinx/source/release-notes.rst
# fdbserver/DataDistribution.actor.cpp
# fdbserver/OldTLogServer_6_0.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/WorkerInterface.actor.h
# fdbserver/worker.actor.cpp
# versions.target
2019-10-24 17:38:15 -07:00
2f34ee684f
fixed indentation issues
2019-10-24 13:21:28 -07:00
b74e5b1543
added sample file for attrition test outside of simulation
2019-10-24 13:10:59 -07:00
7af3239ee7
FastRestore:AtomicOpTest:Debug:1 key per group for ops keyspace
2019-10-23 14:36:34 -07:00
ba7e499efe
FastRestore:AtomicOpTest:Limit 1 actor per client
2019-10-23 14:04:14 -07:00
e676348710
Merge pull request #1955 from fzhjon/mark-ss-failed
...
Add fdbcli and API command to mark storage servers as permanently failed
2019-10-22 23:36:30 -07:00
96d463bab6
FastRestore:Fix bug in applying mutations and increase atomicOp test worload
...
When Applier applies mutations to the destination cluster, it advances the
mutation cursor twice when it should only advance it once.
This makes restore miss some mutations when the applying txn includes
more than 1 mutations.
2019-10-22 23:24:23 -07:00
01b4fb5e5e
CMake:Add performant restore test to CMake
2019-10-21 14:52:03 -07:00
970327b554
FastRestore:Add ParallelRestoreCorrectnessAtomicOpTinyData.txt
2019-10-21 14:42:11 -07:00
450a09e117
Code Review Changes
2019-09-24 15:48:50 -07:00
471e283128
Merge branch 'master' of https://github.com/apple/foundationdb into mark-ss-failed
2019-09-18 11:49:07 -07:00
680f0e3bd8
updated CMakeLists
2019-09-13 09:30:40 -07:00
05eac989da
unpair the cycle test with configure test for now, because tests were timing out
2019-09-13 09:20:06 -07:00
5bf8d61d81
pair configureDatabase tests with cycle tests to ensure that changing the configuration does not corrupt data
2019-09-04 18:41:22 -07:00
807b02551e
updated help message and changed existing workload to use mark as failed feature
2019-08-27 14:39:43 -07:00
10be4b8127
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbserver/VersionedBTree.actor.cpp
2019-08-07 04:20:14 -07:00
da9c4e97d3
Added new pager interface, IPager2, whose write interface enables forcing the user to handle a page update causing a copy to a new Page ID. Implemented FIFOQueue<T> which uses pages of T stored in a Pager2 instance to implement a FIFO queue. Implemented COWPager, a copy-on-write Pager2 in which all page writes cause a change of Page ID. VersionedBTree, still only operating in single-version mode, now uses Pager2.
2019-08-07 02:36:33 -07:00
f453c12f0f
remove object serialization flag from TestRunner
2019-08-06 10:20:01 -07:00
438bc636d5
Rename max_machine_failures_without_losing_X to max_zone_failures_without_losing_X in status.
2019-07-30 14:02:31 -07:00
8149b5b352
Merge pull request #1413 from atn34/change-connection-file
...
Switch cluster file feature
2019-07-26 13:27:37 -07:00
9afd162e2f
remove snap v1 related code
2019-07-25 17:29:31 -07:00
a2f2ad96ff
code review comments and merge to master changes
2019-07-24 15:36:28 -07:00
c18558cf55
enable DD mode in restore based on test spec
2019-07-24 15:36:28 -07:00
24cbe313a1
Snap*Restart in ver 2 and buggify off in restore
2019-07-24 15:36:28 -07:00
9ad9bd4c1f
Merge branch 'master' of https://github.com/apple/foundationdb into change-connection-file
2019-07-24 15:22:26 -07:00
2f9f3c184f
Merge pull request #1870 from alexmiller-apple/txnStateStore-workload
...
Add ability to bulk load data into TxnStateStore
2019-07-22 13:20:39 -07:00
4ac1a0f557
Add ability to bulk load data into TxnStateStore
...
* Changes BulkLoad workload to support a specific volume of data to load
* Changes BulkLoad and Cycle to correctly handle \xff in keyPrefix
* Adds BulkLoad to TxnStateStoreCycleTest
2019-07-19 18:01:24 -07:00
3045826e3c
Merge pull request #1819 from mpilman/flatbuffers-fixes2
...
Flatbuffers fixes2
2019-07-19 16:33:50 -07:00
c3a8ae4752
Merge pull request #1791 from fzhjon/fetch-keys-requests-priority
...
Introduce priority to fetchKeys requests from data distribution
2019-07-19 14:54:51 -07:00
1ac2d01b03
Merge remote-tracking branch 'upstream/master' into flatbuffers-fixes2
2019-07-18 09:50:08 -07:00
d5caf0c1b4
Merge branch 'flatbuffers-fixes2' of github.com:mpilman/foundationdb into flatbuffers-fixes2
2019-07-16 14:47:40 -07:00
f3d3f80ed2
Merge branch 'master' into bugFix
2019-07-15 14:49:19 -07:00
4b0fdabae5
mark test file as IGNORE and comment out dead placeholder code
2019-07-15 09:45:16 -07:00
1a18c859c7
knobified the durability lag rate controls
2019-07-12 18:50:56 -07:00
969957e619
Merge branch 'master' into change-connection-file
2019-07-12 11:39:19 -07:00
1cf036fc9f
Remove SnapTestAttrition from ctest
...
SnapTestAttrition the test file was removed in a53bf928
2019-07-12 10:43:42 -07:00
bc29b1e78a
Use flatbuffers for testing
2019-07-11 23:03:31 -07:00
274cb40eb8
Run tests under valgrind when USE_VALGRIND
2019-07-11 23:03:31 -07:00
35be45d4ce
Remove code accidentally left for testing ctest fix
2019-07-11 10:01:26 -07:00
695ca5e8e9
moved location of DDMetricsExclude test txt file
2019-07-11 09:56:59 -07:00
f12a3909f3
renamed workloads and made code style adjustments
2019-07-11 09:56:58 -07:00
1e9d31597c
removed extra parameter from getRange, added knob to guard new changes, and adjusted style/formatting in several places
2019-07-11 09:56:58 -07:00
f707d186fe
added new priority for fetchkeys requests and adjusted ddmetrics workload to run parallel with mako
2019-07-11 09:56:58 -07:00
3b8998b836
added workload to measure data distribution speeds
2019-07-11 09:56:58 -07:00
62d7ee9424
Fix ctests so each restart test has a seed unique to the first test
...
and any other test
2019-07-11 09:56:51 -07:00
a53bf9289a
remove SnapTestAttrition because it is causing correctness errors
2019-07-09 17:13:24 -07:00
1582a2a24d
Merge branch 'master' of https://github.com/apple/foundationdb into change-connection-file
2019-07-09 13:41:54 -07:00
1b939d5208
Merge pull request #1749 from satherton/feature-redwood
...
Update redwood storage engine to latest correctness-passing version
2019-06-28 16:22:06 -07:00
83fae6cc15
Fix ExternalWorkload not being a part of the old build/test system.
2019-06-25 21:42:35 -07:00
8f06ccb065
Increased testDuration for DifferentClustersSameRV test
...
Unlocking the second database can take a while because the local
ratekkeeper throttles reads until the updateStorage actor catches up
after advanceVersion
2019-06-25 16:56:49 -07:00
0fe6edc254
Merge pull request #1678 from mpilman/features/external-workload
...
Features/external workload
2019-06-25 13:53:19 -07:00
f1f1081202
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbserver/VersionedBTree.actor.cpp
2019-06-24 20:17:49 -07:00
afb0dbcd1c
Merge branch 'master' of https://github.com/apple/foundationdb into change-connection-file
2019-06-20 19:11:29 -07:00
148f922813
Revert "Add CycleTest (currently times out)"
...
This reverts commit f367fcaf25
2019-06-20 18:34:41 -07:00
1611f0385a
Remove symbolication from ctest, as a large number of backtraces times out tests.
...
HugeArenaAllocation in particular is a bit spammy of backtraces,
intentionally, and is causing testing in the CI to time out.
2019-06-19 13:37:59 -07:00
2eff2b7e21
First simple test is working (but very buggy)
2019-06-19 13:03:41 -07:00
d4b7f9b606
Fixed some cmake, compile, and IDE warnings.
2019-06-17 18:55:49 -07:00
e2b6acffce
Merge branch 'master' into add-data-distribution-metrics
2019-06-12 16:12:16 -07:00
b473a8a830
changed on-the-wire format to use serialized flatbuffers, added cycletest to workload, and fixed small bug in trace
2019-06-11 15:45:06 -07:00
f367fcaf25
Add CycleTest (currently times out)
2019-06-11 14:10:34 -07:00
02e173b601
Add changeConnectionFile method to Transaction
...
Also add tests
2019-06-11 13:58:22 -07:00
8144882d7b
Merge branch 'apple-master' into features/local-rk
2019-06-10 19:40:25 -07:00
46b77819aa
Fixed LocalRatekeeper test
2019-06-10 18:25:58 -07:00
100789b354
More bug fixes in handling upperBound changes in modified pages and worst-case delta size calculation. Normalized some formatting in debug statements. Fixed compile error on linux. Updated test specs.
2019-06-05 20:58:47 -07:00
66811b7bd2
update to latest version
2019-06-03 16:49:19 -07:00
a6bee65f11
Merge branch 'master' into add-data-distribution-metrics
2019-05-30 11:11:49 -07:00
d3a179b6f9
Multiple bug fixes
...
- wait for snapTLogFailKeys in a loop, otherwise in some race
condition it can cause a false assert
- in single region, there does not seem to be a guarantee of
tagLocalityListKey for a given DC ID, avoiding that assert for now
- to find the workers that are coordinators, looking up by primary
address is not sufficient in some cases, hence looking by both
primary and secondary address
- test make files to reflect the location of the new test cases
2019-05-28 22:07:46 -07:00
16fc7b6aaa
move SnapTests into restarting/from_6.2.0
2019-05-28 22:07:46 -07:00
c53c4fa898
reduce the snap test durations
2019-05-28 22:07:46 -07:00
4083af0b01
Avoid using trackLatest for TLog pop test cases
2019-05-28 22:07:46 -07:00
61e93a9304
Address review comments and minor fixes
2019-05-28 22:07:46 -07:00
8838ba3d3b
Split SnapTestSimpleRestart into two test cases
2019-05-28 22:07:46 -07:00
8370871e4c
stale RESTORE option related code removed
2019-05-28 22:07:46 -07:00
898bed66c1
Allow only whitelisted binary path for exec op
2019-05-28 22:07:46 -07:00
f129d996fe
Remove dumpAfterTest=true in snap tests
2019-05-28 22:07:46 -07:00
281c785f94
'--restoring' cmd line arg removed for fdbserver
...
'--restoring' command line option was introduced to indicate
simulated fdbserver to restore from snapshot and restart the cluster.
As part of this change that option is removed and restore
information is stored in the restartInfo.ini.
2019-05-28 22:07:46 -07:00
3d5998e9dd
tlog: when pops are disabled, store them & replay
...
In Tlogs, disable pop is done whlie taking snapshots. Earlier, tlogs
were ignoring the pops if it got pop requests when pops were
disabled. In this change, instead of ignoring the pop - it remembers
the list of pops in-memory and plays them once the popping is
enabled.
2019-05-28 22:07:46 -07:00
4bc4c615da
exec op to all tlog, restore change in test &other
...
- exec operation to go to all the TLogs
- minor bug fix in tlog
- restore implementation for the simulator
- restore snap UID to be stored in restartInfo.ini
- test cases added
- indentation and trace file fixes
2019-05-28 22:07:46 -07:00
a7920ef311
Merge branch 'master' of https://github.com/apple/foundationdb into MakoWorkload
2019-05-28 18:21:02 -07:00
ca034846b4
update comments, parameter names and descriptions
2019-05-28 17:29:45 -07:00
0f46b6c926
Add sql_create_report call to generate sql file
...
After all testing iterations are complete, run sql_create_report to generate the sql
file and (for now) archive it with the job.
SNOW-78373
2019-05-28 10:22:24 -07:00
31939fd044
Merge pull request #1597 from atn34/cmake-flatbuffers-off
...
Hardcode flatbuffers to off for ctest
2019-05-23 11:13:54 -07:00
b451c2cd56
Merge pull request #1497 from alexmiller-apple/fastrecovery
...
Add an \xff keyrange that is backed by the txnStateStore.
2019-05-23 10:52:35 -07:00
74a705645b
changed kv format of metrics list and added simulation test
2019-05-22 12:44:32 -07:00
57968d9df7
Merge branch 'master' of https://github.com/apple/foundationdb into MakoWorkload
2019-05-21 16:24:11 -07:00
12a51b2d39
fix bugs, update naming and comments, refine functions
2019-05-20 18:26:30 -07:00
f673be456b
Added jenkins file for Snowflake
...
We (snowflake) want to run build and simulation tests against PRs.
For this we need a jenkins file in the repository. This should not
impact anything else
2019-05-17 11:17:40 -10:00
8082362830
Hardcode flatbuffers to off for ctest
2019-05-16 13:02:06 -07:00
106bb7677d
update
2019-05-15 12:58:12 -07:00
6afce01744
Implementation complete (not yet working)
2019-05-13 14:15:22 -07:00
5678a7417e
Mako Workload
2019-05-09 15:55:05 -07:00
0a7c74eac9
Add TxnStateStoreCycleTest to CMake
2019-04-25 17:09:35 -07:00
797d431934
Add an \xff keyrange that is backed by the txnStateStore.
2019-04-25 17:04:20 -07:00
0dd57dd454
Fix a few typos
2019-04-13 17:29:30 -07:00
d2e74cb2c0
Fix stupid rounding error
2019-04-08 11:05:29 -07:00
bdba8e22eb
Added test and bugfixes
2019-04-08 11:05:29 -07:00
2627bcd35e
Merge branch 'master' into feature-metadata-version
2019-03-10 21:13:28 -07:00
121ed4acf9
Trace the cmake SEED
...
This will be included in the repro instructions, so we need to trace it
to keep track of it.
2019-03-08 15:03:56 -08:00
21f55e1878
Merge pull request #1190 from bnamasivayam/restore-multiple-ranges
...
Add support for restoring multiple ranges.
2019-03-05 10:15:55 -08:00
075fdef31a
Merge branch 'master' into feature-metadata-version
...
# Conflicts:
# fdbclient/DatabaseContext.h
2019-03-03 22:58:45 -08:00
057ebe56e4
fix: unknownCommit handling relied on soleOwnership of the version stamp keys, so we need to use a second key to track the commit version for the metadataVersionKey
...
renamed a confusing option
2019-03-03 21:31:40 -08:00
3da85f3acd
implemented the \xff/metadataVersion key, which can be used by layers to help them cheaply cache metadata and know when their cache is invalid
2019-02-28 17:45:00 -08:00
d2bde4e55b
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-27 16:30:25 -08:00
8afb7fbb9d
Merge pull request #1160 from alexmiller-apple/tstlog-fork
...
Spill-By-Reference TLog Part 2: New and Old TLogServers co-exist harmoniously
2019-02-26 18:00:04 -08:00
7eba50b086
Add support for restoring multiple ranges.
2019-02-25 18:00:28 -08:00
abfe057805
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-25 13:47:16 -08:00
b8910ba7cd
Merge branch 'master' into feature-fix-force-recovery
...
# Conflicts:
# fdbclient/ManagementAPI.actor.h
# fdbserver/DataDistribution.actor.cpp
# fdbserver/storageserver.actor.cpp
# fdbserver/workloads/KillRegion.actor.cpp
2019-02-22 14:38:13 -08:00
6d23eb2d1a
Implement log_version.
...
This mega-commit introduces a new configuration setting, `log_version`,
that controls the TLog implementations and features that are available
within FDB, so that users can opt in to new features if they're willing
to sacrifice backwards compatibility.
2019-02-22 12:15:23 -08:00
2a20ea6ac5
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-22 09:41:47 -08:00
1292f5c59a
Add TEST_LOG_FORMAT to cmake
2019-02-21 14:36:01 -08:00
fa96b8dd33
Merge branch 'master' of https://github.com/apple/foundationdb into add-health-metrics
2019-02-20 16:56:16 -08:00
a20f5482bc
Created StorageStats struct to combine health metrics for storage servers
2019-02-20 11:57:41 -08:00
a6627fa5ca
be less clever for seed generation and seed>0
...
simulator doesn't accept a seed that is 0
2019-02-19 22:06:45 -08:00
e8af6512ec
Generate distinct pseudo-random number for each test
...
Fixes #1161
2019-02-19 22:06:45 -08:00
6b0ef8341c
Merge pull request #1157 from vishesh/task/tls-upgrade
...
Support live TLS migration #1050
2019-02-19 14:30:08 -08:00
8ed89fd711
fixed review comments
2019-02-19 11:26:53 -08:00
a2e4dc042f
Implement automatic package verification
2019-02-19 11:11:43 -08:00
065a45e05f
Merge branch 'master' into feature-fix-force-recovery
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbserver/ClusterController.actor.cpp
# fdbserver/workloads/KillRegion.actor.cpp
2019-02-18 17:09:06 -08:00
6cf4b416f7
move the KillRegionCycle workload into the fast directory
2019-02-18 15:43:01 -08:00
907446d0ce
Merge remote-tracking branch 'apple/master' into task/tls-upgrade
2019-02-14 11:37:38 -08:00
2d7f5f505c
Pass a bytestring to xml.sax.parseString
...
This fixes the ctest prb.
CI was broken because TestRunner errored at:
Traceback (most recent call last):
File "/foundationdb/tests/TestRunner/TestRunner.py", line 373, in <module>
res = run_simulation_test(basedir, args)
File "/foundationdb/tests/TestRunner/TestRunner.py", line 313, in run_simulation_test
options.log_format, return_codes)
File "/foundationdb/tests/TestRunner/TestRunner.py", line 261, in process_traces
parser.processTraces()
File "/foundationdb/tests/TestRunner/TestRunner.py", line 112, in processTraces
obj = self.processLine(line, linenr)
File "/foundationdb/tests/TestRunner/TestRunner.py", line 222, in processLine
xml.sax.parseString(line, handler, errorHandler=errorHandler)
File "/usr/lib/python3.4/xml/sax/__init__.py", line 45, in parseString
inpsrc.setByteStream(BytesIO(string))
TypeError: 'str' does not support the buffer interface
Which means you can't parse a string to parseString. This was fixed by
3.7, but our CI runs 3.4, so we need to .encode() `line` before passing
it, so that it ends up as a `bytes`.
2019-02-11 17:59:52 -08:00
a09afe5906
Added Throttling workload to test native health metrics API
2019-02-04 13:04:25 -08:00
1d7fec3074
Merge commit '048bfc5c368063d9e009513078dab88be0cbd5b0' into task/tls-upgrade-2
...
# Conflicts:
# .gitignore
2019-01-24 17:43:06 -08:00
f5e5f02a7f
Update tests/CMakeLists.txt
...
Co-Authored-By: mpilman <markus@pilman.ch>
2019-01-24 13:16:06 -08:00
83a9d5790f
Update tests/CMakeLists.txt
...
Co-Authored-By: mpilman <markus@pilman.ch>
2019-01-24 13:15:59 -08:00
5b8c98fd22
Update tests/CMakeLists.txt
...
Co-Authored-By: mpilman <markus@pilman.ch>
2019-01-24 13:15:42 -08:00
a50f2d953a
Update tests/CMakeLists.txt
...
set buggify to on by default
Co-Authored-By: mpilman <markus@pilman.ch>
2019-01-24 13:15:14 -08:00
58964af7e1
ctest improvements - #1058
...
- A set of CMake variables controls whether to keep
the simfdb directory and the traces and whether we
want to aggregate the traces into a single file
- Test labels now contain the directory they are in
so that one can now run `ctest -R fast/`
- A different binary can be used for restart tests. CMake
will automatically look for an installed fdb and use that
by default. If none is found, it will use the built one
but it will also print a warning
- CMake will throw an error if there are any text files in
the tests directory that are not associated with a test.
- Moved testing from fdbserver/CMakeLists.txt to
tests/CMakeLists.txt
- Moved fdb testing functions to its own cmake module
2019-01-22 14:34:51 -08:00
6d1f4bda75
Move traces.xml into the testdir so it doesn't grow infinitely.
2019-01-14 20:58:19 -08:00
c0bf8cd7e7
Remove the test result directory if the test passed.
2019-01-14 20:31:02 -08:00
248ae311bf
Type=TestFailure events shouldn't fail tests.
...
This sounds kind of broken, but is needed for now.
2019-01-14 19:16:03 -08:00
23697f06f7
Fix broken method call.
2019-01-14 19:14:25 -08:00
414fb0b6c8
made TestRunner work with XML traces
2019-01-14 19:14:25 -08:00
5ba5499b69
fail tests properly
2019-01-14 19:14:25 -08:00
b17d2234c9
added missing files
2019-01-14 19:14:25 -08:00
31c4ac07ac
WIP: FailureMonitoring use endpointAddressList (create individual endpoints for each address) WIP: g_currentDeliveryPeerAddress WIP: FlowTransport endpoint map WIP: Add peerReference to addressToEndpointMap
2019-01-09 07:46:01 -08:00
b402bd5bcc
Update RandomReadWrite.txt
2018-10-31 09:34:34 +08:00
0d1a8c3f2f
Update comment for alpha parameter
2018-10-30 15:50:45 +08:00
db71b60d72
Merge pull request #819 from satherton/feature-redwood
...
Redwood storage engine, initial/experimental version
2018-10-18 18:38:11 -07:00
ecddeab2ae
fixed review comments; demote killRegionCycle test for now
2018-10-08 10:39:39 -07:00
22f8a4efa9
Normalized all unit test names to begin with "/" if they should be included in random unit testing.
2018-10-05 22:09:58 -07:00
3ea9193fa7
Renamed redwood to redwood-experimental. UnitTest names can now be hidden using # as the first character so that random correctness tests will not run them. Excluded redwood tests from correctness testing. Reverted default storage engine to ssd.
2018-10-05 14:43:54 -07:00
7c1dc305cb
Merge commit 'a72c8f5cb2e79a673abc0ed3d27ef1c51028fb13' into feature-redwood
2018-10-05 10:15:10 -07:00
2fc86c5ff3
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbrpc/AsyncFileCached.actor.h
# fdbserver/IKeyValueStore.h
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/workloads/StatusWorkload.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-09-20 03:39:55 -07:00
200e65fe61
added a workload which tests killing an entire region, and recovering from the failure with data loss.
...
fix: we cannot pop the txs tag from remote logs until they have a full copy of the txnStateStore
fix: we have to modify all of history, we cannot stop after finding a local remote
2018-09-17 18:32:39 -07:00
26c9666f95
Changed default specific unit test.
2018-09-08 19:26:39 -07:00
d8ea3dbf9a
Added the ability to configure a cluster from a JSON file
2018-08-16 17:34:59 -07:00
9c918a28f6
fix: status was reporting no replicas remaining when the remote datacenter was initially configured with usable_regions=2
2018-08-09 13:16:09 -07:00
96389c74cd
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-07-10 16:42:34 -07:00
1bc95862b7
Merge branch 'release-6.0' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-07-10 04:16:02 -07:00
6d76ff67a3
added the connection string to status
2018-07-09 22:11:58 -07:00
b2fc2b4829
Gave status schema validation trace events more descriptive names. Fixed schema in tests.
2018-07-05 17:05:47 -07:00
9d85a05372
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-07-05 12:52:06 -07:00
507b3bacb0
fix: kill all tlogs in one region prevents the remote logs from recovering in that region, do not allow that to prevent us from configuring usable_regions=1.
...
added more recovery states.
2018-07-05 00:08:51 -07:00
dfc8c5154c
Update storage engine list in status schema.
2018-07-04 19:53:56 -07:00
866ccfe344
added the ability to allow the master to finish recovery before all storage servers in both regions have their mutations. This allows you to recover from scenarios where you lose all your tlogs in one dc.
2018-07-04 01:59:04 -04:00
b95a2bd6c1
Merge commit 'b17c8359ec22892ed4daeaa569f2f5e105477251' into feature-redwood
...
# Conflicts:
# flow/Trace.cpp
2018-06-30 23:18:29 -07:00
0123627d67
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-06-22 10:43:07 -07:00
8a8914f046
re-added the ability to configure the number of log routers. Many log routers are needed to get a sufficient number of sockets involved in copying data across the WAN
2018-06-22 00:04:00 -07:00
e8f66df001
Add metrics for watches and mutations on the storage server. The storage server tracks its lag with the logs, and status tries to report a more accurate measure of this lag.
2018-06-21 15:59:43 -07:00
ed8c2485c8
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2018-06-21 14:16:13 -07:00
62eeefcc8a
Update status schema with new field
2018-06-20 12:09:28 -07:00
e5c48d453a
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2018-06-18 22:45:27 -07:00
1ccfb3a0f4
fix: log_anti_quorum was always 0 in simulation
...
removed durableStorageQuorum, because it is no longer a useful configuration parameter
2018-06-18 10:24:57 -07:00
e8c462882b
re-added remote_logs as a parameter, because it could be useful to have a different number of logs between when recruited as primary and remote
2018-06-18 10:22:34 -07:00
0913368651
added usable_regions to specify if we will replicate into a remote region
...
remote replication defaults to the primary replication
removed remote_logs, because they should be specified as an override in the regions object
2018-06-17 19:31:15 -07:00
90c8288c68
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
2018-06-17 14:55:05 -07:00
246abd1207
added full_replication to status
2018-06-14 21:14:18 -07:00
0103b6f5ed
added datacenter_version_difference to status
2018-06-14 19:09:25 -07:00
99e21c869c
fixed a number of status calculations, and re-enabled the status workload
2018-06-14 17:58:57 -07:00
2878f30f29
Merge branch 'master' of github.com:apple/foundationdb into feature-redwood
...
# Conflicts:
# fdbserver/IKeyValueStore.h
# fdbserver/KeyValueStoreMemory.actor.cpp
# fdbserver/storageserver.actor.cpp
2018-06-13 15:56:06 -07:00
ca720e1540
Merge pull request #297 from apple/release-5.2
...
Merge 5.2 to Master
2018-05-08 12:04:20 -07:00
432a295bc2
Add read bytes and read keys info to status. Collect this information directly from StorageMetrics rather than through ratekeeper.
2018-05-04 12:01:40 -07:00
69e831d522
Merge remote-tracking branch 'upstream/release-5.2' into merge-release-5.2
2018-04-28 17:44:52 -07:00
6bddf8aefa
Upgrade DR from 5.1 to 5.2
2018-04-26 17:24:40 -07:00
af61d3596d
Merge branch 'public-master' into feature-redwood
...
# Conflicts:
# fdbserver/DatabaseConfiguration.cpp
# fdbserver/OldTLogServer.actor.cpp
# fdbserver/fdbserver.vcxproj
# fdbserver/fdbserver.vcxproj.filters
2018-04-24 17:22:21 -07:00
19762b847d
Merge branch 'release-5.2'
...
# Conflicts:
# fdbserver/DatabaseConfiguration.cpp
# fdbserver/SimulatedCluster.actor.cpp
2018-04-10 17:02:43 -07:00
2752a28611
Merge branch 'release-5.2' of github.com:apple/foundationdb into feature-redwood
2018-04-06 16:29:37 -07:00
d3fb17d30a
Merge pull request #74 from bnamasivayam/client-profiling-tests
...
Client profiling tests - Part 1
2018-03-23 16:52:49 -07:00
ede5cab192
Merge pull request #89 from yichic/share-log-mutations-5.2
...
Share log mutations 5.2
2018-03-19 12:01:26 -07:00
1f2602d2b3
Fix all review comments
2018-03-19 11:33:33 -07:00
d6559b144f
Share log mutations between backups and DRs which have the same backup range
2018-03-19 11:32:50 -07:00
f12c1d811c
Fix all review comments
2018-03-16 18:09:23 -07:00
26b93ff920
Share log mutations between backups and DRs which have the same backup range
2018-03-16 18:09:23 -07:00
856d2a0a9d
Add correctness tests for Client transaction profiling data format. It also includes format check across upgrades.
2018-03-14 12:39:50 -07:00
91bb8faa45
Merge commit 'f773b9460d31d31b7d421860fc647936f31aa1fa'
...
# Conflicts:
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-03-09 14:47:03 -08:00
b2ef6e1358
Add missing available_bytes fields to test status schemas
2018-03-09 14:17:20 -08:00
aea1f7ba21
Add tests for Client Transaction Profiling correctness
2018-03-05 18:55:23 -08:00
37a6a81634
Merge commit '7f6fc3e039c911cd84b8540f7f799fc38a1c1822' into feature-remote-logs
...
# Conflicts:
# fdbserver/workloads/RestartRecovery.actor.cpp
2018-02-23 12:33:28 -08:00
0bae9880f1
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-21 10:25:11 -08:00
cb25564d38
simulated cluster supports fearless configurations
...
removed unused simulation variables
run the simulation with only 1 coordinator most of the time, since we protect the coordinator from being killed, and protecting too many things is bad for simulation
2018-02-15 18:32:39 -08:00
0a35f167e4
Merge branch 'master' into feature-redwood
...
# Conflicts:
# fdbserver/DiskQueue.actor.cpp
# fdbserver/IDiskQueue.h
# fdbserver/Knobs.cpp
# fdbserver/Knobs.h
# fdbserver/fdbserver.vcxproj
# fdbserver/fdbserver.vcxproj.filters
# fdbserver/worker.actor.cpp
2018-02-12 01:30:02 -08:00
e104c21160
tweaks to fix some places where python format rewrite looked kind of bad
2018-02-07 16:59:59 -08:00
79d94214a4
Merge commit 'f4ffc9752b5ec66ac47f5f684a5d8be06a7eae6e' into feature-remote-logs
2018-01-25 10:12:06 -08:00
57986cfe00
format python files to be roughtly pep8 compliant
2018-01-24 19:06:58 -08:00
7db7a51440
Changes to backup folder structure in BackupContainerBlobStore at the top level inside the backup bucket. All data files now live under data/<backup_name> and there is an 'index' at backups/<backup_name> which indicates what backups exist. Backup names can now contain '/' characters.
2018-01-23 11:46:16 -08:00
698ef4117e
Merge branch 'master' into feature-remote-logs
2018-01-20 10:34:30 -08:00
b78e0a362a
fix: do not pause when running multiple backup tests simultaneously
2018-01-18 12:24:33 -08:00
264dc44dfa
fixed many more bugs associated with running without remote logs
2018-01-17 17:03:17 -08:00
3ec45d38a0
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-01-06 13:54:45 -08:00
b86f68ceb8
Added new test that combines atomic backup/restore. Added randomization to delays in AtomicRestore workload.
2018-01-05 14:43:21 -08:00
5ac4f73978
Merge branch 'release-5.1' into feature-remote-logs
...
# Conflicts:
# fdbclient/NativeAPI.actor.cpp
# fdbrpc/Locality.h
# fdbrpc/simulator.h
# fdbserver/ApplyMetadataMutation.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/LogSystemPeekCursor.actor.cpp
# fdbserver/MasterProxyServer.actor.cpp
# fdbserver/SimulatedCluster.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/TagPartitionedLogSystem.actor.cpp
# fdbserver/WorkerInterface.h
# fdbserver/masterserver.actor.cpp
# flow/Net2.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2018-01-05 11:33:42 -08:00
5015119115
Generalize the message that gets displayed in status if a cluster file's contents are incorrect.
2018-01-05 10:29:47 -08:00
371dee70e6
Improved backup folder structure to be shallower but spread files more uniformly and make each folder's entries lexically sort into version order regardless of numeric length. Improved backup container test to use a random version multiplier on the file versions created in order to test a wide range of versioned folder paths.
2018-01-02 23:22:35 -08:00
86958cb08d
Merge pull request #226 from cie/fix-taskBucket-unblockFuture
...
Modify TaskBucketCorrectness to support chain and multiple tasks
2017-12-20 18:00:54 -08:00
91e5abeaa6
Modify TaskBucketCorrectness to support chain and multiple tasks
2017-12-20 17:02:49 -08:00
47a9a7ab0e
Finished backup container discovery / listing via base URL.
2017-12-12 17:44:03 -08:00
7cf17df821
Merge branch 'master' into log-group-for-unsupported-clients
...
# Conflicts:
# flow/Net2.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2017-11-01 11:31:02 -07:00
fcbeea104f
Update documentation and tests with new connected_clients status schema
2017-11-01 11:25:50 -07:00
cfefab18fb
Merge branch 'master' into add-new-atomic-ops
2017-10-25 18:03:34 -07:00
2f6d55a52f
Add correctness tests for all atomic ops
2017-10-25 13:36:49 -07:00
48901a9223
added a list of tlog IDs that are missing to status
2017-10-24 16:28:50 -07:00
eeebf10030
Modified existing behavior of MIN and AND atomic ops. The new behavior results in a 'SET' if the atomic op is performed on a non -existing key.
...
Added new atomic ops ByteMin and ByteMax that does lexicographic comparison of byte strings.
2017-10-10 13:02:22 -07:00
15962cf079
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# fdbrpc/Locality.cpp
# fdbrpc/Locality.h
# fdbserver/ClusterController.actor.cpp
# fdbserver/ClusterRecruitmentInterface.h
# fdbserver/TLogServer.actor.cpp
# fdbserver/TagPartitionedLogSystem.actor.cpp
# fdbserver/WorkerInterface.h
# fdbserver/fdbserver.vcxproj.filters
# fdbserver/masterserver.actor.cpp
# fdbserver/worker.actor.cpp
# flow/error_definitions.h
2017-10-05 17:09:44 -07:00
11668bb359
Fixing code review comments.
2017-09-29 15:58:36 -07:00
9cb478be6d
Add a VersionStamp+BackupToDBAbort test that fails.
2017-09-29 15:58:36 -07:00
69523ce151
Hackish version of a test, but it does fail.
2017-09-29 15:58:36 -07:00
d4f75630de
Support log group field in status json
2017-09-28 16:31:29 -07:00
0f8ff26029
Merge pull request #158 from bmuppana/master
...
<rdar://problem/34557380> Need a way to map real time to version
2017-09-27 17:56:42 -07:00
0bf5bdb23a
<rdar://problem/34557380> Need a way to map real time to version
2017-09-25 12:51:37 -07:00
e8b895c878
added the ability to disable connection failures for a period of time after one happens
2017-09-18 12:46:29 -07:00
489332533c
all timeouts longer than two minutes have been can be lowered to 60.0 with buggification
...
added a workload that tries for a 50 second maximum latency in the presence of one failure with both buggification and connection failures
2017-09-18 11:04:51 -07:00
34f987f56d
added a test in simulation which ensures that a recovery after a single failure takes less than 15 seconds
2017-09-15 17:55:01 -07:00
76e7988663
Merge branch 'master' into feature-remote-logs
...
# Conflicts:
# fdbserver/ClusterController.actor.cpp
# fdbserver/DataDistribution.actor.cpp
# fdbserver/OldTLogServer.actor.cpp
# fdbserver/TLogServer.actor.cpp
# fdbserver/WorkerInterface.h
# flow/Net2.actor.cpp
2017-09-11 15:15:56 -07:00
560e172c46
tests: Update binding single key get range performance test to specify a limit of 2 and use exact mode.
2017-09-01 12:36:24 -07:00
1cd36b29f0
Moved Redwood unit test spec file so it can be used in testharness.
2017-08-31 10:37:45 -07:00
5d49f2c710
Merge branch 'master' into feature-redwood
...
# Conflicts:
# fdbserver/fdbserver.vcxproj
2017-08-28 17:45:50 -07:00
1ef77ef99a
java tester support on release-5.0
2017-07-17 16:42:50 -07:00
5e4a94120c
update status json schema
2017-07-15 16:18:59 -07:00
d8b82ecbe4
Added use of IndirectShadowPager. Moved MemoryPager code into .cpp because of type conflicts and its implementation doesn’t need to be externally visible anyway. Added start of a performance test. Renamed tests. Correctness/set now has random reopen of disk based pager before verification. Added asserts for when keys or values that are too large to fit in a single page.
2017-07-13 22:11:48 -07:00
800cbc0ccf
added perf tests in python and java for single key get ranges
2017-07-11 19:32:24 -07:00
169e41804e
Update ryw benchmark key count for get range/clear range tests
2017-06-29 15:35:07 -07:00
693ef857ad
added Java perf tests ; made python ryw tests more to what appears to be the spec
2017-06-27 16:26:29 -07:00
b65ad3563c
Merge branch 'master' into feature-redwood
...
# Conflicts:
# fdbserver/fdbserver.vcxproj
# fdbserver/fdbserver.vcxproj.filters
2017-06-09 14:56:41 -07:00
2d0dbd57e8
randomized the delays in atomic switchover workload
2017-06-01 12:08:21 -07:00
1626e16377
Merge branch 'release-4.6' into release-5.0
2017-05-31 16:23:37 -07:00
98604d33a0
Merge branch 'fix-io-timeout-handling'
...
# Conflicts:
# fdbrpc/AsyncFileKAIO.actor.h
# fdbrpc/sim2.actor.cpp
# fdbserver/KeyValueStoreSQLite.actor.cpp
# fdbserver/optimisttest.actor.cpp
# fdbserver/worker.actor.cpp
# fdbserver/workloads/MachineAttrition.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2017-05-26 18:43:08 -07:00
a674cb4ef4
Initial repository commit
2017-05-25 13:48:44 -07:00
negoyal