919a3d1740
Iterating over and coalescing the “internal” records (not user visible, tuple of (key, version, valueIndex, value)) now works forward and backward, and findEqual() now works using this to read split values fully. This is also most of the work needed for range reads.
2017-09-09 01:29:25 -07:00
4fa2415553
Merge branch 'release-5.0'
2017-09-08 17:28:12 -07:00
bb8a245bdb
circus: throughput test scales latency error by the target latency
2017-09-08 17:27:54 -07:00
ea26bc1c43
passed first tests which kill entire datacenters
...
added configuration options for the remote data center and satellite data centers
updated cluster controller recruitment logic
refactors how master writes core state
updated log recovery, and log system peeking
2017-09-07 15:32:08 -07:00
bd1c7e7295
Use addTeamsBestOf() instead of addAllTeams() when team size is greater than 3
2017-09-07 12:31:01 -07:00
c7df951f7c
Using BackupConfig from backup.actor.cpp to reduce intermediate
...
functions.
2017-09-07 08:36:36 -07:00
fe208d6adf
Merge branch 'master' of github.com:apple/foundationdb into backup
2017-09-06 10:01:55 -07:00
b5f79d27f7
Large values are now split and stored, but not yet read correctly.
2017-09-05 16:59:31 -07:00
83810edabc
Backup/Restore tag can be std::string instad of Key.
2017-09-05 11:38:40 -07:00
fe2d104000
Debug macro needs to be 1 instead of just being defined when it is in use.
2017-09-01 21:24:26 -07:00
dc1f7ca6b7
testers now use client locality load balancing
2017-09-01 12:53:01 -07:00
9745334704
Changed random parameter range so that at least two internal keys can always fit in a single page.
2017-09-01 00:17:07 -07:00
cc24072a5d
Add the multi version API to the list of APIs to choose in the APICorrectness tester. Support for the multi-version client already existed.
2017-08-31 16:23:55 -07:00
88e48e9a3e
Bug fix in detection of lack of mutations in a subtree.
2017-08-31 01:23:12 -07:00
1e11353603
Merge branch 'master' into feature-redwood
2017-08-31 00:24:58 -07:00
d61be4c760
Merge branch 'release-5.0'
2017-08-30 12:59:24 -07:00
963e1c3f31
fix: we need to reboot the process even if it will result in too many files, because the check will not succeed without it
2017-08-30 12:58:46 -07:00
7f46dfd829
Merge conflict fix.
2017-08-30 10:29:06 -07:00
8d97a15c3f
BUGGIFY recovery to lock only the minimum number of TLogs required to prevent a quorum.
...
This is to test the quorum logic introduced in the previous patch, and should
flush out any other bugs that rely on TLog locking during recovery.
2017-08-29 14:43:40 -07:00
f8486d1368
Only ensure a quorum of TLogs are unlocked to confirm the epoch hasn't ended.
...
Currently, GRV will wait to hear back from (almost) all TLogs to confirm that
they're unlocked and that the current epoch hasn't ended. This confirms that
there isn't a new set of proxies and using the commit version from the old set
of proxies would violate causal consistency.
However, during recovery, we ensure that no quorum of TLogs exists before
starting a new epoch and allowing new commits on the new TLogs. Thus, we only
need to wait until we have a quorum of TLogs that are unlocked.
This should be a significant improvement in latency particularly for the cases
when we start running >10 TLogs.
2017-08-29 14:43:40 -07:00
4c1d61cd08
Assorted minor changes.
...
In which we:
* Clarify some math in a comment
* Remove misleading debugging information
* Add a useful trace event
2017-08-29 14:43:40 -07:00
dbfa94f735
LF -> CRLF
...
It appears a previous patch left parts of this file ending with LF, and the
majority of the file ends in CRLF. I see no reason to keep this inconsistency,
but these line ending wars are going to drive me insane.
2017-08-29 14:43:40 -07:00
6020d70863
Added trace event to track reboots initiated by ConsistencyCheck workload in simulation
2017-08-29 11:41:27 -07:00
c95a1be5ec
Add trace event for rebooting process during simulation for consistency check
2017-08-29 11:00:44 -07:00
b716ad90f8
Missed in last commit.
2017-08-28 17:46:18 -07:00
5d49f2c710
Merge branch 'master' into feature-redwood
...
# Conflicts:
# fdbserver/fdbserver.vcxproj
2017-08-28 17:45:50 -07:00
5049421a0e
More comments, debug output improvements.
2017-08-28 17:26:53 -07:00
86774f6e42
Merge branch 'release-5.0'
2017-08-28 17:17:00 -07:00
03478561b9
fix: Set lock aware at the transaction level for latency probe to avoid having to fill the shard cache every time.
2017-08-28 17:16:46 -07:00
9a0a3b6329
Merge commit '66528becb82d826e81fa644bb378212584ab580e'
2017-08-28 16:47:59 -07:00
9fe927127f
choose leader on the perferred process class
2017-08-28 14:41:04 -07:00
44e0df78c5
Added support for tracking roles for simulation workers
...
Fixed the exclusion and inclusion address simulation API and integration within workloads
Added more information within trace events for simulation
2017-08-28 11:25:37 -07:00
581bd6c8ed
Added option to delay the displaying of the simulation workers
2017-08-28 10:53:56 -07:00
14ef5bed42
Major bug fix. In certain situations, some mutations were being applied to two adjacent pages, which had a lot of crazy side effects.
2017-08-28 06:28:49 -07:00
d9136743f7
Bug fix, reformatted and cleaned up a lot of debug output.
2017-08-28 03:53:29 -07:00
0fd57e9183
Many bug fixes.
2017-08-28 01:57:01 -07:00
300b5a17ed
Merge branch 'release-5.0'
2017-08-25 18:55:33 -07:00
1070a5ed66
Several bug fixes.
2017-08-25 15:48:32 -07:00
272b4b984c
fix: fixed a rare bug where we do not wait for a file in the process of being deleted to shutdown before rebooting a machine
2017-08-25 10:12:58 -07:00
888093463b
Checkpointing progress on large rewrite of how mutations are stored and applied. Not working yet.
2017-08-24 17:25:53 -07:00
26a5b5e422
rollback workload now clogs the communication between one of the proxies and the tlogs, since that is what will cause a rollback
2017-08-23 16:08:13 -07:00
4c706d33e9
Merge branch 'release-5.0'
2017-08-23 14:59:43 -07:00
be941b4bd1
sending void to committed could cause self to be deleted, so call cleanup before sending
2017-08-23 13:56:18 -07:00
7729f663e9
Ensured that the circus id is always lowercase
2017-08-23 13:45:00 -07:00
f9308b8fa6
Merge pull request #145 from cie/alexmiller/simrefactor
...
Refactor simulation to pull all configuration parameters into one struct.
2017-08-23 12:54:21 -07:00
4b40f817f1
fix: is recovery is cancelled before the copy is complete, remove the tlog
2017-08-23 12:26:03 -07:00
8056b78414
Merge branch 'release-5.0'
2017-08-22 13:51:19 -07:00
814e471689
Added support for displaying initial workers via printf within simulation using a workload
2017-08-22 13:38:24 -07:00
0b817f95f2
Bug fixes, but there’s still an issue with crossing page boundaries at the same key.
2017-08-22 11:30:44 -07:00
7b78035365
Have SimulationConfig wrap DatabaseConfiguration to reduce code duplication.
...
This effectively turns initializing SimulationConfig into the equivalent of
building a config string and calling buildConfiguration on it.
2017-08-22 10:13:57 -07:00
9b25c72971
Pull database config and cluster config into one struct.
...
This will allow us to specify custom situations to be chosen more frequently,
and in particular control machines and processes.
2017-08-21 22:35:44 -07:00
aff31b7f36
Checkpointing. Clears almost work, but two edge cases aren’t handled correctly yet involving range clears that must cross leaf page boundaries.
2017-08-21 22:29:57 -07:00
5ee07b1a9e
Merge branch 'release-5.0'
2017-08-14 16:56:58 -07:00
de1b590a8a
The TLog did not delete data from removed logs
...
The TLog continued to make data from removed logs persistent
2017-08-11 18:08:09 -07:00
50fb44be92
Merge branch 'release-5.0'
...
# Conflicts:
# versions.target
2017-08-09 23:36:12 -07:00
2335fc73f2
fix: peek cursors were being timed out every 10 minutes, instead of 10 minutes after the last use
...
fix: if an interface is changed while we are not waiting in getMore, we will not reset the sequence to 0.
2017-08-09 15:58:06 -07:00
47a37f3f1e
Merge pull request #135 from cie/switch-for-data-distribution
...
Add a switch to turn off data distribution in CLI
2017-08-07 12:54:08 -07:00
de03491475
Switched to new commit buffer type which can support more than set, but so far only set is implemented.
2017-08-04 00:01:25 -07:00
c22708b6d6
added tag localities
...
fix: remote logs need to stop the master when they are stopped
2017-08-03 16:16:36 -07:00
dbcd3526d7
Memory lifetime bug fix and previously missed changes for enforcing only reading committed versions.
2017-08-03 15:07:29 -07:00
ca7437ecf6
Merge branch 'release-5.0'
2017-08-02 22:07:01 -07:00
d0fbc41338
set LOCK_AWARE on several transactions used for getting cluster info for the consistency check
2017-07-28 18:50:32 -07:00
6a8a5c41b0
Add a switch to turn off data distribution in CLI
2017-07-28 18:14:55 -07:00
4243486f54
fix: TLogMetrics was being track latested with the wrong ID
2017-07-28 14:37:23 -07:00
37e5e2acbb
Fix parentheses issue in StorageMetrics.actor.h
2017-07-27 12:03:36 -07:00
cdc62e265c
Merge pull request #133 from cie/shard-system-keyspace
...
Shard system keyspace
2017-07-26 17:09:13 -07:00
41c90bcdea
Merge commit '89ac94853c70d08289e7fb58055bc5d0cd4e494d'
2017-07-26 15:35:36 -07:00
d8e308c18f
Enable use of incremental delete when deleting disk queue and sqlite KVS sqlite files.
2017-07-26 14:11:11 -07:00
53e1ae9f60
shard system keyspace
2017-07-26 13:47:31 -07:00
31314d06d4
Added test stats. Added enforcement of only reading committed versions, which is a currently an implementation limitation.
2017-07-25 16:10:19 -07:00
4aaee86c2a
Moved MetricLogger actor to fdbclient so applications other than fdbserver can use it.
2017-07-24 13:13:06 -07:00
2ae445782e
fix: cannot rely on the bestServer’s version because other logs may have higher versions
2017-07-21 19:21:49 -07:00
f6826f1e15
fix: log routers were popped at too high of a version
...
fix: make sure tlogs make everything durable
fix: make cluster controller’s temporary remote log recruitment not interfere with better master exists
2017-07-20 16:26:05 -07:00
7fec378830
do not continue copying data from prior generations after being locked
2017-07-19 15:11:18 -07:00
5852a6301b
fixed even more bugs
2017-07-15 15:15:03 -07:00
c860f09d8a
Merge branch 'release-5.0'
2017-07-14 16:01:15 -07:00
660729839c
moved Notified.h from flow -> fdbclient ; flow bindings package does better job when excluding testers
2017-07-14 15:49:30 -07:00
d61f74e52b
Debug output changes.
2017-07-14 13:39:58 -07:00
8b3569e27e
Bug fix.
2017-07-14 11:37:08 -07:00
fc0557252b
Bug fixes. Btree now uses page 0 as root and will write initial page only if necessary. Added debug printf macro.
2017-07-14 11:36:49 -07:00
d8b82ecbe4
Added use of IndirectShadowPager. Moved MemoryPager code into .cpp because of type conflicts and its implementation doesn’t need to be externally visible anyway. Added start of a performance test. Renamed tests. Correctness/set now has random reopen of disk based pager before verification. Added asserts for when keys or values that are too large to fit in a single page.
2017-07-13 22:11:48 -07:00
5ac4de8775
fix: the same tag could be in the server tags list twice
2017-07-13 16:31:55 -07:00
b56f5643d5
Bug fixes.
2017-07-13 14:51:39 -07:00
57ba9d36af
fixed a large number of bugs
2017-07-13 12:29:21 -07:00
bea061fd75
Bug fix, building tree root (1 or more top levels) must happen inside commitSubtree() otherwise the root page logical ID (1) can be written out of version order.
2017-07-13 11:32:14 -07:00
f75b6f333b
Merge branch 'release-5.0'
2017-07-13 11:21:18 -07:00
415458deef
made LogSet reference counted,
...
fixed a few bugs
2017-07-11 15:48:10 -07:00
81ae263ad9
implemented setPeekCursor
...
removed oldTLogServer
first compiling version
2017-07-10 17:41:32 -07:00
979ebcef6c
changed to using a vector of logSets instead of a duplicate set of logs for remote servers
...
finished porting changes to the tlog
everything but peeking is finished in the TagPartitionedLogSystem
2017-07-09 14:46:16 -07:00
c508e8bdf9
Adjacent internal btree pages can now contain the same keys at different versions.
2017-07-04 23:49:18 -07:00
1a71df1871
Lots of bug fixes and debug output added. Unitttest for set works…pretty often.
2017-07-04 23:41:48 -07:00
aa1c903b52
fix: do not log that data distribution is initialized until readyToStart is ready
2017-06-30 16:21:59 -07:00
0906250e78
merged everything from feature-remote-logs besides the tlog and tagpartitionedlogsystem
...
re-included tags in messages to the tlog
previously never committed the LogRouter
2017-06-29 15:50:19 -07:00
a2d229ff00
Merge branch 'release-5.0'
2017-06-29 08:16:22 -07:00
69c862ed6e
updated release notes for 5.0.1
2017-06-28 16:52:45 -07:00
663535e00a
add the same number for shared metrics and local metrics
2017-06-28 15:21:54 -07:00
fe37d0b056
added additional logging about bytesInput and bytesDurable
2017-06-28 13:29:40 -07:00
31d562ff7b
Merge branch 'release-5.0'
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbserver/DatabaseConfiguration.cpp
# versions.target
2017-06-27 11:16:08 -07:00
9fd5955e92
Merge branch 'master' into removing-old-dc-code
2017-06-26 16:27:10 -07:00
15cb498aa7
removed fast_recovery_double and fast_recovery_triple from the fdbcli
2017-06-23 16:18:23 -07:00
533dca95d8
fix: bytesDurable was not correctly increased when a log was removed
...
re-added many TLogMetrics
added a new role for the shared tlog
2017-06-22 17:21:42 -07:00
6d19580789
Merge branch 'release-5.0' of github.com:apple/foundationdb into release-5.0
...
# Conflicts:
# fdbrpc/simulator.h
2017-06-19 17:39:37 -07:00
96f40d8eb0
Added support for optionally re-including excluded servers
...
Removed unneeded code to protect coordinators
Ensured that simulation exclusion list is updated for excluded processes
2017-06-19 16:51:07 -07:00
9553458b78
Updated simulation to support managing exclusion and inclusion address
...
Added method for identifying acceptable availability process classes
Extended cluster availability function to ensure coordinators can be auto configured
Fixed availability function to allow protected processes to be considered as dead if not available
Added debug trace events for providing machine state when considering availability
Added trace event for protected coordinators
2017-06-19 16:48:15 -07:00
f405c8d88e
Merge branch 'release-4.6' into release-5.0
...
# Conflicts:
# fdbrpc/AsyncFileKAIO.actor.h
# fdbrpc/sim2.actor.cpp
# fdbserver/optimisttest.actor.cpp
# versions.target
2017-06-15 17:40:19 -07:00
5de6c703cf
Fixed line endings which were changed to UNIX style during a recent merge.
2017-06-15 16:57:35 -07:00
4bdcd8fc12
Merge branch 'release-4.6' into release-5.0
...
# Conflicts:
# bindings/bindingtester/run_binding_tester.sh
# fdbrpc/AsyncFileKAIO.actor.h
2017-06-14 16:43:53 -07:00
b65ad3563c
Merge branch 'master' into feature-redwood
...
# Conflicts:
# fdbserver/fdbserver.vcxproj
# fdbserver/fdbserver.vcxproj.filters
2017-06-09 14:56:41 -07:00
cefaa2391d
fix: tlog actor needs to be cancelled when worker shutdown
2017-06-02 17:56:47 -07:00
766dc23e26
fix: do not use TLS in protectedAddresses
2017-06-02 13:52:21 -07:00
b108c1ea0d
fix: do not delay before setting logData->version
2017-06-02 11:27:37 -07:00
2d0dbd57e8
randomized the delays in atomic switchover workload
2017-06-01 12:08:21 -07:00
bfcbb5623f
fixed build from merge error
2017-06-01 12:07:30 -07:00
276073d91b
Merge branch 'release-4.6' into release-5.0
2017-06-01 11:54:54 -07:00
fa4fdb1f1d
Merge branch 'fix-io-timeout-handling' into release-5.0
...
# Conflicts:
# fdbserver/optimisttest.actor.cpp
2017-05-31 17:03:15 -07:00
1626e16377
Merge branch 'release-4.6' into release-5.0
2017-05-31 16:23:37 -07:00
333f2e4865
Added connection fault disabler within setup of backup submission. It should be reviewed to determine the amount of time to wait before disabling
2017-05-31 14:21:50 -07:00
98604d33a0
Merge branch 'fix-io-timeout-handling'
...
# Conflicts:
# fdbrpc/AsyncFileKAIO.actor.h
# fdbrpc/sim2.actor.cpp
# fdbserver/KeyValueStoreSQLite.actor.cpp
# fdbserver/optimisttest.actor.cpp
# fdbserver/worker.actor.cpp
# fdbserver/workloads/MachineAttrition.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2017-05-26 18:43:08 -07:00
ba606c2135
Removed NOT_IN_CLEAN macro from file
2017-05-26 14:52:06 -07:00
b28ed397a2
Fixed printf field width specifier to reduce compilation warnings within OS X
2017-05-26 14:51:34 -07:00
0b9ed67e12
Fixed support for RemoveServers Workload
...
Added availability functions to simulation
2017-05-26 14:20:11 -07:00
16cc0821b1
Removed dead machine option from simulation
2017-05-25 16:29:02 -07:00
a674cb4ef4
Initial repository commit
2017-05-25 13:48:44 -07:00
Stephen Atherton