6758c649fc
Catch and update processClass change from DBSource
2017-09-25 10:36:03 -07:00
cce4eeb52d
fix: the master was sending the cluster controller uninitialized configurations
2017-09-22 16:59:24 -07:00
180438d41e
fix: use the number of present logServers rather than the total size of the vector
2017-09-22 16:19:16 -07:00
738ae21c3a
fix: an optimization in buggified locking can cause recovery to break because it would not restart if a locked process was killed when the remaining logs cannot obtain a quorum
2017-09-22 15:07:57 -07:00
585c9bf68f
Quick fix to reduce CPU usage of ensureEpochLive.
...
It is suspected that policy recomputations are driving proxy CPU usage up, and
thus latency and throughput down. To quickly confirm this theory, we're
forcing ensureEpochLive to wait until it has RF responses, which means we'll
probably only validate the policy once per call.
2017-09-21 18:22:24 -07:00
fbd67ea547
fix: excluded servers are worst fit for master rather than never assign (so that we can recover if every process has been excluded)
...
fix: better master exists did not use exclusions because the configuration was reset
2017-09-20 11:48:26 -07:00
cb43563b2d
fix: toMap properly lists the redundancy mode of the cluster
2017-09-19 16:35:42 -07:00
f75dfc3153
do not register with the master until recovery of the queue is complete, to avoid having the master wait a long time for a peek response
2017-09-18 17:39:12 -07:00
567d663afd
Fix SimulationConfig never generating a custom config.
...
A 0 was changed to a 1 when rewriting code, and `case 0:` was never being hit. :(
Thankfully, it looks like nothing was broken by this in the meantime.
2017-09-18 17:29:36 -07:00
e8b895c878
added the ability to disable connection failures for a period of time after one happens
2017-09-18 12:46:29 -07:00
489332533c
all timeouts longer than two minutes have been can be lowered to 60.0 with buggification
...
added a workload that tries for a 50 second maximum latency in the presence of one failure with both buggification and connection failures
2017-09-18 11:04:51 -07:00
34f987f56d
added a test in simulation which ensures that a recovery after a single failure takes less than 15 seconds
2017-09-15 17:55:01 -07:00
d9b64899c5
fix: we need to wait for log server failures if we have not locked all of the logs
2017-09-15 13:11:21 -07:00
36c98f18e9
do not register a worker with the cluster controller until it has finished recovering all files from disk
2017-09-15 10:57:58 -07:00
f3b7aa615d
fix: seed storage servers are recruited based on the storage policy
2017-09-14 17:06:00 -07:00
9404d226d0
Merge branch 'release-5.0'
2017-09-13 16:49:00 -07:00
cb92194772
Fixed problem with master being recruited on excluded servers
2017-09-13 16:48:27 -07:00
5e14f19875
Merge pull request #147 from cie/alexmiller/grvtlogs
...
Only verify a quorum of TLogs are unlocked for a GRV request
2017-09-13 16:07:25 -07:00
d6b3be98fe
Fix whitespace.
2017-09-13 15:49:39 -07:00
06a9c7a772
Remove unnecessary policy recomputations in confirmEpochLive.
...
Watching for interface changes on readied servers was done as a workaround for
a case where all futures could be ready, but the policy verification would
never succeed. This turns out to be because stopping a tlog causes an error to
be returned. However, if a TLog is stopped, then we know that we can't do any
more commits, so we can just immediately stop trying and never mark our future
as ready.
2017-09-13 15:45:09 -07:00
8cb53fd608
Merge pull request #149 from cie/choose-leader-on-stateless-processes
...
choose leader on the perferred process class
2017-09-13 13:58:49 -07:00
4fa2415553
Merge branch 'release-5.0'
2017-09-08 17:28:12 -07:00
bb8a245bdb
circus: throughput test scales latency error by the target latency
2017-09-08 17:27:54 -07:00
bd1c7e7295
Use addTeamsBestOf() instead of addAllTeams() when team size is greater than 3
2017-09-07 12:31:01 -07:00
c7df951f7c
Using BackupConfig from backup.actor.cpp to reduce intermediate
...
functions.
2017-09-07 08:36:36 -07:00
fe208d6adf
Merge branch 'master' of github.com:apple/foundationdb into backup
2017-09-06 10:01:55 -07:00
83810edabc
Backup/Restore tag can be std::string instad of Key.
2017-09-05 11:38:40 -07:00
dc1f7ca6b7
testers now use client locality load balancing
2017-09-01 12:53:01 -07:00
cc24072a5d
Add the multi version API to the list of APIs to choose in the APICorrectness tester. Support for the multi-version client already existed.
2017-08-31 16:23:55 -07:00
d61be4c760
Merge branch 'release-5.0'
2017-08-30 12:59:24 -07:00
963e1c3f31
fix: we need to reboot the process even if it will result in too many files, because the check will not succeed without it
2017-08-30 12:58:46 -07:00
8d97a15c3f
BUGGIFY recovery to lock only the minimum number of TLogs required to prevent a quorum.
...
This is to test the quorum logic introduced in the previous patch, and should
flush out any other bugs that rely on TLog locking during recovery.
2017-08-29 14:43:40 -07:00
f8486d1368
Only ensure a quorum of TLogs are unlocked to confirm the epoch hasn't ended.
...
Currently, GRV will wait to hear back from (almost) all TLogs to confirm that
they're unlocked and that the current epoch hasn't ended. This confirms that
there isn't a new set of proxies and using the commit version from the old set
of proxies would violate causal consistency.
However, during recovery, we ensure that no quorum of TLogs exists before
starting a new epoch and allowing new commits on the new TLogs. Thus, we only
need to wait until we have a quorum of TLogs that are unlocked.
This should be a significant improvement in latency particularly for the cases
when we start running >10 TLogs.
2017-08-29 14:43:40 -07:00
4c1d61cd08
Assorted minor changes.
...
In which we:
* Clarify some math in a comment
* Remove misleading debugging information
* Add a useful trace event
2017-08-29 14:43:40 -07:00
dbfa94f735
LF -> CRLF
...
It appears a previous patch left parts of this file ending with LF, and the
majority of the file ends in CRLF. I see no reason to keep this inconsistency,
but these line ending wars are going to drive me insane.
2017-08-29 14:43:40 -07:00
6020d70863
Added trace event to track reboots initiated by ConsistencyCheck workload in simulation
2017-08-29 11:41:27 -07:00
c95a1be5ec
Add trace event for rebooting process during simulation for consistency check
2017-08-29 11:00:44 -07:00
86774f6e42
Merge branch 'release-5.0'
2017-08-28 17:17:00 -07:00
03478561b9
fix: Set lock aware at the transaction level for latency probe to avoid having to fill the shard cache every time.
2017-08-28 17:16:46 -07:00
9a0a3b6329
Merge commit '66528becb82d826e81fa644bb378212584ab580e'
2017-08-28 16:47:59 -07:00
9fe927127f
choose leader on the perferred process class
2017-08-28 14:41:04 -07:00
44e0df78c5
Added support for tracking roles for simulation workers
...
Fixed the exclusion and inclusion address simulation API and integration within workloads
Added more information within trace events for simulation
2017-08-28 11:25:37 -07:00
581bd6c8ed
Added option to delay the displaying of the simulation workers
2017-08-28 10:53:56 -07:00
300b5a17ed
Merge branch 'release-5.0'
2017-08-25 18:55:33 -07:00
272b4b984c
fix: fixed a rare bug where we do not wait for a file in the process of being deleted to shutdown before rebooting a machine
2017-08-25 10:12:58 -07:00
26a5b5e422
rollback workload now clogs the communication between one of the proxies and the tlogs, since that is what will cause a rollback
2017-08-23 16:08:13 -07:00
4c706d33e9
Merge branch 'release-5.0'
2017-08-23 14:59:43 -07:00
be941b4bd1
sending void to committed could cause self to be deleted, so call cleanup before sending
2017-08-23 13:56:18 -07:00
7729f663e9
Ensured that the circus id is always lowercase
2017-08-23 13:45:00 -07:00
f9308b8fa6
Merge pull request #145 from cie/alexmiller/simrefactor
...
Refactor simulation to pull all configuration parameters into one struct.
2017-08-23 12:54:21 -07:00
4b40f817f1
fix: is recovery is cancelled before the copy is complete, remove the tlog
2017-08-23 12:26:03 -07:00
8056b78414
Merge branch 'release-5.0'
2017-08-22 13:51:19 -07:00
814e471689
Added support for displaying initial workers via printf within simulation using a workload
2017-08-22 13:38:24 -07:00
7b78035365
Have SimulationConfig wrap DatabaseConfiguration to reduce code duplication.
...
This effectively turns initializing SimulationConfig into the equivalent of
building a config string and calling buildConfiguration on it.
2017-08-22 10:13:57 -07:00
9b25c72971
Pull database config and cluster config into one struct.
...
This will allow us to specify custom situations to be chosen more frequently,
and in particular control machines and processes.
2017-08-21 22:35:44 -07:00
5ee07b1a9e
Merge branch 'release-5.0'
2017-08-14 16:56:58 -07:00
de1b590a8a
The TLog did not delete data from removed logs
...
The TLog continued to make data from removed logs persistent
2017-08-11 18:08:09 -07:00
50fb44be92
Merge branch 'release-5.0'
...
# Conflicts:
# versions.target
2017-08-09 23:36:12 -07:00
2335fc73f2
fix: peek cursors were being timed out every 10 minutes, instead of 10 minutes after the last use
...
fix: if an interface is changed while we are not waiting in getMore, we will not reset the sequence to 0.
2017-08-09 15:58:06 -07:00
47a37f3f1e
Merge pull request #135 from cie/switch-for-data-distribution
...
Add a switch to turn off data distribution in CLI
2017-08-07 12:54:08 -07:00
ca7437ecf6
Merge branch 'release-5.0'
2017-08-02 22:07:01 -07:00
d0fbc41338
set LOCK_AWARE on several transactions used for getting cluster info for the consistency check
2017-07-28 18:50:32 -07:00
6a8a5c41b0
Add a switch to turn off data distribution in CLI
2017-07-28 18:14:55 -07:00
4243486f54
fix: TLogMetrics was being track latested with the wrong ID
2017-07-28 14:37:23 -07:00
37e5e2acbb
Fix parentheses issue in StorageMetrics.actor.h
2017-07-27 12:03:36 -07:00
cdc62e265c
Merge pull request #133 from cie/shard-system-keyspace
...
Shard system keyspace
2017-07-26 17:09:13 -07:00
41c90bcdea
Merge commit '89ac94853c70d08289e7fb58055bc5d0cd4e494d'
2017-07-26 15:35:36 -07:00
d8e308c18f
Enable use of incremental delete when deleting disk queue and sqlite KVS sqlite files.
2017-07-26 14:11:11 -07:00
53e1ae9f60
shard system keyspace
2017-07-26 13:47:31 -07:00
4aaee86c2a
Moved MetricLogger actor to fdbclient so applications other than fdbserver can use it.
2017-07-24 13:13:06 -07:00
7fec378830
do not continue copying data from prior generations after being locked
2017-07-19 15:11:18 -07:00
c860f09d8a
Merge branch 'release-5.0'
2017-07-14 16:01:15 -07:00
660729839c
moved Notified.h from flow -> fdbclient ; flow bindings package does better job when excluding testers
2017-07-14 15:49:30 -07:00
f75b6f333b
Merge branch 'release-5.0'
2017-07-13 11:21:18 -07:00
aa1c903b52
fix: do not log that data distribution is initialized until readyToStart is ready
2017-06-30 16:21:59 -07:00
a2d229ff00
Merge branch 'release-5.0'
2017-06-29 08:16:22 -07:00
69c862ed6e
updated release notes for 5.0.1
2017-06-28 16:52:45 -07:00
663535e00a
add the same number for shared metrics and local metrics
2017-06-28 15:21:54 -07:00
fe37d0b056
added additional logging about bytesInput and bytesDurable
2017-06-28 13:29:40 -07:00
31d562ff7b
Merge branch 'release-5.0'
...
# Conflicts:
# fdbclient/ManagementAPI.actor.cpp
# fdbserver/DatabaseConfiguration.cpp
# versions.target
2017-06-27 11:16:08 -07:00
9fd5955e92
Merge branch 'master' into removing-old-dc-code
2017-06-26 16:27:10 -07:00
15cb498aa7
removed fast_recovery_double and fast_recovery_triple from the fdbcli
2017-06-23 16:18:23 -07:00
533dca95d8
fix: bytesDurable was not correctly increased when a log was removed
...
re-added many TLogMetrics
added a new role for the shared tlog
2017-06-22 17:21:42 -07:00
6d19580789
Merge branch 'release-5.0' of github.com:apple/foundationdb into release-5.0
...
# Conflicts:
# fdbrpc/simulator.h
2017-06-19 17:39:37 -07:00
96f40d8eb0
Added support for optionally re-including excluded servers
...
Removed unneeded code to protect coordinators
Ensured that simulation exclusion list is updated for excluded processes
2017-06-19 16:51:07 -07:00
9553458b78
Updated simulation to support managing exclusion and inclusion address
...
Added method for identifying acceptable availability process classes
Extended cluster availability function to ensure coordinators can be auto configured
Fixed availability function to allow protected processes to be considered as dead if not available
Added debug trace events for providing machine state when considering availability
Added trace event for protected coordinators
2017-06-19 16:48:15 -07:00
f405c8d88e
Merge branch 'release-4.6' into release-5.0
...
# Conflicts:
# fdbrpc/AsyncFileKAIO.actor.h
# fdbrpc/sim2.actor.cpp
# fdbserver/optimisttest.actor.cpp
# versions.target
2017-06-15 17:40:19 -07:00
5de6c703cf
Fixed line endings which were changed to UNIX style during a recent merge.
2017-06-15 16:57:35 -07:00
4bdcd8fc12
Merge branch 'release-4.6' into release-5.0
...
# Conflicts:
# bindings/bindingtester/run_binding_tester.sh
# fdbrpc/AsyncFileKAIO.actor.h
2017-06-14 16:43:53 -07:00
cefaa2391d
fix: tlog actor needs to be cancelled when worker shutdown
2017-06-02 17:56:47 -07:00
766dc23e26
fix: do not use TLS in protectedAddresses
2017-06-02 13:52:21 -07:00
b108c1ea0d
fix: do not delay before setting logData->version
2017-06-02 11:27:37 -07:00
2d0dbd57e8
randomized the delays in atomic switchover workload
2017-06-01 12:08:21 -07:00
bfcbb5623f
fixed build from merge error
2017-06-01 12:07:30 -07:00
276073d91b
Merge branch 'release-4.6' into release-5.0
2017-06-01 11:54:54 -07:00
fa4fdb1f1d
Merge branch 'fix-io-timeout-handling' into release-5.0
...
# Conflicts:
# fdbserver/optimisttest.actor.cpp
2017-05-31 17:03:15 -07:00
1626e16377
Merge branch 'release-4.6' into release-5.0
2017-05-31 16:23:37 -07:00
333f2e4865
Added connection fault disabler within setup of backup submission. It should be reviewed to determine the amount of time to wait before disabling
2017-05-31 14:21:50 -07:00
98604d33a0
Merge branch 'fix-io-timeout-handling'
...
# Conflicts:
# fdbrpc/AsyncFileKAIO.actor.h
# fdbrpc/sim2.actor.cpp
# fdbserver/KeyValueStoreSQLite.actor.cpp
# fdbserver/optimisttest.actor.cpp
# fdbserver/worker.actor.cpp
# fdbserver/workloads/MachineAttrition.actor.cpp
# tests/fast/SidebandWithStatus.txt
# tests/rare/LargeApiCorrectnessStatus.txt
# tests/slow/DDBalanceAndRemoveStatus.txt
2017-05-26 18:43:08 -07:00
ba606c2135
Removed NOT_IN_CLEAN macro from file
2017-05-26 14:52:06 -07:00
b28ed397a2
Fixed printf field width specifier to reduce compilation warnings within OS X
2017-05-26 14:51:34 -07:00
0b9ed67e12
Fixed support for RemoveServers Workload
...
Added availability functions to simulation
2017-05-26 14:20:11 -07:00
16cc0821b1
Removed dead machine option from simulation
2017-05-25 16:29:02 -07:00
a674cb4ef4
Initial repository commit
2017-05-25 13:48:44 -07:00
Yichi Chiang