wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
foundationdb/fdbserver/ClusterController.actor.cpp

2771 lines
118 KiB
C++
Raw Normal View History

Initial repository commit
2017-05-26 04:48:44 +08:00
/*
* ClusterController.actor.cpp
*
* This source file is part of the FoundationDB open source project
*
Minor fix on ratekeeper work registration.
2019-02-16 09:29:52 +08:00
* Copyright 2013-2019 Apple Inc. and the FoundationDB project authors
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* http://www.apache.org/licenses/LICENSE-2.0
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#include "fdbrpc/FailureMonitor.h"
#include "flow/ActorCollection.h"
Use proper fwd decl in NativeAPI Also NativeAPI.h -> NativeAPI.actor.h
2019-02-18 07:41:16 +08:00
#include "fdbclient/NativeAPI.actor.h"
Initial repository commit
2017-05-26 04:48:44 +08:00
#include "fdbserver/CoordinationInterface.h"
Add a new DataDistributor role. Let cluster controller to start a new data distributor role by sending a message to a chosen worker. Change MasterInterface usage in DataDistribution to masterId Add DataDistributor rejoin handling. This allows the data distributor to tell the new cluster controller of its existence so that the controller doesn't spawn a new one. I.e., there should be only ONE data distributor in the cluster. If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries to recruit one as DD. CC also monitors DD and restarts one if it failed. The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for the new DD. Add GetRecoveryInfo RPC to master server, which is called by data distributor to obtain the recovery Transaction version from the master server.
2018-12-14 05:31:37 +08:00
#include "fdbserver/DataDistributorInterface.h"
Adjust all includes to be relative to the root. Remove the use of relative paths. A header at foo/bar.h could be included by files under foo/ with "bar.h", but would be included everywhere else as "foo/bar.h". Adjust so that every include references such a header with the latter form. Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-20 01:30:13 +08:00
#include "fdbserver/Knobs.h"
Use ACTOR forward declarations in MoveKeys Also MoveKeys.h -> MoveKeys.actor.h
2019-02-18 10:55:52 +08:00
#include "fdbserver/MoveKeys.actor.h"
Use fwd decls in WorkerInterface Also WorkerInterface.h -> WorkerInterface.actor.h
2019-02-18 11:13:26 +08:00
#include "fdbserver/WorkerInterface.actor.h"
Adjust all includes to be relative to the root. Remove the use of relative paths. A header at foo/bar.h could be included by files under foo/ with "bar.h", but would be included everywhere else as "foo/bar.h". Adjust so that every include references such a header with the latter form. Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-20 01:30:13 +08:00
#include "fdbserver/LeaderElection.h"
Add used worker IDs to cluster controller This "usedIds" is updated when receiving a master registration message, so that when recruiting new data distributor, existing assignment is known.
2019-01-29 01:25:15 +08:00
#include "fdbserver/LogSystemConfig.h"
Adjust all includes to be relative to the root. Remove the use of relative paths. A header at foo/bar.h could be included by files under foo/ with "bar.h", but would be included everywhere else as "foo/bar.h". Adjust so that every include references such a header with the latter form. Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-20 01:30:13 +08:00
#include "fdbserver/WaitFailure.h"
#include "fdbserver/ClusterRecruitmentInterface.h"
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
#include "fdbserver/RatekeeperInterface.h"
Adjust all includes to be relative to the root. Remove the use of relative paths. A header at foo/bar.h could be included by files under foo/ with "bar.h", but would be included everywhere else as "foo/bar.h". Adjust so that every include references such a header with the latter form. Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-20 01:30:13 +08:00
#include "fdbserver/ServerDBInfo.h"
#include "fdbserver/Status.h"
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
#include "fdbserver/LatencyBandConfig.h"
Initial repository commit
2017-05-26 04:48:44 +08:00
#include <algorithm>
#include "fdbclient/DatabaseContext.h"
Adjust all includes to be relative to the root. Remove the use of relative paths. A header at foo/bar.h could be included by files under foo/ with "bar.h", but would be included everywhere else as "foo/bar.h". Adjust so that every include references such a header with the latter form. Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-20 01:30:13 +08:00
#include "fdbserver/RecoveryState.h"
Initial repository commit
2017-05-26 04:48:44 +08:00
#include "fdbclient/ReadYourWrites.h"
#include "fdbrpc/Replication.h"
#include "fdbrpc/ReplicationUtils.h"
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
#include "fdbclient/KeyBackedTypes.h"
Stop performing self-moves. (e.g. a = std::move(a)) self-moves are frowned upon in C++, and in our code this generally happens from calls to swap as part of trying to implement a "unordered erase" function via swap-to-the-end-and-pop_back. For convenience, a swapAndPop() function is now offered that performs this, while disallowing self-moves.
2018-08-02 09:09:54 +08:00
#include "flow/Util.h"
Fix more instances of actorcompiler.h being in the wrong place.
2018-08-11 08:24:52 +08:00
#include "flow/actorcompiler.h" // This must be the last #include.
Initial repository commit
2017-05-26 04:48:44 +08:00
void failAfter( Future<Void> trigger, Endpoint e );
struct WorkerInfo : NonCopyable {
Future<Void> watcher;
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
ReplyPromise<RegisterWorkerReply> reply;
Initial repository commit
2017-05-26 04:48:44 +08:00
Generation gen;
int reboots;
keep track of the last time a process became available to set a better starting value for remoteStartTime
2018-11-05 06:33:03 +08:00
double lastAvailableTime;
Initial repository commit
2017-05-26 04:48:44 +08:00
ProcessClass initialClass;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
ClusterControllerPriorityInfo priorityInfo;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
WorkerDetails details;
Add actors to store halt request futures Address best fitness in checking better DD or RK.
2019-03-23 09:00:16 +08:00
Future<Void> haltRatekeeper;
Future<Void> haltDistributor;
Initial repository commit
2017-05-26 04:48:44 +08:00
keep track of the last time a process became available to set a better starting value for remoteStartTime
2018-11-05 06:33:03 +08:00
WorkerInfo() : gen(-1), reboots(0), lastAvailableTime(now()), priorityInfo(ProcessClass::UnsetFit, false, ClusterControllerPriorityInfo::FitnessUnknown) {}
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
WorkerInfo( Future<Void> watcher, ReplyPromise<RegisterWorkerReply> reply, Generation gen, WorkerInterface interf, ProcessClass initialClass, ProcessClass processClass, ClusterControllerPriorityInfo priorityInfo, bool degraded ) :
watcher(watcher), reply(reply), gen(gen), reboots(0), lastAvailableTime(now()), initialClass(initialClass), priorityInfo(priorityInfo), details(interf, processClass, degraded) {}
Initial repository commit
2017-05-26 04:48:44 +08:00
Remove noexcept macro and replace with BOOST_NOEXCEPT. BOOST_NOEXCEPT does what the noexcept macro was supposed to do, but in a way that is correctly maintained over time.
2019-01-26 08:49:59 +08:00
WorkerInfo( WorkerInfo&& r ) BOOST_NOEXCEPT : watcher(std::move(r.watcher)), reply(std::move(r.reply)), gen(r.gen),
fixed merge conflict
2019-03-09 11:51:06 +08:00
reboots(r.reboots), lastAvailableTime(r.lastAvailableTime), initialClass(r.initialClass), priorityInfo(r.priorityInfo), details(std::move(r.details)) {}
Remove noexcept macro and replace with BOOST_NOEXCEPT. BOOST_NOEXCEPT does what the noexcept macro was supposed to do, but in a way that is correctly maintained over time.
2019-01-26 08:49:59 +08:00
void operator=( WorkerInfo&& r ) BOOST_NOEXCEPT {
Initial repository commit
2017-05-26 04:48:44 +08:00
watcher = std::move(r.watcher);
reply = std::move(r.reply);
gen = r.gen;
reboots = r.reboots;
keep track of the last time a process became available to set a better starting value for remoteStartTime
2018-11-05 06:33:03 +08:00
lastAvailableTime = r.lastAvailableTime;
Initial repository commit
2017-05-26 04:48:44 +08:00
initialClass = r.initialClass;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
priorityInfo = r.priorityInfo;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
details = std::move(r.details);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
};
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
struct WorkerFitnessInfo {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
WorkerDetails worker;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
ProcessClass::Fitness fitness;
int used;
WorkerFitnessInfo() : fitness(ProcessClass::NeverAssign), used(0) {}
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
WorkerFitnessInfo(WorkerDetails worker, ProcessClass::Fitness fitness, int used) : worker(worker), fitness(fitness), used(used) {}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
};
Initial repository commit
2017-05-26 04:48:44 +08:00
class ClusterControllerData {
public:
struct DBInfo {
Reference<AsyncVar<ClientDBInfo>> clientInfo;
Reference<AsyncVar<ServerDBInfo>> serverInfo;
ProcessIssuesMap clientsWithIssues, workersWithIssues;
std::map<NetworkAddress, double> incompatibleConnections;
ClientVersionMap clientVersionMap;
Status: TLS client status Use ClientStatusInfo structure for each network address (client), instead of passing each status info as a parameter.
2019-02-27 08:20:05 +08:00
std::map<NetworkAddress, ClientStatusInfo> clientStatusInfoMap;
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
AsyncTrigger forceMasterFailure;
Initial repository commit
2017-05-26 04:48:44 +08:00
int64_t masterRegistrationCount;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
bool recoveryStalled;
added the (untested) ability to force a recovery to the remote datacenter, even if that results in data loss. If the DR lag is more than 1 week there could be potential data corruption if any primary storage servers are still alive.
2018-07-01 21:39:04 +08:00
bool forceRecovery;
Initial repository commit
2017-05-26 04:48:44 +08:00
DatabaseConfiguration config; // Asynchronously updated via master registration
Add fullyRecoveredConfig for checking exclusions
2017-11-16 09:15:24 +08:00
DatabaseConfiguration fullyRecoveredConfig;
Initial repository commit
2017-05-26 04:48:44 +08:00
Database db;
The cluster controller will increase its failure monitoring delay after there have been many unfinishedRecoveries
2018-09-01 01:51:55 +08:00
int unfinishedRecoveries;
int logGenerations;
Initial repository commit
2017-05-26 04:48:44 +08:00
The cluster controller will increase its failure monitoring delay after there have been many unfinishedRecoveries
2018-09-01 01:51:55 +08:00
DBInfo() : masterRegistrationCount(0), recoveryStalled(false), forceRecovery(false), unfinishedRecoveries(0), logGenerations(0),
Initial repository commit
2017-05-26 04:48:44 +08:00
clientInfo( new AsyncVar<ClientDBInfo>( ClientDBInfo() ) ),
This is the first step in eliminating the usage of database names in our code. The C API remains the same, but underneath that all usage of database names is eliminated.
2018-08-17 01:24:12 +08:00
serverInfo( new AsyncVar<ServerDBInfo>( ServerDBInfo() ) ),
Initial repository commit
2017-05-26 04:48:44 +08:00
db( DatabaseContext::create( clientInfo, Future<Void>(), LocalityData(), true, TaskDefaultEndpoint, true ) ) // SOMEDAY: Locality!
{
Add DataDistributorInterface to ServerDBInfo Also change the Proxy and QuietDatabase to use the DataDistributorInterface.
2019-01-19 03:30:18 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
void setDistributor(const DataDistributorInterface& interf) {
Add DataDistributorInterface to ServerDBInfo Also change the Proxy and QuietDatabase to use the DataDistributorInterface.
2019-01-19 03:30:18 +08:00
ServerDBInfo newInfo = serverInfo->get();
newInfo.id = g_random->randomUniqueID();
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
newInfo.distributor = interf;
Fix wait failure bug on cluster controller The setDistributor() sets an AsyncVar and then runs waitFailureClient. This ordering is wrong because the AsyncVar::set triggers the other loop to run first, which will wait on Never(). The correct code should wait on the Future returned by the waitFailureClient.
2019-01-30 02:14:11 +08:00
serverInfo->set( newInfo );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
void setRatekeeper(const RatekeeperInterface& interf) {
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
ServerDBInfo newInfo = serverInfo->get();
newInfo.id = g_random->randomUniqueID();
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
newInfo.ratekeeper = interf;
serverInfo->set( newInfo );
}
void clearInterf(ProcessClass::ClassType t) {
ServerDBInfo newInfo = serverInfo->get();
newInfo.id = g_random->randomUniqueID();
if (t == ProcessClass::DataDistributorClass) {
newInfo.distributor = Optional<DataDistributorInterface>();
} else if (t == ProcessClass::RateKeeperClass) {
newInfo.ratekeeper = Optional<RatekeeperInterface>();
}
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
serverInfo->set( newInfo );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
};
struct UpdateWorkerList {
Future<Void> init( Database const& db ) {
return update(this, db);
}
void set(Optional<Standalone<StringRef>> processID, Optional<ProcessData> data ) {
delta[processID] = data;
anyDelta.set(true);
}
private:
std::map<Optional<Standalone<StringRef>>, Optional<ProcessData>> delta;
AsyncVar<bool> anyDelta;
ACTOR static Future<Void> update( UpdateWorkerList* self, Database db ) {
// The Database we are using is based on worker registrations to this cluster controller, which come only
// from master servers that we started, so it shouldn't be possible for multiple cluster controllers to fight.
state Transaction tr(db);
loop {
try {
tr.clear( workerListKeys );
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( tr.commit() );
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
} catch (Error& e) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( tr.onError(e) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
loop {
// Wait for some changes
while (!self->anyDelta.get())
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( self->anyDelta.onChange() );
Initial repository commit
2017-05-26 04:48:44 +08:00
self->anyDelta.set(false);
state std::map<Optional<Standalone<StringRef>>, Optional<ProcessData>> delta;
delta.swap( self->delta );
TraceEvent("UpdateWorkerList").detail("DeltaCount", delta.size());
// Do a transaction to write the changes
loop {
try {
for(auto w = delta.begin(); w != delta.end(); ++w) {
if (w->second.present()) {
tr.set( workerListKeyFor( w->first.get() ), workerListValue( w->second.get()) );
} else
tr.clear( workerListKeyFor( w->first.get() ) );
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( tr.commit() );
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
} catch (Error& e) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( tr.onError(e) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
}
}
};
bool workerAvailable( WorkerInfo const& worker, bool checkStable ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
return ( now() - startTime < 2 * FLOW_KNOBS->SERVER_REQUEST_INTERVAL ) || ( IFailureMonitor::failureMonitor().getState(worker.details.interf.storage.getEndpoint()).isAvailable() && ( !checkStable || worker.reboots < 2 ) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
WorkerDetails getStorageWorker( RecruitStorageRequest const& req ) {
Initial repository commit
2017-05-26 04:48:44 +08:00
std::set<Optional<Standalone<StringRef>>> excludedMachines( req.excludeMachines.begin(), req.excludeMachines.end() );
fixed merge problems
2017-09-12 07:37:10 +08:00
std::set<Optional<Standalone<StringRef>>> includeDCs( req.includeDCs.begin(), req.includeDCs.end() );
Initial repository commit
2017-05-26 04:48:44 +08:00
std::set<AddressExclusion> excludedAddresses( req.excludeAddresses.begin(), req.excludeAddresses.end() );
for( auto& it : id_worker )
if( workerAvailable( it.second, false ) &&
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
!excludedMachines.count(it.second.details.interf.locality.zoneId()) &&
( includeDCs.size() == 0 || includeDCs.count(it.second.details.interf.locality.dcId()) ) &&
!addressExcluded(excludedAddresses, it.second.details.interf.address()) &&
it.second.details.processClass.machineClassFitness( ProcessClass::Storage ) <= ProcessClass::UnsetFit ) {
return it.second.details;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
if( req.criticalRecruitment ) {
ProcessClass::Fitness bestFit = ProcessClass::NeverAssign;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
Optional<WorkerDetails> bestInfo;
Initial repository commit
2017-05-26 04:48:44 +08:00
for( auto& it : id_worker ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
ProcessClass::Fitness fit = it.second.details.processClass.machineClassFitness( ProcessClass::Storage );
Initial repository commit
2017-05-26 04:48:44 +08:00
if( workerAvailable( it.second, false ) &&
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
!excludedMachines.count(it.second.details.interf.locality.zoneId()) &&
( includeDCs.size() == 0 || includeDCs.count(it.second.details.interf.locality.dcId()) ) &&
!addressExcluded(excludedAddresses, it.second.details.interf.address()) &&
Initial repository commit
2017-05-26 04:48:44 +08:00
fit < bestFit ) {
bestFit = fit;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
bestInfo = it.second.details;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
if( bestInfo.present() ) {
return bestInfo.get();
}
}
throw no_more_servers();
}
removed LocalitySetRef and IRepPolicyRef typedefs, because for clarity the Ref suffix is reserved for arena allocated objects instead of reference counted objects.
2019-03-14 04:14:39 +08:00
std::vector<WorkerDetails> getWorkersForSeedServers( DatabaseConfiguration const& conf, Reference<IReplicationPolicy> const& policy, Optional<Optional<Standalone<StringRef>>> const& dcId = Optional<Optional<Standalone<StringRef>>>() ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
std::map<ProcessClass::Fitness, vector<WorkerDetails>> fitness_workers;
std::vector<WorkerDetails> results;
removed LocalitySetRef and IRepPolicyRef typedefs, because for clarity the Ref suffix is reserved for arena allocated objects instead of reference counted objects.
2019-03-14 04:14:39 +08:00
Reference<LocalitySet> logServerSet = Reference<LocalitySet>(new LocalityMap<WorkerDetails>());
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
LocalityMap<WorkerDetails>* logServerMap = (LocalityMap<WorkerDetails>*) logServerSet.getPtr();
fix: seed storage servers are recruited based on the storage policy
2017-09-15 08:06:00 +08:00
bool bCompleted = false;
for( auto& it : id_worker ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
auto fitness = it.second.details.processClass.machineClassFitness( ProcessClass::Storage );
if( workerAvailable(it.second, false) && !conf.isExcludedServer(it.second.details.interf.address()) && fitness != ProcessClass::NeverAssign && ( !dcId.present() || it.second.details.interf.locality.dcId()==dcId.get() ) ) {
fitness_workers[ fitness ].push_back(it.second.details);
fix: seed storage servers are recruited based on the storage policy
2017-09-15 08:06:00 +08:00
}
}
for( auto& it : fitness_workers ) {
for (auto& worker : it.second ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
logServerMap->add(worker.interf.locality, &worker);
fix: seed storage servers are recruited based on the storage policy
2017-09-15 08:06:00 +08:00
}
std::vector<LocalityEntry> bestSet;
Merge branch 'master' into feature-remote-logs # Conflicts: # fdbrpc/Locality.cpp # fdbrpc/Locality.h # fdbserver/ClusterController.actor.cpp # fdbserver/ClusterRecruitmentInterface.h # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/fdbserver.vcxproj.filters # fdbserver/masterserver.actor.cpp # fdbserver/worker.actor.cpp # flow/error_definitions.h
2017-10-06 08:09:44 +08:00
if( logServerSet->selectReplicas(policy, bestSet) ) {
fix: seed storage servers are recruited based on the storage policy
2017-09-15 08:06:00 +08:00
results.reserve(bestSet.size());
for (auto& entry : bestSet) {
auto object = logServerMap->getObject(entry);
results.push_back(*object);
}
bCompleted = true;
break;
}
}
logServerSet->clear();
logServerSet.clear();
if (!bCompleted) {
throw no_more_servers();
}
return results;
}
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
std::vector<WorkerDetails> getWorkersForTlogs( DatabaseConfiguration const& conf, int32_t required, int32_t desired, Reference<IReplicationPolicy> const& policy, std::map< Optional<Standalone<StringRef>>, int>& id_used, bool checkStable = false, std::set<Optional<Key>> dcIds = std::set<Optional<Key>>(), std::vector<UID> exclusionWorkerIds = {}) {
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
std::map<std::pair<ProcessClass::Fitness,bool>, vector<WorkerDetails>> fitness_workers;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
std::vector<WorkerDetails> results;
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
std::vector<LocalityData> unavailableLocals;
removed LocalitySetRef and IRepPolicyRef typedefs, because for clarity the Ref suffix is reserved for arena allocated objects instead of reference counted objects.
2019-03-14 04:14:39 +08:00
Reference<LocalitySet> logServerSet;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
LocalityMap<WorkerDetails>* logServerMap;
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
bool bCompleted = false;
Initial repository commit
2017-05-26 04:48:44 +08:00
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
logServerSet = Reference<LocalitySet>(new LocalityMap<WorkerDetails>());
logServerMap = (LocalityMap<WorkerDetails>*) logServerSet.getPtr();
Initial repository commit
2017-05-26 04:48:44 +08:00
for( auto& it : id_worker ) {
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
if (std::find(exclusionWorkerIds.begin(), exclusionWorkerIds.end(), it.second.details.interf.id()) == exclusionWorkerIds.end()) {
auto fitness = it.second.details.processClass.machineClassFitness(ProcessClass::TLog);
if (workerAvailable(it.second, checkStable) && !conf.isExcludedServer(it.second.details.interf.address()) && fitness != ProcessClass::NeverAssign && (!dcIds.size() || dcIds.count(it.second.details.interf.locality.dcId()))) {
fitness_workers[std::make_pair(fitness, it.second.details.degraded)].push_back(it.second.details);
}
else {
unavailableLocals.push_back(it.second.details.interf.locality);
}
Fixed support for RemoveServers Workload Added availability functions to simulation
2017-05-27 05:20:11 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
results.reserve(results.size() + id_worker.size());
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
for (int fitness = ProcessClass::BestFit; fitness != ProcessClass::NeverAssign && !bCompleted; fitness++)
Initial repository commit
2017-05-26 04:48:44 +08:00
{
auto fitnessEnum = (ProcessClass::Fitness) fitness;
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
for(int addingDegraded = 0; addingDegraded < 2; addingDegraded++) {
auto workerItr = fitness_workers.find(std::make_pair(fitnessEnum,(bool)addingDegraded));
fixed review comments
2019-03-13 05:38:54 +08:00
if (workerItr != fitness_workers.end()) {
for (auto& worker : workerItr->second ) {
logServerMap->add(worker.interf.locality, &worker);
}
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
}
fixed review comments
2019-03-13 05:38:54 +08:00
if (logServerSet->size() < (addingDegraded == 0 ? desired : required)) {
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
TraceEvent(SevWarn,"GWFTADTooFew", id).detail("Fitness", fitness).detail("Processes", logServerSet->size()).detail("Required", required).detail("TLogPolicy", policy->info()).detail("DesiredLogs", desired).detail("AddingDegraded", addingDegraded);
}
else if (logServerSet->size() == required || logServerSet->size() <= desired) {
if (logServerSet->validate(policy)) {
for (auto& object : logServerMap->getObjects()) {
results.push_back(*object);
}
bCompleted = true;
break;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
TraceEvent(SevWarn,"GWFTADNotAcceptable", id).detail("Fitness", fitness).detail("Processes", logServerSet->size()).detail("Required", required).detail("TLogPolicy",policy->info()).detail("DesiredLogs", desired).detail("AddingDegraded", addingDegraded);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
// Try to select the desired size, if larger
else {
std::vector<LocalityEntry> bestSet;
std::vector<LocalityData> tLocalities;
// Try to find the best team of servers to fulfill the policy
if (findBestPolicySet(bestSet, logServerSet, policy, desired, SERVER_KNOBS->POLICY_RATING_TESTS, SERVER_KNOBS->POLICY_GENERATIONS)) {
results.reserve(results.size() + bestSet.size());
for (auto& entry : bestSet) {
auto object = logServerMap->getObject(entry);
ASSERT(object);
results.push_back(*object);
tLocalities.push_back(object->interf.locality);
}
TraceEvent("GWFTADBestResults", id).detail("Fitness", fitness).detail("Processes", logServerSet->size()).detail("BestCount", bestSet.size()).detail("BestZones", ::describeZones(tLocalities))
.detail("BestDataHalls", ::describeDataHalls(tLocalities)).detail("TLogPolicy", policy->info()).detail("TotalResults", results.size()).detail("DesiredLogs", desired).detail("AddingDegraded", addingDegraded);
bCompleted = true;
break;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
TraceEvent(SevWarn,"GWFTADNoBest", id).detail("Fitness", fitness).detail("Processes", logServerSet->size()).detail("Required", required).detail("TLogPolicy", policy->info()).detail("DesiredLogs", desired).detail("AddingDegraded", addingDegraded);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
}
// If policy cannot be satisfied
first working version of non-copying recovery working with fearless configurations
2018-04-09 12:24:05 +08:00
if (!bCompleted) {
std::vector<LocalityData> tLocalities;
for (auto& object : logServerMap->getObjects()) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
tLocalities.push_back(object->interf.locality);
first working version of non-copying recovery working with fearless configurations
2018-04-09 12:24:05 +08:00
}
TraceEvent(SevWarn, "GetTLogTeamFailed").detail("Policy", policy->info()).detail("Processes", logServerSet->size()).detail("Workers", id_worker.size()).detail("FitnessGroups", fitness_workers.size())
.detail("TLogZones", ::describeZones(tLocalities)).detail("TLogDataHalls", ::describeDataHalls(tLocalities)).detail("MissingZones", ::describeZones(unavailableLocals))
.detail("MissingDataHalls", ::describeDataHalls(unavailableLocals)).detail("Required", required).detail("DesiredLogs", desired).detail("RatingTests",SERVER_KNOBS->POLICY_RATING_TESTS)
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
.detail("CheckStable", checkStable).detail("NumExclusionWorkers", exclusionWorkerIds.size()).detail("PolicyGenerations",SERVER_KNOBS->POLICY_GENERATIONS).backtrace();
Initial repository commit
2017-05-26 04:48:44 +08:00
logServerSet->clear();
logServerSet.clear();
throw no_more_servers();
}
for (auto& result : results) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
id_used[result.interf.locality.processId()]++;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
first working version of non-copying recovery working with fearless configurations
2018-04-09 12:24:05 +08:00
TraceEvent("GetTLogTeamDone").detail("Completed", bCompleted).detail("Policy", policy->info()).detail("Results", results.size()).detail("Processes", logServerSet->size()).detail("Workers", id_worker.size())
.detail("Required", required).detail("Desired", desired).detail("RatingTests",SERVER_KNOBS->POLICY_RATING_TESTS).detail("PolicyGenerations",SERVER_KNOBS->POLICY_GENERATIONS);
Initial repository commit
2017-05-26 04:48:44 +08:00
logServerSet->clear();
logServerSet.clear();
return results;
}
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
//FIXME: This logic will fallback unnecessarily when usable dcs > 1 because it does not check all combinations of potential satellite locations
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
std::vector<WorkerDetails> getWorkersForSatelliteLogs( const DatabaseConfiguration& conf, const RegionInfo& region, const RegionInfo& remoteRegion, std::map< Optional<Standalone<StringRef>>, int>& id_used, bool& satelliteFallback, bool checkStable = false ) {
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
int startDC = 0;
loop {
if(startDC > 0 && startDC >= region.satellites.size() + 1 - (satelliteFallback ? region.satelliteTLogUsableDcsFallback : region.satelliteTLogUsableDcs)) {
if(satelliteFallback || region.satelliteTLogUsableDcsFallback == 0) {
throw no_more_servers();
} else {
spend less time before using satellite fallback, because the database will be unavailable during this waiting time
2018-07-03 00:50:52 +08:00
if(now() - startTime < SERVER_KNOBS->WAIT_FOR_GOOD_RECRUITMENT_DELAY) {
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
throw operation_failed();
}
satelliteFallback = true;
startDC = 0;
}
}
try {
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
bool remoteDCUsedAsSatellite = false;
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
std::set<Optional<Key>> satelliteDCs;
for(int s = startDC; s < std::min<int>(startDC + (satelliteFallback ? region.satelliteTLogUsableDcsFallback : region.satelliteTLogUsableDcs), region.satellites.size()); s++) {
satelliteDCs.insert(region.satellites[s].dcId);
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
if (region.satellites[s].dcId == remoteRegion.dcId) {
remoteDCUsedAsSatellite = true;
}
}
std::vector<UID> exclusionWorkerIds;
// FIXME: If remote DC is used as satellite then this logic only ensures that required number of remote TLogs can be recruited. It does not balance the number of desired TLogs
// across the satellite and remote sides.
if (remoteDCUsedAsSatellite) {
Addressed Review Comments
2019-03-20 06:23:14 +08:00
std::map< Optional<Standalone<StringRef>>, int> tmpIdUsed;
auto remoteLogs = getWorkersForTlogs(conf, conf.getRemoteTLogReplicationFactor(), conf.getRemoteTLogReplicationFactor(), conf.getRemoteTLogPolicy(), tmpIdUsed, false, { remoteRegion.dcId }, {});
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
std::transform(remoteLogs.begin(), remoteLogs.end(), std::back_inserter(exclusionWorkerIds), [](const WorkerDetails &in) { return in.interf.id(); });
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
}
if(satelliteFallback) {
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
return getWorkersForTlogs( conf, region.satelliteTLogReplicationFactorFallback, conf.getDesiredSatelliteLogs(region.dcId)*region.satelliteTLogUsableDcsFallback/region.satelliteTLogUsableDcs, region.satelliteTLogPolicyFallback, id_used, checkStable, satelliteDCs, exclusionWorkerIds);
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
} else {
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
return getWorkersForTlogs( conf, region.satelliteTLogReplicationFactor, conf.getDesiredSatelliteLogs(region.dcId), region.satelliteTLogPolicy, id_used, checkStable, satelliteDCs, exclusionWorkerIds);
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
}
} catch (Error &e) {
if(e.code() != error_code_no_more_servers) {
throw;
}
}
startDC++;
}
}
Add actors to store halt request futures Address best fitness in checking better DD or RK.
2019-03-23 09:00:16 +08:00
ProcessClass::Fitness getBestFitnessForRoleInDatacenter(ProcessClass::ClusterRole role) {
ProcessClass::Fitness bestFitness = ProcessClass::NeverAssign;
for (const auto& it : id_worker) {
if (it.second.priorityInfo.isExcluded || it.second.details.interf.locality.dcId() != clusterControllerDcId) {
continue;
}
bestFitness = std::min(bestFitness, it.second.details.processClass.machineClassFitness(role));
}
return bestFitness;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
WorkerFitnessInfo getWorkerForRoleInDatacenter(Optional<Standalone<StringRef>> const& dcId, ProcessClass::ClusterRole role, ProcessClass::Fitness unacceptableFitness, DatabaseConfiguration const& conf, std::map< Optional<Standalone<StringRef>>, int>& id_used, bool checkStable = false ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
std::map<std::pair<ProcessClass::Fitness,int>, vector<WorkerDetails>> fitness_workers;
Initial repository commit
2017-05-26 04:48:44 +08:00
for( auto& it : id_worker ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
auto fitness = it.second.details.processClass.machineClassFitness( role );
if(conf.isExcludedServer(it.second.details.interf.address())) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
fitness = std::max(fitness, ProcessClass::ExcludeFit);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if( workerAvailable(it.second, checkStable) && fitness < unacceptableFitness && it.second.details.interf.locality.dcId()==dcId ) {
fitness_workers[ std::make_pair(fitness, id_used[it.first]) ].push_back(it.second.details);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
for( auto& it : fitness_workers ) {
auto& w = it.second;
g_random->randomShuffle(w);
for( int i=0; i < w.size(); i++ ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
id_used[w[i].interf.locality.processId()]++;
Initial repository commit
2017-05-26 04:48:44 +08:00
return WorkerFitnessInfo(w[i], it.first.first, it.first.second);
}
}
throw no_more_servers();
}
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
vector<WorkerDetails> getWorkersForRoleInDatacenter(Optional<Standalone<StringRef>> const& dcId, ProcessClass::ClusterRole role, int amount, DatabaseConfiguration const& conf, std::map< Optional<Standalone<StringRef>>, int>& id_used, Optional<WorkerFitnessInfo> minWorker = Optional<WorkerFitnessInfo>(), bool checkStable = false ) {
std::map<std::pair<ProcessClass::Fitness,int>, vector<WorkerDetails>> fitness_workers;
vector<WorkerDetails> results;
Initial repository commit
2017-05-26 04:48:44 +08:00
if (amount <= 0)
return results;
for( auto& it : id_worker ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
auto fitness = it.second.details.processClass.machineClassFitness( role );
if( workerAvailable(it.second, checkStable) && !conf.isExcludedServer(it.second.details.interf.address()) && it.second.details.interf.locality.dcId() == dcId &&
( !minWorker.present() || ( it.second.details.interf.id() != minWorker.get().worker.interf.id() && ( fitness < minWorker.get().fitness || (fitness == minWorker.get().fitness && id_used[it.first] <= minWorker.get().used ) ) ) ) ) {
fitness_workers[ std::make_pair(fitness, id_used[it.first]) ].push_back(it.second.details);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
for( auto& it : fitness_workers ) {
auto& w = it.second;
g_random->randomShuffle(w);
for( int i=0; i < w.size(); i++ ) {
results.push_back(w[i]);
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
id_used[w[i].interf.locality.processId()]++;
Initial repository commit
2017-05-26 04:48:44 +08:00
if( results.size() == amount )
return results;
}
}
return results;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
struct RoleFitness {
ProcessClass::Fitness bestFit;
ProcessClass::Fitness worstFit;
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
ProcessClass::ClusterRole role;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
int count;
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
bool worstIsDegraded;
Initial repository commit
2017-05-26 04:48:44 +08:00
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
RoleFitness(int bestFit, int worstFit, int count, ProcessClass::ClusterRole role) : bestFit((ProcessClass::Fitness)bestFit), worstFit((ProcessClass::Fitness)worstFit), count(count), role(role), worstIsDegraded(false) {}
Initial repository commit
2017-05-26 04:48:44 +08:00
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
RoleFitness(int fitness, int count, ProcessClass::ClusterRole role) : bestFit((ProcessClass::Fitness)fitness), worstFit((ProcessClass::Fitness)fitness), count(count), role(role), worstIsDegraded(false) {}
Initial repository commit
2017-05-26 04:48:44 +08:00
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
RoleFitness() : bestFit(ProcessClass::NeverAssign), worstFit(ProcessClass::NeverAssign), role(ProcessClass::NoRole), count(0), worstIsDegraded(false) {}
Enable checkUsingDesiredClasses() in consistency check
2017-10-25 03:58:54 +08:00
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
RoleFitness(RoleFitness first, RoleFitness second, ProcessClass::ClusterRole role) : bestFit(std::min(first.worstFit, second.worstFit)), worstFit(std::max(first.worstFit, second.worstFit)), count(first.count + second.count), role(role) {
if(first.worstFit > second.worstFit) {
worstIsDegraded = first.worstIsDegraded;
} else if(second.worstFit > first.worstFit) {
worstIsDegraded = second.worstIsDegraded;
} else {
worstIsDegraded = first.worstIsDegraded || second.worstIsDegraded;
}
}
Enable checkUsingDesiredClasses() in consistency check
2017-10-25 03:58:54 +08:00
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
RoleFitness( vector<WorkerDetails> workers, ProcessClass::ClusterRole role ) : role(role) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
worstFit = ProcessClass::BestFit;
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
worstIsDegraded = false;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
bestFit = ProcessClass::NeverAssign;
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
for(auto& it : workers) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
auto thisFit = it.processClass.machineClassFitness( role );
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
if(thisFit > worstFit) {
worstFit = thisFit;
worstIsDegraded = it.degraded;
} else if(thisFit == worstFit) {
worstIsDegraded = worstIsDegraded || it.degraded;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
bestFit = std::min(bestFit, thisFit);
}
count = workers.size();
fix: degraded is only used for tlog recruitment, so we should not use it in the fitness calculation for other roles
2019-03-22 02:23:49 +08:00
//degraded is only used for recruitment of tlogs
if(role != ProcessClass::TLog) {
worstIsDegraded = false;
}
Enable checkUsingDesiredClasses() in consistency check
2017-10-25 03:58:54 +08:00
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
bool operator < (RoleFitness const& r) const {
if (worstFit != r.worstFit) return worstFit < r.worstFit;
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
if (worstIsDegraded != r.worstIsDegraded) return r.worstIsDegraded;
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
// FIXME: TLog recruitment process does not guarantee the best fit is not worsened.
Update fdbserver/ClusterController.actor.cpp Co-Authored-By: bnamasivayam <36455962+bnamasivayam@users.noreply.github.com>
2019-01-15 08:57:00 +08:00
if (role != ProcessClass::TLog && role != ProcessClass::LogRouter && bestFit != r.bestFit) return bestFit < r.bestFit;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
return count > r.count;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
bool betterFitness (RoleFitness const& r) const {
fix: tlogFitness did not consider it better to have one tlog of a better fitness fix: checkStable was not used in all places in better master exists fix: we need to call checkOutstanding on worker registration in all cases fix: in case persistentData is keyValueStoreMemory, we need to make sure it is fully recovered before writing to it
2018-01-05 03:33:02 +08:00
if (worstFit != r.worstFit) return worstFit < r.worstFit;
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
if (worstIsDegraded != r.worstIsDegraded) return r.worstFit;
fix: tlogFitness did not consider it better to have one tlog of a better fitness fix: checkStable was not used in all places in better master exists fix: we need to call checkOutstanding on worker registration in all cases fix: in case persistentData is keyValueStoreMemory, we need to make sure it is fully recovered before writing to it
2018-01-05 03:33:02 +08:00
if (bestFit != r.bestFit) return bestFit < r.bestFit;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
return false;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Swapped OkayFit and UnsetFit, because generally if machine classes are set on one machine they are set everywhere and it helps with wait_for_good_recruitment logic wait_for_good_recruitment now requires that you have the desired count of each roll remote recruitment is given a much longer wait_for_good_recruitment time interval, which does not start until enough remote machines have registered
2018-06-23 01:15:24 +08:00
bool betterCount (RoleFitness const& r) const {
simplified betterCount logic
2018-06-23 01:38:36 +08:00
if(count > r.count) return true;
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
if(worstFit != r.worstFit) return worstFit < r.worstFit;
if (worstIsDegraded != r.worstIsDegraded) return r.worstFit;
return false;
Swapped OkayFit and UnsetFit, because generally if machine classes are set on one machine they are set everywhere and it helps with wait_for_good_recruitment logic wait_for_good_recruitment now requires that you have the desired count of each roll remote recruitment is given a much longer wait_for_good_recruitment time interval, which does not start until enough remote machines have registered
2018-06-23 01:15:24 +08:00
}
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
bool operator == (RoleFitness const& r) const { return worstFit == r.worstFit && bestFit == r.bestFit && count == r.count && worstIsDegraded == r.worstIsDegraded; }
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
std::string toString() const { return format("%d %d %d %d", bestFit, worstFit, count, worstIsDegraded); }
Initial repository commit
2017-05-26 04:48:44 +08:00
};
std::set<Optional<Standalone<StringRef>>> getDatacenters( DatabaseConfiguration const& conf, bool checkStable = false ) {
std::set<Optional<Standalone<StringRef>>> result;
for( auto& it : id_worker )
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if( workerAvailable( it.second, checkStable ) && !conf.isExcludedServer( it.second.details.interf.address() ) )
result.insert(it.second.details.interf.locality.dcId());
Initial repository commit
2017-05-26 04:48:44 +08:00
return result;
}
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
void updateKnownIds(std::map< Optional<Standalone<StringRef>>, int>* id_used) {
(*id_used)[masterProcessId]++;
(*id_used)[clusterControllerProcessId]++;
if (db.serverInfo->get().distributor.present()) {
(*id_used)[db.serverInfo->get().distributor.get().locality.processId()]++;
}
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
if (db.serverInfo->get().ratekeeper.present()) {
(*id_used)[db.serverInfo->get().ratekeeper.get().locality.processId()]++;
}
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
}
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
RecruitRemoteFromConfigurationReply findRemoteWorkersForConfiguration( RecruitRemoteFromConfigurationRequest const& req ) {
RecruitRemoteFromConfigurationReply result;
Initial repository commit
2017-05-26 04:48:44 +08:00
std::map< Optional<Standalone<StringRef>>, int> id_used;
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
updateKnownIds(&id_used);
Initial repository commit
2017-05-26 04:48:44 +08:00
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
std::set<Optional<Key>> remoteDC;
remoteDC.insert(req.dcId);
added usable_regions to specify if we will replicate into a remote region remote replication defaults to the primary replication removed remote_logs, because they should be specified as an override in the regions object
2018-06-18 10:31:15 +08:00
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
auto remoteLogs = getWorkersForTlogs( req.configuration, req.configuration.getRemoteTLogReplicationFactor(), req.configuration.getDesiredRemoteLogs(), req.configuration.getRemoteTLogPolicy(), id_used, false, remoteDC, req.exclusionWorkerIds );
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
for(int i = 0; i < remoteLogs.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.remoteTLogs.push_back(remoteLogs[i].interf);
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
fix: we must commit to the number of log routers we are going to use when recruiting the primary, because it determines the number of log router tags that will be attached to mutations
2018-03-07 08:31:21 +08:00
auto logRouters = getWorkersForRoleInDatacenter( req.dcId, ProcessClass::LogRouter, req.logRouterCount, req.configuration, id_used );
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
for(int i = 0; i < logRouters.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.logRouters.push_back(logRouters[i].interf);
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
Swapped OkayFit and UnsetFit, because generally if machine classes are set on one machine they are set everywhere and it helps with wait_for_good_recruitment logic wait_for_good_recruitment now requires that you have the desired count of each roll remote recruitment is given a much longer wait_for_good_recruitment time interval, which does not start until enough remote machines have registered
2018-06-23 01:15:24 +08:00
if(!remoteStartTime.present()) {
keep track of the last time a process became available to set a better starting value for remoteStartTime
2018-11-05 06:33:03 +08:00
double maxAvailableTime = 0;
for(auto& it : result.remoteTLogs) {
maxAvailableTime = std::max(maxAvailableTime, id_worker[it.locality.processId()].lastAvailableTime);
}
for(auto& it : result.logRouters) {
maxAvailableTime = std::max(maxAvailableTime, id_worker[it.locality.processId()].lastAvailableTime);
}
remoteStartTime = maxAvailableTime;
Swapped OkayFit and UnsetFit, because generally if machine classes are set on one machine they are set everywhere and it helps with wait_for_good_recruitment logic wait_for_good_recruitment now requires that you have the desired count of each roll remote recruitment is given a much longer wait_for_good_recruitment time interval, which does not start until enough remote machines have registered
2018-06-23 01:15:24 +08:00
}
if( now() - remoteStartTime.get() < SERVER_KNOBS->WAIT_FOR_GOOD_REMOTE_RECRUITMENT_DELAY &&
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
( ( RoleFitness(SERVER_KNOBS->EXPECTED_TLOG_FITNESS, req.configuration.getDesiredRemoteLogs(), ProcessClass::TLog).betterCount(RoleFitness(remoteLogs, ProcessClass::TLog)) ) ||
( RoleFitness(SERVER_KNOBS->EXPECTED_LOG_ROUTER_FITNESS, req.logRouterCount, ProcessClass::LogRouter).betterCount(RoleFitness(logRouters, ProcessClass::LogRouter)) ) ) ) {
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
throw operation_failed();
}
Initial repository commit
2017-05-26 04:48:44 +08:00
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
return result;
}
Initial repository commit
2017-05-26 04:48:44 +08:00
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
ErrorOr<RecruitFromConfigurationReply> findWorkersForConfiguration( RecruitFromConfigurationRequest const& req, Optional<Key> dcId ) {
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
RecruitFromConfigurationReply result;
std::map< Optional<Standalone<StringRef>>, int> id_used;
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
updateKnownIds(&id_used);
Merge branch 'master' into feature-remote-logs # Conflicts: # fdbrpc/Locality.cpp # fdbrpc/Locality.h # fdbserver/ClusterController.actor.cpp # fdbserver/ClusterRecruitmentInterface.h # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/fdbserver.vcxproj.filters # fdbserver/masterserver.actor.cpp # fdbserver/worker.actor.cpp # flow/error_definitions.h
2017-10-06 08:09:44 +08:00
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
ASSERT(dcId.present());
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
std::set<Optional<Key>> primaryDC;
primaryDC.insert(dcId);
result.dcId = dcId;
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
RegionInfo region;
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
RegionInfo remoteRegion;
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
for(auto& r : req.configuration.regions) {
first working version of non-copying recovery working with fearless configurations
2018-04-09 12:24:05 +08:00
if(r.dcId == dcId.get()) {
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
region = r;
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
}
else {
remoteRegion = r;
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
}
}
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
Merge branch 'master' into feature-remote-logs # Conflicts: # fdbrpc/Locality.cpp # fdbrpc/Locality.h # fdbserver/ClusterController.actor.cpp # fdbserver/ClusterRecruitmentInterface.h # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/fdbserver.vcxproj.filters # fdbserver/masterserver.actor.cpp # fdbserver/worker.actor.cpp # flow/error_definitions.h
2017-10-06 08:09:44 +08:00
if(req.recruitSeedServers) {
auto primaryStorageServers = getWorkersForSeedServers( req.configuration, req.configuration.storagePolicy, dcId );
first working version of non-copying recovery working with fearless configurations
2018-04-09 12:24:05 +08:00
for(int i = 0; i < primaryStorageServers.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.storageServers.push_back(primaryStorageServers[i].interf);
Merge branch 'master' into feature-remote-logs # Conflicts: # fdbrpc/Locality.cpp # fdbrpc/Locality.h # fdbserver/ClusterController.actor.cpp # fdbserver/ClusterRecruitmentInterface.h # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/fdbserver.vcxproj.filters # fdbserver/masterserver.actor.cpp # fdbserver/worker.actor.cpp # flow/error_definitions.h
2017-10-06 08:09:44 +08:00
}
}
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
Merge branch 'master' into feature-remote-logs # Conflicts: # fdbrpc/Locality.cpp # fdbrpc/Locality.h # fdbserver/ClusterController.actor.cpp # fdbserver/ClusterRecruitmentInterface.h # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/fdbserver.vcxproj.filters # fdbserver/masterserver.actor.cpp # fdbserver/worker.actor.cpp # flow/error_definitions.h
2017-10-06 08:09:44 +08:00
auto tlogs = getWorkersForTlogs( req.configuration, req.configuration.tLogReplicationFactor, req.configuration.getDesiredLogs(), req.configuration.tLogPolicy, id_used, false, primaryDC );
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
for(int i = 0; i < tlogs.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.tLogs.push_back(tlogs[i].interf);
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
std::vector<WorkerDetails> satelliteLogs;
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
if(region.satelliteTLogReplicationFactor > 0) {
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
satelliteLogs = getWorkersForSatelliteLogs( req.configuration, region, remoteRegion, id_used, result.satelliteFallback );
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
for(int i = 0; i < satelliteLogs.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.satelliteTLogs.push_back(satelliteLogs[i].interf);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
auto first_resolver = getWorkerForRoleInDatacenter( dcId, ProcessClass::Resolver, ProcessClass::ExcludeFit, req.configuration, id_used );
auto first_proxy = getWorkerForRoleInDatacenter( dcId, ProcessClass::Proxy, ProcessClass::ExcludeFit, req.configuration, id_used );
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
auto proxies = getWorkersForRoleInDatacenter( dcId, ProcessClass::Proxy, req.configuration.getDesiredProxies()-1, req.configuration, id_used, first_proxy );
auto resolvers = getWorkersForRoleInDatacenter( dcId, ProcessClass::Resolver, req.configuration.getDesiredResolvers()-1, req.configuration, id_used, first_resolver );
proxies.push_back(first_proxy.worker);
resolvers.push_back(first_resolver.worker);
Initial repository commit
2017-05-26 04:48:44 +08:00
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
for(int i = 0; i < resolvers.size(); i++)
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.resolvers.push_back(resolvers[i].interf);
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
for(int i = 0; i < proxies.size(); i++)
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.proxies.push_back(proxies[i].interf);
Initial repository commit
2017-05-26 04:48:44 +08:00
first version of non-copying recovery. Upgrades are broken, and it has not been tested using fearless configurations yet
2018-03-30 06:12:38 +08:00
auto oldLogRouters = getWorkersForRoleInDatacenter( dcId, ProcessClass::LogRouter, req.maxOldLogRouters, req.configuration, id_used );
for(int i = 0; i < oldLogRouters.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.oldLogRouters.push_back(oldLogRouters[i].interf);
first version of non-copying recovery. Upgrades are broken, and it has not been tested using fearless configurations yet
2018-03-30 06:12:38 +08:00
}
fix: we must commit to the number of log routers we are going to use when recruiting the primary, because it determines the number of log router tags that will be attached to mutations
2018-03-07 08:31:21 +08:00
Initial repository commit
2017-05-26 04:48:44 +08:00
if( now() - startTime < SERVER_KNOBS->WAIT_FOR_GOOD_RECRUITMENT_DELAY &&
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
( RoleFitness(SERVER_KNOBS->EXPECTED_TLOG_FITNESS, req.configuration.getDesiredLogs(), ProcessClass::TLog).betterCount(RoleFitness(tlogs, ProcessClass::TLog)) ||
( region.satelliteTLogReplicationFactor > 0 && RoleFitness(SERVER_KNOBS->EXPECTED_TLOG_FITNESS, req.configuration.getDesiredSatelliteLogs(dcId), ProcessClass::TLog).betterCount(RoleFitness(satelliteLogs, ProcessClass::TLog)) ) ||
RoleFitness(SERVER_KNOBS->EXPECTED_PROXY_FITNESS, req.configuration.getDesiredProxies(), ProcessClass::Proxy).betterCount(RoleFitness(proxies, ProcessClass::Proxy)) ||
RoleFitness(SERVER_KNOBS->EXPECTED_RESOLVER_FITNESS, req.configuration.getDesiredResolvers(), ProcessClass::Resolver).betterCount(RoleFitness(resolvers, ProcessClass::Resolver)) ) ) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
return operation_failed();
Initial repository commit
2017-05-26 04:48:44 +08:00
}
return result;
}
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
RecruitFromConfigurationReply findWorkersForConfiguration( RecruitFromConfigurationRequest const& req ) {
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
if(req.configuration.regions.size() > 1) {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
std::vector<RegionInfo> regions = req.configuration.regions;
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
if(regions[0].priority == regions[1].priority && regions[1].dcId == clusterControllerDcId.get()) {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
std::swap(regions[0], regions[1]);
}
fix: treat the local region as best when version difference is too large re-check requests when the version difference becomes small
2018-07-07 05:44:11 +08:00
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
if(regions[1].dcId == clusterControllerDcId.get() && regions[1].priority >= 0 && (!versionDifferenceUpdated || datacenterVersionDifference >= SERVER_KNOBS->MAX_VERSION_DIFFERENCE)) {
fix: treat the local region as best when version difference is too large re-check requests when the version difference becomes small
2018-07-07 05:44:11 +08:00
std::swap(regions[0], regions[1]);
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
bool setPrimaryDesired = false;
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
try {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
auto reply = findWorkersForConfiguration(req, regions[0].dcId);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
setPrimaryDesired = true;
vector<Optional<Key>> dcPriority;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
dcPriority.push_back(regions[0].dcId);
dcPriority.push_back(regions[1].dcId);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
desiredDcIds.set(dcPriority);
if(reply.isError()) {
throw reply.getError();
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
} else if(regions[0].dcId == clusterControllerDcId.get()) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
return reply.get();
}
throw no_more_servers();
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
} catch( Error& e ) {
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
if (now() - startTime < SERVER_KNOBS->WAIT_FOR_GOOD_REMOTE_RECRUITMENT_DELAY && regions[1].dcId != clusterControllerDcId.get()) {
fix: the cluster controller needs to continue to retry recruitment until after wait_for_good_remote_recruitment_delay
2018-06-26 09:20:16 +08:00
throw operation_failed();
}
if (e.code() != error_code_no_more_servers || regions[1].priority < 0) {
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
throw;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
TraceEvent(SevWarn, "AttemptingRecruitmentInRemoteDC", id).error(e);
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
auto reply = findWorkersForConfiguration(req, regions[1].dcId);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
if(!setPrimaryDesired) {
vector<Optional<Key>> dcPriority;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
dcPriority.push_back(regions[1].dcId);
dcPriority.push_back(regions[0].dcId);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
desiredDcIds.set(dcPriority);
}
if(reply.isError()) {
throw reply.getError();
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
} else if (regions[1].dcId == clusterControllerDcId.get()) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
return reply.get();
}
throw;
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
added support for single region configurations
2018-03-16 01:59:30 +08:00
} else if(req.configuration.regions.size() == 1) {
vector<Optional<Key>> dcPriority;
dcPriority.push_back(req.configuration.regions[0].dcId);
desiredDcIds.set(dcPriority);
auto reply = findWorkersForConfiguration(req, req.configuration.regions[0].dcId);
if(reply.isError()) {
throw reply.getError();
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
} else if (req.configuration.regions[0].dcId == clusterControllerDcId.get()) {
added support for single region configurations
2018-03-16 01:59:30 +08:00
return reply.get();
}
throw no_more_servers();
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
} else {
RecruitFromConfigurationReply result;
std::map< Optional<Standalone<StringRef>>, int> id_used;
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
updateKnownIds(&id_used);
Merge branch 'master' into feature-remote-logs # Conflicts: # fdbrpc/Locality.cpp # fdbrpc/Locality.h # fdbserver/ClusterController.actor.cpp # fdbserver/ClusterRecruitmentInterface.h # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/fdbserver.vcxproj.filters # fdbserver/masterserver.actor.cpp # fdbserver/worker.actor.cpp # flow/error_definitions.h
2017-10-06 08:09:44 +08:00
auto tlogs = getWorkersForTlogs( req.configuration, req.configuration.tLogReplicationFactor, req.configuration.getDesiredLogs(), req.configuration.tLogPolicy, id_used );
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
for(int i = 0; i < tlogs.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.tLogs.push_back(tlogs[i].interf);
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
Merge branch 'master' into feature-remote-logs # Conflicts: # fdbrpc/Locality.cpp # fdbrpc/Locality.h # fdbserver/ClusterController.actor.cpp # fdbserver/ClusterRecruitmentInterface.h # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/fdbserver.vcxproj.filters # fdbserver/masterserver.actor.cpp # fdbserver/worker.actor.cpp # flow/error_definitions.h
2017-10-06 08:09:44 +08:00
if(req.recruitSeedServers) {
auto primaryStorageServers = getWorkersForSeedServers( req.configuration, req.configuration.storagePolicy );
for(int i = 0; i < primaryStorageServers.size(); i++)
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.storageServers.push_back(primaryStorageServers[i].interf);
Merge branch 'master' into feature-remote-logs # Conflicts: # fdbrpc/Locality.cpp # fdbrpc/Locality.h # fdbserver/ClusterController.actor.cpp # fdbserver/ClusterRecruitmentInterface.h # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/fdbserver.vcxproj.filters # fdbserver/masterserver.actor.cpp # fdbserver/worker.actor.cpp # flow/error_definitions.h
2017-10-06 08:09:44 +08:00
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
auto datacenters = getDatacenters( req.configuration );
RoleFitness bestFitness;
int numEquivalent = 1;
Optional<Key> bestDC;
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
for(auto dcId : datacenters ) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
try {
//SOMEDAY: recruitment in other DCs besides the clusterControllerDcID will not account for the processes used by the master and cluster controller properly.
auto used = id_used;
auto first_resolver = getWorkerForRoleInDatacenter( dcId, ProcessClass::Resolver, ProcessClass::ExcludeFit, req.configuration, used );
auto first_proxy = getWorkerForRoleInDatacenter( dcId, ProcessClass::Proxy, ProcessClass::ExcludeFit, req.configuration, used );
auto proxies = getWorkersForRoleInDatacenter( dcId, ProcessClass::Proxy, req.configuration.getDesiredProxies()-1, req.configuration, used, first_proxy );
auto resolvers = getWorkersForRoleInDatacenter( dcId, ProcessClass::Resolver, req.configuration.getDesiredResolvers()-1, req.configuration, used, first_resolver );
proxies.push_back(first_proxy.worker);
resolvers.push_back(first_resolver.worker);
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
auto fitness = RoleFitness( RoleFitness(proxies, ProcessClass::Proxy), RoleFitness(resolvers, ProcessClass::Resolver), ProcessClass::NoRole );
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
if(dcId == clusterControllerDcId) {
bestFitness = fitness;
bestDC = dcId;
for(int i = 0; i < resolvers.size(); i++)
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.resolvers.push_back(resolvers[i].interf);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
for(int i = 0; i < proxies.size(); i++)
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.proxies.push_back(proxies[i].interf);
first version of non-copying recovery. Upgrades are broken, and it has not been tested using fearless configurations yet
2018-03-30 06:12:38 +08:00
auto oldLogRouters = getWorkersForRoleInDatacenter( dcId, ProcessClass::LogRouter, req.maxOldLogRouters, req.configuration, used );
for(int i = 0; i < oldLogRouters.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.oldLogRouters.push_back(oldLogRouters[i].interf);
first version of non-copying recovery. Upgrades are broken, and it has not been tested using fearless configurations yet
2018-03-30 06:12:38 +08:00
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
break;
} else {
if(fitness < bestFitness) {
bestFitness = fitness;
numEquivalent = 1;
bestDC = dcId;
} else if( fitness == bestFitness && g_random->random01() < 1.0/++numEquivalent ) {
bestDC = dcId;
}
}
} catch( Error &e ) {
if(e.code() != error_code_no_more_servers) {
throw;
}
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
if(bestDC != clusterControllerDcId) {
vector<Optional<Key>> dcPriority;
dcPriority.push_back(bestDC);
desiredDcIds.set(dcPriority);
throw no_more_servers();
}
//If this cluster controller dies, do not prioritize recruiting the next one in the same DC
desiredDcIds.set(vector<Optional<Key>>());
Attempt to normalize trace events: * Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check. * Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase. * Use seconds instead of milliseconds in details. Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed. This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-09 02:11:08 +08:00
TraceEvent("FindWorkersForConfig").detail("Replication", req.configuration.tLogReplicationFactor)
.detail("DesiredLogs", req.configuration.getDesiredLogs()).detail("ActualLogs", result.tLogs.size())
.detail("DesiredProxies", req.configuration.getDesiredProxies()).detail("ActualProxies", result.proxies.size())
.detail("DesiredResolvers", req.configuration.getDesiredResolvers()).detail("ActualResolvers", result.resolvers.size());
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
if( now() - startTime < SERVER_KNOBS->WAIT_FOR_GOOD_RECRUITMENT_DELAY &&
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
( RoleFitness(SERVER_KNOBS->EXPECTED_TLOG_FITNESS, req.configuration.getDesiredLogs(), ProcessClass::TLog).betterCount(RoleFitness(tlogs, ProcessClass::TLog)) ||
RoleFitness(std::min(SERVER_KNOBS->EXPECTED_PROXY_FITNESS, SERVER_KNOBS->EXPECTED_RESOLVER_FITNESS), std::max(SERVER_KNOBS->EXPECTED_PROXY_FITNESS, SERVER_KNOBS->EXPECTED_RESOLVER_FITNESS), req.configuration.getDesiredProxies()+req.configuration.getDesiredResolvers(), ProcessClass::NoRole).betterCount(bestFitness) ) ) {
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
throw operation_failed();
}
return result;
}
}
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
void checkRegions(const std::vector<RegionInfo>& regions) {
if(desiredDcIds.get().present() && desiredDcIds.get().get().size() == 2 && desiredDcIds.get().get()[0].get() == regions[0].dcId && desiredDcIds.get().get()[1].get() == regions[1].dcId) {
return;
}
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
try {
std::map< Optional<Standalone<StringRef>>, int> id_used;
getWorkerForRoleInDatacenter(regions[0].dcId, ProcessClass::ClusterController, ProcessClass::ExcludeFit, db.config, id_used, true);
getWorkerForRoleInDatacenter(regions[0].dcId, ProcessClass::Master, ProcessClass::ExcludeFit, db.config, id_used, true);
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
std::set<Optional<Key>> primaryDC;
primaryDC.insert(regions[0].dcId);
fix: desiredTLogCount was used instead of getDesiredLogs(), which caused problems with recruitment when desiredTLogCount was -1. canKillProcess logic was wrong. We still need to configure usable_regions because if datacenterVersionDifference is too large we cannot complete data movement.
2018-07-05 04:22:32 +08:00
getWorkersForTlogs(db.config, db.config.tLogReplicationFactor, db.config.getDesiredLogs(), db.config.tLogPolicy, id_used, true, primaryDC);
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
if(regions[0].satelliteTLogReplicationFactor > 0) {
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
bool satelliteFallback = false;
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
getWorkersForSatelliteLogs(db.config, regions[0], regions[1], id_used, satelliteFallback, true);
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
getWorkerForRoleInDatacenter( regions[0].dcId, ProcessClass::Resolver, ProcessClass::ExcludeFit, db.config, id_used, true );
getWorkerForRoleInDatacenter( regions[0].dcId, ProcessClass::Proxy, ProcessClass::ExcludeFit, db.config, id_used, true );
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
vector<Optional<Key>> dcPriority;
dcPriority.push_back(regions[0].dcId);
dcPriority.push_back(regions[1].dcId);
desiredDcIds.set(dcPriority);
} catch( Error &e ) {
if(e.code() != error_code_no_more_servers) {
throw;
}
}
}
void checkRecoveryStalled() {
fix: it is not safe to drop logs supporting the current primary datacenter, because configuring usable_regions down will drop the storage servers in the remote region, leaving you will no remaining logs
2018-07-15 07:26:45 +08:00
if( (db.serverInfo->get().recoveryState == RecoveryState::RECRUITING || db.serverInfo->get().recoveryState == RecoveryState::ACCEPTING_COMMITS || db.serverInfo->get().recoveryState == RecoveryState::ALL_LOGS_RECRUITED) && db.recoveryStalled ) {
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
if (db.config.regions.size() > 1) {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
auto regions = db.config.regions;
if(clusterControllerDcId.get() == regions[0].dcId) {
std::swap(regions[0], regions[1]);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
ASSERT(clusterControllerDcId.get() == regions[1].dcId);
checkRegions(regions);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
}
}
//FIXME: determine when to fail the cluster controller when a primaryDC has not been set
Initial repository commit
2017-05-26 04:48:44 +08:00
bool betterMasterExists() {
ServerDBInfo dbi = db.serverInfo->get();
fix: kill all tlogs in one region prevents the remote logs from recovering in that region, do not allow that to prevent us from configuring usable_regions=1. added more recovery states.
2018-07-05 15:08:51 +08:00
if(dbi.recoveryState < RecoveryState::ACCEPTING_COMMITS) {
Initial repository commit
2017-05-26 04:48:44 +08:00
return false;
}
Merge release-6.0 into master
2018-09-22 07:14:39 +08:00
// Do not trigger better master exists if the cluster controller is excluded, since the master will change anyways once the cluster controller is moved
if(id_worker[clusterControllerProcessId].priorityInfo.isExcluded) {
return false;
}
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
if (db.config.regions.size() > 1 && db.config.regions[0].priority > db.config.regions[1].priority &&
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
db.config.regions[0].dcId != clusterControllerDcId.get() && versionDifferenceUpdated && datacenterVersionDifference < SERVER_KNOBS->MAX_VERSION_DIFFERENCE) {
checkRegions(db.config.regions);
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
Fix double recoveries
2017-11-17 08:58:55 +08:00
// Get master process
Initial repository commit
2017-05-26 04:48:44 +08:00
auto masterWorker = id_worker.find(dbi.master.locality.processId());
Fix double recoveries
2017-11-17 08:58:55 +08:00
if(masterWorker == id_worker.end()) {
Initial repository commit
2017-05-26 04:48:44 +08:00
return false;
Enable checkUsingDesiredClasses() in consistency check
2017-10-25 03:58:54 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
Fix double recoveries
2017-11-17 08:58:55 +08:00
// Get tlog processes
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
std::vector<WorkerDetails> tlogs;
std::vector<WorkerDetails> remote_tlogs;
std::vector<WorkerDetails> satellite_tlogs;
std::vector<WorkerDetails> log_routers;
fix: we may not be able to recruit enough log routers, in this case put multiple log routers on the same worker, but also properly rank this configuration lower in better master exists
2018-04-27 13:18:07 +08:00
std::set<NetworkAddress> logRouterAddresses;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
for( auto& logSet : dbi.logSystemConfig.tLogs ) {
for( auto& it : logSet.tLogs ) {
auto tlogWorker = id_worker.find(it.interf().locality.processId());
if ( tlogWorker == id_worker.end() )
return false;
if ( tlogWorker->second.priorityInfo.isExcluded )
return true;
removed hasBestPolicy
2018-06-16 03:36:19 +08:00
if(logSet.isLocal && logSet.locality == tagLocalitySatellite) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
satellite_tlogs.push_back(tlogWorker->second.details);
removed hasBestPolicy
2018-06-16 03:36:19 +08:00
}
else if(logSet.isLocal) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
tlogs.push_back(tlogWorker->second.details);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
} else {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
remote_tlogs.push_back(tlogWorker->second.details);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
}
for( auto& it : logSet.logRouters ) {
auto tlogWorker = id_worker.find(it.interf().locality.processId());
if ( tlogWorker == id_worker.end() )
return false;
if ( tlogWorker->second.priorityInfo.isExcluded )
return true;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if( !logRouterAddresses.count( tlogWorker->second.details.interf.address() ) ) {
logRouterAddresses.insert( tlogWorker->second.details.interf.address() );
log_routers.push_back(tlogWorker->second.details);
fix: we may not be able to recruit enough log routers, in this case put multiple log routers on the same worker, but also properly rank this configuration lower in better master exists
2018-04-27 13:18:07 +08:00
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Fix double recoveries
2017-11-17 08:58:55 +08:00
// Get proxy classes
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
std::vector<WorkerDetails> proxyClasses;
Initial repository commit
2017-05-26 04:48:44 +08:00
for(auto& it : dbi.client.proxies ) {
auto proxyWorker = id_worker.find(it.locality.processId());
if ( proxyWorker == id_worker.end() )
return false;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
if ( proxyWorker->second.priorityInfo.isExcluded )
Fix double recoveries
2017-11-17 08:58:55 +08:00
return true;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
proxyClasses.push_back(proxyWorker->second.details);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Fix double recoveries
2017-11-17 08:58:55 +08:00
// Get resolver classes
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
std::vector<WorkerDetails> resolverClasses;
Initial repository commit
2017-05-26 04:48:44 +08:00
for(auto& it : dbi.resolvers ) {
auto resolverWorker = id_worker.find(it.locality.processId());
if ( resolverWorker == id_worker.end() )
return false;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
if ( resolverWorker->second.priorityInfo.isExcluded )
Fix double recoveries
2017-11-17 08:58:55 +08:00
return true;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
resolverClasses.push_back(resolverWorker->second.details);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Fix double recoveries
2017-11-17 08:58:55 +08:00
// Check master fitness. Don't return false if master is excluded in case all the processes are excluded, we still need master for recovery.
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
ProcessClass::Fitness oldMasterFit = masterWorker->second.details.processClass.machineClassFitness( ProcessClass::Master );
Fix double recoveries
2017-11-17 08:58:55 +08:00
if(db.config.isExcludedServer(dbi.master.address())) {
oldMasterFit = std::max(oldMasterFit, ProcessClass::ExcludeFit);
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
std::map< Optional<Standalone<StringRef>>, int> id_used;
id_used[clusterControllerProcessId]++;
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
if (db.serverInfo->get().distributor.present()) {
id_used[db.serverInfo->get().distributor.get().locality.processId()]++;
Fix segfault error The usedIds is updated by master registration request, which populates the usedIds map. However, this request may contain processes that cluster controller is not aware, i.e., not in id_worker map. This is ok until I added tracing the usedIds, which silently insert an empty entry into id_worker map for the unknown process. This new entry can cause crashing failure when trying to access its LocalityData. Remove AsyncTrigger for usedIds, and change to serverInfo->onChange. Use const & to avoid unnecessary copies in WorkerInterface's LocalityData and getExtraTLogEligibleMachines().
2019-02-01 14:23:49 +08:00
}
Fix a segfault bug due to uncopied ratekeeper interface
2019-02-19 06:57:21 +08:00
if (db.serverInfo->get().ratekeeper.present()) {
id_used[db.serverInfo->get().ratekeeper.get().locality.processId()]++;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
WorkerFitnessInfo mworker = getWorkerForRoleInDatacenter(clusterControllerDcId, ProcessClass::Master, ProcessClass::NeverAssign, db.config, id_used, true);
Fix double recoveries
2017-11-17 08:58:55 +08:00
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
if ( oldMasterFit < mworker.fitness )
Fix double recoveries
2017-11-17 08:58:55 +08:00
return false;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if ( oldMasterFit > mworker.fitness || ( dbi.master.locality.processId() == clusterControllerProcessId && mworker.worker.interf.locality.processId() != clusterControllerProcessId ) )
Fix double recoveries
2017-11-17 08:58:55 +08:00
return true;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
std::set<Optional<Key>> primaryDC;
std::set<Optional<Key>> remoteDC;
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
RegionInfo region;
ensure that the processId exists in id_worker before accessing it
2019-03-23 09:54:39 +08:00
RegionInfo remoteRegion;
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
if (db.config.regions.size()) {
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
primaryDC.insert(clusterControllerDcId);
for(auto& r : db.config.regions) {
if(r.dcId != clusterControllerDcId.get()) {
ASSERT(remoteDC.empty());
remoteDC.insert(r.dcId);
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
remoteRegion = r;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
} else {
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
ASSERT(region.dcId == StringRef());
region = r;
}
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
Fix double recoveries
2017-11-17 08:58:55 +08:00
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
// Check tLog fitness
RoleFitness oldTLogFit(tlogs, ProcessClass::TLog);
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
auto newTLogs = getWorkersForTlogs(db.config, db.config.tLogReplicationFactor, db.config.getDesiredLogs(), db.config.tLogPolicy, id_used, true, primaryDC);
RoleFitness newTLogFit(newTLogs, ProcessClass::TLog);
Fix double recoveries
2017-11-17 08:58:55 +08:00
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
if(oldTLogFit < newTLogFit) return false;
Fix double recoveries
2017-11-17 08:58:55 +08:00
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
bool oldSatelliteFallback = false;
for(auto& logSet : dbi.logSystemConfig.tLogs) {
if(logSet.isLocal && logSet.locality == tagLocalitySatellite) {
oldSatelliteFallback = logSet.tLogPolicy->info() != region.satelliteTLogPolicy->info();
ASSERT(!oldSatelliteFallback || logSet.tLogPolicy->info() == region.satelliteTLogPolicyFallback->info());
break;
}
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
RoleFitness oldSatelliteTLogFit(satellite_tlogs, ProcessClass::TLog);
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
bool newSatelliteFallback = false;
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
auto newSatelliteTLogs = region.satelliteTLogReplicationFactor > 0 ? getWorkersForSatelliteLogs(db.config, region, remoteRegion, id_used, newSatelliteFallback, true) : satellite_tlogs;
RoleFitness newSatelliteTLogFit(newSatelliteTLogs, ProcessClass::TLog);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
if(oldSatelliteTLogFit < newSatelliteTLogFit)
return false;
if(!oldSatelliteFallback && newSatelliteFallback)
return false;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
RoleFitness oldRemoteTLogFit(remote_tlogs, ProcessClass::TLog);
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
std::vector<UID> exclusionWorkerIds;
auto fn = [](const WorkerDetails &in) { return in.interf.id(); };
std::transform(newTLogs.begin(), newTLogs.end(), std::back_inserter(exclusionWorkerIds), fn);
std::transform(newSatelliteTLogs.begin(), newSatelliteTLogs.end(), std::back_inserter(exclusionWorkerIds), fn);
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
RoleFitness newRemoteTLogFit(
(db.config.usableRegions > 1 && dbi.recoveryState == RecoveryState::FULLY_RECOVERED) ?
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
getWorkersForTlogs(db.config, db.config.getRemoteTLogReplicationFactor(), db.config.getDesiredRemoteLogs(), db.config.getRemoteTLogPolicy(), id_used, true, remoteDC, exclusionWorkerIds)
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
: remote_tlogs, ProcessClass::TLog);
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
if(oldRemoteTLogFit < newRemoteTLogFit) return false;
re-added the ability to configure the number of log routers. Many log routers are needed to get a sufficient number of sockets involved in copying data across the WAN
2018-06-22 15:04:00 +08:00
int oldRouterCount = oldTLogFit.count * std::max<int>(1, db.config.desiredLogRouterCount / std::max(1,oldTLogFit.count));
int newRouterCount = newTLogFit.count * std::max<int>(1, db.config.desiredLogRouterCount / std::max(1,newTLogFit.count));
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
RoleFitness oldLogRoutersFit(log_routers, ProcessClass::LogRouter);
fix: kill all tlogs in one region prevents the remote logs from recovering in that region, do not allow that to prevent us from configuring usable_regions=1. added more recovery states.
2018-07-05 15:08:51 +08:00
RoleFitness newLogRoutersFit((db.config.usableRegions > 1 && dbi.recoveryState == RecoveryState::FULLY_RECOVERED) ? getWorkersForRoleInDatacenter( *remoteDC.begin(), ProcessClass::LogRouter, newRouterCount, db.config, id_used, Optional<WorkerFitnessInfo>(), true ) : log_routers, ProcessClass::LogRouter);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
re-added the ability to configure the number of log routers. Many log routers are needed to get a sufficient number of sockets involved in copying data across the WAN
2018-06-22 15:04:00 +08:00
if(oldLogRoutersFit.count < oldRouterCount) {
fix: we may not be able to recruit enough log routers, in this case put multiple log routers on the same worker, but also properly rank this configuration lower in better master exists
2018-04-27 13:18:07 +08:00
oldLogRoutersFit.worstFit = ProcessClass::NeverAssign;
}
re-added the ability to configure the number of log routers. Many log routers are needed to get a sufficient number of sockets involved in copying data across the WAN
2018-06-22 15:04:00 +08:00
if(newLogRoutersFit.count < newRouterCount) {
fix: we may not be able to recruit enough log routers, in this case put multiple log routers on the same worker, but also properly rank this configuration lower in better master exists
2018-04-27 13:18:07 +08:00
newLogRoutersFit.worstFit = ProcessClass::NeverAssign;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
if(oldLogRoutersFit < newLogRoutersFit) return false;
Fix double recoveries
2017-11-17 08:58:55 +08:00
// Check proxy/resolver fitness
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
RoleFitness oldInFit(RoleFitness(proxyClasses, ProcessClass::Proxy), RoleFitness(resolverClasses, ProcessClass::Resolver), ProcessClass::NoRole);
Initial repository commit
2017-05-26 04:48:44 +08:00
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
auto first_resolver = getWorkerForRoleInDatacenter( clusterControllerDcId, ProcessClass::Resolver, ProcessClass::ExcludeFit, db.config, id_used, true );
auto first_proxy = getWorkerForRoleInDatacenter( clusterControllerDcId, ProcessClass::Proxy, ProcessClass::ExcludeFit, db.config, id_used, true );
Initial repository commit
2017-05-26 04:48:44 +08:00
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
auto proxies = getWorkersForRoleInDatacenter( clusterControllerDcId, ProcessClass::Proxy, db.config.getDesiredProxies()-1, db.config, id_used, first_proxy, true );
auto resolvers = getWorkersForRoleInDatacenter( clusterControllerDcId, ProcessClass::Resolver, db.config.getDesiredResolvers()-1, db.config, id_used, first_resolver, true );
proxies.push_back(first_proxy.worker);
resolvers.push_back(first_resolver.worker);
Initial repository commit
2017-05-26 04:48:44 +08:00
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
RoleFitness newInFit(RoleFitness(proxies, ProcessClass::Proxy), RoleFitness(resolvers, ProcessClass::Resolver), ProcessClass::NoRole);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
if(oldInFit.betterFitness(newInFit)) return false;
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
if(oldTLogFit > newTLogFit || oldInFit > newInFit || (oldSatelliteFallback && !newSatelliteFallback) || oldSatelliteTLogFit > newSatelliteTLogFit || oldRemoteTLogFit > newRemoteTLogFit || oldLogRoutersFit > newLogRoutersFit) {
Attempt to normalize trace events: * Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check. * Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase. * Use seconds instead of milliseconds in details. Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed. This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-09 02:11:08 +08:00
TraceEvent("BetterMasterExists", id).detail("OldMasterFit", oldMasterFit).detail("NewMasterFit", mworker.fitness)
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
.detail("OldTLogFit", oldTLogFit.toString()).detail("NewTLogFit", newTLogFit.toString())
.detail("OldInFit", oldInFit.toString()).detail("NewInFit", newInFit.toString())
.detail("OldSatelliteFit", oldSatelliteTLogFit.toString()).detail("NewSatelliteFit", newSatelliteTLogFit.toString())
.detail("OldRemoteFit", oldRemoteTLogFit.toString()).detail("NewRemoteFit", newRemoteTLogFit.toString())
.detail("OldRouterFit", oldLogRoutersFit.toString()).detail("NewRouterFit", newLogRoutersFit.toString())
.detail("OldSatelliteFallback", oldSatelliteFallback).detail("NewSatelliteFallback", newSatelliteFallback);
Initial repository commit
2017-05-26 04:48:44 +08:00
return true;
}
Fix double recoveries
2017-11-17 08:58:55 +08:00
Initial repository commit
2017-05-26 04:48:44 +08:00
return false;
}
Recruit a new ratekeeper before halting the old
2019-03-20 06:21:46 +08:00
bool isProxyOrResolver(Optional<Key> processId) {
ASSERT(masterProcessId.present());
Update fdbserver/ClusterController.actor.cpp
2019-03-23 08:00:52 +08:00
if (processId == masterProcessId) return false;
Recruit a new ratekeeper before halting the old
2019-03-20 06:21:46 +08:00
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
auto& dbInfo = db.serverInfo->get();
for (const MasterProxyInterface& interf : dbInfo.client.proxies) {
if (interf.locality.processId() == processId) return true;
}
for (const ResolverInterface& interf: dbInfo.resolvers) {
if (interf.locality.processId() == processId) return true;
}
return false;
}
bool onMasterIsBetter(const WorkerDetails& worker, ProcessClass::ClusterRole role) {
ASSERT(masterProcessId.present());
Fix ratekeeper and data distributor recruiting bug Avoid multiple concurrent recuriting of ratekeepers with a recruiting flag. Fix endless recruiting when the chosen worker is a proxy or a resolver -- prefer master in this case.
2019-03-21 01:00:31 +08:00
const auto& pid = worker.interf.locality.processId();
if ((role != ProcessClass::DataDistributor && role != ProcessClass::RateKeeper) || pid == masterProcessId.get()) {
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
return false;
}
Update fdbserver/ClusterController.actor.cpp
2019-03-23 08:00:58 +08:00
return isProxyOrResolver(pid);
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
}
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
std::map< Optional<Standalone<StringRef>>, int> getUsedIds() {
Add used worker IDs to cluster controller This "usedIds" is updated when receiving a master registration message, so that when recruiting new data distributor, existing assignment is known.
2019-01-29 01:25:15 +08:00
std::map<Optional<Standalone<StringRef>>, int> idUsed;
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
updateKnownIds(&idUsed);
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
auto& dbInfo = db.serverInfo->get();
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
for (const auto& tlogset : dbInfo.logSystemConfig.tLogs) {
Various review comments fixes
2019-02-01 02:10:41 +08:00
for (const auto& tlog: tlogset.tLogs) {
if (tlog.present()) {
idUsed[tlog.interf().locality.processId()]++;
Add used worker IDs to cluster controller This "usedIds" is updated when receiving a master registration message, so that when recruiting new data distributor, existing assignment is known.
2019-01-29 01:25:15 +08:00
}
}
}
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
for (const MasterProxyInterface& interf : dbInfo.client.proxies) {
Add used worker IDs to cluster controller This "usedIds" is updated when receiving a master registration message, so that when recruiting new data distributor, existing assignment is known.
2019-01-29 01:25:15 +08:00
ASSERT(interf.locality.processId().present());
idUsed[interf.locality.processId()]++;
}
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
for (const ResolverInterface& interf: dbInfo.resolvers) {
Add used worker IDs to cluster controller This "usedIds" is updated when receiving a master registration message, so that when recruiting new data distributor, existing assignment is known.
2019-01-29 01:25:15 +08:00
ASSERT(interf.locality.processId().present());
idUsed[interf.locality.processId()]++;
}
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
return idUsed;
Add used worker IDs to cluster controller This "usedIds" is updated when receiving a master registration message, so that when recruiting new data distributor, existing assignment is known.
2019-01-29 01:25:15 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
std::map< Optional<Standalone<StringRef>>, WorkerInfo > id_worker;
Add used worker IDs to cluster controller This "usedIds" is updated when receiving a master registration message, so that when recruiting new data distributor, existing assignment is known.
2019-01-29 01:25:15 +08:00
std::map< Optional<Standalone<StringRef>>, ProcessClass > id_class; //contains the mapping from process id to process class from the database
Initial repository commit
2017-05-26 04:48:44 +08:00
Standalone<RangeResultRef> lastProcessClasses;
bool gotProcessClasses;
Add fullyRecoveredConfig for checking exclusions
2017-11-16 09:15:24 +08:00
bool gotFullyRecoveredConfig;
Initial repository commit
2017-05-26 04:48:44 +08:00
Optional<Standalone<StringRef>> masterProcessId;
Use processId instead of address for comparison
2017-10-26 02:35:29 +08:00
Optional<Standalone<StringRef>> clusterControllerProcessId;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
Optional<Standalone<StringRef>> clusterControllerDcId;
AsyncVar<Optional<vector<Optional<Key>>>> desiredDcIds; //desired DC priorities
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
AsyncVar<std::pair<bool,Optional<vector<Optional<Key>>>>> changingDcIds; //current DC priorities to change first, and whether that is the cluster controller
AsyncVar<std::pair<bool,Optional<vector<Optional<Key>>>>> changedDcIds; //current DC priorities to change second, and whether the cluster controller has been changed
Initial repository commit
2017-05-26 04:48:44 +08:00
UID id;
std::vector<RecruitFromConfigurationRequest> outstandingRecruitmentRequests;
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
std::vector<RecruitRemoteFromConfigurationRequest> outstandingRemoteRecruitmentRequests;
Initial repository commit
2017-05-26 04:48:44 +08:00
std::vector<std::pair<RecruitStorageRequest, double>> outstandingStorageRequests;
ActorCollection ac;
UpdateWorkerList updateWorkerList;
checking outstanding requests can be CPU intensive, so rate limit checking requests
2018-06-28 14:02:08 +08:00
Future<Void> outstandingRequestChecker;
Initial repository commit
2017-05-26 04:48:44 +08:00
DBInfo db;
Database cx;
double startTime;
Swapped OkayFit and UnsetFit, because generally if machine classes are set on one machine they are set everywhere and it helps with wait_for_good_recruitment logic wait_for_good_recruitment now requires that you have the desired count of each roll remote recruitment is given a much longer wait_for_good_recruitment time interval, which does not start until enough remote machines have registered
2018-06-23 01:15:24 +08:00
Optional<double> remoteStartTime;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
Version datacenterVersionDifference;
bool versionDifferenceUpdated;
Add DataDistributorInterface to ServerDBInfo Also change the Proxy and QuietDatabase to use the DataDistributorInterface.
2019-01-19 03:30:18 +08:00
PromiseStream<Future<Void>> addActor;
Fix the issue of slow dying Data Distributor Test with: -r simulation -f ./foundationdb/tests/slow/CommitBug.txt -s 67828576 -b on The test has the following event sequence: - Time 113.3s, CC noticed DD failure, cleard DD interface. - 1s later, DD rejoined and registered with CC. - Time 131.7s, DD actor cancelled. This old DD raced to register with CC and the failure monitor is not installed because monitorDataDistributor is stalled waiting for new DD. - Time 161.4s, new DD running. New DD recruting was delayed due to no servers in the period. Fix by disabling DD registration during the recruting process.
2019-03-18 11:55:59 +08:00
bool recruitingDistributor;
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
Optional<UID> recruitingRatekeeperID;
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
AsyncVar<bool> recruitRatekeeper;
Initial repository commit
2017-05-26 04:48:44 +08:00
fix: the cluster controller did not pass in its own locality when creating its database object, therefore it was not using locality aware load balancing
2018-09-29 03:12:06 +08:00
ClusterControllerData( ClusterControllerFullInterface const& ccInterface, LocalityData const& locality )
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
: clusterControllerProcessId(locality.processId()), clusterControllerDcId(locality.dcId()),
id(ccInterface.id()), ac(false), outstandingRequestChecker(Void()), gotProcessClasses(false),
gotFullyRecoveredConfig(false), startTime(now()), datacenterVersionDifference(0),
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
versionDifferenceUpdated(false), recruitingDistributor(false), recruitRatekeeper(false)
Initial repository commit
2017-05-26 04:48:44 +08:00
{
auto serverInfo = db.serverInfo->get();
serverInfo.id = g_random->randomUniqueID();
serverInfo.masterLifetime.ccID = id;
serverInfo.clusterInterface = ccInterface;
fix: the cluster controller did not pass in its own locality when creating its database object, therefore it was not using locality aware load balancing
2018-09-29 03:12:06 +08:00
serverInfo.myLocality = locality;
Initial repository commit
2017-05-26 04:48:44 +08:00
db.serverInfo->set( serverInfo );
cx = openDBOnServer(db.serverInfo, TaskDefaultEndpoint, true, true);
}
~ClusterControllerData() {
ac.clear(false);
id_worker.clear();
}
};
ACTOR Future<Void> clusterWatchDatabase( ClusterControllerData* cluster, ClusterControllerData::DBInfo* db )
{
state MasterInterface iMaster;
// SOMEDAY: If there is already a non-failed master referenced by zkMasterInfo, use that one until it fails
// When this someday is implemented, make sure forced failures still cause the master to be recruited again
loop {
TraceEvent("CCWDB", cluster->id);
try {
state double recoveryStart = now();
TraceEvent("CCWDB", cluster->id).detail("Recruiting", "Master");
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
//We must recruit the master in the same data center as the cluster controller.
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
//This should always be possible, because we can recruit the master on the same process as the cluster controller.
std::map< Optional<Standalone<StringRef>>, int> id_used;
id_used[cluster->clusterControllerProcessId]++;
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
if (cluster->db.serverInfo->get().distributor.present()) {
id_used[cluster->db.serverInfo->get().distributor.get().locality.processId()]++;
Fix segfault error The usedIds is updated by master registration request, which populates the usedIds map. However, this request may contain processes that cluster controller is not aware, i.e., not in id_worker map. This is ok until I added tracing the usedIds, which silently insert an empty entry into id_worker map for the unknown process. This new entry can cause crashing failure when trying to access its LocalityData. Remove AsyncTrigger for usedIds, and change to serverInfo->onChange. Use const & to avoid unnecessary copies in WorkerInterface's LocalityData and getExtraTLogEligibleMachines().
2019-02-01 14:23:49 +08:00
}
Fix a segfault bug due to uncopied ratekeeper interface
2019-02-19 06:57:21 +08:00
if (cluster->db.serverInfo->get().ratekeeper.present()) {
id_used[cluster->db.serverInfo->get().ratekeeper.get().locality.processId()]++;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
state WorkerFitnessInfo masterWorker = cluster->getWorkerForRoleInDatacenter(cluster->clusterControllerDcId, ProcessClass::Master, ProcessClass::NeverAssign, db->config, id_used);
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if( ( masterWorker.worker.processClass.machineClassFitness( ProcessClass::Master ) > SERVER_KNOBS->EXPECTED_MASTER_FITNESS || masterWorker.worker.interf.locality.processId() == cluster->clusterControllerProcessId )
fix: the cluster controller did not consider the master sharing the same process as the cluster controller as bad in all needed locations waited too long for good recruitment locations, which would add too much time to recoveries of clusters that do not use machine classes
2018-02-07 03:30:05 +08:00
&& now() - cluster->startTime < SERVER_KNOBS->WAIT_FOR_GOOD_RECRUITMENT_DELAY ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
TraceEvent("CCWDB", cluster->id).detail("Fitness", masterWorker.worker.processClass.machineClassFitness( ProcessClass::Master ));
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( delay(SERVER_KNOBS->ATTEMPT_RECRUITMENT_DELAY) );
Initial repository commit
2017-05-26 04:48:44 +08:00
continue;
}
RecruitMasterRequest rmq;
rmq.lifetime = db->serverInfo->get().masterLifetime;
added the (untested) ability to force a recovery to the remote datacenter, even if that results in data loss. If the DR lag is more than 1 week there could be potential data corruption if any primary storage servers are still alive.
2018-07-01 21:39:04 +08:00
rmq.forceRecovery = db->forceRecovery;
Initial repository commit
2017-05-26 04:48:44 +08:00
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
cluster->masterProcessId = masterWorker.worker.interf.locality.processId();
The cluster controller will increase its failure monitoring delay after there have been many unfinishedRecoveries
2018-09-01 01:51:55 +08:00
cluster->db.unfinishedRecoveries++;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
state Future<ErrorOr<MasterInterface>> fNewMaster = masterWorker.worker.interf.master.tryGetReply( rmq );
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
wait( ready(fNewMaster) || db->forceMasterFailure.onTrigger() );
if (fNewMaster.isReady() && fNewMaster.get().present()) {
TraceEvent("CCWDB", cluster->id).detail("Recruited", fNewMaster.get().get().id());
Initial repository commit
2017-05-26 04:48:44 +08:00
// for status tool
TraceEvent("RecruitedMasterWorker", cluster->id)
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
.detail("Address", fNewMaster.get().get().address())
This is the first step in eliminating the usage of database names in our code. The C API remains the same, but underneath that all usage of database names is eliminated.
2018-08-17 01:24:12 +08:00
.trackLatest("RecruitedMasterWorker");
Initial repository commit
2017-05-26 04:48:44 +08:00
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
iMaster = fNewMaster.get().get();
Initial repository commit
2017-05-26 04:48:44 +08:00
db->masterRegistrationCount = 0;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
db->recoveryStalled = false;
Initial repository commit
2017-05-26 04:48:44 +08:00
This is the first step in eliminating the usage of database names in our code. The C API remains the same, but underneath that all usage of database names is eliminated.
2018-08-17 01:24:12 +08:00
auto dbInfo = ServerDBInfo();
Initial repository commit
2017-05-26 04:48:44 +08:00
dbInfo.master = iMaster;
dbInfo.id = g_random->randomUniqueID();
dbInfo.masterLifetime = db->serverInfo->get().masterLifetime;
++dbInfo.masterLifetime;
dbInfo.clusterInterface = db->serverInfo->get().clusterInterface;
Add DataDistributorInterface to ServerDBInfo Also change the Proxy and QuietDatabase to use the DataDistributorInterface.
2019-01-19 03:30:18 +08:00
dbInfo.distributor = db->serverInfo->get().distributor;
Fix a segfault bug due to uncopied ratekeeper interface
2019-02-19 06:57:21 +08:00
dbInfo.ratekeeper = db->serverInfo->get().ratekeeper;
Initial repository commit
2017-05-26 04:48:44 +08:00
TraceEvent("CCWDB", cluster->id).detail("Lifetime", dbInfo.masterLifetime.toString()).detail("ChangeID", dbInfo.id);
db->serverInfo->set( dbInfo );
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
state Future<Void> spinDelay = delay(SERVER_KNOBS->MASTER_SPIN_DELAY); // Don't retry master recovery more than once per second, but don't delay the "first" recovery after more than a second of normal operation
Initial repository commit
2017-05-26 04:48:44 +08:00
TraceEvent("CCWDB", cluster->id).detail("Watching", iMaster.id());
// Master failure detection is pretty sensitive, but if we are in the middle of a very long recovery we really don't want to have to start over
loop choose {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when (wait( waitFailureClient( iMaster.waitFailure, db->masterRegistrationCount ?
Initial repository commit
2017-05-26 04:48:44 +08:00
SERVER_KNOBS->MASTER_FAILURE_REACTION_TIME : (now() - recoveryStart) * SERVER_KNOBS->MASTER_FAILURE_SLOPE_DURING_RECOVERY,
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
db->masterRegistrationCount ? -SERVER_KNOBS->MASTER_FAILURE_REACTION_TIME/SERVER_KNOBS->SECONDS_BEFORE_NO_FAILURE_DELAY : SERVER_KNOBS->MASTER_FAILURE_SLOPE_DURING_RECOVERY ) || db->forceMasterFailure.onTrigger() )) { break; }
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when (wait( db->serverInfo->onChange() )) {}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
wait(spinDelay);
Initial repository commit
2017-05-26 04:48:44 +08:00
TEST(true); // clusterWatchDatabase() master failed
TraceEvent(SevWarn,"DetectedFailedMaster", cluster->id).detail("OldMaster", iMaster.id());
} else {
TEST(true); //clusterWatchDatabas() !newMaster.present()
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( delay(SERVER_KNOBS->MASTER_SPIN_DELAY) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
} catch (Error& e) {
TraceEvent("CCWDB", cluster->id).error(e, true).detail("Master", iMaster.id());
if (e.code() == error_code_actor_cancelled) throw;
bool ok = e.code() == error_code_no_more_servers;
Attempt to normalize trace events: * Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check. * Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase. * Use seconds instead of milliseconds in details. Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed. This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-09 02:11:08 +08:00
TraceEvent(ok ? SevWarn : SevError,"ClusterWatchDatabaseRetrying", cluster->id).error(e);
Initial repository commit
2017-05-26 04:48:44 +08:00
if (!ok)
throw e;
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( delay(SERVER_KNOBS->ATTEMPT_RECRUITMENT_DELAY) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
}
void addIssue( ProcessIssuesMap& issueMap, NetworkAddress const& addr, std::string const& issue, UID& issueID ) {
auto& e = issueMap[addr];
e.first = issue;
e.second = issueID = g_random->randomUniqueID();
if (!issue.size()) issueMap.erase(addr);
}
void removeIssue( ProcessIssuesMap& issueMap, NetworkAddress const& addr, std::string const& issue, UID& issueID ) {
if (!issue.size()) return;
if ( issueMap.count(addr) && issueMap[addr].second == issueID )
issueMap.erase( addr );
}
ACTOR Future<Void> clusterGetServerInfo(
ClusterControllerData::DBInfo* db,
UID knownServerInfoID,
std::string issues,
std::vector<NetworkAddress> incompatiblePeers,
ReplyPromise<ServerDBInfo> reply)
{
state UID issueID;
Listen to multiple addresses and start using vector<NetworkAdddress> in Endpoint - This patch will make FDB listen to multiple addresses given via command line. Although, we'll still use first address in most places, this patch starts using vector<NetworkAddress> in Endpoint at some basic places. - When sending packets to an endpoint, pick a random network address in endpoints - Renames Endpoint::address to Endpoint::addresses since it now holds a vector of addresses.
2018-10-31 04:44:37 +08:00
addIssue( db->workersWithIssues, reply.getEndpoint().getPrimaryAddress(), issues, issueID );
Initial repository commit
2017-05-26 04:48:44 +08:00
for(auto it : incompatiblePeers) {
db->incompatibleConnections[it] = now() + SERVER_KNOBS->INCOMPATIBLE_PEERS_LOGGING_INTERVAL;
}
while (db->serverInfo->get().id == knownServerInfoID) {
choose {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when (wait( db->serverInfo->onChange() )) {}
when (wait( delayJittered( 300 ) )) { break; } // The server might be long gone!
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
Listen to multiple addresses and start using vector<NetworkAdddress> in Endpoint - This patch will make FDB listen to multiple addresses given via command line. Although, we'll still use first address in most places, this patch starts using vector<NetworkAddress> in Endpoint at some basic places. - When sending packets to an endpoint, pick a random network address in endpoints - Renames Endpoint::address to Endpoint::addresses since it now holds a vector of addresses.
2018-10-31 04:44:37 +08:00
removeIssue( db->workersWithIssues, reply.getEndpoint().getPrimaryAddress(), issues, issueID );
Initial repository commit
2017-05-26 04:48:44 +08:00
reply.send( db->serverInfo->get() );
return Void();
}
ACTOR Future<Void> clusterOpenDatabase(
ClusterControllerData::DBInfo* db,
UID knownClientInfoID,
std::string issues,
Standalone<VectorRef<ClientVersionRef>> supportedVersions,
Status: Count connected coordinators per client A client will always try to connect all coordinators. This commit let Status track the number of connected coordinators for each client. This allows us to do canary in coordinators. For example, when we switch from non-TLS to TLS, we can switch 1 coordinator from non-TLS to TLS. This can help check if a client has the ability to connect through TLS. We can make the non-TLS to TLS switch for each coordinators one by one. This avoid the risk of losing connection in the switch.
2019-03-06 13:00:19 +08:00
int connectedCoordinatorsNum,
Support log group field in status json
2017-09-29 07:31:29 +08:00
Standalone<StringRef> traceLogGroup,
Initial repository commit
2017-05-26 04:48:44 +08:00
ReplyPromise<ClientDBInfo> reply)
{
// NOTE: The client no longer expects this function to return errors
state UID issueID;
Listen to multiple addresses and start using vector<NetworkAdddress> in Endpoint - This patch will make FDB listen to multiple addresses given via command line. Although, we'll still use first address in most places, this patch starts using vector<NetworkAddress> in Endpoint at some basic places. - When sending packets to an endpoint, pick a random network address in endpoints - Renames Endpoint::address to Endpoint::addresses since it now holds a vector of addresses.
2018-10-31 04:44:37 +08:00
addIssue( db->clientsWithIssues, reply.getEndpoint().getPrimaryAddress(), issues, issueID );
Initial repository commit
2017-05-26 04:48:44 +08:00
if(supportedVersions.size() > 0) {
Listen to multiple addresses and start using vector<NetworkAdddress> in Endpoint - This patch will make FDB listen to multiple addresses given via command line. Although, we'll still use first address in most places, this patch starts using vector<NetworkAddress> in Endpoint at some basic places. - When sending packets to an endpoint, pick a random network address in endpoints - Renames Endpoint::address to Endpoint::addresses since it now holds a vector of addresses.
2018-10-31 04:44:37 +08:00
db->clientVersionMap[reply.getEndpoint().getPrimaryAddress()] = supportedVersions;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Status: Count connected coordinators per client A client will always try to connect all coordinators. This commit let Status track the number of connected coordinators for each client. This allows us to do canary in coordinators. For example, when we switch from non-TLS to TLS, we can switch 1 coordinator from non-TLS to TLS. This can help check if a client has the ability to connect through TLS. We can make the non-TLS to TLS switch for each coordinators one by one. This avoid the risk of losing connection in the switch.
2019-03-06 13:00:19 +08:00
TLS Status: Resolve review comments Use connectedCoordinatorsNumDelayed to reduce the load on cluster controller; Set connectedCoordinatorsNum to null by default for monitorLeader()
2019-03-12 08:10:06 +08:00
db->clientStatusInfoMap[reply.getEndpoint().getPrimaryAddress()] = ClientStatusInfo(traceLogGroup.toString(), connectedCoordinatorsNum);
Support log group field in status json
2017-09-29 07:31:29 +08:00
Initial repository commit
2017-05-26 04:48:44 +08:00
while (db->clientInfo->get().id == knownClientInfoID) {
choose {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when (wait( db->clientInfo->onChange() )) {}
when (wait( delayJittered( 300 ) )) { break; } // The client might be long gone!
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
Listen to multiple addresses and start using vector<NetworkAdddress> in Endpoint - This patch will make FDB listen to multiple addresses given via command line. Although, we'll still use first address in most places, this patch starts using vector<NetworkAddress> in Endpoint at some basic places. - When sending packets to an endpoint, pick a random network address in endpoints - Renames Endpoint::address to Endpoint::addresses since it now holds a vector of addresses.
2018-10-31 04:44:37 +08:00
removeIssue( db->clientsWithIssues, reply.getEndpoint().getPrimaryAddress(), issues, issueID );
db->clientVersionMap.erase(reply.getEndpoint().getPrimaryAddress());
Status: TLS client status Use ClientStatusInfo structure for each network address (client), instead of passing each status info as a parameter.
2019-02-27 08:20:05 +08:00
db->clientStatusInfoMap.erase(reply.getEndpoint().getPrimaryAddress());
Initial repository commit
2017-05-26 04:48:44 +08:00
reply.send( db->clientInfo->get() );
return Void();
}
void checkOutstandingRecruitmentRequests( ClusterControllerData* self ) {
for( int i = 0; i < self->outstandingRecruitmentRequests.size(); i++ ) {
RecruitFromConfigurationRequest& req = self->outstandingRecruitmentRequests[i];
try {
req.reply.send( self->findWorkersForConfiguration( req ) );
Stop performing self-moves. (e.g. a = std::move(a)) self-moves are frowned upon in C++, and in our code this generally happens from calls to swap as part of trying to implement a "unordered erase" function via swap-to-the-end-and-pop_back. For convenience, a swapAndPop() function is now offered that performs this, while disallowing self-moves.
2018-08-02 09:09:54 +08:00
swapAndPop( &self->outstandingRecruitmentRequests, i-- );
Initial repository commit
2017-05-26 04:48:44 +08:00
} catch (Error& e) {
if (e.code() == error_code_no_more_servers || e.code() == error_code_operation_failed) {
TraceEvent(SevWarn, "RecruitTLogMatchingSetNotAvailable", self->id).error(e);
} else {
TraceEvent(SevError, "RecruitTLogsRequestError", self->id).error(e);
throw;
}
}
}
}
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
void checkOutstandingRemoteRecruitmentRequests( ClusterControllerData* self ) {
for( int i = 0; i < self->outstandingRemoteRecruitmentRequests.size(); i++ ) {
RecruitRemoteFromConfigurationRequest& req = self->outstandingRemoteRecruitmentRequests[i];
try {
req.reply.send( self->findRemoteWorkersForConfiguration( req ) );
Stop performing self-moves. (e.g. a = std::move(a)) self-moves are frowned upon in C++, and in our code this generally happens from calls to swap as part of trying to implement a "unordered erase" function via swap-to-the-end-and-pop_back. For convenience, a swapAndPop() function is now offered that performs this, while disallowing self-moves.
2018-08-02 09:09:54 +08:00
swapAndPop( &self->outstandingRemoteRecruitmentRequests, i-- );
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
} catch (Error& e) {
if (e.code() == error_code_no_more_servers || e.code() == error_code_operation_failed) {
TraceEvent(SevWarn, "RecruitRemoteTLogMatchingSetNotAvailable", self->id).error(e);
} else {
TraceEvent(SevError, "RecruitRemoteTLogsRequestError", self->id).error(e);
throw;
}
}
}
}
Initial repository commit
2017-05-26 04:48:44 +08:00
void checkOutstandingStorageRequests( ClusterControllerData* self ) {
for( int i = 0; i < self->outstandingStorageRequests.size(); i++ ) {
auto& req = self->outstandingStorageRequests[i];
try {
if(req.second < now()) {
req.first.reply.sendError(timed_out());
Stop performing self-moves. (e.g. a = std::move(a)) self-moves are frowned upon in C++, and in our code this generally happens from calls to swap as part of trying to implement a "unordered erase" function via swap-to-the-end-and-pop_back. For convenience, a swapAndPop() function is now offered that performs this, while disallowing self-moves.
2018-08-02 09:09:54 +08:00
swapAndPop( &self->outstandingStorageRequests, i-- );
Initial repository commit
2017-05-26 04:48:44 +08:00
} else {
if(!self->gotProcessClasses && !req.first.criticalRecruitment)
throw no_more_servers();
auto worker = self->getStorageWorker(req.first);
RecruitStorageReply rep;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
rep.worker = worker.interf;
rep.processClass = worker.processClass;
Initial repository commit
2017-05-26 04:48:44 +08:00
req.first.reply.send( rep );
Stop performing self-moves. (e.g. a = std::move(a)) self-moves are frowned upon in C++, and in our code this generally happens from calls to swap as part of trying to implement a "unordered erase" function via swap-to-the-end-and-pop_back. For convenience, a swapAndPop() function is now offered that performs this, while disallowing self-moves.
2018-08-02 09:09:54 +08:00
swapAndPop( &self->outstandingStorageRequests, i-- );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
} catch (Error& e) {
if (e.code() == error_code_no_more_servers) {
TraceEvent(SevWarn, "RecruitStorageNotAvailable", self->id).error(e);
} else {
TraceEvent(SevError, "RecruitStorageError", self->id).error(e);
throw;
}
}
}
}
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
void checkBetterDDOrRK(ClusterControllerData* self) {
Fix a race of DD, RK, Master failure If all DD, RK, Master run on the same process and failed. Recruiting of new DD or RK could try to use the old master worker interface, which is an invalid one and causes recruitment to be stuck. Fix by adding a delay and checking master is valid before recruitment.
2019-03-21 07:03:36 +08:00
if (!self->masterProcessId.present() || self->db.serverInfo->get().recoveryState < RecoveryState::ACCEPTING_COMMITS) {
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
return;
}
auto& masterWorker = self->id_worker[self->masterProcessId.get()];
const ServerDBInfo& db = self->db.serverInfo->get();
Add actors to store halt request futures Address best fitness in checking better DD or RK.
2019-03-23 09:00:16 +08:00
auto bestFitnessForRK = self->getBestFitnessForRoleInDatacenter(ProcessClass::RateKeeper);
auto bestFitnessForDD = self->getBestFitnessForRoleInDatacenter(ProcessClass::DataDistributor);
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
Fix test stuck due to invalid worker in cluster controller Test case: -r simulation --crash -f ./tests/rare/CloggedCycleWithKills.txt -s 688927581 -b off
2019-03-21 13:18:53 +08:00
if (!self->recruitingRatekeeperID.present() && db.ratekeeper.present() && self->id_worker.count(db.ratekeeper.get().locality.processId())) {
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
auto& rkWorker = self->id_worker[db.ratekeeper.get().locality.processId()];
auto rkFitness = rkWorker.details.processClass.machineClassFitness(ProcessClass::RateKeeper);
if (self->isProxyOrResolver(rkWorker.details.interf.locality.processId()) ||
Add actors to store halt request futures Address best fitness in checking better DD or RK.
2019-03-23 09:00:16 +08:00
rkFitness > bestFitnessForRK || rkWorker.priorityInfo.isExcluded) {
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
TraceEvent("CC_HaltRK", self->id).detail("RKID", db.ratekeeper.get().id())
Move ratekeeper or data distributor from excluded servers
2019-03-22 08:07:12 +08:00
.detail("Excluded", rkWorker.priorityInfo.isExcluded)
Add actors to store halt request futures Address best fitness in checking better DD or RK.
2019-03-23 09:00:16 +08:00
.detail("Fitness", rkFitness).detail("BestFitness", bestFitnessForRK);
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
self->recruitRatekeeper.trigger();
}
}
Fix test stuck due to invalid worker in cluster controller Test case: -r simulation --crash -f ./tests/rare/CloggedCycleWithKills.txt -s 688927581 -b off
2019-03-21 13:18:53 +08:00
if (!self->recruitingDistributor && db.distributor.present() && self->id_worker.count(db.distributor.get().locality.processId())) {
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
auto& ddWorker = self->id_worker[db.distributor.get().locality.processId()];
auto ddFitness = ddWorker.details.processClass.machineClassFitness(ProcessClass::DataDistributor);
if (self->isProxyOrResolver(ddWorker.details.interf.locality.processId()) ||
Add actors to store halt request futures Address best fitness in checking better DD or RK.
2019-03-23 09:00:16 +08:00
ddFitness > bestFitnessForDD || ddWorker.priorityInfo.isExcluded) {
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
TraceEvent("CC_HaltDD", self->id).detail("DDID", db.distributor.get().id())
Move ratekeeper or data distributor from excluded servers
2019-03-22 08:07:12 +08:00
.detail("Excluded", ddWorker.priorityInfo.isExcluded)
Add actors to store halt request futures Address best fitness in checking better DD or RK.
2019-03-23 09:00:16 +08:00
.detail("Fitness", ddFitness).detail("BestFitness", bestFitnessForDD);
ensure that the processId exists in id_worker before accessing it
2019-03-23 09:54:39 +08:00
ddWorker.haltDistributor = brokenPromiseToNever(db.distributor.get().haltDataDistributor.getReply(HaltDataDistributorRequest(self->id)));
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
}
}
}
checking outstanding requests can be CPU intensive, so rate limit checking requests
2018-06-28 14:02:08 +08:00
ACTOR Future<Void> doCheckOutstandingRequests( ClusterControllerData* self ) {
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
try {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( delay(SERVER_KNOBS->CHECK_OUTSTANDING_INTERVAL) );
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
checkOutstandingRecruitmentRequests( self );
checkOutstandingRemoteRecruitmentRequests( self );
checkOutstandingStorageRequests( self );
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
checkBetterDDOrRK(self);
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
self->checkRecoveryStalled();
if (self->betterMasterExists()) {
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
self->db.forceMasterFailure.trigger();
TraceEvent("MasterRegistrationKill", self->id).detail("MasterId", self->db.serverInfo->get().master.id());
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
}
} catch( Error &e ) {
if(e.code() != error_code_operation_failed && e.code() != error_code_no_more_servers) {
TraceEvent(SevError, "CheckOutstandingError").error(e);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
return Void();
}
checking outstanding requests can be CPU intensive, so rate limit checking requests
2018-06-28 14:02:08 +08:00
void checkOutstandingRequests( ClusterControllerData* self ) {
if( !self->outstandingRequestChecker.isReady() )
Initial repository commit
2017-05-26 04:48:44 +08:00
return;
checking outstanding requests can be CPU intensive, so rate limit checking requests
2018-06-28 14:02:08 +08:00
self->outstandingRequestChecker = doCheckOutstandingRequests(self);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
ACTOR Future<Void> rebootAndCheck( ClusterControllerData* cluster, Optional<Standalone<StringRef>> processID ) {
Fix minor IDE build errors
2019-02-18 10:46:59 +08:00
{
auto watcher = cluster->id_worker.find(processID);
ASSERT(watcher != cluster->id_worker.end());
Initial repository commit
2017-05-26 04:48:44 +08:00
Fix minor IDE build errors
2019-02-18 10:46:59 +08:00
watcher->second.lastAvailableTime = now();
watcher->second.reboots++;
wait( delay( g_network->isSimulated() ? SERVER_KNOBS->SIM_SHUTDOWN_TIMEOUT : SERVER_KNOBS->SHUTDOWN_TIMEOUT ) );
}
Initial repository commit
2017-05-26 04:48:44 +08:00
Fix minor IDE build errors
2019-02-18 10:46:59 +08:00
{
auto watcher = cluster->id_worker.find(processID);
if(watcher != cluster->id_worker.end()) {
watcher->second.reboots--;
if( watcher->second.reboots < 2 )
checkOutstandingRequests( cluster );
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
return Void();
}
ACTOR Future<Void> workerAvailabilityWatch( WorkerInterface worker, ProcessClass startingClass, ClusterControllerData* cluster ) {
Merge release-6.0 into master
2018-09-22 07:14:39 +08:00
state Future<Void> failed = worker.address() == g_network->getLocalAddress() ? Never() : waitFailureClient( worker.waitFailure, SERVER_KNOBS->WORKER_FAILURE_TIME );
Initial repository commit
2017-05-26 04:48:44 +08:00
cluster->updateWorkerList.set( worker.locality.processId(), ProcessData(worker.locality, startingClass, worker.address()) );
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
// This switching avoids a race where the worker can be added to id_worker map after the workerAvailabilityWatch fails for the worker.
wait(delay(0));
Initial repository commit
2017-05-26 04:48:44 +08:00
loop {
choose {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when( wait( IFailureMonitor::failureMonitor().onStateEqual( worker.storage.getEndpoint(), FailureStatus(IFailureMonitor::failureMonitor().getState( worker.storage.getEndpoint() ).isAvailable()) ) ) ) {
Initial repository commit
2017-05-26 04:48:44 +08:00
if( IFailureMonitor::failureMonitor().getState( worker.storage.getEndpoint() ).isAvailable() ) {
cluster->ac.add( rebootAndCheck( cluster, worker.locality.processId() ) );
checkOutstandingRequests( cluster );
}
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when( wait( failed ) ) { // remove workers that have failed
Fix cluster controller review comments
2017-10-05 06:48:55 +08:00
WorkerInfo& failedWorkerInfo = cluster->id_worker[ worker.locality.processId() ];
Add initialClass to RegisterWorkerRequest
2017-10-05 08:11:12 +08:00
if (!failedWorkerInfo.reply.isSet()) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
failedWorkerInfo.reply.send( RegisterWorkerReply(failedWorkerInfo.details.processClass, failedWorkerInfo.priorityInfo) );
Add initialClass to RegisterWorkerRequest
2017-10-05 08:11:12 +08:00
}
Fix a race of DD, RK, Master failure If all DD, RK, Master run on the same process and failed. Recruiting of new DD or RK could try to use the old master worker interface, which is an invalid one and causes recruitment to be stuck. Fix by adding a delay and checking master is valid before recruitment.
2019-03-21 07:03:36 +08:00
if (worker.locality.processId() == cluster->masterProcessId) {
cluster->masterProcessId = Optional<Key>();
}
Initial repository commit
2017-05-26 04:48:44 +08:00
cluster->id_worker.erase( worker.locality.processId() );
cluster->updateWorkerList.set( worker.locality.processId(), Optional<ProcessData>() );
return Void();
}
}
}
}
struct FailureStatusInfo {
FailureStatus status;
double lastRequestTime;
double penultimateRequestTime;
FailureStatusInfo() : lastRequestTime(0), penultimateRequestTime(0) {}
void insertRequest(double now) {
penultimateRequestTime = lastRequestTime;
lastRequestTime = now;
}
double latency(double now) const {
return std::max( now - lastRequestTime, lastRequestTime - penultimateRequestTime );
}
};
//The failure monitor client relies on the fact that the failure detection server will not declare itself failed
The cluster controller will increase its failure monitoring delay after there have been many unfinishedRecoveries
2018-09-01 01:51:55 +08:00
ACTOR Future<Void> failureDetectionServer( UID uniqueID, ClusterControllerData::DBInfo* db, FutureStream< FailureMonitoringRequest > requests ) {
Initial repository commit
2017-05-26 04:48:44 +08:00
state Version currentVersion = 0;
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
state std::map<NetworkAddressList, FailureStatusInfo> currentStatus; // The status at currentVersion
Initial repository commit
2017-05-26 04:48:44 +08:00
state std::deque<SystemFailureStatus> statusHistory; // The last change in statusHistory is from currentVersion-1 to currentVersion
state Future<Void> periodically = Void();
state double lastT = 0;
loop choose {
when ( FailureMonitoringRequest req = waitNext( requests ) ) {
if ( req.senderStatus.present() ) {
// Update the status of requester, if necessary
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
auto& stat = currentStatus[ req.addresses ];
Initial repository commit
2017-05-26 04:48:44 +08:00
auto& newStat = req.senderStatus.get();
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
ASSERT( !newStat.failed || req.addresses != g_network->getLocalAddresses() );
Initial repository commit
2017-05-26 04:48:44 +08:00
stat.insertRequest(now());
if (req.senderStatus != stat.status) {
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
TraceEvent("FailureDetectionStatus", uniqueID).detail("System", describe(req.addresses)).detail("Status", newStat.failed ? "Failed" : "OK").detail("Why", "Request");
statusHistory.push_back( SystemFailureStatus( req.addresses, newStat ) );
Initial repository commit
2017-05-26 04:48:44 +08:00
++currentVersion;
if (req.senderStatus == FailureStatus()){
// failureMonitorClient reports explicitly that it is failed
ASSERT(false); // This can't happen at the moment; if that changes, make this a TEST instead
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
currentStatus.erase(req.addresses);
Initial repository commit
2017-05-26 04:48:44 +08:00
} else {
TEST(true);
stat.status = newStat;
}
while (statusHistory.size() > currentStatus.size())
statusHistory.pop_front();
}
}
// Return delta-compressed status changes to requester
Version reqVersion = req.failureInformationVersion;
if (reqVersion > currentVersion){
req.reply.sendError( future_version() );
ASSERT(false);
} else {
TEST(true); // failureDetectionServer sending failure data to requester
FailureMonitoringReply reply;
reply.failureInformationVersion = currentVersion;
added knobs so that the client failure monitoring update rate and the server failure monitoring update rate are separate knobs
2017-12-02 05:04:32 +08:00
if( req.senderStatus.present() ) {
reply.clientRequestIntervalMS = FLOW_KNOBS->SERVER_REQUEST_INTERVAL * 1000;
reply.considerServerFailedTimeoutMS = CLIENT_KNOBS->FAILURE_TIMEOUT_DELAY * 1000;
} else {
reply.clientRequestIntervalMS = FLOW_KNOBS->CLIENT_REQUEST_INTERVAL * 1000;
reply.considerServerFailedTimeoutMS = CLIENT_KNOBS->CLIENT_FAILURE_TIMEOUT_DELAY * 1000;
}
Initial repository commit
2017-05-26 04:48:44 +08:00
ASSERT( currentVersion >= (int64_t)statusHistory.size());
if (reqVersion < currentVersion - (int64_t)statusHistory.size() || reqVersion == 0) {
// Send everything
TEST(true); // failureDetectionServer sending all current data to requester
reply.allOthersFailed = true;
for(auto it = currentStatus.begin(); it != currentStatus.end(); ++it)
reply.changes.push_back( reply.arena, SystemFailureStatus( it->first, it->second.status ) );
} else {
TEST(true); // failureDetectionServer sending delta-compressed data to requester
// SOMEDAY: Send only the last change for a given address?
reply.allOthersFailed = false;
for(int v = reqVersion - currentVersion + statusHistory.size(); v < statusHistory.size(); v++) {
reply.changes.push_back( reply.arena, statusHistory[v] );
}
}
req.reply.send( reply );
}
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when ( wait( periodically ) ) {
added knobs so that the client failure monitoring update rate and the server failure monitoring update rate are separate knobs
2017-12-02 05:04:32 +08:00
periodically = delay( FLOW_KNOBS->SERVER_REQUEST_INTERVAL );
Initial repository commit
2017-05-26 04:48:44 +08:00
double t = now();
if (lastT != 0 && t - lastT > 1)
TraceEvent("LongDelayOnClusterController").detail("Duration", t - lastT);
lastT = t;
// Adapt to global unresponsiveness
vector<double> delays;
for(auto it=currentStatus.begin(); it!=currentStatus.end(); it++)
if (it->second.penultimateRequestTime) {
delays.push_back(it->second.latency(t));
call all methods which could disable a trace event before it is initialized. In practice this means calling .error first, then .suppressFor, then all your details.
2018-08-02 05:30:57 +08:00
//TraceEvent("FDData", uniqueID).detail("S", it->first.toString()).detail("L", it->second.latency(t));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
int pivot = std::max(0, (int)delays.size()-2);
double pivotDelay = 0;
if (delays.size()) {
std::nth_element(delays.begin(), delays.begin()+pivot, delays.end());
pivotDelay = *(delays.begin()+pivot);
}
added knobs so that the client failure monitoring update rate and the server failure monitoring update rate are separate knobs
2017-12-02 05:04:32 +08:00
pivotDelay = std::max(0.0, pivotDelay - FLOW_KNOBS->SERVER_REQUEST_INTERVAL);
Initial repository commit
2017-05-26 04:48:44 +08:00
call all methods which could disable a trace event before it is initialized. In practice this means calling .error first, then .suppressFor, then all your details.
2018-08-02 05:30:57 +08:00
//TraceEvent("FailureDetectionPoll", uniqueID).detail("PivotDelay", pivotDelay).detail("Clients", currentStatus.size());
Attempt to normalize trace events: * Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check. * Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase. * Use seconds instead of milliseconds in details. Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed. This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-09 02:11:08 +08:00
//TraceEvent("FailureDetectionAcceptableDelay").detail("Delay", acceptableDelay1000);
Initial repository commit
2017-05-26 04:48:44 +08:00
The cluster controller will increase its failure monitoring delay after there have been many unfinishedRecoveries
2018-09-01 01:51:55 +08:00
bool tooManyLogGenerations = std::max(db->unfinishedRecoveries, db->logGenerations) > CLIENT_KNOBS->FAILURE_MAX_GENERATIONS;
Initial repository commit
2017-05-26 04:48:44 +08:00
for(auto it = currentStatus.begin(); it != currentStatus.end(); ) {
double delay = t - it->second.lastRequestTime;
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
if ( it->first != g_network->getLocalAddresses() && ( tooManyLogGenerations ?
The cluster controller will increase its failure monitoring delay after there have been many unfinishedRecoveries
2018-09-01 01:51:55 +08:00
( delay > CLIENT_KNOBS->FAILURE_EMERGENCY_DELAY ) :
( delay > pivotDelay * 2 + FLOW_KNOBS->SERVER_REQUEST_INTERVAL + CLIENT_KNOBS->FAILURE_MIN_DELAY || delay > CLIENT_KNOBS->FAILURE_MAX_DELAY ) ) ) {
Initial repository commit
2017-05-26 04:48:44 +08:00
//printf("Failure Detection Server: Status of '%s' is now '%s' after %f sec\n", it->first.toString().c_str(), "Failed", now() - it->second.lastRequestTime);
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
TraceEvent("FailureDetectionStatus", uniqueID).detail("System", describe(it->first)).detail("Status","Failed").detail("Why", "Timeout").detail("LastRequestAge", delay)
The cluster controller will increase its failure monitoring delay after there have been many unfinishedRecoveries
2018-09-01 01:51:55 +08:00
.detail("PivotDelay", pivotDelay).detail("UnfinishedRecoveries", db->unfinishedRecoveries).detail("LogGenerations", db->logGenerations);
Initial repository commit
2017-05-26 04:48:44 +08:00
statusHistory.push_back( SystemFailureStatus( it->first, FailureStatus(true) ) );
++currentVersion;
it = currentStatus.erase(it);
while (statusHistory.size() > currentStatus.size())
statusHistory.pop_front();
} else {
++it;
}
}
}
}
}
ACTOR Future<vector<TLogInterface>> requireAll( vector<Future<Optional<vector<TLogInterface>>>> in ) {
state vector<TLogInterface> out;
state int i;
for(i=0; i<in.size(); i++) {
Optional<vector<TLogInterface>> x = wait(in[i]);
if (!x.present()) throw recruitment_failed();
out.insert(out.end(), x.get().begin(), x.get().end());
}
return out;
}
void clusterRecruitStorage( ClusterControllerData* self, RecruitStorageRequest req ) {
try {
if(!self->gotProcessClasses && !req.criticalRecruitment)
throw no_more_servers();
auto worker = self->getStorageWorker(req);
RecruitStorageReply rep;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
rep.worker = worker.interf;
rep.processClass = worker.processClass;
Initial repository commit
2017-05-26 04:48:44 +08:00
req.reply.send( rep );
} catch ( Error& e ) {
if (e.code() == error_code_no_more_servers) {
self->outstandingStorageRequests.push_back( std::make_pair(req, now() + SERVER_KNOBS->RECRUITMENT_TIMEOUT) );
TraceEvent(SevWarn, "RecruitStorageNotAvailable", self->id).error(e);
} else {
TraceEvent(SevError, "RecruitStorageError", self->id).error(e);
throw; // Any other error will bring down the cluster controller
}
}
}
ACTOR Future<Void> clusterRecruitFromConfiguration( ClusterControllerData* self, RecruitFromConfigurationRequest req ) {
// At the moment this doesn't really need to be an actor (it always completes immediately)
TEST(true); //ClusterController RecruitTLogsRequest
loop {
try {
req.reply.send( self->findWorkersForConfiguration( req ) );
return Void();
} catch (Error& e) {
if (e.code() == error_code_no_more_servers && now() - self->startTime >= SERVER_KNOBS->WAIT_FOR_GOOD_RECRUITMENT_DELAY) {
self->outstandingRecruitmentRequests.push_back( req );
TraceEvent(SevWarn, "RecruitFromConfigurationNotAvailable", self->id).error(e);
return Void();
} else if(e.code() == error_code_operation_failed || e.code() == error_code_no_more_servers) {
//recruitment not good enough, try again
}
else {
TraceEvent(SevError, "RecruitFromConfigurationError", self->id).error(e);
throw; // goodbye, cluster controller
}
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( delay(SERVER_KNOBS->ATTEMPT_RECRUITMENT_DELAY) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
ACTOR Future<Void> clusterRecruitRemoteFromConfiguration( ClusterControllerData* self, RecruitRemoteFromConfigurationRequest req ) {
// At the moment this doesn't really need to be an actor (it always completes immediately)
TEST(true); //ClusterController RecruitTLogsRequest
loop {
try {
req.reply.send( self->findRemoteWorkersForConfiguration( req ) );
return Void();
} catch (Error& e) {
Swapped OkayFit and UnsetFit, because generally if machine classes are set on one machine they are set everywhere and it helps with wait_for_good_recruitment logic wait_for_good_recruitment now requires that you have the desired count of each roll remote recruitment is given a much longer wait_for_good_recruitment time interval, which does not start until enough remote machines have registered
2018-06-23 01:15:24 +08:00
if (e.code() == error_code_no_more_servers && self->remoteStartTime.present() && now() - self->remoteStartTime.get() >= SERVER_KNOBS->WAIT_FOR_GOOD_REMOTE_RECRUITMENT_DELAY) {
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
self->outstandingRemoteRecruitmentRequests.push_back( req );
TraceEvent(SevWarn, "RecruitRemoteFromConfigurationNotAvailable", self->id).error(e);
return Void();
} else if(e.code() == error_code_operation_failed || e.code() == error_code_no_more_servers) {
//recruitment not good enough, try again
}
else {
TraceEvent(SevError, "RecruitRemoteFromConfigurationError", self->id).error(e);
throw; // goodbye, cluster controller
}
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( delay(SERVER_KNOBS->ATTEMPT_RECRUITMENT_DELAY) );
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
}
Initial repository commit
2017-05-26 04:48:44 +08:00
void clusterRegisterMaster( ClusterControllerData* self, RegisterMasterRequest const& req ) {
req.reply.send( Void() );
This is the first step in eliminating the usage of database names in our code. The C API remains the same, but underneath that all usage of database names is eliminated.
2018-08-17 01:24:12 +08:00
TraceEvent("MasterRegistrationReceived", self->id).detail("MasterId", req.id).detail("Master", req.mi.toString()).detail("Tlogs", describe(req.logSystemConfig.tLogs)).detail("Resolvers", req.resolvers.size())
fix: desiredTLogCount was used instead of getDesiredLogs(), which caused problems with recruitment when desiredTLogCount was -1. canKillProcess logic was wrong. We still need to configure usable_regions because if datacenterVersionDifference is too large we cannot complete data movement.
2018-07-05 04:22:32 +08:00
.detail("RecoveryState", (int)req.recoveryState).detail("RegistrationCount", req.registrationCount).detail("Proxies", req.proxies.size()).detail("RecoveryCount", req.recoveryCount).detail("Stalled", req.recoveryStalled);
Initial repository commit
2017-05-26 04:48:44 +08:00
//make sure the request comes from an active database
auto db = &self->db;
if ( db->serverInfo->get().master.id() != req.id || req.registrationCount <= db->masterRegistrationCount ) {
This is the first step in eliminating the usage of database names in our code. The C API remains the same, but underneath that all usage of database names is eliminated.
2018-08-17 01:24:12 +08:00
TraceEvent("MasterRegistrationNotFound", self->id).detail("MasterId", req.id).detail("ExistingId", db->serverInfo->get().master.id()).detail("RegCount", req.registrationCount).detail("ExistingRegCount", db->masterRegistrationCount);
Initial repository commit
2017-05-26 04:48:44 +08:00
return;
}
The cluster controller will increase its failure monitoring delay after there have been many unfinishedRecoveries
2018-09-01 01:51:55 +08:00
if ( req.recoveryState == RecoveryState::FULLY_RECOVERED ) {
self->db.unfinishedRecoveries = 0;
self->db.logGenerations = 0;
ASSERT( !req.logSystemConfig.oldTLogs.size() );
} else {
self->db.logGenerations = std::max<int>(self->db.logGenerations, req.logSystemConfig.oldTLogs.size());
}
Initial repository commit
2017-05-26 04:48:44 +08:00
db->masterRegistrationCount = req.registrationCount;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
db->recoveryStalled = req.recoveryStalled;
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
if ( req.configuration.present() ) {
db->config = req.configuration.get();
fix: kill all tlogs in one region prevents the remote logs from recovering in that region, do not allow that to prevent us from configuring usable_regions=1. added more recovery states.
2018-07-05 15:08:51 +08:00
if ( req.recoveryState >= RecoveryState::ACCEPTING_COMMITS ) {
Add fullyRecoveredConfig for checking exclusions
2017-11-16 09:15:24 +08:00
self->gotFullyRecoveredConfig = true;
db->fullyRecoveredConfig = req.configuration.get();
for ( auto& it : self->id_worker ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
bool isExcludedFromConfig = db->fullyRecoveredConfig.isExcludedServer(it.second.details.interf.address());
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
if ( it.second.priorityInfo.isExcluded != isExcludedFromConfig ) {
it.second.priorityInfo.isExcluded = isExcludedFromConfig;
if( !it.second.reply.isSet() ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
it.second.reply.send( RegisterWorkerReply( it.second.details.processClass, it.second.priorityInfo ) );
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
Add fullyRecoveredConfig for checking exclusions
2017-11-16 09:15:24 +08:00
}
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
}
}
}
Initial repository commit
2017-05-26 04:48:44 +08:00
bool isChanged = false;
auto dbInfo = self->db.serverInfo->get();
if (dbInfo.recoveryState != req.recoveryState) {
dbInfo.recoveryState = req.recoveryState;
isChanged = true;
}
if (dbInfo.priorCommittedLogServers != req.priorCommittedLogServers) {
dbInfo.priorCommittedLogServers = req.priorCommittedLogServers;
isChanged = true;
}
// Construct the client information
if (db->clientInfo->get().proxies != req.proxies) {
isChanged = true;
ClientDBInfo clientInfo;
clientInfo.id = g_random->randomUniqueID();
clientInfo.proxies = req.proxies;
clientInfo.clientTxnInfoSampleRate = db->clientInfo->get().clientTxnInfoSampleRate;
clientInfo.clientTxnInfoSizeLimit = db->clientInfo->get().clientTxnInfoSizeLimit;
db->clientInfo->set( clientInfo );
dbInfo.client = db->clientInfo->get();
}
if( !dbInfo.logSystemConfig.isEqual(req.logSystemConfig) ) {
isChanged = true;
dbInfo.logSystemConfig = req.logSystemConfig;
}
if( dbInfo.resolvers != req.resolvers ) {
isChanged = true;
dbInfo.resolvers = req.resolvers;
}
if( dbInfo.recoveryCount != req.recoveryCount ) {
isChanged = true;
dbInfo.recoveryCount = req.recoveryCount;
}
if( isChanged ) {
dbInfo.id = g_random->randomUniqueID();
self->db.serverInfo->set( dbInfo );
}
checking outstanding requests can be CPU intensive, so rate limit checking requests
2018-06-28 14:02:08 +08:00
checkOutstandingRequests(self);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
void registerWorker( RegisterWorkerRequest req, ClusterControllerData *self ) {
WorkerInterface w = req.wi;
Introduce prevChangeID to CandidacyRequest and LeaderHeartbeatRequest
2017-10-13 08:11:58 +08:00
ProcessClass newProcessClass = req.processClass;
Initial repository commit
2017-05-26 04:48:44 +08:00
auto info = self->id_worker.find( w.locality.processId() );
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
ClusterControllerPriorityInfo newPriorityInfo = req.priorityInfo;
fix: processClassFitness could be wrong if the client changed their class while rebooting
2019-03-22 08:56:04 +08:00
newPriorityInfo.processClassFitness = newProcessClass.machineClassFitness(ProcessClass::ClusterController);
Initial repository commit
2017-05-26 04:48:44 +08:00
suppressed spammy trace events, and avoid reporting a long master recovery duration when the cluster is first created
2018-11-05 15:07:56 +08:00
if(info == self->id_worker.end()) {
TraceEvent("ClusterControllerActualWorkers", self->id).detail("WorkerId",w.id()).detailext("ProcessId", w.locality.processId()).detailext("ZoneId", w.locality.zoneId()).detailext("DataHall", w.locality.dataHallId()).detail("PClass", req.processClass.toString()).detail("Workers", self->id_worker.size());
} else {
TraceEvent("ClusterControllerWorkerAlreadyRegistered", self->id).suppressFor(1.0).detail("WorkerId",w.id()).detailext("ProcessId", w.locality.processId()).detailext("ZoneId", w.locality.zoneId()).detailext("DataHall", w.locality.dataHallId()).detail("PClass", req.processClass.toString()).detail("Workers", self->id_worker.size());
}
Use processId instead of address for comparison
2017-10-26 02:35:29 +08:00
if ( w.address() == g_network->getLocalAddress() ) {
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
if(self->changingDcIds.get().first) {
if(self->changingDcIds.get().second.present()) {
newPriorityInfo.dcFitness = ClusterControllerPriorityInfo::calculateDCFitness( w.locality.dcId(), self->changingDcIds.get().second.get() );
}
} else if(self->changedDcIds.get().second.present()) {
newPriorityInfo.dcFitness = ClusterControllerPriorityInfo::calculateDCFitness( w.locality.dcId(), self->changedDcIds.get().second.get() );
}
} else {
if(!self->changingDcIds.get().first) {
if(self->changingDcIds.get().second.present()) {
newPriorityInfo.dcFitness = ClusterControllerPriorityInfo::calculateDCFitness( w.locality.dcId(), self->changingDcIds.get().second.get() );
}
} else if(self->changedDcIds.get().second.present()) {
newPriorityInfo.dcFitness = ClusterControllerPriorityInfo::calculateDCFitness( w.locality.dcId(), self->changedDcIds.get().second.get() );
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
Use processId instead of address for comparison
2017-10-26 02:35:29 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
// Check process class and exclusive property
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if ( info == self->id_worker.end() || info->second.details.interf.id() != w.id() || req.generation >= info->second.gen ) {
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
if ( self->gotProcessClasses ) {
auto classIter = self->id_class.find(w.locality.processId());
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
if( classIter != self->id_class.end() && (classIter->second.classSource() == ProcessClass::DBSource || req.initialClass.classType() == ProcessClass::UnsetClass)) {
newProcessClass = classIter->second;
} else {
newProcessClass = req.initialClass;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
newPriorityInfo.processClassFitness = newProcessClass.machineClassFitness(ProcessClass::ClusterController);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Add fullyRecoveredConfig for checking exclusions
2017-11-16 09:15:24 +08:00
if ( self->gotFullyRecoveredConfig ) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
newPriorityInfo.isExcluded = self->db.fullyRecoveredConfig.isExcludedServer(w.address());
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
// Notify the worker to register again with new process class/exclusive property
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
if ( !req.reply.isSet() && newPriorityInfo != req.priorityInfo ) {
req.reply.send( RegisterWorkerReply(newProcessClass, newPriorityInfo) );
Introduce prevChangeID to CandidacyRequest and LeaderHeartbeatRequest
2017-10-13 08:11:58 +08:00
}
}
make sure id_worker is updated before setting ratekeeper or data distribution
2019-03-23 08:08:54 +08:00
if( info == self->id_worker.end() ) {
self->id_worker[w.locality.processId()] = WorkerInfo( workerAvailabilityWatch( w, newProcessClass, self ), req.reply, req.generation, w, req.initialClass, newProcessClass, newPriorityInfo, req.degraded );
checkOutstandingRequests( self );
} else if( info->second.details.interf.id() != w.id() || req.generation >= info->second.gen ) {
if (!info->second.reply.isSet()) {
info->second.reply.send( Never() );
}
info->second.reply = req.reply;
info->second.details.processClass = newProcessClass;
info->second.priorityInfo = newPriorityInfo;
info->second.initialClass = req.initialClass;
info->second.details.degraded = req.degraded;
info->second.gen = req.generation;
if(info->second.details.interf.id() != w.id()) {
info->second.details.interf = w;
info->second.watcher = workerAvailabilityWatch( w, newProcessClass, self );
}
checkOutstandingRequests( self );
} else {
TEST(true); // Received an old worker registration request.
}
Fix the issue of slow dying Data Distributor Test with: -r simulation -f ./foundationdb/tests/slow/CommitBug.txt -s 67828576 -b on The test has the following event sequence: - Time 113.3s, CC noticed DD failure, cleard DD interface. - 1s later, DD rejoined and registered with CC. - Time 131.7s, DD actor cancelled. This old DD raced to register with CC and the failure monitor is not installed because monitorDataDistributor is stalled waiting for new DD. - Time 161.4s, new DD running. New DD recruting was delayed due to no servers in the period. Fix by disabling DD registration during the recruting process.
2019-03-18 11:55:59 +08:00
if (req.distributorInterf.present() && !self->db.serverInfo->get().distributor.present() &&
self->clusterControllerDcId == req.distributorInterf.get().locality.dcId() &&
!self->recruitingDistributor) {
Piggyback data distributor interface in worker registration This allows cluster controller to know data distributor during worker registration phase, thus avoiding recruiting a new data distributor after starting. Also change the worker to skip creating a new data distributor if there is already one running on the worker, which can trigger operation timeout in tests.
2019-01-29 03:29:39 +08:00
const DataDistributorInterface& di = req.distributorInterf.get();
Move ratekeeper or data distributor from excluded servers
2019-03-22 08:07:12 +08:00
TraceEvent("CC_RegisterDataDistributor", self->id).detail("DDID", di.id());
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
self->db.setDistributor(di);
Piggyback data distributor interface in worker registration This allows cluster controller to know data distributor during worker registration phase, thus avoiding recruiting a new data distributor after starting. Also change the worker to skip creating a new data distributor if there is already one running on the worker, which can trigger operation timeout in tests.
2019-01-29 03:29:39 +08:00
}
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
if (req.ratekeeperInterf.present()) {
startRatekeeper is responsible for updating serverDBInfo
2019-03-23 08:56:16 +08:00
if((self->recruitingRatekeeperID.present() && self->recruitingRatekeeperID.get() != req.ratekeeperInterf.get().id()) ||
fix: halt if datacenters are different
2019-03-23 14:53:21 +08:00
self->clusterControllerDcId != w.locality.dcId()) {
startRatekeeper is responsible for updating serverDBInfo
2019-03-23 08:56:16 +08:00
TraceEvent("CC_HaltRatekeeper", self->id).detail("RKID", req.ratekeeperInterf.get().id())
.detail("DcID", printable(self->clusterControllerDcId))
ensure that the processId exists in id_worker before accessing it
2019-03-23 09:54:39 +08:00
.detail("ReqDcID", printable(w.locality.dcId()))
startRatekeeper is responsible for updating serverDBInfo
2019-03-23 08:56:16 +08:00
.detail("RecruitingRKID", self->recruitingRatekeeperID.present() ? self->recruitingRatekeeperID.get() : UID());
ensure that the processId exists in id_worker before accessing it
2019-03-23 09:54:39 +08:00
self->id_worker[w.locality.processId()].haltRatekeeper = brokenPromiseToNever(req.ratekeeperInterf.get().haltRatekeeper.getReply(HaltRatekeeperRequest(self->id)));
startRatekeeper is responsible for updating serverDBInfo
2019-03-23 08:56:16 +08:00
} else if(!self->recruitingRatekeeperID.present()) {
Fix Ratekeeper and DataDistributor placement Make sure both RateKeeper and DataDistributor are placed in the same data center as the Master. Make sure only one RateKeeper is live in the cluster as well.
2019-03-16 08:06:15 +08:00
const RatekeeperInterface& rki = req.ratekeeperInterf.get();
Move ratekeeper or data distributor from excluded servers
2019-03-22 08:07:12 +08:00
const auto& ratekeeper = self->db.serverInfo->get().ratekeeper;
Fix test stuck due to invalid worker in cluster controller Test case: -r simulation --crash -f ./tests/rare/CloggedCycleWithKills.txt -s 688927581 -b off
2019-03-21 13:18:53 +08:00
TraceEvent("CC_RegisterRatekeeper", self->id).detail("RKID", rki.id());
ensure that the processId exists in id_worker before accessing it
2019-03-23 09:54:39 +08:00
if (ratekeeper.present() && ratekeeper.get().id() != rki.id() && self->id_worker.count(ratekeeper.get().locality.processId())) {
Restore master PID in cluster controller worker registration CC may think master failed and clear the master PID, which can block both data distributor and ratekeeper recruitment. Fix by restoring it during worker registration.
2019-03-23 05:45:12 +08:00
TraceEvent("CC_HaltRatekeeper", self->id).detail("RKID", ratekeeper.get().id())
.detail("DcID", printable(self->clusterControllerDcId))
ensure that the processId exists in id_worker before accessing it
2019-03-23 09:54:39 +08:00
.detail("ReqDcID", printable(w.locality.dcId()))
Restore master PID in cluster controller worker registration CC may think master failed and clear the master PID, which can block both data distributor and ratekeeper recruitment. Fix by restoring it during worker registration.
2019-03-23 05:45:12 +08:00
.detail("RecruitingRKID", self->recruitingRatekeeperID.present() ? self->recruitingRatekeeperID.get() : UID());
Add actors to store halt request futures Address best fitness in checking better DD or RK.
2019-03-23 09:00:16 +08:00
self->id_worker[ratekeeper.get().locality.processId()].haltRatekeeper = brokenPromiseToNever(ratekeeper.get().haltRatekeeper.getReply(HaltRatekeeperRequest(self->id)));
Fix ratekeeper recruitment bug
2019-03-21 04:54:15 +08:00
}
startRatekeeper is responsible for updating serverDBInfo
2019-03-23 08:56:16 +08:00
if(!ratekeeper.present() || ratekeeper.get().id() != rki.id()) {
self->db.setRatekeeper(rki);
Fix ratekeeper recruitment bug
2019-03-21 04:54:15 +08:00
}
Fix Ratekeeper and DataDistributor placement Make sure both RateKeeper and DataDistributor are placed in the same data center as the Master. Make sure only one RateKeeper is live in the cluster as well.
2019-03-16 08:06:15 +08:00
}
Piggyback data distributor interface in worker registration This allows cluster controller to know data distributor during worker registration phase, thus avoiding recruiting a new data distributor after starting. Also change the worker to skip creating a new data distributor if there is already one running on the worker, which can trigger operation timeout in tests.
2019-01-29 03:29:39 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Fixing PR comments <rdar://problem/34557380> Need a way to map real time to version
2017-09-28 07:31:38 +08:00
#define TIME_KEEPER_VERSION LiteralStringRef("1")
ACTOR Future<Void> timeKeeperSetVersion(ClusterControllerData *self) {
Merge branch 'release-5.1' into feature-remote-logs # Conflicts: # fdbclient/NativeAPI.actor.cpp # fdbrpc/Locality.h # fdbrpc/simulator.h # fdbserver/ApplyMetadataMutation.h # fdbserver/ClusterController.actor.cpp # fdbserver/LogSystemPeekCursor.actor.cpp # fdbserver/MasterProxyServer.actor.cpp # fdbserver/SimulatedCluster.actor.cpp # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/masterserver.actor.cpp # flow/Net2.actor.cpp # tests/fast/SidebandWithStatus.txt # tests/rare/LargeApiCorrectnessStatus.txt # tests/slow/DDBalanceAndRemoveStatus.txt
2018-01-06 03:33:42 +08:00
state Reference<ReadYourWritesTransaction> tr = Reference<ReadYourWritesTransaction>(new ReadYourWritesTransaction(self->cx));
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
loop {
try {
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr->setOption(FDBTransactionOptions::LOCK_AWARE);
make timekeeper priority system immediate
2018-01-09 10:21:00 +08:00
tr->setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
tr->set(timeKeeperVersionKey, TIME_KEEPER_VERSION);
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr->commit());
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
break;
} catch (Error &e) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr->onError(e));
Fixing PR comments <rdar://problem/34557380> Need a way to map real time to version
2017-09-28 07:31:38 +08:00
}
}
return Void();
}
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
// This actor periodically gets read version and writes it to cluster with current timestamp as key. To avoid running
// out of space, it limits the max number of entries and clears old entries on each update. This mapping is used from
// backup and restore to get the version information for a timestamp.
ACTOR Future<Void> timeKeeper(ClusterControllerData *self) {
state KeyBackedMap<int64_t, Version> versionMap(timeKeeperPrefixRange.begin);
call debug transaction on timekeeper
2018-11-03 03:56:29 +08:00
TraceEvent("TimeKeeperStarted");
Fixing PR comments <rdar://problem/34557380> Need a way to map real time to version
2017-09-28 07:31:38 +08:00
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(timeKeeperSetVersion(self));
Fixing PR comments <rdar://problem/34557380> Need a way to map real time to version
2017-09-28 07:31:38 +08:00
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
loop {
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
state Reference<ReadYourWritesTransaction> tr = Reference<ReadYourWritesTransaction>(new ReadYourWritesTransaction(self->cx));
loop {
try {
call debug transaction on timekeeper
2018-11-03 03:56:29 +08:00
if(!g_network->isSimulated()) {
addressed review comments
2018-11-05 12:26:23 +08:00
// This is done to provide an arbitrary logged transaction every ~10s.
// FIXME: replace or augment this with logging on the proxy which tracks
// how long it is taking to hear responses from each other component.
call debug transaction on timekeeper
2018-11-03 03:56:29 +08:00
UID debugID = g_random->randomUniqueID();
Renamed TimeKeeperStarted to TimeKeeperCommit
2018-11-03 03:57:03 +08:00
TraceEvent("TimeKeeperCommit", debugID);
call debug transaction on timekeeper
2018-11-03 03:56:29 +08:00
tr->debugTransaction(debugID);
}
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr->setOption(FDBTransactionOptions::LOCK_AWARE);
make timekeeper priority system immediate
2018-01-09 10:21:00 +08:00
tr->setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
Optional<Value> disableValue = wait( tr->get(timeKeeperDisableKey) );
if(disableValue.present()) {
break;
}
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
Version v = tr->getReadVersion().get();
int64_t currentTime = (int64_t)now();
versionMap.set(tr, currentTime, v);
added the ability to disable timeKeeper; disabled timeKeeper before consistency check in simulation
2017-09-29 04:13:24 +08:00
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
int64_t ttl = currentTime - SERVER_KNOBS->TIME_KEEPER_DELAY * SERVER_KNOBS->TIME_KEEPER_MAX_ENTRIES;
if (ttl > 0) {
versionMap.erase(tr, 0, ttl);
added the ability to disable timeKeeper; disabled timeKeeper before consistency check in simulation
2017-09-29 04:13:24 +08:00
}
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr->commit());
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
break;
} catch (Error &e) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr->onError(e));
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
}
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(delay(SERVER_KNOBS->TIME_KEEPER_DELAY));
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
}
}
Initial repository commit
2017-05-26 04:48:44 +08:00
ACTOR Future<Void> statusServer(FutureStream< StatusRequest> requests,
ClusterControllerData *self,
ServerCoordinators coordinators)
{
// Seconds since the END of the last GetStatus executed
state double last_request_time = 0.0;
// Place to accumulate a batch of requests to respond to
state std::vector<StatusRequest> requests_batch;
loop {
try {
// Wait til first request is ready
StatusRequest req = waitNext(requests);
requests_batch.push_back(req);
// Earliest time at which we may begin a new request
double next_allowed_request_time = last_request_time + SERVER_KNOBS->STATUS_MIN_TIME_BETWEEN_REQUESTS;
// Wait if needed to satisfy min_time knob, also allows more requets to queue up.
double minwait = std::max(next_allowed_request_time - now(), 0.0);
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(delay(minwait));
Initial repository commit
2017-05-26 04:48:44 +08:00
// Get all requests that are ready right *now*, before GetStatus() begins.
// All of these requests will be responded to with the next GetStatus() result.
Throttle status requests with MAX_STATUS_REQUESTS_PER_SECOND knob, whenever status batching is used.
2019-01-29 07:37:30 +08:00
// If requests are batched, do not respond to more than MAX_STATUS_REQUESTS_PER_SECOND
// requests per second
while (requests.isReady()) {
auto req = requests.pop();
if (SERVER_KNOBS->STATUS_MIN_TIME_BETWEEN_REQUESTS > 0.0 &&
requests_batch.size() + 1 >
SERVER_KNOBS->STATUS_MIN_TIME_BETWEEN_REQUESTS * SERVER_KNOBS->MAX_STATUS_REQUESTS_PER_SECOND) {
Increase severity to SevWarnAlways for TooManyStatusRequests trace Co-Authored-By: tclinken <trevorclinkenbeard@gmail.com>
2019-01-29 09:50:50 +08:00
TraceEvent(SevWarnAlways, "TooManyStatusRequests").suppressFor(1.0).detail("BatchSize", requests_batch.size());
Throttle status requests with MAX_STATUS_REQUESTS_PER_SECOND knob, whenever status batching is used.
2019-01-29 07:37:30 +08:00
req.reply.sendError(server_overloaded());
} else {
requests_batch.push_back(req);
}
}
Initial repository commit
2017-05-26 04:48:44 +08:00
// Get status but trap errors to send back to client.
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
vector<WorkerDetails> workers;
Initial repository commit
2017-05-26 04:48:44 +08:00
for(auto& it : self->id_worker)
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
workers.push_back(it.second.details);
Initial repository commit
2017-05-26 04:48:44 +08:00
std::vector<NetworkAddress> incompatibleConnections;
for(auto it = self->db.incompatibleConnections.begin(); it != self->db.incompatibleConnections.end();) {
if(it->second < now()) {
it = self->db.incompatibleConnections.erase(it);
} else {
incompatibleConnections.push_back(it->first);
it++;
}
}
Status: TLS client status Use ClientStatusInfo structure for each network address (client), instead of passing each status info as a parameter.
2019-02-27 08:20:05 +08:00
state ErrorOr<StatusReply> result = wait(errorOr(clusterGetStatus(self->db.serverInfo, self->cx, workers, self->db.workersWithIssues, self->db.clientsWithIssues, self->db.clientVersionMap, self->db.clientStatusInfoMap, coordinators, incompatibleConnections, self->datacenterVersionDifference)));
Initial repository commit
2017-05-26 04:48:44 +08:00
if (result.isError() && result.getError().code() == error_code_actor_cancelled)
throw result.getError();
// Update last_request_time now because GetStatus is finished and the delay is to be measured between requests
last_request_time = now();
while (!requests_batch.empty())
{
if (result.isError())
requests_batch.back().reply.sendError(result.getError());
else
requests_batch.back().reply.send(result.get());
requests_batch.pop_back();
fixed merge problem
2018-08-22 23:47:47 +08:00
wait( yield() );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
catch (Error &e) {
TraceEvent(SevError, "StatusServerError").error(e);
throw e;
}
}
}
ACTOR Future<Void> monitorProcessClasses(ClusterControllerData *self) {
state ReadYourWritesTransaction trVer( self->db.db );
loop {
try {
trVer.setOption( FDBTransactionOptions::ACCESS_SYSTEM_KEYS );
trVer.setOption( FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE );
Optional<Value> val = wait(trVer.get(processClassVersionKey));
if (val.present())
break;
Standalone<RangeResultRef> processClasses = wait( trVer.getRange( processClassKeys, CLIENT_KNOBS->TOO_MANY ) );
ASSERT( !processClasses.more && processClasses.size() < CLIENT_KNOBS->TOO_MANY );
trVer.clear(processClassKeys);
trVer.set(processClassVersionKey, processClassVersionValue);
for (auto it : processClasses) {
UID processUid = decodeProcessClassKeyOld(it.key);
trVer.set(processClassKeyFor(processUid.toString()), it.value);
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(trVer.commit());
Initial repository commit
2017-05-26 04:48:44 +08:00
TraceEvent("ProcessClassUpgrade");
break;
}
catch(Error &e) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( trVer.onError(e) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
loop {
state ReadYourWritesTransaction tr( self->db.db );
loop {
try {
tr.setOption( FDBTransactionOptions::ACCESS_SYSTEM_KEYS );
tr.setOption( FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE );
Standalone<RangeResultRef> processClasses = wait( tr.getRange( processClassKeys, CLIENT_KNOBS->TOO_MANY ) );
ASSERT( !processClasses.more && processClasses.size() < CLIENT_KNOBS->TOO_MANY );
if(processClasses != self->lastProcessClasses || !self->gotProcessClasses) {
self->id_class.clear();
for( int i = 0; i < processClasses.size(); i++ ) {
auto c = decodeProcessClassValue( processClasses[i].value );
ASSERT( c.classSource() != ProcessClass::CommandLineSource );
self->id_class[decodeProcessClassKey( processClasses[i].key )] = c;
}
for( auto& w : self->id_worker ) {
auto classIter = self->id_class.find(w.first);
Catch and update processClass change from DBSource
2017-09-26 01:36:03 +08:00
ProcessClass newProcessClass;
Initial repository commit
2017-05-26 04:48:44 +08:00
if( classIter != self->id_class.end() && (classIter->second.classSource() == ProcessClass::DBSource || w.second.initialClass.classType() == ProcessClass::UnsetClass) ) {
Catch and update processClass change from DBSource
2017-09-26 01:36:03 +08:00
newProcessClass = classIter->second;
Initial repository commit
2017-05-26 04:48:44 +08:00
} else {
Catch and update processClass change from DBSource
2017-09-26 01:36:03 +08:00
newProcessClass = w.second.initialClass;
}
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if (newProcessClass != w.second.details.processClass) {
w.second.details.processClass = newProcessClass;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
w.second.priorityInfo.processClassFitness = newProcessClass.machineClassFitness(ProcessClass::ClusterController);
Fix cluster controller review comments
2017-10-05 06:48:55 +08:00
if (!w.second.reply.isSet()) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
w.second.reply.send( RegisterWorkerReply(w.second.details.processClass, w.second.priorityInfo) );
Fix cluster controller review comments
2017-10-05 06:48:55 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
self->lastProcessClasses = processClasses;
self->gotProcessClasses = true;
checkOutstandingRequests( self );
}
state Future<Void> watchFuture = tr.watch(processClassChangeKey);
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr.commit());
wait(watchFuture);
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
}
catch(Error &e) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( tr.onError(e) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
}
}
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
ACTOR Future<Void> monitorServerInfoConfig(ClusterControllerData::DBInfo* db) {
loop {
state ReadYourWritesTransaction tr(db->db);
loop {
try {
tr.setOption(FDBTransactionOptions::READ_SYSTEM_KEYS);
tr.setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
tr.setOption(FDBTransactionOptions::READ_LOCK_AWARE);
Optional<Value> configVal = wait(tr.get(latencyBandConfigKey));
Optional<LatencyBandConfig> config;
if(configVal.present()) {
config = LatencyBandConfig::parse(configVal.get());
}
ServerDBInfo serverInfo = db->serverInfo->get();
if(config != serverInfo.latencyBandConfig) {
TraceEvent("LatencyBandConfigChanged").detail("Present", config.present());
serverInfo.id = g_random->randomUniqueID();
serverInfo.latencyBandConfig = config;
db->serverInfo->set(serverInfo);
}
state Future<Void> configChangeFuture = tr.watch(latencyBandConfigKey);
Merge commit '3b2700d25334c53d13496ca16682642aac951beb' into track-server-request-latencies # Conflicts: # fdbclient/MasterProxyInterface.h # fdbserver/ClusterController.actor.cpp # fdbserver/MasterProxyServer.actor.cpp # fdbserver/ServerDBInfo.h # fdbserver/Status.actor.cpp # fdbserver/fdbserver.vcxproj # fdbserver/storageserver.actor.cpp
2019-01-25 03:43:26 +08:00
wait(tr.commit());
wait(configChangeFuture);
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
break;
}
catch (Error &e) {
Merge commit '3b2700d25334c53d13496ca16682642aac951beb' into track-server-request-latencies # Conflicts: # fdbclient/MasterProxyInterface.h # fdbserver/ClusterController.actor.cpp # fdbserver/MasterProxyServer.actor.cpp # fdbserver/ServerDBInfo.h # fdbserver/Status.actor.cpp # fdbserver/fdbserver.vcxproj # fdbserver/storageserver.actor.cpp
2019-01-25 03:43:26 +08:00
wait(tr.onError(e));
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
}
}
}
}
Initial repository commit
2017-05-26 04:48:44 +08:00
ACTOR Future<Void> monitorClientTxnInfoConfigs(ClusterControllerData::DBInfo* db) {
loop {
state ReadYourWritesTransaction tr(db->db);
loop {
try {
tr.setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr.setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
Address code review comments. * Fixed memory corruption with SystemData key constants * Removed duplication in ClusterController * Reworked fdbcli actions to better represent explicit vs default assignments
2017-10-04 11:57:39 +08:00
state Optional<Value> rateVal = wait(tr.get(fdbClientInfoTxnSampleRate));
state Optional<Value> limitVal = wait(tr.get(fdbClientInfoTxnSizeLimit));
Initial repository commit
2017-05-26 04:48:44 +08:00
ClientDBInfo clientInfo = db->clientInfo->get();
onMasterProxiesChanged was being triggered when any member of ClientDBInfo changed. Change the behavior to be triggered only when proxies field in ClientDBInfo is changed.
2017-11-02 09:29:56 +08:00
double sampleRate = rateVal.present() ? BinaryReader::fromStringRef<double>(rateVal.get(), Unversioned()) : std::numeric_limits<double>::infinity();
int64_t sizeLimit = limitVal.present() ? BinaryReader::fromStringRef<int64_t>(limitVal.get(), Unversioned()) : -1;
if (sampleRate != clientInfo.clientTxnInfoSampleRate || sizeLimit != clientInfo.clientTxnInfoSampleRate) {
Initial repository commit
2017-05-26 04:48:44 +08:00
clientInfo.id = g_random->randomUniqueID();
onMasterProxiesChanged was being triggered when any member of ClientDBInfo changed. Change the behavior to be triggered only when proxies field in ClientDBInfo is changed.
2017-11-02 09:29:56 +08:00
clientInfo.clientTxnInfoSampleRate = sampleRate;
clientInfo.clientTxnInfoSizeLimit = sizeLimit;
Initial repository commit
2017-05-26 04:48:44 +08:00
db->clientInfo->set(clientInfo);
}
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
Address code review comments. * Fixed memory corruption with SystemData key constants * Removed duplication in ClusterController * Reworked fdbcli actions to better represent explicit vs default assignments
2017-10-04 11:57:39 +08:00
state Future<Void> watchRateFuture = tr.watch(fdbClientInfoTxnSampleRate);
state Future<Void> watchLimitFuture = tr.watch(fdbClientInfoTxnSizeLimit);
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr.commit());
Initial repository commit
2017-05-26 04:48:44 +08:00
choose {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when(wait(watchRateFuture)) { break; }
when (wait(watchLimitFuture)) { break; }
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
catch (Error &e) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr.onError(e));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
}
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
ACTOR Future<Void> updatedChangingDatacenters(ClusterControllerData *self) {
//do not change the cluster controller until all the processes have had a chance to register
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( delay(SERVER_KNOBS->WAIT_FOR_GOOD_RECRUITMENT_DELAY) );
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
loop {
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
state Future<Void> onChange = self->desiredDcIds.onChange();
if(!self->desiredDcIds.get().present()) {
self->changingDcIds.set(std::make_pair(false,self->desiredDcIds.get()));
} else {
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
auto& worker = self->id_worker[self->clusterControllerProcessId];
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
uint8_t newFitness = ClusterControllerPriorityInfo::calculateDCFitness( worker.details.interf.locality.dcId(), self->desiredDcIds.get().get() );
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
self->changingDcIds.set(std::make_pair(worker.priorityInfo.dcFitness > newFitness,self->desiredDcIds.get()));
added trace events for cluster controller changing datacenters
2018-07-03 01:06:54 +08:00
TraceEvent("UpdateChangingDatacenter", self->id).detail("OldFitness", worker.priorityInfo.dcFitness).detail("NewFitness", newFitness);
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
if ( worker.priorityInfo.dcFitness > newFitness ) {
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
worker.priorityInfo.dcFitness = newFitness;
if(!worker.reply.isSet()) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
worker.reply.send( RegisterWorkerReply( worker.details.processClass, worker.priorityInfo ) );
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
}
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
} else {
state int currentFit = ProcessClass::BestFit;
while(currentFit <= ProcessClass::NeverAssign) {
bool updated = false;
for ( auto& it : self->id_worker ) {
if( ( !it.second.priorityInfo.isExcluded && it.second.priorityInfo.processClassFitness == currentFit ) || currentFit == ProcessClass::NeverAssign ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
uint8_t fitness = ClusterControllerPriorityInfo::calculateDCFitness( it.second.details.interf.locality.dcId(), self->changingDcIds.get().second.get() );
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
if ( it.first != self->clusterControllerProcessId && it.second.priorityInfo.dcFitness != fitness ) {
updated = true;
it.second.priorityInfo.dcFitness = fitness;
if(!it.second.reply.isSet()) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
it.second.reply.send( RegisterWorkerReply( it.second.details.processClass, it.second.priorityInfo ) );
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
}
}
}
}
if(updated && currentFit < ProcessClass::NeverAssign) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( delay(SERVER_KNOBS->CC_CLASS_DELAY) );
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
}
currentFit++;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
}
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(onChange);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
}
ACTOR Future<Void> updatedChangedDatacenters(ClusterControllerData *self) {
state Future<Void> changeDelay = delay(SERVER_KNOBS->CC_CHANGE_DELAY);
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
state Future<Void> onChange = self->changingDcIds.onChange();
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
loop {
choose {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when( wait(onChange) ) {
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
changeDelay = delay(SERVER_KNOBS->CC_CHANGE_DELAY);
onChange = self->changingDcIds.onChange();
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when( wait(changeDelay) ) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
changeDelay = Never();
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
onChange = self->changingDcIds.onChange();
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
self->changedDcIds.set(self->changingDcIds.get());
if(self->changedDcIds.get().second.present()) {
added trace events for cluster controller changing datacenters
2018-07-03 01:06:54 +08:00
TraceEvent("UpdateChangedDatacenter", self->id).detail("CCFirst", self->changedDcIds.get().first);
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
if( !self->changedDcIds.get().first ) {
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
auto& worker = self->id_worker[self->clusterControllerProcessId];
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
uint8_t newFitness = ClusterControllerPriorityInfo::calculateDCFitness( worker.details.interf.locality.dcId(), self->changedDcIds.get().second.get() );
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
if( worker.priorityInfo.dcFitness != newFitness ) {
worker.priorityInfo.dcFitness = newFitness;
if(!worker.reply.isSet()) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
worker.reply.send( RegisterWorkerReply( worker.details.processClass, worker.priorityInfo ) );
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
} else {
state int currentFit = ProcessClass::BestFit;
while(currentFit <= ProcessClass::NeverAssign) {
bool updated = false;
for ( auto& it : self->id_worker ) {
if( ( !it.second.priorityInfo.isExcluded && it.second.priorityInfo.processClassFitness == currentFit ) || currentFit == ProcessClass::NeverAssign ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
uint8_t fitness = ClusterControllerPriorityInfo::calculateDCFitness( it.second.details.interf.locality.dcId(), self->changedDcIds.get().second.get() );
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
if ( it.first != self->clusterControllerProcessId && it.second.priorityInfo.dcFitness != fitness ) {
updated = true;
it.second.priorityInfo.dcFitness = fitness;
if(!it.second.reply.isSet()) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
it.second.reply.send( RegisterWorkerReply( it.second.details.processClass, it.second.priorityInfo ) );
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
}
}
}
}
if(updated && currentFit < ProcessClass::NeverAssign) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( delay(SERVER_KNOBS->CC_CLASS_DELAY) );
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
}
currentFit++;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
}
}
}
}
}
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
ACTOR Future<Void> updateDatacenterVersionDifference( ClusterControllerData *self ) {
fixed compile issue
2018-06-22 07:34:36 +08:00
state double lastLogTime = 0;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
loop {
self->versionDifferenceUpdated = false;
fix: kill all tlogs in one region prevents the remote logs from recovering in that region, do not allow that to prevent us from configuring usable_regions=1. added more recovery states.
2018-07-05 15:08:51 +08:00
if(self->db.serverInfo->get().recoveryState >= RecoveryState::ACCEPTING_COMMITS && self->db.config.usableRegions == 1) {
fix: treat the local region as best when version difference is too large re-check requests when the version difference becomes small
2018-07-07 05:44:11 +08:00
bool oldDifferenceTooLarge = !self->versionDifferenceUpdated || self->datacenterVersionDifference >= SERVER_KNOBS->MAX_VERSION_DIFFERENCE;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
self->versionDifferenceUpdated = true;
self->datacenterVersionDifference = 0;
fix: treat the local region as best when version difference is too large re-check requests when the version difference becomes small
2018-07-07 05:44:11 +08:00
if(oldDifferenceTooLarge) {
checkOutstandingRequests(self);
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(self->db.serverInfo->onChange());
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
continue;
}
state Optional<TLogInterface> primaryLog;
state Optional<TLogInterface> remoteLog;
tlogs serve reads to log routers at a low priority, to prevent them from using all their resources catching up a remote dc that has been down for a long time increase the amount of memory ratekeeper budgets for tlogs so that there is a gap after the spill threshold to prevent temporarily overshooting the budget
2018-08-05 01:31:30 +08:00
if(self->db.serverInfo->get().recoveryState >= RecoveryState::ALL_LOGS_RECRUITED) {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
for(auto& logSet : self->db.serverInfo->get().logSystemConfig.tLogs) {
removed hasBestPolicy
2018-06-16 03:36:19 +08:00
if(logSet.isLocal && logSet.locality != tagLocalitySatellite) {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
for(auto& tLog : logSet.tLogs) {
if(tLog.present()) {
primaryLog = tLog.interf();
break;
}
}
}
if(!logSet.isLocal) {
for(auto& tLog : logSet.tLogs) {
if(tLog.present()) {
remoteLog = tLog.interf();
break;
}
}
}
}
}
if(!primaryLog.present() || !remoteLog.present()) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(self->db.serverInfo->onChange());
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
continue;
}
added logging for the datacenter version difference
2018-06-22 07:31:52 +08:00
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
state Future<Void> onChange = self->db.serverInfo->onChange();
loop {
fix: getting metrics did not handle broken_promise errors
2018-07-06 03:30:11 +08:00
state Future<TLogQueuingMetricsReply> primaryMetrics = brokenPromiseToNever( primaryLog.get().getQueuingMetrics.getReply( TLogQueuingMetricsRequest() ) );
state Future<TLogQueuingMetricsReply> remoteMetrics = brokenPromiseToNever( remoteLog.get().getQueuingMetrics.getReply( TLogQueuingMetricsRequest() ) );
added logging for the datacenter version difference
2018-06-22 07:31:52 +08:00
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( ( success(primaryMetrics) && success(remoteMetrics) ) || onChange );
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
if(onChange.isReady()) {
break;
}
fix: treat the local region as best when version difference is too large re-check requests when the version difference becomes small
2018-07-07 05:44:11 +08:00
bool oldDifferenceTooLarge = !self->versionDifferenceUpdated || self->datacenterVersionDifference >= SERVER_KNOBS->MAX_VERSION_DIFFERENCE;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
self->versionDifferenceUpdated = true;
self->datacenterVersionDifference = primaryMetrics.get().v - remoteMetrics.get().v;
fix: treat the local region as best when version difference is too large re-check requests when the version difference becomes small
2018-07-07 05:44:11 +08:00
if(oldDifferenceTooLarge && self->datacenterVersionDifference < SERVER_KNOBS->MAX_VERSION_DIFFERENCE) {
checkOutstandingRequests(self);
}
added logging for the datacenter version difference
2018-06-22 07:31:52 +08:00
if(now() - lastLogTime > SERVER_KNOBS->CLUSTER_CONTROLLER_LOGGING_DELAY) {
lastLogTime = now();
TraceEvent("DatacenterVersionDifference", self->id).detail("Difference", self->datacenterVersionDifference);
}
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( delay(SERVER_KNOBS->VERSION_LAG_METRIC_INTERVAL) || onChange );
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
if(onChange.isReady()) {
break;
}
}
}
}
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
ACTOR Future<Void> doEmptyCommit(Database cx) {
state Transaction tr(cx);
loop {
try {
tr.setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
tr.setOption(FDBTransactionOptions::LOCK_AWARE);
tr.makeSelfConflicting();
wait(tr.commit());
return Void();
} catch( Error &e ) {
wait( tr.onError(e) );
}
}
}
ACTOR Future<Void> handleForcedRecoveries( ClusterControllerData *self, ClusterControllerFullInterface interf ) {
loop {
state ForceRecoveryRequest req = waitNext( interf.clientInterface.forceRecovery.getFuture() );
TraceEvent("ForcedRecoveryStart", self->id).detail("ClusterControllerDcId", printable(self->clusterControllerDcId)).detail("DcId", req.dcId.printable());
state Future<Void> fCommit = doEmptyCommit(self->cx);
knobify force recovery recovery check delay
2019-02-20 08:05:20 +08:00
wait(fCommit || delay(SERVER_KNOBS->FORCE_RECOVERY_CHECK_DELAY));
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
if(!fCommit.isReady() || fCommit.isError()) {
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
if (self->clusterControllerDcId != req.dcId) {
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
vector<Optional<Key>> dcPriority;
dcPriority.push_back(req.dcId);
dcPriority.push_back(self->clusterControllerDcId);
self->desiredDcIds.set(dcPriority);
} else {
self->db.forceRecovery = true;
self->db.forceMasterFailure.trigger();
}
wait(fCommit);
}
TraceEvent("ForcedRecoveryFinish", self->id);
self->db.forceRecovery = false;
req.reply.send(Void());
forgot to fix merge conflicts
2019-02-19 09:09:55 +08:00
}
}
Add a new DataDistributor role. Let cluster controller to start a new data distributor role by sending a message to a chosen worker. Change MasterInterface usage in DataDistribution to masterId Add DataDistributor rejoin handling. This allows the data distributor to tell the new cluster controller of its existence so that the controller doesn't spawn a new one. I.e., there should be only ONE data distributor in the cluster. If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries to recruit one as DD. CC also monitors DD and restarts one if it failed. The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for the new DD. Add GetRecoveryInfo RPC to master server, which is called by data distributor to obtain the recovery Transaction version from the master server.
2018-12-14 05:31:37 +08:00
ACTOR Future<DataDistributorInterface> startDataDistributor( ClusterControllerData *self ) {
Fix a race of DD, RK, Master failure If all DD, RK, Master run on the same process and failed. Recruiting of new DD or RK could try to use the old master worker interface, which is an invalid one and causes recruitment to be stuck. Fix by adding a delay and checking master is valid before recruitment.
2019-03-21 07:03:36 +08:00
wait(delay(0.0)); // If master fails at the same time, give it a chance to clear master PID.
Add a new DataDistributor role. Let cluster controller to start a new data distributor role by sending a message to a chosen worker. Change MasterInterface usage in DataDistribution to masterId Add DataDistributor rejoin handling. This allows the data distributor to tell the new cluster controller of its existence so that the controller doesn't spawn a new one. I.e., there should be only ONE data distributor in the cluster. If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries to recruit one as DD. CC also monitors DD and restarts one if it failed. The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for the new DD. Add GetRecoveryInfo RPC to master server, which is called by data distributor to obtain the recovery Transaction version from the master server.
2018-12-14 05:31:37 +08:00
Restore master PID in cluster controller worker registration CC may think master failed and clear the master PID, which can block both data distributor and ratekeeper recruitment. Fix by restoring it during worker registration.
2019-03-23 05:45:12 +08:00
TraceEvent("CC_StartDataDistributor", self->id);
Update fdbserver/ClusterController.actor.cpp Co-Authored-By: jzhou77 <jingyuzhou@gmail.com>
2019-02-14 03:54:35 +08:00
loop {
Fix segfault and no_more_servers errors
2019-01-31 01:05:12 +08:00
try {
Avoid unnecessary recuriting of DD or RK While waiting for recruting data distributor or ratekeeper, a previous one could already joined. So we can skip this unnecessary recruiting. Revert the change of worker.actor.cpp for ratekeeper. Instead, recruiting ratekeeper should avoid the process with an existing one. This fixes a bug where the ratekeeper interface became zombie, killing other healthy ratekeeper but doing no useful work. Found by: -r simulation --crash -f tests/fast/WriteDuringRead.txt -s 31858110 -b on
2019-03-22 13:20:00 +08:00
state bool no_distributor = !self->db.serverInfo->get().distributor.present();
Fix a race of DD, RK, Master failure If all DD, RK, Master run on the same process and failed. Recruiting of new DD or RK could try to use the old master worker interface, which is an invalid one and causes recruitment to be stuck. Fix by adding a delay and checking master is valid before recruitment.
2019-03-21 07:03:36 +08:00
while (!self->masterProcessId.present() || self->masterProcessId != self->db.serverInfo->get().master.locality.processId() || self->db.serverInfo->get().recoveryState < RecoveryState::ACCEPTING_COMMITS) {
wait(self->db.serverInfo->onChange() || delay(SERVER_KNOBS->WAIT_FOR_GOOD_RECRUITMENT_DELAY));
Fix segfault and no_more_servers errors
2019-01-31 01:05:12 +08:00
}
Avoid unnecessary recuriting of DD or RK While waiting for recruting data distributor or ratekeeper, a previous one could already joined. So we can skip this unnecessary recruiting. Revert the change of worker.actor.cpp for ratekeeper. Instead, recruiting ratekeeper should avoid the process with an existing one. This fixes a bug where the ratekeeper interface became zombie, killing other healthy ratekeeper but doing no useful work. Found by: -r simulation --crash -f tests/fast/WriteDuringRead.txt -s 31858110 -b on
2019-03-22 13:20:00 +08:00
if (no_distributor && self->db.serverInfo->get().distributor.present()) {
return self->db.serverInfo->get().distributor.get();
}
Per review comments Add a knob for starting distributor delay. Move distributor failed variable to a local loop.
2019-02-01 02:51:25 +08:00
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
std::map<Optional<Standalone<StringRef>>, int> id_used = self->getUsedIds();
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
WorkerFitnessInfo data_distributor = self->getWorkerForRoleInDatacenter(self->clusterControllerDcId, ProcessClass::DataDistributor, ProcessClass::NeverAssign, self->db.config, id_used);
state WorkerDetails worker = data_distributor.worker;
if (self->onMasterIsBetter(worker, ProcessClass::DataDistributor)) {
worker = self->id_worker[self->masterProcessId.get()].details;
}
Recruit a new ratekeeper before halting the old
2019-03-20 06:21:46 +08:00
InitializeDataDistributorRequest req(g_random->randomUniqueID());
Move ratekeeper or data distributor from excluded servers
2019-03-22 08:07:12 +08:00
TraceEvent("CC_DataDistributorRecruit", self->id).detail("Addr", worker.interf.address());
Fix segfault and no_more_servers errors
2019-01-31 01:05:12 +08:00
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
ErrorOr<DataDistributorInterface> distributor = wait( worker.interf.dataDistributor.getReplyUnlessFailedFor(req, SERVER_KNOBS->WAIT_FOR_DISTRIBUTOR_JOIN_DELAY, 0) );
Fix segfault and no_more_servers errors
2019-01-31 01:05:12 +08:00
if (distributor.present()) {
Move ratekeeper or data distributor from excluded servers
2019-03-22 08:07:12 +08:00
TraceEvent("CC_DataDistributorRecruited", self->id).detail("Addr", worker.interf.address());
Fix segfault and no_more_servers errors
2019-01-31 01:05:12 +08:00
return distributor.get();
}
}
catch (Error& e) {
Move ratekeeper or data distributor from excluded servers
2019-03-22 08:07:12 +08:00
TraceEvent("CC_DataDistributorRecruitError", self->id).error(e);
Fix segfault and no_more_servers errors
2019-01-31 01:05:12 +08:00
if ( e.code() != error_code_no_more_servers ) {
throw;
}
}
Fix an out-of-memory error
2019-02-11 02:19:36 +08:00
wait( delay(SERVER_KNOBS->ATTEMPT_RECRUITMENT_DELAY) );
Add a new DataDistributor role. Let cluster controller to start a new data distributor role by sending a message to a chosen worker. Change MasterInterface usage in DataDistribution to masterId Add DataDistributor rejoin handling. This allows the data distributor to tell the new cluster controller of its existence so that the controller doesn't spawn a new one. I.e., there should be only ONE data distributor in the cluster. If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries to recruit one as DD. CC also monitors DD and restarts one if it failed. The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for the new DD. Add GetRecoveryInfo RPC to master server, which is called by data distributor to obtain the recovery Transaction version from the master server.
2018-12-14 05:31:37 +08:00
}
}
Fix a segfault bug due to uncopied ratekeeper interface
2019-02-19 06:57:21 +08:00
ACTOR Future<Void> monitorDataDistributor(ClusterControllerData *self) {
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
state Future<Void> initialDelay = delay(SERVER_KNOBS->WAIT_FOR_DISTRIBUTOR_JOIN_DELAY);
Add a new DataDistributor role. Let cluster controller to start a new data distributor role by sending a message to a chosen worker. Change MasterInterface usage in DataDistribution to masterId Add DataDistributor rejoin handling. This allows the data distributor to tell the new cluster controller of its existence so that the controller doesn't spawn a new one. I.e., there should be only ONE data distributor in the cluster. If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries to recruit one as DD. CC also monitors DD and restarts one if it failed. The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for the new DD. Add GetRecoveryInfo RPC to master server, which is called by data distributor to obtain the recovery Transaction version from the master server.
2018-12-14 05:31:37 +08:00
// wait for a while to see if existing data distributor will join.
loop choose {
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
when ( wait(initialDelay) ) { break; }
when ( wait(self->db.serverInfo->onChange()) ) { // Rejoins via worker registration
if ( self->db.serverInfo->get().distributor.present() ) {
Move ratekeeper or data distributor from excluded servers
2019-03-22 08:07:12 +08:00
TraceEvent("CC_InfoChange", self->id)
.detail("DDID", self->db.serverInfo->get().distributor.get().id());
Fix wait failure bug on cluster controller The setDistributor() sets an AsyncVar and then runs waitFailureClient. This ordering is wrong because the AsyncVar::set triggers the other loop to run first, which will wait on Never(). The correct code should wait on the Future returned by the waitFailureClient.
2019-01-30 02:14:11 +08:00
break;
}
Piggyback data distributor interface in worker registration This allows cluster controller to know data distributor during worker registration phase, thus avoiding recruiting a new data distributor after starting. Also change the worker to skip creating a new data distributor if there is already one running on the worker, which can trigger operation timeout in tests.
2019-01-29 03:29:39 +08:00
}
Add a new DataDistributor role. Let cluster controller to start a new data distributor role by sending a message to a chosen worker. Change MasterInterface usage in DataDistribution to masterId Add DataDistributor rejoin handling. This allows the data distributor to tell the new cluster controller of its existence so that the controller doesn't spawn a new one. I.e., there should be only ONE data distributor in the cluster. If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries to recruit one as DD. CC also monitors DD and restarts one if it failed. The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for the new DD. Add GetRecoveryInfo RPC to master server, which is called by data distributor to obtain the recovery Transaction version from the master server.
2018-12-14 05:31:37 +08:00
}
Per review comments Add a knob for starting distributor delay. Move distributor failed variable to a local loop.
2019-02-01 02:51:25 +08:00
loop {
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
if ( self->db.serverInfo->get().distributor.present() ) {
wait( waitFailureClient( self->db.serverInfo->get().distributor.get().waitFailure, SERVER_KNOBS->DD_FAILURE_TIME ) );
TraceEvent("ClusterController", self->id)
.detail("DataDistributorDied", self->db.serverInfo->get().distributor.get().id());
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
self->db.clearInterf(ProcessClass::DataDistributorClass);
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
} else {
Fix the issue of slow dying Data Distributor Test with: -r simulation -f ./foundationdb/tests/slow/CommitBug.txt -s 67828576 -b on The test has the following event sequence: - Time 113.3s, CC noticed DD failure, cleard DD interface. - 1s later, DD rejoined and registered with CC. - Time 131.7s, DD actor cancelled. This old DD raced to register with CC and the failure monitor is not installed because monitorDataDistributor is stalled waiting for new DD. - Time 161.4s, new DD running. New DD recruting was delayed due to no servers in the period. Fix by disabling DD registration during the recruting process.
2019-03-18 11:55:59 +08:00
self->recruitingDistributor = true;
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
DataDistributorInterface distributorInterf = wait( startDataDistributor(self) );
Fix the issue of slow dying Data Distributor Test with: -r simulation -f ./foundationdb/tests/slow/CommitBug.txt -s 67828576 -b on The test has the following event sequence: - Time 113.3s, CC noticed DD failure, cleard DD interface. - 1s later, DD rejoined and registered with CC. - Time 131.7s, DD actor cancelled. This old DD raced to register with CC and the failure monitor is not installed because monitorDataDistributor is stalled waiting for new DD. - Time 161.4s, new DD running. New DD recruting was delayed due to no servers in the period. Fix by disabling DD registration during the recruting process.
2019-03-18 11:55:59 +08:00
self->recruitingDistributor = false;
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
self->db.setDistributor(distributorInterf);
}
}
}
Fix ratekeeper recruitment bug
2019-03-21 04:54:15 +08:00
ACTOR Future<Void> startRatekeeper(ClusterControllerData *self) {
Fix a race of DD, RK, Master failure If all DD, RK, Master run on the same process and failed. Recruiting of new DD or RK could try to use the old master worker interface, which is an invalid one and causes recruitment to be stuck. Fix by adding a delay and checking master is valid before recruitment.
2019-03-21 07:03:36 +08:00
wait(delay(0.0)); // If master fails at the same time, give it a chance to clear master PID.
Restore master PID in cluster controller worker registration CC may think master failed and clear the master PID, which can block both data distributor and ratekeeper recruitment. Fix by restoring it during worker registration.
2019-03-23 05:45:12 +08:00
TraceEvent("CC_StartRatekeeper", self->id);
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
loop {
try {
Avoid unnecessary recuriting of DD or RK While waiting for recruting data distributor or ratekeeper, a previous one could already joined. So we can skip this unnecessary recruiting. Revert the change of worker.actor.cpp for ratekeeper. Instead, recruiting ratekeeper should avoid the process with an existing one. This fixes a bug where the ratekeeper interface became zombie, killing other healthy ratekeeper but doing no useful work. Found by: -r simulation --crash -f tests/fast/WriteDuringRead.txt -s 31858110 -b on
2019-03-22 13:20:00 +08:00
state bool no_ratekeeper = !self->db.serverInfo->get().ratekeeper.present();
Fix a race of DD, RK, Master failure If all DD, RK, Master run on the same process and failed. Recruiting of new DD or RK could try to use the old master worker interface, which is an invalid one and causes recruitment to be stuck. Fix by adding a delay and checking master is valid before recruitment.
2019-03-21 07:03:36 +08:00
while (!self->masterProcessId.present() || self->masterProcessId != self->db.serverInfo->get().master.locality.processId() || self->db.serverInfo->get().recoveryState < RecoveryState::ACCEPTING_COMMITS) {
wait(self->db.serverInfo->onChange() || delay(SERVER_KNOBS->WAIT_FOR_GOOD_RECRUITMENT_DELAY));
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
}
Avoid unnecessary recuriting of DD or RK While waiting for recruting data distributor or ratekeeper, a previous one could already joined. So we can skip this unnecessary recruiting. Revert the change of worker.actor.cpp for ratekeeper. Instead, recruiting ratekeeper should avoid the process with an existing one. This fixes a bug where the ratekeeper interface became zombie, killing other healthy ratekeeper but doing no useful work. Found by: -r simulation --crash -f tests/fast/WriteDuringRead.txt -s 31858110 -b on
2019-03-22 13:20:00 +08:00
if (no_ratekeeper && self->db.serverInfo->get().ratekeeper.present()) {
// Existing ratekeeper registers while waiting, so skip.
return Void();
}
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
std::map<Optional<Standalone<StringRef>>, int> id_used = self->getUsedIds();
Recruit a new ratekeeper before halting the old
2019-03-20 06:21:46 +08:00
WorkerFitnessInfo rkWorker = self->getWorkerForRoleInDatacenter(self->clusterControllerDcId, ProcessClass::RateKeeper, ProcessClass::NeverAssign, self->db.config, id_used);
InitializeRatekeeperRequest req(g_random->randomUniqueID());
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
state WorkerDetails worker = rkWorker.worker;
if (self->onMasterIsBetter(worker, ProcessClass::RateKeeper)) {
worker = self->id_worker[self->masterProcessId.get()].details;
}
Avoid unnecessary recuriting of DD or RK While waiting for recruting data distributor or ratekeeper, a previous one could already joined. So we can skip this unnecessary recruiting. Revert the change of worker.actor.cpp for ratekeeper. Instead, recruiting ratekeeper should avoid the process with an existing one. This fixes a bug where the ratekeeper interface became zombie, killing other healthy ratekeeper but doing no useful work. Found by: -r simulation --crash -f tests/fast/WriteDuringRead.txt -s 31858110 -b on
2019-03-22 13:20:00 +08:00
if (self->db.serverInfo->get().ratekeeper.present() && self->db.serverInfo->get().ratekeeper.get().locality.processId() == worker.interf.locality.processId()) {
throw no_more_servers(); // Avoid recruiting an existing one.
}
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
self->recruitingRatekeeperID = req.reqId;
Fix ratekeeper and data distributor recruiting bug Avoid multiple concurrent recuriting of ratekeepers with a recruiting flag. Fix endless recruiting when the chosen worker is a proxy or a resolver -- prefer master in this case.
2019-03-21 01:00:31 +08:00
TraceEvent("ClusterController_RecruitRatekeeper", self->id).detail("Addr", worker.interf.address()).detail("RKID", req.reqId);
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
ErrorOr<RatekeeperInterface> interf = wait( worker.interf.ratekeeper.getReplyUnlessFailedFor(req, SERVER_KNOBS->WAIT_FOR_RATEKEEPER_JOIN_DELAY, 0) );
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
if (interf.present()) {
Fix ratekeeper recruitment bug
2019-03-21 04:54:15 +08:00
TraceEvent("ClusterController_RatekeeperRecruited", self->id).detail("Addr", worker.interf.address());
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
self->recruitRatekeeper.set(false);
startRatekeeper is responsible for updating serverDBInfo
2019-03-23 08:56:16 +08:00
self->recruitingRatekeeperID = Optional<UID>();
const auto& ratekeeper = self->db.serverInfo->get().ratekeeper;
TraceEvent("CC_RegisterRatekeeper", self->id).detail("RKID", interf.get().id());
ensure that the processId exists in id_worker before accessing it
2019-03-23 09:54:39 +08:00
if (ratekeeper.present() && ratekeeper.get().id() != interf.get().id() && self->id_worker.count(ratekeeper.get().locality.processId())) {
startRatekeeper is responsible for updating serverDBInfo
2019-03-23 08:56:16 +08:00
TraceEvent("CC_HaltRatekeeper", self->id).detail("RKID", ratekeeper.get().id())
.detail("DcID", printable(self->clusterControllerDcId));
Add actors to store halt request futures Address best fitness in checking better DD or RK.
2019-03-23 09:00:16 +08:00
self->id_worker[ratekeeper.get().locality.processId()].haltRatekeeper = brokenPromiseToNever(ratekeeper.get().haltRatekeeper.getReply(HaltRatekeeperRequest(self->id)));
startRatekeeper is responsible for updating serverDBInfo
2019-03-23 08:56:16 +08:00
}
if(!ratekeeper.present() || ratekeeper.get().id() != interf.get().id()) {
self->db.setRatekeeper(interf.get());
}
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
checkOutstandingRequests(self);
Fix ratekeeper recruitment bug
2019-03-21 04:54:15 +08:00
return Void();
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
}
}
catch (Error& e) {
Fix review comment for PR 1176
2019-03-13 02:34:16 +08:00
TraceEvent("ClusterController_RatekeeperRecruitError", self->id).error(e);
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
if ( e.code() != error_code_no_more_servers ) {
throw;
}
}
wait( delay(SERVER_KNOBS->ATTEMPT_RECRUITMENT_DELAY) );
}
}
Minor fix on ratekeeper work registration.
2019-02-16 09:29:52 +08:00
ACTOR Future<Void> monitorRatekeeper(ClusterControllerData *self) {
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
state Future<Void> initialDelay = delay(SERVER_KNOBS->WAIT_FOR_RATEKEEPER_JOIN_DELAY);
// wait for a while to see if an existing ratekeeper will join.
loop choose {
when ( wait(initialDelay) ) { break; }
when ( wait(self->db.serverInfo->onChange()) ) { // Rejoins via worker registration
if ( self->db.serverInfo->get().ratekeeper.present() ) {
TraceEvent("ClusterController_GotRateKeeper", self->id)
.detail("RKID", self->db.serverInfo->get().ratekeeper.get().id());
break;
}
}
}
loop {
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
if ( self->db.serverInfo->get().ratekeeper.present() && !self->recruitRatekeeper.get() ) {
choose {
when(wait(waitFailureClient( self->db.serverInfo->get().ratekeeper.get().waitFailure, SERVER_KNOBS->RATEKEEPER_FAILURE_TIME ))) {
TraceEvent("ClusterController_RateKeeperDied", self->id)
.detail("RKID", self->db.serverInfo->get().ratekeeper.get().id());
self->db.clearInterf(ProcessClass::RateKeeperClass);
Fix ratekeeper and data distributor recruiting bug Avoid multiple concurrent recuriting of ratekeepers with a recruiting flag. Fix endless recruiting when the chosen worker is a proxy or a resolver -- prefer master in this case.
2019-03-21 01:00:31 +08:00
}
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
when(wait(self->recruitRatekeeper.onChange())) {}
Recruit a new ratekeeper before halting the old
2019-03-20 06:21:46 +08:00
}
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
} else {
wait( startRatekeeper(self) );
Add a new DataDistributor role. Let cluster controller to start a new data distributor role by sending a message to a chosen worker. Change MasterInterface usage in DataDistribution to masterId Add DataDistributor rejoin handling. This allows the data distributor to tell the new cluster controller of its existence so that the controller doesn't spawn a new one. I.e., there should be only ONE data distributor in the cluster. If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries to recruit one as DD. CC also monitors DD and restarts one if it failed. The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for the new DD. Add GetRecoveryInfo RPC to master server, which is called by data distributor to obtain the recovery Transaction version from the master server.
2018-12-14 05:31:37 +08:00
}
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
}
}
fix: the cluster controller did not pass in its own locality when creating its database object, therefore it was not using locality aware load balancing
2018-09-29 03:12:06 +08:00
ACTOR Future<Void> clusterControllerCore( ClusterControllerFullInterface interf, Future<Void> leaderFail, ServerCoordinators coordinators, LocalityData locality ) {
state ClusterControllerData self( interf, locality );
Initial repository commit
2017-05-26 04:48:44 +08:00
state Future<Void> coordinationPingDelay = delay( SERVER_KNOBS->WORKER_COORDINATION_PING_DELAY );
state uint64_t step = 0;
Add DataDistributorInterface to ServerDBInfo Also change the Proxy and QuietDatabase to use the DataDistributorInterface.
2019-01-19 03:30:18 +08:00
state Future<ErrorOr<Void>> error = errorOr( actorCollection( self.addActor.getFuture() ) );
self.addActor.send( failureDetectionServer( self.id, &self.db, interf.clientInterface.failureMonitoring.getFuture() ) );
self.addActor.send( clusterWatchDatabase( &self, &self.db ) ); // Start the master database
self.addActor.send( self.updateWorkerList.init( self.db.db ) );
self.addActor.send( statusServer( interf.clientInterface.databaseStatus.getFuture(), &self, coordinators));
self.addActor.send( timeKeeper(&self) );
self.addActor.send( monitorProcessClasses(&self) );
self.addActor.send( monitorClientTxnInfoConfigs(&self.db) );
self.addActor.send( updatedChangingDatacenters(&self) );
self.addActor.send( updatedChangedDatacenters(&self) );
self.addActor.send( updateDatacenterVersionDifference(&self) );
forgot to fix merge conflicts
2019-02-19 09:09:55 +08:00
self.addActor.send( handleForcedRecoveries(&self, interf) );
Fix a segfault bug due to uncopied ratekeeper interface
2019-02-19 06:57:21 +08:00
self.addActor.send( monitorDataDistributor(&self) );
Minor fix on ratekeeper work registration.
2019-02-16 09:29:52 +08:00
self.addActor.send( monitorRatekeeper(&self) );
Initial repository commit
2017-05-26 04:48:44 +08:00
//printf("%s: I am the cluster controller\n", g_network->getLocalAddress().toString().c_str());
loop choose {
when( ErrorOr<Void> err = wait( error ) ) {
if (err.isError()) {
Add the roles running on a process as a field on trace events in the form of a comma delimited string of role abbreviations.
2018-09-06 06:06:14 +08:00
endRole(Role::CLUSTER_CONTROLLER, interf.id(), "Stop Received Error", false, err.getError());
Initial repository commit
2017-05-26 04:48:44 +08:00
}
else {
Add the roles running on a process as a field on trace events in the form of a comma delimited string of role abbreviations.
2018-09-06 06:06:14 +08:00
endRole(Role::CLUSTER_CONTROLLER, interf.id(), "Stop Received Signal", true);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
// We shut down normally even if there was a serious error (so this fdbserver may be re-elected cluster controller)
return Void();
}
when( OpenDatabaseRequest req = waitNext( interf.clientInterface.openDatabase.getFuture() ) ) {
Status: Count connected coordinators per client A client will always try to connect all coordinators. This commit let Status track the number of connected coordinators for each client. This allows us to do canary in coordinators. For example, when we switch from non-TLS to TLS, we can switch 1 coordinator from non-TLS to TLS. This can help check if a client has the ability to connect through TLS. We can make the non-TLS to TLS switch for each coordinators one by one. This avoid the risk of losing connection in the switch.
2019-03-06 13:00:19 +08:00
self.addActor.send( clusterOpenDatabase( &self.db, req.knownClientInfoID, req.issues.toString(), req.supportedVersions, req.connectedCoordinatorsNum, req.traceLogGroup, req.reply ) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
when( RecruitFromConfigurationRequest req = waitNext( interf.recruitFromConfiguration.getFuture() ) ) {
Add DataDistributorInterface to ServerDBInfo Also change the Proxy and QuietDatabase to use the DataDistributorInterface.
2019-01-19 03:30:18 +08:00
self.addActor.send( clusterRecruitFromConfiguration( &self, req ) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
when( RecruitRemoteFromConfigurationRequest req = waitNext( interf.recruitRemoteFromConfiguration.getFuture() ) ) {
Add DataDistributorInterface to ServerDBInfo Also change the Proxy and QuietDatabase to use the DataDistributorInterface.
2019-01-19 03:30:18 +08:00
self.addActor.send( clusterRecruitRemoteFromConfiguration( &self, req ) );
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
when( RecruitStorageRequest req = waitNext( interf.recruitStorage.getFuture() ) ) {
clusterRecruitStorage( &self, req );
}
when( RegisterWorkerRequest req = waitNext( interf.registerWorker.getFuture() ) ) {
registerWorker( req, &self );
}
when( GetWorkersRequest req = waitNext( interf.getWorkers.getFuture() ) ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
vector<WorkerDetails> workers;
Enable checkUsingDesiredClasses() in consistency check
2017-10-25 03:58:54 +08:00
for(auto& it : self.id_worker) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if ( (req.flags & GetWorkersRequest::NON_EXCLUDED_PROCESSES_ONLY) && self.db.config.isExcludedServer(it.second.details.interf.address()) ) {
Enable checkUsingDesiredClasses() in consistency check
2017-10-25 03:58:54 +08:00
continue;
}
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if ( (req.flags & GetWorkersRequest::TESTER_CLASS_ONLY) && it.second.details.processClass.classType() != ProcessClass::TesterClass ) {
Enable checkUsingDesiredClasses() in consistency check
2017-10-25 03:58:54 +08:00
continue;
}
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
workers.push_back(it.second.details);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Enable checkUsingDesiredClasses() in consistency check
2017-10-25 03:58:54 +08:00
req.reply.send( workers );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
when( GetClientWorkersRequest req = waitNext( interf.clientInterface.getClientWorkers.getFuture() ) ) {
vector<ClientWorkerInterface> workers;
for(auto& it : self.id_worker) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if (it.second.details.processClass.classType() != ProcessClass::TesterClass) {
workers.push_back(it.second.details.interf.clientInterface);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
req.reply.send(workers);
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when( wait( coordinationPingDelay ) ) {
Initial repository commit
2017-05-26 04:48:44 +08:00
CoordinationPingMessage message(self.id, step++);
for(auto& it : self.id_worker)
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
it.second.details.interf.coordinationPing.send(message);
Initial repository commit
2017-05-26 04:48:44 +08:00
coordinationPingDelay = delay( SERVER_KNOBS->WORKER_COORDINATION_PING_DELAY );
TraceEvent("CoordinationPingSent", self.id).detail("TimeStep", message.timeStep);
}
when( RegisterMasterRequest req = waitNext( interf.registerMaster.getFuture() ) ) {
clusterRegisterMaster( &self, req );
}
when( GetServerDBInfoRequest req = waitNext( interf.getServerDBInfo.getFuture() ) ) {
Add DataDistributorInterface to ServerDBInfo Also change the Proxy and QuietDatabase to use the DataDistributorInterface.
2019-01-19 03:30:18 +08:00
self.addActor.send( clusterGetServerInfo( &self.db, req.knownServerInfoID, req.issues.toString(), req.incompatiblePeers, req.reply ) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when( wait( leaderFail ) ) {
Initial repository commit
2017-05-26 04:48:44 +08:00
// We are no longer the leader if this has changed.
Add the roles running on a process as a field on trace events in the form of a comma delimited string of role abbreviations.
2018-09-06 06:06:14 +08:00
endRole(Role::CLUSTER_CONTROLLER, interf.id(), "Leader Replaced", true);
Initial repository commit
2017-05-26 04:48:44 +08:00
TEST(true); // Lost Cluster Controller Role
return Void();
}
when( ReplyPromise<Void> ping = waitNext( interf.clientInterface.ping.getFuture() ) ) {
ping.send( Void() );
}
}
}
fix: the cluster controller did not pass in its own locality when creating its database object, therefore it was not using locality aware load balancing
2018-09-29 03:12:06 +08:00
ACTOR Future<Void> clusterController( ServerCoordinators coordinators, Reference<AsyncVar<Optional<ClusterControllerFullInterface>>> currentCC, bool hasConnected, Reference<AsyncVar<ClusterControllerPriorityInfo>> asyncPriorityInfo, LocalityData locality ) {
Initial repository commit
2017-05-26 04:48:44 +08:00
loop {
state ClusterControllerFullInterface cci;
state bool inRole = false;
cci.initEndpoints();
try {
//Register as a possible leader; wait to be elected
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
state Future<Void> leaderFail = tryBecomeLeader( coordinators, cci, currentCC, hasConnected, asyncPriorityInfo );
Initial repository commit
2017-05-26 04:48:44 +08:00
while (!currentCC->get().present() || currentCC->get().get() != cci) {
choose {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when( wait(currentCC->onChange()) ) {}
when( wait(leaderFail) ) { ASSERT(false); throw internal_error(); }
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
hasConnected = true;
Add the roles running on a process as a field on trace events in the form of a comma delimited string of role abbreviations.
2018-09-06 06:06:14 +08:00
startRole(Role::CLUSTER_CONTROLLER, cci.id(), UID());
Initial repository commit
2017-05-26 04:48:44 +08:00
inRole = true;
Merge branch 'release-6.0' # Conflicts: # documentation/sphinx/source/release-notes.rst # fdbclient/ManagementAPI.actor.cpp # fdbserver/ClusterController.actor.cpp # fdbserver/DataDistribution.actor.cpp # fdbserver/LogSystemDiskQueueAdapter.actor.cpp # fdbserver/SimulatedCluster.actor.cpp # fdbserver/TLogServer.actor.cpp
2018-10-04 07:57:18 +08:00
wait( clusterControllerCore( cci, leaderFail, coordinators, locality ) );
Initial repository commit
2017-05-26 04:48:44 +08:00
} catch(Error& e) {
if (inRole)
Add the roles running on a process as a field on trace events in the form of a comma delimited string of role abbreviations.
2018-09-06 06:06:14 +08:00
endRole(Role::CLUSTER_CONTROLLER, cci.id(), "Error", e.code() == error_code_actor_cancelled || e.code() == error_code_coordinators_changed, e);
Initial repository commit
2017-05-26 04:48:44 +08:00
else
TraceEvent( e.code() == error_code_coordinators_changed ? SevInfo : SevError, "ClusterControllerCandidateError", cci.id()).error(e);
throw;
}
}
}
fix: the cluster controller did not pass in its own locality when creating its database object, therefore it was not using locality aware load balancing
2018-09-29 03:12:06 +08:00
ACTOR Future<Void> clusterController( Reference<ClusterConnectionFile> connFile, Reference<AsyncVar<Optional<ClusterControllerFullInterface>>> currentCC, Reference<AsyncVar<ClusterControllerPriorityInfo>> asyncPriorityInfo, Future<Void> recoveredDiskFiles, LocalityData locality ) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(recoveredDiskFiles);
Initial repository commit
2017-05-26 04:48:44 +08:00
state bool hasConnected = false;
loop {
try {
ServerCoordinators coordinators( connFile );
Merge branch 'release-6.0' # Conflicts: # documentation/sphinx/source/release-notes.rst # fdbclient/ManagementAPI.actor.cpp # fdbserver/ClusterController.actor.cpp # fdbserver/DataDistribution.actor.cpp # fdbserver/LogSystemDiskQueueAdapter.actor.cpp # fdbserver/SimulatedCluster.actor.cpp # fdbserver/TLogServer.actor.cpp
2018-10-04 07:57:18 +08:00
wait( clusterController( coordinators, currentCC, hasConnected, asyncPriorityInfo, locality ) );
Initial repository commit
2017-05-26 04:48:44 +08:00
} catch( Error &e ) {
if( e.code() != error_code_coordinators_changed )
throw; // Expected to terminate fdbserver
}
hasConnected = true;
}
Make RecoveryState an enum class. This means that all the == 7 or != 0 checks go away, and explicit names must be used.
2018-05-09 08:17:17 +08:00
}