wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
foundationdb/fdbserver/ClusterController.actor.cpp

2956 lines
127 KiB
C++
Raw Normal View History

Initial repository commit
2017-05-26 04:48:44 +08:00
/*
* ClusterController.actor.cpp
*
* This source file is part of the FoundationDB open source project
*
Minor fix on ratekeeper work registration.
2019-02-16 09:29:52 +08:00
* Copyright 2013-2019 Apple Inc. and the FoundationDB project authors
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* http://www.apache.org/licenses/LICENSE-2.0
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#include "fdbrpc/FailureMonitor.h"
#include "flow/ActorCollection.h"
Use proper fwd decl in NativeAPI Also NativeAPI.h -> NativeAPI.actor.h
2019-02-18 07:41:16 +08:00
#include "fdbclient/NativeAPI.actor.h"
Initial repository commit
2017-05-26 04:48:44 +08:00
#include "fdbserver/CoordinationInterface.h"
Add a new DataDistributor role. Let cluster controller to start a new data distributor role by sending a message to a chosen worker. Change MasterInterface usage in DataDistribution to masterId Add DataDistributor rejoin handling. This allows the data distributor to tell the new cluster controller of its existence so that the controller doesn't spawn a new one. I.e., there should be only ONE data distributor in the cluster. If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries to recruit one as DD. CC also monitors DD and restarts one if it failed. The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for the new DD. Add GetRecoveryInfo RPC to master server, which is called by data distributor to obtain the recovery Transaction version from the master server.
2018-12-14 05:31:37 +08:00
#include "fdbserver/DataDistributorInterface.h"
Adjust all includes to be relative to the root. Remove the use of relative paths. A header at foo/bar.h could be included by files under foo/ with "bar.h", but would be included everywhere else as "foo/bar.h". Adjust so that every include references such a header with the latter form. Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-20 01:30:13 +08:00
#include "fdbserver/Knobs.h"
Use ACTOR forward declarations in MoveKeys Also MoveKeys.h -> MoveKeys.actor.h
2019-02-18 10:55:52 +08:00
#include "fdbserver/MoveKeys.actor.h"
Use fwd decls in WorkerInterface Also WorkerInterface.h -> WorkerInterface.actor.h
2019-02-18 11:13:26 +08:00
#include "fdbserver/WorkerInterface.actor.h"
Adjust all includes to be relative to the root. Remove the use of relative paths. A header at foo/bar.h could be included by files under foo/ with "bar.h", but would be included everywhere else as "foo/bar.h". Adjust so that every include references such a header with the latter form. Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-20 01:30:13 +08:00
#include "fdbserver/LeaderElection.h"
Add used worker IDs to cluster controller This "usedIds" is updated when receiving a master registration message, so that when recruiting new data distributor, existing assignment is known.
2019-01-29 01:25:15 +08:00
#include "fdbserver/LogSystemConfig.h"
Adjust all includes to be relative to the root. Remove the use of relative paths. A header at foo/bar.h could be included by files under foo/ with "bar.h", but would be included everywhere else as "foo/bar.h". Adjust so that every include references such a header with the latter form. Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-20 01:30:13 +08:00
#include "fdbserver/WaitFailure.h"
#include "fdbserver/ClusterRecruitmentInterface.h"
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
#include "fdbserver/RatekeeperInterface.h"
Adjust all includes to be relative to the root. Remove the use of relative paths. A header at foo/bar.h could be included by files under foo/ with "bar.h", but would be included everywhere else as "foo/bar.h". Adjust so that every include references such a header with the latter form. Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-20 01:30:13 +08:00
#include "fdbserver/ServerDBInfo.h"
#include "fdbserver/Status.h"
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
#include "fdbserver/LatencyBandConfig.h"
Initial repository commit
2017-05-26 04:48:44 +08:00
#include <algorithm>
#include "fdbclient/DatabaseContext.h"
Adjust all includes to be relative to the root. Remove the use of relative paths. A header at foo/bar.h could be included by files under foo/ with "bar.h", but would be included everywhere else as "foo/bar.h". Adjust so that every include references such a header with the latter form. Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-20 01:30:13 +08:00
#include "fdbserver/RecoveryState.h"
Initial repository commit
2017-05-26 04:48:44 +08:00
#include "fdbclient/ReadYourWrites.h"
#include "fdbrpc/Replication.h"
#include "fdbrpc/ReplicationUtils.h"
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
#include "fdbclient/KeyBackedTypes.h"
Stop performing self-moves. (e.g. a = std::move(a)) self-moves are frowned upon in C++, and in our code this generally happens from calls to swap as part of trying to implement a "unordered erase" function via swap-to-the-end-and-pop_back. For convenience, a swapAndPop() function is now offered that performs this, while disallowing self-moves.
2018-08-02 09:09:54 +08:00
#include "flow/Util.h"
Fix more instances of actorcompiler.h being in the wrong place.
2018-08-11 08:24:52 +08:00
#include "flow/actorcompiler.h" // This must be the last #include.
Initial repository commit
2017-05-26 04:48:44 +08:00
void failAfter( Future<Void> trigger, Endpoint e );
struct WorkerInfo : NonCopyable {
Future<Void> watcher;
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
ReplyPromise<RegisterWorkerReply> reply;
Initial repository commit
2017-05-26 04:48:44 +08:00
Generation gen;
int reboots;
ProcessClass initialClass;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
ClusterControllerPriorityInfo priorityInfo;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
WorkerDetails details;
Add actors to store halt request futures Address best fitness in checking better DD or RK.
2019-03-23 09:00:16 +08:00
Future<Void> haltRatekeeper;
Future<Void> haltDistributor;
Initial repository commit
2017-05-26 04:48:44 +08:00
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
WorkerInfo() : gen(-1), reboots(0), priorityInfo(ProcessClass::UnsetFit, false, ClusterControllerPriorityInfo::FitnessUnknown) {}
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
WorkerInfo( Future<Void> watcher, ReplyPromise<RegisterWorkerReply> reply, Generation gen, WorkerInterface interf, ProcessClass initialClass, ProcessClass processClass, ClusterControllerPriorityInfo priorityInfo, bool degraded ) :
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
watcher(watcher), reply(reply), gen(gen), reboots(0), initialClass(initialClass), priorityInfo(priorityInfo), details(interf, processClass, degraded) {}
Initial repository commit
2017-05-26 04:48:44 +08:00
Remove noexcept macro and replace with BOOST_NOEXCEPT. BOOST_NOEXCEPT does what the noexcept macro was supposed to do, but in a way that is correctly maintained over time.
2019-01-26 08:49:59 +08:00
WorkerInfo( WorkerInfo&& r ) BOOST_NOEXCEPT : watcher(std::move(r.watcher)), reply(std::move(r.reply)), gen(r.gen),
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
reboots(r.reboots), initialClass(r.initialClass), priorityInfo(r.priorityInfo), details(std::move(r.details)) {}
Remove noexcept macro and replace with BOOST_NOEXCEPT. BOOST_NOEXCEPT does what the noexcept macro was supposed to do, but in a way that is correctly maintained over time.
2019-01-26 08:49:59 +08:00
void operator=( WorkerInfo&& r ) BOOST_NOEXCEPT {
Initial repository commit
2017-05-26 04:48:44 +08:00
watcher = std::move(r.watcher);
reply = std::move(r.reply);
gen = r.gen;
reboots = r.reboots;
initialClass = r.initialClass;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
priorityInfo = r.priorityInfo;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
details = std::move(r.details);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
};
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
struct WorkerFitnessInfo {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
WorkerDetails worker;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
ProcessClass::Fitness fitness;
int used;
WorkerFitnessInfo() : fitness(ProcessClass::NeverAssign), used(0) {}
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
WorkerFitnessInfo(WorkerDetails worker, ProcessClass::Fitness fitness, int used) : worker(worker), fitness(fitness), used(used) {}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
};
Initial repository commit
2017-05-26 04:48:44 +08:00
class ClusterControllerData {
public:
struct DBInfo {
Reference<AsyncVar<ClientDBInfo>> clientInfo;
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
Reference<AsyncVar<CachedSerialization<ServerDBInfo>>> serverInfo;
during recovery, only send the full serverDBInfo to processes that are part of the new generation
2019-12-10 05:17:49 +08:00
CachedSerialization<ServerDBInfo> serverInfoMasterOnly;
std::set<NetworkAddress> requiredAddresses;
Clients keep a persistent connection open with coordinators to get updates to the list of proxies Status still needs to be updated with client information with information from the coordinators
2019-07-24 10:22:44 +08:00
ProcessIssuesMap workersWithIssues;
Initial repository commit
2017-05-26 04:48:44 +08:00
std::map<NetworkAddress, double> incompatibleConnections;
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
AsyncTrigger forceMasterFailure;
Initial repository commit
2017-05-26 04:48:44 +08:00
int64_t masterRegistrationCount;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
bool recoveryStalled;
added the (untested) ability to force a recovery to the remote datacenter, even if that results in data loss. If the DR lag is more than 1 week there could be potential data corruption if any primary storage servers are still alive.
2018-07-01 21:39:04 +08:00
bool forceRecovery;
Initial repository commit
2017-05-26 04:48:44 +08:00
DatabaseConfiguration config; // Asynchronously updated via master registration
Add fullyRecoveredConfig for checking exclusions
2017-11-16 09:15:24 +08:00
DatabaseConfiguration fullyRecoveredConfig;
Initial repository commit
2017-05-26 04:48:44 +08:00
Database db;
The cluster controller will increase its failure monitoring delay after there have been many unfinishedRecoveries
2018-09-01 01:51:55 +08:00
int unfinishedRecoveries;
int logGenerations;
added status information on connected clients
2019-07-26 08:15:31 +08:00
std::map<NetworkAddress, std::pair<double, OpenDatabaseRequest>> clientStatus;
Initial repository commit
2017-05-26 04:48:44 +08:00
The cluster controller will increase its failure monitoring delay after there have been many unfinishedRecoveries
2018-09-01 01:51:55 +08:00
DBInfo() : masterRegistrationCount(0), recoveryStalled(false), forceRecovery(false), unfinishedRecoveries(0), logGenerations(0),
Initial repository commit
2017-05-26 04:48:44 +08:00
clientInfo( new AsyncVar<ClientDBInfo>( ClientDBInfo() ) ),
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
serverInfo( new AsyncVar<CachedSerialization<ServerDBInfo>>( CachedSerialization<ServerDBInfo>() ) ),
A giant translation of TaskFooPriority -> TaskPriority::Foo This is so that APIs that take priorities don't take ints, which are common and easy to accidentally pass the wrong thing.
2019-06-25 17:47:35 +08:00
db( DatabaseContext::create( clientInfo, Future<Void>(), LocalityData(), true, TaskPriority::DefaultEndpoint, true ) ) // SOMEDAY: Locality!
Initial repository commit
2017-05-26 04:48:44 +08:00
{
Add DataDistributorInterface to ServerDBInfo Also change the Proxy and QuietDatabase to use the DataDistributorInterface.
2019-01-19 03:30:18 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
fix: we were recruiting one too few oldLogRouters code cleanup
2020-01-03 07:05:44 +08:00
void addRequiredAddresses(const std::vector<WorkerInterface>& interfaces) {
for(auto& it : interfaces) {
requiredAddresses.insert(it.address());
}
}
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
void setDistributor(const DataDistributorInterface& interf) {
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
CachedSerialization<ServerDBInfo> newInfoCache = serverInfo->get();
auto& newInfo = newInfoCache.mutate();
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
newInfo.id = deterministicRandom()->randomUniqueID();
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
newInfo.distributor = interf;
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
serverInfo->set( newInfoCache );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
void setRatekeeper(const RatekeeperInterface& interf) {
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
CachedSerialization<ServerDBInfo> newInfoCache = serverInfo->get();
auto& newInfo = newInfoCache.mutate();
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
newInfo.id = deterministicRandom()->randomUniqueID();
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
newInfo.ratekeeper = interf;
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
serverInfo->set( newInfoCache );
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
}
void clearInterf(ProcessClass::ClassType t) {
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
CachedSerialization<ServerDBInfo> newInfoCache = serverInfo->get();
auto& newInfo = newInfoCache.mutate();
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
newInfo.id = deterministicRandom()->randomUniqueID();
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
if (t == ProcessClass::DataDistributorClass) {
newInfo.distributor = Optional<DataDistributorInterface>();
Changes to ratekeeper camel case
2019-03-27 23:24:25 +08:00
} else if (t == ProcessClass::RatekeeperClass) {
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
newInfo.ratekeeper = Optional<RatekeeperInterface>();
}
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
serverInfo->set( newInfoCache );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
};
struct UpdateWorkerList {
Future<Void> init( Database const& db ) {
return update(this, db);
}
void set(Optional<Standalone<StringRef>> processID, Optional<ProcessData> data ) {
delta[processID] = data;
anyDelta.set(true);
}
private:
std::map<Optional<Standalone<StringRef>>, Optional<ProcessData>> delta;
AsyncVar<bool> anyDelta;
ACTOR static Future<Void> update( UpdateWorkerList* self, Database db ) {
// The Database we are using is based on worker registrations to this cluster controller, which come only
// from master servers that we started, so it shouldn't be possible for multiple cluster controllers to fight.
state Transaction tr(db);
loop {
try {
tr.clear( workerListKeys );
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( tr.commit() );
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
} catch (Error& e) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( tr.onError(e) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
loop {
// Wait for some changes
while (!self->anyDelta.get())
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( self->anyDelta.onChange() );
Initial repository commit
2017-05-26 04:48:44 +08:00
self->anyDelta.set(false);
state std::map<Optional<Standalone<StringRef>>, Optional<ProcessData>> delta;
delta.swap( self->delta );
TraceEvent("UpdateWorkerList").detail("DeltaCount", delta.size());
// Do a transaction to write the changes
loop {
try {
for(auto w = delta.begin(); w != delta.end(); ++w) {
if (w->second.present()) {
tr.set( workerListKeyFor( w->first.get() ), workerListValue( w->second.get()) );
} else
tr.clear( workerListKeyFor( w->first.get() ) );
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( tr.commit() );
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
} catch (Error& e) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( tr.onError(e) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
}
}
};
bool workerAvailable( WorkerInfo const& worker, bool checkStable ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
return ( now() - startTime < 2 * FLOW_KNOBS->SERVER_REQUEST_INTERVAL ) || ( IFailureMonitor::failureMonitor().getState(worker.details.interf.storage.getEndpoint()).isAvailable() && ( !checkStable || worker.reboots < 2 ) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
WorkerDetails getStorageWorker( RecruitStorageRequest const& req ) {
Initial repository commit
2017-05-26 04:48:44 +08:00
std::set<Optional<Standalone<StringRef>>> excludedMachines( req.excludeMachines.begin(), req.excludeMachines.end() );
fixed merge problems
2017-09-12 07:37:10 +08:00
std::set<Optional<Standalone<StringRef>>> includeDCs( req.includeDCs.begin(), req.includeDCs.end() );
Initial repository commit
2017-05-26 04:48:44 +08:00
std::set<AddressExclusion> excludedAddresses( req.excludeAddresses.begin(), req.excludeAddresses.end() );
for( auto& it : id_worker )
if( workerAvailable( it.second, false ) &&
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
!excludedMachines.count(it.second.details.interf.locality.zoneId()) &&
( includeDCs.size() == 0 || includeDCs.count(it.second.details.interf.locality.dcId()) ) &&
!addressExcluded(excludedAddresses, it.second.details.interf.address()) &&
it.second.details.processClass.machineClassFitness( ProcessClass::Storage ) <= ProcessClass::UnsetFit ) {
return it.second.details;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
if( req.criticalRecruitment ) {
ProcessClass::Fitness bestFit = ProcessClass::NeverAssign;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
Optional<WorkerDetails> bestInfo;
Initial repository commit
2017-05-26 04:48:44 +08:00
for( auto& it : id_worker ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
ProcessClass::Fitness fit = it.second.details.processClass.machineClassFitness( ProcessClass::Storage );
Initial repository commit
2017-05-26 04:48:44 +08:00
if( workerAvailable( it.second, false ) &&
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
!excludedMachines.count(it.second.details.interf.locality.zoneId()) &&
( includeDCs.size() == 0 || includeDCs.count(it.second.details.interf.locality.dcId()) ) &&
!addressExcluded(excludedAddresses, it.second.details.interf.address()) &&
Initial repository commit
2017-05-26 04:48:44 +08:00
fit < bestFit ) {
bestFit = fit;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
bestInfo = it.second.details;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
if( bestInfo.present() ) {
return bestInfo.get();
}
}
throw no_more_servers();
}
removed LocalitySetRef and IRepPolicyRef typedefs, because for clarity the Ref suffix is reserved for arena allocated objects instead of reference counted objects.
2019-03-14 04:14:39 +08:00
std::vector<WorkerDetails> getWorkersForSeedServers( DatabaseConfiguration const& conf, Reference<IReplicationPolicy> const& policy, Optional<Optional<Standalone<StringRef>>> const& dcId = Optional<Optional<Standalone<StringRef>>>() ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
std::map<ProcessClass::Fitness, vector<WorkerDetails>> fitness_workers;
std::vector<WorkerDetails> results;
removed LocalitySetRef and IRepPolicyRef typedefs, because for clarity the Ref suffix is reserved for arena allocated objects instead of reference counted objects.
2019-03-14 04:14:39 +08:00
Reference<LocalitySet> logServerSet = Reference<LocalitySet>(new LocalityMap<WorkerDetails>());
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
LocalityMap<WorkerDetails>* logServerMap = (LocalityMap<WorkerDetails>*) logServerSet.getPtr();
fix: seed storage servers are recruited based on the storage policy
2017-09-15 08:06:00 +08:00
bool bCompleted = false;
for( auto& it : id_worker ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
auto fitness = it.second.details.processClass.machineClassFitness( ProcessClass::Storage );
if( workerAvailable(it.second, false) && !conf.isExcludedServer(it.second.details.interf.address()) && fitness != ProcessClass::NeverAssign && ( !dcId.present() || it.second.details.interf.locality.dcId()==dcId.get() ) ) {
fitness_workers[ fitness ].push_back(it.second.details);
fix: seed storage servers are recruited based on the storage policy
2017-09-15 08:06:00 +08:00
}
}
for( auto& it : fitness_workers ) {
for (auto& worker : it.second ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
logServerMap->add(worker.interf.locality, &worker);
fix: seed storage servers are recruited based on the storage policy
2017-09-15 08:06:00 +08:00
}
std::vector<LocalityEntry> bestSet;
Merge branch 'master' into feature-remote-logs # Conflicts: # fdbrpc/Locality.cpp # fdbrpc/Locality.h # fdbserver/ClusterController.actor.cpp # fdbserver/ClusterRecruitmentInterface.h # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/fdbserver.vcxproj.filters # fdbserver/masterserver.actor.cpp # fdbserver/worker.actor.cpp # flow/error_definitions.h
2017-10-06 08:09:44 +08:00
if( logServerSet->selectReplicas(policy, bestSet) ) {
fix: seed storage servers are recruited based on the storage policy
2017-09-15 08:06:00 +08:00
results.reserve(bestSet.size());
for (auto& entry : bestSet) {
auto object = logServerMap->getObject(entry);
results.push_back(*object);
}
bCompleted = true;
break;
}
}
logServerSet->clear();
logServerSet.clear();
if (!bCompleted) {
throw no_more_servers();
}
return results;
}
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
std::vector<WorkerDetails> getWorkersForTlogs( DatabaseConfiguration const& conf, int32_t required, int32_t desired, Reference<IReplicationPolicy> const& policy, std::map< Optional<Standalone<StringRef>>, int>& id_used, bool checkStable = false, std::set<Optional<Key>> dcIds = std::set<Optional<Key>>(), std::vector<UID> exclusionWorkerIds = {}) {
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
std::map<std::pair<ProcessClass::Fitness,bool>, vector<WorkerDetails>> fitness_workers;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
std::vector<WorkerDetails> results;
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
std::vector<LocalityData> unavailableLocals;
removed LocalitySetRef and IRepPolicyRef typedefs, because for clarity the Ref suffix is reserved for arena allocated objects instead of reference counted objects.
2019-03-14 04:14:39 +08:00
Reference<LocalitySet> logServerSet;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
LocalityMap<WorkerDetails>* logServerMap;
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
bool bCompleted = false;
Initial repository commit
2017-05-26 04:48:44 +08:00
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
logServerSet = Reference<LocalitySet>(new LocalityMap<WorkerDetails>());
logServerMap = (LocalityMap<WorkerDetails>*) logServerSet.getPtr();
Initial repository commit
2017-05-26 04:48:44 +08:00
for( auto& it : id_worker ) {
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
if (std::find(exclusionWorkerIds.begin(), exclusionWorkerIds.end(), it.second.details.interf.id()) == exclusionWorkerIds.end()) {
auto fitness = it.second.details.processClass.machineClassFitness(ProcessClass::TLog);
if (workerAvailable(it.second, checkStable) && !conf.isExcludedServer(it.second.details.interf.address()) && fitness != ProcessClass::NeverAssign && (!dcIds.size() || dcIds.count(it.second.details.interf.locality.dcId()))) {
fitness_workers[std::make_pair(fitness, it.second.details.degraded)].push_back(it.second.details);
}
else {
unavailableLocals.push_back(it.second.details.interf.locality);
}
Fixed support for RemoveServers Workload Added availability functions to simulation
2017-05-27 05:20:11 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
results.reserve(results.size() + id_worker.size());
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
for (int fitness = ProcessClass::BestFit; fitness != ProcessClass::NeverAssign && !bCompleted; fitness++)
Initial repository commit
2017-05-26 04:48:44 +08:00
{
auto fitnessEnum = (ProcessClass::Fitness) fitness;
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
for(int addingDegraded = 0; addingDegraded < 2; addingDegraded++) {
auto workerItr = fitness_workers.find(std::make_pair(fitnessEnum,(bool)addingDegraded));
fixed review comments
2019-03-13 05:38:54 +08:00
if (workerItr != fitness_workers.end()) {
for (auto& worker : workerItr->second ) {
logServerMap->add(worker.interf.locality, &worker);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
fixed review comments
2019-03-13 05:38:54 +08:00
if (logServerSet->size() < (addingDegraded == 0 ? desired : required)) {
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
}
else if (logServerSet->size() == required || logServerSet->size() <= desired) {
if (logServerSet->validate(policy)) {
for (auto& object : logServerMap->getObjects()) {
results.push_back(*object);
}
bCompleted = true;
break;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
TraceEvent(SevWarn,"GWFTADNotAcceptable", id).detail("Fitness", fitness).detail("Processes", logServerSet->size()).detail("Required", required).detail("TLogPolicy",policy->info()).detail("DesiredLogs", desired).detail("AddingDegraded", addingDegraded);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
// Try to select the desired size, if larger
else {
std::vector<LocalityEntry> bestSet;
std::vector<LocalityData> tLocalities;
// Try to find the best team of servers to fulfill the policy
Address review comments.
2020-02-21 01:34:01 +08:00
if (findBestPolicySet(bestSet, logServerSet, policy, desired, SERVER_KNOBS->POLICY_RATING_TESTS,
SERVER_KNOBS->POLICY_GENERATIONS)) {
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
results.reserve(results.size() + bestSet.size());
for (auto& entry : bestSet) {
auto object = logServerMap->getObject(entry);
ASSERT(object);
results.push_back(*object);
tLocalities.push_back(object->interf.locality);
}
TraceEvent("GWFTADBestResults", id).detail("Fitness", fitness).detail("Processes", logServerSet->size()).detail("BestCount", bestSet.size()).detail("BestZones", ::describeZones(tLocalities))
.detail("BestDataHalls", ::describeDataHalls(tLocalities)).detail("TLogPolicy", policy->info()).detail("TotalResults", results.size()).detail("DesiredLogs", desired).detail("AddingDegraded", addingDegraded);
bCompleted = true;
break;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
TraceEvent(SevWarn,"GWFTADNoBest", id).detail("Fitness", fitness).detail("Processes", logServerSet->size()).detail("Required", required).detail("TLogPolicy", policy->info()).detail("DesiredLogs", desired).detail("AddingDegraded", addingDegraded);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
}
// If policy cannot be satisfied
first working version of non-copying recovery working with fearless configurations
2018-04-09 12:24:05 +08:00
if (!bCompleted) {
std::vector<LocalityData> tLocalities;
for (auto& object : logServerMap->getObjects()) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
tLocalities.push_back(object->interf.locality);
first working version of non-copying recovery working with fearless configurations
2018-04-09 12:24:05 +08:00
}
TraceEvent(SevWarn, "GetTLogTeamFailed").detail("Policy", policy->info()).detail("Processes", logServerSet->size()).detail("Workers", id_worker.size()).detail("FitnessGroups", fitness_workers.size())
.detail("TLogZones", ::describeZones(tLocalities)).detail("TLogDataHalls", ::describeDataHalls(tLocalities)).detail("MissingZones", ::describeZones(unavailableLocals))
.detail("MissingDataHalls", ::describeDataHalls(unavailableLocals)).detail("Required", required).detail("DesiredLogs", desired).detail("RatingTests",SERVER_KNOBS->POLICY_RATING_TESTS)
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
.detail("CheckStable", checkStable).detail("NumExclusionWorkers", exclusionWorkerIds.size()).detail("PolicyGenerations",SERVER_KNOBS->POLICY_GENERATIONS).backtrace();
Initial repository commit
2017-05-26 04:48:44 +08:00
logServerSet->clear();
logServerSet.clear();
throw no_more_servers();
}
for (auto& result : results) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
id_used[result.interf.locality.processId()]++;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
first working version of non-copying recovery working with fearless configurations
2018-04-09 12:24:05 +08:00
TraceEvent("GetTLogTeamDone").detail("Completed", bCompleted).detail("Policy", policy->info()).detail("Results", results.size()).detail("Processes", logServerSet->size()).detail("Workers", id_worker.size())
.detail("Required", required).detail("Desired", desired).detail("RatingTests",SERVER_KNOBS->POLICY_RATING_TESTS).detail("PolicyGenerations",SERVER_KNOBS->POLICY_GENERATIONS);
Initial repository commit
2017-05-26 04:48:44 +08:00
return results;
}
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
//FIXME: This logic will fallback unnecessarily when usable dcs > 1 because it does not check all combinations of potential satellite locations
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
std::vector<WorkerDetails> getWorkersForSatelliteLogs( const DatabaseConfiguration& conf, const RegionInfo& region, const RegionInfo& remoteRegion, std::map< Optional<Standalone<StringRef>>, int>& id_used, bool& satelliteFallback, bool checkStable = false ) {
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
int startDC = 0;
loop {
if(startDC > 0 && startDC >= region.satellites.size() + 1 - (satelliteFallback ? region.satelliteTLogUsableDcsFallback : region.satelliteTLogUsableDcs)) {
if(satelliteFallback || region.satelliteTLogUsableDcsFallback == 0) {
throw no_more_servers();
} else {
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
if(!goodRecruitmentTime.isReady()) {
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
throw operation_failed();
}
satelliteFallback = true;
startDC = 0;
}
}
try {
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
bool remoteDCUsedAsSatellite = false;
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
std::set<Optional<Key>> satelliteDCs;
added the ability to configure satellite_logs by satellite location, this will overwrite the region configure if both are present
2019-10-15 09:30:15 +08:00
int32_t desiredSatelliteTLogs = 0;
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
for(int s = startDC; s < std::min<int>(startDC + (satelliteFallback ? region.satelliteTLogUsableDcsFallback : region.satelliteTLogUsableDcs), region.satellites.size()); s++) {
satelliteDCs.insert(region.satellites[s].dcId);
added the ability to configure satellite_logs by satellite location, this will overwrite the region configure if both are present
2019-10-15 09:30:15 +08:00
if(region.satellites[s].satelliteDesiredTLogCount == -1 || desiredSatelliteTLogs == -1) {
desiredSatelliteTLogs = -1;
} else {
desiredSatelliteTLogs += region.satellites[s].satelliteDesiredTLogCount;
}
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
if (region.satellites[s].dcId == remoteRegion.dcId) {
remoteDCUsedAsSatellite = true;
}
}
std::vector<UID> exclusionWorkerIds;
// FIXME: If remote DC is used as satellite then this logic only ensures that required number of remote TLogs can be recruited. It does not balance the number of desired TLogs
// across the satellite and remote sides.
if (remoteDCUsedAsSatellite) {
Addressed Review Comments
2019-03-20 06:23:14 +08:00
std::map< Optional<Standalone<StringRef>>, int> tmpIdUsed;
auto remoteLogs = getWorkersForTlogs(conf, conf.getRemoteTLogReplicationFactor(), conf.getRemoteTLogReplicationFactor(), conf.getRemoteTLogPolicy(), tmpIdUsed, false, { remoteRegion.dcId }, {});
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
std::transform(remoteLogs.begin(), remoteLogs.end(), std::back_inserter(exclusionWorkerIds), [](const WorkerDetails &in) { return in.interf.id(); });
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
}
if(satelliteFallback) {
added the ability to configure satellite_logs by satellite location, this will overwrite the region configure if both are present
2019-10-15 09:30:15 +08:00
return getWorkersForTlogs( conf, region.satelliteTLogReplicationFactorFallback, desiredSatelliteTLogs>0 ? desiredSatelliteTLogs : conf.getDesiredSatelliteLogs(region.dcId)*region.satelliteTLogUsableDcsFallback/region.satelliteTLogUsableDcs, region.satelliteTLogPolicyFallback, id_used, checkStable, satelliteDCs, exclusionWorkerIds);
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
} else {
added the ability to configure satellite_logs by satellite location, this will overwrite the region configure if both are present
2019-10-15 09:30:15 +08:00
return getWorkersForTlogs( conf, region.satelliteTLogReplicationFactor, desiredSatelliteTLogs>0 ? desiredSatelliteTLogs : conf.getDesiredSatelliteLogs(region.dcId), region.satelliteTLogPolicy, id_used, checkStable, satelliteDCs, exclusionWorkerIds);
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
}
} catch (Error &e) {
if(e.code() != error_code_no_more_servers) {
throw;
}
}
startDC++;
}
}
Add actors to store halt request futures Address best fitness in checking better DD or RK.
2019-03-23 09:00:16 +08:00
ProcessClass::Fitness getBestFitnessForRoleInDatacenter(ProcessClass::ClusterRole role) {
ProcessClass::Fitness bestFitness = ProcessClass::NeverAssign;
for (const auto& it : id_worker) {
if (it.second.priorityInfo.isExcluded || it.second.details.interf.locality.dcId() != clusterControllerDcId) {
continue;
}
bestFitness = std::min(bestFitness, it.second.details.processClass.machineClassFitness(role));
}
return bestFitness;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
WorkerFitnessInfo getWorkerForRoleInDatacenter(Optional<Standalone<StringRef>> const& dcId, ProcessClass::ClusterRole role, ProcessClass::Fitness unacceptableFitness, DatabaseConfiguration const& conf, std::map< Optional<Standalone<StringRef>>, int>& id_used, bool checkStable = false ) {
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
std::map<std::pair<ProcessClass::Fitness,int>, std::pair<vector<WorkerDetails>,vector<WorkerDetails>>> fitness_workers;
Initial repository commit
2017-05-26 04:48:44 +08:00
for( auto& it : id_worker ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
auto fitness = it.second.details.processClass.machineClassFitness( role );
if(conf.isExcludedServer(it.second.details.interf.address())) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
fitness = std::max(fitness, ProcessClass::ExcludeFit);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if( workerAvailable(it.second, checkStable) && fitness < unacceptableFitness && it.second.details.interf.locality.dcId()==dcId ) {
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
if ((db.serverInfo->get().read().distributor.present() && db.serverInfo->get().read().distributor.get().locality.processId() == it.first) ||
(db.serverInfo->get().read().ratekeeper.present() && db.serverInfo->get().read().ratekeeper.get().locality.processId() == it.first)) {
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
fitness_workers[ std::make_pair(fitness, id_used[it.first]) ].second.push_back(it.second.details);
} else {
fitness_workers[ std::make_pair(fitness, id_used[it.first]) ].first.push_back(it.second.details);
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
for( auto& it : fitness_workers ) {
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
for( int j=0; j < 2; j++ ) {
auto& w = j==0 ? it.second.first : it.second.second;
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
deterministicRandom()->randomShuffle(w);
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
for( int i=0; i < w.size(); i++ ) {
id_used[w[i].interf.locality.processId()]++;
one proxy or resolver with best fitness no longer prevents more proxies or resolvers from being recruited with good fitness
2019-10-15 09:32:17 +08:00
return WorkerFitnessInfo(w[i], std::max(ProcessClass::GoodFit, it.first.first), it.first.second);
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
throw no_more_servers();
}
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
vector<WorkerDetails> getWorkersForRoleInDatacenter(Optional<Standalone<StringRef>> const& dcId, ProcessClass::ClusterRole role, int amount, DatabaseConfiguration const& conf, std::map< Optional<Standalone<StringRef>>, int>& id_used, Optional<WorkerFitnessInfo> minWorker = Optional<WorkerFitnessInfo>(), bool checkStable = false ) {
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
std::map<std::pair<ProcessClass::Fitness,int>, std::pair<vector<WorkerDetails>,vector<WorkerDetails>>> fitness_workers;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
vector<WorkerDetails> results;
refactor getWorkersForRoleInDatacenter
2020-03-17 02:50:17 +08:00
if(minWorker.present()) {
results.push_back(minWorker.get().worker);
}
if (amount <= results.size()) {
Initial repository commit
2017-05-26 04:48:44 +08:00
return results;
refactor getWorkersForRoleInDatacenter
2020-03-17 02:50:17 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
for( auto& it : id_worker ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
auto fitness = it.second.details.processClass.machineClassFitness( role );
if( workerAvailable(it.second, checkStable) && !conf.isExcludedServer(it.second.details.interf.address()) && it.second.details.interf.locality.dcId() == dcId &&
( !minWorker.present() || ( it.second.details.interf.id() != minWorker.get().worker.interf.id() && ( fitness < minWorker.get().fitness || (fitness == minWorker.get().fitness && id_used[it.first] <= minWorker.get().used ) ) ) ) ) {
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
if ((db.serverInfo->get().read().distributor.present() && db.serverInfo->get().read().distributor.get().locality.processId() == it.first) ||
(db.serverInfo->get().read().ratekeeper.present() && db.serverInfo->get().read().ratekeeper.get().locality.processId() == it.first)) {
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
fitness_workers[ std::make_pair(fitness, id_used[it.first]) ].second.push_back(it.second.details);
} else {
fitness_workers[ std::make_pair(fitness, id_used[it.first]) ].first.push_back(it.second.details);
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
for( auto& it : fitness_workers ) {
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
for( int j=0; j < 2; j++ ) {
auto& w = j==0 ? it.second.first : it.second.second;
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
deterministicRandom()->randomShuffle(w);
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
for( int i=0; i < w.size(); i++ ) {
results.push_back(w[i]);
id_used[w[i].interf.locality.processId()]++;
if( results.size() == amount )
return results;
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
return results;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
struct RoleFitness {
ProcessClass::Fitness bestFit;
ProcessClass::Fitness worstFit;
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
ProcessClass::ClusterRole role;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
int count;
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
bool worstIsDegraded;
Initial repository commit
2017-05-26 04:48:44 +08:00
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
RoleFitness(int bestFit, int worstFit, int count, ProcessClass::ClusterRole role) : bestFit((ProcessClass::Fitness)bestFit), worstFit((ProcessClass::Fitness)worstFit), count(count), role(role), worstIsDegraded(false) {}
Initial repository commit
2017-05-26 04:48:44 +08:00
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
RoleFitness(int fitness, int count, ProcessClass::ClusterRole role) : bestFit((ProcessClass::Fitness)fitness), worstFit((ProcessClass::Fitness)fitness), count(count), role(role), worstIsDegraded(false) {}
Initial repository commit
2017-05-26 04:48:44 +08:00
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
RoleFitness() : bestFit(ProcessClass::NeverAssign), worstFit(ProcessClass::NeverAssign), role(ProcessClass::NoRole), count(0), worstIsDegraded(false) {}
Enable checkUsingDesiredClasses() in consistency check
2017-10-25 03:58:54 +08:00
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
RoleFitness( vector<WorkerDetails> workers, ProcessClass::ClusterRole role ) : role(role) {
one proxy or resolver with best fitness no longer prevents more proxies or resolvers from being recruited with good fitness
2019-10-15 09:32:17 +08:00
worstFit = ProcessClass::GoodFit;
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
worstIsDegraded = false;
fix: tlogFitness did not consider it better to have one tlog of a better fitness fix: checkStable was not used in all places in better master exists fix: we need to call checkOutstanding on worker registration in all cases fix: in case persistentData is keyValueStoreMemory, we need to make sure it is fully recovered before writing to it
2018-01-05 03:33:02 +08:00
bestFit = ProcessClass::NeverAssign;
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
for(auto& it : workers) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
auto thisFit = it.processClass.machineClassFitness( role );
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
if(thisFit > worstFit) {
worstFit = thisFit;
worstIsDegraded = it.degraded;
} else if(thisFit == worstFit) {
worstIsDegraded = worstIsDegraded || it.degraded;
}
fix: tlogFitness did not consider it better to have one tlog of a better fitness fix: checkStable was not used in all places in better master exists fix: we need to call checkOutstanding on worker registration in all cases fix: in case persistentData is keyValueStoreMemory, we need to make sure it is fully recovered before writing to it
2018-01-05 03:33:02 +08:00
bestFit = std::min(bestFit, thisFit);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
count = workers.size();
fix: degraded is only used for tlog recruitment, so we should not use it in the fitness calculation for other roles
2019-03-22 02:23:49 +08:00
//degraded is only used for recruitment of tlogs
if(role != ProcessClass::TLog) {
worstIsDegraded = false;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
bool operator < (RoleFitness const& r) const {
if (worstFit != r.worstFit) return worstFit < r.worstFit;
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
if (worstIsDegraded != r.worstIsDegraded) return r.worstIsDegraded;
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
// FIXME: TLog recruitment process does not guarantee the best fit is not worsened.
Update fdbserver/ClusterController.actor.cpp Co-Authored-By: bnamasivayam <36455962+bnamasivayam@users.noreply.github.com>
2019-01-15 08:57:00 +08:00
if (role != ProcessClass::TLog && role != ProcessClass::LogRouter && bestFit != r.bestFit) return bestFit < r.bestFit;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
return count > r.count;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
bool betterFitness (RoleFitness const& r) const {
fix: tlogFitness did not consider it better to have one tlog of a better fitness fix: checkStable was not used in all places in better master exists fix: we need to call checkOutstanding on worker registration in all cases fix: in case persistentData is keyValueStoreMemory, we need to make sure it is fully recovered before writing to it
2018-01-05 03:33:02 +08:00
if (worstFit != r.worstFit) return worstFit < r.worstFit;
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
if (worstIsDegraded != r.worstIsDegraded) return r.worstFit;
fix: tlogFitness did not consider it better to have one tlog of a better fitness fix: checkStable was not used in all places in better master exists fix: we need to call checkOutstanding on worker registration in all cases fix: in case persistentData is keyValueStoreMemory, we need to make sure it is fully recovered before writing to it
2018-01-05 03:33:02 +08:00
if (bestFit != r.bestFit) return bestFit < r.bestFit;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
return false;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Swapped OkayFit and UnsetFit, because generally if machine classes are set on one machine they are set everywhere and it helps with wait_for_good_recruitment logic wait_for_good_recruitment now requires that you have the desired count of each roll remote recruitment is given a much longer wait_for_good_recruitment time interval, which does not start until enough remote machines have registered
2018-06-23 01:15:24 +08:00
bool betterCount (RoleFitness const& r) const {
simplified betterCount logic
2018-06-23 01:38:36 +08:00
if(count > r.count) return true;
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
if(worstFit != r.worstFit) return worstFit < r.worstFit;
if (worstIsDegraded != r.worstIsDegraded) return r.worstFit;
return false;
Swapped OkayFit and UnsetFit, because generally if machine classes are set on one machine they are set everywhere and it helps with wait_for_good_recruitment logic wait_for_good_recruitment now requires that you have the desired count of each roll remote recruitment is given a much longer wait_for_good_recruitment time interval, which does not start until enough remote machines have registered
2018-06-23 01:15:24 +08:00
}
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
bool operator == (RoleFitness const& r) const { return worstFit == r.worstFit && bestFit == r.bestFit && count == r.count && worstIsDegraded == r.worstIsDegraded; }
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
std::string toString() const { return format("%d %d %d %d", bestFit, worstFit, count, worstIsDegraded); }
Initial repository commit
2017-05-26 04:48:44 +08:00
};
fix: better master exists must check if fitness is better for proxies or resolvers before looking at the count of either of them
2019-10-18 04:18:31 +08:00
struct RoleFitnessPair {
RoleFitness proxy;
RoleFitness resolver;
RoleFitnessPair() {}
RoleFitnessPair(RoleFitness const& proxy, RoleFitness const& resolver) : proxy(proxy), resolver(resolver) {}
bool operator < (RoleFitnessPair const& r) const {
if(proxy.betterFitness(r.proxy)) {
return true;
}
if(r.proxy.betterFitness(proxy)) {
return false;
}
if(resolver.betterFitness(r.resolver)) {
return true;
}
if(r.resolver.betterFitness(resolver)) {
return false;
}
if(proxy.count != r.proxy.count) {
return proxy.count > r.proxy.count;
}
return resolver.count > r.resolver.count;
}
bool operator == (RoleFitnessPair const& r) const { return proxy == r.proxy && resolver == r.resolver; }
};
Initial repository commit
2017-05-26 04:48:44 +08:00
std::set<Optional<Standalone<StringRef>>> getDatacenters( DatabaseConfiguration const& conf, bool checkStable = false ) {
std::set<Optional<Standalone<StringRef>>> result;
for( auto& it : id_worker )
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if( workerAvailable( it.second, checkStable ) && !conf.isExcludedServer( it.second.details.interf.address() ) )
result.insert(it.second.details.interf.locality.dcId());
Initial repository commit
2017-05-26 04:48:44 +08:00
return result;
}
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
void updateKnownIds(std::map< Optional<Standalone<StringRef>>, int>* id_used) {
(*id_used)[masterProcessId]++;
(*id_used)[clusterControllerProcessId]++;
}
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
RecruitRemoteFromConfigurationReply findRemoteWorkersForConfiguration( RecruitRemoteFromConfigurationRequest const& req ) {
RecruitRemoteFromConfigurationReply result;
Initial repository commit
2017-05-26 04:48:44 +08:00
std::map< Optional<Standalone<StringRef>>, int> id_used;
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
updateKnownIds(&id_used);
Initial repository commit
2017-05-26 04:48:44 +08:00
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
std::set<Optional<Key>> remoteDC;
remoteDC.insert(req.dcId);
added usable_regions to specify if we will replicate into a remote region remote replication defaults to the primary replication removed remote_logs, because they should be specified as an override in the regions object
2018-06-18 10:31:15 +08:00
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
auto remoteLogs = getWorkersForTlogs( req.configuration, req.configuration.getRemoteTLogReplicationFactor(), req.configuration.getDesiredRemoteLogs(), req.configuration.getRemoteTLogPolicy(), id_used, false, remoteDC, req.exclusionWorkerIds );
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
for(int i = 0; i < remoteLogs.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.remoteTLogs.push_back(remoteLogs[i].interf);
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
fix: we must commit to the number of log routers we are going to use when recruiting the primary, because it determines the number of log router tags that will be attached to mutations
2018-03-07 08:31:21 +08:00
auto logRouters = getWorkersForRoleInDatacenter( req.dcId, ProcessClass::LogRouter, req.logRouterCount, req.configuration, id_used );
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
for(int i = 0; i < logRouters.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.logRouters.push_back(logRouters[i].interf);
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
Swapped OkayFit and UnsetFit, because generally if machine classes are set on one machine they are set everywhere and it helps with wait_for_good_recruitment logic wait_for_good_recruitment now requires that you have the desired count of each roll remote recruitment is given a much longer wait_for_good_recruitment time interval, which does not start until enough remote machines have registered
2018-06-23 01:15:24 +08:00
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
if( !goodRemoteRecruitmentTime.isReady() &&
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
( ( RoleFitness(SERVER_KNOBS->EXPECTED_TLOG_FITNESS, req.configuration.getDesiredRemoteLogs(), ProcessClass::TLog).betterCount(RoleFitness(remoteLogs, ProcessClass::TLog)) ) ||
( RoleFitness(SERVER_KNOBS->EXPECTED_LOG_ROUTER_FITNESS, req.logRouterCount, ProcessClass::LogRouter).betterCount(RoleFitness(logRouters, ProcessClass::LogRouter)) ) ) ) {
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
throw operation_failed();
}
Initial repository commit
2017-05-26 04:48:44 +08:00
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
return result;
}
Initial repository commit
2017-05-26 04:48:44 +08:00
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
ErrorOr<RecruitFromConfigurationReply> findWorkersForConfiguration( RecruitFromConfigurationRequest const& req, Optional<Key> dcId ) {
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
RecruitFromConfigurationReply result;
std::map< Optional<Standalone<StringRef>>, int> id_used;
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
updateKnownIds(&id_used);
Merge branch 'master' into feature-remote-logs # Conflicts: # fdbrpc/Locality.cpp # fdbrpc/Locality.h # fdbserver/ClusterController.actor.cpp # fdbserver/ClusterRecruitmentInterface.h # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/fdbserver.vcxproj.filters # fdbserver/masterserver.actor.cpp # fdbserver/worker.actor.cpp # flow/error_definitions.h
2017-10-06 08:09:44 +08:00
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
ASSERT(dcId.present());
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
std::set<Optional<Key>> primaryDC;
primaryDC.insert(dcId);
result.dcId = dcId;
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
RegionInfo region;
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
RegionInfo remoteRegion;
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
for(auto& r : req.configuration.regions) {
first working version of non-copying recovery working with fearless configurations
2018-04-09 12:24:05 +08:00
if(r.dcId == dcId.get()) {
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
region = r;
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
}
else {
remoteRegion = r;
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
}
}
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
Merge branch 'master' into feature-remote-logs # Conflicts: # fdbrpc/Locality.cpp # fdbrpc/Locality.h # fdbserver/ClusterController.actor.cpp # fdbserver/ClusterRecruitmentInterface.h # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/fdbserver.vcxproj.filters # fdbserver/masterserver.actor.cpp # fdbserver/worker.actor.cpp # flow/error_definitions.h
2017-10-06 08:09:44 +08:00
if(req.recruitSeedServers) {
auto primaryStorageServers = getWorkersForSeedServers( req.configuration, req.configuration.storagePolicy, dcId );
first working version of non-copying recovery working with fearless configurations
2018-04-09 12:24:05 +08:00
for(int i = 0; i < primaryStorageServers.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.storageServers.push_back(primaryStorageServers[i].interf);
Merge branch 'master' into feature-remote-logs # Conflicts: # fdbrpc/Locality.cpp # fdbrpc/Locality.h # fdbserver/ClusterController.actor.cpp # fdbserver/ClusterRecruitmentInterface.h # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/fdbserver.vcxproj.filters # fdbserver/masterserver.actor.cpp # fdbserver/worker.actor.cpp # flow/error_definitions.h
2017-10-06 08:09:44 +08:00
}
}
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
Merge branch 'master' into feature-remote-logs # Conflicts: # fdbrpc/Locality.cpp # fdbrpc/Locality.h # fdbserver/ClusterController.actor.cpp # fdbserver/ClusterRecruitmentInterface.h # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/fdbserver.vcxproj.filters # fdbserver/masterserver.actor.cpp # fdbserver/worker.actor.cpp # flow/error_definitions.h
2017-10-06 08:09:44 +08:00
auto tlogs = getWorkersForTlogs( req.configuration, req.configuration.tLogReplicationFactor, req.configuration.getDesiredLogs(), req.configuration.tLogPolicy, id_used, false, primaryDC );
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
for(int i = 0; i < tlogs.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.tLogs.push_back(tlogs[i].interf);
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
std::vector<WorkerDetails> satelliteLogs;
do not recruit satellite tlogs when usable regions=1
2020-03-14 01:24:52 +08:00
if(region.satelliteTLogReplicationFactor > 0 && req.configuration.usableRegions > 1) {
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
satelliteLogs = getWorkersForSatelliteLogs( req.configuration, region, remoteRegion, id_used, result.satelliteFallback );
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
for(int i = 0; i < satelliteLogs.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.satelliteTLogs.push_back(satelliteLogs[i].interf);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
auto first_resolver = getWorkerForRoleInDatacenter( dcId, ProcessClass::Resolver, ProcessClass::ExcludeFit, req.configuration, id_used );
auto first_proxy = getWorkerForRoleInDatacenter( dcId, ProcessClass::Proxy, ProcessClass::ExcludeFit, req.configuration, id_used );
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
refactor getWorkersForRoleInDatacenter
2020-03-17 02:50:17 +08:00
auto proxies = getWorkersForRoleInDatacenter( dcId, ProcessClass::Proxy, req.configuration.getDesiredProxies(), req.configuration, id_used, first_proxy );
auto resolvers = getWorkersForRoleInDatacenter( dcId, ProcessClass::Resolver, req.configuration.getDesiredResolvers(), req.configuration, id_used, first_resolver );
Initial repository commit
2017-05-26 04:48:44 +08:00
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
for(int i = 0; i < resolvers.size(); i++)
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.resolvers.push_back(resolvers[i].interf);
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
for(int i = 0; i < proxies.size(); i++)
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.proxies.push_back(proxies[i].interf);
Initial repository commit
2017-05-26 04:48:44 +08:00
recruit oldLogRouters on TLogs, do not recruit oldLogRouters on the cluster controller if possible
2019-12-10 05:12:13 +08:00
if(req.maxOldLogRouters > 0) {
fix: we were recruiting one too few oldLogRouters code cleanup
2020-01-03 07:05:44 +08:00
if(tlogs.size() == 1) {
result.oldLogRouters.push_back(tlogs[0].interf);
recruit oldLogRouters on TLogs, do not recruit oldLogRouters on the cluster controller if possible
2019-12-10 05:12:13 +08:00
} else {
fix: we were recruiting one too few oldLogRouters code cleanup
2020-01-03 07:05:44 +08:00
for(int i = 0; i < tlogs.size(); i++) {
if(tlogs[i].interf.locality.processId() != clusterControllerProcessId) {
result.oldLogRouters.push_back(tlogs[i].interf);
recruit oldLogRouters on TLogs, do not recruit oldLogRouters on the cluster controller if possible
2019-12-10 05:12:13 +08:00
}
}
}
first version of non-copying recovery. Upgrades are broken, and it has not been tested using fearless configurations yet
2018-03-30 06:12:38 +08:00
}
fix: we must commit to the number of log routers we are going to use when recruiting the primary, because it determines the number of log router tags that will be attached to mutations
2018-03-07 08:31:21 +08:00
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
if( !goodRecruitmentTime.isReady() &&
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
( RoleFitness(SERVER_KNOBS->EXPECTED_TLOG_FITNESS, req.configuration.getDesiredLogs(), ProcessClass::TLog).betterCount(RoleFitness(tlogs, ProcessClass::TLog)) ||
do not recruit satellite tlogs when usable regions=1
2020-03-14 01:24:52 +08:00
( region.satelliteTLogReplicationFactor > 0 && req.configuration.usableRegions > 1 && RoleFitness(SERVER_KNOBS->EXPECTED_TLOG_FITNESS, req.configuration.getDesiredSatelliteLogs(dcId), ProcessClass::TLog).betterCount(RoleFitness(satelliteLogs, ProcessClass::TLog)) ) ||
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
RoleFitness(SERVER_KNOBS->EXPECTED_PROXY_FITNESS, req.configuration.getDesiredProxies(), ProcessClass::Proxy).betterCount(RoleFitness(proxies, ProcessClass::Proxy)) ||
RoleFitness(SERVER_KNOBS->EXPECTED_RESOLVER_FITNESS, req.configuration.getDesiredResolvers(), ProcessClass::Resolver).betterCount(RoleFitness(resolvers, ProcessClass::Resolver)) ) ) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
return operation_failed();
Initial repository commit
2017-05-26 04:48:44 +08:00
}
return result;
}
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
RecruitFromConfigurationReply findWorkersForConfiguration( RecruitFromConfigurationRequest const& req ) {
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
if(req.configuration.regions.size() > 1) {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
std::vector<RegionInfo> regions = req.configuration.regions;
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
if(regions[0].priority == regions[1].priority && regions[1].dcId == clusterControllerDcId.get()) {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
std::swap(regions[0], regions[1]);
}
fix: treat the local region as best when version difference is too large re-check requests when the version difference becomes small
2018-07-07 05:44:11 +08:00
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
if(regions[1].dcId == clusterControllerDcId.get() && regions[1].priority >= 0 && (!versionDifferenceUpdated || datacenterVersionDifference >= SERVER_KNOBS->MAX_VERSION_DIFFERENCE)) {
fix: treat the local region as best when version difference is too large re-check requests when the version difference becomes small
2018-07-07 05:44:11 +08:00
std::swap(regions[0], regions[1]);
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
bool setPrimaryDesired = false;
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
try {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
auto reply = findWorkersForConfiguration(req, regions[0].dcId);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
setPrimaryDesired = true;
vector<Optional<Key>> dcPriority;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
dcPriority.push_back(regions[0].dcId);
dcPriority.push_back(regions[1].dcId);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
desiredDcIds.set(dcPriority);
if(reply.isError()) {
throw reply.getError();
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
} else if(regions[0].dcId == clusterControllerDcId.get()) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
return reply.get();
}
throw no_more_servers();
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
} catch( Error& e ) {
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
if (!goodRemoteRecruitmentTime.isReady() && regions[1].dcId != clusterControllerDcId.get()) {
fix: the cluster controller needs to continue to retry recruitment until after wait_for_good_remote_recruitment_delay
2018-06-26 09:20:16 +08:00
throw operation_failed();
}
if (e.code() != error_code_no_more_servers || regions[1].priority < 0) {
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
throw;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
TraceEvent(SevWarn, "AttemptingRecruitmentInRemoteDC", id).error(e);
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
auto reply = findWorkersForConfiguration(req, regions[1].dcId);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
if(!setPrimaryDesired) {
vector<Optional<Key>> dcPriority;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
dcPriority.push_back(regions[1].dcId);
dcPriority.push_back(regions[0].dcId);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
desiredDcIds.set(dcPriority);
}
if(reply.isError()) {
throw reply.getError();
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
} else if (regions[1].dcId == clusterControllerDcId.get()) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
return reply.get();
}
throw;
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
added support for single region configurations
2018-03-16 01:59:30 +08:00
} else if(req.configuration.regions.size() == 1) {
vector<Optional<Key>> dcPriority;
dcPriority.push_back(req.configuration.regions[0].dcId);
desiredDcIds.set(dcPriority);
auto reply = findWorkersForConfiguration(req, req.configuration.regions[0].dcId);
if(reply.isError()) {
throw reply.getError();
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
} else if (req.configuration.regions[0].dcId == clusterControllerDcId.get()) {
added support for single region configurations
2018-03-16 01:59:30 +08:00
return reply.get();
}
throw no_more_servers();
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
} else {
RecruitFromConfigurationReply result;
std::map< Optional<Standalone<StringRef>>, int> id_used;
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
updateKnownIds(&id_used);
Merge branch 'master' into feature-remote-logs # Conflicts: # fdbrpc/Locality.cpp # fdbrpc/Locality.h # fdbserver/ClusterController.actor.cpp # fdbserver/ClusterRecruitmentInterface.h # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/fdbserver.vcxproj.filters # fdbserver/masterserver.actor.cpp # fdbserver/worker.actor.cpp # flow/error_definitions.h
2017-10-06 08:09:44 +08:00
auto tlogs = getWorkersForTlogs( req.configuration, req.configuration.tLogReplicationFactor, req.configuration.getDesiredLogs(), req.configuration.tLogPolicy, id_used );
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
for(int i = 0; i < tlogs.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.tLogs.push_back(tlogs[i].interf);
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
recruit oldLogRouters on TLogs, do not recruit oldLogRouters on the cluster controller if possible
2019-12-10 05:12:13 +08:00
if(req.maxOldLogRouters > 0) {
fix: we were recruiting one too few oldLogRouters code cleanup
2020-01-03 07:05:44 +08:00
if(tlogs.size() == 1) {
result.oldLogRouters.push_back(tlogs[0].interf);
recruit oldLogRouters on TLogs, do not recruit oldLogRouters on the cluster controller if possible
2019-12-10 05:12:13 +08:00
} else {
fix: we were recruiting one too few oldLogRouters code cleanup
2020-01-03 07:05:44 +08:00
for(int i = 0; i < tlogs.size(); i++) {
if(tlogs[i].interf.locality.processId() != clusterControllerProcessId) {
result.oldLogRouters.push_back(tlogs[i].interf);
recruit oldLogRouters on TLogs, do not recruit oldLogRouters on the cluster controller if possible
2019-12-10 05:12:13 +08:00
}
}
}
}
Merge branch 'master' into feature-remote-logs # Conflicts: # fdbrpc/Locality.cpp # fdbrpc/Locality.h # fdbserver/ClusterController.actor.cpp # fdbserver/ClusterRecruitmentInterface.h # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/fdbserver.vcxproj.filters # fdbserver/masterserver.actor.cpp # fdbserver/worker.actor.cpp # flow/error_definitions.h
2017-10-06 08:09:44 +08:00
if(req.recruitSeedServers) {
auto primaryStorageServers = getWorkersForSeedServers( req.configuration, req.configuration.storagePolicy );
for(int i = 0; i < primaryStorageServers.size(); i++)
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.storageServers.push_back(primaryStorageServers[i].interf);
Merge branch 'master' into feature-remote-logs # Conflicts: # fdbrpc/Locality.cpp # fdbrpc/Locality.h # fdbserver/ClusterController.actor.cpp # fdbserver/ClusterRecruitmentInterface.h # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/fdbserver.vcxproj.filters # fdbserver/masterserver.actor.cpp # fdbserver/worker.actor.cpp # flow/error_definitions.h
2017-10-06 08:09:44 +08:00
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
auto datacenters = getDatacenters( req.configuration );
fix: better master exists must check if fitness is better for proxies or resolvers before looking at the count of either of them
2019-10-18 04:18:31 +08:00
RoleFitnessPair bestFitness;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
int numEquivalent = 1;
Optional<Key> bestDC;
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
for(auto dcId : datacenters ) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
try {
//SOMEDAY: recruitment in other DCs besides the clusterControllerDcID will not account for the processes used by the master and cluster controller properly.
auto used = id_used;
auto first_resolver = getWorkerForRoleInDatacenter( dcId, ProcessClass::Resolver, ProcessClass::ExcludeFit, req.configuration, used );
auto first_proxy = getWorkerForRoleInDatacenter( dcId, ProcessClass::Proxy, ProcessClass::ExcludeFit, req.configuration, used );
refactor getWorkersForRoleInDatacenter
2020-03-17 02:50:17 +08:00
auto proxies = getWorkersForRoleInDatacenter( dcId, ProcessClass::Proxy, req.configuration.getDesiredProxies(), req.configuration, used, first_proxy );
auto resolvers = getWorkersForRoleInDatacenter( dcId, ProcessClass::Resolver, req.configuration.getDesiredResolvers(), req.configuration, used, first_resolver );
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
fix: better master exists must check if fitness is better for proxies or resolvers before looking at the count of either of them
2019-10-18 04:18:31 +08:00
RoleFitnessPair fitness( RoleFitness(proxies, ProcessClass::Proxy), RoleFitness(resolvers, ProcessClass::Resolver) );
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
if(dcId == clusterControllerDcId) {
bestFitness = fitness;
bestDC = dcId;
for(int i = 0; i < resolvers.size(); i++)
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.resolvers.push_back(resolvers[i].interf);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
for(int i = 0; i < proxies.size(); i++)
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.proxies.push_back(proxies[i].interf);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
break;
} else {
if(fitness < bestFitness) {
bestFitness = fitness;
numEquivalent = 1;
bestDC = dcId;
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
} else if( fitness == bestFitness && deterministicRandom()->random01() < 1.0/++numEquivalent ) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
bestDC = dcId;
}
}
} catch( Error &e ) {
if(e.code() != error_code_no_more_servers) {
throw;
}
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
if(bestDC != clusterControllerDcId) {
vector<Optional<Key>> dcPriority;
dcPriority.push_back(bestDC);
desiredDcIds.set(dcPriority);
throw no_more_servers();
}
//If this cluster controller dies, do not prioritize recruiting the next one in the same DC
desiredDcIds.set(vector<Optional<Key>>());
Attempt to normalize trace events: * Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check. * Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase. * Use seconds instead of milliseconds in details. Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed. This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-09 02:11:08 +08:00
TraceEvent("FindWorkersForConfig").detail("Replication", req.configuration.tLogReplicationFactor)
.detail("DesiredLogs", req.configuration.getDesiredLogs()).detail("ActualLogs", result.tLogs.size())
.detail("DesiredProxies", req.configuration.getDesiredProxies()).detail("ActualProxies", result.proxies.size())
.detail("DesiredResolvers", req.configuration.getDesiredResolvers()).detail("ActualResolvers", result.resolvers.size());
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
if( !goodRecruitmentTime.isReady() &&
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
( RoleFitness(SERVER_KNOBS->EXPECTED_TLOG_FITNESS, req.configuration.getDesiredLogs(), ProcessClass::TLog).betterCount(RoleFitness(tlogs, ProcessClass::TLog)) ||
fix: better master exists must check if fitness is better for proxies or resolvers before looking at the count of either of them
2019-10-18 04:18:31 +08:00
RoleFitness(SERVER_KNOBS->EXPECTED_PROXY_FITNESS, req.configuration.getDesiredProxies(), ProcessClass::Proxy).betterCount(bestFitness.proxy) ||
RoleFitness(SERVER_KNOBS->EXPECTED_RESOLVER_FITNESS, req.configuration.getDesiredResolvers(), ProcessClass::Resolver).betterCount(bestFitness.resolver) ) ) {
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
throw operation_failed();
}
return result;
}
}
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
void checkRegions(const std::vector<RegionInfo>& regions) {
if(desiredDcIds.get().present() && desiredDcIds.get().get().size() == 2 && desiredDcIds.get().get()[0].get() == regions[0].dcId && desiredDcIds.get().get()[1].get() == regions[1].dcId) {
return;
}
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
try {
std::map< Optional<Standalone<StringRef>>, int> id_used;
getWorkerForRoleInDatacenter(regions[0].dcId, ProcessClass::ClusterController, ProcessClass::ExcludeFit, db.config, id_used, true);
getWorkerForRoleInDatacenter(regions[0].dcId, ProcessClass::Master, ProcessClass::ExcludeFit, db.config, id_used, true);
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
std::set<Optional<Key>> primaryDC;
primaryDC.insert(regions[0].dcId);
fix: desiredTLogCount was used instead of getDesiredLogs(), which caused problems with recruitment when desiredTLogCount was -1. canKillProcess logic was wrong. We still need to configure usable_regions because if datacenterVersionDifference is too large we cannot complete data movement.
2018-07-05 04:22:32 +08:00
getWorkersForTlogs(db.config, db.config.tLogReplicationFactor, db.config.getDesiredLogs(), db.config.tLogPolicy, id_used, true, primaryDC);
do not recruit satellite tlogs when usable regions=1
2020-03-14 01:24:52 +08:00
if(regions[0].satelliteTLogReplicationFactor > 0 && db.config.usableRegions > 1) {
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
bool satelliteFallback = false;
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
getWorkersForSatelliteLogs(db.config, regions[0], regions[1], id_used, satelliteFallback, true);
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
getWorkerForRoleInDatacenter( regions[0].dcId, ProcessClass::Resolver, ProcessClass::ExcludeFit, db.config, id_used, true );
getWorkerForRoleInDatacenter( regions[0].dcId, ProcessClass::Proxy, ProcessClass::ExcludeFit, db.config, id_used, true );
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
vector<Optional<Key>> dcPriority;
dcPriority.push_back(regions[0].dcId);
dcPriority.push_back(regions[1].dcId);
desiredDcIds.set(dcPriority);
} catch( Error &e ) {
if(e.code() != error_code_no_more_servers) {
throw;
}
}
}
void checkRecoveryStalled() {
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
if( (db.serverInfo->get().read().recoveryState == RecoveryState::RECRUITING || db.serverInfo->get().read().recoveryState == RecoveryState::ACCEPTING_COMMITS || db.serverInfo->get().read().recoveryState == RecoveryState::ALL_LOGS_RECRUITED) && db.recoveryStalled ) {
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
if (db.config.regions.size() > 1) {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
auto regions = db.config.regions;
if(clusterControllerDcId.get() == regions[0].dcId) {
std::swap(regions[0], regions[1]);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
ASSERT(clusterControllerDcId.get() == regions[1].dcId);
checkRegions(regions);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
}
}
//FIXME: determine when to fail the cluster controller when a primaryDC has not been set
Initial repository commit
2017-05-26 04:48:44 +08:00
bool betterMasterExists() {
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
const ServerDBInfo dbi = db.serverInfo->get().read();
Initial repository commit
2017-05-26 04:48:44 +08:00
fix: kill all tlogs in one region prevents the remote logs from recovering in that region, do not allow that to prevent us from configuring usable_regions=1. added more recovery states.
2018-07-05 15:08:51 +08:00
if(dbi.recoveryState < RecoveryState::ACCEPTING_COMMITS) {
Initial repository commit
2017-05-26 04:48:44 +08:00
return false;
}
Merge release-6.0 into master
2018-09-22 07:14:39 +08:00
// Do not trigger better master exists if the cluster controller is excluded, since the master will change anyways once the cluster controller is moved
if(id_worker[clusterControllerProcessId].priorityInfo.isExcluded) {
return false;
}
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
if (db.config.regions.size() > 1 && db.config.regions[0].priority > db.config.regions[1].priority &&
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
db.config.regions[0].dcId != clusterControllerDcId.get() && versionDifferenceUpdated && datacenterVersionDifference < SERVER_KNOBS->MAX_VERSION_DIFFERENCE) {
checkRegions(db.config.regions);
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
Fix double recoveries
2017-11-17 08:58:55 +08:00
// Get master process
Initial repository commit
2017-05-26 04:48:44 +08:00
auto masterWorker = id_worker.find(dbi.master.locality.processId());
Fix double recoveries
2017-11-17 08:58:55 +08:00
if(masterWorker == id_worker.end()) {
Initial repository commit
2017-05-26 04:48:44 +08:00
return false;
Enable checkUsingDesiredClasses() in consistency check
2017-10-25 03:58:54 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
Fix double recoveries
2017-11-17 08:58:55 +08:00
// Get tlog processes
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
std::vector<WorkerDetails> tlogs;
std::vector<WorkerDetails> remote_tlogs;
std::vector<WorkerDetails> satellite_tlogs;
std::vector<WorkerDetails> log_routers;
fix: we may not be able to recruit enough log routers, in this case put multiple log routers on the same worker, but also properly rank this configuration lower in better master exists
2018-04-27 13:18:07 +08:00
std::set<NetworkAddress> logRouterAddresses;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
for( auto& logSet : dbi.logSystemConfig.tLogs ) {
for( auto& it : logSet.tLogs ) {
auto tlogWorker = id_worker.find(it.interf().locality.processId());
if ( tlogWorker == id_worker.end() )
return false;
if ( tlogWorker->second.priorityInfo.isExcluded )
return true;
removed hasBestPolicy
2018-06-16 03:36:19 +08:00
if(logSet.isLocal && logSet.locality == tagLocalitySatellite) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
satellite_tlogs.push_back(tlogWorker->second.details);
removed hasBestPolicy
2018-06-16 03:36:19 +08:00
}
else if(logSet.isLocal) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
tlogs.push_back(tlogWorker->second.details);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
} else {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
remote_tlogs.push_back(tlogWorker->second.details);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
}
for( auto& it : logSet.logRouters ) {
auto tlogWorker = id_worker.find(it.interf().locality.processId());
if ( tlogWorker == id_worker.end() )
return false;
if ( tlogWorker->second.priorityInfo.isExcluded )
return true;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if( !logRouterAddresses.count( tlogWorker->second.details.interf.address() ) ) {
logRouterAddresses.insert( tlogWorker->second.details.interf.address() );
log_routers.push_back(tlogWorker->second.details);
fix: we may not be able to recruit enough log routers, in this case put multiple log routers on the same worker, but also properly rank this configuration lower in better master exists
2018-04-27 13:18:07 +08:00
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Fix double recoveries
2017-11-17 08:58:55 +08:00
// Get proxy classes
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
std::vector<WorkerDetails> proxyClasses;
Initial repository commit
2017-05-26 04:48:44 +08:00
for(auto& it : dbi.client.proxies ) {
auto proxyWorker = id_worker.find(it.locality.processId());
if ( proxyWorker == id_worker.end() )
return false;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
if ( proxyWorker->second.priorityInfo.isExcluded )
Fix double recoveries
2017-11-17 08:58:55 +08:00
return true;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
proxyClasses.push_back(proxyWorker->second.details);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Fix double recoveries
2017-11-17 08:58:55 +08:00
// Get resolver classes
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
std::vector<WorkerDetails> resolverClasses;
Initial repository commit
2017-05-26 04:48:44 +08:00
for(auto& it : dbi.resolvers ) {
auto resolverWorker = id_worker.find(it.locality.processId());
if ( resolverWorker == id_worker.end() )
return false;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
if ( resolverWorker->second.priorityInfo.isExcluded )
Fix double recoveries
2017-11-17 08:58:55 +08:00
return true;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
resolverClasses.push_back(resolverWorker->second.details);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Fix double recoveries
2017-11-17 08:58:55 +08:00
// Check master fitness. Don't return false if master is excluded in case all the processes are excluded, we still need master for recovery.
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
ProcessClass::Fitness oldMasterFit = masterWorker->second.details.processClass.machineClassFitness( ProcessClass::Master );
Fix double recoveries
2017-11-17 08:58:55 +08:00
if(db.config.isExcludedServer(dbi.master.address())) {
oldMasterFit = std::max(oldMasterFit, ProcessClass::ExcludeFit);
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
std::map< Optional<Standalone<StringRef>>, int> id_used;
id_used[clusterControllerProcessId]++;
WorkerFitnessInfo mworker = getWorkerForRoleInDatacenter(clusterControllerDcId, ProcessClass::Master, ProcessClass::NeverAssign, db.config, id_used, true);
one proxy or resolver with best fitness no longer prevents more proxies or resolvers from being recruited with good fitness
2019-10-15 09:32:17 +08:00
auto newMasterFit = mworker.worker.processClass.machineClassFitness( ProcessClass::Master );
if(db.config.isExcludedServer(mworker.worker.interf.address())) {
newMasterFit = std::max(newMasterFit, ProcessClass::ExcludeFit);
}
Fix double recoveries
2017-11-17 08:58:55 +08:00
one proxy or resolver with best fitness no longer prevents more proxies or resolvers from being recruited with good fitness
2019-10-15 09:32:17 +08:00
if ( oldMasterFit < newMasterFit )
Fix double recoveries
2017-11-17 08:58:55 +08:00
return false;
one proxy or resolver with best fitness no longer prevents more proxies or resolvers from being recruited with good fitness
2019-10-15 09:32:17 +08:00
if ( oldMasterFit > newMasterFit || ( dbi.master.locality.processId() == clusterControllerProcessId && mworker.worker.interf.locality.processId() != clusterControllerProcessId ) )
Fix double recoveries
2017-11-17 08:58:55 +08:00
return true;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
std::set<Optional<Key>> primaryDC;
std::set<Optional<Key>> remoteDC;
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
RegionInfo region;
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
RegionInfo remoteRegion;
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
if (db.config.regions.size()) {
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
primaryDC.insert(clusterControllerDcId);
for(auto& r : db.config.regions) {
if(r.dcId != clusterControllerDcId.get()) {
ASSERT(remoteDC.empty());
remoteDC.insert(r.dcId);
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
remoteRegion = r;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
} else {
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
ASSERT(region.dcId == StringRef());
region = r;
}
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
Fix double recoveries
2017-11-17 08:58:55 +08:00
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
// Check tLog fitness
RoleFitness oldTLogFit(tlogs, ProcessClass::TLog);
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
auto newTLogs = getWorkersForTlogs(db.config, db.config.tLogReplicationFactor, db.config.getDesiredLogs(), db.config.tLogPolicy, id_used, true, primaryDC);
RoleFitness newTLogFit(newTLogs, ProcessClass::TLog);
Fix double recoveries
2017-11-17 08:58:55 +08:00
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
if(oldTLogFit < newTLogFit) return false;
Fix double recoveries
2017-11-17 08:58:55 +08:00
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
bool oldSatelliteFallback = false;
one proxy or resolver with best fitness no longer prevents more proxies or resolvers from being recruited with good fitness
2019-10-15 09:32:17 +08:00
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
for(auto& logSet : dbi.logSystemConfig.tLogs) {
if(logSet.isLocal && logSet.locality == tagLocalitySatellite) {
oldSatelliteFallback = logSet.tLogPolicy->info() != region.satelliteTLogPolicy->info();
ASSERT(!oldSatelliteFallback || logSet.tLogPolicy->info() == region.satelliteTLogPolicyFallback->info());
break;
}
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
RoleFitness oldSatelliteTLogFit(satellite_tlogs, ProcessClass::TLog);
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
bool newSatelliteFallback = false;
do not recruit satellite tlogs when usable regions=1
2020-03-14 01:24:52 +08:00
auto newSatelliteTLogs = (region.satelliteTLogReplicationFactor > 0 && db.config.usableRegions > 1) ? getWorkersForSatelliteLogs(db.config, region, remoteRegion, id_used, newSatelliteFallback, true) : satellite_tlogs;
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
RoleFitness newSatelliteTLogFit(newSatelliteTLogs, ProcessClass::TLog);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
fix: the cluster controller would not change to a new set of satellite tlogs when they become available in a better satellite location
2019-10-15 09:31:23 +08:00
std::map<Optional<Key>,int32_t> satellite_priority;
for(auto& r : region.satellites) {
satellite_priority[r.dcId] = r.priority;
}
int32_t oldSatelliteRegionFit = std::numeric_limits<int32_t>::max();
for(auto& it : satellite_tlogs) {
if(satellite_priority.count(it.interf.locality.dcId())) {
oldSatelliteRegionFit = std::min(oldSatelliteRegionFit, satellite_priority[it.interf.locality.dcId()]);
} else {
oldSatelliteRegionFit = -1;
}
}
int32_t newSatelliteRegionFit = std::numeric_limits<int32_t>::max();
for(auto& it : newSatelliteTLogs) {
if(satellite_priority.count(it.interf.locality.dcId())) {
newSatelliteRegionFit = std::min(newSatelliteRegionFit, satellite_priority[it.interf.locality.dcId()]);
} else {
newSatelliteRegionFit = -1;
}
}
if(oldSatelliteFallback && !newSatelliteFallback)
return true;
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
if(!oldSatelliteFallback && newSatelliteFallback)
return false;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
fix: the cluster controller would not change to a new set of satellite tlogs when they become available in a better satellite location
2019-10-15 09:31:23 +08:00
if(oldSatelliteRegionFit < newSatelliteRegionFit)
return true;
if(oldSatelliteRegionFit > newSatelliteRegionFit)
return false;
if(oldSatelliteTLogFit < newSatelliteTLogFit)
return false;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
RoleFitness oldRemoteTLogFit(remote_tlogs, ProcessClass::TLog);
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
std::vector<UID> exclusionWorkerIds;
auto fn = [](const WorkerDetails &in) { return in.interf.id(); };
std::transform(newTLogs.begin(), newTLogs.end(), std::back_inserter(exclusionWorkerIds), fn);
std::transform(newSatelliteTLogs.begin(), newSatelliteTLogs.end(), std::back_inserter(exclusionWorkerIds), fn);
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
RoleFitness newRemoteTLogFit(
fix: the cluster controller would not recruit more remote logs before the database became fully_recovered
2020-01-11 04:21:48 +08:00
(db.config.usableRegions > 1 && (dbi.recoveryState == RecoveryState::ALL_LOGS_RECRUITED || dbi.recoveryState == RecoveryState::FULLY_RECOVERED)) ?
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
getWorkersForTlogs(db.config, db.config.getRemoteTLogReplicationFactor(), db.config.getDesiredRemoteLogs(), db.config.getRemoteTLogPolicy(), id_used, true, remoteDC, exclusionWorkerIds)
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
: remote_tlogs, ProcessClass::TLog);
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
if(oldRemoteTLogFit < newRemoteTLogFit) return false;
re-added the ability to configure the number of log routers. Many log routers are needed to get a sufficient number of sockets involved in copying data across the WAN
2018-06-22 15:04:00 +08:00
int oldRouterCount = oldTLogFit.count * std::max<int>(1, db.config.desiredLogRouterCount / std::max(1,oldTLogFit.count));
int newRouterCount = newTLogFit.count * std::max<int>(1, db.config.desiredLogRouterCount / std::max(1,newTLogFit.count));
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
RoleFitness oldLogRoutersFit(log_routers, ProcessClass::LogRouter);
fix: kill all tlogs in one region prevents the remote logs from recovering in that region, do not allow that to prevent us from configuring usable_regions=1. added more recovery states.
2018-07-05 15:08:51 +08:00
RoleFitness newLogRoutersFit((db.config.usableRegions > 1 && dbi.recoveryState == RecoveryState::FULLY_RECOVERED) ? getWorkersForRoleInDatacenter( *remoteDC.begin(), ProcessClass::LogRouter, newRouterCount, db.config, id_used, Optional<WorkerFitnessInfo>(), true ) : log_routers, ProcessClass::LogRouter);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
re-added the ability to configure the number of log routers. Many log routers are needed to get a sufficient number of sockets involved in copying data across the WAN
2018-06-22 15:04:00 +08:00
if(oldLogRoutersFit.count < oldRouterCount) {
fix: we may not be able to recruit enough log routers, in this case put multiple log routers on the same worker, but also properly rank this configuration lower in better master exists
2018-04-27 13:18:07 +08:00
oldLogRoutersFit.worstFit = ProcessClass::NeverAssign;
}
re-added the ability to configure the number of log routers. Many log routers are needed to get a sufficient number of sockets involved in copying data across the WAN
2018-06-22 15:04:00 +08:00
if(newLogRoutersFit.count < newRouterCount) {
fix: we may not be able to recruit enough log routers, in this case put multiple log routers on the same worker, but also properly rank this configuration lower in better master exists
2018-04-27 13:18:07 +08:00
newLogRoutersFit.worstFit = ProcessClass::NeverAssign;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
if(oldLogRoutersFit < newLogRoutersFit) return false;
Fix double recoveries
2017-11-17 08:58:55 +08:00
// Check proxy/resolver fitness
fix: better master exists must check if fitness is better for proxies or resolvers before looking at the count of either of them
2019-10-18 04:18:31 +08:00
RoleFitnessPair oldInFit(RoleFitness(proxyClasses, ProcessClass::Proxy), RoleFitness(resolverClasses, ProcessClass::Resolver));
Initial repository commit
2017-05-26 04:48:44 +08:00
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
auto first_resolver = getWorkerForRoleInDatacenter( clusterControllerDcId, ProcessClass::Resolver, ProcessClass::ExcludeFit, db.config, id_used, true );
auto first_proxy = getWorkerForRoleInDatacenter( clusterControllerDcId, ProcessClass::Proxy, ProcessClass::ExcludeFit, db.config, id_used, true );
Initial repository commit
2017-05-26 04:48:44 +08:00
refactor getWorkersForRoleInDatacenter
2020-03-17 02:50:17 +08:00
auto proxies = getWorkersForRoleInDatacenter( clusterControllerDcId, ProcessClass::Proxy, db.config.getDesiredProxies(), db.config, id_used, first_proxy, true );
auto resolvers = getWorkersForRoleInDatacenter( clusterControllerDcId, ProcessClass::Resolver, db.config.getDesiredResolvers(), db.config, id_used, first_resolver, true );
Initial repository commit
2017-05-26 04:48:44 +08:00
fix: better master exists must check if fitness is better for proxies or resolvers before looking at the count of either of them
2019-10-18 04:18:31 +08:00
RoleFitnessPair newInFit(RoleFitness(proxies, ProcessClass::Proxy), RoleFitness(resolvers, ProcessClass::Resolver));
if(oldInFit.proxy.betterFitness(newInFit.proxy) || oldInFit.resolver.betterFitness(newInFit.resolver)) {
return false;
}
if(oldTLogFit > newTLogFit || oldInFit > newInFit || oldSatelliteTLogFit > newSatelliteTLogFit || oldRemoteTLogFit > newRemoteTLogFit || oldLogRoutersFit > newLogRoutersFit) {
one proxy or resolver with best fitness no longer prevents more proxies or resolvers from being recruited with good fitness
2019-10-15 09:32:17 +08:00
TraceEvent("BetterMasterExists", id).detail("OldMasterFit", oldMasterFit).detail("NewMasterFit", newMasterFit)
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
.detail("OldTLogFit", oldTLogFit.toString()).detail("NewTLogFit", newTLogFit.toString())
fix: better master exists must check if fitness is better for proxies or resolvers before looking at the count of either of them
2019-10-18 04:18:31 +08:00
.detail("OldProxyFit", oldInFit.proxy.toString()).detail("NewProxyFit", newInFit.proxy.toString())
.detail("OldResolverFit", oldInFit.resolver.toString()).detail("NewResolverFit", newInFit.resolver.toString())
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
.detail("OldSatelliteFit", oldSatelliteTLogFit.toString()).detail("NewSatelliteFit", newSatelliteTLogFit.toString())
.detail("OldRemoteFit", oldRemoteTLogFit.toString()).detail("NewRemoteFit", newRemoteTLogFit.toString())
.detail("OldRouterFit", oldLogRoutersFit.toString()).detail("NewRouterFit", newLogRoutersFit.toString())
.detail("OldSatelliteFallback", oldSatelliteFallback).detail("NewSatelliteFallback", newSatelliteFallback);
Initial repository commit
2017-05-26 04:48:44 +08:00
return true;
}
Fix double recoveries
2017-11-17 08:58:55 +08:00
Initial repository commit
2017-05-26 04:48:44 +08:00
return false;
}
fix: dd test recruitment should prefer the location of ratekeeper over other used processes
2019-11-14 04:58:55 +08:00
bool isUsedNotMaster(Optional<Key> processId) {
Recruit a new ratekeeper before halting the old
2019-03-20 06:21:46 +08:00
ASSERT(masterProcessId.present());
Update fdbserver/ClusterController.actor.cpp
2019-03-23 08:00:52 +08:00
if (processId == masterProcessId) return false;
Recruit a new ratekeeper before halting the old
2019-03-20 06:21:46 +08:00
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
auto& dbInfo = db.serverInfo->get().read();
fix: dd test recruitment should prefer the location of ratekeeper over other used processes
2019-11-14 04:58:55 +08:00
for (const auto& tlogset : dbInfo.logSystemConfig.tLogs) {
for (const auto& tlog: tlogset.tLogs) {
if (tlog.present() && tlog.interf().locality.processId() == processId) return true;
}
}
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
for (const MasterProxyInterface& interf : dbInfo.client.proxies) {
if (interf.locality.processId() == processId) return true;
}
for (const ResolverInterface& interf: dbInfo.resolvers) {
if (interf.locality.processId() == processId) return true;
}
Fix bug where DD or RK could be halted and re-recruited in a loop for certain valid process class configurations. Specifically, recruitment of DD or RK takes into account that master process is preferred over proxy, resolver or cc. But check for better DD only looks for better machine class ignoring that the new recruit could share a proxy or resolver or CC. Also try to balance the distribution of the DD and RK role if there are enough processes to do so.
2019-11-13 06:22:36 +08:00
if (processId == clusterControllerProcessId) return true;
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
return false;
}
bool onMasterIsBetter(const WorkerDetails& worker, ProcessClass::ClusterRole role) {
ASSERT(masterProcessId.present());
Fix ratekeeper and data distributor recruiting bug Avoid multiple concurrent recuriting of ratekeepers with a recruiting flag. Fix endless recruiting when the chosen worker is a proxy or a resolver -- prefer master in this case.
2019-03-21 01:00:31 +08:00
const auto& pid = worker.interf.locality.processId();
Changes to ratekeeper camel case
2019-03-27 23:24:25 +08:00
if ((role != ProcessClass::DataDistributor && role != ProcessClass::Ratekeeper) || pid == masterProcessId.get()) {
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
return false;
}
fix: dd test recruitment should prefer the location of ratekeeper over other used processes
2019-11-14 04:58:55 +08:00
return isUsedNotMaster(pid);
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
}
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
std::map< Optional<Standalone<StringRef>>, int> getUsedIds() {
Add used worker IDs to cluster controller This "usedIds" is updated when receiving a master registration message, so that when recruiting new data distributor, existing assignment is known.
2019-01-29 01:25:15 +08:00
std::map<Optional<Standalone<StringRef>>, int> idUsed;
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
updateKnownIds(&idUsed);
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
auto& dbInfo = db.serverInfo->get().read();
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
for (const auto& tlogset : dbInfo.logSystemConfig.tLogs) {
Various review comments fixes
2019-02-01 02:10:41 +08:00
for (const auto& tlog: tlogset.tLogs) {
if (tlog.present()) {
idUsed[tlog.interf().locality.processId()]++;
Add used worker IDs to cluster controller This "usedIds" is updated when receiving a master registration message, so that when recruiting new data distributor, existing assignment is known.
2019-01-29 01:25:15 +08:00
}
}
}
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
for (const MasterProxyInterface& interf : dbInfo.client.proxies) {
Add used worker IDs to cluster controller This "usedIds" is updated when receiving a master registration message, so that when recruiting new data distributor, existing assignment is known.
2019-01-29 01:25:15 +08:00
ASSERT(interf.locality.processId().present());
idUsed[interf.locality.processId()]++;
}
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
for (const ResolverInterface& interf: dbInfo.resolvers) {
Add used worker IDs to cluster controller This "usedIds" is updated when receiving a master registration message, so that when recruiting new data distributor, existing assignment is known.
2019-01-29 01:25:15 +08:00
ASSERT(interf.locality.processId().present());
idUsed[interf.locality.processId()]++;
}
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
return idUsed;
Add used worker IDs to cluster controller This "usedIds" is updated when receiving a master registration message, so that when recruiting new data distributor, existing assignment is known.
2019-01-29 01:25:15 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
std::map< Optional<Standalone<StringRef>>, WorkerInfo > id_worker;
Add used worker IDs to cluster controller This "usedIds" is updated when receiving a master registration message, so that when recruiting new data distributor, existing assignment is known.
2019-01-29 01:25:15 +08:00
std::map< Optional<Standalone<StringRef>>, ProcessClass > id_class; //contains the mapping from process id to process class from the database
Initial repository commit
2017-05-26 04:48:44 +08:00
Standalone<RangeResultRef> lastProcessClasses;
bool gotProcessClasses;
Add fullyRecoveredConfig for checking exclusions
2017-11-16 09:15:24 +08:00
bool gotFullyRecoveredConfig;
Initial repository commit
2017-05-26 04:48:44 +08:00
Optional<Standalone<StringRef>> masterProcessId;
Use processId instead of address for comparison
2017-10-26 02:35:29 +08:00
Optional<Standalone<StringRef>> clusterControllerProcessId;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
Optional<Standalone<StringRef>> clusterControllerDcId;
AsyncVar<Optional<vector<Optional<Key>>>> desiredDcIds; //desired DC priorities
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
AsyncVar<std::pair<bool,Optional<vector<Optional<Key>>>>> changingDcIds; //current DC priorities to change first, and whether that is the cluster controller
AsyncVar<std::pair<bool,Optional<vector<Optional<Key>>>>> changedDcIds; //current DC priorities to change second, and whether the cluster controller has been changed
Initial repository commit
2017-05-26 04:48:44 +08:00
UID id;
std::vector<RecruitFromConfigurationRequest> outstandingRecruitmentRequests;
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
std::vector<RecruitRemoteFromConfigurationRequest> outstandingRemoteRecruitmentRequests;
Initial repository commit
2017-05-26 04:48:44 +08:00
std::vector<std::pair<RecruitStorageRequest, double>> outstandingStorageRequests;
ActorCollection ac;
UpdateWorkerList updateWorkerList;
checking outstanding requests can be CPU intensive, so rate limit checking requests
2018-06-28 14:02:08 +08:00
Future<Void> outstandingRequestChecker;
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
Future<Void> outstandingRemoteRequestChecker;
Initial repository commit
2017-05-26 04:48:44 +08:00
DBInfo db;
Database cx;
double startTime;
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
Future<Void> goodRecruitmentTime;
Future<Void> goodRemoteRecruitmentTime;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
Version datacenterVersionDifference;
bool versionDifferenceUpdated;
Add DataDistributorInterface to ServerDBInfo Also change the Proxy and QuietDatabase to use the DataDistributorInterface.
2019-01-19 03:30:18 +08:00
PromiseStream<Future<Void>> addActor;
Fix the issue of slow dying Data Distributor Test with: -r simulation -f ./foundationdb/tests/slow/CommitBug.txt -s 67828576 -b on The test has the following event sequence: - Time 113.3s, CC noticed DD failure, cleard DD interface. - 1s later, DD rejoined and registered with CC. - Time 131.7s, DD actor cancelled. This old DD raced to register with CC and the failure monitor is not installed because monitorDataDistributor is stalled waiting for new DD. - Time 161.4s, new DD running. New DD recruting was delayed due to no servers in the period. Fix by disabling DD registration during the recruting process.
2019-03-18 11:55:59 +08:00
bool recruitingDistributor;
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
Optional<UID> recruitingRatekeeperID;
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
AsyncVar<bool> recruitRatekeeper;
Initial repository commit
2017-05-26 04:48:44 +08:00
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
CounterCollection clusterControllerMetrics;
Counter openDatabaseRequests;
Counter registerWorkerRequests;
Counter getWorkersRequests;
Counter getClientWorkersRequests;
Counter registerMasterRequests;
Counter getServerDBInfoRequests;
Counter statusRequests;
Counter failureMonitoringRequests;
Counter serversFailed;
Counter serversUnfailed;
fix: the cluster controller did not pass in its own locality when creating its database object, therefore it was not using locality aware load balancing
2018-09-29 03:12:06 +08:00
ClusterControllerData( ClusterControllerFullInterface const& ccInterface, LocalityData const& locality )
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
: clusterControllerProcessId(locality.processId()), clusterControllerDcId(locality.dcId()),
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
id(ccInterface.id()), ac(false), outstandingRequestChecker(Void()), outstandingRemoteRequestChecker(Void()), gotProcessClasses(false),
updateGoodRemoteRecruitmentTime was unnecessary because the only way findRemoteWorkers would return would be after a new server has joined which already resets goodRemoteRecruitmentTime
2020-02-21 05:46:22 +08:00
gotFullyRecoveredConfig(false), startTime(now()), goodRecruitmentTime(Never()),
goodRemoteRecruitmentTime(Never()), datacenterVersionDifference(0),
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
versionDifferenceUpdated(false), recruitingDistributor(false), recruitRatekeeper(false),
clusterControllerMetrics("ClusterController", id.toString()),
openDatabaseRequests("OpenDatabaseRequests", clusterControllerMetrics),
registerWorkerRequests("RegisterWorkerRequests", clusterControllerMetrics),
getWorkersRequests("GetWorkersRequests", clusterControllerMetrics),
getClientWorkersRequests("GetClientWorkersRequests", clusterControllerMetrics),
registerMasterRequests("RegisterMasterRequests", clusterControllerMetrics),
getServerDBInfoRequests("GetServerDBInfoRequests", clusterControllerMetrics),
statusRequests("StatusRequests", clusterControllerMetrics),
failureMonitoringRequests("FailureMonitoringRequests", clusterControllerMetrics),
serversFailed("ServersFailed", clusterControllerMetrics),
serversUnfailed("ServersUnfailed", clusterControllerMetrics)
Initial repository commit
2017-05-26 04:48:44 +08:00
{
during recovery, only send the full serverDBInfo to processes that are part of the new generation
2019-12-10 05:17:49 +08:00
auto& serverInfo = db.serverInfoMasterOnly.mutate();
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
serverInfo.id = deterministicRandom()->randomUniqueID();
Initial repository commit
2017-05-26 04:48:44 +08:00
serverInfo.masterLifetime.ccID = id;
serverInfo.clusterInterface = ccInterface;
fix: the cluster controller did not pass in its own locality when creating its database object, therefore it was not using locality aware load balancing
2018-09-29 03:12:06 +08:00
serverInfo.myLocality = locality;
during recovery, only send the full serverDBInfo to processes that are part of the new generation
2019-12-10 05:17:49 +08:00
db.serverInfo->set( db.serverInfoMasterOnly );
A giant translation of TaskFooPriority -> TaskPriority::Foo This is so that APIs that take priorities don't take ints, which are common and easy to accidentally pass the wrong thing.
2019-06-25 17:47:35 +08:00
cx = openDBOnServer(db.serverInfo, TaskPriority::DefaultEndpoint, true, true);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
~ClusterControllerData() {
ac.clear(false);
id_worker.clear();
}
};
ACTOR Future<Void> clusterWatchDatabase( ClusterControllerData* cluster, ClusterControllerData::DBInfo* db )
{
state MasterInterface iMaster;
// SOMEDAY: If there is already a non-failed master referenced by zkMasterInfo, use that one until it fails
// When this someday is implemented, make sure forced failures still cause the master to be recruited again
loop {
TraceEvent("CCWDB", cluster->id);
try {
state double recoveryStart = now();
TraceEvent("CCWDB", cluster->id).detail("Recruiting", "Master");
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
//We must recruit the master in the same data center as the cluster controller.
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
//This should always be possible, because we can recruit the master on the same process as the cluster controller.
std::map< Optional<Standalone<StringRef>>, int> id_used;
id_used[cluster->clusterControllerProcessId]++;
state WorkerFitnessInfo masterWorker = cluster->getWorkerForRoleInDatacenter(cluster->clusterControllerDcId, ProcessClass::Master, ProcessClass::NeverAssign, db->config, id_used);
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if( ( masterWorker.worker.processClass.machineClassFitness( ProcessClass::Master ) > SERVER_KNOBS->EXPECTED_MASTER_FITNESS || masterWorker.worker.interf.locality.processId() == cluster->clusterControllerProcessId )
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
&& !cluster->goodRecruitmentTime.isReady() ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
TraceEvent("CCWDB", cluster->id).detail("Fitness", masterWorker.worker.processClass.machineClassFitness( ProcessClass::Master ));
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( delay(SERVER_KNOBS->ATTEMPT_RECRUITMENT_DELAY) );
Initial repository commit
2017-05-26 04:48:44 +08:00
continue;
}
RecruitMasterRequest rmq;
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
rmq.lifetime = db->serverInfo->get().read().masterLifetime;
added the (untested) ability to force a recovery to the remote datacenter, even if that results in data loss. If the DR lag is more than 1 week there could be potential data corruption if any primary storage servers are still alive.
2018-07-01 21:39:04 +08:00
rmq.forceRecovery = db->forceRecovery;
Initial repository commit
2017-05-26 04:48:44 +08:00
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
cluster->masterProcessId = masterWorker.worker.interf.locality.processId();
The cluster controller will increase its failure monitoring delay after there have been many unfinishedRecoveries
2018-09-01 01:51:55 +08:00
cluster->db.unfinishedRecoveries++;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
state Future<ErrorOr<MasterInterface>> fNewMaster = masterWorker.worker.interf.master.tryGetReply( rmq );
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
wait( ready(fNewMaster) || db->forceMasterFailure.onTrigger() );
if (fNewMaster.isReady() && fNewMaster.get().present()) {
TraceEvent("CCWDB", cluster->id).detail("Recruited", fNewMaster.get().get().id());
Initial repository commit
2017-05-26 04:48:44 +08:00
// for status tool
TraceEvent("RecruitedMasterWorker", cluster->id)
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
.detail("Address", fNewMaster.get().get().address())
This is the first step in eliminating the usage of database names in our code. The C API remains the same, but underneath that all usage of database names is eliminated.
2018-08-17 01:24:12 +08:00
.trackLatest("RecruitedMasterWorker");
Initial repository commit
2017-05-26 04:48:44 +08:00
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
iMaster = fNewMaster.get().get();
Initial repository commit
2017-05-26 04:48:44 +08:00
db->masterRegistrationCount = 0;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
db->recoveryStalled = false;
Initial repository commit
2017-05-26 04:48:44 +08:00
during recovery, only send the full serverDBInfo to processes that are part of the new generation
2019-12-10 05:17:49 +08:00
db->serverInfoMasterOnly = CachedSerialization<ServerDBInfo>();
auto& dbInfo = db->serverInfoMasterOnly.mutate();
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
Initial repository commit
2017-05-26 04:48:44 +08:00
dbInfo.master = iMaster;
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
dbInfo.id = deterministicRandom()->randomUniqueID();
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
dbInfo.masterLifetime = db->serverInfo->get().read().masterLifetime;
Initial repository commit
2017-05-26 04:48:44 +08:00
++dbInfo.masterLifetime;
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
dbInfo.clusterInterface = db->serverInfo->get().read().clusterInterface;
dbInfo.distributor = db->serverInfo->get().read().distributor;
dbInfo.ratekeeper = db->serverInfo->get().read().ratekeeper;
Fix cases where latency band config could be discarded during recovery or process start.
2019-11-21 03:44:18 +08:00
dbInfo.latencyBandConfig = db->serverInfo->get().read().latencyBandConfig;
Initial repository commit
2017-05-26 04:48:44 +08:00
TraceEvent("CCWDB", cluster->id).detail("Lifetime", dbInfo.masterLifetime.toString()).detail("ChangeID", dbInfo.id);
during recovery, only send the full serverDBInfo to processes that are part of the new generation
2019-12-10 05:17:49 +08:00
db->requiredAddresses.clear();
db->serverInfo->set( db->serverInfoMasterOnly );
Initial repository commit
2017-05-26 04:48:44 +08:00
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
state Future<Void> spinDelay = delay(SERVER_KNOBS->MASTER_SPIN_DELAY); // Don't retry master recovery more than once per second, but don't delay the "first" recovery after more than a second of normal operation
Initial repository commit
2017-05-26 04:48:44 +08:00
TraceEvent("CCWDB", cluster->id).detail("Watching", iMaster.id());
// Master failure detection is pretty sensitive, but if we are in the middle of a very long recovery we really don't want to have to start over
loop choose {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when (wait( waitFailureClient( iMaster.waitFailure, db->masterRegistrationCount ?
Initial repository commit
2017-05-26 04:48:44 +08:00
SERVER_KNOBS->MASTER_FAILURE_REACTION_TIME : (now() - recoveryStart) * SERVER_KNOBS->MASTER_FAILURE_SLOPE_DURING_RECOVERY,
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
db->masterRegistrationCount ? -SERVER_KNOBS->MASTER_FAILURE_REACTION_TIME/SERVER_KNOBS->SECONDS_BEFORE_NO_FAILURE_DELAY : SERVER_KNOBS->MASTER_FAILURE_SLOPE_DURING_RECOVERY ) || db->forceMasterFailure.onTrigger() )) { break; }
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when (wait( db->serverInfo->onChange() )) {}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
wait(spinDelay);
Initial repository commit
2017-05-26 04:48:44 +08:00
TEST(true); // clusterWatchDatabase() master failed
TraceEvent(SevWarn,"DetectedFailedMaster", cluster->id).detail("OldMaster", iMaster.id());
} else {
TEST(true); //clusterWatchDatabas() !newMaster.present()
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( delay(SERVER_KNOBS->MASTER_SPIN_DELAY) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
} catch (Error& e) {
TraceEvent("CCWDB", cluster->id).error(e, true).detail("Master", iMaster.id());
if (e.code() == error_code_actor_cancelled) throw;
bool ok = e.code() == error_code_no_more_servers;
Attempt to normalize trace events: * Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check. * Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase. * Use seconds instead of milliseconds in details. Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed. This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-09 02:11:08 +08:00
TraceEvent(ok ? SevWarn : SevError,"ClusterWatchDatabaseRetrying", cluster->id).error(e);
Initial repository commit
2017-05-26 04:48:44 +08:00
if (!ok)
throw e;
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( delay(SERVER_KNOBS->ATTEMPT_RECRUITMENT_DELAY) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
}
Add support for a client or worker having multiple issues.
2019-03-22 23:29:41 +08:00
ACTOR Future<Void> clusterGetServerInfo(ClusterControllerData::DBInfo* db, UID knownServerInfoID,
Standalone<VectorRef<StringRef>> issues,
std::vector<NetworkAddress> incompatiblePeers,
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
ReplyPromise<CachedSerialization<ServerDBInfo>> reply) {
Add support for a client or worker having multiple issues.
2019-03-22 23:29:41 +08:00
state Optional<UID> issueID;
during recovery, only send the full serverDBInfo to processes that are part of the new generation
2019-12-10 05:17:49 +08:00
state bool useMasterOnly = false;
Add support for a client or worker having multiple issues.
2019-03-22 23:29:41 +08:00
setIssues(db->workersWithIssues, reply.getEndpoint().getPrimaryAddress(), issues, issueID);
Initial repository commit
2017-05-26 04:48:44 +08:00
for(auto it : incompatiblePeers) {
db->incompatibleConnections[it] = now() + SERVER_KNOBS->INCOMPATIBLE_PEERS_LOGGING_INTERVAL;
}
during recovery, only send the full serverDBInfo to processes that are part of the new generation
2019-12-10 05:17:49 +08:00
loop {
useMasterOnly = db->serverInfo->get().read().recoveryState < RecoveryState::ACCEPTING_COMMITS && !db->requiredAddresses.count(reply.getEndpoint().getPrimaryAddress());
if((useMasterOnly ? db->serverInfoMasterOnly.read().id : db->serverInfo->get().read().id) != knownServerInfoID) {
break;
}
Initial repository commit
2017-05-26 04:48:44 +08:00
choose {
add yieldedFutures to prevent slow tasks
2019-09-12 07:26:48 +08:00
when (wait( yieldedFuture(db->serverInfo->onChange()) )) {}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when (wait( delayJittered( 300 ) )) { break; } // The server might be long gone!
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
Add support for a client or worker having multiple issues.
2019-03-22 23:29:41 +08:00
removeIssues(db->workersWithIssues, reply.getEndpoint().getPrimaryAddress(), issueID);
Initial repository commit
2017-05-26 04:48:44 +08:00
during recovery, only send the full serverDBInfo to processes that are part of the new generation
2019-12-10 05:17:49 +08:00
reply.send( useMasterOnly ? db->serverInfoMasterOnly : db->serverInfo->get() );
Initial repository commit
2017-05-26 04:48:44 +08:00
return Void();
}
added status information on connected clients
2019-07-26 08:15:31 +08:00
ACTOR Future<Void> clusterOpenDatabase(ClusterControllerData::DBInfo* db, OpenDatabaseRequest req) {
db->clientStatus[req.reply.getEndpoint().getPrimaryAddress()] = std::make_pair(now(), req);
if(db->clientStatus.size() > 10000) {
TraceEvent(SevWarnAlways, "TooManyClientStatusEntries").suppressFor(1.0);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
added status information on connected clients
2019-07-26 08:15:31 +08:00
while (db->clientInfo->get().id == req.knownClientInfoID) {
Initial repository commit
2017-05-26 04:48:44 +08:00
choose {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when (wait( db->clientInfo->onChange() )) {}
added status information on connected clients
2019-07-26 08:15:31 +08:00
when (wait( delayJittered( SERVER_KNOBS->COORDINATOR_REGISTER_INTERVAL ) )) { break; } // The client might be long gone!
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
added status information on connected clients
2019-07-26 08:15:31 +08:00
req.reply.send( db->clientInfo->get() );
Initial repository commit
2017-05-26 04:48:44 +08:00
return Void();
}
void checkOutstandingRecruitmentRequests( ClusterControllerData* self ) {
for( int i = 0; i < self->outstandingRecruitmentRequests.size(); i++ ) {
RecruitFromConfigurationRequest& req = self->outstandingRecruitmentRequests[i];
try {
during recovery, only send the full serverDBInfo to processes that are part of the new generation
2019-12-10 05:17:49 +08:00
RecruitFromConfigurationReply rep = self->findWorkersForConfiguration( req );
fix: we were recruiting one too few oldLogRouters code cleanup
2020-01-03 07:05:44 +08:00
self->db.addRequiredAddresses(rep.oldLogRouters);
self->db.addRequiredAddresses(rep.proxies);
self->db.addRequiredAddresses(rep.resolvers);
self->db.addRequiredAddresses(rep.satelliteTLogs);
self->db.addRequiredAddresses(rep.tLogs);
during recovery, only send the full serverDBInfo to processes that are part of the new generation
2019-12-10 05:17:49 +08:00
self->db.serverInfo->trigger();
req.reply.send( rep );
Stop performing self-moves. (e.g. a = std::move(a)) self-moves are frowned upon in C++, and in our code this generally happens from calls to swap as part of trying to implement a "unordered erase" function via swap-to-the-end-and-pop_back. For convenience, a swapAndPop() function is now offered that performs this, while disallowing self-moves.
2018-08-02 09:09:54 +08:00
swapAndPop( &self->outstandingRecruitmentRequests, i-- );
Initial repository commit
2017-05-26 04:48:44 +08:00
} catch (Error& e) {
if (e.code() == error_code_no_more_servers || e.code() == error_code_operation_failed) {
TraceEvent(SevWarn, "RecruitTLogMatchingSetNotAvailable", self->id).error(e);
} else {
TraceEvent(SevError, "RecruitTLogsRequestError", self->id).error(e);
throw;
}
}
}
}
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
void checkOutstandingRemoteRecruitmentRequests( ClusterControllerData* self ) {
for( int i = 0; i < self->outstandingRemoteRecruitmentRequests.size(); i++ ) {
RecruitRemoteFromConfigurationRequest& req = self->outstandingRemoteRecruitmentRequests[i];
try {
during recovery, only send the full serverDBInfo to processes that are part of the new generation
2019-12-10 05:17:49 +08:00
RecruitRemoteFromConfigurationReply rep = self->findRemoteWorkersForConfiguration( req );
fix: we were recruiting one too few oldLogRouters code cleanup
2020-01-03 07:05:44 +08:00
self->db.addRequiredAddresses(rep.remoteTLogs);
self->db.addRequiredAddresses(rep.logRouters);
during recovery, only send the full serverDBInfo to processes that are part of the new generation
2019-12-10 05:17:49 +08:00
self->db.serverInfo->trigger();
req.reply.send( rep );
Stop performing self-moves. (e.g. a = std::move(a)) self-moves are frowned upon in C++, and in our code this generally happens from calls to swap as part of trying to implement a "unordered erase" function via swap-to-the-end-and-pop_back. For convenience, a swapAndPop() function is now offered that performs this, while disallowing self-moves.
2018-08-02 09:09:54 +08:00
swapAndPop( &self->outstandingRemoteRecruitmentRequests, i-- );
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
} catch (Error& e) {
if (e.code() == error_code_no_more_servers || e.code() == error_code_operation_failed) {
TraceEvent(SevWarn, "RecruitRemoteTLogMatchingSetNotAvailable", self->id).error(e);
} else {
TraceEvent(SevError, "RecruitRemoteTLogsRequestError", self->id).error(e);
throw;
}
}
}
}
Initial repository commit
2017-05-26 04:48:44 +08:00
void checkOutstandingStorageRequests( ClusterControllerData* self ) {
for( int i = 0; i < self->outstandingStorageRequests.size(); i++ ) {
auto& req = self->outstandingStorageRequests[i];
try {
if(req.second < now()) {
req.first.reply.sendError(timed_out());
Stop performing self-moves. (e.g. a = std::move(a)) self-moves are frowned upon in C++, and in our code this generally happens from calls to swap as part of trying to implement a "unordered erase" function via swap-to-the-end-and-pop_back. For convenience, a swapAndPop() function is now offered that performs this, while disallowing self-moves.
2018-08-02 09:09:54 +08:00
swapAndPop( &self->outstandingStorageRequests, i-- );
Initial repository commit
2017-05-26 04:48:44 +08:00
} else {
if(!self->gotProcessClasses && !req.first.criticalRecruitment)
throw no_more_servers();
auto worker = self->getStorageWorker(req.first);
RecruitStorageReply rep;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
rep.worker = worker.interf;
rep.processClass = worker.processClass;
Initial repository commit
2017-05-26 04:48:44 +08:00
req.first.reply.send( rep );
Stop performing self-moves. (e.g. a = std::move(a)) self-moves are frowned upon in C++, and in our code this generally happens from calls to swap as part of trying to implement a "unordered erase" function via swap-to-the-end-and-pop_back. For convenience, a swapAndPop() function is now offered that performs this, while disallowing self-moves.
2018-08-02 09:09:54 +08:00
swapAndPop( &self->outstandingStorageRequests, i-- );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
} catch (Error& e) {
if (e.code() == error_code_no_more_servers) {
TraceEvent(SevWarn, "RecruitStorageNotAvailable", self->id).error(e);
} else {
TraceEvent(SevError, "RecruitStorageError", self->id).error(e);
throw;
}
}
}
}
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
void checkBetterDDOrRK(ClusterControllerData* self) {
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
if (!self->masterProcessId.present() || self->db.serverInfo->get().read().recoveryState < RecoveryState::ACCEPTING_COMMITS) {
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
return;
}
Fix bug where DD or RK could be halted and re-recruited in a loop for certain valid process class configurations. Specifically, recruitment of DD or RK takes into account that master process is preferred over proxy, resolver or cc. But check for better DD only looks for better machine class ignoring that the new recruit could share a proxy or resolver or CC. Also try to balance the distribution of the DD and RK role if there are enough processes to do so.
2019-11-13 06:22:36 +08:00
std::map<Optional<Standalone<StringRef>>, int> id_used = self->getUsedIds();
WorkerDetails newRKWorker = self->getWorkerForRoleInDatacenter(self->clusterControllerDcId, ProcessClass::Ratekeeper, ProcessClass::NeverAssign, self->db.config, id_used, true).worker;
if (self->onMasterIsBetter(newRKWorker, ProcessClass::Ratekeeper)) {
newRKWorker = self->id_worker[self->masterProcessId.get()].details;
}
fix: dd test recruitment should prefer the location of ratekeeper over other used processes
2019-11-14 04:58:55 +08:00
id_used = self->getUsedIds();
for(auto& it : id_used) {
it.second *= 2;
}
id_used[newRKWorker.interf.locality.processId()]++;
Fix bug where DD or RK could be halted and re-recruited in a loop for certain valid process class configurations. Specifically, recruitment of DD or RK takes into account that master process is preferred over proxy, resolver or cc. But check for better DD only looks for better machine class ignoring that the new recruit could share a proxy or resolver or CC. Also try to balance the distribution of the DD and RK role if there are enough processes to do so.
2019-11-13 06:22:36 +08:00
WorkerDetails newDDWorker = self->getWorkerForRoleInDatacenter(self->clusterControllerDcId, ProcessClass::DataDistributor, ProcessClass::NeverAssign, self->db.config, id_used, true).worker;
if (self->onMasterIsBetter(newDDWorker, ProcessClass::DataDistributor)) {
newDDWorker = self->id_worker[self->masterProcessId.get()].details;
}
auto bestFitnessForRK = newRKWorker.processClass.machineClassFitness(ProcessClass::Ratekeeper);
if(self->db.config.isExcludedServer(newRKWorker.interf.address())) {
bestFitnessForRK = std::max(bestFitnessForRK, ProcessClass::ExcludeFit);
}
auto bestFitnessForDD = newDDWorker.processClass.machineClassFitness(ProcessClass::DataDistributor);
if(self->db.config.isExcludedServer(newDDWorker.interf.address())) {
bestFitnessForDD = std::max(bestFitnessForDD, ProcessClass::ExcludeFit);
}
This commit tries to distribute RK and DD among other empty available processes.
2019-11-13 09:52:42 +08:00
//TraceEvent("CheckBetterDDorRKNewRecruits", self->id).detail("MasterProcessId", self->masterProcessId)
//.detail("NewRecruitRKProcessId", newRKWorker.interf.locality.processId()).detail("NewRecruiteDDProcessId", newDDWorker.interf.locality.processId());
Fix bug where DD or RK could be halted and re-recruited in a loop for certain valid process class configurations. Specifically, recruitment of DD or RK takes into account that master process is preferred over proxy, resolver or cc. But check for better DD only looks for better machine class ignoring that the new recruit could share a proxy or resolver or CC. Also try to balance the distribution of the DD and RK role if there are enough processes to do so.
2019-11-13 06:22:36 +08:00
Optional<Standalone<StringRef>> currentRKProcessId;
Optional<Standalone<StringRef>> currentDDProcessId;
This commit tries to distribute RK and DD among other empty available processes.
2019-11-13 09:52:42 +08:00
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
auto& db = self->db.serverInfo->get().read();
fix: dd test recruitment should prefer the location of ratekeeper over other used processes
2019-11-14 04:58:55 +08:00
bool ratekeeperHealthy = false;
once we recruit a ratekeeper, do not allow any other ratekeepers to register
2019-03-25 02:04:39 +08:00
if (db.ratekeeper.present() && self->id_worker.count(db.ratekeeper.get().locality.processId()) &&
(!self->recruitingRatekeeperID.present() || (self->recruitingRatekeeperID.get() == db.ratekeeper.get().id()))) {
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
auto& rkWorker = self->id_worker[db.ratekeeper.get().locality.processId()];
Fix bug where DD or RK could be halted and re-recruited in a loop for certain valid process class configurations. Specifically, recruitment of DD or RK takes into account that master process is preferred over proxy, resolver or cc. But check for better DD only looks for better machine class ignoring that the new recruit could share a proxy or resolver or CC. Also try to balance the distribution of the DD and RK role if there are enough processes to do so.
2019-11-13 06:22:36 +08:00
currentRKProcessId = rkWorker.details.interf.locality.processId();
Changes to ratekeeper camel case
2019-03-27 23:24:25 +08:00
auto rkFitness = rkWorker.details.processClass.machineClassFitness(ProcessClass::Ratekeeper);
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
if(rkWorker.priorityInfo.isExcluded) {
Fix fitness assignment statements Found by MacOS build.
2019-03-24 10:16:04 +08:00
rkFitness = ProcessClass::ExcludeFit;
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
}
fix: dd test recruitment should prefer the location of ratekeeper over other used processes
2019-11-14 04:58:55 +08:00
if (self->isUsedNotMaster(rkWorker.details.interf.locality.processId()) || bestFitnessForRK < rkFitness
Fix bug where DD or RK could be halted and re-recruited in a loop for certain valid process class configurations. Specifically, recruitment of DD or RK takes into account that master process is preferred over proxy, resolver or cc. But check for better DD only looks for better machine class ignoring that the new recruit could share a proxy or resolver or CC. Also try to balance the distribution of the DD and RK role if there are enough processes to do so.
2019-11-13 06:22:36 +08:00
|| (rkFitness == bestFitnessForRK && rkWorker.details.interf.locality.processId() == self->masterProcessId && newRKWorker.interf.locality.processId() != self->masterProcessId)) {
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCHaltRK", self->id).detail("RKID", db.ratekeeper.get().id())
Move ratekeeper or data distributor from excluded servers
2019-03-22 08:07:12 +08:00
.detail("Excluded", rkWorker.priorityInfo.isExcluded)
Add actors to store halt request futures Address best fitness in checking better DD or RK.
2019-03-23 09:00:16 +08:00
.detail("Fitness", rkFitness).detail("BestFitness", bestFitnessForRK);
Fix: recruit ratekeeper is not triggerred
2019-03-24 00:20:54 +08:00
self->recruitRatekeeper.set(true);
fix: dd test recruitment should prefer the location of ratekeeper over other used processes
2019-11-14 04:58:55 +08:00
} else {
ratekeeperHealthy = true;
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
}
}
Fix test stuck due to invalid worker in cluster controller Test case: -r simulation --crash -f ./tests/rare/CloggedCycleWithKills.txt -s 688927581 -b off
2019-03-21 13:18:53 +08:00
if (!self->recruitingDistributor && db.distributor.present() && self->id_worker.count(db.distributor.get().locality.processId())) {
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
auto& ddWorker = self->id_worker[db.distributor.get().locality.processId()];
auto ddFitness = ddWorker.details.processClass.machineClassFitness(ProcessClass::DataDistributor);
Fix bug where DD or RK could be halted and re-recruited in a loop for certain valid process class configurations. Specifically, recruitment of DD or RK takes into account that master process is preferred over proxy, resolver or cc. But check for better DD only looks for better machine class ignoring that the new recruit could share a proxy or resolver or CC. Also try to balance the distribution of the DD and RK role if there are enough processes to do so.
2019-11-13 06:22:36 +08:00
currentDDProcessId = ddWorker.details.interf.locality.processId();
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
if(ddWorker.priorityInfo.isExcluded) {
Fix fitness assignment statements Found by MacOS build.
2019-03-24 10:16:04 +08:00
ddFitness = ProcessClass::ExcludeFit;
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
}
fix: dd test recruitment should prefer the location of ratekeeper over other used processes
2019-11-14 04:58:55 +08:00
if (self->isUsedNotMaster(ddWorker.details.interf.locality.processId()) || bestFitnessForDD < ddFitness
Fix bug where DD or RK could be halted and re-recruited in a loop for certain valid process class configurations. Specifically, recruitment of DD or RK takes into account that master process is preferred over proxy, resolver or cc. But check for better DD only looks for better machine class ignoring that the new recruit could share a proxy or resolver or CC. Also try to balance the distribution of the DD and RK role if there are enough processes to do so.
2019-11-13 06:22:36 +08:00
|| (ddFitness == bestFitnessForDD && ddWorker.details.interf.locality.processId() == self->masterProcessId && newDDWorker.interf.locality.processId() != self->masterProcessId)
fix: dd test recruitment should prefer the location of ratekeeper over other used processes
2019-11-14 04:58:55 +08:00
|| (ddFitness == bestFitnessForDD && newRKWorker.interf.locality.processId() != newDDWorker.interf.locality.processId() && ratekeeperHealthy && currentRKProcessId.present() && currentDDProcessId == currentRKProcessId
&& (newRKWorker.interf.locality.processId() != self->masterProcessId && newDDWorker.interf.locality.processId() != self->masterProcessId) )) {
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCHaltDD", self->id).detail("DDID", db.distributor.get().id())
Move ratekeeper or data distributor from excluded servers
2019-03-22 08:07:12 +08:00
.detail("Excluded", ddWorker.priorityInfo.isExcluded)
This commit tries to distribute RK and DD among other empty available processes.
2019-11-13 09:52:42 +08:00
.detail("Fitness", ddFitness).detail("BestFitness", bestFitnessForDD)
.detail("CurrentRateKeeperProcessId", currentRKProcessId.present() ? currentRKProcessId.get() : LiteralStringRef("None"))
fix: dd test recruitment should prefer the location of ratekeeper over other used processes
2019-11-14 04:58:55 +08:00
.detail("CurrentDDProcessId", currentDDProcessId)
.detail("MasterProcessID", self->masterProcessId)
.detail("NewRKWorkers", newRKWorker.interf.locality.processId())
.detail("NewDDWorker", newDDWorker.interf.locality.processId());
ensure that the processId exists in id_worker before accessing it
2019-03-23 09:54:39 +08:00
ddWorker.haltDistributor = brokenPromiseToNever(db.distributor.get().haltDataDistributor.getReply(HaltDataDistributorRequest(self->id)));
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
}
}
}
checking outstanding requests can be CPU intensive, so rate limit checking requests
2018-06-28 14:02:08 +08:00
ACTOR Future<Void> doCheckOutstandingRequests( ClusterControllerData* self ) {
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
try {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( delay(SERVER_KNOBS->CHECK_OUTSTANDING_INTERVAL) );
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
while( !self->goodRecruitmentTime.isReady() ) {
wait(self->goodRecruitmentTime);
}
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
checkOutstandingRecruitmentRequests( self );
checkOutstandingStorageRequests( self );
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
checkBetterDDOrRK(self);
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
self->checkRecoveryStalled();
if (self->betterMasterExists()) {
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
self->db.forceMasterFailure.trigger();
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
TraceEvent("MasterRegistrationKill", self->id).detail("MasterId", self->db.serverInfo->get().read().master.id());
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
}
} catch( Error &e ) {
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
if(e.code() != error_code_no_more_servers) {
TraceEvent(SevError, "CheckOutstandingError").error(e);
}
}
return Void();
}
ACTOR Future<Void> doCheckOutstandingRemoteRequests( ClusterControllerData* self ) {
try {
wait( delay(SERVER_KNOBS->CHECK_OUTSTANDING_INTERVAL) );
updateGoodRemoteRecruitmentTime was unnecessary because the only way findRemoteWorkers would return would be after a new server has joined which already resets goodRemoteRecruitmentTime
2020-02-21 05:46:22 +08:00
while( !self->goodRemoteRecruitmentTime.isReady() ) {
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
wait(self->goodRemoteRecruitmentTime);
}
checkOutstandingRemoteRecruitmentRequests( self );
} catch( Error &e ) {
if(e.code() != error_code_no_more_servers) {
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
TraceEvent(SevError, "CheckOutstandingError").error(e);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
return Void();
}
checking outstanding requests can be CPU intensive, so rate limit checking requests
2018-06-28 14:02:08 +08:00
void checkOutstandingRequests( ClusterControllerData* self ) {
updateGoodRemoteRecruitmentTime was unnecessary because the only way findRemoteWorkers would return would be after a new server has joined which already resets goodRemoteRecruitmentTime
2020-02-21 05:46:22 +08:00
if( self->outstandingRemoteRequestChecker.isReady() ) {
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
self->outstandingRemoteRequestChecker = doCheckOutstandingRemoteRequests(self);
}
Initial repository commit
2017-05-26 04:48:44 +08:00
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
if( self->outstandingRequestChecker.isReady() ) {
self->outstandingRequestChecker = doCheckOutstandingRequests(self);
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
ACTOR Future<Void> rebootAndCheck( ClusterControllerData* cluster, Optional<Standalone<StringRef>> processID ) {
Fix minor IDE build errors
2019-02-18 10:46:59 +08:00
{
auto watcher = cluster->id_worker.find(processID);
ASSERT(watcher != cluster->id_worker.end());
Initial repository commit
2017-05-26 04:48:44 +08:00
Fix minor IDE build errors
2019-02-18 10:46:59 +08:00
watcher->second.reboots++;
wait( delay( g_network->isSimulated() ? SERVER_KNOBS->SIM_SHUTDOWN_TIMEOUT : SERVER_KNOBS->SHUTDOWN_TIMEOUT ) );
}
Initial repository commit
2017-05-26 04:48:44 +08:00
Fix minor IDE build errors
2019-02-18 10:46:59 +08:00
{
auto watcher = cluster->id_worker.find(processID);
if(watcher != cluster->id_worker.end()) {
watcher->second.reboots--;
if( watcher->second.reboots < 2 )
checkOutstandingRequests( cluster );
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
return Void();
}
ACTOR Future<Void> workerAvailabilityWatch( WorkerInterface worker, ProcessClass startingClass, ClusterControllerData* cluster ) {
run clang-format on changes
2019-05-30 04:43:21 +08:00
state Future<Void> failed =
(worker.address() == g_network->getLocalAddress() || startingClass.classType() == ProcessClass::TesterClass)
? Never()
: waitFailureClient(worker.waitFailure, SERVER_KNOBS->WORKER_FAILURE_TIME);
Initial repository commit
2017-05-26 04:48:44 +08:00
cluster->updateWorkerList.set( worker.locality.processId(), ProcessData(worker.locality, startingClass, worker.address()) );
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
// This switching avoids a race where the worker can be added to id_worker map after the workerAvailabilityWatch fails for the worker.
wait(delay(0));
Initial repository commit
2017-05-26 04:48:44 +08:00
loop {
choose {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when( wait( IFailureMonitor::failureMonitor().onStateEqual( worker.storage.getEndpoint(), FailureStatus(IFailureMonitor::failureMonitor().getState( worker.storage.getEndpoint() ).isAvailable()) ) ) ) {
Initial repository commit
2017-05-26 04:48:44 +08:00
if( IFailureMonitor::failureMonitor().getState( worker.storage.getEndpoint() ).isAvailable() ) {
cluster->ac.add( rebootAndCheck( cluster, worker.locality.processId() ) );
checkOutstandingRequests( cluster );
}
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when( wait( failed ) ) { // remove workers that have failed
Fix cluster controller review comments
2017-10-05 06:48:55 +08:00
WorkerInfo& failedWorkerInfo = cluster->id_worker[ worker.locality.processId() ];
Add initialClass to RegisterWorkerRequest
2017-10-05 08:11:12 +08:00
if (!failedWorkerInfo.reply.isSet()) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
failedWorkerInfo.reply.send( RegisterWorkerReply(failedWorkerInfo.details.processClass, failedWorkerInfo.priorityInfo) );
Add initialClass to RegisterWorkerRequest
2017-10-05 08:11:12 +08:00
}
Fix a race of DD, RK, Master failure If all DD, RK, Master run on the same process and failed. Recruiting of new DD or RK could try to use the old master worker interface, which is an invalid one and causes recruitment to be stuck. Fix by adding a delay and checking master is valid before recruitment.
2019-03-21 07:03:36 +08:00
if (worker.locality.processId() == cluster->masterProcessId) {
cluster->masterProcessId = Optional<Key>();
}
Initial repository commit
2017-05-26 04:48:44 +08:00
cluster->id_worker.erase( worker.locality.processId() );
cluster->updateWorkerList.set( worker.locality.processId(), Optional<ProcessData>() );
return Void();
}
}
}
}
struct FailureStatusInfo {
FailureStatus status;
double lastRequestTime;
double penultimateRequestTime;
FailureStatusInfo() : lastRequestTime(0), penultimateRequestTime(0) {}
void insertRequest(double now) {
penultimateRequestTime = lastRequestTime;
lastRequestTime = now;
}
double latency(double now) const {
return std::max( now - lastRequestTime, lastRequestTime - penultimateRequestTime );
}
};
//The failure monitor client relies on the fact that the failure detection server will not declare itself failed
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
ACTOR Future<Void> failureDetectionServer( UID uniqueID, ClusterControllerData* self, FutureStream< FailureMonitoringRequest > requests ) {
Initial repository commit
2017-05-26 04:48:44 +08:00
state Version currentVersion = 0;
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
state std::map<NetworkAddressList, FailureStatusInfo> currentStatus; // The status at currentVersion
Initial repository commit
2017-05-26 04:48:44 +08:00
state std::deque<SystemFailureStatus> statusHistory; // The last change in statusHistory is from currentVersion-1 to currentVersion
state Future<Void> periodically = Void();
state double lastT = 0;
loop choose {
when ( FailureMonitoringRequest req = waitNext( requests ) ) {
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
++self->failureMonitoringRequests;
Initial repository commit
2017-05-26 04:48:44 +08:00
if ( req.senderStatus.present() ) {
// Update the status of requester, if necessary
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
auto& stat = currentStatus[ req.addresses ];
Initial repository commit
2017-05-26 04:48:44 +08:00
auto& newStat = req.senderStatus.get();
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
ASSERT( !newStat.failed || req.addresses != g_network->getLocalAddresses() );
Initial repository commit
2017-05-26 04:48:44 +08:00
stat.insertRequest(now());
if (req.senderStatus != stat.status) {
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
if(newStat.failed) {
++self->serversFailed;
}
else {
++self->serversUnfailed;
}
changed NetworkAddressList to at most two addresses for performance
2019-03-24 08:54:46 +08:00
TraceEvent("FailureDetectionStatus", uniqueID).detail("System", req.addresses.toString()).detail("Status", newStat.failed ? "Failed" : "OK").detail("Why", "Request");
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
statusHistory.push_back( SystemFailureStatus( req.addresses, newStat ) );
Initial repository commit
2017-05-26 04:48:44 +08:00
++currentVersion;
if (req.senderStatus == FailureStatus()){
// failureMonitorClient reports explicitly that it is failed
ASSERT(false); // This can't happen at the moment; if that changes, make this a TEST instead
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
currentStatus.erase(req.addresses);
Initial repository commit
2017-05-26 04:48:44 +08:00
} else {
TEST(true);
stat.status = newStat;
}
while (statusHistory.size() > currentStatus.size())
statusHistory.pop_front();
}
}
// Return delta-compressed status changes to requester
Version reqVersion = req.failureInformationVersion;
if (reqVersion > currentVersion){
req.reply.sendError( future_version() );
ASSERT(false);
} else {
TEST(true); // failureDetectionServer sending failure data to requester
FailureMonitoringReply reply;
reply.failureInformationVersion = currentVersion;
added knobs so that the client failure monitoring update rate and the server failure monitoring update rate are separate knobs
2017-12-02 05:04:32 +08:00
if( req.senderStatus.present() ) {
reply.clientRequestIntervalMS = FLOW_KNOBS->SERVER_REQUEST_INTERVAL * 1000;
reply.considerServerFailedTimeoutMS = CLIENT_KNOBS->FAILURE_TIMEOUT_DELAY * 1000;
} else {
reply.clientRequestIntervalMS = FLOW_KNOBS->CLIENT_REQUEST_INTERVAL * 1000;
reply.considerServerFailedTimeoutMS = CLIENT_KNOBS->CLIENT_FAILURE_TIMEOUT_DELAY * 1000;
}
Initial repository commit
2017-05-26 04:48:44 +08:00
ASSERT( currentVersion >= (int64_t)statusHistory.size());
if (reqVersion < currentVersion - (int64_t)statusHistory.size() || reqVersion == 0) {
// Send everything
TEST(true); // failureDetectionServer sending all current data to requester
reply.allOthersFailed = true;
for(auto it = currentStatus.begin(); it != currentStatus.end(); ++it)
reply.changes.push_back( reply.arena, SystemFailureStatus( it->first, it->second.status ) );
} else {
TEST(true); // failureDetectionServer sending delta-compressed data to requester
// SOMEDAY: Send only the last change for a given address?
reply.allOthersFailed = false;
for(int v = reqVersion - currentVersion + statusHistory.size(); v < statusHistory.size(); v++) {
reply.changes.push_back( reply.arena, statusHistory[v] );
}
}
req.reply.send( reply );
}
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when ( wait( periodically ) ) {
added knobs so that the client failure monitoring update rate and the server failure monitoring update rate are separate knobs
2017-12-02 05:04:32 +08:00
periodically = delay( FLOW_KNOBS->SERVER_REQUEST_INTERVAL );
Initial repository commit
2017-05-26 04:48:44 +08:00
double t = now();
if (lastT != 0 && t - lastT > 1)
TraceEvent("LongDelayOnClusterController").detail("Duration", t - lastT);
lastT = t;
// Adapt to global unresponsiveness
vector<double> delays;
for(auto it=currentStatus.begin(); it!=currentStatus.end(); it++)
if (it->second.penultimateRequestTime) {
delays.push_back(it->second.latency(t));
call all methods which could disable a trace event before it is initialized. In practice this means calling .error first, then .suppressFor, then all your details.
2018-08-02 05:30:57 +08:00
//TraceEvent("FDData", uniqueID).detail("S", it->first.toString()).detail("L", it->second.latency(t));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
int pivot = std::max(0, (int)delays.size()-2);
double pivotDelay = 0;
if (delays.size()) {
std::nth_element(delays.begin(), delays.begin()+pivot, delays.end());
pivotDelay = *(delays.begin()+pivot);
}
added knobs so that the client failure monitoring update rate and the server failure monitoring update rate are separate knobs
2017-12-02 05:04:32 +08:00
pivotDelay = std::max(0.0, pivotDelay - FLOW_KNOBS->SERVER_REQUEST_INTERVAL);
Initial repository commit
2017-05-26 04:48:44 +08:00
call all methods which could disable a trace event before it is initialized. In practice this means calling .error first, then .suppressFor, then all your details.
2018-08-02 05:30:57 +08:00
//TraceEvent("FailureDetectionPoll", uniqueID).detail("PivotDelay", pivotDelay).detail("Clients", currentStatus.size());
Attempt to normalize trace events: * Detail names now all start with an uppercase character and contain no underscores. Ideally these should be head-first camel case, though that was harder to check. * Type names have the same rules, except they allow one underscore (to support a usage pattern Context_Type). The first character after the underscore is also uppercase. * Use seconds instead of milliseconds in details. Added a check when events are logged in simulation that logs a message to stderr if the first two rules above aren't followed. This probably doesn't address every instance of the above problems, but all of the events I was able to hit in simulation pass the check.
2018-06-09 02:11:08 +08:00
//TraceEvent("FailureDetectionAcceptableDelay").detail("Delay", acceptableDelay1000);
Initial repository commit
2017-05-26 04:48:44 +08:00
renamed a variable
2020-03-05 10:49:21 +08:00
bool useEmergencyDelay = (std::max(self->db.unfinishedRecoveries, self->db.logGenerations) > CLIENT_KNOBS->FAILURE_MAX_GENERATIONS) ||
Do not allow the cluster controller to mark any process as failed within 30 seconds of startup
2020-03-05 10:45:26 +08:00
(now() - self->startTime < CLIENT_KNOBS->FAILURE_EMERGENCY_DELAY);
The cluster controller will increase its failure monitoring delay after there have been many unfinishedRecoveries
2018-09-01 01:51:55 +08:00
Initial repository commit
2017-05-26 04:48:44 +08:00
for(auto it = currentStatus.begin(); it != currentStatus.end(); ) {
double delay = t - it->second.lastRequestTime;
renamed a variable
2020-03-05 10:49:21 +08:00
if ( it->first != g_network->getLocalAddresses() && ( useEmergencyDelay ?
The cluster controller will increase its failure monitoring delay after there have been many unfinishedRecoveries
2018-09-01 01:51:55 +08:00
( delay > CLIENT_KNOBS->FAILURE_EMERGENCY_DELAY ) :
( delay > pivotDelay * 2 + FLOW_KNOBS->SERVER_REQUEST_INTERVAL + CLIENT_KNOBS->FAILURE_MIN_DELAY || delay > CLIENT_KNOBS->FAILURE_MAX_DELAY ) ) ) {
Initial repository commit
2017-05-26 04:48:44 +08:00
//printf("Failure Detection Server: Status of '%s' is now '%s' after %f sec\n", it->first.toString().c_str(), "Failed", now() - it->second.lastRequestTime);
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
TraceEvent("FailureDetectionStatus", uniqueID).detail("System", describe(it->first)).detail("Status","Failed").detail("Why", "Timeout").detail("LastRequestAge", delay)
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
.detail("PivotDelay", pivotDelay).detail("UnfinishedRecoveries", self->db.unfinishedRecoveries).detail("LogGenerations", self->db.logGenerations);
++self->serversFailed;
Initial repository commit
2017-05-26 04:48:44 +08:00
statusHistory.push_back( SystemFailureStatus( it->first, FailureStatus(true) ) );
++currentVersion;
it = currentStatus.erase(it);
while (statusHistory.size() > currentStatus.size())
statusHistory.pop_front();
} else {
++it;
}
}
}
}
}
ACTOR Future<vector<TLogInterface>> requireAll( vector<Future<Optional<vector<TLogInterface>>>> in ) {
state vector<TLogInterface> out;
state int i;
for(i=0; i<in.size(); i++) {
Optional<vector<TLogInterface>> x = wait(in[i]);
if (!x.present()) throw recruitment_failed();
out.insert(out.end(), x.get().begin(), x.get().end());
}
return out;
}
void clusterRecruitStorage( ClusterControllerData* self, RecruitStorageRequest req ) {
try {
if(!self->gotProcessClasses && !req.criticalRecruitment)
throw no_more_servers();
auto worker = self->getStorageWorker(req);
RecruitStorageReply rep;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
rep.worker = worker.interf;
rep.processClass = worker.processClass;
Initial repository commit
2017-05-26 04:48:44 +08:00
req.reply.send( rep );
} catch ( Error& e ) {
if (e.code() == error_code_no_more_servers) {
self->outstandingStorageRequests.push_back( std::make_pair(req, now() + SERVER_KNOBS->RECRUITMENT_TIMEOUT) );
TraceEvent(SevWarn, "RecruitStorageNotAvailable", self->id).error(e);
} else {
TraceEvent(SevError, "RecruitStorageError", self->id).error(e);
throw; // Any other error will bring down the cluster controller
}
}
}
ACTOR Future<Void> clusterRecruitFromConfiguration( ClusterControllerData* self, RecruitFromConfigurationRequest req ) {
// At the moment this doesn't really need to be an actor (it always completes immediately)
TEST(true); //ClusterController RecruitTLogsRequest
loop {
try {
during recovery, only send the full serverDBInfo to processes that are part of the new generation
2019-12-10 05:17:49 +08:00
auto rep = self->findWorkersForConfiguration( req );
fix: we were recruiting one too few oldLogRouters code cleanup
2020-01-03 07:05:44 +08:00
self->db.addRequiredAddresses(rep.oldLogRouters);
self->db.addRequiredAddresses(rep.proxies);
self->db.addRequiredAddresses(rep.resolvers);
self->db.addRequiredAddresses(rep.satelliteTLogs);
self->db.addRequiredAddresses(rep.tLogs);
during recovery, only send the full serverDBInfo to processes that are part of the new generation
2019-12-10 05:17:49 +08:00
self->db.serverInfo->trigger();
req.reply.send( rep );
Initial repository commit
2017-05-26 04:48:44 +08:00
return Void();
} catch (Error& e) {
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
if (e.code() == error_code_no_more_servers && self->goodRecruitmentTime.isReady()) {
Initial repository commit
2017-05-26 04:48:44 +08:00
self->outstandingRecruitmentRequests.push_back( req );
TraceEvent(SevWarn, "RecruitFromConfigurationNotAvailable", self->id).error(e);
return Void();
} else if(e.code() == error_code_operation_failed || e.code() == error_code_no_more_servers) {
//recruitment not good enough, try again
}
else {
TraceEvent(SevError, "RecruitFromConfigurationError", self->id).error(e);
throw; // goodbye, cluster controller
}
}
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
wait( lowPriorityDelay(SERVER_KNOBS->ATTEMPT_RECRUITMENT_DELAY) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
ACTOR Future<Void> clusterRecruitRemoteFromConfiguration( ClusterControllerData* self, RecruitRemoteFromConfigurationRequest req ) {
// At the moment this doesn't really need to be an actor (it always completes immediately)
TEST(true); //ClusterController RecruitTLogsRequest
loop {
try {
during recovery, only send the full serverDBInfo to processes that are part of the new generation
2019-12-10 05:17:49 +08:00
RecruitRemoteFromConfigurationReply rep = self->findRemoteWorkersForConfiguration( req );
fix: we were recruiting one too few oldLogRouters code cleanup
2020-01-03 07:05:44 +08:00
self->db.addRequiredAddresses(rep.remoteTLogs);
self->db.addRequiredAddresses(rep.logRouters);
during recovery, only send the full serverDBInfo to processes that are part of the new generation
2019-12-10 05:17:49 +08:00
self->db.serverInfo->trigger();
req.reply.send( rep );
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
return Void();
} catch (Error& e) {
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
if (e.code() == error_code_no_more_servers && self->goodRemoteRecruitmentTime.isReady()) {
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
self->outstandingRemoteRecruitmentRequests.push_back( req );
TraceEvent(SevWarn, "RecruitRemoteFromConfigurationNotAvailable", self->id).error(e);
return Void();
} else if(e.code() == error_code_operation_failed || e.code() == error_code_no_more_servers) {
//recruitment not good enough, try again
}
else {
TraceEvent(SevError, "RecruitRemoteFromConfigurationError", self->id).error(e);
throw; // goodbye, cluster controller
}
}
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
wait( lowPriorityDelay(SERVER_KNOBS->ATTEMPT_RECRUITMENT_DELAY) );
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
}
Initial repository commit
2017-05-26 04:48:44 +08:00
void clusterRegisterMaster( ClusterControllerData* self, RegisterMasterRequest const& req ) {
req.reply.send( Void() );
This is the first step in eliminating the usage of database names in our code. The C API remains the same, but underneath that all usage of database names is eliminated.
2018-08-17 01:24:12 +08:00
TraceEvent("MasterRegistrationReceived", self->id).detail("MasterId", req.id).detail("Master", req.mi.toString()).detail("Tlogs", describe(req.logSystemConfig.tLogs)).detail("Resolvers", req.resolvers.size())
fix: desiredTLogCount was used instead of getDesiredLogs(), which caused problems with recruitment when desiredTLogCount was -1. canKillProcess logic was wrong. We still need to configure usable_regions because if datacenterVersionDifference is too large we cannot complete data movement.
2018-07-05 04:22:32 +08:00
.detail("RecoveryState", (int)req.recoveryState).detail("RegistrationCount", req.registrationCount).detail("Proxies", req.proxies.size()).detail("RecoveryCount", req.recoveryCount).detail("Stalled", req.recoveryStalled);
Initial repository commit
2017-05-26 04:48:44 +08:00
//make sure the request comes from an active database
auto db = &self->db;
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
if ( db->serverInfo->get().read().master.id() != req.id || req.registrationCount <= db->masterRegistrationCount ) {
TraceEvent("MasterRegistrationNotFound", self->id).detail("MasterId", req.id).detail("ExistingId", db->serverInfo->get().read().master.id()).detail("RegCount", req.registrationCount).detail("ExistingRegCount", db->masterRegistrationCount);
Initial repository commit
2017-05-26 04:48:44 +08:00
return;
}
The cluster controller will increase its failure monitoring delay after there have been many unfinishedRecoveries
2018-09-01 01:51:55 +08:00
if ( req.recoveryState == RecoveryState::FULLY_RECOVERED ) {
self->db.unfinishedRecoveries = 0;
self->db.logGenerations = 0;
ASSERT( !req.logSystemConfig.oldTLogs.size() );
} else {
self->db.logGenerations = std::max<int>(self->db.logGenerations, req.logSystemConfig.oldTLogs.size());
}
Initial repository commit
2017-05-26 04:48:44 +08:00
db->masterRegistrationCount = req.registrationCount;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
db->recoveryStalled = req.recoveryStalled;
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
if ( req.configuration.present() ) {
db->config = req.configuration.get();
fix: kill all tlogs in one region prevents the remote logs from recovering in that region, do not allow that to prevent us from configuring usable_regions=1. added more recovery states.
2018-07-05 15:08:51 +08:00
if ( req.recoveryState >= RecoveryState::ACCEPTING_COMMITS ) {
Add fullyRecoveredConfig for checking exclusions
2017-11-16 09:15:24 +08:00
self->gotFullyRecoveredConfig = true;
db->fullyRecoveredConfig = req.configuration.get();
for ( auto& it : self->id_worker ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
bool isExcludedFromConfig = db->fullyRecoveredConfig.isExcludedServer(it.second.details.interf.address());
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
if ( it.second.priorityInfo.isExcluded != isExcludedFromConfig ) {
it.second.priorityInfo.isExcluded = isExcludedFromConfig;
if( !it.second.reply.isSet() ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
it.second.reply.send( RegisterWorkerReply( it.second.details.processClass, it.second.priorityInfo ) );
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
Add fullyRecoveredConfig for checking exclusions
2017-11-16 09:15:24 +08:00
}
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
}
}
}
Initial repository commit
2017-05-26 04:48:44 +08:00
bool isChanged = false;
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
auto cachedInfo = self->db.serverInfo->get();
auto& dbInfo = cachedInfo.mutate();
Initial repository commit
2017-05-26 04:48:44 +08:00
if (dbInfo.recoveryState != req.recoveryState) {
dbInfo.recoveryState = req.recoveryState;
isChanged = true;
}
if (dbInfo.priorCommittedLogServers != req.priorCommittedLogServers) {
dbInfo.priorCommittedLogServers = req.priorCommittedLogServers;
isChanged = true;
}
// Construct the client information
if (db->clientInfo->get().proxies != req.proxies) {
isChanged = true;
ClientDBInfo clientInfo;
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
clientInfo.id = deterministicRandom()->randomUniqueID();
Initial repository commit
2017-05-26 04:48:44 +08:00
clientInfo.proxies = req.proxies;
clientInfo.clientTxnInfoSampleRate = db->clientInfo->get().clientTxnInfoSampleRate;
clientInfo.clientTxnInfoSizeLimit = db->clientInfo->get().clientTxnInfoSizeLimit;
db->clientInfo->set( clientInfo );
dbInfo.client = db->clientInfo->get();
}
if( !dbInfo.logSystemConfig.isEqual(req.logSystemConfig) ) {
isChanged = true;
dbInfo.logSystemConfig = req.logSystemConfig;
}
if( dbInfo.resolvers != req.resolvers ) {
isChanged = true;
dbInfo.resolvers = req.resolvers;
}
if( dbInfo.recoveryCount != req.recoveryCount ) {
isChanged = true;
dbInfo.recoveryCount = req.recoveryCount;
}
if( isChanged ) {
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
dbInfo.id = deterministicRandom()->randomUniqueID();
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
self->db.serverInfo->set( cachedInfo );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
checking outstanding requests can be CPU intensive, so rate limit checking requests
2018-06-28 14:02:08 +08:00
checkOutstandingRequests(self);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
void registerWorker( RegisterWorkerRequest req, ClusterControllerData *self ) {
Restore master PID in worker registration This fix is lost during merge.
2019-03-24 11:55:03 +08:00
const WorkerInterface& w = req.wi;
Introduce prevChangeID to CandidacyRequest and LeaderHeartbeatRequest
2017-10-13 08:11:58 +08:00
ProcessClass newProcessClass = req.processClass;
Initial repository commit
2017-05-26 04:48:44 +08:00
auto info = self->id_worker.find( w.locality.processId() );
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
ClusterControllerPriorityInfo newPriorityInfo = req.priorityInfo;
fix: processClassFitness could be wrong if the client changed their class while rebooting
2019-03-22 08:56:04 +08:00
newPriorityInfo.processClassFitness = newProcessClass.machineClassFitness(ProcessClass::ClusterController);
Initial repository commit
2017-05-26 04:48:44 +08:00
suppressed spammy trace events, and avoid reporting a long master recovery duration when the cluster is first created
2018-11-05 15:07:56 +08:00
if(info == self->id_worker.end()) {
Defer formatting in traces to make them cheaper This is the first part of making `TraceEvent` cheaper. The main idea is to defer calls to any code that formats string. These are the main changes: - TraceEvent::detail now takes a c-string instead of std::string for literals. This prevents unnecessary allocations if the trace is not going to be printed in the first place (for example for SevDebug). Before that `detail` expected a `std::string` as key, which mean that any string literal would be copied on each call. - Templates Traceable and SpecialTraceMetricType. These templates can be specialized for any type that needs to be printed. The actual formatting will be deferred to after the `enabled` check. This provides two benefits: (1) if a TraceEvent is disabled, we don't pay for the formatting and (2) TraceEvent can trace types that it doesn't know about. - TraceEvent::enabled will be set in the constructor if the Severity is passed. This will make sure that `TraceEvent::init` is not called. - `TraceEvent::detail` will be inlined. So for disabled TraceEvent calls, a call to detail will only introduce a if-branch which is much cheaper than a function call.
2019-03-15 09:40:28 +08:00
TraceEvent("ClusterControllerActualWorkers", self->id).detail("WorkerId",w.id()).detail("ProcessId", w.locality.processId()).detail("ZoneId", w.locality.zoneId()).detail("DataHall", w.locality.dataHallId()).detail("PClass", req.processClass.toString()).detail("Workers", self->id_worker.size());
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
self->goodRecruitmentTime = lowPriorityDelay(SERVER_KNOBS->WAIT_FOR_GOOD_RECRUITMENT_DELAY);
updateGoodRemoteRecruitmentTime was unnecessary because the only way findRemoteWorkers would return would be after a new server has joined which already resets goodRemoteRecruitmentTime
2020-02-21 05:46:22 +08:00
self->goodRemoteRecruitmentTime = lowPriorityDelay(SERVER_KNOBS->WAIT_FOR_GOOD_REMOTE_RECRUITMENT_DELAY);
suppressed spammy trace events, and avoid reporting a long master recovery duration when the cluster is first created
2018-11-05 15:07:56 +08:00
} else {
Defer formatting in traces to make them cheaper This is the first part of making `TraceEvent` cheaper. The main idea is to defer calls to any code that formats string. These are the main changes: - TraceEvent::detail now takes a c-string instead of std::string for literals. This prevents unnecessary allocations if the trace is not going to be printed in the first place (for example for SevDebug). Before that `detail` expected a `std::string` as key, which mean that any string literal would be copied on each call. - Templates Traceable and SpecialTraceMetricType. These templates can be specialized for any type that needs to be printed. The actual formatting will be deferred to after the `enabled` check. This provides two benefits: (1) if a TraceEvent is disabled, we don't pay for the formatting and (2) TraceEvent can trace types that it doesn't know about. - TraceEvent::enabled will be set in the constructor if the Severity is passed. This will make sure that `TraceEvent::init` is not called. - `TraceEvent::detail` will be inlined. So for disabled TraceEvent calls, a call to detail will only introduce a if-branch which is much cheaper than a function call.
2019-03-15 09:40:28 +08:00
TraceEvent("ClusterControllerWorkerAlreadyRegistered", self->id).suppressFor(1.0).detail("WorkerId",w.id()).detail("ProcessId", w.locality.processId()).detail("ZoneId", w.locality.zoneId()).detail("DataHall", w.locality.dataHallId()).detail("PClass", req.processClass.toString()).detail("Workers", self->id_worker.size());
suppressed spammy trace events, and avoid reporting a long master recovery duration when the cluster is first created
2018-11-05 15:07:56 +08:00
}
Use processId instead of address for comparison
2017-10-26 02:35:29 +08:00
if ( w.address() == g_network->getLocalAddress() ) {
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
if(self->changingDcIds.get().first) {
if(self->changingDcIds.get().second.present()) {
newPriorityInfo.dcFitness = ClusterControllerPriorityInfo::calculateDCFitness( w.locality.dcId(), self->changingDcIds.get().second.get() );
}
} else if(self->changedDcIds.get().second.present()) {
newPriorityInfo.dcFitness = ClusterControllerPriorityInfo::calculateDCFitness( w.locality.dcId(), self->changedDcIds.get().second.get() );
}
} else {
if(!self->changingDcIds.get().first) {
if(self->changingDcIds.get().second.present()) {
newPriorityInfo.dcFitness = ClusterControllerPriorityInfo::calculateDCFitness( w.locality.dcId(), self->changingDcIds.get().second.get() );
}
} else if(self->changedDcIds.get().second.present()) {
newPriorityInfo.dcFitness = ClusterControllerPriorityInfo::calculateDCFitness( w.locality.dcId(), self->changedDcIds.get().second.get() );
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
Use processId instead of address for comparison
2017-10-26 02:35:29 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
// Check process class and exclusive property
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if ( info == self->id_worker.end() || info->second.details.interf.id() != w.id() || req.generation >= info->second.gen ) {
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
if ( self->gotProcessClasses ) {
auto classIter = self->id_class.find(w.locality.processId());
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
if( classIter != self->id_class.end() && (classIter->second.classSource() == ProcessClass::DBSource || req.initialClass.classType() == ProcessClass::UnsetClass)) {
newProcessClass = classIter->second;
} else {
newProcessClass = req.initialClass;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
newPriorityInfo.processClassFitness = newProcessClass.machineClassFitness(ProcessClass::ClusterController);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Add fullyRecoveredConfig for checking exclusions
2017-11-16 09:15:24 +08:00
if ( self->gotFullyRecoveredConfig ) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
newPriorityInfo.isExcluded = self->db.fullyRecoveredConfig.isExcludedServer(w.address());
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
// Notify the worker to register again with new process class/exclusive property
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
if ( !req.reply.isSet() && newPriorityInfo != req.priorityInfo ) {
req.reply.send( RegisterWorkerReply(newProcessClass, newPriorityInfo) );
Introduce prevChangeID to CandidacyRequest and LeaderHeartbeatRequest
2017-10-13 08:11:58 +08:00
}
}
if( info == self->id_worker.end() ) {
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
self->id_worker[w.locality.processId()] = WorkerInfo( workerAvailabilityWatch( w, newProcessClass, self ), req.reply, req.generation, w, req.initialClass, newProcessClass, newPriorityInfo, req.degraded );
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
if (!self->masterProcessId.present() && w.locality.processId() == self->db.serverInfo->get().read().master.locality.processId()) {
Restore master PID in worker registration This fix is lost during merge.
2019-03-24 11:55:03 +08:00
self->masterProcessId = w.locality.processId();
}
Initial repository commit
2017-05-26 04:48:44 +08:00
checkOutstandingRequests( self );
make sure id_worker is updated before setting ratekeeper or data distribution
2019-03-23 08:08:54 +08:00
} else if( info->second.details.interf.id() != w.id() || req.generation >= info->second.gen ) {
Catch and update processClass change from DBSource
2017-09-26 01:36:03 +08:00
if (!info->second.reply.isSet()) {
info->second.reply.send( Never() );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
info->second.reply = req.reply;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
info->second.details.processClass = newProcessClass;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
info->second.priorityInfo = newPriorityInfo;
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
info->second.initialClass = req.initialClass;
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
info->second.details.degraded = req.degraded;
Initial repository commit
2017-05-26 04:48:44 +08:00
info->second.gen = req.generation;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if(info->second.details.interf.id() != w.id()) {
info->second.details.interf = w;
Introduce prevChangeID to CandidacyRequest and LeaderHeartbeatRequest
2017-10-13 08:11:58 +08:00
info->second.watcher = workerAvailabilityWatch( w, newProcessClass, self );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
fix: tlogFitness did not consider it better to have one tlog of a better fitness fix: checkStable was not used in all places in better master exists fix: we need to call checkOutstanding on worker registration in all cases fix: in case persistentData is keyValueStoreMemory, we need to make sure it is fully recovered before writing to it
2018-01-05 03:33:02 +08:00
checkOutstandingRequests( self );
make sure id_worker is updated before setting ratekeeper or data distribution
2019-03-23 08:08:54 +08:00
} else {
TEST(true); // Received an old worker registration request.
Initial repository commit
2017-05-26 04:48:44 +08:00
}
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
if (req.distributorInterf.present() && !self->db.serverInfo->get().read().distributor.present() &&
Fix the issue of slow dying Data Distributor Test with: -r simulation -f ./foundationdb/tests/slow/CommitBug.txt -s 67828576 -b on The test has the following event sequence: - Time 113.3s, CC noticed DD failure, cleard DD interface. - 1s later, DD rejoined and registered with CC. - Time 131.7s, DD actor cancelled. This old DD raced to register with CC and the failure monitor is not installed because monitorDataDistributor is stalled waiting for new DD. - Time 161.4s, new DD running. New DD recruting was delayed due to no servers in the period. Fix by disabling DD registration during the recruting process.
2019-03-18 11:55:59 +08:00
self->clusterControllerDcId == req.distributorInterf.get().locality.dcId() &&
!self->recruitingDistributor) {
Piggyback data distributor interface in worker registration This allows cluster controller to know data distributor during worker registration phase, thus avoiding recruiting a new data distributor after starting. Also change the worker to skip creating a new data distributor if there is already one running on the worker, which can trigger operation timeout in tests.
2019-01-29 03:29:39 +08:00
const DataDistributorInterface& di = req.distributorInterf.get();
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCRegisterDataDistributor", self->id).detail("DDID", di.id());
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
self->db.setDistributor(di);
Piggyback data distributor interface in worker registration This allows cluster controller to know data distributor during worker registration phase, thus avoiding recruiting a new data distributor after starting. Also change the worker to skip creating a new data distributor if there is already one running on the worker, which can trigger operation timeout in tests.
2019-01-29 03:29:39 +08:00
}
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
if (req.ratekeeperInterf.present()) {
startRatekeeper is responsible for updating serverDBInfo
2019-03-23 08:56:16 +08:00
if((self->recruitingRatekeeperID.present() && self->recruitingRatekeeperID.get() != req.ratekeeperInterf.get().id()) ||
fix: halt if datacenters are different
2019-03-23 14:53:21 +08:00
self->clusterControllerDcId != w.locality.dcId()) {
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCHaltRegisteringRatekeeper", self->id).detail("RKID", req.ratekeeperInterf.get().id())
startRatekeeper is responsible for updating serverDBInfo
2019-03-23 08:56:16 +08:00
.detail("DcID", printable(self->clusterControllerDcId))
ensure that the processId exists in id_worker before accessing it
2019-03-23 09:54:39 +08:00
.detail("ReqDcID", printable(w.locality.dcId()))
startRatekeeper is responsible for updating serverDBInfo
2019-03-23 08:56:16 +08:00
.detail("RecruitingRKID", self->recruitingRatekeeperID.present() ? self->recruitingRatekeeperID.get() : UID());
ensure that the processId exists in id_worker before accessing it
2019-03-23 09:54:39 +08:00
self->id_worker[w.locality.processId()].haltRatekeeper = brokenPromiseToNever(req.ratekeeperInterf.get().haltRatekeeper.getReply(HaltRatekeeperRequest(self->id)));
startRatekeeper is responsible for updating serverDBInfo
2019-03-23 08:56:16 +08:00
} else if(!self->recruitingRatekeeperID.present()) {
Fix Ratekeeper and DataDistributor placement Make sure both RateKeeper and DataDistributor are placed in the same data center as the Master. Make sure only one RateKeeper is live in the cluster as well.
2019-03-16 08:06:15 +08:00
const RatekeeperInterface& rki = req.ratekeeperInterf.get();
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
const auto& ratekeeper = self->db.serverInfo->get().read().ratekeeper;
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCRegisterRatekeeper", self->id).detail("RKID", rki.id());
ensure that the processId exists in id_worker before accessing it
2019-03-23 09:54:39 +08:00
if (ratekeeper.present() && ratekeeper.get().id() != rki.id() && self->id_worker.count(ratekeeper.get().locality.processId())) {
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCHaltPreviousRatekeeper", self->id).detail("RKID", ratekeeper.get().id())
Restore master PID in cluster controller worker registration CC may think master failed and clear the master PID, which can block both data distributor and ratekeeper recruitment. Fix by restoring it during worker registration.
2019-03-23 05:45:12 +08:00
.detail("DcID", printable(self->clusterControllerDcId))
ensure that the processId exists in id_worker before accessing it
2019-03-23 09:54:39 +08:00
.detail("ReqDcID", printable(w.locality.dcId()))
Restore master PID in cluster controller worker registration CC may think master failed and clear the master PID, which can block both data distributor and ratekeeper recruitment. Fix by restoring it during worker registration.
2019-03-23 05:45:12 +08:00
.detail("RecruitingRKID", self->recruitingRatekeeperID.present() ? self->recruitingRatekeeperID.get() : UID());
Add actors to store halt request futures Address best fitness in checking better DD or RK.
2019-03-23 09:00:16 +08:00
self->id_worker[ratekeeper.get().locality.processId()].haltRatekeeper = brokenPromiseToNever(ratekeeper.get().haltRatekeeper.getReply(HaltRatekeeperRequest(self->id)));
Fix ratekeeper recruitment bug
2019-03-21 04:54:15 +08:00
}
once we recruit a ratekeeper, do not allow any other ratekeepers to register
2019-03-25 02:04:39 +08:00
if(!ratekeeper.present() || ratekeeper.get().id() != rki.id()) {
startRatekeeper is responsible for updating serverDBInfo
2019-03-23 08:56:16 +08:00
self->db.setRatekeeper(rki);
Fix ratekeeper recruitment bug
2019-03-21 04:54:15 +08:00
}
Fix Ratekeeper and DataDistributor placement Make sure both RateKeeper and DataDistributor are placed in the same data center as the Master. Make sure only one RateKeeper is live in the cluster as well.
2019-03-16 08:06:15 +08:00
}
Piggyback data distributor interface in worker registration This allows cluster controller to know data distributor during worker registration phase, thus avoiding recruiting a new data distributor after starting. Also change the worker to skip creating a new data distributor if there is already one running on the worker, which can trigger operation timeout in tests.
2019-01-29 03:29:39 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Fixing PR comments <rdar://problem/34557380> Need a way to map real time to version
2017-09-28 07:31:38 +08:00
#define TIME_KEEPER_VERSION LiteralStringRef("1")
ACTOR Future<Void> timeKeeperSetVersion(ClusterControllerData *self) {
Merge branch 'release-5.1' into feature-remote-logs # Conflicts: # fdbclient/NativeAPI.actor.cpp # fdbrpc/Locality.h # fdbrpc/simulator.h # fdbserver/ApplyMetadataMutation.h # fdbserver/ClusterController.actor.cpp # fdbserver/LogSystemPeekCursor.actor.cpp # fdbserver/MasterProxyServer.actor.cpp # fdbserver/SimulatedCluster.actor.cpp # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/masterserver.actor.cpp # flow/Net2.actor.cpp # tests/fast/SidebandWithStatus.txt # tests/rare/LargeApiCorrectnessStatus.txt # tests/slow/DDBalanceAndRemoveStatus.txt
2018-01-06 03:33:42 +08:00
state Reference<ReadYourWritesTransaction> tr = Reference<ReadYourWritesTransaction>(new ReadYourWritesTransaction(self->cx));
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
loop {
try {
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr->setOption(FDBTransactionOptions::LOCK_AWARE);
make timekeeper priority system immediate
2018-01-09 10:21:00 +08:00
tr->setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
tr->set(timeKeeperVersionKey, TIME_KEEPER_VERSION);
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr->commit());
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
break;
} catch (Error &e) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr->onError(e));
Fixing PR comments <rdar://problem/34557380> Need a way to map real time to version
2017-09-28 07:31:38 +08:00
}
}
return Void();
}
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
// This actor periodically gets read version and writes it to cluster with current timestamp as key. To avoid running
// out of space, it limits the max number of entries and clears old entries on each update. This mapping is used from
// backup and restore to get the version information for a timestamp.
ACTOR Future<Void> timeKeeper(ClusterControllerData *self) {
state KeyBackedMap<int64_t, Version> versionMap(timeKeeperPrefixRange.begin);
call debug transaction on timekeeper
2018-11-03 03:56:29 +08:00
TraceEvent("TimeKeeperStarted");
Fixing PR comments <rdar://problem/34557380> Need a way to map real time to version
2017-09-28 07:31:38 +08:00
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(timeKeeperSetVersion(self));
Fixing PR comments <rdar://problem/34557380> Need a way to map real time to version
2017-09-28 07:31:38 +08:00
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
loop {
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
state Reference<ReadYourWritesTransaction> tr = Reference<ReadYourWritesTransaction>(new ReadYourWritesTransaction(self->cx));
loop {
try {
call debug transaction on timekeeper
2018-11-03 03:56:29 +08:00
if(!g_network->isSimulated()) {
addressed review comments
2018-11-05 12:26:23 +08:00
// This is done to provide an arbitrary logged transaction every ~10s.
// FIXME: replace or augment this with logging on the proxy which tracks
// how long it is taking to hear responses from each other component.
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
UID debugID = deterministicRandom()->randomUniqueID();
Renamed TimeKeeperStarted to TimeKeeperCommit
2018-11-03 03:57:03 +08:00
TraceEvent("TimeKeeperCommit", debugID);
call debug transaction on timekeeper
2018-11-03 03:56:29 +08:00
tr->debugTransaction(debugID);
}
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr->setOption(FDBTransactionOptions::LOCK_AWARE);
make timekeeper priority system immediate
2018-01-09 10:21:00 +08:00
tr->setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
Optional<Value> disableValue = wait( tr->get(timeKeeperDisableKey) );
if(disableValue.present()) {
break;
}
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
Version v = tr->getReadVersion().get();
int64_t currentTime = (int64_t)now();
versionMap.set(tr, currentTime, v);
added the ability to disable timeKeeper; disabled timeKeeper before consistency check in simulation
2017-09-29 04:13:24 +08:00
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
int64_t ttl = currentTime - SERVER_KNOBS->TIME_KEEPER_DELAY * SERVER_KNOBS->TIME_KEEPER_MAX_ENTRIES;
if (ttl > 0) {
versionMap.erase(tr, 0, ttl);
added the ability to disable timeKeeper; disabled timeKeeper before consistency check in simulation
2017-09-29 04:13:24 +08:00
}
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr->commit());
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
break;
} catch (Error &e) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr->onError(e));
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
}
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(delay(SERVER_KNOBS->TIME_KEEPER_DELAY));
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
}
}
Initial repository commit
2017-05-26 04:48:44 +08:00
ACTOR Future<Void> statusServer(FutureStream< StatusRequest> requests,
ClusterControllerData *self,
ServerCoordinators coordinators)
{
// Seconds since the END of the last GetStatus executed
state double last_request_time = 0.0;
// Place to accumulate a batch of requests to respond to
state std::vector<StatusRequest> requests_batch;
loop {
try {
// Wait til first request is ready
StatusRequest req = waitNext(requests);
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
++self->statusRequests;
Initial repository commit
2017-05-26 04:48:44 +08:00
requests_batch.push_back(req);
// Earliest time at which we may begin a new request
double next_allowed_request_time = last_request_time + SERVER_KNOBS->STATUS_MIN_TIME_BETWEEN_REQUESTS;
// Wait if needed to satisfy min_time knob, also allows more requets to queue up.
double minwait = std::max(next_allowed_request_time - now(), 0.0);
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(delay(minwait));
Initial repository commit
2017-05-26 04:48:44 +08:00
// Get all requests that are ready right *now*, before GetStatus() begins.
// All of these requests will be responded to with the next GetStatus() result.
Throttle status requests with MAX_STATUS_REQUESTS_PER_SECOND knob, whenever status batching is used.
2019-01-29 07:37:30 +08:00
// If requests are batched, do not respond to more than MAX_STATUS_REQUESTS_PER_SECOND
// requests per second
while (requests.isReady()) {
auto req = requests.pop();
if (SERVER_KNOBS->STATUS_MIN_TIME_BETWEEN_REQUESTS > 0.0 &&
requests_batch.size() + 1 >
SERVER_KNOBS->STATUS_MIN_TIME_BETWEEN_REQUESTS * SERVER_KNOBS->MAX_STATUS_REQUESTS_PER_SECOND) {
Increase severity to SevWarnAlways for TooManyStatusRequests trace Co-Authored-By: tclinken <trevorclinkenbeard@gmail.com>
2019-01-29 09:50:50 +08:00
TraceEvent(SevWarnAlways, "TooManyStatusRequests").suppressFor(1.0).detail("BatchSize", requests_batch.size());
Throttle status requests with MAX_STATUS_REQUESTS_PER_SECOND knob, whenever status batching is used.
2019-01-29 07:37:30 +08:00
req.reply.sendError(server_overloaded());
} else {
requests_batch.push_back(req);
}
}
Initial repository commit
2017-05-26 04:48:44 +08:00
// Get status but trap errors to send back to client.
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
vector<WorkerDetails> workers;
Initial repository commit
2017-05-26 04:48:44 +08:00
for(auto& it : self->id_worker)
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
workers.push_back(it.second.details);
Initial repository commit
2017-05-26 04:48:44 +08:00
std::vector<NetworkAddress> incompatibleConnections;
for(auto it = self->db.incompatibleConnections.begin(); it != self->db.incompatibleConnections.end();) {
if(it->second < now()) {
it = self->db.incompatibleConnections.erase(it);
} else {
incompatibleConnections.push_back(it->first);
it++;
}
}
added status information on connected clients
2019-07-26 08:15:31 +08:00
state ErrorOr<StatusReply> result = wait(errorOr(clusterGetStatus(self->db.serverInfo, self->cx, workers, self->db.workersWithIssues, &self->db.clientStatus, coordinators, incompatibleConnections, self->datacenterVersionDifference)));
Status: TLS client status Use ClientStatusInfo structure for each network address (client), instead of passing each status info as a parameter.
2019-02-27 08:20:05 +08:00
Initial repository commit
2017-05-26 04:48:44 +08:00
if (result.isError() && result.getError().code() == error_code_actor_cancelled)
throw result.getError();
// Update last_request_time now because GetStatus is finished and the delay is to be measured between requests
last_request_time = now();
while (!requests_batch.empty())
{
if (result.isError())
requests_batch.back().reply.sendError(result.getError());
else
requests_batch.back().reply.send(result.get());
requests_batch.pop_back();
fixed merge problem
2018-08-22 23:47:47 +08:00
wait( yield() );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
catch (Error &e) {
TraceEvent(SevError, "StatusServerError").error(e);
throw e;
}
}
}
ACTOR Future<Void> monitorProcessClasses(ClusterControllerData *self) {
state ReadYourWritesTransaction trVer( self->db.db );
loop {
try {
trVer.setOption( FDBTransactionOptions::ACCESS_SYSTEM_KEYS );
trVer.setOption( FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE );
Optional<Value> val = wait(trVer.get(processClassVersionKey));
if (val.present())
break;
Standalone<RangeResultRef> processClasses = wait( trVer.getRange( processClassKeys, CLIENT_KNOBS->TOO_MANY ) );
ASSERT( !processClasses.more && processClasses.size() < CLIENT_KNOBS->TOO_MANY );
trVer.clear(processClassKeys);
trVer.set(processClassVersionKey, processClassVersionValue);
for (auto it : processClasses) {
UID processUid = decodeProcessClassKeyOld(it.key);
trVer.set(processClassKeyFor(processUid.toString()), it.value);
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(trVer.commit());
Initial repository commit
2017-05-26 04:48:44 +08:00
TraceEvent("ProcessClassUpgrade");
break;
}
catch(Error &e) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( trVer.onError(e) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
loop {
state ReadYourWritesTransaction tr( self->db.db );
loop {
try {
tr.setOption( FDBTransactionOptions::ACCESS_SYSTEM_KEYS );
tr.setOption( FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE );
Standalone<RangeResultRef> processClasses = wait( tr.getRange( processClassKeys, CLIENT_KNOBS->TOO_MANY ) );
ASSERT( !processClasses.more && processClasses.size() < CLIENT_KNOBS->TOO_MANY );
if(processClasses != self->lastProcessClasses || !self->gotProcessClasses) {
self->id_class.clear();
for( int i = 0; i < processClasses.size(); i++ ) {
auto c = decodeProcessClassValue( processClasses[i].value );
ASSERT( c.classSource() != ProcessClass::CommandLineSource );
self->id_class[decodeProcessClassKey( processClasses[i].key )] = c;
}
for( auto& w : self->id_worker ) {
auto classIter = self->id_class.find(w.first);
Catch and update processClass change from DBSource
2017-09-26 01:36:03 +08:00
ProcessClass newProcessClass;
Initial repository commit
2017-05-26 04:48:44 +08:00
if( classIter != self->id_class.end() && (classIter->second.classSource() == ProcessClass::DBSource || w.second.initialClass.classType() == ProcessClass::UnsetClass) ) {
Catch and update processClass change from DBSource
2017-09-26 01:36:03 +08:00
newProcessClass = classIter->second;
Initial repository commit
2017-05-26 04:48:44 +08:00
} else {
Catch and update processClass change from DBSource
2017-09-26 01:36:03 +08:00
newProcessClass = w.second.initialClass;
}
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if (newProcessClass != w.second.details.processClass) {
w.second.details.processClass = newProcessClass;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
w.second.priorityInfo.processClassFitness = newProcessClass.machineClassFitness(ProcessClass::ClusterController);
Fix cluster controller review comments
2017-10-05 06:48:55 +08:00
if (!w.second.reply.isSet()) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
w.second.reply.send( RegisterWorkerReply(w.second.details.processClass, w.second.priorityInfo) );
Fix cluster controller review comments
2017-10-05 06:48:55 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
self->lastProcessClasses = processClasses;
self->gotProcessClasses = true;
checkOutstandingRequests( self );
}
state Future<Void> watchFuture = tr.watch(processClassChangeKey);
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr.commit());
wait(watchFuture);
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
}
catch(Error &e) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( tr.onError(e) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
}
}
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
ACTOR Future<Void> monitorServerInfoConfig(ClusterControllerData::DBInfo* db) {
loop {
state ReadYourWritesTransaction tr(db->db);
loop {
try {
tr.setOption(FDBTransactionOptions::READ_SYSTEM_KEYS);
tr.setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
tr.setOption(FDBTransactionOptions::READ_LOCK_AWARE);
Optional<Value> configVal = wait(tr.get(latencyBandConfigKey));
Optional<LatencyBandConfig> config;
if(configVal.present()) {
config = LatencyBandConfig::parse(configVal.get());
}
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
auto cachedInfo = db->serverInfo->get();
auto& serverInfo = cachedInfo.mutate();
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
if(config != serverInfo.latencyBandConfig) {
TraceEvent("LatencyBandConfigChanged").detail("Present", config.present());
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
serverInfo.id = deterministicRandom()->randomUniqueID();
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
serverInfo.latencyBandConfig = config;
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
db->serverInfo->set(cachedInfo);
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
}
state Future<Void> configChangeFuture = tr.watch(latencyBandConfigKey);
Merge commit '3b2700d25334c53d13496ca16682642aac951beb' into track-server-request-latencies # Conflicts: # fdbclient/MasterProxyInterface.h # fdbserver/ClusterController.actor.cpp # fdbserver/MasterProxyServer.actor.cpp # fdbserver/ServerDBInfo.h # fdbserver/Status.actor.cpp # fdbserver/fdbserver.vcxproj # fdbserver/storageserver.actor.cpp
2019-01-25 03:43:26 +08:00
wait(tr.commit());
wait(configChangeFuture);
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
break;
}
catch (Error &e) {
Merge commit '3b2700d25334c53d13496ca16682642aac951beb' into track-server-request-latencies # Conflicts: # fdbclient/MasterProxyInterface.h # fdbserver/ClusterController.actor.cpp # fdbserver/MasterProxyServer.actor.cpp # fdbserver/ServerDBInfo.h # fdbserver/Status.actor.cpp # fdbserver/fdbserver.vcxproj # fdbserver/storageserver.actor.cpp
2019-01-25 03:43:26 +08:00
wait(tr.onError(e));
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
}
}
}
}
Initial repository commit
2017-05-26 04:48:44 +08:00
ACTOR Future<Void> monitorClientTxnInfoConfigs(ClusterControllerData::DBInfo* db) {
loop {
state ReadYourWritesTransaction tr(db->db);
loop {
try {
tr.setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr.setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
Address code review comments. * Fixed memory corruption with SystemData key constants * Removed duplication in ClusterController * Reworked fdbcli actions to better represent explicit vs default assignments
2017-10-04 11:57:39 +08:00
state Optional<Value> rateVal = wait(tr.get(fdbClientInfoTxnSampleRate));
state Optional<Value> limitVal = wait(tr.get(fdbClientInfoTxnSizeLimit));
Initial repository commit
2017-05-26 04:48:44 +08:00
ClientDBInfo clientInfo = db->clientInfo->get();
onMasterProxiesChanged was being triggered when any member of ClientDBInfo changed. Change the behavior to be triggered only when proxies field in ClientDBInfo is changed.
2017-11-02 09:29:56 +08:00
double sampleRate = rateVal.present() ? BinaryReader::fromStringRef<double>(rateVal.get(), Unversioned()) : std::numeric_limits<double>::infinity();
int64_t sizeLimit = limitVal.present() ? BinaryReader::fromStringRef<int64_t>(limitVal.get(), Unversioned()) : -1;
if (sampleRate != clientInfo.clientTxnInfoSampleRate || sizeLimit != clientInfo.clientTxnInfoSampleRate) {
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
clientInfo.id = deterministicRandom()->randomUniqueID();
onMasterProxiesChanged was being triggered when any member of ClientDBInfo changed. Change the behavior to be triggered only when proxies field in ClientDBInfo is changed.
2017-11-02 09:29:56 +08:00
clientInfo.clientTxnInfoSampleRate = sampleRate;
clientInfo.clientTxnInfoSizeLimit = sizeLimit;
Initial repository commit
2017-05-26 04:48:44 +08:00
db->clientInfo->set(clientInfo);
}
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
Address code review comments. * Fixed memory corruption with SystemData key constants * Removed duplication in ClusterController * Reworked fdbcli actions to better represent explicit vs default assignments
2017-10-04 11:57:39 +08:00
state Future<Void> watchRateFuture = tr.watch(fdbClientInfoTxnSampleRate);
state Future<Void> watchLimitFuture = tr.watch(fdbClientInfoTxnSizeLimit);
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr.commit());
Initial repository commit
2017-05-26 04:48:44 +08:00
choose {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when(wait(watchRateFuture)) { break; }
when (wait(watchLimitFuture)) { break; }
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
catch (Error &e) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr.onError(e));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
}
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
ACTOR Future<Void> updatedChangingDatacenters(ClusterControllerData *self) {
//do not change the cluster controller until all the processes have had a chance to register
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( delay(SERVER_KNOBS->WAIT_FOR_GOOD_RECRUITMENT_DELAY) );
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
loop {
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
state Future<Void> onChange = self->desiredDcIds.onChange();
if(!self->desiredDcIds.get().present()) {
self->changingDcIds.set(std::make_pair(false,self->desiredDcIds.get()));
} else {
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
auto& worker = self->id_worker[self->clusterControllerProcessId];
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
uint8_t newFitness = ClusterControllerPriorityInfo::calculateDCFitness( worker.details.interf.locality.dcId(), self->desiredDcIds.get().get() );
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
self->changingDcIds.set(std::make_pair(worker.priorityInfo.dcFitness > newFitness,self->desiredDcIds.get()));
added trace events for cluster controller changing datacenters
2018-07-03 01:06:54 +08:00
TraceEvent("UpdateChangingDatacenter", self->id).detail("OldFitness", worker.priorityInfo.dcFitness).detail("NewFitness", newFitness);
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
if ( worker.priorityInfo.dcFitness > newFitness ) {
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
worker.priorityInfo.dcFitness = newFitness;
if(!worker.reply.isSet()) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
worker.reply.send( RegisterWorkerReply( worker.details.processClass, worker.priorityInfo ) );
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
}
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
} else {
state int currentFit = ProcessClass::BestFit;
while(currentFit <= ProcessClass::NeverAssign) {
bool updated = false;
for ( auto& it : self->id_worker ) {
if( ( !it.second.priorityInfo.isExcluded && it.second.priorityInfo.processClassFitness == currentFit ) || currentFit == ProcessClass::NeverAssign ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
uint8_t fitness = ClusterControllerPriorityInfo::calculateDCFitness( it.second.details.interf.locality.dcId(), self->changingDcIds.get().second.get() );
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
if ( it.first != self->clusterControllerProcessId && it.second.priorityInfo.dcFitness != fitness ) {
updated = true;
it.second.priorityInfo.dcFitness = fitness;
if(!it.second.reply.isSet()) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
it.second.reply.send( RegisterWorkerReply( it.second.details.processClass, it.second.priorityInfo ) );
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
}
}
}
}
if(updated && currentFit < ProcessClass::NeverAssign) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( delay(SERVER_KNOBS->CC_CLASS_DELAY) );
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
}
currentFit++;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
}
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(onChange);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
}
ACTOR Future<Void> updatedChangedDatacenters(ClusterControllerData *self) {
state Future<Void> changeDelay = delay(SERVER_KNOBS->CC_CHANGE_DELAY);
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
state Future<Void> onChange = self->changingDcIds.onChange();
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
loop {
choose {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when( wait(onChange) ) {
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
changeDelay = delay(SERVER_KNOBS->CC_CHANGE_DELAY);
onChange = self->changingDcIds.onChange();
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when( wait(changeDelay) ) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
changeDelay = Never();
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
onChange = self->changingDcIds.onChange();
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
self->changedDcIds.set(self->changingDcIds.get());
if(self->changedDcIds.get().second.present()) {
added trace events for cluster controller changing datacenters
2018-07-03 01:06:54 +08:00
TraceEvent("UpdateChangedDatacenter", self->id).detail("CCFirst", self->changedDcIds.get().first);
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
if( !self->changedDcIds.get().first ) {
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
auto& worker = self->id_worker[self->clusterControllerProcessId];
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
uint8_t newFitness = ClusterControllerPriorityInfo::calculateDCFitness( worker.details.interf.locality.dcId(), self->changedDcIds.get().second.get() );
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
if( worker.priorityInfo.dcFitness != newFitness ) {
worker.priorityInfo.dcFitness = newFitness;
if(!worker.reply.isSet()) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
worker.reply.send( RegisterWorkerReply( worker.details.processClass, worker.priorityInfo ) );
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
} else {
state int currentFit = ProcessClass::BestFit;
while(currentFit <= ProcessClass::NeverAssign) {
bool updated = false;
for ( auto& it : self->id_worker ) {
if( ( !it.second.priorityInfo.isExcluded && it.second.priorityInfo.processClassFitness == currentFit ) || currentFit == ProcessClass::NeverAssign ) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
uint8_t fitness = ClusterControllerPriorityInfo::calculateDCFitness( it.second.details.interf.locality.dcId(), self->changedDcIds.get().second.get() );
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
if ( it.first != self->clusterControllerProcessId && it.second.priorityInfo.dcFitness != fitness ) {
updated = true;
it.second.priorityInfo.dcFitness = fitness;
if(!it.second.reply.isSet()) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
it.second.reply.send( RegisterWorkerReply( it.second.details.processClass, it.second.priorityInfo ) );
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
}
}
}
}
if(updated && currentFit < ProcessClass::NeverAssign) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( delay(SERVER_KNOBS->CC_CLASS_DELAY) );
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
}
currentFit++;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
}
}
}
}
}
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
ACTOR Future<Void> updateDatacenterVersionDifference( ClusterControllerData *self ) {
fixed compile issue
2018-06-22 07:34:36 +08:00
state double lastLogTime = 0;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
loop {
self->versionDifferenceUpdated = false;
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
if(self->db.serverInfo->get().read().recoveryState >= RecoveryState::ACCEPTING_COMMITS && self->db.config.usableRegions == 1) {
fix: treat the local region as best when version difference is too large re-check requests when the version difference becomes small
2018-07-07 05:44:11 +08:00
bool oldDifferenceTooLarge = !self->versionDifferenceUpdated || self->datacenterVersionDifference >= SERVER_KNOBS->MAX_VERSION_DIFFERENCE;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
self->versionDifferenceUpdated = true;
self->datacenterVersionDifference = 0;
fix: treat the local region as best when version difference is too large re-check requests when the version difference becomes small
2018-07-07 05:44:11 +08:00
if(oldDifferenceTooLarge) {
checkOutstandingRequests(self);
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(self->db.serverInfo->onChange());
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
continue;
}
state Optional<TLogInterface> primaryLog;
state Optional<TLogInterface> remoteLog;
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
if(self->db.serverInfo->get().read().recoveryState >= RecoveryState::ALL_LOGS_RECRUITED) {
for(auto& logSet : self->db.serverInfo->get().read().logSystemConfig.tLogs) {
removed hasBestPolicy
2018-06-16 03:36:19 +08:00
if(logSet.isLocal && logSet.locality != tagLocalitySatellite) {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
for(auto& tLog : logSet.tLogs) {
if(tLog.present()) {
primaryLog = tLog.interf();
break;
}
}
}
if(!logSet.isLocal) {
for(auto& tLog : logSet.tLogs) {
if(tLog.present()) {
remoteLog = tLog.interf();
break;
}
}
}
}
}
if(!primaryLog.present() || !remoteLog.present()) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(self->db.serverInfo->onChange());
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
continue;
}
added logging for the datacenter version difference
2018-06-22 07:31:52 +08:00
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
state Future<Void> onChange = self->db.serverInfo->onChange();
loop {
fix: getting metrics did not handle broken_promise errors
2018-07-06 03:30:11 +08:00
state Future<TLogQueuingMetricsReply> primaryMetrics = brokenPromiseToNever( primaryLog.get().getQueuingMetrics.getReply( TLogQueuingMetricsRequest() ) );
state Future<TLogQueuingMetricsReply> remoteMetrics = brokenPromiseToNever( remoteLog.get().getQueuingMetrics.getReply( TLogQueuingMetricsRequest() ) );
added logging for the datacenter version difference
2018-06-22 07:31:52 +08:00
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( ( success(primaryMetrics) && success(remoteMetrics) ) || onChange );
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
if(onChange.isReady()) {
break;
}
fix: do not report datacenter version difference before both datacenters report a correct version
2020-05-11 08:49:09 +08:00
if(primaryMetrics.get().v > 0 && remoteMetrics.get().v > 0) {
bool oldDifferenceTooLarge = !self->versionDifferenceUpdated || self->datacenterVersionDifference >= SERVER_KNOBS->MAX_VERSION_DIFFERENCE;
self->versionDifferenceUpdated = true;
self->datacenterVersionDifference = primaryMetrics.get().v - remoteMetrics.get().v;
if(oldDifferenceTooLarge && self->datacenterVersionDifference < SERVER_KNOBS->MAX_VERSION_DIFFERENCE) {
checkOutstandingRequests(self);
}
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
fix: do not report datacenter version difference before both datacenters report a correct version
2020-05-11 08:49:09 +08:00
if(now() - lastLogTime > SERVER_KNOBS->CLUSTER_CONTROLLER_LOGGING_DELAY) {
lastLogTime = now();
TraceEvent("DatacenterVersionDifference", self->id).detail("Difference", self->datacenterVersionDifference);
}
added logging for the datacenter version difference
2018-06-22 07:31:52 +08:00
}
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait( delay(SERVER_KNOBS->VERSION_LAG_METRIC_INTERVAL) || onChange );
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
if(onChange.isReady()) {
break;
}
}
}
}
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
ACTOR Future<Void> doEmptyCommit(Database cx) {
state Transaction tr(cx);
loop {
try {
tr.setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
tr.setOption(FDBTransactionOptions::LOCK_AWARE);
tr.makeSelfConflicting();
wait(tr.commit());
return Void();
} catch( Error &e ) {
wait( tr.onError(e) );
}
}
}
ACTOR Future<Void> handleForcedRecoveries( ClusterControllerData *self, ClusterControllerFullInterface interf ) {
loop {
state ForceRecoveryRequest req = waitNext( interf.clientInterface.forceRecovery.getFuture() );
Remove trace-calls to printable (in non-workloads)
2019-03-19 06:03:43 +08:00
TraceEvent("ForcedRecoveryStart", self->id).detail("ClusterControllerDcId", self->clusterControllerDcId).detail("DcId", req.dcId.printable());
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
state Future<Void> fCommit = doEmptyCommit(self->cx);
knobify force recovery recovery check delay
2019-02-20 08:05:20 +08:00
wait(fCommit || delay(SERVER_KNOBS->FORCE_RECOVERY_CHECK_DELAY));
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
if(!fCommit.isReady() || fCommit.isError()) {
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
if (self->clusterControllerDcId != req.dcId) {
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
vector<Optional<Key>> dcPriority;
dcPriority.push_back(req.dcId);
dcPriority.push_back(self->clusterControllerDcId);
self->desiredDcIds.set(dcPriority);
} else {
self->db.forceRecovery = true;
self->db.forceMasterFailure.trigger();
}
wait(fCommit);
}
TraceEvent("ForcedRecoveryFinish", self->id);
self->db.forceRecovery = false;
req.reply.send(Void());
forgot to fix merge conflicts
2019-02-19 09:09:55 +08:00
}
}
Add a new DataDistributor role. Let cluster controller to start a new data distributor role by sending a message to a chosen worker. Change MasterInterface usage in DataDistribution to masterId Add DataDistributor rejoin handling. This allows the data distributor to tell the new cluster controller of its existence so that the controller doesn't spawn a new one. I.e., there should be only ONE data distributor in the cluster. If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries to recruit one as DD. CC also monitors DD and restarts one if it failed. The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for the new DD. Add GetRecoveryInfo RPC to master server, which is called by data distributor to obtain the recovery Transaction version from the master server.
2018-12-14 05:31:37 +08:00
ACTOR Future<DataDistributorInterface> startDataDistributor( ClusterControllerData *self ) {
Fix a race of DD, RK, Master failure If all DD, RK, Master run on the same process and failed. Recruiting of new DD or RK could try to use the old master worker interface, which is an invalid one and causes recruitment to be stuck. Fix by adding a delay and checking master is valid before recruitment.
2019-03-21 07:03:36 +08:00
wait(delay(0.0)); // If master fails at the same time, give it a chance to clear master PID.
Add a new DataDistributor role. Let cluster controller to start a new data distributor role by sending a message to a chosen worker. Change MasterInterface usage in DataDistribution to masterId Add DataDistributor rejoin handling. This allows the data distributor to tell the new cluster controller of its existence so that the controller doesn't spawn a new one. I.e., there should be only ONE data distributor in the cluster. If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries to recruit one as DD. CC also monitors DD and restarts one if it failed. The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for the new DD. Add GetRecoveryInfo RPC to master server, which is called by data distributor to obtain the recovery Transaction version from the master server.
2018-12-14 05:31:37 +08:00
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCStartDataDistributor", self->id);
Update fdbserver/ClusterController.actor.cpp Co-Authored-By: jzhou77 <jingyuzhou@gmail.com>
2019-02-14 03:54:35 +08:00
loop {
Fix segfault and no_more_servers errors
2019-01-31 01:05:12 +08:00
try {
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
state bool no_distributor = !self->db.serverInfo->get().read().distributor.present();
while (!self->masterProcessId.present() || self->masterProcessId != self->db.serverInfo->get().read().master.locality.processId() || self->db.serverInfo->get().read().recoveryState < RecoveryState::ACCEPTING_COMMITS) {
Fix a race of DD, RK, Master failure If all DD, RK, Master run on the same process and failed. Recruiting of new DD or RK could try to use the old master worker interface, which is an invalid one and causes recruitment to be stuck. Fix by adding a delay and checking master is valid before recruitment.
2019-03-21 07:03:36 +08:00
wait(self->db.serverInfo->onChange() || delay(SERVER_KNOBS->WAIT_FOR_GOOD_RECRUITMENT_DELAY));
Fix segfault and no_more_servers errors
2019-01-31 01:05:12 +08:00
}
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
if (no_distributor && self->db.serverInfo->get().read().distributor.present()) {
return self->db.serverInfo->get().read().distributor.get();
Fix segfault and no_more_servers errors
2019-01-31 01:05:12 +08:00
}
Per review comments Add a knob for starting distributor delay. Move distributor failed variable to a local loop.
2019-02-01 02:51:25 +08:00
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
std::map<Optional<Standalone<StringRef>>, int> id_used = self->getUsedIds();
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
WorkerFitnessInfo data_distributor = self->getWorkerForRoleInDatacenter(self->clusterControllerDcId, ProcessClass::DataDistributor, ProcessClass::NeverAssign, self->db.config, id_used);
state WorkerDetails worker = data_distributor.worker;
if (self->onMasterIsBetter(worker, ProcessClass::DataDistributor)) {
worker = self->id_worker[self->masterProcessId.get()].details;
}
once we recruit a ratekeeper, do not allow any other ratekeepers to register
2019-03-25 02:04:39 +08:00
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
InitializeDataDistributorRequest req(deterministicRandom()->randomUniqueID());
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCDataDistributorRecruit", self->id).detail("Addr", worker.interf.address());
Fix segfault and no_more_servers errors
2019-01-31 01:05:12 +08:00
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
ErrorOr<DataDistributorInterface> distributor = wait( worker.interf.dataDistributor.getReplyUnlessFailedFor(req, SERVER_KNOBS->WAIT_FOR_DISTRIBUTOR_JOIN_DELAY, 0) );
Fix segfault and no_more_servers errors
2019-01-31 01:05:12 +08:00
if (distributor.present()) {
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCDataDistributorRecruited", self->id).detail("Addr", worker.interf.address());
Fix segfault and no_more_servers errors
2019-01-31 01:05:12 +08:00
return distributor.get();
}
}
catch (Error& e) {
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCDataDistributorRecruitError", self->id).error(e);
Fix segfault and no_more_servers errors
2019-01-31 01:05:12 +08:00
if ( e.code() != error_code_no_more_servers ) {
throw;
}
}
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
wait( lowPriorityDelay(SERVER_KNOBS->ATTEMPT_RECRUITMENT_DELAY) );
Add a new DataDistributor role. Let cluster controller to start a new data distributor role by sending a message to a chosen worker. Change MasterInterface usage in DataDistribution to masterId Add DataDistributor rejoin handling. This allows the data distributor to tell the new cluster controller of its existence so that the controller doesn't spawn a new one. I.e., there should be only ONE data distributor in the cluster. If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries to recruit one as DD. CC also monitors DD and restarts one if it failed. The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for the new DD. Add GetRecoveryInfo RPC to master server, which is called by data distributor to obtain the recovery Transaction version from the master server.
2018-12-14 05:31:37 +08:00
}
}
Fix a segfault bug due to uncopied ratekeeper interface
2019-02-19 06:57:21 +08:00
ACTOR Future<Void> monitorDataDistributor(ClusterControllerData *self) {
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
while(self->db.serverInfo->get().read().recoveryState < RecoveryState::ACCEPTING_COMMITS) {
ratekeeper needs to be recruited for proxies to make progress, so if one has not registered with the cluster controller by the time we are accepting commits, recruit a new one
2019-03-25 07:48:24 +08:00
wait(self->db.serverInfo->onChange());
Add a new DataDistributor role. Let cluster controller to start a new data distributor role by sending a message to a chosen worker. Change MasterInterface usage in DataDistribution to masterId Add DataDistributor rejoin handling. This allows the data distributor to tell the new cluster controller of its existence so that the controller doesn't spawn a new one. I.e., there should be only ONE data distributor in the cluster. If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries to recruit one as DD. CC also monitors DD and restarts one if it failed. The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for the new DD. Add GetRecoveryInfo RPC to master server, which is called by data distributor to obtain the recovery Transaction version from the master server.
2018-12-14 05:31:37 +08:00
}
Per review comments Add a knob for starting distributor delay. Move distributor failed variable to a local loop.
2019-02-01 02:51:25 +08:00
loop {
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
if ( self->db.serverInfo->get().read().distributor.present() ) {
wait( waitFailureClient( self->db.serverInfo->get().read().distributor.get().waitFailure, SERVER_KNOBS->DD_FAILURE_TIME ) );
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCDataDistributorDied", self->id)
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
.detail("DistributorId", self->db.serverInfo->get().read().distributor.get().id());
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
self->db.clearInterf(ProcessClass::DataDistributorClass);
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
} else {
Fix the issue of slow dying Data Distributor Test with: -r simulation -f ./foundationdb/tests/slow/CommitBug.txt -s 67828576 -b on The test has the following event sequence: - Time 113.3s, CC noticed DD failure, cleard DD interface. - 1s later, DD rejoined and registered with CC. - Time 131.7s, DD actor cancelled. This old DD raced to register with CC and the failure monitor is not installed because monitorDataDistributor is stalled waiting for new DD. - Time 161.4s, new DD running. New DD recruting was delayed due to no servers in the period. Fix by disabling DD registration during the recruting process.
2019-03-18 11:55:59 +08:00
self->recruitingDistributor = true;
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
DataDistributorInterface distributorInterf = wait( startDataDistributor(self) );
Fix the issue of slow dying Data Distributor Test with: -r simulation -f ./foundationdb/tests/slow/CommitBug.txt -s 67828576 -b on The test has the following event sequence: - Time 113.3s, CC noticed DD failure, cleard DD interface. - 1s later, DD rejoined and registered with CC. - Time 131.7s, DD actor cancelled. This old DD raced to register with CC and the failure monitor is not installed because monitorDataDistributor is stalled waiting for new DD. - Time 161.4s, new DD running. New DD recruting was delayed due to no servers in the period. Fix by disabling DD registration during the recruting process.
2019-03-18 11:55:59 +08:00
self->recruitingDistributor = false;
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
self->db.setDistributor(distributorInterf);
}
}
}
Fix ratekeeper recruitment bug
2019-03-21 04:54:15 +08:00
ACTOR Future<Void> startRatekeeper(ClusterControllerData *self) {
Fix a race of DD, RK, Master failure If all DD, RK, Master run on the same process and failed. Recruiting of new DD or RK could try to use the old master worker interface, which is an invalid one and causes recruitment to be stuck. Fix by adding a delay and checking master is valid before recruitment.
2019-03-21 07:03:36 +08:00
wait(delay(0.0)); // If master fails at the same time, give it a chance to clear master PID.
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCStartRatekeeper", self->id);
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
loop {
try {
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
state bool no_ratekeeper = !self->db.serverInfo->get().read().ratekeeper.present();
while (!self->masterProcessId.present() || self->masterProcessId != self->db.serverInfo->get().read().master.locality.processId() || self->db.serverInfo->get().read().recoveryState < RecoveryState::ACCEPTING_COMMITS) {
Fix a race of DD, RK, Master failure If all DD, RK, Master run on the same process and failed. Recruiting of new DD or RK could try to use the old master worker interface, which is an invalid one and causes recruitment to be stuck. Fix by adding a delay and checking master is valid before recruitment.
2019-03-21 07:03:36 +08:00
wait(self->db.serverInfo->onChange() || delay(SERVER_KNOBS->WAIT_FOR_GOOD_RECRUITMENT_DELAY));
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
}
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
if (no_ratekeeper && self->db.serverInfo->get().read().ratekeeper.present()) {
Avoid unnecessary recuriting of DD or RK While waiting for recruting data distributor or ratekeeper, a previous one could already joined. So we can skip this unnecessary recruiting. Revert the change of worker.actor.cpp for ratekeeper. Instead, recruiting ratekeeper should avoid the process with an existing one. This fixes a bug where the ratekeeper interface became zombie, killing other healthy ratekeeper but doing no useful work. Found by: -r simulation --crash -f tests/fast/WriteDuringRead.txt -s 31858110 -b on
2019-03-22 13:20:00 +08:00
// Existing ratekeeper registers while waiting, so skip.
return Void();
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
}
std::map<Optional<Standalone<StringRef>>, int> id_used = self->getUsedIds();
Changes to ratekeeper camel case
2019-03-27 23:24:25 +08:00
WorkerFitnessInfo rkWorker = self->getWorkerForRoleInDatacenter(self->clusterControllerDcId, ProcessClass::Ratekeeper, ProcessClass::NeverAssign, self->db.config, id_used);
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
InitializeRatekeeperRequest req(deterministicRandom()->randomUniqueID());
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
state WorkerDetails worker = rkWorker.worker;
Changes to ratekeeper camel case
2019-03-27 23:24:25 +08:00
if (self->onMasterIsBetter(worker, ProcessClass::Ratekeeper)) {
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
worker = self->id_worker[self->masterProcessId.get()].details;
}
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
self->recruitingRatekeeperID = req.reqId;
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCRecruitRatekeeper", self->id).detail("Addr", worker.interf.address()).detail("RKID", req.reqId);
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
ErrorOr<RatekeeperInterface> interf = wait( worker.interf.ratekeeper.getReplyUnlessFailedFor(req, SERVER_KNOBS->WAIT_FOR_RATEKEEPER_JOIN_DELAY, 0) );
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
if (interf.present()) {
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
self->recruitRatekeeper.set(false);
once we recruit a ratekeeper, do not allow any other ratekeepers to register
2019-03-25 02:04:39 +08:00
self->recruitingRatekeeperID = interf.get().id();
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
const auto& ratekeeper = self->db.serverInfo->get().read().ratekeeper;
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCRatekeeperRecruited", self->id).detail("Addr", worker.interf.address()).detail("RKID", interf.get().id());
ensure that the processId exists in id_worker before accessing it
2019-03-23 09:54:39 +08:00
if (ratekeeper.present() && ratekeeper.get().id() != interf.get().id() && self->id_worker.count(ratekeeper.get().locality.processId())) {
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCHaltRatekeeperAfterRecruit", self->id).detail("RKID", ratekeeper.get().id())
startRatekeeper is responsible for updating serverDBInfo
2019-03-23 08:56:16 +08:00
.detail("DcID", printable(self->clusterControllerDcId));
Add actors to store halt request futures Address best fitness in checking better DD or RK.
2019-03-23 09:00:16 +08:00
self->id_worker[ratekeeper.get().locality.processId()].haltRatekeeper = brokenPromiseToNever(ratekeeper.get().haltRatekeeper.getReply(HaltRatekeeperRequest(self->id)));
startRatekeeper is responsible for updating serverDBInfo
2019-03-23 08:56:16 +08:00
}
once we recruit a ratekeeper, do not allow any other ratekeepers to register
2019-03-25 02:04:39 +08:00
if(!ratekeeper.present() || ratekeeper.get().id() != interf.get().id()) {
startRatekeeper is responsible for updating serverDBInfo
2019-03-23 08:56:16 +08:00
self->db.setRatekeeper(interf.get());
}
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
checkOutstandingRequests(self);
Fix ratekeeper recruitment bug
2019-03-21 04:54:15 +08:00
return Void();
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
}
}
catch (Error& e) {
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCRatekeeperRecruitError", self->id).error(e);
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
if ( e.code() != error_code_no_more_servers ) {
throw;
}
}
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
wait( lowPriorityDelay(SERVER_KNOBS->ATTEMPT_RECRUITMENT_DELAY) );
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
}
}
Minor fix on ratekeeper work registration.
2019-02-16 09:29:52 +08:00
ACTOR Future<Void> monitorRatekeeper(ClusterControllerData *self) {
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
while(self->db.serverInfo->get().read().recoveryState < RecoveryState::ACCEPTING_COMMITS) {
ratekeeper needs to be recruited for proxies to make progress, so if one has not registered with the cluster controller by the time we are accepting commits, recruit a new one
2019-03-25 07:48:24 +08:00
wait(self->db.serverInfo->onChange());
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
}
loop {
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
if ( self->db.serverInfo->get().read().ratekeeper.present() && !self->recruitRatekeeper.get() ) {
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
choose {
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
when(wait(waitFailureClient( self->db.serverInfo->get().read().ratekeeper.get().waitFailure, SERVER_KNOBS->RATEKEEPER_FAILURE_TIME ))) {
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCRatekeeperDied", self->id)
the cluster controller caches the serialization of serverDBInfo, to avoid regenerating it many times
2019-09-11 05:27:22 +08:00
.detail("RKID", self->db.serverInfo->get().read().ratekeeper.get().id());
Changes to ratekeeper camel case
2019-03-27 23:24:25 +08:00
self->db.clearInterf(ProcessClass::RatekeeperClass);
Fix ratekeeper and data distributor recruiting bug Avoid multiple concurrent recuriting of ratekeepers with a recruiting flag. Fix endless recruiting when the chosen worker is a proxy or a resolver -- prefer master in this case.
2019-03-21 01:00:31 +08:00
}
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
when(wait(self->recruitRatekeeper.onChange())) {}
Recruit a new ratekeeper before halting the old
2019-03-20 06:21:46 +08:00
}
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
} else {
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
wait( startRatekeeper(self) );
Add a new DataDistributor role. Let cluster controller to start a new data distributor role by sending a message to a chosen worker. Change MasterInterface usage in DataDistribution to masterId Add DataDistributor rejoin handling. This allows the data distributor to tell the new cluster controller of its existence so that the controller doesn't spawn a new one. I.e., there should be only ONE data distributor in the cluster. If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries to recruit one as DD. CC also monitors DD and restarts one if it failed. The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for the new DD. Add GetRecoveryInfo RPC to master server, which is called by data distributor to obtain the recovery Transaction version from the master server.
2018-12-14 05:31:37 +08:00
}
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
}
}
fix: the cluster controller did not pass in its own locality when creating its database object, therefore it was not using locality aware load balancing
2018-09-29 03:12:06 +08:00
ACTOR Future<Void> clusterControllerCore( ClusterControllerFullInterface interf, Future<Void> leaderFail, ServerCoordinators coordinators, LocalityData locality ) {
state ClusterControllerData self( interf, locality );
Initial repository commit
2017-05-26 04:48:44 +08:00
state Future<Void> coordinationPingDelay = delay( SERVER_KNOBS->WORKER_COORDINATION_PING_DELAY );
state uint64_t step = 0;
Add DataDistributorInterface to ServerDBInfo Also change the Proxy and QuietDatabase to use the DataDistributorInterface.
2019-01-19 03:30:18 +08:00
state Future<ErrorOr<Void>> error = errorOr( actorCollection( self.addActor.getFuture() ) );
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
self.addActor.send( failureDetectionServer( self.id, &self, interf.clientInterface.failureMonitoring.getFuture() ) );
Add DataDistributorInterface to ServerDBInfo Also change the Proxy and QuietDatabase to use the DataDistributorInterface.
2019-01-19 03:30:18 +08:00
self.addActor.send( clusterWatchDatabase( &self, &self.db ) ); // Start the master database
self.addActor.send( self.updateWorkerList.init( self.db.db ) );
self.addActor.send( statusServer( interf.clientInterface.databaseStatus.getFuture(), &self, coordinators));
self.addActor.send( timeKeeper(&self) );
self.addActor.send( monitorProcessClasses(&self) );
Add a monitorServerInfoConfig() call back This was deleted during a code refactor in ef868f5. Because no tests were complaining, we didn't find this until now.
2019-07-26 06:17:26 +08:00
self.addActor.send( monitorServerInfoConfig(&self.db) );
Add DataDistributorInterface to ServerDBInfo Also change the Proxy and QuietDatabase to use the DataDistributorInterface.
2019-01-19 03:30:18 +08:00
self.addActor.send( monitorClientTxnInfoConfigs(&self.db) );
self.addActor.send( updatedChangingDatacenters(&self) );
self.addActor.send( updatedChangedDatacenters(&self) );
self.addActor.send( updateDatacenterVersionDifference(&self) );
forgot to fix merge conflicts
2019-02-19 09:09:55 +08:00
self.addActor.send( handleForcedRecoveries(&self, interf) );
Fix a segfault bug due to uncopied ratekeeper interface
2019-02-19 06:57:21 +08:00
self.addActor.send( monitorDataDistributor(&self) );
Minor fix on ratekeeper work registration.
2019-02-16 09:29:52 +08:00
self.addActor.send( monitorRatekeeper(&self) );
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
self.addActor.send( traceCounters("ClusterControllerMetrics", self.id, SERVER_KNOBS->STORAGE_LOGGING_DELAY, &self.clusterControllerMetrics, self.id.toString() + "/ClusterControllerMetrics") );
Initial repository commit
2017-05-26 04:48:44 +08:00
//printf("%s: I am the cluster controller\n", g_network->getLocalAddress().toString().c_str());
loop choose {
when( ErrorOr<Void> err = wait( error ) ) {
if (err.isError()) {
Add the roles running on a process as a field on trace events in the form of a comma delimited string of role abbreviations.
2018-09-06 06:06:14 +08:00
endRole(Role::CLUSTER_CONTROLLER, interf.id(), "Stop Received Error", false, err.getError());
Initial repository commit
2017-05-26 04:48:44 +08:00
}
else {
Add the roles running on a process as a field on trace events in the form of a comma delimited string of role abbreviations.
2018-09-06 06:06:14 +08:00
endRole(Role::CLUSTER_CONTROLLER, interf.id(), "Stop Received Signal", true);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
// We shut down normally even if there was a serious error (so this fdbserver may be re-elected cluster controller)
return Void();
}
when( OpenDatabaseRequest req = waitNext( interf.clientInterface.openDatabase.getFuture() ) ) {
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
++self.openDatabaseRequests;
added status information on connected clients
2019-07-26 08:15:31 +08:00
self.addActor.send(clusterOpenDatabase(&self.db, req));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
when( RecruitFromConfigurationRequest req = waitNext( interf.recruitFromConfiguration.getFuture() ) ) {
Add DataDistributorInterface to ServerDBInfo Also change the Proxy and QuietDatabase to use the DataDistributorInterface.
2019-01-19 03:30:18 +08:00
self.addActor.send( clusterRecruitFromConfiguration( &self, req ) );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
when( RecruitRemoteFromConfigurationRequest req = waitNext( interf.recruitRemoteFromConfiguration.getFuture() ) ) {
Add DataDistributorInterface to ServerDBInfo Also change the Proxy and QuietDatabase to use the DataDistributorInterface.
2019-01-19 03:30:18 +08:00
self.addActor.send( clusterRecruitRemoteFromConfiguration( &self, req ) );
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
when( RecruitStorageRequest req = waitNext( interf.recruitStorage.getFuture() ) ) {
clusterRecruitStorage( &self, req );
}
when( RegisterWorkerRequest req = waitNext( interf.registerWorker.getFuture() ) ) {
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
++self.registerWorkerRequests;
Initial repository commit
2017-05-26 04:48:44 +08:00
registerWorker( req, &self );
}
when( GetWorkersRequest req = waitNext( interf.getWorkers.getFuture() ) ) {
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
++self.getWorkersRequests;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
vector<WorkerDetails> workers;
Enable checkUsingDesiredClasses() in consistency check
2017-10-25 03:58:54 +08:00
for(auto& it : self.id_worker) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if ( (req.flags & GetWorkersRequest::NON_EXCLUDED_PROCESSES_ONLY) && self.db.config.isExcludedServer(it.second.details.interf.address()) ) {
Enable checkUsingDesiredClasses() in consistency check
2017-10-25 03:58:54 +08:00
continue;
}
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if ( (req.flags & GetWorkersRequest::TESTER_CLASS_ONLY) && it.second.details.processClass.classType() != ProcessClass::TesterClass ) {
Enable checkUsingDesiredClasses() in consistency check
2017-10-25 03:58:54 +08:00
continue;
}
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
workers.push_back(it.second.details);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Enable checkUsingDesiredClasses() in consistency check
2017-10-25 03:58:54 +08:00
req.reply.send( workers );
Initial repository commit
2017-05-26 04:48:44 +08:00
}
when( GetClientWorkersRequest req = waitNext( interf.clientInterface.getClientWorkers.getFuture() ) ) {
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
++self.getClientWorkersRequests;
Initial repository commit
2017-05-26 04:48:44 +08:00
vector<ClientWorkerInterface> workers;
for(auto& it : self.id_worker) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if (it.second.details.processClass.classType() != ProcessClass::TesterClass) {
workers.push_back(it.second.details.interf.clientInterface);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
req.reply.send(workers);
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when( wait( coordinationPingDelay ) ) {
Initial repository commit
2017-05-26 04:48:44 +08:00
CoordinationPingMessage message(self.id, step++);
for(auto& it : self.id_worker)
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
it.second.details.interf.coordinationPing.send(message);
Initial repository commit
2017-05-26 04:48:44 +08:00
coordinationPingDelay = delay( SERVER_KNOBS->WORKER_COORDINATION_PING_DELAY );
TraceEvent("CoordinationPingSent", self.id).detail("TimeStep", message.timeStep);
}
when( RegisterMasterRequest req = waitNext( interf.registerMaster.getFuture() ) ) {
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
++self.registerMasterRequests;
Initial repository commit
2017-05-26 04:48:44 +08:00
clusterRegisterMaster( &self, req );
}
when( GetServerDBInfoRequest req = waitNext( interf.getServerDBInfo.getFuture() ) ) {
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
++self.getServerDBInfoRequests;
Add support for a client or worker having multiple issues.
2019-03-22 23:29:41 +08:00
self.addActor.send(
clusterGetServerInfo(&self.db, req.knownServerInfoID, req.issues, req.incompatiblePeers, req.reply));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when( wait( leaderFail ) ) {
Initial repository commit
2017-05-26 04:48:44 +08:00
// We are no longer the leader if this has changed.
Add the roles running on a process as a field on trace events in the form of a comma delimited string of role abbreviations.
2018-09-06 06:06:14 +08:00
endRole(Role::CLUSTER_CONTROLLER, interf.id(), "Leader Replaced", true);
Initial repository commit
2017-05-26 04:48:44 +08:00
TEST(true); // Lost Cluster Controller Role
return Void();
}
when( ReplyPromise<Void> ping = waitNext( interf.clientInterface.ping.getFuture() ) ) {
ping.send( Void() );
}
}
}
If a process receives a message in its clusterControllerInterface before becoming the cluster controller, if the process does not become the cluster controller in the next minute it should destroy the interface to prevent a memory leak.
2019-05-30 07:57:13 +08:00
ACTOR Future<Void> replaceInterface( ClusterControllerFullInterface interf ) {
loop {
if( interf.hasMessage() ) {
wait(delay(SERVER_KNOBS->REPLACE_INTERFACE_DELAY));
return Void();
}
wait(delay(SERVER_KNOBS->REPLACE_INTERFACE_CHECK_DELAY));
}
}
fix: the cluster controller did not pass in its own locality when creating its database object, therefore it was not using locality aware load balancing
2018-09-29 03:12:06 +08:00
ACTOR Future<Void> clusterController( ServerCoordinators coordinators, Reference<AsyncVar<Optional<ClusterControllerFullInterface>>> currentCC, bool hasConnected, Reference<AsyncVar<ClusterControllerPriorityInfo>> asyncPriorityInfo, LocalityData locality ) {
Initial repository commit
2017-05-26 04:48:44 +08:00
loop {
state ClusterControllerFullInterface cci;
state bool inRole = false;
cci.initEndpoints();
try {
//Register as a possible leader; wait to be elected
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
state Future<Void> leaderFail = tryBecomeLeader( coordinators, cci, currentCC, hasConnected, asyncPriorityInfo );
If a process receives a message in its clusterControllerInterface before becoming the cluster controller, if the process does not become the cluster controller in the next minute it should destroy the interface to prevent a memory leak.
2019-05-30 07:57:13 +08:00
state Future<Void> shouldReplace = replaceInterface( cci );
Initial repository commit
2017-05-26 04:48:44 +08:00
while (!currentCC->get().present() || currentCC->get().get() != cci) {
choose {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when( wait(currentCC->onChange()) ) {}
when( wait(leaderFail) ) { ASSERT(false); throw internal_error(); }
If a process receives a message in its clusterControllerInterface before becoming the cluster controller, if the process does not become the cluster controller in the next minute it should destroy the interface to prevent a memory leak.
2019-05-30 07:57:13 +08:00
when( wait(shouldReplace) ) { break; }
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
If a process receives a message in its clusterControllerInterface before becoming the cluster controller, if the process does not become the cluster controller in the next minute it should destroy the interface to prevent a memory leak.
2019-05-30 07:57:13 +08:00
if(!shouldReplace.isReady()) {
shouldReplace = Future<Void>();
hasConnected = true;
startRole(Role::CLUSTER_CONTROLLER, cci.id(), UID());
inRole = true;
Initial repository commit
2017-05-26 04:48:44 +08:00
If a process receives a message in its clusterControllerInterface before becoming the cluster controller, if the process does not become the cluster controller in the next minute it should destroy the interface to prevent a memory leak.
2019-05-30 07:57:13 +08:00
wait( clusterControllerCore( cci, leaderFail, coordinators, locality ) );
}
Initial repository commit
2017-05-26 04:48:44 +08:00
} catch(Error& e) {
if (inRole)
Add the roles running on a process as a field on trace events in the form of a comma delimited string of role abbreviations.
2018-09-06 06:06:14 +08:00
endRole(Role::CLUSTER_CONTROLLER, cci.id(), "Error", e.code() == error_code_actor_cancelled || e.code() == error_code_coordinators_changed, e);
Initial repository commit
2017-05-26 04:48:44 +08:00
else
TraceEvent( e.code() == error_code_coordinators_changed ? SevInfo : SevError, "ClusterControllerCandidateError", cci.id()).error(e);
throw;
}
}
}
fix: the cluster controller did not pass in its own locality when creating its database object, therefore it was not using locality aware load balancing
2018-09-29 03:12:06 +08:00
ACTOR Future<Void> clusterController( Reference<ClusterConnectionFile> connFile, Reference<AsyncVar<Optional<ClusterControllerFullInterface>>> currentCC, Reference<AsyncVar<ClusterControllerPriorityInfo>> asyncPriorityInfo, Future<Void> recoveredDiskFiles, LocalityData locality ) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(recoveredDiskFiles);
Initial repository commit
2017-05-26 04:48:44 +08:00
state bool hasConnected = false;
loop {
try {
ServerCoordinators coordinators( connFile );
Merge branch 'release-6.0' # Conflicts: # documentation/sphinx/source/release-notes.rst # fdbclient/ManagementAPI.actor.cpp # fdbserver/ClusterController.actor.cpp # fdbserver/DataDistribution.actor.cpp # fdbserver/LogSystemDiskQueueAdapter.actor.cpp # fdbserver/SimulatedCluster.actor.cpp # fdbserver/TLogServer.actor.cpp
2018-10-04 07:57:18 +08:00
wait( clusterController( coordinators, currentCC, hasConnected, asyncPriorityInfo, locality ) );
Initial repository commit
2017-05-26 04:48:44 +08:00
} catch( Error &e ) {
if( e.code() != error_code_coordinators_changed )
throw; // Expected to terminate fdbserver
}
hasConnected = true;
}
Make RecoveryState an enum class. This means that all the == 7 or != 0 checks go away, and explicit names must be used.
2018-05-09 08:17:17 +08:00
}