wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
foundationdb/fdbserver/ClusterController.actor.cpp

3514 lines
140 KiB
C++
Raw Normal View History

Initial repository commit
2017-05-26 04:48:44 +08:00
/*
* ClusterController.actor.cpp
*
* This source file is part of the FoundationDB open source project
*
Minor fix on ratekeeper work registration.
2019-02-16 09:29:52 +08:00
* Copyright 2013-2019 Apple Inc. and the FoundationDB project authors
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* http://www.apache.org/licenses/LICENSE-2.0
remove trailing whitespace from our copyright headers ; fixed formatting of python setup.py
2018-02-22 02:25:11 +08:00
*
Initial repository commit
2017-05-26 04:48:44 +08:00
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
Recruit backup worker during master recovery Right now recruit the same number as TLogs. The backup worker does nothing.
2019-05-16 07:13:04 +08:00
#include <algorithm>
#include <iterator>
#include <map>
#include <set>
#include <vector>
Initial repository commit
2017-05-26 04:48:44 +08:00
#include "fdbrpc/FailureMonitor.h"
#include "flow/ActorCollection.h"
Use proper fwd decl in NativeAPI Also NativeAPI.h -> NativeAPI.actor.h
2019-02-18 07:41:16 +08:00
#include "fdbclient/NativeAPI.actor.h"
Add backup role, class, and worker skeleton
2019-04-25 06:12:37 +08:00
#include "fdbserver/BackupInterface.h"
Initial repository commit
2017-05-26 04:48:44 +08:00
#include "fdbserver/CoordinationInterface.h"
Add a new DataDistributor role. Let cluster controller to start a new data distributor role by sending a message to a chosen worker. Change MasterInterface usage in DataDistribution to masterId Add DataDistributor rejoin handling. This allows the data distributor to tell the new cluster controller of its existence so that the controller doesn't spawn a new one. I.e., there should be only ONE data distributor in the cluster. If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries to recruit one as DD. CC also monitors DD and restarts one if it failed. The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for the new DD. Add GetRecoveryInfo RPC to master server, which is called by data distributor to obtain the recovery Transaction version from the master server.
2018-12-14 05:31:37 +08:00
#include "fdbserver/DataDistributorInterface.h"
Adjust all includes to be relative to the root. Remove the use of relative paths. A header at foo/bar.h could be included by files under foo/ with "bar.h", but would be included everywhere else as "foo/bar.h". Adjust so that every include references such a header with the latter form. Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-20 01:30:13 +08:00
#include "fdbserver/Knobs.h"
Use ACTOR forward declarations in MoveKeys Also MoveKeys.h -> MoveKeys.actor.h
2019-02-18 10:55:52 +08:00
#include "fdbserver/MoveKeys.actor.h"
Use fwd decls in WorkerInterface Also WorkerInterface.h -> WorkerInterface.actor.h
2019-02-18 11:13:26 +08:00
#include "fdbserver/WorkerInterface.actor.h"
Adjust all includes to be relative to the root. Remove the use of relative paths. A header at foo/bar.h could be included by files under foo/ with "bar.h", but would be included everywhere else as "foo/bar.h". Adjust so that every include references such a header with the latter form. Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-20 01:30:13 +08:00
#include "fdbserver/LeaderElection.h"
Add used worker IDs to cluster controller This "usedIds" is updated when receiving a master registration message, so that when recruiting new data distributor, existing assignment is known.
2019-01-29 01:25:15 +08:00
#include "fdbserver/LogSystemConfig.h"
Adjust all includes to be relative to the root. Remove the use of relative paths. A header at foo/bar.h could be included by files under foo/ with "bar.h", but would be included everywhere else as "foo/bar.h". Adjust so that every include references such a header with the latter form. Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-20 01:30:13 +08:00
#include "fdbserver/WaitFailure.h"
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
#include "fdbserver/RatekeeperInterface.h"
Adjust all includes to be relative to the root. Remove the use of relative paths. A header at foo/bar.h could be included by files under foo/ with "bar.h", but would be included everywhere else as "foo/bar.h". Adjust so that every include references such a header with the latter form. Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-20 01:30:13 +08:00
#include "fdbserver/ServerDBInfo.h"
#include "fdbserver/Status.h"
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
#include "fdbserver/LatencyBandConfig.h"
Initial repository commit
2017-05-26 04:48:44 +08:00
#include "fdbclient/DatabaseContext.h"
Adjust all includes to be relative to the root. Remove the use of relative paths. A header at foo/bar.h could be included by files under foo/ with "bar.h", but would be included everywhere else as "foo/bar.h". Adjust so that every include references such a header with the latter form. Signed-off-by: Robert Escriva <rescriva@dropbox.com>
2018-10-20 01:30:13 +08:00
#include "fdbserver/RecoveryState.h"
Initial repository commit
2017-05-26 04:48:44 +08:00
#include "fdbclient/ReadYourWrites.h"
#include "fdbrpc/Replication.h"
#include "fdbrpc/ReplicationUtils.h"
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
#include "fdbclient/KeyBackedTypes.h"
Stop performing self-moves. (e.g. a = std::move(a)) self-moves are frowned upon in C++, and in our code this generally happens from calls to swap as part of trying to implement a "unordered erase" function via swap-to-the-end-and-pop_back. For convenience, a swapAndPop() function is now offered that performs this, while disallowing self-moves.
2018-08-02 09:09:54 +08:00
#include "flow/Util.h"
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
#include "flow/actorcompiler.h" // This must be the last #include.
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
void failAfter(Future<Void> trigger, Endpoint e);
Initial repository commit
2017-05-26 04:48:44 +08:00
struct WorkerInfo : NonCopyable {
Future<Void> watcher;
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
ReplyPromise<RegisterWorkerReply> reply;
Initial repository commit
2017-05-26 04:48:44 +08:00
Generation gen;
int reboots;
ProcessClass initialClass;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
ClusterControllerPriorityInfo priorityInfo;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
WorkerDetails details;
Add actors to store halt request futures Address best fitness in checking better DD or RK.
2019-03-23 09:00:16 +08:00
Future<Void> haltRatekeeper;
Future<Void> haltDistributor;
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
Standalone<VectorRef<StringRef>> issues;
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
WorkerInfo()
: gen(-1), reboots(0),
priorityInfo(ProcessClass::UnsetFit, false, ClusterControllerPriorityInfo::FitnessUnknown) {}
WorkerInfo(Future<Void> watcher,
ReplyPromise<RegisterWorkerReply> reply,
Generation gen,
WorkerInterface interf,
ProcessClass initialClass,
ProcessClass processClass,
ClusterControllerPriorityInfo priorityInfo,
bool degraded,
Standalone<VectorRef<StringRef>> issues)
: watcher(watcher), reply(reply), gen(gen), reboots(0), initialClass(initialClass), priorityInfo(priorityInfo),
details(interf, processClass, degraded), issues(issues) {}
Initial repository commit
2017-05-26 04:48:44 +08:00
Replace BOOST_NOEXCEPT with noexcept
2020-06-10 08:33:41 +08:00
WorkerInfo(WorkerInfo&& r) noexcept
: watcher(std::move(r.watcher)), reply(std::move(r.reply)), gen(r.gen), reboots(r.reboots),
initialClass(r.initialClass), priorityInfo(r.priorityInfo), details(std::move(r.details)),
haltRatekeeper(r.haltRatekeeper), haltDistributor(r.haltDistributor), issues(r.issues) {}
void operator=(WorkerInfo&& r) noexcept {
Initial repository commit
2017-05-26 04:48:44 +08:00
watcher = std::move(r.watcher);
reply = std::move(r.reply);
gen = r.gen;
reboots = r.reboots;
initialClass = r.initialClass;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
priorityInfo = r.priorityInfo;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
details = std::move(r.details);
Merging with Master.
2019-11-13 05:01:29 +08:00
haltRatekeeper = r.haltRatekeeper;
haltDistributor = r.haltDistributor;
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
issues = r.issues;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
};
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
struct WorkerFitnessInfo {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
WorkerDetails worker;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
ProcessClass::Fitness fitness;
int used;
WorkerFitnessInfo() : fitness(ProcessClass::NeverAssign), used(0) {}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
WorkerFitnessInfo(WorkerDetails worker, ProcessClass::Fitness fitness, int used)
: worker(worker), fitness(fitness), used(used) {}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
};
Initial repository commit
2017-05-26 04:48:44 +08:00
class ClusterControllerData {
public:
struct DBInfo {
Reference<AsyncVar<ClientDBInfo>> clientInfo;
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
Reference<AsyncVar<ServerDBInfo>> serverInfo;
Initial repository commit
2017-05-26 04:48:44 +08:00
std::map<NetworkAddress, double> incompatibleConnections;
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
AsyncTrigger forceMasterFailure;
Initial repository commit
2017-05-26 04:48:44 +08:00
int64_t masterRegistrationCount;
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
int64_t dbInfoCount;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
bool recoveryStalled;
added the (untested) ability to force a recovery to the remote datacenter, even if that results in data loss. If the DR lag is more than 1 week there could be potential data corruption if any primary storage servers are still alive.
2018-07-01 21:39:04 +08:00
bool forceRecovery;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
DatabaseConfiguration config; // Asynchronously updated via master registration
Add fullyRecoveredConfig for checking exclusions
2017-11-16 09:15:24 +08:00
DatabaseConfiguration fullyRecoveredConfig;
Initial repository commit
2017-05-26 04:48:44 +08:00
Database db;
The cluster controller will increase its failure monitoring delay after there have been many unfinishedRecoveries
2018-09-01 01:51:55 +08:00
int unfinishedRecoveries;
int logGenerations;
Merging with Master.
2019-11-13 05:01:29 +08:00
bool cachePopulated;
added status information on connected clients
2019-07-26 08:15:31 +08:00
std::map<NetworkAddress, std::pair<double, OpenDatabaseRequest>> clientStatus;
Initial repository commit
2017-05-26 04:48:44 +08:00
Add uses of makeReference
2020-11-07 15:50:55 +08:00
DBInfo()
: masterRegistrationCount(0), recoveryStalled(false), forceRecovery(false), unfinishedRecoveries(0),
logGenerations(0), cachePopulated(false), clientInfo(new AsyncVar<ClientDBInfo>()), dbInfoCount(0),
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
serverInfo(new AsyncVar<ServerDBInfo>()), db(DatabaseContext::create(clientInfo,
Future<Void>(),
LocalityData(),
true,
TaskPriority::DefaultEndpoint,
true)) // SOMEDAY: Locality!
Add uses of makeReference
2020-11-07 15:50:55 +08:00
{}
Initial repository commit
2017-05-26 04:48:44 +08:00
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
void setDistributor(const DataDistributorInterface& interf) {
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
auto newInfo = serverInfo->get();
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
newInfo.id = deterministicRandom()->randomUniqueID();
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
newInfo.infoGeneration = ++dbInfoCount;
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
newInfo.distributor = interf;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
serverInfo->set(newInfo);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
void setRatekeeper(const RatekeeperInterface& interf) {
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
auto newInfo = serverInfo->get();
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
newInfo.id = deterministicRandom()->randomUniqueID();
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
newInfo.infoGeneration = ++dbInfoCount;
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
newInfo.ratekeeper = interf;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
serverInfo->set(newInfo);
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
}
void clearInterf(ProcessClass::ClassType t) {
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
auto newInfo = serverInfo->get();
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
newInfo.id = deterministicRandom()->randomUniqueID();
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
newInfo.infoGeneration = ++dbInfoCount;
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
if (t == ProcessClass::DataDistributorClass) {
newInfo.distributor = Optional<DataDistributorInterface>();
Changes to ratekeeper camel case
2019-03-27 23:24:25 +08:00
} else if (t == ProcessClass::RatekeeperClass) {
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
newInfo.ratekeeper = Optional<RatekeeperInterface>();
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
serverInfo->set(newInfo);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
};
struct UpdateWorkerList {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
Future<Void> init(Database const& db) { return update(this, db); }
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
void set(Optional<Standalone<StringRef>> processID, Optional<ProcessData> data) {
Initial repository commit
2017-05-26 04:48:44 +08:00
delta[processID] = data;
anyDelta.set(true);
}
private:
std::map<Optional<Standalone<StringRef>>, Optional<ProcessData>> delta;
AsyncVar<bool> anyDelta;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR static Future<Void> update(UpdateWorkerList* self, Database db) {
Initial repository commit
2017-05-26 04:48:44 +08:00
// The Database we are using is based on worker registrations to this cluster controller, which come only
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
// from master servers that we started, so it shouldn't be possible for multiple cluster controllers to
// fight.
Initial repository commit
2017-05-26 04:48:44 +08:00
state Transaction tr(db);
loop {
try {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
tr.clear(workerListKeys);
wait(tr.commit());
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
} catch (Error& e) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
wait(tr.onError(e));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
loop {
// Wait for some changes
while (!self->anyDelta.get())
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
wait(self->anyDelta.onChange());
Initial repository commit
2017-05-26 04:48:44 +08:00
self->anyDelta.set(false);
state std::map<Optional<Standalone<StringRef>>, Optional<ProcessData>> delta;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
delta.swap(self->delta);
Initial repository commit
2017-05-26 04:48:44 +08:00
TraceEvent("UpdateWorkerList").detail("DeltaCount", delta.size());
// Do a transaction to write the changes
loop {
try {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto w = delta.begin(); w != delta.end(); ++w) {
Initial repository commit
2017-05-26 04:48:44 +08:00
if (w->second.present()) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
tr.set(workerListKeyFor(w->first.get()), workerListValue(w->second.get()));
Initial repository commit
2017-05-26 04:48:44 +08:00
} else
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
tr.clear(workerListKeyFor(w->first.get()));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
wait(tr.commit());
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
} catch (Error& e) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
wait(tr.onError(e));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
}
}
};
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
bool workerAvailable(WorkerInfo const& worker, bool checkStable) {
return (now() - startTime < 2 * FLOW_KNOBS->SERVER_REQUEST_INTERVAL) ||
(IFailureMonitor::failureMonitor().getState(worker.details.interf.storage.getEndpoint()).isAvailable() &&
(!checkStable || worker.reboots < 2));
}
bool isLongLivedStateless(Optional<Key> const& processId) {
return (db.serverInfo->get().distributor.present() &&
db.serverInfo->get().distributor.get().locality.processId() == processId) ||
(db.serverInfo->get().ratekeeper.present() &&
db.serverInfo->get().ratekeeper.get().locality.processId() == processId);
}
WorkerDetails getStorageWorker(RecruitStorageRequest const& req) {
std::set<Optional<Standalone<StringRef>>> excludedMachines(req.excludeMachines.begin(),
req.excludeMachines.end());
std::set<Optional<Standalone<StringRef>>> includeDCs(req.includeDCs.begin(), req.includeDCs.end());
std::set<AddressExclusion> excludedAddresses(req.excludeAddresses.begin(), req.excludeAddresses.end());
for (auto& it : id_worker)
if (workerAvailable(it.second, false) &&
!excludedMachines.count(it.second.details.interf.locality.zoneId()) &&
(includeDCs.size() == 0 || includeDCs.count(it.second.details.interf.locality.dcId())) &&
!addressExcluded(excludedAddresses, it.second.details.interf.address()) &&
(!it.second.details.interf.secondaryAddress().present() ||
!addressExcluded(excludedAddresses, it.second.details.interf.secondaryAddress().get())) &&
it.second.details.processClass.machineClassFitness(ProcessClass::Storage) <= ProcessClass::UnsetFit) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
return it.second.details;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (req.criticalRecruitment) {
Initial repository commit
2017-05-26 04:48:44 +08:00
ProcessClass::Fitness bestFit = ProcessClass::NeverAssign;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
Optional<WorkerDetails> bestInfo;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& it : id_worker) {
ProcessClass::Fitness fit = it.second.details.processClass.machineClassFitness(ProcessClass::Storage);
if (workerAvailable(it.second, false) &&
!excludedMachines.count(it.second.details.interf.locality.zoneId()) &&
(includeDCs.size() == 0 || includeDCs.count(it.second.details.interf.locality.dcId())) &&
!addressExcluded(excludedAddresses, it.second.details.interf.address()) && fit < bestFit) {
Initial repository commit
2017-05-26 04:48:44 +08:00
bestFit = fit;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
bestInfo = it.second.details;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (bestInfo.present()) {
Initial repository commit
2017-05-26 04:48:44 +08:00
return bestInfo.get();
}
}
throw no_more_servers();
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
std::vector<WorkerDetails> getWorkersForSeedServers(
DatabaseConfiguration const& conf,
Reference<IReplicationPolicy> const& policy,
Optional<Optional<Standalone<StringRef>>> const& dcId = Optional<Optional<Standalone<StringRef>>>()) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
std::map<ProcessClass::Fitness, vector<WorkerDetails>> fitness_workers;
std::vector<WorkerDetails> results;
removed LocalitySetRef and IRepPolicyRef typedefs, because for clarity the Ref suffix is reserved for arena allocated objects instead of reference counted objects.
2019-03-14 04:14:39 +08:00
Reference<LocalitySet> logServerSet = Reference<LocalitySet>(new LocalityMap<WorkerDetails>());
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
LocalityMap<WorkerDetails>* logServerMap = (LocalityMap<WorkerDetails>*)logServerSet.getPtr();
fix: seed storage servers are recruited based on the storage policy
2017-09-15 08:06:00 +08:00
bool bCompleted = false;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& it : id_worker) {
auto fitness = it.second.details.processClass.machineClassFitness(ProcessClass::Storage);
if (workerAvailable(it.second, false) && !conf.isExcludedServer(it.second.details.interf.addresses()) &&
fitness != ProcessClass::NeverAssign &&
(!dcId.present() || it.second.details.interf.locality.dcId() == dcId.get())) {
fitness_workers[fitness].push_back(it.second.details);
fix: seed storage servers are recruited based on the storage policy
2017-09-15 08:06:00 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& it : fitness_workers) {
for (auto& worker : it.second) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
logServerMap->add(worker.interf.locality, &worker);
fix: seed storage servers are recruited based on the storage policy
2017-09-15 08:06:00 +08:00
}
std::vector<LocalityEntry> bestSet;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (logServerSet->selectReplicas(policy, bestSet)) {
fix: seed storage servers are recruited based on the storage policy
2017-09-15 08:06:00 +08:00
results.reserve(bestSet.size());
for (auto& entry : bestSet) {
auto object = logServerMap->getObject(entry);
results.push_back(*object);
}
bCompleted = true;
break;
}
}
logServerSet->clear();
logServerSet.clear();
if (!bCompleted) {
throw no_more_servers();
}
return results;
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
std::vector<WorkerDetails> getWorkersForTlogs(DatabaseConfiguration const& conf,
int32_t required,
int32_t desired,
Add trace to debug
2020-08-07 16:02:07 +08:00
Reference<IReplicationPolicy> const& policy,
std::map<Optional<Standalone<StringRef>>, int>& id_used,
bool checkStable = false,
std::set<Optional<Key>> dcIds = std::set<Optional<Key>>(),
std::vector<UID> exclusionWorkerIds = {}) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
std::map<std::pair<ProcessClass::Fitness, bool>, vector<WorkerDetails>> fitness_workers;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
std::vector<WorkerDetails> results;
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
std::vector<LocalityData> unavailableLocals;
removed LocalitySetRef and IRepPolicyRef typedefs, because for clarity the Ref suffix is reserved for arena allocated objects instead of reference counted objects.
2019-03-14 04:14:39 +08:00
Reference<LocalitySet> logServerSet;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
LocalityMap<WorkerDetails>* logServerMap;
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
bool bCompleted = false;
Initial repository commit
2017-05-26 04:48:44 +08:00
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
logServerSet = Reference<LocalitySet>(new LocalityMap<WorkerDetails>());
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
logServerMap = (LocalityMap<WorkerDetails>*)logServerSet.getPtr();
for (auto& it : id_worker) {
if (std::find(exclusionWorkerIds.begin(), exclusionWorkerIds.end(), it.second.details.interf.id()) ==
exclusionWorkerIds.end()) {
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
auto fitness = it.second.details.processClass.machineClassFitness(ProcessClass::TLog);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (workerAvailable(it.second, checkStable) &&
!conf.isExcludedServer(it.second.details.interf.addresses()) &&
fitness != ProcessClass::NeverAssign &&
(!dcIds.size() || dcIds.count(it.second.details.interf.locality.dcId()))) {
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
fitness_workers[std::make_pair(fitness, it.second.details.degraded)].push_back(it.second.details);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} else {
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
unavailableLocals.push_back(it.second.details.interf.locality);
}
Fixed support for RemoveServers Workload Added availability functions to simulation
2017-05-27 05:20:11 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
results.reserve(results.size() + id_worker.size());
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (int fitness = ProcessClass::BestFit; fitness != ProcessClass::NeverAssign && !bCompleted; fitness++) {
auto fitnessEnum = (ProcessClass::Fitness)fitness;
for (int addingDegraded = 0; addingDegraded < 2; addingDegraded++) {
auto workerItr = fitness_workers.find(std::make_pair(fitnessEnum, (bool)addingDegraded));
fixed review comments
2019-03-13 05:38:54 +08:00
if (workerItr != fitness_workers.end()) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& worker : workerItr->second) {
fixed review comments
2019-03-13 05:38:54 +08:00
logServerMap->add(worker.interf.locality, &worker);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
Merge remote-tracking branch 'negoyal/fdb_cache_subfeature2' into features/cache-initialization
2020-01-08 11:53:09 +08:00
fixed review comments
2019-03-13 05:38:54 +08:00
if (logServerSet->size() < (addingDegraded == 0 ? desired : required)) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} else if (logServerSet->size() == required || logServerSet->size() <= desired) {
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
if (logServerSet->validate(policy)) {
for (auto& object : logServerMap->getObjects()) {
results.push_back(*object);
}
bCompleted = true;
break;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
RemoveServersSafely:Safety check toKill1 to avoid cluster getting stuck toKill1 and toKill2 are a random subset of all processes. If simply kill all processes in toKill1 or toKill2, we may kill too many processes to make the cluster unavailable and stuck. Similar as what toKill2 were modified if it can cause cluster unavailable, we should do the same thing for toKill1
2020-07-29 12:07:23 +08:00
TraceEvent(SevWarn, "GWFTADNotAcceptable", id)
.detail("Fitness", fitness)
.detail("Processes", logServerSet->size())
.detail("Required", required)
.detail("TLogPolicy", policy->info())
.detail("DesiredLogs", desired)
.detail("AddingDegraded", addingDegraded);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
// Try to select the desired size, if larger
else {
std::vector<LocalityEntry> bestSet;
std::vector<LocalityData> tLocalities;
// Try to find the best team of servers to fulfill the policy
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (findBestPolicySet(bestSet,
logServerSet,
policy,
desired,
SERVER_KNOBS->POLICY_RATING_TESTS,
Address review comments.
2020-02-21 01:34:01 +08:00
SERVER_KNOBS->POLICY_GENERATIONS)) {
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
results.reserve(results.size() + bestSet.size());
for (auto& entry : bestSet) {
auto object = logServerMap->getObject(entry);
ASSERT(object);
results.push_back(*object);
tLocalities.push_back(object->interf.locality);
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
TraceEvent("GWFTADBestResults", id)
.detail("Fitness", fitness)
.detail("Processes", logServerSet->size())
.detail("BestCount", bestSet.size())
.detail("BestZones", ::describeZones(tLocalities))
.detail("BestDataHalls", ::describeDataHalls(tLocalities))
.detail("TLogPolicy", policy->info())
.detail("TotalResults", results.size())
.detail("DesiredLogs", desired)
.detail("AddingDegraded", addingDegraded);
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
bCompleted = true;
break;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
TraceEvent(SevWarn, "GWFTADNoBest", id)
.detail("Fitness", fitness)
.detail("Processes", logServerSet->size())
.detail("Required", required)
.detail("TLogPolicy", policy->info())
.detail("DesiredLogs", desired)
.detail("AddingDegraded", addingDegraded);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
}
// If policy cannot be satisfied
first working version of non-copying recovery working with fearless configurations
2018-04-09 12:24:05 +08:00
if (!bCompleted) {
std::vector<LocalityData> tLocalities;
for (auto& object : logServerMap->getObjects()) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
tLocalities.push_back(object->interf.locality);
first working version of non-copying recovery working with fearless configurations
2018-04-09 12:24:05 +08:00
}
RemoveServersSafely:Safety check toKill1 to avoid cluster getting stuck toKill1 and toKill2 are a random subset of all processes. If simply kill all processes in toKill1 or toKill2, we may kill too many processes to make the cluster unavailable and stuck. Similar as what toKill2 were modified if it can cause cluster unavailable, we should do the same thing for toKill1
2020-07-29 12:07:23 +08:00
TraceEvent(SevWarn, "GetTLogTeamFailed")
.detail("Policy", policy->info())
.detail("Processes", logServerSet->size())
.detail("Workers", id_worker.size())
.detail("FitnessGroups", fitness_workers.size())
.detail("TLogZones", ::describeZones(tLocalities))
.detail("TLogDataHalls", ::describeDataHalls(tLocalities))
.detail("MissingZones", ::describeZones(unavailableLocals))
.detail("MissingDataHalls", ::describeDataHalls(unavailableLocals))
.detail("Required", required)
.detail("DesiredLogs", desired)
.detail("RatingTests", SERVER_KNOBS->POLICY_RATING_TESTS)
.detail("CheckStable", checkStable)
.detail("NumExclusionWorkers", exclusionWorkerIds.size())
.detail("PolicyGenerations", SERVER_KNOBS->POLICY_GENERATIONS)
.backtrace();
Initial repository commit
2017-05-26 04:48:44 +08:00
logServerSet->clear();
logServerSet.clear();
throw no_more_servers();
}
for (auto& result : results) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
id_used[result.interf.locality.processId()]++;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
TraceEvent("GetTLogTeamDone")
.detail("Completed", bCompleted)
.detail("Policy", policy->info())
.detail("Results", results.size())
.detail("Processes", logServerSet->size())
.detail("Workers", id_worker.size())
.detail("Required", required)
.detail("Desired", desired)
.detail("RatingTests", SERVER_KNOBS->POLICY_RATING_TESTS)
.detail("PolicyGenerations", SERVER_KNOBS->POLICY_GENERATIONS);
Initial repository commit
2017-05-26 04:48:44 +08:00
return results;
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
// FIXME: This logic will fallback unnecessarily when usable dcs > 1 because it does not check all combinations of
// potential satellite locations
std::vector<WorkerDetails> getWorkersForSatelliteLogs(const DatabaseConfiguration& conf,
const RegionInfo& region,
const RegionInfo& remoteRegion,
std::map<Optional<Standalone<StringRef>>, int>& id_used,
bool& satelliteFallback,
bool checkStable = false) {
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
int startDC = 0;
loop {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (startDC > 0 && startDC >= region.satellites.size() + 1 -
(satelliteFallback ? region.satelliteTLogUsableDcsFallback
: region.satelliteTLogUsableDcs)) {
if (satelliteFallback || region.satelliteTLogUsableDcsFallback == 0) {
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
throw no_more_servers();
} else {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!goodRecruitmentTime.isReady()) {
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
throw operation_failed();
}
satelliteFallback = true;
startDC = 0;
}
}
try {
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
bool remoteDCUsedAsSatellite = false;
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
std::set<Optional<Key>> satelliteDCs;
added the ability to configure satellite_logs by satellite location, this will overwrite the region configure if both are present
2019-10-15 09:30:15 +08:00
int32_t desiredSatelliteTLogs = 0;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (int s = startDC;
s < std::min<int>(startDC + (satelliteFallback ? region.satelliteTLogUsableDcsFallback
: region.satelliteTLogUsableDcs),
region.satellites.size());
s++) {
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
satelliteDCs.insert(region.satellites[s].dcId);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (region.satellites[s].satelliteDesiredTLogCount == -1 || desiredSatelliteTLogs == -1) {
added the ability to configure satellite_logs by satellite location, this will overwrite the region configure if both are present
2019-10-15 09:30:15 +08:00
desiredSatelliteTLogs = -1;
} else {
desiredSatelliteTLogs += region.satellites[s].satelliteDesiredTLogCount;
}
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
if (region.satellites[s].dcId == remoteRegion.dcId) {
remoteDCUsedAsSatellite = true;
}
}
std::vector<UID> exclusionWorkerIds;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
// FIXME: If remote DC is used as satellite then this logic only ensures that required number of remote
// TLogs can be recruited. It does not balance the number of desired TLogs across the satellite and
// remote sides.
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
if (remoteDCUsedAsSatellite) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
std::map<Optional<Standalone<StringRef>>, int> tmpIdUsed;
auto remoteLogs = getWorkersForTlogs(conf,
conf.getRemoteTLogReplicationFactor(),
conf.getRemoteTLogReplicationFactor(),
conf.getRemoteTLogPolicy(),
tmpIdUsed,
false,
{ remoteRegion.dcId },
{});
std::transform(remoteLogs.begin(),
remoteLogs.end(),
std::back_inserter(exclusionWorkerIds),
[](const WorkerDetails& in) { return in.interf.id(); });
}
if (satelliteFallback) {
return getWorkersForTlogs(conf,
region.satelliteTLogReplicationFactorFallback,
desiredSatelliteTLogs > 0 ? desiredSatelliteTLogs
: conf.getDesiredSatelliteLogs(region.dcId) *
region.satelliteTLogUsableDcsFallback /
region.satelliteTLogUsableDcs,
region.satelliteTLogPolicyFallback,
id_used,
checkStable,
satelliteDCs,
exclusionWorkerIds);
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
} else {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
return getWorkersForTlogs(conf,
region.satelliteTLogReplicationFactor,
desiredSatelliteTLogs > 0 ? desiredSatelliteTLogs
: conf.getDesiredSatelliteLogs(region.dcId),
region.satelliteTLogPolicy,
id_used,
checkStable,
satelliteDCs,
exclusionWorkerIds);
}
} catch (Error& e) {
if (e.code() != error_code_no_more_servers) {
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
throw;
}
}
startDC++;
}
}
Add actors to store halt request futures Address best fitness in checking better DD or RK.
2019-03-23 09:00:16 +08:00
ProcessClass::Fitness getBestFitnessForRoleInDatacenter(ProcessClass::ClusterRole role) {
ProcessClass::Fitness bestFitness = ProcessClass::NeverAssign;
for (const auto& it : id_worker) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (it.second.priorityInfo.isExcluded ||
it.second.details.interf.locality.dcId() != clusterControllerDcId) {
Add actors to store halt request futures Address best fitness in checking better DD or RK.
2019-03-23 09:00:16 +08:00
continue;
}
bestFitness = std::min(bestFitness, it.second.details.processClass.machineClassFitness(role));
}
return bestFitness;
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
WorkerFitnessInfo getWorkerForRoleInDatacenter(Optional<Standalone<StringRef>> const& dcId,
ProcessClass::ClusterRole role,
ProcessClass::Fitness unacceptableFitness,
DatabaseConfiguration const& conf,
std::map<Optional<Standalone<StringRef>>, int>& id_used,
bool checkStable = false) {
std::map<std::pair<ProcessClass::Fitness, int>, std::pair<vector<WorkerDetails>, vector<WorkerDetails>>>
fitness_workers;
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& it : id_worker) {
auto fitness = it.second.details.processClass.machineClassFitness(role);
if (conf.isExcludedServer(it.second.details.interf.addresses())) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
fitness = std::max(fitness, ProcessClass::ExcludeFit);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (workerAvailable(it.second, checkStable) && fitness < unacceptableFitness &&
it.second.details.interf.locality.dcId() == dcId) {
Merging with Master.
2019-11-13 05:01:29 +08:00
if (isLongLivedStateless(it.first)) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
fitness_workers[std::make_pair(fitness, id_used[it.first])].second.push_back(it.second.details);
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
} else {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
fitness_workers[std::make_pair(fitness, id_used[it.first])].first.push_back(it.second.details);
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& it : fitness_workers) {
for (int j = 0; j < 2; j++) {
auto& w = j == 0 ? it.second.first : it.second.second;
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
deterministicRandom()->randomShuffle(w);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (int i = 0; i < w.size(); i++) {
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
id_used[w[i].interf.locality.processId()]++;
one proxy or resolver with best fitness no longer prevents more proxies or resolvers from being recruited with good fitness
2019-10-15 09:32:17 +08:00
return WorkerFitnessInfo(w[i], std::max(ProcessClass::GoodFit, it.first.first), it.first.second);
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
throw no_more_servers();
}
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
vector<WorkerDetails> getWorkersForRoleInDatacenter(
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
Optional<Standalone<StringRef>> const& dcId,
ProcessClass::ClusterRole role,
int amount,
DatabaseConfiguration const& conf,
std::map<Optional<Standalone<StringRef>>, int>& id_used,
Optional<WorkerFitnessInfo> minWorker = Optional<WorkerFitnessInfo>(),
bool checkStable = false) {
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
std::map<std::pair<ProcessClass::Fitness, int>, std::pair<vector<WorkerDetails>, vector<WorkerDetails>>>
fitness_workers;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
vector<WorkerDetails> results;
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
if (minWorker.present()) {
results.push_back(minWorker.get().worker);
refactor getWorkersForRoleInDatacenter
2020-03-17 02:50:17 +08:00
}
if (amount <= results.size()) {
Initial repository commit
2017-05-26 04:48:44 +08:00
return results;
refactor getWorkersForRoleInDatacenter
2020-03-17 02:50:17 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
for (auto& it : id_worker) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
auto fitness = it.second.details.processClass.machineClassFitness(role);
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
if (workerAvailable(it.second, checkStable) &&
!conf.isExcludedServer(it.second.details.interf.addresses()) &&
it.second.details.interf.locality.dcId() == dcId &&
(!minWorker.present() ||
(it.second.details.interf.id() != minWorker.get().worker.interf.id() &&
(fitness < minWorker.get().fitness ||
(fitness == minWorker.get().fitness && id_used[it.first] <= minWorker.get().used))))) {
Merging with Master.
2019-11-13 05:01:29 +08:00
if (isLongLivedStateless(it.first)) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
fitness_workers[std::make_pair(fitness, id_used[it.first])].second.push_back(it.second.details);
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
} else {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
fitness_workers[std::make_pair(fitness, id_used[it.first])].first.push_back(it.second.details);
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& it : fitness_workers) {
for (int j = 0; j < 2; j++) {
auto& w = j == 0 ? it.second.first : it.second.second;
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
deterministicRandom()->randomShuffle(w);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (int i = 0; i < w.size(); i++) {
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
results.push_back(w[i]);
id_used[w[i].interf.locality.processId()]++;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (results.size() == amount)
return results;
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
return results;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
struct RoleFitness {
ProcessClass::Fitness bestFit;
ProcessClass::Fitness worstFit;
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
ProcessClass::ClusterRole role;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
int count;
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
bool worstIsDegraded;
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
RoleFitness(int bestFit, int worstFit, int count, ProcessClass::ClusterRole role)
: bestFit((ProcessClass::Fitness)bestFit), worstFit((ProcessClass::Fitness)worstFit), count(count),
role(role), worstIsDegraded(false) {}
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
RoleFitness(int fitness, int count, ProcessClass::ClusterRole role)
: bestFit((ProcessClass::Fitness)fitness), worstFit((ProcessClass::Fitness)fitness), count(count), role(role),
worstIsDegraded(false) {}
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
RoleFitness()
: bestFit(ProcessClass::NeverAssign), worstFit(ProcessClass::NeverAssign), role(ProcessClass::NoRole),
count(0), worstIsDegraded(false) {}
Enable checkUsingDesiredClasses() in consistency check
2017-10-25 03:58:54 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
RoleFitness(vector<WorkerDetails> workers, ProcessClass::ClusterRole role) : role(role) {
one proxy or resolver with best fitness no longer prevents more proxies or resolvers from being recruited with good fitness
2019-10-15 09:32:17 +08:00
worstFit = ProcessClass::GoodFit;
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
worstIsDegraded = false;
fix: tlogFitness did not consider it better to have one tlog of a better fitness fix: checkStable was not used in all places in better master exists fix: we need to call checkOutstanding on worker registration in all cases fix: in case persistentData is keyValueStoreMemory, we need to make sure it is fully recovered before writing to it
2018-01-05 03:33:02 +08:00
bestFit = ProcessClass::NeverAssign;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& it : workers) {
auto thisFit = it.processClass.machineClassFitness(role);
if (thisFit > worstFit) {
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
worstFit = thisFit;
worstIsDegraded = it.degraded;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} else if (thisFit == worstFit) {
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
worstIsDegraded = worstIsDegraded || it.degraded;
}
fix: tlogFitness did not consider it better to have one tlog of a better fitness fix: checkStable was not used in all places in better master exists fix: we need to call checkOutstanding on worker registration in all cases fix: in case persistentData is keyValueStoreMemory, we need to make sure it is fully recovered before writing to it
2018-01-05 03:33:02 +08:00
bestFit = std::min(bestFit, thisFit);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
count = workers.size();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
// degraded is only used for recruitment of tlogs
if (role != ProcessClass::TLog) {
fix: degraded is only used for tlog recruitment, so we should not use it in the fitness calculation for other roles
2019-03-22 02:23:49 +08:00
worstIsDegraded = false;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
bool operator<(RoleFitness const& r) const {
if (worstFit != r.worstFit)
return worstFit < r.worstFit;
if (worstIsDegraded != r.worstIsDegraded)
return r.worstIsDegraded;
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
// FIXME: TLog recruitment process does not guarantee the best fit is not worsened.
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (role != ProcessClass::TLog && role != ProcessClass::LogRouter && bestFit != r.bestFit)
return bestFit < r.bestFit;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
return count > r.count;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Remove using namespace std::rel_ops This causes the following to not compile anymore \#include <utility> \#include <vector> using namespace std::rel_ops; int main() { std::vector<int> xs; return xs.rbegin() != xs.rend(); } See https://godbolt.org/z/s1977n
2020-07-11 05:37:47 +08:00
bool operator>(RoleFitness const& r) const { return r < *this; }
bool operator<=(RoleFitness const& r) const { return !(*this > r); }
bool operator>=(RoleFitness const& r) const { return !(*this < r); }
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
bool betterFitness(RoleFitness const& r) const {
if (worstFit != r.worstFit)
return worstFit < r.worstFit;
if (worstIsDegraded != r.worstIsDegraded)
return r.worstFit;
if (bestFit != r.bestFit)
return bestFit < r.bestFit;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
return false;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
bool betterCount(RoleFitness const& r) const {
if (count > r.count)
return true;
if (worstFit != r.worstFit)
return worstFit < r.worstFit;
if (worstIsDegraded != r.worstIsDegraded)
return r.worstFit;
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
return false;
Swapped OkayFit and UnsetFit, because generally if machine classes are set on one machine they are set everywhere and it helps with wait_for_good_recruitment logic wait_for_good_recruitment now requires that you have the desired count of each roll remote recruitment is given a much longer wait_for_good_recruitment time interval, which does not start until enough remote machines have registered
2018-06-23 01:15:24 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
bool operator==(RoleFitness const& r) const {
return worstFit == r.worstFit && bestFit == r.bestFit && count == r.count &&
worstIsDegraded == r.worstIsDegraded;
}
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
std::string toString() const { return format("%d %d %d %d", bestFit, worstFit, count, worstIsDegraded); }
Initial repository commit
2017-05-26 04:48:44 +08:00
};
fix: better master exists must check if fitness is better for proxies or resolvers before looking at the count of either of them
2019-10-18 04:18:31 +08:00
struct RoleFitnessPair {
RoleFitness proxy;
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
RoleFitness grvProxy;
fix: better master exists must check if fitness is better for proxies or resolvers before looking at the count of either of them
2019-10-18 04:18:31 +08:00
RoleFitness resolver;
RoleFitnessPair() {}
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
RoleFitnessPair(RoleFitness const& proxy, RoleFitness const& grvProxy, RoleFitness const& resolver)
: proxy(proxy), grvProxy(grvProxy), resolver(resolver) {}
fix: better master exists must check if fitness is better for proxies or resolvers before looking at the count of either of them
2019-10-18 04:18:31 +08:00
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
bool operator<(RoleFitnessPair const& r) const {
if (proxy.betterFitness(r.proxy)) {
fix: better master exists must check if fitness is better for proxies or resolvers before looking at the count of either of them
2019-10-18 04:18:31 +08:00
return true;
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (r.proxy.betterFitness(proxy)) {
fix: better master exists must check if fitness is better for proxies or resolvers before looking at the count of either of them
2019-10-18 04:18:31 +08:00
return false;
}
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
if (grvProxy.betterFitness(r.grvProxy)) {
return true;
}
if (r.grvProxy.betterFitness(grvProxy)) {
return false;
}
if (resolver.betterFitness(r.resolver)) {
fix: better master exists must check if fitness is better for proxies or resolvers before looking at the count of either of them
2019-10-18 04:18:31 +08:00
return true;
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (r.resolver.betterFitness(resolver)) {
fix: better master exists must check if fitness is better for proxies or resolvers before looking at the count of either of them
2019-10-18 04:18:31 +08:00
return false;
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (proxy.count != r.proxy.count) {
fix: better master exists must check if fitness is better for proxies or resolvers before looking at the count of either of them
2019-10-18 04:18:31 +08:00
return proxy.count > r.proxy.count;
}
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
if (grvProxy.count != r.grvProxy.count) {
return grvProxy.count > r.grvProxy.count;
}
fix: better master exists must check if fitness is better for proxies or resolvers before looking at the count of either of them
2019-10-18 04:18:31 +08:00
return resolver.count > r.resolver.count;
}
Remove using namespace std::rel_ops This causes the following to not compile anymore \#include <utility> \#include <vector> using namespace std::rel_ops; int main() { std::vector<int> xs; return xs.rbegin() != xs.rend(); } See https://godbolt.org/z/s1977n
2020-07-11 05:37:47 +08:00
bool operator>(RoleFitnessPair const& r) const { return r < *this; }
bool operator<=(RoleFitnessPair const& r) const { return !(*this > r); }
bool operator>=(RoleFitnessPair const& r) const { return !(*this < r); }
fix: better master exists must check if fitness is better for proxies or resolvers before looking at the count of either of them
2019-10-18 04:18:31 +08:00
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
bool operator==(RoleFitnessPair const& r) const {
return proxy == r.proxy && grvProxy == r.grvProxy && resolver == r.resolver;
}
fix: better master exists must check if fitness is better for proxies or resolvers before looking at the count of either of them
2019-10-18 04:18:31 +08:00
};
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
std::set<Optional<Standalone<StringRef>>> getDatacenters(DatabaseConfiguration const& conf,
bool checkStable = false) {
Initial repository commit
2017-05-26 04:48:44 +08:00
std::set<Optional<Standalone<StringRef>>> result;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& it : id_worker)
if (workerAvailable(it.second, checkStable) && !conf.isExcludedServer(it.second.details.interf.addresses()))
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.insert(it.second.details.interf.locality.dcId());
Initial repository commit
2017-05-26 04:48:44 +08:00
return result;
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
void updateKnownIds(std::map<Optional<Standalone<StringRef>>, int>* id_used) {
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
(*id_used)[masterProcessId]++;
(*id_used)[clusterControllerProcessId]++;
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
RecruitRemoteFromConfigurationReply findRemoteWorkersForConfiguration(
RecruitRemoteFromConfigurationRequest const& req) {
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
RecruitRemoteFromConfigurationReply result;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
std::map<Optional<Standalone<StringRef>>, int> id_used;
Initial repository commit
2017-05-26 04:48:44 +08:00
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
updateKnownIds(&id_used);
Initial repository commit
2017-05-26 04:48:44 +08:00
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
std::set<Optional<Key>> remoteDC;
remoteDC.insert(req.dcId);
added usable_regions to specify if we will replicate into a remote region remote replication defaults to the primary replication removed remote_logs, because they should be specified as an override in the regions object
2018-06-18 10:31:15 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
auto remoteLogs = getWorkersForTlogs(req.configuration,
req.configuration.getRemoteTLogReplicationFactor(),
req.configuration.getDesiredRemoteLogs(),
req.configuration.getRemoteTLogPolicy(),
id_used,
false,
remoteDC,
req.exclusionWorkerIds);
for (int i = 0; i < remoteLogs.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.remoteTLogs.push_back(remoteLogs[i].interf);
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
auto logRouters = getWorkersForRoleInDatacenter(
req.dcId, ProcessClass::LogRouter, req.logRouterCount, req.configuration, id_used);
for (int i = 0; i < logRouters.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.logRouters.push_back(logRouters[i].interf);
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!goodRemoteRecruitmentTime.isReady() &&
((RoleFitness(
SERVER_KNOBS->EXPECTED_TLOG_FITNESS, req.configuration.getDesiredRemoteLogs(), ProcessClass::TLog)
.betterCount(RoleFitness(remoteLogs, ProcessClass::TLog))) ||
(RoleFitness(SERVER_KNOBS->EXPECTED_LOG_ROUTER_FITNESS, req.logRouterCount, ProcessClass::LogRouter)
.betterCount(RoleFitness(logRouters, ProcessClass::LogRouter))))) {
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
throw operation_failed();
}
Initial repository commit
2017-05-26 04:48:44 +08:00
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
return result;
}
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ErrorOr<RecruitFromConfigurationReply> findWorkersForConfiguration(RecruitFromConfigurationRequest const& req,
Optional<Key> dcId) {
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
RecruitFromConfigurationReply result;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
std::map<Optional<Standalone<StringRef>>, int> id_used;
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
updateKnownIds(&id_used);
Merge branch 'master' into feature-remote-logs # Conflicts: # fdbrpc/Locality.cpp # fdbrpc/Locality.h # fdbserver/ClusterController.actor.cpp # fdbserver/ClusterRecruitmentInterface.h # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/fdbserver.vcxproj.filters # fdbserver/masterserver.actor.cpp # fdbserver/worker.actor.cpp # flow/error_definitions.h
2017-10-06 08:09:44 +08:00
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
ASSERT(dcId.present());
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
std::set<Optional<Key>> primaryDC;
primaryDC.insert(dcId);
result.dcId = dcId;
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
RegionInfo region;
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
RegionInfo remoteRegion;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& r : req.configuration.regions) {
if (r.dcId == dcId.get()) {
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
region = r;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} else {
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
remoteRegion = r;
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
}
}
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (req.recruitSeedServers) {
auto primaryStorageServers =
getWorkersForSeedServers(req.configuration, req.configuration.storagePolicy, dcId);
for (int i = 0; i < primaryStorageServers.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.storageServers.push_back(primaryStorageServers[i].interf);
Merge branch 'master' into feature-remote-logs # Conflicts: # fdbrpc/Locality.cpp # fdbrpc/Locality.h # fdbserver/ClusterController.actor.cpp # fdbserver/ClusterRecruitmentInterface.h # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/fdbserver.vcxproj.filters # fdbserver/masterserver.actor.cpp # fdbserver/worker.actor.cpp # flow/error_definitions.h
2017-10-06 08:09:44 +08:00
}
}
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
auto tlogs = getWorkersForTlogs(req.configuration,
req.configuration.tLogReplicationFactor,
req.configuration.getDesiredLogs(),
req.configuration.tLogPolicy,
id_used,
false,
primaryDC);
for (int i = 0; i < tlogs.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.tLogs.push_back(tlogs[i].interf);
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
std::vector<WorkerDetails> satelliteLogs;
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
if (region.satelliteTLogReplicationFactor > 0 && req.configuration.usableRegions > 1) {
satelliteLogs =
getWorkersForSatelliteLogs(req.configuration, region, remoteRegion, id_used, result.satelliteFallback);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (int i = 0; i < satelliteLogs.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.satelliteTLogs.push_back(satelliteLogs[i].interf);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
auto first_commit_proxy = getWorkerForRoleInDatacenter(
dcId, ProcessClass::CommitProxy, ProcessClass::ExcludeFit, req.configuration, id_used);
auto first_grv_proxy = getWorkerForRoleInDatacenter(
dcId, ProcessClass::GrvProxy, ProcessClass::ExcludeFit, req.configuration, id_used);
auto first_resolver = getWorkerForRoleInDatacenter(
dcId, ProcessClass::Resolver, ProcessClass::ExcludeFit, req.configuration, id_used);
auto commit_proxies = getWorkersForRoleInDatacenter(dcId,
ProcessClass::CommitProxy,
req.configuration.getDesiredCommitProxies(),
req.configuration,
id_used,
first_commit_proxy);
auto grv_proxies = getWorkersForRoleInDatacenter(dcId,
ProcessClass::GrvProxy,
req.configuration.getDesiredGrvProxies(),
req.configuration,
id_used,
first_grv_proxy);
auto resolvers = getWorkersForRoleInDatacenter(dcId,
ProcessClass::Resolver,
req.configuration.getDesiredResolvers(),
req.configuration,
id_used,
first_resolver);
for (int i = 0; i < commit_proxies.size(); i++)
result.commitProxies.push_back(commit_proxies[i].interf);
for (int i = 0; i < grv_proxies.size(); i++)
result.grvProxies.push_back(grv_proxies[i].interf);
for (int i = 0; i < resolvers.size(); i++)
pass joshua
2020-07-29 22:26:55 +08:00
result.resolvers.push_back(resolvers[i].interf);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (req.maxOldLogRouters > 0) {
if (tlogs.size() == 1) {
fix: we were recruiting one too few oldLogRouters code cleanup
2020-01-03 07:05:44 +08:00
result.oldLogRouters.push_back(tlogs[0].interf);
recruit oldLogRouters on TLogs, do not recruit oldLogRouters on the cluster controller if possible
2019-12-10 05:12:13 +08:00
} else {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (int i = 0; i < tlogs.size(); i++) {
if (tlogs[i].interf.locality.processId() != clusterControllerProcessId) {
fix: we were recruiting one too few oldLogRouters code cleanup
2020-01-03 07:05:44 +08:00
result.oldLogRouters.push_back(tlogs[i].interf);
recruit oldLogRouters on TLogs, do not recruit oldLogRouters on the cluster controller if possible
2019-12-10 05:12:13 +08:00
}
}
}
first version of non-copying recovery. Upgrades are broken, and it has not been tested using fearless configurations yet
2018-03-30 06:12:38 +08:00
}
fix: we must commit to the number of log routers we are going to use when recruiting the primary, because it determines the number of log router tags that will be attached to mutations
2018-03-07 08:31:21 +08:00
Rename backupLoggingEnabled as backupWorkerEnabled To highlight the changes for 7.0 backup changes. By default, backup_worker_enabled flag is set for 7.0 version.
2020-02-05 02:09:16 +08:00
if (req.configuration.backupWorkerEnabled) {
Add a DB configuration option for backup workers Right now, the default is to keep the old backup behavior, i.e., do NOT use backup workers. Specifically, if BackupType is not set (or is set to default), the master will not recruit backup workers and will not add pseudo locality for backup workers. The StartFullBackupTaskFunc is updated to check if backup worker is enabled. Only when it is not enabled, starting a backup will wait on all backup workers to be started.
2020-01-28 05:14:52 +08:00
const int nBackup = std::max<int>(
(req.configuration.desiredLogRouterCount > 0 ? req.configuration.desiredLogRouterCount : tlogs.size()),
req.maxOldLogRouters);
auto backupWorkers =
getWorkersForRoleInDatacenter(dcId, ProcessClass::Backup, nBackup, req.configuration, id_used);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
std::transform(backupWorkers.begin(),
backupWorkers.end(),
std::back_inserter(result.backupWorkers),
Add a DB configuration option for backup workers Right now, the default is to keep the old backup behavior, i.e., do NOT use backup workers. Specifically, if BackupType is not set (or is set to default), the master will not recruit backup workers and will not add pseudo locality for backup workers. The StartFullBackupTaskFunc is updated to check if backup worker is enabled. Only when it is not enabled, starting a backup will wait on all backup workers to be started.
2020-01-28 05:14:52 +08:00
[](const WorkerDetails& w) { return w.interf; });
}
Recruit backup worker during master recovery Right now recruit the same number as TLogs. The backup worker does nothing.
2019-05-16 07:13:04 +08:00
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
if (!goodRecruitmentTime.isReady() &&
(RoleFitness(SERVER_KNOBS->EXPECTED_TLOG_FITNESS, req.configuration.getDesiredLogs(), ProcessClass::TLog)
.betterCount(RoleFitness(tlogs, ProcessClass::TLog)) ||
(region.satelliteTLogReplicationFactor > 0 && req.configuration.usableRegions > 1 &&
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
RoleFitness(SERVER_KNOBS->EXPECTED_TLOG_FITNESS,
req.configuration.getDesiredSatelliteLogs(dcId),
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
ProcessClass::TLog)
.betterCount(RoleFitness(satelliteLogs, ProcessClass::TLog))) ||
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
RoleFitness(SERVER_KNOBS->EXPECTED_COMMIT_PROXY_FITNESS,
req.configuration.getDesiredCommitProxies(),
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
ProcessClass::CommitProxy)
.betterCount(RoleFitness(commit_proxies, ProcessClass::CommitProxy)) ||
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
RoleFitness(SERVER_KNOBS->EXPECTED_GRV_PROXY_FITNESS,
req.configuration.getDesiredGrvProxies(),
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
ProcessClass::GrvProxy)
.betterCount(RoleFitness(grv_proxies, ProcessClass::GrvProxy)) ||
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
RoleFitness(SERVER_KNOBS->EXPECTED_RESOLVER_FITNESS,
req.configuration.getDesiredResolvers(),
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
ProcessClass::Resolver)
.betterCount(RoleFitness(resolvers, ProcessClass::Resolver)))) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
return operation_failed();
Initial repository commit
2017-05-26 04:48:44 +08:00
}
return result;
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
RecruitFromConfigurationReply findWorkersForConfiguration(RecruitFromConfigurationRequest const& req) {
if (req.configuration.regions.size() > 1) {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
std::vector<RegionInfo> regions = req.configuration.regions;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (regions[0].priority == regions[1].priority && regions[1].dcId == clusterControllerDcId.get()) {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
std::swap(regions[0], regions[1]);
}
fix: treat the local region as best when version difference is too large re-check requests when the version difference becomes small
2018-07-07 05:44:11 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (regions[1].dcId == clusterControllerDcId.get() && regions[1].priority >= 0 &&
(!versionDifferenceUpdated || datacenterVersionDifference >= SERVER_KNOBS->MAX_VERSION_DIFFERENCE)) {
fix: treat the local region as best when version difference is too large re-check requests when the version difference becomes small
2018-07-07 05:44:11 +08:00
std::swap(regions[0], regions[1]);
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
bool setPrimaryDesired = false;
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
try {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
auto reply = findWorkersForConfiguration(req, regions[0].dcId);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
setPrimaryDesired = true;
vector<Optional<Key>> dcPriority;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
dcPriority.push_back(regions[0].dcId);
dcPriority.push_back(regions[1].dcId);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
desiredDcIds.set(dcPriority);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (reply.isError()) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
throw reply.getError();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} else if (regions[0].dcId == clusterControllerDcId.get()) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
return reply.get();
}
throw no_more_servers();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} catch (Error& e) {
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
if (!goodRemoteRecruitmentTime.isReady() && regions[1].dcId != clusterControllerDcId.get()) {
fix: the cluster controller needs to continue to retry recruitment until after wait_for_good_remote_recruitment_delay
2018-06-26 09:20:16 +08:00
throw operation_failed();
}
if (e.code() != error_code_no_more_servers || regions[1].priority < 0) {
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
throw;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
TraceEvent(SevWarn, "AttemptingRecruitmentInRemoteDC", id).error(e);
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
auto reply = findWorkersForConfiguration(req, regions[1].dcId);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!setPrimaryDesired) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
vector<Optional<Key>> dcPriority;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
dcPriority.push_back(regions[1].dcId);
dcPriority.push_back(regions[0].dcId);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
desiredDcIds.set(dcPriority);
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (reply.isError()) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
throw reply.getError();
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
} else if (regions[1].dcId == clusterControllerDcId.get()) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
return reply.get();
}
throw;
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} else if (req.configuration.regions.size() == 1) {
added support for single region configurations
2018-03-16 01:59:30 +08:00
vector<Optional<Key>> dcPriority;
dcPriority.push_back(req.configuration.regions[0].dcId);
desiredDcIds.set(dcPriority);
auto reply = findWorkersForConfiguration(req, req.configuration.regions[0].dcId);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (reply.isError()) {
added support for single region configurations
2018-03-16 01:59:30 +08:00
throw reply.getError();
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
} else if (req.configuration.regions[0].dcId == clusterControllerDcId.get()) {
added support for single region configurations
2018-03-16 01:59:30 +08:00
return reply.get();
}
throw no_more_servers();
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
} else {
RecruitFromConfigurationReply result;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
std::map<Optional<Standalone<StringRef>>, int> id_used;
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
updateKnownIds(&id_used);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
auto tlogs = getWorkersForTlogs(req.configuration,
req.configuration.tLogReplicationFactor,
req.configuration.getDesiredLogs(),
req.configuration.tLogPolicy,
id_used);
for (int i = 0; i < tlogs.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.tLogs.push_back(tlogs[i].interf);
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (req.maxOldLogRouters > 0) {
if (tlogs.size() == 1) {
fix: we were recruiting one too few oldLogRouters code cleanup
2020-01-03 07:05:44 +08:00
result.oldLogRouters.push_back(tlogs[0].interf);
recruit oldLogRouters on TLogs, do not recruit oldLogRouters on the cluster controller if possible
2019-12-10 05:12:13 +08:00
} else {
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
for (int i = 0; i < tlogs.size(); i++) {
if (tlogs[i].interf.locality.processId() != clusterControllerProcessId) {
fix: we were recruiting one too few oldLogRouters code cleanup
2020-01-03 07:05:44 +08:00
result.oldLogRouters.push_back(tlogs[i].interf);
recruit oldLogRouters on TLogs, do not recruit oldLogRouters on the cluster controller if possible
2019-12-10 05:12:13 +08:00
}
}
}
}
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
if (req.recruitSeedServers) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
auto primaryStorageServers =
getWorkersForSeedServers(req.configuration, req.configuration.storagePolicy);
for (int i = 0; i < primaryStorageServers.size(); i++)
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.storageServers.push_back(primaryStorageServers[i].interf);
Merge branch 'master' into feature-remote-logs # Conflicts: # fdbrpc/Locality.cpp # fdbrpc/Locality.h # fdbserver/ClusterController.actor.cpp # fdbserver/ClusterRecruitmentInterface.h # fdbserver/TLogServer.actor.cpp # fdbserver/TagPartitionedLogSystem.actor.cpp # fdbserver/WorkerInterface.h # fdbserver/fdbserver.vcxproj.filters # fdbserver/masterserver.actor.cpp # fdbserver/worker.actor.cpp # flow/error_definitions.h
2017-10-06 08:09:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
auto datacenters = getDatacenters(req.configuration);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
fix: better master exists must check if fitness is better for proxies or resolvers before looking at the count of either of them
2019-10-18 04:18:31 +08:00
RoleFitnessPair bestFitness;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
int numEquivalent = 1;
Optional<Key> bestDC;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto dcId : datacenters) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
try {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
// SOMEDAY: recruitment in other DCs besides the clusterControllerDcID will not account for the
// processes used by the master and cluster controller properly.
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
auto used = id_used;
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
auto first_commit_proxy = getWorkerForRoleInDatacenter(
dcId, ProcessClass::CommitProxy, ProcessClass::ExcludeFit, req.configuration, used);
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
auto first_grv_proxy = getWorkerForRoleInDatacenter(
dcId, ProcessClass::GrvProxy, ProcessClass::ExcludeFit, req.configuration, used);
auto first_resolver = getWorkerForRoleInDatacenter(
dcId, ProcessClass::Resolver, ProcessClass::ExcludeFit, req.configuration, used);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
auto commit_proxies = getWorkersForRoleInDatacenter(dcId,
ProcessClass::CommitProxy,
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
req.configuration.getDesiredCommitProxies(),
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
req.configuration,
used,
first_commit_proxy);
auto grv_proxies = getWorkersForRoleInDatacenter(dcId,
ProcessClass::GrvProxy,
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
req.configuration.getDesiredGrvProxies(),
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
req.configuration,
used,
first_grv_proxy);
auto resolvers = getWorkersForRoleInDatacenter(dcId,
ProcessClass::Resolver,
req.configuration.getDesiredResolvers(),
req.configuration,
used,
first_resolver);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
RoleFitnessPair fitness(RoleFitness(commit_proxies, ProcessClass::CommitProxy),
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
RoleFitness(grv_proxies, ProcessClass::GrvProxy),
RoleFitness(resolvers, ProcessClass::Resolver));
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (dcId == clusterControllerDcId) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
bestFitness = fitness;
bestDC = dcId;
Fixed formatting issues and removed GRV related code in MasterProxy
2020-08-12 09:54:54 +08:00
for (int i = 0; i < resolvers.size(); i++) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
result.resolvers.push_back(resolvers[i].interf);
Fixed formatting issues and removed GRV related code in MasterProxy
2020-08-12 09:54:54 +08:00
}
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
for (int i = 0; i < commit_proxies.size(); i++) {
result.commitProxies.push_back(commit_proxies[i].interf);
Fixed formatting issues and removed GRV related code in MasterProxy
2020-08-12 09:54:54 +08:00
}
for (int i = 0; i < grv_proxies.size(); i++) {
result.grvProxies.push_back(grv_proxies[i].interf);
}
first version of non-copying recovery. Upgrades are broken, and it has not been tested using fearless configurations yet
2018-03-30 06:12:38 +08:00
Rename backupLoggingEnabled as backupWorkerEnabled To highlight the changes for 7.0 backup changes. By default, backup_worker_enabled flag is set for 7.0 version.
2020-02-05 02:09:16 +08:00
if (req.configuration.backupWorkerEnabled) {
Add a DB configuration option for backup workers Right now, the default is to keep the old backup behavior, i.e., do NOT use backup workers. Specifically, if BackupType is not set (or is set to default), the master will not recruit backup workers and will not add pseudo locality for backup workers. The StartFullBackupTaskFunc is updated to check if backup worker is enabled. Only when it is not enabled, starting a backup will wait on all backup workers to be started.
2020-01-28 05:14:52 +08:00
const int nBackup = std::max<int>(tlogs.size(), req.maxOldLogRouters);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
auto backupWorkers = getWorkersForRoleInDatacenter(
dcId, ProcessClass::Backup, nBackup, req.configuration, id_used);
std::transform(backupWorkers.begin(),
backupWorkers.end(),
Add a DB configuration option for backup workers Right now, the default is to keep the old backup behavior, i.e., do NOT use backup workers. Specifically, if BackupType is not set (or is set to default), the master will not recruit backup workers and will not add pseudo locality for backup workers. The StartFullBackupTaskFunc is updated to check if backup worker is enabled. Only when it is not enabled, starting a backup will wait on all backup workers to be started.
2020-01-28 05:14:52 +08:00
std::back_inserter(result.backupWorkers),
[](const WorkerDetails& w) { return w.interf; });
}
Recruit backup workers for single region Enable log router tags for single region, which are popped by backup workers. Need to add noop for backup workers if there is no active backups.
2019-12-11 05:28:49 +08:00
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
break;
} else {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (fitness < bestFitness) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
bestFitness = fitness;
numEquivalent = 1;
bestDC = dcId;
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
} else if (fitness == bestFitness &&
deterministicRandom()->random01() < 1.0 / ++numEquivalent) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
bestDC = dcId;
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} catch (Error& e) {
if (e.code() != error_code_no_more_servers) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
throw;
}
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (bestDC != clusterControllerDcId) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
vector<Optional<Key>> dcPriority;
dcPriority.push_back(bestDC);
desiredDcIds.set(dcPriority);
throw no_more_servers();
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
// If this cluster controller dies, do not prioritize recruiting the next one in the same DC
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
desiredDcIds.set(vector<Optional<Key>>());
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
TraceEvent("FindWorkersForConfig")
.detail("Replication", req.configuration.tLogReplicationFactor)
.detail("DesiredLogs", req.configuration.getDesiredLogs())
.detail("ActualLogs", result.tLogs.size())
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
.detail("DesiredCommitProxies", req.configuration.getDesiredCommitProxies())
.detail("ActualCommitProxies", result.commitProxies.size())
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
.detail("DesiredGrvProxies", req.configuration.getDesiredGrvProxies())
.detail("ActualGrvProxies", result.grvProxies.size())
.detail("DesiredResolvers", req.configuration.getDesiredResolvers())
pass joshua
2020-07-29 22:26:55 +08:00
.detail("ActualResolvers", result.resolvers.size());
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
if (!goodRecruitmentTime.isReady() &&
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
(RoleFitness(
SERVER_KNOBS->EXPECTED_TLOG_FITNESS, req.configuration.getDesiredLogs(), ProcessClass::TLog)
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
.betterCount(RoleFitness(tlogs, ProcessClass::TLog)) ||
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
RoleFitness(SERVER_KNOBS->EXPECTED_COMMIT_PROXY_FITNESS,
req.configuration.getDesiredCommitProxies(),
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
ProcessClass::CommitProxy)
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
.betterCount(bestFitness.proxy) ||
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
RoleFitness(SERVER_KNOBS->EXPECTED_GRV_PROXY_FITNESS,
req.configuration.getDesiredGrvProxies(),
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
ProcessClass::GrvProxy)
.betterCount(bestFitness.grvProxy) ||
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
RoleFitness(SERVER_KNOBS->EXPECTED_RESOLVER_FITNESS,
req.configuration.getDesiredResolvers(),
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
ProcessClass::Resolver)
.betterCount(bestFitness.resolver))) {
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
throw operation_failed();
}
return result;
}
}
Add comments to HA code and loadBalance code
2021-02-11 05:45:06 +08:00
// Check if txn system is recruited successfully in each region
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
void checkRegions(const std::vector<RegionInfo>& regions) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (desiredDcIds.get().present() && desiredDcIds.get().get().size() == 2 &&
desiredDcIds.get().get()[0].get() == regions[0].dcId &&
desiredDcIds.get().get()[1].get() == regions[1].dcId) {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
return;
}
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
try {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
std::map<Optional<Standalone<StringRef>>, int> id_used;
getWorkerForRoleInDatacenter(
regions[0].dcId, ProcessClass::ClusterController, ProcessClass::ExcludeFit, db.config, id_used, true);
getWorkerForRoleInDatacenter(
regions[0].dcId, ProcessClass::Master, ProcessClass::ExcludeFit, db.config, id_used, true);
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
std::set<Optional<Key>> primaryDC;
primaryDC.insert(regions[0].dcId);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
getWorkersForTlogs(db.config,
db.config.tLogReplicationFactor,
db.config.getDesiredLogs(),
db.config.tLogPolicy,
id_used,
true,
primaryDC);
if (regions[0].satelliteTLogReplicationFactor > 0 && db.config.usableRegions > 1) {
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
bool satelliteFallback = false;
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
getWorkersForSatelliteLogs(db.config, regions[0], regions[1], id_used, satelliteFallback, true);
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
getWorkerForRoleInDatacenter(
regions[0].dcId, ProcessClass::Resolver, ProcessClass::ExcludeFit, db.config, id_used, true);
getWorkerForRoleInDatacenter(
regions[0].dcId, ProcessClass::CommitProxy, ProcessClass::ExcludeFit, db.config, id_used, true);
getWorkerForRoleInDatacenter(
regions[0].dcId, ProcessClass::GrvProxy, ProcessClass::ExcludeFit, db.config, id_used, true);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
vector<Optional<Key>> dcPriority;
dcPriority.push_back(regions[0].dcId);
dcPriority.push_back(regions[1].dcId);
desiredDcIds.set(dcPriority);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} catch (Error& e) {
if (e.code() != error_code_no_more_servers) {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
throw;
}
}
}
void checkRecoveryStalled() {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if ((db.serverInfo->get().recoveryState == RecoveryState::RECRUITING ||
db.serverInfo->get().recoveryState == RecoveryState::ACCEPTING_COMMITS ||
db.serverInfo->get().recoveryState == RecoveryState::ALL_LOGS_RECRUITED) &&
db.recoveryStalled) {
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
if (db.config.regions.size() > 1) {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
auto regions = db.config.regions;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (clusterControllerDcId.get() == regions[0].dcId) {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
std::swap(regions[0], regions[1]);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
ASSERT(clusterControllerDcId.get() == regions[1].dcId);
checkRegions(regions);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
// FIXME: determine when to fail the cluster controller when a primaryDC has not been set
Initial repository commit
2017-05-26 04:48:44 +08:00
bool betterMasterExists() {
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
const ServerDBInfo dbi = db.serverInfo->get();
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (dbi.recoveryState < RecoveryState::ACCEPTING_COMMITS) {
Initial repository commit
2017-05-26 04:48:44 +08:00
return false;
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
// Do not trigger better master exists if the cluster controller is excluded, since the master will change
// anyways once the cluster controller is moved
if (id_worker[clusterControllerProcessId].priorityInfo.isExcluded) {
Merge release-6.0 into master
2018-09-22 07:14:39 +08:00
return false;
}
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
if (db.config.regions.size() > 1 && db.config.regions[0].priority > db.config.regions[1].priority &&
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
db.config.regions[0].dcId != clusterControllerDcId.get() && versionDifferenceUpdated &&
datacenterVersionDifference < SERVER_KNOBS->MAX_VERSION_DIFFERENCE) {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
checkRegions(db.config.regions);
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
Fix double recoveries
2017-11-17 08:58:55 +08:00
// Get master process
Initial repository commit
2017-05-26 04:48:44 +08:00
auto masterWorker = id_worker.find(dbi.master.locality.processId());
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (masterWorker == id_worker.end()) {
Initial repository commit
2017-05-26 04:48:44 +08:00
return false;
Enable checkUsingDesiredClasses() in consistency check
2017-10-25 03:58:54 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
Fix double recoveries
2017-11-17 08:58:55 +08:00
// Get tlog processes
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
std::vector<WorkerDetails> tlogs;
std::vector<WorkerDetails> remote_tlogs;
std::vector<WorkerDetails> satellite_tlogs;
std::vector<WorkerDetails> log_routers;
fix: we may not be able to recruit enough log routers, in this case put multiple log routers on the same worker, but also properly rank this configuration lower in better master exists
2018-04-27 13:18:07 +08:00
std::set<NetworkAddress> logRouterAddresses;
Update better master exist with backup workers During recruitment, if there is no desired log router count, use tlog size instead, because the number of backup workers has to be larger than 0.
2020-01-18 04:49:34 +08:00
std::vector<WorkerDetails> backup_workers;
std::set<NetworkAddress> backup_addresses;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& logSet : dbi.logSystemConfig.tLogs) {
for (auto& it : logSet.tLogs) {
more work towards shrinking locality
2020-04-19 12:29:38 +08:00
auto tlogWorker = id_worker.find(it.interf().filteredLocality.processId());
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (tlogWorker == id_worker.end())
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
return false;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (tlogWorker->second.priorityInfo.isExcluded)
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
return true;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (logSet.isLocal && logSet.locality == tagLocalitySatellite) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
satellite_tlogs.push_back(tlogWorker->second.details);
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
} else if (logSet.isLocal) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
tlogs.push_back(tlogWorker->second.details);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
} else {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
remote_tlogs.push_back(tlogWorker->second.details);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
}
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
for (auto& it : logSet.logRouters) {
more work towards shrinking locality
2020-04-19 12:29:38 +08:00
auto tlogWorker = id_worker.find(it.interf().filteredLocality.processId());
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (tlogWorker == id_worker.end())
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
return false;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (tlogWorker->second.priorityInfo.isExcluded)
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
return true;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!logRouterAddresses.count(tlogWorker->second.details.interf.address())) {
logRouterAddresses.insert(tlogWorker->second.details.interf.address());
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
log_routers.push_back(tlogWorker->second.details);
fix: we may not be able to recruit enough log routers, in this case put multiple log routers on the same worker, but also properly rank this configuration lower in better master exists
2018-04-27 13:18:07 +08:00
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
Update better master exist with backup workers During recruitment, if there is no desired log router count, use tlog size instead, because the number of backup workers has to be larger than 0.
2020-01-18 04:49:34 +08:00
for (const auto& worker : logSet.backupWorkers) {
auto workerIt = id_worker.find(worker.interf().locality.processId());
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (workerIt == id_worker.end())
return false;
if (workerIt->second.priorityInfo.isExcluded)
return true;
Update better master exist with backup workers During recruitment, if there is no desired log router count, use tlog size instead, because the number of backup workers has to be larger than 0.
2020-01-18 04:49:34 +08:00
if (backup_addresses.count(workerIt->second.details.interf.address()) == 0) {
backup_addresses.insert(workerIt->second.details.interf.address());
backup_workers.push_back(workerIt->second.details);
}
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
// Get commit proxy classes
std::vector<WorkerDetails> commitProxyClasses;
for (auto& it : dbi.client.commitProxies) {
auto commitProxyWorker = id_worker.find(it.processId);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (commitProxyWorker == id_worker.end())
return false;
if (commitProxyWorker->second.priorityInfo.isExcluded)
return true;
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
commitProxyClasses.push_back(commitProxyWorker->second.details);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
// Get grv proxy classes
std::vector<WorkerDetails> grvProxyClasses;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& it : dbi.client.grvProxies) {
Pass watches test
2020-07-15 15:37:41 +08:00
auto grvProxyWorker = id_worker.find(it.processId);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (grvProxyWorker == id_worker.end())
Pass watches test
2020-07-15 15:37:41 +08:00
return false;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (grvProxyWorker->second.priorityInfo.isExcluded)
Pass watches test
2020-07-15 15:37:41 +08:00
return true;
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
grvProxyClasses.push_back(grvProxyWorker->second.details);
Pass watches test
2020-07-15 15:37:41 +08:00
}
Fix double recoveries
2017-11-17 08:58:55 +08:00
// Get resolver classes
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
std::vector<WorkerDetails> resolverClasses;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& it : dbi.resolvers) {
Initial repository commit
2017-05-26 04:48:44 +08:00
auto resolverWorker = id_worker.find(it.locality.processId());
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (resolverWorker == id_worker.end())
Initial repository commit
2017-05-26 04:48:44 +08:00
return false;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (resolverWorker->second.priorityInfo.isExcluded)
Fix double recoveries
2017-11-17 08:58:55 +08:00
return true;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
resolverClasses.push_back(resolverWorker->second.details);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
// Check master fitness. Don't return false if master is excluded in case all the processes are excluded, we
// still need master for recovery.
ProcessClass::Fitness oldMasterFit =
masterWorker->second.details.processClass.machineClassFitness(ProcessClass::Master);
if (db.config.isExcludedServer(dbi.master.addresses())) {
Fix double recoveries
2017-11-17 08:58:55 +08:00
oldMasterFit = std::max(oldMasterFit, ProcessClass::ExcludeFit);
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
std::map<Optional<Standalone<StringRef>>, int> id_used;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
id_used[clusterControllerProcessId]++;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
WorkerFitnessInfo mworker = getWorkerForRoleInDatacenter(
clusterControllerDcId, ProcessClass::Master, ProcessClass::NeverAssign, db.config, id_used, true);
auto newMasterFit = mworker.worker.processClass.machineClassFitness(ProcessClass::Master);
if (db.config.isExcludedServer(mworker.worker.interf.addresses())) {
one proxy or resolver with best fitness no longer prevents more proxies or resolvers from being recruited with good fitness
2019-10-15 09:32:17 +08:00
newMasterFit = std::max(newMasterFit, ProcessClass::ExcludeFit);
}
Fix double recoveries
2017-11-17 08:58:55 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (oldMasterFit < newMasterFit)
Fix double recoveries
2017-11-17 08:58:55 +08:00
return false;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (oldMasterFit > newMasterFit || (dbi.master.locality.processId() == clusterControllerProcessId &&
mworker.worker.interf.locality.processId() != clusterControllerProcessId))
Fix double recoveries
2017-11-17 08:58:55 +08:00
return true;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
std::set<Optional<Key>> primaryDC;
std::set<Optional<Key>> remoteDC;
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
RegionInfo region;
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
RegionInfo remoteRegion;
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
if (db.config.regions.size()) {
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
primaryDC.insert(clusterControllerDcId);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& r : db.config.regions) {
if (r.dcId != clusterControllerDcId.get()) {
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
ASSERT(remoteDC.empty());
remoteDC.insert(r.dcId);
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
remoteRegion = r;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
} else {
added region-based configuration to support a large variety of fearless setups. Currently only 1 primary 1 remote setups are allowed.
2018-03-06 11:27:46 +08:00
ASSERT(region.dcId == StringRef());
region = r;
}
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
Fix double recoveries
2017-11-17 08:58:55 +08:00
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
// Check tLog fitness
RoleFitness oldTLogFit(tlogs, ProcessClass::TLog);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
auto newTLogs = getWorkersForTlogs(db.config,
db.config.tLogReplicationFactor,
db.config.getDesiredLogs(),
db.config.tLogPolicy,
id_used,
true,
primaryDC);
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
RoleFitness newTLogFit(newTLogs, ProcessClass::TLog);
Fix double recoveries
2017-11-17 08:58:55 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (oldTLogFit < newTLogFit)
return false;
Fix double recoveries
2017-11-17 08:58:55 +08:00
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
bool oldSatelliteFallback = false;
one proxy or resolver with best fitness no longer prevents more proxies or resolvers from being recruited with good fitness
2019-10-15 09:32:17 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& logSet : dbi.logSystemConfig.tLogs) {
if (region.satelliteTLogPolicy.isValid() && logSet.isLocal && logSet.locality == tagLocalitySatellite) {
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
oldSatelliteFallback = logSet.tLogPolicy->info() != region.satelliteTLogPolicy->info();
Fixed a crash found by nightly correctness.
2020-11-04 01:28:04 +08:00
ASSERT(!oldSatelliteFallback ||
(region.satelliteTLogPolicyFallback.isValid() &&
logSet.tLogPolicy->info() == region.satelliteTLogPolicyFallback->info()));
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
break;
}
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
RoleFitness oldSatelliteTLogFit(satellite_tlogs, ProcessClass::TLog);
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
bool newSatelliteFallback = false;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
auto newSatelliteTLogs =
(region.satelliteTLogReplicationFactor > 0 && db.config.usableRegions > 1)
? getWorkersForSatelliteLogs(db.config, region, remoteRegion, id_used, newSatelliteFallback, true)
: satellite_tlogs;
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
RoleFitness newSatelliteTLogFit(newSatelliteTLogs, ProcessClass::TLog);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
std::map<Optional<Key>, int32_t> satellite_priority;
for (auto& r : region.satellites) {
fix: the cluster controller would not change to a new set of satellite tlogs when they become available in a better satellite location
2019-10-15 09:31:23 +08:00
satellite_priority[r.dcId] = r.priority;
}
int32_t oldSatelliteRegionFit = std::numeric_limits<int32_t>::max();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& it : satellite_tlogs) {
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
if (satellite_priority.count(it.interf.locality.dcId())) {
fix: the cluster controller would not change to a new set of satellite tlogs when they become available in a better satellite location
2019-10-15 09:31:23 +08:00
oldSatelliteRegionFit = std::min(oldSatelliteRegionFit, satellite_priority[it.interf.locality.dcId()]);
} else {
oldSatelliteRegionFit = -1;
}
}
int32_t newSatelliteRegionFit = std::numeric_limits<int32_t>::max();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& it : newSatelliteTLogs) {
if (satellite_priority.count(it.interf.locality.dcId())) {
fix: the cluster controller would not change to a new set of satellite tlogs when they become available in a better satellite location
2019-10-15 09:31:23 +08:00
newSatelliteRegionFit = std::min(newSatelliteRegionFit, satellite_priority[it.interf.locality.dcId()]);
} else {
newSatelliteRegionFit = -1;
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (oldSatelliteFallback && !newSatelliteFallback)
return true;
if (!oldSatelliteFallback && newSatelliteFallback)
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
return false;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (oldSatelliteRegionFit < newSatelliteRegionFit)
return true;
if (oldSatelliteRegionFit > newSatelliteRegionFit)
fix: the cluster controller would not change to a new set of satellite tlogs when they become available in a better satellite location
2019-10-15 09:31:23 +08:00
return false;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (oldSatelliteTLogFit < newSatelliteTLogFit)
fix: the cluster controller would not change to a new set of satellite tlogs when they become available in a better satellite location
2019-10-15 09:31:23 +08:00
return false;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
RoleFitness oldRemoteTLogFit(remote_tlogs, ProcessClass::TLog);
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
std::vector<UID> exclusionWorkerIds;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
auto fn = [](const WorkerDetails& in) { return in.interf.id(); };
Support config where the primary and remote DC's can be used as satellites.
2019-03-19 03:17:59 +08:00
std::transform(newTLogs.begin(), newTLogs.end(), std::back_inserter(exclusionWorkerIds), fn);
std::transform(newSatelliteTLogs.begin(), newSatelliteTLogs.end(), std::back_inserter(exclusionWorkerIds), fn);
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
RoleFitness newRemoteTLogFit(
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
(db.config.usableRegions > 1 && (dbi.recoveryState == RecoveryState::ALL_LOGS_RECRUITED ||
dbi.recoveryState == RecoveryState::FULLY_RECOVERED))
? getWorkersForTlogs(db.config,
db.config.getRemoteTLogReplicationFactor(),
db.config.getDesiredRemoteLogs(),
db.config.getRemoteTLogPolicy(),
id_used,
true,
remoteDC,
exclusionWorkerIds)
: remote_tlogs,
ProcessClass::TLog);
if (oldRemoteTLogFit < newRemoteTLogFit)
return false;
int oldRouterCount =
oldTLogFit.count * std::max<int>(1, db.config.desiredLogRouterCount / std::max(1, oldTLogFit.count));
int newRouterCount =
newTLogFit.count * std::max<int>(1, db.config.desiredLogRouterCount / std::max(1, newTLogFit.count));
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
RoleFitness oldLogRoutersFit(log_routers, ProcessClass::LogRouter);
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
RoleFitness newLogRoutersFit(
(db.config.usableRegions > 1 && dbi.recoveryState == RecoveryState::FULLY_RECOVERED)
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
? getWorkersForRoleInDatacenter(*remoteDC.begin(),
ProcessClass::LogRouter,
newRouterCount,
db.config,
id_used,
Optional<WorkerFitnessInfo>(),
true)
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
: log_routers,
ProcessClass::LogRouter);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (oldLogRoutersFit.count < oldRouterCount) {
fix: we may not be able to recruit enough log routers, in this case put multiple log routers on the same worker, but also properly rank this configuration lower in better master exists
2018-04-27 13:18:07 +08:00
oldLogRoutersFit.worstFit = ProcessClass::NeverAssign;
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (newLogRoutersFit.count < newRouterCount) {
fix: we may not be able to recruit enough log routers, in this case put multiple log routers on the same worker, but also properly rank this configuration lower in better master exists
2018-04-27 13:18:07 +08:00
newLogRoutersFit.worstFit = ProcessClass::NeverAssign;
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (oldLogRoutersFit < newLogRoutersFit)
return false;
Pass watches test
2020-07-15 15:37:41 +08:00
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
// Check proxy/grvProxy/resolver fitness
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
RoleFitnessPair oldInFit(RoleFitness(commitProxyClasses, ProcessClass::CommitProxy),
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
RoleFitness(grvProxyClasses, ProcessClass::GrvProxy),
RoleFitness(resolverClasses, ProcessClass::Resolver));
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
auto first_commit_proxy = getWorkerForRoleInDatacenter(
clusterControllerDcId, ProcessClass::CommitProxy, ProcessClass::ExcludeFit, db.config, id_used, true);
auto first_grv_proxy = getWorkerForRoleInDatacenter(
clusterControllerDcId, ProcessClass::GrvProxy, ProcessClass::ExcludeFit, db.config, id_used, true);
auto first_resolver = getWorkerForRoleInDatacenter(
clusterControllerDcId, ProcessClass::Resolver, ProcessClass::ExcludeFit, db.config, id_used, true);
auto commit_proxies = getWorkersForRoleInDatacenter(clusterControllerDcId,
ProcessClass::CommitProxy,
db.config.getDesiredCommitProxies(),
db.config,
id_used,
first_commit_proxy,
true);
auto grv_proxies = getWorkersForRoleInDatacenter(clusterControllerDcId,
ProcessClass::GrvProxy,
db.config.getDesiredGrvProxies(),
db.config,
id_used,
first_grv_proxy,
true);
auto resolvers = getWorkersForRoleInDatacenter(clusterControllerDcId,
ProcessClass::Resolver,
db.config.getDesiredResolvers(),
db.config,
id_used,
first_resolver,
true);
Initial repository commit
2017-05-26 04:48:44 +08:00
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
RoleFitnessPair newInFit(RoleFitness(commit_proxies, ProcessClass::CommitProxy),
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
RoleFitness(grv_proxies, ProcessClass::GrvProxy),
RoleFitness(resolvers, ProcessClass::Resolver));
if (oldInFit.proxy.betterFitness(newInFit.proxy) || oldInFit.grvProxy.betterFitness(newInFit.grvProxy) ||
oldInFit.resolver.betterFitness(newInFit.resolver)) {
fix: better master exists must check if fitness is better for proxies or resolvers before looking at the count of either of them
2019-10-18 04:18:31 +08:00
return false;
}
Update better master exist with backup workers During recruitment, if there is no desired log router count, use tlog size instead, because the number of backup workers has to be larger than 0.
2020-01-18 04:49:34 +08:00
Better master exists should not trigger if it will lower the total number of processes being recruited
2021-03-17 03:14:19 +08:00
if (oldTLogFit.count + oldInFit.proxy.count + oldInFit.grvProxy.count + oldInFit.resolver.count >
newTLogFit.count + newInFit.proxy.count + newInFit.grvProxy.count + newInFit.resolver.count) {
return false;
}
Update better master exist with backup workers During recruitment, if there is no desired log router count, use tlog size instead, because the number of backup workers has to be larger than 0.
2020-01-18 04:49:34 +08:00
// Check backup worker fitness
RoleFitness oldBackupWorkersFit(backup_workers, ProcessClass::Backup);
const int nBackup = backup_addresses.size();
RoleFitness newBackupWorkersFit(
Better master exists should not trigger if it will lower the total number of processes being recruited
2021-03-17 03:14:19 +08:00
getWorkersForRoleInDatacenter(
clusterControllerDcId, ProcessClass::Backup, nBackup, db.config, id_used, true),
Update better master exist with backup workers During recruitment, if there is no desired log router count, use tlog size instead, because the number of backup workers has to be larger than 0.
2020-01-18 04:49:34 +08:00
ProcessClass::Backup);
Remove debug trace events
2020-07-23 03:20:22 +08:00
if (oldTLogFit > newTLogFit || oldInFit > newInFit || oldSatelliteTLogFit > newSatelliteTLogFit ||
Update better master exist with backup workers During recruitment, if there is no desired log router count, use tlog size instead, because the number of backup workers has to be larger than 0.
2020-01-18 04:49:34 +08:00
oldRemoteTLogFit > newRemoteTLogFit || oldLogRoutersFit > newLogRoutersFit ||
oldBackupWorkersFit > newBackupWorkersFit) {
TraceEvent("BetterMasterExists", id)
.detail("OldMasterFit", oldMasterFit)
.detail("NewMasterFit", newMasterFit)
.detail("OldTLogFit", oldTLogFit.toString())
.detail("NewTLogFit", newTLogFit.toString())
.detail("OldProxyFit", oldInFit.proxy.toString())
.detail("NewProxyFit", newInFit.proxy.toString())
Resolve comments to make GRV proxy a separate process class
2020-08-06 15:01:57 +08:00
.detail("OldGrvProxyFit", oldInFit.grvProxy.toString())
.detail("NewGrvProxyFit", newInFit.grvProxy.toString())
Update better master exist with backup workers During recruitment, if there is no desired log router count, use tlog size instead, because the number of backup workers has to be larger than 0.
2020-01-18 04:49:34 +08:00
.detail("OldResolverFit", oldInFit.resolver.toString())
.detail("NewResolverFit", newInFit.resolver.toString())
.detail("OldSatelliteFit", oldSatelliteTLogFit.toString())
.detail("NewSatelliteFit", newSatelliteTLogFit.toString())
.detail("OldRemoteFit", oldRemoteTLogFit.toString())
.detail("NewRemoteFit", newRemoteTLogFit.toString())
.detail("OldRouterFit", oldLogRoutersFit.toString())
.detail("NewRouterFit", newLogRoutersFit.toString())
.detail("OldBackupWorkerFit", oldBackupWorkersFit.toString())
.detail("NewBackupWorkerFit", newBackupWorkersFit.toString())
.detail("OldSatelliteFallback", oldSatelliteFallback)
.detail("NewSatelliteFallback", newSatelliteFallback);
Initial repository commit
2017-05-26 04:48:44 +08:00
return true;
}
Fix double recoveries
2017-11-17 08:58:55 +08:00
Initial repository commit
2017-05-26 04:48:44 +08:00
return false;
}
fix: dd test recruitment should prefer the location of ratekeeper over other used processes
2019-11-14 04:58:55 +08:00
bool isUsedNotMaster(Optional<Key> processId) {
Recruit a new ratekeeper before halting the old
2019-03-20 06:21:46 +08:00
ASSERT(masterProcessId.present());
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (processId == masterProcessId)
return false;
Recruit a new ratekeeper before halting the old
2019-03-20 06:21:46 +08:00
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
auto& dbInfo = db.serverInfo->get();
fix: dd test recruitment should prefer the location of ratekeeper over other used processes
2019-11-14 04:58:55 +08:00
for (const auto& tlogset : dbInfo.logSystemConfig.tLogs) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (const auto& tlog : tlogset.tLogs) {
if (tlog.present() && tlog.interf().filteredLocality.processId() == processId)
return true;
fix: dd test recruitment should prefer the location of ratekeeper over other used processes
2019-11-14 04:58:55 +08:00
}
}
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
for (const CommitProxyInterface& interf : dbInfo.client.commitProxies) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (interf.processId == processId)
return true;
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
}
Pass watches test
2020-07-15 15:37:41 +08:00
for (const GrvProxyInterface& interf : dbInfo.client.grvProxies) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (interf.processId == processId)
return true;
Pass watches test
2020-07-15 15:37:41 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (const ResolverInterface& interf : dbInfo.resolvers) {
if (interf.locality.processId() == processId)
return true;
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (processId == clusterControllerProcessId)
return true;
Fix bug where DD or RK could be halted and re-recruited in a loop for certain valid process class configurations. Specifically, recruitment of DD or RK takes into account that master process is preferred over proxy, resolver or cc. But check for better DD only looks for better machine class ignoring that the new recruit could share a proxy or resolver or CC. Also try to balance the distribution of the DD and RK role if there are enough processes to do so.
2019-11-13 06:22:36 +08:00
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
return false;
}
bool onMasterIsBetter(const WorkerDetails& worker, ProcessClass::ClusterRole role) {
ASSERT(masterProcessId.present());
Fix ratekeeper and data distributor recruiting bug Avoid multiple concurrent recuriting of ratekeepers with a recruiting flag. Fix endless recruiting when the chosen worker is a proxy or a resolver -- prefer master in this case.
2019-03-21 01:00:31 +08:00
const auto& pid = worker.interf.locality.processId();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if ((role != ProcessClass::DataDistributor && role != ProcessClass::Ratekeeper) ||
pid == masterProcessId.get()) {
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
return false;
}
fix: dd test recruitment should prefer the location of ratekeeper over other used processes
2019-11-14 04:58:55 +08:00
return isUsedNotMaster(pid);
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
std::map<Optional<Standalone<StringRef>>, int> getUsedIds() {
Add used worker IDs to cluster controller This "usedIds" is updated when receiving a master registration message, so that when recruiting new data distributor, existing assignment is known.
2019-01-29 01:25:15 +08:00
std::map<Optional<Standalone<StringRef>>, int> idUsed;
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
updateKnownIds(&idUsed);
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
auto& dbInfo = db.serverInfo->get();
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
for (const auto& tlogset : dbInfo.logSystemConfig.tLogs) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (const auto& tlog : tlogset.tLogs) {
Various review comments fixes
2019-02-01 02:10:41 +08:00
if (tlog.present()) {
more work towards shrinking locality
2020-04-19 12:29:38 +08:00
idUsed[tlog.interf().filteredLocality.processId()]++;
Add used worker IDs to cluster controller This "usedIds" is updated when receiving a master registration message, so that when recruiting new data distributor, existing assignment is known.
2019-01-29 01:25:15 +08:00
}
}
}
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
for (const CommitProxyInterface& interf : dbInfo.client.commitProxies) {
fix compiler errors
2020-05-02 05:30:50 +08:00
ASSERT(interf.processId.present());
idUsed[interf.processId]++;
Add used worker IDs to cluster controller This "usedIds" is updated when receiving a master registration message, so that when recruiting new data distributor, existing assignment is known.
2019-01-29 01:25:15 +08:00
}
Pass watches test
2020-07-15 15:37:41 +08:00
for (const GrvProxyInterface& interf : dbInfo.client.grvProxies) {
ASSERT(interf.processId.present());
idUsed[interf.processId]++;
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (const ResolverInterface& interf : dbInfo.resolvers) {
Add used worker IDs to cluster controller This "usedIds" is updated when receiving a master registration message, so that when recruiting new data distributor, existing assignment is known.
2019-01-29 01:25:15 +08:00
ASSERT(interf.locality.processId().present());
idUsed[interf.locality.processId()]++;
}
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
return idUsed;
Add used worker IDs to cluster controller This "usedIds" is updated when receiving a master registration message, so that when recruiting new data distributor, existing assignment is known.
2019-01-29 01:25:15 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
std::map<Optional<Standalone<StringRef>>, WorkerInfo> id_worker;
std::map<Optional<Standalone<StringRef>>, ProcessClass>
id_class; // contains the mapping from process id to process class from the database
Initial repository commit
2017-05-26 04:48:44 +08:00
Standalone<RangeResultRef> lastProcessClasses;
bool gotProcessClasses;
Add fullyRecoveredConfig for checking exclusions
2017-11-16 09:15:24 +08:00
bool gotFullyRecoveredConfig;
Initial repository commit
2017-05-26 04:48:44 +08:00
Optional<Standalone<StringRef>> masterProcessId;
Use processId instead of address for comparison
2017-10-26 02:35:29 +08:00
Optional<Standalone<StringRef>> clusterControllerProcessId;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
Optional<Standalone<StringRef>> clusterControllerDcId;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
AsyncVar<Optional<vector<Optional<Key>>>> desiredDcIds; // desired DC priorities
AsyncVar<std::pair<bool, Optional<vector<Optional<Key>>>>>
changingDcIds; // current DC priorities to change first, and whether that is the cluster controller
AsyncVar<std::pair<bool, Optional<vector<Optional<Key>>>>>
changedDcIds; // current DC priorities to change second, and whether the cluster controller has been changed
Initial repository commit
2017-05-26 04:48:44 +08:00
UID id;
std::vector<RecruitFromConfigurationRequest> outstandingRecruitmentRequests;
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
std::vector<RecruitRemoteFromConfigurationRequest> outstandingRemoteRecruitmentRequests;
Initial repository commit
2017-05-26 04:48:44 +08:00
std::vector<std::pair<RecruitStorageRequest, double>> outstandingStorageRequests;
ActorCollection ac;
UpdateWorkerList updateWorkerList;
checking outstanding requests can be CPU intensive, so rate limit checking requests
2018-06-28 14:02:08 +08:00
Future<Void> outstandingRequestChecker;
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
Future<Void> outstandingRemoteRequestChecker;
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
AsyncTrigger updateDBInfo;
code cleanup based on review comments
2020-04-18 06:05:01 +08:00
std::set<Endpoint> updateDBInfoEndpoints;
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
std::set<Endpoint> removedDBInfoEndpoints;
Initial repository commit
2017-05-26 04:48:44 +08:00
DBInfo db;
Database cx;
double startTime;
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
Future<Void> goodRecruitmentTime;
Future<Void> goodRemoteRecruitmentTime;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
Version datacenterVersionDifference;
Add DataDistributorInterface to ServerDBInfo Also change the Proxy and QuietDatabase to use the DataDistributorInterface.
2019-01-19 03:30:18 +08:00
PromiseStream<Future<Void>> addActor;
Add backup role, class, and worker skeleton
2019-04-25 06:12:37 +08:00
bool versionDifferenceUpdated;
Fix the issue of slow dying Data Distributor Test with: -r simulation -f ./foundationdb/tests/slow/CommitBug.txt -s 67828576 -b on The test has the following event sequence: - Time 113.3s, CC noticed DD failure, cleard DD interface. - 1s later, DD rejoined and registered with CC. - Time 131.7s, DD actor cancelled. This old DD raced to register with CC and the failure monitor is not installed because monitorDataDistributor is stalled waiting for new DD. - Time 161.4s, new DD running. New DD recruting was delayed due to no servers in the period. Fix by disabling DD registration during the recruting process.
2019-03-18 11:55:59 +08:00
bool recruitingDistributor;
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
Optional<UID> recruitingRatekeeperID;
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
AsyncVar<bool> recruitRatekeeper;
Initial repository commit
2017-05-26 04:48:44 +08:00
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
CounterCollection clusterControllerMetrics;
Counter openDatabaseRequests;
Counter registerWorkerRequests;
Counter getWorkersRequests;
Counter getClientWorkersRequests;
Counter registerMasterRequests;
Counter statusRequests;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ClusterControllerData(ClusterControllerFullInterface const& ccInterface, LocalityData const& locality)
: clusterControllerProcessId(locality.processId()), clusterControllerDcId(locality.dcId()), id(ccInterface.id()),
ac(false), outstandingRequestChecker(Void()), outstandingRemoteRequestChecker(Void()), gotProcessClasses(false),
gotFullyRecoveredConfig(false), startTime(now()), goodRecruitmentTime(Never()),
goodRemoteRecruitmentTime(Never()), datacenterVersionDifference(0), versionDifferenceUpdated(false),
recruitingDistributor(false), recruitRatekeeper(false),
clusterControllerMetrics("ClusterController", id.toString()),
openDatabaseRequests("OpenDatabaseRequests", clusterControllerMetrics),
registerWorkerRequests("RegisterWorkerRequests", clusterControllerMetrics),
getWorkersRequests("GetWorkersRequests", clusterControllerMetrics),
getClientWorkersRequests("GetClientWorkersRequests", clusterControllerMetrics),
registerMasterRequests("RegisterMasterRequests", clusterControllerMetrics),
statusRequests("StatusRequests", clusterControllerMetrics) {
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
auto serverInfo = ServerDBInfo();
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
serverInfo.id = deterministicRandom()->randomUniqueID();
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
serverInfo.infoGeneration = ++db.dbInfoCount;
Initial repository commit
2017-05-26 04:48:44 +08:00
serverInfo.masterLifetime.ccID = id;
serverInfo.clusterInterface = ccInterface;
fix: the cluster controller did not pass in its own locality when creating its database object, therefore it was not using locality aware load balancing
2018-09-29 03:12:06 +08:00
serverInfo.myLocality = locality;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
db.serverInfo->set(serverInfo);
A giant translation of TaskFooPriority -> TaskPriority::Foo This is so that APIs that take priorities don't take ints, which are common and easy to accidentally pass the wrong thing.
2019-06-25 17:47:35 +08:00
cx = openDBOnServer(db.serverInfo, TaskPriority::DefaultEndpoint, true, true);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
~ClusterControllerData() {
ac.clear(false);
id_worker.clear();
}
};
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> clusterWatchDatabase(ClusterControllerData* cluster, ClusterControllerData::DBInfo* db) {
Initial repository commit
2017-05-26 04:48:44 +08:00
state MasterInterface iMaster;
// SOMEDAY: If there is already a non-failed master referenced by zkMasterInfo, use that one until it fails
// When this someday is implemented, make sure forced failures still cause the master to be recruited again
loop {
TraceEvent("CCWDB", cluster->id);
try {
state double recoveryStart = now();
TraceEvent("CCWDB", cluster->id).detail("Recruiting", "Master");
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
// We must recruit the master in the same data center as the cluster controller.
// This should always be possible, because we can recruit the master on the same process as the cluster
// controller.
std::map<Optional<Standalone<StringRef>>, int> id_used;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
id_used[cluster->clusterControllerProcessId]++;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
state WorkerFitnessInfo masterWorker = cluster->getWorkerForRoleInDatacenter(
cluster->clusterControllerDcId, ProcessClass::Master, ProcessClass::NeverAssign, db->config, id_used);
if ((masterWorker.worker.processClass.machineClassFitness(ProcessClass::Master) >
SERVER_KNOBS->EXPECTED_MASTER_FITNESS ||
masterWorker.worker.interf.locality.processId() == cluster->clusterControllerProcessId) &&
!cluster->goodRecruitmentTime.isReady()) {
TraceEvent("CCWDB", cluster->id)
.detail("Fitness", masterWorker.worker.processClass.machineClassFitness(ProcessClass::Master));
wait(delay(SERVER_KNOBS->ATTEMPT_RECRUITMENT_DELAY));
Initial repository commit
2017-05-26 04:48:44 +08:00
continue;
}
RecruitMasterRequest rmq;
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
rmq.lifetime = db->serverInfo->get().masterLifetime;
added the (untested) ability to force a recovery to the remote datacenter, even if that results in data loss. If the DR lag is more than 1 week there could be potential data corruption if any primary storage servers are still alive.
2018-07-01 21:39:04 +08:00
rmq.forceRecovery = db->forceRecovery;
Initial repository commit
2017-05-26 04:48:44 +08:00
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
cluster->masterProcessId = masterWorker.worker.interf.locality.processId();
The cluster controller will increase its failure monitoring delay after there have been many unfinishedRecoveries
2018-09-01 01:51:55 +08:00
cluster->db.unfinishedRecoveries++;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
state Future<ErrorOr<MasterInterface>> fNewMaster = masterWorker.worker.interf.master.tryGetReply(rmq);
wait(ready(fNewMaster) || db->forceMasterFailure.onTrigger());
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
if (fNewMaster.isReady() && fNewMaster.get().present()) {
TraceEvent("CCWDB", cluster->id).detail("Recruited", fNewMaster.get().get().id());
Initial repository commit
2017-05-26 04:48:44 +08:00
// for status tool
TraceEvent("RecruitedMasterWorker", cluster->id)
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
.detail("Address", fNewMaster.get().get().address())
.trackLatest("RecruitedMasterWorker");
Initial repository commit
2017-05-26 04:48:44 +08:00
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
iMaster = fNewMaster.get().get();
Initial repository commit
2017-05-26 04:48:44 +08:00
db->masterRegistrationCount = 0;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
db->recoveryStalled = false;
Initial repository commit
2017-05-26 04:48:44 +08:00
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
auto dbInfo = ServerDBInfo();
Initial repository commit
2017-05-26 04:48:44 +08:00
dbInfo.master = iMaster;
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
dbInfo.id = deterministicRandom()->randomUniqueID();
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
dbInfo.infoGeneration = ++db->dbInfoCount;
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
dbInfo.masterLifetime = db->serverInfo->get().masterLifetime;
Initial repository commit
2017-05-26 04:48:44 +08:00
++dbInfo.masterLifetime;
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
dbInfo.clusterInterface = db->serverInfo->get().clusterInterface;
dbInfo.distributor = db->serverInfo->get().distributor;
dbInfo.ratekeeper = db->serverInfo->get().ratekeeper;
dbInfo.latencyBandConfig = db->serverInfo->get().latencyBandConfig;
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
TraceEvent("CCWDB", cluster->id)
.detail("Lifetime", dbInfo.masterLifetime.toString())
.detail("ChangeID", dbInfo.id);
db->serverInfo->set(dbInfo);
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
state Future<Void> spinDelay = delay(
SERVER_KNOBS
->MASTER_SPIN_DELAY); // Don't retry master recovery more than once per second, but don't delay
// the "first" recovery after more than a second of normal operation
Initial repository commit
2017-05-26 04:48:44 +08:00
TraceEvent("CCWDB", cluster->id).detail("Watching", iMaster.id());
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
// Master failure detection is pretty sensitive, but if we are in the middle of a very long recovery we
// really don't want to have to start over
Initial repository commit
2017-05-26 04:48:44 +08:00
loop choose {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(wait(waitFailureClient(
iMaster.waitFailure,
db->masterRegistrationCount
? SERVER_KNOBS->MASTER_FAILURE_REACTION_TIME
: (now() - recoveryStart) * SERVER_KNOBS->MASTER_FAILURE_SLOPE_DURING_RECOVERY,
db->masterRegistrationCount ? -SERVER_KNOBS->MASTER_FAILURE_REACTION_TIME /
SERVER_KNOBS->SECONDS_BEFORE_NO_FAILURE_DELAY
: SERVER_KNOBS->MASTER_FAILURE_SLOPE_DURING_RECOVERY) ||
db->forceMasterFailure.onTrigger())) {
break;
}
when(wait(db->serverInfo->onChange())) {}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
wait(spinDelay);
Initial repository commit
2017-05-26 04:48:44 +08:00
TEST(true); // clusterWatchDatabase() master failed
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
TraceEvent(SevWarn, "DetectedFailedMaster", cluster->id).detail("OldMaster", iMaster.id());
Initial repository commit
2017-05-26 04:48:44 +08:00
} else {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
TEST(true); // clusterWatchDatabas() !newMaster.present()
wait(delay(SERVER_KNOBS->MASTER_SPIN_DELAY));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
} catch (Error& e) {
TraceEvent("CCWDB", cluster->id).error(e, true).detail("Master", iMaster.id());
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (e.code() == error_code_actor_cancelled)
throw;
Initial repository commit
2017-05-26 04:48:44 +08:00
bool ok = e.code() == error_code_no_more_servers;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
TraceEvent(ok ? SevWarn : SevError, "ClusterWatchDatabaseRetrying", cluster->id).error(e);
Initial repository commit
2017-05-26 04:48:44 +08:00
if (!ok)
throw e;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
wait(delay(SERVER_KNOBS->ATTEMPT_RECRUITMENT_DELAY));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> clusterGetServerInfo(ClusterControllerData::DBInfo* db,
UID knownServerInfoID,
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
ReplyPromise<ServerDBInfo> reply) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
while (db->serverInfo->get().id == knownServerInfoID) {
Initial repository commit
2017-05-26 04:48:44 +08:00
choose {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(wait(yieldedFuture(db->serverInfo->onChange()))) {}
when(wait(delayJittered(300))) { break; } // The server might be long gone!
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
reply.send(db->serverInfo->get());
Initial repository commit
2017-05-26 04:48:44 +08:00
return Void();
}
added status information on connected clients
2019-07-26 08:15:31 +08:00
ACTOR Future<Void> clusterOpenDatabase(ClusterControllerData::DBInfo* db, OpenDatabaseRequest req) {
db->clientStatus[req.reply.getEndpoint().getPrimaryAddress()] = std::make_pair(now(), req);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (db->clientStatus.size() > 10000) {
added status information on connected clients
2019-07-26 08:15:31 +08:00
TraceEvent(SevWarnAlways, "TooManyClientStatusEntries").suppressFor(1.0);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Merge remote-tracking branch 'negoyal/fdb_cache_subfeature2' into features/cache-initialization
2020-01-08 11:53:09 +08:00
added status information on connected clients
2019-07-26 08:15:31 +08:00
while (db->clientInfo->get().id == req.knownClientInfoID) {
Initial repository commit
2017-05-26 04:48:44 +08:00
choose {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(wait(db->clientInfo->onChange())) {}
when(wait(delayJittered(SERVER_KNOBS->COORDINATOR_REGISTER_INTERVAL))) {
break;
} // The client might be long gone!
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
req.reply.send(db->clientInfo->get());
Initial repository commit
2017-05-26 04:48:44 +08:00
return Void();
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
void checkOutstandingRecruitmentRequests(ClusterControllerData* self) {
for (int i = 0; i < self->outstandingRecruitmentRequests.size(); i++) {
Initial repository commit
2017-05-26 04:48:44 +08:00
RecruitFromConfigurationRequest& req = self->outstandingRecruitmentRequests[i];
try {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
RecruitFromConfigurationReply rep = self->findWorkersForConfiguration(req);
req.reply.send(rep);
swapAndPop(&self->outstandingRecruitmentRequests, i--);
Initial repository commit
2017-05-26 04:48:44 +08:00
} catch (Error& e) {
if (e.code() == error_code_no_more_servers || e.code() == error_code_operation_failed) {
TraceEvent(SevWarn, "RecruitTLogMatchingSetNotAvailable", self->id).error(e);
} else {
TraceEvent(SevError, "RecruitTLogsRequestError", self->id).error(e);
throw;
}
}
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
void checkOutstandingRemoteRecruitmentRequests(ClusterControllerData* self) {
for (int i = 0; i < self->outstandingRemoteRecruitmentRequests.size(); i++) {
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
RecruitRemoteFromConfigurationRequest& req = self->outstandingRemoteRecruitmentRequests[i];
try {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
RecruitRemoteFromConfigurationReply rep = self->findRemoteWorkersForConfiguration(req);
req.reply.send(rep);
swapAndPop(&self->outstandingRemoteRecruitmentRequests, i--);
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
} catch (Error& e) {
if (e.code() == error_code_no_more_servers || e.code() == error_code_operation_failed) {
TraceEvent(SevWarn, "RecruitRemoteTLogMatchingSetNotAvailable", self->id).error(e);
} else {
TraceEvent(SevError, "RecruitRemoteTLogsRequestError", self->id).error(e);
throw;
}
}
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
void checkOutstandingStorageRequests(ClusterControllerData* self) {
for (int i = 0; i < self->outstandingStorageRequests.size(); i++) {
Initial repository commit
2017-05-26 04:48:44 +08:00
auto& req = self->outstandingStorageRequests[i];
try {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (req.second < now()) {
Initial repository commit
2017-05-26 04:48:44 +08:00
req.first.reply.sendError(timed_out());
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
swapAndPop(&self->outstandingStorageRequests, i--);
Initial repository commit
2017-05-26 04:48:44 +08:00
} else {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!self->gotProcessClasses && !req.first.criticalRecruitment)
Initial repository commit
2017-05-26 04:48:44 +08:00
throw no_more_servers();
auto worker = self->getStorageWorker(req.first);
RecruitStorageReply rep;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
rep.worker = worker.interf;
rep.processClass = worker.processClass;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
req.first.reply.send(rep);
swapAndPop(&self->outstandingStorageRequests, i--);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
} catch (Error& e) {
if (e.code() == error_code_no_more_servers) {
StorageEngineSwitch:Graceful switch When fdbcli change storeType for storage engines, we switch the store type of storage servers one by one gracefully. This avoids recruiting multiple storage servers on the same process, which can cause OOM error.
2019-08-13 01:08:12 +08:00
TraceEvent(SevWarn, "RecruitStorageNotAvailable", self->id)
StorageEngineSwitch:Clean up unnecessary trace And do not trigger storage recruitment unnecessarily.
2019-08-20 04:47:48 +08:00
.suppressFor(1.0)
StorageEngineSwitch:Graceful switch When fdbcli change storeType for storage engines, we switch the store type of storage servers one by one gracefully. This avoids recruiting multiple storage servers on the same process, which can cause OOM error.
2019-08-13 01:08:12 +08:00
.detail("OutstandingReq", i)
.detail("IsCriticalRecruitment", req.first.criticalRecruitment)
.error(e);
Initial repository commit
2017-05-26 04:48:44 +08:00
} else {
TraceEvent(SevError, "RecruitStorageError", self->id).error(e);
throw;
}
}
}
}
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
void checkBetterDDOrRK(ClusterControllerData* self) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!self->masterProcessId.present() ||
self->db.serverInfo->get().recoveryState < RecoveryState::ACCEPTING_COMMITS) {
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
return;
}
Fix bug where DD or RK could be halted and re-recruited in a loop for certain valid process class configurations. Specifically, recruitment of DD or RK takes into account that master process is preferred over proxy, resolver or cc. But check for better DD only looks for better machine class ignoring that the new recruit could share a proxy or resolver or CC. Also try to balance the distribution of the DD and RK role if there are enough processes to do so.
2019-11-13 06:22:36 +08:00
std::map<Optional<Standalone<StringRef>>, int> id_used = self->getUsedIds();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
WorkerDetails newRKWorker = self->getWorkerForRoleInDatacenter(self->clusterControllerDcId,
ProcessClass::Ratekeeper,
ProcessClass::NeverAssign,
self->db.config,
id_used,
true)
.worker;
Fix bug where DD or RK could be halted and re-recruited in a loop for certain valid process class configurations. Specifically, recruitment of DD or RK takes into account that master process is preferred over proxy, resolver or cc. But check for better DD only looks for better machine class ignoring that the new recruit could share a proxy or resolver or CC. Also try to balance the distribution of the DD and RK role if there are enough processes to do so.
2019-11-13 06:22:36 +08:00
if (self->onMasterIsBetter(newRKWorker, ProcessClass::Ratekeeper)) {
newRKWorker = self->id_worker[self->masterProcessId.get()].details;
}
fix: dd test recruitment should prefer the location of ratekeeper over other used processes
2019-11-14 04:58:55 +08:00
id_used = self->getUsedIds();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& it : id_used) {
fix: dd test recruitment should prefer the location of ratekeeper over other used processes
2019-11-14 04:58:55 +08:00
it.second *= 2;
}
id_used[newRKWorker.interf.locality.processId()]++;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
WorkerDetails newDDWorker = self->getWorkerForRoleInDatacenter(self->clusterControllerDcId,
ProcessClass::DataDistributor,
ProcessClass::NeverAssign,
self->db.config,
id_used,
true)
.worker;
Fix bug where DD or RK could be halted and re-recruited in a loop for certain valid process class configurations. Specifically, recruitment of DD or RK takes into account that master process is preferred over proxy, resolver or cc. But check for better DD only looks for better machine class ignoring that the new recruit could share a proxy or resolver or CC. Also try to balance the distribution of the DD and RK role if there are enough processes to do so.
2019-11-13 06:22:36 +08:00
if (self->onMasterIsBetter(newDDWorker, ProcessClass::DataDistributor)) {
newDDWorker = self->id_worker[self->masterProcessId.get()].details;
}
auto bestFitnessForRK = newRKWorker.processClass.machineClassFitness(ProcessClass::Ratekeeper);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (self->db.config.isExcludedServer(newRKWorker.interf.addresses())) {
Fix bug where DD or RK could be halted and re-recruited in a loop for certain valid process class configurations. Specifically, recruitment of DD or RK takes into account that master process is preferred over proxy, resolver or cc. But check for better DD only looks for better machine class ignoring that the new recruit could share a proxy or resolver or CC. Also try to balance the distribution of the DD and RK role if there are enough processes to do so.
2019-11-13 06:22:36 +08:00
bestFitnessForRK = std::max(bestFitnessForRK, ProcessClass::ExcludeFit);
}
auto bestFitnessForDD = newDDWorker.processClass.machineClassFitness(ProcessClass::DataDistributor);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (self->db.config.isExcludedServer(newDDWorker.interf.addresses())) {
Fix bug where DD or RK could be halted and re-recruited in a loop for certain valid process class configurations. Specifically, recruitment of DD or RK takes into account that master process is preferred over proxy, resolver or cc. But check for better DD only looks for better machine class ignoring that the new recruit could share a proxy or resolver or CC. Also try to balance the distribution of the DD and RK role if there are enough processes to do so.
2019-11-13 06:22:36 +08:00
bestFitnessForDD = std::max(bestFitnessForDD, ProcessClass::ExcludeFit);
}
This commit tries to distribute RK and DD among other empty available processes.
2019-11-13 09:52:42 +08:00
//TraceEvent("CheckBetterDDorRKNewRecruits", self->id).detail("MasterProcessId", self->masterProcessId)
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
//.detail("NewRecruitRKProcessId", newRKWorker.interf.locality.processId()).detail("NewRecruiteDDProcessId",
// newDDWorker.interf.locality.processId());
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
Fix bug where DD or RK could be halted and re-recruited in a loop for certain valid process class configurations. Specifically, recruitment of DD or RK takes into account that master process is preferred over proxy, resolver or cc. But check for better DD only looks for better machine class ignoring that the new recruit could share a proxy or resolver or CC. Also try to balance the distribution of the DD and RK role if there are enough processes to do so.
2019-11-13 06:22:36 +08:00
Optional<Standalone<StringRef>> currentRKProcessId;
Optional<Standalone<StringRef>> currentDDProcessId;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
auto& db = self->db.serverInfo->get();
fix: dd test recruitment should prefer the location of ratekeeper over other used processes
2019-11-14 04:58:55 +08:00
bool ratekeeperHealthy = false;
once we recruit a ratekeeper, do not allow any other ratekeepers to register
2019-03-25 02:04:39 +08:00
if (db.ratekeeper.present() && self->id_worker.count(db.ratekeeper.get().locality.processId()) &&
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
(!self->recruitingRatekeeperID.present() || (self->recruitingRatekeeperID.get() == db.ratekeeper.get().id()))) {
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
auto& rkWorker = self->id_worker[db.ratekeeper.get().locality.processId()];
Fix bug where DD or RK could be halted and re-recruited in a loop for certain valid process class configurations. Specifically, recruitment of DD or RK takes into account that master process is preferred over proxy, resolver or cc. But check for better DD only looks for better machine class ignoring that the new recruit could share a proxy or resolver or CC. Also try to balance the distribution of the DD and RK role if there are enough processes to do so.
2019-11-13 06:22:36 +08:00
currentRKProcessId = rkWorker.details.interf.locality.processId();
Changes to ratekeeper camel case
2019-03-27 23:24:25 +08:00
auto rkFitness = rkWorker.details.processClass.machineClassFitness(ProcessClass::Ratekeeper);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (rkWorker.priorityInfo.isExcluded) {
Fix fitness assignment statements Found by MacOS build.
2019-03-24 10:16:04 +08:00
rkFitness = ProcessClass::ExcludeFit;
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (self->isUsedNotMaster(rkWorker.details.interf.locality.processId()) || bestFitnessForRK < rkFitness ||
(rkFitness == bestFitnessForRK && rkWorker.details.interf.locality.processId() == self->masterProcessId &&
newRKWorker.interf.locality.processId() != self->masterProcessId)) {
TraceEvent("CCHaltRK", self->id)
.detail("RKID", db.ratekeeper.get().id())
.detail("Excluded", rkWorker.priorityInfo.isExcluded)
.detail("Fitness", rkFitness)
.detail("BestFitness", bestFitnessForRK);
Fix: recruit ratekeeper is not triggerred
2019-03-24 00:20:54 +08:00
self->recruitRatekeeper.set(true);
fix: dd test recruitment should prefer the location of ratekeeper over other used processes
2019-11-14 04:58:55 +08:00
} else {
ratekeeperHealthy = true;
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!self->recruitingDistributor && db.distributor.present() &&
self->id_worker.count(db.distributor.get().locality.processId())) {
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
auto& ddWorker = self->id_worker[db.distributor.get().locality.processId()];
auto ddFitness = ddWorker.details.processClass.machineClassFitness(ProcessClass::DataDistributor);
Fix bug where DD or RK could be halted and re-recruited in a loop for certain valid process class configurations. Specifically, recruitment of DD or RK takes into account that master process is preferred over proxy, resolver or cc. But check for better DD only looks for better machine class ignoring that the new recruit could share a proxy or resolver or CC. Also try to balance the distribution of the DD and RK role if there are enough processes to do so.
2019-11-13 06:22:36 +08:00
currentDDProcessId = ddWorker.details.interf.locality.processId();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (ddWorker.priorityInfo.isExcluded) {
Fix fitness assignment statements Found by MacOS build.
2019-03-24 10:16:04 +08:00
ddFitness = ProcessClass::ExcludeFit;
the data distributor and ratekeeper are not included in id_used, but when comparing equally good options we prefer to avoid sharing with those roles excluded data distributor and ratekeeper were improperly killed when the best option was also excluded
2019-03-24 04:25:36 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (self->isUsedNotMaster(ddWorker.details.interf.locality.processId()) || bestFitnessForDD < ddFitness ||
(ddFitness == bestFitnessForDD && ddWorker.details.interf.locality.processId() == self->masterProcessId &&
newDDWorker.interf.locality.processId() != self->masterProcessId) ||
(ddFitness == bestFitnessForDD &&
newRKWorker.interf.locality.processId() != newDDWorker.interf.locality.processId() && ratekeeperHealthy &&
currentRKProcessId.present() && currentDDProcessId == currentRKProcessId &&
(newRKWorker.interf.locality.processId() != self->masterProcessId &&
newDDWorker.interf.locality.processId() != self->masterProcessId))) {
TraceEvent("CCHaltDD", self->id)
.detail("DDID", db.distributor.get().id())
.detail("Excluded", ddWorker.priorityInfo.isExcluded)
.detail("Fitness", ddFitness)
.detail("BestFitness", bestFitnessForDD)
.detail("CurrentRateKeeperProcessId",
currentRKProcessId.present() ? currentRKProcessId.get() : LiteralStringRef("None"))
.detail("CurrentDDProcessId", currentDDProcessId)
.detail("MasterProcessID", self->masterProcessId)
.detail("NewRKWorkers", newRKWorker.interf.locality.processId())
.detail("NewDDWorker", newDDWorker.interf.locality.processId());
ddWorker.haltDistributor = brokenPromiseToNever(
db.distributor.get().haltDataDistributor.getReply(HaltDataDistributorRequest(self->id)));
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
}
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> doCheckOutstandingRequests(ClusterControllerData* self) {
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
try {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
wait(delay(SERVER_KNOBS->CHECK_OUTSTANDING_INTERVAL));
while (!self->goodRecruitmentTime.isReady()) {
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
wait(self->goodRecruitmentTime);
}
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
checkOutstandingRecruitmentRequests(self);
checkOutstandingStorageRequests(self);
Move placement monitoring inside doCheckOutstandingRequests
2019-03-20 06:58:25 +08:00
checkBetterDDOrRK(self);
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
self->checkRecoveryStalled();
if (self->betterMasterExists()) {
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
self->db.forceMasterFailure.trigger();
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
TraceEvent("MasterRegistrationKill", self->id).detail("MasterId", self->db.serverInfo->get().master.id());
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} catch (Error& e) {
if (e.code() != error_code_no_more_servers) {
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
TraceEvent(SevError, "CheckOutstandingError").error(e);
}
}
return Void();
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> doCheckOutstandingRemoteRequests(ClusterControllerData* self) {
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
try {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
wait(delay(SERVER_KNOBS->CHECK_OUTSTANDING_INTERVAL));
while (!self->goodRemoteRecruitmentTime.isReady()) {
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
wait(self->goodRemoteRecruitmentTime);
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
checkOutstandingRemoteRecruitmentRequests(self);
} catch (Error& e) {
if (e.code() != error_code_no_more_servers) {
added a fallback satellite configuration, so that we can use two satellites if available, but do not have to failover to the remote datacenter if one satellite is down
2018-06-29 14:15:32 +08:00
TraceEvent(SevError, "CheckOutstandingError").error(e);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
return Void();
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
void checkOutstandingRequests(ClusterControllerData* self) {
if (self->outstandingRemoteRequestChecker.isReady()) {
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
self->outstandingRemoteRequestChecker = doCheckOutstandingRemoteRequests(self);
}
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (self->outstandingRequestChecker.isReady()) {
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
self->outstandingRequestChecker = doCheckOutstandingRequests(self);
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> rebootAndCheck(ClusterControllerData* cluster, Optional<Standalone<StringRef>> processID) {
Fix minor IDE build errors
2019-02-18 10:46:59 +08:00
{
auto watcher = cluster->id_worker.find(processID);
ASSERT(watcher != cluster->id_worker.end());
Initial repository commit
2017-05-26 04:48:44 +08:00
Fix minor IDE build errors
2019-02-18 10:46:59 +08:00
watcher->second.reboots++;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
wait(delay(g_network->isSimulated() ? SERVER_KNOBS->SIM_SHUTDOWN_TIMEOUT : SERVER_KNOBS->SHUTDOWN_TIMEOUT));
Fix minor IDE build errors
2019-02-18 10:46:59 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
Fix minor IDE build errors
2019-02-18 10:46:59 +08:00
{
auto watcher = cluster->id_worker.find(processID);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (watcher != cluster->id_worker.end()) {
Fix minor IDE build errors
2019-02-18 10:46:59 +08:00
watcher->second.reboots--;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (watcher->second.reboots < 2)
checkOutstandingRequests(cluster);
Fix minor IDE build errors
2019-02-18 10:46:59 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
return Void();
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> workerAvailabilityWatch(WorkerInterface worker,
ProcessClass startingClass,
ClusterControllerData* cluster) {
run clang-format on changes
2019-05-30 04:43:21 +08:00
state Future<Void> failed =
(worker.address() == g_network->getLocalAddress() || startingClass.classType() == ProcessClass::TesterClass)
? Never()
: waitFailureClient(worker.waitFailure, SERVER_KNOBS->WORKER_FAILURE_TIME);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
cluster->updateWorkerList.set(worker.locality.processId(),
ProcessData(worker.locality, startingClass, worker.stableAddress()));
code cleanup based on review comments
2020-04-18 06:05:01 +08:00
cluster->updateDBInfoEndpoints.insert(worker.updateServerDBInfo.getEndpoint());
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
cluster->updateDBInfo.trigger();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
// This switching avoids a race where the worker can be added to id_worker map after the workerAvailabilityWatch
// fails for the worker.
Re-enable CheckDesiredClasses after making necessary changes for multi-region setup. Fixed a couple of bugs 1) A rare race condition where a worker is being roles even after it died. 2) Fix how RoleFitness is calculated for TLog and LogRouter. Only worst fitness is compared to see if a better fit is available.
2019-01-11 02:28:32 +08:00
wait(delay(0));
Initial repository commit
2017-05-26 04:48:44 +08:00
loop {
choose {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(wait(IFailureMonitor::failureMonitor().onStateEqual(
worker.storage.getEndpoint(),
FailureStatus(
IFailureMonitor::failureMonitor().getState(worker.storage.getEndpoint()).isAvailable())))) {
if (IFailureMonitor::failureMonitor().getState(worker.storage.getEndpoint()).isAvailable()) {
cluster->ac.add(rebootAndCheck(cluster, worker.locality.processId()));
checkOutstandingRequests(cluster);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(wait(failed)) { // remove workers that have failed
WorkerInfo& failedWorkerInfo = cluster->id_worker[worker.locality.processId()];
Register caches through keyspace This also removes the old mechanism that registers them through the serverDBInfo. Caches do now self-recruit at startup
2019-12-07 05:28:44 +08:00
Add initialClass to RegisterWorkerRequest
2017-10-05 08:11:12 +08:00
if (!failedWorkerInfo.reply.isSet()) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
failedWorkerInfo.reply.send(
RegisterWorkerReply(failedWorkerInfo.details.processClass, failedWorkerInfo.priorityInfo));
Add initialClass to RegisterWorkerRequest
2017-10-05 08:11:12 +08:00
}
Fix a race of DD, RK, Master failure If all DD, RK, Master run on the same process and failed. Recruiting of new DD or RK could try to use the old master worker interface, which is an invalid one and causes recruitment to be stuck. Fix by adding a delay and checking master is valid before recruitment.
2019-03-21 07:03:36 +08:00
if (worker.locality.processId() == cluster->masterProcessId) {
cluster->masterProcessId = Optional<Key>();
}
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
cluster->removedDBInfoEndpoints.insert(worker.updateServerDBInfo.getEndpoint());
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
cluster->id_worker.erase(worker.locality.processId());
cluster->updateWorkerList.set(worker.locality.processId(), Optional<ProcessData>());
Initial repository commit
2017-05-26 04:48:44 +08:00
return Void();
}
}
}
}
struct FailureStatusInfo {
FailureStatus status;
double lastRequestTime;
double penultimateRequestTime;
FailureStatusInfo() : lastRequestTime(0), penultimateRequestTime(0) {}
void insertRequest(double now) {
penultimateRequestTime = lastRequestTime;
lastRequestTime = now;
}
double latency(double now) const {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
return std::max(now - lastRequestTime, lastRequestTime - penultimateRequestTime);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
};
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<vector<TLogInterface>> requireAll(vector<Future<Optional<vector<TLogInterface>>>> in) {
Initial repository commit
2017-05-26 04:48:44 +08:00
state vector<TLogInterface> out;
state int i;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (i = 0; i < in.size(); i++) {
Initial repository commit
2017-05-26 04:48:44 +08:00
Optional<vector<TLogInterface>> x = wait(in[i]);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!x.present())
throw recruitment_failed();
Initial repository commit
2017-05-26 04:48:44 +08:00
out.insert(out.end(), x.get().begin(), x.get().end());
}
return out;
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
void clusterRecruitStorage(ClusterControllerData* self, RecruitStorageRequest req) {
Initial repository commit
2017-05-26 04:48:44 +08:00
try {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!self->gotProcessClasses && !req.criticalRecruitment)
Initial repository commit
2017-05-26 04:48:44 +08:00
throw no_more_servers();
auto worker = self->getStorageWorker(req);
RecruitStorageReply rep;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
rep.worker = worker.interf;
rep.processClass = worker.processClass;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
req.reply.send(rep);
} catch (Error& e) {
Initial repository commit
2017-05-26 04:48:44 +08:00
if (e.code() == error_code_no_more_servers) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
self->outstandingStorageRequests.push_back(std::make_pair(req, now() + SERVER_KNOBS->RECRUITMENT_TIMEOUT));
StorageEngineSwitch:Graceful switch When fdbcli change storeType for storage engines, we switch the store type of storage servers one by one gracefully. This avoids recruiting multiple storage servers on the same process, which can cause OOM error.
2019-08-13 01:08:12 +08:00
TraceEvent(SevWarn, "RecruitStorageNotAvailable", self->id)
.detail("IsCriticalRecruitment", req.criticalRecruitment)
.error(e);
Initial repository commit
2017-05-26 04:48:44 +08:00
} else {
TraceEvent(SevError, "RecruitStorageError", self->id).error(e);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
throw; // Any other error will bring down the cluster controller
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> clusterRecruitFromConfiguration(ClusterControllerData* self, RecruitFromConfigurationRequest req) {
Initial repository commit
2017-05-26 04:48:44 +08:00
// At the moment this doesn't really need to be an actor (it always completes immediately)
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
TEST(true); // ClusterController RecruitTLogsRequest
Initial repository commit
2017-05-26 04:48:44 +08:00
loop {
try {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
auto rep = self->findWorkersForConfiguration(req);
req.reply.send(rep);
Initial repository commit
2017-05-26 04:48:44 +08:00
return Void();
} catch (Error& e) {
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
if (e.code() == error_code_no_more_servers && self->goodRecruitmentTime.isReady()) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
self->outstandingRecruitmentRequests.push_back(req);
Initial repository commit
2017-05-26 04:48:44 +08:00
TraceEvent(SevWarn, "RecruitFromConfigurationNotAvailable", self->id).error(e);
return Void();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} else if (e.code() == error_code_operation_failed || e.code() == error_code_no_more_servers) {
// recruitment not good enough, try again
} else {
Initial repository commit
2017-05-26 04:48:44 +08:00
TraceEvent(SevError, "RecruitFromConfigurationError", self->id).error(e);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
throw; // goodbye, cluster controller
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
wait(lowPriorityDelay(SERVER_KNOBS->ATTEMPT_RECRUITMENT_DELAY));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> clusterRecruitRemoteFromConfiguration(ClusterControllerData* self,
RecruitRemoteFromConfigurationRequest req) {
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
// At the moment this doesn't really need to be an actor (it always completes immediately)
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
TEST(true); // ClusterController RecruitTLogsRequest Remote
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
loop {
try {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
RecruitRemoteFromConfigurationReply rep = self->findRemoteWorkersForConfiguration(req);
req.reply.send(rep);
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
return Void();
} catch (Error& e) {
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
if (e.code() == error_code_no_more_servers && self->goodRemoteRecruitmentTime.isReady()) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
self->outstandingRemoteRecruitmentRequests.push_back(req);
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
TraceEvent(SevWarn, "RecruitRemoteFromConfigurationNotAvailable", self->id).error(e);
return Void();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} else if (e.code() == error_code_operation_failed || e.code() == error_code_no_more_servers) {
// recruitment not good enough, try again
} else {
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
TraceEvent(SevError, "RecruitRemoteFromConfigurationError", self->id).error(e);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
throw; // goodbye, cluster controller
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
wait(lowPriorityDelay(SERVER_KNOBS->ATTEMPT_RECRUITMENT_DELAY));
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
void clusterRegisterMaster(ClusterControllerData* self, RegisterMasterRequest const& req) {
req.reply.send(Void());
Initial repository commit
2017-05-26 04:48:44 +08:00
Fix oldest backup epoch for backup workers The oldest backup epoch is piggybacked in LogSystemConfig from master to cluster controller and then to all workers. Previously, this epoch is set to the current master epoch, which is wrong.
2020-03-18 05:45:07 +08:00
TraceEvent("MasterRegistrationReceived", self->id)
.detail("MasterId", req.id)
.detail("Master", req.mi.toString())
.detail("Tlogs", describe(req.logSystemConfig.tLogs))
.detail("Resolvers", req.resolvers.size())
.detail("RecoveryState", (int)req.recoveryState)
.detail("RegistrationCount", req.registrationCount)
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
.detail("CommitProxies", req.commitProxies.size())
Fixed formatting issues and removed GRV related code in MasterProxy
2020-08-12 09:54:54 +08:00
.detail("GrvProxies", req.grvProxies.size())
Fix oldest backup epoch for backup workers The oldest backup epoch is piggybacked in LogSystemConfig from master to cluster controller and then to all workers. Previously, this epoch is set to the current master epoch, which is wrong.
2020-03-18 05:45:07 +08:00
.detail("RecoveryCount", req.recoveryCount)
.detail("Stalled", req.recoveryStalled)
.detail("OldestBackupEpoch", req.logSystemConfig.oldestBackupEpoch);
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
// make sure the request comes from an active database
Initial repository commit
2017-05-26 04:48:44 +08:00
auto db = &self->db;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (db->serverInfo->get().master.id() != req.id || req.registrationCount <= db->masterRegistrationCount) {
TraceEvent("MasterRegistrationNotFound", self->id)
.detail("MasterId", req.id)
.detail("ExistingId", db->serverInfo->get().master.id())
.detail("RegCount", req.registrationCount)
.detail("ExistingRegCount", db->masterRegistrationCount);
Initial repository commit
2017-05-26 04:48:44 +08:00
return;
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (req.recoveryState == RecoveryState::FULLY_RECOVERED) {
The cluster controller will increase its failure monitoring delay after there have been many unfinishedRecoveries
2018-09-01 01:51:55 +08:00
self->db.unfinishedRecoveries = 0;
self->db.logGenerations = 0;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ASSERT(!req.logSystemConfig.oldTLogs.size());
The cluster controller will increase its failure monitoring delay after there have been many unfinishedRecoveries
2018-09-01 01:51:55 +08:00
} else {
self->db.logGenerations = std::max<int>(self->db.logGenerations, req.logSystemConfig.oldTLogs.size());
}
Initial repository commit
2017-05-26 04:48:44 +08:00
db->masterRegistrationCount = req.registrationCount;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
db->recoveryStalled = req.recoveryStalled;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (req.configuration.present()) {
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
db->config = req.configuration.get();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (req.recoveryState >= RecoveryState::ACCEPTING_COMMITS) {
Add fullyRecoveredConfig for checking exclusions
2017-11-16 09:15:24 +08:00
self->gotFullyRecoveredConfig = true;
db->fullyRecoveredConfig = req.configuration.get();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& it : self->id_worker) {
bool isExcludedFromConfig =
db->fullyRecoveredConfig.isExcludedServer(it.second.details.interf.addresses());
if (it.second.priorityInfo.isExcluded != isExcludedFromConfig) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
it.second.priorityInfo.isExcluded = isExcludedFromConfig;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!it.second.reply.isSet()) {
it.second.reply.send(
RegisterWorkerReply(it.second.details.processClass, it.second.priorityInfo));
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
Add fullyRecoveredConfig for checking exclusions
2017-11-16 09:15:24 +08:00
}
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
}
}
}
Initial repository commit
2017-05-26 04:48:44 +08:00
bool isChanged = false;
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
auto dbInfo = self->db.serverInfo->get();
Initial repository commit
2017-05-26 04:48:44 +08:00
if (dbInfo.recoveryState != req.recoveryState) {
dbInfo.recoveryState = req.recoveryState;
isChanged = true;
}
if (dbInfo.priorCommittedLogServers != req.priorCommittedLogServers) {
dbInfo.priorCommittedLogServers = req.priorCommittedLogServers;
isChanged = true;
}
// Construct the client information
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
if (db->clientInfo->get().commitProxies != req.commitProxies ||
db->clientInfo->get().grvProxies != req.grvProxies) {
Initial repository commit
2017-05-26 04:48:44 +08:00
isChanged = true;
ClientDBInfo clientInfo;
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
clientInfo.id = deterministicRandom()->randomUniqueID();
Rename proxy to commit proxy
2020-09-11 08:44:15 +08:00
clientInfo.commitProxies = req.commitProxies;
Pass watches test
2020-07-15 15:37:41 +08:00
clientInfo.grvProxies = req.grvProxies;
Initial repository commit
2017-05-26 04:48:44 +08:00
clientInfo.clientTxnInfoSampleRate = db->clientInfo->get().clientTxnInfoSampleRate;
clientInfo.clientTxnInfoSizeLimit = db->clientInfo->get().clientTxnInfoSizeLimit;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
db->clientInfo->set(clientInfo);
Initial repository commit
2017-05-26 04:48:44 +08:00
dbInfo.client = db->clientInfo->get();
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!dbInfo.logSystemConfig.isEqual(req.logSystemConfig)) {
Initial repository commit
2017-05-26 04:48:44 +08:00
isChanged = true;
dbInfo.logSystemConfig = req.logSystemConfig;
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (dbInfo.resolvers != req.resolvers) {
Initial repository commit
2017-05-26 04:48:44 +08:00
isChanged = true;
dbInfo.resolvers = req.resolvers;
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (dbInfo.recoveryCount != req.recoveryCount) {
Initial repository commit
2017-05-26 04:48:44 +08:00
isChanged = true;
dbInfo.recoveryCount = req.recoveryCount;
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (isChanged) {
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
dbInfo.id = deterministicRandom()->randomUniqueID();
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
dbInfo.infoGeneration = ++self->db.dbInfoCount;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
self->db.serverInfo->set(dbInfo);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
checking outstanding requests can be CPU intensive, so rate limit checking requests
2018-06-28 14:02:08 +08:00
checkOutstandingRequests(self);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
void registerWorker(RegisterWorkerRequest req, ClusterControllerData* self) {
Restore master PID in worker registration This fix is lost during merge.
2019-03-24 11:55:03 +08:00
const WorkerInterface& w = req.wi;
Introduce prevChangeID to CandidacyRequest and LeaderHeartbeatRequest
2017-10-13 08:11:58 +08:00
ProcessClass newProcessClass = req.processClass;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
auto info = self->id_worker.find(w.locality.processId());
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
ClusterControllerPriorityInfo newPriorityInfo = req.priorityInfo;
fix: processClassFitness could be wrong if the client changed their class while rebooting
2019-03-22 08:56:04 +08:00
newPriorityInfo.processClassFitness = newProcessClass.machineClassFitness(ProcessClass::ClusterController);
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto it : req.incompatiblePeers) {
fix compilation errors
2020-04-07 11:58:43 +08:00
self->db.incompatibleConnections[it] = now() + SERVER_KNOBS->INCOMPATIBLE_PEERS_LOGGING_INTERVAL;
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
}
many bug fixes
2020-04-11 04:45:16 +08:00
self->removedDBInfoEndpoints.erase(w.updateServerDBInfo.getEndpoint());
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (info == self->id_worker.end()) {
TraceEvent("ClusterControllerActualWorkers", self->id)
.detail("WorkerId", w.id())
.detail("ProcessId", w.locality.processId())
.detail("ZoneId", w.locality.zoneId())
.detail("DataHall", w.locality.dataHallId())
.detail("PClass", req.processClass.toString())
.detail("Workers", self->id_worker.size());
The cluster controller waits until no new workers register for 1.0 before starting a bad recruitment
2020-02-20 08:48:30 +08:00
self->goodRecruitmentTime = lowPriorityDelay(SERVER_KNOBS->WAIT_FOR_GOOD_RECRUITMENT_DELAY);
updateGoodRemoteRecruitmentTime was unnecessary because the only way findRemoteWorkers would return would be after a new server has joined which already resets goodRemoteRecruitmentTime
2020-02-21 05:46:22 +08:00
self->goodRemoteRecruitmentTime = lowPriorityDelay(SERVER_KNOBS->WAIT_FOR_GOOD_REMOTE_RECRUITMENT_DELAY);
suppressed spammy trace events, and avoid reporting a long master recovery duration when the cluster is first created
2018-11-05 15:07:56 +08:00
} else {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
TraceEvent("ClusterControllerWorkerAlreadyRegistered", self->id)
.suppressFor(1.0)
.detail("WorkerId", w.id())
.detail("ProcessId", w.locality.processId())
.detail("ZoneId", w.locality.zoneId())
.detail("DataHall", w.locality.dataHallId())
.detail("PClass", req.processClass.toString())
.detail("Workers", self->id_worker.size());
}
if (w.address() == g_network->getLocalAddress()) {
if (self->changingDcIds.get().first) {
if (self->changingDcIds.get().second.present()) {
newPriorityInfo.dcFitness = ClusterControllerPriorityInfo::calculateDCFitness(
w.locality.dcId(), self->changingDcIds.get().second.get());
}
} else if (self->changedDcIds.get().second.present()) {
newPriorityInfo.dcFitness = ClusterControllerPriorityInfo::calculateDCFitness(
w.locality.dcId(), self->changedDcIds.get().second.get());
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
}
} else {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!self->changingDcIds.get().first) {
if (self->changingDcIds.get().second.present()) {
newPriorityInfo.dcFitness = ClusterControllerPriorityInfo::calculateDCFitness(
w.locality.dcId(), self->changingDcIds.get().second.get());
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} else if (self->changedDcIds.get().second.present()) {
newPriorityInfo.dcFitness = ClusterControllerPriorityInfo::calculateDCFitness(
w.locality.dcId(), self->changedDcIds.get().second.get());
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
Use processId instead of address for comparison
2017-10-26 02:35:29 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
// Check process class and exclusive property
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (info == self->id_worker.end() || info->second.details.interf.id() != w.id() ||
req.generation >= info->second.gen) {
if (self->gotProcessClasses) {
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
auto classIter = self->id_class.find(w.locality.processId());
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (classIter != self->id_class.end() && (classIter->second.classSource() == ProcessClass::DBSource ||
req.initialClass.classType() == ProcessClass::UnsetClass)) {
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
newProcessClass = classIter->second;
} else {
newProcessClass = req.initialClass;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
newPriorityInfo.processClassFitness = newProcessClass.machineClassFitness(ProcessClass::ClusterController);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (self->gotFullyRecoveredConfig) {
many bug fixes
2020-04-11 04:45:16 +08:00
newPriorityInfo.isExcluded = self->db.fullyRecoveredConfig.isExcludedServer(w.addresses());
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Introduce prevChangeID to CandidacyRequest and LeaderHeartbeatRequest
2017-10-13 08:11:58 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (info == self->id_worker.end()) {
self->id_worker[w.locality.processId()] = WorkerInfo(workerAvailabilityWatch(w, newProcessClass, self),
req.reply,
req.generation,
w,
req.initialClass,
newProcessClass,
newPriorityInfo,
req.degraded,
req.issues);
if (!self->masterProcessId.present() &&
w.locality.processId() == self->db.serverInfo->get().master.locality.processId()) {
Restore master PID in worker registration This fix is lost during merge.
2019-03-24 11:55:03 +08:00
self->masterProcessId = w.locality.processId();
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
checkOutstandingRequests(self);
} else if (info->second.details.interf.id() != w.id() || req.generation >= info->second.gen) {
Catch and update processClass change from DBSource
2017-09-26 01:36:03 +08:00
if (!info->second.reply.isSet()) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
info->second.reply.send(Never());
Initial repository commit
2017-05-26 04:48:44 +08:00
}
info->second.reply = req.reply;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
info->second.details.processClass = newProcessClass;
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
info->second.priorityInfo = newPriorityInfo;
Change excluded cluster controller
2017-11-15 05:57:37 +08:00
info->second.initialClass = req.initialClass;
tlog recruitment will prefer non-degraded processes, however it will not choose less than desired number of tlogs to avoid degraded processes better master exists will switch the master to avoid degraded processes
2019-03-09 03:40:00 +08:00
info->second.details.degraded = req.degraded;
Initial repository commit
2017-05-26 04:48:44 +08:00
info->second.gen = req.generation;
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
info->second.issues = req.issues;
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (info->second.details.interf.id() != w.id()) {
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
self->removedDBInfoEndpoints.insert(info->second.details.interf.updateServerDBInfo.getEndpoint());
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
info->second.details.interf = w;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
info->second.watcher = workerAvailabilityWatch(w, newProcessClass, self);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
checkOutstandingRequests(self);
make sure id_worker is updated before setting ratekeeper or data distribution
2019-03-23 08:08:54 +08:00
} else {
TEST(true); // Received an old worker registration request.
Initial repository commit
2017-05-26 04:48:44 +08:00
}
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
if (req.distributorInterf.present() && !self->db.serverInfo->get().distributor.present() &&
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
self->clusterControllerDcId == req.distributorInterf.get().locality.dcId() && !self->recruitingDistributor) {
Piggyback data distributor interface in worker registration This allows cluster controller to know data distributor during worker registration phase, thus avoiding recruiting a new data distributor after starting. Also change the worker to skip creating a new data distributor if there is already one running on the worker, which can trigger operation timeout in tests.
2019-01-29 03:29:39 +08:00
const DataDistributorInterface& di = req.distributorInterf.get();
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCRegisterDataDistributor", self->id).detail("DDID", di.id());
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
self->db.setDistributor(di);
Piggyback data distributor interface in worker registration This allows cluster controller to know data distributor during worker registration phase, thus avoiding recruiting a new data distributor after starting. Also change the worker to skip creating a new data distributor if there is already one running on the worker, which can trigger operation timeout in tests.
2019-01-29 03:29:39 +08:00
}
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
if (req.ratekeeperInterf.present()) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if ((self->recruitingRatekeeperID.present() &&
self->recruitingRatekeeperID.get() != req.ratekeeperInterf.get().id()) ||
self->clusterControllerDcId != w.locality.dcId()) {
Recruit backup workers for old epochs If there are unfinished ranges in the old epochs, the new master will recruit backup workers responsible for finishing these ranges. These workers remains in the cluster until the next epoch, when it will remove itself.
2019-07-30 01:37:42 +08:00
TraceEvent("CCHaltRegisteringRatekeeper", self->id)
.detail("RKID", req.ratekeeperInterf.get().id())
.detail("DcID", printable(self->clusterControllerDcId))
.detail("ReqDcID", printable(w.locality.dcId()))
.detail("RecruitingRKID",
self->recruitingRatekeeperID.present() ? self->recruitingRatekeeperID.get() : UID());
Add backup role, class, and worker skeleton
2019-04-25 06:12:37 +08:00
self->id_worker[w.locality.processId()].haltRatekeeper = brokenPromiseToNever(
req.ratekeeperInterf.get().haltRatekeeper.getReply(HaltRatekeeperRequest(self->id)));
Recruit backup workers for old epochs If there are unfinished ranges in the old epochs, the new master will recruit backup workers responsible for finishing these ranges. These workers remains in the cluster until the next epoch, when it will remove itself.
2019-07-30 01:37:42 +08:00
} else if (!self->recruitingRatekeeperID.present()) {
Fix Ratekeeper and DataDistributor placement Make sure both RateKeeper and DataDistributor are placed in the same data center as the Master. Make sure only one RateKeeper is live in the cluster as well.
2019-03-16 08:06:15 +08:00
const RatekeeperInterface& rki = req.ratekeeperInterf.get();
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
const auto& ratekeeper = self->db.serverInfo->get().ratekeeper;
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCRegisterRatekeeper", self->id).detail("RKID", rki.id());
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (ratekeeper.present() && ratekeeper.get().id() != rki.id() &&
self->id_worker.count(ratekeeper.get().locality.processId())) {
TraceEvent("CCHaltPreviousRatekeeper", self->id)
.detail("RKID", ratekeeper.get().id())
.detail("DcID", printable(self->clusterControllerDcId))
.detail("ReqDcID", printable(w.locality.dcId()))
.detail("RecruitingRKID",
self->recruitingRatekeeperID.present() ? self->recruitingRatekeeperID.get() : UID());
self->id_worker[ratekeeper.get().locality.processId()].haltRatekeeper =
brokenPromiseToNever(ratekeeper.get().haltRatekeeper.getReply(HaltRatekeeperRequest(self->id)));
}
if (!ratekeeper.present() || ratekeeper.get().id() != rki.id()) {
startRatekeeper is responsible for updating serverDBInfo
2019-03-23 08:56:16 +08:00
self->db.setRatekeeper(rki);
Fix ratekeeper recruitment bug
2019-03-21 04:54:15 +08:00
}
Fix Ratekeeper and DataDistributor placement Make sure both RateKeeper and DataDistributor are placed in the same data center as the Master. Make sure only one RateKeeper is live in the cluster as well.
2019-03-16 08:06:15 +08:00
}
Piggyback data distributor interface in worker registration This allows cluster controller to know data distributor during worker registration phase, thus avoiding recruiting a new data distributor after starting. Also change the worker to skip creating a new data distributor if there is already one running on the worker, which can trigger operation timeout in tests.
2019-01-29 03:29:39 +08:00
}
Merging with Master.
2019-11-13 05:01:29 +08:00
// Notify the worker to register again with new process class/exclusive property
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!req.reply.isSet() && newPriorityInfo != req.priorityInfo) {
req.reply.send(RegisterWorkerReply(newProcessClass, newPriorityInfo));
Merging with Master.
2019-11-13 05:01:29 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Fixing PR comments <rdar://problem/34557380> Need a way to map real time to version
2017-09-28 07:31:38 +08:00
#define TIME_KEEPER_VERSION LiteralStringRef("1")
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> timeKeeperSetVersion(ClusterControllerData* self) {
Add uses of makeReference
2020-11-07 15:50:55 +08:00
state Reference<ReadYourWritesTransaction> tr = makeReference<ReadYourWritesTransaction>(self->cx);
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
loop {
try {
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr->setOption(FDBTransactionOptions::LOCK_AWARE);
make timekeeper priority system immediate
2018-01-09 10:21:00 +08:00
tr->setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
tr->set(timeKeeperVersionKey, TIME_KEEPER_VERSION);
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr->commit());
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
break;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} catch (Error& e) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr->onError(e));
Fixing PR comments <rdar://problem/34557380> Need a way to map real time to version
2017-09-28 07:31:38 +08:00
}
}
return Void();
}
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
// This actor periodically gets read version and writes it to cluster with current timestamp as key. To avoid running
// out of space, it limits the max number of entries and clears old entries on each update. This mapping is used from
// backup and restore to get the version information for a timestamp.
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> timeKeeper(ClusterControllerData* self) {
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
state KeyBackedMap<int64_t, Version> versionMap(timeKeeperPrefixRange.begin);
call debug transaction on timekeeper
2018-11-03 03:56:29 +08:00
TraceEvent("TimeKeeperStarted");
Fixing PR comments <rdar://problem/34557380> Need a way to map real time to version
2017-09-28 07:31:38 +08:00
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(timeKeeperSetVersion(self));
Fixing PR comments <rdar://problem/34557380> Need a way to map real time to version
2017-09-28 07:31:38 +08:00
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
loop {
Add uses of makeReference
2020-11-07 15:50:55 +08:00
state Reference<ReadYourWritesTransaction> tr = makeReference<ReadYourWritesTransaction>(self->cx);
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
loop {
try {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!g_network->isSimulated()) {
addressed review comments
2018-11-05 12:26:23 +08:00
// This is done to provide an arbitrary logged transaction every ~10s.
// FIXME: replace or augment this with logging on the proxy which tracks
// how long it is taking to hear responses from each other component.
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
UID debugID = deterministicRandom()->randomUniqueID();
Renamed TimeKeeperStarted to TimeKeeperCommit
2018-11-03 03:57:03 +08:00
TraceEvent("TimeKeeperCommit", debugID);
call debug transaction on timekeeper
2018-11-03 03:56:29 +08:00
tr->debugTransaction(debugID);
}
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr->setOption(FDBTransactionOptions::LOCK_AWARE);
make timekeeper priority system immediate
2018-01-09 10:21:00 +08:00
tr->setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
Optional<Value> disableValue = wait(tr->get(timeKeeperDisableKey));
if (disableValue.present()) {
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
break;
}
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
Version v = tr->getReadVersion().get();
int64_t currentTime = (int64_t)now();
versionMap.set(tr, currentTime, v);
added the ability to disable timeKeeper; disabled timeKeeper before consistency check in simulation
2017-09-29 04:13:24 +08:00
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
int64_t ttl = currentTime - SERVER_KNOBS->TIME_KEEPER_DELAY * SERVER_KNOBS->TIME_KEEPER_MAX_ENTRIES;
if (ttl > 0) {
versionMap.erase(tr, 0, ttl);
added the ability to disable timeKeeper; disabled timeKeeper before consistency check in simulation
2017-09-29 04:13:24 +08:00
}
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr->commit());
Don't ignore TimeKeeper failures.
2017-10-19 05:31:31 +08:00
break;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} catch (Error& e) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr->onError(e));
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
}
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(delay(SERVER_KNOBS->TIME_KEEPER_DELAY));
<rdar://problem/34557380> Need a way to map real time to version
2017-09-26 03:40:24 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> statusServer(FutureStream<StatusRequest> requests,
ClusterControllerData* self,
ServerCoordinators coordinators) {
Initial repository commit
2017-05-26 04:48:44 +08:00
// Seconds since the END of the last GetStatus executed
state double last_request_time = 0.0;
// Place to accumulate a batch of requests to respond to
state std::vector<StatusRequest> requests_batch;
loop {
try {
// Wait til first request is ready
StatusRequest req = waitNext(requests);
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
++self->statusRequests;
Initial repository commit
2017-05-26 04:48:44 +08:00
requests_batch.push_back(req);
// Earliest time at which we may begin a new request
double next_allowed_request_time = last_request_time + SERVER_KNOBS->STATUS_MIN_TIME_BETWEEN_REQUESTS;
// Wait if needed to satisfy min_time knob, also allows more requets to queue up.
double minwait = std::max(next_allowed_request_time - now(), 0.0);
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(delay(minwait));
Initial repository commit
2017-05-26 04:48:44 +08:00
// Get all requests that are ready right *now*, before GetStatus() begins.
// All of these requests will be responded to with the next GetStatus() result.
Throttle status requests with MAX_STATUS_REQUESTS_PER_SECOND knob, whenever status batching is used.
2019-01-29 07:37:30 +08:00
// If requests are batched, do not respond to more than MAX_STATUS_REQUESTS_PER_SECOND
// requests per second
while (requests.isReady()) {
auto req = requests.pop();
if (SERVER_KNOBS->STATUS_MIN_TIME_BETWEEN_REQUESTS > 0.0 &&
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
requests_batch.size() + 1 >
SERVER_KNOBS->STATUS_MIN_TIME_BETWEEN_REQUESTS * SERVER_KNOBS->MAX_STATUS_REQUESTS_PER_SECOND) {
TraceEvent(SevWarnAlways, "TooManyStatusRequests")
.suppressFor(1.0)
.detail("BatchSize", requests_batch.size());
Throttle status requests with MAX_STATUS_REQUESTS_PER_SECOND knob, whenever status batching is used.
2019-01-29 07:37:30 +08:00
req.reply.sendError(server_overloaded());
} else {
requests_batch.push_back(req);
}
}
Initial repository commit
2017-05-26 04:48:44 +08:00
// Get status but trap errors to send back to client.
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
vector<WorkerDetails> workers;
code cleanup based on review comments
2020-04-18 06:05:01 +08:00
std::vector<ProcessIssues> workerIssues;
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& it : self->id_worker) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
workers.push_back(it.second.details);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (it.second.issues.size()) {
code cleanup based on review comments
2020-04-18 06:05:01 +08:00
workerIssues.push_back(ProcessIssues(it.second.details.interf.address(), it.second.issues));
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
}
}
Initial repository commit
2017-05-26 04:48:44 +08:00
std::vector<NetworkAddress> incompatibleConnections;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto it = self->db.incompatibleConnections.begin(); it != self->db.incompatibleConnections.end();) {
if (it->second < now()) {
Initial repository commit
2017-05-26 04:48:44 +08:00
it = self->db.incompatibleConnections.erase(it);
} else {
incompatibleConnections.push_back(it->first);
it++;
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
state ErrorOr<StatusReply> result = wait(errorOr(clusterGetStatus(self->db.serverInfo,
self->cx,
workers,
workerIssues,
&self->db.clientStatus,
coordinators,
incompatibleConnections,
self->datacenterVersionDifference)));
Status: TLS client status Use ClientStatusInfo structure for each network address (client), instead of passing each status info as a parameter.
2019-02-27 08:20:05 +08:00
Initial repository commit
2017-05-26 04:48:44 +08:00
if (result.isError() && result.getError().code() == error_code_actor_cancelled)
throw result.getError();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
// Update last_request_time now because GetStatus is finished and the delay is to be measured between
// requests
Initial repository commit
2017-05-26 04:48:44 +08:00
last_request_time = now();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
while (!requests_batch.empty()) {
Initial repository commit
2017-05-26 04:48:44 +08:00
if (result.isError())
requests_batch.back().reply.sendError(result.getError());
else
requests_batch.back().reply.send(result.get());
requests_batch.pop_back();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
wait(yield());
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} catch (Error& e) {
Initial repository commit
2017-05-26 04:48:44 +08:00
TraceEvent(SevError, "StatusServerError").error(e);
throw e;
}
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> monitorProcessClasses(ClusterControllerData* self) {
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
state ReadYourWritesTransaction trVer(self->db.db);
Initial repository commit
2017-05-26 04:48:44 +08:00
loop {
try {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
trVer.setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
trVer.setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
Initial repository commit
2017-05-26 04:48:44 +08:00
Optional<Value> val = wait(trVer.get(processClassVersionKey));
if (val.present())
break;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
Standalone<RangeResultRef> processClasses = wait(trVer.getRange(processClassKeys, CLIENT_KNOBS->TOO_MANY));
ASSERT(!processClasses.more && processClasses.size() < CLIENT_KNOBS->TOO_MANY);
Initial repository commit
2017-05-26 04:48:44 +08:00
trVer.clear(processClassKeys);
trVer.set(processClassVersionKey, processClassVersionValue);
for (auto it : processClasses) {
UID processUid = decodeProcessClassKeyOld(it.key);
trVer.set(processClassKeyFor(processUid.toString()), it.value);
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(trVer.commit());
Initial repository commit
2017-05-26 04:48:44 +08:00
TraceEvent("ProcessClassUpgrade");
break;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} catch (Error& e) {
wait(trVer.onError(e));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
loop {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
state ReadYourWritesTransaction tr(self->db.db);
Initial repository commit
2017-05-26 04:48:44 +08:00
loop {
try {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
tr.setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr.setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
Standalone<RangeResultRef> processClasses = wait(tr.getRange(processClassKeys, CLIENT_KNOBS->TOO_MANY));
ASSERT(!processClasses.more && processClasses.size() < CLIENT_KNOBS->TOO_MANY);
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (processClasses != self->lastProcessClasses || !self->gotProcessClasses) {
Initial repository commit
2017-05-26 04:48:44 +08:00
self->id_class.clear();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (int i = 0; i < processClasses.size(); i++) {
auto c = decodeProcessClassValue(processClasses[i].value);
ASSERT(c.classSource() != ProcessClass::CommandLineSource);
self->id_class[decodeProcessClassKey(processClasses[i].key)] = c;
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& w : self->id_worker) {
Initial repository commit
2017-05-26 04:48:44 +08:00
auto classIter = self->id_class.find(w.first);
Catch and update processClass change from DBSource
2017-09-26 01:36:03 +08:00
ProcessClass newProcessClass;
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (classIter != self->id_class.end() &&
(classIter->second.classSource() == ProcessClass::DBSource ||
w.second.initialClass.classType() == ProcessClass::UnsetClass)) {
Catch and update processClass change from DBSource
2017-09-26 01:36:03 +08:00
newProcessClass = classIter->second;
Initial repository commit
2017-05-26 04:48:44 +08:00
} else {
Catch and update processClass change from DBSource
2017-09-26 01:36:03 +08:00
newProcessClass = w.second.initialClass;
}
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if (newProcessClass != w.second.details.processClass) {
w.second.details.processClass = newProcessClass;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
w.second.priorityInfo.processClassFitness =
newProcessClass.machineClassFitness(ProcessClass::ClusterController);
Fix cluster controller review comments
2017-10-05 06:48:55 +08:00
if (!w.second.reply.isSet()) {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
w.second.reply.send(
RegisterWorkerReply(w.second.details.processClass, w.second.priorityInfo));
Fix cluster controller review comments
2017-10-05 06:48:55 +08:00
}
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
self->lastProcessClasses = processClasses;
self->gotProcessClasses = true;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
checkOutstandingRequests(self);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
state Future<Void> watchFuture = tr.watch(processClassChangeKey);
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr.commit());
wait(watchFuture);
Initial repository commit
2017-05-26 04:48:44 +08:00
break;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} catch (Error& e) {
wait(tr.onError(e));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
}
}
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
ACTOR Future<Void> monitorServerInfoConfig(ClusterControllerData::DBInfo* db) {
loop {
state ReadYourWritesTransaction tr(db->db);
loop {
try {
tr.setOption(FDBTransactionOptions::READ_SYSTEM_KEYS);
tr.setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
tr.setOption(FDBTransactionOptions::READ_LOCK_AWARE);
Optional<Value> configVal = wait(tr.get(latencyBandConfigKey));
Optional<LatencyBandConfig> config;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (configVal.present()) {
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
config = LatencyBandConfig::parse(configVal.get());
}
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
auto serverInfo = db->serverInfo->get();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (config != serverInfo.latencyBandConfig) {
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
TraceEvent("LatencyBandConfigChanged").detail("Present", config.present());
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
serverInfo.id = deterministicRandom()->randomUniqueID();
fix compilation errors
2020-04-07 11:58:43 +08:00
serverInfo.infoGeneration = ++db->dbInfoCount;
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
serverInfo.latencyBandConfig = config;
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
db->serverInfo->set(serverInfo);
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
}
state Future<Void> configChangeFuture = tr.watch(latencyBandConfigKey);
Merge commit '3b2700d25334c53d13496ca16682642aac951beb' into track-server-request-latencies # Conflicts: # fdbclient/MasterProxyInterface.h # fdbserver/ClusterController.actor.cpp # fdbserver/MasterProxyServer.actor.cpp # fdbserver/ServerDBInfo.h # fdbserver/Status.actor.cpp # fdbserver/fdbserver.vcxproj # fdbserver/storageserver.actor.cpp
2019-01-25 03:43:26 +08:00
wait(tr.commit());
wait(configChangeFuture);
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
break;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} catch (Error& e) {
Merge remote-tracking branch 'negoyal/fdb_cache_subfeature2' into features/cache-initialization
2020-01-08 11:53:09 +08:00
wait(tr.onError(e));
Added the ability to configure the latency band settings by setting a special key in \xff keyspace.
2019-01-19 08:18:34 +08:00
}
}
}
}
Initial repository commit
2017-05-26 04:48:44 +08:00
ACTOR Future<Void> monitorClientTxnInfoConfigs(ClusterControllerData::DBInfo* db) {
loop {
state ReadYourWritesTransaction tr(db->db);
loop {
try {
tr.setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr.setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
Address code review comments. * Fixed memory corruption with SystemData key constants * Removed duplication in ClusterController * Reworked fdbcli actions to better represent explicit vs default assignments
2017-10-04 11:57:39 +08:00
state Optional<Value> rateVal = wait(tr.get(fdbClientInfoTxnSampleRate));
state Optional<Value> limitVal = wait(tr.get(fdbClientInfoTxnSizeLimit));
Initial repository commit
2017-05-26 04:48:44 +08:00
ClientDBInfo clientInfo = db->clientInfo->get();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
double sampleRate = rateVal.present()
? BinaryReader::fromStringRef<double>(rateVal.get(), Unversioned())
: std::numeric_limits<double>::infinity();
int64_t sizeLimit =
limitVal.present() ? BinaryReader::fromStringRef<int64_t>(limitVal.get(), Unversioned()) : -1;
if (sampleRate != clientInfo.clientTxnInfoSampleRate ||
sizeLimit != clientInfo.clientTxnInfoSampleRate) {
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
clientInfo.id = deterministicRandom()->randomUniqueID();
onMasterProxiesChanged was being triggered when any member of ClientDBInfo changed. Change the behavior to be triggered only when proxies field in ClientDBInfo is changed.
2017-11-02 09:29:56 +08:00
clientInfo.clientTxnInfoSampleRate = sampleRate;
clientInfo.clientTxnInfoSizeLimit = sizeLimit;
Initial repository commit
2017-05-26 04:48:44 +08:00
db->clientInfo->set(clientInfo);
}
The failure monitor is given a list of all IP addresses associated with a process The connect packet includes the correct remote address Did a lot of code cleanup Simulation test mixed TLS and non-TLS listeners on the same process
2019-02-01 10:20:14 +08:00
Address code review comments. * Fixed memory corruption with SystemData key constants * Removed duplication in ClusterController * Reworked fdbcli actions to better represent explicit vs default assignments
2017-10-04 11:57:39 +08:00
state Future<Void> watchRateFuture = tr.watch(fdbClientInfoTxnSampleRate);
state Future<Void> watchLimitFuture = tr.watch(fdbClientInfoTxnSizeLimit);
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr.commit());
Initial repository commit
2017-05-26 04:48:44 +08:00
choose {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
when(wait(watchRateFuture)) { break; }
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(wait(watchLimitFuture)) { break; }
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} catch (Error& e) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(tr.onError(e));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> updatedChangingDatacenters(ClusterControllerData* self) {
// do not change the cluster controller until all the processes have had a chance to register
wait(delay(SERVER_KNOBS->WAIT_FOR_GOOD_RECRUITMENT_DELAY));
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
loop {
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
state Future<Void> onChange = self->desiredDcIds.onChange();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!self->desiredDcIds.get().present()) {
self->changingDcIds.set(std::make_pair(false, self->desiredDcIds.get()));
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
} else {
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
auto& worker = self->id_worker[self->clusterControllerProcessId];
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
uint8_t newFitness = ClusterControllerPriorityInfo::calculateDCFitness(
worker.details.interf.locality.dcId(), self->desiredDcIds.get().get());
self->changingDcIds.set(
std::make_pair(worker.priorityInfo.dcFitness > newFitness, self->desiredDcIds.get()));
TraceEvent("UpdateChangingDatacenter", self->id)
.detail("OldFitness", worker.priorityInfo.dcFitness)
.detail("NewFitness", newFitness);
if (worker.priorityInfo.dcFitness > newFitness) {
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
worker.priorityInfo.dcFitness = newFitness;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!worker.reply.isSet()) {
worker.reply.send(RegisterWorkerReply(worker.details.processClass, worker.priorityInfo));
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
}
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
} else {
state int currentFit = ProcessClass::BestFit;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
while (currentFit <= ProcessClass::NeverAssign) {
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
bool updated = false;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& it : self->id_worker) {
if ((!it.second.priorityInfo.isExcluded &&
it.second.priorityInfo.processClassFitness == currentFit) ||
currentFit == ProcessClass::NeverAssign) {
uint8_t fitness = ClusterControllerPriorityInfo::calculateDCFitness(
it.second.details.interf.locality.dcId(), self->changingDcIds.get().second.get());
if (it.first != self->clusterControllerProcessId &&
it.second.priorityInfo.dcFitness != fitness) {
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
updated = true;
it.second.priorityInfo.dcFitness = fitness;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!it.second.reply.isSet()) {
it.second.reply.send(
RegisterWorkerReply(it.second.details.processClass, it.second.priorityInfo));
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
}
}
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (updated && currentFit < ProcessClass::NeverAssign) {
wait(delay(SERVER_KNOBS->CC_CLASS_DELAY));
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
}
currentFit++;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
}
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(onChange);
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> updatedChangedDatacenters(ClusterControllerData* self) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
state Future<Void> changeDelay = delay(SERVER_KNOBS->CC_CHANGE_DELAY);
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
state Future<Void> onChange = self->changingDcIds.onChange();
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
loop {
choose {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(wait(onChange)) {
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
changeDelay = delay(SERVER_KNOBS->CC_CHANGE_DELAY);
onChange = self->changingDcIds.onChange();
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(wait(changeDelay)) {
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
changeDelay = Never();
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
onChange = self->changingDcIds.onChange();
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
self->changedDcIds.set(self->changingDcIds.get());
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (self->changedDcIds.get().second.present()) {
added trace events for cluster controller changing datacenters
2018-07-03 01:06:54 +08:00
TraceEvent("UpdateChangedDatacenter", self->id).detail("CCFirst", self->changedDcIds.get().first);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!self->changedDcIds.get().first) {
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
auto& worker = self->id_worker[self->clusterControllerProcessId];
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
uint8_t newFitness = ClusterControllerPriorityInfo::calculateDCFitness(
worker.details.interf.locality.dcId(), self->changedDcIds.get().second.get());
if (worker.priorityInfo.dcFitness != newFitness) {
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
worker.priorityInfo.dcFitness = newFitness;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!worker.reply.isSet()) {
worker.reply.send(
RegisterWorkerReply(worker.details.processClass, worker.priorityInfo));
when the cluster controller is changing itself to a better dc fitness, it should notify itself first so another process does not take over
2018-06-29 15:10:29 +08:00
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
} else {
state int currentFit = ProcessClass::BestFit;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
while (currentFit <= ProcessClass::NeverAssign) {
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
bool updated = false;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& it : self->id_worker) {
if ((!it.second.priorityInfo.isExcluded &&
it.second.priorityInfo.processClassFitness == currentFit) ||
currentFit == ProcessClass::NeverAssign) {
uint8_t fitness = ClusterControllerPriorityInfo::calculateDCFitness(
it.second.details.interf.locality.dcId(),
self->changedDcIds.get().second.get());
if (it.first != self->clusterControllerProcessId &&
it.second.priorityInfo.dcFitness != fitness) {
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
updated = true;
it.second.priorityInfo.dcFitness = fitness;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!it.second.reply.isSet()) {
it.second.reply.send(RegisterWorkerReply(it.second.details.processClass,
it.second.priorityInfo));
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
}
}
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (updated && currentFit < ProcessClass::NeverAssign) {
wait(delay(SERVER_KNOBS->CC_CLASS_DELAY));
update better machine classes first to give them a higher chance of becoming the next cluster controller
2018-06-29 16:11:59 +08:00
}
currentFit++;
}
re-enabled better master exists the cluster controller can choose a better data center for itself and let the workers know where the next cluster controller should be recruited
2018-02-10 08:48:55 +08:00
}
}
}
}
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> updateDatacenterVersionDifference(ClusterControllerData* self) {
fixed compile issue
2018-06-22 07:34:36 +08:00
state double lastLogTime = 0;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
loop {
self->versionDifferenceUpdated = false;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (self->db.serverInfo->get().recoveryState >= RecoveryState::ACCEPTING_COMMITS &&
self->db.config.usableRegions == 1) {
bool oldDifferenceTooLarge = !self->versionDifferenceUpdated ||
self->datacenterVersionDifference >= SERVER_KNOBS->MAX_VERSION_DIFFERENCE;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
self->versionDifferenceUpdated = true;
self->datacenterVersionDifference = 0;
fix: treat the local region as best when version difference is too large re-check requests when the version difference becomes small
2018-07-07 05:44:11 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (oldDifferenceTooLarge) {
fix: treat the local region as best when version difference is too large re-check requests when the version difference becomes small
2018-07-07 05:44:11 +08:00
checkOutstandingRequests(self);
}
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(self->db.serverInfo->onChange());
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
continue;
}
state Optional<TLogInterface> primaryLog;
state Optional<TLogInterface> remoteLog;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (self->db.serverInfo->get().recoveryState >= RecoveryState::ALL_LOGS_RECRUITED) {
for (auto& logSet : self->db.serverInfo->get().logSystemConfig.tLogs) {
if (logSet.isLocal && logSet.locality != tagLocalitySatellite) {
for (auto& tLog : logSet.tLogs) {
if (tLog.present()) {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
primaryLog = tLog.interf();
break;
}
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!logSet.isLocal) {
for (auto& tLog : logSet.tLogs) {
if (tLog.present()) {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
remoteLog = tLog.interf();
break;
}
}
}
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!primaryLog.present() || !remoteLog.present()) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(self->db.serverInfo->onChange());
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
continue;
}
added logging for the datacenter version difference
2018-06-22 07:31:52 +08:00
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
state Future<Void> onChange = self->db.serverInfo->onChange();
loop {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
state Future<TLogQueuingMetricsReply> primaryMetrics =
brokenPromiseToNever(primaryLog.get().getQueuingMetrics.getReply(TLogQueuingMetricsRequest()));
state Future<TLogQueuingMetricsReply> remoteMetrics =
brokenPromiseToNever(remoteLog.get().getQueuingMetrics.getReply(TLogQueuingMetricsRequest()));
added logging for the datacenter version difference
2018-06-22 07:31:52 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
wait((success(primaryMetrics) && success(remoteMetrics)) || onChange);
if (onChange.isReady()) {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
break;
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (primaryMetrics.get().v > 0 && remoteMetrics.get().v > 0) {
bool oldDifferenceTooLarge = !self->versionDifferenceUpdated ||
self->datacenterVersionDifference >= SERVER_KNOBS->MAX_VERSION_DIFFERENCE;
fix: do not report datacenter version difference before both datacenters report a correct version
2020-05-11 08:49:09 +08:00
self->versionDifferenceUpdated = true;
self->datacenterVersionDifference = primaryMetrics.get().v - remoteMetrics.get().v;
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (oldDifferenceTooLarge && self->datacenterVersionDifference < SERVER_KNOBS->MAX_VERSION_DIFFERENCE) {
fix: do not report datacenter version difference before both datacenters report a correct version
2020-05-11 08:49:09 +08:00
checkOutstandingRequests(self);
}
fix: treat the local region as best when version difference is too large re-check requests when the version difference becomes small
2018-07-07 05:44:11 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (now() - lastLogTime > SERVER_KNOBS->CLUSTER_CONTROLLER_LOGGING_DELAY) {
fix: do not report datacenter version difference before both datacenters report a correct version
2020-05-11 08:49:09 +08:00
lastLogTime = now();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
TraceEvent("DatacenterVersionDifference", self->id)
.detail("Difference", self->datacenterVersionDifference);
fix: do not report datacenter version difference before both datacenters report a correct version
2020-05-11 08:49:09 +08:00
}
added logging for the datacenter version difference
2018-06-22 07:31:52 +08:00
}
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
wait(delay(SERVER_KNOBS->VERSION_LAG_METRIC_INTERVAL) || onChange);
if (onChange.isReady()) {
The master will tell the cluster controller if it is going to take a long time to recruit new logs in its DC; the cluster controller can determine if the other DC would be better and recruit there. The cluster controller will not switch to the other data center if remote logs are too far behind. We will not recruit in DCs with negative priority.
2018-06-14 09:14:14 +08:00
break;
}
}
}
}
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
ACTOR Future<Void> doEmptyCommit(Database cx) {
state Transaction tr(cx);
loop {
try {
tr.setOption(FDBTransactionOptions::PRIORITY_SYSTEM_IMMEDIATE);
tr.setOption(FDBTransactionOptions::LOCK_AWARE);
tr.makeSelfConflicting();
wait(tr.commit());
return Void();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} catch (Error& e) {
wait(tr.onError(e));
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
}
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> handleForcedRecoveries(ClusterControllerData* self, ClusterControllerFullInterface interf) {
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
loop {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
state ForceRecoveryRequest req = waitNext(interf.clientInterface.forceRecovery.getFuture());
TraceEvent("ForcedRecoveryStart", self->id)
.detail("ClusterControllerDcId", self->clusterControllerDcId)
.detail("DcId", req.dcId.printable());
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
state Future<Void> fCommit = doEmptyCommit(self->cx);
knobify force recovery recovery check delay
2019-02-20 08:05:20 +08:00
wait(fCommit || delay(SERVER_KNOBS->FORCE_RECOVERY_CHECK_DELAY));
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!fCommit.isReady() || fCommit.isError()) {
Remove clusterControllerDcId check This is no longer needed since it'll be set in the ctor.
2019-03-23 08:08:58 +08:00
if (self->clusterControllerDcId != req.dcId) {
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
vector<Optional<Key>> dcPriority;
dcPriority.push_back(req.dcId);
dcPriority.push_back(self->clusterControllerDcId);
self->desiredDcIds.set(dcPriority);
} else {
self->db.forceRecovery = true;
self->db.forceMasterFailure.trigger();
}
wait(fCommit);
}
TraceEvent("ForcedRecoveryFinish", self->id);
self->db.forceRecovery = false;
req.reply.send(Void());
forgot to fix merge conflicts
2019-02-19 09:09:55 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<DataDistributorInterface> startDataDistributor(ClusterControllerData* self) {
wait(delay(0.0)); // If master fails at the same time, give it a chance to clear master PID.
Add a new DataDistributor role. Let cluster controller to start a new data distributor role by sending a message to a chosen worker. Change MasterInterface usage in DataDistribution to masterId Add DataDistributor rejoin handling. This allows the data distributor to tell the new cluster controller of its existence so that the controller doesn't spawn a new one. I.e., there should be only ONE data distributor in the cluster. If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries to recruit one as DD. CC also monitors DD and restarts one if it failed. The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for the new DD. Add GetRecoveryInfo RPC to master server, which is called by data distributor to obtain the recovery Transaction version from the master server.
2018-12-14 05:31:37 +08:00
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCStartDataDistributor", self->id);
Update fdbserver/ClusterController.actor.cpp Co-Authored-By: jzhou77 <jingyuzhou@gmail.com>
2019-02-14 03:54:35 +08:00
loop {
Fix segfault and no_more_servers errors
2019-01-31 01:05:12 +08:00
try {
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
state bool no_distributor = !self->db.serverInfo->get().distributor.present();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
while (!self->masterProcessId.present() ||
self->masterProcessId != self->db.serverInfo->get().master.locality.processId() ||
self->db.serverInfo->get().recoveryState < RecoveryState::ACCEPTING_COMMITS) {
Fix a race of DD, RK, Master failure If all DD, RK, Master run on the same process and failed. Recruiting of new DD or RK could try to use the old master worker interface, which is an invalid one and causes recruitment to be stuck. Fix by adding a delay and checking master is valid before recruitment.
2019-03-21 07:03:36 +08:00
wait(self->db.serverInfo->onChange() || delay(SERVER_KNOBS->WAIT_FOR_GOOD_RECRUITMENT_DELAY));
Fix segfault and no_more_servers errors
2019-01-31 01:05:12 +08:00
}
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
if (no_distributor && self->db.serverInfo->get().distributor.present()) {
return self->db.serverInfo->get().distributor.get();
Fix segfault and no_more_servers errors
2019-01-31 01:05:12 +08:00
}
Per review comments Add a knob for starting distributor delay. Move distributor failed variable to a local loop.
2019-02-01 02:51:25 +08:00
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
std::map<Optional<Standalone<StringRef>>, int> id_used = self->getUsedIds();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
WorkerFitnessInfo data_distributor = self->getWorkerForRoleInDatacenter(self->clusterControllerDcId,
ProcessClass::DataDistributor,
ProcessClass::NeverAssign,
self->db.config,
id_used);
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
state WorkerDetails worker = data_distributor.worker;
if (self->onMasterIsBetter(worker, ProcessClass::DataDistributor)) {
worker = self->id_worker[self->masterProcessId.get()].details;
}
Merge remote-tracking branch 'negoyal/fdb_cache_subfeature2' into features/cache-initialization
2020-01-08 11:53:09 +08:00
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
InitializeDataDistributorRequest req(deterministicRandom()->randomUniqueID());
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCDataDistributorRecruit", self->id).detail("Addr", worker.interf.address());
Fix segfault and no_more_servers errors
2019-01-31 01:05:12 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ErrorOr<DataDistributorInterface> distributor = wait(worker.interf.dataDistributor.getReplyUnlessFailedFor(
req, SERVER_KNOBS->WAIT_FOR_DISTRIBUTOR_JOIN_DELAY, 0));
Fix segfault and no_more_servers errors
2019-01-31 01:05:12 +08:00
if (distributor.present()) {
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCDataDistributorRecruited", self->id).detail("Addr", worker.interf.address());
Fix segfault and no_more_servers errors
2019-01-31 01:05:12 +08:00
return distributor.get();
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} catch (Error& e) {
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCDataDistributorRecruitError", self->id).error(e);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (e.code() != error_code_no_more_servers) {
Fix segfault and no_more_servers errors
2019-01-31 01:05:12 +08:00
throw;
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
wait(lowPriorityDelay(SERVER_KNOBS->ATTEMPT_RECRUITMENT_DELAY));
Add a new DataDistributor role. Let cluster controller to start a new data distributor role by sending a message to a chosen worker. Change MasterInterface usage in DataDistribution to masterId Add DataDistributor rejoin handling. This allows the data distributor to tell the new cluster controller of its existence so that the controller doesn't spawn a new one. I.e., there should be only ONE data distributor in the cluster. If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries to recruit one as DD. CC also monitors DD and restarts one if it failed. The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for the new DD. Add GetRecoveryInfo RPC to master server, which is called by data distributor to obtain the recovery Transaction version from the master server.
2018-12-14 05:31:37 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> monitorDataDistributor(ClusterControllerData* self) {
while (self->db.serverInfo->get().recoveryState < RecoveryState::ACCEPTING_COMMITS) {
ratekeeper needs to be recruited for proxies to make progress, so if one has not registered with the cluster controller by the time we are accepting commits, recruit a new one
2019-03-25 07:48:24 +08:00
wait(self->db.serverInfo->onChange());
Add a new DataDistributor role. Let cluster controller to start a new data distributor role by sending a message to a chosen worker. Change MasterInterface usage in DataDistribution to masterId Add DataDistributor rejoin handling. This allows the data distributor to tell the new cluster controller of its existence so that the controller doesn't spawn a new one. I.e., there should be only ONE data distributor in the cluster. If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries to recruit one as DD. CC also monitors DD and restarts one if it failed. The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for the new DD. Add GetRecoveryInfo RPC to master server, which is called by data distributor to obtain the recovery Transaction version from the master server.
2018-12-14 05:31:37 +08:00
}
Per review comments Add a knob for starting distributor delay. Move distributor failed variable to a local loop.
2019-02-01 02:51:25 +08:00
loop {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (self->db.serverInfo->get().distributor.present()) {
wait(waitFailureClient(self->db.serverInfo->get().distributor.get().waitFailure,
SERVER_KNOBS->DD_FAILURE_TIME));
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCDataDistributorDied", self->id)
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
.detail("DistributorId", self->db.serverInfo->get().distributor.get().id());
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
self->db.clearInterf(ProcessClass::DataDistributorClass);
Final cleanup per review comments Make distributor interface optional in ServerDBInfo and many other small changes.
2019-02-13 07:50:44 +08:00
} else {
Fix the issue of slow dying Data Distributor Test with: -r simulation -f ./foundationdb/tests/slow/CommitBug.txt -s 67828576 -b on The test has the following event sequence: - Time 113.3s, CC noticed DD failure, cleard DD interface. - 1s later, DD rejoined and registered with CC. - Time 131.7s, DD actor cancelled. This old DD raced to register with CC and the failure monitor is not installed because monitorDataDistributor is stalled waiting for new DD. - Time 161.4s, new DD running. New DD recruting was delayed due to no servers in the period. Fix by disabling DD registration during the recruting process.
2019-03-18 11:55:59 +08:00
self->recruitingDistributor = true;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
DataDistributorInterface distributorInterf = wait(startDataDistributor(self));
Fix the issue of slow dying Data Distributor Test with: -r simulation -f ./foundationdb/tests/slow/CommitBug.txt -s 67828576 -b on The test has the following event sequence: - Time 113.3s, CC noticed DD failure, cleard DD interface. - 1s later, DD rejoined and registered with CC. - Time 131.7s, DD actor cancelled. This old DD raced to register with CC and the failure monitor is not installed because monitorDataDistributor is stalled waiting for new DD. - Time 161.4s, new DD running. New DD recruting was delayed due to no servers in the period. Fix by disabling DD registration during the recruting process.
2019-03-18 11:55:59 +08:00
self->recruitingDistributor = false;
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
self->db.setDistributor(distributorInterf);
}
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> startRatekeeper(ClusterControllerData* self) {
wait(delay(0.0)); // If master fails at the same time, give it a chance to clear master PID.
Fix a race of DD, RK, Master failure If all DD, RK, Master run on the same process and failed. Recruiting of new DD or RK could try to use the old master worker interface, which is an invalid one and causes recruitment to be stuck. Fix by adding a delay and checking master is valid before recruitment.
2019-03-21 07:03:36 +08:00
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCStartRatekeeper", self->id);
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
loop {
try {
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
state bool no_ratekeeper = !self->db.serverInfo->get().ratekeeper.present();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
while (!self->masterProcessId.present() ||
self->masterProcessId != self->db.serverInfo->get().master.locality.processId() ||
self->db.serverInfo->get().recoveryState < RecoveryState::ACCEPTING_COMMITS) {
Fix a race of DD, RK, Master failure If all DD, RK, Master run on the same process and failed. Recruiting of new DD or RK could try to use the old master worker interface, which is an invalid one and causes recruitment to be stuck. Fix by adding a delay and checking master is valid before recruitment.
2019-03-21 07:03:36 +08:00
wait(self->db.serverInfo->onChange() || delay(SERVER_KNOBS->WAIT_FOR_GOOD_RECRUITMENT_DELAY));
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
}
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
if (no_ratekeeper && self->db.serverInfo->get().ratekeeper.present()) {
Avoid unnecessary recuriting of DD or RK While waiting for recruting data distributor or ratekeeper, a previous one could already joined. So we can skip this unnecessary recruiting. Revert the change of worker.actor.cpp for ratekeeper. Instead, recruiting ratekeeper should avoid the process with an existing one. This fixes a bug where the ratekeeper interface became zombie, killing other healthy ratekeeper but doing no useful work. Found by: -r simulation --crash -f tests/fast/WriteDuringRead.txt -s 31858110 -b on
2019-03-22 13:20:00 +08:00
// Existing ratekeeper registers while waiting, so skip.
return Void();
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
}
std::map<Optional<Standalone<StringRef>>, int> id_used = self->getUsedIds();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
WorkerFitnessInfo rkWorker = self->getWorkerForRoleInDatacenter(self->clusterControllerDcId,
ProcessClass::Ratekeeper,
ProcessClass::NeverAssign,
self->db.config,
id_used);
Replace g_random and g_nondeterministic_random with functions deterministicRandom() and nondeterministicRandom() that return thread_local random number generators. Delete g_debug_random and trace_random. Allow only deterministicRandom() to be seeded, and require it to be seeded from each thread on which it is used.
2019-05-11 05:01:52 +08:00
InitializeRatekeeperRequest req(deterministicRandom()->randomUniqueID());
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
state WorkerDetails worker = rkWorker.worker;
Changes to ratekeeper camel case
2019-03-27 23:24:25 +08:00
if (self->onMasterIsBetter(worker, ProcessClass::Ratekeeper)) {
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
worker = self->id_worker[self->masterProcessId.get()].details;
}
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
First round of review comment fixes for PR#1307
2019-03-20 02:29:19 +08:00
self->recruitingRatekeeperID = req.reqId;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
TraceEvent("CCRecruitRatekeeper", self->id)
.detail("Addr", worker.interf.address())
.detail("RKID", req.reqId);
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ErrorOr<RatekeeperInterface> interf = wait(
worker.interf.ratekeeper.getReplyUnlessFailedFor(req, SERVER_KNOBS->WAIT_FOR_RATEKEEPER_JOIN_DELAY, 0));
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
if (interf.present()) {
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
self->recruitRatekeeper.set(false);
once we recruit a ratekeeper, do not allow any other ratekeepers to register
2019-03-25 02:04:39 +08:00
self->recruitingRatekeeperID = interf.get().id();
properly cache serialization of serverDBInfo
2020-04-12 10:30:05 +08:00
const auto& ratekeeper = self->db.serverInfo->get().ratekeeper;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
TraceEvent("CCRatekeeperRecruited", self->id)
.detail("Addr", worker.interf.address())
.detail("RKID", interf.get().id());
if (ratekeeper.present() && ratekeeper.get().id() != interf.get().id() &&
self->id_worker.count(ratekeeper.get().locality.processId())) {
TraceEvent("CCHaltRatekeeperAfterRecruit", self->id)
.detail("RKID", ratekeeper.get().id())
.detail("DcID", printable(self->clusterControllerDcId));
self->id_worker[ratekeeper.get().locality.processId()].haltRatekeeper =
brokenPromiseToNever(ratekeeper.get().haltRatekeeper.getReply(HaltRatekeeperRequest(self->id)));
}
if (!ratekeeper.present() || ratekeeper.get().id() != interf.get().id()) {
startRatekeeper is responsible for updating serverDBInfo
2019-03-23 08:56:16 +08:00
self->db.setRatekeeper(interf.get());
}
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
checkOutstandingRequests(self);
Fix ratekeeper recruitment bug
2019-03-21 04:54:15 +08:00
return Void();
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} catch (Error& e) {
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCRatekeeperRecruitError", self->id).error(e);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (e.code() != error_code_no_more_servers) {
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
throw;
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
wait(lowPriorityDelay(SERVER_KNOBS->ATTEMPT_RECRUITMENT_DELAY));
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> monitorRatekeeper(ClusterControllerData* self) {
while (self->db.serverInfo->get().recoveryState < RecoveryState::ACCEPTING_COMMITS) {
ratekeeper needs to be recruited for proxies to make progress, so if one has not registered with the cluster controller by the time we are accepting commits, recruit a new one
2019-03-25 07:48:24 +08:00
wait(self->db.serverInfo->onChange());
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
}
loop {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (self->db.serverInfo->get().ratekeeper.present() && !self->recruitRatekeeper.get()) {
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
choose {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(wait(waitFailureClient(self->db.serverInfo->get().ratekeeper.get().waitFailure,
SERVER_KNOBS->RATEKEEPER_FAILURE_TIME))) {
Remove additional underscores
2019-07-05 23:12:25 +08:00
TraceEvent("CCRatekeeperDied", self->id)
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
.detail("RKID", self->db.serverInfo->get().ratekeeper.get().id());
Changes to ratekeeper camel case
2019-03-27 23:24:25 +08:00
self->db.clearInterf(ProcessClass::RatekeeperClass);
Fix ratekeeper and data distributor recruiting bug Avoid multiple concurrent recuriting of ratekeepers with a recruiting flag. Fix endless recruiting when the chosen worker is a proxy or a resolver -- prefer master in this case.
2019-03-21 01:00:31 +08:00
}
simplified ratekeeper monitoring loop
2019-03-23 09:22:45 +08:00
when(wait(self->recruitRatekeeper.onChange())) {}
Recruit a new ratekeeper before halting the old
2019-03-20 06:21:46 +08:00
}
Separate Ratekeeper from data distribution. Add a new role for ratekeeper. Remove StorageServerChanges from data distribution. Ratekeeper monitors storage servers, which borrows the idea from DataDistribution.
2019-02-15 08:24:46 +08:00
} else {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
wait(startRatekeeper(self));
Add a new DataDistributor role. Let cluster controller to start a new data distributor role by sending a message to a chosen worker. Change MasterInterface usage in DataDistribution to masterId Add DataDistributor rejoin handling. This allows the data distributor to tell the new cluster controller of its existence so that the controller doesn't spawn a new one. I.e., there should be only ONE data distributor in the cluster. If DataDistributor (DD) doesn't join in a while, then ClusterController (CC) tries to recruit one as DD. CC also monitors DD and restarts one if it failed. The Proxy is also monitoring the DD. If DD failed, the Proxy will ask CC for the new DD. Add GetRecoveryInfo RPC to master server, which is called by data distributor to obtain the recovery Transaction version from the master server.
2018-12-14 05:31:37 +08:00
}
fix: forced recoveries now require a target dcid which will become the new primary location. During the forced recovery, the configuration will be changed to make that location primary, and usable_regions will be set to 1. If the target dcid is already the primary location, the forced recovery will do nothing. This makes forced recoveries idempotent, so it is safe to the client to re-send forced recovery commands to the cluster controller. fix: the cluster controller attempts to do a commit to determine if the cluster is alive, since its own internal recoveryState might not be up-to-date. fix: forceMasterFailure on the cluster controller did not always cause the current master to be re-recruited
2019-02-19 06:54:28 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> dbInfoUpdater(ClusterControllerData* self) {
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
state Future<Void> dbInfoChange = self->db.serverInfo->onChange();
state Future<Void> updateDBInfo = self->updateDBInfo.onTrigger();
loop {
choose {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(wait(updateDBInfo)) { wait(delay(SERVER_KNOBS->DBINFO_BATCH_DELAY) || dbInfoChange); }
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
when(wait(dbInfoChange)) {}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
code cleanup based on review comments
2020-04-18 06:05:01 +08:00
UpdateServerDBInfoRequest req;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (dbInfoChange.isReady()) {
for (auto& it : self->id_worker) {
code cleanup based on review comments
2020-04-18 06:05:01 +08:00
req.broadcastInfo.push_back(it.second.details.interf.updateServerDBInfo.getEndpoint());
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
}
} else {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto it : self->removedDBInfoEndpoints) {
fixed improper use of std::set erase
2020-04-18 07:45:22 +08:00
self->updateDBInfoEndpoints.erase(it);
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
req.broadcastInfo =
std::vector<Endpoint>(self->updateDBInfoEndpoints.begin(), self->updateDBInfoEndpoints.end());
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
}
code cleanup based on review comments
2020-04-18 06:05:01 +08:00
self->updateDBInfoEndpoints.clear();
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
self->removedDBInfoEndpoints.clear();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
dbInfoChange = self->db.serverInfo->onChange();
updateDBInfo = self->updateDBInfo.onTrigger();
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
req.serializedDbInfo =
BinaryWriter::toValue(self->db.serverInfo->get(), AssumeVersion(g_network->protocolVersion()));
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
many bug fixes
2020-04-11 04:45:16 +08:00
TraceEvent("DBInfoStartBroadcast", self->id);
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
choose {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(std::vector<Endpoint> notUpdated =
wait(broadcastDBInfoRequest(req, SERVER_KNOBS->DBINFO_SEND_AMOUNT, Optional<Endpoint>(), false))) {
remove spammy trace event
2020-09-01 01:37:00 +08:00
TraceEvent("DBInfoFinishBroadcast", self->id).detail("NotUpdated", notUpdated.size());
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (notUpdated.size()) {
fixed improper use of std::set erase
2020-04-18 07:45:22 +08:00
self->updateDBInfoEndpoints.insert(notUpdated.begin(), notUpdated.end());
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
self->updateDBInfo.trigger();
}
}
code cleanup
2020-04-11 08:02:11 +08:00
when(wait(dbInfoChange)) {}
implemented a tree broadcast for txn state message for proxies, and serverDBInfo for workers
2020-04-06 14:09:36 +08:00
}
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> clusterControllerCore(ClusterControllerFullInterface interf,
Future<Void> leaderFail,
ServerCoordinators coordinators,
LocalityData locality) {
state ClusterControllerData self(interf, locality);
state Future<Void> coordinationPingDelay = delay(SERVER_KNOBS->WORKER_COORDINATION_PING_DELAY);
Initial repository commit
2017-05-26 04:48:44 +08:00
state uint64_t step = 0;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
state Future<ErrorOr<Void>> error = errorOr(actorCollection(self.addActor.getFuture()));
self.addActor.send(clusterWatchDatabase(&self, &self.db)); // Start the master database
self.addActor.send(self.updateWorkerList.init(self.db.db));
self.addActor.send(statusServer(interf.clientInterface.databaseStatus.getFuture(), &self, coordinators));
self.addActor.send(timeKeeper(&self));
self.addActor.send(monitorProcessClasses(&self));
self.addActor.send(monitorServerInfoConfig(&self.db));
self.addActor.send(monitorClientTxnInfoConfigs(&self.db));
self.addActor.send(updatedChangingDatacenters(&self));
self.addActor.send(updatedChangedDatacenters(&self));
self.addActor.send(updateDatacenterVersionDifference(&self));
self.addActor.send(handleForcedRecoveries(&self, interf));
self.addActor.send(monitorDataDistributor(&self));
self.addActor.send(monitorRatekeeper(&self));
self.addActor.send(dbInfoUpdater(&self));
self.addActor.send(traceCounters("ClusterControllerMetrics",
self.id,
SERVER_KNOBS->STORAGE_LOGGING_DELAY,
&self.clusterControllerMetrics,
self.id.toString() + "/ClusterControllerMetrics"));
self.addActor.send(traceRole(Role::CLUSTER_CONTROLLER, interf.id()));
// printf("%s: I am the cluster controller\n", g_network->getLocalAddress().toString().c_str());
Initial repository commit
2017-05-26 04:48:44 +08:00
loop choose {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(ErrorOr<Void> err = wait(error)) {
Initial repository commit
2017-05-26 04:48:44 +08:00
if (err.isError()) {
Add the roles running on a process as a field on trace events in the form of a comma delimited string of role abbreviations.
2018-09-06 06:06:14 +08:00
endRole(Role::CLUSTER_CONTROLLER, interf.id(), "Stop Received Error", false, err.getError());
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} else {
Add the roles running on a process as a field on trace events in the form of a comma delimited string of role abbreviations.
2018-09-06 06:06:14 +08:00
endRole(Role::CLUSTER_CONTROLLER, interf.id(), "Stop Received Signal", true);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
// We shut down normally even if there was a serious error (so this fdbserver may be re-elected cluster
// controller)
Initial repository commit
2017-05-26 04:48:44 +08:00
return Void();
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(OpenDatabaseRequest req = waitNext(interf.clientInterface.openDatabase.getFuture())) {
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
++self.openDatabaseRequests;
added status information on connected clients
2019-07-26 08:15:31 +08:00
self.addActor.send(clusterOpenDatabase(&self.db, req));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(RecruitFromConfigurationRequest req = waitNext(interf.recruitFromConfiguration.getFuture())) {
self.addActor.send(clusterRecruitFromConfiguration(&self, req));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(RecruitRemoteFromConfigurationRequest req = waitNext(interf.recruitRemoteFromConfiguration.getFuture())) {
self.addActor.send(clusterRecruitRemoteFromConfiguration(&self, req));
cluster controller changes were not maintained during merge
2017-09-12 08:40:46 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(RecruitStorageRequest req = waitNext(interf.recruitStorage.getFuture())) {
clusterRecruitStorage(&self, req);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(RegisterWorkerRequest req = waitNext(interf.registerWorker.getFuture())) {
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
++self.registerWorkerRequests;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
registerWorker(req, &self);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(GetWorkersRequest req = waitNext(interf.getWorkers.getFuture())) {
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
++self.getWorkersRequests;
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
vector<WorkerDetails> workers;
Enable checkUsingDesiredClasses() in consistency check
2017-10-25 03:58:54 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& it : self.id_worker) {
if ((req.flags & GetWorkersRequest::NON_EXCLUDED_PROCESSES_ONLY) &&
self.db.config.isExcludedServer(it.second.details.interf.addresses())) {
Enable checkUsingDesiredClasses() in consistency check
2017-10-25 03:58:54 +08:00
continue;
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if ((req.flags & GetWorkersRequest::TESTER_CLASS_ONLY) &&
it.second.details.processClass.classType() != ProcessClass::TesterClass) {
Enable checkUsingDesiredClasses() in consistency check
2017-10-25 03:58:54 +08:00
continue;
}
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
workers.push_back(it.second.details);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
Enable checkUsingDesiredClasses() in consistency check
2017-10-25 03:58:54 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
req.reply.send(workers);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(GetClientWorkersRequest req = waitNext(interf.clientInterface.getClientWorkers.getFuture())) {
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
++self.getClientWorkersRequests;
Initial repository commit
2017-05-26 04:48:44 +08:00
vector<ClientWorkerInterface> workers;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& it : self.id_worker) {
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
if (it.second.details.processClass.classType() != ProcessClass::TesterClass) {
workers.push_back(it.second.details.interf.clientInterface);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
req.reply.send(workers);
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(wait(coordinationPingDelay)) {
Initial repository commit
2017-05-26 04:48:44 +08:00
CoordinationPingMessage message(self.id, step++);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
for (auto& it : self.id_worker)
replaced std::pair<WorkerInterface,ProcessClass> with a struct named WorkerDetails
2019-03-09 00:25:07 +08:00
it.second.details.interf.coordinationPing.send(message);
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
coordinationPingDelay = delay(SERVER_KNOBS->WORKER_COORDINATION_PING_DELAY);
Initial repository commit
2017-05-26 04:48:44 +08:00
TraceEvent("CoordinationPingSent", self.id).detail("TimeStep", message.timeStep);
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(RegisterMasterRequest req = waitNext(interf.registerMaster.getFuture())) {
Add cluster controller metrics
2019-10-04 06:29:11 +08:00
++self.registerMasterRequests;
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
clusterRegisterMaster(&self, req);
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(GetServerDBInfoRequest req = waitNext(interf.getServerDBInfo.getFuture())) {
self.addActor.send(clusterGetServerInfo(&self.db, req.knownServerInfoID, req.reply));
Initial repository commit
2017-05-26 04:48:44 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(wait(leaderFail)) {
Initial repository commit
2017-05-26 04:48:44 +08:00
// We are no longer the leader if this has changed.
Add the roles running on a process as a field on trace events in the form of a comma delimited string of role abbreviations.
2018-09-06 06:06:14 +08:00
endRole(Role::CLUSTER_CONTROLLER, interf.id(), "Leader Replaced", true);
Initial repository commit
2017-05-26 04:48:44 +08:00
TEST(true); // Lost Cluster Controller Role
return Void();
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(ReplyPromise<Void> ping = waitNext(interf.clientInterface.ping.getFuture())) { ping.send(Void()); }
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> replaceInterface(ClusterControllerFullInterface interf) {
If a process receives a message in its clusterControllerInterface before becoming the cluster controller, if the process does not become the cluster controller in the next minute it should destroy the interface to prevent a memory leak.
2019-05-30 07:57:13 +08:00
loop {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (interf.hasMessage()) {
If a process receives a message in its clusterControllerInterface before becoming the cluster controller, if the process does not become the cluster controller in the next minute it should destroy the interface to prevent a memory leak.
2019-05-30 07:57:13 +08:00
wait(delay(SERVER_KNOBS->REPLACE_INTERFACE_DELAY));
return Void();
}
wait(delay(SERVER_KNOBS->REPLACE_INTERFACE_CHECK_DELAY));
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> clusterController(ServerCoordinators coordinators,
Reference<AsyncVar<Optional<ClusterControllerFullInterface>>> currentCC,
bool hasConnected,
Reference<AsyncVar<ClusterControllerPriorityInfo>> asyncPriorityInfo,
LocalityData locality) {
Initial repository commit
2017-05-26 04:48:44 +08:00
loop {
state ClusterControllerFullInterface cci;
state bool inRole = false;
cci.initEndpoints();
try {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
// Register as a possible leader; wait to be elected
state Future<Void> leaderFail =
tryBecomeLeader(coordinators, cci, currentCC, hasConnected, asyncPriorityInfo);
state Future<Void> shouldReplace = replaceInterface(cci);
Initial repository commit
2017-05-26 04:48:44 +08:00
while (!currentCC->get().present() || currentCC->get().get() != cci) {
choose {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
when(wait(currentCC->onChange())) {}
when(wait(leaderFail)) {
ASSERT(false);
throw internal_error();
}
when(wait(shouldReplace)) { break; }
Initial repository commit
2017-05-26 04:48:44 +08:00
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
if (!shouldReplace.isReady()) {
If a process receives a message in its clusterControllerInterface before becoming the cluster controller, if the process does not become the cluster controller in the next minute it should destroy the interface to prevent a memory leak.
2019-05-30 07:57:13 +08:00
shouldReplace = Future<Void>();
hasConnected = true;
startRole(Role::CLUSTER_CONTROLLER, cci.id(), UID());
inRole = true;
Initial repository commit
2017-05-26 04:48:44 +08:00
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
wait(clusterControllerCore(cci, leaderFail, coordinators, locality));
If a process receives a message in its clusterControllerInterface before becoming the cluster controller, if the process does not become the cluster controller in the next minute it should destroy the interface to prevent a memory leak.
2019-05-30 07:57:13 +08:00
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
} catch (Error& e) {
Initial repository commit
2017-05-26 04:48:44 +08:00
if (inRole)
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
endRole(Role::CLUSTER_CONTROLLER,
cci.id(),
"Error",
e.code() == error_code_actor_cancelled || e.code() == error_code_coordinators_changed,
e);
Initial repository commit
2017-05-26 04:48:44 +08:00
else
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
TraceEvent(e.code() == error_code_coordinators_changed ? SevInfo : SevError,
"ClusterControllerCandidateError",
cci.id())
.error(e);
Initial repository commit
2017-05-26 04:48:44 +08:00
throw;
}
}
}
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ACTOR Future<Void> clusterController(Reference<ClusterConnectionFile> connFile,
Reference<AsyncVar<Optional<ClusterControllerFullInterface>>> currentCC,
Reference<AsyncVar<ClusterControllerPriorityInfo>> asyncPriorityInfo,
Future<Void> recoveredDiskFiles,
LocalityData locality) {
Rewrite all `Void _ = wait(...)` -> `wait(...)`. This takes advantage of the new actorcompiler functionality to avoid having duplicate definitions of `Void _` when trying to feed the un-actorompiled source through clang.
2018-08-11 04:57:10 +08:00
wait(recoveredDiskFiles);
Initial repository commit
2017-05-26 04:48:44 +08:00
state bool hasConnected = false;
loop {
try {
apply clang-format to *.c, *.cpp, *.h, *.hpp files
2021-03-11 02:06:03 +08:00
ServerCoordinators coordinators(connFile);
wait(clusterController(coordinators, currentCC, hasConnected, asyncPriorityInfo, locality));
} catch (Error& e) {
if (e.code() != error_code_coordinators_changed)
Initial repository commit
2017-05-26 04:48:44 +08:00
throw; // Expected to terminate fdbserver
}
hasConnected = true;
}
Make RecoveryState an enum class. This means that all the == 7 or != 0 checks go away, and explicit names must be used.
2018-05-09 08:17:17 +08:00
}