wbtiger
/
foundationdb
mirror of https://github.com/apple/foundationdb.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into trace-log-refactor 

# Conflicts:
#	fdbserver/QuietDatabase.actor.cpp
#	fdbserver/Status.actor.cpp
#	flow/Trace.cpp
This commit is contained in:
A.J. Beamon 2018-06-08 13:24:30 -07:00
parent 5848055620 943dfa278b
commit 0ca51989bb
118 changed files with 1590 additions and 1542 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
bindings/flow/tester/Tester.actor.cpp
View File

@ -1816,7 +1816,7 @@ int main( int argc, char** argv ) {
}
catch (std::exception& e) {
fprintf(stderr, "std::exception: %s\n", e.what());
TraceEvent(SevError, "MainError").error(unknown_error()).detail("std::exception", e.what());
TraceEvent(SevError, "MainError").error(unknown_error()).detail("RootException", e.what());
flushAndExit(FDB_EXIT_MAIN_EXCEPTION);
}
}

24
documentation/sphinx/source/downloads.rst
View File

@ -10,38 +10,38 @@ macOS
The macOS installation package is supported on macOS 10.7+. It includes the client and (optionally) the server.
* `FoundationDB-5.2.1.pkg <https://www.foundationdb.org/downloads/5.2.1/macOS/installers/FoundationDB-5.2.1.pkg>`_
* `FoundationDB-5.2.3.pkg <https://www.foundationdb.org/downloads/5.2.3/macOS/installers/FoundationDB-5.2.3.pkg>`_
Ubuntu
------
The Ubuntu packages are supported on 64-bit Ubuntu 12.04+, but beware of the Linux kernel bug in Ubuntu 12.x.
* `foundationdb-clients-5.2.1-1_amd64.deb <https://www.foundationdb.org/downloads/5.2.1/ubuntu/installers/foundationdb-clients_5.2.1-1_amd64.deb>`_
* `foundationdb-server-5.2.1-1_amd64.deb <https://www.foundationdb.org/downloads/5.2.1/ubuntu/installers/foundationdb-server_5.2.1-1_amd64.deb>`_ (depends on the clients package)
* `foundationdb-clients-5.2.3-1_amd64.deb <https://www.foundationdb.org/downloads/5.2.3/ubuntu/installers/foundationdb-clients_5.2.3-1_amd64.deb>`_
* `foundationdb-server-5.2.3-1_amd64.deb <https://www.foundationdb.org/downloads/5.2.3/ubuntu/installers/foundationdb-server_5.2.3-1_amd64.deb>`_ (depends on the clients package)
RHEL/CentOS EL6
---------------
The RHEL/CentOS EL6 packages are supported on 64-bit RHEL/CentOS 6.x.
* `foundationdb-clients-5.2.1-1.el6.x86_64.rpm <https://www.foundationdb.org/downloads/5.2.1/rhel6/installers/foundationdb-clients-5.2.1-1.el6.x86_64.rpm>`_
* `foundationdb-server-5.2.1-1.el6.x86_64.rpm <https://www.foundationdb.org/downloads/5.2.1/rhel6/installers/foundationdb-server-5.2.1-1.el6.x86_64.rpm>`_ (depends on the clients package)
* `foundationdb-clients-5.2.3-1.el6.x86_64.rpm <https://www.foundationdb.org/downloads/5.2.3/rhel6/installers/foundationdb-clients-5.2.3-1.el6.x86_64.rpm>`_
* `foundationdb-server-5.2.3-1.el6.x86_64.rpm <https://www.foundationdb.org/downloads/5.2.3/rhel6/installers/foundationdb-server-5.2.3-1.el6.x86_64.rpm>`_ (depends on the clients package)
RHEL/CentOS EL7
---------------
The RHEL/CentOS EL7 packages are supported on 64-bit RHEL/CentOS 7.x.
* `foundationdb-clients-5.2.1-1.el7.x86_64.rpm <https://www.foundationdb.org/downloads/5.2.1/rhel7/installers/foundationdb-clients-5.2.1-1.el7.x86_64.rpm>`_
* `foundationdb-server-5.2.1-1.el7.x86_64.rpm <https://www.foundationdb.org/downloads/5.2.1/rhel7/installers/foundationdb-server-5.2.1-1.el7.x86_64.rpm>`_ (depends on the clients package)
* `foundationdb-clients-5.2.3-1.el7.x86_64.rpm <https://www.foundationdb.org/downloads/5.2.3/rhel7/installers/foundationdb-clients-5.2.3-1.el7.x86_64.rpm>`_
* `foundationdb-server-5.2.3-1.el7.x86_64.rpm <https://www.foundationdb.org/downloads/5.2.3/rhel7/installers/foundationdb-server-5.2.3-1.el7.x86_64.rpm>`_ (depends on the clients package)
Windows
-------
The Windows installer is supported on 64-bit Windows XP and later. It includes the client and (optionally) the server.
* `foundationdb-5.2.1-x64.msi <https://www.foundationdb.org/downloads/5.2.1/windows/installers/foundationdb-5.2.1-x64.msi>`_
* `foundationdb-5.2.3-x64.msi <https://www.foundationdb.org/downloads/5.2.3/windows/installers/foundationdb-5.2.3-x64.msi>`_
API Language Bindings
=====================
@ -58,18 +58,18 @@ On macOS and Windows, the FoundationDB Python API bindings are installed as part
If you need to use the FoundationDB Python API from other Python installations or paths, download the Python package:
* `foundationdb-5.2.1.tar.gz <https://www.foundationdb.org/downloads/5.2.1/bindings/python/foundationdb-5.2.1.tar.gz>`_
* `foundationdb-5.2.3.tar.gz <https://www.foundationdb.org/downloads/5.2.3/bindings/python/foundationdb-5.2.3.tar.gz>`_
Ruby 1.9.3/2.0.0+
-----------------
* `fdb-5.2.1.gem <https://www.foundationdb.org/downloads/5.2.1/bindings/ruby/fdb-5.2.1.gem>`_
* `fdb-5.2.3.gem <https://www.foundationdb.org/downloads/5.2.3/bindings/ruby/fdb-5.2.3.gem>`_
Java 8+
-------
* `fdb-java-5.2.1.jar <https://www.foundationdb.org/downloads/5.2.1/bindings/java/fdb-java-5.2.1.jar>`_
* `fdb-java-5.2.1-javadoc.jar <https://www.foundationdb.org/downloads/5.2.1/bindings/java/fdb-java-5.2.1-javadoc.jar>`_
* `fdb-java-5.2.3.jar <https://www.foundationdb.org/downloads/5.2.3/bindings/java/fdb-java-5.2.3.jar>`_
* `fdb-java-5.2.3-javadoc.jar <https://www.foundationdb.org/downloads/5.2.3/bindings/java/fdb-java-5.2.3-javadoc.jar>`_
Go 1.1+
-------

2
documentation/sphinx/source/old-release-notes/release-notes-450.rst
View File

@ -83,7 +83,7 @@ Fixes
* A multi-version API external client that was a duplicate could cause a crash at client shutdown. <rdar://problem/25838039>
* A multi-version API external client that failed to load would cause a crash at client shutdown. <rdar://problem/26611919>
* Status could report an incorrect database size in rare scenarios. <rdar://problem/25295683>
* Data distribution would stopped working (both exclude and new recruitment) if a call to commit on the storage server could hang indefinitely. <rdar://problem/26276166>
* Data distribution would stop working (both exclude and new recruitment) if a call to commit on the storage server could hang indefinitely. <rdar://problem/26276166>
* ``fdbcli`` would quit on some commands that were not errors or continue after some real errors when run with ``--exec``. <rdar://problem/25755317>
* Trace files generated by clients using the multi-version API could have name collisions. <rdar://problem/25884486>

1
documentation/sphinx/source/release-notes.rst
View File

@ -29,6 +29,7 @@ Other Changes
-------------
* Does not support upgrades from any version older than 5.0.
* Renamed the trace log attribute ``logGroup`` to ``LogGroup``.
Earlier release notes
---------------------

4
fdbbackup/backup.actor.cpp
View File

@ -1159,7 +1159,7 @@ ACTOR Future<std::string> getLayerStatus(Reference<ReadYourWritesTransaction> tr
if (backupVersion[i].get().present()) {
double seconds_behind = ((double)readVer - BinaryReader::fromStringRef<Version>(backupVersion[i].get().get(), Unversioned())) / CLIENT_KNOBS->CORE_VERSIONSPERSECOND;
tagRoot.create("seconds_behind") = seconds_behind;
//TraceEvent("BackupMetrics").detail("secondsBehind", seconds_behind);
//TraceEvent("BackupMetrics").detail("SecondsBehind", seconds_behind);
}
tagRoot.create("backup_state") = BackupAgentBase::getStateText(status);
@ -2999,7 +2999,7 @@ int main(int argc, char* argv[]) {
TraceEvent(SevError, "MainError").error(e);
status = FDB_EXIT_MAIN_ERROR;
} catch (std::exception& e) {
TraceEvent(SevError, "MainError").error(unknown_error()).detail("std::exception", e.what());
TraceEvent(SevError, "MainError").error(unknown_error()).detail("RootException", e.what());
status = FDB_EXIT_MAIN_EXCEPTION;
}

6
fdbcli/fdbcli.actor.cpp
View File

@ -2060,7 +2060,7 @@ void fdbcli_comp_cmd(std::string const& text, std::vector<std::string>& lc) {
void LogCommand(std::string line, UID randomID, std::string errMsg) {
printf("%s\n", errMsg.c_str());
TraceEvent(SevInfo, "CLICommandLog", randomID).detail("command", printable(StringRef(line))).detail("error", printable(StringRef(errMsg)));
TraceEvent(SevInfo, "CLICommandLog", randomID).detail("Command", printable(StringRef(line))).detail("Error", printable(StringRef(errMsg)));
}
struct CLIOptions {
@ -2305,7 +2305,7 @@ ACTOR Future<int> cli(CLIOptions opt, LineNoise* plinenoise) {
try {
state UID randomID = g_random->randomUniqueID();
TraceEvent(SevInfo, "CLICommandLog", randomID).detail("command", printable(StringRef(line)));
TraceEvent(SevInfo, "CLICommandLog", randomID).detail("Command", printable(StringRef(line)));
bool malformed, partial;
state std::vector<std::vector<StringRef>> parsed = parseLine(line, malformed, partial);
@ -3079,7 +3079,7 @@ ACTOR Future<int> cli(CLIOptions opt, LineNoise* plinenoise) {
is_error = true;
}
TraceEvent(SevInfo, "CLICommandLog", randomID).detail("command", printable(StringRef(line))).detail("is_error", is_error);
TraceEvent(SevInfo, "CLICommandLog", randomID).detail("Command", printable(StringRef(line))).detail("IsError", is_error);
} catch (Error& e) {
if(e.code() != error_code_actor_cancelled)

2
fdbclient/BackupAgent.h
View File

@ -423,7 +423,7 @@ bool copyParameter(Reference<Task> source, Reference<Task> dest, Key key);
Version getVersionFromString(std::string const& value);
Standalone<VectorRef<KeyRangeRef>> getLogRanges(Version beginVersion, Version endVersion, Key destUidValue, int blockSize = CLIENT_KNOBS->LOG_RANGE_BLOCK_SIZE);
Standalone<VectorRef<KeyRangeRef>> getApplyRanges(Version beginVersion, Version endVersion, Key backupUid);
Future<Void> eraseLogData(Database cx, Key logUidValue, Key destUidValue, Optional<Version> beginVersion = Optional<Version>(), Optional<Version> endVersion = Optional<Version>(), bool checkBackupUid = false, Version backupUid = 0);
Future<Void> eraseLogData(Database cx, Key logUidValue, Key destUidValue, Optional<Version> endVersion = Optional<Version>(), bool checkBackupUid = false, Version backupUid = 0);
Key getApplyKey( Version version, Key backupUid );
std::pair<uint64_t, uint32_t> decodeBKMutationLogKey(Key key);
Standalone<VectorRef<MutationRef>> decodeBackupLogValue(StringRef value);

256
fdbclient/BackupAgentBase.actor.cpp
View File

@ -77,7 +77,7 @@ Standalone<VectorRef<KeyRangeRef>> getLogRanges(Version beginVersion, Version en
Key baLogRangePrefix = destUidValue.withPrefix(backupLogKeys.begin);
//TraceEvent("getLogRanges").detail("destUidValue", destUidValue).detail("prefix", printable(StringRef(baLogRangePrefix)));
//TraceEvent("GetLogRanges").detail("DestUidValue", destUidValue).detail("Prefix", printable(StringRef(baLogRangePrefix)));
for (int64_t vblock = beginVersion / blockSize; vblock < (endVersion + blockSize - 1) / blockSize; ++vblock) {
int64_t tb = vblock * blockSize / CLIENT_KNOBS->LOG_RANGE_BLOCK_SIZE;
@ -100,7 +100,7 @@ Standalone<VectorRef<KeyRangeRef>> getApplyRanges(Version beginVersion, Version
Key baLogRangePrefix = backupUid.withPrefix(applyLogKeys.begin);
//TraceEvent("getLogRanges").detail("backupUid", backupUid).detail("prefix", printable(StringRef(baLogRangePrefix)));
//TraceEvent("GetLogRanges").detail("BackupUid", backupUid).detail("Prefix", printable(StringRef(baLogRangePrefix)));
for (int64_t vblock = beginVersion / CLIENT_KNOBS->APPLY_BLOCK_SIZE; vblock < (endVersion + CLIENT_KNOBS->APPLY_BLOCK_SIZE - 1) / CLIENT_KNOBS->APPLY_BLOCK_SIZE; ++vblock) {
int64_t tb = vblock * CLIENT_KNOBS->APPLY_BLOCK_SIZE / CLIENT_KNOBS->LOG_RANGE_BLOCK_SIZE;
@ -148,8 +148,8 @@ Standalone<VectorRef<MutationRef>> decodeBackupLogValue(StringRef value) {
memcpy(&protocolVersion, value.begin(), sizeof(uint64_t));
offset += sizeof(uint64_t);
if (protocolVersion <= 0x0FDB00A200090001){
TraceEvent(SevError, "decodeBackupLogValue").detail("incompatible_protocol_version", protocolVersion)
.detail("valueSize", value.size()).detail("value", printable(value));
TraceEvent(SevError, "DecodeBackupLogValue").detail("IncompatibleProtocolVersion", protocolVersion)
.detail("ValueSize", value.size()).detail("Value", printable(value));
throw incompatible_protocol_version();
}
@ -188,14 +188,14 @@ Standalone<VectorRef<MutationRef>> decodeBackupLogValue(StringRef value) {
ASSERT(consumed == totalBytes);
if (value.size() != offset) {
TraceEvent(SevError, "BA_decodeBackupLogValue").detail("unexpected_extra_data_size", value.size()).detail("offset", offset).detail("totalBytes", totalBytes).detail("consumed", consumed).detail("originalOffset", originalOffset);
TraceEvent(SevError, "BA_DecodeBackupLogValue").detail("UnexpectedExtraDataSize", value.size()).detail("Offset", offset).detail("TotalBytes", totalBytes).detail("Consumed", consumed).detail("OriginalOffset", originalOffset);
throw restore_corrupted_data();
}
return result;
}
catch (Error& e) {
TraceEvent(e.code() == error_code_restore_missing_data ? SevWarn : SevError, "BA_decodeBackupLogValue").error(e).GetLastError().detail("valueSize", value.size()).detail("value", printable(value));
TraceEvent(e.code() == error_code_restore_missing_data ? SevWarn : SevError, "BA_DecodeBackupLogValue").error(e).GetLastError().detail("ValueSize", value.size()).detail("Value", printable(value));
throw;
}
}
@ -207,8 +207,8 @@ void decodeBackupLogValue(Arena& arena, VectorRef<MutationRef>& result, int& mut
memcpy(&protocolVersion, value.begin(), sizeof(uint64_t));
offset += sizeof(uint64_t);
if (protocolVersion <= 0x0FDB00A200090001){
TraceEvent(SevError, "decodeBackupLogValue").detail("incompatible_protocol_version", protocolVersion)
.detail("valueSize", value.size()).detail("value", printable(value));
TraceEvent(SevError, "DecodeBackupLogValue").detail("IncompatibleProtocolVersion", protocolVersion)
.detail("ValueSize", value.size()).detail("Value", printable(value));
throw incompatible_protocol_version();
}
@ -280,7 +280,7 @@ void decodeBackupLogValue(Arena& arena, VectorRef<MutationRef>& result, int& mut
}
else {
Version ver = key_version->rangeContaining(logValue.param1).value();
//TraceEvent("ApplyMutation").detail("logValue", logValue.toString()).detail("version", version).detail("ver", ver).detail("apply", version > ver && ver != invalidVersion);
//TraceEvent("ApplyMutation").detail("LogValue", logValue.toString()).detail("Version", version).detail("Ver", ver).detail("Apply", version > ver && ver != invalidVersion);
if (version > ver && ver != invalidVersion) {
if(removePrefix.size()) {
logValue.param1 = logValue.param1.removePrefix(removePrefix);
@ -298,12 +298,12 @@ void decodeBackupLogValue(Arena& arena, VectorRef<MutationRef>& result, int& mut
ASSERT(consumed == totalBytes);
if (value.size() != offset) {
TraceEvent(SevError, "BA_decodeBackupLogValue").detail("unexpected_extra_data_size", value.size()).detail("offset", offset).detail("totalBytes", totalBytes).detail("consumed", consumed).detail("originalOffset", originalOffset);
TraceEvent(SevError, "BA_DecodeBackupLogValue").detail("UnexpectedExtraDataSize", value.size()).detail("Offset", offset).detail("TotalBytes", totalBytes).detail("Consumed", consumed).detail("OriginalOffset", originalOffset);
throw restore_corrupted_data();
}
}
catch (Error& e) {
TraceEvent(e.code() == error_code_restore_missing_data ? SevWarn : SevError, "BA_decodeBackupLogValue").error(e).GetLastError().detail("valueSize", value.size()).detail("value", printable(value));
TraceEvent(e.code() == error_code_restore_missing_data ? SevWarn : SevError, "BA_DecodeBackupLogValue").error(e).GetLastError().detail("ValueSize", value.size()).detail("Value", printable(value));
throw;
}
}
@ -312,7 +312,7 @@ ACTOR Future<Void> logErrorWorker(Reference<ReadYourWritesTransaction> tr, Key k
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr->setOption(FDBTransactionOptions::LOCK_AWARE);
if(now() - lastErrorTime > CLIENT_KNOBS->BACKUP_ERROR_DELAY) {
TraceEvent("BA_logError").detail("key", printable(keyErrors)).detail("message", message);
TraceEvent("BA_LogError").detail("Key", printable(keyErrors)).detail("Message", message);
lastErrorTime = now();
}
tr->set(keyErrors, message);
@ -423,19 +423,19 @@ ACTOR Future<Void> readCommitted(Database cx, PromiseStream<RCGroup> results, Fu
int index(0);
for (auto & s : rangevalue){
uint64_t groupKey = groupBy(s.key).first;
//TraceEvent("log_readCommitted").detail("groupKey", groupKey).detail("skipGroup", skipGroup).detail("nextKey", printable(nextKey.key)).detail("end", printable(end.key)).detail("valuesize", value.size()).detail("index",index++).detail("size",s.value.size());
//TraceEvent("Log_ReadCommitted").detail("GroupKey", groupKey).detail("SkipGroup", skipGroup).detail("NextKey", printable(nextKey.key)).detail("End", printable(end.key)).detail("Valuesize", value.size()).detail("Index",index++).detail("Size",s.value.size());
if (groupKey != skipGroup){
if (rcGroup.version == -1){
rcGroup.version = tr.getReadVersion().get();
rcGroup.groupKey = groupKey;
}
else if (rcGroup.groupKey != groupKey) {
//TraceEvent("log_readCommitted").detail("sendGroup0", rcGroup.groupKey).detail("itemSize", rcGroup.items.size()).detail("data_length",rcGroup.items[0].value.size());
//TraceEvent("Log_ReadCommitted").detail("SendGroup0", rcGroup.groupKey).detail("ItemSize", rcGroup.items.size()).detail("Data_length",rcGroup.items[0].value.size());
//state uint32_t len(0);
//for (size_t j = 0; j < rcGroup.items.size(); ++j) {
// len += rcGroup.items[j].value.size();
//}
//TraceEvent("SendGroup").detail("groupKey", rcGroup.groupKey).detail("version", rcGroup.version).detail("length", len).detail("releaser.remaining", releaser.remaining);
//TraceEvent("SendGroup").detail("GroupKey", rcGroup.groupKey).detail("Version", rcGroup.version).detail("Length", len).detail("Releaser.remaining", releaser.remaining);
releaser.remaining -= rcGroup.items.expectedSize(); //its the responsibility of the caller to release after this point
ASSERT(releaser.remaining >= 0);
results.send(rcGroup);
@ -454,7 +454,7 @@ ACTOR Future<Void> readCommitted(Database cx, PromiseStream<RCGroup> results, Fu
if (rcGroup.version != -1){
releaser.remaining -= rcGroup.items.expectedSize(); //its the responsibility of the caller to release after this point
ASSERT(releaser.remaining >= 0);
//TraceEvent("log_readCommitted").detail("sendGroup1", rcGroup.groupKey).detail("itemSize", rcGroup.items.size()).detail("data_length", rcGroup.items[0].value.size());
//TraceEvent("Log_ReadCommitted").detail("SendGroup1", rcGroup.groupKey).detail("ItemSize", rcGroup.items.size()).detail("DataLength", rcGroup.items[0].value.size());
results.send(rcGroup);
}
@ -619,151 +619,117 @@ ACTOR Future<Void> applyMutations(Database cx, Key uid, Key addPrefix, Key remov
beginVersion = newEndVersion;
}
} catch( Error &e ) {
TraceEvent(e.code() == error_code_restore_missing_data ? SevWarnAlways : SevError, "AM_error").error(e);
TraceEvent(e.code() == error_code_restore_missing_data ? SevWarnAlways : SevError, "ApplyMutationsError").error(e);
throw;
}
}
ACTOR Future<Void> _clearLogRanges(Reference<ReadYourWritesTransaction> tr, bool clearVersionHistory, Key logUidValue, Key destUidValue, Version beginVersion, Version endVersion) {
ACTOR static Future<Void> _eraseLogData(Database cx, Key logUidValue, Key destUidValue, Optional<Version> endVersion, bool checkBackupUid, Version backupUid) {
state Key backupLatestVersionsPath = destUidValue.withPrefix(backupLatestVersionsPrefix);
state Key backupLatestVersionsKey = logUidValue.withPrefix(backupLatestVersionsPath);
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr->setOption(FDBTransactionOptions::LOCK_AWARE);
state Standalone<RangeResultRef> backupVersions = wait(tr->getRange(KeyRangeRef(backupLatestVersionsPath, strinc(backupLatestVersionsPath)), CLIENT_KNOBS->TOO_MANY));
// Make sure version history key does exist and lower the beginVersion if needed
bool foundSelf = false;
for (auto backupVersion : backupVersions) {
Key currLogUidValue = backupVersion.key.removePrefix(backupLatestVersionsPrefix).removePrefix(destUidValue);
if (currLogUidValue == logUidValue) {
foundSelf = true;
beginVersion = std::min(beginVersion, BinaryReader::fromStringRef<Version>(backupVersion.value, Unversioned()));
}
}
// Do not clear anything if version history key cannot be found
if (!foundSelf) {
if (!destUidValue.size()) {
return Void();
}
Version nextSmallestVersion = endVersion;
bool clearLogRangesRequired = true;
state Reference<ReadYourWritesTransaction> tr(new ReadYourWritesTransaction(cx));
loop{
try {
tr->setOption(FDBTransactionOptions::LOCK_AWARE);
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
// More than one backup/DR with the same range
if (backupVersions.size() > 1) {
for (auto backupVersion : backupVersions) {
Key currLogUidValue = backupVersion.key.removePrefix(backupLatestVersionsPrefix).removePrefix(destUidValue);
Version currVersion = BinaryReader::fromStringRef<Version>(backupVersion.value, Unversioned());
if (currLogUidValue == logUidValue) {
continue;
} else if (currVersion > beginVersion) {
nextSmallestVersion = std::min(currVersion, nextSmallestVersion);
} else {
// If we can find a version less than or equal to beginVersion, clearing log ranges is not required
clearLogRangesRequired = false;
break;
}
}
}
if (clearVersionHistory && backupVersions.size() == 1) {
// Clear version history
tr->clear(prefixRange(backupLatestVersionsPath));
// Clear everything under blog/[destUid]
tr->clear(prefixRange(destUidValue.withPrefix(backupLogKeys.begin)));
// Disable committing mutations into blog
tr->clear(prefixRange(destUidValue.withPrefix(logRangesRange.begin)));
} else {
if (clearVersionHistory) {
// Clear current backup version history
tr->clear(backupLatestVersionsKey);
} else {
// Update current backup latest version
tr->set(backupLatestVersionsKey, BinaryWriter::toValue<Version>(endVersion, Unversioned()));
}
// Clear log ranges if needed
if (clearLogRangesRequired) {
Standalone<VectorRef<KeyRangeRef>> ranges = getLogRanges(beginVersion, nextSmallestVersion, destUidValue);
for (auto& range : ranges) {
tr->clear(range);
}
}
}
return Void();
}
// The difference between beginVersion and endVersion should not be too large
Future<Void> clearLogRanges(Reference<ReadYourWritesTransaction> tr, bool clearVersionHistory, Key logUidValue, Key destUidValue, Version beginVersion, Version endVersion) {
return _clearLogRanges(tr, clearVersionHistory, logUidValue, destUidValue, beginVersion, endVersion);
}
ACTOR static Future<Void> _eraseLogData(Database cx, Key logUidValue, Key destUidValue, Optional<Version> beginVersion, Optional<Version> endVersion, bool checkBackupUid, Version backupUid) {
if ((beginVersion.present() && endVersion.present() && endVersion.get() <= beginVersion.get()) || !destUidValue.size())
return Void();
state Version currBeginVersion;
state Version endVersionValue;
state Version currEndVersion;
state bool clearVersionHistory;
ASSERT(beginVersion.present() == endVersion.present());
if (beginVersion.present()) {
currBeginVersion = beginVersion.get();
endVersionValue = endVersion.get();
clearVersionHistory = false;
} else {
// If beginVersion and endVersion are not presented, it means backup is done and we need to clear version history.
// Set currBeginVersion to INTMAX_MAX and it will be set to the correct version in clearLogRanges().
// Set endVersionValue to INTMAX_MAX since we need to clear log ranges up to next smallest version.
currBeginVersion = endVersionValue = currEndVersion = INTMAX_MAX;
clearVersionHistory = true;
}
while (currBeginVersion < endVersionValue || clearVersionHistory) {
state Reference<ReadYourWritesTransaction> tr(new ReadYourWritesTransaction(cx));
loop{
try {
tr->setOption(FDBTransactionOptions::LOCK_AWARE);
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
if (checkBackupUid) {
Subspace sourceStates = Subspace(databaseBackupPrefixRange.begin).get(BackupAgentBase::keySourceStates).get(logUidValue);
Optional<Value> v = wait( tr->get( sourceStates.pack(DatabaseBackupAgent::keyFolderId) ) );
if(v.present() && BinaryReader::fromStringRef<Version>(v.get(), Unversioned()) > backupUid)
return Void();
}
if (!clearVersionHistory) {
currEndVersion = std::min(currBeginVersion + CLIENT_KNOBS->CLEAR_LOG_RANGE_COUNT * CLIENT_KNOBS->LOG_RANGE_BLOCK_SIZE, endVersionValue);
}
Void _ = wait(clearLogRanges(tr, clearVersionHistory, logUidValue, destUidValue, currBeginVersion, currEndVersion));
Void _ = wait(tr->commit());
if (clearVersionHistory) {
if (checkBackupUid) {
Subspace sourceStates = Subspace(databaseBackupPrefixRange.begin).get(BackupAgentBase::keySourceStates).get(logUidValue);
Optional<Value> v = wait( tr->get( sourceStates.pack(DatabaseBackupAgent::keyFolderId) ) );
if(v.present() && BinaryReader::fromStringRef<Version>(v.get(), Unversioned()) > backupUid)
return Void();
}
state Standalone<RangeResultRef> backupVersions = wait(tr->getRange(KeyRangeRef(backupLatestVersionsPath, strinc(backupLatestVersionsPath)), CLIENT_KNOBS->TOO_MANY));
// Make sure version history key does exist and lower the beginVersion if needed
state Version currBeginVersion = invalidVersion;
for (auto backupVersion : backupVersions) {
Key currLogUidValue = backupVersion.key.removePrefix(backupLatestVersionsPrefix).removePrefix(destUidValue);
if (currLogUidValue == logUidValue) {
currBeginVersion = BinaryReader::fromStringRef<Version>(backupVersion.value, Unversioned());
break;
}
}
// Do not clear anything if version history key cannot be found
if (currBeginVersion == invalidVersion) {
return Void();
}
state Version currEndVersion = currBeginVersion + CLIENT_KNOBS->CLEAR_LOG_RANGE_COUNT * CLIENT_KNOBS->LOG_RANGE_BLOCK_SIZE;
if(endVersion.present()) {
currEndVersion = std::min(currEndVersion, endVersion.get());
}
state Version nextSmallestVersion = currEndVersion;
bool clearLogRangesRequired = true;
// More than one backup/DR with the same range
if (backupVersions.size() > 1) {
for (auto backupVersion : backupVersions) {
Key currLogUidValue = backupVersion.key.removePrefix(backupLatestVersionsPrefix).removePrefix(destUidValue);
Version currVersion = BinaryReader::fromStringRef<Version>(backupVersion.value, Unversioned());
if (currLogUidValue == logUidValue) {
continue;
} else if (currVersion > currBeginVersion) {
nextSmallestVersion = std::min(currVersion, nextSmallestVersion);
} else {
// If we can find a version less than or equal to beginVersion, clearing log ranges is not required
clearLogRangesRequired = false;
break;
}
}
}
if (!endVersion.present() && backupVersions.size() == 1) {
// Clear version history
tr->clear(prefixRange(backupLatestVersionsPath));
// Clear everything under blog/[destUid]
tr->clear(prefixRange(destUidValue.withPrefix(backupLogKeys.begin)));
// Disable committing mutations into blog
tr->clear(prefixRange(destUidValue.withPrefix(logRangesRange.begin)));
} else {
if (!endVersion.present() && currEndVersion >= nextSmallestVersion) {
// Clear current backup version history
tr->clear(backupLatestVersionsKey);
} else {
// Update current backup latest version
tr->set(backupLatestVersionsKey, BinaryWriter::toValue<Version>(currEndVersion, Unversioned()));
}
currBeginVersion = currEndVersion;
break;
} catch (Error &e) {
Void _ = wait(tr->onError(e));
// Clear log ranges if needed
if (clearLogRangesRequired) {
Standalone<VectorRef<KeyRangeRef>> ranges = getLogRanges(currBeginVersion, nextSmallestVersion, destUidValue);
for (auto& range : ranges) {
tr->clear(range);
}
}
}
Void _ = wait(tr->commit());
if (!endVersion.present() && (backupVersions.size() == 1 || currEndVersion >= nextSmallestVersion)) {
return Void();
}
if(endVersion.present() && currEndVersion == endVersion.get()) {
return Void();
}
tr->reset();
} catch (Error &e) {
Void _ = wait(tr->onError(e));
}
}
return Void();
}
Future<Void> eraseLogData(Database cx, Key logUidValue, Key destUidValue, Optional<Version> beginVersion, Optional<Version> endVersion, bool checkBackupUid, Version backupUid) {
return _eraseLogData(cx, logUidValue, destUidValue, beginVersion, endVersion, checkBackupUid, backupUid);
}
Future<Void> eraseLogData(Database cx, Key logUidValue, Key destUidValue, Optional<Version> endVersion, bool checkBackupUid, Version backupUid) {
return _eraseLogData(cx, logUidValue, destUidValue, endVersion, checkBackupUid, backupUid);
}

4
fdbclient/BackupContainer.actor.cpp
View File

@ -1184,7 +1184,7 @@ Reference<IBackupContainer> IBackupContainer::openContainer(std::string url)
m.detail("Description", "Invalid container specification. See help.").detail("URL", url);
if(e.code() == error_code_backup_invalid_url)
m.detail("lastOpenError", lastOpenError);
m.detail("LastOpenError", lastOpenError);
throw;
}
}
@ -1222,7 +1222,7 @@ ACTOR Future<std::vector<std::string>> listContainers_impl(std::string baseURL)
m.detail("Description", "Invalid backup container URL prefix. See help.").detail("URL", baseURL);
if(e.code() == error_code_backup_invalid_url)
m.detail("lastOpenError", IBackupContainer::lastOpenError);
m.detail("LastOpenError", IBackupContainer::lastOpenError);
throw;
}
}

2
fdbclient/CoordinationInterface.h
View File

@ -93,7 +93,7 @@ private:
struct LeaderInfo {
UID changeID;
uint64_t mask = ~(127ll << 57);
static const uint64_t mask = ~(127ll << 57);
Value serializedInfo;
bool forward; // If true, serializedInfo is a connection string instead!

71
fdbclient/DatabaseBackupAgent.actor.cpp
View File

@ -109,7 +109,7 @@ namespace dbBackup {
Future<Void> checkTaskVersion(Tr tr, Reference<Task> task, StringRef name, uint32_t version) {
uint32_t taskVersion = task->getVersion();
if (taskVersion > version) {
TraceEvent(SevError, "BA_BackupRangeTaskFunc_execute").detail("taskVersion", taskVersion).detail("Name", printable(name)).detail("Version", version);
TraceEvent(SevError, "BA_BackupRangeTaskFunc_execute").detail("TaskVersion", taskVersion).detail("Name", printable(name)).detail("Version", version);
Void _ = wait(logError(tr, Subspace(databaseBackupPrefixRange.begin).get(BackupAgentBase::keyErrors).pack(task->params[BackupAgentBase::keyConfigLogUid]),
format("ERROR: %s task version `%lu' is greater than supported version `%lu'", task->params[Task::reservedTaskParamKeyType].toString().c_str(), (unsigned long)taskVersion, (unsigned long)version)));
@ -299,7 +299,7 @@ namespace dbBackup {
return Void();
}
//TraceEvent("DBA_Range").detail("range", printable(KeyRangeRef(rangeBegin, rangeEnd))).detail("version", values.second).detail("size", values.first.size()).detail("LogUID", printable(task->params[BackupAgentBase::keyConfigLogUid])).detail("addPrefix", printable(addPrefix)).detail("removePrefix", printable(removePrefix));
//TraceEvent("DBA_Range").detail("Range", printable(KeyRangeRef(rangeBegin, rangeEnd))).detail("Version", values.second).detail("Size", values.first.size()).detail("LogUID", printable(task->params[BackupAgentBase::keyConfigLogUid])).detail("AddPrefix", printable(addPrefix)).detail("RemovePrefix", printable(removePrefix));
Subspace krv(conf.get(DatabaseBackupAgent::keyRangeVersions));
state KeyRange versionRange = singleKeyRange(krv.pack(values.second));
@ -334,7 +334,7 @@ namespace dbBackup {
break;
if( values.first[valueLoc].key >= backupVersions.get()[versionLoc].key ) {
//TraceEvent("DBA_set", debugID).detail("key", printable(values.first[valueLoc].key)).detail("value", printable(values.first[valueLoc].value));
//TraceEvent("DBA_Set", debugID).detail("Key", printable(values.first[valueLoc].key)).detail("Value", printable(values.first[valueLoc].value));
tr->set(values.first[valueLoc].key.removePrefix(removePrefix).withPrefix(addPrefix), values.first[valueLoc].value);
bytesSet += values.first[valueLoc].expectedSize() - removePrefix.expectedSize() + addPrefix.expectedSize();
}
@ -347,7 +347,7 @@ namespace dbBackup {
Void _ = wait(tr->commit());
Params.bytesWritten().set(task, Params.bytesWritten().getOrDefault(task) + bytesSet);
//TraceEvent("DBA_setComplete", debugID).detail("ver", values.second).detail("logVersion", logVersion).detail("readVersion", readVer).detail("commitVer", tr.getCommittedVersion()).detail("range", printable(versionRange));
//TraceEvent("DBA_SetComplete", debugID).detail("Ver", values.second).detail("LogVersion", logVersion).detail("ReadVersion", readVer).detail("CommitVer", tr.getCommittedVersion()).detail("Range", printable(versionRange));
if(backupVersions.get().more) {
tr->reset();
@ -431,7 +431,7 @@ namespace dbBackup {
srcTr.setOption(FDBTransactionOptions::LOCK_AWARE);
Version readVersion = wait(srcTr.getReadVersion());
tr->set(states.pack(DatabaseBackupAgent::keyCopyStop), BinaryWriter::toValue(readVersion, Unversioned()));
TraceEvent("DBA_FinishFullBackup").detail("copyStop", readVersion);
TraceEvent("DBA_FinishFullBackup").detail("CopyStop", readVersion);
Void _ = wait(taskBucket->finish(tr, task));
return Void();
@ -476,21 +476,20 @@ namespace dbBackup {
Void _ = wait(checkTaskVersion(cx, task, EraseLogRangeTaskFunc::name, EraseLogRangeTaskFunc::version));
Version beginVersion = BinaryReader::fromStringRef<Version>(task->params[DatabaseBackupAgent::keyBeginVersion], Unversioned());
Version endVersion = BinaryReader::fromStringRef<Version>(task->params[DatabaseBackupAgent::keyEndVersion], Unversioned());
Void _ = wait(eraseLogData(taskBucket->src, task->params[BackupAgentBase::keyConfigLogUid], task->params[BackupAgentBase::destUid], Optional<Version>(beginVersion), Optional<Version>(endVersion), true, BinaryReader::fromStringRef<Version>(task->params[BackupAgentBase::keyFolderId], Unversioned())));
Void _ = wait(eraseLogData(taskBucket->src, task->params[BackupAgentBase::keyConfigLogUid], task->params[BackupAgentBase::destUid], Optional<Version>(endVersion), true, BinaryReader::fromStringRef<Version>(task->params[BackupAgentBase::keyFolderId], Unversioned())));
return Void();
}
ACTOR static Future<Key> addTask(Reference<ReadYourWritesTransaction> tr, Reference<TaskBucket> taskBucket, Reference<Task> parentTask, Version beginVersion, Version endVersion, TaskCompletionKey completionKey, Reference<TaskFuture> waitFor = Reference<TaskFuture>()) {
ACTOR static Future<Key> addTask(Reference<ReadYourWritesTransaction> tr, Reference<TaskBucket> taskBucket, Reference<Task> parentTask, Version endVersion, TaskCompletionKey completionKey, Reference<TaskFuture> waitFor = Reference<TaskFuture>()) {
Key doneKey = wait(completionKey.get(tr, taskBucket));
Reference<Task> task(new Task(EraseLogRangeTaskFunc::name, EraseLogRangeTaskFunc::version, doneKey, 1));
copyDefaultParameters(parentTask, task);
task->params[DatabaseBackupAgent::keyBeginVersion] = BinaryWriter::toValue(beginVersion, Unversioned());
task->params[DatabaseBackupAgent::keyBeginVersion] = BinaryWriter::toValue(1, Unversioned()); //FIXME: remove in 6.X, only needed for 5.2 backward compatibility
task->params[DatabaseBackupAgent::keyEndVersion] = BinaryWriter::toValue(endVersion, Unversioned());
if (!waitFor) {
@ -732,7 +731,7 @@ namespace dbBackup {
state Version stopVersionData = stopValue.present() ? BinaryReader::fromStringRef<Version>(stopValue.get(), Unversioned()) : -1;
if(endVersion - beginVersion > g_random->randomInt64(0, CLIENT_KNOBS->BACKUP_VERSION_DELAY)) {
TraceEvent("DBA_CopyLogs").detail("beginVersion", beginVersion).detail("applyVersion", applyVersion).detail("endVersion", endVersion).detail("stopVersionData", stopVersionData).detail("LogUID", printable(task->params[BackupAgentBase::keyConfigLogUid]));
TraceEvent("DBA_CopyLogs").detail("BeginVersion", beginVersion).detail("ApplyVersion", applyVersion).detail("EndVersion", endVersion).detail("StopVersionData", stopVersionData).detail("LogUID", printable(task->params[BackupAgentBase::keyConfigLogUid]));
}
if ((stopVersionData == -1) || (stopVersionData >= applyVersion)) {
@ -749,7 +748,7 @@ namespace dbBackup {
// Do not erase at the first time
if (prevBeginVersion > 0) {
addTaskVector.push_back(EraseLogRangeTaskFunc::addTask(tr, taskBucket, task, prevBeginVersion, beginVersion, TaskCompletionKey::joinWith(allPartsDone)));
addTaskVector.push_back(EraseLogRangeTaskFunc::addTask(tr, taskBucket, task, beginVersion, TaskCompletionKey::joinWith(allPartsDone)));
}
Void _ = wait(waitForAll(addTaskVector) && taskBucket->finish(tr, task));
@ -814,7 +813,7 @@ namespace dbBackup {
Optional<Value> endValue = wait(tr2.get(task->params[BackupAgentBase::keyConfigLogUid].withPrefix(applyMutationsEndRange.begin)));
Version endVersion = endValue.present() ? BinaryReader::fromStringRef<Version>(endValue.get(), Unversioned()) : -1;
//TraceEvent("DBA_FinishedFullBackup").detail("applied", appliedVersion).detail("endVer", endVersion);
//TraceEvent("DBA_FinishedFullBackup").detail("Applied", appliedVersion).detail("EndVer", endVersion);
if(appliedVersion < endVersion) {
Void _ = wait(delay(FLOW_KNOBS->PREVENT_FAST_SPIN_DELAY));
task->params[FinishedFullBackupTaskFunc::keyInsertTask] = StringRef();
@ -856,7 +855,7 @@ namespace dbBackup {
}
Version backupUid = BinaryReader::fromStringRef<Version>(task->params[BackupAgentBase::keyFolderId], Unversioned());
Void _ = wait(eraseLogData(taskBucket->src, logUidValue, destUidValue, Optional<Version>(), Optional<Version>(), true, backupUid));
Void _ = wait(eraseLogData(taskBucket->src, logUidValue, destUidValue, Optional<Version>(), true, backupUid));
return Void();
}
@ -936,7 +935,7 @@ namespace dbBackup {
Optional<Value> stopWhenDone = wait(fStopWhenDone);
if(endVersion - beginVersion > g_random->randomInt64(0, CLIENT_KNOBS->BACKUP_VERSION_DELAY)) {
TraceEvent("DBA_CopyDiffLogs").detail("beginVersion", beginVersion).detail("endVersion", endVersion).detail("LogUID", printable(task->params[BackupAgentBase::keyConfigLogUid]));
TraceEvent("DBA_CopyDiffLogs").detail("BeginVersion", beginVersion).detail("EndVersion", endVersion).detail("LogUID", printable(task->params[BackupAgentBase::keyConfigLogUid]));
}
if (!stopWhenDone.present()) {
@ -952,7 +951,7 @@ namespace dbBackup {
}
if (prevBeginVersion > 0) {
addTaskVector.push_back(EraseLogRangeTaskFunc::addTask(tr, taskBucket, task, prevBeginVersion, beginVersion, TaskCompletionKey::joinWith(allPartsDone)));
addTaskVector.push_back(EraseLogRangeTaskFunc::addTask(tr, taskBucket, task, beginVersion, TaskCompletionKey::joinWith(allPartsDone)));
}
Void _ = wait(waitForAll(addTaskVector) && taskBucket->finish(tr, task));
@ -1212,7 +1211,7 @@ namespace dbBackup {
}
}
TraceEvent("DBA_abort_old_backup").detail("tagName", tagNameKey.printable());
TraceEvent("DBA_AbortOldBackup").detail("TagName", tagNameKey.printable());
Void _ = wait(srcDrAgent.abortBackup(cx, tagNameKey, false, true));
return Void();
@ -1405,7 +1404,7 @@ namespace dbBackup {
state UID logUid = BinaryReader::fromStringRef<UID>(task->params[DatabaseBackupAgent::keyConfigLogUid], Unversioned());
TraceEvent("DBA_Complete").detail("restoreVersion", restoreVersion).detail("differential", stopWhenDone.present()).detail("LogUID", printable(task->params[BackupAgentBase::keyConfigLogUid]));
TraceEvent("DBA_Complete").detail("RestoreVersion", restoreVersion).detail("Differential", stopWhenDone.present()).detail("LogUID", printable(task->params[BackupAgentBase::keyConfigLogUid]));
// Start the complete task, if differential is not enabled
if (stopWhenDone.present()) {
@ -1634,7 +1633,7 @@ public:
state UID logUid = wait(backupAgent->getLogUid(cx, tagName));
state Key drVersionKey = backupAgent->config.get(BinaryWriter::toValue(logUid, Unversioned())).pack(DatabaseBackupAgent::keyDrVersion);
TraceEvent("DRU_watchLatestDrVersion").detail("drVersionKey", drVersionKey.printable()).detail("logUid", BinaryWriter::toValue(logUid, Unversioned()).printable());
TraceEvent("DRU_WatchLatestDrVersion").detail("DrVersionKey", drVersionKey.printable()).detail("LogUid", BinaryWriter::toValue(logUid, Unversioned()).printable());
loop {
state Reference<ReadYourWritesTransaction> tr(new ReadYourWritesTransaction(cx));
@ -1645,7 +1644,7 @@ public:
tr->setOption(FDBTransactionOptions::LOCK_AWARE);
Optional<Value> drVersion = wait(tr->get(drVersionKey));
TraceEvent("DRU_versionCheck").detail("current", drVersion.present() ? BinaryReader::fromStringRef<int>(drVersion.get(), Unversioned()) : -1).detail("expected", DatabaseBackupAgent::LATEST_DR_VERSION).detail("logUid", BinaryWriter::toValue(logUid, Unversioned()).printable());
TraceEvent("DRU_VersionCheck").detail("Current", drVersion.present() ? BinaryReader::fromStringRef<int>(drVersion.get(), Unversioned()) : -1).detail("Expected", DatabaseBackupAgent::LATEST_DR_VERSION).detail("LogUid", BinaryWriter::toValue(logUid, Unversioned()).printable());
if (drVersion.present() && BinaryReader::fromStringRef<int>(drVersion.get(), Unversioned()) == DatabaseBackupAgent::LATEST_DR_VERSION) {
return Void();
}
@ -1825,8 +1824,8 @@ public:
else
Void _ = wait(checkDatabaseLock(tr, logUid));
TraceEvent("DBA_Submit").detail("logUid", logUid).detail("lock", lockDB).detail("LogUID", printable(logUidValue)).detail("tag", printable(tagName))
.detail("key", printable(backupAgent->states.get(logUidValue).pack(DatabaseBackupAgent::keyFolderId))).detail("mapPrefix", printable(mapPrefix));
TraceEvent("DBA_Submit").detail("LogUid", logUid).detail("Lock", lockDB).detail("LogUID", printable(logUidValue)).detail("Tag", printable(tagName))
.detail("Key", printable(backupAgent->states.get(logUidValue).pack(DatabaseBackupAgent::keyFolderId))).detail("MapPrefix", printable(mapPrefix));
return Void();
}
@ -1834,7 +1833,7 @@ public:
ACTOR static Future<Void> unlockBackup(DatabaseBackupAgent* backupAgent, Reference<ReadYourWritesTransaction> tr, Key tagName) {
UID logUid = wait(backupAgent->getLogUid(tr, tagName));
Void _ = wait(unlockDatabase(tr, logUid));
TraceEvent("DBA_unlock").detail("tag", printable(tagName));
TraceEvent("DBA_Unlock").detail("Tag", printable(tagName));
return Void();
}
@ -1843,7 +1842,7 @@ public:
state UID destlogUid = wait(backupAgent->getLogUid(dest, tagName));
state int status = wait(backupAgent->getStateValue(dest, destlogUid));
TraceEvent("DBA_switchover_start").detail("status", status);
TraceEvent("DBA_SwitchoverStart").detail("Status", status);
if (status != BackupAgentBase::STATE_DIFFERENTIAL && status != BackupAgentBase::STATE_COMPLETED) {
throw backup_duplicate();
}
@ -1872,7 +1871,7 @@ public:
}
}
TraceEvent("DBA_switchover_locked").detail("version", commitVersion);
TraceEvent("DBA_SwitchoverLocked").detail("Version", commitVersion);
// Wait for the destination to apply mutations up to the lock commit before switching over.
state ReadYourWritesTransaction tr2(dest);
@ -1881,11 +1880,11 @@ public:
tr2.setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr2.setOption(FDBTransactionOptions::LOCK_AWARE);
state Optional<Value> backupUid = wait(tr2.get(backupAgent->states.get(BinaryWriter::toValue(destlogUid, Unversioned())).pack(DatabaseBackupAgent::keyFolderId)));
TraceEvent("DBA_switchover_backupUID").detail("uid", printable(backupUid)).detail("key", printable(backupAgent->states.get(BinaryWriter::toValue(destlogUid, Unversioned())).pack(DatabaseBackupAgent::keyFolderId)));
TraceEvent("DBA_SwitchoverBackupUID").detail("Uid", printable(backupUid)).detail("Key", printable(backupAgent->states.get(BinaryWriter::toValue(destlogUid, Unversioned())).pack(DatabaseBackupAgent::keyFolderId)));
if(!backupUid.present())
throw backup_duplicate();
Optional<Value> v = wait(tr2.get(BinaryWriter::toValue(destlogUid, Unversioned()).withPrefix(applyMutationsBeginRange.begin)));
TraceEvent("DBA_switchover_version").detail("version", v.present() ? BinaryReader::fromStringRef<Version>(v.get(), Unversioned()) : 0);
TraceEvent("DBA_SwitchoverVersion").detail("Version", v.present() ? BinaryReader::fromStringRef<Version>(v.get(), Unversioned()) : 0);
if( v.present() && BinaryReader::fromStringRef<Version>(v.get(), Unversioned()) >= commitVersion)
break;
@ -1898,7 +1897,7 @@ public:
}
}
TraceEvent("DBA_switchover_ready");
TraceEvent("DBA_SwitchoverReady");
try {
Void _ = wait( backupAgent->discontinueBackup(dest, tagName) );
@ -1909,7 +1908,7 @@ public:
int _ = wait( backupAgent->waitBackup(dest, tagName, true) );
TraceEvent("DBA_switchover_stopped");
TraceEvent("DBA_SwitchoverStopped");
state ReadYourWritesTransaction tr3(dest);
loop {
@ -1917,7 +1916,7 @@ public:
tr3.setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr3.setOption(FDBTransactionOptions::LOCK_AWARE);
Version destVersion = wait(tr3.getReadVersion());
TraceEvent("DBA_switchover_version_upgrade").detail("src", commitVersion).detail("dest", destVersion);
TraceEvent("DBA_SwitchoverVersionUpgrade").detail("Src", commitVersion).detail("Dest", destVersion);
if (destVersion <= commitVersion) {
TEST(true); // Forcing dest backup cluster to higher version
tr3.set(minRequiredCommitVersionKey, BinaryWriter::toValue(commitVersion+1, Unversioned()));
@ -1930,7 +1929,7 @@ public:
}
}
TraceEvent("DBA_switchover_version_upgraded");
TraceEvent("DBA_SwitchoverVersionUpgraded");
try {
Void _ = wait( drAgent.submitBackup(backupAgent->taskBucket->src, tagName, backupRanges, false, addPrefix, removePrefix, true, true) );
@ -1939,15 +1938,15 @@ public:
throw;
}
TraceEvent("DBA_switchover_submitted");
TraceEvent("DBA_SwitchoverSubmitted");
int _ = wait( drAgent.waitSubmitted(backupAgent->taskBucket->src, tagName) );
TraceEvent("DBA_switchover_started");
TraceEvent("DBA_SwitchoverStarted");
Void _ = wait( backupAgent->unlockBackup(dest, tagName) );
TraceEvent("DBA_switchover_unlocked");
TraceEvent("DBA_SwitchoverUnlocked");
return Void();
}
@ -1957,7 +1956,7 @@ public:
state UID logUid = wait(backupAgent->getLogUid(tr, tagName));
state int status = wait(backupAgent->getStateValue(tr, logUid));
TraceEvent("DBA_discontinue").detail("status", status);
TraceEvent("DBA_Discontinue").detail("Status", status);
if (!DatabaseBackupAgent::isRunnable((BackupAgentBase::enumState)status)) {
throw backup_unneeded();
}
@ -2018,7 +2017,7 @@ public:
tr->set(StringRef(backupAgent->states.get(logUidValue).pack(DatabaseBackupAgent::keyStateStatus)), StringRef(DatabaseBackupAgent::getStateText(BackupAgentBase::STATE_PARTIALLY_ABORTED)));
Void _ = wait(tr->commit());
TraceEvent("DBA_Abort").detail("commitVersion", tr->getCommittedVersion());
TraceEvent("DBA_Abort").detail("CommitVersion", tr->getCommittedVersion());
break;
}
catch (Error &e) {
@ -2044,7 +2043,7 @@ public:
if (lastApplied.present()) {
Version current = tr->getReadVersion().get();
Version applied = BinaryReader::fromStringRef<Version>(lastApplied.get(), Unversioned());
TraceEvent("DBA_abort_version_upgrade").detail("src", applied).detail("dest", current);
TraceEvent("DBA_AbortVersionUpgrade").detail("Src", applied).detail("Dest", current);
if (current <= applied) {
TEST(true); // Upgrading version of local database.
// The +1 is because we want to make sure that a versionstamped operation can't reuse

3
fdbclient/DatabaseContext.h
View File

@ -141,7 +141,8 @@ public:
int64_t transactionsTooOld;
int64_t transactionsFutureVersions;
int64_t transactionsNotCommitted;
int64_t transactionsMaybeCommitted;
int64_t transactionsMaybeCommitted;
int64_t transactionsResourceConstrained;
ContinuousSample<double> latencies, readLatencies, commitLatencies, GRVLatencies, mutationsPerCommit, bytesPerCommit;
int outstandingWatches;

75
fdbclient/FileBackupAgent.actor.cpp
View File

@ -705,7 +705,7 @@ namespace fileBackup {
if (taskVersion > version) {
state Error err = task_invalid_version();
TraceEvent(SevWarn, "BA_BackupRangeTaskFunc_execute").detail("taskVersion", taskVersion).detail("Name", printable(name)).detail("Version", version);
TraceEvent(SevWarn, "BA_BackupRangeTaskFunc_Execute").detail("TaskVersion", taskVersion).detail("Name", printable(name)).detail("Version", version);
if (KeyBackedConfig::TaskParams.uid().exists(task)) {
std::string msg = format("%s task version `%lu' is greater than supported version `%lu'", task->params[Task::reservedTaskParamKeyType].toString().c_str(), (unsigned long)taskVersion, (unsigned long)version);
Void _ = wait(BackupConfig(task).logError(cx, err, msg));
@ -724,7 +724,7 @@ namespace fileBackup {
state Subspace tagNames = backupAgent->subspace.get(BackupAgentBase::keyTagName);
Optional<Value> uidStr = wait(tr->get(tagNames.pack(Key(tagName))));
if (!uidStr.present()) {
TraceEvent(SevWarn, "FileBackupAbortIncompatibleBackup_TagNotFound").detail("tagName", tagName.c_str());
TraceEvent(SevWarn, "FileBackupAbortIncompatibleBackup_TagNotFound").detail("TagName", tagName.c_str());
return Void();
}
state UID uid = BinaryReader::fromStringRef<UID>(uidStr.get(), Unversioned());
@ -737,8 +737,8 @@ namespace fileBackup {
state EBackupState status = !statusStr.present() ? FileBackupAgent::STATE_NEVERRAN : BackupAgentBase::getState(statusStr.get().toString());
TraceEvent(SevInfo, "FileBackupAbortIncompatibleBackup")
.detail("tagName", tagName.c_str())
.detail("status", BackupAgentBase::getStateText(status));
.detail("TagName", tagName.c_str())
.detail("Status", BackupAgentBase::getStateText(status));
// Clear the folder id to prevent future tasks from executing at all
tr->clear(singleKeyRange(StringRef(globalConfig.pack(FileBackupAgent::keyFolderId))));
@ -770,8 +770,8 @@ namespace fileBackup {
TEST(true); // Canceling old backup task
TraceEvent(SevInfo, "FileBackupCancelOldTask")
.detail("task", printable(task->params[Task::reservedTaskParamKeyType]))
.detail("tagName", tagName);
.detail("Task", printable(task->params[Task::reservedTaskParamKeyType]))
.detail("TagName", tagName);
Void _ = wait(abortFiveZeroBackup(&backupAgent, tr, tagName));
Void _ = wait(taskBucket->finish(tr, task));
@ -779,7 +779,7 @@ namespace fileBackup {
}
virtual StringRef getName() const {
TraceEvent(SevError, "FileBackupError").detail("cause", "AbortFiveZeroBackupTaskFunc::name() should never be called");
TraceEvent(SevError, "FileBackupError").detail("Cause", "AbortFiveZeroBackupTaskFunc::name() should never be called");
ASSERT(false);
return StringRef();
}
@ -812,9 +812,9 @@ namespace fileBackup {
throw backup_unneeded();
}
TraceEvent(SevInfo, "FBA_abortFileOneBackup")
.detail("tagName", tagName.c_str())
.detail("status", BackupAgentBase::getStateText(status));
TraceEvent(SevInfo, "FBA_AbortFileOneBackup")
.detail("TagName", tagName.c_str())
.detail("Status", BackupAgentBase::getStateText(status));
// Cancel backup task through tag
Void _ = wait(tag.cancel(tr));
@ -840,8 +840,8 @@ namespace fileBackup {
TEST(true); // Canceling 5.1 backup task
TraceEvent(SevInfo, "FileBackupCancelFiveOneTask")
.detail("task", printable(task->params[Task::reservedTaskParamKeyType]))
.detail("tagName", tagName);
.detail("Task", printable(task->params[Task::reservedTaskParamKeyType]))
.detail("TagName", tagName);
Void _ = wait(abortFiveOneBackup(&backupAgent, tr, tagName));
Void _ = wait(taskBucket->finish(tr, task));
@ -849,7 +849,7 @@ namespace fileBackup {
}
virtual StringRef getName() const {
TraceEvent(SevError, "FileBackupError").detail("cause", "AbortFiveOneBackupTaskFunc::name() should never be called");
TraceEvent(SevError, "FileBackupError").detail("Cause", "AbortFiveOneBackupTaskFunc::name() should never be called");
ASSERT(false);
return StringRef();
}
@ -1911,30 +1911,25 @@ namespace fileBackup {
state Reference<FlowLock> lock(new FlowLock(CLIENT_KNOBS->BACKUP_LOCK_BYTES));
Void _ = wait(checkTaskVersion(cx, task, EraseLogRangeTaskFunc::name, EraseLogRangeTaskFunc::version));
state Version beginVersion = Params.beginVersion().get(task);
state Version endVersion = Params.endVersion().get(task);
state Key destUidValue = Params.destUidValue().get(task);
state BackupConfig config(task);
state Key logUidValue = config.getUidAsKey();
if (beginVersion == 0) {
Void _ = wait(eraseLogData(cx, logUidValue, destUidValue));
} else {
Void _ = wait(eraseLogData(cx, logUidValue, destUidValue, Optional<Version>(beginVersion), Optional<Version>(endVersion)));
}
Void _ = wait(eraseLogData(cx, logUidValue, destUidValue, endVersion != 0 ? Optional<Version>(endVersion) : Optional<Version>()));
return Void();
}
ACTOR static Future<Key> addTask(Reference<ReadYourWritesTransaction> tr, Reference<TaskBucket> taskBucket, UID logUid, TaskCompletionKey completionKey, Key destUidValue, Version beginVersion = 0, Version endVersion = 0, Reference<TaskFuture> waitFor = Reference<TaskFuture>()) {
ACTOR static Future<Key> addTask(Reference<ReadYourWritesTransaction> tr, Reference<TaskBucket> taskBucket, UID logUid, TaskCompletionKey completionKey, Key destUidValue, Version endVersion = 0, Reference<TaskFuture> waitFor = Reference<TaskFuture>()) {
Key key = wait(addBackupTask(EraseLogRangeTaskFunc::name,
EraseLogRangeTaskFunc::version,
tr, taskBucket, completionKey,
BackupConfig(logUid),
waitFor,
[=](Reference<Task> task) {
Params.beginVersion().set(task, beginVersion);
Params.beginVersion().set(task, 1); //FIXME: remove in 6.X, only needed for 5.2 backward compatibility
Params.endVersion().set(task, endVersion);
Params.destUidValue().set(task, destUidValue);
},
@ -2039,7 +2034,7 @@ namespace fileBackup {
// Do not erase at the first time
if (prevBeginVersion > 0) {
state Key destUidValue = wait(config.destUidValue().getOrThrow(tr));
Key _ = wait(EraseLogRangeTaskFunc::addTask(tr, taskBucket, config.getUid(), TaskCompletionKey::joinWith(logDispatchBatchFuture), destUidValue, prevBeginVersion, beginVersion));
Key _ = wait(EraseLogRangeTaskFunc::addTask(tr, taskBucket, config.getUid(), TaskCompletionKey::joinWith(logDispatchBatchFuture), destUidValue, beginVersion));
}
Void _ = wait(taskBucket->finish(tr, task));
@ -3406,10 +3401,10 @@ public:
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr->setOption(FDBTransactionOptions::LOCK_AWARE);
TraceEvent(SevInfo, "FBA_submitBackup")
.detail("tagName", tagName.c_str())
.detail("stopWhenDone", stopWhenDone)
.detail("outContainer", outContainer.toString());
TraceEvent(SevInfo, "FBA_SubmitBackup")
.detail("TagName", tagName.c_str())
.detail("StopWhenDone", stopWhenDone)
.detail("OutContainer", outContainer.toString());
state KeyBackedTag tag = makeBackupTag(tagName);
Optional<UidAndAbortedFlagT> uidAndAbortedFlag = wait(tag.get(tr));
@ -3481,8 +3476,8 @@ public:
tr->set(destUidLookupPath, destUidValue);
}
}
Version initVersion = 1;
tr->set(config.getUidAsKey().withPrefix(destUidValue).withPrefix(backupLatestVersionsPrefix), BinaryWriter::toValue<Version>(initVersion, Unversioned()));
tr->set(config.getUidAsKey().withPrefix(destUidValue).withPrefix(backupLatestVersionsPrefix), BinaryWriter::toValue<Version>(tr->getReadVersion().get(), Unversioned()));
config.destUidValue().set(tr, destUidValue);
// Point the tag to this new uid
@ -3632,10 +3627,10 @@ public:
// and clear the mutation logging config and data - but set its state as COMPLETED instead of ABORTED.
state Optional<Version> latestRestorableVersion = wait(config.getLatestRestorableVersion(tr));
TraceEvent(SevInfo, "FBA_discontinueBackup")
TraceEvent(SevInfo, "FBA_DiscontinueBackup")
.detail("AlreadyRestorable", latestRestorableVersion.present() ? "Yes" : "No")
.detail("tagName", tag.tagName.c_str())
.detail("status", BackupAgentBase::getStateText(status));
.detail("TagName", tag.tagName.c_str())
.detail("Status", BackupAgentBase::getStateText(status));
if(latestRestorableVersion.present()) {
// Cancel all backup tasks through tag
@ -3679,9 +3674,9 @@ public:
throw backup_unneeded();
}
TraceEvent(SevInfo, "FBA_abortBackup")
.detail("tagName", tagName.c_str())
.detail("status", BackupAgentBase::getStateText(status));
TraceEvent(SevInfo, "FBA_AbortBackup")
.detail("TagName", tagName.c_str())
.detail("Status", BackupAgentBase::getStateText(status));
// Cancel backup task through tag
Void _ = wait(tag.cancel(tr));
@ -3946,7 +3941,7 @@ public:
Void _ = wait( lockDatabase(&tr, randomUid) );
Void _ = wait(tr.commit());
commitVersion = tr.getCommittedVersion();
TraceEvent("AS_locked").detail("commitVer", commitVersion);
TraceEvent("AS_Locked").detail("CommitVer", commitVersion);
break;
} catch( Error &e ) {
Void _ = wait(tr.onError(e));
@ -3958,7 +3953,7 @@ public:
try {
Optional<Version> restoreVersion = wait( backupConfig.getLatestRestorableVersion(ryw_tr) );
if(restoreVersion.present() && restoreVersion.get() >= commitVersion) {
TraceEvent("AS_restoreVersion").detail("restoreVer", restoreVersion.get());
TraceEvent("AS_RestoreVersion").detail("RestoreVer", restoreVersion.get());
break;
} else {
ryw_tr->reset();
@ -3974,7 +3969,7 @@ public:
try {
Void _ = wait( discontinueBackup(backupAgent, ryw_tr, tagName) );
Void _ = wait( ryw_tr->commit() );
TraceEvent("AS_discontinuedBackup");
TraceEvent("AS_DiscontinuedBackup");
break;
} catch( Error &e ) {
if(e.code() == error_code_backup_unneeded || e.code() == error_code_backup_duplicate){
@ -3985,7 +3980,7 @@ public:
}
int _ = wait( waitBackup(backupAgent, cx, tagName.toString(), true) );
TraceEvent("AS_backupStopped");
TraceEvent("AS_BackupStopped");
ryw_tr->reset();
loop {
@ -3995,7 +3990,7 @@ public:
ryw_tr->addReadConflictRange(range);
ryw_tr->clear(range);
Void _ = wait( ryw_tr->commit() );
TraceEvent("AS_clearedRange");
TraceEvent("AS_ClearedRange");
break;
} catch( Error &e ) {
Void _ = wait( ryw_tr->onError(e) );
@ -4004,7 +3999,7 @@ public:
Reference<IBackupContainer> bc = wait(backupConfig.backupContainer().getOrThrow(cx));
TraceEvent("AS_startRestore");
TraceEvent("AS_StartRestore");
Version ver = wait( restore(backupAgent, cx, tagName, KeyRef(bc->getURL()), true, -1, true, range, addPrefix, removePrefix, true, randomUid) );
return ver;
}

8
fdbclient/KeyBackedTypes.h
View File

@ -123,10 +123,10 @@ public:
auto backtrace = platform::get_backtrace();
return map(get(tr, snapshot), [=](Optional<T> val) -> T {
if (!val.present()) {
TraceEvent(SevInfo, "KeyBackedProperty_keyNotFound")
.detail("key", printable(keyCopy))
.detail("err", err.code())
.detail("parentTrace", backtrace.c_str());
TraceEvent(SevInfo, "KeyBackedProperty_KeyNotFound")
.detail("Key", printable(keyCopy))
.detail("Err", err.code())
.detail("ParentTrace", backtrace.c_str());
throw err;
}

1
fdbclient/Knobs.cpp
View File

@ -48,6 +48,7 @@ ClientKnobs::ClientKnobs(bool randomize) {
init( DEFAULT_BACKOFF, .01 ); if( randomize && BUGGIFY ) DEFAULT_BACKOFF = g_random->random01();
init( DEFAULT_MAX_BACKOFF, 1.0 );
init( BACKOFF_GROWTH_RATE, 2.0 );
init( RESOURCE_CONSTRAINED_MAX_BACKOFF, 30.0 );
init( TRANSACTION_SIZE_LIMIT, 1e7 );
init( KEY_SIZE_LIMIT, 1e4 );

1
fdbclient/Knobs.h
View File

@ -46,6 +46,7 @@ public:
double DEFAULT_BACKOFF;
double DEFAULT_MAX_BACKOFF;
double BACKOFF_GROWTH_RATE;
double RESOURCE_CONSTRAINED_MAX_BACKOFF;
int64_t TRANSACTION_SIZE_LIMIT;
int64_t KEY_SIZE_LIMIT;

18
fdbclient/ManagementAPI.actor.cpp
View File

@ -192,13 +192,13 @@ ConfigurationResult::Type buildConfiguration( std::vector<StringRef> const& mode
std::string mode = it.toString();
auto m = configForToken( mode );
if( !m.size() ) {
TraceEvent(SevWarnAlways, "UnknownOption").detail("option", mode);
TraceEvent(SevWarnAlways, "UnknownOption").detail("Option", mode);
return ConfigurationResult::UNKNOWN_OPTION;
}
for( auto t = m.begin(); t != m.end(); ++t ) {
if( outConf.count( t->first ) ) {
TraceEvent(SevWarnAlways, "ConflictingOption").detail("option", printable(StringRef(t->first)));
TraceEvent(SevWarnAlways, "ConflictingOption").detail("Option", printable(StringRef(t->first)));
return ConfigurationResult::CONFLICTING_OPTIONS;
}
outConf[t->first] = t->second;
@ -1107,7 +1107,7 @@ ACTOR Future<int> setDDMode( Database cx, int mode ) {
Void _ = wait( tr.commit() );
return oldMode;
} catch (Error& e) {
TraceEvent("setDDModeRetrying").error(e);
TraceEvent("SetDDModeRetrying").error(e);
Void _ = wait (tr.onError(e));
}
}
@ -1249,7 +1249,7 @@ ACTOR Future<Void> lockDatabase( Transaction* tr, UID id ) {
if(BinaryReader::fromStringRef<UID>(val.get().substr(10), Unversioned()) == id) {
return Void();
} else {
//TraceEvent("DBA_lock_locked").detail("expecting", id).detail("lock", BinaryReader::fromStringRef<UID>(val.get().substr(10), Unversioned()));
//TraceEvent("DBA_LockLocked").detail("Expecting", id).detail("Lock", BinaryReader::fromStringRef<UID>(val.get().substr(10), Unversioned()));
throw database_locked();
}
}
@ -1268,7 +1268,7 @@ ACTOR Future<Void> lockDatabase( Reference<ReadYourWritesTransaction> tr, UID id
if(BinaryReader::fromStringRef<UID>(val.get().substr(10), Unversioned()) == id) {
return Void();
} else {
//TraceEvent("DBA_lock_locked").detail("expecting", id).detail("lock", BinaryReader::fromStringRef<UID>(val.get().substr(10), Unversioned()));
//TraceEvent("DBA_LockLocked").detail("Expecting", id).detail("Lock", BinaryReader::fromStringRef<UID>(val.get().substr(10), Unversioned()));
throw database_locked();
}
}
@ -1302,7 +1302,7 @@ ACTOR Future<Void> unlockDatabase( Transaction* tr, UID id ) {
return Void();
if(val.present() && BinaryReader::fromStringRef<UID>(val.get().substr(10), Unversioned()) != id) {
//TraceEvent("DBA_unlock_locked").detail("expecting", id).detail("lock", BinaryReader::fromStringRef<UID>(val.get().substr(10), Unversioned()));
//TraceEvent("DBA_UnlockLocked").detail("Expecting", id).detail("Lock", BinaryReader::fromStringRef<UID>(val.get().substr(10), Unversioned()));
throw database_locked();
}
@ -1319,7 +1319,7 @@ ACTOR Future<Void> unlockDatabase( Reference<ReadYourWritesTransaction> tr, UID
return Void();
if(val.present() && BinaryReader::fromStringRef<UID>(val.get().substr(10), Unversioned()) != id) {
//TraceEvent("DBA_unlock_locked").detail("expecting", id).detail("lock", BinaryReader::fromStringRef<UID>(val.get().substr(10), Unversioned()));
//TraceEvent("DBA_UnlockLocked").detail("Expecting", id).detail("Lock", BinaryReader::fromStringRef<UID>(val.get().substr(10), Unversioned()));
throw database_locked();
}
@ -1348,7 +1348,7 @@ ACTOR Future<Void> checkDatabaseLock( Transaction* tr, UID id ) {
Optional<Value> val = wait( tr->get(databaseLockedKey) );
if (val.present() && BinaryReader::fromStringRef<UID>(val.get().substr(10), Unversioned()) != id) {
//TraceEvent("DBA_check_locked").detail("expecting", id).detail("lock", BinaryReader::fromStringRef<UID>(val.get().substr(10), Unversioned())).backtrace();
//TraceEvent("DBA_CheckLocked").detail("Expecting", id).detail("Lock", BinaryReader::fromStringRef<UID>(val.get().substr(10), Unversioned())).backtrace();
throw database_locked();
}
@ -1361,7 +1361,7 @@ ACTOR Future<Void> checkDatabaseLock( Reference<ReadYourWritesTransaction> tr, U
Optional<Value> val = wait( tr->get(databaseLockedKey) );
if (val.present() && BinaryReader::fromStringRef<UID>(val.get().substr(10), Unversioned()) != id) {
//TraceEvent("DBA_check_locked").detail("expecting", id).detail("lock", BinaryReader::fromStringRef<UID>(val.get().substr(10), Unversioned())).backtrace();
//TraceEvent("DBA_CheckLocked").detail("Expecting", id).detail("Lock", BinaryReader::fromStringRef<UID>(val.get().substr(10), Unversioned())).backtrace();
throw database_locked();
}

2
fdbclient/MasterProxyInterface.h
View File

@ -55,7 +55,7 @@ struct MasterProxyInterface {
void initEndpoints() {
getConsistentReadVersion.getEndpoint(TaskProxyGetConsistentReadVersion);
getRawCommittedVersion.getEndpoint(TaskProxyGetRawCommittedVersion);
commit.getEndpoint(TaskProxyCommit);
commit.getEndpoint(TaskProxyCommitDispatcher);
getKeyServersLocations.getEndpoint(TaskProxyGetKeyServersLocations);
}
};

44
fdbclient/MetricLogger.actor.cpp
View File

@ -446,12 +446,12 @@ TEST_CASE("fdbserver/metrics/TraceEvents") {
stringMetric = Standalone<StringRef>(StringRef((uint8_t *)s, strlen(s)), arena);
TraceEvent("Dummy")
.detail("a", x)
.detail("x", 1.5 * x)
.detail("d", s)
.detail("j", sin(2.0 * x))
.detail("k", sin(3.0 * x))
.detail("s", sstart + (double)chunk * sin(10.0 * i / chunk));
.detail("A", x)
.detail("X", 1.5 * x)
.detail("D", s)
.detail("J", sin(2.0 * x))
.detail("K", sin(3.0 * x))
.detail("S", sstart + (double)chunk * sin(10.0 * i / chunk));
}
Void _ = wait(delay(w));
@ -460,14 +460,14 @@ TEST_CASE("fdbserver/metrics/TraceEvents") {
intMetric = x;
boolMetric = x % 2 > 0;
TraceEvent("Dummy")
.detail("a", x)
.detail("x", 1.5 * x)
.detail("b", x*2)
.detail("y", 3.0 * x)
.detail("d", d[x % 3])
.detail("j", sin(2.0 * x))
.detail("k", sin(3.0 * x))
.detail("s", sstart + (double)chunk * sin(40.0 * i / chunk));
.detail("A", x)
.detail("X", 1.5 * x)
.detail("B", x*2)
.detail("Y", 3.0 * x)
.detail("D", d[x % 3])
.detail("J", sin(2.0 * x))
.detail("K", sin(3.0 * x))
.detail("S", sstart + (double)chunk * sin(40.0 * i / chunk));
}
Void _ = wait(delay(w));
@ -476,14 +476,14 @@ TEST_CASE("fdbserver/metrics/TraceEvents") {
intMetric = x;
boolMetric = x % 2 > 0;
TraceEvent("Dummy")
.detail("a", x)
.detail("x", 1.5 * x)
.detail("c", x*3)
.detail("z", 4.5 * x)
.detail("d", d[x % 3])
.detail("j", sin(2.0 * x))
.detail("k", sin(3.0 * x))
.detail("s", sstart + (double)chunk * sin(160.0 * i / chunk));
.detail("A", x)
.detail("X", 1.5 * x)
.detail("C", x*3)
.detail("Z", 4.5 * x)
.detail("D", d[x % 3])
.detail("J", sin(2.0 * x))
.detail("K", sin(3.0 * x))
.detail("S", sstart + (double)chunk * sin(160.0 * i / chunk));
}
Void _ = wait(delay(w));

82
fdbclient/MonitorLeader.actor.cpp
View File

@ -325,45 +325,53 @@ ACTOR Future<Void> monitorNominee( Key key, ClientLeaderRegInterface coord, Asyn
}
// Also used in fdbserver/LeaderElection.actor.cpp!
Optional<LeaderInfo> getLeader( vector<Optional<LeaderInfo>> nominees ) {
// bool represents if the LeaderInfo is a majority answer or not.
// This function also masks the first 7 bits of changeId of the nominees and returns the Leader with masked changeId
Optional<std::pair<LeaderInfo, bool>> getLeader( const vector<Optional<LeaderInfo>>& nominees ) {
vector<LeaderInfo> maskedNominees;
maskedNominees.reserve(nominees.size());
for (auto &nominee : nominees) {
if (nominee.present()) {
maskedNominees.push_back(nominee.get());
maskedNominees.back().changeID = UID(maskedNominees.back().changeID.first() & LeaderInfo::mask, maskedNominees.back().changeID.second());
}
}
// If any coordinator says that the quorum is forwarded, then it is
for(int i=0; i<nominees.size(); i++)
if (nominees[i].present() && nominees[i].get().forward)
return nominees[i].get();
for(int i=0; i<maskedNominees.size(); i++)
if (maskedNominees[i].forward)
return std::pair<LeaderInfo, bool>(maskedNominees[i], true);
if(!maskedNominees.size())
return Optional<std::pair<LeaderInfo, bool>>();
std::sort(maskedNominees.begin(), maskedNominees.end(),
[](const LeaderInfo& l, const LeaderInfo& r) { return l.changeID < r.changeID; });
if(!nominees.size())
return Optional<LeaderInfo>();
// There is a leader if a majority of the nominees are the same.
// If there is a majority, the median item is in it.
int bestCount = 0;
Optional<LeaderInfo> currentNominee;
for(int i=0; i<nominees.size(); i++) {
if( (nominees[i].present() != currentNominee.present()) || (currentNominee.present() && !currentNominee.get().equalInternalId(nominees[i].get()) ) ) {
if(bestCount > 0) {
bestCount--;
} else {
bestCount = 1;
currentNominee = nominees[i];
LeaderInfo bestNominee;
LeaderInfo currentNominee;
int curCount = 0;
for (int i = 0; i < maskedNominees.size(); i++) {
if (currentNominee == maskedNominees[i]) {
curCount++;
}
else {
if (curCount > bestCount) {
bestNominee = currentNominee;
bestCount = curCount;
}
} else {
bestCount++;
currentNominee = maskedNominees[i];
curCount = 1;
}
}
if(!currentNominee.present())
return Optional<LeaderInfo>();
int amountBest = 0;
for(int i=0; i<nominees.size(); i++) {
if( nominees[i].present() && currentNominee.get().equalInternalId(nominees[i].get()) ) {
amountBest++;
}
if (curCount > bestCount) {
bestNominee = currentNominee;
bestCount = curCount;
}
if(amountBest >= nominees.size()/2 + 1) {
return currentNominee;
}
return Optional<LeaderInfo>();
bool majority = bestCount >= nominees.size() / 2 + 1;
return std::pair<LeaderInfo, bool>(bestNominee, majority);
}
struct MonitorLeaderInfo {
@ -389,12 +397,12 @@ ACTOR Future<MonitorLeaderInfo> monitorLeaderOneGeneration( Reference<ClusterCon
allActors = waitForAll(actors);
loop {
Optional<LeaderInfo> leader = getLeader(nominees);
TraceEvent("MonitorLeaderChange").detail("NewLeader", leader.present() ? leader.get().changeID : UID(1,1));
Optional<std::pair<LeaderInfo, bool>> leader = getLeader(nominees);
TraceEvent("MonitorLeaderChange").detail("NewLeader", leader.present() ? leader.get().first.changeID : UID(1,1));
if (leader.present()) {
if( leader.get().forward ) {
TraceEvent("MonitorLeaderForwarding").detail("NewConnStr", leader.get().serializedInfo.toString()).detail("OldConnStr", info.intermediateConnFile->getConnectionString().toString());
info.intermediateConnFile = Reference<ClusterConnectionFile>(new ClusterConnectionFile(connFile->getFilename(), ClusterConnectionString(leader.get().serializedInfo.toString())));
if( leader.get().first.forward ) {
TraceEvent("MonitorLeaderForwarding").detail("NewConnStr", leader.get().first.serializedInfo.toString()).detail("OldConnStr", info.intermediateConnFile->getConnectionString().toString());
info.intermediateConnFile = Reference<ClusterConnectionFile>(new ClusterConnectionFile(connFile->getFilename(), ClusterConnectionString(leader.get().first.serializedInfo.toString())));
return info;
}
if(connFile != info.intermediateConnFile) {
@ -410,7 +418,7 @@ ACTOR Future<MonitorLeaderInfo> monitorLeaderOneGeneration( Reference<ClusterCon
info.hasConnected = true;
connFile->notifyConnected();
outSerializedLeaderInfo->set( leader.get().serializedInfo );
outSerializedLeaderInfo->set( leader.get().first.serializedInfo );
}
Void _ = wait( nomineeChange.onTrigger() || allActors );
}

2
fdbclient/MultiVersionTransaction.actor.cpp
View File

@ -1240,7 +1240,7 @@ ThreadFuture<Reference<ICluster>> MultiVersionApi::createCluster(const char *clu
}
else {
for( auto it : externalClients ) {
TraceEvent("CreatingClusterOnExternalClient").detail("LibraryPath", it.second->libPath).detail("failed", it.second->failed);
TraceEvent("CreatingClusterOnExternalClient").detail("LibraryPath", it.second->libPath).detail("Failed", it.second->failed);
}
return mapThreadFuture<Reference<ICluster>, Reference<ICluster>>(clusterFuture, [this, clusterFile](ErrorOr<Reference<ICluster>> cluster) {
if(cluster.isError()) {

91
fdbclient/NativeAPI.actor.cpp
View File

@ -221,20 +221,21 @@ ACTOR Future<Void> databaseLogger( DatabaseContext *cx ) {
.detail("FutureVersions", cx->transactionsFutureVersions)
.detail("NotCommitted", cx->transactionsNotCommitted)
.detail("MaybeCommitted", cx->transactionsMaybeCommitted)
.detail("MeanLatency", 1000 * cx->latencies.mean())
.detail("MedianLatency", 1000 * cx->latencies.median())
.detail("Latency90", 1000 * cx->latencies.percentile(0.90))
.detail("Latency98", 1000 * cx->latencies.percentile(0.98))
.detail("MaxLatency", 1000 * cx->latencies.max())
.detail("MeanRowReadLatency", 1000 * cx->readLatencies.mean())
.detail("MedianRowReadLatency", 1000 * cx->readLatencies.median())
.detail("MaxRowReadLatency", 1000 * cx->readLatencies.max())
.detail("MeanGRVLatency", 1000 * cx->GRVLatencies.mean())
.detail("MedianGRVLatency", 1000 * cx->GRVLatencies.median())
.detail("MaxGRVLatency", 1000 * cx->GRVLatencies.max())
.detail("MeanCommitLatency", 1000 * cx->commitLatencies.mean())
.detail("MedianCommitLatency", 1000 * cx->commitLatencies.median())
.detail("MaxCommitLatency", 1000 * cx->commitLatencies.max())
.detail("ResourceConstrained", cx->transactionsResourceConstrained)
.detail("MeanLatency", cx->latencies.mean())
.detail("MedianLatency", cx->latencies.median())
.detail("Latency90", cx->latencies.percentile(0.90))
.detail("Latency98", cx->latencies.percentile(0.98))
.detail("MaxLatency", cx->latencies.max())
.detail("MeanRowReadLatency", cx->readLatencies.mean())
.detail("MedianRowReadLatency", cx->readLatencies.median())
.detail("MaxRowReadLatency", cx->readLatencies.max())
.detail("MeanGRVLatency", cx->GRVLatencies.mean())
.detail("MedianGRVLatency", cx->GRVLatencies.median())
.detail("MaxGRVLatency", cx->GRVLatencies.max())
.detail("MeanCommitLatency", cx->commitLatencies.mean())
.detail("MedianCommitLatency", cx->commitLatencies.median())
.detail("MaxCommitLatency", cx->commitLatencies.max())
.detail("MeanMutationsPerCommit", cx->mutationsPerCommit.mean())
.detail("MedianMutationsPerCommit", cx->mutationsPerCommit.median())
.detail("MaxMutationsPerCommit", cx->mutationsPerCommit.max())
@ -470,7 +471,7 @@ DatabaseContext::DatabaseContext(
int taskID, LocalityData clientLocality, bool enableLocalityLoadBalance, bool lockAware )
: clientInfo(clientInfo), masterProxiesChangeTrigger(), cluster(cluster), clientInfoMonitor(clientInfoMonitor), dbName(dbName), dbId(dbId),
transactionReadVersions(0), transactionLogicalReads(0), transactionPhysicalReads(0), transactionCommittedMutations(0), transactionCommittedMutationBytes(0), transactionsCommitStarted(0),
transactionsCommitCompleted(0), transactionsTooOld(0), transactionsFutureVersions(0), transactionsNotCommitted(0), transactionsMaybeCommitted(0), taskID(taskID),
transactionsCommitCompleted(0), transactionsTooOld(0), transactionsFutureVersions(0), transactionsNotCommitted(0), transactionsMaybeCommitted(0), transactionsResourceConstrained(0), taskID(taskID),
outstandingWatches(0), maxOutstandingWatches(CLIENT_KNOBS->DEFAULT_MAX_OUTSTANDING_WATCHES), clientLocality(clientLocality), enableLocalityLoadBalance(enableLocalityLoadBalance), lockAware(lockAware),
latencies(1000), readLatencies(1000), commitLatencies(1000), GRVLatencies(1000), mutationsPerCommit(1000), bytesPerCommit(1000)
{
@ -969,7 +970,7 @@ AddressExclusion AddressExclusion::parse( StringRef const& key ) {
std::string s = key.toString();
int a,b,c,d,port,count=-1;
if (sscanf(s.c_str(), "%d.%d.%d.%d%n", &a,&b,&c,&d, &count)<4) {
TraceEvent(SevWarnAlways, "AddressExclusionParseError").detail("s", printable(key));
TraceEvent(SevWarnAlways, "AddressExclusionParseError").detail("String", printable(key));
return AddressExclusion();
}
s = s.substr(count);
@ -977,7 +978,7 @@ AddressExclusion AddressExclusion::parse( StringRef const& key ) {
if (!s.size())
return AddressExclusion( ip );
if (sscanf( s.c_str(), ":%d%n", &port, &count ) < 1 || count != s.size()) {
TraceEvent(SevWarnAlways, "AddressExclusionParseError").detail("s", printable(key));
TraceEvent(SevWarnAlways, "AddressExclusionParseError").detail("String", printable(key));
return AddressExclusion();
}
return AddressExclusion( ip, port );
@ -1249,11 +1250,11 @@ ACTOR Future<Key> getKey( Database cx, KeySelector k, Future<Version> version, T
try {
if( info.debugID.present() )
g_traceBatch.addEvent("TransactionDebug", info.debugID.get().first(), "NativeAPI.getKey.Before"); //.detail("StartKey", printable(k.getKey())).detail("offset",k.offset).detail("orEqual",k.orEqual);
g_traceBatch.addEvent("TransactionDebug", info.debugID.get().first(), "NativeAPI.getKey.Before"); //.detail("StartKey", printable(k.getKey())).detail("Offset",k.offset).detail("OrEqual",k.orEqual);
++cx->transactionPhysicalReads;
GetKeyReply reply = wait( loadBalance( ssi.second, &StorageServerInterface::getKey, GetKeyRequest(k, version.get()), TaskDefaultPromiseEndpoint, false, cx->enableLocalityLoadBalance ? &cx->queueModel : NULL ) );
if( info.debugID.present() )
g_traceBatch.addEvent("TransactionDebug", info.debugID.get().first(), "NativeAPI.getKey.After"); //.detail("NextKey",printable(reply.sel.key)).detail("offset", reply.sel.offset).detail("orEqual", k.orEqual);
g_traceBatch.addEvent("TransactionDebug", info.debugID.get().first(), "NativeAPI.getKey.After"); //.detail("NextKey",printable(reply.sel.key)).detail("Offset", reply.sel.offset).detail("OrEqual", k.orEqual);
k = reply.sel;
if (!k.offset && k.orEqual) {
return k.getKey();
@ -1265,7 +1266,7 @@ ACTOR Future<Key> getKey( Database cx, KeySelector k, Future<Version> version, T
Void _ = wait(delay(CLIENT_KNOBS->WRONG_SHARD_SERVER_DELAY, info.taskID));
} else {
if(e.code() != error_code_actor_cancelled) {
TraceEvent(SevInfo, "getKeyError")
TraceEvent(SevInfo, "GetKeyError")
.error(e)
.detail("AtKey", printable(k.getKey()))
.detail("Offset", k.offset);
@ -1290,7 +1291,7 @@ ACTOR Future<Version> waitForCommittedVersion( Database cx, Version version ) {
}
}
} catch (Error& e) {
TraceEvent(SevError, "waitForCommittedVersionError").error(e);
TraceEvent(SevError, "WaitForCommittedVersionError").error(e);
throw;
}
}
@ -1323,7 +1324,7 @@ ACTOR Future< Void > watchValue( Future<Version> version, Key key, Optional<Valu
//cannot do this until the storage server is notified on knownCommittedVersion changes from tlog (faster than the current update loop)
Version v = wait( waitForCommittedVersion( cx, resp ) );
//TraceEvent("watcherCommitted").detail("committedVersion", v).detail("watchVersion", resp).detail("key", printable( key )).detail("value", printable(value));
//TraceEvent("WatcherCommitted").detail("CommittedVersion", v).detail("WatchVersion", resp).detail("Key", printable( key )).detail("Value", printable(value));
if( v - resp < 50000000 ) // False if there is a master failure between getting the response and getting the committed version, Dependent on SERVER_KNOBS->MAX_VERSIONS_IN_FLIGHT
return Void();
@ -1484,7 +1485,7 @@ ACTOR Future<Standalone<RangeResultRef>> getExactRange( Database cx, Version ver
Void _ = wait( delay(CLIENT_KNOBS->WRONG_SHARD_SERVER_DELAY, info.taskID ));
break;
} else {
TraceEvent(SevInfo, "getExactRangeError")
TraceEvent(SevInfo, "GetExactRangeError")
.error(e)
.detail("ShardBegin", printable(locations[shard].first.begin))
.detail("ShardEnd", printable(locations[shard].first.end));
@ -1663,11 +1664,11 @@ ACTOR Future<Standalone<RangeResultRef>> getRange( Database cx, Reference<Transa
/*TraceEvent("TransactionDebugGetRangeInfo", info.debugID.get())
.detail("ReqBeginKey", printable(req.begin.getKey()))
.detail("ReqEndKey", printable(req.end.getKey()))
.detail("originalBegin", originalBegin.toString())
.detail("originalEnd", originalEnd.toString())
.detail("OriginalBegin", originalBegin.toString())
.detail("OriginalEnd", originalEnd.toString())
.detail("Begin", begin.toString())
.detail("End", end.toString())
.detail("shard", printable(shard))
.detail("Shard", printable(shard))
.detail("ReqLimit", req.limit)
.detail("ReqLimitBytes", req.limitBytes)
.detail("ReqVersion", req.version)
@ -2165,9 +2166,10 @@ void Transaction::addWriteConflictRange( const KeyRangeRef& keys ) {
t.write_conflict_ranges.push_back_deep( req.arena, r );
}
double Transaction::getBackoff() {
double Transaction::getBackoff(int errCode) {
double b = backoff * g_random->random01();
backoff = std::min(backoff * CLIENT_KNOBS->BACKOFF_GROWTH_RATE, options.maxBackoff);
backoff = errCode == error_code_proxy_memory_limit_exceeded ? std::min(backoff * CLIENT_KNOBS->BACKOFF_GROWTH_RATE, CLIENT_KNOBS->RESOURCE_CONSTRAINED_MAX_BACKOFF) :
std::min(backoff * CLIENT_KNOBS->BACKOFF_GROWTH_RATE, options.maxBackoff);
return b;
}
@ -2446,8 +2448,8 @@ ACTOR static Future<Void> tryCommit( Database cx, Reference<TransactionLogInfo>
// The user needs to be informed that we aren't sure whether the commit happened. Standard retry loops retry it anyway (relying on transaction idempotence) but a client might do something else.
throw commit_unknown_result();
} else {
if (e.code() != error_code_transaction_too_old && e.code() != error_code_not_committed && e.code() != error_code_database_locked)
TraceEvent(SevError, "tryCommitError").error(e);
if (e.code() != error_code_transaction_too_old && e.code() != error_code_not_committed && e.code() != error_code_database_locked && e.code() != error_code_proxy_memory_limit_exceeded)
TraceEvent(SevError, "TryCommitError").error(e);
if (trLogInfo)
trLogInfo->addLog(FdbClientLogEvents::EventCommitError(startTime, static_cast<int>(e.code()), req));
throw;
@ -2456,11 +2458,6 @@ ACTOR static Future<Void> tryCommit( Database cx, Reference<TransactionLogInfo>
}
Future<Void> Transaction::commitMutations() {
cx->transactionsCommitStarted++;
if(options.readOnly)
return transaction_read_only();
try {
//if this is a read-only transaction return immediately
if( !tr.transaction.write_conflict_ranges.size() && !tr.transaction.mutations.size() ) {
@ -2471,6 +2468,11 @@ Future<Void> Transaction::commitMutations() {
return Void();
}
cx->transactionsCommitStarted++;
if(options.readOnly)
return transaction_read_only();
cx->mutationsPerCommit.addSample(tr.transaction.mutations.size());
cx->bytesPerCommit.addSample(tr.transaction.mutations.expectedSize());
@ -2479,8 +2481,8 @@ Future<Void> Transaction::commitMutations() {
TraceEvent(!g_network->isSimulated() ? SevWarnAlways : SevWarn, "LargeTransaction")
.detail("Size", transactionSize)
.detail("NumMutations", tr.transaction.mutations.size())
.detail("readConflictSize", tr.transaction.read_conflict_ranges.expectedSize())
.detail("writeConflictSize", tr.transaction.write_conflict_ranges.expectedSize())
.detail("ReadConflictSize", tr.transaction.read_conflict_ranges.expectedSize())
.detail("WriteConflictSize", tr.transaction.write_conflict_ranges.expectedSize())
.suppressFor(1.0);
}
@ -2796,14 +2798,17 @@ Future<Void> Transaction::onError( Error const& e ) {
}
if (e.code() == error_code_not_committed ||
e.code() == error_code_commit_unknown_result ||
e.code() == error_code_database_locked)
e.code() == error_code_database_locked ||
e.code() == error_code_proxy_memory_limit_exceeded)
{
if(e.code() == error_code_not_committed)
cx->transactionsNotCommitted++;
if(e.code() == error_code_commit_unknown_result)
cx->transactionsMaybeCommitted++;
if (e.code() == error_code_proxy_memory_limit_exceeded)
cx->transactionsResourceConstrained++;
double backoff = getBackoff();
double backoff = getBackoff(e.code());
reset();
return delay( backoff, info.taskID );
}
@ -2922,7 +2927,7 @@ ACTOR Future< StorageMetrics > waitStorageMetrics(
}
} catch (Error& e) {
if (e.code() != error_code_wrong_shard_server && e.code() != error_code_all_alternatives_failed) {
TraceEvent(SevError, "waitStorageMetricsError").error(e);
TraceEvent(SevError, "WaitStorageMetricsError").error(e);
throw;
}
cx->invalidateCache(keys);
@ -2963,7 +2968,7 @@ ACTOR Future< Standalone<VectorRef<KeyRef>> > splitStorageMetrics( Database cx,
else {
results.push_back_deep( results.arena(), keys.begin );
try {
//TraceEvent("SplitStorageMetrics").detail("locations", locations.size());
//TraceEvent("SplitStorageMetrics").detail("Locations", locations.size());
state int i = 0;
for(; i<locations.size(); i++) {
@ -2979,7 +2984,7 @@ ACTOR Future< Standalone<VectorRef<KeyRef>> > splitStorageMetrics( Database cx,
}
used = res.used;
//TraceEvent("SplitStorageMetricsResult").detail("used", used.bytes).detail("location", i).detail("size", res.splits.size());
//TraceEvent("SplitStorageMetricsResult").detail("Used", used.bytes).detail("Location", i).detail("Size", res.splits.size());
}
if( used.allLessOrEqual( limit * CLIENT_KNOBS->STORAGE_METRICS_UNFAIR_SPLIT_LIMIT ) ) {
@ -2990,7 +2995,7 @@ ACTOR Future< Standalone<VectorRef<KeyRef>> > splitStorageMetrics( Database cx,
return results;
} catch (Error& e) {
if (e.code() != error_code_wrong_shard_server && e.code() != error_code_all_alternatives_failed) {
TraceEvent(SevError, "splitStorageMetricsError").error(e);
TraceEvent(SevError, "SplitStorageMetricsError").error(e);
throw;
}
cx->invalidateCache( keys );

2
fdbclient/NativeAPI.h
View File

@ -284,7 +284,7 @@ public:
void reset();
void fullReset();
double getBackoff();
double getBackoff(int errCode);
void debugTransaction(UID dID) { info.debugID = dID; }
Future<Void> commitMutations();

48
fdbclient/RYWIterator.cpp
View File

@ -558,7 +558,7 @@ TEST_CASE("fdbclient/WriteMap/random") {
KeyRangeRef range = RandomTestImpl::getRandomRange(arena);
writes.addConflictRange(range);
conflictMap.insert(range, true);
TraceEvent("RWMT_addConflictRange").detail("range", printable(range));
TraceEvent("RWMT_AddConflictRange").detail("Range", printable(range));
}
else if(r == 1) {
KeyRangeRef range = RandomTestImpl::getRandomRange(arena);
@ -567,7 +567,7 @@ TEST_CASE("fdbclient/WriteMap/random") {
conflictMap.insert(range, false);
clearMap.insert(range, false);
unreadableMap.insert(range, true);
TraceEvent("RWMT_addUnmodifiedAndUnreadableRange").detail("range", printable(range));
TraceEvent("RWMT_AddUnmodifiedAndUnreadableRange").detail("Range", printable(range));
}
else if (r == 2) {
bool addConflict = g_random->random01() < 0.5;
@ -578,7 +578,7 @@ TEST_CASE("fdbclient/WriteMap/random") {
conflictMap.insert(range, true);
clearMap.insert(range, true);
unreadableMap.insert(range, false);
TraceEvent("RWMT_clear").detail("range", printable(range)).detail("addConflict", addConflict);
TraceEvent("RWMT_Clear").detail("Range", printable(range)).detail("AddConflict", addConflict);
}
else if (r == 3) {
bool addConflict = g_random->random01() < 0.5;
@ -590,7 +590,7 @@ TEST_CASE("fdbclient/WriteMap/random") {
conflictMap.insert(key, true);
clearMap.insert(key, false);
unreadableMap.insert(key, true);
TraceEvent("RWMT_setVersionstampedValue").detail("key", printable(key)).detail("value", value.size()).detail("addConflict", addConflict);
TraceEvent("RWMT_SetVersionstampedValue").detail("Key", printable(key)).detail("Value", value.size()).detail("AddConflict", addConflict);
}
else if (r == 4) {
bool addConflict = g_random->random01() < 0.5;
@ -602,7 +602,7 @@ TEST_CASE("fdbclient/WriteMap/random") {
conflictMap.insert(key, true);
clearMap.insert(key, false);
unreadableMap.insert(key, true);
TraceEvent("RWMT_setVersionstampedKey").detail("key", printable(key)).detail("value", value.size()).detail("addConflict", addConflict);
TraceEvent("RWMT_SetVersionstampedKey").detail("Key", printable(key)).detail("Value", value.size()).detail("AddConflict", addConflict);
}
else if (r == 5) {
bool addConflict = g_random->random01() < 0.5;
@ -622,7 +622,7 @@ TEST_CASE("fdbclient/WriteMap/random") {
if (addConflict)
conflictMap.insert(key, true);
clearMap.insert(key, false);
TraceEvent("RWMT_and").detail("key", printable(key)).detail("value", value.size()).detail("addConflict", addConflict);
TraceEvent("RWMT_And").detail("Key", printable(key)).detail("Value", value.size()).detail("AddConflict", addConflict);
}
else {
bool addConflict = g_random->random01() < 0.5;
@ -636,7 +636,7 @@ TEST_CASE("fdbclient/WriteMap/random") {
if (addConflict)
conflictMap.insert(key, true);
clearMap.insert(key, false);
TraceEvent("RWMT_set").detail("key", printable(key)).detail("value", value.size()).detail("addConflict", addConflict);
TraceEvent("RWMT_Set").detail("Key", printable(key)).detail("Value", value.size()).detail("AddConflict", addConflict);
}
}
@ -648,23 +648,23 @@ TEST_CASE("fdbclient/WriteMap/random") {
for (; it.beginKey() < allKeys.end; ++it) {
if (it.is_operation()) {
ASSERT(setIter != setEnd);
TraceEvent("RWMT_checkOperation")
.detail("wm_key", printable(it.beginKey().toStandaloneStringRef()))
.detail("wm_size", it.op().size())
.detail("wm_value", it.op().top().value.present() ? std::to_string(it.op().top().value.get().size()) : "Not Found")
.detail("wm_type", (int)it.op().top().type)
.detail("sm_key", printable(setIter->first))
.detail("sm_size", setIter->second.size())
.detail("sm_value", setIter->second.top().value.present() ? std::to_string(setIter->second.top().value.get().size()) : "Not Found")
.detail("sm_type", (int)setIter->second.top().type);
TraceEvent("RWMT_CheckOperation")
.detail("WmKey", printable(it.beginKey().toStandaloneStringRef()))
.detail("WmSize", it.op().size())
.detail("WmValue", it.op().top().value.present() ? std::to_string(it.op().top().value.get().size()) : "Not Found")
.detail("WmType", (int)it.op().top().type)
.detail("SmKey", printable(setIter->first))
.detail("SmSize", setIter->second.size())
.detail("SmValue", setIter->second.top().value.present() ? std::to_string(setIter->second.top().value.get().size()) : "Not Found")
.detail("SmType", (int)setIter->second.top().type);
ASSERT(it.beginKey() == setIter->first && it.op() == setIter->second);
++setIter;
}
}
TraceEvent("RWMT_checkOperationFinal")
.detail("wm_key", printable(it.beginKey().toStandaloneStringRef()))
.detail("sm_iter", setIter == setEnd);
TraceEvent("RWMT_CheckOperationFinal")
.detail("WmKey", printable(it.beginKey().toStandaloneStringRef()))
.detail("SmIter", setIter == setEnd);
ASSERT(it.beginKey() >= allKeys.end && setIter == setEnd);
@ -711,11 +711,11 @@ TEST_CASE("fdbclient/WriteMap/random") {
auto unreadableEnd = unreadableRanges.end();
while (it.beginKey() < allKeys.end && unreadableIter != unreadableEnd) {
TraceEvent("RWMT_checkUnreadable")
.detail("writeMapRange", printable(KeyRangeRef(it.beginKey().toStandaloneStringRef(), it.endKey().toStandaloneStringRef())))
.detail("unreadableMapRange", printable(unreadableIter.range()))
.detail("writeMapValue", it.is_unreadable())
.detail("unreadableMapValue", unreadableIter.value());
TraceEvent("RWMT_CheckUnreadable")
.detail("WriteMapRange", printable(KeyRangeRef(it.beginKey().toStandaloneStringRef(), it.endKey().toStandaloneStringRef())))
.detail("UnreadableMapRange", printable(unreadableIter.range()))
.detail("WriteMapValue", it.is_unreadable())
.detail("UnreadableMapValue", unreadableIter.value());
ASSERT(unreadableIter.value() == it.is_unreadable());
if (unreadableIter.range().end < it.endKey()) {
++unreadableIter;

52
fdbclient/ReadYourWrites.actor.cpp
View File

@ -490,15 +490,15 @@ public:
//TraceEvent("RYWSelectorsStartForward", randomID).detail("ByteLimit", limits.bytes).detail("RowLimit", limits.rows);
loop {
/*TraceEvent("RYWSelectors", randomID).detail("begin", begin.toString())
.detail("end", end.toString())
.detail("reached", limits.isReached())
.detail("itemsPastEnd", itemsPastEnd)
.detail("endOffset", -end.offset)
.detail("itBegin", printable(it.beginKey().toStandaloneStringRef()))
.detail("itEnd", printable(itEnd.beginKey().toStandaloneStringRef()))
.detail("unknown", it.is_unknown_range())
.detail("requests", requestCount);*/
/*TraceEvent("RYWSelectors", randomID).detail("Begin", begin.toString())
.detail("End", end.toString())
.detail("Reached", limits.isReached())
.detail("ItemsPastEnd", itemsPastEnd)
.detail("EndOffset", -end.offset)
.detail("ItBegin", printable(it.beginKey().toStandaloneStringRef()))
.detail("ItEnd", printable(itEnd.beginKey().toStandaloneStringRef()))
.detail("Unknown", it.is_unknown_range())
.detail("Requests", requestCount);*/
if( !result.size() && actualBeginOffset >= actualEndOffset && begin.getKey() >= end.getKey() ) {
return RangeResultRef(false, false);
@ -589,13 +589,13 @@ public:
ASSERT( !requestLimit.hasRowLimit() || requestLimit.rows > 0 );
ASSERT( requestLimit.hasRowLimit() || requestLimit.hasByteLimit() );
//TraceEvent("RYWIssuing", randomID).detail("begin", read_begin.toString()).detail("end", read_end.toString()).detail("bytes", requestLimit.bytes).detail("rows", requestLimit.rows).detail("limits", limits.bytes).detail("reached", limits.isReached()).detail("requestCount", requestCount).detail("singleClears", singleClears).detail("ucEnd", printable(ucEnd.beginKey().toStandaloneStringRef())).detail("minRows", requestLimit.minRows);
//TraceEvent("RYWIssuing", randomID).detail("Begin", read_begin.toString()).detail("End", read_end.toString()).detail("Bytes", requestLimit.bytes).detail("Rows", requestLimit.rows).detail("Limits", limits.bytes).detail("Reached", limits.isReached()).detail("RequestCount", requestCount).detail("SingleClears", singleClears).detail("UcEnd", printable(ucEnd.beginKey().toStandaloneStringRef())).detail("MinRows", requestLimit.minRows);
additionalRows = 0;
Standalone<RangeResultRef> snapshot_read = wait( ryw->tr.getRange( read_begin, read_end, requestLimit, true, false ) );
KeyRangeRef range = getKnownKeyRange( snapshot_read, read_begin, read_end, ryw->arena );
//TraceEvent("RYWCacheInsert", randomID).detail("Range", printable(range)).detail("expectedSize", snapshot_read.expectedSize()).detail("rows", snapshot_read.size()).detail("results", printable(snapshot_read)).detail("more", snapshot_read.more).detail("readToBegin", snapshot_read.readToBegin).detail("readThroughEnd", snapshot_read.readThroughEnd).detail("readThrough", printable(snapshot_read.readThrough));
//TraceEvent("RYWCacheInsert", randomID).detail("Range", printable(range)).detail("ExpectedSize", snapshot_read.expectedSize()).detail("Rows", snapshot_read.size()).detail("Results", printable(snapshot_read)).detail("More", snapshot_read.more).detail("ReadToBegin", snapshot_read.readToBegin).detail("ReadThroughEnd", snapshot_read.readThroughEnd).detail("ReadThrough", printable(snapshot_read.readThrough));
if( ryw->cache.insert( range, snapshot_read ) )
ryw->arena.dependsOn(snapshot_read.arena());
@ -615,7 +615,7 @@ public:
itemsPastEnd += maxCount - count;
//TraceEvent("RYWaddKV", randomID).detail("key", printable(it.beginKey().toStandaloneStringRef())).detail("count", count).detail("maxCount", maxCount).detail("itemsPastEnd", itemsPastEnd);
//TraceEvent("RYWaddKV", randomID).detail("Key", printable(it.beginKey().toStandaloneStringRef())).detail("Count", count).detail("MaxCount", maxCount).detail("ItemsPastEnd", itemsPastEnd);
if( count ) result.append( result.arena(), start, count );
++it;
} else
@ -756,19 +756,19 @@ public:
resolveKeySelectorFromCache( begin, itEnd, ryw->getMaxReadKey(), &readToBegin, &readThroughEnd, &actualBeginOffset );
}
//TraceEvent("RYWSelectorsStartReverse", randomID).detail("byteLimit", limits.bytes).detail("rowLimit", limits.rows);
//TraceEvent("RYWSelectorsStartReverse", randomID).detail("ByteLimit", limits.bytes).detail("RowLimit", limits.rows);
loop {
/*TraceEvent("RYWSelectors", randomID).detail("begin", begin.toString())
.detail("end", end.toString())
.detail("reached", limits.isReached())
.detail("itemsPastBegin", itemsPastBegin)
.detail("endOffset", end.offset)
.detail("itBegin", printable(it.beginKey().toStandaloneStringRef()))
.detail("itEnd", printable(itEnd.beginKey().toStandaloneStringRef()))
.detail("unknown", it.is_unknown_range())
.detail("kv", it.is_kv())
.detail("requests", requestCount);*/
/*TraceEvent("RYWSelectors", randomID).detail("Begin", begin.toString())
.detail("End", end.toString())
.detail("Reached", limits.isReached())
.detail("ItemsPastBegin", itemsPastBegin)
.detail("EndOffset", end.offset)
.detail("ItBegin", printable(it.beginKey().toStandaloneStringRef()))
.detail("ItEnd", printable(itEnd.beginKey().toStandaloneStringRef()))
.detail("Unknown", it.is_unknown_range())
.detail("Kv", it.is_kv())
.detail("Requests", requestCount);*/
if(!result.size() && actualBeginOffset >= actualEndOffset && begin.getKey() >= end.getKey()) {
return RangeResultRef(false, false);
@ -862,13 +862,13 @@ public:
ASSERT( !requestLimit.hasRowLimit() || requestLimit.rows > 0 );
ASSERT( requestLimit.hasRowLimit() || requestLimit.hasByteLimit() );
//TraceEvent("RYWIssuing", randomID).detail("begin", read_begin.toString()).detail("end", read_end.toString()).detail("bytes", requestLimit.bytes).detail("rows", requestLimit.rows).detail("limits", limits.bytes).detail("reached", limits.isReached()).detail("requestCount", requestCount).detail("singleClears", singleClears).detail("ucEnd", printable(ucEnd.beginKey().toStandaloneStringRef())).detail("minRows", requestLimit.minRows);
//TraceEvent("RYWIssuing", randomID).detail("Begin", read_begin.toString()).detail("End", read_end.toString()).detail("Bytes", requestLimit.bytes).detail("Rows", requestLimit.rows).detail("Limits", limits.bytes).detail("Reached", limits.isReached()).detail("RequestCount", requestCount).detail("SingleClears", singleClears).detail("UcEnd", printable(ucEnd.beginKey().toStandaloneStringRef())).detail("MinRows", requestLimit.minRows);
additionalRows = 0;
Standalone<RangeResultRef> snapshot_read = wait( ryw->tr.getRange( read_begin, read_end, requestLimit, true, true ) );
KeyRangeRef range = getKnownKeyRangeBack( snapshot_read, read_begin, read_end, ryw->arena );
//TraceEvent("RYWCacheInsert", randomID).detail("Range", printable(range)).detail("expectedSize", snapshot_read.expectedSize()).detail("rows", snapshot_read.size()).detail("results", printable(snapshot_read)).detail("more", snapshot_read.more).detail("readToBegin", snapshot_read.readToBegin).detail("readThroughEnd", snapshot_read.readThroughEnd).detail("readThrough", printable(snapshot_read.readThrough));
//TraceEvent("RYWCacheInsert", randomID).detail("Range", printable(range)).detail("ExpectedSize", snapshot_read.expectedSize()).detail("Rows", snapshot_read.size()).detail("Results", printable(snapshot_read)).detail("More", snapshot_read.more).detail("ReadToBegin", snapshot_read.readToBegin).detail("ReadThroughEnd", snapshot_read.readThroughEnd).detail("ReadThrough", printable(snapshot_read.readThrough));
RangeResultRef reversed;
reversed.resize(ryw->arena, snapshot_read.size());
@ -895,7 +895,7 @@ public:
}
itemsPastBegin += maxCount - count;
//TraceEvent("RYWaddKV", randomID).detail("key", printable(it.beginKey().toStandaloneStringRef())).detail("count", count).detail("maxCount", maxCount).detail("itemsPastBegin", itemsPastBegin);
//TraceEvent("RYWaddKV", randomID).detail("Key", printable(it.beginKey().toStandaloneStringRef())).detail("Count", count).detail("MaxCount", maxCount).detail("ItemsPastBegin", itemsPastBegin);
if( count ) {
int size = result.size();
result.resize(result.arena(),size+count);

1
fdbclient/ReadYourWrites.h
View File

@ -119,7 +119,6 @@ public:
void cancel();
void reset();
double getBackoff() { return tr.getBackoff(); }
void debugTransaction(UID dID) { tr.debugTransaction(dID); }
Future<Void> debug_onIdle() { return reading; }

2
fdbclient/SnapshotCache.h
View File

@ -331,7 +331,7 @@ public:
void dump() {
for( auto it = entries.begin(); it != entries.end(); ++it ) {
TraceEvent("CacheDump").detail("begin", printable(it->beginKey)).detail("end", printable(it->endKey.toStandaloneStringRef())).detail("values", printable(it->values));
TraceEvent("CacheDump").detail("Begin", printable(it->beginKey)).detail("End", printable(it->endKey.toStandaloneStringRef())).detail("Values", printable(it->values));
}
}

2
fdbclient/SystemData.cpp
View File

@ -453,7 +453,7 @@ Key logRangesEncodeKey(KeyRef keyBegin, UID logUid) {
// Returns the start key and optionally the logRange Uid
KeyRef logRangesDecodeKey(KeyRef key, UID* logUid) {
if (key.size() < logRangesRange.begin.size() + sizeof(UID)) {
TraceEvent(SevError, "InvalidDecodeKey").detail("key", printable(key));
TraceEvent(SevError, "InvalidDecodeKey").detail("Key", printable(key));
ASSERT(false);
}

54
fdbclient/TaskBucket.actor.cpp
View File

@ -229,17 +229,17 @@ public:
ACTOR static Future<bool> taskVerify(Reference<TaskBucket> tb, Reference<ReadYourWritesTransaction> tr, Reference<Task> task) {
if (task->params.find(Task::reservedTaskParamValidKey) == task->params.end()) {
TraceEvent("TB_taskVerify_invalidTask")
.detail("task", printable(task->params[Task::reservedTaskParamKeyType]))
.detail("reservedTaskParamValidKey", "missing");
TraceEvent("TB_TaskVerifyInvalidTask")
.detail("Task", printable(task->params[Task::reservedTaskParamKeyType]))
.detail("ReservedTaskParamValidKey", "missing");
return false;
}
if (task->params.find(Task::reservedTaskParamValidValue) == task->params.end()) {
TraceEvent("TB_taskVerify_invalidTask")
.detail("task", printable(task->params[Task::reservedTaskParamKeyType]))
.detail("reservedTaskParamValidKey", printable(task->params[Task::reservedTaskParamValidKey]))
.detail("reservedTaskParamValidValue", "missing");
TraceEvent("TB_TaskVerifyInvalidTask")
.detail("Task", printable(task->params[Task::reservedTaskParamKeyType]))
.detail("ReservedTaskParamValidKey", printable(task->params[Task::reservedTaskParamValidKey]))
.detail("ReservedTaskParamValidValue", "missing");
return false;
}
@ -248,20 +248,20 @@ public:
Optional<Value> keyValue = wait(tr->get(task->params[Task::reservedTaskParamValidKey]));
if (!keyValue.present()) {
TraceEvent("TB_taskVerify_invalidTask")
.detail("task", printable(task->params[Task::reservedTaskParamKeyType]))
.detail("reservedTaskParamValidKey", printable(task->params[Task::reservedTaskParamValidKey]))
.detail("reservedTaskParamValidValue", printable(task->params[Task::reservedTaskParamValidValue]))
.detail("keyValue", "missing");
TraceEvent("TB_TaskVerifyInvalidTask")
.detail("Task", printable(task->params[Task::reservedTaskParamKeyType]))
.detail("ReservedTaskParamValidKey", printable(task->params[Task::reservedTaskParamValidKey]))
.detail("ReservedTaskParamValidValue", printable(task->params[Task::reservedTaskParamValidValue]))
.detail("KeyValue", "missing");
return false;
}
if (keyValue.get().compare(StringRef(task->params[Task::reservedTaskParamValidValue]))) {
TraceEvent("TB_taskVerify_abortedTask")
.detail("task", printable(task->params[Task::reservedTaskParamKeyType]))
.detail("reservedTaskParamValidKey", printable(task->params[Task::reservedTaskParamValidKey]))
.detail("reservedTaskParamValidValue", printable(task->params[Task::reservedTaskParamValidValue]))
.detail("keyValue", printable(keyValue.get()));
TraceEvent("TB_TaskVerifyAbortedTask")
.detail("Task", printable(task->params[Task::reservedTaskParamKeyType]))
.detail("ReservedTaskParamValidKey", printable(task->params[Task::reservedTaskParamValidKey]))
.detail("ReservedTaskParamValidValue", printable(task->params[Task::reservedTaskParamValidValue]))
.detail("KeyValue", printable(keyValue.get()));
return false;
}
@ -704,14 +704,14 @@ public:
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
tr->setOption(FDBTransactionOptions::LOCK_AWARE);
Standalone<RangeResultRef> values = wait(tr->getRange(subspace.range(), CLIENT_KNOBS->TOO_MANY));
TraceEvent("TaskBucket").detail("debugPrintRange", "Print DB Range").detail("key", printable(subspace.key())).detail("count", values.size()).detail("msg", printable(msg));
TraceEvent("TaskBucket").detail("DebugPrintRange", "Print DB Range").detail("Key", printable(subspace.key())).detail("Count", values.size()).detail("Msg", printable(msg));
/*
printf("debugPrintRange key: (%d) %s\n", values.size(), printable(subspace.key()).c_str());
for (auto & s : values) {
printf(" key: %-40s value: %s\n", printable(s.key).c_str(), printable(s.value).c_str());
TraceEvent("TaskBucket").detail("debugPrintRange", printable(msg))
.detail("key", printable(s.key))
.detail("value", printable(s.value));
TraceEvent("TaskBucket").detail("DebugPrintRange", printable(msg))
.detail("Key", printable(s.key))
.detail("Value", printable(s.value));
}*/
return Void();
@ -835,9 +835,9 @@ ACTOR static Future<Key> actorAddTask(TaskBucket* tb, Reference<ReadYourWritesTr
Optional<Value> validationValue = wait(tr->get(validationKey));
if (!validationValue.present()) {
TraceEvent(SevError, "TB_addTask_invalidKey")
.detail("task", printable(task->params[Task::reservedTaskParamKeyType]))
.detail("validationKey", printable(validationKey));
TraceEvent(SevError, "TB_AddTaskInvalidKey")
.detail("Task", printable(task->params[Task::reservedTaskParamKeyType]))
.detail("ValidationKey", printable(validationKey));
throw invalid_option_value();
}
@ -1103,9 +1103,9 @@ public:
Optional<Value> validationValue = wait(tr->get(validationKey));
if (!validationValue.present()) {
TraceEvent(SevError, "TB_onSetAddTask_invalidKey")
.detail("task", printable(task->params[Task::reservedTaskParamKeyType]))
.detail("validationKey", printable(validationKey));
TraceEvent(SevError, "TB_OnSetAddTaskInvalidKey")
.detail("Task", printable(task->params[Task::reservedTaskParamKeyType]))
.detail("ValidationKey", printable(validationKey));
throw invalid_option_value();
}

2
fdbclient/VersionedMap.h
View File

@ -635,7 +635,7 @@ public:
int count=0, height=0;
PTreeImpl::validate<MapPair<K,std::pair<T,Version>>>( root, at, NULL, NULL, count, height );
if ( height > 100 )
TraceEvent(SevWarnAlways, "DiabolicalPTreeSize").detail("size", count).detail("height", height);
TraceEvent(SevWarnAlways, "DiabolicalPTreeSize").detail("Size", count).detail("Height", height);
}
private:
Tree root;

4
fdbclient/vexillographer/fdb.options
View File

@ -46,8 +46,8 @@ description is not currently required but encouraged.
paramType="Int" paramDescription="max total size of trace files"
description="Sets the maximum size of all the trace output files put together. This value should be in the range ``[0, INT64_MAX]``. If the value is set to 0, there is no limit on the total size of the files. The default is a maximum size of 104,857,600 bytes. If the default roll size is used, this means that a maximum of 10 trace files will be written at a time."/>
<Option name="trace_log_group" code="33"
paramType="String" paramDescription="value of the logGroup attribute"
description="Sets the 'logGroup' attribute with the specified value for all events in the trace output files. The default log group is 'default'."/>
paramType="String" paramDescription="value of the LogGroup attribute"
description="Sets the 'LogGroup' attribute with the specified value for all events in the trace output files. The default log group is 'default'."/>
<Option name="knob" code="40"
paramType="String" paramDescription="knob_name=knob_value"
description="Set internal tuning or debugging knobs"/>

6
fdbrpc/AsyncFileCached.actor.h
View File

@ -229,15 +229,15 @@ private:
ACTOR static Future<Reference<IAsyncFile>> open_impl( std::string filename, int flags, int mode, Reference<EvictablePageCache> pageCache ) {
try {
TraceEvent("AFCUnderlyingOpenBegin").detail("filename", filename);
TraceEvent("AFCUnderlyingOpenBegin").detail("Filename", filename);
if(flags & IAsyncFile::OPEN_CACHED_READ_ONLY)
flags = flags & ~IAsyncFile::OPEN_READWRITE | IAsyncFile::OPEN_READONLY;
else
flags = flags & ~IAsyncFile::OPEN_READONLY | IAsyncFile::OPEN_READWRITE;
state Reference<IAsyncFile> f = wait( IAsyncFileSystem::filesystem()->open(filename, flags | IAsyncFile::OPEN_UNCACHED | IAsyncFile::OPEN_UNBUFFERED, mode) );
TraceEvent("AFCUnderlyingOpenEnd").detail("filename", filename);
TraceEvent("AFCUnderlyingOpenEnd").detail("Filename", filename);
int64_t l = wait( f->size() );
TraceEvent("AFCUnderlyingSize").detail("filename", filename).detail("size", l);
TraceEvent("AFCUnderlyingSize").detail("Filename", filename).detail("Size", l);
auto& of = openFiles[filename];
of.f = new AsyncFileCached(f, filename, l, pageCache);
of.opened = Future<Reference<IAsyncFile>>();

12
fdbrpc/AsyncFileEIO.actor.h
View File

@ -83,10 +83,10 @@ public:
TraceEvent(notFound ? SevWarn : SevWarnAlways, "FileOpenError").error(e).GetLastError().detail("File", filename).detail("Flags", flags).detail("Mode", mode);
throw e;
}
TraceEvent("AsyncFileOpened").detail("Filename", filename).detail("fd", r->result).detail("Flags", flags).suppressFor(1.0);
TraceEvent("AsyncFileOpened").detail("Filename", filename).detail("Fd", r->result).detail("Flags", flags).suppressFor(1.0);
if ((flags & OPEN_LOCK) && !lock_fd(r->result)) {
TraceEvent(SevError, "UnableToLockFile").detail("filename", filename).GetLastError();
TraceEvent(SevError, "UnableToLockFile").detail("Filename", filename).GetLastError();
throw io_error();
}
@ -142,7 +142,7 @@ public:
struct stat buf;
if (fstat( fd, &buf )) {
TraceEvent("AsyncFileEIOFStatError").detail("fd",fd).GetLastError();
TraceEvent("AsyncFileEIOFStatError").detail("Fd",fd).GetLastError();
return io_error();
}
return buf.st_size;
@ -183,7 +183,7 @@ public:
// rename() is atomic
if (rename( part_filename.c_str(), final_filename.c_str() )) {
TraceEvent("AsyncFileEIORenameError").detail("filename", final_filename).GetLastError();
TraceEvent("AsyncFileEIORenameError").detail("Filename", final_filename).GetLastError();
throw io_error();
}
@ -254,7 +254,7 @@ private:
static void error( const char* context, int fd, eio_req* r, Reference<ErrorInfo> const& err = Reference<ErrorInfo>() ) {
Error e = io_error();
errno = r->errorno;
TraceEvent(context).detail("fd", fd).detail("Result", r->result).GetLastError().error(e);
TraceEvent(context).detail("Fd", fd).detail("Result", r->result).GetLastError().error(e);
if (err) err->set(e);
else throw e;
}
@ -264,7 +264,7 @@ private:
state eio_req* r = eio_close(fd, 0, eio_callback, &p);
Void _ = wait( p.getFuture() );
if (r->result) error( "CloseError", fd, r );
TraceEvent("AsyncFileClosed").detail("fd", fd).suppressFor(1.0);
TraceEvent("AsyncFileClosed").detail("Fd", fd).suppressFor(1.0);
}
ACTOR static Future<int> read_impl( int fd, void* data, int length, int64_t offset ) {

24
fdbrpc/AsyncFileKAIO.actor.h
View File

@ -113,7 +113,7 @@ public:
int ecode = errno; // Save errno in case it is modified before it is used below
TraceEvent ev("AsyncFileKAIOOpenFailed");
ev.detail("Filename", filename).detailf("Flags", "%x", flags)
.detailf("OSFlags", "%x", openFlags(flags) | O_DIRECT).detailf("mode", "0%o", mode).error(e).GetLastError();
.detailf("OSFlags", "%x", openFlags(flags) | O_DIRECT).detailf("Mode", "0%o", mode).error(e).GetLastError();
if(ecode == EINVAL)
ev.detail("Description", "Invalid argument - Does the target filesystem support KAIO?");
return e;
@ -121,8 +121,8 @@ public:
TraceEvent("AsyncFileKAIOOpen")
.detail("Filename", filename)
.detail("Flags", flags)
.detail("mode", mode)
.detail("fd", fd);
.detail("Mode", mode)
.detail("Fd", fd);
}
Reference<AsyncFileKAIO> r(new AsyncFileKAIO( fd, flags, filename ));
@ -136,14 +136,14 @@ public:
lockDesc.l_len = 0; // "Specifying 0 for l_len has the special meaning: lock all bytes starting at the location specified by l_whence and l_start through to the end of file, no matter how large the file grows."
lockDesc.l_pid = 0;
if (fcntl(fd, F_SETLK, &lockDesc) == -1) {
TraceEvent(SevError, "UnableToLockFile").detail("filename", filename).GetLastError();
TraceEvent(SevError, "UnableToLockFile").detail("Filename", filename).GetLastError();
return io_error();
}
}
struct stat buf;
if (fstat( fd, &buf )) {
TraceEvent("AsyncFileKAIOFStatError").detail("fd",fd).detail("filename", filename).GetLastError();
TraceEvent("AsyncFileKAIOFStatError").detail("Fd",fd).detail("Filename", filename).GetLastError();
return io_error();
}
@ -262,7 +262,7 @@ public:
result = fallocate( fd, 0, 0, size);
if (result != 0) {
int fallocateErrCode = errno;
TraceEvent("AsyncFileKAIOAllocateError").detail("fd",fd).detail("filename", filename).GetLastError();
TraceEvent("AsyncFileKAIOAllocateError").detail("Fd",fd).detail("Filename", filename).GetLastError();
if ( fallocateErrCode == EOPNOTSUPP ) {
// Mark fallocate as unsupported. Try again with truncate.
ctx.fallocateSupported = false;
@ -280,7 +280,7 @@ public:
KAIOLogEvent(logFile, id, OpLogEntry::TRUNCATE, OpLogEntry::COMPLETE, size / 4096, result);
if(result != 0) {
TraceEvent("AsyncFileKAIOTruncateError").detail("fd",fd).detail("filename", filename).GetLastError();
TraceEvent("AsyncFileKAIOTruncateError").detail("Fd",fd).detail("Filename", filename).GetLastError();
return io_error();
}
@ -409,7 +409,7 @@ public:
double elapsed = timer_monotonic() - begin;
g_network->networkMetrics.secSquaredSubmit += elapsed*elapsed/2;
//TraceEvent("Launched").detail("n", rc).detail("queued", ctx.queue.size()).detail("ms", elapsed*1e3).detail("oustanding", ctx.outstanding+rc);
//TraceEvent("Launched").detail("N", rc).detail("Queued", ctx.queue.size()).detail("Elapsed", elapsed).detail("Outstanding", ctx.outstanding+rc);
//printf("launched: %d/%d in %f us (%d outstanding; lowest prio %d)\n", rc, ctx.queue.size(), elapsed*1e6, ctx.outstanding + rc, toStart[n-1]->getTask());
if (rc<0) {
if (errno == EAGAIN) {
@ -478,16 +478,16 @@ private:
fstat( aio_fildes, &fst );
errno = -r;
TraceEvent("AsyncFileKAIOIOError").GetLastError().detail("fd", aio_fildes).detail("op", aio_lio_opcode).detail("nbytes", nbytes).detail("offset", offset).detail("ptr", int64_t(buf))
.detail("Size", fst.st_size).detail("filename", owner->filename);
TraceEvent("AsyncFileKAIOIOError").GetLastError().detail("Fd", aio_fildes).detail("Op", aio_lio_opcode).detail("Nbytes", nbytes).detail("Offset", offset).detail("Ptr", int64_t(buf))
.detail("Size", fst.st_size).detail("Filename", owner->filename);
}
deliver( result, owner->failed, r, getTask() );
delete this;
}
void timeout(bool warnOnly) {
TraceEvent(SevWarnAlways, "AsyncFileKAIOTimeout").detail("fd", aio_fildes).detail("op", aio_lio_opcode).detail("nbytes", nbytes).detail("offset", offset).detail("ptr", int64_t(buf))
.detail("filename", owner->filename);
TraceEvent(SevWarnAlways, "AsyncFileKAIOTimeout").detail("Fd", aio_fildes).detail("Op", aio_lio_opcode).detail("Nbytes", nbytes).detail("Offset", offset).detail("Ptr", int64_t(buf))
.detail("Filename", owner->filename);
g_network->setGlobal(INetwork::enASIOTimedOut, (flowGlobalType)true);
if(!warnOnly)

4
fdbrpc/AsyncFileNonDurable.actor.h
View File

@ -200,7 +200,7 @@ public:
state int currentTaskID = g_network->getCurrentTask();
state Future<Void> shutdown = success(currentProcess->shutdownSignal.getFuture());
//TraceEvent("AsyncFileNonDurableOpenBegin").detail("Filename", filename).detail("addr", g_simulator.getCurrentProcess()->address);
//TraceEvent("AsyncFileNonDurableOpenBegin").detail("Filename", filename).detail("Addr", g_simulator.getCurrentProcess()->address);
Void _ = wait( g_simulator.onMachine( currentProcess ) );
try {
Void _ = wait(success(wrappedFile) || shutdown);
@ -238,7 +238,7 @@ public:
state Error err = e;
std::string currentFilename = ( wrappedFile.isReady() && !wrappedFile.isError() ) ? wrappedFile.get()->getFilename() : actualFilename;
currentProcess->machine->openFiles.erase( currentFilename );
//TraceEvent("AsyncFileNonDurableOpenError").detail("Filename", filename).detail("address", currentProcess->address).error(e, true).detail("addr", g_simulator.getCurrentProcess()->address);
//TraceEvent("AsyncFileNonDurableOpenError").detail("Filename", filename).detail("Address", currentProcess->address).error(e, true).detail("Addr", g_simulator.getCurrentProcess()->address);
Void _ = wait( g_simulator.onProcess( currentProcess, currentTaskID ) );
throw err;
}

10
fdbrpc/FlowTests.actor.cpp
View File

@ -1098,11 +1098,11 @@ TEST_CASE("flow/flow/YieldedAsyncMap/basic")
//yam.triggerRange(0, 4);
state Future<Void> y2 = yam.onChange(1);
Void _ = wait(reportErrors(y0, "y0"));
Void _ = wait(reportErrors(y1, "y1"));
Void _ = wait(reportErrors(y1a, "y1a"));
Void _ = wait(reportErrors(y1b, "y1b"));
Void _ = wait(reportErrors(timeout(y2, 5, Void()), "y2"));
Void _ = wait(reportErrors(y0, "Y0"));
Void _ = wait(reportErrors(y1, "Y1"));
Void _ = wait(reportErrors(y1a, "Y1a"));
Void _ = wait(reportErrors(y1b, "Y1b"));
Void _ = wait(reportErrors(timeout(y2, 5, Void()), "Y2"));
return Void();
}

6
fdbrpc/FlowTransport.actor.cpp
View File

@ -516,14 +516,14 @@ static void scanPackets( TransportData* transport, uint8_t*& unprocessed_begin,
uint32_t calculatedChecksum = crc32c_append(0, p, packetLen);
if (calculatedChecksum != packetChecksum) {
if (isBuggifyEnabled) {
TraceEvent(SevInfo, "ChecksumMismatchExp").detail("packetChecksum", (int)packetChecksum).detail("calculatedChecksum", (int)calculatedChecksum);
TraceEvent(SevInfo, "ChecksumMismatchExp").detail("PacketChecksum", (int)packetChecksum).detail("CalculatedChecksum", (int)calculatedChecksum);
} else {
TraceEvent(SevWarnAlways, "ChecksumMismatchUnexp").detail("packetChecksum", (int)packetChecksum).detail("calculatedChecksum", (int)calculatedChecksum);
TraceEvent(SevWarnAlways, "ChecksumMismatchUnexp").detail("PacketChecksum", (int)packetChecksum).detail("CalculatedChecksum", (int)calculatedChecksum);
}
throw checksum_failed();
} else {
if (isBuggifyEnabled) {
TraceEvent(SevError, "ChecksumMatchUnexp").detail("packetChecksum", (int)packetChecksum).detail("calculatedChecksum", (int)calculatedChecksum);
TraceEvent(SevError, "ChecksumMatchUnexp").detail("PacketChecksum", (int)packetChecksum).detail("CalculatedChecksum", (int)calculatedChecksum);
}
}
}

20
fdbrpc/batcher.actor.h
View File

@ -47,7 +47,7 @@ bool firstInBatch(CommitTransactionRequest x) {
}
ACTOR template <class X>
Future<Void> batcher(PromiseStream<std::vector<X>> out, FutureStream<X> in, double avgMinDelay, double* avgMaxDelay, double emptyBatchTimeout, int maxCount, int desiredBytes, int maxBytes, Optional<PromiseStream<Void>> batchStartedStream, int taskID = TaskDefaultDelay, Counter* counter = 0)
Future<Void> batcher(PromiseStream<std::pair<std::vector<X>, int> > out, FutureStream<X> in, double avgMinDelay, double* avgMaxDelay, double emptyBatchTimeout, int maxCount, int desiredBytes, int maxBytes, Optional<PromiseStream<Void>> batchStartedStream, int64_t *commitBatchesMemBytesCount, int64_t commitBatchesMemBytesLimit, int taskID = TaskDefaultDelay, Counter* counter = 0)
{
Void _ = wait( delayJittered(*avgMaxDelay, taskID) ); // smooth out
// This is set up to deliver even zero-size batches if emptyBatchTimeout elapses, because that's what master proxy wants. The source control history
@ -68,6 +68,15 @@ Future<Void> batcher(PromiseStream<std::vector<X>> out, FutureStream<X> in, doub
while (!timeout.isReady() && !(batch.size() == maxCount || batchBytes >= desiredBytes)) {
choose {
when ( X x = waitNext(in) ) {
int bytes = getBytes(x);
// Drop requests if memory is under severe pressure
if (*commitBatchesMemBytesCount + bytes > commitBatchesMemBytesLimit) {
x.reply.sendError(proxy_memory_limit_exceeded());
TraceEvent(SevWarnAlways, "ProxyCommitBatchMemoryThresholdExceeded").detail("CommitBatchesMemBytesCount", *commitBatchesMemBytesCount).detail("CommitBatchesMemLimit", commitBatchesMemBytesLimit).suppressFor(60, true);
continue;
}
// Process requests in the normal case
if (counter) ++*counter;
logOnReceive(x);
if (!batch.size()) {
@ -79,10 +88,9 @@ Future<Void> batcher(PromiseStream<std::vector<X>> out, FutureStream<X> in, doub
timeout = delayJittered(*avgMaxDelay - (now() - lastBatch), taskID);
}
int bytes = getBytes( x );
bool first = firstInBatch( x );
if((batchBytes + bytes > maxBytes || first) && batch.size()) {
out.send(batch);
out.send({ batch, batchBytes });
lastBatch = now();
if(batchStartedStream.present())
batchStartedStream.get().send(Void());
@ -93,14 +101,14 @@ Future<Void> batcher(PromiseStream<std::vector<X>> out, FutureStream<X> in, doub
batch.push_back(x);
batchBytes += bytes;
*commitBatchesMemBytesCount += bytes;
}
when ( Void _ = wait( timeout ) ) {}
}
}
out.send(batch);
out.send({std::move(batch), batchBytes});
lastBatch = now();
}
}
#endif
#endif

56
fdbrpc/sim2.actor.cpp
View File

@ -633,7 +633,7 @@ private:
std::string sourceFilename = self->filename + ".part";
if(machineCache.count(sourceFilename)) {
TraceEvent("SimpleFileRename").detail("From", sourceFilename).detail("To", self->filename).detail("sourceCount", machineCache.count(sourceFilename)).detail("fileCount", machineCache.count(self->filename));
TraceEvent("SimpleFileRename").detail("From", sourceFilename).detail("To", self->filename).detail("SourceCount", machineCache.count(sourceFilename)).detail("FileCount", machineCache.count(self->filename));
renameFile( sourceFilename.c_str(), self->filename.c_str() );
ASSERT(!machineCache.count(self->filename));
@ -866,7 +866,7 @@ public:
diskSpace.totalSpace = 5e9 + g_random->random01() * 100e9; //Total space between 5GB and 105GB
diskSpace.baseFreeSpace = std::min<int64_t>(diskSpace.totalSpace, std::max(5e9, (g_random->random01() * (1 - .075) + .075) * diskSpace.totalSpace) + totalFileSize); //Minimum 5GB or 7.5% total disk space, whichever is higher
TraceEvent("Sim2DiskSpaceInitialization").detail("TotalSpace", diskSpace.totalSpace).detail("BaseFreeSpace", diskSpace.baseFreeSpace).detail("totalFileSize", totalFileSize).detail("NumFiles", numFiles);
TraceEvent("Sim2DiskSpaceInitialization").detail("TotalSpace", diskSpace.totalSpace).detail("BaseFreeSpace", diskSpace.baseFreeSpace).detail("TotalFileSize", totalFileSize).detail("NumFiles", numFiles);
}
else {
int64_t maxDelta = std::min(5.0, (now() - diskSpace.lastUpdate)) * (BUGGIFY ? 10e6 : 1e6); //External processes modifying the disk
@ -880,7 +880,7 @@ public:
free = std::max<int64_t>(0, diskSpace.baseFreeSpace - totalFileSize);
if(free == 0)
TraceEvent(SevWarnAlways, "Sim2NoFreeSpace").detail("TotalSpace", diskSpace.totalSpace).detail("BaseFreeSpace", diskSpace.baseFreeSpace).detail("totalFileSize", totalFileSize).detail("NumFiles", numFiles);
TraceEvent(SevWarnAlways, "Sim2NoFreeSpace").detail("TotalSpace", diskSpace.totalSpace).detail("BaseFreeSpace", diskSpace.baseFreeSpace).detail("TotalFileSize", totalFileSize).detail("NumFiles", numFiles);
}
virtual bool isAddressOnThisHost( NetworkAddress const& addr ) {
return addr.ip == getCurrentProcess()->address.ip;
@ -964,8 +964,8 @@ public:
for( int i = 0; i < machine.processes.size(); i++ ) {
if( machine.processes[i]->locality.zoneId() != locality.zoneId() ) { // SOMEDAY: compute ip from locality to avoid this check
TraceEvent("Sim2Mismatch").detail("IP", format("%x", ip))
.detailext("zoneId", locality.zoneId()).detail("NewName", name)
.detailext("ExistingmachineId", machine.processes[i]->locality.zoneId()).detail("ExistingName", machine.processes[i]->name);
.detailext("ZoneId", locality.zoneId()).detail("NewName", name)
.detailext("ExistingMachineId", machine.processes[i]->locality.zoneId()).detail("ExistingName", machine.processes[i]->name);
ASSERT( false );
}
ASSERT( machine.processes[i]->address.port != port );
@ -993,7 +993,7 @@ public:
m->setGlobal(enNetworkConnections, (flowGlobalType) m->network);
m->setGlobal(enASIOTimedOut, (flowGlobalType) false);
TraceEvent("NewMachine").detail("Name", name).detail("Address", m->address).detailext("zoneId", m->locality.zoneId()).detail("Excluded", m->excluded).detail("Cleared", m->cleared);
TraceEvent("NewMachine").detail("Name", name).detail("Address", m->address).detailext("ZoneId", m->locality.zoneId()).detail("Excluded", m->excluded).detail("Cleared", m->cleared);
// FIXME: Sometimes, connections to/from this process will explicitly close
@ -1123,21 +1123,21 @@ public:
if (tooManyDead) {
newKt = Reboot;
canSurvive = false;
TraceEvent("KillChanged").detail("KillType", kt).detail("NewKillType", newKt).detail("tLogPolicy", tLogPolicy->info()).detail("Reason", "tLogPolicy validates against dead processes.");
TraceEvent("KillChanged").detail("KillType", kt).detail("NewKillType", newKt).detail("TLogPolicy", tLogPolicy->info()).detail("Reason", "tLogPolicy validates against dead processes.");
}
// Reboot and Delete if remaining machines do NOT fulfill policies
else if ((kt != RebootAndDelete) && (kt != RebootProcessAndDelete) && notEnoughLeft) {
newKt = (g_random->random01() < 0.33) ? RebootAndDelete : Reboot;
canSurvive = false;
TraceEvent("KillChanged").detail("KillType", kt).detail("NewKillType", newKt).detail("tLogPolicy", tLogPolicy->info()).detail("Reason", "tLogPolicy does not validates against remaining processes.");
TraceEvent("KillChanged").detail("KillType", kt).detail("NewKillType", newKt).detail("TLogPolicy", tLogPolicy->info()).detail("Reason", "tLogPolicy does not validates against remaining processes.");
}
else if ((kt != RebootAndDelete) && (kt != RebootProcessAndDelete) && (nQuorum > uniqueMachines.size())) {
newKt = (g_random->random01() < 0.33) ? RebootAndDelete : Reboot;
canSurvive = false;
TraceEvent("KillChanged").detail("KillType", kt).detail("NewKillType", newKt).detail("storagePolicy", storagePolicy->info()).detail("Quorum", nQuorum).detail("Machines", uniqueMachines.size()).detail("Reason", "Not enough unique machines to perform auto configuration of coordinators.");
TraceEvent("KillChanged").detail("KillType", kt).detail("NewKillType", newKt).detail("StoragePolicy", storagePolicy->info()).detail("Quorum", nQuorum).detail("Machines", uniqueMachines.size()).detail("Reason", "Not enough unique machines to perform auto configuration of coordinators.");
}
else {
TraceEvent("CanSurviveKills").detail("KillType", kt).detail("tLogPolicy", tLogPolicy->info()).detail("storagePolicy", storagePolicy->info()).detail("Quorum", nQuorum).detail("Machines", uniqueMachines.size());
TraceEvent("CanSurviveKills").detail("KillType", kt).detail("TLogPolicy", tLogPolicy->info()).detail("StoragePolicy", storagePolicy->info()).detail("Quorum", nQuorum).detail("Machines", uniqueMachines.size());
}
}
if (newKillType) *newKillType = newKt;
@ -1145,7 +1145,7 @@ public:
}
virtual void destroyProcess( ISimulator::ProcessInfo *p ) {
TraceEvent("ProcessDestroyed").detail("Name", p->name).detail("Address", p->address).detailext("zoneId", p->locality.zoneId());
TraceEvent("ProcessDestroyed").detail("Name", p->name).detail("Address", p->address).detailext("ZoneId", p->locality.zoneId());
currentlyRebootingProcesses.insert(std::pair<NetworkAddress, ProcessInfo*>(p->address, p));
std::vector<ProcessInfo*>& processes = machines[ p->locality.zoneId().get() ].processes;
if( p != processes.back() ) {
@ -1202,7 +1202,7 @@ public:
}
}
virtual void killProcess( ProcessInfo* machine, KillType kt ) {
TraceEvent("attemptingKillProcess");
TraceEvent("AttemptingKillProcess");
if (kt < RebootAndDelete ) {
killProcess_internal( machine, kt );
}
@ -1241,7 +1241,7 @@ public:
// Do nothing, if no processes to kill
if (processesOnMachine == 0) {
TraceEvent(SevWarn, "AbortedKill", zoneId).detailext("ZoneId", zoneId).detail("Reason", "The target had no processes running.").detail("processes", processesOnMachine).detail("processesPerMachine", processesPerMachine).backtrace();
TraceEvent(SevWarn, "AbortedKill", zoneId).detailext("ZoneId", zoneId).detail("Reason", "The target had no processes running.").detail("Processes", processesOnMachine).detail("ProcessesPerMachine", processesPerMachine).backtrace();
if (ktFinal) *ktFinal = None;
return false;
}
@ -1285,10 +1285,10 @@ public:
if ((kt != Reboot) && (!killIsSafe)) {
kt = Reboot;
}
TraceEvent("ChangedKillMachine", zoneId).detailext("ZoneId", zoneId).detail("KillType", kt).detail("OrigKillType", ktOrig).detail("ProcessesLeft", processesLeft.size()).detail("ProcessesDead", processesDead.size()).detail("TotalProcesses", machines.size()).detail("processesPerMachine", processesPerMachine).detail("Protected", protectedWorker).detail("Unavailable", unavailable).detail("Excluded", excluded).detail("Cleared", cleared).detail("ProtectedTotal", protectedAddresses.size()).detail("tLogPolicy", tLogPolicy->info()).detail("storagePolicy", storagePolicy->info());
TraceEvent("ChangedKillMachine", zoneId).detailext("ZoneId", zoneId).detail("KillType", kt).detail("OrigKillType", ktOrig).detail("ProcessesLeft", processesLeft.size()).detail("ProcessesDead", processesDead.size()).detail("TotalProcesses", machines.size()).detail("ProcessesPerMachine", processesPerMachine).detail("Protected", protectedWorker).detail("Unavailable", unavailable).detail("Excluded", excluded).detail("Cleared", cleared).detail("ProtectedTotal", protectedAddresses.size()).detail("TLogPolicy", tLogPolicy->info()).detail("StoragePolicy", storagePolicy->info());
}
else if ((kt == KillInstantly) || (kt == InjectFaults)) {
TraceEvent("DeadMachine", zoneId).detailext("ZoneId", zoneId).detail("KillType", kt).detail("ProcessesLeft", processesLeft.size()).detail("ProcessesDead", processesDead.size()).detail("TotalProcesses", machines.size()).detail("processesPerMachine", processesPerMachine).detail("tLogPolicy", tLogPolicy->info()).detail("storagePolicy", storagePolicy->info());
TraceEvent("DeadMachine", zoneId).detailext("ZoneId", zoneId).detail("KillType", kt).detail("ProcessesLeft", processesLeft.size()).detail("ProcessesDead", processesDead.size()).detail("TotalProcesses", machines.size()).detail("ProcessesPerMachine", processesPerMachine).detail("TLogPolicy", tLogPolicy->info()).detail("StoragePolicy", storagePolicy->info());
for (auto process : processesLeft) {
TraceEvent("DeadMachineSurvivors", zoneId).detailext("ZoneId", zoneId).detail("KillType", kt).detail("ProcessesLeft", processesLeft.size()).detail("ProcessesDead", processesDead.size()).detail("SurvivingProcess", describe(*process));
}
@ -1297,7 +1297,7 @@ public:
}
}
else {
TraceEvent("ClearMachine", zoneId).detailext("ZoneId", zoneId).detail("KillType", kt).detail("ProcessesLeft", processesLeft.size()).detail("ProcessesDead", processesDead.size()).detail("TotalProcesses", machines.size()).detail("processesPerMachine", processesPerMachine).detail("tLogPolicy", tLogPolicy->info()).detail("storagePolicy", storagePolicy->info());
TraceEvent("ClearMachine", zoneId).detailext("ZoneId", zoneId).detail("KillType", kt).detail("ProcessesLeft", processesLeft.size()).detail("ProcessesDead", processesDead.size()).detail("TotalProcesses", machines.size()).detail("ProcessesPerMachine", processesPerMachine).detail("TLogPolicy", tLogPolicy->info()).detail("StoragePolicy", storagePolicy->info());
for (auto process : processesLeft) {
TraceEvent("ClearMachineSurvivors", zoneId).detailext("ZoneId", zoneId).detail("KillType", kt).detail("ProcessesLeft", processesLeft.size()).detail("ProcessesDead", processesDead.size()).detail("SurvivingProcess", describe(*process));
}
@ -1312,7 +1312,7 @@ public:
// Check if any processes on machine are rebooting
if( processesOnMachine != processesPerMachine && kt >= RebootAndDelete ) {
TEST(true); //Attempted reboot, but the target did not have all of its processes running
TraceEvent(SevWarn, "AbortedKill", zoneId).detail("KillType", kt).detailext("ZoneId", zoneId).detail("Reason", "Machine processes does not match number of processes per machine").detail("processes", processesOnMachine).detail("processesPerMachine", processesPerMachine).backtrace();
TraceEvent(SevWarn, "AbortedKill", zoneId).detail("KillType", kt).detailext("ZoneId", zoneId).detail("Reason", "Machine processes does not match number of processes per machine").detail("Processes", processesOnMachine).detail("ProcessesPerMachine", processesPerMachine).backtrace();
if (ktFinal) *ktFinal = None;
return false;
}
@ -1320,24 +1320,24 @@ public:
// Check if any processes on machine are rebooting
if ( processesOnMachine != processesPerMachine) {
TEST(true); //Attempted reboot, but the target did not have all of its processes running
TraceEvent(SevWarn, "AbortedKill", zoneId).detail("KillType", kt).detailext("ZoneId", zoneId).detail("Reason", "Machine processes does not match number of processes per machine").detail("processes", processesOnMachine).detail("processesPerMachine", processesPerMachine).backtrace();
TraceEvent(SevWarn, "AbortedKill", zoneId).detail("KillType", kt).detailext("ZoneId", zoneId).detail("Reason", "Machine processes does not match number of processes per machine").detail("Processes", processesOnMachine).detail("ProcessesPerMachine", processesPerMachine).backtrace();
if (ktFinal) *ktFinal = None;
return false;
}
TraceEvent("KillMachine", zoneId).detailext("ZoneId", zoneId).detail("Kt", kt).detail("KtOrig", ktOrig).detail("KillableMachines", processesOnMachine).detail("ProcessPerMachine", processesPerMachine).detail("KillChanged", kt!=ktOrig).detail("killIsSafe", killIsSafe);
TraceEvent("KillMachine", zoneId).detailext("ZoneId", zoneId).detail("Kt", kt).detail("KtOrig", ktOrig).detail("KillableMachines", processesOnMachine).detail("ProcessPerMachine", processesPerMachine).detail("KillChanged", kt!=ktOrig).detail("KillIsSafe", killIsSafe);
if (kt < RebootAndDelete ) {
if(kt == InjectFaults && machines[zoneId].machineProcess != nullptr)
killProcess_internal( machines[zoneId].machineProcess, kt );
for (auto& process : machines[zoneId].processes) {
TraceEvent("KillMachineProcess", zoneId).detail("KillType", kt).detail("Process", process->toString()).detail("startingClass", process->startingClass.toString()).detail("failed", process->failed).detail("excluded", process->excluded).detail("cleared", process->cleared).detail("rebooting", process->rebooting);
TraceEvent("KillMachineProcess", zoneId).detail("KillType", kt).detail("Process", process->toString()).detail("StartingClass", process->startingClass.toString()).detail("Failed", process->failed).detail("Excluded", process->excluded).detail("Cleared", process->cleared).detail("Rebooting", process->rebooting);
if (process->startingClass != ProcessClass::TesterClass)
killProcess_internal( process, kt );
}
}
else if ( kt == Reboot || killIsSafe) {
for (auto& process : machines[zoneId].processes) {
TraceEvent("KillMachineProcess", zoneId).detail("KillType", kt).detail("Process", process->toString()).detail("startingClass", process->startingClass.toString()).detail("failed", process->failed).detail("excluded", process->excluded).detail("cleared", process->cleared).detail("rebooting", process->rebooting);
TraceEvent("KillMachineProcess", zoneId).detail("KillType", kt).detail("Process", process->toString()).detail("StartingClass", process->startingClass.toString()).detail("Failed", process->failed).detail("Excluded", process->excluded).detail("Cleared", process->cleared).detail("Rebooting", process->rebooting);
if (process->startingClass != ProcessClass::TesterClass)
doReboot(process, kt );
}
@ -1367,7 +1367,7 @@ public:
if ((kt != Reboot) && (protectedAddresses.count(procRecord->address))) {
kt = Reboot;
TraceEvent(SevWarn, "DcKillChanged").detailext("DataCenter", dcId).detail("KillType", kt).detail("OrigKillType", ktOrig)
.detail("Reason", "Datacenter has protected process").detail("ProcessAddress", procRecord->address).detail("failed", procRecord->failed).detail("rebooting", procRecord->rebooting).detail("excluded", procRecord->excluded).detail("cleared", procRecord->cleared).detail("Process", describe(*procRecord));
.detail("Reason", "Datacenter has protected process").detail("ProcessAddress", procRecord->address).detail("Failed", procRecord->failed).detail("Rebooting", procRecord->rebooting).detail("Excluded", procRecord->excluded).detail("Cleared", procRecord->cleared).detail("Process", describe(*procRecord));
}
datacenterZones[processZoneId.get()] ++;
dcProcesses ++;
@ -1404,7 +1404,7 @@ public:
TraceEvent(SevWarn, "DcKillChanged").detailext("DataCenter", dcId).detail("KillType", kt).detail("OrigKillType", ktOrig);
}
else {
TraceEvent("DeadDataCenter").detailext("DataCenter", dcId).detail("KillType", kt).detail("DcZones", datacenterZones.size()).detail("DcProcesses", dcProcesses).detail("ProcessesDead", processesDead.size()).detail("ProcessesLeft", processesLeft.size()).detail("tLogPolicy", tLogPolicy->info()).detail("storagePolicy", storagePolicy->info());
TraceEvent("DeadDataCenter").detailext("DataCenter", dcId).detail("KillType", kt).detail("DcZones", datacenterZones.size()).detail("DcProcesses", dcProcesses).detail("ProcessesDead", processesDead.size()).detail("ProcessesLeft", processesLeft.size()).detail("TLogPolicy", tLogPolicy->info()).detail("StoragePolicy", storagePolicy->info());
for (auto process : processesLeft) {
auto zoneId = process->locality.zoneId();
TraceEvent("DeadDcSurvivors", zoneId).detailext("ZoneId", zoneId).detail("KillType", kt).detail("ProcessesLeft", processesLeft.size()).detail("ProcessesDead", processesDead.size()).detail("SurvivingProcess", describe(*process));
@ -1420,7 +1420,7 @@ public:
for (auto& datacenterZone : datacenterZones) {
killMachine( datacenterZone.first, kt, (kt == RebootAndDelete), true, &ktResult);
if (ktResult != kt) {
TraceEvent(SevWarn, "killDCFail")
TraceEvent(SevWarn, "KillDCFail")
.detailext("Zone", datacenterZone.first)
.detail("KillType", kt)
.detail("KillTypeResult", ktResult)
@ -1430,7 +1430,7 @@ public:
ktMin = std::min<KillType>( ktResult, ktMin );
}
TraceEvent("killDataCenter")
TraceEvent("KillDataCenter")
.detail("DcZones", datacenterZones.size())
.detail("DcProcesses", dcProcesses)
.detailext("DCID", dcId)
@ -1546,7 +1546,7 @@ public:
/*auto elapsed = getCPUTicks() - before;
currentProcess->cpuTicks += elapsed;
if (g_random->random01() < 0.01){
TraceEvent("st").detail("cpu", currentProcess->cpuTicks);
TraceEvent("TaskDuration").detail("CpuTicks", currentProcess->cpuTicks);
currentProcess->cpuTicks = 0;
}*/
} catch (Error& e) {
@ -1627,7 +1627,7 @@ static double networkLatency() {
}
ACTOR void doReboot( ISimulator::ProcessInfo *p, ISimulator::KillType kt ) {
TraceEvent("RebootingProcessAttempt").detailext("ZoneId", p->locality.zoneId()).detail("KillType", kt).detail("Process", p->toString()).detail("startingClass", p->startingClass.toString()).detail("failed", p->failed).detail("excluded", p->excluded).detail("cleared", p->cleared).detail("rebooting", p->rebooting).detail("TaskDefaultDelay", TaskDefaultDelay);
TraceEvent("RebootingProcessAttempt").detailext("ZoneId", p->locality.zoneId()).detail("KillType", kt).detail("Process", p->toString()).detail("StartingClass", p->startingClass.toString()).detail("Failed", p->failed).detail("Excluded", p->excluded).detail("Cleared", p->cleared).detail("Rebooting", p->rebooting).detail("TaskDefaultDelay", TaskDefaultDelay);
Void _ = wait( g_sim2.delay( 0, TaskDefaultDelay, p ) ); // Switch to the machine in question
@ -1641,7 +1641,7 @@ ACTOR void doReboot( ISimulator::ProcessInfo *p, ISimulator::KillType kt ) {
if( p->rebooting )
return;
TraceEvent("RebootingProcess").detail("KillType", kt).detail("Address", p->address).detailext("ZoneId", p->locality.zoneId()).detailext("DataHall", p->locality.dataHallId()).detail("Locality", p->locality.toString()).detail("failed", p->failed).detail("excluded", p->excluded).detail("cleared", p->cleared).backtrace();
TraceEvent("RebootingProcess").detail("KillType", kt).detail("Address", p->address).detailext("ZoneId", p->locality.zoneId()).detailext("DataHall", p->locality.dataHallId()).detail("Locality", p->locality.toString()).detail("Failed", p->failed).detail("Excluded", p->excluded).detail("Cleared", p->cleared).backtrace();
p->rebooting = true;
if ((kt == ISimulator::RebootAndDelete) || (kt == ISimulator::RebootProcessAndDelete)) {
p->cleared = true;

28
fdbserver/ApplyMetadataMutation.h
View File

@ -106,7 +106,7 @@ static void applyMetadataMutations(UID const& dbgid, Arena &arena, VectorRef<Mut
MutationRef privatized = m;
privatized.param1 = m.param1.withPrefix(systemKeys.begin, arena);
TraceEvent(SevDebug, "SendingPrivateMutation", dbgid).detail("Original", m.toString()).detail("Privatized", privatized.toString()).detail("Server", serverKeysDecodeServer(m.param1))
.detail("tagKey", printable(serverTagKeyFor( serverKeysDecodeServer(m.param1) ))).detail("tag", decodeServerTagValue( txnStateStore->readValue( serverTagKeyFor( serverKeysDecodeServer(m.param1) ) ).get().get() ).toString());
.detail("TagKey", printable(serverTagKeyFor( serverKeysDecodeServer(m.param1) ))).detail("Tag", decodeServerTagValue( txnStateStore->readValue( serverTagKeyFor( serverKeysDecodeServer(m.param1) ) ).get().get() ).toString());
toCommit->addTag( decodeServerTagValue( txnStateStore->readValue( serverTagKeyFor( serverKeysDecodeServer(m.param1) ) ).get().get() ) );
toCommit->addTypedMessage(privatized);
@ -118,7 +118,7 @@ static void applyMetadataMutations(UID const& dbgid, Arena &arena, VectorRef<Mut
if(toCommit) {
MutationRef privatized = m;
privatized.param1 = m.param1.withPrefix(systemKeys.begin, arena);
TraceEvent("ServerTag", dbgid).detail("server", id).detail("tag", tag.toString());
TraceEvent("ServerTag", dbgid).detail("Server", id).detail("Tag", tag.toString());
toCommit->addTag(tag);
toCommit->addTypedMessage(LogProtocolMessage());
@ -151,7 +151,7 @@ static void applyMetadataMutations(UID const& dbgid, Arena &arena, VectorRef<Mut
if(Optional<StringRef>(m.param2) != txnStateStore->readValue(m.param1).get().cast_to<StringRef>()) { // FIXME: Make this check more specific, here or by reading configuration whenever there is a change
if(!m.param1.startsWith( excludedServersPrefix ) && m.param1 != excludedServersVersionKey) {
auto t = txnStateStore->readValue(m.param1).get();
TraceEvent("MutationRequiresRestart", dbgid).detail("M", m.toString()).detail("PrevValue", t.present() ? printable(t.get()) : "(none)").detail("toCommit", toCommit!=NULL);
TraceEvent("MutationRequiresRestart", dbgid).detail("M", m.toString()).detail("PrevValue", t.present() ? printable(t.get()) : "(none)").detail("ToCommit", toCommit!=NULL);
if(confChange) *confChange = true;
}
}
@ -227,8 +227,8 @@ static void applyMetadataMutations(UID const& dbgid, Arena &arena, VectorRef<Mut
}
// Log the modification
TraceEvent("LogRangeAdd").detail("logRanges", vecBackupKeys->size()).detail("mutationKey", printable(m.param1))
.detail("logRangeBegin", printable(logRangeBegin)).detail("logRangeEnd", printable(logRangeEnd));
TraceEvent("LogRangeAdd").detail("LogRanges", vecBackupKeys->size()).detail("MutationKey", printable(m.param1))
.detail("LogRangeBegin", printable(logRangeBegin)).detail("LogRangeEnd", printable(logRangeEnd));
}
}
else if (m.param1.startsWith(globalKeysPrefix)) {
@ -254,7 +254,7 @@ static void applyMetadataMutations(UID const& dbgid, Arena &arena, VectorRef<Mut
}
else if (m.param1 == minRequiredCommitVersionKey) {
Version requested = BinaryReader::fromStringRef<Version>(m.param2, Unversioned());
TraceEvent("MinRequiredCommitVersion", dbgid).detail("min", requested).detail("current", popVersion).detail("hasConf", !!confChange);
TraceEvent("MinRequiredCommitVersion", dbgid).detail("Min", requested).detail("Current", popVersion).detail("HasConf", !!confChange);
if(!initialCommit) txnStateStore->set(KeyValueRef(m.param1, m.param2));
if (confChange) *confChange = true;
TEST(true); // Recovering at a higher version.
@ -292,7 +292,7 @@ static void applyMetadataMutations(UID const& dbgid, Arena &arena, VectorRef<Mut
auto serverKeysCleared = txnStateStore->readRange( range & serverTagKeys ).get(); // read is expected to be immediately available
for(auto &kv : serverKeysCleared) {
Tag tag = decodeServerTagValue(kv.value);
TraceEvent("ServerTagRemove").detail("popVersion", popVersion).detail("tag", tag.toString()).detail("server", decodeServerTagKey(kv.key));
TraceEvent("ServerTagRemove").detail("PopVersion", popVersion).detail("Tag", tag.toString()).detail("Server", decodeServerTagKey(kv.key));
logSystem->pop( popVersion, decodeServerTagValue(kv.value) );
(*tag_popped)[tag] = popVersion;
@ -320,7 +320,7 @@ static void applyMetadataMutations(UID const& dbgid, Arena &arena, VectorRef<Mut
auto serverKeysCleared = txnStateStore->readRange( range & serverTagHistoryKeys ).get(); // read is expected to be immediately available
for(auto &kv : serverKeysCleared) {
Tag tag = decodeServerTagValue(kv.value);
TraceEvent("ServerTagHistoryRemove").detail("popVersion", popVersion).detail("tag", tag.toString()).detail("version", decodeServerTagHistoryKey(kv.key));
TraceEvent("ServerTagHistoryRemove").detail("PopVersion", popVersion).detail("Tag", tag.toString()).detail("Version", decodeServerTagHistoryKey(kv.key));
logSystem->pop( popVersion, tag );
(*tag_popped)[tag] = popVersion;
}
@ -362,8 +362,8 @@ static void applyMetadataMutations(UID const& dbgid, Arena &arena, VectorRef<Mut
KeyRangeRef commonLogRange(range & logRangesRange);
TraceEvent("LogRangeClear")
.detail("rangeBegin", printable(range.begin)).detail("rangeEnd", printable(range.end))
.detail("intersectBegin", printable(commonLogRange.begin)).detail("intersectEnd", printable(commonLogRange.end));
.detail("RangeBegin", printable(range.begin)).detail("RangeEnd", printable(range.end))
.detail("IntersectBegin", printable(commonLogRange.begin)).detail("IntersectEnd", printable(commonLogRange.end));
// Remove the key range from the vector, if defined
if (vecBackupKeys) {
@ -374,7 +374,7 @@ static void applyMetadataMutations(UID const& dbgid, Arena &arena, VectorRef<Mut
// read is expected to be immediately available
auto logRangesAffected = txnStateStore->readRange(commonLogRange).get();
TraceEvent("LogRangeClearBegin").detail("affectedLogRanges", logRangesAffected.size());
TraceEvent("LogRangeClearBegin").detail("AffectedLogRanges", logRangesAffected.size());
// Add the backup name to the backup locations that do not have it
for (auto logRangeAffected : logRangesAffected)
@ -385,9 +385,9 @@ static void applyMetadataMutations(UID const& dbgid, Arena &arena, VectorRef<Mut
// Decode the log destination and key value
logKeyEnd = logRangesDecodeValue(logRangeAffected.value, &logDestination);
TraceEvent("LogRangeErase").detail("affectedKey", printable(logRangeAffected.key)).detail("affectedValue", printable(logRangeAffected.value))
.detail("logKeyBegin", printable(logKeyBegin)).detail("logKeyEnd", printable(logKeyEnd))
.detail("logDestination", printable(logDestination));
TraceEvent("LogRangeErase").detail("AffectedKey", printable(logRangeAffected.key)).detail("AffectedValue", printable(logRangeAffected.value))
.detail("LogKeyBegin", printable(logKeyBegin)).detail("LogKeyEnd", printable(logKeyEnd))
.detail("LogDestination", printable(logDestination));
// Identify the locations to place the backup key
auto logRanges = vecBackupKeys->modify(KeyRangeRef(logKeyBegin, logKeyEnd));

42
fdbserver/ClusterController.actor.cpp
View File

@ -273,7 +273,7 @@ public:
logServerMap->add(worker.first.locality, &worker);
}
if (logServerSet->size() < required) {
TraceEvent(SevWarn,"GWFTADTooFew", id).detail("Fitness", fitness).detail("Processes", logServerSet->size()).detail("Required", required).detail("tLogPolicy", policy->info()).detail("DesiredLogs", desired);
TraceEvent(SevWarn,"GWFTADTooFew", id).detail("Fitness", fitness).detail("Processes", logServerSet->size()).detail("Required", required).detail("TLogPolicy", policy->info()).detail("DesiredLogs", desired);
}
else if (logServerSet->size() <= desired) {
if (logServerSet->validate(policy)) {
@ -283,7 +283,7 @@ public:
bCompleted = true;
break;
}
TraceEvent(SevWarn,"GWFTADNotAcceptable", id).detail("Fitness", fitness).detail("Processes", logServerSet->size()).detail("Required", required).detail("tLogPolicy",policy->info()).detail("DesiredLogs", desired);
TraceEvent(SevWarn,"GWFTADNotAcceptable", id).detail("Fitness", fitness).detail("Processes", logServerSet->size()).detail("Required", required).detail("TLogPolicy",policy->info()).detail("DesiredLogs", desired);
}
// Try to select the desired size, if larger
else {
@ -300,11 +300,11 @@ public:
tLocalities.push_back(object->first.locality);
}
TraceEvent("GWFTADBestResults", id).detail("Fitness", fitness).detail("Processes", logServerSet->size()).detail("BestCount", bestSet.size()).detail("BestZones", ::describeZones(tLocalities))
.detail("BestDataHalls", ::describeDataHalls(tLocalities)).detail("tLogPolicy", policy->info()).detail("TotalResults", results.size()).detail("DesiredLogs", desired);
.detail("BestDataHalls", ::describeDataHalls(tLocalities)).detail("TLogPolicy", policy->info()).detail("TotalResults", results.size()).detail("DesiredLogs", desired);
bCompleted = true;
break;
}
TraceEvent(SevWarn,"GWFTADNoBest", id).detail("Fitness", fitness).detail("Processes", logServerSet->size()).detail("Required", required).detail("tLogPolicy", policy->info()).detail("DesiredLogs", desired);
TraceEvent(SevWarn,"GWFTADNoBest", id).detail("Fitness", fitness).detail("Processes", logServerSet->size()).detail("Required", required).detail("TLogPolicy", policy->info()).detail("DesiredLogs", desired);
}
}
@ -318,7 +318,7 @@ public:
TraceEvent(SevWarn, "GetTLogTeamFailed").detail("Policy", policy->info()).detail("Processes", logServerSet->size()).detail("Workers", id_worker.size()).detail("FitnessGroups", fitness_workers.size())
.detail("TLogZones", ::describeZones(tLocalities)).detail("TLogDataHalls", ::describeDataHalls(tLocalities)).detail("MissingZones", ::describeZones(unavailableLocals))
.detail("MissingDataHalls", ::describeDataHalls(unavailableLocals)).detail("Required", required).detail("DesiredLogs", desired).detail("RatingTests",SERVER_KNOBS->POLICY_RATING_TESTS)
.detail("checkStable", checkStable).detail("PolicyGenerations",SERVER_KNOBS->POLICY_GENERATIONS).backtrace();
.detail("CheckStable", checkStable).detail("PolicyGenerations",SERVER_KNOBS->POLICY_GENERATIONS).backtrace();
logServerSet->clear();
logServerSet.clear();
@ -675,10 +675,10 @@ public:
}
//If this cluster controller dies, do not prioritize recruiting the next one in the same DC
desiredDcIds.set(vector<Optional<Key>>());
TraceEvent("findWorkersForConfig").detail("replication", req.configuration.tLogReplicationFactor)
.detail("desiredLogs", req.configuration.getDesiredLogs()).detail("actualLogs", result.tLogs.size())
.detail("desiredProxies", req.configuration.getDesiredProxies()).detail("actualProxies", result.proxies.size())
.detail("desiredResolvers", req.configuration.getDesiredResolvers()).detail("actualResolvers", result.resolvers.size());
TraceEvent("FindWorkersForConfig").detail("Replication", req.configuration.tLogReplicationFactor)
.detail("DesiredLogs", req.configuration.getDesiredLogs()).detail("ActualLogs", result.tLogs.size())
.detail("DesiredProxies", req.configuration.getDesiredProxies()).detail("ActualProxies", result.proxies.size())
.detail("DesiredResolvers", req.configuration.getDesiredResolvers()).detail("ActualResolvers", result.resolvers.size());
if( now() - startTime < SERVER_KNOBS->WAIT_FOR_GOOD_RECRUITMENT_DELAY &&
( RoleFitness(tlogs, ProcessClass::TLog) > RoleFitness(SERVER_KNOBS->EXPECTED_TLOG_FITNESS, req.configuration.getDesiredLogs()) ||
@ -882,13 +882,13 @@ public:
if(oldInFit.betterFitness(newInFit)) return false;
if(oldTLogFit > newTLogFit || oldInFit > newInFit || oldSatelliteTLogFit > newSatelliteTLogFit || oldRemoteTLogFit > newRemoteTLogFit || oldLogRoutersFit > newLogRoutersFit) {
TraceEvent("BetterMasterExists", id).detail("oldMasterFit", oldMasterFit).detail("newMasterFit", mworker.fitness)
.detail("oldTLogFitC", oldTLogFit.count).detail("newTLogFitC", newTLogFit.count)
.detail("oldTLogWorstFitT", oldTLogFit.worstFit).detail("newTLogWorstFitT", newTLogFit.worstFit)
.detail("oldTLogBestFitT", oldTLogFit.bestFit).detail("newTLogBestFitT", newTLogFit.bestFit)
.detail("oldInFitW", oldInFit.worstFit).detail("newInFitW", newInFit.worstFit)
.detail("oldInFitB", oldInFit.bestFit).detail("newInFitB", newInFit.bestFit)
.detail("oldInFitC", oldInFit.count).detail("newInFitC", newInFit.count);
TraceEvent("BetterMasterExists", id).detail("OldMasterFit", oldMasterFit).detail("NewMasterFit", mworker.fitness)
.detail("OldTLogFitC", oldTLogFit.count).detail("NewTLogFitC", newTLogFit.count)
.detail("OldTLogWorstFitT", oldTLogFit.worstFit).detail("NewTLogWorstFitT", newTLogFit.worstFit)
.detail("OldTLogBestFitT", oldTLogFit.bestFit).detail("NewTLogBestFitT", newTLogFit.bestFit)
.detail("OldInFitW", oldInFit.worstFit).detail("NewInFitW", newInFit.worstFit)
.detail("OldInFitB", oldInFit.bestFit).detail("NewInFitB", newInFit.bestFit)
.detail("OldInFitC", oldInFit.count).detail("NewInFitC", newInFit.count);
return true;
}
@ -1022,7 +1022,7 @@ ACTOR Future<Void> clusterWatchDatabase( ClusterControllerData* cluster, Cluster
if (e.code() == error_code_actor_cancelled) throw;
bool ok = e.code() == error_code_no_more_servers;
TraceEvent(ok ? SevWarn : SevError,"clusterWatchDatabaseRetrying", cluster->id).error(e);
TraceEvent(ok ? SevWarn : SevError,"ClusterWatchDatabaseRetrying", cluster->id).error(e);
if (!ok)
throw e;
Void _ = wait( delay(SERVER_KNOBS->ATTEMPT_RECRUITMENT_DELAY) );
@ -1350,7 +1350,7 @@ ACTOR Future<Void> failureDetectionServer( UID uniqueID, FutureStream< FailureMo
pivotDelay = std::max(0.0, pivotDelay - FLOW_KNOBS->SERVER_REQUEST_INTERVAL);
TraceEvent("FailureDetectionPoll", uniqueID).detail("PivotDelay", pivotDelay).detail("Clients", currentStatus.size());
//TraceEvent("FailureDetectionAcceptableDelay").detail("ms", acceptableDelay*1000);
//TraceEvent("FailureDetectionAcceptableDelay").detail("Delay", acceptableDelay1000);
for(auto it = currentStatus.begin(); it != currentStatus.end(); ) {
double delay = t - it->second.lastRequestTime;
@ -1454,13 +1454,13 @@ ACTOR Future<Void> clusterRecruitRemoteFromConfiguration( ClusterControllerData*
void clusterRegisterMaster( ClusterControllerData* self, RegisterMasterRequest const& req ) {
req.reply.send( Void() );
TraceEvent("MasterRegistrationReceived", self->id).detail("dbName", printable(req.dbName)).detail("MasterId", req.id).detail("Master", req.mi.toString()).detail("Tlogs", describe(req.logSystemConfig.tLogs)).detail("Resolvers", req.resolvers.size())
TraceEvent("MasterRegistrationReceived", self->id).detail("DbName", printable(req.dbName)).detail("MasterId", req.id).detail("Master", req.mi.toString()).detail("Tlogs", describe(req.logSystemConfig.tLogs)).detail("Resolvers", req.resolvers.size())
.detail("RecoveryState", req.recoveryState).detail("RegistrationCount", req.registrationCount).detail("Proxies", req.proxies.size()).detail("RecoveryCount", req.recoveryCount);
//make sure the request comes from an active database
auto db = &self->db;
if ( db->serverInfo->get().master.id() != req.id || req.registrationCount <= db->masterRegistrationCount ) {
TraceEvent("MasterRegistrationNotFound", self->id).detail("dbName", printable(req.dbName)).detail("MasterId", req.id).detail("existingId", db->serverInfo->get().master.id()).detail("RegCount", req.registrationCount).detail("ExistingRegCount", db->masterRegistrationCount);
TraceEvent("MasterRegistrationNotFound", self->id).detail("DbName", printable(req.dbName)).detail("MasterId", req.id).detail("ExistingId", db->serverInfo->get().master.id()).detail("RegCount", req.registrationCount).detail("ExistingRegCount", db->masterRegistrationCount);
return;
}
@ -1537,7 +1537,7 @@ void registerWorker( RegisterWorkerRequest req, ClusterControllerData *self ) {
auto info = self->id_worker.find( w.locality.processId() );
ClusterControllerPriorityInfo newPriorityInfo = req.priorityInfo;
TraceEvent("ClusterControllerActualWorkers", self->id).detail("WorkerID",w.id()).detailext("ProcessID", w.locality.processId()).detailext("ZoneId", w.locality.zoneId()).detailext("DataHall", w.locality.dataHallId()).detail("pClass", req.processClass.toString()).detail("Workers", self->id_worker.size()).detail("Registered", (info == self->id_worker.end() ? "False" : "True")).backtrace();
TraceEvent("ClusterControllerActualWorkers", self->id).detail("WorkerId",w.id()).detailext("ProcessId", w.locality.processId()).detailext("ZoneId", w.locality.zoneId()).detailext("DataHall", w.locality.dataHallId()).detail("PClass", req.processClass.toString()).detail("Workers", self->id_worker.size()).detail("Registered", (info == self->id_worker.end() ? "False" : "True")).backtrace();
if ( w.address() == g_network->getLocalAddress() ) {
self->clusterControllerProcessId = w.locality.processId();

6
fdbserver/CoordinatedState.actor.cpp
View File

@ -114,9 +114,9 @@ struct CoordinatedStateImpl {
UniqueGeneration wgen = wait( self->replicatedWrite( self, GenerationRegWriteRequest( KeyValueRef(self->coordinators.clusterKey, v), self->gen ) ) );
self->stage = 6;
TraceEvent("CoordinatedStateSet").detail("gen", self->gen.generation).detail("wgen", wgen.generation)
.detail("genu", self->gen.uid).detail("wgenu", wgen.uid)
.detail("cgen", self->conflictGen);
TraceEvent("CoordinatedStateSet").detail("Gen", self->gen.generation).detail("Wgen", wgen.generation)
.detail("Genu", self->gen.uid).detail("Wgenu", wgen.uid)
.detail("Cgen", self->conflictGen);
if (wgen == self->gen)
return Void();

6
fdbserver/Coordination.actor.cpp
View File

@ -150,11 +150,11 @@ ACTOR Future<Void> localGenerationReg( GenerationRegInterface interf, OnDemandSt
store->set( KeyValueRef( wrq.kv.key, BinaryWriter::toValue(v, IncludeVersion()) ) );
Void _ = wait(store->commit());
TraceEvent("GenerationRegWrote").detail("From", wrq.reply.getEndpoint().address).detail("Key", printable(wrq.kv.key))
.detail("reqGen", wrq.gen.generation).detail("Returning", v.writeGen.generation);
.detail("ReqGen", wrq.gen.generation).detail("Returning", v.writeGen.generation);
wrq.reply.send( v.writeGen );
} else {
TraceEvent("GenerationRegWriteFail").detail("From", wrq.reply.getEndpoint().address).detail("Key", printable(wrq.kv.key))
.detail("reqGen", wrq.gen.generation).detail("readGen", v.readGen.generation).detail("writeGen", v.writeGen.generation);
.detail("ReqGen", wrq.gen.generation).detail("ReadGen", v.readGen.generation).detail("WriteGen", v.writeGen.generation);
wrq.reply.send( std::max( v.readGen, v.writeGen ) );
}
}
@ -416,7 +416,7 @@ ACTOR Future<Void> coordinationServer(std::string dataFolder) {
state GenerationRegInterface myInterface( g_network );
state OnDemandStore store( dataFolder, myID );
TraceEvent("CoordinationServer", myID).detail("myInterfaceAddr", myInterface.read.getEndpoint().address).detail("Folder", dataFolder);
TraceEvent("CoordinationServer", myID).detail("MyInterfaceAddr", myInterface.read.getEndpoint().address).detail("Folder", dataFolder);
try {
Void _ = wait( localGenerationReg(myInterface, &store) || leaderServer(myLeaderInterface, &store) || store.getError() );

42
fdbserver/DataDistribution.actor.cpp
View File

@ -323,8 +323,8 @@ ACTOR Future<Void> storageServerFailureTracker(
: waitFailureClient(server.waitFailure, SERVER_KNOBS->DATA_DISTRIBUTION_FAILURE_REACTION_TIME, 0, TaskDataDistribution) ) )
{
status->isFailed = !status->isFailed;
TraceEvent("StatusMapChange", masterId).detail("ServerID", server.id()).detail("Status", status->toString()).
detail("Available", IFailureMonitor::failureMonitor().getState(server.waitFailure.getEndpoint()).isAvailable());
TraceEvent("StatusMapChange", masterId).detail("ServerID", server.id()).detail("Status", status->toString())
.detail("Available", IFailureMonitor::failureMonitor().getState(server.waitFailure.getEndpoint()).isAvailable());
}
when ( Void _ = wait( status->isUnhealthy() ? waitForAllDataRemoved(cx, server.id(), addedVersion) : Never() ) ) { break; }
}
@ -384,7 +384,7 @@ ACTOR Future<Reference<InitialDataDistribution>> getInitialDataDistribution( Dat
Void _ = wait( tr.onError(e) );
ASSERT(!succeeded); //We shouldn't be retrying if we have already started modifying result in this loop
TraceEvent("getInitialTeamsRetry", masterId);
TraceEvent("GetInitialTeamsRetry", masterId);
}
}
@ -463,7 +463,7 @@ ACTOR Future<Reference<InitialDataDistribution>> getInitialDataDistribution( Dat
Void _ = wait( tr.onError(e) );
ASSERT(!succeeded); //We shouldn't be retrying if we have already started modifying result in this loop
TraceEvent("getInitialTeamsKeyServersRetry", masterId);
TraceEvent("GetInitialTeamsKeyServersRetry", masterId);
}
}
@ -962,7 +962,7 @@ struct DDTeamCollection {
}
if(totalServers.size() < configuration.storageTeamSize ) {
TraceEvent(SevWarn, "DataDistributionBuildTeams", masterId).detail("Reason","Not enough servers for a team").detail("Servers",totalServers.size()).detail("teamSize", configuration.storageTeamSize);
TraceEvent(SevWarn, "DataDistributionBuildTeams", masterId).detail("Reason","Not enough servers for a team").detail("Servers",totalServers.size()).detail("TeamSize", configuration.storageTeamSize);
return addedTeams;
}
@ -1170,7 +1170,7 @@ struct DDTeamCollection {
}
allServers.push_back( newServer.id() );
TraceEvent("AddedStorageServer", masterId).detail("ServerID", newServer.id()).detail("ProcessClass", processClass.toString()).detail("WaitFailureToken", newServer.waitFailure.getEndpoint().token).detail("address", newServer.waitFailure.getEndpoint().address);
TraceEvent("AddedStorageServer", masterId).detail("ServerID", newServer.id()).detail("ProcessClass", processClass.toString()).detail("WaitFailureToken", newServer.waitFailure.getEndpoint().token).detail("Address", newServer.waitFailure.getEndpoint().address);
auto &r = server_info[newServer.id()] = Reference<TCServerInfo>( new TCServerInfo( newServer, processClass ) );
r->tracker = storageServerTracker( this, cx, r.getPtr(), &server_status, lock, masterId, &server_info, serverChanges, serverFailures, errorOut, addedVersion );
restartTeamBuilder.trigger();
@ -1254,7 +1254,7 @@ ACTOR Future<Void> teamTracker( DDTeamCollection *self, Reference<IDataDistribut
try {
loop {
TraceEvent("TeamHealthChangeDetected", self->masterId).detail("isReady", self->initialFailureReactionDelay.isReady() );
TraceEvent("TeamHealthChangeDetected", self->masterId).detail("IsReady", self->initialFailureReactionDelay.isReady() );
// Check if the number of degraded machines has changed
state vector<Future<Void>> change;
auto servers = team->getServerIDs();
@ -1287,8 +1287,8 @@ ACTOR Future<Void> teamTracker( DDTeamCollection *self, Reference<IDataDistribut
if( serversLeft != lastServersLeft || anyUndesired != lastAnyUndesired || anyWrongConfiguration != lastWrongConfiguration || wrongSize || recheck ) {
TraceEvent("TeamHealthChanged", self->masterId)
.detail("Team", team->getDesc()).detail("serversLeft", serversLeft)
.detail("lastServersLeft", lastServersLeft).detail("ContainsUndesiredServer", anyUndesired)
.detail("Team", team->getDesc()).detail("ServersLeft", serversLeft)
.detail("LastServersLeft", lastServersLeft).detail("ContainsUndesiredServer", anyUndesired)
.detail("HealthyTeamsCount", self->healthyTeamCount).detail("IsWrongConfiguration", anyWrongConfiguration);
bool healthy = matchesPolicy && !anyUndesired && team->getServerIDs().size() == self->configuration.storageTeamSize && team->getServerIDs().size() == serversLeft;
@ -1394,7 +1394,7 @@ ACTOR Future<Void> teamTracker( DDTeamCollection *self, Reference<IDataDistribut
}
}
} else {
TraceEvent("TeamHealthNotReady", self->masterId).detail("healthyTeamCount", self->healthyTeamCount);
TraceEvent("TeamHealthNotReady", self->masterId).detail("HealthyTeamCount", self->healthyTeamCount);
}
}
@ -1771,7 +1771,7 @@ ACTOR Future<Void> initializeStorage( DDTeamCollection *self, RecruitStorageRepl
isr.interfaceId = interfaceId;
TraceEvent("DDRecruiting").detail("State", "Sending request to worker").detail("WorkerID", candidateWorker.worker.id())
.detail("WorkerLocality", candidateWorker.worker.locality.toString()).detail("interf", interfaceId).detail("addr", candidateWorker.worker.address());
.detail("WorkerLocality", candidateWorker.worker.locality.toString()).detail("Interf", interfaceId).detail("Addr", candidateWorker.worker.address());
self->recruitingIds.insert(interfaceId);
self->recruitingLocalities.insert(candidateWorker.worker.address());
@ -1782,7 +1782,7 @@ ACTOR Future<Void> initializeStorage( DDTeamCollection *self, RecruitStorageRepl
self->recruitingStream.set(self->recruitingStream.get()-1);
TraceEvent("DDRecruiting").detail("State", "Finished request").detail("WorkerID", candidateWorker.worker.id())
.detail("WorkerLocality", candidateWorker.worker.locality.toString()).detail("interf", interfaceId).detail("addr", candidateWorker.worker.address());
.detail("WorkerLocality", candidateWorker.worker.locality.toString()).detail("Interf", interfaceId).detail("Addr", candidateWorker.worker.address());
if( newServer.isError() ) {
TraceEvent(SevWarn, "DDRecruitmentError").error(newServer.getError());
@ -1874,17 +1874,17 @@ ACTOR Future<Void> updateReplicasKey(DDTeamCollection* self, Optional<Key> dcId)
Void _ = wait(self->initialFailureReactionDelay);
Void _ = wait(delay(FLOW_KNOBS->PREVENT_FAST_SPIN_DELAY, TaskLowPriority)); //After the team trackers wait on the initial failure reaction delay, they yield. We want to make sure every tracker has had the opportunity to send their relocations to the queue.
while(self->zeroHealthyTeams->get() || self->processingUnhealthy->get()) {
TraceEvent("DDUpdatingStalled", self->masterId).detail("dcId", printable(dcId)).detail("zeroHealthy", self->zeroHealthyTeams->get()).detail("processingUnhealthy", self->processingUnhealthy->get());
TraceEvent("DDUpdatingStalled", self->masterId).detail("DcId", printable(dcId)).detail("ZeroHealthy", self->zeroHealthyTeams->get()).detail("ProcessingUnhealthy", self->processingUnhealthy->get());
Void _ = wait(self->zeroHealthyTeams->onChange() || self->processingUnhealthy->onChange());
}
TraceEvent("DDUpdatingReplicas", self->masterId).detail("dcId", printable(dcId)).detail("replicas", self->configuration.storageTeamSize);
TraceEvent("DDUpdatingReplicas", self->masterId).detail("DcId", printable(dcId)).detail("Replicas", self->configuration.storageTeamSize);
state Transaction tr(self->cx);
loop {
try {
tr.addReadConflictRange(singleKeyRange(datacenterReplicasKeyFor(dcId)));
tr.set(datacenterReplicasKeyFor(dcId), datacenterReplicasValue(self->configuration.storageTeamSize));
Void _ = wait( tr.commit() );
TraceEvent("DDUpdatedReplicas", self->masterId).detail("dcId", printable(dcId)).detail("replicas", self->configuration.storageTeamSize);
TraceEvent("DDUpdatedReplicas", self->masterId).detail("DcId", printable(dcId)).detail("Replicas", self->configuration.storageTeamSize);
return Void();
} catch( Error &e ) {
Void _ = wait( tr.onError(e) );
@ -1916,9 +1916,9 @@ ACTOR Future<Void> dataDistributionTeamCollection(
state PromiseStream<Void> serverRemoved;
state Future<Void> error = actorCollection( self.addActor.getFuture() );
TraceEvent("DDTeamCollectionBegin", masterId).detail("primary", primary);
TraceEvent("DDTeamCollectionBegin", masterId).detail("Primary", primary);
Void _ = wait( readyToStart );
TraceEvent("DDTeamCollectionReadyToStart", masterId).detail("primary", primary);
TraceEvent("DDTeamCollectionReadyToStart", masterId).detail("Primary", primary);
try {
self.init( *initData );
initData = Reference<InitialDataDistribution>();
@ -1964,7 +1964,7 @@ ACTOR Future<Void> dataDistributionTeamCollection(
}
} catch (Error& e) {
if (e.code() != error_code_movekeys_conflict)
TraceEvent(SevError, "dataDistributionTeamCollectionError", masterId).error(e);
TraceEvent(SevError, "DataDistributionTeamCollectionError", masterId).error(e);
throw e;
}
}
@ -2124,9 +2124,9 @@ ACTOR Future<Void> dataDistribution(
TraceEvent("DDInitTookMoveKeysLock", mi.id());
state Reference<InitialDataDistribution> initData = wait( getInitialDataDistribution(cx, mi.id(), lock, configuration.remoteTLogReplicationFactor > 0 ? remoteDcIds : std::vector<Optional<Key>>() ) );
if(initData->shards.size() > 1) {
TraceEvent("DDInitGotInitialDD", mi.id()).detail("b", printable(initData->shards.end()[-2].key)).detail("e", printable(initData->shards.end()[-1].key)).detail("src", describe(initData->shards.end()[-2].primarySrc)).detail("dest", describe(initData->shards.end()[-2].primaryDest)).trackLatest("InitialDD");
TraceEvent("DDInitGotInitialDD", mi.id()).detail("B", printable(initData->shards.end()[-2].key)).detail("E", printable(initData->shards.end()[-1].key)).detail("Src", describe(initData->shards.end()[-2].primarySrc)).detail("Dest", describe(initData->shards.end()[-2].primaryDest)).trackLatest("InitialDD");
} else {
TraceEvent("DDInitGotInitialDD", mi.id()).detail("b","").detail("e", "").detail("src", "[no items]").detail("dest", "[no items]").trackLatest("InitialDD");
TraceEvent("DDInitGotInitialDD", mi.id()).detail("B","").detail("E", "").detail("Src", "[no items]").detail("Dest", "[no items]").trackLatest("InitialDD");
}
if (initData->mode) break;
@ -2210,7 +2210,7 @@ ACTOR Future<Void> dataDistribution(
if( e.code() != error_code_movekeys_conflict )
throw err;
bool ddEnabled = wait( isDataDistributionEnabled(cx) );
TraceEvent("DataDistributionMoveKeysConflict").detail("ddEnabled", ddEnabled);
TraceEvent("DataDistributionMoveKeysConflict").detail("DataDistributionEnabled", ddEnabled);
if( ddEnabled )
throw err;
}

22
fdbserver/DataDistributionQueue.actor.cpp
View File

@ -486,10 +486,10 @@ struct DDQueueData {
for( int i = 0; i < it->second.ledger.size() - 1; i++ ) {
if( it->second.ledger[i] < it->second.ledger[i+1] )
TraceEvent(SevError, "DDQueueValidateError12").detail("Problem", "ascending ledger problem")
.detail("ledgerLevel", i).detail("ledgerValueA", it->second.ledger[i]).detail("ledgerValueB", it->second.ledger[i+1]);
.detail("LedgerLevel", i).detail("LedgerValueA", it->second.ledger[i]).detail("LedgerValueB", it->second.ledger[i+1]);
if( it->second.ledger[i] < 0.0 )
TraceEvent(SevError, "DDQueueValidateError13").detail("Problem", "negative ascending problem")
.detail("ledgerLevel", i).detail("ledgerValue", it->second.ledger[i]);
.detail("LedgerLevel", i).detail("LedgerValue", it->second.ledger[i]);
}
}
@ -833,7 +833,7 @@ struct DDQueueData {
//logRelocation( rd, "LaunchedRelocation" );
}
if( now() - startTime > .001 && g_random->random01()<0.001 )
TraceEvent(SevWarnAlways, "LaunchingQueueSlowx1000").detail("elapsed", now() - startTime );
TraceEvent(SevWarnAlways, "LaunchingQueueSlowx1000").detail("Elapsed", now() - startTime );
/*if( startedHere > 0 ) {
TraceEvent("StartedDDRelocators", mi.id())
@ -984,7 +984,7 @@ ACTOR Future<Void> dataDistributionRelocator( DDQueueData *self, RelocateData rd
error = e;
}
//TraceEvent("RelocateShardFinished", masterId).detail("relocateId", relocateShardInterval.pairID);
//TraceEvent("RelocateShardFinished", masterId).detail("RelocateId", relocateShardInterval.pairID);
if( error.code() != error_code_move_to_removed_server ) {
if( !error.code() ) {
@ -1001,7 +1001,7 @@ ACTOR Future<Void> dataDistributionRelocator( DDQueueData *self, RelocateData rd
if( !error.code() ) {
TraceEvent(relocateShardInterval.end(), masterId).detail("Result","Success");
if(rd.keys.begin == keyServersPrefix) {
TraceEvent("MovedKeyServerKeys").detail("dest", destination.getDesc()).trackLatest("MovedKeyServers");
TraceEvent("MovedKeyServerKeys").detail("Dest", destination.getDesc()).trackLatest("MovedKeyServers");
}
if( !signalledTransferComplete ) {
@ -1057,11 +1057,11 @@ ACTOR Future<bool> rebalanceTeams( DDQueueData* self, int priority, Reference<ID
for( int i = 0; i < shards.size(); i++ ) {
if( moveShard == shards[i] ) {
TraceEvent(priority == PRIORITY_REBALANCE_OVERUTILIZED_TEAM ? "BgDDMountainChopper" : "BgDDValleyFiller", self->mi.id())
.detail("sourceBytes", sourceBytes)
.detail("destBytes", destBytes)
.detail("shardBytes", metrics.bytes)
.detail("sourceTeam", sourceTeam->getDesc())
.detail("destTeam", destTeam->getDesc());
.detail("SourceBytes", sourceBytes)
.detail("DestBytes", destBytes)
.detail("ShardBytes", metrics.bytes)
.detail("SourceTeam", sourceTeam->getDesc())
.detail("DestTeam", destTeam->getDesc());
self->input.send( RelocateShard( moveShard, priority ) );
return true;
@ -1258,7 +1258,7 @@ ACTOR Future<Void> dataDistributionQueue(
} catch (Error& e) {
if (e.code() != error_code_broken_promise && // FIXME: Get rid of these broken_promise errors every time we are killed by the master dying
e.code() != error_code_movekeys_conflict)
TraceEvent(SevError, "dataDistributionQueueError", mi.id()).error(e);
TraceEvent(SevError, "DataDistributionQueueError", mi.id()).error(e);
throw e;
}
}

4
fdbserver/DataDistributionTracker.actor.cpp
View File

@ -344,7 +344,7 @@ ACTOR Future<Void> shardSplitter(
.detail("MetricsBytes", metrics.bytes)
.detail("Bandwidth", bandwidthStatus == BandwidthStatusHigh ? "High" : bandwidthStatus == BandwidthStatusNormal ? "Normal" : "Low")
.detail("BytesPerKSec", metrics.bytesPerKSecond)
.detail("numShards", numShards);
.detail("NumShards", numShards);
}
if( numShards > 1 ) {
@ -695,7 +695,7 @@ ACTOR Future<Void> dataDistributionTracker(
when( Void _ = wait( self.sizeChanges.getResult() ) ) {}
}
} catch (Error& e) {
TraceEvent(SevError, "dataDistributionTrackerError", self.masterId).error(e);
TraceEvent(SevError, "DataDistributionTrackerError", self.masterId).error(e);
throw e;
}
}

70
fdbserver/DiskQueue.actor.cpp
View File

@ -136,7 +136,7 @@ public:
Future<Standalone<StringRef>> readFirstAndLastPages( compare_pages compare ) { return readFirstAndLastPages(this,compare); }
void setStartPage( int file, int64_t page ) {
TraceEvent("RDQSetStart", dbgid).detail("f",file).detail("p",page).detail("file0name", files[0].dbgFilename);
TraceEvent("RDQSetStart", dbgid).detail("FileNum",file).detail("PageNum",page).detail("File0Name", files[0].dbgFilename);
readingFile = file;
readingPage = page;
}
@ -235,7 +235,7 @@ public:
if(p > 0) {
toSync.push_back( files[1].syncQueue );
/*TraceEvent("RDQWriteAndSwap", this->dbgid).detail("File1name", files[1].dbgFilename).detail("File1size", files[1].size)
.detail("writingPos", writingPos).detail("writingBytes", p);*/
.detail("WritingPos", writingPos).detail("WritingBytes", p);*/
waitfor.push_back( files[1].f->write( pageData.begin(), p, writingPos ) );
pageData = pageData.substr( p );
}
@ -247,19 +247,19 @@ public:
} else {
// Extend files[1] to accomodate the new write and about 10MB or 2x current size for future writes.
/*TraceEvent("RDQExtend", this->dbgid).detail("File1name", files[1].dbgFilename).detail("File1size", files[1].size)
.detail("extensionBytes", fileExtensionBytes);*/
.detail("ExtensionBytes", fileExtensionBytes);*/
int64_t minExtension = pageData.size() + writingPos - files[1].size;
files[1].size += std::min(std::max(fileExtensionBytes, minExtension), files[0].size+files[1].size+minExtension);
waitfor.push_back( files[1].f->truncate( files[1].size ) );
if(fileSizeWarningLimit > 0 && files[1].size > fileSizeWarningLimit) {
TraceEvent(SevWarnAlways, "DiskQueueFileTooLarge", dbgid).detail("filename", filename(1)).detail("size", files[1].size).suppressFor(1.0);
TraceEvent(SevWarnAlways, "DiskQueueFileTooLarge", dbgid).detail("Filename", filename(1)).detail("Size", files[1].size).suppressFor(1.0);
}
}
}
/*TraceEvent("RDQWrite", this->dbgid).detail("File1name", files[1].dbgFilename).detail("File1size", files[1].size)
.detail("writingPos", writingPos).detail("writingBytes", pageData.size());*/
.detail("WritingPos", writingPos).detail("WritingBytes", pageData.size());*/
files[1].size = std::max( files[1].size, writingPos + pageData.size() );
toSync.push_back( files[1].syncQueue );
waitfor.push_back( files[1].f->write( pageData.begin(), pageData.size(), writingPos ) );
@ -322,7 +322,7 @@ public:
delete pageMem;
TEST(true); // push error
TEST(2==syncFiles.size()); // push spanning both files error
TraceEvent(SevError, "RDQ_pushAndCommit_Error", dbgid).detail("InitialFilename0", filename).error(e, true);
TraceEvent(SevError, "RDQPushAndCommitError", dbgid).detail("InitialFilename0", filename).error(e, true);
if (errorPromise.canBeSet()) errorPromise.sendError(e);
if (pushing.canBeSet()) pushing.sendError(e);
@ -512,7 +512,7 @@ public:
bool middleValid = compare( &firstPage[1], middlePage );
TraceEvent("RDQBS", self->dbgid).detail("b", begin).detail("e", end).detail("m", middle).detail("v", middleValid).detail("file0name", self->files[0].dbgFilename);
TraceEvent("RDQBS", self->dbgid).detail("Begin", begin).detail("End", end).detail("Middle", middle).detail("Valid", middleValid).detail("File0Name", self->files[0].dbgFilename);
if (middleValid)
begin = middle;
@ -533,7 +533,7 @@ public:
return result.str;
} catch (Error& e) {
bool ok = e.code() == error_code_file_not_found;
TraceEvent(ok ? SevInfo : SevError, "RDQ_rfl_Error", self->dbgid).detail("file0name", self->files[0].dbgFilename).error(e, true);
TraceEvent(ok ? SevInfo : SevError, "RDQReadFirstAndLastPagesError", self->dbgid).detail("File0Name", self->files[0].dbgFilename).error(e, true);
if (!self->error.isSet()) self->error.sendError(e);
throw;
}
@ -588,7 +588,7 @@ public:
return result;
} catch (Error& e) {
TEST(true); // Read next page error
TraceEvent(SevError, "RDQ_rnp_Error", self->dbgid).detail("file0name", self->files[0].dbgFilename).error(e, true);
TraceEvent(SevError, "RDQReadNextPageError", self->dbgid).detail("File0Name", self->files[0].dbgFilename).error(e, true);
if (!self->error.isSet()) self->error.sendError(e);
throw;
}
@ -633,7 +633,7 @@ public:
return Void();
} catch (Error& e) {
TraceEvent(SevError, "RDQ_tblrp_Error", self->dbgid).detail("file0name", self->files[0].dbgFilename).error(e);
TraceEvent(SevError, "RDQTruncateBeforeLastReadPageError", self->dbgid).detail("File0Name", self->files[0].dbgFilename).error(e);
if (!self->error.isSet()) self->error.sendError(e);
throw;
}
@ -676,7 +676,7 @@ public:
TraceEvent(SevError, "DQPopUncommittedData", dbgid)
.detail("UpTo", upTo)
.detail("LastCommittedSeq", lastCommittedSeq)
.detail("file0name", rawQueue->files[0].dbgFilename);
.detail("File0Name", rawQueue->files[0].dbgFilename);
}
if (upTo.lo > poppedSeq) {
poppedSeq = upTo.lo;
@ -712,17 +712,17 @@ public:
if( pushedPageCount() >= 8000 ) {
TraceEvent( warnAlwaysForMemory ? SevWarnAlways : SevWarn, "DiskQueueMemoryWarning", dbgid)
.detail("pushed_pages", pushedPageCount())
.detail("nextPageSeq", nextPageSeq)
.detail("PushedPages", pushedPageCount())
.detail("NextPageSeq", nextPageSeq)
.detail("Details", format("%d pages", pushedPageCount()))
.detail("file0name", rawQueue->files[0].dbgFilename)
.detail("File0Name", rawQueue->files[0].dbgFilename)
.suppressFor(1.0);
if(g_network->isSimulated())
warnAlwaysForMemory = false;
}
/*TraceEvent("DQCommit", dbgid).detail("Pages", pushedPageCount()).detail("lastPoppedSeq", lastPoppedSeq).detail("poppedSeq", poppedSeq).detail("nextPageSeq", nextPageSeq)
.detail("RawFile0Size", rawQueue->files[0].size).detail("RawFile1Size", rawQueue->files[1].size).detail("writingPos", rawQueue->writingPos)
/*TraceEvent("DQCommit", dbgid).detail("Pages", pushedPageCount()).detail("LastPoppedSeq", lastPoppedSeq).detail("PoppedSeq", poppedSeq).detail("NextPageSeq", nextPageSeq)
.detail("RawFile0Size", rawQueue->files[0].size).detail("RawFile1Size", rawQueue->files[1].size).detail("WritingPos", rawQueue->writingPos)
.detail("RawFile0Name", rawQueue->files[0].dbgFilename);*/
lastCommittedSeq = backPage().endSeq();
@ -742,17 +742,17 @@ public:
virtual Future<Void> getError() { return rawQueue->getError(); }
virtual Future<Void> onClosed() { return rawQueue->onClosed(); }
virtual void dispose() {
TraceEvent("DQDestroy", dbgid).detail("lastPoppedSeq", lastPoppedSeq).detail("poppedSeq", poppedSeq).detail("nextPageSeq", nextPageSeq).detail("file0name", rawQueue->files[0].dbgFilename);
TraceEvent("DQDestroy", dbgid).detail("LastPoppedSeq", lastPoppedSeq).detail("PoppedSeq", poppedSeq).detail("NextPageSeq", nextPageSeq).detail("File0Name", rawQueue->files[0].dbgFilename);
rawQueue->dispose();
delete this;
}
virtual void close() {
TraceEvent("DQClose", dbgid)
.detail("lastPoppedSeq", lastPoppedSeq)
.detail("poppedSeq", poppedSeq)
.detail("nextPageSeq", nextPageSeq)
.detail("poppedCommitted", rawQueue->dbg_file0BeginSeq + rawQueue->files[0].popped + rawQueue->files[1].popped)
.detail("file0name", rawQueue->files[0].dbgFilename);
.detail("LastPoppedSeq", lastPoppedSeq)
.detail("PoppedSeq", poppedSeq)
.detail("NextPageSeq", nextPageSeq)
.detail("PoppedCommitted", rawQueue->dbg_file0BeginSeq + rawQueue->files[0].popped + rawQueue->files[1].popped)
.detail("File0Name", rawQueue->files[0].dbgFilename);
rawQueue->close();
delete this;
}
@ -818,9 +818,9 @@ private:
if (pushedPageCount() == 8000) {
TraceEvent("DiskQueueHighPageCount", dbgid)
.detail("pushed_pages", pushedPageCount())
.detail("nextPageSeq", nextPageSeq)
.detail("file0name", rawQueue->files[0].dbgFilename);
.detail("PushedPages", pushedPageCount())
.detail("NextPageSeq", nextPageSeq)
.detail("File0Name", rawQueue->files[0].dbgFilename);
}
}
@ -857,7 +857,7 @@ private:
}
self->readBufPos = self->nextReadLocation % sizeof(Page) - sizeof(PageHeader);
if (self->readBufPos < 0) { self->nextReadLocation -= self->readBufPos; self->readBufPos = 0; }
TraceEvent("DQRecStart", self->dbgid).detail("readBufPos", self->readBufPos).detail("nextReadLoc", self->nextReadLocation).detail("file0name", self->rawQueue->files[0].dbgFilename);
TraceEvent("DQRecStart", self->dbgid).detail("ReadBufPos", self->readBufPos).detail("NextReadLoc", self->nextReadLocation).detail("File0Name", self->rawQueue->files[0].dbgFilename);
}
loop {
@ -873,7 +873,7 @@ private:
Standalone<StringRef> page = wait( self->rawQueue->readNextPage() );
if (!page.size()) {
TraceEvent("DQRecEOF", self->dbgid).detail("nextReadLocation", self->nextReadLocation).detail("file0name", self->rawQueue->files[0].dbgFilename);
TraceEvent("DQRecEOF", self->dbgid).detail("NextReadLocation", self->nextReadLocation).detail("File0Name", self->rawQueue->files[0].dbgFilename);
break;
}
ASSERT( page.size() == sizeof(Page) );
@ -881,12 +881,12 @@ private:
self->readBufArena = page.arena();
self->readBufPage = (Page*)page.begin();
if (!self->readBufPage->checkHash() || self->readBufPage->seq < self->nextReadLocation/sizeof(Page)*sizeof(Page)) {
TraceEvent("DQRecInvalidPage", self->dbgid).detail("nextReadLocation", self->nextReadLocation).detail("hashCheck", self->readBufPage->checkHash())
.detail("seq", self->readBufPage->seq).detail("expect", self->nextReadLocation/sizeof(Page)*sizeof(Page)).detail("file0name", self->rawQueue->files[0].dbgFilename);
TraceEvent("DQRecInvalidPage", self->dbgid).detail("NextReadLocation", self->nextReadLocation).detail("HashCheck", self->readBufPage->checkHash())
.detail("Seq", self->readBufPage->seq).detail("Expect", self->nextReadLocation/sizeof(Page)*sizeof(Page)).detail("File0Name", self->rawQueue->files[0].dbgFilename);
Void _ = wait( self->rawQueue->truncateBeforeLastReadPage() );
break;
}
//TraceEvent("DQRecPage", self->dbgid).detail("nextReadLoc", self->nextReadLocation).detail("Seq", self->readBufPage->seq).detail("Pop", self->readBufPage->popped).detail("Payload", self->readBufPage->payloadSize).detail("file0name", self->rawQueue->files[0].dbgFilename);
//TraceEvent("DQRecPage", self->dbgid).detail("NextReadLoc", self->nextReadLocation).detail("Seq", self->readBufPage->seq).detail("Pop", self->readBufPage->popped).detail("Payload", self->readBufPage->payloadSize).detail("File0Name", self->rawQueue->files[0].dbgFilename);
ASSERT( self->readBufPage->seq == self->nextReadLocation/sizeof(Page)*sizeof(Page) );
self->lastPoppedSeq = self->readBufPage->popped;
}
@ -902,7 +902,7 @@ private:
self->nextPageSeq = self->nextReadLocation/sizeof(Page)*sizeof(Page);
if (self->nextReadLocation % sizeof(Page) > 36) self->nextPageSeq += sizeof(Page);
TraceEvent("DQRecovered", self->dbgid).detail("lastPoppedSeq", self->lastPoppedSeq).detail("poppedSeq", self->poppedSeq).detail("nextPageSeq", self->nextPageSeq).detail("file0name", self->rawQueue->files[0].dbgFilename);
TraceEvent("DQRecovered", self->dbgid).detail("LastPoppedSeq", self->lastPoppedSeq).detail("PoppedSeq", self->poppedSeq).detail("NextPageSeq", self->nextPageSeq).detail("File0Name", self->rawQueue->files[0].dbgFilename);
self->recovered = true;
ASSERT( self->poppedSeq <= self->endLocation() );
self->recoveryFirstPages = Standalone<StringRef>();
@ -938,7 +938,7 @@ private:
state int sizeNum;
for( sizeNum=0; sizeNum < self->rawQueue->files[fileNum].size; sizeNum += sizeof(Page) ) {
int _ = wait( self->rawQueue->files[fileNum].f->read( testPage.get(), sizeof(Page), sizeNum ) );
TraceEvent("PageData").detail("file", self->rawQueue->files[fileNum].dbgFilename).detail("sizeNum", sizeNum).detail("seq", testPage->seq).detail("hash", testPage->checkHash()).detail("popped", testPage->popped);
TraceEvent("PageData").detail("File", self->rawQueue->files[fileNum].dbgFilename).detail("SizeNum", sizeNum).detail("Seq", testPage->seq).detail("Hash", testPage->checkHash()).detail("Popped", testPage->popped);
}
}
*/
@ -962,7 +962,7 @@ private:
.detail("Page1Seq", p[1].seq)
.detail("Location", loc)
.detail("Context", context)
.detail("file0name", rawQueue->files[0].dbgFilename);
.detail("File0Name", rawQueue->files[0].dbgFilename);
for(int i=recoveryFirstPages.size() / sizeof(Page) - 2; i>=0; i--)
if ( p[i].checkHash() && p[i].seq <= (size_t)loc ) {
@ -976,7 +976,7 @@ private:
.detail("PageSequence", p[i].seq)
.detail("Location", loc)
.detail("Context", context)
.detail("file0name", rawQueue->files[0].dbgFilename);
.detail("File0Name", rawQueue->files[0].dbgFilename);
ok = true;
break;
}
@ -989,7 +989,7 @@ private:
.detail("Page1Seq", p[1].seq)
.detail("Location", loc)
.detail("Context", context)
.detail("file0name", rawQueue->files[0].dbgFilename);
.detail("File0Name", rawQueue->files[0].dbgFilename);
ASSERT( ok );
}

28
fdbserver/KeyValueStoreMemory.actor.cpp
View File

@ -408,9 +408,9 @@ private:
bool ok = count < 1e6;
if( !ok ) {
TraceEvent(/*ok ? SevInfo : */SevWarnAlways, "KVSMemCommit_queue", id)
.detail("bytes", total)
.detail("log", log)
.detail("ops", count)
.detail("Bytes", total)
.detail("Log", log)
.detail("Ops", count)
.detail("LastLoggedLocation", log_location)
.detail("Details", count);
}
@ -490,9 +490,9 @@ private:
if (h.op == OpSnapshotItem) { // snapshot data item
/*if (p1 < uncommittedNextKey) {
TraceEvent(SevError, "RecSnapshotBack", self->id)
.detail("nextKey", printable(uncommittedNextKey))
.detail("p1", printable(p1))
.detail("nextlocation", self->log->getNextReadLocation());
.detail("NextKey", printable(uncommittedNextKey))
.detail("P1", printable(p1))
.detail("Nextlocation", self->log->getNextReadLocation());
}
ASSERT( p1 >= uncommittedNextKey );*/
if( p1 >= uncommittedNextKey )
@ -502,9 +502,9 @@ private:
++dbgSnapshotItemCount;
} else if (h.op == OpSnapshotEnd || h.op == OpSnapshotAbort) { // snapshot complete
TraceEvent("RecSnapshotEnd", self->id)
.detail("nextKey", printable(uncommittedNextKey))
.detail("nextlocation", self->log->getNextReadLocation())
.detail("isSnapshotEnd", h.op == OpSnapshotEnd);
.detail("NextKey", printable(uncommittedNextKey))
.detail("Nextlocation", self->log->getNextReadLocation())
.detail("IsSnapshotEnd", h.op == OpSnapshotEnd);
if(h.op == OpSnapshotEnd) {
uncommittedPrevSnapshotEnd = uncommittedSnapshotEnd;
@ -531,7 +531,7 @@ private:
} else if (h.op == OpRollback) { // rollback previous transaction
recoveryQueue.rollback();
TraceEvent("KVSMemRecSnapshotRollback", self->id)
.detail("nextKey", printable(uncommittedNextKey));
.detail("NextKey", printable(uncommittedNextKey));
uncommittedNextKey = self->recoveredSnapshotKey;
uncommittedPrevSnapshotEnd = self->previousSnapshotEnd;
uncommittedSnapshotEnd = self->currentSnapshotEnd;
@ -646,10 +646,10 @@ private:
if (next == self->data.end()) {
auto thisSnapshotEnd = self->log_op( OpSnapshotEnd, StringRef(), StringRef() );
//TraceEvent("SnapshotEnd", self->id)
// .detail("lastKey", printable(lastKey.present() ? lastKey.get() : LiteralStringRef("<none>")))
// .detail("currentSnapshotEndLoc", self->currentSnapshotEnd)
// .detail("previousSnapshotEndLoc", self->previousSnapshotEnd)
// .detail("thisSnapshotEnd", thisSnapshotEnd)
// .detail("LastKey", printable(lastKey.present() ? lastKey.get() : LiteralStringRef("<none>")))
// .detail("CurrentSnapshotEndLoc", self->currentSnapshotEnd)
// .detail("PreviousSnapshotEndLoc", self->previousSnapshotEnd)
// .detail("ThisSnapshotEnd", thisSnapshotEnd)
// .detail("Items", snapItems)
// .detail("CommittedWrites", self->notifiedCommittedWriteBytes.get())
// .detail("SnapshotSize", snapshotBytes);

22
fdbserver/KeyValueStoreSQLite.actor.cpp
View File

@ -269,7 +269,7 @@ struct SQLiteDB : NonCopyable {
TraceEvent("BTreeIntegrityCheckBegin").detail("Filename", filename);
char* e = sqlite3BtreeIntegrityCheck(btree, tables, 3, 1000, &errors, verbose);
if (!(g_network->isSimulated() && (g_simulator.getCurrentProcess()->fault_injection_p1 || g_simulator.getCurrentProcess()->rebooting))) {
TraceEvent((errors||e) ? SevError : SevInfo, "BTreeIntegrityCheck").detail("filename", filename).detail("ErrorTotal", errors);
TraceEvent((errors||e) ? SevError : SevInfo, "BTreeIntegrityCheck").detail("Filename", filename).detail("ErrorTotal", errors);
if(e != nullptr) {
// e is a string containing 1 or more lines. Create a separate trace event for each line.
char *lineStart = e;
@ -282,7 +282,7 @@ struct SQLiteDB : NonCopyable {
// If the line length found is not zero then print a trace event
if(*lineStart != '\0')
TraceEvent(SevError, "BTreeIntegrityCheck").detail("filename", filename).detail("ErrorDetail", lineStart);
TraceEvent(SevError, "BTreeIntegrityCheck").detail("Filename", filename).detail("ErrorDetail", lineStart);
lineStart = lineEnd;
}
}
@ -1262,7 +1262,7 @@ int SQLiteDB::checkAllPageChecksums() {
TraceEvent("SQLitePageChecksumScanEnd")
.detail("Elapsed", DEBUG_DETERMINISM ? 0 : timer()-startT)
.detail("filename", filename)
.detail("Filename", filename)
.detail("CorruptPages", corruptPages)
.detail("ReadErrors", readErrors)
.detail("TotalErrors", totalErrors);
@ -1275,7 +1275,7 @@ int SQLiteDB::checkAllPageChecksums() {
void SQLiteDB::open(bool writable) {
ASSERT( !haveMutex );
double startT = timer();
//TraceEvent("KVThreadInitStage").detail("Stage",1).detail("filename", filename).detail("writable", writable);
//TraceEvent("KVThreadInitStage").detail("Stage",1).detail("Filename", filename).detail("Writable", writable);
// First try to open an existing file
std::string apath = abspath(filename);
@ -1283,7 +1283,7 @@ void SQLiteDB::open(bool writable) {
ErrorOr<Reference<IAsyncFile>> dbFile = waitForAndGet( errorOr( IAsyncFileSystem::filesystem()->open( apath, IAsyncFile::OPEN_READWRITE | IAsyncFile::OPEN_LOCK, 0 ) ) );
ErrorOr<Reference<IAsyncFile>> walFile = waitForAndGet( errorOr( IAsyncFileSystem::filesystem()->open( walpath, IAsyncFile::OPEN_READWRITE | IAsyncFile::OPEN_LOCK, 0 ) ) );
//TraceEvent("KVThreadInitStage").detail("Stage",15).detail("filename", apath).detail("writable", writable).detail("isErr", dbFile.isError());
//TraceEvent("KVThreadInitStage").detail("Stage",15).detail("Filename", apath).detail("Writable", writable).detail("IsErr", dbFile.isError());
if (writable) {
if (dbFile.isError() && dbFile.getError().code() == error_code_file_not_found && !fileExists(apath) && // db file is missing
@ -1318,7 +1318,7 @@ void SQLiteDB::open(bool writable) {
if (dbFile.isError()) throw dbFile.getError(); // If we've failed to open the file, throw an exception
if (walFile.isError()) throw walFile.getError(); // If we've failed to open the file, throw an exception
//TraceEvent("KVThreadInitStage").detail("Stage",2).detail("filename", filename).detail("writable", writable);
//TraceEvent("KVThreadInitStage").detail("Stage",2).detail("Filename", filename).detail("Writable", writable);
// Now that the file itself is open and locked, let sqlite open the database
// Note that VFSAsync will also call g_network->open (including for the WAL), so its flags are important, too
@ -1330,7 +1330,7 @@ void SQLiteDB::open(bool writable) {
sqlite3_extended_result_codes(db, 1);
//TraceEvent("KVThreadInitStage").detail("Stage",3).detail("filename", filename).detail("writable", writable);
//TraceEvent("KVThreadInitStage").detail("Stage",3).detail("Filename", filename).detail("Writable", writable);
//Statement(*this, "PRAGMA cache_size = 100").execute();
@ -1346,7 +1346,7 @@ void SQLiteDB::open(bool writable) {
Statement(*this, "PRAGMA wal_autocheckpoint = -1").nextRow();
}
//TraceEvent("KVThreadInitStage").detail("Stage",4).detail("filename", filename).detail("writable", writable);
//TraceEvent("KVThreadInitStage").detail("Stage",4).detail("Filename", filename).detail("Writable", writable);
sqlite3_mutex_enter(db->mutex);
haveMutex = true;
@ -1356,7 +1356,7 @@ void SQLiteDB::open(bool writable) {
this->dbFile = dbFile.get();
this->walFile = walFile.get();
TraceEvent("KVThreadInitTime").detail("Elapsed", DEBUG_DETERMINISM ? 0 : timer()-startT).detail("filename", filename).detail("writable", writable);
TraceEvent("KVThreadInitTime").detail("Elapsed", DEBUG_DETERMINISM ? 0 : timer()-startT).detail("Filename", filename).detail("Writable", writable);
ASSERT(vfsAsyncIsOpen(filename));
}
@ -1392,7 +1392,7 @@ void SQLiteDB::createFromScratch() {
ASSERT( freetable == table+1 );
endTransaction();
} else {
TraceEvent("pgnoRoot").detail("value", pgnoRoot);
TraceEvent("pgnoRoot").detail("Value", pgnoRoot);
checkError("CheckTables", SQLITE_CORRUPT);
}
}
@ -1662,7 +1662,7 @@ private:
checkFreePages();
++writesComplete;
if (t3-a.issuedTime > 10.0*g_random->random01())
TraceEvent("KVCommit10s_sample", dbgid).detail("Queued", t1-a.issuedTime).detail("Commit", t2-t1).detail("Checkpoint", t3-t2);
TraceEvent("KVCommit10sSample", dbgid).detail("Queued", t1-a.issuedTime).detail("Commit", t2-t1).detail("Checkpoint", t3-t2);
diskBytesUsed = waitForAndGet( conn.dbFile->size() ) + waitForAndGet( conn.walFile->size() );

4
fdbserver/Knobs.cpp
View File

@ -227,6 +227,7 @@ ServerKnobs::ServerKnobs(bool randomize, ClientKnobs* clientKnobs) {
init( COMMIT_TRANSACTION_BATCH_INTERVAL_LATENCY_FRACTION, 0.1 );
init( COMMIT_TRANSACTION_BATCH_INTERVAL_SMOOTHER_ALPHA, 0.1 );
init( COMMIT_TRANSACTION_BATCH_COUNT_MAX, 32768 ); if( randomize && BUGGIFY ) COMMIT_TRANSACTION_BATCH_COUNT_MAX = 1000; // Do NOT increase this number beyond 32768, as CommitIds only budget 2 bytes for storing transaction id within each batch
init( COMMIT_BATCHES_MEM_BYTES_HARD_LIMIT, 8LL << 30 ); if (randomize && BUGGIFY) COMMIT_BATCHES_MEM_BYTES_HARD_LIMIT = g_random->randomInt64(100LL << 20, 8LL << 30);
// these settings disable batch bytes scaling. Try COMMIT_TRANSACTION_BATCH_BYTES_MAX=1e6, COMMIT_TRANSACTION_BATCH_BYTES_SCALE_BASE=50000, COMMIT_TRANSACTION_BATCH_BYTES_SCALE_POWER=0.5?
init( COMMIT_TRANSACTION_BATCH_BYTES_MIN, 100000 );
@ -238,6 +239,8 @@ ServerKnobs::ServerKnobs(bool randomize, ClientKnobs* clientKnobs) {
init( RESOLVER_COALESCE_TIME, 1.0 );
init( BUGGIFIED_ROW_LIMIT, APPLY_MUTATION_BYTES ); if( randomize && BUGGIFY ) BUGGIFIED_ROW_LIMIT = g_random->randomInt(3, 30);
init( PROXY_SPIN_DELAY, 0.01 );
init( COMMIT_BATCHES_MEM_FRACTION_OF_TOTAL, 0.5 );
init( COMMIT_BATCHES_MEM_TO_TOTAL_MEM_SCALE_FACTOR, 10.0 );
// Master Server
init( MASTER_LOGGING_DELAY, 1.0 );
@ -293,6 +296,7 @@ ServerKnobs::ServerKnobs(bool randomize, ClientKnobs* clientKnobs) {
init( MIN_REBOOT_TIME, 4.0 ); if( longReboots ) MIN_REBOOT_TIME = 10.0;
init( MAX_REBOOT_TIME, 5.0 ); if( longReboots ) MAX_REBOOT_TIME = 20.0;
init( LOG_DIRECTORY, "."); // Will be set to the command line flag.
init(SERVER_MEM_LIMIT, 8LL << 30);
//Ratekeeper
bool slowRateKeeper = randomize && BUGGIFY;

4
fdbserver/Knobs.h
View File

@ -178,6 +178,9 @@ public:
int COMMIT_TRANSACTION_BATCH_BYTES_MAX;
double COMMIT_TRANSACTION_BATCH_BYTES_SCALE_BASE;
double COMMIT_TRANSACTION_BATCH_BYTES_SCALE_POWER;
int64_t COMMIT_BATCHES_MEM_BYTES_HARD_LIMIT;
double COMMIT_BATCHES_MEM_FRACTION_OF_TOTAL;
double COMMIT_BATCHES_MEM_TO_TOTAL_MEM_SCALE_FACTOR;
double TRANSACTION_BUDGET_TIME;
double RESOLVER_COALESCE_TIME;
@ -236,6 +239,7 @@ public:
double MIN_REBOOT_TIME;
double MAX_REBOOT_TIME;
std::string LOG_DIRECTORY;
int64_t SERVER_MEM_LIMIT;
//Ratekeeper
double SMOOTHING_AMOUNT;

30
fdbserver/LeaderElection.actor.cpp
View File

@ -25,7 +25,7 @@
#include "fdbserver/CoordinationInterface.h"
#include "fdbclient/MonitorLeader.h"
extern Optional<LeaderInfo> getLeader( vector<Optional<LeaderInfo>> nominees );
Optional<std::pair<LeaderInfo, bool>> getLeader( const vector<Optional<LeaderInfo>>& nominees );
ACTOR Future<Void> submitCandidacy( Key key, LeaderElectionRegInterface coord, LeaderInfo myInfo, UID prevChangeID, Reference<AsyncVar<vector<Optional<LeaderInfo>>>> nominees, int index ) {
loop {
@ -106,44 +106,44 @@ ACTOR Future<Void> tryBecomeLeaderInternal( ServerCoordinators coordinators, Val
candidacies = waitForAll(cand);
loop {
state Optional<LeaderInfo> leader = getLeader( nominees->get() );
if( leader.present() && leader.get().forward ) {
state Optional<std::pair<LeaderInfo, bool>> leader = getLeader( nominees->get() );
if( leader.present() && leader.get().first.forward ) {
// These coordinators are forwarded to another set. But before we change our own cluster file, we need to make
// sure that a majority of coordinators know that.
// SOMEDAY: Wait briefly to see if other coordinators will tell us they already know, to save communication?
Void _ = wait( changeLeaderCoordinators( coordinators, leader.get().serializedInfo ) );
Void _ = wait( changeLeaderCoordinators( coordinators, leader.get().first.serializedInfo ) );
if(!hasConnected) {
TraceEvent(SevWarnAlways, "IncorrectClusterFileContentsAtConnection").detail("Filename", coordinators.ccf->getFilename())
.detail("ConnectionStringFromFile", coordinators.ccf->getConnectionString().toString())
.detail("CurrentConnectionString", leader.get().serializedInfo.toString());
.detail("CurrentConnectionString", leader.get().first.serializedInfo.toString());
}
coordinators.ccf->setConnectionString( ClusterConnectionString( leader.get().serializedInfo.toString() ) );
coordinators.ccf->setConnectionString( ClusterConnectionString( leader.get().first.serializedInfo.toString() ) );
TraceEvent("LeaderForwarding").detail("ConnStr", coordinators.ccf->getConnectionString().toString());
throw coordinators_changed();
}
if (leader.present()) {
if (leader.present() && leader.get().second) {
hasConnected = true;
coordinators.ccf->notifyConnected();
}
if (leader.present() && leader.get().changeID == myInfo.changeID) {
if (leader.present() && leader.get().second && leader.get().first.equalInternalId(myInfo)) {
TraceEvent("BecomingLeader", myInfo.changeID);
ASSERT( leader.get().serializedInfo == proposedSerializedInterface );
outSerializedLeader->set( leader.get().serializedInfo );
ASSERT( leader.get().first.serializedInfo == proposedSerializedInterface );
outSerializedLeader->set( leader.get().first.serializedInfo );
iAmLeader = true;
break;
}
if (leader.present()) {
TraceEvent("LeaderChanged", myInfo.changeID).detail("ToID", leader.get().changeID);
if (leader.get().serializedInfo != proposedSerializedInterface) // We never set outSerializedLeader to our own interface unless we are ready to become leader!
outSerializedLeader->set( leader.get().serializedInfo );
TraceEvent("LeaderChanged", myInfo.changeID).detail("ToID", leader.get().first.changeID);
if (leader.get().first.serializedInfo != proposedSerializedInterface) // We never set outSerializedLeader to our own interface unless we are ready to become leader!
outSerializedLeader->set( leader.get().first.serializedInfo );
}
// If more than 2*SERVER_KNOBS->POLLING_FREQUENCY elapses while we are nominated by some coordinator but there is no leader,
// we might be breaking the leader election process for someone with better communications but lower ID, so change IDs.
if (!leader.present() && std::count( nominees->get().begin(), nominees->get().end(), myInfo )) {
if ((!leader.present() || !leader.get().second) && std::count( nominees->get().begin(), nominees->get().end(), myInfo )) {
if (!badCandidateTimeout.isValid())
badCandidateTimeout = delay( SERVER_KNOBS->POLLING_FREQUENCY*2, TaskCoordinationReply );
} else
@ -211,4 +211,4 @@ ACTOR Future<Void> tryBecomeLeaderInternal( ServerCoordinators coordinators, Val
if (SERVER_KNOBS->BUGGIFY_ALL_COORDINATION || BUGGIFY) Void _ = wait( delay( SERVER_KNOBS->BUGGIFIED_EVENTUAL_CONSISTENCY * g_random->random01() ) );
return Void(); // We are no longer leader
}
}

12
fdbserver/LogRouter.actor.cpp
View File

@ -204,7 +204,7 @@ ACTOR Future<Void> pullAsyncData( LogRouterData *self ) {
Void _ = wait(self->minPopped.whenAtLeast(std::min(self->version.get(), ver - SERVER_KNOBS->MAX_READ_TRANSACTION_LIFE_VERSIONS)));
commitMessages(self, ver, messages);
self->version.set( ver );
//TraceEvent("LogRouterVersion").detail("ver",ver);
//TraceEvent("LogRouterVersion").detail("Ver",ver);
}
lastVer = ver;
ver = r->version().version;
@ -249,7 +249,7 @@ void peekMessagesFromMemory( LogRouterData* self, TLogPeekRequest const& req, Bi
ASSERT( !messages.getLength() );
auto& deque = get_version_messages(self, req.tag);
//TraceEvent("tLogPeekMem", self->dbgid).detail("Tag", printable(req.tag1)).detail("pDS", self->persistentDataSequence).detail("pDDS", self->persistentDataDurableSequence).detail("Oldest", map1.empty() ? 0 : map1.begin()->key ).detail("OldestMsgCount", map1.empty() ? 0 : map1.begin()->value.size());
//TraceEvent("TLogPeekMem", self->dbgid).detail("Tag", printable(req.tag1)).detail("PDS", self->persistentDataSequence).detail("PDDS", self->persistentDataDurableSequence).detail("Oldest", map1.empty() ? 0 : map1.begin()->key ).detail("OldestMsgCount", map1.empty() ? 0 : map1.begin()->value.size());
auto it = std::lower_bound(deque.begin(), deque.end(), std::make_pair(req.begin, LengthPrefixedStringRef()), CompareFirst<std::pair<Version, LengthPrefixedStringRef>>());
@ -258,7 +258,7 @@ void peekMessagesFromMemory( LogRouterData* self, TLogPeekRequest const& req, Bi
if(it->first != currentVersion) {
if (messages.getLength() >= SERVER_KNOBS->DESIRED_TOTAL_BYTES) {
endVersion = currentVersion + 1;
//TraceEvent("tLogPeekMessagesReached2", self->dbgid);
//TraceEvent("TLogPeekMessagesReached2", self->dbgid);
break;
}
@ -280,7 +280,7 @@ Version poppedVersion( LogRouterData* self, Tag tag) {
ACTOR Future<Void> logRouterPeekMessages( LogRouterData* self, TLogPeekRequest req ) {
state BinaryWriter messages(Unversioned());
//TraceEvent("LogRouterPeek1", self->dbgid).detail("from", req.reply.getEndpoint().address).detail("ver", self->version.get()).detail("begin", req.begin);
//TraceEvent("LogRouterPeek1", self->dbgid).detail("From", req.reply.getEndpoint().address).detail("Ver", self->version.get()).detail("Begin", req.begin);
if( req.returnIfBlocked && self->version.get() < req.begin ) {
//TraceEvent("LogRouterPeek2", self->dbgid);
req.reply.sendError(end_of_stream());
@ -297,7 +297,7 @@ ACTOR Future<Void> logRouterPeekMessages( LogRouterData* self, TLogPeekRequest r
if(poppedVer > req.begin || req.begin < self->startVersion) {
//This should only happen if a packet is sent multiple times and the reply is not needed.
// Since we are using popped differently, do not send a reply.
TraceEvent(SevWarnAlways, "LogRouterPeekPopped", self->dbgid).detail("begin", req.begin).detail("popped", poppedVer).detail("start", self->startVersion);
TraceEvent(SevWarnAlways, "LogRouterPeekPopped", self->dbgid).detail("Begin", req.begin).detail("Popped", poppedVer).detail("Start", self->startVersion);
req.reply.send( Never() );
return Void();
}
@ -410,7 +410,7 @@ ACTOR Future<Void> logRouter(
Reference<AsyncVar<ServerDBInfo>> db)
{
try {
TraceEvent("LogRouterStart", interf.id()).detail("start", req.startVersion).detail("tag", req.routerTag.toString()).detail("localities", req.tLogLocalities.size()).detail("hasBestPolicy", req.hasBestPolicy).detail("locality", req.locality);
TraceEvent("LogRouterStart", interf.id()).detail("Start", req.startVersion).detail("Tag", req.routerTag.toString()).detail("Localities", req.tLogLocalities.size()).detail("HasBestPolicy", req.hasBestPolicy).detail("Locality", req.locality);
state Future<Void> core = logRouterCore(interf, req, db);
loop choose{
when(Void _ = wait(core)) { return Void(); }

8
fdbserver/LogSystemDiskQueueAdapter.actor.cpp
View File

@ -29,14 +29,14 @@ public:
while (self->recoveryQueueDataSize < bytes) {
if (self->recoveryLoc == self->logSystem->getEnd()) {
// Recovery will be complete once the current recoveryQueue is consumed, so we no longer need self->logSystem
TraceEvent("PeekNextEnd").detail("queue", self->recoveryQueue.size()).detail("bytes", bytes).detail("loc", self->recoveryLoc).detail("end", self->logSystem->getEnd());
TraceEvent("PeekNextEnd").detail("Queue", self->recoveryQueue.size()).detail("Bytes", bytes).detail("Loc", self->recoveryLoc).detail("End", self->logSystem->getEnd());
self->logSystem.clear();
break;
}
if(!self->cursor->hasMessage()) {
Void _ = wait( self->cursor->getMore() );
TraceEvent("PeekNextGetMore").detail("queue", self->recoveryQueue.size()).detail("bytes", bytes).detail("loc", self->recoveryLoc).detail("end", self->logSystem->getEnd());
TraceEvent("PeekNextGetMore").detail("Queue", self->recoveryQueue.size()).detail("Bytes", bytes).detail("Loc", self->recoveryLoc).detail("End", self->logSystem->getEnd());
if(self->recoveryQueueDataSize == 0) {
self->recoveryQueueLoc = self->recoveryLoc;
}
@ -51,7 +51,7 @@ public:
self->cursor->nextMessage();
if(!self->cursor->hasMessage()) self->recoveryLoc = self->cursor->version().version;
//TraceEvent("PeekNextResults").detail("from", self->recoveryLoc).detail("queue", self->recoveryQueue.size()).detail("bytes", bytes).detail("has", self->cursor->hasMessage()).detail("end", self->logSystem->getEnd());
//TraceEvent("PeekNextResults").detail("From", self->recoveryLoc).detail("Queue", self->recoveryQueue.size()).detail("Bytes", bytes).detail("Has", self->cursor->hasMessage()).detail("End", self->logSystem->getEnd());
}
if(self->recoveryQueue.size() > 1) {
self->recoveryQueue[0] = concatenate(self->recoveryQueue.begin(), self->recoveryQueue.end());
@ -63,7 +63,7 @@ public:
ASSERT(self->recoveryQueue[0].size() == self->recoveryQueueDataSize);
//TraceEvent("PeekNextReturn").detail("bytes", bytes).detail("queueSize", self->recoveryQueue.size());
//TraceEvent("PeekNextReturn").detail("Bytes", bytes).detail("QueueSize", self->recoveryQueue.size());
bytes = std::min(bytes, self->recoveryQueue[0].size());
Standalone<StringRef> result( self->recoveryQueue[0].substr(0,bytes), self->recoveryQueue[0].arena() );
self->recoveryQueue[0].contents() = self->recoveryQueue[0].substr(bytes);

52
fdbserver/LogSystemPeekCursor.actor.cpp
View File

@ -26,13 +26,13 @@
ILogSystem::ServerPeekCursor::ServerPeekCursor( Reference<AsyncVar<OptionalInterface<TLogInterface>>> const& interf, Tag tag, Version begin, Version end, bool returnIfBlocked, bool parallelGetMore )
: interf(interf), tag(tag), messageVersion(begin), end(end), hasMsg(false), rd(results.arena, results.messages, Unversioned()), randomID(g_random->randomUniqueID()), poppedVersion(0), returnIfBlocked(returnIfBlocked), sequence(0), parallelGetMore(parallelGetMore) {
this->results.maxKnownVersion = 0;
//TraceEvent("SPC_starting", randomID).detail("tag", tag.toString()).detail("begin", begin).detail("end", end).backtrace();
//TraceEvent("SPC_Starting", randomID).detail("Tag", tag.toString()).detail("Begin", begin).detail("End", end).backtrace();
}
ILogSystem::ServerPeekCursor::ServerPeekCursor( TLogPeekReply const& results, LogMessageVersion const& messageVersion, LogMessageVersion const& end, int32_t messageLength, int32_t rawLength, bool hasMsg, Version poppedVersion, Tag tag )
: results(results), tag(tag), rd(results.arena, results.messages, Unversioned()), messageVersion(messageVersion), end(end), messageLength(messageLength), rawLength(rawLength), hasMsg(hasMsg), randomID(g_random->randomUniqueID()), poppedVersion(poppedVersion), returnIfBlocked(false), sequence(0), parallelGetMore(false)
{
//TraceEvent("SPC_clone", randomID);
//TraceEvent("SPC_Clone", randomID);
this->results.maxKnownVersion = 0;
if(hasMsg)
nextMessage();
@ -55,12 +55,12 @@ ArenaReader* ILogSystem::ServerPeekCursor::reader() {
}
bool ILogSystem::ServerPeekCursor::hasMessage() {
//TraceEvent("SPC_hasMessage", randomID).detail("hasMsg", hasMsg);
//TraceEvent("SPC_HasMessage", randomID).detail("HasMsg", hasMsg);
return hasMsg;
}
void ILogSystem::ServerPeekCursor::nextMessage() {
//TraceEvent("SPC_nextMessage", randomID).detail("messageVersion", messageVersion.toString());
//TraceEvent("SPC_NextMessage", randomID).detail("MessageVersion", messageVersion.toString());
ASSERT(hasMsg);
if (rd.empty()) {
messageVersion.reset(std::min(results.end, end.version));
@ -73,7 +73,7 @@ void ILogSystem::ServerPeekCursor::nextMessage() {
Version ver;
rd >> dummy >> ver;
//TraceEvent("SPC_process_Seq", randomID).detail("messageVersion", messageVersion.toString()).detail("ver", ver).detail("tag", tag.toString());
//TraceEvent("SPC_ProcessSeq", randomID).detail("MessageVersion", messageVersion.toString()).detail("Ver", ver).detail("Tag", tag.toString());
//ASSERT( ver >= messageVersion.version );
messageVersion.reset(ver);
@ -96,11 +96,11 @@ void ILogSystem::ServerPeekCursor::nextMessage() {
rawLength = messageLength + sizeof(messageLength);
messageLength -= (sizeof(messageVersion.sub) + sizeof(tagCount) + tagCount*sizeof(Tag));
hasMsg = true;
//TraceEvent("SPC_nextMessageB", randomID).detail("messageVersion", messageVersion.toString());
//TraceEvent("SPC_NextMessageB", randomID).detail("MessageVersion", messageVersion.toString());
}
StringRef ILogSystem::ServerPeekCursor::getMessage() {
//TraceEvent("SPC_getMessage", randomID);
//TraceEvent("SPC_GetMessage", randomID);
return StringRef( (uint8_t const*)rd.readBytes(messageLength), messageLength);
}
@ -114,7 +114,7 @@ const std::vector<Tag>& ILogSystem::ServerPeekCursor::getTags() {
}
void ILogSystem::ServerPeekCursor::advanceTo(LogMessageVersion n) {
//TraceEvent("SPC_advanceTo", randomID).detail("n", n.toString());
//TraceEvent("SPC_AdvanceTo", randomID).detail("N", n.toString());
while( messageVersion < n && hasMessage() ) {
getMessage();
nextMessage();
@ -157,7 +157,7 @@ ACTOR Future<Void> serverPeekParallelGetMore( ILogSystem::ServerPeekCursor* self
self->hasMsg = true;
self->nextMessage();
self->advanceTo(skipSeq);
//TraceEvent("SPC_getMoreB", self->randomID).detail("has", self->hasMessage()).detail("end", res.end).detail("popped", res.popped.present() ? res.popped.get() : 0);
//TraceEvent("SPC_GetMoreB", self->randomID).detail("Has", self->hasMessage()).detail("End", res.end).detail("Popped", res.popped.present() ? res.popped.get() : 0);
return Void();
}
when( Void _ = wait( self->interfaceChanged ) ) {
@ -202,7 +202,7 @@ ACTOR Future<Void> serverPeekGetMore( ILogSystem::ServerPeekCursor* self, int ta
self->hasMsg = true;
self->nextMessage();
self->advanceTo(skipSeq);
//TraceEvent("SPC_getMoreB", self->randomID).detail("has", self->hasMessage()).detail("end", res.end).detail("popped", res.popped.present() ? res.popped.get() : 0);
//TraceEvent("SPC_GetMoreB", self->randomID).detail("Has", self->hasMessage()).detail("End", res.end).detail("Popped", res.popped.present() ? res.popped.get() : 0);
return Void();
}
when( Void _ = wait( self->interf->onChange() ) ) {}
@ -218,7 +218,7 @@ ACTOR Future<Void> serverPeekGetMore( ILogSystem::ServerPeekCursor* self, int ta
}
Future<Void> ILogSystem::ServerPeekCursor::getMore(int taskID) {
//TraceEvent("SPC_getMore", randomID).detail("hasMessage", hasMessage()).detail("more", !more.isValid() || more.isReady()).detail("messageVersion", messageVersion.toString()).detail("end", end.toString());
//TraceEvent("SPC_GetMore", randomID).detail("HasMessage", hasMessage()).detail("More", !more.isValid() || more.isReady()).detail("MessageVersion", messageVersion.toString()).detail("End", end.toString());
if( hasMessage() )
return Void();
if( !more.isValid() || more.isReady() ) {
@ -269,7 +269,7 @@ ILogSystem::MergedPeekCursor::MergedPeekCursor( std::vector<Reference<AsyncVar<O
tLogPolicy(tLogPolicy), tLogReplicationFactor(tLogReplicationFactor), collectTags(false) {
for( int i = 0; i < logServers.size(); i++ ) {
Reference<ILogSystem::ServerPeekCursor> cursor( new ILogSystem::ServerPeekCursor( logServers[i], tag, begin, end, bestServer >= 0, parallelGetMore ) );
//TraceEvent("MPC_starting", randomID).detail("cursor", cursor->randomID).detail("end", end);
//TraceEvent("MPC_Starting", randomID).detail("Cursor", cursor->randomID).detail("End", end);
serverCursors.push_back( cursor );
}
sortedVersions.resize(serverCursors.size());
@ -432,7 +432,7 @@ void ILogSystem::MergedPeekCursor::advanceTo(LogMessageVersion n) {
ACTOR Future<Void> mergedPeekGetMore(ILogSystem::MergedPeekCursor* self, LogMessageVersion startVersion, int taskID) {
loop {
//TraceEvent("MPC_getMoreA", self->randomID).detail("start", startVersion.toString());
//TraceEvent("MPC_GetMoreA", self->randomID).detail("Start", startVersion.toString());
if(self->bestServer >= 0 && self->serverCursors[self->bestServer]->isActive()) {
ASSERT(!self->serverCursors[self->bestServer]->hasMessage());
Void _ = wait( self->serverCursors[self->bestServer]->getMore(taskID) || self->serverCursors[self->bestServer]->onFailed() );
@ -444,7 +444,7 @@ ACTOR Future<Void> mergedPeekGetMore(ILogSystem::MergedPeekCursor* self, LogMess
Void _ = wait(quorum(q, 1));
}
self->calcHasMessage();
//TraceEvent("MPC_getMoreB", self->randomID).detail("hasMessage", self->hasMessage()).detail("start", startVersion.toString()).detail("seq", self->version().toString());
//TraceEvent("MPC_GetMoreB", self->randomID).detail("HasMessage", self->hasMessage()).detail("Start", startVersion.toString()).detail("Seq", self->version().toString());
if (self->hasMessage() || self->version() > startVersion) {
self->messageArena = Arena();
return Void();
@ -546,7 +546,7 @@ ArenaReader* ILogSystem::SetPeekCursor::reader() { return serverCursors[currentS
void ILogSystem::SetPeekCursor::calcHasMessage() {
if(bestSet >= 0 && bestServer >= 0) {
if(nextVersion.present()) {
//TraceEvent("LPC_calcNext").detail("ver", messageVersion.toString()).detail("tag", tag.toString()).detail("hasNextMessage", hasNextMessage).detail("nextVersion", nextVersion.get().toString());
//TraceEvent("LPC_CalcNext").detail("Ver", messageVersion.toString()).detail("Tag", tag.toString()).detail("HasNextMessage", hasNextMessage).detail("NextVersion", nextVersion.get().toString());
serverCursors[bestSet][bestServer]->advanceTo( nextVersion.get() );
}
if( serverCursors[bestSet][bestServer]->hasMessage() ) {
@ -555,7 +555,7 @@ void ILogSystem::SetPeekCursor::calcHasMessage() {
currentCursor = bestServer;
hasNextMessage = true;
//TraceEvent("LPC_calc1").detail("ver", messageVersion.toString()).detail("tag", tag.toString()).detail("hasNextMessage", hasNextMessage);
//TraceEvent("LPC_Calc1").detail("Ver", messageVersion.toString()).detail("Tag", tag.toString()).detail("HasNextMessage", hasNextMessage);
for (auto& cursors : serverCursors) {
for(auto& c : cursors) {
@ -578,10 +578,10 @@ void ILogSystem::SetPeekCursor::calcHasMessage() {
if(useBestSet) {
updateMessage(bestSet, false); // Use Quorum logic
//TraceEvent("LPC_calc2").detail("ver", messageVersion.toString()).detail("tag", tag.toString()).detail("hasNextMessage", hasNextMessage);
//TraceEvent("LPC_Calc2").detail("Ver", messageVersion.toString()).detail("Tag", tag.toString()).detail("HasNextMessage", hasNextMessage);
if(!hasNextMessage) {
updateMessage(bestSet, true);
//TraceEvent("LPC_calc3").detail("ver", messageVersion.toString()).detail("tag", tag.toString()).detail("hasNextMessage", hasNextMessage);
//TraceEvent("LPC_Calc3").detail("Ver", messageVersion.toString()).detail("Tag", tag.toString()).detail("HasNextMessage", hasNextMessage);
}
} else {
for(int i = 0; i < logSets.size() && !hasNextMessage; i++) {
@ -589,13 +589,13 @@ void ILogSystem::SetPeekCursor::calcHasMessage() {
updateMessage(i, false); // Use Quorum logic
}
}
//TraceEvent("LPC_calc4").detail("ver", messageVersion.toString()).detail("tag", tag.toString()).detail("hasNextMessage", hasNextMessage);
//TraceEvent("LPC_Calc4").detail("Ver", messageVersion.toString()).detail("Tag", tag.toString()).detail("HasNextMessage", hasNextMessage);
for(int i = 0; i < logSets.size() && !hasNextMessage; i++) {
if(i != bestSet) {
updateMessage(i, true);
}
}
//TraceEvent("LPC_calc5").detail("ver", messageVersion.toString()).detail("tag", tag.toString()).detail("hasNextMessage", hasNextMessage);
//TraceEvent("LPC_Calc5").detail("Ver", messageVersion.toString()).detail("Tag", tag.toString()).detail("HasNextMessage", hasNextMessage);
}
}
@ -607,7 +607,7 @@ void ILogSystem::SetPeekCursor::updateMessage(int logIdx, bool usePolicy) {
auto& serverCursor = serverCursors[logIdx][i];
if (nextVersion.present()) serverCursor->advanceTo(nextVersion.get());
sortedVersions.push_back(std::pair<LogMessageVersion, int>(serverCursor->version(), i));
//TraceEvent("LPC_update1").detail("ver", messageVersion.toString()).detail("tag", tag.toString()).detail("hasNextMessage", hasNextMessage).detail("serverVer", serverCursor->version().toString()).detail("i", i);
//TraceEvent("LPC_Update1").detail("Ver", messageVersion.toString()).detail("Tag", tag.toString()).detail("HasNextMessage", hasNextMessage).detail("ServerVer", serverCursor->version().toString()).detail("I", i);
}
if(usePolicy) {
@ -687,10 +687,10 @@ void ILogSystem::SetPeekCursor::advanceTo(LogMessageVersion n) {
ACTOR Future<Void> setPeekGetMore(ILogSystem::SetPeekCursor* self, LogMessageVersion startVersion, int taskID) {
loop {
//TraceEvent("LPC_getMore1", self->randomID).detail("start", startVersion.toString()).detail("t", self->tag);
//TraceEvent("LPC_GetMore1", self->randomID).detail("Start", startVersion.toString()).detail("Tag", self->tag);
if(self->bestServer >= 0 && self->bestSet >= 0 && self->serverCursors[self->bestSet][self->bestServer]->isActive()) {
ASSERT(!self->serverCursors[self->bestSet][self->bestServer]->hasMessage());
//TraceEvent("LPC_getMore2", self->randomID).detail("start", startVersion.toString()).detail("t", self->tag);
//TraceEvent("LPC_GetMore2", self->randomID).detail("Start", startVersion.toString()).detail("Tag", self->tag);
Void _ = wait( self->serverCursors[self->bestSet][self->bestServer]->getMore(taskID) || self->serverCursors[self->bestSet][self->bestServer]->onFailed() );
self->useBestSet = true;
} else {
@ -713,7 +713,7 @@ ACTOR Future<Void> setPeekGetMore(ILogSystem::SetPeekCursor* self, LogMessageVer
return Void();
}
//TraceEvent("LPC_getMore3", self->randomID).detail("start", startVersion.toString()).detail("t", self->tag.toString()).detail("bestSetSize", self->serverCursors[self->bestSet].size());
//TraceEvent("LPC_GetMore3", self->randomID).detail("Start", startVersion.toString()).detail("Tag", self->tag.toString()).detail("BestSetSize", self->serverCursors[self->bestSet].size());
vector<Future<Void>> q;
for (auto& c : self->serverCursors[self->bestSet]) {
if (!c->hasMessage()) {
@ -727,7 +727,7 @@ ACTOR Future<Void> setPeekGetMore(ILogSystem::SetPeekCursor* self, LogMessageVer
} else {
//FIXME: this will peeking way too many cursors when satellites exist, and does not need to peek bestSet cursors since we cannot get anymore data from them
vector<Future<Void>> q;
//TraceEvent("LPC_getMore4", self->randomID).detail("start", startVersion.toString()).detail("t", self->tag);
//TraceEvent("LPC_GetMore4", self->randomID).detail("Start", startVersion.toString()).detail("Tag", self->tag);
for(auto& cursors : self->serverCursors) {
for (auto& c :cursors) {
if (!c->hasMessage()) {
@ -740,7 +740,7 @@ ACTOR Future<Void> setPeekGetMore(ILogSystem::SetPeekCursor* self, LogMessageVer
}
}
self->calcHasMessage();
//TraceEvent("LPC_getMoreB", self->randomID).detail("hasMessage", self->hasMessage()).detail("start", startVersion.toString()).detail("seq", self->version().toString());
//TraceEvent("LPC_GetMoreB", self->randomID).detail("HasMessage", self->hasMessage()).detail("Start", startVersion.toString()).detail("Seq", self->version().toString());
if (self->hasMessage() || self->version() > startVersion)
return Void();
}

74
fdbserver/MasterProxyServer.actor.cpp
View File

@ -58,15 +58,16 @@ struct ProxyStats {
Future<Void> logger;
explicit ProxyStats(UID id, Version* pVersion, NotifiedVersion* pCommittedVersion)
explicit ProxyStats(UID id, Version* pVersion, NotifiedVersion* pCommittedVersion, int64_t *commitBatchesMemBytesCountPtr)
: cc("ProxyStats", id.toString()),
txnStartIn("txnStartIn", cc), txnStartOut("txnStartOut", cc), txnStartBatch("txnStartBatch", cc), txnSystemPriorityStartIn("txnSystemPriorityStartIn", cc), txnSystemPriorityStartOut("txnSystemPriorityStartOut", cc), txnBatchPriorityStartIn("txnBatchPriorityStartIn", cc), txnBatchPriorityStartOut("txnBatchPriorityStartOut", cc),
txnDefaultPriorityStartIn("txnDefaultPriorityStartIn", cc), txnDefaultPriorityStartOut("txnDefaultPriorityStartOut", cc), txnCommitIn("txnCommitIn", cc), txnCommitVersionAssigned("txnCommitVersionAssigned", cc), txnCommitResolving("txnCommitResolving", cc), txnCommitResolved("txnCommitResolved", cc), txnCommitOut("txnCommitOut", cc),
txnCommitOutSuccess("txnCommitOutSuccess", cc), txnConflicts("txnConflicts", cc), commitBatchIn("commitBatchIn", cc), commitBatchOut("commitBatchOut", cc), mutationBytes("mutationBytes", cc), mutations("mutations", cc), conflictRanges("conflictRanges", cc), lastCommitVersionAssigned(0)
txnStartIn("TxnStartIn", cc), txnStartOut("TxnStartOut", cc), txnStartBatch("TxnStartBatch", cc), txnSystemPriorityStartIn("TxnSystemPriorityStartIn", cc), txnSystemPriorityStartOut("TxnSystemPriorityStartOut", cc), txnBatchPriorityStartIn("TxnBatchPriorityStartIn", cc), txnBatchPriorityStartOut("TxnBatchPriorityStartOut", cc),
txnDefaultPriorityStartIn("TxnDefaultPriorityStartIn", cc), txnDefaultPriorityStartOut("TxnDefaultPriorityStartOut", cc), txnCommitIn("TxnCommitIn", cc), txnCommitVersionAssigned("TxnCommitVersionAssigned", cc), txnCommitResolving("TxnCommitResolving", cc), txnCommitResolved("TxnCommitResolved", cc), txnCommitOut("TxnCommitOut", cc),
txnCommitOutSuccess("TxnCommitOutSuccess", cc), txnConflicts("TxnConflicts", cc), commitBatchIn("CommitBatchIn", cc), commitBatchOut("CommitBatchOut", cc), mutationBytes("MutationBytes", cc), mutations("Mutations", cc), conflictRanges("ConflictRanges", cc), lastCommitVersionAssigned(0)
{
specialCounter(cc, "lastAssignedCommitVersion", [this](){return this->lastCommitVersionAssigned;});
specialCounter(cc, "version", [pVersion](){return *pVersion; });
specialCounter(cc, "committedVersion", [pCommittedVersion](){ return pCommittedVersion->get(); });
specialCounter(cc, "LastAssignedCommitVersion", [this](){return this->lastCommitVersionAssigned;});
specialCounter(cc, "Version", [pVersion](){return *pVersion; });
specialCounter(cc, "CommittedVersion", [pCommittedVersion](){ return pCommittedVersion->get(); });
specialCounter(cc, "CommitBatchesMemBytesCount", [commitBatchesMemBytesCountPtr]() { return *commitBatchesMemBytesCountPtr; });
logger = traceCounters("ProxyMetrics", id, SERVER_KNOBS->WORKER_LOGGING_INTERVAL, &cc, "ProxyMetrics");
}
};
@ -152,7 +153,7 @@ ACTOR Future<Void> queueTransactionStartRequests(std::priority_queue< std::pair<
ACTOR void discardCommit(UID id, Future<LogSystemDiskQueueAdapter::CommitMessage> fcm, Future<Void> dummyCommitState) {
ASSERT(!dummyCommitState.isReady());
LogSystemDiskQueueAdapter::CommitMessage cm = wait(fcm);
TraceEvent("Discarding", id).detail("count", cm.messages.size());
TraceEvent("Discarding", id).detail("Count", cm.messages.size());
cm.acknowledge.send(Void());
ASSERT(dummyCommitState.isReady());
}
@ -167,6 +168,7 @@ DESCR struct SingleKeyMutation {
struct ProxyCommitData {
UID dbgid;
int64_t commitBatchesMemBytesCount;
ProxyStats stats;
MasterInterface master;
vector<ResolverInterface> resolvers;
@ -221,13 +223,14 @@ struct ProxyCommitData {
}
ProxyCommitData(UID dbgid, MasterInterface master, RequestStream<GetReadVersionRequest> getConsistentReadVersion, Version recoveryTransactionVersion, RequestStream<CommitTransactionRequest> commit, Reference<AsyncVar<ServerDBInfo>> db, bool firstProxy)
: dbgid(dbgid), stats(dbgid, &version, &committedVersion), master(master),
: dbgid(dbgid), stats(dbgid, &version, &committedVersion, &commitBatchesMemBytesCount), master(master),
logAdapter(NULL), txnStateStore(NULL),
committedVersion(recoveryTransactionVersion), version(0),
lastVersionTime(0), commitVersionRequestNumber(1), mostRecentProcessedRequestNumber(0),
getConsistentReadVersion(getConsistentReadVersion), commit(commit), lastCoalesceTime(0),
localCommitBatchesStarted(0), locked(false), firstProxy(firstProxy),
cx(openDBOnServer(db, TaskDefaultEndpoint, true, true)), singleKeyMutationEvent(LiteralStringRef("SingleKeyMutation"))
cx(openDBOnServer(db, TaskDefaultEndpoint, true, true)), singleKeyMutationEvent(LiteralStringRef("SingleKeyMutation")),
commitBatchesMemBytesCount(0)
{}
};
@ -315,7 +318,8 @@ struct ResolutionRequestBuilder {
ACTOR Future<Void> commitBatch(
ProxyCommitData* self,
vector<CommitTransactionRequest> trs,
double *commitBatchTime)
double *commitBatchTime,
int currentBatchMemBytesCount)
{
state int64_t localBatchNumber = ++self->localCommitBatchesStarted;
state LogPushData toCommit(self->logSystem);
@ -325,6 +329,9 @@ ACTOR Future<Void> commitBatch(
ASSERT(SERVER_KNOBS->MAX_READ_TRANSACTION_LIFE_VERSIONS <= SERVER_KNOBS->MAX_VERSIONS_IN_FLIGHT); // since we are using just the former to limit the number of versions actually in flight!
// Active load balancing runs at a very high priority (to obtain accurate estimate of memory used by commit batches) so we need to downgrade here
Void _ = wait(delay(0, TaskProxyCommit));
self->lastVersionTime = t1;
++self->stats.commitBatchIn;
@ -339,7 +346,7 @@ ACTOR Future<Void> commitBatch(
if(localBatchNumber == 2 && !debugID.present() && self->firstProxy && !g_network->isSimulated()) {
debugID = g_random->randomUniqueID();
TraceEvent("SecondCommitBatch", self->dbgid).detail("debugID", debugID.get());
TraceEvent("SecondCommitBatch", self->dbgid).detail("DebugID", debugID.get());
}
if (debugID.present())
@ -374,7 +381,7 @@ ACTOR Future<Void> commitBatch(
r->value().push_back(std::make_pair(versionReply.resolverChangesVersion,it.dest));
}
//TraceEvent("ProxyGotVer", self->dbgid).detail("commit", commitVersion).detail("prev", prevVersion);
//TraceEvent("ProxyGotVer", self->dbgid).detail("Commit", commitVersion).detail("Prev", prevVersion);
if (debugID.present())
g_traceBatch.addEvent("CommitDebug", debugID.get().first(), "MasterProxyServer.commitBatch.GotCommitVersion");
@ -730,8 +737,8 @@ ACTOR Future<Void> commitBatch(
// if (debugMutation("BackupProxyCommit", commitVersion, backupMutation)) {
// TraceEvent("BackupProxyCommitTo", self->dbgid).detail("To", describe(tags)).detail("BackupMutation", backupMutation.toString())
// .detail("BackupMutationSize", val.size()).detail("Version", commitVersion).detail("destPath", printable(logRangeMutation.first))
// .detail("partIndex", part).detail("partIndexEndian", bigEndian32(part)).detail("partData", printable(backupMutation.param1));
// .detail("BackupMutationSize", val.size()).detail("Version", commitVersion).detail("DestPath", printable(logRangeMutation.first))
// .detail("PartIndex", part).detail("PartIndexEndian", bigEndian32(part)).detail("PartData", printable(backupMutation.param1));
// }
}
}
@ -857,7 +864,7 @@ ACTOR Future<Void> commitBatch(
}
self->keyResolvers.coalesce(allKeys);
if(self->keyResolvers.size() != lastSize)
TraceEvent("KeyResolverSize", self->dbgid).detail("size", self->keyResolvers.size());
TraceEvent("KeyResolverSize", self->dbgid).detail("Size", self->keyResolvers.size());
}
// Dynamic batching for commits
@ -868,6 +875,8 @@ ACTOR Future<Void> commitBatch(
target_latency * SERVER_KNOBS->COMMIT_TRANSACTION_BATCH_INTERVAL_SMOOTHER_ALPHA + *commitBatchTime * (1-SERVER_KNOBS->COMMIT_TRANSACTION_BATCH_INTERVAL_SMOOTHER_ALPHA)));
self->commitBatchesMemBytesCount -= currentBatchMemBytesCount;
ASSERT_ABORT(self->commitBatchesMemBytesCount >= 0);
return Void();
}
@ -1004,15 +1013,15 @@ ACTOR static Future<Void> transactionStarter(
forwardPromise(GRVTimer, delayJittered(SERVER_KNOBS->START_TRANSACTION_BATCH_QUEUE_CHECK_INTERVAL, TaskProxyGRVTimer));
/*TraceEvent("GRVBatch", proxy.id())
.detail("elapsed", elapsed)
.detail("nTransactionToStart", nTransactionsToStart)
.detail("transactionRate", transactionRate)
.detail("transactionQueueSize", transactionQueue.size())
.detail("numTransactionsStarted", transactionsStarted[0] + transactionsStarted[1])
.detail("numSystemTransactionsStarted", systemTransactionsStarted[0] + systemTransactionsStarted[1])
.detail("numNonSystemTransactionsStarted", transactionsStarted[0] + transactionsStarted[1] - systemTransactionsStarted[0] - systemTransactionsStarted[1])
.detail("transactionBudget", transactionBudget)
.detail("lastLeftToStart", leftToStart);*/
.detail("Elapsed", elapsed)
.detail("NTransactionToStart", nTransactionsToStart)
.detail("TransactionRate", transactionRate)
.detail("TransactionQueueSize", transactionQueue.size())
.detail("NumTransactionsStarted", transactionsStarted[0] + transactionsStarted[1])
.detail("NumSystemTransactionsStarted", systemTransactionsStarted[0] + systemTransactionsStarted[1])
.detail("NumNonSystemTransactionsStarted", transactionsStarted[0] + transactionsStarted[1] - systemTransactionsStarted[0] - systemTransactionsStarted[1])
.detail("TransactionBudget", transactionBudget)
.detail("LastLeftToStart", leftToStart);*/
// dynamic batching
ReplyPromise<GetReadVersionReply> GRVReply;
@ -1156,9 +1165,8 @@ ACTOR Future<Void> masterProxyServerCore(
state ProxyCommitData commitData(proxy.id(), master, proxy.getConsistentReadVersion, recoveryTransactionVersion, proxy.commit, db, firstProxy);
state Future<Sequence> sequenceFuture = (Sequence)0;
state PromiseStream< vector<CommitTransactionRequest> > batchedCommits;
state PromiseStream< std::pair<vector<CommitTransactionRequest>, int> > batchedCommits;
state Future<Void> commitBatcher;
state Future<Void> lastCommitComplete = Void();
state PromiseStream<Future<Void>> addActor;
@ -1192,6 +1200,10 @@ ACTOR Future<Void> masterProxyServerCore(
commitData.logAdapter = new LogSystemDiskQueueAdapter(commitData.logSystem, txsTag, false);
commitData.txnStateStore = keyValueStoreLogSystem(commitData.logAdapter, proxy.id(), 2e9, true, true);
onError = onError || commitData.logSystem->onError();
// ((SERVER_MEM_LIMIT * COMMIT_BATCHES_MEM_FRACTION_OF_TOTAL) / COMMIT_BATCHES_MEM_TO_TOTAL_MEM_SCALE_FACTOR) is only a approximate formula for limiting the memory used.
// COMMIT_BATCHES_MEM_TO_TOTAL_MEM_SCALE_FACTOR is an estimate based on experiments and not an accurate one.
state int64_t commitBatchesMemoryLimit = std::min(SERVER_KNOBS->COMMIT_BATCHES_MEM_BYTES_HARD_LIMIT, static_cast<int64_t>((SERVER_KNOBS->SERVER_MEM_LIMIT * SERVER_KNOBS->COMMIT_BATCHES_MEM_FRACTION_OF_TOTAL) / SERVER_KNOBS->COMMIT_BATCHES_MEM_TO_TOTAL_MEM_SCALE_FACTOR));
TraceEvent(SevInfo, "CommitBatchesMemoryLimit").detail("BytesLimit", commitBatchesMemoryLimit);
addActor.send(transactionStarter(proxy, master, db, addActor, &commitData));
addActor.send(readRequestServer(proxy, &commitData));
@ -1203,7 +1215,7 @@ ACTOR Future<Void> masterProxyServerCore(
(int)std::min<double>(SERVER_KNOBS->COMMIT_TRANSACTION_BATCH_BYTES_MAX,
std::max<double>(SERVER_KNOBS->COMMIT_TRANSACTION_BATCH_BYTES_MIN,
SERVER_KNOBS->COMMIT_TRANSACTION_BATCH_BYTES_SCALE_BASE * pow(db->get().client.proxies.size(), SERVER_KNOBS->COMMIT_TRANSACTION_BATCH_BYTES_SCALE_POWER)));
commitBatcher = batcher(batchedCommits, proxy.commit.getFuture(), SERVER_KNOBS->COMMIT_TRANSACTION_BATCH_INTERVAL_FROM_IDLE, &commitBatchInterval, SERVER_KNOBS->MAX_COMMIT_BATCH_INTERVAL, SERVER_KNOBS->COMMIT_TRANSACTION_BATCH_COUNT_MAX, commitBatchByteLimit, CLIENT_KNOBS->TRANSACTION_SIZE_LIMIT, commitData.commitBatchStartNotifications, TaskProxyCommitBatcher, &commitData.stats.txnCommitIn);
commitBatcher = batcher(batchedCommits, proxy.commit.getFuture(), SERVER_KNOBS->COMMIT_TRANSACTION_BATCH_INTERVAL_FROM_IDLE, &commitBatchInterval, SERVER_KNOBS->MAX_COMMIT_BATCH_INTERVAL, SERVER_KNOBS->COMMIT_TRANSACTION_BATCH_COUNT_MAX, commitBatchByteLimit, CLIENT_KNOBS->TRANSACTION_SIZE_LIMIT, commitData.commitBatchStartNotifications, &commitData.commitBatchesMemBytesCount, commitBatchesMemoryLimit, TaskProxyCommitBatcher, &commitData.stats.txnCommitIn);
loop choose{
when( Void _ = wait( dbInfoChange ) ) {
dbInfoChange = db->onChange();
@ -1215,13 +1227,15 @@ ACTOR Future<Void> masterProxyServerCore(
}
}
when(Void _ = wait(onError)) {}
when(vector<CommitTransactionRequest> trs = waitNext(batchedCommits.getFuture())) {
when(std::pair<vector<CommitTransactionRequest>, int> batchedRequests = waitNext(batchedCommits.getFuture())) {
const vector<CommitTransactionRequest> &trs = batchedRequests.first;
int batchBytes = batchedRequests.second;
//TraceEvent("MasterProxyCTR", proxy.id()).detail("CommitTransactions", trs.size()).detail("TransactionRate", transactionRate).detail("TransactionQueue", transactionQueue.size()).detail("ReleasedTransactionCount", transactionCount);
if (trs.size() || (db->get().recoveryState >= RecoveryState::FULLY_RECOVERED && now() - lastCommit >= SERVER_KNOBS->MAX_COMMIT_BATCH_INTERVAL)) {
lastCommit = now();
if (trs.size() || lastCommitComplete.isReady()) {
lastCommitComplete = commitBatch(&commitData, trs, &commitBatchInterval);
lastCommitComplete = commitBatch(&commitData, trs, &commitBatchInterval, batchBytes);
addActor.send(lastCommitComplete);
}
}

8
fdbserver/MoveKeys.actor.cpp
View File

@ -315,7 +315,7 @@ ACTOR Future<Void> startMoveKeys( Database occ, KeyRange keys, vector<UID> serve
Void _ = wait( tr.onError(e) );
if(retries%10 == 0) {
TraceEvent(retries == 50 ? SevWarnAlways : SevWarn, "startMoveKeysRetrying", relocationIntervalId)
TraceEvent(retries == 50 ? SevWarnAlways : SevWarn, "StartMoveKeysRetrying", relocationIntervalId)
.detail("Keys", printable(keys))
.detail("BeginKey", printable(begin))
.detail("NumTries", retries)
@ -556,7 +556,7 @@ ACTOR Future<Void> finishMoveKeys( Database occ, KeyRange keys, vector<UID> dest
ASSERT(false);
}
waitInterval = TraceInterval("RelocateShard_FinishMoveKeys_WaitDurable");
waitInterval = TraceInterval("RelocateShard_FinishMoveKeysWaitDurable");
TraceEvent(SevDebug, waitInterval.begin(), relocationIntervalId)
.detail("KeyBegin", printable(keys.begin))
.detail("KeyEnd", printable(keys.end));
@ -627,7 +627,7 @@ ACTOR Future<Void> finishMoveKeys( Database occ, KeyRange keys, vector<UID> dest
Void _ = wait( tr.onError(error) );
retries++;
if(retries%10 == 0) {
TraceEvent(retries == 20 ? SevWarnAlways : SevWarn, "RelocateShard_finishMoveKeysRetrying", relocationIntervalId)
TraceEvent(retries == 20 ? SevWarnAlways : SevWarn, "RelocateShard_FinishMoveKeysRetrying", relocationIntervalId)
.error(err)
.detail("KeyBegin", printable(keys.begin))
.detail("KeyEnd", printable(keys.end))
@ -771,7 +771,7 @@ ACTOR Future<Void> removeStorageServer( Database cx, UID serverID, MoveKeysLock
TraceEvent(SevWarn,"NoCanRemove").detail("Count", noCanRemoveCount++).detail("ServerID", serverID);
Void _ = wait( delayJittered(SERVER_KNOBS->REMOVE_RETRY_DELAY, TaskDataDistributionLaunch) );
tr.reset();
TraceEvent("RemoveStorageServerRetrying").detail("canRemove", canRemove);
TraceEvent("RemoveStorageServerRetrying").detail("CanRemove", canRemove);
} else {
state Future<Optional<Value>> fListKey = tr.get( serverListKeyFor(serverID) );

66
fdbserver/OldTLogServer.actor.cpp
View File

@ -407,8 +407,8 @@ namespace oldTLog {
explicit LogData(TLogData* tLogData, TLogInterface interf) : tLogData(tLogData), knownCommittedVersion(0), tli(interf), logId(interf.id()),
cc("TLog", interf.id().toString()),
bytesInput("bytesInput", cc),
bytesDurable("bytesDurable", cc),
bytesInput("BytesInput", cc),
bytesDurable("BytesDurable", cc),
// These are initialized differently on init() or recovery
recoveryCount(), stopped(false), initialized(false), queueCommittingVersion(0), newPersistentDataVersion(invalidVersion), recovery(Void())
{
@ -419,22 +419,22 @@ namespace oldTLog {
version.initMetric(LiteralStringRef("TLog.Version"), cc.id);
queueCommittedVersion.initMetric(LiteralStringRef("TLog.QueueCommittedVersion"), cc.id);
specialCounter(cc, "version", [this](){ return this->version.get(); });
specialCounter(cc, "sharedBytesInput", [tLogData](){ return tLogData->bytesInput; });
specialCounter(cc, "sharedBytesDurable", [tLogData](){ return tLogData->bytesDurable; });
specialCounter(cc, "kvstoreBytesUsed", [tLogData](){ return tLogData->persistentData->getStorageBytes().used; });
specialCounter(cc, "kvstoreBytesFree", [tLogData](){ return tLogData->persistentData->getStorageBytes().free; });
specialCounter(cc, "kvstoreBytesAvailable", [tLogData](){ return tLogData->persistentData->getStorageBytes().available; });
specialCounter(cc, "kvstoreBytesTotal", [tLogData](){ return tLogData->persistentData->getStorageBytes().total; });
specialCounter(cc, "queueDiskBytesUsed", [tLogData](){ return tLogData->rawPersistentQueue->getStorageBytes().used; });
specialCounter(cc, "queueDiskBytesFree", [tLogData](){ return tLogData->rawPersistentQueue->getStorageBytes().free; });
specialCounter(cc, "queueDiskBytesAvailable", [tLogData](){ return tLogData->rawPersistentQueue->getStorageBytes().available; });
specialCounter(cc, "queueDiskBytesTotal", [tLogData](){ return tLogData->rawPersistentQueue->getStorageBytes().total; });
specialCounter(cc, "Version", [this](){ return this->version.get(); });
specialCounter(cc, "SharedBytesInput", [tLogData](){ return tLogData->bytesInput; });
specialCounter(cc, "SharedBytesDurable", [tLogData](){ return tLogData->bytesDurable; });
specialCounter(cc, "KvstoreBytesUsed", [tLogData](){ return tLogData->persistentData->getStorageBytes().used; });
specialCounter(cc, "KvstoreBytesFree", [tLogData](){ return tLogData->persistentData->getStorageBytes().free; });
specialCounter(cc, "KvstoreBytesAvailable", [tLogData](){ return tLogData->persistentData->getStorageBytes().available; });
specialCounter(cc, "KvstoreBytesTotal", [tLogData](){ return tLogData->persistentData->getStorageBytes().total; });
specialCounter(cc, "QueueDiskBytesUsed", [tLogData](){ return tLogData->rawPersistentQueue->getStorageBytes().used; });
specialCounter(cc, "QueueDiskBytesFree", [tLogData](){ return tLogData->rawPersistentQueue->getStorageBytes().free; });
specialCounter(cc, "QueueDiskBytesAvailable", [tLogData](){ return tLogData->rawPersistentQueue->getStorageBytes().available; });
specialCounter(cc, "QueueDiskBytesTotal", [tLogData](){ return tLogData->rawPersistentQueue->getStorageBytes().total; });
}
~LogData() {
tLogData->bytesDurable += bytesInput.getValue() - bytesDurable.getValue();
TraceEvent("TLogBytesWhenRemoved", tli.id()).detail("sharedBytesInput", tLogData->bytesInput).detail("sharedBytesDurable", tLogData->bytesDurable).detail("localBytesInput", bytesInput.getValue()).detail("localBytesDurable", bytesDurable.getValue());
TraceEvent("TLogBytesWhenRemoved", tli.id()).detail("SharedBytesInput", tLogData->bytesInput).detail("SharedBytesDurable", tLogData->bytesDurable).detail("LocalBytesInput", bytesInput.getValue()).detail("LocalBytesDurable", bytesDurable.getValue());
ASSERT_ABORT(tLogData->bytesDurable <= tLogData->bytesInput);
endRole(tli.id(), "TLog", "Error", true);
@ -460,7 +460,7 @@ namespace oldTLog {
TEST( logData->stopped );
TEST( !logData->stopped );
TraceEvent("TLogStop", logData->logId).detail("Ver", stopVersion).detail("isStopped", logData->stopped).detail("queueCommitted", logData->queueCommittedVersion.get());
TraceEvent("TLogStop", logData->logId).detail("Ver", stopVersion).detail("IsStopped", logData->stopped).detail("QueueCommitted", logData->queueCommittedVersion.get());
logData->stopped = true;
if(logData->recoverySuccessful.canBeSet()) {
@ -476,7 +476,7 @@ namespace oldTLog {
result.end = stopVersion;
result.knownCommittedVersion = logData->knownCommittedVersion;
TraceEvent("TLogStop2", self->dbgid).detail("logId", logData->logId).detail("Ver", stopVersion).detail("isStopped", logData->stopped).detail("queueCommitted", logData->queueCommittedVersion.get());
TraceEvent("TLogStop2", self->dbgid).detail("LogId", logData->logId).detail("Ver", stopVersion).detail("IsStopped", logData->stopped).detail("QueueCommitted", logData->queueCommittedVersion.get());
reply.send( result );
@ -504,7 +504,7 @@ namespace oldTLog {
ASSERT( newPersistentDataVersion > logData->persistentDataVersion );
ASSERT( logData->persistentDataVersion == logData->persistentDataDurableVersion );
//TraceEvent("updatePersistentData", self->dbgid).detail("seq", newPersistentDataSeq);
//TraceEvent("UpdatePersistentData", self->dbgid).detail("Seq", newPersistentDataSeq);
state bool anyData = false;
state Map<OldTag, LogData::TagData>::iterator tag;
@ -565,7 +565,7 @@ namespace oldTLog {
}
if(logData->bytesDurable.getValue() > logData->bytesInput.getValue() || self->bytesDurable > self->bytesInput) {
TraceEvent(SevError, "BytesDurableTooLarge", logData->logId).detail("sharedBytesInput", self->bytesInput).detail("sharedBytesDurable", self->bytesDurable).detail("localBytesInput", logData->bytesInput.getValue()).detail("localBytesDurable", logData->bytesDurable.getValue());
TraceEvent(SevError, "BytesDurableTooLarge", logData->logId).detail("SharedBytesInput", self->bytesInput).detail("SharedBytesDurable", self->bytesDurable).detail("LocalBytesInput", logData->bytesInput.getValue()).detail("LocalBytesDurable", logData->bytesDurable.getValue());
}
ASSERT(logData->bytesDurable.getValue() <= logData->bytesInput.getValue());
@ -623,7 +623,7 @@ namespace oldTLog {
Void _ = wait( logData->queueCommittedVersion.whenAtLeast( nextVersion ) );
Void _ = wait( delay(0, TaskUpdateStorage) );
//TraceEvent("TlogUpdatePersist", self->dbgid).detail("logId", logData->logId).detail("nextVersion", nextVersion).detail("version", logData->version.get()).detail("persistentDataDurableVer", logData->persistentDataDurableVersion).detail("queueCommitVer", logData->queueCommittedVersion.get()).detail("persistDataVer", logData->persistentDataVersion);
//TraceEvent("TlogUpdatePersist", self->dbgid).detail("LogId", logData->logId).detail("NextVersion", nextVersion).detail("Version", logData->version.get()).detail("PersistentDataDurableVer", logData->persistentDataDurableVersion).detail("QueueCommitVer", logData->queueCommittedVersion.get()).detail("PersistDataVer", logData->persistentDataVersion);
if (nextVersion > logData->persistentDataVersion) {
self->updatePersist = updatePersistentData(self, logData, nextVersion);
Void _ = wait( self->updatePersist );
@ -670,7 +670,7 @@ namespace oldTLog {
nextVersion = std::max<Version>(nextVersion, logData->persistentDataVersion);
//TraceEvent("UpdateStorageVer", logData->logId).detail("nextVersion", nextVersion).detail("persistentDataVersion", logData->persistentDataVersion).detail("totalSize", totalSize);
//TraceEvent("UpdateStorageVer", logData->logId).detail("NextVersion", nextVersion).detail("PersistentDataVersion", logData->persistentDataVersion).detail("TotalSize", totalSize);
Void _ = wait( logData->queueCommittedVersion.whenAtLeast( nextVersion ) );
Void _ = wait( delay(0, TaskUpdateStorage) );
@ -794,7 +794,7 @@ namespace oldTLog {
self->bytesInput += addedBytes;
bytesInput += addedBytes;
//TraceEvent("TLogPushed", self->dbgid).detail("Bytes", addedBytes).detail("MessageBytes", messages.size()).detail("Tags", tags.size()).detail("expectedBytes", expectedBytes).detail("mCount", mCount).detail("tCount", tCount);
//TraceEvent("TLogPushed", self->dbgid).detail("Bytes", addedBytes).detail("MessageBytes", messages.size()).detail("Tags", tags.size()).detail("ExpectedBytes", expectedBytes).detail("MCount", mCount).detail("TCount", tCount);
}
Version poppedVersion( Reference<LogData> self, OldTag tag) {
@ -897,7 +897,7 @@ namespace oldTLog {
return Void();
}
//TraceEvent("tLogPeekMessages0", self->dbgid).detail("reqBeginEpoch", req.begin.epoch).detail("reqBeginSeq", req.begin.sequence).detail("epoch", self->epoch()).detail("persistentDataSeq", self->persistentDataSequence).detail("Tag1", printable(req.tag1)).detail("Tag2", printable(req.tag2));
//TraceEvent("tLogPeekMessages0", self->dbgid).detail("ReqBeginEpoch", req.begin.epoch).detail("ReqBeginSeq", req.begin.sequence).detail("Epoch", self->epoch()).detail("PersistentDataSeq", self->persistentDataSequence).detail("Tag1", printable(req.tag1)).detail("Tag2", printable(req.tag2));
// Wait until we have something to return that the caller doesn't already have
if( logData->version.get() < req.begin ) {
Void _ = wait( logData->version.whenAtLeast( req.begin ) );
@ -907,7 +907,7 @@ namespace oldTLog {
state Version endVersion = logData->version.get() + 1;
//grab messages from disk
//TraceEvent("tLogPeekMessages", self->dbgid).detail("reqBeginEpoch", req.begin.epoch).detail("reqBeginSeq", req.begin.sequence).detail("epoch", self->epoch()).detail("persistentDataSeq", self->persistentDataSequence).detail("Tag1", printable(req.tag1)).detail("Tag2", printable(req.tag2));
//TraceEvent("tLogPeekMessages", self->dbgid).detail("ReqBeginEpoch", req.begin.epoch).detail("ReqBeginSeq", req.begin.sequence).detail("Epoch", self->epoch()).detail("PersistentDataSeq", self->persistentDataSequence).detail("Tag1", printable(req.tag1)).detail("Tag2", printable(req.tag2));
if( req.begin <= logData->persistentDataDurableVersion ) {
// Just in case the durable version changes while we are waiting for the read, we grab this data from memory. We may or may not actually send it depending on
// whether we get enough data from disk.
@ -959,7 +959,7 @@ namespace oldTLog {
reply.messages = messages.toStringRef();
reply.end = endVersion;
}
//TraceEvent("TlogPeek", self->dbgid).detail("logId", logData->logId).detail("endVer", reply.end).detail("msgBytes", reply.messages.expectedSize()).detail("ForAddress", req.reply.getEndpoint().address);
//TraceEvent("TlogPeek", self->dbgid).detail("LogId", logData->logId).detail("EndVer", reply.end).detail("MsgBytes", reply.messages.expectedSize()).detail("ForAddress", req.reply.getEndpoint().address);
if(req.sequence.present()) {
auto& trackerData = self->peekTracker[peekId];
@ -1026,7 +1026,7 @@ namespace oldTLog {
continue;
}
TraceEvent("commitQueueNewLog", self->dbgid).detail("logId", logData->logId).detail("version", logData->version.get()).detail("committing", logData->queueCommittingVersion).detail("commmitted", logData->queueCommittedVersion.get());
TraceEvent("CommitQueueNewLog", self->dbgid).detail("LogId", logData->logId).detail("Version", logData->version.get()).detail("Committing", logData->queueCommittingVersion).detail("Commmitted", logData->queueCommittedVersion.get());
loop {
if(logData->stopped && logData->version.get() == std::max(logData->queueCommittingVersion, logData->queueCommittedVersion.get())) {
@ -1074,8 +1074,8 @@ namespace oldTLog {
}
if ( isDisplaced )
{
TraceEvent("TLogDisplaced", tli.id()).detail("Reason", "DBInfoDoesNotContain").detail("recoveryCount", recoveryCount).detail("infRecoveryCount", inf.recoveryCount).detail("recoveryState", inf.recoveryState)
.detail("logSysConf", describe(inf.logSystemConfig.tLogs)).detail("priorLogs", describe(inf.priorCommittedLogServers)).detail("oldLogGens", inf.logSystemConfig.oldTLogs.size());
TraceEvent("TLogDisplaced", tli.id()).detail("Reason", "DBInfoDoesNotContain").detail("RecoveryCount", recoveryCount).detail("InfRecoveryCount", inf.recoveryCount).detail("RecoveryState", inf.recoveryState)
.detail("LogSysConf", describe(inf.logSystemConfig.tLogs)).detail("PriorLogs", describe(inf.priorCommittedLogServers)).detail("OldLogGens", inf.logSystemConfig.oldTLogs.size());
if (BUGGIFY) Void _ = wait( delay( SERVER_KNOBS->BUGGIFY_WORKER_REMOVED_MAX_LAG * g_random->random01() ) );
throw worker_removed();
}
@ -1167,7 +1167,7 @@ namespace oldTLog {
}
void removeLog( TLogData* self, Reference<LogData> logData ) {
TraceEvent("TLogRemoved", logData->logId).detail("input", logData->bytesInput.getValue()).detail("durable", logData->bytesDurable.getValue());
TraceEvent("TLogRemoved", logData->logId).detail("Input", logData->bytesInput.getValue()).detail("Durable", logData->bytesDurable.getValue());
logData->stopped = true;
if(logData->recoverySuccessful.canBeSet()) {
logData->recoverySuccessful.send(false);
@ -1196,7 +1196,7 @@ namespace oldTLog {
return Void();
}
TraceEvent("newLogData", self->dbgid).detail("logId", logData->logId);
TraceEvent("NewLogData", self->dbgid).detail("LogId", logData->logId);
logData->initialized = true;
self->newLogData.trigger();
@ -1293,7 +1293,7 @@ namespace oldTLog {
logData->removed = rejoinMasters(self, recruited, logData->recoveryCount, registerWithMaster.getFuture());
removed.push_back(errorOr(logData->removed));
TraceEvent("TLogRestorePersistentStateVer", id1).detail("ver", ver);
TraceEvent("TLogRestorePersistentStateVer", id1).detail("Ver", ver);
// Restore popped keys. Pop operations that took place after the last (committed) updatePersistentDataVersion might be lost, but
// that is fine because we will get the corresponding data back, too.
@ -1342,8 +1342,8 @@ namespace oldTLog {
lastVer = qe.version;
}
//TraceEvent("TLogRecoveredQE", self->dbgid).detail("logId", qe.id).detail("ver", qe.version).detail("MessageBytes", qe.messages.size()).detail("Tags", qe.tags.size())
// .detail("Tag0", qe.tags.size() ? qe.tags[0].tag : invalidTag).detail("version", logData->version.get());
//TraceEvent("TLogRecoveredQE", self->dbgid).detail("LogId", qe.id).detail("Ver", qe.version).detail("MessageBytes", qe.messages.size()).detail("Tags", qe.tags.size())
// .detail("Tag0", qe.tags.size() ? qe.tags[0].tag : invalidTag).detail("Version", logData->version.get());
if(logData) {
logData->knownCommittedVersion = std::max(logData->knownCommittedVersion, qe.knownCommittedVersion);
@ -1376,7 +1376,7 @@ namespace oldTLog {
for(auto it : self->id_data) {
if(it.second->queueCommittedVersion.get() == 0) {
TraceEvent("TLogZeroVersion", self->dbgid).detail("logId", it.first);
TraceEvent("TLogZeroVersion", self->dbgid).detail("LogId", it.first);
it.second->queueCommittedVersion.set(it.second->version.get());
}
self->sharedActors.send( tLogCore( self, it.second ) );

12
fdbserver/QuietDatabase.actor.cpp
View File

@ -50,7 +50,7 @@ ACTOR Future<WorkerInterface> getMasterWorker( Database cx, Reference<AsyncVar<S
for( int i = 0; i < workers.size(); i++ ) {
if( workers[i].first.address() == dbInfo->get().master.address() ) {
TraceEvent("GetMasterWorker").detail("Database", printable(cx->dbName)).detail("Stage", "GotWorkers").detail("masterId", dbInfo->get().master.id()).detail("workerId", workers[i].first.id());
TraceEvent("GetMasterWorker").detail("Database", printable(cx->dbName)).detail("Stage", "GotWorkers").detail("MasterId", dbInfo->get().master.id()).detail("WorkerId", workers[i].first.id());
return workers[i].first;
}
}
@ -94,8 +94,8 @@ int64_t getQueueSize( TraceEventFields md ) {
double inputRoughness, durableRoughness;
int64_t inputBytes, durableBytes;
sscanf(md.getValue("bytesInput").c_str(), "%lf %lf %lld", &inputRate, &inputRoughness, &inputBytes);
sscanf(md.getValue("bytesDurable").c_str(), "%lf %lf %lld", &durableRate, &durableRoughness, &durableBytes);
sscanf(md.getValue("BytesInput").c_str(), "%lf %lf %lld", &inputRate, &inputRoughness, &inputBytes);
sscanf(md.getValue("BytesDurable").c_str(), "%lf %lf %lld", &durableRate, &durableRoughness, &durableBytes);
return inputBytes - durableBytes;
}
@ -330,9 +330,9 @@ ACTOR Future<Void> waitForQuietDatabase( Database cx, Reference<AsyncVar<ServerD
Void _ = wait( success( dataInFlight ) && success( tLogQueueSize ) && success( dataDistributionQueueSize )
&& success( storageQueueSize ) && success( dataDistributionActive ) && success( storageServersRecruiting ) );
TraceEvent(("QuietDatabase" + phase).c_str())
.detail("dataInFlight", dataInFlight.get()).detail("maxTLogQueueSize", tLogQueueSize.get()).detail("dataDistributionQueueSize", dataDistributionQueueSize.get())
.detail("maxStorageQueueSize", storageQueueSize.get()).detail("dataDistributionActive", dataDistributionActive.get())
.detail("storageServersRecruiting", storageServersRecruiting.get());
.detail("DataInFlight", dataInFlight.get()).detail("MaxTLogQueueSize", tLogQueueSize.get()).detail("DataDistributionQueueSize", dataDistributionQueueSize.get())
.detail("MaxStorageQueueSize", storageQueueSize.get()).detail("DataDistributionActive", dataDistributionActive.get())
.detail("StorageServersRecruiting", storageServersRecruiting.get());
if( dataInFlight.get() > dataInFlightGate || tLogQueueSize.get() > maxTLogQueueGate
|| dataDistributionQueueSize.get() > maxDataDistributionQueueSize || storageQueueSize.get() > maxStorageServerQueueGate

4
fdbserver/Ratekeeper.actor.cpp
View File

@ -312,7 +312,7 @@ void updateRate( Ratekeeper* self ) {
.detail("ActualTPS", actualTPS)
.detail("InputRate", inputRate)
.detail("VerySmoothDurableBytesRate", ss.verySmoothDurableBytes.smoothRate())
.detail("b", b);
.detail("B", b);
}*/
// Don't let any storage server use up its target bytes faster than its MVCC window!
@ -423,7 +423,7 @@ void updateRate( Ratekeeper* self ) {
if( tl.lastReply.bytesInput - tl.lastReply.bytesDurable > tl.lastReply.storageBytes.free - minFreeSpace / 2 ) {
if(now() - self->lastWarning > 5.0) {
self->lastWarning = now();
TraceEvent(SevWarnAlways, "RkTlogMinFreeSpaceZero").detail("reasonId", tl.id);
TraceEvent(SevWarnAlways, "RkTlogMinFreeSpaceZero").detail("ReasonId", tl.id);
}
reasonID = tl.id;
limitReason = limitReason_t::log_server_min_free_space;

6
fdbserver/Resolver.actor.cpp
View File

@ -90,9 +90,9 @@ ACTOR Future<Void> resolveBatch(
while( self->totalStateBytes.get() > SERVER_KNOBS->RESOLVER_STATE_MEMORY_LIMIT && self->recentStateTransactionSizes.size() &&
proxyInfo.lastVersion > self->recentStateTransactionSizes.front().first && req.version > self->neededVersion.get() ) {
/*TraceEvent("ResolveBatchDelay").detail("From", proxyAddress).detail("stateBytes", self->totalStateBytes.get()).detail("RecentStateTransactionSize", self->recentStateTransactionSizes.size())
.detail("lastVersion", proxyInfo.lastVersion).detail("RequestVersion", req.version).detail("NeededVersion", self->neededVersion.get())
.detail("recentStateVer", self->recentStateTransactions.begin()->key);*/
/*TraceEvent("ResolveBatchDelay").detail("From", proxyAddress).detail("StateBytes", self->totalStateBytes.get()).detail("RecentStateTransactionSize", self->recentStateTransactionSizes.size())
.detail("LastVersion", proxyInfo.lastVersion).detail("RequestVersion", req.version).detail("NeededVersion", self->neededVersion.get())
.detail("RecentStateVer", self->recentStateTransactions.begin()->key);*/
Void _ = wait( self->totalStateBytes.onChange() || self->neededVersion.onChange() );
}

22
fdbserver/SimulatedCluster.actor.cpp
View File

@ -165,7 +165,7 @@ ACTOR Future<Void> runDr( Reference<ClusterConnectionFile> connFile ) {
Reference<Cluster> extraCluster = Cluster::createCluster(extraFile, -1);
state Database extraDB = extraCluster->createDatabase(LiteralStringRef("DB")).get();
TraceEvent("StartingDrAgents").detail("connFile", connFile->getConnectionString().toString()).detail("extraString", extraFile->getConnectionString().toString());
TraceEvent("StartingDrAgents").detail("ConnFile", connFile->getConnectionString().toString()).detail("ExtraString", extraFile->getConnectionString().toString());
state DatabaseBackupAgent dbAgent = DatabaseBackupAgent(cx);
state DatabaseBackupAgent extraAgent = DatabaseBackupAgent(extraDB);
@ -219,7 +219,7 @@ ACTOR Future<ISimulator::KillType> simulatedFDBDRebooter(
TraceEvent("SimulatedFDBDPreWait").detail("Cycles", cycles).detail("RandomId", randomId)
.detail("Address", NetworkAddress(ip, port, true, false))
.detailext("ZoneId", localities.zoneId())
.detail("waitTime", waitTime).detail("Port", port);
.detail("WaitTime", waitTime).detail("Port", port);
Void _ = wait( delay( waitTime ) );
@ -422,8 +422,8 @@ ACTOR Future<Void> simulatedMachine(
.detail("CFolder0", coordFolders[0])
.detail("MachineIPs", toIPVectorString(ips))
.detail("SSL", sslEnabled)
.detail("processes", processes.size())
.detail("bootCount", bootCount)
.detail("Processes", processes.size())
.detail("BootCount", bootCount)
.detail("ProcessClass", processClass.toString())
.detail("Restarting", restarting)
.detail("UseSeedFile", useSeedFile)
@ -657,12 +657,12 @@ ACTOR Future<Void> restartSimulatedSystem(
g_simulator.processesPerMachine = processesPerMachine;
}
catch (Error& e) {
TraceEvent(SevError, "restartSimulationError").error(e);
TraceEvent(SevError, "RestartSimulationError").error(e);
}
TraceEvent("RestartSimulatorSettings")
.detail("desiredCoordinators", g_simulator.desiredCoordinators)
.detail("processesPerMachine", g_simulator.processesPerMachine);
.detail("DesiredCoordinators", g_simulator.desiredCoordinators)
.detail("ProcessesPerMachine", g_simulator.processesPerMachine);
Void _ = wait(delay(1.0));
@ -951,7 +951,7 @@ void setupSimulatedSystem( vector<Future<Void>> *systemActors, std::string baseF
ASSERT(g_simulator.storagePolicy && g_simulator.tLogPolicy);
ASSERT(!g_simulator.hasRemoteReplication || g_simulator.remoteTLogPolicy);
ASSERT(!g_simulator.hasSatelliteReplication || g_simulator.satelliteTLogPolicy);
TraceEvent("simulatorConfig").detail("ConfigString", printable(StringRef(startingConfigString)));
TraceEvent("SimulatorConfig").detail("ConfigString", printable(StringRef(startingConfigString)));
const int dataCenters = simconfig.datacenters;
const int machineCount = simconfig.machine_count;
@ -1063,9 +1063,9 @@ void setupSimulatedSystem( vector<Future<Void>> *systemActors, std::string baseF
g_simulator.processesPerMachine = processesPerMachine;
TraceEvent("SetupSimulatorSettings")
.detail("desiredCoordinators", g_simulator.desiredCoordinators)
.detail("physicalDatacenters", g_simulator.physicalDatacenters)
.detail("processesPerMachine", g_simulator.processesPerMachine);
.detail("DesiredCoordinators", g_simulator.desiredCoordinators)
.detail("PhysicalDatacenters", g_simulator.physicalDatacenters)
.detail("ProcessesPerMachine", g_simulator.processesPerMachine);
// SOMEDAY: add locality for testers to simulate network topology
// FIXME: Start workers with tester class instead, at least sometimes run tests with the testers-only flag

54
fdbserver/Status.actor.cpp
View File

@ -408,15 +408,15 @@ struct RolesInfo {
obj["id"] = iface.id().shortString();
obj["role"] = role;
try {
obj["stored_bytes"] = parseInt64(metrics.getValue("bytesStored"));
obj["kvstore_used_bytes"] = parseInt64(metrics.getValue("kvstoreBytesUsed"));
obj["kvstore_free_bytes"] = parseInt64(metrics.getValue("kvstoreBytesFree"));
obj["kvstore_available_bytes"] = parseInt64(metrics.getValue("kvstoreBytesAvailable"));
obj["kvstore_total_bytes"] = parseInt64(metrics.getValue("kvstoreBytesTotal"));
obj["input_bytes"] = parseCounter(metrics.getValue("bytesInput"));
obj["durable_bytes"] = parseCounter(metrics.getValue("bytesDurable"));
obj["stored_bytes"] = parseInt64(metrics.getValue("BytesStored"));
obj["kvstore_used_bytes"] = parseInt64(metrics.getValue("KvstoreBytesUsed"));
obj["kvstore_free_bytes"] = parseInt64(metrics.getValue("KvstoreBytesFree"));
obj["kvstore_available_bytes"] = parseInt64(metrics.getValue("KvstoreBytesAvailable"));
obj["kvstore_total_bytes"] = parseInt64(metrics.getValue("KvstoreBytesTotal"));
obj["input_bytes"] = parseCounter(metrics.getValue("BytesInput"));
obj["durable_bytes"] = parseCounter(metrics.getValue("BytesDurable"));
obj["query_queue_max"] = parseInt(metrics.getValue("QueryQueueMax"));
obj["finished_queries"] = parseCounter(metrics.getValue("finishedQueries"));
obj["finished_queries"] = parseCounter(metrics.getValue("FinishedQueries"));
Version version = parseInt64(metrics.getValue("version"));
obj["data_version"] = version;
@ -436,17 +436,17 @@ struct RolesInfo {
obj["id"] = iface.id().shortString();
obj["role"] = role;
try {
obj["kvstore_used_bytes"] = parseInt64(metrics.getValue("kvstoreBytesUsed"));
obj["kvstore_free_bytes"] = parseInt64(metrics.getValue("kvstoreBytesFree"));
obj["kvstore_available_bytes"] = parseInt64(metrics.getValue("kvstoreBytesAvailable"));
obj["kvstore_total_bytes"] = parseInt64(metrics.getValue("kvstoreBytesTotal"));
obj["queue_disk_used_bytes"] = parseInt64(metrics.getValue("queueDiskBytesUsed"));
obj["queue_disk_free_bytes"] = parseInt64(metrics.getValue("queueDiskBytesFree"));
obj["queue_disk_available_bytes"] = parseInt64(metrics.getValue("queueDiskBytesAvailable"));
obj["queue_disk_total_bytes"] = parseInt64(metrics.getValue("queueDiskBytesTotal"));
obj["input_bytes"] = parseCounter(metrics.getValue("bytesInput"));
obj["durable_bytes"] = parseCounter(metrics.getValue("bytesDurable"));
obj["data_version"] = parseInt64(metrics.getValue("version"));
obj["kvstore_used_bytes"] = parseInt64(metrics.getValue("KvstoreBytesUsed"));
obj["kvstore_free_bytes"] = parseInt64(metrics.getValue("KvstoreBytesFree"));
obj["kvstore_available_bytes"] = parseInt64(metrics.getValue("KvstoreBytesAvailable"));
obj["kvstore_total_bytes"] = parseInt64(metrics.getValue("KvstoreBytesTotal"));
obj["queue_disk_used_bytes"] = parseInt64(metrics.getValue("QueueDiskBytesUsed"));
obj["queue_disk_free_bytes"] = parseInt64(metrics.getValue("QueueDiskBytesFree"));
obj["queue_disk_available_bytes"] = parseInt64(metrics.getValue("QueueDiskBytesAvailable"));
obj["queue_disk_total_bytes"] = parseInt64(metrics.getValue("QueueDiskBytesTotal"));
obj["input_bytes"] = parseCounter(metrics.getValue("BytesInput"));
obj["durable_bytes"] = parseCounter(metrics.getValue("BytesDurable"));
obj["data_version"] = parseInt64(metrics.getValue("Version"));
} catch (Error& e) {
if(e.code() != error_code_attribute_not_found)
throw e;
@ -1214,11 +1214,11 @@ ACTOR static Future<StatusObject> workloadStatusFetcher(Reference<AsyncVar<struc
StatusObject mutations=makeCounter(), mutationBytes=makeCounter(), txnConflicts=makeCounter(), txnStartOut=makeCounter(), txnCommitOutSuccess=makeCounter();
for (auto &ps : proxyStats) {
mutations = addCounters( mutations, parseCounter(ps.getValue("mutations")) );
mutationBytes = addCounters( mutationBytes, parseCounter(ps.getValue("mutationBytes")) );
txnConflicts = addCounters( txnConflicts, parseCounter(ps.getValue("txnConflicts")) );
txnStartOut = addCounters( txnStartOut, parseCounter(ps.getValue("txnStartOut")) );
txnCommitOutSuccess = addCounters( txnCommitOutSuccess, parseCounter(ps.getValue("txnCommitOutSuccess")) );
mutations = addCounters( mutations, parseCounter(ps.getValue("Mutations")) );
mutationBytes = addCounters( mutationBytes, parseCounter(ps.getValue("MutationBytes")) );
txnConflicts = addCounters( txnConflicts, parseCounter(ps.getValue("TxnConflicts")) );
txnStartOut = addCounters( txnStartOut, parseCounter(ps.getValue("TxnStartOut")) );
txnCommitOutSuccess = addCounters( txnCommitOutSuccess, parseCounter(ps.getValue("TxnCommitOutSuccess")) );
}
operationsObj["writes"] = mutations;
@ -1307,9 +1307,9 @@ ACTOR static Future<StatusObject> workloadStatusFetcher(Reference<AsyncVar<struc
StatusObject readBytes = makeCounter();
for(auto &ss : storageServers.get()) {
reads = addCounters(reads, parseCounter(ss.second.getValue("finishedQueries")));
readKeys = addCounters(readKeys, parseCounter(ss.second.getValue("rowsQueried")));
readBytes = addCounters(readBytes, parseCounter(ss.second.getValue("bytesQueried")));
reads = addCounters(reads, parseCounter(ss.second.getValue("FinishedQueries")));
readKeys = addCounters(readKeys, parseCounter(ss.second.getValue("RowsQueried")));
readBytes = addCounters(readBytes, parseCounter(ss.second.getValue("BytesQueried")));
}
operationsObj["reads"] = reads;

10
fdbserver/StorageMetrics.actor.h
View File

@ -72,7 +72,7 @@ struct StorageMetricSample {
}
// If we didn't return above, we didn't find anything.
TraceEvent(SevWarnAlways, "CannotSplitLastSampleKey").detail("range", printable(range)).detail("offset", offset);
TraceEvent(SevWarnAlways, "CannotSplitLastSampleKey").detail("Range", printable(range)).detail("Offset", offset);
return front ? range.end : range.begin;
}
};
@ -349,10 +349,10 @@ struct StorageServerMetrics {
if (sb.free < 1e9 && g_random->random01() < 0.1)
TraceEvent(SevWarn, "PhysicalDiskMetrics")
.detail("free", sb.free)
.detail("total", sb.total)
.detail("available", sb.available)
.detail("load", rep.load.bytes);
.detail("Free", sb.free)
.detail("Total", sb.total)
.detail("Available", sb.available)
.detail("Load", rep.load.bytes);
rep.free.bytes = sb.free;
rep.free.iosPerKSecond = 10e6;

222
fdbserver/TLogServer.actor.cpp
View File

@ -283,43 +283,43 @@ struct TLogData : NonCopyable {
struct LogData : NonCopyable, public ReferenceCounted<LogData> {
struct TagData : NonCopyable, public ReferenceCounted<TagData> {
std::deque<std::pair<Version, LengthPrefixedStringRef>> version_messages;
bool nothing_persistent; // true means tag is *known* to have no messages in persistentData. false means nothing.
bool popped_recently; // `popped` has changed since last updatePersistentData
std::deque<std::pair<Version, LengthPrefixedStringRef>> versionMessages;
bool nothingPersistent; // true means tag is *known* to have no messages in persistentData. false means nothing.
bool poppedRecently; // `popped` has changed since last updatePersistentData
Version popped; // see popped version tracking contract below
bool update_version_sizes;
bool updateVersionSizes;
bool unpoppedRecovered;
Tag tag;
TagData( Tag tag, Version popped, bool nothing_persistent, bool popped_recently, bool unpoppedRecovered ) : tag(tag), nothing_persistent(nothing_persistent), popped(popped), popped_recently(popped_recently), unpoppedRecovered(unpoppedRecovered), update_version_sizes(tag != txsTag) {}
TagData( Tag tag, Version popped, bool nothingPersistent, bool poppedRecently, bool unpoppedRecovered ) : tag(tag), nothingPersistent(nothingPersistent), popped(popped), poppedRecently(poppedRecently), unpoppedRecovered(unpoppedRecovered), updateVersionSizes(tag != txsTag) {}
TagData(TagData&& r) noexcept(true) : version_messages(std::move(r.version_messages)), nothing_persistent(r.nothing_persistent), popped_recently(r.popped_recently), popped(r.popped), update_version_sizes(r.update_version_sizes), tag(r.tag), unpoppedRecovered(r.unpoppedRecovered) {}
TagData(TagData&& r) noexcept(true) : versionMessages(std::move(r.versionMessages)), nothingPersistent(r.nothingPersistent), poppedRecently(r.poppedRecently), popped(r.popped), updateVersionSizes(r.updateVersionSizes), tag(r.tag), unpoppedRecovered(r.unpoppedRecovered) {}
void operator= (TagData&& r) noexcept(true) {
version_messages = std::move(r.version_messages);
nothing_persistent = r.nothing_persistent;
popped_recently = r.popped_recently;
versionMessages = std::move(r.versionMessages);
nothingPersistent = r.nothingPersistent;
poppedRecently = r.poppedRecently;
popped = r.popped;
update_version_sizes = r.update_version_sizes;
updateVersionSizes = r.updateVersionSizes;
tag = r.tag;
unpoppedRecovered = r.unpoppedRecovered;
}
// Erase messages not needed to update *from* versions >= before (thus, messages with toversion <= before)
ACTOR Future<Void> eraseMessagesBefore( TagData *self, Version before, int64_t* gBytesErased, Reference<LogData> tlogData, int taskID ) {
while(!self->version_messages.empty() && self->version_messages.front().first < before) {
Version version = self->version_messages.front().first;
while(!self->versionMessages.empty() && self->versionMessages.front().first < before) {
Version version = self->versionMessages.front().first;
std::pair<int, int> &sizes = tlogData->version_sizes[version];
int64_t messagesErased = 0;
while(!self->version_messages.empty() && self->version_messages.front().first == version) {
auto const& m = self->version_messages.front();
while(!self->versionMessages.empty() && self->versionMessages.front().first == version) {
auto const& m = self->versionMessages.front();
++messagesErased;
if(self->update_version_sizes) {
if(self->updateVersionSizes) {
sizes.first -= m.second.expectedSize();
}
self->version_messages.pop_front();
self->versionMessages.pop_front();
}
int64_t bytesErased = messagesErased * SERVER_KNOBS->VERSION_MESSAGES_ENTRY_BYTES_WITH_OVERHEAD;
@ -336,7 +336,7 @@ struct LogData : NonCopyable, public ReferenceCounted<LogData> {
}
};
Map<Version, IDiskQueue::location> version_location; // For the version of each entry that was push()ed, the end location of the serialized bytes
Map<Version, IDiskQueue::location> versionLocation; // For the version of each entry that was push()ed, the end location of the serialized bytes
/*
Popped version tracking contract needed by log system to implement ILogCursor::popped():
@ -378,8 +378,8 @@ struct LogData : NonCopyable, public ReferenceCounted<LogData> {
}
//only callable after getTagData returns a null reference
Reference<TagData> createTagData(Tag tag, Version popped, bool nothing_persistent, bool popped_recently, bool unpoppedRecovered) {
Reference<TagData> newTagData = Reference<TagData>( new TagData(tag, popped, nothing_persistent, popped_recently, unpoppedRecovered) );
Reference<TagData> createTagData(Tag tag, Version popped, bool nothingPersistent, bool poppedRecently, bool unpoppedRecovered) {
Reference<TagData> newTagData = Reference<TagData>( new TagData(tag, popped, nothingPersistent, poppedRecently, unpoppedRecovered) );
int idx = tag.locality >= 0 ? 2*tag.locality : 1-(2*tag.locality);
tag_data[idx][tag.id] = newTagData;
return newTagData;
@ -408,7 +408,7 @@ struct LogData : NonCopyable, public ReferenceCounted<LogData> {
UID recruitmentID;
explicit LogData(TLogData* tLogData, TLogInterface interf, Tag remoteTag, bool isPrimary, int logRouterTags, UID recruitmentID) : tLogData(tLogData), knownCommittedVersion(1), logId(interf.id()),
cc("TLog", interf.id().toString()), bytesInput("bytesInput", cc), bytesDurable("bytesDurable", cc), remoteTag(remoteTag), isPrimary(isPrimary), logRouterTags(logRouterTags), recruitmentID(recruitmentID),
cc("TLog", interf.id().toString()), bytesInput("BytesInput", cc), bytesDurable("BytesDurable", cc), remoteTag(remoteTag), isPrimary(isPrimary), logRouterTags(logRouterTags), recruitmentID(recruitmentID),
logSystem(new AsyncVar<Reference<ILogSystem>>()), logRouterPoppedVersion(0), durableKnownCommittedVersion(0),
// These are initialized differently on init() or recovery
recoveryCount(), stopped(false), initialized(false), queueCommittingVersion(0), newPersistentDataVersion(invalidVersion), unrecoveredBefore(1), recoveredAt(1), unpoppedRecoveredTags(0),
@ -421,22 +421,22 @@ struct LogData : NonCopyable, public ReferenceCounted<LogData> {
version.initMetric(LiteralStringRef("TLog.Version"), cc.id);
queueCommittedVersion.initMetric(LiteralStringRef("TLog.QueueCommittedVersion"), cc.id);
specialCounter(cc, "version", [this](){ return this->version.get(); });
specialCounter(cc, "sharedBytesInput", [tLogData](){ return tLogData->bytesInput; });
specialCounter(cc, "sharedBytesDurable", [tLogData](){ return tLogData->bytesDurable; });
specialCounter(cc, "kvstoreBytesUsed", [tLogData](){ return tLogData->persistentData->getStorageBytes().used; });
specialCounter(cc, "kvstoreBytesFree", [tLogData](){ return tLogData->persistentData->getStorageBytes().free; });
specialCounter(cc, "kvstoreBytesAvailable", [tLogData](){ return tLogData->persistentData->getStorageBytes().available; });
specialCounter(cc, "kvstoreBytesTotal", [tLogData](){ return tLogData->persistentData->getStorageBytes().total; });
specialCounter(cc, "queueDiskBytesUsed", [tLogData](){ return tLogData->rawPersistentQueue->getStorageBytes().used; });
specialCounter(cc, "queueDiskBytesFree", [tLogData](){ return tLogData->rawPersistentQueue->getStorageBytes().free; });
specialCounter(cc, "queueDiskBytesAvailable", [tLogData](){ return tLogData->rawPersistentQueue->getStorageBytes().available; });
specialCounter(cc, "queueDiskBytesTotal", [tLogData](){ return tLogData->rawPersistentQueue->getStorageBytes().total; });
specialCounter(cc, "Version", [this](){ return this->version.get(); });
specialCounter(cc, "SharedBytesInput", [tLogData](){ return tLogData->bytesInput; });
specialCounter(cc, "SharedBytesDurable", [tLogData](){ return tLogData->bytesDurable; });
specialCounter(cc, "KvstoreBytesUsed", [tLogData](){ return tLogData->persistentData->getStorageBytes().used; });
specialCounter(cc, "KvstoreBytesFree", [tLogData](){ return tLogData->persistentData->getStorageBytes().free; });
specialCounter(cc, "KvstoreBytesAvailable", [tLogData](){ return tLogData->persistentData->getStorageBytes().available; });
specialCounter(cc, "KvstoreBytesTotal", [tLogData](){ return tLogData->persistentData->getStorageBytes().total; });
specialCounter(cc, "QueueDiskBytesUsed", [tLogData](){ return tLogData->rawPersistentQueue->getStorageBytes().used; });
specialCounter(cc, "QueueDiskBytesFree", [tLogData](){ return tLogData->rawPersistentQueue->getStorageBytes().free; });
specialCounter(cc, "QueueDiskBytesAvailable", [tLogData](){ return tLogData->rawPersistentQueue->getStorageBytes().available; });
specialCounter(cc, "QueueDiskBytesTotal", [tLogData](){ return tLogData->rawPersistentQueue->getStorageBytes().total; });
}
~LogData() {
tLogData->bytesDurable += bytesInput.getValue() - bytesDurable.getValue();
TraceEvent("TLogBytesWhenRemoved", logId).detail("sharedBytesInput", tLogData->bytesInput).detail("sharedBytesDurable", tLogData->bytesDurable).detail("localBytesInput", bytesInput.getValue()).detail("localBytesDurable", bytesDurable.getValue());
TraceEvent("TLogBytesWhenRemoved", logId).detail("SharedBytesInput", tLogData->bytesInput).detail("SharedBytesDurable", tLogData->bytesDurable).detail("LocalBytesInput", bytesInput.getValue()).detail("LocalBytesDurable", bytesDurable.getValue());
ASSERT_ABORT(tLogData->bytesDurable <= tLogData->bytesInput);
endRole(logId, "TLog", "Error", true);
@ -468,28 +468,28 @@ void TLogQueue::push( T const& qe, Reference<LogData> logData ) {
*(uint32_t*)wr.getData() = wr.getLength() - sizeof(uint32_t) - sizeof(uint8_t);
auto loc = queue->push( wr.toStringRef() );
//TraceEvent("TLogQueueVersionWritten", dbgid).detail("Size", wr.getLength() - sizeof(uint32_t) - sizeof(uint8_t)).detail("Loc", loc);
logData->version_location[qe.version] = loc;
logData->versionLocation[qe.version] = loc;
}
void TLogQueue::pop( Version upTo, Reference<LogData> logData ) {
// Keep only the given and all subsequent version numbers
// Find the first version >= upTo
auto v = logData->version_location.lower_bound(upTo);
if (v == logData->version_location.begin()) return;
auto v = logData->versionLocation.lower_bound(upTo);
if (v == logData->versionLocation.begin()) return;
if(v == logData->version_location.end()) {
v = logData->version_location.lastItem();
if(v == logData->versionLocation.end()) {
v = logData->versionLocation.lastItem();
}
else {
v.decrementNonEnd();
}
queue->pop( v->value );
logData->version_location.erase( logData->version_location.begin(), v ); // ... and then we erase that previous version and all prior versions
logData->versionLocation.erase( logData->versionLocation.begin(), v ); // ... and then we erase that previous version and all prior versions
}
void TLogQueue::updateVersionSizes( const TLogQueueEntry& result, TLogData* tLog ) {
auto it = tLog->id_data.find(result.id);
if(it != tLog->id_data.end()) {
it->second->version_location[result.version] = queue->getNextReadLocation();
it->second->versionLocation[result.version] = queue->getNextReadLocation();
}
}
@ -500,7 +500,7 @@ ACTOR Future<Void> tLogLock( TLogData* self, ReplyPromise< TLogLockResult > repl
TEST( logData->stopped );
TEST( !logData->stopped );
TraceEvent("TLogStop", logData->logId).detail("Ver", stopVersion).detail("isStopped", logData->stopped).detail("queueCommitted", logData->queueCommittedVersion.get());
TraceEvent("TLogStop", logData->logId).detail("Ver", stopVersion).detail("IsStopped", logData->stopped).detail("QueueCommitted", logData->queueCommittedVersion.get());
logData->stopped = true;
if(!logData->recoveryComplete.isSet()) {
@ -516,24 +516,24 @@ ACTOR Future<Void> tLogLock( TLogData* self, ReplyPromise< TLogLockResult > repl
result.end = stopVersion;
result.knownCommittedVersion = logData->knownCommittedVersion;
TraceEvent("TLogStop2", self->dbgid).detail("logId", logData->logId).detail("Ver", stopVersion).detail("isStopped", logData->stopped).detail("queueCommitted", logData->queueCommittedVersion.get()).detail("knownCommitted", result.knownCommittedVersion);
TraceEvent("TLogStop2", self->dbgid).detail("LogId", logData->logId).detail("Ver", stopVersion).detail("IsStopped", logData->stopped).detail("QueueCommitted", logData->queueCommittedVersion.get()).detail("KnownCommitted", result.knownCommittedVersion);
reply.send( result );
return Void();
}
void updatePersistentPopped( TLogData* self, Reference<LogData> logData, Reference<LogData::TagData> data ) {
if (!data->popped_recently) return;
if (!data->poppedRecently) return;
self->persistentData->set(KeyValueRef( persistTagPoppedKey(logData->logId, data->tag), persistTagPoppedValue(data->popped) ));
data->popped_recently = false;
data->poppedRecently = false;
if (data->nothing_persistent) return;
if (data->nothingPersistent) return;
self->persistentData->clear( KeyRangeRef(
persistTagMessagesKey( logData->logId, data->tag, Version(0) ),
persistTagMessagesKey( logData->logId, data->tag, data->popped ) ) );
if (data->popped > logData->persistentDataVersion)
data->nothing_persistent = true;
data->nothingPersistent = true;
}
ACTOR Future<Void> updatePersistentData( TLogData* self, Reference<LogData> logData, Version newPersistentDataVersion ) {
@ -543,30 +543,30 @@ ACTOR Future<Void> updatePersistentData( TLogData* self, Reference<LogData> logD
ASSERT( newPersistentDataVersion > logData->persistentDataVersion );
ASSERT( logData->persistentDataVersion == logData->persistentDataDurableVersion );
//TraceEvent("updatePersistentData", self->dbgid).detail("seq", newPersistentDataSeq);
//TraceEvent("UpdatePersistentData", self->dbgid).detail("Seq", newPersistentDataSeq);
state bool anyData = false;
// For all existing tags
state int tag_locality = 0;
state int tag_id = 0;
state int tagLocality = 0;
state int tagId = 0;
for(tag_locality = 0; tag_locality < logData->tag_data.size(); tag_locality++) {
for(tag_id = 0; tag_id < logData->tag_data[tag_locality].size(); tag_id++) {
state Reference<LogData::TagData> tagData = logData->tag_data[tag_locality][tag_id];
for(tagLocality = 0; tagLocality < logData->tag_data.size(); tagLocality++) {
for(tagId = 0; tagId < logData->tag_data[tagLocality].size(); tagId++) {
state Reference<LogData::TagData> tagData = logData->tag_data[tagLocality][tagId];
if(tagData) {
state Version currentVersion = 0;
// Clear recently popped versions from persistentData if necessary
updatePersistentPopped( self, logData, tagData );
// Transfer unpopped messages with version numbers less than newPersistentDataVersion to persistentData
state std::deque<std::pair<Version, LengthPrefixedStringRef>>::iterator msg = tagData->version_messages.begin();
while(msg != tagData->version_messages.end() && msg->first <= newPersistentDataVersion) {
state std::deque<std::pair<Version, LengthPrefixedStringRef>>::iterator msg = tagData->versionMessages.begin();
while(msg != tagData->versionMessages.end() && msg->first <= newPersistentDataVersion) {
currentVersion = msg->first;
anyData = true;
tagData->nothing_persistent = false;
tagData->nothingPersistent = false;
BinaryWriter wr( Unversioned() );
for(; msg != tagData->version_messages.end() && msg->first == currentVersion; ++msg)
for(; msg != tagData->versionMessages.end() && msg->first == currentVersion; ++msg)
wr << msg->second.toStringRef();
self->persistentData->set( KeyValueRef( persistTagMessagesKey( logData->logId, tagData->tag, currentVersion ), wr.toStringRef() ) );
@ -574,7 +574,7 @@ ACTOR Future<Void> updatePersistentData( TLogData* self, Reference<LogData> logD
Future<Void> f = yield(TaskUpdateStorage);
if(!f.isReady()) {
Void _ = wait(f);
msg = std::upper_bound(tagData->version_messages.begin(), tagData->version_messages.end(), std::make_pair(currentVersion, LengthPrefixedStringRef()), CompareFirst<std::pair<Version, LengthPrefixedStringRef>>());
msg = std::upper_bound(tagData->versionMessages.begin(), tagData->versionMessages.end(), std::make_pair(currentVersion, LengthPrefixedStringRef()), CompareFirst<std::pair<Version, LengthPrefixedStringRef>>());
}
}
@ -595,10 +595,10 @@ ACTOR Future<Void> updatePersistentData( TLogData* self, Reference<LogData> logD
TEST(anyData); // TLog moved data to persistentData
logData->persistentDataDurableVersion = newPersistentDataVersion;
for(tag_locality = 0; tag_locality < logData->tag_data.size(); tag_locality++) {
for(tag_id = 0; tag_id < logData->tag_data[tag_locality].size(); tag_id++) {
if(logData->tag_data[tag_locality][tag_id]) {
Void _ = wait(logData->tag_data[tag_locality][tag_id]->eraseMessagesBefore( newPersistentDataVersion+1, &self->bytesDurable, logData, TaskUpdateStorage ));
for(tagLocality = 0; tagLocality < logData->tag_data.size(); tagLocality++) {
for(tagId = 0; tagId < logData->tag_data[tagLocality].size(); tagId++) {
if(logData->tag_data[tagLocality][tagId]) {
Void _ = wait(logData->tag_data[tagLocality][tagId]->eraseMessagesBefore( newPersistentDataVersion+1, &self->bytesDurable, logData, TaskUpdateStorage ));
Void _ = wait(yield(TaskUpdateStorage));
}
}
@ -617,7 +617,7 @@ ACTOR Future<Void> updatePersistentData( TLogData* self, Reference<LogData> logD
}
if(logData->bytesDurable.getValue() > logData->bytesInput.getValue() || self->bytesDurable > self->bytesInput) {
TraceEvent(SevError, "BytesDurableTooLarge", logData->logId).detail("sharedBytesInput", self->bytesInput).detail("sharedBytesDurable", self->bytesDurable).detail("localBytesInput", logData->bytesInput.getValue()).detail("localBytesDurable", logData->bytesDurable.getValue());
TraceEvent(SevError, "BytesDurableTooLarge", logData->logId).detail("SharedBytesInput", self->bytesInput).detail("SharedBytesDurable", self->bytesDurable).detail("LocalBytesInput", logData->bytesInput.getValue()).detail("LocalBytesDurable", logData->bytesDurable.getValue());
}
ASSERT(logData->bytesDurable.getValue() <= logData->bytesInput.getValue());
@ -647,8 +647,8 @@ ACTOR Future<Void> updateStorage( TLogData* self ) {
state Version nextVersion = 0;
state int totalSize = 0;
state int tag_locality = 0;
state int tag_id = 0;
state int tagLocality = 0;
state int tagId = 0;
state Reference<LogData::TagData> tagData;
if(logData->stopped) {
@ -656,11 +656,11 @@ ACTOR Future<Void> updateStorage( TLogData* self ) {
while(logData->persistentDataDurableVersion != logData->version.get()) {
std::vector<std::pair<std::deque<std::pair<Version, LengthPrefixedStringRef>>::iterator, std::deque<std::pair<Version, LengthPrefixedStringRef>>::iterator>> iters;
for(tag_locality = 0; tag_locality < logData->tag_data.size(); tag_locality++) {
for(tag_id = 0; tag_id < logData->tag_data[tag_locality].size(); tag_id++) {
tagData = logData->tag_data[tag_locality][tag_id];
for(tagLocality = 0; tagLocality < logData->tag_data.size(); tagLocality++) {
for(tagId = 0; tagId < logData->tag_data[tagLocality].size(); tagId++) {
tagData = logData->tag_data[tagLocality][tagId];
if(tagData) {
iters.push_back(std::make_pair(tagData->version_messages.begin(), tagData->version_messages.end()));
iters.push_back(std::make_pair(tagData->versionMessages.begin(), tagData->versionMessages.end()));
}
}
}
@ -686,7 +686,7 @@ ACTOR Future<Void> updateStorage( TLogData* self ) {
Void _ = wait( logData->queueCommittedVersion.whenAtLeast( nextVersion ) );
Void _ = wait( delay(0, TaskUpdateStorage) );
//TraceEvent("TlogUpdatePersist", self->dbgid).detail("logId", logData->logId).detail("nextVersion", nextVersion).detail("version", logData->version.get()).detail("persistentDataDurableVer", logData->persistentDataDurableVersion).detail("queueCommitVer", logData->queueCommittedVersion.get()).detail("persistDataVer", logData->persistentDataVersion);
//TraceEvent("TlogUpdatePersist", self->dbgid).detail("LogId", logData->logId).detail("NextVersion", nextVersion).detail("Version", logData->version.get()).detail("PersistentDataDurableVer", logData->persistentDataDurableVersion).detail("QueueCommitVer", logData->queueCommittedVersion.get()).detail("PersistDataVer", logData->persistentDataVersion);
if (nextVersion > logData->persistentDataVersion) {
self->updatePersist = updatePersistentData(self, logData, nextVersion);
Void _ = wait( self->updatePersist );
@ -718,12 +718,12 @@ ACTOR Future<Void> updateStorage( TLogData* self ) {
++sizeItr;
nextVersion = sizeItr == logData->version_sizes.end() ? logData->version.get() : sizeItr->key;
for(tag_locality = 0; tag_locality < logData->tag_data.size(); tag_locality++) {
for(tag_id = 0; tag_id < logData->tag_data[tag_locality].size(); tag_id++) {
tagData = logData->tag_data[tag_locality][tag_id];
for(tagLocality = 0; tagLocality < logData->tag_data.size(); tagLocality++) {
for(tagId = 0; tagId < logData->tag_data[tagLocality].size(); tagId++) {
tagData = logData->tag_data[tagLocality][tagId];
if(tagData) {
auto it = std::lower_bound(tagData->version_messages.begin(), tagData->version_messages.end(), std::make_pair(prevVersion, LengthPrefixedStringRef()), CompareFirst<std::pair<Version, LengthPrefixedStringRef>>());
for(; it != tagData->version_messages.end() && it->first < nextVersion; ++it) {
auto it = std::lower_bound(tagData->versionMessages.begin(), tagData->versionMessages.end(), std::make_pair(prevVersion, LengthPrefixedStringRef()), CompareFirst<std::pair<Version, LengthPrefixedStringRef>>());
for(; it != tagData->versionMessages.end() && it->first < nextVersion; ++it) {
totalSize += it->second.expectedSize();
}
@ -737,7 +737,7 @@ ACTOR Future<Void> updateStorage( TLogData* self ) {
nextVersion = std::max<Version>(nextVersion, logData->persistentDataVersion);
//TraceEvent("UpdateStorageVer", logData->logId).detail("nextVersion", nextVersion).detail("persistentDataVersion", logData->persistentDataVersion).detail("totalSize", totalSize);
//TraceEvent("UpdateStorageVer", logData->logId).detail("NextVersion", nextVersion).detail("PersistentDataVersion", logData->persistentDataVersion).detail("TotalSize", totalSize);
Void _ = wait( logData->queueCommittedVersion.whenAtLeast( nextVersion ) );
Void _ = wait( delay(0, TaskUpdateStorage) );
@ -820,12 +820,12 @@ void commitMessages( Reference<LogData> self, Version version, const std::vector
}
if (version >= tagData->popped) {
tagData->version_messages.push_back(std::make_pair(version, LengthPrefixedStringRef((uint32_t*)(block.end() - msg.message.size()))));
if(tagData->version_messages.back().second.expectedSize() > SERVER_KNOBS->MAX_MESSAGE_SIZE) {
TraceEvent(SevWarnAlways, "LargeMessage").detail("Size", tagData->version_messages.back().second.expectedSize());
tagData->versionMessages.push_back(std::make_pair(version, LengthPrefixedStringRef((uint32_t*)(block.end() - msg.message.size()))));
if(tagData->versionMessages.back().second.expectedSize() > SERVER_KNOBS->MAX_MESSAGE_SIZE) {
TraceEvent(SevWarnAlways, "LargeMessage").detail("Size", tagData->versionMessages.back().second.expectedSize());
}
if (tag != txsTag) {
expectedBytes += tagData->version_messages.back().second.expectedSize();
expectedBytes += tagData->versionMessages.back().second.expectedSize();
}
// The factor of VERSION_MESSAGES_OVERHEAD is intended to be an overestimate of the actual memory used to store this data in a std::deque.
@ -845,7 +845,7 @@ void commitMessages( Reference<LogData> self, Version version, const std::vector
self->bytesInput += addedBytes;
bytesInput += addedBytes;
//TraceEvent("TLogPushed", self->dbgid).detail("Bytes", addedBytes).detail("MessageBytes", messages.size()).detail("Tags", tags.size()).detail("expectedBytes", expectedBytes).detail("mCount", mCount).detail("tCount", tCount);
//TraceEvent("TLogPushed", self->dbgid).detail("Bytes", addedBytes).detail("MessageBytes", messages.size()).detail("Tags", tags.size()).detail("ExpectedBytes", expectedBytes).detail("MCount", mCount).detail("TCount", tCount);
}
void commitMessages( Reference<LogData> self, Version version, Arena arena, StringRef messages, int64_t& bytesInput ) {
@ -878,13 +878,13 @@ Version poppedVersion( Reference<LogData> self, Tag tag) {
return tagData->popped;
}
std::deque<std::pair<Version, LengthPrefixedStringRef>> & get_version_messages( Reference<LogData> self, Tag tag ) {
std::deque<std::pair<Version, LengthPrefixedStringRef>> & getVersionMessages( Reference<LogData> self, Tag tag ) {
auto tagData = self->getTagData(tag);
if (!tagData) {
static std::deque<std::pair<Version, LengthPrefixedStringRef>> empty;
return empty;
}
return tagData->version_messages;
return tagData->versionMessages;
};
ACTOR Future<Void> tLogPop( TLogData* self, TLogPopRequest req, Reference<LogData> logData ) {
@ -893,12 +893,12 @@ ACTOR Future<Void> tLogPop( TLogData* self, TLogPopRequest req, Reference<LogDat
tagData = logData->createTagData(req.tag, req.to, true, true, false);
} else if (req.to > tagData->popped) {
tagData->popped = req.to;
tagData->popped_recently = true;
tagData->poppedRecently = true;
if(tagData->unpoppedRecovered && req.to > logData->recoveredAt) {
tagData->unpoppedRecovered = false;
logData->unpoppedRecoveredTags--;
TraceEvent("TLogPoppedTag", logData->logId).detail("tags", logData->unpoppedRecoveredTags).detail("tag", req.tag.toString()).detail("durableKCVer", logData->durableKnownCommittedVersion).detail("recoveredAt", logData->recoveredAt);
TraceEvent("TLogPoppedTag", logData->logId).detail("Tags", logData->unpoppedRecoveredTags).detail("Tag", req.tag.toString()).detail("DurableKCVer", logData->durableKnownCommittedVersion).detail("RecoveredAt", logData->recoveredAt);
if(logData->unpoppedRecoveredTags == 0 && logData->durableKnownCommittedVersion >= logData->recoveredAt && logData->recoveryComplete.canBeSet()) {
logData->recoveryComplete.send(Void());
}
@ -916,8 +916,8 @@ ACTOR Future<Void> tLogPop( TLogData* self, TLogPopRequest req, Reference<LogDat
void peekMessagesFromMemory( Reference<LogData> self, TLogPeekRequest const& req, BinaryWriter& messages, Version& endVersion ) {
ASSERT( !messages.getLength() );
auto& deque = get_version_messages(self, req.tag);
//TraceEvent("tLogPeekMem", self->dbgid).detail("Tag", printable(req.tag1)).detail("pDS", self->persistentDataSequence).detail("pDDS", self->persistentDataDurableSequence).detail("Oldest", map1.empty() ? 0 : map1.begin()->key ).detail("OldestMsgCount", map1.empty() ? 0 : map1.begin()->value.size());
auto& deque = getVersionMessages(self, req.tag);
//TraceEvent("TLogPeekMem", self->dbgid).detail("Tag", printable(req.tag1)).detail("PDS", self->persistentDataSequence).detail("PDDS", self->persistentDataDurableSequence).detail("Oldest", map1.empty() ? 0 : map1.begin()->key ).detail("OldestMsgCount", map1.empty() ? 0 : map1.begin()->value.size());
Version begin = std::max( req.begin, self->persistentDataDurableVersion+1 );
auto it = std::lower_bound(deque.begin(), deque.end(), std::make_pair(begin, LengthPrefixedStringRef()), CompareFirst<std::pair<Version, LengthPrefixedStringRef>>());
@ -927,7 +927,7 @@ void peekMessagesFromMemory( Reference<LogData> self, TLogPeekRequest const& req
if(it->first != currentVersion) {
if (messages.getLength() >= SERVER_KNOBS->DESIRED_TOTAL_BYTES) {
endVersion = currentVersion + 1;
//TraceEvent("tLogPeekMessagesReached2", self->dbgid);
//TraceEvent("TLogPeekMessagesReached2", self->dbgid);
break;
}
@ -971,7 +971,7 @@ ACTOR Future<Void> tLogPeekMessages( TLogData* self, TLogPeekRequest req, Refere
return Void();
}
//TraceEvent("tLogPeekMessages0", self->dbgid).detail("reqBeginEpoch", req.begin.epoch).detail("reqBeginSeq", req.begin.sequence).detail("epoch", self->epoch()).detail("persistentDataSeq", self->persistentDataSequence).detail("Tag1", printable(req.tag1)).detail("Tag2", printable(req.tag2));
//TraceEvent("TLogPeekMessages0", self->dbgid).detail("ReqBeginEpoch", req.begin.epoch).detail("ReqBeginSeq", req.begin.sequence).detail("Epoch", self->epoch()).detail("PersistentDataSeq", self->persistentDataSequence).detail("Tag1", printable(req.tag1)).detail("Tag2", printable(req.tag2));
// Wait until we have something to return that the caller doesn't already have
if( logData->version.get() < req.begin ) {
Void _ = wait( logData->version.whenAtLeast( req.begin ) );
@ -1007,7 +1007,7 @@ ACTOR Future<Void> tLogPeekMessages( TLogData* self, TLogPeekRequest req, Refere
state Version endVersion = logData->version.get() + 1;
//grab messages from disk
//TraceEvent("tLogPeekMessages", self->dbgid).detail("reqBeginEpoch", req.begin.epoch).detail("reqBeginSeq", req.begin.sequence).detail("epoch", self->epoch()).detail("persistentDataSeq", self->persistentDataSequence).detail("Tag1", printable(req.tag1)).detail("Tag2", printable(req.tag2));
//TraceEvent("TLogPeekMessages", self->dbgid).detail("ReqBeginEpoch", req.begin.epoch).detail("ReqBeginSeq", req.begin.sequence).detail("Epoch", self->epoch()).detail("PersistentDataSeq", self->persistentDataSequence).detail("Tag1", printable(req.tag1)).detail("Tag2", printable(req.tag2));
if( req.begin <= logData->persistentDataDurableVersion ) {
// Just in case the durable version changes while we are waiting for the read, we grab this data from memory. We may or may not actually send it depending on
// whether we get enough data from disk.
@ -1043,7 +1043,7 @@ ACTOR Future<Void> tLogPeekMessages( TLogData* self, TLogPeekRequest req, Refere
reply.messages = messages.toStringRef();
reply.end = endVersion;
//TraceEvent("TlogPeek", self->dbgid).detail("logId", logData->logId).detail("endVer", reply.end).detail("msgBytes", reply.messages.expectedSize()).detail("ForAddress", req.reply.getEndpoint().address);
//TraceEvent("TlogPeek", self->dbgid).detail("LogId", logData->logId).detail("EndVer", reply.end).detail("MsgBytes", reply.messages.expectedSize()).detail("ForAddress", req.reply.getEndpoint().address);
if(req.sequence.present()) {
auto& trackerData = self->peekTracker[peekId];
@ -1087,7 +1087,7 @@ ACTOR Future<Void> doQueueCommit( TLogData* self, Reference<LogData> logData ) {
logData->durableKnownCommittedVersion = knownCommittedVersion;
if(logData->unpoppedRecoveredTags == 0 && knownCommittedVersion >= logData->recoveredAt && logData->recoveryComplete.canBeSet()) {
TraceEvent("TLogRecoveryComplete", logData->logId).detail("tags", logData->unpoppedRecoveredTags).detail("durableKCVer", logData->durableKnownCommittedVersion).detail("recoveredAt", logData->recoveredAt);
TraceEvent("TLogRecoveryComplete", logData->logId).detail("Tags", logData->unpoppedRecoveredTags).detail("DurableKCVer", logData->durableKnownCommittedVersion).detail("RecoveredAt", logData->recoveredAt);
logData->recoveryComplete.send(Void());
}
@ -1121,7 +1121,7 @@ ACTOR Future<Void> commitQueue( TLogData* self ) {
continue;
}
TraceEvent("commitQueueNewLog", self->dbgid).detail("logId", logData->logId).detail("version", logData->version.get()).detail("committing", logData->queueCommittingVersion).detail("commmitted", logData->queueCommittedVersion.get());
TraceEvent("CommitQueueNewLog", self->dbgid).detail("LogId", logData->logId).detail("Version", logData->version.get()).detail("Committing", logData->queueCommittingVersion).detail("Commmitted", logData->queueCommittedVersion.get());
if(logData->committingQueue.canBeSet()) {
logData->committingQueue.send(Void());
}
@ -1268,8 +1268,8 @@ ACTOR Future<Void> rejoinMasters( TLogData* self, TLogInterface tli, DBRecoveryC
}
if ( isDisplaced )
{
TraceEvent("TLogDisplaced", tli.id()).detail("Reason", "DBInfoDoesNotContain").detail("recoveryCount", recoveryCount).detail("infRecoveryCount", inf.recoveryCount).detail("recoveryState", inf.recoveryState)
.detail("logSysConf", describe(inf.logSystemConfig.tLogs)).detail("priorLogs", describe(inf.priorCommittedLogServers)).detail("oldLogGens", inf.logSystemConfig.oldTLogs.size());
TraceEvent("TLogDisplaced", tli.id()).detail("Reason", "DBInfoDoesNotContain").detail("RecoveryCount", recoveryCount).detail("InfRecoveryCount", inf.recoveryCount).detail("RecoveryState", inf.recoveryState)
.detail("LogSysConf", describe(inf.logSystemConfig.tLogs)).detail("PriorLogs", describe(inf.priorCommittedLogServers)).detail("OldLogGens", inf.logSystemConfig.oldTLogs.size());
if (BUGGIFY) Void _ = wait( delay( SERVER_KNOBS->BUGGIFY_WORKER_REMOVED_MAX_LAG * g_random->random01() ) );
throw worker_removed();
}
@ -1305,7 +1305,7 @@ ACTOR Future<Void> respondToRecovered( TLogInterface tli, Promise<Void> recovery
}
finishedRecovery = false;
}
TraceEvent("TLogRespondToRecovered", tli.id()).detail("finished", finishedRecovery);
TraceEvent("TLogRespondToRecovered", tli.id()).detail("Finished", finishedRecovery);
loop {
TLogRecoveryFinishedRequest req = waitNext( tli.recoveryFinished.getFuture() );
if(finishedRecovery) {
@ -1386,7 +1386,7 @@ ACTOR Future<Void> serveTLogInterface( TLogData* self, TLogInterface tli, Refere
logData->addActor.send( tLogPop( self, req, logData ) );
}
when( TLogCommitRequest req = waitNext( tli.commit.getFuture() ) ) {
//TraceEvent("TLogCommitReq", logData->logId).detail("ver", req.version).detail("prevVer", req.prevVersion).detail("logVer", logData->version.get());
//TraceEvent("TLogCommitReq", logData->logId).detail("Ver", req.version).detail("PrevVer", req.prevVersion).detail("LogVer", logData->version.get());
ASSERT(logData->isPrimary);
TEST(logData->stopped); // TLogCommitRequest while stopped
if (!logData->stopped)
@ -1415,7 +1415,7 @@ ACTOR Future<Void> serveTLogInterface( TLogData* self, TLogInterface tli, Refere
}
void removeLog( TLogData* self, Reference<LogData> logData ) {
TraceEvent("TLogRemoved", logData->logId).detail("input", logData->bytesInput.getValue()).detail("durable", logData->bytesDurable.getValue());
TraceEvent("TLogRemoved", logData->logId).detail("Input", logData->bytesInput.getValue()).detail("Durable", logData->bytesDurable.getValue());
logData->stopped = true;
if(!logData->recoveryComplete.isSet()) {
logData->recoveryComplete.sendError(end_of_stream());
@ -1702,7 +1702,7 @@ ACTOR Future<Void> restorePersistentState( TLogData* self, LocalityData locality
logData->removed = rejoinMasters(self, recruited, logData->recoveryCount, registerWithMaster.getFuture(), false);
removed.push_back(errorOr(logData->removed));
TraceEvent("TLogRestorePersistentStateVer", id1).detail("ver", ver);
TraceEvent("TLogRestorePersistentStateVer", id1).detail("Ver", ver);
// Restore popped keys. Pop operations that took place after the last (committed) updatePersistentDataVersion might be lost, but
// that is fine because we will get the corresponding data back, too.
@ -1748,8 +1748,8 @@ ACTOR Future<Void> restorePersistentState( TLogData* self, LocalityData locality
}
}
//TraceEvent("TLogRecoveredQE", self->dbgid).detail("logId", qe.id).detail("ver", qe.version).detail("MessageBytes", qe.messages.size()).detail("Tags", qe.tags.size())
// .detail("Tag0", qe.tags.size() ? qe.tags[0].tag : invalidTag).detail("version", logData->version.get());
//TraceEvent("TLogRecoveredQE", self->dbgid).detail("LogId", qe.id).detail("Ver", qe.version).detail("MessageBytes", qe.messages.size()).detail("Tags", qe.tags.size())
// .detail("Tag0", qe.tags.size() ? qe.tags[0].tag : invalidTag).detail("Version", logData->version.get());
if(logData) {
logData->knownCommittedVersion = std::max(logData->knownCommittedVersion, qe.knownCommittedVersion);
@ -1782,7 +1782,7 @@ ACTOR Future<Void> restorePersistentState( TLogData* self, LocalityData locality
for(auto it : self->id_data) {
if(it.second->queueCommittedVersion.get() == 0) {
TraceEvent("TLogZeroVersion", self->dbgid).detail("logId", it.first);
TraceEvent("TLogZeroVersion", self->dbgid).detail("LogId", it.first);
it.second->queueCommittedVersion.set(it.second->version.get());
}
it.second->recoveryComplete.sendError(end_of_stream());
@ -1835,9 +1835,9 @@ ACTOR Future<Void> updateLogSystem(TLogData* self, Reference<LogData> logData, L
} else {
logData->logSystem->get()->pop(logData->logRouterPoppedVersion, logData->remoteTag, logData->durableKnownCommittedVersion, logData->locality);
}
TraceEvent("TLogUpdate", self->dbgid).detail("logId", logData->logId).detail("recruitmentID", logData->recruitmentID).detail("dbRecruitmentID", self->dbInfo->get().logSystemConfig.recruitmentID).detail("recoverFrom", recoverFrom.toString()).detail("dbInfo", self->dbInfo->get().logSystemConfig.toString()).detail("found", found).detail("logSystem", (bool) logSystem->get() ).detail("recoveryState", self->dbInfo->get().recoveryState);
TraceEvent("TLogUpdate", self->dbgid).detail("LogId", logData->logId).detail("RecruitmentID", logData->recruitmentID).detail("DbRecruitmentID", self->dbInfo->get().logSystemConfig.recruitmentID).detail("RecoverFrom", recoverFrom.toString()).detail("DbInfo", self->dbInfo->get().logSystemConfig.toString()).detail("Found", found).detail("LogSystem", (bool) logSystem->get() ).detail("RecoveryState", self->dbInfo->get().recoveryState);
for(auto it : self->dbInfo->get().logSystemConfig.oldTLogs) {
TraceEvent("TLogUpdateOld", self->dbgid).detail("logId", logData->logId).detail("dbInfo", it.toString());
TraceEvent("TLogUpdateOld", self->dbgid).detail("LogId", logData->logId).detail("DbInfo", it.toString());
}
Void _ = wait( self->dbInfo->onChange() );
}
@ -1857,7 +1857,7 @@ ACTOR Future<Void> tLogStart( TLogData* self, InitializeTLogRequest req, Localit
for(auto it : self->id_data) {
if( !it.second->stopped ) {
TraceEvent("TLogStoppedByNewRecruitment", self->dbgid).detail("stoppedId", it.first.toString()).detail("recruitedId", recruited.id()).detail("endEpoch", it.second->logSystem->get().getPtr() != 0);
TraceEvent("TLogStoppedByNewRecruitment", self->dbgid).detail("StoppedId", it.first.toString()).detail("RecruitedId", recruited.id()).detail("EndEpoch", it.second->logSystem->get().getPtr() != 0);
if(!it.second->isPrimary && it.second->logSystem->get()) {
it.second->removed = it.second->removed && it.second->logSystem->get()->endEpoch();
}
@ -1898,7 +1898,7 @@ ACTOR Future<Void> tLogStart( TLogData* self, InitializeTLogRequest req, Localit
logData->unpoppedRecoveredTags = req.allTags.size();
Void _ = wait( initPersistentState( self, logData, req.allTags ) || logData->removed );
TraceEvent("TLogRecover", self->dbgid).detail("logId", logData->logId).detail("at", req.recoverAt).detail("known", req.knownCommittedVersion).detail("unrecovered", logData->unrecoveredBefore).detail("tags", describe(req.recoverTags)).detail("locality", req.locality).detail("logRouterTags", logData->logRouterTags);
TraceEvent("TLogRecover", self->dbgid).detail("LogId", logData->logId).detail("At", req.recoverAt).detail("Known", req.knownCommittedVersion).detail("Unrecovered", logData->unrecoveredBefore).detail("Tags", describe(req.recoverTags)).detail("Locality", req.locality).detail("LogRouterTags", logData->logRouterTags);
if(logData->recoveryComplete.isSet()) {
throw worker_removed();
@ -1916,7 +1916,7 @@ ACTOR Future<Void> tLogStart( TLogData* self, InitializeTLogRequest req, Localit
Void _ = wait(pullAsyncData(self, logData, tags, logData->unrecoveredBefore, req.knownCommittedVersion, true) || logData->removed);
}
TraceEvent("TLogPullComplete", self->dbgid).detail("logId", logData->logId);
TraceEvent("TLogPullComplete", self->dbgid).detail("LogId", logData->logId);
if(req.isPrimary && !req.recoverTags.empty() && !logData->stopped && req.knownCommittedVersion < req.recoverAt) {
Void _ = wait(pullAsyncData(self, logData, req.recoverTags, req.knownCommittedVersion + 1, req.recoverAt, false) || logData->removed);
@ -1948,11 +1948,11 @@ ACTOR Future<Void> tLogStart( TLogData* self, InitializeTLogRequest req, Localit
//so we need to pop all tags that did not have data at the recovery version.
std::vector<Future<Void>> popFutures;
std::set<Tag> allTags(req.allTags.begin(), req.allTags.end());
for(int tag_locality = 0; tag_locality < logData->tag_data.size(); tag_locality++) {
for(int tag_id = 0; tag_id < logData->tag_data[tag_locality].size(); tag_id++) {
auto data = logData->tag_data[tag_locality][tag_id];
for(int tagLocality = 0; tagLocality < logData->tag_data.size(); tagLocality++) {
for(int tagId = 0; tagId < logData->tag_data[tagLocality].size(); tagId++) {
auto data = logData->tag_data[tagLocality][tagId];
if(data && !allTags.count(data->tag) && data->tag.locality != tagLocalityLogRouter) {
TraceEvent("TLogPopOnRecover", self->dbgid).detail("logId", logData->logId).detail("tag", data->tag.toString()).detail("ver", req.recoverAt);
TraceEvent("TLogPopOnRecover", self->dbgid).detail("LogId", logData->logId).detail("Tag", data->tag.toString()).detail("Ver", req.recoverAt);
popFutures.push_back(tLogPop(self, TLogPopRequest(req.recoverAt, 0, data->tag), logData));
}
}
@ -1960,7 +1960,7 @@ ACTOR Future<Void> tLogStart( TLogData* self, InitializeTLogRequest req, Localit
Void _ = wait(waitForAll(popFutures));
TraceEvent("TLogPull2Complete", self->dbgid).detail("logId", logData->logId);
TraceEvent("TLogPull2Complete", self->dbgid).detail("LogId", logData->logId);
logData->addActor.send( respondToRecovered( recruited, logData->recoveryComplete ) );
} else {
// Brand new tlog, initialization has already been done by caller
@ -1993,7 +1993,7 @@ ACTOR Future<Void> tLogStart( TLogData* self, InitializeTLogRequest req, Localit
req.reply.send( recruited );
TraceEvent("TLogReady", logData->logId).detail("allTags", describe(req.allTags));
TraceEvent("TLogReady", logData->logId).detail("AllTags", describe(req.allTags));
updater = Void();
Void _ = wait( tLogCore( self, logData, recruited ) );

76
fdbserver/TagPartitionedLogSystem.actor.cpp
View File

@ -428,7 +428,7 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
}
if(begin >= lastBegin) {
TraceEvent("TLogPeekAllCurrentOnly", dbgid).detail("tag", tag.toString()).detail("begin", begin).detail("end", end).detail("bestLogs", bestSet >= 0 ? localSets[bestSet]->logServerString() : "no best set");
TraceEvent("TLogPeekAllCurrentOnly", dbgid).detail("Tag", tag.toString()).detail("Begin", begin).detail("End", end).detail("BestLogs", bestSet >= 0 ? localSets[bestSet]->logServerString() : "no best set");
return Reference<ILogSystem::SetPeekCursor>( new ILogSystem::SetPeekCursor( localSets, bestSet == -1 ? nextBestSet : bestSet,
bestSet >= 0 ? localSets[bestSet]->bestLocationFor( tag ) : -1, tag, begin, end, parallelGetMore ) );
} else {
@ -436,7 +436,7 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
std::vector< LogMessageVersion > epochEnds;
if(lastBegin < end) {
TraceEvent("TLogPeekAllAddingCurrent", dbgid).detail("tag", tag.toString()).detail("begin", begin).detail("end", end).detail("bestLogs", bestSet >= 0 ? localSets[bestSet]->logServerString() : "no best set");
TraceEvent("TLogPeekAllAddingCurrent", dbgid).detail("Tag", tag.toString()).detail("Begin", begin).detail("End", end).detail("BestLogs", bestSet >= 0 ? localSets[bestSet]->logServerString() : "no best set");
cursors.push_back( Reference<ILogSystem::SetPeekCursor>( new ILogSystem::SetPeekCursor( localSets, bestSet == -1 ? nextBestSet : bestSet,
bestSet >= 0 ? localSets[bestSet]->bestLocationFor( tag ) : -1, tag, lastBegin, end, parallelGetMore)) );
}
@ -446,7 +446,7 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
if(tag == txsTag) {
break;
}
TraceEvent("TLogPeekAllDead", dbgid).detail("tag", tag.toString()).detail("begin", begin).detail("end", end).detail("lastBegin", lastBegin).detail("oldLogDataSize", oldLogData.size());
TraceEvent("TLogPeekAllDead", dbgid).detail("Tag", tag.toString()).detail("Begin", begin).detail("End", end).detail("LastBegin", lastBegin).detail("OldLogDataSize", oldLogData.size());
if(throwIfDead) {
throw worker_removed();
} else {
@ -473,8 +473,8 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
}
if(thisBegin < lastBegin) {
if(thisBegin < end) {
TraceEvent("TLogPeekAllAddingOld", dbgid).detail("tag", tag.toString()).detail("begin", begin).detail("end", end)
.detail("bestLogs", bestOldSet >= 0 ? localOldSets[bestOldSet]->logServerString() : "no best set").detail("lastBegin", lastBegin).detail("thisBegin", thisBegin);
TraceEvent("TLogPeekAllAddingOld", dbgid).detail("Tag", tag.toString()).detail("Begin", begin).detail("End", end)
.detail("BestLogs", bestOldSet >= 0 ? localOldSets[bestOldSet]->logServerString() : "no best set").detail("LastBegin", lastBegin).detail("ThisBegin", thisBegin);
cursors.push_back( Reference<ILogSystem::SetPeekCursor>( new ILogSystem::SetPeekCursor( localOldSets, bestOldSet == -1 ? nextBestOldSet : bestOldSet,
bestOldSet >= 0 ? localOldSets[bestOldSet]->bestLocationFor( tag ) : -1, tag, thisBegin, std::min(lastBegin, end), parallelGetMore)) );
epochEnds.push_back(LogMessageVersion(std::min(lastBegin, end)));
@ -502,21 +502,21 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
}
}
if(bestSet == -1) {
TraceEvent("TLogPeekRemoteNoBestSet", dbgid).detail("tag", tag.toString()).detail("begin", begin);
TraceEvent("TLogPeekRemoteNoBestSet", dbgid).detail("Tag", tag.toString()).detail("Begin", begin);
return Reference<ILogSystem::ServerPeekCursor>( new ILogSystem::ServerPeekCursor( Reference<AsyncVar<OptionalInterface<TLogInterface>>>(), tag, begin, getPeekEnd(), false, false ) );
}
if(begin >= lastBegin) {
TraceEvent("TLogPeekRemoteBestOnly", dbgid).detail("tag", tag.toString()).detail("begin", begin).detail("bestSet", bestSet).detail("bestSetStart", lastBegin).detail("logRouterIds", tLogs[bestSet]->logRouterString());
TraceEvent("TLogPeekRemoteBestOnly", dbgid).detail("Tag", tag.toString()).detail("Begin", begin).detail("BestSet", bestSet).detail("BestSetStart", lastBegin).detail("LogRouterIds", tLogs[bestSet]->logRouterString());
return Reference<ILogSystem::MergedPeekCursor>( new ILogSystem::MergedPeekCursor( tLogs[bestSet]->logRouters, -1, (int)tLogs[bestSet]->logRouters.size(), tag, begin, getPeekEnd(), false, std::vector<LocalityData>(), IRepPolicyRef(), 0 ) );
} else {
std::vector< Reference<ILogSystem::IPeekCursor> > cursors;
std::vector< LogMessageVersion > epochEnds;
TraceEvent("TLogPeekRemoteAddingBest", dbgid).detail("tag", tag.toString()).detail("begin", begin).detail("bestSet", bestSet).detail("bestSetStart", lastBegin).detail("logRouterIds", tLogs[bestSet]->logRouterString());
TraceEvent("TLogPeekRemoteAddingBest", dbgid).detail("Tag", tag.toString()).detail("Begin", begin).detail("BestSet", bestSet).detail("BestSetStart", lastBegin).detail("LogRouterIds", tLogs[bestSet]->logRouterString());
cursors.push_back( Reference<ILogSystem::MergedPeekCursor>( new ILogSystem::MergedPeekCursor( tLogs[bestSet]->logRouters, -1, (int)tLogs[bestSet]->logRouters.size(), tag, lastBegin, getPeekEnd(), false, std::vector<LocalityData>(), IRepPolicyRef(), 0 ) ) );
int i = 0;
while(begin < lastBegin) {
if(i == oldLogData.size()) {
TraceEvent("TLogPeekRemoteDead", dbgid).detail("tag", tag.toString()).detail("begin", begin).detail("lastBegin", lastBegin).detail("oldLogDataSize", oldLogData.size());
TraceEvent("TLogPeekRemoteDead", dbgid).detail("Tag", tag.toString()).detail("Begin", begin).detail("LastBegin", lastBegin).detail("OldLogDataSize", oldLogData.size());
return Reference<ILogSystem::ServerPeekCursor>( new ILogSystem::ServerPeekCursor( Reference<AsyncVar<OptionalInterface<TLogInterface>>>(), tag, begin, getPeekEnd(), false, false ) );
}
@ -533,13 +533,13 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
}
}
if(bestOldSet == -1) {
TraceEvent("TLogPeekRemoteNoOldBestSet", dbgid).detail("tag", tag.toString()).detail("begin", begin);
TraceEvent("TLogPeekRemoteNoOldBestSet", dbgid).detail("Tag", tag.toString()).detail("Begin", begin);
return Reference<ILogSystem::ServerPeekCursor>( new ILogSystem::ServerPeekCursor( Reference<AsyncVar<OptionalInterface<TLogInterface>>>(), tag, begin, getPeekEnd(), false, false ) );
}
if(thisBegin < lastBegin) {
TraceEvent("TLogPeekRemoteAddingOldBest", dbgid).detail("tag", tag.toString()).detail("begin", begin).detail("bestOldSet", bestOldSet).detail("logRouterIds", oldLogData[i].tLogs[bestOldSet]->logRouterString())
.detail("lastBegin", lastBegin).detail("thisBegin", thisBegin).detail("bestStartVer", oldLogData[i].tLogs[bestOldSet]->startVersion);
TraceEvent("TLogPeekRemoteAddingOldBest", dbgid).detail("Tag", tag.toString()).detail("Begin", begin).detail("BestOldSet", bestOldSet).detail("LogRouterIds", oldLogData[i].tLogs[bestOldSet]->logRouterString())
.detail("LastBegin", lastBegin).detail("ThisBegin", thisBegin).detail("BestStartVer", oldLogData[i].tLogs[bestOldSet]->startVersion);
cursors.push_back( Reference<ILogSystem::MergedPeekCursor>( new ILogSystem::MergedPeekCursor( oldLogData[i].tLogs[bestOldSet]->logRouters, -1, (int)oldLogData[i].tLogs[bestOldSet]->logRouters.size(), tag,
thisBegin, lastBegin, false, std::vector<LocalityData>(), IRepPolicyRef(), 0 ) ) );
epochEnds.push_back(LogMessageVersion(lastBegin));
@ -554,7 +554,7 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
virtual Reference<IPeekCursor> peek( UID dbgid, Version begin, Tag tag, bool parallelGetMore ) {
if(!tLogs.size()) {
TraceEvent("TLogPeekNoLogSets", dbgid).detail("tag", tag.toString()).detail("begin", begin);
TraceEvent("TLogPeekNoLogSets", dbgid).detail("Tag", tag.toString()).detail("Begin", begin);
return Reference<ILogSystem::ServerPeekCursor>( new ILogSystem::ServerPeekCursor( Reference<AsyncVar<OptionalInterface<TLogInterface>>>(), tag, begin, getPeekEnd(), false, false ) );
}
@ -567,7 +567,7 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
virtual Reference<IPeekCursor> peek( UID dbgid, Version begin, std::vector<Tag> tags, bool parallelGetMore ) {
if(tags.empty()) {
TraceEvent("TLogPeekNoTags", dbgid).detail("begin", begin);
TraceEvent("TLogPeekNoTags", dbgid).detail("Begin", begin);
return Reference<ILogSystem::ServerPeekCursor>( new ILogSystem::ServerPeekCursor( Reference<AsyncVar<OptionalInterface<TLogInterface>>>(), invalidTag, begin, getPeekEnd(), false, false ) );
}
@ -595,26 +595,26 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
}
}
if(bestSet == -1) {
TraceEvent("TLogPeekLocalNoBestSet", dbgid).detail("tag", tag.toString()).detail("begin", begin).detail("end", end);
TraceEvent("TLogPeekLocalNoBestSet", dbgid).detail("Tag", tag.toString()).detail("Begin", begin).detail("End", end);
return Reference<ILogSystem::ServerPeekCursor>( new ILogSystem::ServerPeekCursor( Reference<AsyncVar<OptionalInterface<TLogInterface>>>(), tag, begin, getPeekEnd(), false, false ) );
}
if(begin >= tLogs[bestSet]->startVersion) {
TraceEvent("TLogPeekLocalBestOnly", dbgid).detail("tag", tag.toString()).detail("begin", begin).detail("end", end).detail("bestSet", bestSet).detail("bestSetStart", tLogs[bestSet]->startVersion).detail("logId", tLogs[bestSet]->logServers[tLogs[bestSet]->bestLocationFor( tag )]->get().id());
TraceEvent("TLogPeekLocalBestOnly", dbgid).detail("Tag", tag.toString()).detail("Begin", begin).detail("End", end).detail("BestSet", bestSet).detail("BestSetStart", tLogs[bestSet]->startVersion).detail("LogId", tLogs[bestSet]->logServers[tLogs[bestSet]->bestLocationFor( tag )]->get().id());
return Reference<ILogSystem::ServerPeekCursor>( new ILogSystem::ServerPeekCursor( tLogs[bestSet]->logServers[tLogs[bestSet]->bestLocationFor( tag )], tag, begin, end, false, false ) );
} else {
std::vector< Reference<ILogSystem::IPeekCursor> > cursors;
std::vector< LogMessageVersion > epochEnds;
if(tLogs[bestSet]->startVersion < end) {
TraceEvent("TLogPeekLocalAddingBest", dbgid).detail("tag", tag.toString()).detail("begin", begin).detail("end", end).detail("bestSet", bestSet).detail("bestSetStart", tLogs[bestSet]->startVersion).detail("logId", tLogs[bestSet]->logServers[tLogs[bestSet]->bestLocationFor( tag )]->get().id());
TraceEvent("TLogPeekLocalAddingBest", dbgid).detail("Tag", tag.toString()).detail("Begin", begin).detail("End", end).detail("BestSet", bestSet).detail("BestSetStart", tLogs[bestSet]->startVersion).detail("LogId", tLogs[bestSet]->logServers[tLogs[bestSet]->bestLocationFor( tag )]->get().id());
cursors.push_back( Reference<ILogSystem::ServerPeekCursor>( new ILogSystem::ServerPeekCursor( tLogs[bestSet]->logServers[tLogs[bestSet]->bestLocationFor( tag )], tag, tLogs[bestSet]->startVersion, end, false, false ) ) );
}
Version lastBegin = tLogs[bestSet]->startVersion;
int i = 0;
while(begin < lastBegin) {
if(i == oldLogData.size()) {
TraceEvent("TLogPeekLocalDead", dbgid).detail("tag", tag.toString()).detail("begin", begin).detail("end", end).detail("lastBegin", lastBegin).detail("oldLogDataSize", oldLogData.size());
TraceEvent("TLogPeekLocalDead", dbgid).detail("Tag", tag.toString()).detail("Begin", begin).detail("End", end).detail("LastBegin", lastBegin).detail("OldLogDataSize", oldLogData.size());
throw worker_removed();
}
@ -630,7 +630,7 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
}
if(foundSpecial) {
TraceEvent("TLogPeekLocalFoundSpecial", dbgid).detail("tag", tag.toString()).detail("begin", begin).detail("end", end);
TraceEvent("TLogPeekLocalFoundSpecial", dbgid).detail("Tag", tag.toString()).detail("Begin", begin).detail("End", end);
cursors.push_back(peekAll(dbgid, begin, std::min(lastBegin, end), tag, false, true));
epochEnds.push_back(LogMessageVersion(std::min(lastBegin, end)));
break;
@ -644,8 +644,8 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
Version thisBegin = std::max(oldLogData[i].tLogs[bestOldSet]->startVersion, begin);
if(thisBegin < lastBegin) {
if(thisBegin < end) {
TraceEvent("TLogPeekLocalAddingOldBest", dbgid).detail("tag", tag.toString()).detail("begin", begin).detail("end", end)
.detail("logServers", oldLogData[i].tLogs[bestOldSet]->logServerString()).detail("thisBegin", thisBegin).detail("lastBegin", lastBegin);
TraceEvent("TLogPeekLocalAddingOldBest", dbgid).detail("Tag", tag.toString()).detail("Begin", begin).detail("End", end)
.detail("LogServers", oldLogData[i].tLogs[bestOldSet]->logServerString()).detail("ThisBegin", thisBegin).detail("LastBegin", lastBegin);
cursors.push_back( Reference<ILogSystem::MergedPeekCursor>( new ILogSystem::MergedPeekCursor( oldLogData[i].tLogs[bestOldSet]->logServers, oldLogData[i].tLogs[bestOldSet]->bestLocationFor( tag ), oldLogData[i].tLogs[bestOldSet]->logServers.size() + 1 - oldLogData[i].tLogs[bestOldSet]->tLogReplicationFactor, tag,
thisBegin, std::min(lastBegin, end), false, oldLogData[i].tLogs[bestOldSet]->tLogLocalities, oldLogData[i].tLogs[bestOldSet]->tLogPolicy, oldLogData[i].tLogs[bestOldSet]->tLogReplicationFactor)));
epochEnds.push_back(LogMessageVersion(std::min(lastBegin, end)));
@ -701,7 +701,7 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
std::vector<Reference<LogSet>> localSets;
for(auto& log : tLogs) {
if(log->isLocal && log->logServers.size()) {
TraceEvent("TLogPeekLogRouterLocalSet", dbgid).detail("tag", tag.toString()).detail("begin", begin).detail("logServers", log->logServerString());
TraceEvent("TLogPeekLogRouterLocalSet", dbgid).detail("Tag", tag.toString()).detail("Begin", begin).detail("LogServers", log->logServerString());
localSets.push_back(log);
if(log->hasBestPolicy && (log->locality == tag.locality || tag.locality == tagLocalitySpecial || log->locality == tagLocalitySpecial || log->locality == tagLocalityUpgraded)) {
bestSet = localSets.size()-1;
@ -713,14 +713,14 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
}
}
TraceEvent("TLogPeekLogRouterSets", dbgid).detail("tag", tag.toString()).detail("begin", begin);
TraceEvent("TLogPeekLogRouterSets", dbgid).detail("Tag", tag.toString()).detail("Begin", begin);
//FIXME: do this merge on one of the logs in the other data center to avoid sending multiple copies across the WAN
return Reference<ILogSystem::SetPeekCursor>( new ILogSystem::SetPeekCursor( localSets, bestSet == -1 ? nextBestSet : bestSet,
bestSet >= 0 ? localSets[bestSet]->bestLocationFor( tag ) : -1, tag, begin, getPeekEnd(), false ) );
} else {
for( auto& log : tLogs ) {
if( log->logServers.size() && log->isLocal && log->hasBestPolicy ) {
TraceEvent("TLogPeekLogRouterBestOnly", dbgid).detail("tag", tag.toString()).detail("begin", begin).detail("logId", log->logServers[log->bestLocationFor( tag )]->get().id());
TraceEvent("TLogPeekLogRouterBestOnly", dbgid).detail("Tag", tag.toString()).detail("Begin", begin).detail("LogId", log->logServers[log->bestLocationFor( tag )]->get().id());
return Reference<ILogSystem::ServerPeekCursor>( new ILogSystem::ServerPeekCursor( log->logServers[log->bestLocationFor( tag )], tag, begin, getPeekEnd(), false, false ) );
}
}
@ -745,7 +745,7 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
std::vector<Reference<LogSet>> localSets;
for(auto& log : old.tLogs) {
if(log->isLocal && log->logServers.size()) {
TraceEvent("TLogPeekLogRouterOldLocalSet", dbgid).detail("tag", tag.toString()).detail("begin", begin).detail("logServers", log->logServerString());
TraceEvent("TLogPeekLogRouterOldLocalSet", dbgid).detail("Tag", tag.toString()).detail("Begin", begin).detail("LogServers", log->logServerString());
localSets.push_back(log);
if(log->hasBestPolicy && (log->locality == tag.locality || tag.locality == tagLocalitySpecial || log->locality == tagLocalitySpecial || log->locality == tagLocalityUpgraded)) {
bestSet = localSets.size()-1;
@ -757,7 +757,7 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
}
}
TraceEvent("TLogPeekLogRouterOldSets", dbgid).detail("tag", tag.toString()).detail("begin", begin).detail("oldEpoch", old.epochEnd);
TraceEvent("TLogPeekLogRouterOldSets", dbgid).detail("Tag", tag.toString()).detail("Begin", begin).detail("OldEpoch", old.epochEnd);
//FIXME: do this merge on one of the logs in the other data center to avoid sending multiple copies across the WAN
return Reference<ILogSystem::SetPeekCursor>( new ILogSystem::SetPeekCursor( localSets, bestSet == -1 ? nextBestSet : bestSet,
bestSet >= 0 ? localSets[bestSet]->bestLocationFor( tag ) : -1, tag, begin, old.epochEnd, false ) );
@ -1128,7 +1128,7 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
TraceEvent("GetDurableResult", dbgid).detail("Required", requiredCount).detail("Present", results.size()).detail("ServerState", sServerState)
.detail("RecoveryVersion", ((safe_range_end > 0) && (safe_range_end-1 < results.size())) ? results[ safe_range_end-1 ].end : -1)
.detail("EndVersion", results[ new_safe_range_begin ].end).detail("SafeBegin", safe_range_begin).detail("SafeEnd", safe_range_end)
.detail("NewSafeBegin", new_safe_range_begin).detail("knownCommittedVersion", knownCommittedVersion).detail("epochEnd", lockInfo.epochEnd);
.detail("NewSafeBegin", new_safe_range_begin).detail("KnownCommittedVersion", knownCommittedVersion).detail("EpochEnd", lockInfo.epochEnd);
return std::make_pair(knownCommittedVersion, results[ new_safe_range_begin ].end);
}
@ -1259,7 +1259,7 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
break;
}
if(!lockedLocalities.count(log->locality)) {
TraceEvent("EpochEndLockExtra").detail("locality", log->locality);
TraceEvent("EpochEndLockExtra").detail("Locality", log->locality);
TEST(true); //locking old generations for version information
lockedLocalities.insert(log->locality);
LogLockInfo lockResult;
@ -1348,7 +1348,7 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
}
if(!found) {
TraceEvent("RecruitingOldLogRoutersAddingLocality").detail("locality", locality).detail("lastStart", lastStart);
TraceEvent("RecruitingOldLogRoutersAddingLocality").detail("Locality", locality).detail("LastStart", lastStart);
Reference<LogSet> newLogSet( new LogSet() );
newLogSet->locality = locality;
newLogSet->startVersion = lastStart;
@ -1399,7 +1399,7 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
}
if(!found) {
TraceEvent("RecruitingOldLogRoutersAddingLocality").detail("locality", locality).detail("lastStart", lastStart);
TraceEvent("RecruitingOldLogRoutersAddingLocality").detail("Locality", locality).detail("LastStart", lastStart);
Reference<LogSet> newLogSet( new LogSet() );
newLogSet->locality = locality;
newLogSet->startVersion = lastStart;
@ -1559,7 +1559,7 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
for( int i = 0; i < remoteWorkers.remoteTLogs.size(); i++ )
remoteTLogInitializationReplies.push_back( transformErrors( throwErrorOr( remoteWorkers.remoteTLogs[i].tLog.getReplyUnlessFailedFor( remoteTLogReqs[i], SERVER_KNOBS->TLOG_TIMEOUT, SERVER_KNOBS->MASTER_FAILURE_SLOPE_DURING_RECOVERY ) ), master_recovery_failed() ) );
TraceEvent("RemoteLogRecruitment_InitializingRemoteLogs").detail("startVersion", logSet->startVersion).detail("localStart", self->tLogs[0]->startVersion).detail("logRouterTags", self->logRouterTags);
TraceEvent("RemoteLogRecruitment_InitializingRemoteLogs").detail("StartVersion", logSet->startVersion).detail("LocalStart", self->tLogs[0]->startVersion).detail("LogRouterTags", self->logRouterTags);
Void _ = wait( waitForAll(remoteTLogInitializationReplies) && waitForAll(logRouterInitializationReplies) && oldRouterRecruitment );
for( int i = 0; i < logRouterInitializationReplies.size(); i++ ) {
@ -1660,7 +1660,7 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
}
state Future<Void> oldRouterRecruitment = Never();
TraceEvent("NewEpochStartVersion").detail("startVersion", logSystem->tLogs[0]->startVersion).detail("epochEnd", oldLogSystem->knownCommittedVersion + 1).detail("locality", primaryLocality);
TraceEvent("NewEpochStartVersion").detail("StartVersion", logSystem->tLogs[0]->startVersion).detail("EpochEnd", oldLogSystem->knownCommittedVersion + 1).detail("Locality", primaryLocality);
if(logSystem->tLogs[0]->startVersion < oldLogSystem->knownCommittedVersion + 1) {
oldRouterRecruitment = TagPartitionedLogSystem::recruitOldLogRouters(oldLogSystem.getPtr(), recr.oldLogRouters, recoveryCount, primaryLocality, logSystem->tLogs[0]->startVersion, localities, logSystem->tLogs[0]->tLogPolicy, logSystem->tLogs[0]->hasBestPolicy, false);
} else {
@ -1823,7 +1823,7 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
loop {
choose {
when (TLogLockResult data = wait( tlog->get().present() ? brokenPromiseToNever( tlog->get().interf().lock.getReply<TLogLockResult>() ) : Never() )) {
TraceEvent("TLogLocked", myID).detail("TLog", tlog->get().id()).detail("end", data.end);
TraceEvent("TLogLocked", myID).detail("TLog", tlog->get().id()).detail("End", data.end);
return data;
}
when (Void _ = wait(tlog->onChange())) {}
@ -1972,11 +1972,11 @@ struct TagPartitionedLogSystem : ILogSystem, ReferenceCounted<TagPartitionedLogS
TEST(can_omit >= prevState.tLogReplicationFactor - prevState.tLogWriteAntiQuorum); // Locking a subset of the TLogs while ending an epoch.
const bool reboot_a_tlog = g_network->now() - g_simulator.lastConnectionFailure > g_simulator.connectionFailuresDisableDuration && BUGGIFY && g_random->random01() < 0.25;
TraceEvent(SevInfo, "MasterRecoveryTLogLocking", dbgid)
.detail("locks", tlogs.size() - can_omit)
.detail("skipped", can_omit)
.detail("replication", prevState.tLogReplicationFactor)
.detail("antiquorum", prevState.tLogWriteAntiQuorum)
.detail("reboot_buggify", reboot_a_tlog);
detail("Locks", tlogs.size() - can_omit)
detail("Skipped", can_omit)
detail("Replication", prevState.tLogReplicationFactor)
detail("Antiquorum", prevState.tLogWriteAntiQuorum)
detail("RebootBuggify", reboot_a_tlog);
for (int i = 0; i < tlogs.size() - can_omit; i++) {
const int index = tlogs[i].second;
Future<TLogLockResult>& entry = tLogReply->at(index);

16
fdbserver/VFSAsync.cpp Executable file → Normal file
View File

@ -92,7 +92,7 @@ std::map<std::string, std::pair<uint32_t,int>> VFSAsyncFile::filename_lockCount_
static int asyncClose(sqlite3_file *pFile){
VFSAsyncFile *p = (VFSAsyncFile*)pFile;
/*TraceEvent("VFSAsyncClose").detail("fd", p->file->debugFD())
/*TraceEvent("VFSAsyncClose").detail("Fd", p->file->debugFD())
.detail("Filename", p->filename).detail("ZCRefs", p->debug_zcrefs)
.detail("ZCReads", p->debug_zcreads).detail("NormalReads", p->debug_reads).backtrace();*/
//printf("Closing %s: %d zcrefs, %d/%d reads zc\n", filename.c_str(), debug_zcrefs, debug_zcreads, debug_zcreads+debug_reads);
@ -201,7 +201,7 @@ static int asyncSync(sqlite3_file *pFile, int flags){
} catch (Error& e) {
TraceEvent("VFSSyncError")
.detail("Filename", p->filename)
.detail("sqlite3_file", (int64_t)pFile)
.detail("Sqlite3File", (int64_t)pFile)
.detail("IAsyncFile", (int64_t)p->file.getPtr())
.error(e);
@ -225,7 +225,7 @@ static int VFSAsyncFileSize(sqlite3_file *pFile, sqlite_int64 *pSize){
static int asyncLock(sqlite3_file *pFile, int eLock){
VFSAsyncFile *p = (VFSAsyncFile*)pFile;
//TraceEvent("FileLock").detail("File", p->filename).detail("fd", p->file->debugFD()).detail("PrevLockLevel", p->lockLevel).detail("Op", eLock).detail("LockCount", *p->pLockCount);
//TraceEvent("FileLock").detail("File", p->filename).detail("Fd", p->file->debugFD()).detail("PrevLockLevel", p->lockLevel).detail("Op", eLock).detail("LockCount", *p->pLockCount);
return eLock == EXCLUSIVE_LOCK ? SQLITE_BUSY : SQLITE_OK;
}
@ -373,7 +373,7 @@ static int asyncDeviceCharacteristics(sqlite3_file *pFile){ return 0; }
} else if (flags & SQLITE_SHM_SHARED) {
for(int i=ofst; i<ofst+n; i++)
if ( memInfo->exclusiveLocks[i] != ((pDbFd->sharedMemoryExclusiveLocks>>i)&1) ) {
//TraceEvent("ShmLocked").detail("File", DEBUG_DETERMINISM ? 0 : (int64_t)pDbFd).detail("Acquiring", "Shared").detail("i", i).detail("Exclusive", memInfo->exclusiveLocks[i]).detail("MyExclusive", pDbFd->sharedMemoryExclusiveLocks);
//TraceEvent("ShmLocked").detail("File", DEBUG_DETERMINISM ? 0 : (int64_t)pDbFd).detail("Acquiring", "Shared").detail("I", i).detail("Exclusive", memInfo->exclusiveLocks[i]).detail("MyExclusive", pDbFd->sharedMemoryExclusiveLocks);
return SQLITE_BUSY;
}
for(int i=ofst; i<ofst+n; i++)
@ -386,7 +386,7 @@ static int asyncDeviceCharacteristics(sqlite3_file *pFile){ return 0; }
if ( memInfo->exclusiveLocks[i] != ((pDbFd->sharedMemoryExclusiveLocks>>i)&1) ||
memInfo->sharedLocks[i] != ((pDbFd->sharedMemorySharedLocks>>i)&1) )
{
//TraceEvent("ShmLocked").detail("File", DEBUG_DETERMINISM ? 0 : (int64_t)pDbFd).detail("Acquiring", "Exclusive").detail("i", i).detail("Exclusive", memInfo->exclusiveLocks[i]).detail("MyExclusive", pDbFd->sharedMemoryExclusiveLocks).detail("Shared", memInfo->sharedLocks[i]).detail("MyShared", pDbFd->sharedMemorySharedLocks);
//TraceEvent("ShmLocked").detail("File", DEBUG_DETERMINISM ? 0 : (int64_t)pDbFd).detail("Acquiring", "Exclusive").detail("I", i).detail("Exclusive", memInfo->exclusiveLocks[i]).detail("MyExclusive", pDbFd->sharedMemoryExclusiveLocks).detail("Shared", memInfo->sharedLocks[i]).detail("MyShared", pDbFd->sharedMemorySharedLocks);
return SQLITE_BUSY;
}
for(int i=ofst; i<ofst+n; i++)
@ -448,7 +448,7 @@ static int asyncDeviceCharacteristics(sqlite3_file *pFile){ return 0; }
}
VFSAsyncFile::~VFSAsyncFile() {
//TraceEvent("VFSAsyncFileDel").detail("filename", filename);
//TraceEvent("VFSAsyncFileDel").detail("Filename", filename);
if (!--filename_lockCount_openCount[filename].second) {
filename_lockCount_openCount.erase(filename);
@ -523,9 +523,9 @@ static int asyncOpen(
/*TraceEvent("VFSOpened")
.detail("Filename", p->filename)
.detail("fd", DEBUG_DETERMINISM ? 0 : p->file->debugFD())
.detail("Fd", DEBUG_DETERMINISM ? 0 : p->file->debugFD())
.detail("Flags", flags)
.detail("sqlite3_file", DEBUG_DETERMINISM ? 0 : (int64_t)pFile)
.detail("Sqlite3File", DEBUG_DETERMINISM ? 0 : (int64_t)pFile)
.detail("IAsyncFile", DEBUG_DETERMINISM ? 0 : (int64_t)p->file.getPtr());*/
} catch (Error& e) {
TraceEvent("SQLiteOpenFail").detail("Filename", p->filename).error(e);

9
fdbserver/fdbserver.actor.cpp
View File

@ -849,7 +849,7 @@ int main(int argc, char* argv[]) {
std::string testServersStr;
NetworkAddress publicAddress, listenAddress;
const char *targetKey = NULL;
uint64_t memLimit = 8LL << 30;
uint64_t memLimit = 8LL << 30; // Nice to maintain the same default value for memLimit and SERVER_KNOBS->SERVER_MEM_LIMIT and SERVER_KNOBS->COMMIT_BATCHES_MEM_BYTES_HARD_LIMIT
uint64_t storageMemLimit = 1LL << 30;
bool buggifyEnabled = false, machineIdOverride = false, restarting = false;
Optional<Standalone<StringRef>> zoneId;
@ -1390,7 +1390,9 @@ int main(int argc, char* argv[]) {
CLIENT_KNOBS = clientKnobs;
if (!serverKnobs->setKnob( "log_directory", logFolder )) ASSERT(false);
if (role != Simulation) {
if (!serverKnobs->setKnob("commit_batches_mem_bytes_hard_limit", std::to_string(memLimit))) ASSERT(false);
}
for(auto k=knobs.begin(); k!=knobs.end(); ++k) {
try {
if (!flowKnobs->setKnob( k->first, k->second ) &&
@ -1408,6 +1410,7 @@ int main(int argc, char* argv[]) {
throw;
}
}
if (!serverKnobs->setKnob("server_mem_limit", std::to_string(memLimit))) ASSERT(false);
if (role == SkipListTest) {
skipListTest();
@ -1775,7 +1778,7 @@ int main(int argc, char* argv[]) {
flushAndExit(FDB_EXIT_MAIN_ERROR);
} catch (std::exception& e) {
fprintf(stderr, "std::exception: %s\n", e.what());
TraceEvent(SevError, "MainError").error(unknown_error()).detail("std::exception", e.what());
TraceEvent(SevError, "MainError").error(unknown_error()).detail("RootException", e.what());
//printf("\n%d tests passed; %d tests failed\n", passCount, failCount);
flushAndExit(FDB_EXIT_MAIN_EXCEPTION);
}

30
fdbserver/masterserver.actor.cpp
View File

@ -260,7 +260,7 @@ ACTOR Future<Void> newProxies( Reference<MasterData> self, RecruitFromConfigurat
req.recoveryCount = self->cstate.myDBState.recoveryCount + 1;
req.recoveryTransactionVersion = self->recoveryTransactionVersion;
req.firstProxy = i == 0;
TraceEvent("ProxyReplies",self->dbgid).detail("workerID", recr.proxies[i].id());
TraceEvent("ProxyReplies",self->dbgid).detail("WorkerID", recr.proxies[i].id());
initializationReplies.push_back( transformErrors( throwErrorOr( recr.proxies[i].masterProxy.getReplyUnlessFailedFor( req, SERVER_KNOBS->TLOG_TIMEOUT, SERVER_KNOBS->MASTER_FAILURE_SLOPE_DURING_RECOVERY ) ), master_recovery_failed() ) );
}
@ -278,7 +278,7 @@ ACTOR Future<Void> newResolvers( Reference<MasterData> self, RecruitFromConfigur
req.recoveryCount = self->cstate.myDBState.recoveryCount + 1;
req.proxyCount = recr.proxies.size();
req.resolverCount = recr.resolvers.size();
TraceEvent("ResolverReplies",self->dbgid).detail("workerID", recr.resolvers[i].id());
TraceEvent("ResolverReplies",self->dbgid).detail("WorkerID", recr.resolvers[i].id());
initializationReplies.push_back( transformErrors( throwErrorOr( recr.resolvers[i].resolver.getReplyUnlessFailedFor( req, SERVER_KNOBS->TLOG_TIMEOUT, SERVER_KNOBS->MASTER_FAILURE_SLOPE_DURING_RECOVERY ) ), master_recovery_failed() ) );
}
@ -292,7 +292,7 @@ ACTOR Future<Void> newTLogServers( Reference<MasterData> self, RecruitFromConfig
if(self->configuration.remoteTLogReplicationFactor > 0) {
state Optional<Key> remoteDcId = self->remoteDcIds.size() ? self->remoteDcIds[0] : Optional<Key>();
if( !self->dcId_locality.count(recr.dcId) ) {
TraceEvent(SevWarn, "UnknownPrimaryDCID", self->dbgid).detail("primaryId", printable(recr.dcId));
TraceEvent(SevWarn, "UnknownPrimaryDCID", self->dbgid).detail("PrimaryId", printable(recr.dcId));
int8_t loc = self->getNextLocality();
Standalone<CommitTransactionRef> tr;
tr.set(tr.arena(), tagLocalityListKeyFor(recr.dcId), tagLocalityListValue(loc));
@ -301,7 +301,7 @@ ACTOR Future<Void> newTLogServers( Reference<MasterData> self, RecruitFromConfig
}
if( !self->dcId_locality.count(remoteDcId) ) {
TraceEvent(SevWarn, "UnknownRemoteDCID", self->dbgid).detail("remoteId", printable(remoteDcId));
TraceEvent(SevWarn, "UnknownRemoteDCID", self->dbgid).detail("RemoteId", printable(remoteDcId));
int8_t loc = self->getNextLocality();
Standalone<CommitTransactionRef> tr;
tr.set(tr.arena(), tagLocalityListKeyFor(remoteDcId), tagLocalityListValue(loc));
@ -456,7 +456,7 @@ ACTOR Future<Void> updateRegistration( Reference<MasterData> self, Reference<ILo
trigger = self->registrationTrigger.onTrigger();
TraceEvent("MasterUpdateRegistration", self->dbgid).detail("RecoveryCount", self->cstate.myDBState.recoveryCount).detail("logs", describe(logSystem->getLogSystemConfig().tLogs));
TraceEvent("MasterUpdateRegistration", self->dbgid).detail("RecoveryCount", self->cstate.myDBState.recoveryCount).detail("Logs", describe(logSystem->getLogSystemConfig().tLogs));
if (!self->cstateUpdated.isSet()) {
Void _ = wait(sendMasterRegistration(self.getPtr(), logSystem->getLogSystemConfig(), self->provisionalProxies, self->resolvers, self->cstate.myDBState.recoveryCount, self->cstate.prevDBState.getPriorCommittedLogServers() ));
@ -543,7 +543,7 @@ ACTOR Future<Void> recruitEverything( Reference<MasterData> self, vector<Storage
.detail("DesiredProxies", self->configuration.getDesiredProxies())
.detail("RequiredResolvers", 1)
.detail("DesiredResolvers", self->configuration.getDesiredResolvers())
.detail("storeType", self->configuration.storageServerStoreType)
.detail("StoreType", self->configuration.storageServerStoreType)
.trackLatest("MasterRecoveryState");
//FIXME: we only need log routers for the same locality as the master
@ -614,13 +614,13 @@ ACTOR Future<Void> readTransactionSystemState( Reference<MasterData> self, Refer
if ( self->recoveryTransactionVersion < minRequiredCommitVersion ) self->recoveryTransactionVersion = minRequiredCommitVersion;
}
TraceEvent("MasterRecovering", self->dbgid).detail("lastEpochEnd", self->lastEpochEnd).detail("recoveryTransactionVersion", self->recoveryTransactionVersion);
TraceEvent("MasterRecovering", self->dbgid).detail("LastEpochEnd", self->lastEpochEnd).detail("RecoveryTransactionVersion", self->recoveryTransactionVersion);
Standalone<VectorRef<KeyValueRef>> rawConf = wait( self->txnStateStore->readRange( configKeys ) );
self->configuration.fromKeyValues( rawConf );
self->originalConfiguration = self->configuration;
self->hasConfiguration = true;
TraceEvent("MasterRecoveredConfig", self->dbgid).detail("conf", self->configuration.toString()).trackLatest("RecoveredConfig");
TraceEvent("MasterRecoveredConfig", self->dbgid).detail("Conf", self->configuration.toString()).trackLatest("RecoveredConfig");
Standalone<VectorRef<KeyValueRef>> rawLocalities = wait( self->txnStateStore->readRange( tagLocalityListKeys ) );
self->dcId_locality.clear();
@ -930,7 +930,7 @@ ACTOR Future<Void> resolutionBalancing(Reference<MasterData> self) {
for (int i = 0; i < futures.size(); i++) {
total += futures[i].get();
metrics.insert(std::make_pair(futures[i].get(), i), NoMetric());
//TraceEvent("ResolverMetric").detail("i", i).detail("metric", futures[i].get());
//TraceEvent("ResolverMetric").detail("I", i).detail("Metric", futures[i].get());
}
if( metrics.lastItem()->first - metrics.begin()->first > SERVER_KNOBS->MIN_BALANCE_DIFFERENCE ) {
try {
@ -950,7 +950,7 @@ ACTOR Future<Void> resolutionBalancing(Reference<MasterData> self) {
ResolutionSplitReply split = wait( brokenPromiseToNever(self->resolvers[metrics.lastItem()->second].split.getReply(req, TaskResolutionMetrics)) );
KeyRangeRef moveRange = range.second ? KeyRangeRef( range.first.begin, split.key ) : KeyRangeRef( split.key, range.first.end );
movedRanges.push_back_deep(movedRanges.arena(), ResolverMoveRef(moveRange, dest));
TraceEvent("MovingResolutionRange").detail("src", src).detail("dest", dest).detail("amount", amount).detail("startRange", printable(range.first)).detail("moveRange", printable(moveRange)).detail("used", split.used).detail("KeyResolverRanges", key_resolver.size());
TraceEvent("MovingResolutionRange").detail("Src", src).detail("Dest", dest).detail("Amount", amount).detail("StartRange", printable(range.first)).detail("MoveRange", printable(moveRange)).detail("Used", split.used).detail("KeyResolverRanges", key_resolver.size());
amount -= split.used;
if(moveRange != range.first || amount <= 0 )
break;
@ -958,7 +958,7 @@ ACTOR Future<Void> resolutionBalancing(Reference<MasterData> self) {
for(auto& it : movedRanges)
key_resolver.insert(it.range, it.dest);
//for(auto& it : key_resolver.ranges())
// TraceEvent("KeyResolver").detail("range", printable(it.range())).detail("value", it.value());
// TraceEvent("KeyResolver").detail("Range", printable(it.range())).detail("Value", it.value());
self->resolverChangesVersion = self->version + 1;
for (auto& p : self->proxies)
@ -1204,7 +1204,7 @@ ACTOR Future<Void> masterCore( Reference<MasterData> self ) {
self->addActor.send( waitResolverFailure( self->resolvers ) );
self->addActor.send( waitProxyFailure( self->proxies ) );
self->addActor.send( provideVersions(self) );
self->addActor.send( reportErrors(updateRegistration(self, self->logSystem), "updateRegistration", self->dbgid) );
self->addActor.send( reportErrors(updateRegistration(self, self->logSystem), "UpdateRegistration", self->dbgid) );
self->registrationTrigger.trigger();
Void _ = wait(discardCommit(self->txnStateStore, self->txnStateLogAdapter));
@ -1255,14 +1255,14 @@ ACTOR Future<Void> masterCore( Reference<MasterData> self ) {
double recoveryDuration = now() - recoverStartTime;
TraceEvent((recoveryDuration > 4 && !g_network->isSimulated()) ? SevWarnAlways : SevInfo, "MasterRecoveryDuration", self->dbgid)
.detail("recoveryDuration", recoveryDuration)
.detail("RecoveryDuration", recoveryDuration)
.trackLatest("MasterRecoveryDuration");
TraceEvent("MasterRecoveryState", self->dbgid)
.detail("StatusCode", RecoveryStatus::fully_recovered)
.detail("Status", RecoveryStatus::names[RecoveryStatus::fully_recovered])
.detail("storeType", self->configuration.storageServerStoreType)
.detail("recoveryDuration", recoveryDuration)
.detail("StoreType", self->configuration.storageServerStoreType)
.detail("RecoveryDuration", recoveryDuration)
.trackLatest("MasterRecoveryState");
// Now that the master is recovered we can start auxiliary services that happen to run here

92
fdbserver/storageserver.actor.cpp
View File

@ -407,38 +407,38 @@ public:
Counters(StorageServer* self)
: cc("StorageServer", self->thisServerID.toString()),
getKeyQueries("getKeyQueries", cc),
getValueQueries("getValueQueries",cc),
getRangeQueries("getRangeQueries", cc),
getKeyQueries("GetKeyQueries", cc),
getValueQueries("GetValueQueries",cc),
getRangeQueries("GetRangeQueries", cc),
allQueries("QueryQueue", cc),
finishedQueries("finishedQueries", cc),
rowsQueried("rowsQueried", cc),
bytesQueried("bytesQueried", cc),
bytesInput("bytesInput", cc),
bytesDurable("bytesDurable", cc),
bytesFetched("bytesFetched", cc),
mutationBytes("mutationBytes", cc),
updateBatches("updateBatches", cc),
updateVersions("updateVersions", cc),
loops("loops", cc)
finishedQueries("FinishedQueries", cc),
rowsQueried("RowsQueried", cc),
bytesQueried("BytesQueried", cc),
bytesInput("BytesInput", cc),
bytesDurable("BytesDurable", cc),
bytesFetched("BytesFetched", cc),
mutationBytes("MutationBytes", cc),
updateBatches("UpdateBatches", cc),
updateVersions("UpdateVersions", cc),
loops("Loops", cc)
{
specialCounter(cc, "lastTLogVersion", [self](){return self->lastTLogVersion; });
specialCounter(cc, "version", [self](){return self->version.get(); });
specialCounter(cc, "storageVersion", [self](){return self->storageVersion(); });
specialCounter(cc, "durableVersion", [self](){return self->durableVersion.get(); });
specialCounter(cc, "desiredOldestVersion", [self](){return self->desiredOldestVersion.get(); });
specialCounter(cc, "LastTLogVersion", [self](){ return self->lastTLogVersion; });
specialCounter(cc, "Version", [self](){ return self->version.get(); });
specialCounter(cc, "StorageVersion", [self](){ return self->storageVersion(); });
specialCounter(cc, "DurableVersion", [self](){ return self->durableVersion.get(); });
specialCounter(cc, "DesiredOldestVersion", [self](){ return self->desiredOldestVersion.get(); });
specialCounter(cc, "FetchKeysFetchActive", [self](){return self->fetchKeysParallelismLock.activePermits(); });
specialCounter(cc, "FetchKeysWaiting", [self](){return self->fetchKeysParallelismLock.waiters(); });
specialCounter(cc, "FetchKeysFetchActive", [self](){ return self->fetchKeysParallelismLock.activePermits(); });
specialCounter(cc, "FetchKeysWaiting", [self](){ return self->fetchKeysParallelismLock.waiters(); });
specialCounter(cc, "QueryQueueMax", [self](){return self->getAndResetMaxQueryQueueSize(); });
specialCounter(cc, "QueryQueueMax", [self](){ return self->getAndResetMaxQueryQueueSize(); });
specialCounter(cc, "bytesStored", [self](){return self->metrics.byteSample.getEstimate(allKeys); });
specialCounter(cc, "BytesStored", [self](){ return self->metrics.byteSample.getEstimate(allKeys); });
specialCounter(cc, "kvstoreBytesUsed", [self](){ return self->storage.getStorageBytes().used; });
specialCounter(cc, "kvstoreBytesFree", [self](){ return self->storage.getStorageBytes().free; });
specialCounter(cc, "kvstoreBytesAvailable", [self](){ return self->storage.getStorageBytes().available; });
specialCounter(cc, "kvstoreBytesTotal", [self](){ return self->storage.getStorageBytes().total; });
specialCounter(cc, "KvstoreBytesUsed", [self](){ return self->storage.getStorageBytes().used; });
specialCounter(cc, "KvstoreBytesFree", [self](){ return self->storage.getStorageBytes().free; });
specialCounter(cc, "KvstoreBytesAvailable", [self](){ return self->storage.getStorageBytes().available; });
specialCounter(cc, "KvstoreBytesTotal", [self](){ return self->storage.getStorageBytes().total; });
}
} counters;
@ -773,7 +773,7 @@ ACTOR Future<Void> watchValue_impl( StorageServer* data, WatchValueRequest req )
GetValueRequest getReq( req.key, latest, req.debugID );
state Future<Void> getValue = getValueQ( data, getReq ); //we are relying on the delay zero at the top of getValueQ, if removed we need one here
GetValueReply reply = wait( getReq.reply.getFuture() );
//TraceEvent("watcherCheckValue").detail("key", printable( req.key ) ).detail("value", printable( req.value ) ).detail("currentValue", printable( v ) ).detail("ver", latest);
//TraceEvent("WatcherCheckValue").detail("Key", printable( req.key ) ).detail("Value", printable( req.value ) ).detail("CurrentValue", printable( v ) ).detail("Ver", latest);
debugMutation("ShardWatchValue", latest, MutationRef(MutationRef::DebugKey, req.key, reply.value.present() ? StringRef( reply.value.get() ) : LiteralStringRef("<null>") ) );
@ -999,10 +999,10 @@ ACTOR Future<GetKeyValuesReply> readRange( StorageServer* data, Version version,
if (more) { // if there might be more data, begin reading right after what we already found to find out
//if (track) printf("more\n");
if (!(limit<=0 || *pLimitBytes<=0 || result.data.end()[-1].key == atStorageVersion.end()[-1].key))
TraceEvent(SevError, "readRangeIssue", data->thisServerID).detail("rB", printable(readBegin)).detail("rE", printable(readEnd))
.detail("vStart", vStart ? printable(vStart.key()) : "nil").detail("vEnd", vEnd ? printable(vEnd.key()) : "nil")
.detail("atStorageVersionBack", printable(atStorageVersion.end()[-1].key)).detail("resultBack", printable(result.data.end()[-1].key))
.detail("limit", limit).detail("*pLimitBytes", *pLimitBytes).detail("resultSize", result.data.size()).detail("prevSize", prevSize);
TraceEvent(SevError, "ReadRangeIssue", data->thisServerID).detail("ReadBegin", printable(readBegin)).detail("ReadEnd", printable(readEnd))
.detail("VStart", vStart ? printable(vStart.key()) : "nil").detail("VEnd", vEnd ? printable(vEnd.key()) : "nil")
.detail("AtStorageVersionBack", printable(atStorageVersion.end()[-1].key)).detail("ResultBack", printable(result.data.end()[-1].key))
.detail("Limit", limit).detail("LimitBytes", *pLimitBytes).detail("ResultSize", result.data.size()).detail("PrevSize", prevSize);
readBegin = readBeginTemp = keyAfter( result.data.end()[-1].key );
ASSERT( limit<=0 || *pLimitBytes<=0 || result.data.end()[-1].key == atStorageVersion.end()[-1].key );
} else if (vStart && vStart->isClearTo()){ // if vStart is a clear, skip it.
@ -1018,11 +1018,11 @@ ACTOR Future<GetKeyValuesReply> readRange( StorageServer* data, Version version,
ASSERT( result.data.size() == 0 || *pLimitBytes + result.data.end()[-1].expectedSize() + sizeof(KeyValueRef) > 0 );
/*if (*pLimitBytes <= 0)
TraceEvent(SevWarn, "readRangeLimitExceeded")
.detail("version", version)
.detail("begin", printable(range.begin) )
.detail("end", printable(range.end) )
.detail("limitReamin", limit)
.detail("lBytesRemain", *pLimitBytes); */
.detail("Version", version)
.detail("Begin", printable(range.begin) )
.detail("End", printable(range.end) )
.detail("LimitReamin", limit)
.detail("LimitBytesRemain", *pLimitBytes); */
/*GetKeyValuesReply correct = wait( readRangeOld(data, version, range, originalLimit, originalLimitBytes) );
bool prefix_equal = true;
@ -1203,7 +1203,7 @@ ACTOR Future<Void> getKeyValues( StorageServer* data, GetKeyValuesRequest req )
if( req.debugID.present() )
g_traceBatch.addEvent("TransactionDebug", req.debugID.get().first(), "storageserver.getKeyValues.AfterVersion");
//.detail("shardBegin", printable(shard.begin)).detail("shardEnd", printable(shard.end));
//.detail("ShardBegin", printable(shard.begin)).detail("ShardEnd", printable(shard.end));
//} catch (Error& e) { TraceEvent("WrongShardServer", data->thisServerID).detail("Begin", req.begin.toString()).detail("End", req.end.toString()).detail("Version", version).detail("Shard", "None").detail("In", "getKeyValues>getShardKeyRange"); throw e; }
if ( !selectorInRange(req.end, shard) && !(req.end.isFirstGreaterOrEqual() && req.end.getKey() == shard.end) ) {
@ -1219,7 +1219,7 @@ ACTOR Future<Void> getKeyValues( StorageServer* data, GetKeyValuesRequest req )
state Key end = wait(fEnd);
if( req.debugID.present() )
g_traceBatch.addEvent("TransactionDebug", req.debugID.get().first(), "storageserver.getKeyValues.AfterKeys");
//.detail("off1",offset1).detail("off2",offset2).detail("reqBegin",printable(req.begin.getKey())).detail("reqEnd",printable(req.end.getKey()));
//.detail("Off1",offset1).detail("Off2",offset2).detail("ReqBegin",printable(req.begin.getKey())).detail("ReqEnd",printable(req.end.getKey()));
// Offsets of zero indicate begin/end keys in this shard, which obviously means we can answer the query
// An end offset of 1 is also OK because the end key is exclusive, so if the first key of the next shard is the end the last actual key returned must be from this shard.
@ -1236,7 +1236,7 @@ ACTOR Future<Void> getKeyValues( StorageServer* data, GetKeyValuesRequest req )
if (begin >= end) {
if( req.debugID.present() )
g_traceBatch.addEvent("TransactionDebug", req.debugID.get().first(), "storageserver.getKeyValues.Send");
//.detail("begin",printable(begin)).detail("end",printable(end));
//.detail("Begin",printable(begin)).detail("End",printable(end));
GetKeyValuesReply none;
none.version = version;
@ -1253,7 +1253,7 @@ ACTOR Future<Void> getKeyValues( StorageServer* data, GetKeyValuesRequest req )
if( req.debugID.present() )
g_traceBatch.addEvent("TransactionDebug", req.debugID.get().first(), "storageserver.getKeyValues.AfterReadRange");
//.detail("begin",printable(begin)).detail("end",printable(end)).detail("sizeOf",r.data.size());
//.detail("Begin",printable(begin)).detail("End",printable(end)).detail("SizeOf",r.data.size());
data->checkChangeCounter( changeCounter, KeyRangeRef( std::min<KeyRef>(begin, std::min<KeyRef>(req.begin.getKey(), req.end.getKey())), std::max<KeyRef>(end, std::max<KeyRef>(req.begin.getKey(), req.end.getKey())) ) );
if (EXPENSIVE_VALIDATION) {
for (int i = 0; i < r.data.size(); i++)
@ -1890,7 +1890,7 @@ ACTOR Future<Void> fetchKeys( StorageServer *data, AddingShard* shard ) {
debug_getRangeRetries++;
if (debug_nextRetryToLog==debug_getRangeRetries){
debug_nextRetryToLog += std::min(debug_nextRetryToLog, 1024);
TraceEvent(SevWarn, "FetchPast", data->thisServerID).detail("totalAttempts", debug_getRangeRetries).detail("FKID", interval.pairID).detail("V", lastFV).detail("N", fetchVersion).detail("E", data->version.get());
TraceEvent(SevWarn, "FetchPast", data->thisServerID).detail("TotalAttempts", debug_getRangeRetries).detail("FKID", interval.pairID).detail("V", lastFV).detail("N", fetchVersion).detail("E", data->version.get());
}
} else if (e.code() == error_code_future_version) {
TEST(true); // fetchKeys got future_version, so there must be a huge storage lag somewhere. Keep trying.
@ -2298,7 +2298,7 @@ private:
.detail("FromVersion", fromVersion)
.detail("ToVersion", rollbackVersion)
.detail("AtVersion", currentVersion)
.detail("storageVersion", data->storageVersion());
.detail("StorageVersion", data->storageVersion());
ASSERT( rollbackVersion >= data->storageVersion() );
}
// Don't let oldestVersion (and thus storageVersion) go into the rolled back range of versions
@ -2508,8 +2508,8 @@ ACTOR Future<Void> update( StorageServer* data, bool* pReceivedUpdate )
data->version.set( ver ); // Triggers replies to waiting gets for new version(s)
if (data->otherError.getFuture().isReady()) data->otherError.getFuture().get();
//TraceEvent("StorageServerUpdated", data->thisServerID).detail("ver", ver).detail("dataVersion", data->version.get())
// .detail("lastTLogVersion", data->lastTLogVersion).detail("newOldest", updater.newOldestVersion).detail("desiredOldest",data->desiredOldestVersion.get())
//TraceEvent("StorageServerUpdated", data->thisServerID).detail("Ver", ver).detail("DataVersion", data->version.get())
// .detail("LastTLogVersion", data->lastTLogVersion).detail("NewOldest", updater.newOldestVersion).detail("DesiredOldest",data->desiredOldestVersion.get())
// .detail("MAX_READ_TRANSACTION_LIFE_VERSIONS", SERVER_KNOBS->MAX_READ_TRANSACTION_LIFE_VERSIONS);
// Trigger updateStorage if necessary
@ -3324,9 +3324,9 @@ ACTOR Future<Void> replaceInterface( StorageServer* self, StorageServerInterface
}
self->allHistory = self->history;
TraceEvent("SSTag", self->thisServerID).detail("myTag", self->tag.toString());
TraceEvent("SSTag", self->thisServerID).detail("MyTag", self->tag.toString());
for(auto it : self->history) {
TraceEvent("SSHistory", self->thisServerID).detail("ver", it.first).detail("tag", it.second.toString());
TraceEvent("SSHistory", self->thisServerID).detail("Ver", it.first).detail("Tag", it.second.toString());
}
if(self->history.size() && BUGGIFY) {

14
fdbserver/tester.actor.cpp
View File

@ -965,7 +965,7 @@ vector<TestSpec> readTests( ifstream& ifs ) {
if(value == "true")
spec.phases = TestWorkload::CHECK;
} else if( attrib == "StderrSeverity" ) {
TraceEvent("StderrSeverity").detail("newSeverity", value);
TraceEvent("StderrSeverity").detail("NewSeverity", value);
}
else if (attrib == "ClientInfoLogging") {
if (value == "false") {
@ -1127,9 +1127,9 @@ ACTOR Future<Void> runTests( Reference<ClusterConnectionFile> connFile, test_typ
Reference<AsyncVar<Optional<ClusterControllerFullInterface>>> cc( new AsyncVar<Optional<ClusterControllerFullInterface>> );
Reference<AsyncVar<Optional<ClusterInterface>>> ci( new AsyncVar<Optional<ClusterInterface>> );
vector<Future<Void>> actors;
actors.push_back( reportErrors(monitorLeader( connFile, cc ), "monitorLeader") );
actors.push_back( reportErrors(extractClusterInterface( cc,ci ),"extractClusterInterface") );
actors.push_back( reportErrors(failureMonitorClient( ci, false ),"failureMonitorClient") );
actors.push_back( reportErrors(monitorLeader( connFile, cc ), "MonitorLeader") );
actors.push_back( reportErrors(extractClusterInterface( cc,ci ),"ExtractClusterInterface") );
actors.push_back( reportErrors(failureMonitorClient( ci, false ),"FailureMonitorClient") );
if(whatToRun == TEST_TYPE_CONSISTENCY_CHECK) {
TestSpec spec;
@ -1166,11 +1166,11 @@ ACTOR Future<Void> runTests( Reference<ClusterConnectionFile> connFile, test_typ
if (at == TEST_HERE) {
Reference<AsyncVar<ServerDBInfo>> db( new AsyncVar<ServerDBInfo> );
vector<TesterInterface> iTesters(1);
actors.push_back( reportErrors(monitorServerDBInfo( cc, Reference<ClusterConnectionFile>(), LocalityData(), db ), "monitorServerDBInfo") ); // FIXME: Locality
actors.push_back( reportErrors(testerServerCore( iTesters[0], connFile, db, locality ), "testerServerCore") );
actors.push_back( reportErrors(monitorServerDBInfo( cc, Reference<ClusterConnectionFile>(), LocalityData(), db ), "MonitorServerDBInfo") ); // FIXME: Locality
actors.push_back( reportErrors(testerServerCore( iTesters[0], connFile, db, locality ), "TesterServerCore") );
tests = runTests( cc, ci, iTesters, testSpecs, startingConfiguration, locality );
} else {
tests = reportErrors(runTests(cc, ci, testSpecs, at, minTestersExpected, startingConfiguration, locality), "runTests");
tests = reportErrors(runTests(cc, ci, testSpecs, at, minTestersExpected, startingConfiguration, locality), "RunTests");
}
choose {

10
fdbserver/worker.actor.cpp
View File

@ -973,7 +973,7 @@ ACTOR Future<Void> fdbd(
{
try {
ServerCoordinators coordinators( connFile );
TraceEvent("StartingFDBD").detailext("ZoneID", localities.zoneId()).detailext("machineId", localities.machineId()).detail("DiskPath", dataFolder).detail("CoordPath", coordFolder);
TraceEvent("StartingFDBD").detailext("ZoneID", localities.zoneId()).detailext("MachineId", localities.machineId()).detail("DiskPath", dataFolder).detail("CoordPath", coordFolder);
// SOMEDAY: start the services on the machine in a staggered fashion in simulation?
Reference<AsyncVar<Optional<ClusterControllerFullInterface>>> cc( new AsyncVar<Optional<ClusterControllerFullInterface>> );
@ -984,10 +984,10 @@ ACTOR Future<Void> fdbd(
state Promise<Void> recoveredDiskFiles;
if ( coordFolder.size() )
v.push_back( fileNotFoundToNever( coordinationServer( coordFolder ) ) ); //SOMEDAY: remove the fileNotFound wrapper and make DiskQueue construction safe from errors setting up their files
v.push_back( reportErrors( processClass == ProcessClass::TesterClass ? monitorLeader( connFile, cc ) : clusterController( connFile, cc , asyncPriorityInfo, recoveredDiskFiles.getFuture()), "clusterController") );
v.push_back( reportErrors(extractClusterInterface( cc, ci ), "extractClusterInterface") );
v.push_back( reportErrors(failureMonitorClient( ci, true ), "failureMonitorClient") );
v.push_back( reportErrorsExcept(workerServer(connFile, cc, localities, asyncPriorityInfo, processClass, dataFolder, memoryLimit, metricsConnFile, metricsPrefix, recoveredDiskFiles), "workerServer", UID(), &normalWorkerErrors()) );
v.push_back( reportErrors( processClass == ProcessClass::TesterClass ? monitorLeader( connFile, cc ) : clusterController( connFile, cc , asyncPriorityInfo, recoveredDiskFiles.getFuture()), "ClusterController") );
v.push_back( reportErrors(extractClusterInterface( cc, ci ), "ExtractClusterInterface") );
v.push_back( reportErrors(failureMonitorClient( ci, true ), "FailureMonitorClient") );
v.push_back( reportErrorsExcept(workerServer(connFile, cc, localities, asyncPriorityInfo, processClass, dataFolder, memoryLimit, metricsConnFile, metricsPrefix, recoveredDiskFiles), "WorkerServer", UID(), &normalWorkerErrors()) );
state Future<Void> firstConnect = reportErrors( printOnFirstConnected(ci), "ClusterFirstConnectedError" );
Void _ = wait( quorum(v,1) );

14
fdbserver/workloads/AsyncFileRead.actor.cpp
View File

@ -60,13 +60,13 @@ struct IOLog {
void dumpMetrics(std::string name){
double elapsed = now() - startTime;
TraceEvent("ProcessLog")
.detail("name", name)
.detail("hz", count / elapsed)
.detail("latency_ms", 1e3 * sumSq / elapsed / 2.0)
.detail("avg_latency_ms", 1e3 * sum / count)
.detail("max_latency_ms", 1e3 * max)
.detail("startTime", startTime)
.detail("elapsed", elapsed);
.detail("Name", name)
.detail("Hz", count / elapsed)
.detail("Latency", sumSq / elapsed / 2.0)
.detail("AvgLatency", sum / count)
.detail("MaxLatency", max)
.detail("StartTime", startTime)
.detail("Elapsed", elapsed);
}
};

8
fdbserver/workloads/AtomicOps.actor.cpp
View File

@ -45,7 +45,7 @@ struct AtomicOpsWorkload : TestWorkload {
// Atomic OPs Min and And have modified behavior from api version 510. Hence allowing testing for older version (500) with a 10% probability
// Actual change of api Version happens in setup
apiVersion500 = ((sharedRandomNumber % 10) == 0);
TraceEvent("AtomicOpsApiVersion500").detail("apiVersion500", apiVersion500);
TraceEvent("AtomicOpsApiVersion500").detail("ApiVersion500", apiVersion500);
int64_t randNum = sharedRandomNumber / 10;
if(opType == -1)
@ -87,7 +87,7 @@ struct AtomicOpsWorkload : TestWorkload {
default:
ASSERT(false);
}
TraceEvent("AtomicWorkload").detail("opType", opType);
TraceEvent("AtomicWorkload").detail("OpType", opType);
}
virtual std::string description() { return "AtomicOps"; }
@ -188,7 +188,7 @@ struct AtomicOpsWorkload : TestWorkload {
}
if(tr.get(LiteralStringRef("xlogResult")).get() != tr.get(LiteralStringRef("xopsResult")).get()) {
TraceEvent(SevError, "LogMismatch").detail("logResult", printable(tr.get(LiteralStringRef("xlogResult")).get())).detail("opsResult", printable(tr.get(LiteralStringRef("xopsResult")).get().get()));
TraceEvent(SevError, "LogMismatch").detail("LogResult", printable(tr.get(LiteralStringRef("xlogResult")).get())).detail("OpsResult", printable(tr.get(LiteralStringRef("xopsResult")).get().get()));
}
if( self->opType == MutationRef::AddValue ) {
@ -202,7 +202,7 @@ struct AtomicOpsWorkload : TestWorkload {
logResult += intValue;
}
if(logResult != opsResult) {
TraceEvent(SevError, "LogAddMismatch").detail("logResult", logResult).detail("opResult", opsResult).detail("opsResultStr", printable(opsResultStr)).detail("size", opsResultStr.size());
TraceEvent(SevError, "LogAddMismatch").detail("LogResult", logResult).detail("OpResult", opsResult).detail("OpsResultStr", printable(opsResultStr)).detail("Size", opsResultStr.size());
}
}
break;

28
fdbserver/workloads/AtomicOpsApiCorrectness.actor.cpp
View File

@ -111,7 +111,7 @@ public:
break;
}
catch (Error& e) {
TraceEvent(SevInfo, "AtomicOpApiThrow").detail("errCode", e.code());
TraceEvent(SevInfo, "AtomicOpApiThrow").detail("ErrCode", e.code());
Void _ = wait(delay(1));
}
}
@ -120,7 +120,7 @@ public:
ASSERT(outputVal.present() && outputVal.get().size() == sizeof(uint64_t));
memcpy(&output, outputVal.get().begin(), outputVal.get().size());
if (output != intValue) {
TraceEvent(SevError, "AtomicOpSetOnNonExistingKeyUnexpectedOutput").detail("opOn", "StorageServer").detail("op", opType).detail("ExpectedOutput", intValue).detail("ActualOutput", output);
TraceEvent(SevError, "AtomicOpSetOnNonExistingKeyUnexpectedOutput").detail("OpOn", "StorageServer").detail("Op", opType).detail("ExpectedOutput", intValue).detail("ActualOutput", output);
self->testFailed = true;
}
@ -130,7 +130,7 @@ public:
ASSERT(outputVal.present() && outputVal.get().size() == sizeof(uint64_t));
memcpy(&output, outputVal.get().begin(), outputVal.get().size());
if (output != intValue) {
TraceEvent(SevError, "AtomicOpSetOnNonExistingKeyUnexpectedOutput").detail("opOn", "RYWLayer").detail("op", opType).detail("ExpectedOutput", intValue).detail("ActualOutput", output);
TraceEvent(SevError, "AtomicOpSetOnNonExistingKeyUnexpectedOutput").detail("OpOn", "RYWLayer").detail("Op", opType).detail("ExpectedOutput", intValue).detail("ActualOutput", output);
self->testFailed = true;
}
return Void();
@ -149,7 +149,7 @@ public:
break;
}
catch (Error& e) {
TraceEvent(SevInfo, "AtomicOpApiThrow").detail("errCode", e.code());
TraceEvent(SevInfo, "AtomicOpApiThrow").detail("ErrCode", e.code());
Void _ = wait(delay(1));
}
}
@ -158,7 +158,7 @@ public:
ASSERT(outputVal.present() && outputVal.get().size() == sizeof(uint64_t));
memcpy(&output, outputVal.get().begin(), outputVal.get().size());
if (output != 0) {
TraceEvent(SevError, "AtomicOpUnsetOnNonExistingKeyUnexpectedOutput").detail("opOn", "StorageServer").detail("op", opType).detail("ExpectedOutput", 0).detail("ActualOutput", output);
TraceEvent(SevError, "AtomicOpUnsetOnNonExistingKeyUnexpectedOutput").detail("OpOn", "StorageServer").detail("Op", opType).detail("ExpectedOutput", 0).detail("ActualOutput", output);
self->testFailed = true;
}
@ -168,7 +168,7 @@ public:
ASSERT(outputVal.present() && outputVal.get().size() == sizeof(uint64_t));
memcpy(&output, outputVal.get().begin(), outputVal.get().size());
if (output != 0) {
TraceEvent(SevError, "AtomicOpUnsetOnNonExistingKeyUnexpectedOutput").detail("opOn", "RYWLayer").detail("op", opType).detail("ExpectedOutput", 0).detail("ActualOutput", output);
TraceEvent(SevError, "AtomicOpUnsetOnNonExistingKeyUnexpectedOutput").detail("OpOn", "RYWLayer").detail("Op", opType).detail("ExpectedOutput", 0).detail("ActualOutput", output);
self->testFailed = true;
}
return Void();
@ -197,7 +197,7 @@ public:
break;
}
catch (Error& e) {
TraceEvent(SevInfo, "AtomicOpApiThrow").detail("errCode", e.code());
TraceEvent(SevInfo, "AtomicOpApiThrow").detail("ErrCode", e.code());
Void _ = wait(delay(1));
}
}
@ -205,7 +205,7 @@ public:
ASSERT(outputVal.present());
Value output = outputVal.get();
if (output != opFunc(existingVal, otherVal)) {
TraceEvent(SevError, "AtomicOpOnEmptyValueUnexpectedOutput").detail("opOn", "StorageServer").detail("op", opType).detail("ExpectedOutput", opFunc(existingVal, otherVal).toString()).detail("ActualOutput", output.toString());
TraceEvent(SevError, "AtomicOpOnEmptyValueUnexpectedOutput").detail("OpOn", "StorageServer").detail("Op", opType).detail("ExpectedOutput", opFunc(existingVal, otherVal).toString()).detail("ActualOutput", output.toString());
self->testFailed = true;
}
@ -214,7 +214,7 @@ public:
ASSERT(outputVal.present());
Value output = outputVal.get();
if (output != opFunc(existingVal, otherVal)) {
TraceEvent(SevError, "AtomicOpOnEmptyValueUnexpectedOutput").detail("opOn", "RYWLayer").detail("op", opType).detail("ExpectedOutput", opFunc(existingVal, otherVal).toString()).detail("ActualOutput", output.toString());
TraceEvent(SevError, "AtomicOpOnEmptyValueUnexpectedOutput").detail("OpOn", "RYWLayer").detail("Op", opType).detail("ExpectedOutput", opFunc(existingVal, otherVal).toString()).detail("ActualOutput", output.toString());
self->testFailed = true;
}
return Void();
@ -240,7 +240,7 @@ public:
break;
}
catch (Error& e) {
TraceEvent(SevInfo, "AtomicOpApiThrow").detail("errCode", e.code());
TraceEvent(SevInfo, "AtomicOpApiThrow").detail("ErrCode", e.code());
Void _ = wait(delay(1));
}
}
@ -250,7 +250,7 @@ public:
ASSERT(outputVal.present() && outputVal.get().size() == sizeof(uint64_t));
memcpy(&output, outputVal.get().begin(), outputVal.get().size());
if (output != opFunc(intValue1, intValue2)) {
TraceEvent(SevError, "AtomicOpApiCorrectnessUnexpectedOutput").detail("opOn", "StorageServer").detail("InValue1", intValue1).detail("InValue2", intValue2).detail("AtomicOp", opType).detail("ExpectedOutput", opFunc(intValue1, intValue2)).detail("ActualOutput", output);
TraceEvent(SevError, "AtomicOpApiCorrectnessUnexpectedOutput").detail("OpOn", "StorageServer").detail("InValue1", intValue1).detail("InValue2", intValue2).detail("AtomicOp", opType).detail("ExpectedOutput", opFunc(intValue1, intValue2)).detail("ActualOutput", output);
self->testFailed = true;
}
@ -261,7 +261,7 @@ public:
ASSERT(outputVal.present() && outputVal.get().size() == sizeof(uint64_t));
memcpy(&output, outputVal.get().begin(), outputVal.get().size());
if (output != opFunc(intValue1, intValue2)) {
TraceEvent(SevError, "AtomicOpApiCorrectnessUnexpectedOutput").detail("opOn", "RYWLayer").detail("InValue1", intValue1).detail("InValue2", intValue2).detail("AtomicOp", opType).detail("ExpectedOutput", opFunc(intValue1, intValue2)).detail("ActualOutput", output);
TraceEvent(SevError, "AtomicOpApiCorrectnessUnexpectedOutput").detail("OpOn", "RYWLayer").detail("InValue1", intValue1).detail("InValue2", intValue2).detail("AtomicOp", opType).detail("ExpectedOutput", opFunc(intValue1, intValue2)).detail("ActualOutput", output);
self->testFailed = true;
}
@ -272,7 +272,7 @@ public:
state int currentApiVersion = getApiVersion(cx);
state Key key = self->getTestKey("test_key_min_");
TraceEvent("AtomicOpCorrectnessApiWorkload").detail("opType", "MIN");
TraceEvent("AtomicOpCorrectnessApiWorkload").detail("OpType", "MIN");
// API Version 500
setApiVersion(&cx, 500);
TraceEvent(SevInfo, "Running Atomic Op Min Correctness Test Api Version 500");
@ -305,7 +305,7 @@ public:
state int currentApiVersion = getApiVersion(cx);
state Key key = self->getTestKey("test_key_and_");
TraceEvent("AtomicOpCorrectnessApiWorkload").detail("opType", "AND");
TraceEvent("AtomicOpCorrectnessApiWorkload").detail("OpType", "AND");
// API Version 500
setApiVersion(&cx, 500);
TraceEvent(SevInfo, "Running Atomic Op AND Correctness Test Api Version 500");

10
fdbserver/workloads/AtomicSwitchover.actor.cpp
View File

@ -100,13 +100,13 @@ struct AtomicSwitchoverWorkload : TestWorkload {
while (src != srcFuture.get().end() && bkp != bkpFuture.get().end()) {
KeyRef bkpKey = bkp->key.substr(backupPrefix.size());
if (src->key != bkpKey && src->value != bkp->value) {
TraceEvent(SevError, "MismatchKeyAndValue").detail("srcKey", printable(src->key)).detail("srcVal", printable(src->value)).detail("bkpKey", printable(bkpKey)).detail("bkpVal", printable(bkp->value));
TraceEvent(SevError, "MismatchKeyAndValue").detail("SrcKey", printable(src->key)).detail("SrcVal", printable(src->value)).detail("BkpKey", printable(bkpKey)).detail("BkpVal", printable(bkp->value));
}
else if (src->key != bkpKey) {
TraceEvent(SevError, "MismatchKey").detail("srcKey", printable(src->key)).detail("srcVal", printable(src->value)).detail("bkpKey", printable(bkpKey)).detail("bkpVal", printable(bkp->value));
TraceEvent(SevError, "MismatchKey").detail("SrcKey", printable(src->key)).detail("SrcVal", printable(src->value)).detail("BkpKey", printable(bkpKey)).detail("BkpVal", printable(bkp->value));
}
else if (src->value != bkp->value) {
TraceEvent(SevError, "MismatchValue").detail("srcKey", printable(src->key)).detail("srcVal", printable(src->value)).detail("bkpKey", printable(bkpKey)).detail("bkpVal", printable(bkp->value));
TraceEvent(SevError, "MismatchValue").detail("SrcKey", printable(src->key)).detail("SrcVal", printable(src->value)).detail("BkpKey", printable(bkpKey)).detail("BkpVal", printable(bkp->value));
}
begin = std::min(src->key, bkpKey);
if (src->key == bkpKey) {
@ -121,12 +121,12 @@ struct AtomicSwitchoverWorkload : TestWorkload {
}
}
while (src != srcFuture.get().end() && !bkpFuture.get().more) {
TraceEvent(SevError, "MissingBkpKey").detail("srcKey", printable(src->key)).detail("srcVal", printable(src->value));
TraceEvent(SevError, "MissingBkpKey").detail("SrcKey", printable(src->key)).detail("SrcVal", printable(src->value));
begin = src->key;
++src;
}
while (bkp != bkpFuture.get().end() && !srcFuture.get().more) {
TraceEvent(SevError, "MissingSrcKey").detail("bkpKey", printable(bkp->key.substr(backupPrefix.size()))).detail("bkpVal", printable(bkp->value));
TraceEvent(SevError, "MissingSrcKey").detail("BkpKey", printable(bkp->key.substr(backupPrefix.size()))).detail("BkpVal", printable(bkp->value));
begin = bkp->key;
++bkp;
}

128
fdbserver/workloads/BackupCorrectness.actor.cpp
View File

@ -83,8 +83,8 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
backupRanges.push_back_deep(backupRanges.arena(), KeyRangeRef(start, *i));
// Track the added range
TraceEvent("BARW_BackupCorrectness_Range", randomID).detail("rangeBegin", (beginRange < endRange) ? printable(beginRange) : printable(endRange))
.detail("rangeEnd", (beginRange < endRange) ? printable(endRange) : printable(beginRange));
TraceEvent("BARW_BackupCorrectnessRange", randomID).detail("RangeBegin", (beginRange < endRange) ? printable(beginRange) : printable(endRange))
.detail("RangeEnd", (beginRange < endRange) ? printable(endRange) : printable(beginRange));
}
}
}
@ -101,17 +101,17 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
if (clientId != 0)
return Void();
TraceEvent(SevInfo, "BARW_Param").detail("locked", locked);
TraceEvent(SevInfo, "BARW_Param").detail("backupAfter", backupAfter);
TraceEvent(SevInfo, "BARW_Param").detail("restoreAfter", restoreAfter);
TraceEvent(SevInfo, "BARW_Param").detail("performRestore", performRestore);
TraceEvent(SevInfo, "BARW_Param").detail("backupTag", printable(backupTag).c_str());
TraceEvent(SevInfo, "BARW_Param").detail("backupRangesCount", backupRangesCount);
TraceEvent(SevInfo, "BARW_Param").detail("backupRangeLengthMax", backupRangeLengthMax);
TraceEvent(SevInfo, "BARW_Param").detail("abortAndRestartAfter", abortAndRestartAfter);
TraceEvent(SevInfo, "BARW_Param").detail("differentialBackup", differentialBackup);
TraceEvent(SevInfo, "BARW_Param").detail("stopDifferentialAfter", stopDifferentialAfter);
TraceEvent(SevInfo, "BARW_Param").detail("agentRequest", agentRequest);
TraceEvent(SevInfo, "BARW_Param").detail("Locked", locked);
TraceEvent(SevInfo, "BARW_Param").detail("BackupAfter", backupAfter);
TraceEvent(SevInfo, "BARW_Param").detail("RestoreAfter", restoreAfter);
TraceEvent(SevInfo, "BARW_Param").detail("PerformRestore", performRestore);
TraceEvent(SevInfo, "BARW_Param").detail("BackupTag", printable(backupTag).c_str());
TraceEvent(SevInfo, "BARW_Param").detail("BackupRangesCount", backupRangesCount);
TraceEvent(SevInfo, "BARW_Param").detail("BackupRangeLengthMax", backupRangeLengthMax);
TraceEvent(SevInfo, "BARW_Param").detail("AbortAndRestartAfter", abortAndRestartAfter);
TraceEvent(SevInfo, "BARW_Param").detail("DifferentialBackup", differentialBackup);
TraceEvent(SevInfo, "BARW_Param").detail("StopDifferentialAfter", stopDifferentialAfter);
TraceEvent(SevInfo, "BARW_Param").detail("AgentRequest", agentRequest);
return _start(cx, this);
}
@ -150,19 +150,19 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
Void _ = wait( delay( startDelay ));
if (startDelay || BUGGIFY) {
TraceEvent("BARW_doBackupAbortBackup1", randomID).detail("tag", printable(tag)).detail("startDelay", startDelay);
TraceEvent("BARW_DoBackupAbortBackup1", randomID).detail("Tag", printable(tag)).detail("StartDelay", startDelay);
try {
Void _ = wait(backupAgent->abortBackup(cx, tag.toString()));
}
catch (Error& e) {
TraceEvent("BARW_doBackupAbortBackupException", randomID).detail("tag", printable(tag)).error(e);
TraceEvent("BARW_DoBackupAbortBackupException", randomID).detail("Tag", printable(tag)).error(e);
if (e.code() != error_code_backup_unneeded)
throw;
}
}
TraceEvent("BARW_doBackupSubmitBackup", randomID).detail("tag", printable(tag)).detail("stopWhenDone", stopDifferentialDelay ? "False" : "True");
TraceEvent("BARW_DoBackupSubmitBackup", randomID).detail("Tag", printable(tag)).detail("StopWhenDone", stopDifferentialDelay ? "False" : "True");
state std::string backupContainer = "file://simfdb/backups/";
state Future<Void> status = statusLoop(cx, tag.toString());
@ -171,7 +171,7 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
Void _ = wait(backupAgent->submitBackup(cx, StringRef(backupContainer), g_random->randomInt(0, 100), tag.toString(), backupRanges, stopDifferentialDelay ? false : true));
}
catch (Error& e) {
TraceEvent("BARW_doBackupSubmitBackupException", randomID).detail("tag", printable(tag)).error(e);
TraceEvent("BARW_DoBackupSubmitBackupException", randomID).detail("Tag", printable(tag)).error(e);
if (e.code() != error_code_backup_unneeded && e.code() != error_code_backup_duplicate)
throw;
}
@ -182,12 +182,12 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
if (stopDifferentialDelay) {
TEST(!stopDifferentialFuture.isReady()); //Restore starts at specified time
Void _ = wait(stopDifferentialFuture);
TraceEvent("BARW_doBackupWaitToDiscontinue", randomID).detail("tag", printable(tag)).detail("differentialAfter", stopDifferentialDelay);
TraceEvent("BARW_DoBackupWaitToDiscontinue", randomID).detail("Tag", printable(tag)).detail("DifferentialAfter", stopDifferentialDelay);
try {
if (BUGGIFY) {
state KeyBackedTag backupTag = makeBackupTag(tag.toString());
TraceEvent("BARW_doBackupWaitForRestorable", randomID).detail("tag", backupTag.tagName);
TraceEvent("BARW_DoBackupWaitForRestorable", randomID).detail("Tag", backupTag.tagName);
// Wait until the backup is in a restorable state
state int resultWait = wait(backupAgent->waitBackup(cx, backupTag.tagName, false));
UidAndAbortedFlagT uidFlag = wait(backupTag.getOrThrow(cx));
@ -202,56 +202,56 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
restorable = desc.maxRestorableVersion.present();
}
TraceEvent("BARW_lastBackupContainer", randomID)
.detail("backupTag", printable(tag))
.detail("lastBackupContainer", lastBackupContainer ? lastBackupContainer->getURL() : "")
.detail("logUid", logUid).detail("waitStatus", resultWait).detail("restorable", restorable);
TraceEvent("BARW_LastBackupContainer", randomID)
.detail("BackupTag", printable(tag))
.detail("LastBackupContainer", lastBackupContainer ? lastBackupContainer->getURL() : "")
.detail("LogUid", logUid).detail("WaitStatus", resultWait).detail("Restorable", restorable);
// Do not check the backup, if aborted
if (resultWait == BackupAgentBase::STATE_ABORTED) {
}
// Ensure that a backup container was found
else if (!lastBackupContainer) {
TraceEvent("BARW_missingBackupContainer", randomID).detail("logUid", logUid).detail("backupTag", printable(tag)).detail("waitStatus", resultWait);
TraceEvent("BARW_MissingBackupContainer", randomID).detail("LogUid", logUid).detail("BackupTag", printable(tag)).detail("WaitStatus", resultWait);
printf("BackupCorrectnessMissingBackupContainer tag: %s status: %d\n", printable(tag).c_str(), resultWait);
}
// Check that backup is restorable
else {
if(!restorable) {
TraceEvent("BARW_notRestorable", randomID).detail("logUid", logUid).detail("backupTag", printable(tag))
.detail("backupFolder", lastBackupContainer->getURL()).detail("waitStatus", resultWait);
TraceEvent("BARW_NotRestorable", randomID).detail("LogUid", logUid).detail("BackupTag", printable(tag))
.detail("BackupFolder", lastBackupContainer->getURL()).detail("WaitStatus", resultWait);
printf("BackupCorrectnessNotRestorable: tag: %s\n", printable(tag).c_str());
}
}
// Abort the backup, if not the first backup because the second backup may have aborted the backup by now
if (startDelay) {
TraceEvent("BARW_doBackupAbortBackup2", randomID).detail("tag", printable(tag))
.detail("waitStatus", resultWait)
.detail("lastBackupContainer", lastBackupContainer ? lastBackupContainer->getURL() : "")
.detail("restorable", restorable);
TraceEvent("BARW_DoBackupAbortBackup2", randomID).detail("Tag", printable(tag))
.detail("WaitStatus", resultWait)
.detail("LastBackupContainer", lastBackupContainer ? lastBackupContainer->getURL() : "")
.detail("Restorable", restorable);
Void _ = wait(backupAgent->abortBackup(cx, tag.toString()));
}
else {
TraceEvent("BARW_doBackupDiscontinueBackup", randomID).detail("tag", printable(tag)).detail("differentialAfter", stopDifferentialDelay);
TraceEvent("BARW_DoBackupDiscontinueBackup", randomID).detail("Tag", printable(tag)).detail("DifferentialAfter", stopDifferentialDelay);
Void _ = wait(backupAgent->discontinueBackup(cx, tag));
}
}
else {
TraceEvent("BARW_doBackupDiscontinueBackup", randomID).detail("tag", printable(tag)).detail("differentialAfter", stopDifferentialDelay);
TraceEvent("BARW_DoBackupDiscontinueBackup", randomID).detail("Tag", printable(tag)).detail("DifferentialAfter", stopDifferentialDelay);
Void _ = wait(backupAgent->discontinueBackup(cx, tag));
}
}
catch (Error& e) {
TraceEvent("BARW_doBackupDiscontinueBackupException", randomID).detail("tag", printable(tag)).error(e);
TraceEvent("BARW_DoBackupDiscontinueBackupException", randomID).detail("Tag", printable(tag)).error(e);
if (e.code() != error_code_backup_unneeded && e.code() != error_code_backup_duplicate)
throw;
}
}
// Wait for the backup to complete
TraceEvent("BARW_doBackupWaitBackup", randomID).detail("tag", printable(tag));
TraceEvent("BARW_DoBackupWaitBackup", randomID).detail("Tag", printable(tag));
state int statusValue = wait(backupAgent->waitBackup(cx, tag.toString(), true));
state std::string statusText;
@ -260,8 +260,8 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
statusText = _statusText;
// Can we validate anything about status?
TraceEvent("BARW_doBackupComplete", randomID).detail("tag", printable(tag))
.detail("status", statusText).detail("statusValue", statusValue);
TraceEvent("BARW_DoBackupComplete", randomID).detail("Tag", printable(tag))
.detail("Status", statusText).detail("StatusValue", statusValue);
return Void();
}
@ -287,7 +287,7 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
if (rowCount > 0) {
try {
Version _ = wait(backupAgent->restore(cx, self->backupTag, KeyRef(lastBackupContainer), true, -1, true, normalKeys, Key(), Key(), self->locked));
TraceEvent(SevError, "BARW_restore_allowed_overwritting_database", randomID);
TraceEvent(SevError, "BARW_RestoreAllowedOverwrittingDatabase", randomID);
ASSERT(false);
}
catch (Error &e) {
@ -304,9 +304,9 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
state FileBackupAgent backupAgent;
state Future<Void> extraBackup;
state bool extraTasks = false;
TraceEvent("BARW_Arguments").detail("backupTag", printable(self->backupTag)).detail("performRestore", self->performRestore)
.detail("backupAfter", self->backupAfter).detail("restoreAfter", self->restoreAfter)
.detail("abortAndRestartAfter", self->abortAndRestartAfter).detail("differentialAfter", self->stopDifferentialAfter);
TraceEvent("BARW_Arguments").detail("BackupTag", printable(self->backupTag)).detail("PerformRestore", self->performRestore)
.detail("BackupAfter", self->backupAfter).detail("RestoreAfter", self->restoreAfter)
.detail("AbortAndRestartAfter", self->abortAndRestartAfter).detail("DifferentialAfter", self->stopDifferentialAfter);
state UID randomID = g_nondeterministic_random->randomUniqueID();
if(self->allowPauses && BUGGIFY) {
@ -324,17 +324,17 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
// backup
Void _ = wait(delay(self->backupAfter));
TraceEvent("BARW_doBackup1", randomID).detail("tag", printable(self->backupTag));
TraceEvent("BARW_DoBackup1", randomID).detail("Tag", printable(self->backupTag));
state Promise<Void> submitted;
state Future<Void> b = doBackup(self, 0, &backupAgent, cx, self->backupTag, self->backupRanges, self->stopDifferentialAfter, submitted);
if (self->abortAndRestartAfter) {
TraceEvent("BARW_doBackup2", randomID).detail("tag", printable(self->backupTag)).detail("abortWait", self->abortAndRestartAfter);
TraceEvent("BARW_DoBackup2", randomID).detail("Tag", printable(self->backupTag)).detail("AbortWait", self->abortAndRestartAfter);
Void _ = wait(submitted.getFuture());
b = b && doBackup(self, self->abortAndRestartAfter, &backupAgent, cx, self->backupTag, self->backupRanges, self->stopDifferentialAfter, Promise<Void>());
}
TraceEvent("BARW_doBackupWait", randomID).detail("backupTag", printable(self->backupTag)).detail("abortAndRestartAfter", self->abortAndRestartAfter);
TraceEvent("BARW_DoBackupWait", randomID).detail("BackupTag", printable(self->backupTag)).detail("AbortAndRestartAfter", self->abortAndRestartAfter);
try {
Void _ = wait(b);
} catch( Error &e ) {
@ -344,7 +344,7 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
throw;
return Void();
}
TraceEvent("BARW_doBackupDone", randomID).detail("backupTag", printable(self->backupTag)).detail("abortAndRestartAfter", self->abortAndRestartAfter);
TraceEvent("BARW_DoBackupDone", randomID).detail("BackupTag", printable(self->backupTag)).detail("AbortAndRestartAfter", self->abortAndRestartAfter);
state KeyBackedTag keyBackedTag = makeBackupTag(self->backupTag.toString());
UidAndAbortedFlagT uidFlag = wait(keyBackedTag.getOrThrow(cx));
@ -354,12 +354,12 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
// Occasionally start yet another backup that might still be running when we restore
if (!self->locked && BUGGIFY) {
TraceEvent("BARW_submitBackup2", randomID).detail("tag", printable(self->backupTag));
TraceEvent("BARW_SubmitBackup2", randomID).detail("Tag", printable(self->backupTag));
try {
extraBackup = backupAgent.submitBackup(cx, LiteralStringRef("file://simfdb/backups/"), g_random->randomInt(0, 100), self->backupTag.toString(), self->backupRanges, true);
}
catch (Error& e) {
TraceEvent("BARW_submitBackup2Exception", randomID).detail("backupTag", printable(self->backupTag)).error(e);
TraceEvent("BARW_SubmitBackup2Exception", randomID).detail("BackupTag", printable(self->backupTag)).error(e);
if (e.code() != error_code_backup_unneeded && e.code() != error_code_backup_duplicate)
throw;
}
@ -379,7 +379,7 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
}));
// restore database
TraceEvent("BARW_restore", randomID).detail("lastBackupContainer", lastBackupContainer->getURL()).detail("restoreAfter", self->restoreAfter).detail("backupTag", printable(self->backupTag));
TraceEvent("BARW_Restore", randomID).detail("LastBackupContainer", lastBackupContainer->getURL()).detail("RestoreAfter", self->restoreAfter).detail("BackupTag", printable(self->backupTag));
state std::vector<Future<Version>> restores;
state std::vector<Standalone<StringRef>> restoreTags;
@ -417,23 +417,23 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
}
if (extraBackup.isValid()) {
TraceEvent("BARW_waitExtraBackup", randomID).detail("backupTag", printable(self->backupTag));
TraceEvent("BARW_WaitExtraBackup", randomID).detail("BackupTag", printable(self->backupTag));
extraTasks = true;
try {
Void _ = wait(extraBackup);
}
catch (Error& e) {
TraceEvent("BARW_extraBackupException", randomID).detail("backupTag", printable(self->backupTag)).error(e);
TraceEvent("BARW_ExtraBackupException", randomID).detail("BackupTag", printable(self->backupTag)).error(e);
if (e.code() != error_code_backup_unneeded && e.code() != error_code_backup_duplicate)
throw;
}
TraceEvent("BARW_abortBackupExtra", randomID).detail("backupTag", printable(self->backupTag));
TraceEvent("BARW_AbortBackupExtra", randomID).detail("BackupTag", printable(self->backupTag));
try {
Void _ = wait(backupAgent.abortBackup(cx, self->backupTag.toString()));
}
catch (Error& e) {
TraceEvent("BARW_abortBackupExtraException", randomID).error(e);
TraceEvent("BARW_AbortBackupExtraException", randomID).error(e);
if (e.code() != error_code_backup_unneeded)
throw;
}
@ -449,7 +449,7 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
loop {
state Reference<ReadYourWritesTransaction> tr(new ReadYourWritesTransaction(cx));
TraceEvent("BARW_check_leftoverkeys", randomID).detail("backupTag", printable(self->backupTag));
TraceEvent("BARW_CheckLeftoverKeys", randomID).detail("BackupTag", printable(self->backupTag));
try {
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
@ -459,12 +459,12 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
// Check the left over tasks
// We have to wait for the list to empty since an abort and get status
// can leave extra tasks in the queue
TraceEvent("BARW_check_leftovertasks", randomID).detail("backupTag", printable(self->backupTag));
TraceEvent("BARW_CheckLeftoverTasks", randomID).detail("BackupTag", printable(self->backupTag));
state int64_t taskCount = wait( backupAgent.getTaskCount(tr) );
state int waitCycles = 0;
if ((taskCount) && (0)) {
TraceEvent("BARW_EndingNonzeroTaskCount", randomID).detail("backupTag", printable(self->backupTag)).detail("taskCount", taskCount).detail("waitCycles", waitCycles);
TraceEvent("BARW_EndingNonzeroTaskCount", randomID).detail("BackupTag", printable(self->backupTag)).detail("TaskCount", taskCount).detail("WaitCycles", waitCycles);
printf("EndingNonZeroTasks: %ld\n", (long) taskCount);
Void _ = wait(TaskBucket::debugPrintRange(cx, LiteralStringRef("\xff"), StringRef()));
}
@ -472,7 +472,7 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
loop {
waitCycles ++;
TraceEvent("BARW_NonzeroTaskWait", randomID).detail("backupTag", printable(self->backupTag)).detail("taskCount", taskCount).detail("waitCycles", waitCycles);
TraceEvent("BARW_NonzeroTaskWait", randomID).detail("BackupTag", printable(self->backupTag)).detail("TaskCount", taskCount).detail("WaitCycles", waitCycles);
printf("%.6f %-10s Wait #%4d for %lld tasks to end\n", now(), randomID.toString().c_str(), waitCycles, (long long) taskCount);
Void _ = wait(delay(5.0));
@ -488,7 +488,7 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
if (taskCount) {
displaySystemKeys ++;
TraceEvent(SevError, "BARW_NonzeroTaskCount", randomID).detail("backupTag", printable(self->backupTag)).detail("taskCount", taskCount).detail("waitCycles", waitCycles);
TraceEvent(SevError, "BARW_NonzeroTaskCount", randomID).detail("BackupTag", printable(self->backupTag)).detail("TaskCount", taskCount).detail("WaitCycles", waitCycles);
printf("BackupCorrectnessLeftOverLogTasks: %ld\n", (long) taskCount);
}
@ -500,10 +500,10 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
if (agentValues.size() > 0) {
displaySystemKeys ++;
printf("BackupCorrectnessLeftOverMutationKeys: (%d) %s\n", agentValues.size(), printable(backupAgentKey).c_str());
TraceEvent(SevError, "BackupCorrectnessLeftOverMutationKeys", randomID).detail("backupTag", printable(self->backupTag))
.detail("LeftOverKeys", agentValues.size()).detail("keySpace", printable(backupAgentKey));
TraceEvent(SevError, "BackupCorrectnessLeftOverMutationKeys", randomID).detail("BackupTag", printable(self->backupTag))
.detail("LeftOverKeys", agentValues.size()).detail("KeySpace", printable(backupAgentKey));
for (auto & s : agentValues) {
TraceEvent("BARW_LeftOverKey", randomID).detail("key", printable(StringRef(s.key.toString()))).detail("value", printable(StringRef(s.value.toString())));
TraceEvent("BARW_LeftOverKey", randomID).detail("Key", printable(StringRef(s.key.toString()))).detail("Value", printable(StringRef(s.value.toString())));
printf(" Key: %-50s Value: %s\n", printable(StringRef(s.key.toString())).c_str(), printable(StringRef(s.value.toString())).c_str());
}
}
@ -513,7 +513,7 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
Optional<Value> latestVersion = wait(tr->get(backupLatestVersionsKey));
if (latestVersion.present()) {
TraceEvent(SevError, "BackupCorrectnessLeftOverVersionKey", randomID).detail("backupTag", printable(self->backupTag)).detail("backupLatestVersionsKey", backupLatestVersionsKey.printable()).detail("destUidValue", destUidValue.printable());
TraceEvent(SevError, "BackupCorrectnessLeftOverVersionKey", randomID).detail("BackupTag", printable(self->backupTag)).detail("BackupLatestVersionsKey", backupLatestVersionsKey.printable()).detail("DestUidValue", destUidValue.printable());
} else {
printf("No left over backup version key\n");
}
@ -526,8 +526,8 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
if (logValues.size() > 0) {
displaySystemKeys ++;
printf("BackupCorrectnessLeftOverLogKeys: (%d) %s\n", logValues.size(), printable(backupLogValuesKey).c_str());
TraceEvent(SevError, "BackupCorrectnessLeftOverLogKeys", randomID).detail("backupTag", printable(self->backupTag))
.detail("LeftOverKeys", logValues.size()).detail("keySpace", printable(backupLogValuesKey));
TraceEvent(SevError, "BackupCorrectnessLeftOverLogKeys", randomID).detail("BackupTag", printable(self->backupTag))
.detail("LeftOverKeys", logValues.size()).detail("KeySpace", printable(backupLogValuesKey));
}
else {
printf("No left over backup log keys\n");
@ -537,7 +537,7 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
break;
}
catch (Error &e) {
TraceEvent("BARW_checkException", randomID).error(e);
TraceEvent("BARW_CheckException", randomID).error(e);
Void _ = wait(tr->onError(e));
}
}
@ -546,7 +546,7 @@ struct BackupAndRestoreCorrectnessWorkload : TestWorkload {
Void _ = wait(TaskBucket::debugPrintRange(cx, LiteralStringRef("\xff"), StringRef()));
}
TraceEvent("BARW_complete", randomID).detail("backupTag", printable(self->backupTag));
TraceEvent("BARW_Complete", randomID).detail("BackupTag", printable(self->backupTag));
// Decrement the backup agent requets
if (self->agentRequest) {

2
fdbserver/workloads/BackupToDBAbort.actor.cpp
View File

@ -73,7 +73,7 @@ struct BackupToDBAbort : TestWorkload {
ACTOR static Future<Void> _start(BackupToDBAbort* self, Database cx) {
state DatabaseBackupAgent backupAgent(cx);
TraceEvent("BDBA_Start").detail("delay", self->abortDelay);
TraceEvent("BDBA_Start").detail("Delay", self->abortDelay);
Void _ = wait(delay(self->abortDelay));
TraceEvent("BDBA_Wait");
int _ = wait( backupAgent.waitBackup(self->extraDB, BackupAgentBase::getDefaultTag(), false) );

108
fdbserver/workloads/BackupToDBCorrectness.actor.cpp
View File

@ -98,8 +98,8 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
backupRanges.push_back_deep(backupRanges.arena(), (beginRange < endRange) ? KeyRangeRef(beginRange, endRange) : KeyRangeRef(endRange, beginRange));
// Track the added range
TraceEvent("BackupCorrectness_Range", randomID).detail("rangeBegin", (beginRange < endRange) ? printable(beginRange) : printable(endRange))
.detail("rangeEnd", (beginRange < endRange) ? printable(endRange) : printable(beginRange));
TraceEvent("BackupCorrectness_Range", randomID).detail("RangeBegin", (beginRange < endRange) ? printable(beginRange) : printable(endRange))
.detail("RangeEnd", (beginRange < endRange) ? printable(endRange) : printable(beginRange));
}
}
@ -107,7 +107,7 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
Reference<Cluster> extraCluster = Cluster::createCluster(extraFile, -1);
extraDB = extraCluster->createDatabase(LiteralStringRef("DB")).get();
TraceEvent("BARW_start").detail("locked", locked);
TraceEvent("BARW_Start").detail("Locked", locked);
}
virtual std::string description() {
@ -151,13 +151,13 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
while (src != srcFuture.get().end() && bkp != bkpFuture.get().end()) {
KeyRef bkpKey = bkp->key.substr(backupPrefix.size());
if (src->key != bkpKey && src->value != bkp->value) {
TraceEvent(SevError, "MismatchKeyAndValue").detail("srcKey", printable(src->key)).detail("srcVal", printable(src->value)).detail("bkpKey", printable(bkpKey)).detail("bkpVal", printable(bkp->value));
TraceEvent(SevError, "MismatchKeyAndValue").detail("SrcKey", printable(src->key)).detail("SrcVal", printable(src->value)).detail("BkpKey", printable(bkpKey)).detail("BkpVal", printable(bkp->value));
}
else if (src->key != bkpKey) {
TraceEvent(SevError, "MismatchKey").detail("srcKey", printable(src->key)).detail("srcVal", printable(src->value)).detail("bkpKey", printable(bkpKey)).detail("bkpVal", printable(bkp->value));
TraceEvent(SevError, "MismatchKey").detail("SrcKey", printable(src->key)).detail("SrcVal", printable(src->value)).detail("BkpKey", printable(bkpKey)).detail("BkpVal", printable(bkp->value));
}
else if (src->value != bkp->value) {
TraceEvent(SevError, "MismatchValue").detail("srcKey", printable(src->key)).detail("srcVal", printable(src->value)).detail("bkpKey", printable(bkpKey)).detail("bkpVal", printable(bkp->value));
TraceEvent(SevError, "MismatchValue").detail("SrcKey", printable(src->key)).detail("SrcVal", printable(src->value)).detail("BkpKey", printable(bkpKey)).detail("BkpVal", printable(bkp->value));
}
begin = std::min(src->key, bkpKey);
if (src->key == bkpKey) {
@ -172,12 +172,12 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
}
}
while (src != srcFuture.get().end() && !bkpFuture.get().more) {
TraceEvent(SevError, "MissingBkpKey").detail("srcKey", printable(src->key)).detail("srcVal", printable(src->value));
TraceEvent(SevError, "MissingBkpKey").detail("SrcKey", printable(src->key)).detail("SrcVal", printable(src->value));
begin = src->key;
++src;
}
while (bkp != bkpFuture.get().end() && !srcFuture.get().more) {
TraceEvent(SevError, "MissingSrcKey").detail("bkpKey", printable(bkp->key.substr(backupPrefix.size()))).detail("bkpVal", printable(bkp->value));
TraceEvent(SevError, "MissingSrcKey").detail("BkpKey", printable(bkp->key.substr(backupPrefix.size()))).detail("BkpVal", printable(bkp->value));
begin = bkp->key;
++bkp;
}
@ -209,13 +209,13 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
Void _ = wait( delay( startDelay ));
if (startDelay || BUGGIFY) {
TraceEvent("BARW_doBackup abortBackup1", randomID).detail("tag", printable(tag)).detail("startDelay", startDelay);
TraceEvent("BARW_DoBackupAbortBackup1", randomID).detail("Tag", printable(tag)).detail("StartDelay", startDelay);
try {
Void _ = wait(backupAgent->abortBackup(cx, tag));
}
catch (Error& e) {
TraceEvent("BARW_doBackup abortBackup Exception", randomID).detail("tag", printable(tag)).error(e);
TraceEvent("BARW_DoBackupAbortBackupException", randomID).detail("Tag", printable(tag)).error(e);
if (e.code() != error_code_backup_unneeded)
throw;
}
@ -226,7 +226,7 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
// pointless in this test) because separating them causes rare errors where the SubmitBackup commit result is indeterminite but the
// submission was in fact successful and the backup actually completes before the retry of SubmitBackup so this second call to submit
// fails because the destination range is no longer empty.
TraceEvent("BARW_doBackup clearAndSubmitBackup", randomID).detail("tag", printable(tag)).detail("stopWhenDone", stopDifferentialDelay ? "False" : "True");
TraceEvent("BARW_DoBackupClearAndSubmitBackup", randomID).detail("Tag", printable(tag)).detail("StopWhenDone", stopDifferentialDelay ? "False" : "True");
try {
state Reference<ReadYourWritesTransaction> tr2(new ReadYourWritesTransaction(self->extraDB));
@ -250,7 +250,7 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
}
}
catch (Error &e) {
TraceEvent("BARW_doBackup submitBackup Exception", randomID).detail("tag", printable(tag)).error(e);
TraceEvent("BARW_DoBackupSubmitBackupException", randomID).detail("Tag", printable(tag)).error(e);
if (e.code() != error_code_backup_unneeded && e.code() != error_code_backup_duplicate) {
throw e;
}
@ -264,37 +264,37 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
if (stopDifferentialDelay) {
TEST(!stopDifferentialFuture.isReady()); //Restore starts at specified time
Void _ = wait(stopDifferentialFuture);
TraceEvent("BARW_doBackup waitToDiscontinue", randomID).detail("tag", printable(tag)).detail("differentialAfter", stopDifferentialDelay);
TraceEvent("BARW_DoBackupWaitToDiscontinue", randomID).detail("Tag", printable(tag)).detail("DifferentialAfter", stopDifferentialDelay);
state bool aborted = false;
try {
if (BUGGIFY) {
TraceEvent("BARW_doBackup waitForRestorable", randomID).detail("tag", printable(tag));
TraceEvent("BARW_DoBackupWaitForRestorable", randomID).detail("Tag", printable(tag));
// Wait until the backup is in a restorable state
state int resultWait = wait(backupAgent->waitBackup(cx, tag, false));
TraceEvent("BARW_lastBackupFolder", randomID).detail("backupTag", printable(tag))
.detail("logUid", logUid).detail("waitStatus", resultWait);
TraceEvent("BARW_LastBackupFolder", randomID).detail("BackupTag", printable(tag))
.detail("LogUid", logUid).detail("WaitStatus", resultWait);
// Abort the backup, if not the first backup because the second backup may have aborted the backup by now
if (startDelay) {
TraceEvent("BARW_doBackup abortBackup2", randomID).detail("tag", printable(tag)).detail("waitStatus", resultWait);
TraceEvent("BARW_DoBackupAbortBackup2", randomID).detail("Tag", printable(tag)).detail("WaitStatus", resultWait);
aborted = true;
Void _ = wait(backupAgent->abortBackup(cx, tag));
}
else {
TraceEvent("BARW_doBackup discontinueBackup", randomID).detail("tag", printable(tag)).detail("differentialAfter", stopDifferentialDelay);
TraceEvent("BARW_DoBackupDiscontinueBackup", randomID).detail("Tag", printable(tag)).detail("DifferentialAfter", stopDifferentialDelay);
Void _ = wait(backupAgent->discontinueBackup(cx, tag));
}
}
else {
TraceEvent("BARW_doBackup discontinueBackup", randomID).detail("tag", printable(tag)).detail("differentialAfter", stopDifferentialDelay);
TraceEvent("BARW_DoBackupDiscontinueBackup", randomID).detail("Tag", printable(tag)).detail("DifferentialAfter", stopDifferentialDelay);
Void _ = wait(backupAgent->discontinueBackup(cx, tag));
}
}
catch (Error& e) {
TraceEvent("BARW_doBackup discontinueBackup Exception", randomID).detail("tag", printable(tag)).error(e);
TraceEvent("BARW_DoBackupDiscontinueBackupException", randomID).detail("Tag", printable(tag)).error(e);
if (e.code() != error_code_backup_unneeded && e.code() != error_code_backup_duplicate)
throw;
}
@ -305,7 +305,7 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
}
// Wait for the backup to complete
TraceEvent("BARW_doBackup waitBackup", randomID).detail("tag", printable(tag));
TraceEvent("BARW_DoBackupWaitBackup", randomID).detail("Tag", printable(tag));
UID _destUid = wait(backupAgent->getDestUid(cx, logUid));
self->destUid = _destUid;
@ -319,8 +319,8 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
statusText = _statusText;
// Can we validate anything about status?
TraceEvent("BARW_doBackup complete", randomID).detail("tag", printable(tag))
.detail("status", statusText).detail("statusValue", statusValue);
TraceEvent("BARW_DoBackupComplete", randomID).detail("Tag", printable(tag))
.detail("Status", statusText).detail("StatusValue", statusValue);
return Void();
}
@ -336,7 +336,7 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
loop {
state Reference<ReadYourWritesTransaction> tr(new ReadYourWritesTransaction(cx));
TraceEvent("BARW_check_leftoverkeys", randomID).detail("backupTag", printable(tag));
TraceEvent("BARW_CheckLeftoverKeys", randomID).detail("BackupTag", printable(tag));
try {
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
@ -344,12 +344,12 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
// Check the left over tasks
// We have to wait for the list to empty since an abort and get status
// can leave extra tasks in the queue
TraceEvent("BARW_check_leftovertasks", randomID).detail("backupTag", printable(tag));
TraceEvent("BARW_CheckLeftoverTasks", randomID).detail("BackupTag", printable(tag));
state int64_t taskCount = wait( backupAgent->getTaskCount(tr) );
state int waitCycles = 0;
if ((taskCount) && (0)) {
TraceEvent("BARW_EndingNonzeroTaskCount", randomID).detail("backupTag", printable(tag)).detail("taskCount", taskCount).detail("waitCycles", waitCycles);
TraceEvent("BARW_EndingNonzeroTaskCount", randomID).detail("BackupTag", printable(tag)).detail("TaskCount", taskCount).detail("WaitCycles", waitCycles);
printf("EndingNonZeroTasks: %ld\n", (long) taskCount);
Void _ = wait(TaskBucket::debugPrintRange(cx, LiteralStringRef("\xff"), StringRef()));
}
@ -357,7 +357,7 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
loop {
waitCycles ++;
TraceEvent("BARW_NonzeroTaskWait", randomID).detail("backupTag", printable(tag)).detail("taskCount", taskCount).detail("waitCycles", waitCycles);
TraceEvent("BARW_NonzeroTaskWait", randomID).detail("BackupTag", printable(tag)).detail("TaskCount", taskCount).detail("WaitCycles", waitCycles);
printf("%.6f %-10s Wait #%4d for %lld tasks to end\n", now(), randomID.toString().c_str(), waitCycles, (long long) taskCount);
Void _ = wait(delay(5.0));
@ -373,8 +373,8 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
if (taskCount) {
displaySystemKeys ++;
TraceEvent(SevError, "BARW_NonzeroTaskCount", randomID).detail("backupTag", printable(tag)).detail("taskCount", taskCount).detail("waitCycles", waitCycles);
printf("BackupCorrectnessLeftOverLogTasks: %ld\n", (long) taskCount);
TraceEvent(SevError, "BARW_NonzeroTaskCount", randomID).detail("BackupTag", printable(tag)).detail("TaskCount", taskCount).detail("WaitCycles", waitCycles);
printf("BackupCorrectnessLeftoverLogTasks: %ld\n", (long) taskCount);
}
Standalone<RangeResultRef> agentValues = wait(tr->getRange(KeyRange(KeyRangeRef(backupAgentKey, strinc(backupAgentKey))), 100));
@ -382,11 +382,11 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
// Error if the system keyspace for the backup tag is not empty
if (agentValues.size() > 0) {
displaySystemKeys++;
printf("BackupCorrectnessLeftOverMutationKeys: (%d) %s\n", agentValues.size(), printable(backupAgentKey).c_str());
TraceEvent(SevError, "BackupCorrectnessLeftOverMutationKeys", randomID).detail("backupTag", printable(tag))
.detail("LeftOverKeys", agentValues.size()).detail("keySpace", printable(backupAgentKey));
printf("BackupCorrectnessLeftoverMutationKeys: (%d) %s\n", agentValues.size(), printable(backupAgentKey).c_str());
TraceEvent(SevError, "BackupCorrectnessLeftoverMutationKeys", randomID).detail("BackupTag", printable(tag))
.detail("LeftoverKeys", agentValues.size()).detail("KeySpace", printable(backupAgentKey));
for (auto & s : agentValues) {
TraceEvent("BARW_LeftOverKey", randomID).detail("key", printable(StringRef(s.key.toString()))).detail("value", printable(StringRef(s.value.toString())));
TraceEvent("BARW_LeftoverKey", randomID).detail("Key", printable(StringRef(s.key.toString()))).detail("Value", printable(StringRef(s.value.toString())));
printf(" Key: %-50s Value: %s\n", printable(StringRef(s.key.toString())).c_str(), printable(StringRef(s.value.toString())).c_str());
}
}
@ -396,7 +396,7 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
Optional<Value> latestVersion = wait(tr->get(backupLatestVersionsKey));
if (latestVersion.present()) {
TraceEvent(SevError, "BackupCorrectnessLeftOverVersionKey", randomID).detail("backupTag", printable(tag)).detail("key", backupLatestVersionsKey.printable()).detail("value", BinaryReader::fromStringRef<Version>(latestVersion.get(), Unversioned()));
TraceEvent(SevError, "BackupCorrectnessLeftoverVersionKey", randomID).detail("BackupTag", printable(tag)).detail("Key", backupLatestVersionsKey.printable()).detail("Value", BinaryReader::fromStringRef<Version>(latestVersion.get(), Unversioned()));
} else {
printf("No left over backup version key\n");
}
@ -408,11 +408,11 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
// Error if the log/mutation keyspace for the backup tag is not empty
if (logValues.size() > 0) {
displaySystemKeys++;
printf("BackupCorrectnessLeftOverLogKeys: (%d) %s\n", logValues.size(), printable(backupLogValuesKey).c_str());
TraceEvent(SevError, "BackupCorrectnessLeftOverLogKeys", randomID).detail("backupTag", printable(tag))
.detail("LeftOverKeys", logValues.size()).detail("keySpace", printable(backupLogValuesKey)).detail("version", decodeBKMutationLogKey(logValues[0].key).first);
printf("BackupCorrectnessLeftoverLogKeys: (%d) %s\n", logValues.size(), printable(backupLogValuesKey).c_str());
TraceEvent(SevError, "BackupCorrectnessLeftoverLogKeys", randomID).detail("BackupTag", printable(tag))
.detail("LeftoverKeys", logValues.size()).detail("KeySpace", printable(backupLogValuesKey)).detail("Version", decodeBKMutationLogKey(logValues[0].key).first);
for (auto & s : logValues) {
TraceEvent("BARW_LeftOverKey", randomID).detail("key", printable(StringRef(s.key.toString()))).detail("value", printable(StringRef(s.value.toString())));
TraceEvent("BARW_LeftoverKey", randomID).detail("Key", printable(StringRef(s.key.toString()))).detail("Value", printable(StringRef(s.value.toString())));
printf(" Key: %-50s Value: %s\n", printable(StringRef(s.key.toString())).c_str(), printable(StringRef(s.value.toString())).c_str());
}
}
@ -424,7 +424,7 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
break;
}
catch (Error &e) {
TraceEvent("BARW_check Exception", randomID).error(e);
TraceEvent("BARW_CheckException", randomID).error(e);
Void _ = wait(tr->onError(e));
}
}
@ -440,8 +440,8 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
state DatabaseBackupAgent restoreAgent(self->extraDB);
state Future<Void> extraBackup;
state bool extraTasks = false;
TraceEvent("BARW_Arguments").detail("backupTag", printable(self->backupTag)).detail("backupAfter", self->backupAfter)
.detail("abortAndRestartAfter", self->abortAndRestartAfter).detail("differentialAfter", self->stopDifferentialAfter);
TraceEvent("BARW_Arguments").detail("BackupTag", printable(self->backupTag)).detail("BackupAfter", self->backupAfter)
.detail("AbortAndRestartAfter", self->abortAndRestartAfter).detail("DifferentialAfter", self->stopDifferentialAfter);
state UID randomID = g_nondeterministic_random->randomUniqueID();
@ -456,31 +456,31 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
// backup
Void _ = wait(delay(self->backupAfter));
TraceEvent("BARW_doBackup1", randomID).detail("tag", printable(self->backupTag));
TraceEvent("BARW_DoBackup1", randomID).detail("Tag", printable(self->backupTag));
state Promise<Void> submitted;
state Future<Void> b = doBackup(self, 0, &backupAgent, self->extraDB, self->backupTag, self->backupRanges, self->stopDifferentialAfter, submitted);
if (self->abortAndRestartAfter) {
TraceEvent("BARW_doBackup2", randomID).detail("tag", printable(self->backupTag)).detail("abortWait", self->abortAndRestartAfter);
TraceEvent("BARW_DoBackup2", randomID).detail("Tag", printable(self->backupTag)).detail("AbortWait", self->abortAndRestartAfter);
Void _ = wait(submitted.getFuture());
b = b && doBackup(self, self->abortAndRestartAfter, &backupAgent, self->extraDB, self->backupTag, self->backupRanges, self->stopDifferentialAfter, Promise<Void>());
}
TraceEvent("BARW_doBackupWait", randomID).detail("backupTag", printable(self->backupTag)).detail("abortAndRestartAfter", self->abortAndRestartAfter);
TraceEvent("BARW_DoBackupWait", randomID).detail("BackupTag", printable(self->backupTag)).detail("AbortAndRestartAfter", self->abortAndRestartAfter);
Void _ = wait(b);
TraceEvent("BARW_doBackupDone", randomID).detail("backupTag", printable(self->backupTag)).detail("abortAndRestartAfter", self->abortAndRestartAfter);
TraceEvent("BARW_DoBackupDone", randomID).detail("BackupTag", printable(self->backupTag)).detail("AbortAndRestartAfter", self->abortAndRestartAfter);
state UID logUid = wait(backupAgent.getLogUid(self->extraDB, self->backupTag));
// Occasionally start yet another backup that might still be running when we restore
if (!self->locked && BUGGIFY) {
TraceEvent("BARW_submitBackup2", randomID).detail("tag", printable(self->backupTag));
TraceEvent("BARW_SubmitBackup2", randomID).detail("Tag", printable(self->backupTag));
try {
extraBackup = backupAgent.submitBackup(self->extraDB, self->backupTag, self->backupRanges, true, self->extraPrefix, StringRef(), self->locked);
}
catch (Error& e) {
TraceEvent("BARW_submitBackup2 Exception", randomID).detail("backupTag", printable(self->backupTag)).error(e);
TraceEvent("BARW_SubmitBackup2Exception", randomID).detail("BackupTag", printable(self->backupTag)).error(e);
if (e.code() != error_code_backup_unneeded && e.code() != error_code_backup_duplicate)
throw;
}
@ -491,7 +491,7 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
if (self->performRestore) {
// restore database
TraceEvent("BARW_restore", randomID).detail("restoreAfter", self->restoreAfter).detail("backupTag", printable(self->restoreTag));
TraceEvent("BARW_Restore", randomID).detail("RestoreAfter", self->restoreAfter).detail("BackupTag", printable(self->restoreTag));
//Void _ = wait(diffRanges(self->backupRanges, self->backupPrefix, cx, self->extraDB));
state Transaction tr3(cx);
@ -520,7 +520,7 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
Void _ = wait(restoreAgent.submitBackup(cx, self->restoreTag, restoreRange, true, StringRef(), self->backupPrefix, self->locked));
}
catch (Error& e) {
TraceEvent("BARW_doBackup submitBackup Exception", randomID).detail("tag", printable(self->restoreTag)).error(e);
TraceEvent("BARW_DoBackupSubmitBackupException", randomID).detail("Tag", printable(self->restoreTag)).error(e);
if (e.code() != error_code_backup_unneeded && e.code() != error_code_backup_duplicate)
throw;
}
@ -530,23 +530,23 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
}
if (extraBackup.isValid()) {
TraceEvent("BARW_wait extraBackup", randomID).detail("backupTag", printable(self->backupTag));
TraceEvent("BARW_WaitExtraBackup", randomID).detail("BackupTag", printable(self->backupTag));
extraTasks = true;
try {
Void _ = wait(extraBackup);
}
catch (Error& e) {
TraceEvent("BARW_extraBackup Exception", randomID).detail("backupTag", printable(self->backupTag)).error(e);
TraceEvent("BARW_ExtraBackupException", randomID).detail("BackupTag", printable(self->backupTag)).error(e);
if (e.code() != error_code_backup_unneeded && e.code() != error_code_backup_duplicate)
throw;
}
TraceEvent("BARW_abortBackup extra", randomID).detail("backupTag", printable(self->backupTag));
TraceEvent("BARW_AbortBackupExtra", randomID).detail("BackupTag", printable(self->backupTag));
try {
Void _ = wait(backupAgent.abortBackup(self->extraDB, self->backupTag));
}
catch (Error& e) {
TraceEvent("BARW_abortBackup extra Exception", randomID).error(e);
TraceEvent("BARW_AbortBackupExtraException", randomID).error(e);
if (e.code() != error_code_backup_unneeded)
throw;
}
@ -559,7 +559,7 @@ struct BackupToDBCorrectnessWorkload : TestWorkload {
Void _ = wait( checkData(cx, restoreUid, restoreUid, randomID, self->restoreTag, &restoreAgent, self->shareLogRange) );
}
TraceEvent("BARW_complete", randomID).detail("backupTag", printable(self->backupTag));
TraceEvent("BARW_Complete", randomID).detail("BackupTag", printable(self->backupTag));
// Decrement the backup agent requets
if (self->agentRequest) {

78
fdbserver/workloads/BackupToDBUpgrade.actor.cpp
View File

@ -64,8 +64,8 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
backupRanges.push_back_deep(backupRanges.arena(), (beginRange < endRange) ? KeyRangeRef(beginRange, endRange) : KeyRangeRef(endRange, beginRange));
// Track the added range
TraceEvent("DRU_backup_range").detail("rangeBegin", (beginRange < endRange) ? printable(beginRange) : printable(endRange))
.detail("rangeEnd", (beginRange < endRange) ? printable(endRange) : printable(beginRange));
TraceEvent("DRU_BackupRange").detail("RangeBegin", (beginRange < endRange) ? printable(beginRange) : printable(endRange))
.detail("RangeEnd", (beginRange < endRange) ? printable(endRange) : printable(beginRange));
}
}
@ -73,7 +73,7 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
Reference<Cluster> extraCluster = Cluster::createCluster(extraFile, -1);
extraDB = extraCluster->createDatabase(LiteralStringRef("DB")).get();
TraceEvent("DRU_start");
TraceEvent("DRU_Start");
}
virtual std::string description() {
@ -120,9 +120,9 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
}
}
TraceEvent("DRU_doBackupInDifferentialMode").detail("tag", printable(tag));
TraceEvent("DRU_DoBackupInDifferentialMode").detail("Tag", printable(tag));
} catch (Error &e) {
TraceEvent("DRU_doBackupSubmitBackupError").detail("tag", printable(tag)).error(e);
TraceEvent("DRU_DoBackupSubmitBackupError").detail("Tag", printable(tag)).error(e);
if (e.code() != error_code_backup_unneeded && e.code() != error_code_backup_duplicate) {
throw e;
}
@ -146,7 +146,7 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
loop {
state Reference<ReadYourWritesTransaction> tr(new ReadYourWritesTransaction(cx));
TraceEvent("DRU_checkLeftoverkeys").detail("backupTag", printable(tag));
TraceEvent("DRU_CheckLeftoverkeys").detail("BackupTag", printable(tag));
try {
tr->setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
@ -154,12 +154,12 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
// Check the left over tasks
// We have to wait for the list to empty since an abort and get status
// can leave extra tasks in the queue
TraceEvent("DRU_checkLeftovertasks").detail("backupTag", printable(tag));
TraceEvent("DRU_CheckLeftovertasks").detail("BackupTag", printable(tag));
state int64_t taskCount = wait( backupAgent->getTaskCount(tr) );
state int waitCycles = 0;
if ((taskCount) && (0)) {
TraceEvent("DRU_EndingNonzeroTaskCount").detail("backupTag", printable(tag)).detail("taskCount", taskCount).detail("waitCycles", waitCycles);
TraceEvent("DRU_EndingNonzeroTaskCount").detail("BackupTag", printable(tag)).detail("TaskCount", taskCount).detail("WaitCycles", waitCycles);
printf("EndingNonZeroTasks: %ld\n", (long) taskCount);
Void _ = wait(TaskBucket::debugPrintRange(cx, LiteralStringRef("\xff"), StringRef()));
}
@ -167,7 +167,7 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
loop {
waitCycles ++;
TraceEvent("DRU_NonzeroTaskWait").detail("backupTag", printable(tag)).detail("taskCount", taskCount).detail("waitCycles", waitCycles);
TraceEvent("DRU_NonzeroTaskWait").detail("BackupTag", printable(tag)).detail("TaskCount", taskCount).detail("WaitCycles", waitCycles);
printf("%.6f Wait #%4d for %lld tasks to end\n", now(), waitCycles, (long long) taskCount);
Void _ = wait(delay(20.0));
@ -183,8 +183,8 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
if (taskCount) {
displaySystemKeys ++;
TraceEvent(SevError, "DRU_NonzeroTaskCount").detail("backupTag", printable(tag)).detail("taskCount", taskCount).detail("waitCycles", waitCycles);
printf("BackupCorrectnessLeftOverLogTasks: %ld\n", (long) taskCount);
TraceEvent(SevError, "DRU_NonzeroTaskCount").detail("BackupTag", printable(tag)).detail("TaskCount", taskCount).detail("WaitCycles", waitCycles);
printf("BackupCorrectnessLeftoverLogTasks: %ld\n", (long) taskCount);
}
Standalone<RangeResultRef> agentValues = wait(tr->getRange(KeyRange(KeyRangeRef(backupAgentKey, strinc(backupAgentKey))), 100));
@ -192,11 +192,11 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
// Error if the system keyspace for the backup tag is not empty
if (agentValues.size() > 0) {
displaySystemKeys++;
printf("BackupCorrectnessLeftOverMutationKeys: (%d) %s\n", agentValues.size(), printable(backupAgentKey).c_str());
TraceEvent(SevError, "BackupCorrectnessLeftOverMutationKeys").detail("backupTag", printable(tag))
.detail("LeftOverKeys", agentValues.size()).detail("keySpace", printable(backupAgentKey));
printf("BackupCorrectnessLeftoverMutationKeys: (%d) %s\n", agentValues.size(), printable(backupAgentKey).c_str());
TraceEvent(SevError, "BackupCorrectnessLeftoverMutationKeys").detail("BackupTag", printable(tag))
.detail("LeftoverKeys", agentValues.size()).detail("KeySpace", printable(backupAgentKey));
for (auto & s : agentValues) {
TraceEvent("DRU_LeftOverKey").detail("key", printable(StringRef(s.key.toString()))).detail("value", printable(StringRef(s.value.toString())));
TraceEvent("DRU_LeftoverKey").detail("Key", printable(StringRef(s.key.toString()))).detail("Value", printable(StringRef(s.value.toString())));
printf(" Key: %-50s Value: %s\n", printable(StringRef(s.key.toString())).c_str(), printable(StringRef(s.value.toString())).c_str());
}
}
@ -206,7 +206,7 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
Optional<Value> latestVersion = wait(tr->get(backupLatestVersionsKey));
if (latestVersion.present()) {
TraceEvent(SevError, "BackupCorrectnessLeftOverVersionKey").detail("backupTag", printable(tag)).detail("key", backupLatestVersionsKey.printable()).detail("value", BinaryReader::fromStringRef<Version>(latestVersion.get(), Unversioned()));
TraceEvent(SevError, "BackupCorrectnessLeftoverVersionKey").detail("BackupTag", printable(tag)).detail("Key", backupLatestVersionsKey.printable()).detail("Value", BinaryReader::fromStringRef<Version>(latestVersion.get(), Unversioned()));
} else {
printf("No left over backup version key\n");
}
@ -218,11 +218,11 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
// Error if the log/mutation keyspace for the backup tag is not empty
if (logValues.size() > 0) {
displaySystemKeys++;
printf("BackupCorrectnessLeftOverLogKeys: (%d) %s\n", logValues.size(), printable(backupLogValuesKey).c_str());
TraceEvent(SevError, "BackupCorrectnessLeftOverLogKeys").detail("backupTag", printable(tag))
.detail("LeftOverKeys", logValues.size()).detail("keySpace", printable(backupLogValuesKey)).detail("version", decodeBKMutationLogKey(logValues[0].key).first);
printf("BackupCorrectnessLeftoverLogKeys: (%d) %s\n", logValues.size(), printable(backupLogValuesKey).c_str());
TraceEvent(SevError, "BackupCorrectnessLeftoverLogKeys").detail("BackupTag", printable(tag))
.detail("LeftoverKeys", logValues.size()).detail("KeySpace", printable(backupLogValuesKey)).detail("Version", decodeBKMutationLogKey(logValues[0].key).first);
for (auto & s : logValues) {
TraceEvent("DRU_LeftOverKey").detail("key", printable(StringRef(s.key.toString()))).detail("value", printable(StringRef(s.value.toString())));
TraceEvent("DRU_LeftoverKey").detail("Key", printable(StringRef(s.key.toString()))).detail("Value", printable(StringRef(s.value.toString())));
printf(" Key: %-50s Value: %s\n", printable(StringRef(s.key.toString())).c_str(), printable(StringRef(s.value.toString())).c_str());
}
}
@ -234,7 +234,7 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
break;
}
catch (Error &e) {
TraceEvent("DRU_checkError").error(e);
TraceEvent("DRU_CheckError").error(e);
Void _ = wait(tr->onError(e));
}
}
@ -252,12 +252,12 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
try{
Void _ = wait(delay(self->backupAfter));
TraceEvent("DRU_doBackup").detail("tag", printable(self->backupTag));
TraceEvent("DRU_DoBackup").detail("Tag", printable(self->backupTag));
state Future<Void> b = doBackup(self, &backupAgent, self->extraDB, self->backupTag, self->backupRanges);
TraceEvent("DRU_doBackupWait").detail("backupTag", printable(self->backupTag));
TraceEvent("DRU_DoBackupWait").detail("BackupTag", printable(self->backupTag));
Void _ = wait(b);
TraceEvent("DRU_doBackupWaitEnd").detail("backupTag", printable(self->backupTag));
TraceEvent("DRU_DoBackupWaitEnd").detail("BackupTag", printable(self->backupTag));
}
catch (Error& e) {
TraceEvent(SevError, "BackupToDBUpgradeSetuEerror").error(e);
@ -292,13 +292,13 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
while (src != srcFuture.get().end() && bkp != bkpFuture.get().end()) {
KeyRef bkpKey = bkp->key.substr(backupPrefix.size());
if (src->key != bkpKey && src->value != bkp->value) {
TraceEvent(SevError, "MismatchKeyAndValue").detail("srcKey", printable(src->key)).detail("srcVal", printable(src->value)).detail("bkpKey", printable(bkpKey)).detail("bkpVal", printable(bkp->value));
TraceEvent(SevError, "MismatchKeyAndValue").detail("SrcKey", printable(src->key)).detail("SrcVal", printable(src->value)).detail("BkpKey", printable(bkpKey)).detail("BkpVal", printable(bkp->value));
}
else if (src->key != bkpKey) {
TraceEvent(SevError, "MismatchKey").detail("srcKey", printable(src->key)).detail("srcVal", printable(src->value)).detail("bkpKey", printable(bkpKey)).detail("bkpVal", printable(bkp->value));
TraceEvent(SevError, "MismatchKey").detail("SrcKey", printable(src->key)).detail("SrcVal", printable(src->value)).detail("BkpKey", printable(bkpKey)).detail("BkpVal", printable(bkp->value));
}
else if (src->value != bkp->value) {
TraceEvent(SevError, "MismatchValue").detail("srcKey", printable(src->key)).detail("srcVal", printable(src->value)).detail("bkpKey", printable(bkpKey)).detail("bkpVal", printable(bkp->value));
TraceEvent(SevError, "MismatchValue").detail("SrcKey", printable(src->key)).detail("SrcVal", printable(src->value)).detail("BkpKey", printable(bkpKey)).detail("BkpVal", printable(bkp->value));
}
begin = std::min(src->key, bkpKey);
if (src->key == bkpKey) {
@ -313,12 +313,12 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
}
}
while (src != srcFuture.get().end() && !bkpFuture.get().more) {
TraceEvent(SevError, "MissingBkpKey").detail("srcKey", printable(src->key)).detail("srcVal", printable(src->value));
TraceEvent(SevError, "MissingBkpKey").detail("SrcKey", printable(src->key)).detail("SrcVal", printable(src->value));
begin = src->key;
++src;
}
while (bkp != bkpFuture.get().end() && !srcFuture.get().more) {
TraceEvent(SevError, "MissingSrcKey").detail("bkpKey", printable(bkp->key.substr(backupPrefix.size()))).detail("bkpVal", printable(bkp->value));
TraceEvent(SevError, "MissingSrcKey").detail("BkpKey", printable(bkp->key.substr(backupPrefix.size()))).detail("BkpVal", printable(bkp->value));
begin = bkp->key;
++bkp;
}
@ -351,7 +351,7 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
state Future<Void> stopDifferential = delay(self->stopDifferentialAfter);
state Future<Void> waitUpgrade = backupAgent.waitUpgradeToLatestDrVersion(self->extraDB, self->backupTag);
Void _ = wait(success(stopDifferential) && success(waitUpgrade));
TraceEvent("DRU_waitDifferentialEnd").detail("tag", printable(self->backupTag));
TraceEvent("DRU_WaitDifferentialEnd").detail("Tag", printable(self->backupTag));
try {
// Get restore ranges before aborting
@ -379,7 +379,7 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
}
}
TraceEvent("DRU_locked").detail("lockedVersion", commitVersion);
TraceEvent("DRU_Locked").detail("LockedVersion", commitVersion);
// Wait for the destination to apply mutations up to the lock commit before switching over.
state ReadYourWritesTransaction versionCheckTr(self->extraDB);
@ -388,7 +388,7 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
versionCheckTr.setOption(FDBTransactionOptions::ACCESS_SYSTEM_KEYS);
versionCheckTr.setOption(FDBTransactionOptions::LOCK_AWARE);
Optional<Value> v = wait(versionCheckTr.get(BinaryWriter::toValue(logUid, Unversioned()).withPrefix(applyMutationsBeginRange.begin)));
TraceEvent("DRU_applied").detail("appliedVersion", v.present() ? BinaryReader::fromStringRef<Version>(v.get(), Unversioned()) : -1);
TraceEvent("DRU_Applied").detail("AppliedVersion", v.present() ? BinaryReader::fromStringRef<Version>(v.get(), Unversioned()) : -1);
if( v.present() && BinaryReader::fromStringRef<Version>(v.get(), Unversioned()) >= commitVersion)
break;
@ -401,16 +401,16 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
}
}
TraceEvent("DRU_diffRanges");
TraceEvent("DRU_DiffRanges");
Void _ = wait( diffRanges(prevBackupRanges, self->backupPrefix, cx, self->extraDB ) );
// abort backup
TraceEvent("DRU_abortBackup").detail("tag", printable(self->backupTag));
TraceEvent("DRU_AbortBackup").detail("Tag", printable(self->backupTag));
Void _ = wait(backupAgent.abortBackup(self->extraDB, self->backupTag));
Void _ = wait( unlockDatabase(self->extraDB, logUid) );
// restore database
TraceEvent("DRU_prepareRestore").detail("restoreTag", printable(self->restoreTag));
TraceEvent("DRU_PrepareRestore").detail("RestoreTag", printable(self->restoreTag));
state Reference<ReadYourWritesTransaction> tr2(new ReadYourWritesTransaction(cx));
loop{
try{
@ -426,7 +426,7 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
Void _ = wait( tr2->commit() );
break;
} catch( Error &e ) {
TraceEvent("DRU_restoreSetupError").error(e, true);
TraceEvent("DRU_RestoreSetupError").error(e, true);
Void _ = wait( tr2->onError(e) );
}
}
@ -438,11 +438,11 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
// start restoring db
try {
TraceEvent("DRU_restoreDb").detail("restoreTag", printable(self->restoreTag));
TraceEvent("DRU_RestoreDb").detail("RestoreTag", printable(self->restoreTag));
Void _ = wait(restoreAgent.submitBackup(cx, self->restoreTag, restoreRanges, true, StringRef(), self->backupPrefix));
}
catch (Error& e) {
TraceEvent("DRU_restoreSubmitBackupError").detail("tag", printable(self->restoreTag)).error(e);
TraceEvent("DRU_RestoreSubmitBackupError").detail("Tag", printable(self->restoreTag)).error(e);
if (e.code() != error_code_backup_unneeded && e.code() != error_code_backup_duplicate)
throw;
}
@ -454,7 +454,7 @@ struct BackupToDBUpgradeWorkload : TestWorkload {
state UID restoreUid = wait(restoreAgent.getLogUid(cx, self->restoreTag));
Void _ = wait(checkData(cx, restoreUid, restoreUid, self->restoreTag, &restoreAgent, self->shareLogRange));
TraceEvent("DRU_complete").detail("backupTag", printable(self->backupTag));
TraceEvent("DRU_Complete").detail("BackupTag", printable(self->backupTag));
if (g_simulator.drAgents == ISimulator::BackupToDB) {
g_simulator.drAgents = ISimulator::NoBackupAgents;

2
fdbserver/workloads/BulkSetup.actor.h
View File

@ -313,7 +313,7 @@ Future<Void> bulkSetup( Database cx, T* workload, uint64_t nodeCount, Promise<do
Void _ = wait( delay( 5.0 ) ); // Wait for the data distribution in a small test to start
loop {
int64_t inFlight = wait( getDataInFlight( cx, workload->dbInfo ) );
TraceEvent("DynamicWarming").detail("inFlight", inFlight);
TraceEvent("DynamicWarming").detail("InFlight", inFlight);
if( inFlight > 1e6 ) { // Wait for just 1 MB to be in flight
Void _ = wait( delay( 1.0 ) );
} else {

12
fdbserver/workloads/ClientTransactionProfileCorrectness.actor.cpp
View File

@ -105,7 +105,7 @@ struct ClientTransactionProfileCorrectnessWorkload : TestWorkload {
if (clientId == 0) {
samplingProbability = getOption(options, LiteralStringRef("samplingProbability"), g_random->random01() / 10); //rand range 0 - 0.1
trInfoSizeLimit = getOption(options, LiteralStringRef("trInfoSizeLimit"), g_random->randomInt(100 * 1024, 10 * 1024 * 1024)); // 100 KB - 10 MB
TraceEvent(SevInfo, "ClientTransactionProfilingSetup").detail("samplingProbability", samplingProbability).detail("trInfoSizeLimit", trInfoSizeLimit);
TraceEvent(SevInfo, "ClientTransactionProfilingSetup").detail("SamplingProbability", samplingProbability).detail("TrInfoSizeLimit", trInfoSizeLimit);
}
}
@ -160,13 +160,13 @@ struct ClientTransactionProfileCorrectnessWorkload : TestWorkload {
if (trInfoChunks.find(trId) == trInfoChunks.end()) {
// Some of the earlier chunks for this trId should have been deleted.
// Discard this chunk as it is of not much use
TraceEvent(SevInfo, "ClientTransactionProfilingSomeChunksMissing").detail("trId", trId);
TraceEvent(SevInfo, "ClientTransactionProfilingSomeChunksMissing").detail("TrId", trId);
}
else {
// Check if it is the expected chunk. Otherwise discard the whole transaction entry.
// There are scenarios (eg., when deletion is happening) where some chunks get missed.
if (chunkNum != trInfoChunks.find(trId)->second.size() + 1) {
TraceEvent(SevInfo, "ClientTransactionProfilingChunksMissing").detail("trId", trId);
TraceEvent(SevInfo, "ClientTransactionProfilingChunksMissing").detail("TrId", trId);
trInfoChunks.erase(trId);
}
else {
@ -248,11 +248,11 @@ struct ClientTransactionProfileCorrectnessWorkload : TestWorkload {
}
// FIXME: Find a way to check that contentsSize is not greater than a certain limit.
//if (counter != contentsSize) {
// TraceEvent(SevError, "ClientTransactionProfilingIncorrectCtrVal").detail("counter", counter).detail("contentsSize", contentsSize);
// TraceEvent(SevError, "ClientTransactionProfilingIncorrectCtrVal").detail("Counter", counter).detail("ContentsSize", contentsSize);
// return false;
//}
TraceEvent(SevInfo, "ClientTransactionProfilingCtrval").detail("counter", counter);
TraceEvent(SevInfo, "ClientTransactionProfilingContentsSize").detail("contentsSize", contentsSize);
TraceEvent(SevInfo, "ClientTransactionProfilingCtrval").detail("Counter", counter);
TraceEvent(SevInfo, "ClientTransactionProfilingContentsSize").detail("ContentsSize", contentsSize);
// Check if the data format is as expected
return self->checkTxInfoEntriesFormat(txInfoEntries);

8
fdbserver/workloads/ConfigureDatabase.actor.cpp
View File

@ -144,7 +144,7 @@ struct ConfigureDatabaseWorkload : TestWorkload {
//TraceEvent("ConfigureTestLoadData").detail("LoadTime", now() - startTime).detail("AmountLoaded",amtLoaded);
}
else if( randomChoice == 3 ) {
//TraceEvent("ConfigureTestConfigureBegin").detail("newConfig", newConfig);
//TraceEvent("ConfigureTestConfigureBegin").detail("NewConfig", newConfig);
int redundancy = g_random->randomInt( 0, sizeof(redundancies)/sizeof(redundancies[0]));
std::string config = redundancies[redundancy];
if(config == "triple" && g_simulator.physicalDatacenters == 3) {
@ -156,15 +156,15 @@ struct ConfigureDatabaseWorkload : TestWorkload {
if (g_random->random01() < 0.5) config += " resolvers=" + format("%d", randomRoleNumber());
ConfigurationResult::Type _ = wait( changeConfig( cx, config ) );
//TraceEvent("ConfigureTestConfigureEnd").detail("newConfig", newConfig);
//TraceEvent("ConfigureTestConfigureEnd").detail("NewConfig", newConfig);
}
else if( randomChoice == 4 ) {
//TraceEvent("ConfigureTestQuorumBegin").detail("newQuorum", s);
//TraceEvent("ConfigureTestQuorumBegin").detail("NewQuorum", s);
auto ch = autoQuorumChange();
if (g_random->randomInt(0,2))
ch = nameQuorumChange( format("NewName%d", g_random->randomInt(0,100)), ch );
CoordinatorsResult::Type _ = wait( changeQuorum( cx, ch ) );
//TraceEvent("ConfigureTestConfigureEnd").detail("newQuorum", s);
//TraceEvent("ConfigureTestConfigureEnd").detail("NewQuorum", s);
}
else if ( randomChoice == 5) {
ConfigurationResult::Type _ = wait( changeConfig( cx, storeTypes[g_random->randomInt( 0, sizeof(storeTypes)/sizeof(storeTypes[0]))] ) );

12
fdbserver/workloads/ConflictRange.actor.cpp
View File

@ -114,7 +114,7 @@ struct ConflictRangeWorkload : TestWorkload {
int clearedB = g_random->randomInt(0, self->maxKeySpace-1);
clearedBegin = std::min(clearedA, clearedB);
clearedEnd = std::max(clearedA, clearedB)+1;
TraceEvent("ConflictRangeClear").detail("begin",clearedBegin).detail("end",clearedEnd);
TraceEvent("ConflictRangeClear").detail("Begin",clearedBegin).detail("End",clearedEnd);
}
tr0.clear( KeyRangeRef( StringRef( format( "%010d", 0 ) ), StringRef( format( "%010d", self->maxKeySpace ) ) ) );
@ -275,8 +275,8 @@ struct ConflictRangeWorkload : TestWorkload {
}
TraceEvent(SevError, "ConflictRangeError").detail("Info", "Conflict returned, however results are the same")
.detail("randomSets",randomSets).detail("myKeyA",myKeyA).detail("myKeyB",myKeyB).detail("onEqualA",onEqualA).detail("onEqualB",onEqualB)
.detail("offsetA",offsetA).detail("offsetB",offsetB).detail("randomLimit",randomLimit).detail("size",originalResults.size()).detail("results", keyStr1).detail("original", keyStr2);
.detail("RandomSets",randomSets).detail("MyKeyA",myKeyA).detail("MyKeyB",myKeyB).detail("OnEqualA",onEqualA).detail("OnEqualB",onEqualB)
.detail("OffsetA",offsetA).detail("OffsetB",offsetB).detail("RandomLimit",randomLimit).detail("Size",originalResults.size()).detail("Results", keyStr1).detail("Original", keyStr2);
tr4 = Transaction(cx);
Standalone<RangeResultRef> res = wait( tr4.getRange( KeyRangeRef( StringRef( format( "%010d", 0 ) ), StringRef( format( "%010d", self->maxKeySpace ) ) ), 200 ) );
@ -285,7 +285,7 @@ struct ConflictRangeWorkload : TestWorkload {
allKeyEntries += printable( res[i].key ) + " ";
}
TraceEvent("ConflictRangeDump").detail("keys", allKeyEntries);
TraceEvent("ConflictRangeDump").detail("Keys", allKeyEntries);
}
throw not_committed();
} else {
@ -314,8 +314,8 @@ struct ConflictRangeWorkload : TestWorkload {
TraceEvent(SevError, "ConflictRangeError").detail("Info", "No conflict returned, however result sizes do not match")
.detail("OriginalSize", originalResults.size()).detail("NewSize", res.size())
.detail("randomSets",randomSets).detail("myKeyA",myKeyA).detail("myKeyB",myKeyB).detail("onEqualA",onEqualA).detail("onEqualB",onEqualB)
.detail("offsetA",offsetA).detail("offsetB",offsetB).detail("randomLimit",randomLimit).detail("size",originalResults.size()).detail("results", keyStr1).detail("original", keyStr2);
.detail("RandomSets",randomSets).detail("MyKeyA",myKeyA).detail("MyKeyB",myKeyB).detail("OnEqualA",onEqualA).detail("OnEqualB",onEqualB)
.detail("OffsetA",offsetA).detail("OffsetB",offsetB).detail("RandomLimit",randomLimit).detail("Size",originalResults.size()).detail("Results", keyStr1).detail("Original", keyStr2);
}
}
} catch (Error& e) {

18
fdbserver/workloads/ConsistencyCheck.actor.cpp
View File

@ -280,7 +280,7 @@ struct ConsistencyCheckWorkload : TestWorkload
}
}
TraceEvent("ConsistencyCheck_FinishedCheck").detail("repetitions", self->repetitions);
TraceEvent("ConsistencyCheck_FinishedCheck").detail("Repetitions", self->repetitions);
return Void();
}
@ -628,7 +628,7 @@ struct ConsistencyCheckWorkload : TestWorkload
//In a quiescent database, check that the team size is the same as the desired team size
if(self->firstClient && self->performQuiescentChecks && sourceStorageServers.size() != configuration.storageTeamSize)
{
TraceEvent("ConsistencyCheck_InvalidTeamSize").detail("ShardBegin", printable(range.begin)).detail("ShardEnd", printable(range.end)).detail("teamSize", sourceStorageServers.size()).detail("desiredTeamSize", configuration.storageTeamSize);
TraceEvent("ConsistencyCheck_InvalidTeamSize").detail("ShardBegin", printable(range.begin)).detail("ShardEnd", printable(range.end)).detail("TeamSize", sourceStorageServers.size()).detail("DesiredTeamSize", configuration.storageTeamSize);
self->testFailure("Invalid team size");
return false;
}
@ -983,7 +983,7 @@ struct ConsistencyCheckWorkload : TestWorkload
}
if(bytesReadInRange > 0) {
TraceEvent("ConsistencyCheck_ReadRange").detail("range", printable(range)).detail("bytesRead", bytesReadInRange);
TraceEvent("ConsistencyCheck_ReadRange").detail("Range", printable(range)).detail("BytesRead", bytesReadInRange);
}
}
@ -1167,7 +1167,7 @@ struct ConsistencyCheckWorkload : TestWorkload
for( auto it : workers ) {
ISimulator::ProcessInfo* info = g_simulator.getProcessByAddress(it.first.address());
if(!info || info->failed) {
TraceEvent("ConsistencyCheck_FailedWorkerInList").detail("addr", it.first.address());
TraceEvent("ConsistencyCheck_FailedWorkerInList").detail("Addr", it.first.address());
return false;
}
workerAddresses.insert( NetworkAddress(it.first.address().ip, it.first.address().port, true, false) );
@ -1177,7 +1177,7 @@ struct ConsistencyCheckWorkload : TestWorkload
for(int i = 0; i < all.size(); i++) {
if( all[i]->isReliable() && all[i]->name == std::string("Server") && all[i]->startingClass != ProcessClass::TesterClass ) {
if(!workerAddresses.count(all[i]->address)) {
TraceEvent("ConsistencyCheck_WorkerMissingFromList").detail("addr", all[i]->address);
TraceEvent("ConsistencyCheck_WorkerMissingFromList").detail("Addr", all[i]->address);
return false;
}
}
@ -1220,7 +1220,7 @@ struct ConsistencyCheckWorkload : TestWorkload
// Check cluster controller
ProcessClass::Fitness bestClusterControllerFitness = getBestAvailableFitness(nonExcludedClassTypes, ProcessClass::ClusterController);
if (!nonExcludedWorkerProcessMap.count(db.clusterInterface.clientInterface.address()) || nonExcludedWorkerProcessMap[db.clusterInterface.clientInterface.address()].machineClassFitness(ProcessClass::ClusterController) != bestClusterControllerFitness) {
TraceEvent("ConsistencyCheck_ClusterControllerNotBest").detail("bestClusterControllerFitness", bestClusterControllerFitness).detail("existingClusterControllerFit", nonExcludedWorkerProcessMap.count(db.clusterInterface.clientInterface.address()) ? nonExcludedWorkerProcessMap[db.clusterInterface.clientInterface.address()].machineClassFitness(ProcessClass::ClusterController) : -1);
TraceEvent("ConsistencyCheck_ClusterControllerNotBest").detail("BestClusterControllerFitness", bestClusterControllerFitness).detail("ExistingClusterControllerFit", nonExcludedWorkerProcessMap.count(db.clusterInterface.clientInterface.address()) ? nonExcludedWorkerProcessMap[db.clusterInterface.clientInterface.address()].machineClassFitness(ProcessClass::ClusterController) : -1);
return false;
}
@ -1234,7 +1234,7 @@ struct ConsistencyCheckWorkload : TestWorkload
}
if (!allWorkerProcessMap.count(db.master.address()) || (!nonExcludedWorkerProcessMap.count(db.master.address()) && bestMasterFitness != ProcessClass::ExcludeFit) || nonExcludedWorkerProcessMap[db.master.address()].machineClassFitness(ProcessClass::Master) != bestMasterFitness) {
TraceEvent("ConsistencyCheck_MasterNotBest").detail("bestMasterFitness", bestMasterFitness).detail("existingMasterFit", nonExcludedWorkerProcessMap.count(db.master.address()) ? nonExcludedWorkerProcessMap[db.master.address()].machineClassFitness(ProcessClass::Master) : -1);
TraceEvent("ConsistencyCheck_MasterNotBest").detail("BestMasterFitness", bestMasterFitness).detail("ExistingMasterFit", nonExcludedWorkerProcessMap.count(db.master.address()) ? nonExcludedWorkerProcessMap[db.master.address()].machineClassFitness(ProcessClass::Master) : -1);
return false;
}
@ -1242,7 +1242,7 @@ struct ConsistencyCheckWorkload : TestWorkload
ProcessClass::Fitness bestMasterProxyFitness = getBestAvailableFitness(nonExcludedClassTypes, ProcessClass::Proxy);
for (auto masterProxy : db.client.proxies) {
if (!nonExcludedWorkerProcessMap.count(masterProxy.address()) || nonExcludedWorkerProcessMap[masterProxy.address()].machineClassFitness(ProcessClass::Proxy) != bestMasterProxyFitness) {
TraceEvent("ConsistencyCheck_ProxyNotBest").detail("bestMasterProxyFitness", bestMasterProxyFitness).detail("existingMasterProxyFitness", nonExcludedWorkerProcessMap.count(masterProxy.address()) ? nonExcludedWorkerProcessMap[masterProxy.address()].machineClassFitness(ProcessClass::Proxy) : -1);
TraceEvent("ConsistencyCheck_ProxyNotBest").detail("BestMasterProxyFitness", bestMasterProxyFitness).detail("ExistingMasterProxyFitness", nonExcludedWorkerProcessMap.count(masterProxy.address()) ? nonExcludedWorkerProcessMap[masterProxy.address()].machineClassFitness(ProcessClass::Proxy) : -1);
return false;
}
}
@ -1251,7 +1251,7 @@ struct ConsistencyCheckWorkload : TestWorkload
ProcessClass::Fitness bestResolverFitness = getBestAvailableFitness(nonExcludedClassTypes, ProcessClass::Resolver);
for (auto resolver : db.resolvers) {
if (!nonExcludedWorkerProcessMap.count(resolver.address()) || nonExcludedWorkerProcessMap[resolver.address()].machineClassFitness(ProcessClass::Resolver) != bestResolverFitness) {
TraceEvent("ConsistencyCheck_ResolverNotBest").detail("bestResolverFitness", bestResolverFitness).detail("existingResolverFitness", nonExcludedWorkerProcessMap.count(resolver.address()) ? nonExcludedWorkerProcessMap[resolver.address()].machineClassFitness(ProcessClass::Resolver) : -1);
TraceEvent("ConsistencyCheck_ResolverNotBest").detail("BestResolverFitness", bestResolverFitness).detail("ExistingResolverFitness", nonExcludedWorkerProcessMap.count(resolver.address()) ? nonExcludedWorkerProcessMap[resolver.address()].machineClassFitness(ProcessClass::Resolver) : -1);
return false;
}
}

2
fdbserver/workloads/DDBalance.actor.cpp
View File

@ -202,7 +202,7 @@ struct DDBalanceWorkload : TestWorkload {
}
if(maxMovedAmount < end-begin) {
TraceEvent(SevError, "LostKeys").detail("maxMoved",maxMovedAmount).detail("ShouldHaveMoved",end-begin).detail("ClientId", self->clientId).detail("MoverId", moverId)
TraceEvent(SevError, "LostKeys").detail("MaxMoved",maxMovedAmount).detail("ShouldHaveMoved",end-begin).detail("ClientId", self->clientId).detail("MoverId", moverId)
.detail("CurrentBin", sourceBin).detail("NextBin", destinationBin);
ASSERT( false );
}

2
fdbserver/workloads/DDMetrics.actor.cpp
View File

@ -50,7 +50,7 @@ struct DDMetricsWorkload : TestWorkload {
ACTOR Future<Void> work( Database cx, DDMetricsWorkload *self ) {
try {
TraceEvent("DDMetricsWaiting").detail("startDelay", self->startDelay);
TraceEvent("DDMetricsWaiting").detail("StartDelay", self->startDelay);
Void _ = wait( delay( self->startDelay ) );
TraceEvent("DDMetricsStarting");
state double startTime = now();

14
fdbserver/workloads/FastTriggeredWatches.actor.cpp
View File

@ -82,12 +82,12 @@ struct FastTriggeredWatchesWorkload : TestWorkload {
tr.set( key, value.get() );
else
tr.clear( key );
//TraceEvent("FTWSetBegin").detail("key", printable(key)).detail("value", printable(value));
//TraceEvent("FTWSetBegin").detail("Key", printable(key)).detail("Value", printable(value));
Void _ = wait( tr.commit() );
//TraceEvent("FTWSetEnd").detail("key", printable(key)).detail("value", printable(value)).detail("ver", tr.getCommittedVersion());
//TraceEvent("FTWSetEnd").detail("Key", printable(key)).detail("Value", printable(value)).detail("Ver", tr.getCommittedVersion());
return tr.getCommittedVersion();
} catch( Error &e ) {
//TraceEvent("FTWSetError").detail("key", printable(key)).detail("value", printable(value)).error(e);
//TraceEvent("FTWSetError").detail("Key", printable(key)).detail("Value", printable(value)).error(e);
Void _ = wait( tr.onError(e) );
}
}
@ -117,23 +117,23 @@ struct FastTriggeredWatchesWorkload : TestWorkload {
getDuration = now() - watchEnd;
}
lastReadVersion = tr.getReadVersion().get();
//TraceEvent("FTWGet").detail("key", printable(setKey)).detail("value", printable(val)).detail("ver", tr.getReadVersion().get());
//TraceEvent("FTWGet").detail("Key", printable(setKey)).detail("Value", printable(val)).detail("Ver", tr.getReadVersion().get());
if( val == setValue )
break;
ASSERT( first );
state Future<Void> watchFuture = tr.watch( setKey );
Void _ = wait( tr.commit() );
//TraceEvent("FTWStartWatch").detail("key", printable(setKey));
//TraceEvent("FTWStartWatch").detail("Key", printable(setKey));
Void _ = wait( watchFuture );
watchEnd = now();
first = false;
} catch( Error &e ) {
//TraceEvent("FTWWatchError").detail("key", printable(setKey)).error(e);
//TraceEvent("FTWWatchError").detail("Key", printable(setKey)).error(e);
Void _ = wait( tr.onError(e) );
}
}
Version ver = wait( setFuture );
//TraceEvent("FTWWatchDone").detail("key", printable(setKey));
//TraceEvent("FTWWatchDone").detail("Key", printable(setKey));
ASSERT( lastReadVersion - ver >= SERVER_KNOBS->MAX_VERSIONS_IN_FLIGHT || lastReadVersion - ver < SERVER_KNOBS->VERSIONS_PER_SECOND*(12+getDuration) );
if( now() - testStart > self->testDuration )

2
fdbserver/workloads/FileSystem.actor.cpp
View File

@ -310,7 +310,7 @@ struct FileSystemWorkload : TestWorkload {
begin = begin + transfered;
}
if( self->loggingQueries ) {
TraceEvent("DeletionQueryResults").detail("serverID", serverID)
TraceEvent("DeletionQueryResults").detail("ServerID", serverID)
.detail("PathBase", base).detail("DeletedFiles", deletedFiles);
}
return Optional<Version>(Version(0));

68
fdbserver/workloads/FuzzApiCorrectness.actor.cpp
View File

@ -68,15 +68,15 @@ struct ExceptionContract {
{
Severity s = (i->second == Possible) ? SevWarn : SevInfo;
TraceEvent evt(s, func.c_str());
evt.error(e).detail("thrown", true)
.detail("expected", i->second == Possible ? "possible" : "always").backtrace();
evt.error(e).detail("Thrown", true)
.detail("Expected", i->second == Possible ? "possible" : "always").backtrace();
if (augment)
augment(evt);
return;
}
TraceEvent evt(SevError, func.c_str());
evt.error(e).detail("thrown", true).detail("expected", "never").backtrace();
evt.error(e).detail("Thrown", true).detail("Expected", "never").backtrace();
if (augment)
augment(evt);
throw e;
@ -87,7 +87,7 @@ struct ExceptionContract {
for (auto i : expected) {
if (i.second == Always) {
TraceEvent evt(SevError, func.c_str());
evt.detail("thrown", false).detail("expected", "always").error(Error::fromUnvalidatedCode(i.first)).backtrace();
evt.detail("Thrown", false).detail("Expected", "always").error(Error::fromUnvalidatedCode(i.first)).backtrace();
if (augment)
augment(evt);
}
@ -145,13 +145,13 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
maxClearSize = 1<<g_random->randomInt(0, 20);
conflictRange = KeyRangeRef( LiteralStringRef("\xfe"), LiteralStringRef("\xfe\x00") );
TraceEvent("FuzzApiCorrectnessConfiguration")
.detail("nodes", nodes)
.detail("initialKeyDensity", initialKeyDensity)
.detail("adjacentKeys", adjacentKeys)
.detail("valueSizeMin", valueSizeRange.first)
.detail("valueSizeRange", valueSizeRange.second)
.detail("maxClearSize", maxClearSize)
.detail("useSystemKeys", useSystemKeys);
.detail("Nodes", nodes)
.detail("InitialKeyDensity", initialKeyDensity)
.detail("AdjacentKeys", adjacentKeys)
.detail("ValueSizeMin", valueSizeRange.first)
.detail("ValueSizeRange", valueSizeRange.second)
.detail("MaxClearSize", maxClearSize)
.detail("UseSystemKeys", useSystemKeys);
TraceEvent("RemapEventSeverity").detail("TargetEvent", "Net2_LargePacket").detail("OriginalSeverity", SevWarnAlways).detail("NewSeverity", SevInfo);
TraceEvent("RemapEventSeverity").detail("TargetEvent", "LargeTransaction").detail("OriginalSeverity", SevWarnAlways).detail("NewSeverity", SevInfo);
@ -227,7 +227,7 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
}
}
Void _ = wait( unsafeThreadFutureToFuture( tr->commit() ) );
//TraceEvent("WDRInitBatch").detail("i", i).detail("CommittedVersion", tr->getCommittedVersion());
//TraceEvent("WDRInitBatch").detail("I", i).detail("CommittedVersion", tr->getCommittedVersion());
break;
} catch( Error &e ) {
Void _ = wait( unsafeThreadFutureToFuture( tr->onError( e ) ) );
@ -287,7 +287,7 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
operations.push_back(testCases[operationType](++self->operationId, self, tr));
} catch( Error &e ) {
TraceEvent(SevWarn, "IgnoredOperation").error(e)
.detail("operation", operationType).detail("id", self->operationId);
.detail("Operation", operationType).detail("Id", self->operationId);
}
}
@ -373,7 +373,7 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
virtual ThreadFuture<value_type> createFuture( Reference<ITransaction> tr ) = 0;
virtual Void errorCheck(Reference<ITransaction> tr, value_type result) { return Void(); }
virtual void augmentTrace(TraceEvent &e) const { e.detail("id", id); }
virtual void augmentTrace(TraceEvent &e) const { e.detail("Id", id); }
protected:
unsigned int id;
@ -577,7 +577,7 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
void augmentTrace(TraceEvent &e) const {
base_type::augmentTrace(e);
e.detail("version", v);
e.detail("Version", v);
}
};
@ -600,7 +600,7 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
void augmentTrace(TraceEvent &e) const {
base_type::augmentTrace(e);
e.detail("key", printable(key));
e.detail("Key", printable(key));
}
};
@ -623,7 +623,7 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
void augmentTrace(TraceEvent &e) const {
base_type::augmentTrace(e);
e.detail("keysel", keysel.toString());
e.detail("KeySel", keysel.toString());
}
};
@ -661,7 +661,7 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
void augmentTrace(TraceEvent &e) const {
base_type::augmentTrace(e);
e.detail("keysel1", keysel1.toString()).detail("keysel2", keysel2.toString()).detail("limit", limit);
e.detail("KeySel1", keysel1.toString()).detail("KeySel2", keysel2.toString()).detail("Limit", limit);
}
};
@ -690,10 +690,10 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
void augmentTrace(TraceEvent &e) const {
base_type::augmentTrace(e);
e.detail("keysel1", keysel1.toString()).detail("keysel2", keysel2.toString());
e.detail("KeySel1", keysel1.toString()).detail("KeySel2", keysel2.toString());
std::stringstream ss;
ss << "(" << limits.rows << ", " << limits.minRows << ", " << limits.bytes << ")";
e.detail("limits", ss.str());
e.detail("Limits", ss.str());
}
};
@ -732,7 +732,7 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
void augmentTrace(TraceEvent &e) const {
base_type::augmentTrace(e);
e.detail("key1", printable(key1)).detail("key2", printable(key2)).detail("limit", limit);
e.detail("Key1", printable(key1)).detail("Key2", printable(key2)).detail("Limit", limit);
}
};
@ -762,10 +762,10 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
void augmentTrace(TraceEvent &e) const {
base_type::augmentTrace(e);
e.detail("key1", printable(key1)).detail("key2", printable(key2));
e.detail("Key1", printable(key1)).detail("Key2", printable(key2));
std::stringstream ss;
ss << "(" << limits.rows << ", " << limits.minRows << ", " << limits.bytes << ")";
e.detail("limits", ss.str());
e.detail("Limits", ss.str());
}
};
@ -786,7 +786,7 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
void augmentTrace(TraceEvent &e) const {
base_type::augmentTrace(e);
e.detail("key", printable(key));
e.detail("Key", printable(key));
}
};
@ -811,7 +811,7 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
void augmentTrace(TraceEvent &e) const {
base_type::augmentTrace(e);
e.detail("key1", printable(key1)).detail("key2", printable(key2));
e.detail("Key1", printable(key1)).detail("Key2", printable(key2));
}
};
@ -879,7 +879,7 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
void augmentTrace(TraceEvent &e) const {
base_type::augmentTrace(e);
e.detail("key", printable(key)).detail("value", printable(value)).detail("op", op).detail("pos", pos);
e.detail("Key", printable(key)).detail("Value", printable(value)).detail("Op", op).detail("Pos", pos);
}
};
@ -908,7 +908,7 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
void augmentTrace(TraceEvent &e) const {
base_type::augmentTrace(e);
e.detail("key", printable(key)).detail("value", printable(value));
e.detail("Key", printable(key)).detail("Value", printable(value));
}
};
@ -937,7 +937,7 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
void augmentTrace(TraceEvent &e) const {
base_type::augmentTrace(e);
e.detail("key1", printable(key1)).detail("key2", printable(key2));
e.detail("Key1", printable(key1)).detail("Key2", printable(key2));
}
};
@ -966,7 +966,7 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
void augmentTrace(TraceEvent &e) const {
base_type::augmentTrace(e);
e.detail("key1", printable(key1)).detail("key2", printable(key2));
e.detail("Key1", printable(key1)).detail("Key2", printable(key2));
}
};
@ -991,7 +991,7 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
void augmentTrace(TraceEvent &e) const {
base_type::augmentTrace(e);
e.detail("key", printable(key));
e.detail("Key", printable(key));
}
};
@ -1017,7 +1017,7 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
void augmentTrace(TraceEvent &e) const {
base_type::augmentTrace(e);
e.detail("key", printable(key));
e.detail("Key", printable(key));
}
};
@ -1042,7 +1042,7 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
void augmentTrace(TraceEvent &e) const {
base_type::augmentTrace(e);
e.detail("key1", printable(key1)).detail("key2", printable(key2));
e.detail("Key1", printable(key1)).detail("Key2", printable(key2));
}
};
@ -1103,7 +1103,7 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
void augmentTrace(TraceEvent &e) const {
base_type::augmentTrace(e);
e.detail("op", op).detail("val", printable(val));
e.detail("Op", op).detail("Val", printable(val));
}
};
@ -1132,7 +1132,7 @@ struct FuzzApiCorrectnessWorkload : TestWorkload {
void augmentTrace(TraceEvent &e) const {
base_type::augmentTrace(e);
e.detail("errorcode", errorcode);
e.detail("ErrorCode", errorcode);
}
};

6
fdbserver/workloads/LockDatabase.actor.cpp
View File

@ -82,13 +82,13 @@ struct LockDatabaseWorkload : TestWorkload {
Void _ = wait( unlockDatabase(&tr, lockID) );
state Standalone<RangeResultRef> data2 = wait( tr.getRange(normalKeys, 50000) );
if(data.size() != data2.size()) {
TraceEvent(SevError, "DataChangedWhileLocked").detail("beforeSize", data.size()).detail("afterSize", data2.size());
TraceEvent(SevError, "DataChangedWhileLocked").detail("BeforeSize", data.size()).detail("AfterSize", data2.size());
self->ok = false;
} else if(data != data2) {
TraceEvent(SevError, "DataChangedWhileLocked").detail("size", data.size());
TraceEvent(SevError, "DataChangedWhileLocked").detail("Size", data.size());
for(int i = 0; i < data.size(); i++) {
if( data[i] != data2[i] ) {
TraceEvent(SevError, "DataChangedWhileLocked").detail("i", i).detail("before", printable(data[i])).detail("after", printable(data2[i]));
TraceEvent(SevError, "DataChangedWhileLocked").detail("I", i).detail("Before", printable(data[i])).detail("After", printable(data2[i]));
}
}
self->ok = false;

2
fdbserver/workloads/LowLatency.actor.cpp
View File

@ -73,7 +73,7 @@ struct LowLatencyWorkload : TestWorkload {
}
}
if(now() - operationStart > self->maxLatency) {
TraceEvent(SevError, "LatencyTooLarge").detail("maxLatency", self->maxLatency).detail("observedLatency", now() - operationStart);
TraceEvent(SevError, "LatencyTooLarge").detail("MaxLatency", self->maxLatency).detail("ObservedLatency", now() - operationStart);
self->ok = false;
}
if( now() - testStart > self->testDuration )

14
fdbserver/workloads/MachineAttrition.actor.cpp
View File

@ -144,9 +144,9 @@ struct MachineAttritionWorkload : TestWorkload {
g_simulator.killDataCenter( target, kt );
} else {
while ( killedMachines < self->machinesToKill && self->machines.size() > self->machinesToLeave) {
TraceEvent("WorkerKillBegin").detail("killedMachines", killedMachines)
.detail("machinesToKill", self->machinesToKill).detail("machinesToLeave", self->machinesToLeave)
.detail("machines", self->machines.size());
TraceEvent("WorkerKillBegin").detail("KilledMachines", killedMachines)
.detail("MachinesToKill", self->machinesToKill).detail("MachinesToLeave", self->machinesToLeave)
.detail("Machines", self->machines.size());
TEST(true); // Killing a machine
Void _ = wait( delay( delayBeforeKill ) );
@ -170,10 +170,10 @@ struct MachineAttritionWorkload : TestWorkload {
LocalityData targetMachine = self->machines.back();
TraceEvent("Assassination").detail("TargetMachine", targetMachine.toString())
.detailext("zoneId", targetMachine.zoneId())
.detail("Reboot", self->reboot).detail("killedMachines", killedMachines)
.detail("machinesToKill", self->machinesToKill).detail("machinesToLeave", self->machinesToLeave)
.detail("machines", self->machines.size()).detail("Replace", self->replacement);
.detailext("ZoneId", targetMachine.zoneId())
.detail("Reboot", self->reboot).detail("KilledMachines", killedMachines)
.detail("MachinesToKill", self->machinesToKill).detail("MachinesToLeave", self->machinesToLeave)
.detail("Machines", self->machines.size()).detail("Replace", self->replacement);
if (self->reboot) {
if( g_random->random01() > 0.5 ) {

14
fdbserver/workloads/MemoryLifetime.actor.cpp
View File

@ -90,7 +90,7 @@ struct MemoryLifetime : KVWorkload {
state Value getRange_newValue = self->randomValue();
state bool getRange_isSnapshot = g_random->random01() < 0.5;
//TraceEvent("MemoryLifetimeCheck").detail("isReverse", getRange_isReverse).detail("startKey", printable(getRange_startKey)).detail("randomStart", getRange_randomStart).detail("newValue", getRange_newValue.size()).detail("isSnapshot", getRange_isSnapshot);
//TraceEvent("MemoryLifetimeCheck").detail("IsReverse", getRange_isReverse).detail("StartKey", printable(getRange_startKey)).detail("RandomStart", getRange_randomStart).detail("NewValue", getRange_newValue.size()).detail("IsSnapshot", getRange_isSnapshot);
if(getRange_randomStart)
tr.set(getRange_startKey, getRange_newValue);
state Standalone<RangeResultRef> getRange_res1 = wait( tr.getRange(getRange_queryRange, GetRangeLimits(4000), getRange_isSnapshot, getRange_isReverse) );
@ -103,16 +103,16 @@ struct MemoryLifetime : KVWorkload {
for(int i = 0; i < getRange_res1.size(); i++) {
if(getRange_res1[i].key != getRange_res2[i].key) {
TraceEvent(SevError, "MemoryLifetimeCheckKeyError")
.detail("key1", printable(getRange_res1[i].key)).detail("key2", printable(getRange_res2[i].key))
.detail("value1", getRange_res1[i].value.size()).detail("value2", getRange_res2[i].value.size())
.detail("i", i).detail("size", getRange_res2.size());
.detail("Key1", printable(getRange_res1[i].key)).detail("Key2", printable(getRange_res2[i].key))
.detail("Value1", getRange_res1[i].value.size()).detail("Value2", getRange_res2[i].value.size())
.detail("I", i).detail("Size", getRange_res2.size());
ASSERT(false);
}
if(getRange_res1[i].value != getRange_res2[i].value) {
TraceEvent(SevError, "MemoryLifetimeCheckValueError")
.detail("key1", printable(getRange_res1[i].key)).detail("key2", printable(getRange_res2[i].key))
.detail("value1", getRange_res1[i].value.size()).detail("value2", getRange_res2[i].value.size())
.detail("i", i).detail("size", getRange_res2.size());
.detail("Key1", printable(getRange_res1[i].key)).detail("Key2", printable(getRange_res2[i].key))
.detail("Value1", getRange_res1[i].value.size()).detail("Value2", getRange_res2[i].value.size())
.detail("I", i).detail("Size", getRange_res2.size());
ASSERT(false);
}
}

2
fdbserver/workloads/Ping.actor.cpp
View File

@ -163,7 +163,7 @@ struct PingWorkload : TestWorkload {
self->totalMessageLatency += elapsed;
self->maxMessageLatency += std::max(0.0, elapsed*1000.0 - self->maxMessageLatency.getValue());
++self->messages;
if (self->logging) TraceEvent("Ping").detail("ms", elapsed*1000.0).detail("To", addr);
if (self->logging) TraceEvent("Ping").detail("Elapsed", elapsed).detail("To", addr);
}
}

2
fdbserver/workloads/RandomMoveKeys.actor.cpp
View File

@ -67,7 +67,7 @@ struct MoveKeysWorkload : TestWorkload {
state int oldMode = wait( setDDMode( cx, 0 ) );
TraceEvent("RMKStartModeSetting");
Void _ = wait( timeout( reportErrors( self->worker( cx, self ), "moveKeysWorkloadWorkerError" ), self->testDuration, Void() ) );
Void _ = wait( timeout( reportErrors( self->worker( cx, self ), "MoveKeysWorkloadWorkerError" ), self->testDuration, Void() ) );
// Always set the DD mode back, even if we die with an error
TraceEvent("RMKDoneMoving");
int _ = wait( setDDMode( cx, oldMode ) );

10
fdbserver/workloads/RandomSelector.actor.cpp
View File

@ -437,7 +437,7 @@ struct RandomSelectorWorkload : TestWorkload {
randomByteLimit = g_random->randomInt( 0, (self->maxOffset+self->maxKeySpace)*512);
reverse = g_random->random01() > 0.5 ? false : true;
//TraceEvent("RYOWgetRange").detail("KeyA", myKeyA).detail("KeyB", myKeyB).detail("onEqualA",onEqualA).detail("onEqualB",onEqualB).detail("offsetA",offsetA).detail("offsetB",offsetB).detail("randomLimit",randomLimit).detail("randomByteLimit", randomByteLimit).detail("reverse", reverse);
//TraceEvent("RYOWgetRange").detail("KeyA", myKeyA).detail("KeyB", myKeyB).detail("OnEqualA",onEqualA).detail("OnEqualB",onEqualB).detail("OffsetA",offsetA).detail("OffsetB",offsetB).detail("RandomLimit",randomLimit).detail("RandomByteLimit", randomByteLimit).detail("Reverse", reverse);
state Standalone<RangeResultRef> getRangeTest1;
Standalone<RangeResultRef> getRangeTest = wait( trRYOW.getRange(KeySelectorRef(StringRef(clientID + "b/" + myKeyA),onEqualA,offsetA),KeySelectorRef(StringRef(clientID + "b/" + myKeyB),onEqualB,offsetB),randomLimit,false,reverse) );
@ -449,7 +449,7 @@ struct RandomSelectorWorkload : TestWorkload {
bool fail = false;
if( getRangeTest1.size() != getRangeTest2.size() ) {
TraceEvent(SevError, "RanSelTestFailure").detail("Reason", "The getRange results did not match sizes").detail("size1", getRangeTest1.size()).detail("size2",getRangeTest2.size()).detail("limit",randomLimit).detail("byteLimit", randomByteLimit).detail("bytes1", getRangeTest1.expectedSize()).detail("bytes2", getRangeTest2.expectedSize()).detail("reverse", reverse);
TraceEvent(SevError, "RanSelTestFailure").detail("Reason", "The getRange results did not match sizes").detail("Size1", getRangeTest1.size()).detail("Size2",getRangeTest2.size()).detail("Limit",randomLimit).detail("ByteLimit", randomByteLimit).detail("Bytes1", getRangeTest1.expectedSize()).detail("Bytes2", getRangeTest2.expectedSize()).detail("Reverse", reverse);
fail = true;
self->fail=true;
}
@ -459,7 +459,7 @@ struct RandomSelectorWorkload : TestWorkload {
std::string valueA = printable(getRangeTest1[k].value);
std::string keyB = printable(getRangeTest2[k].key);
std::string valueB = printable(getRangeTest2[k].value);
TraceEvent(SevError, "RanSelTestFailure").detail("Reason", "The getRange results did not match contents").detail("KeyA",keyA).detail("ValueA",valueA).detail("KeyB",keyB).detail("ValueB",valueB).detail("reverse", reverse);
TraceEvent(SevError, "RanSelTestFailure").detail("Reason", "The getRange results did not match contents").detail("KeyA",keyA).detail("ValueA",valueA).detail("KeyB",keyB).detail("ValueB",valueB).detail("Reverse", reverse);
fail = true;
self->fail=true;
}
@ -475,7 +475,7 @@ struct RandomSelectorWorkload : TestWorkload {
outStr2 = outStr2 + printable(getRangeTest2[k].key) + " " + format("%d", getRangeTest2[k].value.size()) + " ";
}
TraceEvent("RanSelTestLog").detail("RYOW",outStr1).detail("normal",outStr2);
TraceEvent("RanSelTestLog").detail("RYOW",outStr1).detail("Normal",outStr2);
}
@ -505,7 +505,7 @@ struct RandomSelectorWorkload : TestWorkload {
}
for(int k = 0; k < finalTest1.size(); k++)
if(finalTest1[k].value != finalTest2[k].value) {
TraceEvent(SevError, "RanSelTestFailure").detail("Reason", "The final results did not match contents").detail("KeyA",printable(finalTest1[k].key)).detail("ValueA",printable(finalTest1[k].value)).detail("KeyB",printable(finalTest2[k].key)).detail("ValueB",printable(finalTest2[k].value)).detail("reverse", reverse);
TraceEvent(SevError, "RanSelTestFailure").detail("Reason", "The final results did not match contents").detail("KeyA",printable(finalTest1[k].key)).detail("ValueA",printable(finalTest1[k].value)).detail("KeyB",printable(finalTest2[k].key)).detail("ValueB",printable(finalTest2[k].value)).detail("Reverse", reverse);
self->fail=true;
}
break;

8
fdbserver/workloads/ReadWrite.actor.cpp
View File

@ -313,10 +313,10 @@ struct ReadWriteWorkload : KVWorkload {
elapsed += self->periodicLoggingInterval;
Void _ = wait( delayUntil(start + elapsed) );
TraceEvent("RW_RowReadLatency").detail("mean", self->readLatencies.mean()).detail("median", 1000 * self->readLatencies.median()).detail("percentile5", 1000 * self->readLatencies.percentile(.05)).detail("percentile95", 1000 * self->readLatencies.percentile(.95)).detail("Count", self->readLatencyCount).detail("Elapsed", elapsed);
TraceEvent("RW_GRVLatency").detail("mean", 1000 * self->GRVLatencies.mean()).detail("median", 1000 * self->GRVLatencies.median()).detail("percentile5", 1000 * self->GRVLatencies.percentile(.05)).detail("percentile95", 1000 * self->GRVLatencies.percentile(.95));
TraceEvent("RW_CommitLatency").detail("mean", 1000 * self->commitLatencies.mean()).detail("median", 1000 * self->commitLatencies.median()).detail("percentile5", 1000 * self->commitLatencies.percentile(.05)).detail("percentile95", 1000 * self->commitLatencies.percentile(.95));
TraceEvent("RW_TotalLatency").detail("mean", 1000 * self->latencies.mean()).detail("median", 1000 * self->latencies.median()).detail("percentile5", 1000 * self->latencies.percentile(.05)).detail("percentile95", 1000 * self->latencies.percentile(.95));
TraceEvent("RW_RowReadLatency").detail("Mean", self->readLatencies.mean()).detail("Median", self->readLatencies.median()).detail("Percentile5", self->readLatencies.percentile(.05)).detail("Percentile95", self->readLatencies.percentile(.95)).detail("Count", self->readLatencyCount).detail("Elapsed", elapsed);
TraceEvent("RW_GRVLatency").detail("Mean", self->GRVLatencies.mean()).detail("Median", self->GRVLatencies.median()).detail("Percentile5", self->GRVLatencies.percentile(.05)).detail("Percentile95", self->GRVLatencies.percentile(.95));
TraceEvent("RW_CommitLatency").detail("Mean", self->commitLatencies.mean()).detail("Median", self->commitLatencies.median()).detail("Percentile5", self->commitLatencies.percentile(.05)).detail("Percentile95", self->commitLatencies.percentile(.95));
TraceEvent("RW_TotalLatency").detail("Mean", self->latencies.mean()).detail("Median", self->latencies.median()).detail("Percentile5", self->latencies.percentile(.05)).detail("Percentile95", self->latencies.percentile(.95));
int64_t ops = (self->aTransactions.getValue() * (self->readsPerTransactionA+self->writesPerTransactionA)) +
(self->bTransactions.getValue() * (self->readsPerTransactionB+self->writesPerTransactionB));

42
fdbserver/workloads/RemoveServersSafely.actor.cpp
View File

@ -170,15 +170,15 @@ struct RemoveServersSafelyWorkload : TestWorkload {
auto processNet = AddressExclusion(processInfo->address.ip, processInfo->address.port);
if (processAddrs.find(processNet) != processAddrs.end()) {
processes.push_back(processInfo);
TraceEvent("RemoveAndKill", functionId).detail("Step", "getProcessItem").detail("ProcessAddress", processInfo->address).detail("Process", describe(*processInfo)).detail("failed", processInfo->failed).detail("excluded", processInfo->excluded).detail("rebooting", processInfo->rebooting).detail("Protected", g_simulator.protectedAddresses.count(processInfo->address));
TraceEvent("RemoveAndKill", functionId).detail("Step", "getProcessItem").detail("ProcessAddress", processInfo->address).detail("Process", describe(*processInfo)).detail("Failed", processInfo->failed).detail("Excluded", processInfo->excluded).detail("Rebooting", processInfo->rebooting).detail("Protected", g_simulator.protectedAddresses.count(processInfo->address));
}
else {
TraceEvent("RemoveAndKill", functionId).detail("Step", "getProcessNoItem").detail("ProcessAddress", processInfo->address).detail("Process", describe(*processInfo)).detail("failed", processInfo->failed).detail("excluded", processInfo->excluded).detail("rebooting", processInfo->rebooting).detail("Protected", g_simulator.protectedAddresses.count(processInfo->address));
TraceEvent("RemoveAndKill", functionId).detail("Step", "getProcessNoItem").detail("ProcessAddress", processInfo->address).detail("Process", describe(*processInfo)).detail("Failed", processInfo->failed).detail("Excluded", processInfo->excluded).detail("Rebooting", processInfo->rebooting).detail("Protected", g_simulator.protectedAddresses.count(processInfo->address));
}
}
TraceEvent("RemoveAndKill", functionId).detail("Step", "getProcesses")
.detail("netAddrSize",netAddrs.size()).detail("processAddrSize",processAddrs.size())
.detail("netAddrs",describe(netAddrs)).detail("processAddrs",describe(processAddrs))
.detail("NetAddrSize",netAddrs.size()).detail("ProcessAddrSize",processAddrs.size())
.detail("NetAddrs",describe(netAddrs)).detail("ProcessAddrs",describe(processAddrs))
.detail("Proceses", processes.size()).detail("MachineProcesses", machineProcesses.size());
return processes;
@ -196,7 +196,7 @@ struct RemoveServersSafelyWorkload : TestWorkload {
}
for (auto& procRecord : procArray) {
procRecord->excluded = true;
TraceEvent("RemoveAndKill").detail("Step", "ExcludeAddress").detail("ProcessAddress", procRecord->address).detail("Process", describe(*procRecord)).detail("failed", procRecord->failed).detail("rebooting", procRecord->rebooting).detail("ClusterAvailable", g_simulator.isAvailable());
TraceEvent("RemoveAndKill").detail("Step", "ExcludeAddress").detail("ProcessAddress", procRecord->address).detail("Process", describe(*procRecord)).detail("Failed", procRecord->failed).detail("Rebooting", procRecord->rebooting).detail("ClusterAvailable", g_simulator.isAvailable());
}
return procArray;
}
@ -215,7 +215,7 @@ struct RemoveServersSafelyWorkload : TestWorkload {
// Only change the exclusion member, if not failed since it will require a reboot to revive it
if (!procRecord->failed)
procRecord->excluded = false;
TraceEvent("RemoveAndKill").detail("Step", "IncludeAddress").detail("ProcessAddress", procRecord->address).detail("Process", describe(*procRecord)).detail("failed", procRecord->failed).detail("rebooting", procRecord->rebooting).detail("ClusterAvailable", g_simulator.isAvailable());
TraceEvent("RemoveAndKill").detail("Step", "IncludeAddress").detail("ProcessAddress", procRecord->address).detail("Process", describe(*procRecord)).detail("Failed", procRecord->failed).detail("Rebooting", procRecord->rebooting).detail("ClusterAvailable", g_simulator.isAvailable());
}
return procArray;
}
@ -288,23 +288,23 @@ struct RemoveServersSafelyWorkload : TestWorkload {
state std::vector<ISimulator::ProcessInfo*> killProcArray;
state bool bClearedFirst;
TraceEvent("RemoveAndKill").detail("Step", "exclude list first").detail("toKill", describe(toKill1)).detail("KillTotal", toKill1.size()).detail("ClusterAvailable", g_simulator.isAvailable());
TraceEvent("RemoveAndKill").detail("Step", "exclude list first").detail("ToKill", describe(toKill1)).detail("KillTotal", toKill1.size()).detail("ClusterAvailable", g_simulator.isAvailable());
self->excludeAddresses(toKill1);
Optional<Void> result = wait( timeout( removeAndKill( self, cx, toKill1, NULL), self->kill1Timeout ) );
bClearedFirst = result.present();
TraceEvent("RemoveAndKill").detail("Step", "excluded list first").detail("excluderesult", bClearedFirst ? "succeeded" : "failed").detail("KillTotal", toKill1.size()).detail("Processes", killProcArray.size()).detail("toKill1", describe(toKill1)).detail("ClusterAvailable", g_simulator.isAvailable());
TraceEvent("RemoveAndKill").detail("Step", "excluded list first").detail("Excluderesult", bClearedFirst ? "succeeded" : "failed").detail("KillTotal", toKill1.size()).detail("Processes", killProcArray.size()).detail("ToKill1", describe(toKill1)).detail("ClusterAvailable", g_simulator.isAvailable());
bClearedFirst=false;
// Include the servers, if unable to exclude
if (!bClearedFirst) {
// Get the updated list of processes which may have changed due to reboots, deletes, etc
TraceEvent("RemoveAndKill").detail("Step", "include all first").detail("KillTotal", toKill1.size()).detail("toKill", describe(toKill1)).detail("ClusterAvailable", g_simulator.isAvailable());
TraceEvent("RemoveAndKill").detail("Step", "include all first").detail("KillTotal", toKill1.size()).detail("ToKill", describe(toKill1)).detail("ClusterAvailable", g_simulator.isAvailable());
Void _ = wait( includeServers( cx, vector<AddressExclusion>(1) ) );
self->includeAddresses(toKill1);
TraceEvent("RemoveAndKill").detail("Step", "included all first").detail("KillTotal", toKill1.size()).detail("toKill", describe(toKill1)).detail("ClusterAvailable", g_simulator.isAvailable());
TraceEvent("RemoveAndKill").detail("Step", "included all first").detail("KillTotal", toKill1.size()).detail("ToKill", describe(toKill1)).detail("ClusterAvailable", g_simulator.isAvailable());
}
// Get the list of protected servers
@ -313,24 +313,24 @@ struct RemoveServersSafelyWorkload : TestWorkload {
// Update the kill networks to the killable processes
toKill2 = self->getNetworks(killProcArray);
TraceEvent("RemoveAndKill").detail("Step", "exclude list second").detail("KillTotal", toKill2.size()).detail("toKill", describe(toKill2)).detail("ClusterAvailable", g_simulator.isAvailable());
TraceEvent("RemoveAndKill").detail("Step", "exclude list second").detail("KillTotal", toKill2.size()).detail("ToKill", describe(toKill2)).detail("ClusterAvailable", g_simulator.isAvailable());
self->excludeAddresses(toKill2);
// The second set of machines is selected so that we can always make progress without it, even after the permitted number of other permanent failures
// so we expect to succeed after a finite amount of time
TraceEvent("RemoveAndKill").detail("Step", "exclude second list").detail("toKill2", describe(toKill2)).detail("KillTotal", toKill2.size())
TraceEvent("RemoveAndKill").detail("Step", "exclude second list").detail("ToKill2", describe(toKill2)).detail("KillTotal", toKill2.size())
.detail("Processes", killProcArray.size()).detail("ClusterAvailable", g_simulator.isAvailable());
Void _ = wait( reportErrors( timeoutError( removeAndKill( self, cx, toKill2, bClearedFirst ? &toKill1 : NULL), self->kill2Timeout ), "RemoveServersSafelyError", UID() ) );
TraceEvent("RemoveAndKill").detail("Step", "excluded second list").detail("KillTotal", toKill1.size()).detail("toKill", describe(toKill2)).detail("ClusterAvailable", g_simulator.isAvailable());
TraceEvent("RemoveAndKill").detail("Step", "excluded second list").detail("KillTotal", toKill1.size()).detail("ToKill", describe(toKill2)).detail("ClusterAvailable", g_simulator.isAvailable());
// Reinclude all of the machine, if buggified
if (BUGGIFY) {
// Get the updated list of processes which may have changed due to reboots, deletes, etc
TraceEvent("RemoveAndKill").detail("Step", "include all second").detail("KillTotal", toKill1.size()).detail("toKill", describe(toKill2)).detail("ClusterAvailable", g_simulator.isAvailable());
TraceEvent("RemoveAndKill").detail("Step", "include all second").detail("KillTotal", toKill1.size()).detail("ToKill", describe(toKill2)).detail("ClusterAvailable", g_simulator.isAvailable());
Void _ = wait( includeServers( cx, vector<AddressExclusion>(1) ) );
self->includeAddresses(toKill2);
TraceEvent("RemoveAndKill").detail("Step", "included all second").detail("KillTotal", toKill1.size()).detail("toKill", describe(toKill2)).detail("ClusterAvailable", g_simulator.isAvailable());
TraceEvent("RemoveAndKill").detail("Step", "included all second").detail("KillTotal", toKill1.size()).detail("ToKill", describe(toKill2)).detail("ClusterAvailable", g_simulator.isAvailable());
}
return Void();
@ -352,15 +352,15 @@ struct RemoveServersSafelyWorkload : TestWorkload {
.detail("Processes", killProcArray.size()).detail("ClusterAvailable", g_simulator.isAvailable()).detail("RemoveViaClear", removeViaClear);
for (auto& killProcess : killProcArray) {
if (g_simulator.protectedAddresses.count(killProcess->address))
TraceEvent("RemoveAndKill", functionId).detail("Step", "NoKill Process").detail("Process", describe(*killProcess)).detail("failed", killProcess->failed).detail("rebooting", killProcess->rebooting).detail("ClusterAvailable", g_simulator.isAvailable()).detail("Protected", g_simulator.protectedAddresses.count(killProcess->address));
TraceEvent("RemoveAndKill", functionId).detail("Step", "NoKill Process").detail("Process", describe(*killProcess)).detail("Failed", killProcess->failed).detail("Rebooting", killProcess->rebooting).detail("ClusterAvailable", g_simulator.isAvailable()).detail("Protected", g_simulator.protectedAddresses.count(killProcess->address));
else if (removeViaClear) {
g_simulator.rebootProcess( killProcess, ISimulator::RebootProcessAndDelete);
TraceEvent("RemoveAndKill", functionId).detail("Step", "Clear Process").detail("Process", describe(*killProcess)).detail("failed", killProcess->failed).detail("rebooting", killProcess->rebooting).detail("ClusterAvailable", g_simulator.isAvailable()).detail("Protected", g_simulator.protectedAddresses.count(killProcess->address));
TraceEvent("RemoveAndKill", functionId).detail("Step", "Clear Process").detail("Process", describe(*killProcess)).detail("Failed", killProcess->failed).detail("Rebooting", killProcess->rebooting).detail("ClusterAvailable", g_simulator.isAvailable()).detail("Protected", g_simulator.protectedAddresses.count(killProcess->address));
}
/*
else {
g_simulator.killProcess( killProcess, ISimulator::KillInstantly );
TraceEvent("RemoveAndKill", functionId).detail("Step", "Kill Process").detail("Process", describe(*killProcess)).detail("failed", killProcess->failed).detail("rebooting", killProcess->rebooting).detail("ClusterAvailable", g_simulator.isAvailable()).detail("Protected", g_simulator.protectedAddresses.count(killProcess->address));
TraceEvent("RemoveAndKill", functionId).detail("Step", "Kill Process").detail("Process", describe(*killProcess)).detail("Failed", killProcess->failed).detail("Rebooting", killProcess->rebooting).detail("ClusterAvailable", g_simulator.isAvailable()).detail("Protected", g_simulator.protectedAddresses.count(killProcess->address));
}
*/
}
@ -400,7 +400,7 @@ struct RemoveServersSafelyWorkload : TestWorkload {
std::copy(toKill.begin(), toKill.end(), std::back_inserter(toKillArray));
killProcArray = self->getProcesses(toKill);
TraceEvent("RemoveAndKill", functionId).detail("Step", "Activate Server Exclusion").detail("KillAddrs", toKill.size()).detail("KillProcs", killProcArray.size()).detail("MissingProcs", toKill.size()!=killProcArray.size()).detail("toKill", describe(toKill)).detail("Addresses", describe(toKillArray)).detail("ClusterAvailable", g_simulator.isAvailable());
TraceEvent("RemoveAndKill", functionId).detail("Step", "Activate Server Exclusion").detail("KillAddrs", toKill.size()).detail("KillProcs", killProcArray.size()).detail("MissingProcs", toKill.size()!=killProcArray.size()).detail("ToKill", describe(toKill)).detail("Addresses", describe(toKillArray)).detail("ClusterAvailable", g_simulator.isAvailable());
Void _ = wait( excludeServers( cx, toKillArray ) );
// We need to skip at least the quorum change if there's nothing to kill, because there might not be enough servers left
@ -410,7 +410,7 @@ struct RemoveServersSafelyWorkload : TestWorkload {
TraceEvent("RemoveAndKill", functionId).detail("Step", "Wait For Server Exclusion").detail("Addresses", describe(toKill)).detail("ClusterAvailable", g_simulator.isAvailable());
Void _ = wait( waitForExcludedServers( cx, toKillArray ) );
TraceEvent("RemoveAndKill", functionId).detail("Step", "coordinators auto").detail("desiredCoordinators", g_simulator.desiredCoordinators).detail("ClusterAvailable", g_simulator.isAvailable());
TraceEvent("RemoveAndKill", functionId).detail("Step", "coordinators auto").detail("DesiredCoordinators", g_simulator.desiredCoordinators).detail("ClusterAvailable", g_simulator.isAvailable());
// Setup the coordinators BEFORE the exclusion
// Otherwise, we may end up with NotEnoughMachinesForCoordinators
@ -420,7 +420,7 @@ struct RemoveServersSafelyWorkload : TestWorkload {
cycle ++;
nQuorum = ((g_simulator.desiredCoordinators+1)/2)*2-1;
CoordinatorsResult::Type result = wait( changeQuorum( cx, autoQuorumChange(nQuorum) ) );
TraceEvent(result==CoordinatorsResult::SUCCESS || result==CoordinatorsResult::SAME_NETWORK_ADDRESSES ? SevInfo : SevWarn, "RemoveAndKillQuorumChangeResult").detail("Step", "coordinators auto").detail("Result", (int)result).detail("attempt", cycle).detail("Quorum", nQuorum).detail("desiredCoordinators", g_simulator.desiredCoordinators);
TraceEvent(result==CoordinatorsResult::SUCCESS || result==CoordinatorsResult::SAME_NETWORK_ADDRESSES ? SevInfo : SevWarn, "RemoveAndKillQuorumChangeResult").detail("Step", "coordinators auto").detail("Result", (int)result).detail("Attempt", cycle).detail("Quorum", nQuorum).detail("DesiredCoordinators", g_simulator.desiredCoordinators);
if (result==CoordinatorsResult::SUCCESS || result==CoordinatorsResult::SAME_NETWORK_ADDRESSES)
break;
}

2
fdbserver/workloads/Rollback.actor.cpp
View File

@ -47,7 +47,7 @@ struct RollbackWorkload : TestWorkload {
virtual Future<Void> start( Database const& cx ) {
if (&g_simulator == g_network && enabled)
return timeout(
reportErrors( rollbackFailureWorker( cx, this, meanDelay ), "rollbackFailureWorkerError" ),
reportErrors( rollbackFailureWorker( cx, this, meanDelay ), "RollbackFailureWorkerError" ),
testDuration, Void() );
return Void();
}

Some files were not shown because too many files have changed in this diff Show More