llvm-project/llvm/unittests/XRay/FDRProducerConsumerTest.cpp

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

223 lines
6.8 KiB
C++
Raw Normal View History

//===- llvm/unittest/XRay/FDRProducerConsumerTest.cpp -----------*- C++ -*-===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
//
// Test for round-trip record writing and reading.
//
//===----------------------------------------------------------------------===//
#include "llvm/Support/DataExtractor.h"
#include "llvm/Support/raw_ostream.h"
#include "llvm/XRay/FDRLogBuilder.h"
#include "llvm/XRay/FDRRecordConsumer.h"
#include "llvm/XRay/FDRRecordProducer.h"
#include "llvm/XRay/FDRRecords.h"
#include "llvm/XRay/FDRTraceWriter.h"
#include "llvm/XRay/FileHeaderReader.h"
#include "gmock/gmock.h"
#include "gtest/gtest.h"
#include <string>
#include <tuple>
namespace llvm {
namespace xray {
namespace {
using ::testing::Eq;
using ::testing::IsEmpty;
using ::testing::Not;
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
using ::testing::SizeIs;
template <class RecordType> std::unique_ptr<Record> MakeRecord();
template <> std::unique_ptr<Record> MakeRecord<NewBufferRecord>() {
return std::make_unique<NewBufferRecord>(1);
}
template <> std::unique_ptr<Record> MakeRecord<NewCPUIDRecord>() {
return std::make_unique<NewCPUIDRecord>(1, 2);
}
template <> std::unique_ptr<Record> MakeRecord<TSCWrapRecord>() {
return std::make_unique<TSCWrapRecord>(1);
}
template <> std::unique_ptr<Record> MakeRecord<WallclockRecord>() {
return std::make_unique<WallclockRecord>(1, 2);
}
template <> std::unique_ptr<Record> MakeRecord<CustomEventRecord>() {
return std::make_unique<CustomEventRecord>(4, 1, 2, "data");
}
template <> std::unique_ptr<Record> MakeRecord<CallArgRecord>() {
return std::make_unique<CallArgRecord>(1);
}
template <> std::unique_ptr<Record> MakeRecord<PIDRecord>() {
return std::make_unique<PIDRecord>(1);
}
template <> std::unique_ptr<Record> MakeRecord<FunctionRecord>() {
return std::make_unique<FunctionRecord>(RecordTypes::ENTER, 1, 2);
}
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
template <> std::unique_ptr<Record> MakeRecord<CustomEventRecordV5>() {
return std::make_unique<CustomEventRecordV5>(4, 1, "data");
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
}
template <> std::unique_ptr<Record> MakeRecord<TypedEventRecord>() {
return std::make_unique<TypedEventRecord>(4, 1, 2, "data");
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
}
template <class T> class RoundTripTest : public ::testing::Test {
public:
RoundTripTest() : Data(), OS(Data) {
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
H.Version = 4;
H.Type = 1;
H.ConstantTSC = true;
H.NonstopTSC = true;
H.CycleFrequency = 3e9;
Writer = std::make_unique<FDRTraceWriter>(OS, H);
Rec = MakeRecord<T>();
}
protected:
std::string Data;
raw_string_ostream OS;
XRayFileHeader H;
std::unique_ptr<FDRTraceWriter> Writer;
std::unique_ptr<Record> Rec;
};
TYPED_TEST_CASE_P(RoundTripTest);
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
template <class T> class RoundTripTestV5 : public ::testing::Test {
public:
RoundTripTestV5() : Data(), OS(Data) {
H.Version = 5;
H.Type = 1;
H.ConstantTSC = true;
H.NonstopTSC = true;
H.CycleFrequency = 3e9;
Writer = std::make_unique<FDRTraceWriter>(OS, H);
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
Rec = MakeRecord<T>();
}
protected:
std::string Data;
raw_string_ostream OS;
XRayFileHeader H;
std::unique_ptr<FDRTraceWriter> Writer;
std::unique_ptr<Record> Rec;
};
TYPED_TEST_CASE_P(RoundTripTestV5);
// This test ensures that the writing and reading implementations are in sync --
// that given write(read(write(R))) == R.
TYPED_TEST_P(RoundTripTest, RoundTripsSingleValue) {
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
// Always write a buffer extents record which will cover the correct size of
// the record, for version 3 and up.
BufferExtents BE(200);
ASSERT_FALSE(errorToBool(BE.apply(*this->Writer)));
auto &R = this->Rec;
ASSERT_FALSE(errorToBool(R->apply(*this->Writer)));
this->OS.flush();
DataExtractor DE(this->Data, sys::IsLittleEndianHost, 8);
uint64_t OffsetPtr = 0;
auto HeaderOrErr = readBinaryFormatHeader(DE, OffsetPtr);
if (!HeaderOrErr)
FAIL() << HeaderOrErr.takeError();
FileBasedRecordProducer P(HeaderOrErr.get(), DE, OffsetPtr);
std::vector<std::unique_ptr<Record>> Records;
LogBuilderConsumer C(Records);
while (DE.isValidOffsetForDataOfSize(OffsetPtr, 1)) {
auto R = P.produce();
if (!R)
FAIL() << R.takeError();
if (auto E = C.consume(std::move(R.get())))
FAIL() << E;
}
ASSERT_THAT(Records, Not(IsEmpty()));
std::string Data2;
raw_string_ostream OS2(Data2);
FDRTraceWriter Writer2(OS2, this->H);
for (auto &P : Records)
ASSERT_FALSE(errorToBool(P->apply(Writer2)));
OS2.flush();
EXPECT_EQ(Data2.substr(sizeof(XRayFileHeader)),
this->Data.substr(sizeof(XRayFileHeader)));
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
ASSERT_THAT(Records, SizeIs(2));
EXPECT_THAT(Records[1]->getRecordType(), Eq(R->getRecordType()));
}
REGISTER_TYPED_TEST_CASE_P(RoundTripTest, RoundTripsSingleValue);
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
// We duplicate the above case for the V5 version using different types and
// encodings.
TYPED_TEST_P(RoundTripTestV5, RoundTripsSingleValue) {
BufferExtents BE(200);
ASSERT_FALSE(errorToBool(BE.apply(*this->Writer)));
auto &R = this->Rec;
ASSERT_FALSE(errorToBool(R->apply(*this->Writer)));
this->OS.flush();
DataExtractor DE(this->Data, sys::IsLittleEndianHost, 8);
uint64_t OffsetPtr = 0;
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
auto HeaderOrErr = readBinaryFormatHeader(DE, OffsetPtr);
if (!HeaderOrErr)
FAIL() << HeaderOrErr.takeError();
FileBasedRecordProducer P(HeaderOrErr.get(), DE, OffsetPtr);
std::vector<std::unique_ptr<Record>> Records;
LogBuilderConsumer C(Records);
while (DE.isValidOffsetForDataOfSize(OffsetPtr, 1)) {
auto R = P.produce();
if (!R)
FAIL() << R.takeError();
if (auto E = C.consume(std::move(R.get())))
FAIL() << E;
}
ASSERT_THAT(Records, Not(IsEmpty()));
std::string Data2;
raw_string_ostream OS2(Data2);
FDRTraceWriter Writer2(OS2, this->H);
for (auto &P : Records)
ASSERT_FALSE(errorToBool(P->apply(Writer2)));
OS2.flush();
EXPECT_EQ(Data2.substr(sizeof(XRayFileHeader)),
this->Data.substr(sizeof(XRayFileHeader)));
ASSERT_THAT(Records, SizeIs(2));
EXPECT_THAT(Records[1]->getRecordType(), Eq(R->getRecordType()));
}
REGISTER_TYPED_TEST_CASE_P(RoundTripTestV5, RoundTripsSingleValue);
// These are the record types we support for v4 and below.
using RecordTypes =
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
::testing::Types<NewBufferRecord, NewCPUIDRecord, TSCWrapRecord,
WallclockRecord, CustomEventRecord, CallArgRecord,
PIDRecord, FunctionRecord>;
INSTANTIATE_TYPED_TEST_CASE_P(Records, RoundTripTest, RecordTypes);
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
// For V5, we have two new types we're supporting.
using RecordTypesV5 =
::testing::Types<NewBufferRecord, NewCPUIDRecord, TSCWrapRecord,
WallclockRecord, CustomEventRecordV5, TypedEventRecord,
CallArgRecord, PIDRecord, FunctionRecord>;
INSTANTIATE_TYPED_TEST_CASE_P(Records, RoundTripTestV5, RecordTypesV5);
} // namespace
} // namespace xray
} // namespace llvm