[XRay] FDRTraceWriter and FDR Trace Loading
Summary:
This is the first step in the larger refactoring and reduction of
D50441.
This step in the process does the following:
- Introduces more granular types of `Record`s representing the many
kinds of records written/read by the Flight Data Recorder (FDR) mode
`Trace` loading function(s).
- Introduces an abstract `RecordVisitor` type meant to handle the
processing of the various `Record` derived types. This `RecordVisitor`
has two implementations in this patch: `RecordInitializer` and
`FDRTraceWriter`.
- We also introduce a convenience interface for building a collection of
`Record` instances called a `LogBuilder`. This allows us to generate
sequences of `Record` instances manually (used in unit tests but
useful otherwise).
- The`FDRTraceWriter` class implements the `RecordVisitor` interface and
handles the writing of metadata records to a `raw_ostream`. We
demonstrate that in the unit test, we can generate in-memory FDR mode
traces using the specific `Record` derived types, which we load
through the `loadTrace(...)` function yielding valid `Trace` objects.
This patch introduces the required types and concepts for us to start
replacing the logic implemented in the `loadFDRLog` function to use the
more granular types. In subsequent patches, we will introduce more
visitor implementations which isolate the verification, printing,
indexing, production/consumption, and finally the conversion of the FDR
mode logs.
The overarching goal of these changes is to make handling FDR mode logs
better tested, more understandable, more extensible, and more
systematic. This will also allow us to better represent the execution
trace, as we improve the fidelity of the events we represent in an XRay
`Trace` object, which we intend to do after FDR mode log processing is
in better shape.
Reviewers: eizan
Reviewed By: eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51210
llvm-svn: 341029
2018-08-30 15:22:21 +08:00
|
|
|
//===- FDRTraceWriter.cpp - XRay FDR Trace Writer ---------------*- C++ -*-===//
|
|
|
|
//
|
|
|
|
// The LLVM Compiler Infrastructure
|
|
|
|
//
|
|
|
|
// This file is distributed under the University of Illinois Open Source
|
|
|
|
// License. See LICENSE.TXT for details.
|
|
|
|
//
|
|
|
|
//===----------------------------------------------------------------------===//
|
|
|
|
//
|
|
|
|
// Test a utility that can write out XRay FDR Mode formatted trace files.
|
|
|
|
//
|
|
|
|
//===----------------------------------------------------------------------===//
|
|
|
|
#include "llvm/XRay/FDRTraceWriter.h"
|
|
|
|
#include <tuple>
|
|
|
|
|
|
|
|
namespace llvm {
|
|
|
|
namespace xray {
|
|
|
|
|
|
|
|
namespace {
|
|
|
|
|
2018-09-01 00:08:38 +08:00
|
|
|
template <size_t Index> struct IndexedWriter {
|
[XRay] FDRTraceWriter and FDR Trace Loading
Summary:
This is the first step in the larger refactoring and reduction of
D50441.
This step in the process does the following:
- Introduces more granular types of `Record`s representing the many
kinds of records written/read by the Flight Data Recorder (FDR) mode
`Trace` loading function(s).
- Introduces an abstract `RecordVisitor` type meant to handle the
processing of the various `Record` derived types. This `RecordVisitor`
has two implementations in this patch: `RecordInitializer` and
`FDRTraceWriter`.
- We also introduce a convenience interface for building a collection of
`Record` instances called a `LogBuilder`. This allows us to generate
sequences of `Record` instances manually (used in unit tests but
useful otherwise).
- The`FDRTraceWriter` class implements the `RecordVisitor` interface and
handles the writing of metadata records to a `raw_ostream`. We
demonstrate that in the unit test, we can generate in-memory FDR mode
traces using the specific `Record` derived types, which we load
through the `loadTrace(...)` function yielding valid `Trace` objects.
This patch introduces the required types and concepts for us to start
replacing the logic implemented in the `loadFDRLog` function to use the
more granular types. In subsequent patches, we will introduce more
visitor implementations which isolate the verification, printing,
indexing, production/consumption, and finally the conversion of the FDR
mode logs.
The overarching goal of these changes is to make handling FDR mode logs
better tested, more understandable, more extensible, and more
systematic. This will also allow us to better represent the execution
trace, as we improve the fidelity of the events we represent in an XRay
`Trace` object, which we intend to do after FDR mode log processing is
in better shape.
Reviewers: eizan
Reviewed By: eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51210
llvm-svn: 341029
2018-08-30 15:22:21 +08:00
|
|
|
template <
|
|
|
|
class Tuple,
|
|
|
|
typename std::enable_if<
|
|
|
|
(Index <
|
|
|
|
std::tuple_size<typename std::remove_reference<Tuple>::type>::value),
|
|
|
|
int>::type = 0>
|
2018-09-01 00:08:38 +08:00
|
|
|
static size_t write(support::endian::Writer &OS, Tuple &&T) {
|
|
|
|
OS.write(std::get<Index>(T));
|
|
|
|
return sizeof(std::get<Index>(T)) + IndexedWriter<Index + 1>::write(OS, T);
|
[XRay] FDRTraceWriter and FDR Trace Loading
Summary:
This is the first step in the larger refactoring and reduction of
D50441.
This step in the process does the following:
- Introduces more granular types of `Record`s representing the many
kinds of records written/read by the Flight Data Recorder (FDR) mode
`Trace` loading function(s).
- Introduces an abstract `RecordVisitor` type meant to handle the
processing of the various `Record` derived types. This `RecordVisitor`
has two implementations in this patch: `RecordInitializer` and
`FDRTraceWriter`.
- We also introduce a convenience interface for building a collection of
`Record` instances called a `LogBuilder`. This allows us to generate
sequences of `Record` instances manually (used in unit tests but
useful otherwise).
- The`FDRTraceWriter` class implements the `RecordVisitor` interface and
handles the writing of metadata records to a `raw_ostream`. We
demonstrate that in the unit test, we can generate in-memory FDR mode
traces using the specific `Record` derived types, which we load
through the `loadTrace(...)` function yielding valid `Trace` objects.
This patch introduces the required types and concepts for us to start
replacing the logic implemented in the `loadFDRLog` function to use the
more granular types. In subsequent patches, we will introduce more
visitor implementations which isolate the verification, printing,
indexing, production/consumption, and finally the conversion of the FDR
mode logs.
The overarching goal of these changes is to make handling FDR mode logs
better tested, more understandable, more extensible, and more
systematic. This will also allow us to better represent the execution
trace, as we improve the fidelity of the events we represent in an XRay
`Trace` object, which we intend to do after FDR mode log processing is
in better shape.
Reviewers: eizan
Reviewed By: eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51210
llvm-svn: 341029
2018-08-30 15:22:21 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
template <
|
|
|
|
class Tuple,
|
|
|
|
typename std::enable_if<
|
|
|
|
(Index >=
|
|
|
|
std::tuple_size<typename std::remove_reference<Tuple>::type>::value),
|
|
|
|
int>::type = 0>
|
2018-09-01 00:08:38 +08:00
|
|
|
static size_t write(support::endian::Writer &OS, Tuple &&) {
|
|
|
|
return 0;
|
|
|
|
}
|
[XRay] FDRTraceWriter and FDR Trace Loading
Summary:
This is the first step in the larger refactoring and reduction of
D50441.
This step in the process does the following:
- Introduces more granular types of `Record`s representing the many
kinds of records written/read by the Flight Data Recorder (FDR) mode
`Trace` loading function(s).
- Introduces an abstract `RecordVisitor` type meant to handle the
processing of the various `Record` derived types. This `RecordVisitor`
has two implementations in this patch: `RecordInitializer` and
`FDRTraceWriter`.
- We also introduce a convenience interface for building a collection of
`Record` instances called a `LogBuilder`. This allows us to generate
sequences of `Record` instances manually (used in unit tests but
useful otherwise).
- The`FDRTraceWriter` class implements the `RecordVisitor` interface and
handles the writing of metadata records to a `raw_ostream`. We
demonstrate that in the unit test, we can generate in-memory FDR mode
traces using the specific `Record` derived types, which we load
through the `loadTrace(...)` function yielding valid `Trace` objects.
This patch introduces the required types and concepts for us to start
replacing the logic implemented in the `loadFDRLog` function to use the
more granular types. In subsequent patches, we will introduce more
visitor implementations which isolate the verification, printing,
indexing, production/consumption, and finally the conversion of the FDR
mode logs.
The overarching goal of these changes is to make handling FDR mode logs
better tested, more understandable, more extensible, and more
systematic. This will also allow us to better represent the execution
trace, as we improve the fidelity of the events we represent in an XRay
`Trace` object, which we intend to do after FDR mode log processing is
in better shape.
Reviewers: eizan
Reviewed By: eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51210
llvm-svn: 341029
2018-08-30 15:22:21 +08:00
|
|
|
};
|
|
|
|
|
2018-08-30 17:04:12 +08:00
|
|
|
template <uint8_t Kind, class... Values>
|
2018-09-01 00:08:38 +08:00
|
|
|
Error writeMetadata(support::endian::Writer &OS, Values &&... Ds) {
|
[XRay] Improve FDR trace handling and error messaging
Summary:
This change covers a number of things spanning LLVM and compiler-rt,
which are related in a non-trivial way.
In LLVM, we have a library that handles the FDR mode even log loading,
which uses C++'s runtime polymorphism feature to better faithfully
represent the events that are written down by the FDR mode runtime. We
do this by interpreting a trace that's serliased in a common format
agreed upon by both the trace loading library and the FDR mode runtime.
This library is under active development, which consists of features
allowing us to reconstitute a higher-level event log.
This event log is used by the conversion and visualisation tools we have
for interpreting XRay traces.
One of the tools we have is a diagnostic tool in llvm-xray called
`fdr-dump` which we've been using to debug our expectations of what the
FDR runtime should be writing and what the logical FDR event log
structures are. We use this fairly extensively to reason about why some
non-trivial traces we're generating with FDR mode runtimes fail to
convert or fail to parse correctly.
One of these failures we've found in manual debugging of some of the
traces we've seen involve an inconsistency between the buffer extents (a
record indicating how many bytes to follow are part of a logical
thread's event log) and the record of the bytes written into the log --
sometimes it turns out the data could be garbage, due to buffers being
recycled, but sometimes we're seeing the buffer extent indicating a log
is "shorter" than the actual records associated with the buffer. This
case happens particularly with function entry records with a call
argument.
This change for now updates the FDR mode runtime to write the bytes for
the function call and arg record before updating the buffer extents
atomically, allowing multiple threads to see a consistent view of the
data in the buffer using the atomic counter associated with a buffer.
What we're trying to prevent here is partial updates where we see the
intermediary updates to the buffer extents (function record size then
call argument record size) becoming observable from another thread, for
instance, one doing the serialization/flushing.
To do both diagnose this issue properly, we need to be able to honour
the extents being set in the `BufferExtents` records marking the
beginning of the logical buffers when reading an FDR trace. Since LLVM
doesn't use C++'s RTTI mechanism, we instead follow the advice in the
documentation for LLVM Style RTTI
(https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on
this RTTI feature to ensure that our file-based record producer (our
streaming "deserializer") can honour the extents of individual buffers
as we interpret traces.
This also sets us up to be able to eventually do smart
skipping/continuation of FDR logs, seeking instead to find BufferExtents
records in cases where we find potentially recoverable errors. In the
meantime, we make this change to operate in a strict mode when reading
logical buffers with extent records.
Reviewers: mboerger
Subscribers: hiraditya, llvm-commits, jfb
Differential Revision: https://reviews.llvm.org/D54201
llvm-svn: 346473
2018-11-09 14:26:48 +08:00
|
|
|
// The first bit in the first byte of metadata records is always set to 1, so
|
|
|
|
// we ensure this is the case when we write out the first byte of the record.
|
|
|
|
uint8_t FirstByte = (static_cast<uint8_t>(Kind) << 1) | uint8_t{0x01u};
|
2018-08-30 17:04:12 +08:00
|
|
|
auto T = std::make_tuple(std::forward<Values>(std::move(Ds))...);
|
2018-09-01 00:08:38 +08:00
|
|
|
// Write in field order.
|
|
|
|
OS.write(FirstByte);
|
|
|
|
auto Bytes = IndexedWriter<0>::write(OS, T);
|
|
|
|
assert(Bytes <= 15 && "Must only ever write at most 16 byte metadata!");
|
|
|
|
// Pad out with appropriate numbers of zero's.
|
|
|
|
for (; Bytes < 15; ++Bytes)
|
|
|
|
OS.write('\0');
|
[XRay] FDRTraceWriter and FDR Trace Loading
Summary:
This is the first step in the larger refactoring and reduction of
D50441.
This step in the process does the following:
- Introduces more granular types of `Record`s representing the many
kinds of records written/read by the Flight Data Recorder (FDR) mode
`Trace` loading function(s).
- Introduces an abstract `RecordVisitor` type meant to handle the
processing of the various `Record` derived types. This `RecordVisitor`
has two implementations in this patch: `RecordInitializer` and
`FDRTraceWriter`.
- We also introduce a convenience interface for building a collection of
`Record` instances called a `LogBuilder`. This allows us to generate
sequences of `Record` instances manually (used in unit tests but
useful otherwise).
- The`FDRTraceWriter` class implements the `RecordVisitor` interface and
handles the writing of metadata records to a `raw_ostream`. We
demonstrate that in the unit test, we can generate in-memory FDR mode
traces using the specific `Record` derived types, which we load
through the `loadTrace(...)` function yielding valid `Trace` objects.
This patch introduces the required types and concepts for us to start
replacing the logic implemented in the `loadFDRLog` function to use the
more granular types. In subsequent patches, we will introduce more
visitor implementations which isolate the verification, printing,
indexing, production/consumption, and finally the conversion of the FDR
mode logs.
The overarching goal of these changes is to make handling FDR mode logs
better tested, more understandable, more extensible, and more
systematic. This will also allow us to better represent the execution
trace, as we improve the fidelity of the events we represent in an XRay
`Trace` object, which we intend to do after FDR mode log processing is
in better shape.
Reviewers: eizan
Reviewed By: eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51210
llvm-svn: 341029
2018-08-30 15:22:21 +08:00
|
|
|
return Error::success();
|
|
|
|
}
|
|
|
|
|
|
|
|
} // namespace
|
|
|
|
|
|
|
|
FDRTraceWriter::FDRTraceWriter(raw_ostream &O, const XRayFileHeader &H)
|
2018-09-01 00:08:38 +08:00
|
|
|
: OS(O, support::endianness::native) {
|
[XRay] FDRTraceWriter and FDR Trace Loading
Summary:
This is the first step in the larger refactoring and reduction of
D50441.
This step in the process does the following:
- Introduces more granular types of `Record`s representing the many
kinds of records written/read by the Flight Data Recorder (FDR) mode
`Trace` loading function(s).
- Introduces an abstract `RecordVisitor` type meant to handle the
processing of the various `Record` derived types. This `RecordVisitor`
has two implementations in this patch: `RecordInitializer` and
`FDRTraceWriter`.
- We also introduce a convenience interface for building a collection of
`Record` instances called a `LogBuilder`. This allows us to generate
sequences of `Record` instances manually (used in unit tests but
useful otherwise).
- The`FDRTraceWriter` class implements the `RecordVisitor` interface and
handles the writing of metadata records to a `raw_ostream`. We
demonstrate that in the unit test, we can generate in-memory FDR mode
traces using the specific `Record` derived types, which we load
through the `loadTrace(...)` function yielding valid `Trace` objects.
This patch introduces the required types and concepts for us to start
replacing the logic implemented in the `loadFDRLog` function to use the
more granular types. In subsequent patches, we will introduce more
visitor implementations which isolate the verification, printing,
indexing, production/consumption, and finally the conversion of the FDR
mode logs.
The overarching goal of these changes is to make handling FDR mode logs
better tested, more understandable, more extensible, and more
systematic. This will also allow us to better represent the execution
trace, as we improve the fidelity of the events we represent in an XRay
`Trace` object, which we intend to do after FDR mode log processing is
in better shape.
Reviewers: eizan
Reviewed By: eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51210
llvm-svn: 341029
2018-08-30 15:22:21 +08:00
|
|
|
// We need to re-construct a header, by writing the fields we care about for
|
|
|
|
// traces, in the format that the runtime would have written.
|
2018-09-01 00:08:38 +08:00
|
|
|
uint32_t BitField =
|
|
|
|
(H.ConstantTSC ? 0x01 : 0x0) | (H.NonstopTSC ? 0x02 : 0x0);
|
|
|
|
|
|
|
|
// For endian-correctness, we need to write these fields in the order they
|
|
|
|
// appear and that we expect, instead of blasting bytes of the struct through.
|
|
|
|
OS.write(H.Version);
|
|
|
|
OS.write(H.Type);
|
|
|
|
OS.write(BitField);
|
|
|
|
OS.write(H.CycleFrequency);
|
|
|
|
ArrayRef<char> FreeFormBytes(H.FreeFormData,
|
|
|
|
sizeof(XRayFileHeader::FreeFormData));
|
|
|
|
OS.write(FreeFormBytes);
|
[XRay] FDRTraceWriter and FDR Trace Loading
Summary:
This is the first step in the larger refactoring and reduction of
D50441.
This step in the process does the following:
- Introduces more granular types of `Record`s representing the many
kinds of records written/read by the Flight Data Recorder (FDR) mode
`Trace` loading function(s).
- Introduces an abstract `RecordVisitor` type meant to handle the
processing of the various `Record` derived types. This `RecordVisitor`
has two implementations in this patch: `RecordInitializer` and
`FDRTraceWriter`.
- We also introduce a convenience interface for building a collection of
`Record` instances called a `LogBuilder`. This allows us to generate
sequences of `Record` instances manually (used in unit tests but
useful otherwise).
- The`FDRTraceWriter` class implements the `RecordVisitor` interface and
handles the writing of metadata records to a `raw_ostream`. We
demonstrate that in the unit test, we can generate in-memory FDR mode
traces using the specific `Record` derived types, which we load
through the `loadTrace(...)` function yielding valid `Trace` objects.
This patch introduces the required types and concepts for us to start
replacing the logic implemented in the `loadFDRLog` function to use the
more granular types. In subsequent patches, we will introduce more
visitor implementations which isolate the verification, printing,
indexing, production/consumption, and finally the conversion of the FDR
mode logs.
The overarching goal of these changes is to make handling FDR mode logs
better tested, more understandable, more extensible, and more
systematic. This will also allow us to better represent the execution
trace, as we improve the fidelity of the events we represent in an XRay
`Trace` object, which we intend to do after FDR mode log processing is
in better shape.
Reviewers: eizan
Reviewed By: eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51210
llvm-svn: 341029
2018-08-30 15:22:21 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
FDRTraceWriter::~FDRTraceWriter() {}
|
|
|
|
|
|
|
|
Error FDRTraceWriter::visit(BufferExtents &R) {
|
|
|
|
return writeMetadata<7u>(OS, R.size());
|
|
|
|
}
|
|
|
|
|
|
|
|
Error FDRTraceWriter::visit(WallclockRecord &R) {
|
|
|
|
return writeMetadata<4u>(OS, R.seconds(), R.nanos());
|
|
|
|
}
|
|
|
|
|
|
|
|
Error FDRTraceWriter::visit(NewCPUIDRecord &R) {
|
2018-09-11 15:27:59 +08:00
|
|
|
return writeMetadata<2u>(OS, R.cpuid(), R.tsc());
|
[XRay] FDRTraceWriter and FDR Trace Loading
Summary:
This is the first step in the larger refactoring and reduction of
D50441.
This step in the process does the following:
- Introduces more granular types of `Record`s representing the many
kinds of records written/read by the Flight Data Recorder (FDR) mode
`Trace` loading function(s).
- Introduces an abstract `RecordVisitor` type meant to handle the
processing of the various `Record` derived types. This `RecordVisitor`
has two implementations in this patch: `RecordInitializer` and
`FDRTraceWriter`.
- We also introduce a convenience interface for building a collection of
`Record` instances called a `LogBuilder`. This allows us to generate
sequences of `Record` instances manually (used in unit tests but
useful otherwise).
- The`FDRTraceWriter` class implements the `RecordVisitor` interface and
handles the writing of metadata records to a `raw_ostream`. We
demonstrate that in the unit test, we can generate in-memory FDR mode
traces using the specific `Record` derived types, which we load
through the `loadTrace(...)` function yielding valid `Trace` objects.
This patch introduces the required types and concepts for us to start
replacing the logic implemented in the `loadFDRLog` function to use the
more granular types. In subsequent patches, we will introduce more
visitor implementations which isolate the verification, printing,
indexing, production/consumption, and finally the conversion of the FDR
mode logs.
The overarching goal of these changes is to make handling FDR mode logs
better tested, more understandable, more extensible, and more
systematic. This will also allow us to better represent the execution
trace, as we improve the fidelity of the events we represent in an XRay
`Trace` object, which we intend to do after FDR mode log processing is
in better shape.
Reviewers: eizan
Reviewed By: eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51210
llvm-svn: 341029
2018-08-30 15:22:21 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
Error FDRTraceWriter::visit(TSCWrapRecord &R) {
|
|
|
|
return writeMetadata<3u>(OS, R.tsc());
|
|
|
|
}
|
|
|
|
|
|
|
|
Error FDRTraceWriter::visit(CustomEventRecord &R) {
|
2018-11-01 08:18:52 +08:00
|
|
|
if (auto E = writeMetadata<5u>(OS, R.size(), R.tsc(), R.cpu()))
|
[XRay] FDRTraceWriter and FDR Trace Loading
Summary:
This is the first step in the larger refactoring and reduction of
D50441.
This step in the process does the following:
- Introduces more granular types of `Record`s representing the many
kinds of records written/read by the Flight Data Recorder (FDR) mode
`Trace` loading function(s).
- Introduces an abstract `RecordVisitor` type meant to handle the
processing of the various `Record` derived types. This `RecordVisitor`
has two implementations in this patch: `RecordInitializer` and
`FDRTraceWriter`.
- We also introduce a convenience interface for building a collection of
`Record` instances called a `LogBuilder`. This allows us to generate
sequences of `Record` instances manually (used in unit tests but
useful otherwise).
- The`FDRTraceWriter` class implements the `RecordVisitor` interface and
handles the writing of metadata records to a `raw_ostream`. We
demonstrate that in the unit test, we can generate in-memory FDR mode
traces using the specific `Record` derived types, which we load
through the `loadTrace(...)` function yielding valid `Trace` objects.
This patch introduces the required types and concepts for us to start
replacing the logic implemented in the `loadFDRLog` function to use the
more granular types. In subsequent patches, we will introduce more
visitor implementations which isolate the verification, printing,
indexing, production/consumption, and finally the conversion of the FDR
mode logs.
The overarching goal of these changes is to make handling FDR mode logs
better tested, more understandable, more extensible, and more
systematic. This will also allow us to better represent the execution
trace, as we improve the fidelity of the events we represent in an XRay
`Trace` object, which we intend to do after FDR mode log processing is
in better shape.
Reviewers: eizan
Reviewed By: eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51210
llvm-svn: 341029
2018-08-30 15:22:21 +08:00
|
|
|
return E;
|
2018-11-01 08:18:52 +08:00
|
|
|
auto D = R.data();
|
|
|
|
ArrayRef<char> Bytes(D.data(), D.size());
|
2018-09-01 00:08:38 +08:00
|
|
|
OS.write(Bytes);
|
[XRay] FDRTraceWriter and FDR Trace Loading
Summary:
This is the first step in the larger refactoring and reduction of
D50441.
This step in the process does the following:
- Introduces more granular types of `Record`s representing the many
kinds of records written/read by the Flight Data Recorder (FDR) mode
`Trace` loading function(s).
- Introduces an abstract `RecordVisitor` type meant to handle the
processing of the various `Record` derived types. This `RecordVisitor`
has two implementations in this patch: `RecordInitializer` and
`FDRTraceWriter`.
- We also introduce a convenience interface for building a collection of
`Record` instances called a `LogBuilder`. This allows us to generate
sequences of `Record` instances manually (used in unit tests but
useful otherwise).
- The`FDRTraceWriter` class implements the `RecordVisitor` interface and
handles the writing of metadata records to a `raw_ostream`. We
demonstrate that in the unit test, we can generate in-memory FDR mode
traces using the specific `Record` derived types, which we load
through the `loadTrace(...)` function yielding valid `Trace` objects.
This patch introduces the required types and concepts for us to start
replacing the logic implemented in the `loadFDRLog` function to use the
more granular types. In subsequent patches, we will introduce more
visitor implementations which isolate the verification, printing,
indexing, production/consumption, and finally the conversion of the FDR
mode logs.
The overarching goal of these changes is to make handling FDR mode logs
better tested, more understandable, more extensible, and more
systematic. This will also allow us to better represent the execution
trace, as we improve the fidelity of the events we represent in an XRay
`Trace` object, which we intend to do after FDR mode log processing is
in better shape.
Reviewers: eizan
Reviewed By: eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51210
llvm-svn: 341029
2018-08-30 15:22:21 +08:00
|
|
|
return Error::success();
|
|
|
|
}
|
|
|
|
|
2018-11-07 12:37:42 +08:00
|
|
|
Error FDRTraceWriter::visit(CustomEventRecordV5 &R) {
|
|
|
|
if (auto E = writeMetadata<5u>(OS, R.size(), R.delta()))
|
|
|
|
return E;
|
|
|
|
auto D = R.data();
|
|
|
|
ArrayRef<char> Bytes(D.data(), D.size());
|
|
|
|
OS.write(Bytes);
|
|
|
|
return Error::success();
|
|
|
|
}
|
|
|
|
|
|
|
|
Error FDRTraceWriter::visit(TypedEventRecord &R) {
|
[XRay] Improve FDR trace handling and error messaging
Summary:
This change covers a number of things spanning LLVM and compiler-rt,
which are related in a non-trivial way.
In LLVM, we have a library that handles the FDR mode even log loading,
which uses C++'s runtime polymorphism feature to better faithfully
represent the events that are written down by the FDR mode runtime. We
do this by interpreting a trace that's serliased in a common format
agreed upon by both the trace loading library and the FDR mode runtime.
This library is under active development, which consists of features
allowing us to reconstitute a higher-level event log.
This event log is used by the conversion and visualisation tools we have
for interpreting XRay traces.
One of the tools we have is a diagnostic tool in llvm-xray called
`fdr-dump` which we've been using to debug our expectations of what the
FDR runtime should be writing and what the logical FDR event log
structures are. We use this fairly extensively to reason about why some
non-trivial traces we're generating with FDR mode runtimes fail to
convert or fail to parse correctly.
One of these failures we've found in manual debugging of some of the
traces we've seen involve an inconsistency between the buffer extents (a
record indicating how many bytes to follow are part of a logical
thread's event log) and the record of the bytes written into the log --
sometimes it turns out the data could be garbage, due to buffers being
recycled, but sometimes we're seeing the buffer extent indicating a log
is "shorter" than the actual records associated with the buffer. This
case happens particularly with function entry records with a call
argument.
This change for now updates the FDR mode runtime to write the bytes for
the function call and arg record before updating the buffer extents
atomically, allowing multiple threads to see a consistent view of the
data in the buffer using the atomic counter associated with a buffer.
What we're trying to prevent here is partial updates where we see the
intermediary updates to the buffer extents (function record size then
call argument record size) becoming observable from another thread, for
instance, one doing the serialization/flushing.
To do both diagnose this issue properly, we need to be able to honour
the extents being set in the `BufferExtents` records marking the
beginning of the logical buffers when reading an FDR trace. Since LLVM
doesn't use C++'s RTTI mechanism, we instead follow the advice in the
documentation for LLVM Style RTTI
(https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on
this RTTI feature to ensure that our file-based record producer (our
streaming "deserializer") can honour the extents of individual buffers
as we interpret traces.
This also sets us up to be able to eventually do smart
skipping/continuation of FDR logs, seeking instead to find BufferExtents
records in cases where we find potentially recoverable errors. In the
meantime, we make this change to operate in a strict mode when reading
logical buffers with extent records.
Reviewers: mboerger
Subscribers: hiraditya, llvm-commits, jfb
Differential Revision: https://reviews.llvm.org/D54201
llvm-svn: 346473
2018-11-09 14:26:48 +08:00
|
|
|
if (auto E = writeMetadata<8u>(OS, R.size(), R.delta(), R.eventType()))
|
2018-11-07 12:37:42 +08:00
|
|
|
return E;
|
|
|
|
auto D = R.data();
|
|
|
|
ArrayRef<char> Bytes(D.data(), D.size());
|
|
|
|
OS.write(Bytes);
|
|
|
|
return Error::success();
|
|
|
|
}
|
|
|
|
|
[XRay] FDRTraceWriter and FDR Trace Loading
Summary:
This is the first step in the larger refactoring and reduction of
D50441.
This step in the process does the following:
- Introduces more granular types of `Record`s representing the many
kinds of records written/read by the Flight Data Recorder (FDR) mode
`Trace` loading function(s).
- Introduces an abstract `RecordVisitor` type meant to handle the
processing of the various `Record` derived types. This `RecordVisitor`
has two implementations in this patch: `RecordInitializer` and
`FDRTraceWriter`.
- We also introduce a convenience interface for building a collection of
`Record` instances called a `LogBuilder`. This allows us to generate
sequences of `Record` instances manually (used in unit tests but
useful otherwise).
- The`FDRTraceWriter` class implements the `RecordVisitor` interface and
handles the writing of metadata records to a `raw_ostream`. We
demonstrate that in the unit test, we can generate in-memory FDR mode
traces using the specific `Record` derived types, which we load
through the `loadTrace(...)` function yielding valid `Trace` objects.
This patch introduces the required types and concepts for us to start
replacing the logic implemented in the `loadFDRLog` function to use the
more granular types. In subsequent patches, we will introduce more
visitor implementations which isolate the verification, printing,
indexing, production/consumption, and finally the conversion of the FDR
mode logs.
The overarching goal of these changes is to make handling FDR mode logs
better tested, more understandable, more extensible, and more
systematic. This will also allow us to better represent the execution
trace, as we improve the fidelity of the events we represent in an XRay
`Trace` object, which we intend to do after FDR mode log processing is
in better shape.
Reviewers: eizan
Reviewed By: eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51210
llvm-svn: 341029
2018-08-30 15:22:21 +08:00
|
|
|
Error FDRTraceWriter::visit(CallArgRecord &R) {
|
|
|
|
return writeMetadata<6u>(OS, R.arg());
|
|
|
|
}
|
|
|
|
|
|
|
|
Error FDRTraceWriter::visit(PIDRecord &R) {
|
|
|
|
return writeMetadata<9u>(OS, R.pid());
|
|
|
|
}
|
|
|
|
|
|
|
|
Error FDRTraceWriter::visit(NewBufferRecord &R) {
|
|
|
|
return writeMetadata<0u>(OS, R.tid());
|
|
|
|
}
|
|
|
|
|
|
|
|
Error FDRTraceWriter::visit(EndBufferRecord &R) {
|
|
|
|
return writeMetadata<1u>(OS, 0);
|
|
|
|
}
|
|
|
|
|
|
|
|
Error FDRTraceWriter::visit(FunctionRecord &R) {
|
2018-09-01 01:49:59 +08:00
|
|
|
// Write out the data in "field" order, to be endian-aware.
|
|
|
|
uint32_t TypeRecordFuncId = uint32_t{R.functionId() & ~uint32_t{0x0Fu << 28}};
|
|
|
|
TypeRecordFuncId <<= 3;
|
|
|
|
TypeRecordFuncId |= static_cast<uint32_t>(R.recordType());
|
|
|
|
TypeRecordFuncId <<= 1;
|
|
|
|
TypeRecordFuncId &= ~uint32_t{0x01};
|
|
|
|
OS.write(TypeRecordFuncId);
|
|
|
|
OS.write(R.delta());
|
[XRay] FDRTraceWriter and FDR Trace Loading
Summary:
This is the first step in the larger refactoring and reduction of
D50441.
This step in the process does the following:
- Introduces more granular types of `Record`s representing the many
kinds of records written/read by the Flight Data Recorder (FDR) mode
`Trace` loading function(s).
- Introduces an abstract `RecordVisitor` type meant to handle the
processing of the various `Record` derived types. This `RecordVisitor`
has two implementations in this patch: `RecordInitializer` and
`FDRTraceWriter`.
- We also introduce a convenience interface for building a collection of
`Record` instances called a `LogBuilder`. This allows us to generate
sequences of `Record` instances manually (used in unit tests but
useful otherwise).
- The`FDRTraceWriter` class implements the `RecordVisitor` interface and
handles the writing of metadata records to a `raw_ostream`. We
demonstrate that in the unit test, we can generate in-memory FDR mode
traces using the specific `Record` derived types, which we load
through the `loadTrace(...)` function yielding valid `Trace` objects.
This patch introduces the required types and concepts for us to start
replacing the logic implemented in the `loadFDRLog` function to use the
more granular types. In subsequent patches, we will introduce more
visitor implementations which isolate the verification, printing,
indexing, production/consumption, and finally the conversion of the FDR
mode logs.
The overarching goal of these changes is to make handling FDR mode logs
better tested, more understandable, more extensible, and more
systematic. This will also allow us to better represent the execution
trace, as we improve the fidelity of the events we represent in an XRay
`Trace` object, which we intend to do after FDR mode log processing is
in better shape.
Reviewers: eizan
Reviewed By: eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51210
llvm-svn: 341029
2018-08-30 15:22:21 +08:00
|
|
|
return Error::success();
|
2018-11-01 08:18:52 +08:00
|
|
|
}
|
[XRay] FDRTraceWriter and FDR Trace Loading
Summary:
This is the first step in the larger refactoring and reduction of
D50441.
This step in the process does the following:
- Introduces more granular types of `Record`s representing the many
kinds of records written/read by the Flight Data Recorder (FDR) mode
`Trace` loading function(s).
- Introduces an abstract `RecordVisitor` type meant to handle the
processing of the various `Record` derived types. This `RecordVisitor`
has two implementations in this patch: `RecordInitializer` and
`FDRTraceWriter`.
- We also introduce a convenience interface for building a collection of
`Record` instances called a `LogBuilder`. This allows us to generate
sequences of `Record` instances manually (used in unit tests but
useful otherwise).
- The`FDRTraceWriter` class implements the `RecordVisitor` interface and
handles the writing of metadata records to a `raw_ostream`. We
demonstrate that in the unit test, we can generate in-memory FDR mode
traces using the specific `Record` derived types, which we load
through the `loadTrace(...)` function yielding valid `Trace` objects.
This patch introduces the required types and concepts for us to start
replacing the logic implemented in the `loadFDRLog` function to use the
more granular types. In subsequent patches, we will introduce more
visitor implementations which isolate the verification, printing,
indexing, production/consumption, and finally the conversion of the FDR
mode logs.
The overarching goal of these changes is to make handling FDR mode logs
better tested, more understandable, more extensible, and more
systematic. This will also allow us to better represent the execution
trace, as we improve the fidelity of the events we represent in an XRay
`Trace` object, which we intend to do after FDR mode log processing is
in better shape.
Reviewers: eizan
Reviewed By: eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51210
llvm-svn: 341029
2018-08-30 15:22:21 +08:00
|
|
|
|
|
|
|
} // namespace xray
|
|
|
|
} // namespace llvm
|