Summary:
This change covers a number of things spanning LLVM and compiler-rt,
which are related in a non-trivial way.
In LLVM, we have a library that handles the FDR mode even log loading,
which uses C++'s runtime polymorphism feature to better faithfully
represent the events that are written down by the FDR mode runtime. We
do this by interpreting a trace that's serliased in a common format
agreed upon by both the trace loading library and the FDR mode runtime.
This library is under active development, which consists of features
allowing us to reconstitute a higher-level event log.
This event log is used by the conversion and visualisation tools we have
for interpreting XRay traces.
One of the tools we have is a diagnostic tool in llvm-xray called
`fdr-dump` which we've been using to debug our expectations of what the
FDR runtime should be writing and what the logical FDR event log
structures are. We use this fairly extensively to reason about why some
non-trivial traces we're generating with FDR mode runtimes fail to
convert or fail to parse correctly.
One of these failures we've found in manual debugging of some of the
traces we've seen involve an inconsistency between the buffer extents (a
record indicating how many bytes to follow are part of a logical
thread's event log) and the record of the bytes written into the log --
sometimes it turns out the data could be garbage, due to buffers being
recycled, but sometimes we're seeing the buffer extent indicating a log
is "shorter" than the actual records associated with the buffer. This
case happens particularly with function entry records with a call
argument.
This change for now updates the FDR mode runtime to write the bytes for
the function call and arg record before updating the buffer extents
atomically, allowing multiple threads to see a consistent view of the
data in the buffer using the atomic counter associated with a buffer.
What we're trying to prevent here is partial updates where we see the
intermediary updates to the buffer extents (function record size then
call argument record size) becoming observable from another thread, for
instance, one doing the serialization/flushing.
To do both diagnose this issue properly, we need to be able to honour
the extents being set in the `BufferExtents` records marking the
beginning of the logical buffers when reading an FDR trace. Since LLVM
doesn't use C++'s RTTI mechanism, we instead follow the advice in the
documentation for LLVM Style RTTI
(https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on
this RTTI feature to ensure that our file-based record producer (our
streaming "deserializer") can honour the extents of individual buffers
as we interpret traces.
This also sets us up to be able to eventually do smart
skipping/continuation of FDR logs, seeking instead to find BufferExtents
records in cases where we find potentially recoverable errors. In the
meantime, we make this change to operate in a strict mode when reading
logical buffers with extent records.
Reviewers: mboerger
Subscribers: hiraditya, llvm-commits, jfb
Differential Revision: https://reviews.llvm.org/D54201
llvm-svn: 346473
Summary:
This change updates the version number for FDR logs to 5, and update the
trace processing to support changes in the custom event records.
In the runtime, since we're already writing down the record preamble to
handle CPU migrations and TSC wraparound, we can use the same TSC delta
encoding in the custom event and typed event records that we use in
function event records. We do the same change to typed events (which
were unsupported before this change in the trace processing) which now
show up in the trace.
Future changes should increase our testing coverage to make custom and
typed events as first class entities in the FDR mode log processing
tools.
This change is also a good example of how we end up supporting new
record types in the FDR mode implementation. This shows the places where
new record types are added and supported.
Depends on D54139.
Reviewers: mboerger
Subscribers: hiraditya, arphaman, jfb, llvm-commits
Differential Revision: https://reviews.llvm.org/D54140
llvm-svn: 346293
Summary:
This change cuts across compiler-rt and llvm, to increment the FDR log
version number to 4, and include the CPU ID in the custom event records.
This is a step towards allowing us to change the `llvm::xray::Trace`
object to start representing both custom and typed events in the stream
of records. Follow-on changes will allow us to change the kinds of
records we're presenting in the stream of traces, to incorporate the
data in custom/typed events.
A follow-on change will handle the typed event case, where it may not
fit within the 15-byte buffer for metadata records.
This work is part of the larger effort to enable writing analysis and
processing tools using a common in-memory representation of the events
found in traces. The work will focus on porting existing tools in LLVM
to use the common representation and informing the design of a
library/framework for expressing trace event analysis as C++ programs.
Reviewers: mboerger, eizan
Subscribers: hiraditya, mgrang, llvm-commits
Differential Revision: https://reviews.llvm.org/D53920
llvm-svn: 345798
Using LLVMTestingSupport in the LLVM_LINK_COMPONENTS breaks the build when
LLVM_TARGETS_TO_BUILD is set to empty.
Libraries that depend on LLVMTestingSupport need to use
target_link_libraries(<target> PRIVATE LLVMTestingSupport) instead.
This required change was already commited by r341899 to fix another build
issue.
This fixes rdar://problem/44615064.
llvm-svn: 342593
Summary:
This more correctly reflects the data written by the FDR mode runtime.
This is a continuation of the work in D50441.
Reviewers: mboerger, eizan
Subscribers: hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51911
llvm-svn: 341905
Summary:
In this change, we implement a `BlockPrinter` which orders records in a
Block that's been indexed by the `BlockIndexer`. This is used in the
`llvm-xray fdr-dump` tool which ties together the various types and
utilities we've been working on, to allow for inspection of XRay FDR
mode traces both with and without verification.
This change is the final step of the refactoring of D50441.
Reviewers: mboerger, eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51846
llvm-svn: 341887
Summary:
This patch implements a `BlockVerifier` type which enforces the
invariants of the log structure of FDR mode logs on a per-block basis.
This ensures that the data we encounter from an FDR mode log
semantically correct (i.e. that records follow the documented "grammar"
for FDR mode log records).
This is another part of the refactoring of D50441.
This is a slightly modified version of rL341628, avoiding the
`std::tuple<...>` constructor that is not constexpr in C++11.
Reviewers: mboerger, eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51723
llvm-svn: 341769
Summary:
This patch implements a `BlockVerifier` type which enforces the
invariants of the log structure of FDR mode logs on a per-block basis.
This ensures that the data we encounter from an FDR mode log
semantically correct (i.e. that records follow the documented "grammar"
for FDR mode log records).
This is another part of the refactoring of D50441.
Reviewers: mboerger, eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51723
llvm-svn: 341628
Summary:
This change adds a `BlockIndexer` type which maintains pointers to
records that belong to the same process+thread pairs. The indexing
happens with order of appearance of records as they are visited.
This version of the indexer currently only supports FDR version 3 logs,
which contain `BufferExtent` records. We will add support for v2 and v1
logs in follow-up patches.
This is another part of D50441.
Reviewers: eizan, kpw, mboerger
Reviewed By: mboerger
Subscribers: mboerger, mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51673
llvm-svn: 341518
We do this instead of using static constexpr char arrays because MSVC
2015 cannot handle the constant initialisation of those along with the
out-of-line storage declaration.
This is a follow-up to D51672.
llvm-svn: 341479
Summary:
This change adds a `RecordPrinter` type which does some basic text
serialization of the FDR record instances. This is one component of the
tool we're building to dump the records from an FDR mode log as-is.
This is a small part of D50441.
Reviewers: eizan, kpw
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51672
llvm-svn: 341447
Previously we've been reading and writing the wrong types which only
worked in little endian implementations. This time we're writing the
same typed values the runtime is using, and reading them appropriately
as well.
llvm-svn: 341241
Before this patch, the FDRTraceWriter would not take endianness into
account when writing data into the output stream.
This is a follow-up to D51289 and D51210.
llvm-svn: 341223
/build/llvm/unittests/XRay/FDRProducerConsumerTest.cpp:90:27: error: declaration of ‘std::unique_ptr<llvm::xray::Record> llvm::xray::{anonymous}::RoundTripTest<T>::Record’ [-fpermissive]
std::unique_ptr<Record> Record;
^~~~~~
In file included from /build/llvm/include/llvm/XRay/FDRLogBuilder.h:12,
from /build/llvm/unittests/XRay/FDRProducerConsumerTest.cpp:15:
/build/llvm/include/llvm/XRay/FDRRecords.h:28:7: error: changes meaning of ‘Record’ from ‘class llvm::xray::Record’ [-fpermissive]
class Record {
^~~~~~
llvm-svn: 341189
Summary:
This patch defines two new base types called `RecordProducer` and
`RecordConsumer` which have default implementations for convenience
(particularly for testing).
A `RecordProducer` implementation has one member function called
`produce()` which serves as a factory constructor for `Record`
instances. This code exercises the `RecordInitializer` code path in the
implementation for `FileBasedRecordProducer`.
A `RecordConsumer` has a single member function called `consume(...)`
which, as the name implies, consumes instances of
`std::unique_ptr<Record>`. We have two implementations, one of which is
used in the test to generate a vector of `std::unique_ptr<Record>`
similar to how the `LogBuilder` implementation works.
We introduce a test in `FDRProducerConsumerTest` which ensures that
records we write through the `FDRTraceWriter` can be loaded by the
`FileBasedRecordProducer`. The record(s) loaded this way are written
again through the `FDRTraceWriter` into a separate string, which we then
compare. This ensures that the read-in bytes to create the `Record`
instances in memory can be replicated when written out through the
`FDRTraceWriter`.
This change depends on D51210 and is part of the refactoring of D50441
into smaller, more focused changes.
Reviewers: eizan, kpw
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51289
llvm-svn: 341180
Summary:
This is the first step in the larger refactoring and reduction of
D50441.
This step in the process does the following:
- Introduces more granular types of `Record`s representing the many
kinds of records written/read by the Flight Data Recorder (FDR) mode
`Trace` loading function(s).
- Introduces an abstract `RecordVisitor` type meant to handle the
processing of the various `Record` derived types. This `RecordVisitor`
has two implementations in this patch: `RecordInitializer` and
`FDRTraceWriter`.
- We also introduce a convenience interface for building a collection of
`Record` instances called a `LogBuilder`. This allows us to generate
sequences of `Record` instances manually (used in unit tests but
useful otherwise).
- The`FDRTraceWriter` class implements the `RecordVisitor` interface and
handles the writing of metadata records to a `raw_ostream`. We
demonstrate that in the unit test, we can generate in-memory FDR mode
traces using the specific `Record` derived types, which we load
through the `loadTrace(...)` function yielding valid `Trace` objects.
This patch introduces the required types and concepts for us to start
replacing the logic implemented in the `loadFDRLog` function to use the
more granular types. In subsequent patches, we will introduce more
visitor implementations which isolate the verification, printing,
indexing, production/consumption, and finally the conversion of the FDR
mode logs.
The overarching goal of these changes is to make handling FDR mode logs
better tested, more understandable, more extensible, and more
systematic. This will also allow us to better represent the execution
trace, as we improve the fidelity of the events we represent in an XRay
`Trace` object, which we intend to do after FDR mode log processing is
in better shape.
Reviewers: eizan
Reviewed By: eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51210
llvm-svn: 341029
Summary:
This change implements the profile loading functionality in LLVM to
support XRay's profiling mode in compiler-rt.
We introduce a type named `llvm::xray::Profile` which allows building a
profile representation. We can load an XRay profile from a file to build
Profile instances, or do it manually through the Profile type's API.
The intent is to get the `llvm-xray` tool to generate `Profile`
instances and use that as the common abstraction through which all
conversion and analysis can be done. In the future we can generate
`Profile` instances from `Trace` instances as well, through conversion
functions.
Some of the key operations supported by the `Profile` API are:
- Path interning (`Profile::internPath(...)`) which returns a unique path
identifier.
- Block appending (`Profile::addBlock(...)`) to add thread-associated
profile information.
- Path ID to Path lookup (`Profile::expandPath(...)`) to look up a
PathID and return the original interned path.
- Block iteration.
A 'Path' in this context represents the function call stack in
leaf-to-root order. This is represented as a path in an internally
managed prefix tree in the `Profile` instance. Having a handle (PathID)
to identify the unique Paths we encounter for a particular Profile
allows us to reduce the amount of memory required to associate profile
data to a particular Path.
This is the first of a series of patches to migrate the `llvm-stacks`
tool towards using a single profile representation.
Depends on D48653.
Reviewers: kpw, eizan
Reviewed By: kpw
Subscribers: kpw, thakis, mgorny, llvm-commits, hiraditya
Differential Revision: https://reviews.llvm.org/D48370
llvm-svn: 341012
Summary:
This change implements the profile loading functionality in LLVM to
support XRay's profiling mode in compiler-rt.
We introduce a type named `llvm::xray::Profile` which allows building a
profile representation. We can load an XRay profile from a file to build
Profile instances, or do it manually through the Profile type's API.
The intent is to get the `llvm-xray` tool to generate `Profile`
instances and use that as the common abstraction through which all
conversion and analysis can be done. In the future we can generate
`Profile` instances from `Trace` instances as well, through conversion
functions.
Some of the key operations supported by the `Profile` API are:
- Path interning (`Profile::internPath(...)`) which returns a unique path
identifier.
- Block appending (`Profile::addBlock(...)`) to add thread-associated
profile information.
- Path ID to Path lookup (`Profile::expandPath(...)`) to look up a
PathID and return the original interned path.
- Block iteration.
A 'Path' in this context represents the function call stack in
leaf-to-root order. This is represented as a path in an internally
managed prefix tree in the `Profile` instance. Having a handle (PathID)
to identify the unique Paths we encounter for a particular Profile
allows us to reduce the amount of memory required to associate profile
data to a particular Path.
This is the first of a series of patches to migrate the `llvm-stacks`
tool towards using a single profile representation.
Depends on D48653.
Reviewers: kpw, eizan
Reviewed By: kpw
Subscribers: mgorny, llvm-commits, hiraditya
Differential Revision: https://reviews.llvm.org/D48370
llvm-svn: 338825
Summary:
In preparation for graph comparison and filtering, this is a library for
representing graphs in LLVM. This will enable easier encapsulation and reuse
of graphs in llvm-xray.
Depends on D28999, D28225
Reviewers: dblaikie, dberris
Reviewed By: dberris
Subscribers: mgorny, llvm-commits
Differential Revision: https://reviews.llvm.org/D29005
llvm-svn: 294717
Summary:
In preparation for graph comparison and filtering, this is a library for
representing graphs in LLVM. This will enable easier encapsulation and reuse
of graphs in llvm-xray.
Depends on D28999, D28225
Reviewers: dblaikie, dberris
Reviewed By: dberris
Subscribers: mgorny, llvm-commits
Differential Revision: https://reviews.llvm.org/D29005
llvm-svn: 294713
Dean Michael Berris