Summary:
This change has a number of fixes for FDR mode in compiler-rt along with
changes to the tooling handling the traces in llvm.
In the runtime, we do the following:
- Advance the "last record" pointer appropriately when writing the
custom event data in the log.
- Add XRAY_NEVER_INSTRUMENT in the rewinding routine.
- When collecting the argument of functions appropriately marked, we
should not attempt to rewind them (and reset the counts of functions
that can be re-wound).
In the tooling, we do the following:
- Remove the state logic in BlockIndexer and instead rely on the
presence/absence of records to indicate blocks.
- Move the verifier into a loop associated with each block.
Reviewers: mboerger, eizan
Subscribers: llvm-commits, hiraditya
Differential Revision: https://reviews.llvm.org/D51965
llvm-svn: 342122
Summary:
In this change, we overhaul the implementation for loading
`llvm::xray::Trace` objects from files by using the combination of
specific FDR Record types and visitors breaking up the logic to
reconstitute an execution trace from flight-data recorder mode traces.
This change allows us to handle out-of-temporal order blocks as written
in files, and more consistently recreate an execution trace spanning
multiple blocks and threads. To do this, we use the `WallclockRecord`
associated with each block to maintain temporal order of blocks, before
attempting to recreate an execution trace.
The new addition in this change is the `TraceExpander` type which can be
thought of as a decompression/decoding routine. This allows us to
maintain the state of an execution environment (thread+process) and
create `XRayRecord` instances that fit nicely into the `Trace`
container. We don't have a specific unit test for the TraceExpander
type, since the end-to-end tests for the `llvm-xray convert` tools
already cover precisely this codepath.
This change completes the refactoring started with D50441.
Depends on D51911.
Reviewers: mboerger, eizan
Subscribers: mgorny, hiraditya, mgrang, llvm-commits
Differential Revision: https://reviews.llvm.org/D51912
llvm-svn: 341906
Summary:
This more correctly reflects the data written by the FDR mode runtime.
This is a continuation of the work in D50441.
Reviewers: mboerger, eizan
Subscribers: hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51911
llvm-svn: 341905
Summary:
In this change, we implement a `BlockPrinter` which orders records in a
Block that's been indexed by the `BlockIndexer`. This is used in the
`llvm-xray fdr-dump` tool which ties together the various types and
utilities we've been working on, to allow for inspection of XRay FDR
mode traces both with and without verification.
This change is the final step of the refactoring of D50441.
Reviewers: mboerger, eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51846
llvm-svn: 341887
Summary:
This patch implements a `BlockVerifier` type which enforces the
invariants of the log structure of FDR mode logs on a per-block basis.
This ensures that the data we encounter from an FDR mode log
semantically correct (i.e. that records follow the documented "grammar"
for FDR mode log records).
This is another part of the refactoring of D50441.
This is a slightly modified version of rL341628, avoiding the
`std::tuple<...>` constructor that is not constexpr in C++11.
Reviewers: mboerger, eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51723
llvm-svn: 341769
Summary:
This patch implements a `BlockVerifier` type which enforces the
invariants of the log structure of FDR mode logs on a per-block basis.
This ensures that the data we encounter from an FDR mode log
semantically correct (i.e. that records follow the documented "grammar"
for FDR mode log records).
This is another part of the refactoring of D50441.
Reviewers: mboerger, eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51723
llvm-svn: 341628
Summary:
This change adds a `BlockIndexer` type which maintains pointers to
records that belong to the same process+thread pairs. The indexing
happens with order of appearance of records as they are visited.
This version of the indexer currently only supports FDR version 3 logs,
which contain `BufferExtent` records. We will add support for v2 and v1
logs in follow-up patches.
This is another part of D50441.
Reviewers: eizan, kpw, mboerger
Reviewed By: mboerger
Subscribers: mboerger, mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51673
llvm-svn: 341518
Summary:
This change adds a `RecordPrinter` type which does some basic text
serialization of the FDR record instances. This is one component of the
tool we're building to dump the records from an FDR mode log as-is.
This is a small part of D50441.
Reviewers: eizan, kpw
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51672
llvm-svn: 341447
Since we changed the storage for the PID in PIDRecord instances, we need
to also update the way we load the data from a DataExtractor through the
RecordInitializer.
llvm-svn: 341243
Previously we've been reading and writing the wrong types which only
worked in little endian implementations. This time we're writing the
same typed values the runtime is using, and reading them appropriately
as well.
llvm-svn: 341241
This change allows us to let the compiler do the right thing for when
handling big-endian and little-endian records for FDR mode function
records.
Previously, we assumed that the encoding was little-endian that reading
the first byte to look for the function id and function record types was
ordered in a little-endian manner. This change allows us to better
handle function records where the first four bytes may actually be
encoded in big-endian thus giving us the wrong bytes where we're seeking
the function information from.
This is a follow-up to D51210 and D51289.
llvm-svn: 341236
This change makes the writer implementation more consistent with the way
fields are written down to avoid assumptions on bitfield order and
padding. We also fix an inconsistency between the type returned by the
`delta()` accessor to match the data member it's returning.
This is a follow-up to D51289 and D51210.
llvm-svn: 341230
This change makes the XRay Trace loading functions first use a
little-endian data extractor, then on failures try a big-endian data
extractor. Without this change, the trace loading facility will not work
with data written from a big-endian machine.
Follow-up to D51210 and D51289.
llvm-svn: 341226
Before this patch, the FDRTraceWriter would not take endianness into
account when writing data into the output stream.
This is a follow-up to D51289 and D51210.
llvm-svn: 341223
This simplifies the implementation of the metadata lookup by using
scoped enums, rather than using enum classes. This way we can get the
number-name mapping without having to resort to comments.
Follow-up to D51289.
llvm-svn: 341205
Summary:
This patch defines two new base types called `RecordProducer` and
`RecordConsumer` which have default implementations for convenience
(particularly for testing).
A `RecordProducer` implementation has one member function called
`produce()` which serves as a factory constructor for `Record`
instances. This code exercises the `RecordInitializer` code path in the
implementation for `FileBasedRecordProducer`.
A `RecordConsumer` has a single member function called `consume(...)`
which, as the name implies, consumes instances of
`std::unique_ptr<Record>`. We have two implementations, one of which is
used in the test to generate a vector of `std::unique_ptr<Record>`
similar to how the `LogBuilder` implementation works.
We introduce a test in `FDRProducerConsumerTest` which ensures that
records we write through the `FDRTraceWriter` can be loaded by the
`FileBasedRecordProducer`. The record(s) loaded this way are written
again through the `FDRTraceWriter` into a separate string, which we then
compare. This ensures that the read-in bytes to create the `Record`
instances in memory can be replicated when written out through the
`FDRTraceWriter`.
This change depends on D51210 and is part of the refactoring of D50441
into smaller, more focused changes.
Reviewers: eizan, kpw
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51289
llvm-svn: 341180
Summary:
This is the first step in the larger refactoring and reduction of
D50441.
This step in the process does the following:
- Introduces more granular types of `Record`s representing the many
kinds of records written/read by the Flight Data Recorder (FDR) mode
`Trace` loading function(s).
- Introduces an abstract `RecordVisitor` type meant to handle the
processing of the various `Record` derived types. This `RecordVisitor`
has two implementations in this patch: `RecordInitializer` and
`FDRTraceWriter`.
- We also introduce a convenience interface for building a collection of
`Record` instances called a `LogBuilder`. This allows us to generate
sequences of `Record` instances manually (used in unit tests but
useful otherwise).
- The`FDRTraceWriter` class implements the `RecordVisitor` interface and
handles the writing of metadata records to a `raw_ostream`. We
demonstrate that in the unit test, we can generate in-memory FDR mode
traces using the specific `Record` derived types, which we load
through the `loadTrace(...)` function yielding valid `Trace` objects.
This patch introduces the required types and concepts for us to start
replacing the logic implemented in the `loadFDRLog` function to use the
more granular types. In subsequent patches, we will introduce more
visitor implementations which isolate the verification, printing,
indexing, production/consumption, and finally the conversion of the FDR
mode logs.
The overarching goal of these changes is to make handling FDR mode logs
better tested, more understandable, more extensible, and more
systematic. This will also allow us to better represent the execution
trace, as we improve the fidelity of the events we represent in an XRay
`Trace` object, which we intend to do after FDR mode log processing is
in better shape.
Reviewers: eizan
Reviewed By: eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51210
llvm-svn: 341029
Summary:
This change implements the profile loading functionality in LLVM to
support XRay's profiling mode in compiler-rt.
We introduce a type named `llvm::xray::Profile` which allows building a
profile representation. We can load an XRay profile from a file to build
Profile instances, or do it manually through the Profile type's API.
The intent is to get the `llvm-xray` tool to generate `Profile`
instances and use that as the common abstraction through which all
conversion and analysis can be done. In the future we can generate
`Profile` instances from `Trace` instances as well, through conversion
functions.
Some of the key operations supported by the `Profile` API are:
- Path interning (`Profile::internPath(...)`) which returns a unique path
identifier.
- Block appending (`Profile::addBlock(...)`) to add thread-associated
profile information.
- Path ID to Path lookup (`Profile::expandPath(...)`) to look up a
PathID and return the original interned path.
- Block iteration.
A 'Path' in this context represents the function call stack in
leaf-to-root order. This is represented as a path in an internally
managed prefix tree in the `Profile` instance. Having a handle (PathID)
to identify the unique Paths we encounter for a particular Profile
allows us to reduce the amount of memory required to associate profile
data to a particular Path.
This is the first of a series of patches to migrate the `llvm-stacks`
tool towards using a single profile representation.
Depends on D48653.
Reviewers: kpw, eizan
Reviewed By: kpw
Subscribers: kpw, thakis, mgorny, llvm-commits, hiraditya
Differential Revision: https://reviews.llvm.org/D48370
llvm-svn: 341012
This patch splits the file trace loading function into two versions, one
that takes a filename and one that takes a `DataExtractor`.
This change is a precursor to larger changes to increase test coverage
for the trace loading implementation.
llvm-svn: 340603
Summary:
This patch moves out the definition of the XRay log file header from
binary logs into its own header and implementation file.
This is one part of the refactoring being done in D50441.
Reviewers: eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51086
llvm-svn: 340389
Summary:
This change uses a single offset pointer used throughout the
implementation of the individual record parsers. This allows us to
report where in a trace file parsing failed.
We're still in an intermediate step here as we prepare to refactor this
further into a set of types and use object-oriented design principles
for a cleaner implementation. The next steps will be to allow us to
parse/dump files in a streaming fashion and incrementally build up the
structures in memory instead of the current all-or-nothing approach.
Reviewers: kpw, eizan
Reviewed By: kpw
Subscribers: hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D50169
llvm-svn: 339092
Summary:
This change implements the profile loading functionality in LLVM to
support XRay's profiling mode in compiler-rt.
We introduce a type named `llvm::xray::Profile` which allows building a
profile representation. We can load an XRay profile from a file to build
Profile instances, or do it manually through the Profile type's API.
The intent is to get the `llvm-xray` tool to generate `Profile`
instances and use that as the common abstraction through which all
conversion and analysis can be done. In the future we can generate
`Profile` instances from `Trace` instances as well, through conversion
functions.
Some of the key operations supported by the `Profile` API are:
- Path interning (`Profile::internPath(...)`) which returns a unique path
identifier.
- Block appending (`Profile::addBlock(...)`) to add thread-associated
profile information.
- Path ID to Path lookup (`Profile::expandPath(...)`) to look up a
PathID and return the original interned path.
- Block iteration.
A 'Path' in this context represents the function call stack in
leaf-to-root order. This is represented as a path in an internally
managed prefix tree in the `Profile` instance. Having a handle (PathID)
to identify the unique Paths we encounter for a particular Profile
allows us to reduce the amount of memory required to associate profile
data to a particular Path.
This is the first of a series of patches to migrate the `llvm-stacks`
tool towards using a single profile representation.
Depends on D48653.
Reviewers: kpw, eizan
Reviewed By: kpw
Subscribers: mgorny, llvm-commits, hiraditya
Differential Revision: https://reviews.llvm.org/D48370
llvm-svn: 338825
Summary:
llvm-xray changes:
- account-mode - process-id {...} shows after thread-id
- convert-mode - process {...} shows after thread
- parses FDR and basic mode pid entries
- Checks version number for FDR log parsing.
Basic logging changes:
- Update header version from 2 -> 3
FDR logging changes:
- Update header version from 2 -> 3
- in writeBufferPreamble, there is an additional PID Metadata record (after thread id record and tsc record)
Test cases changes:
- fdr-mode.cc, fdr-single-thread.cc, fdr-thread-order.cc modified to catch process id output in the log.
Reviewers: dberris
Reviewed By: dberris
Subscribers: hiraditya, llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D49153
llvm-svn: 336974
Summary:
When there's a mismatch of a function argument being right after the
wrong function, print an offset into the file where that happened, to
ease further debugging.
Reviewers: dberris, eizan, kpw
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D42492
llvm-svn: 323758
Summary:
Before this change, the FDR mode implementation relied on at thread-exit
handling to return buffers back to the (global) buffer queue. This
introduces issues with the initialisation of the thread_local objects
which, even through the use of pthread_setspecific(...) may eventually
call into an allocation function. Similar to previous changes in this
line, we're finding that there is a huge potential for deadlocks when
initialising these thread-locals when the memory allocation
implementation is also xray-instrumented.
In this change, we limit the call to pthread_setspecific(...) to provide
a non-null value to associate to the key created with
pthread_key_create(...). While this doesn't completely eliminate the
potential for the deadlock(s), it does allow us to still clean up at
thread exit when we need to. The change is that we don't need to do more
work when starting and ending a thread's lifetime. We also have a test
to make sure that we actually can safely recycle the buffers in case we
end up re-using the buffer(s) available from the queue on multiple
thread entry/exits.
This change cuts across both LLVM and compiler-rt to allow us to update
both the XRay runtime implementation as well as the library support for
loading these new versions of the FDR mode logging. Version 2 of the FDR
logging implementation makes the following changes:
* Introduction of a new 'BufferExtents' metadata record that's outside
of the buffer's contents but are written before the actual buffer.
This data is associated to the Buffer handed out by the BufferQueue
rather than a record that occupies bytes in the actual buffer.
* Removal of the "end of buffer" records. This is in-line with the
changes we described above, to allow for optimistic logging without
explicit record writing at thread exit.
The optimistic logging model operates under the following assumptions:
* Threads writing to the buffers will potentially race with the thread
attempting to flush the log. To avoid this situation from occuring,
we make sure that when we've finalized the logging implementation,
that threads will see this finalization state on the next write, and
either choose to not write records the thread would have written or
write the record(s) in two phases -- first write the record(s), then
update the extents metadata.
* We change the buffer queue implementation so that once it's handed
out a buffer to a thread, that we assume that buffer is marked
"used" to be able to capture partial writes. None of this will be
safe to handle if threads are racing to write the extents records
and the reader thread is attempting to flush the log. The optimism
comes from the finalization routine being required to complete
before we attempt to flush the log.
This is a fairly significant semantics change for the FDR
implementation. This is why we've decided to update the version number
for FDR mode logs. The tools, however, still need to be able to support
older versions of the log until we finally deprecate those earlier
versions.
Reviewers: dblaikie, pelikan, kpw
Subscribers: llvm-commits, hiraditya
Differential Revision: https://reviews.llvm.org/D39526
llvm-svn: 318733
Summary:
The arg1 logging handler changed in compiler-rt to start writing a
different type for entries encountered when logging the first argument
of XRay-instrumented functions. This change allows the trace loader to
support reading these record types as well as prepare for when the
basic (naive) mode implementation starts writing down the argument
payloads.
Without this change, binaries with arg1 logging support enabled start
writing unreadable logs for any of the XRay tracing tools.
Reviewers: pelikan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D38550
llvm-svn: 314967
Summary:
A new FDR metadata record will support logging a function call argument;
appending multiple metadata records will represent a sequence of arguments
meaning that "holes" are not representable by the buffer format. Each
call argument is currently a 64-bit value (useful for "this" pointers and
synchronization objects).
If present, we put this argument to the function call "entry" record it
belongs to, and alter its type to notify the user of its presence.
Reviewers: dberris
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D32840
llvm-svn: 314269
Summary:
This change adds support for explicit tail-exit records to be written by
the XRay runtime. This lets us differentiate the tail exit
records/events in the log, and allows us to treat those exit events
especially in the future. For now we allow printing those out in YAML
(and reading them in).
Reviewers: kpw, pelikan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D37964
llvm-svn: 313514
Summary:
For readers unfamiliar with the XRay code base, reference the compiler-rt
implementation even though we're not allowed to share any code and explain
our little-endian views more clearly.
For code clarity either get rid of obvious comments or explain their
intentions, fix typos, correct coding style according to LLVM's standards
and manually CSE long expressions to point out it is the same expression.
Reviewers: dberris
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D34339
llvm-svn: 313340
Summary:
When extracting the instrumentation map from a binary, we should be able
to recognize the new kinds of instrumentation sleds we've been emitting
with the compiler using -fxray-instrument. This change adds a test for
all the kinds of sleds we currently support (sans the tail-call sled,
which is a bit harder to force in a simple prebuilt input).
Reviewers: kpw, dblaikie
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D36819
llvm-svn: 311305
Summary:
Adding a new restructuredText file to document the trace format produced with
an FDR mode handler and read by llvm-xray toolset.
Fixed two problems in the documentation from differential review. One bad table
and a missing link in the toc.
Original commit was e97c5836a77db803fe53319c53f3bf8e8b26d2b7.
Reviewers: dberris, pelikan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D36041
llvm-svn: 309891
Dean Michael Berris