Previously we've been reading and writing the wrong types which only
worked in little endian implementations. This time we're writing the
same typed values the runtime is using, and reading them appropriately
as well.
llvm-svn: 341241
This change allows us to let the compiler do the right thing for when
handling big-endian and little-endian records for FDR mode function
records.
Previously, we assumed that the encoding was little-endian that reading
the first byte to look for the function id and function record types was
ordered in a little-endian manner. This change allows us to better
handle function records where the first four bytes may actually be
encoded in big-endian thus giving us the wrong bytes where we're seeking
the function information from.
This is a follow-up to D51210 and D51289.
llvm-svn: 341236
This change makes the writer implementation more consistent with the way
fields are written down to avoid assumptions on bitfield order and
padding. We also fix an inconsistency between the type returned by the
`delta()` accessor to match the data member it's returning.
This is a follow-up to D51289 and D51210.
llvm-svn: 341230
This change makes the XRay Trace loading functions first use a
little-endian data extractor, then on failures try a big-endian data
extractor. Without this change, the trace loading facility will not work
with data written from a big-endian machine.
Follow-up to D51210 and D51289.
llvm-svn: 341226
Before this patch, the FDRTraceWriter would not take endianness into
account when writing data into the output stream.
This is a follow-up to D51289 and D51210.
llvm-svn: 341223
This simplifies the implementation of the metadata lookup by using
scoped enums, rather than using enum classes. This way we can get the
number-name mapping without having to resort to comments.
Follow-up to D51289.
llvm-svn: 341205
Summary:
This patch defines two new base types called `RecordProducer` and
`RecordConsumer` which have default implementations for convenience
(particularly for testing).
A `RecordProducer` implementation has one member function called
`produce()` which serves as a factory constructor for `Record`
instances. This code exercises the `RecordInitializer` code path in the
implementation for `FileBasedRecordProducer`.
A `RecordConsumer` has a single member function called `consume(...)`
which, as the name implies, consumes instances of
`std::unique_ptr<Record>`. We have two implementations, one of which is
used in the test to generate a vector of `std::unique_ptr<Record>`
similar to how the `LogBuilder` implementation works.
We introduce a test in `FDRProducerConsumerTest` which ensures that
records we write through the `FDRTraceWriter` can be loaded by the
`FileBasedRecordProducer`. The record(s) loaded this way are written
again through the `FDRTraceWriter` into a separate string, which we then
compare. This ensures that the read-in bytes to create the `Record`
instances in memory can be replicated when written out through the
`FDRTraceWriter`.
This change depends on D51210 and is part of the refactoring of D50441
into smaller, more focused changes.
Reviewers: eizan, kpw
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51289
llvm-svn: 341180
Summary:
This is the first step in the larger refactoring and reduction of
D50441.
This step in the process does the following:
- Introduces more granular types of `Record`s representing the many
kinds of records written/read by the Flight Data Recorder (FDR) mode
`Trace` loading function(s).
- Introduces an abstract `RecordVisitor` type meant to handle the
processing of the various `Record` derived types. This `RecordVisitor`
has two implementations in this patch: `RecordInitializer` and
`FDRTraceWriter`.
- We also introduce a convenience interface for building a collection of
`Record` instances called a `LogBuilder`. This allows us to generate
sequences of `Record` instances manually (used in unit tests but
useful otherwise).
- The`FDRTraceWriter` class implements the `RecordVisitor` interface and
handles the writing of metadata records to a `raw_ostream`. We
demonstrate that in the unit test, we can generate in-memory FDR mode
traces using the specific `Record` derived types, which we load
through the `loadTrace(...)` function yielding valid `Trace` objects.
This patch introduces the required types and concepts for us to start
replacing the logic implemented in the `loadFDRLog` function to use the
more granular types. In subsequent patches, we will introduce more
visitor implementations which isolate the verification, printing,
indexing, production/consumption, and finally the conversion of the FDR
mode logs.
The overarching goal of these changes is to make handling FDR mode logs
better tested, more understandable, more extensible, and more
systematic. This will also allow us to better represent the execution
trace, as we improve the fidelity of the events we represent in an XRay
`Trace` object, which we intend to do after FDR mode log processing is
in better shape.
Reviewers: eizan
Reviewed By: eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51210
llvm-svn: 341029
Summary:
This change implements the profile loading functionality in LLVM to
support XRay's profiling mode in compiler-rt.
We introduce a type named `llvm::xray::Profile` which allows building a
profile representation. We can load an XRay profile from a file to build
Profile instances, or do it manually through the Profile type's API.
The intent is to get the `llvm-xray` tool to generate `Profile`
instances and use that as the common abstraction through which all
conversion and analysis can be done. In the future we can generate
`Profile` instances from `Trace` instances as well, through conversion
functions.
Some of the key operations supported by the `Profile` API are:
- Path interning (`Profile::internPath(...)`) which returns a unique path
identifier.
- Block appending (`Profile::addBlock(...)`) to add thread-associated
profile information.
- Path ID to Path lookup (`Profile::expandPath(...)`) to look up a
PathID and return the original interned path.
- Block iteration.
A 'Path' in this context represents the function call stack in
leaf-to-root order. This is represented as a path in an internally
managed prefix tree in the `Profile` instance. Having a handle (PathID)
to identify the unique Paths we encounter for a particular Profile
allows us to reduce the amount of memory required to associate profile
data to a particular Path.
This is the first of a series of patches to migrate the `llvm-stacks`
tool towards using a single profile representation.
Depends on D48653.
Reviewers: kpw, eizan
Reviewed By: kpw
Subscribers: kpw, thakis, mgorny, llvm-commits, hiraditya
Differential Revision: https://reviews.llvm.org/D48370
llvm-svn: 341012
This patch splits the file trace loading function into two versions, one
that takes a filename and one that takes a `DataExtractor`.
This change is a precursor to larger changes to increase test coverage
for the trace loading implementation.
llvm-svn: 340603
Summary:
This patch moves out the definition of the XRay log file header from
binary logs into its own header and implementation file.
This is one part of the refactoring being done in D50441.
Reviewers: eizan
Subscribers: mgorny, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D51086
llvm-svn: 340389
Summary:
This change uses a single offset pointer used throughout the
implementation of the individual record parsers. This allows us to
report where in a trace file parsing failed.
We're still in an intermediate step here as we prepare to refactor this
further into a set of types and use object-oriented design principles
for a cleaner implementation. The next steps will be to allow us to
parse/dump files in a streaming fashion and incrementally build up the
structures in memory instead of the current all-or-nothing approach.
Reviewers: kpw, eizan
Reviewed By: kpw
Subscribers: hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D50169
llvm-svn: 339092
Summary:
This change implements the profile loading functionality in LLVM to
support XRay's profiling mode in compiler-rt.
We introduce a type named `llvm::xray::Profile` which allows building a
profile representation. We can load an XRay profile from a file to build
Profile instances, or do it manually through the Profile type's API.
The intent is to get the `llvm-xray` tool to generate `Profile`
instances and use that as the common abstraction through which all
conversion and analysis can be done. In the future we can generate
`Profile` instances from `Trace` instances as well, through conversion
functions.
Some of the key operations supported by the `Profile` API are:
- Path interning (`Profile::internPath(...)`) which returns a unique path
identifier.
- Block appending (`Profile::addBlock(...)`) to add thread-associated
profile information.
- Path ID to Path lookup (`Profile::expandPath(...)`) to look up a
PathID and return the original interned path.
- Block iteration.
A 'Path' in this context represents the function call stack in
leaf-to-root order. This is represented as a path in an internally
managed prefix tree in the `Profile` instance. Having a handle (PathID)
to identify the unique Paths we encounter for a particular Profile
allows us to reduce the amount of memory required to associate profile
data to a particular Path.
This is the first of a series of patches to migrate the `llvm-stacks`
tool towards using a single profile representation.
Depends on D48653.
Reviewers: kpw, eizan
Reviewed By: kpw
Subscribers: mgorny, llvm-commits, hiraditya
Differential Revision: https://reviews.llvm.org/D48370
llvm-svn: 338825
Summary:
llvm-xray changes:
- account-mode - process-id {...} shows after thread-id
- convert-mode - process {...} shows after thread
- parses FDR and basic mode pid entries
- Checks version number for FDR log parsing.
Basic logging changes:
- Update header version from 2 -> 3
FDR logging changes:
- Update header version from 2 -> 3
- in writeBufferPreamble, there is an additional PID Metadata record (after thread id record and tsc record)
Test cases changes:
- fdr-mode.cc, fdr-single-thread.cc, fdr-thread-order.cc modified to catch process id output in the log.
Reviewers: dberris
Reviewed By: dberris
Subscribers: hiraditya, llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D49153
llvm-svn: 336974
Summary:
When there's a mismatch of a function argument being right after the
wrong function, print an offset into the file where that happened, to
ease further debugging.
Reviewers: dberris, eizan, kpw
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D42492
llvm-svn: 323758
Summary:
Before this change, the FDR mode implementation relied on at thread-exit
handling to return buffers back to the (global) buffer queue. This
introduces issues with the initialisation of the thread_local objects
which, even through the use of pthread_setspecific(...) may eventually
call into an allocation function. Similar to previous changes in this
line, we're finding that there is a huge potential for deadlocks when
initialising these thread-locals when the memory allocation
implementation is also xray-instrumented.
In this change, we limit the call to pthread_setspecific(...) to provide
a non-null value to associate to the key created with
pthread_key_create(...). While this doesn't completely eliminate the
potential for the deadlock(s), it does allow us to still clean up at
thread exit when we need to. The change is that we don't need to do more
work when starting and ending a thread's lifetime. We also have a test
to make sure that we actually can safely recycle the buffers in case we
end up re-using the buffer(s) available from the queue on multiple
thread entry/exits.
This change cuts across both LLVM and compiler-rt to allow us to update
both the XRay runtime implementation as well as the library support for
loading these new versions of the FDR mode logging. Version 2 of the FDR
logging implementation makes the following changes:
* Introduction of a new 'BufferExtents' metadata record that's outside
of the buffer's contents but are written before the actual buffer.
This data is associated to the Buffer handed out by the BufferQueue
rather than a record that occupies bytes in the actual buffer.
* Removal of the "end of buffer" records. This is in-line with the
changes we described above, to allow for optimistic logging without
explicit record writing at thread exit.
The optimistic logging model operates under the following assumptions:
* Threads writing to the buffers will potentially race with the thread
attempting to flush the log. To avoid this situation from occuring,
we make sure that when we've finalized the logging implementation,
that threads will see this finalization state on the next write, and
either choose to not write records the thread would have written or
write the record(s) in two phases -- first write the record(s), then
update the extents metadata.
* We change the buffer queue implementation so that once it's handed
out a buffer to a thread, that we assume that buffer is marked
"used" to be able to capture partial writes. None of this will be
safe to handle if threads are racing to write the extents records
and the reader thread is attempting to flush the log. The optimism
comes from the finalization routine being required to complete
before we attempt to flush the log.
This is a fairly significant semantics change for the FDR
implementation. This is why we've decided to update the version number
for FDR mode logs. The tools, however, still need to be able to support
older versions of the log until we finally deprecate those earlier
versions.
Reviewers: dblaikie, pelikan, kpw
Subscribers: llvm-commits, hiraditya
Differential Revision: https://reviews.llvm.org/D39526
llvm-svn: 318733
Summary:
The arg1 logging handler changed in compiler-rt to start writing a
different type for entries encountered when logging the first argument
of XRay-instrumented functions. This change allows the trace loader to
support reading these record types as well as prepare for when the
basic (naive) mode implementation starts writing down the argument
payloads.
Without this change, binaries with arg1 logging support enabled start
writing unreadable logs for any of the XRay tracing tools.
Reviewers: pelikan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D38550
llvm-svn: 314967
Summary:
A new FDR metadata record will support logging a function call argument;
appending multiple metadata records will represent a sequence of arguments
meaning that "holes" are not representable by the buffer format. Each
call argument is currently a 64-bit value (useful for "this" pointers and
synchronization objects).
If present, we put this argument to the function call "entry" record it
belongs to, and alter its type to notify the user of its presence.
Reviewers: dberris
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D32840
llvm-svn: 314269
Summary:
This change adds support for explicit tail-exit records to be written by
the XRay runtime. This lets us differentiate the tail exit
records/events in the log, and allows us to treat those exit events
especially in the future. For now we allow printing those out in YAML
(and reading them in).
Reviewers: kpw, pelikan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D37964
llvm-svn: 313514
Summary:
For readers unfamiliar with the XRay code base, reference the compiler-rt
implementation even though we're not allowed to share any code and explain
our little-endian views more clearly.
For code clarity either get rid of obvious comments or explain their
intentions, fix typos, correct coding style according to LLVM's standards
and manually CSE long expressions to point out it is the same expression.
Reviewers: dberris
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D34339
llvm-svn: 313340
Summary:
When extracting the instrumentation map from a binary, we should be able
to recognize the new kinds of instrumentation sleds we've been emitting
with the compiler using -fxray-instrument. This change adds a test for
all the kinds of sleds we currently support (sans the tail-call sled,
which is a bit harder to force in a simple prebuilt input).
Reviewers: kpw, dblaikie
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D36819
llvm-svn: 311305
Summary:
Adding a new restructuredText file to document the trace format produced with
an FDR mode handler and read by llvm-xray toolset.
Fixed two problems in the documentation from differential review. One bad table
and a missing link in the toc.
Original commit was e97c5836a77db803fe53319c53f3bf8e8b26d2b7.
Reviewers: dberris, pelikan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D36041
llvm-svn: 309891
Summary:
Adding a new restructuredText file to document the trace format produced with
an FDR mode handler and read by llvm-xray toolset.
Reviewers: dberris, pelikan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D36041
llvm-svn: 309836
I did this a long time ago with a janky python script, but now
clang-format has built-in support for this. I fed clang-format every
line with a #include and let it re-sort things according to the precise
LLVM rules for include ordering baked into clang-format these days.
I've reverted a number of files where the results of sorting includes
isn't healthy. Either places where we have legacy code relying on
particular include ordering (where possible, I'll fix these separately)
or where we have particular formatting around #include lines that
I didn't want to disturb in this patch.
This patch is *entirely* mechanical. If you get merge conflicts or
anything, just ignore the changes in this patch and run clang-format
over your #include lines in the files.
Sorry for any noise here, but it is important to keep these things
stable. I was seeing an increasing number of patches with irrelevant
re-ordering of #include lines because clang-format was used. This patch
at least isolates that churn, makes it easy to skip when resolving
conflicts, and gets us to a clean baseline (again).
llvm-svn: 304787
Summary:
In D30630 we will start writing custom event records. To avoid breaking
the tools that read the FDR mode records, we skip over these records.
To support these custom event records more effectively, we will have to
expose them in the trace loading API. Those changes will be forthcoming.
Reviewers: kpw, pelikan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D33032
llvm-svn: 302856
Summary:
It is problematic for this reader that it expects to read data from
several threads, but the header or message format does not define
framing. Since the buffers are reused, we can't rely on skipping
zeroed out data as a synchronization method either.
There is an argument that this is not version compatible with the format
the reader expected previously. I argue that since the writer wrote garbage
past the end of buffer record, there is no currently working reader to
compromise.
The corresponding writer change is posted to D31384.
Reviewers: dberris, pelikan
Reviewed By: dberris
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D31385
llvm-svn: 298983
Summary:
The file type packs function trace data onto disk from potentially multiple
threads that are aggregated and flushed during the course of an instrumented
program's runtime.
It is named FDR mode or Flight Data recorder as an analogy to plane
blackboxes, which instrument a running system without access to IO.
The writer code is defined in compiler-rt in xray_fdr_logging.h/cc
Reviewers: rSerge, kcc, dberris
Reviewed By: dberris
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D29697
llvm-svn: 295397
Summary:
powerpc64 big-endian is not supported, but I believe that most logic can
be shared, except for xray_powerpc64.cc.
Also add a function InvalidateInstructionCache to xray_util.h, which is
copied from llvm/Support/Memory.cpp. I'm not sure if I need to add a unittest,
and I don't know how.
Reviewers: dberris, echristo, iteratee, kbarton, hfinkel
Subscribers: mehdi_amini, nemanjai, mgorny, llvm-commits
Differential Revision: https://reviews.llvm.org/D29742
llvm-svn: 294781
Summary:
This change implements the instrumentation map loading library which can
understand both YAML-defined instrumentation maps, and ELF 64-bit object
files that have the XRay instrumentation map section. We break it out
into a library on its own to allow for other applications to deal with
the XRay instrumentation map defined in XRay-instrumented binaries.
This type provides both raw access to the logical representation of the
instrumentation map entries as well as higher level functions for
converting a function ID into a function address.
At this point we only support ELF64 binaries and YAML-defined XRay
instrumentation maps. Future changes should extend this to support
32-bit ELF binaries, as well as other binary formats (like MachO).
As part of this change we also migrate all uses of the extraction logic
that used to be defined in tools/llvm-xray/ to use this new type and
interface for loading from files. We also remove the flag from the
`llvm-xray` tool that required users to specify the type of the
instrumentation map file being provided to instead make the library
auto-detect the file type.
Reviewers: dblaikie
Subscribers: mgorny, varno, llvm-commits
Differential Revision: https://reviews.llvm.org/D29319
llvm-svn: 293721
Summary:
In this change we move the definition of the log reading routines from
the tools directory in LLVM to {include/llvm,lib}/XRay. We improve the
documentation a little bit for the publicly accessible headers, and
adjust the top-matter. This also leads to some refactoring and cleanup
in the tooling code.
In particular, we do the following:
- Rename the class from LogReader to Trace, as it better represents
the logical set of records as opposed to a log.
- Use file type detection instead of asking the user to say what
format the input file is. This allows us to keep the interface
simple and encapsulate the logic of loading the data appropriately.
In future changes we increase the API surface and write dedicated unit
tests for the XRay library.
Depends on D24376.
Reviewers: dblaikie, echristo
Subscribers: mehdi_amini, mgorny, llvm-commits, varno
Differential Revision: https://reviews.llvm.org/D28345
llvm-svn: 291652
Dean Michael Berris