Summary:
This change adds APIs to allow logging implementations to provide a
function for iterating through in-memory buffers (if they hold in-memory
buffers) and a way for users to generically deal with these buffers
in-process. These APIs are:
- __xray_log_set_buffer_iterator(...) and
__xray_log_remove_buffer_iterator(): installs and removes an
iterator function that takes an XRayBuffer and yields the next one.
- __xray_log_process_buffers(...): takes a function pointer that can
take a mode identifier (string) and an XRayBuffer to process this
data as they see fit.
The intent is to have the FDR mode implementation's buffers be
available through this `__xray_log_process_buffers(...)` API, so that
they can be streamed from memory instead of flushed to disk (useful for
getting the data to a network, or doing in-process analysis).
Basic mode logging will not support this mechanism as it's designed to
write the data mostly to disk.
Future implementations will may depend on this API as well, to allow for
programmatically working through the XRay buffers exposed to the
users in some fashion.
Reviewers: eizan, kpw, pelikan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D43495
llvm-svn: 326866
Summary:
Turns out sizeof(packed) isn't as strong as we'd hoped. This makes sure
that when we initialize the padding, all 12 bytes will be zero.
Reviewers: dberris, kpw, eizan
Subscribers: delcypher, llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D42494
llvm-svn: 323755
Summary:
This change allows for registration of multiple logging implementations
through a central mechanism in XRay, mapping an implementation to a
"mode". Modes are strings that are used as keys to determine which
implementation to install through a single API. This mechanism allows
users to choose which implementation to install either from the
environment variable 'XRAY_OPTIONS' with the `xray_mode=` flag, or
programmatically using the `__xray_select_mode(...)` function.
Here, we introduce two API functions for the XRay logging:
__xray_log_register_mode(Mode, Impl): Associates an XRayLogImpl to a
string Mode. We can only have one implementation associated with a given
Mode.
__xray_log_select_mode(Mode): Finds the associated Impl for Mode and
installs it as if by calling `__xray_set_log_impl(...)`.
Along with these changes, we also deprecate the xray_naive_log and
xray_fdr_log flags and encourage users to instead use the xray_mode
flag.
Reviewers: kpw, dblaikie, eizan, pelikan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D40703
llvm-svn: 319759
Summary:
Before this patch, XRay's basic (naive mode) logging would be
initialised and installed in an adhoc manner. This patch ports the
implementation of the basic (naive mode) logging implementation to use
the common XRay framework.
We also make the following changes to reduce the variance between the
usage model of basic mode from FDR (flight data recorder) mode:
- Allow programmatic control of the size of the buffers dedicated to
per-thread records. This removes some hard-coded constants and turns
them into runtime-controllable flags and through an Options
structure.
- Default the `xray_naive_log` option to false. For now, the only way
to start basic mode is to set the environment variable, or set the
default at build-time compiler options. Because of this change we've
had to update a couple of tests relying on basic mode being always
on.
- Removed the reliance on a non-trivially destructible per-thread
resource manager. We use a similar trick done in D39526 to use
pthread_key_create() and pthread_setspecific() to ensure that the
per-thread cleanup handling is performed at thread-exit time.
We also radically simplify the code structure for basic mode, to move
most of the implementation in the `__xray` namespace.
Reviewers: pelikan, eizan, kpw
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D40164
llvm-svn: 318734
Summary:
This change allows the XRay basic (naive) mode logging implementation to
start writing the payload entries through the arg1 logging handler. This
implementation writes out the records that the llvm-xray tool and the
trace reader library will start processing in D38550.
This introduces a new payload record type which logs the data through
the in-memory buffer. It uses the same size/alignment that the normal
XRay record entries use. We use a new record type to indicate these new
entries, so that the trace reader library in LLVM can start reading
these entries.
Depends on D38550.
Reviewers: pelikan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D38551
llvm-svn: 314968
Summary:
Write out records about logged function call first arguments. D32840
implements the reading of this in llvm-xray.
Reviewers: dberris
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D32844
llvm-svn: 314378
Summary:
This change starts differentiating tail exits from normal exits. We also
increase the version number of the "naive" log to version 2, which will
be the starting version where these records start appearing. In FDR mode
we treat the tail exits as normal exits, and are thus subject to the
same treatment with regard to record unwriting.
Updating the version number is important to signal older builds of the
llvm-xray tool that do not deal with the tail exit records must fail
early (and that users should only use the llvm-xray tool built after
the support for tail exits to get accurate handling of these records).
Depends on D37964.
Reviewers: kpw, pelikan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D37965
llvm-svn: 313515
Summary:
Define a build-time configuration option for the XRay runtime to
determine whether the archive will add an entry to the `.preinit_array`
section of the binary. We also allow for initializing the XRay data
structures with an explicit call to __xray_init(). This allows us to
give users the capability to initialize the XRay data structures on
demand.
This can allow us to start porting XRay to platforms where
`.preinit_array` isn't a supported section. It also allows us to limit
the effects of XRay in the initialization sequence for applications that
are sensitive to this kind of interference (i.e. large binaries) or
those that want to package XRay control in libraries.
Future changes should allow us to build two different library archives
for the XRay runtime, and allow clang users to determine which version
to link.
Reviewers: dblaikie, kpw, pelikan
Subscribers: mgorny, llvm-commits
Differential Revision: https://reviews.llvm.org/D36080
llvm-svn: 309909
Summary:
This change attempts to remove all the dependencies we have on
std::mutex and any std::shared_ptr construction in global variables. We
instead use raw pointers to these objects, and construct them on the
heap. In cases where it's possible, we lazily initialize these pointers.
While we do not have a replacement for std::shared_ptr yet in
compiler-rt, we use this work-around to avoid having to statically
initialize the objects as globals. Subsequent changes should allow us to
completely remove our dependency on std::shared_ptr and instead have our
own implementation of the std::shared_ptr and std::weak_ptr semantics
(or completely rewrite the implementaton to not need these
standard-library provided abstractions).
Reviewers: dblaikie, kpw, pelikan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D36078
llvm-svn: 309792
Summary:
This change implements support for the custom event logging sleds and
intrinsics at runtime. For now it only supports handling the sleds in
x86_64, with the implementations for other architectures stubbed out to
do nothing.
NOTE: Work in progress, uploaded for exposition/exploration purposes.
Depends on D27503, D30018, and D33032.
Reviewers: echristo, javed.absar, timshen
Subscribers: mehdi_amini, nemanjai, llvm-commits
Differential Revision: https://reviews.llvm.org/D30630
llvm-svn: 302857
Summary:
This change allows us to provide users and implementers of XRay handlers
a means of converting XRay function id's to addresses. This, in
combination with the facilities provided in D32695, allows users to find
out:
- How many function id's there are defined in the current binary.
- Get the address of the function associated with this function id.
- Patch only specific functions according to their requirements.
While we don't directly provide symbolization support in XRay, having
the function's address lets users determine this information easily
either during runtime, or offline with tools like 'addr2line'.
Reviewers: dblaikie, echristo, pelikan
Subscribers: kpw, llvm-commits
Differential Revision: https://reviews.llvm.org/D32846
llvm-svn: 302210
Summary:
This change allows us to patch/unpatch specific functions using the
function ID. This is useful in cases where implementations might want to
do coverage-style, or more fine-grained control of which functions to
patch or un-patch at runtime.
Depends on D32693.
Reviewers: dblaikie, echristo, kpw
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D32695
llvm-svn: 302112
Summary:
In this patch we document the requirements for implementations that want
to install handlers for the dynamically-controlled XRay "framework".
This clarifies what the expectations are for implementations that
want to install their handlers using this API (similar to how the FDR
logging implementation does so). It also gives users some guarantees on
semantics for the APIs.
If all goes well, users can decide to use the XRay APIs to control the
tracing/logging at the application level, without having to depend on
implementation details of the installed logging implementation. This
lets users choose the implementation that comes with compiler-rt, or
potentially multiple other implementations that use the same APIs.
We also add one convenience function (__xray_remove_log_impl()) for
explicitly removing the currently installed log implementation.
Reviewers: kpw, pelikan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D32579
llvm-svn: 301784
Summary:
Currently the FDR log writer, upon flushing, dumps a sequence of buffers from
its freelist to disk. A reader can read the first buffer up to an EOB record,
but then it is unclear how far ahead to scan to find the next threads traces.
There are a few ways to handle this problem.
1. The reader has externalized knowledge of the buffer size.
2. The size of buffers is in the file header or otherwise encoded in the log.
3. Only write out the portion of the buffer with records. When released, the
buffers are marked with a size.
4. The reader looks for memory that matches a pattern and synchronizes on it.
2 and 3 seem the most flexible and 2 does not rule 3 out.
This is an implementation of 2.
In addition, the function handler for fdr more aggressively checks for
finalization and makes an attempt to release its buffer.
Reviewers: pelikan, dberris
Reviewed By: dberris
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D31384
llvm-svn: 298982
Summary:
This change exercises the end-to-end functionality defined in the FDR
logging implementation. We also prepare for being able to run traces
generated by the FDR logging implementation from being analysed with the
llvm-xray command that comes with the LLVM distribution.
This also unblocks D31385, D31384, and D31345.
Reviewers: kpw, pelikan
Subscribers: llvm-commits, mgorny
Differential Revision: https://reviews.llvm.org/D31452
llvm-svn: 298977
Summary:
Functions with the LOG_ARGS_ENTRY sled kind at their beginning will be handled
in a way to (optionally) pass their first call argument to your logging handler.
For practical and performance reasons, only the first argument is supported, and
only up to 64 bits.
Reviewers: javed.absar, dberris
Reviewed By: dberris
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D29703
llvm-svn: 297000
Summary:
In this change we introduce the notion of a "flight data recorder" mode
for XRay logging, where XRay logs in-memory first, and write out data
on-demand as required (as opposed to the naive implementation that keeps
logging while tracing is "on"). This depends on D26232 where we
implement the core data structure for holding the buffers that threads
will be using to write out records of operation.
This implementation only currently works on x86_64 and depends heavily
on the TSC math to write out smaller records to the inmemory buffers.
Also, this implementation defines two different kinds of records with
different sizes (compared to the current naive implementation): a
MetadataRecord (16 bytes) and a FunctionRecord (8 bytes). MetadataRecord
entries are meant to write out information like the thread ID for which
the metadata record is defined for, whether the execution of a thread
moved to a different CPU, etc. while a FunctionRecord represents the
different kinds of function call entry/exit records we might encounter
in the course of a thread's execution along with a delta from the last
time the logging handler was called.
While this implementation is not exactly what is described in the
original XRay whitepaper, this one gives us an initial implementation
that we can iterate and build upon.
Reviewers: echristo, rSerge, majnemer
Subscribers: mehdi_amini, llvm-commits, mgorny
Differential Revision: https://reviews.llvm.org/D27038
llvm-svn: 293015
Summary:
In this change we introduce the notion of a "flight data recorder" mode
for XRay logging, where XRay logs in-memory first, and write out data
on-demand as required (as opposed to the naive implementation that keeps
logging while tracing is "on"). This depends on D26232 where we
implement the core data structure for holding the buffers that threads
will be using to write out records of operation.
This implementation only currently works on x86_64 and depends heavily
on the TSC math to write out smaller records to the inmemory buffers.
Also, this implementation defines two different kinds of records with
different sizes (compared to the current naive implementation): a
MetadataRecord (16 bytes) and a FunctionRecord (8 bytes). MetadataRecord
entries are meant to write out information like the thread ID for which
the metadata record is defined for, whether the execution of a thread
moved to a different CPU, etc. while a FunctionRecord represents the
different kinds of function call entry/exit records we might encounter
in the course of a thread's execution along with a delta from the last
time the logging handler was called.
While this implementation is not exactly what is described in the
original XRay whitepaper, this one gives us an initial implementation
that we can iterate and build upon.
Reviewers: echristo, rSerge, majnemer
Subscribers: mehdi_amini, llvm-commits, mgorny
Differential Revision: https://reviews.llvm.org/D27038
llvm-svn: 290852
Summary:
This change depends on D23986 which adds tail call-specific sleds. For
now we treat them first as normal exits, and in the future leave room
for implementing this as a different kind of log entry.
The reason for deferring the change is so that we can keep the naive
logging implementation more accurate without additional complexity for
reading the log. The accuracy is gained in effectively interpreting call
stacks like:
A()
B()
C()
Which when tail-call merged will end up not having any exit entries for
A() nor B(), but effectively in turn can be reasoned about as:
A()
B()
C()
Although we lose the fact that A() had called B() then had called C()
with the naive approach, a later iteration that adds the explicit tail
call entries would be a change in the log format and thus necessitate a
version change for the header. We can do this later to have a chance at
releasing some tools (in D21987) that are able to handle the naive log
format, then support higher version numbers of the log format too.
Reviewers: echristo, kcc, rSerge, majnemer
Subscribers: mehdi_amini, llvm-commits, dberris
Differential Revision: https://reviews.llvm.org/D23988
llvm-svn: 284178
Depends on D21612 which implements the building blocks for the compiler-rt
implementation of the XRay runtime. We use a naive in-memory log of fixed-size
entries that get written out to a log file when the buffers are full, and when
the thread exits.
This implementation lays some foundations on to allowing for more complex XRay
records to be written to the log in subsequent changes. It also defines the format
that the function call accounting tool in D21987 will start building upon.
Once D21987 lands, we should be able to start defining more tests using that tool
once the function call accounting tool becomes part of the llvm distribution.
Reviewers: echristo, kcc, rnk, eugenis, majnemer, rSerge
Subscribers: sdardis, rSerge, dberris, tberghammer, danalbert, srhines, majnemer, llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D21982
llvm-svn: 279805
This addresses some comments from D21612, which contains the following changes:
- Update __xray_patch() and __xray_unpatch() API documentation to not imply asynchrony.
- Introduce a scope cleanup mechanism to make sure we can roll-back changes to the XRayPatching global atomic.
- Introduce a few more comments for potential extension points for other platforms (for the implementation details of patching and un-patching).
Reviewers: eugenis, rnk, kcc, echristo, majnemer
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D22911
llvm-svn: 277124
Summary:
This is a fixed-up version of D21612, to address failure identified post-commit.
Original commit description:
This patch implements the initialisation and patching routines for the XRay runtime, along with the necessary trampolines for function entry/exit handling. For now we only define the basic hooks for allowing an implementation to define a handler that gets run on function entry/exit. We expose a minimal API for controlling the behaviour of the runtime (patching, cleanup, and setting the handler to invoke when instrumenting).
Fixes include:
- Gating XRay build to only Linux x86_64 and with the right dependencies in case it is the only library being built
- Including <cstddef> to fix std::size_t issue
Reviewers: kcc, rnk, echristo
Subscribers: mehdi_amini, llvm-commits
Differential Revision: https://reviews.llvm.org/D22611
llvm-svn: 276251
and also the follow-up "[xray] Only build xray on Linux for now"
Two build errors were reported on the llvm-commits list:
[ 88%] Building CXX object lib/xray/CMakeFiles/clang_rt.xray-x86_64.dir/xray_flags.cc.o
/mnt/b/sanitizer-buildbot1/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/lib/xray/xray_init.cc:23:10: fatal error: 'llvm/Support/ELF.h' file not found
#include "llvm/Support/ELF.h"
^
and
In file included from /w/src/llvm.org/projects/compiler-rt/lib/xray/xray_interface.cc:16:
/w/src/llvm.org/projects/compiler-rt/lib/xray/xray_interface_internal.h:36:8: error:
no type named 'size_t' in namespace 'std'
std::size_t Entries;
~~~~~^
llvm-svn: 276186
Summary:
This patch implements the initialisation and patching routines for the XRay runtime, along with the necessary trampolines for function entry/exit handling. For now we only define the basic hooks for allowing an implementation to define a handler that gets run on function entry/exit. We expose a minimal API for controlling the behaviour of the runtime (patching, cleanup, and setting the handler to invoke when instrumenting).
Depends on D19904
Reviewers: echristo, kcc, rnk
Subscribers: rnk, mehdi_amini, llvm-commits
Differential Revision: https://reviews.llvm.org/D21612
llvm-svn: 276117
Dean Michael Berris