llvm-project/clang/docs/SourceBasedCodeCoverage.rst

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

305 lines
12 KiB
ReStructuredText
Raw Normal View History

==========================
Source-based Code Coverage
==========================
.. contents::
:local:
Introduction
============
This document explains how to use clang's source-based code coverage feature.
It's called "source-based" because it operates on AST and preprocessor
information directly. This allows it to generate very precise coverage data.
Clang ships two other code coverage implementations:
* :doc:`SanitizerCoverage` - A low-overhead tool meant for use alongside the
various sanitizers. It can provide up to edge-level coverage.
* gcov - A GCC-compatible coverage implementation which operates on DebugInfo.
This is enabled by ``-ftest-coverage`` or ``--coverage``.
From this point onwards "code coverage" will refer to the source-based kind.
The code coverage workflow
==========================
The code coverage workflow consists of three main steps:
* Compiling with coverage enabled.
* Running the instrumented program.
* Creating coverage reports.
The next few sections work through a complete, copy-'n-paste friendly example
based on this program:
.. code-block:: cpp
% cat <<EOF > foo.cc
#define BAR(x) ((x) || (x))
template <typename T> void foo(T x) {
for (unsigned I = 0; I < 10; ++I) { BAR(I); }
}
int main() {
foo<int>(0);
foo<float>(0);
return 0;
}
EOF
Compiling with coverage enabled
===============================
To compile code with coverage enabled, pass ``-fprofile-instr-generate
-fcoverage-mapping`` to the compiler:
.. code-block:: console
# Step 1: Compile with coverage enabled.
% clang++ -fprofile-instr-generate -fcoverage-mapping foo.cc -o foo
Note that linking together code with and without coverage instrumentation is
supported. Uninstrumented code simply won't be accounted for in reports.
Running the instrumented program
================================
The next step is to run the instrumented program. When the program exits it
will write a **raw profile** to the path specified by the ``LLVM_PROFILE_FILE``
environment variable. If that variable does not exist, the profile is written
to ``default.profraw`` in the current directory of the program. If
``LLVM_PROFILE_FILE`` contains a path to a non-existent directory, the missing
directory structure will be created. Additionally, the following special
**pattern strings** are rewritten:
* "%p" expands out to the process ID.
* "%h" expands out to the hostname of the machine running the program.
* "%Nm" expands out to the instrumented binary's signature. When this pattern
is specified, the runtime creates a pool of N raw profiles which are used for
on-line profile merging. The runtime takes care of selecting a raw profile
from the pool, locking it, and updating it before the program exits. If N is
not specified (i.e the pattern is "%m"), it's assumed that ``N = 1``. N must
be between 1 and 9. The merge pool specifier can only occur once per filename
pattern.
[profile] Add a mode to continuously sync counter updates to a file Add support for continuously syncing profile counter updates to a file. The motivation for this is that programs do not always exit cleanly. On iOS, for example, programs are usually killed via a signal from the OS. Running atexit() handlers after catching a signal is unreliable, so some method for progressively writing out profile data is necessary. The approach taken here is to mmap() the `__llvm_prf_cnts` section onto a raw profile. To do this, the linker must page-align the counter and data sections, and the runtime must ensure that counters are mapped to a page-aligned offset within a raw profile. Continuous mode is (for the moment) incompatible with the online merging mode. This limitation is lifted in https://reviews.llvm.org/D69586. Continuous mode is also (for the moment) incompatible with value profiling, as I'm not sure whether there is interest in this and the implementation may be tricky. As I have not been able to test extensively on non-Darwin platforms, only Darwin support is included for the moment. However, continuous mode may "just work" without modification on Linux and some UNIX-likes. AIUI the default value for the GNU linker's `--section-alignment` flag is set to the page size on many systems. This appears to be true for LLD as well, as its `no_nmagic` option is on by default. Continuous mode will not "just work" on Fuchsia or Windows, as it's not possible to mmap() a section on these platforms. There is a proposal to add a layer of indirection to the profile instrumentation to support these platforms. rdar://54210980 Differential Revision: https://reviews.llvm.org/D68351
2019-09-20 02:56:43 +08:00
* "%c" expands out to nothing, but enables a mode in which profile counter
updates are continuously synced to a file. This means that if the
instrumented program crashes, or is killed by a signal, perfect coverage
information can still be recovered. Continuous mode does not support value
profiling for PGO, and is only supported on Darwin at the moment. Support for
Linux may be mostly complete but requires testing, and support for
Fuchsia/Windows may require more extensive changes: please get involved if
you are interested in porting this feature.
[profile] Add a mode to continuously sync counter updates to a file Add support for continuously syncing profile counter updates to a file. The motivation for this is that programs do not always exit cleanly. On iOS, for example, programs are usually killed via a signal from the OS. Running atexit() handlers after catching a signal is unreliable, so some method for progressively writing out profile data is necessary. The approach taken here is to mmap() the `__llvm_prf_cnts` section onto a raw profile. To do this, the linker must page-align the counter and data sections, and the runtime must ensure that counters are mapped to a page-aligned offset within a raw profile. Continuous mode is (for the moment) incompatible with the online merging mode. This limitation is lifted in https://reviews.llvm.org/D69586. Continuous mode is also (for the moment) incompatible with value profiling, as I'm not sure whether there is interest in this and the implementation may be tricky. As I have not been able to test extensively on non-Darwin platforms, only Darwin support is included for the moment. However, continuous mode may "just work" without modification on Linux and some UNIX-likes. AIUI the default value for the GNU linker's `--section-alignment` flag is set to the page size on many systems. This appears to be true for LLD as well, as its `no_nmagic` option is on by default. Continuous mode will not "just work" on Fuchsia or Windows, as it's not possible to mmap() a section on these platforms. There is a proposal to add a layer of indirection to the profile instrumentation to support these platforms. rdar://54210980 Differential Revision: https://reviews.llvm.org/D68351
2019-09-20 02:56:43 +08:00
.. code-block:: console
# Step 2: Run the program.
% LLVM_PROFILE_FILE="foo.profraw" ./foo
Creating coverage reports
=========================
Raw profiles have to be **indexed** before they can be used to generate
coverage reports. This is done using the "merge" tool in ``llvm-profdata``
(which can combine multiple raw profiles and index them at the same time):
.. code-block:: console
# Step 3(a): Index the raw profile.
% llvm-profdata merge -sparse foo.profraw -o foo.profdata
There are multiple different ways to render coverage reports. The simplest
option is to generate a line-oriented report:
.. code-block:: console
# Step 3(b): Create a line-oriented coverage report.
% llvm-cov show ./foo -instr-profile=foo.profdata
This report includes a summary view as well as dedicated sub-views for
templated functions and their instantiations. For our example program, we get
distinct views for ``foo<int>(...)`` and ``foo<float>(...)``. If
``-show-line-counts-or-regions`` is enabled, ``llvm-cov`` displays sub-line
region counts (even in macro expansions):
.. code-block:: none
1| 20|#define BAR(x) ((x) || (x))
^20 ^2
2| 2|template <typename T> void foo(T x) {
3| 22| for (unsigned I = 0; I < 10; ++I) { BAR(I); }
^22 ^20 ^20^20
4| 2|}
------------------
| void foo<int>(int):
| 2| 1|template <typename T> void foo(T x) {
| 3| 11| for (unsigned I = 0; I < 10; ++I) { BAR(I); }
| ^11 ^10 ^10^10
| 4| 1|}
------------------
| void foo<float>(int):
| 2| 1|template <typename T> void foo(T x) {
| 3| 11| for (unsigned I = 0; I < 10; ++I) { BAR(I); }
| ^11 ^10 ^10^10
| 4| 1|}
------------------
To generate a file-level summary of coverage statistics instead of a
line-oriented report, try:
.. code-block:: console
# Step 3(c): Create a coverage summary.
% llvm-cov report ./foo -instr-profile=foo.profdata
Filename Regions Missed Regions Cover Functions Missed Functions Executed Lines Missed Lines Cover
--------------------------------------------------------------------------------------------------------------------------------------
/tmp/foo.cc 13 0 100.00% 3 0 100.00% 13 0 100.00%
--------------------------------------------------------------------------------------------------------------------------------------
TOTAL 13 0 100.00% 3 0 100.00% 13 0 100.00%
The ``llvm-cov`` tool supports specifying a custom demangler, writing out
reports in a directory structure, and generating html reports. For the full
list of options, please refer to the `command guide
<https://llvm.org/docs/CommandGuide/llvm-cov.html>`_.
A few final notes:
* The ``-sparse`` flag is optional but can result in dramatically smaller
indexed profiles. This option should not be used if the indexed profile will
be reused for PGO.
* Raw profiles can be discarded after they are indexed. Advanced use of the
profile runtime library allows an instrumented program to merge profiling
information directly into an existing raw profile on disk. The details are
out of scope.
* The ``llvm-profdata`` tool can be used to merge together multiple raw or
indexed profiles. To combine profiling data from multiple runs of a program,
try e.g:
.. code-block:: console
% llvm-profdata merge -sparse foo1.profraw foo2.profdata -o foo3.profdata
Exporting coverage data
=======================
Coverage data can be exported into JSON using the ``llvm-cov export``
sub-command. There is a comprehensive reference which defines the structure of
the exported data at a high level in the llvm-cov source code.
Interpreting reports
====================
There are four statistics tracked in a coverage summary:
* Function coverage is the percentage of functions which have been executed at
least once. A function is considered to be executed if any of its
instantiations are executed.
* Instantiation coverage is the percentage of function instantiations which
have been executed at least once. Template functions and static inline
functions from headers are two kinds of functions which may have multiple
instantiations.
* Line coverage is the percentage of code lines which have been executed at
least once. Only executable lines within function bodies are considered to be
code lines.
* Region coverage is the percentage of code regions which have been executed at
least once. A code region may span multiple lines (e.g in a large function
body with no control flow). However, it's also possible for a single line to
contain multiple code regions (e.g in "return x || y && z").
Of these four statistics, function coverage is usually the least granular while
region coverage is the most granular. The project-wide totals for each
statistic are listed in the summary.
Format compatibility guarantees
===============================
* There are no backwards or forwards compatibility guarantees for the raw
profile format. Raw profiles may be dependent on the specific compiler
revision used to generate them. It's inadvisable to store raw profiles for
long periods of time.
* Tools must retain **backwards** compatibility with indexed profile formats.
These formats are not forwards-compatible: i.e, a tool which uses format
version X will not be able to understand format version (X+k).
* Tools must also retain **backwards** compatibility with the format of the
coverage mappings emitted into instrumented binaries. These formats are not
forwards-compatible.
* The JSON coverage export format has a (major, minor, patch) version triple.
Only a major version increment indicates a backwards-incompatible change. A
minor version increment is for added functionality, and patch version
increments are for bugfixes.
Using the profiling runtime without static initializers
=======================================================
By default the compiler runtime uses a static initializer to determine the
profile output path and to register a writer function. To collect profiles
without using static initializers, do this manually:
* Export a ``int __llvm_profile_runtime`` symbol from each instrumented shared
library and executable. When the linker finds a definition of this symbol, it
knows to skip loading the object which contains the profiling runtime's
static initializer.
* Forward-declare ``void __llvm_profile_initialize_file(void)`` and call it
once from each instrumented executable. This function parses
``LLVM_PROFILE_FILE``, sets the output path, and truncates any existing files
at that path. To get the same behavior without truncating existing files,
pass a filename pattern string to ``void __llvm_profile_set_filename(char
*)``. These calls can be placed anywhere so long as they precede all calls
to ``__llvm_profile_write_file``.
* Forward-declare ``int __llvm_profile_write_file(void)`` and call it to write
out a profile. This function returns 0 when it succeeds, and a non-zero value
otherwise. Calling this function multiple times appends profile data to an
existing on-disk raw profile.
In C++ files, declare these as ``extern "C"``.
Collecting coverage reports for the llvm project
================================================
To prepare a coverage report for llvm (and any of its sub-projects), add
``-DLLVM_BUILD_INSTRUMENTED_COVERAGE=On`` to the cmake configuration. Raw
profiles will be written to ``$BUILD_DIR/profiles/``. To prepare an html
report, run ``llvm/utils/prepare-code-coverage-artifact.py``.
To specify an alternate directory for raw profiles, use
``-DLLVM_PROFILE_DATA_DIR``. To change the size of the profile merge pool, use
``-DLLVM_PROFILE_MERGE_POOL_SIZE``.
Drawbacks and limitations
=========================
* Prior to version 2.26, the GNU binutils BFD linker is not able link programs
compiled with ``-fcoverage-mapping`` in its ``--gc-sections`` mode. Possible
workarounds include disabling ``--gc-sections``, upgrading to a newer version
of BFD, or using the Gold linker.
* Code coverage does not handle unpredictable changes in control flow or stack
unwinding in the presence of exceptions precisely. Consider the following
function:
.. code-block:: cpp
int f() {
may_throw();
return 0;
}
If the call to ``may_throw()`` propagates an exception into ``f``, the code
coverage tool may mark the ``return`` statement as executed even though it is
not. A call to ``longjmp()`` can have similar effects.