Commit Graph

159 Commits

Author SHA1 Message Date
Keith Smiley e972e49b11 [llvm-cov] Add error for invalid -path-equivalence format
Differential Revision: https://reviews.llvm.org/D109042
2021-09-10 18:34:37 -07:00
Petr Hosek 8280ece0c9 [Coverage] Support overriding compilation directory
When making compilation relocatable, for example in distributed
compilation scenarios, we want to set compilation dir to a relative
value like `.` but this presents a problem when generating reports
because if the file path is relative as well, for example `..`, you
may end up writing files outside of the output directory.

This change introduces a flag that allows overriding the compilation
directory that's stored inside the profile with a different value that
is absolute.

Differential Revision: https://reviews.llvm.org/D100232
2021-05-11 15:26:45 -07:00
Zequan Wu dd388ba3e0 [llvm-cov] Check path emptyness in path-equivalence after removing dots. 2021-03-24 17:54:38 -07:00
Choongwoo Han 9d8a3e75b4 [llvm-cov] Cache file status information
Currently, getSourceFile accesses file system to check if two paths are
the same file with a thread lock, which is a huge performance bottleneck
in some cases. Currently, it's accessing file system size(files) * size(files) times.

Thus, cache file status information, which reduces file system access to size(files) times.

When I tested it with two binaries and 16 cpu cores,
it saved over 70% of time.

Binary 1: 56 secs -> 3 secs
Binary 2: 17 hours -> 4 hours

Differential Revision: https://reviews.llvm.org/D97061
2021-03-03 10:04:07 -08:00
Alan Phipps 9f2967bcfe [Coverage] Add support for Branch Coverage in LLVM Source-Based Code Coverage
This is an enhancement to LLVM Source-Based Code Coverage in clang to track how
many times individual branch-generating conditions are taken (evaluate to TRUE)
and not taken (evaluate to FALSE).  Individual conditions may comprise larger
boolean expressions using boolean logical operators.  This functionality is
very similar to what is supported by GCOV except that it is very closely
anchored to the ASTs.

Differential Revision: https://reviews.llvm.org/D84467
2021-01-05 09:51:51 -06:00
Kazu Hirata b676f2fee1 [llvm-cov, llvm-symbolizer] Use llvm::erase_if (NFC) 2020-12-26 12:06:27 -08:00
Zequan Wu c75a0a1e9d [llvm-cov] Fix missing slash in -path-equivalence 2020-11-06 14:54:11 -08:00
Zequan Wu e92eeaf3c2 [llvm-cov] don't include all source files when provided source files are filtered out
When all provided source files are filtered out either due to `--ignore-filename-regex` or not part of binary, don't generate coverage reults for all source files. Because if users want to generate coverage results for all source files, they don't even need to provid selected source files or `--ignore-filename-regex`.

Differential Revision: https://reviews.llvm.org/D89359
2020-10-23 19:32:16 -07:00
Jeremy Morse 0a7f41739f Revert "[llvm-cov] don't include all source files when provided source files are filtered out"
This reverts commit c2bd20ef652 and the follow up fix 16605bba6fb.

The tools/llvm-cov/warnings.h continues to fail on Windows platforms even
after the follow up, for example on the llvm-clang-win-x-armv7l builder:

    http://lab.llvm.org:8011/#/builders/60/builds/94
2020-10-16 13:46:12 +01:00
Zequan Wu cd3090819c [llvm-cov] don't include all source files when provided source files are filtered out
When all provided source files are filtered out either due to `--ignore-filename-regex` or not part of binary, don't generate coverage reults for all source files. Because if users want to generate coverage results for all source files, they don't even need to provid selected source files or `--ignore-filename-regex`.

Differential Revision: https://reviews.llvm.org/D89359
2020-10-15 18:11:01 -07:00
Vedant Kumar 3c731ba5f1 [llvm-cov] Allow commas in filenames passed to `-object` flag
Currently, -object takes a comma separated list of objects as an
argument, which prevents it working with path names that contain a
comma. Drop comma-separated support, which requires to set pass the
-object flag multiple times to set multiple objects.

Patch by Andrew Gallagher!

Differential Revision: https://reviews.llvm.org/D87003
2020-09-18 13:46:29 -07:00
Simon Pilgrim 6022efb0e9 CoverageFilters.h - reduce unnecessary includes to forward declarations. NFC. 2020-05-27 16:57:47 +01:00
Alexandre Ganea 0e13a0331f [llvm-cov] Prevent llvm-cov from using too many threads
As reported here: https://reviews.llvm.org/D75153#1987272

Before, each instance of llvm-cov was creating one thread per hardware core, which wasn't needed probably because the number of inputs were small. This was probably causing a thread rlimit issue on large core count systems.

After this patch, the previous behavior is restored (to what was before rG8404aeb5):

If --num-threads is not specified, we create one thread per input, up to num.cores.
When specified, --num-threads indicates any number of threads, with no upper limit.

Differential Revision: https://reviews.llvm.org/D78408
2020-04-24 15:28:25 -04:00
Eli Friedman 9eb1b41811 [llvm-cov] Improve error message for missing profdata
I got a report recently that a user was having trouble interpreting the
meaning of the error message.  Hopefully this is more readable; produces
something like the following:

error: No such file or directory: Could not read profile data!

Differential Revision: https://reviews.llvm.org/D76796
2020-03-30 12:54:07 -07:00
Alexandre Ganea 8404aeb56a [Support] On Windows, ensure hardware_concurrency() extends to all CPU sockets and all NUMA groups
The goal of this patch is to maximize CPU utilization on multi-socket or high core count systems, so that parallel computations such as LLD/ThinLTO can use all hardware threads in the system. Before this patch, on Windows, a maximum of 64 hardware threads could be used at most, in some cases dispatched only on one CPU socket.

== Background ==
Windows doesn't have a flat cpu_set_t like Linux. Instead, it projects hardware CPUs (or NUMA nodes) to applications through a concept of "processor groups". A "processor" is the smallest unit of execution on a CPU, that is, an hyper-thread if SMT is active; a core otherwise. There's a limit of 32-bit processors on older 32-bit versions of Windows, which later was raised to 64-processors with 64-bit versions of Windows. This limit comes from the affinity mask, which historically is represented by the sizeof(void*). Consequently, the concept of "processor groups" was introduced for dealing with systems with more than 64 hyper-threads.

By default, the Windows OS assigns only one "processor group" to each starting application, in a round-robin manner. If the application wants to use more processors, it needs to programmatically enable it, by assigning threads to other "processor groups". This also means that affinity cannot cross "processor group" boundaries; one can only specify a "preferred" group on start-up, but the application is free to allocate more groups if it wants to.

This creates a peculiar situation, where newer CPUs like the AMD EPYC 7702P (64-cores, 128-hyperthreads) are projected by the OS as two (2) "processor groups". This means that by default, an application can only use half of the cores. This situation could only get worse in the years to come, as dies with more cores will appear on the market.

== The problem ==
The heavyweight_hardware_concurrency() API was introduced so that only *one hardware thread per core* was used. Once that API returns, that original intention is lost, only the number of threads is retained. Consider a situation, on Windows, where the system has 2 CPU sockets, 18 cores each, each core having 2 hyper-threads, for a total of 72 hyper-threads. Both heavyweight_hardware_concurrency() and hardware_concurrency() currently return 36, because on Windows they are simply wrappers over std:🧵:hardware_concurrency() -- which can only return processors from the current "processor group".

== The changes in this patch ==
To solve this situation, we capture (and retain) the initial intention until the point of usage, through a new ThreadPoolStrategy class. The number of threads to use is deferred as late as possible, until the moment where the std::threads are created (ThreadPool in the case of ThinLTO).

When using hardware_concurrency(), setting ThreadCount to 0 now means to use all the possible hardware CPU (SMT) threads. Providing a ThreadCount above to the maximum number of threads will have no effect, the maximum will be used instead.
The heavyweight_hardware_concurrency() is similar to hardware_concurrency(), except that only one thread per hardware *core* will be used.

When LLVM_ENABLE_THREADS is OFF, the threading APIs will always return 1, to ensure any caller loops will be exercised at least once.

Differential Revision: https://reviews.llvm.org/D71775
2020-02-14 10:24:22 -05:00
Benjamin Kramer bd31243a34 Fix more implicit conversions. Getting closer to having clang working with gcc 5 again 2020-01-29 02:57:59 +01:00
Benjamin Kramer adcd026838 Make llvm::StringRef to std::string conversions explicit.
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.

This doesn't actually modify StringRef yet, I'll do that in a follow-up.
2020-01-28 23:25:25 +01:00
Ilya Biryukov aa981c1802 Reland 9f3fdb0d7fab: [Driver] Use VFS to check if sanitizer blacklists exist
With updates to various LLVM tools that use SpecialCastList.

It was tempting to use RealFileSystem as the default, but that makes it
too easy to accidentally forget passing VFS in clang code.
2019-11-21 11:56:09 +01:00
Vlad Tsyrklevich 8d24d72f7f Revert "[llvm-cov] Add option to whitelist filenames"
This reverts commit bfed824b57, the
included test fails on many bots including the sanitier bots, e.g. in
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/36140
2019-10-29 22:38:38 -07:00
Vedant Kumar bfed824b57 [llvm-cov] Add option to whitelist filenames
Add the `-whitelist-filename-regex` option to restrict coverage
reporting to file paths that match a whitelist regex.

Patch by Michael Daniels!

rdar://56720320
2019-10-29 18:26:33 -07:00
Jonas Devlieghere 0eaee545ee [llvm] Migrate llvm::make_unique to std::make_unique
Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.

llvm-svn: 369013
2019-08-15 15:54:37 +00:00
Max Moroz a80d9ce5cf Speeding up llvm-cov export with multithreaded renderFiles implementation.
Summary:
CoverageExporterJson::renderFiles accounts for most of the execution time given a large profdata file with multiple binaries.

Proposed solution is to generate JSON for each file in parallel and sort at the end to preserve deterministic output. Also added flags to skip generating parts of the output to trim the output size.

Patch by Sajjad Mirza (@sajjadm).

Reviewers: Dor1s, vsk

Reviewed By: Dor1s, vsk

Subscribers: liaoyuke, mgrang, jdoerfert, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D59277

llvm-svn: 356178
2019-03-14 17:49:27 +00:00
Chandler Carruth 2946cd7010 Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636
2019-01-19 08:50:56 +00:00
Jordan Rupprecht 16a0de2e55 [binutils] NFC: fix clang-tidy warning: use empty() instead of size() == 0
llvm-svn: 349710
2018-12-20 00:57:06 +00:00
Max Moroz 0d12713c2b [llvm-cov] Remove "default:" label in the switch covering all enum values.
Summary:
Fixing the build breakage:
http://lab.llvm.org:8011/builders/lld-x86_64-darwin13/builds/27309

Reviewers: vsk, allevato, Dor1s

Reviewed By: Dor1s

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D54323

llvm-svn: 346514
2018-11-09 16:44:36 +00:00
Max Moroz b2091c930b [llvm-cov] Add lcov tracefile export format.
Summary:
lcov tracefiles are used by various coverage reporting tools and build
systems (e.g., Bazel). It is a simple text-based format to parse and
more convenient to use than the JSON export format, which needs
additional processing to map regions/segments back to line numbers.

It's a little unfortunate that "text" format is now overloaded to refer
specifically to JSON for export, but I wanted to avoid making any
breaking changes to the UI of the llvm-cov tool at this time.

Patch by Tony Allevato (@allevato).

Reviewers: Dor1s, vsk

Reviewed By: Dor1s, vsk

Subscribers: mgorny, llvm-commits

Differential Revision: https://reviews.llvm.org/D54266

llvm-svn: 346506
2018-11-09 16:10:44 +00:00
Sam McCall 79c995c0cc [Support] Listing a directory containing dangling symlinks is not an error.
Summary:
Reporting this as an error required stat()ing every file, as well as seeming
semantically questionable.

Reviewers: vsk, bkramer

Subscribers: mgrang, kristina, llvm-commits, liaoyuke

Differential Revision: https://reviews.llvm.org/D52648

llvm-svn: 343460
2018-10-01 12:17:05 +00:00
Vedant Kumar e302fc597a [Coverage] Delete getCounterMismatches, it's dead code (NFC)
Exactly one counted region is inserted into a function record for every
region in a coverage mapping.

llvm-svn: 339193
2018-08-07 22:25:22 +00:00
Zachary Turner 08426e1f9f Refactor ExecuteAndWait to take StringRefs.
This simplifies some code which had StringRefs to begin with, and
makes other code more complicated which had const char* to begin
with.

In the end, I think this makes for a more idiomatic and platform
agnostic API.  Not all platforms launch process with null terminated
c-string arrays for the environment pointer and argv, but the api
was designed that way because it allowed easy pass-through for
posix-based platforms.  There's a little additional overhead now
since on posix based platforms we'll be takign StringRefs which
were constructed from null terminated strings and then copying
them to null terminate them again, but from a readability and
usability standpoint of the API user, I think this API signature
is strictly better.

llvm-svn: 334518
2018-06-12 17:43:52 +00:00
Adrian Prantl 5f8f34e459 Remove \brief commands from doxygen comments.
We've been running doxygen with the autobrief option for a couple of
years now. This makes the \brief markers into our comments
redundant. Since they are a visual distraction and we don't want to
encourage more \brief markers in new code either, this patch removes
them all.

Patch produced by

  for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done

Differential Revision: https://reviews.llvm.org/D46290

llvm-svn: 331272
2018-05-01 15:54:18 +00:00
Max Moroz 4220f89107 [llvm-cov] Implement -ignore-filename-regex= option for excluding source files.
Summary:
The option is helpful for large projects where it's not feasible to specify sources which
user would like to see in the report. Instead, it allows to black-list specific sources via
regular expressions (e.g. now it's possible to skip all files that have "test" in its name).

This also partially fixes https://bugs.llvm.org/show_bug.cgi?id=34277

Reviewers: vsk, morehouse, liaoyuke

Reviewed By: vsk

Subscribers: kcc, mgorny, llvm-commits

Differential Revision: https://reviews.llvm.org/D43907

llvm-svn: 329581
2018-04-09 15:20:35 +00:00
Max Moroz 650fd6c31c [llvm-cov] Prevent llvm-cov from hanging when a symblink doesn't exist.
Summary:
Previous code hangs indefinitely when trying to iterate through a
symbol link file that points to an non-exist directory. This change
fixes the bug to make the addCollectedPath function exit ealier and
print out correct warning messages.

Patch by Yuke Liao (@liaoyuke).

Reviewers: Dor1s, vsk

Reviewed By: vsk

Subscribers: bruno, mgrang, llvm-commits

Differential Revision: https://reviews.llvm.org/D44960

llvm-svn: 329338
2018-04-05 19:43:24 +00:00
Max Moroz cc254ba4a7 [llvm-cov] Multi-threaded implementation of prepareFileReports method.
Summary:
Local testing has demonstrated a great speed improvement, compare the following:

1) Existing version:
```
$ time llvm-cov show -format=html -output-dir=report -instr-profile=... ...
The tool has been launched:                            00:00:00
Loading coverage data:                                 00:00:00
Get unique source files:                               00:00:33
Creating an index out of the source files:             00:00:34
Going into prepareFileReports:                         00:00:34
Going to emit summary information for each file:       00:28:55 <-- 28:21 min!
Going to emit links to files with no function:         00:28:55
Launching 32 threads for generating HTML files:        00:28:55

real  37m43.651s
user  112m5.540s
sys   7m39.872s
```

2) Multi-threaded version with 32 CPUs:
```
$ time llvm-cov show -format=html -output-dir=report -instr-profile=... ...
The tool has been launched:                            00:00:00
Loading coverage data:                                 00:00:00
Get unique source files:                               00:00:38
Creating an index out of the source files:             00:00:40
Going into prepareFileReports:                         00:00:40
Preparing file reports using 32 threads:               00:00:40
# Creating thread tasks for the following number of files: 16422
Going to emit summary information for each file:       00:01:57 <-- 1:17 min!
Going to emit links to files with no function:         00:01:58
Launching 32 threads for generating HTML files:        00:01:58

real  11m2.044s
user  134m48.124s
sys   7m53.388s
```

Reviewers: vsk, morehouse

Reviewed By: vsk

Subscribers: Dor1s, llvm-commits, kcc

Differential Revision: https://reviews.llvm.org/D41206

llvm-svn: 321871
2018-01-05 16:15:07 +00:00
Max Moroz 1ef3a778ac [llvm-cov] Refactor "export" command implementation and add support for SOURCES.
Summary: Define an interface for Exporter + split JSON exporter into .h and .cpp.

Reviewers: vsk, morehouse

Reviewed By: vsk

Subscribers: llvm-commits, Dor1s, kcc

Differential Revision: https://reviews.llvm.org/D41600

llvm-svn: 321815
2018-01-04 19:33:29 +00:00
Benjamin Kramer 3a13ed60ba Avoid int to string conversion in Twine or raw_ostream contexts.
Some output changes from uppercase hex to lowercase hex, no other functionality change intended.

llvm-svn: 321526
2017-12-28 16:58:54 +00:00
Max Moroz fe4d904917 [llvm-cov] Add an option for "export" command to emit only file summary data.
Summary:
That allows to get the same data as produced by "llvm-cov report",
but in JSON format, which is better for further processing by end users.

Reviewers: vsk

Reviewed By: vsk

Differential Revision: https://reviews.llvm.org/D41085

llvm-svn: 320435
2017-12-11 23:17:46 +00:00
Max Moroz 4a4bfa4e27 [llvm-cov] Generate "report" for given source paths if sources are specified.
Summary:
Documentation says that user can specify sources for both "show" and
"report" commands. "Show" command respects specified sources, but "report" does
not. It is useful to have both "show" and "report" generated for specified
sources. Also added tests to for both commands with sources specified.

Reviewers: vsk, kcc

Reviewed By: vsk

Differential Revision: https://reviews.llvm.org/D38860

llvm-svn: 315685
2017-10-13 14:44:51 +00:00
Sean Eveson ea9dceed77 [llvm-cov] Fix showing title when filtering and not outputting to a directory
Differential Revision: https://reviews.llvm.org/D38507

llvm-svn: 314885
2017-10-04 08:54:37 +00:00
Sean Eveson fa8ef35e78 [llvm-cov] Create directory structure when filtering using -name*= options
Before this change using any of the -name*= command line options with an output
directory would result in a single file (functions.txt/functions.html)
containing the coverage for those specific functions. Now you get the same
directory structure as when not using any -name*= options.

Differential Revision: https://reviews.llvm.org/D38280

llvm-svn: 314396
2017-09-28 10:07:30 +00:00
Sean Eveson 1439fa6236 Revert "[llvm-cov] Create directory structure when filtering using -name*= options"
Test failures.

llvm-svn: 314314
2017-09-27 16:20:07 +00:00
Sean Eveson 51b817479b [llvm-cov] Create directory structure when filtering using -name*= options
Before this change using any of the -name*= command line options with an output
directory would result in a single file (functions.txt/functions.html)
containing the coverage for those specific functions. Now you get the same
directory structure as when not using any -name*= options.

Differential Revision: https://reviews.llvm.org/D38280

llvm-svn: 314310
2017-09-27 15:37:40 +00:00
Vedant Kumar feb3f5272f [llvm-cov] Warn if -show-functions is used without query files
llvm-cov's report mode does not print any output when -show-functions is
specified and no source files are specified. This can be surprising, so
the tool should at least print out an error message when this happens.

rdar://problem/34636859

llvm-svn: 314175
2017-09-25 23:10:03 +00:00
Reid Kleckner 3fc649cb76 [Support] Rename tool_output_file to ToolOutputFile, NFC
This class isn't similar to anything from the STL, so it shouldn't use
the STL naming conventions.

llvm-svn: 314050
2017-09-23 01:03:17 +00:00
Vedant Kumar 18dd9e88ed [llvm-cov] Improve error messaging for function mismatches
Passing "-dump" to llvm-cov will now print more detailed information
about function hash and counter mismatches. This should make it easier
to debug *.profdata files which contain incorrect records, and to debug
other scenarios where coverage goes missing due to mismatch issues.

llvm-svn: 313853
2017-09-21 01:11:30 +00:00
Vedant Kumar b7fdaf2cd4 [llvm-cov] Make report metrics agree with line exec counts, fixes PR34615
Use the same logic as the line-oriented coverage view to determine the
number of covered lines in a function.

Fixes llvm.org/PR34615.

llvm-svn: 313604
2017-09-19 02:00:12 +00:00
Alexander Kornienko 208eecd57f Convenience/safety fix for llvm::sys::Execute(And|No)Wait
Summary:
Change the type of the Redirects parameter of llvm::sys::ExecuteAndWait,
ExecuteNoWait and other APIs that wrap them from `const StringRef **` to
`ArrayRef<Optional<StringRef>>`, which is safer and simplifies the use of these
APIs (no more local StringRef variables just to get a pointer to).

Corresponding clang changes will be posted as a separate patch.

Reviewers: bkramer

Reviewed By: bkramer

Subscribers: vsk, llvm-commits

Differential Revision: https://reviews.llvm.org/D37563

llvm-svn: 313155
2017-09-13 17:03:37 +00:00
Eli Friedman 50479f60c4 [llvm-cov] Allow hiding instantiation/region coverage from summary tables
Region coverage is difficult to explain without going deep into how
coverage is implemented. Instantiation coverage is easier to explain,
but probably not useful in most cases (templates don't exist in C, and
most C++ code contains relatively few templates).

This patch adds the options "-show-region-summary" and
"-show-instantiation-summary" to allow hiding those columns.
"-show-instantiation-summary" is turned off by default.

llvm-svn: 312969
2017-09-11 22:56:20 +00:00
Vedant Kumar 72c3a11488 [llvm-cov] Fix a lifetime issue
This fixes an issue where a std::string was moved to a constructor
which accepted a StringRef.

llvm-svn: 312816
2017-09-08 18:44:49 +00:00
Vedant Kumar 933b37f99f [llvm-cov] Unify region marker placement between text/html modes
Make sure that the text and html emitters always emit the same set of
region markers, and avoid emitting redundant markers for line segments
which don't end on the line they start on.

This is related to D35925, and depends on D36014

Differential Revision: https://reviews.llvm.org/D36020

llvm-svn: 312813
2017-09-08 18:44:46 +00:00
Sean Eveson e15300ecf5 [llvm-cov] Read in function names for filtering from a text file.
Summary: Add a -name-whitelist option, which behaves in the same way as -name, but it reads in multiple function names from the given input file(s).

Reviewers: vsk

Reviewed By: vsk

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D37111

llvm-svn: 312227
2017-08-31 09:11:31 +00:00