Commit Graph

475 Commits

Author SHA1 Message Date
Hans Wennborg 546918cbb4 Revert "[compiler-rt] Add a critical section when flushing gcov counters"
See discussion on PR44792.

This reverts commit 02ce9d8ef5.

It also reverts the follow-up commits
8f46269f0 "[profile] Don't dump counters when forking and don't reset when calling exec** functions"
62c7d8402 "[profile] gcov_mutex must be static"
2020-02-26 13:27:44 +01:00
Calixte Denizet 62c7d84026 [profile] gcov_mutex must be static
Summary: Forget static keyword for gcov_mutex in https://reviews.llvm.org/D74953 and that causes test failure on mac.

Reviewers: erik.pilkington, vsk

Reviewed By: vsk

Subscribers: vsk, dexonsmith, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D75080
2020-02-25 10:54:52 +01:00
Calixte Denizet 8f46269f0c [profile] Don't dump counters when forking and don't reset when calling exec** functions
Summary:
There is no need to write out gcdas when forking because we can just reset the counters in the parent process.
Let say a counter is N before the fork, then fork and this counter is set to 0 in the child process.
In the parent process, the counter is incremented by P and in the child process it's incremented by C.
When dump is ran at exit, parent process will dump N+P for the given counter and the child process will dump 0+C, so when the gcdas are merged the resulting counter will be N+P+C.
About exec** functions, since the current process is replaced by an another one there is no need to reset the counters but just write out the gcdas since the counters are definitely lost.
To avoid to have lists in a bad state, we just lock them during the fork and the flush (if called explicitely) and lock them when an element is added.

Reviewers: marco-c

Reviewed By: marco-c

Subscribers: hiraditya, cfe-commits, #sanitizers, llvm-commits, sylvestre.ledru

Tags: #clang, #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D74953
2020-02-24 10:38:33 +01:00
Petr Hosek d3db13af7e [profile] Support counter relocation at runtime
This is an alternative to the continous mode that was implemented in
D68351. This mode relies on padding and the ability to mmap a file over
the existing mapping which is generally only available on POSIX systems
and isn't suitable for other platforms.

This change instead introduces the ability to relocate counters at
runtime using a level of indirection. On every counter access, we add a
bias to the counter address. This bias is stored in a symbol that's
provided by the profile runtime and is initially set to zero, meaning no
relocation. The runtime can mmap the profile into memory at abitrary
location, and set bias to the offset between the original and the new
counter location, at which point every subsequent counter access will be
to the new location, which allows updating profile directly akin to the
continous mode.

The advantage of this implementation is that doesn't require any special
OS support. The disadvantage is the extra overhead due to additional
instructions required for each counter access (overhead both in terms of
binary size and performance) plus duplication of counters (i.e. one copy
in the binary itself and another copy that's mmapped).

Differential Revision: https://reviews.llvm.org/D69740
2020-01-17 15:02:23 -08:00
Fangrui Song c5b94ea265 [profile] Support merge pool size >= 10
The executable acquires an advisory record lock (`fcntl(fd, F_SETLKW, *)`) on a profile file.
Merge pool size >= 10 may be beneficial when the concurrency is large.

Also fix a small problem about snprintf. It can cause the filename to be truncated after %m.

Reviewed By: davidxl

Differential Revision: https://reviews.llvm.org/D71970
2020-01-12 00:27:18 -08:00
Calixte Denizet 02ce9d8ef5 [compiler-rt] Add a critical section when flushing gcov counters
Summary:
Counters can be flushed in a multi-threaded context for example when the process is forked in different threads (https://github.com/llvm/llvm-project/blob/master/llvm/lib/Transforms/Instrumentation/GCOVProfiling.cpp#L632-L663).
In order to avoid pretty bad things, a critical section is needed around the flush.
We had a lot of crashes in this code in Firefox CI when we switched to clang for linux ccov builds and those crashes disappeared with this patch.

Reviewers: marco-c, froydnj, dmajor, davidxl, vsk

Reviewed By: marco-c, dmajor

Subscribers: ahatanak, froydnj, dmajor, dberris, jfb, #sanitizers, llvm-commits, sylvestre.ledru

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D70910
2019-12-12 09:23:32 +01:00
Vedant Kumar 5a486e0f15 [profile] Avoid allocating a page on the stack, NFC
When writing out a profile, avoid allocating a page on the stack for the
purpose of writing out zeroes, as some embedded environments do not have
enough stack space to accomodate this.

Instead, use a small, fixed-size zero buffer that can be written
repeatedly.

For a synthetic file with >100,000 functions, I did not measure a
significant difference in profile write times. We are removing a
page-length zero-fill `memset()` in favor of several smaller buffered
`fwrite()` calls: in practice, I am not sure there is much of a
difference. The performance impact is only expected to affect the
continuous sync mode (%c) -- zero padding is less than 8 bytes in all
other cases.

rdar://57810014

Differential Revision: https://reviews.llvm.org/D71323
2019-12-11 10:04:00 -08:00
Akira Hatanaka 78a7af456d Revert "[compiler-rt] Add a critical section when flushing gcov counters"
This reverts commit 88f5bf77f9 as it broke
green dragon bots.

http://lab.llvm.org:8080/green/job/clang-stage1-RA/4401/
2019-12-09 19:58:26 -08:00
Calixte Denizet 88f5bf77f9 [compiler-rt] Add a critical section when flushing gcov counters
Summary:
Counters can be flushed in a multi-threaded context for example when the process is forked in different threads (https://github.com/llvm/llvm-project/blob/master/llvm/lib/Transforms/Instrumentation/GCOVProfiling.cpp#L632-L663).
In order to avoid pretty bad things, a critical section is needed around the flush.
We had a lot of crashes in this code in Firefox CI when we switched to clang for linux ccov builds and those crashes disappeared with this patch.

Reviewers: marco-c, froydnj, dmajor, davidxl

Reviewed By: marco-c, dmajor

Subscribers: froydnj, dmajor, dberris, jfb, #sanitizers, llvm-commits, sylvestre.ledru

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D70910
2019-12-09 10:39:55 +01:00
Hans Wennborg 900d8a9a3b [profile] Fix file contention causing dropped counts on Windows under -fprofile-generate
See PR43425:
https://bugs.llvm.org/show_bug.cgi?id=43425

When writing profile data on Windows we were opening profile file with
exclusive read/write access.

In case we are trying to write to the file from multiple processes
simultaneously, subsequent calls to CreateFileA would return
INVALID_HANDLE_VALUE.

To fix this, I changed to open without exclusive access and then take a
lock.

Patch by Michael Holman!

Differential revision: https://reviews.llvm.org/D70330
2019-11-27 15:55:13 +01:00
Petr Hosek f35032e03d Reland "[CMake] Support installation of InstrProfData.inc"
This header fragment is useful on its own for any consumer that wants
to use custom instruction profile runtime with the LLVM instrumentation.
The concrete use case is in Fuchsia's kernel where we want to use
instruction profile instrumentation, but we cannot use the compiler-rt
runtime because it's not designed for use in the kernel environment.
This change allows installing this header as part of compiler-rt.

Differential Revision: https://reviews.llvm.org/D64532
2019-11-22 14:09:46 -08:00
Petr Hosek 262b10ba1f Revert "[CMake] Support installation of InstrProfData.inc"
This reverts commit f11bc1776f since it's
failing to build on some bots.
2019-11-22 12:00:23 -08:00
Petr Hosek f11bc1776f [CMake] Support installation of InstrProfData.inc
This header fragment is useful on its own for any consumer that wants
to use custom instruction profile runtime with the LLVM instrumentation.
The concrete use case is in Fuchsia's kernel where we want to use
instruction profile instrumentation, but we cannot use the compiler-rt
runtime because it's not designed for use in the kernel environment.
This change allows installing this header as part of compiler-rt.

Differential Revision: https://reviews.llvm.org/D64532
2019-11-22 11:52:50 -08:00
Vedant Kumar 0d4211f4e7 [profile] Address unused function warnings on Windows after D69586
This '#ifdef's out two functions which are unused on Windows, to prevent
-Wunused-function warnings.
2019-11-19 12:51:12 -08:00
Vedant Kumar 1aacf58819 [profile] Unbreak Fuchsia/Windows after D68351
Continuous mode is not yet supported on Fuchsia/Windows, however an
error should not be reported unless the user attempted to actually
enable continuous mode.
2019-11-19 12:29:57 -08:00
Vedant Kumar 2492b5a125 [profile] Support online merging with continuous sync mode
Make it possible to use online profile merging ("%m" mode) with
continuous sync ("%c" mode).

To implement this, the merged profile is locked in the runtime
initialization step and either a) filled out for the first time or b)
checked for compatibility. Then, the profile can simply be mmap()'d with
MAP_SHARED set. With the mmap() in place, counter updates from every
process which uses an image are mapped onto the same set of physical
pages assigned by the filesystem cache. After the mmap() is set up, the
profile is unlocked.

Differential Revision: https://reviews.llvm.org/D69586
2019-11-18 12:56:58 -08:00
Vedant Kumar e7aab320b1 [profile] Factor out logic for mmap'ing merged profile, NFC
Split out the logic to get the size of a merged profile and to do a
compatibility check. This can be shared with both the continuous+merging
mode implementation, as well as the runtime-allocated counters
implementation planned for Fuchsia.

Lifted out of D69586.

Differential Revision: https://reviews.llvm.org/D70135
2019-11-13 07:52:15 -08:00
Douglas Yung 7ebde1bf67 Add a shim for setenv on PS4 since it does not exist.
A few years back a similar change was made for getenv since neither function is supported on the PS4 platform.

Recently, commit d889d1e added a call to setenv in compiler-rt which was causing linking errors because the symbol was not found. This fixes that issue by putting in a shim similar to how we previously dealt with the lack of getenv.

Differential Revision: https://reviews.llvm.org/D70033
2019-11-12 15:05:45 -08:00
Martin Storsjö ec630194fe [compiler-rt] [profile] Fix building for MinGW after d889d1efef
This commit added use of a Windows API in InstrProfilingPort.h.
When _MSC_VER is defined (for MSVC), windows.h is already included
earlier in the same header (for atomics), but MinGW, the gcc
atomics builtins are used instead. Therefore explicitly include
windows.h here, where the API is used.
2019-11-01 09:20:51 +02:00
Vedant Kumar 197bad5089 [profile] Fifth speculative fix for Android after D68351
Use the printf macros from inttypes.h to sidestep -Wformat issues:

/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:425:14: error: format specifies type 'long long' but the argument has type 'off_t' (aka 'long') [-Werror,-Wformat]
             CurrentFileOffset, PageSize);
             ^~~~~~~~~~~~~~~~~
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/profile/InstrProfilingPort.h:114:50: note: expanded from macro 'PROF_ERR'
  fprintf(stderr, "LLVM Profile Error: " Format, __VA_ARGS__);
                                         ~~~~~~  ^~~~~~~~~~~
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:461:41: error: format specifies type 'unsigned long long' but the argument has type 'uint64_t' (aka 'unsigned long') [-Werror,-Wformat]
        strerror(errno), CountersBegin, PageAlignedCountersLength, Fileno,
                                        ^~~~~~~~~~~~~~~~~~~~~~~~~
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/profile/InstrProfilingPort.h:114:50: note: expanded from macro 'PROF_ERR'
  fprintf(stderr, "LLVM Profile Error: " Format, __VA_ARGS__);
                                         ~~~~~~  ^~~~~~~~~~~
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:462:9: error: format specifies type 'unsigned long long' but the argument has type 'uint64_t' (aka 'unsigned long') [-Werror,-Wformat]
        FileOffsetToCounters);
        ^~~~~~~~~~~~~~~~~~~~
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/profile/InstrProfilingPort.h:114:50: note: expanded from macro 'PROF_ERR'
  fprintf(stderr, "LLVM Profile Error: " Format, __VA_ARGS__);
2019-10-31 17:03:52 -07:00
Vedant Kumar a49795d8ab [profile] Third speculative fix for Windows after D68351
_putenv on Windows takes 1 argument, whereas setenv elsewhere takes 3.
Just treat the two platforms differently.

http://lab.llvm.org:8011/builders/sanitizer-windows/builds/53547
2019-10-31 16:35:58 -07:00
Vedant Kumar 6dfe8e590e [profile] Second speculative fix for Windows
VLAs in C appear to not work on Windows, so use COMPILER_RT_ALLOCA:

C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\lib\profile\InstrProfilingWriter.c(264): error C2057: expected constant expression
C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\lib\profile\InstrProfilingWriter.c(264): error C2466: cannot allocate an array of constant size 0
C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\lib\profile\InstrProfilingWriter.c(264): error C2133: 'Zeroes': unknown size
2019-10-31 16:25:25 -07:00
Vedant Kumar aa7f3a405d [profile] Speculative fix for Windows after D68351
setenv() appears to not be available on Windows:

http://lab.llvm.org:8011/builders/sanitizer-windows/builds/53545/steps/stage%201%20build/logs/stdio
2019-10-31 16:23:39 -07:00
Vedant Kumar d889d1efef [profile] Add a mode to continuously sync counter updates to a file
Add support for continuously syncing profile counter updates to a file.

The motivation for this is that programs do not always exit cleanly. On
iOS, for example, programs are usually killed via a signal from the OS.
Running atexit() handlers after catching a signal is unreliable, so some
method for progressively writing out profile data is necessary.

The approach taken here is to mmap() the `__llvm_prf_cnts` section onto
a raw profile. To do this, the linker must page-align the counter and
data sections, and the runtime must ensure that counters are mapped to a
page-aligned offset within a raw profile.

Continuous mode is (for the moment) incompatible with the online merging
mode. This limitation is lifted in https://reviews.llvm.org/D69586.

Continuous mode is also (for the moment) incompatible with value
profiling, as I'm not sure whether there is interest in this and the
implementation may be tricky.

As I have not been able to test extensively on non-Darwin platforms,
only Darwin support is included for the moment. However, continuous mode
may "just work" without modification on Linux and some UNIX-likes. AIUI
the default value for the GNU linker's `--section-alignment` flag is set
to the page size on many systems. This appears to be true for LLD as
well, as its `no_nmagic` option is on by default. Continuous mode will
not "just work" on Fuchsia or Windows, as it's not possible to mmap() a
section on these platforms. There is a proposal to add a layer of
indirection to the profile instrumentation to support these platforms.

rdar://54210980

Differential Revision: https://reviews.llvm.org/D68351
2019-10-31 16:04:09 -07:00
Vedant Kumar 937241b0d9 [profile] Do not cache __llvm_profile_get_filename result
When the %m filename pattern is used, the filename is unique to each
image, so the cached value is wrong.

It struck me that the full filename isn't something that's recomputed
often, so perhaps it doesn't need to be cached at all. David Li pointed
out we can go further and just hide lprofCurFilename. This may regress
workflows that depend on using the set-filename API to change filenames
across all loaded DSOs, but this is expected to be very rare.

rdar://55137071

Differential Revision: https://reviews.llvm.org/D69137

llvm-svn: 375301
2019-10-18 23:33:40 +00:00
Dmitri Gribenko 57256af307 Revert "clang-misexpect: Profile Guided Validation of Performance Annotations in LLVM"
This reverts commit r371584. It introduced a dependency from compiler-rt
to llvm/include/ADT, which is problematic for multiple reasons.

One is that it is a novel dependency edge, which needs cross-compliation
machinery for llvm/include/ADT (yes, it is true that right now
compiler-rt included only header-only libraries, however, if we allow
compiler-rt to depend on anything from ADT, other libraries will
eventually get used).

Secondly, depending on ADT from compiler-rt exposes ADT symbols from
compiler-rt, which would cause ODR violations when Clang is built with
the profile library.

llvm-svn: 371598
2019-09-11 09:16:17 +00:00
Petr Hosek 394a8ed8f1 clang-misexpect: Profile Guided Validation of Performance Annotations in LLVM
This patch contains the basic functionality for reporting potentially
incorrect usage of __builtin_expect() by comparing the developer's
annotation against a collected PGO profile. A more detailed proposal and
discussion appears on the CFE-dev mailing list
(http://lists.llvm.org/pipermail/cfe-dev/2019-July/062971.html) and a
prototype of the initial frontend changes appear here in D65300

We revised the work in D65300 by moving the misexpect check into the
LLVM backend, and adding support for IR and sampling based profiles, in
addition to frontend instrumentation.

We add new misexpect metadata tags to those instructions directly
influenced by the llvm.expect intrinsic (branch, switch, and select)
when lowering the intrinsics. The misexpect metadata contains
information about the expected target of the intrinsic so that we can
check against the correct PGO counter when emitting diagnostics, and the
compiler's values for the LikelyBranchWeight and UnlikelyBranchWeight.
We use these branch weight values to determine when to emit the
diagnostic to the user.

A future patch should address the comment at the top of
LowerExpectIntrisic.cpp to hoist the LikelyBranchWeight and
UnlikelyBranchWeight values into a shared space that can be accessed
outside of the LowerExpectIntrinsic pass. Once that is done, the
misexpect metadata can be updated to be smaller.

In the long term, it is possible to reconstruct portions of the
misexpect metadata from the existing profile data. However, we have
avoided this to keep the code simple, and because some kind of metadata
tag will be required to identify which branch/switch/select instructions
are influenced by the use of llvm.expect

Patch By: paulkirth
Differential Revision: https://reviews.llvm.org/D66324

llvm-svn: 371584
2019-09-11 01:09:16 +00:00
Nico Weber 98f800dabe compiler-rt: Fix warning if COMPILER_RT_HAS_FCNTL_LCK is 0
Fixes "warning: implicit declaration of function 'flock' is invalid in C99"
for flock().

llvm-svn: 369534
2019-08-21 15:24:25 +00:00
Petr Hosek edf504f503 [Fuchsia] Create the VMO during initialization, not during exit
We want to avoid doing expensive work during atexit since the process
might be terminated before we can publish the VMO and write out the
symbolizer markup, so move the VMO creation to the initialization
phase and only write data during the atexit phase.

Differential Revision: https://reviews.llvm.org/D66323

llvm-svn: 369180
2019-08-17 00:54:22 +00:00
Nico Weber 62a0585530 compiler-rt: Rename .cc file in lib/profile to .cpp
See https://reviews.llvm.org/D58620 for discussion.

Note how the comment in the file already said ".cpp" :)

llvm-svn: 367460
2019-07-31 18:21:08 +00:00
Nico Weber b206c3e3e4 gn build: Add build files for compiler-rt/lib/profile
Differential Revision: https://reviews.llvm.org/D65518

llvm-svn: 367450
2019-07-31 17:15:32 +00:00
Pirama Arumuga Nainar 6caa8da072 [profile] In Android, do not mkdir() dirs in GCOV_PREFIX
Summary:
In Android, attempting to mkdir() or even stat() top-level directories
like /data causes noisy selinux denials.  During whole-system coverage
instrumentation, this causes a deluge of noisy messages that drown out
legitimate selinux denials, that should be audited and fixed.

To avoid this, skip creating any directory in GCOV_PREFIX (thereby
assuming that it exists).

- Android platform ensures that the GCOV_PREFIX used in Android is
created and read/writable by all processes.
- This only affects the Android platform (by checking against
__ANDROID_API_FUTURE__) and for apps built with Clang coverage, the
runtime will still create any non-existent parent directories for the
coverage files.

Reviewers: srhines, davidxl

Subscribers: krytarowski, #sanitizers, danalbert, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D65245

llvm-svn: 367064
2019-07-25 22:10:56 +00:00
Sajjad Mirza 6694b2b36b (Reland with changes) Adding a function for setting coverage output file.
Summary:
User code can open a file on its own and pass it to the runtime, rather than
specifying a name and having the runtime open the file. This supports the use
case where a process cannot open a file on its own but can receive a file
descriptor from another process.

Relanding https://reviews.llvm.org/D62541. The original revision unlocked
the file before calling flush, this revision fixes that.

Reviewers: Dor1s, davidxl

Reviewed By: Dor1s

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D63581

llvm-svn: 364231
2019-06-24 21:32:50 +00:00
Hans Wennborg 05d44139ee Revert r362676 "[Profile]: Add runtime interface to specify file handle for profile data."
This caused instrumented Clang to become crashy. See llvm-commits thread
for repro steps.

This also reverts follow-up r362716 which added test cases.

> Author: Sajjad Mirza
>
> Differential Revision: http://reviews.llvm.org/D62541

llvm-svn: 363134
2019-06-12 08:44:32 +00:00
Xinliang David Li c1867557d9 [Profile]: Add runtime interface to specify file handle for profile data.
Author: Sajjad Mirza

Differential Revision: http://reviews.llvm.org/D62541

llvm-svn: 362676
2019-06-06 06:35:18 +00:00
Max Moroz 52fa90a348 This change adds an API to allow setting the flag to indicate that the profile data has been dumped to the file.
Summary:
The main use is for users to disable dumping profile data to the file
for certain processes in case the processes don't have permission to
write to the disks, and trying to do so would result in side effects
such as crashes.

Patch by Yuke Liao (@liaoyuke).

Additional context (Chromium use case):
- https://bugs.chromium.org/p/chromium/issues/detail?id=842424
- https://bugs.chromium.org/p/chromium/issues/detail?id=957655
- https://chromium-review.googlesource.com/c/chromium/src/+/1610093

Reviewers: Dor1s, vsk, davidxl

Reviewed By: Dor1s, davidxl

Subscribers: delcypher, davidxl, sajjadm, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D62078

llvm-svn: 361194
2019-05-20 20:02:20 +00:00
Nico Weber 37dc0ced7a Fix some gcc warnings in compiler-rt
- Several "warning: extra ';' [-Wpedantic]"
- One "C++ style comments are not allowed in ISO C90 [enabled by default]"
  in a file that uses C style comments everywhere but in one place

llvm-svn: 360430
2019-05-10 14:15:13 +00:00
Petr Hosek 156226bb70 [compiler-rt] Set the ZX_VMO_RESIZABLE option for zx_vmo_create
Currently VMO in Zircon create using the zx_vmo_create is resizable
by default, but we'll be changing this in the future, requiring an
explicit flag to make the VMO resizable.

Prepare for this change by passing ZX_VMO_RESIZABLE option to all
zx_vmo_create calls that need resizable VMO.

Differential Revision: https://reviews.llvm.org/D61450

llvm-svn: 359803
2019-05-02 17:24:53 +00:00
Dmitry Mikulin 312b5f86b7 The error message for mismatched value sites is very cryptic.
Make it more readable for an average user.

Differential Revision: https://reviews.llvm.org/D60896

llvm-svn: 359043
2019-04-23 22:26:55 +00:00
Manman Ren e73ae9a142 Reland compiler-rt support for order file instrumentation.
r355343 was landed and was reverted in r355363 due to build breakage.
This patch adds Linux/Windows support on top of r355343.

In this patch, Darwin should be working with testing case. Linux should be working,
I will enable the testing case in a follwup diff. Windows/Other should be building.
Correct implementation for Other platforms will be added.

Thanks David for reviewing the original diff, helping me with issues on Linux, and
giving suggestions for adding support for Other platforms.

llvm-svn: 355701
2019-03-08 15:30:56 +00:00
Manman Ren ff4bb36d7c Revert compiler-rt diffs for order file instrumentation to get bot green!
This caused issues on Linux/Windows and other platforms.

r355343 355350 355350

llvm-svn: 355363
2019-03-05 01:21:40 +00:00
Manman Ren 31b31e5111 Attemp to fix windows profile-rt build breakage.
Followup to D57530.

llvm-svn: 355357
2019-03-05 00:52:00 +00:00
Manman Ren 03d5348132 Attemp to fix build brokage due to D57530.
By adding implementations for __llvm_profile_begin_orderfile for non-Darwin platforms.

llvm-svn: 355350
2019-03-04 23:41:57 +00:00
Manman Ren 4737abc71c Order File Instrumentation: dump the data in compiler-rt
The profile data will be dumped in a file default_xxx.profraw.order.

Differential Revision: https://reviews.llvm.org/D57530

llvm-svn: 355343
2019-03-04 22:28:38 +00:00
Rong Xu 23452e1c85 [PGO] Update InstrProfData.inc to sync with llvm
llvm-svn: 355119
2019-02-28 19:06:02 +00:00
Reid Kleckner 8b6af00173 [llvm-cov] Fix llvm-cov on Windows and un-XFAIL test
Summary:
The llvm-cov tool needs to be able to find coverage names in the
executable, so the .lprfn and .lcovmap sections cannot be merged into
.rdata.

Also, the linker merges .lprfn$M into .lprfn, so llvm-cov needs to
handle that when looking up sections. It has to support running on both
relocatable object files and linked PE files.

Lastly, when loading .lprfn from a PE file, llvm-cov needs to skip the
leading zero byte added by the profile runtime.

Reviewers: vsk

Subscribers: hiraditya, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D58661

llvm-svn: 354840
2019-02-26 02:30:00 +00:00
Ana Pazos bbb8129b2c Set hidden attribute on lprofMergeValueProfData
Summary:
The changes in https://reviews.llvm.org/D44847 cause load time failure
due to lprofMergeValueProfData in Android libs enabled with profile generation:

"dlopen failed: cannot locate symbol "lprofMergeValueProfData" referenced by..."

Marking lprofMergeValueProfData as hidden so the correct in-module definition
 is picked by the linker.

Reviewers: davidxl

Reviewed By: davidxl

Subscribers: efriedma, xur, davidxl, llvm-commits

Differential Revision: https://reviews.llvm.org/D55893

llvm-svn: 354064
2019-02-14 21:38:40 +00:00
Martin Storsjo 7eb4ec2879 [profile] Provide lprofGetHostName for all windows environments
This function doesn't use anything MSVC specific but works fine
for any _WIN32 target.

Differential Revision: https://reviews.llvm.org/D58106

llvm-svn: 353918
2019-02-13 07:27:05 +00:00
Francis Visoiu Mistrih c716bee053 [InstrProf] Fix darwin errors after r353547
The macros were changed but not updated in the Darwin version.

Fail here: http://green.lab.llvm.org/green/job/clang-stage1-configure-RA/53739/

rdar://47925820

llvm-svn: 353560
2019-02-08 20:37:40 +00:00
Reid Kleckner 987d331fab [InstrProf] Implement static profdata registration
Summary:
The motivating use case is eliminating duplicate profile data registered
for the same inline function in two object files. Before this change,
users would observe multiple symbol definition errors with VC link, but
links with LLD would succeed.

Users (Mozilla) have reported that PGO works well with clang-cl and LLD,
but when using LLD without this static registration, we would get into a
"relocation against a discarded section" situation. I'm not sure what
happens in that situation, but I suspect that duplicate, unused profile
information was retained. If so, this change will reduce the size of
such binaries with LLD.

Now, Windows uses static registration and is in line with all the other
platforms.

Reviewers: davidxl, wmi, inglorion, void, calixte

Subscribers: mgorny, krytarowski, eraman, fedor.sergeev, hiraditya, #sanitizers, dmajor, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D57929

llvm-svn: 353547
2019-02-08 19:03:50 +00:00
Reid Kleckner c20ff80845 Silence -Wformat warnings about GetLastError returning ULONG
llvm-svn: 353485
2019-02-07 23:58:48 +00:00
Reid Kleckner 0a142346b3 [InstrProf] Merge COFF .lprof* and .lcovmap sections into .data/.rdata
There is no reason for these sections to remain separate in the final
DLL or EXE.

I have not yet added a InstrProfilingPlatformWindows.c for these, since
avoiding dynamic profile data registration is a larger project for
later.

llvm-svn: 353221
2019-02-05 21:26:31 +00:00
Chandler Carruth 2946cd7010 Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636
2019-01-19 08:50:56 +00:00
Rong Xu 3e9e7fb961 [profile] Sync up InstrProfData.inc with llvm copy /NFC
llvm-svn: 351257
2019-01-15 21:59:17 +00:00
Kamil Rytarowski edbe2b3f02 Add support for LLVM profile for NetBSD
Summary:
NetBSD uses typical UNIX interfaces.

All tests pass except instrprof-dlopen-dlclose-gcov.test, as there
is not supported semantics of atexit(3) in dlopen(3)ed+dlclose(3)d
DSO.

NetBSD also ships an older version of LLVM profile (ABI v.2 predating
ABI v.4 in upstream version) inside libc. That copy has been manually
removed during the porting and testing process of the upstream version
to NetBSD. Otherwise there were conflicts between them two.

Reviewers: joerg, vitalybuka, vsk

Subscribers: srhines, fedor.sergeev, llvm-commits, mgorny, #sanitizers

Tags: #sanitizers

Differential Revision: https://reviews.llvm.org/D55997

llvm-svn: 349994
2018-12-22 06:56:19 +00:00
Marco Castelluccio ed4bb266c8 [GCOV] Close file mapping handle on Windows, so flushed gcda files can be removed while the process is in execution
llvm-svn: 346300
2018-11-07 09:38:26 +00:00
Matt Davis 839ec9d9a4 [compiler-rt] Remove unused formal parameters from allocateOneNode. NFC.
Summary: This is just a minor cleanup to the allocateOneNode interface.  The formals are no-longer used, so I just removed them.

Reviewers: davidxl, void

Reviewed By: davidxl

Subscribers: dberris, llvm-commits

Differential Revision: https://reviews.llvm.org/D52509

llvm-svn: 344073
2018-10-09 20:10:28 +00:00
Martin Storsjo 6dc8968124 [Windows] Include BaseTsd.h with lowercase
This fixes building on a case sensitive filesystem with mingw-w64
headers, where all headers are lowercase. This header actually also
is named with a lowercase name in the Windows SDK as well.

Differential Revision: https://reviews.llvm.org/D51877

llvm-svn: 341857
2018-09-10 20:28:06 +00:00
Matt Davis 89e8af6d96 [compiler-rt] Add a routine to specify the mode used when creating profile dirs.
Summary:
This patch introduces `llvm_profile_set_dir_mode` and `llvm_profile_get_dir_mode` to
the compiler-rt profile API. 

Originally, profile data was placed into a directory that was created with a hard-coded
mode value of 0755 (for non-win32 builds).  In certain cases, it can be helpful to create
directories with a different mode other than 0755.  This patch introduces set/get
routines to allow users to specify a desired mode.  The default remains at 0755.

Reviewers: void, probinson

Reviewed By: probinson

Subscribers: probinson, dberris, cfe-commits

Differential Revision: https://reviews.llvm.org/D49953

llvm-svn: 338456
2018-07-31 23:37:24 +00:00
Petr Hosek 47e5fcba57 [profile] Support profiling runtime on Fuchsia
This ports the profiling runtime on Fuchsia and enables the
instrumentation. Unlike on other platforms, Fuchsia doesn't use
files to dump the instrumentation data since on Fuchsia, filesystem
may not be accessible to the instrumented process. We instead use
the data sink to pass the profiling data to the system the same
sanitizer runtimes do.

Differential Revision: https://reviews.llvm.org/D47208

llvm-svn: 337881
2018-07-25 03:01:35 +00:00
Igor Kudrin 356c2aeffe [profile] Fix finding the first and last directory separators on Windows.
Until now, our code preferred backslashes to slashes, whereas Windows
allows using both types of directory separators in one path string.

Differential Revision: https://reviews.llvm.org/D49664

llvm-svn: 337826
2018-07-24 13:06:19 +00:00
Igor Kudrin 63600c7487 [profile] Fix a possible memory leak in parseFilenamePattern().
Differential Revision: https://reviews.llvm.org/D49666

llvm-svn: 337823
2018-07-24 12:28:53 +00:00
Teresa Johnson 73053b221f [profile] Add interface to get profile filename
Summary:
Add __llvm_profile_get_filename interface to get the profile filename,
which can be used for identifying which profile file belongs to an app
when multiple binaries are instrumented and dumping profiles into the
same directory. The filename includes the path.

Reviewers: davidxl

Subscribers: delcypher, #sanitizers, llvm-commits

Differential Revision: https://reviews.llvm.org/D49529

llvm-svn: 337482
2018-07-19 19:03:50 +00:00
Marco Castelluccio e21cfa7e7f Fix reading 32 bit gcov tag values on little-endian machines
Summary:
The write buffer contains signed chars, which means the shift operations caused values such as the arc tag value (0x01a10000) to be read incorrectly (0xffa10000).

This fixes a regression from https://reviews.llvm.org/D49132.

Reviewers: uweigand, davidxl

Reviewed By: uweigand

Subscribers: llvm-commits, #sanitizers

Differential Revision: https://reviews.llvm.org/D49161

llvm-svn: 336775
2018-07-11 09:30:25 +00:00
Ulrich Weigand dd4d2ac607 [gcov] Fix fallout from r336693
Fix building GCDAProfiling.c with pre-C99 compilers.
This caused a build bot failure.

llvm-svn: 336706
2018-07-10 17:08:05 +00:00
Ulrich Weigand e508cd2a04 [gcov] Fix gcov profiling on big-endian machines
Two fixes required to handle big-endian systems:

- 64-bit counter values are stored in a mixed-endian format in the
gcov files: a 32-bit low-part followed by a 32-bit high part. Note that
this is already implemented correctly on the LLVM side, see
GCOVBuffer::readInt64.

- The tag values (e.g. arcs tag, object summary tag, ...) are aways
written as the same sequence of bytes independent of byte order. But
when *reading* them back in, the code reads them as 32-bit values in
host byte order. For the comparisons to work correctly, this should
instead always read them as little-endian values.

Fixes PR 38121.

Reviewed By: marco-c

Differential Revision: https://reviews.llvm.org/D49132

llvm-svn: 336693
2018-07-10 16:08:27 +00:00
Marco Castelluccio 2827420aab Reapply "Make __gcov_flush flush counters for all shared libraries"
This reapplies r336365, after marking tests as failing on various
configurations.

llvm-svn: 336678
2018-07-10 14:12:03 +00:00
Dan Liew b1f95697c1 [CMake] Add compiler-rt header files to the list of sources for targets
when building with an IDE so that header files show up in the UI.
This massively improves the development workflow in IDEs.

To implement this a new function `compiler_rt_process_sources(...)` has
been added that adds header files to the list of sources when the
generator is an IDE. For non-IDE generators (e.g. Ninja/Makefile) no
changes are made to the list of source files.

The function can be passed a list of headers via the
`ADDITIONAL_HEADERS` argument. For each runtime library a list of
explicit header files has been added and passed via
`ADDITIONAL_HEADERS`. For `tsan` and `sanitizer_common` a list of
headers was already present but it was stale and has been updated
to reflect the current state of the source tree.

The original version of this patch used file globbing (`*.{h,inc,def}`)
to find the headers but the approach was changed due to this being a
CMake anti-pattern (if the list of headers changes CMake won't
automatically re-generate if globbing is used).

The LLVM repo contains a similar function named `llvm_process_sources()`
but we don't use it here for several reasons:

* It depends on the `LLVM_ENABLE_OPTION` cache variable which is
  not set in standalone compiler-rt builds.
* We would have to `include(LLVMProcessSources)` which I'd like to
  avoid because it would include a bunch of stuff we don't need.

Differential Revision: https://reviews.llvm.org/D48422

llvm-svn: 336663
2018-07-10 13:00:17 +00:00
Michael Zolotukhin 9664ca9dce Revert "Make __gcov_flush flush counters for all shared libraries"
This reverts r336365: the added tests are failing on various
configurations (e.g. on green-dragon).

llvm-svn: 336474
2018-07-07 00:07:00 +00:00
Marco Castelluccio 32d18beb8c Make __gcov_flush flush counters for all shared libraries
Summary:
This will make the behavior of __gcov_flush match the GCC behavior.

I would like to rename __gcov_flush to __llvm_gcov_flush (in case of programs linking to libraries built with different compilers), but I guess we can't for compatibility reasons.

Reviewers: davidxl

Reviewed By: davidxl

Subscribers: samsonov, vitalybuka, pcc, kcc, junbuml, glider, fhahn, eugenis, dvyukov, davidxl, srhines, chh, llvm-commits, #sanitizers

Differential Revision: https://reviews.llvm.org/D48538

llvm-svn: 336365
2018-07-05 15:52:59 +00:00
Chih-Hung Hsieh 7222e8e30b [profile] Add llvm_gcov_flush to be called outside a shared library
__gcov_flush is hidden.
For applications to dump profiling data of selected .so files,
they can use dlsym to find and call llvm_gcov_flush in each .so file.

Differential Revision: https://reviews.llvm.org/D45454

llvm-svn: 336019
2018-06-29 21:45:55 +00:00
Petr Hosek 1608ca6d64 [CMake] Build shared version of runtimes for Fuchsia
Fuchsia is no longer treated as UNIX which means we need to explicitly
enable building of shared versions of runtimes.

Differential Revision: https://reviews.llvm.org/D46609

llvm-svn: 331922
2018-05-09 21:24:06 +00:00
Reid Kleckner 963aba3452 Fix clang-cl warnings in compiler-rt
The profile library was missing some includes and was erroneously using
ftruncate. WinASan was using `= {0}` to initialize structs, which
creates -Wmissing-field-initializers and -Wmissing-braces warnings with
clang. Use `= {}` instead, since this is C++.

llvm-svn: 330616
2018-04-23 17:05:47 +00:00
Rong Xu 95ab7582f0 [profile] Fix value profile runtime merging issues
This patch fixes the following issues:
(1) The strong definition of the merge hook function was not working which
breaks the online value profile merging. This patch removes the weak
attribute of VPMergeHook and assigns the value dynamically.
(2) Truncate the proifle file so that we don't have garbage data at the end of
the file.
(3) Add new __llvm_profile_instrument_target_value() interface to do the value
profile update in batch. This is needed as the original incremental by 1
in __llvm_profile_instrument_target() is too slow for online merge.

Differential Revision: https://reviews.llvm.org/D44847

llvm-svn: 328987
2018-04-02 16:57:00 +00:00
Hiroshi Inoue 93eaad7dac [NFC] fix trivial typos in comments
"the the" -> "the"

llvm-svn: 323080
2018-01-22 07:51:37 +00:00
Marco Castelluccio 8dfdfa65ed Hide some symbols to avoid a crash on shutdown when using code coverage
Summary:
gcov / gcda-based profiling crashes when shared libraries are unloaded

Patch by Benoit Belley and test by Marco Castelluccio for Firefox

See https://bugs.llvm.org/show_bug.cgi?id=27224 & https://bugzilla.mozilla.org/show_bug.cgi?id=1401230

Reviewers: davidxl, rnk, void

Subscribers: jessicah, marco-c, belleyb, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D38124

llvm-svn: 321703
2018-01-03 11:21:30 +00:00
Marco Castelluccio ba3a619139 Flush gcda files before unlocking them
Summary: Fixes https://bugs.llvm.org/show_bug.cgi?id=35464.

Reviewers: zturner, rnk, void

Subscribers: sylvestre.ledru, llvm-commits, #sanitizers

Differential Revision: https://reviews.llvm.org/D40610

llvm-svn: 321702
2018-01-03 11:13:45 +00:00
Reid Kleckner d0711dfba1 Include process.h for getpid on Windows in instr profiling
llvm-svn: 321313
2017-12-21 21:48:55 +00:00
Vedant Kumar d7c9336a84 [profile] Port the runtime to Solaris (retry)
This includes a few nice bits of refactoring (e.g splitting out the
exclusive locking code into a common utility).

Hopefully the Windows support is fixed now.

Patch by Rainer Orth!

Differential Revision: https://reviews.llvm.org/D40944

llvm-svn: 320731
2017-12-14 19:01:04 +00:00
Vedant Kumar e8e8599ade Revert "(HEAD -> master, origin/master, origin/HEAD) [profile] Port the runtime to Solaris"
This reverts commit r320726. It looks like flock isn't available on
Windows:

http://lab.llvm.org:8011/builders/sanitizer-windows/builds/21317/steps/build%20compiler-rt/logs/stdio

llvm-svn: 320728
2017-12-14 18:50:13 +00:00
Vedant Kumar 5b0d5b45a2 [profile] Port the runtime to Solaris
This includes a few nice bits of refactoring (e.g splitting out the
exclusive locking code into a common utility).

Patch by Rainer Orth!

Differential Revision: https://reviews.llvm.org/D40944

llvm-svn: 320726
2017-12-14 18:43:14 +00:00
Vedant Kumar ede2449fd1 [profile] Update InstrProfData.inc to sync with llvm
llvm-svn: 318230
2017-11-14 23:57:58 +00:00
Marco Castelluccio ff4da375ae Implement flock for Windows in compiler-rt
Summary:
This patch implements flock for Windows, needed to make gcda writing work in a multiprocessing scenario.

Fixes https://bugs.llvm.org/show_bug.cgi?id=34923.

Reviewers: zturner

Reviewed By: zturner

Subscribers: rnk, zturner, llvm-commits

Differential Revision: https://reviews.llvm.org/D38891

llvm-svn: 317705
2017-11-08 19:11:54 +00:00
Marco Castelluccio 17103fc12f Use O_BINARY when opening GCDA file on Windows
Summary:
Fixes https://bugs.llvm.org/show_bug.cgi?id=34922.

Apparently, the mode in **fdopen** gets simply ignored and Windows only cares about the mode of the original **open**.

I have verified this both with the simple case from bug 34922 and with a full Firefox build.

Reviewers: zturner

Reviewed By: zturner

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D38984

llvm-svn: 316048
2017-10-18 00:22:01 +00:00
Vedant Kumar 4265d16d45 [profile] Update InstrProfData.inc
llvm-svn: 313599
2017-09-18 23:37:32 +00:00
Alex Lorenz 341317fda4 Revert r312240
The buildbots have shown that -Wstrict-prototypes behaves differently in GCC
and Clang so we should keep it disabled until Clang follows GCC's behaviour

llvm-svn: 312246
2017-08-31 15:51:23 +00:00
Alex Lorenz fbac1ae062 Build LLVM with -Wstrict-prototypes enabled
Clang 5 supports -Wstrict-prototypes. We should use it to catch any C
declarations that declare a non-prototype function.

rdar://33705313

Differential Revision: https://reviews.llvm.org/D36669

llvm-svn: 312240
2017-08-31 13:23:24 +00:00
Xinliang David Li c7c5303fa3 [Profile] create a copy of profile file name from environment
Original patch by Max Moroz.

Differential Revsion: http://reviews.llvm.org/D36903

llvm-svn: 311607
2017-08-23 21:39:33 +00:00
Hans Wennborg c5e7cee0a2 [profile] Fix warning about C++ style comment in C file
llvm-svn: 311496
2017-08-22 21:54:37 +00:00
Xinliang David Li 1c9320cd4d Revert r310857 due to internal test failure
llvm-svn: 310907
2017-08-15 03:13:01 +00:00
Xinliang David Li 533d4c90bd [PGO] Add support for relocate profile dumping directory
Differential Revsion: http://reviews.llvm.org/D36648

llvm-svn: 310857
2017-08-14 16:51:05 +00:00
Dehao Chen ab360d6015 Change INSTR_PROF_DEFAULT_NUM_VAL_PER_SITE from 8 to 16.
Summary: In the current implementation, the defaul number of values per site tracked by value profiler is 8, which is too small and could introduce inaccuracies to profile. Changing it to 16 will be able to gain more accurate value profiler.

Reviewers: davidxl, tejohnson

Reviewed By: tejohnson

Subscribers: sanjoy, llvm-commits

Differential Revision: https://reviews.llvm.org/D35964

llvm-svn: 309388
2017-07-28 15:00:30 +00:00
Vedant Kumar ff3227e77d [profile] Move __llvm_profile_filename into a separate object
Users can specify the path a raw profile is written to by passing
-fprofile-instr-generate=<path>, but this functionality broke on Darwin
after __llvm_profile_filename was made weak [1], resulting in profiles
being written to "default.profraw" even when <path> is specified.

The situation is that instrumented programs provide a weak definition of
__llvm_profile_filename, which conflicts with a weak redefinition
provided by the profiling runtime.

The linker appears to pick the 'winning' definition arbitrarily: on
Darwin, it usually prefers the larger definition, which is probably why
the instrprof-override-filename.c test has been passing.

The fix is to move the runtime's definition into a separate object file
within the archive. This means that the linker won't "see" the runtime's
definition unless the user program has not provided one. I couldn't
think of a great way to test this other than to mimic the Darwin
failure: use -fprofile-instr-generate=<some-small-path>.

Testing: check-{clang,profile}, modified instrprof-override-filename.c.

[1] [Profile] deprecate __llvm_profile_override_default_filename
https://reviews.llvm.org/D22613
https://reviews.llvm.org/D22614

Differential Revision: https://reviews.llvm.org/D34797

llvm-svn: 306710
2017-06-29 17:42:24 +00:00
Xinliang David Li f50cc3ed8a [PGO] Reduce IO in profile dumping with merging
Differential Revision: http://reviews.llvm.org/D34709

llvm-svn: 306561
2017-06-28 16:46:06 +00:00
Xinliang David Li 1f0d0b2a4c [Profile] Remove redundant call
llvm-svn: 306480
2017-06-27 22:42:24 +00:00
Xinliang David Li 967669f6c1 [PGO] Refactor file/buffer writer callback interfaces /NFC
Introduces a 'owner' struct to include the overridable write
method and the write context in C.

This allows easy introdution of new member API to help reduce
profile merge time in the follow up patch.

llvm-svn: 306432
2017-06-27 17:28:01 +00:00
Vedant Kumar 4afdcb0975 [profile] Sync up InstrProfData.inc (NFC)
llvm-svn: 300383
2017-04-15 00:10:33 +00:00
Xinliang David Li 3bb31c8c49 [Profile] PE binary coverage bug fix
PR/32584

Differential Revision: https://reviews.llvm.org/D32023

llvm-svn: 300278
2017-04-13 23:37:15 +00:00
Xinliang David Li 577b9d41d6 Revert 299954 : test failure needs to be fixed
llvm-svn: 299960
2017-04-11 16:27:26 +00:00
Xinliang David Li a53e6702d2 [Profile] PE binary coverage bug fix
PR/32584

Differential Revision: https://reviews.llvm.org/D31939

llvm-svn: 299954
2017-04-11 15:51:39 +00:00