Following up on commit 177176f75c, if we
failed to setDeleteDisposition(true) during TempFile creation, then
don't try to setDeleteDisposition(false) during TempFile::keep, since it
will likely fail as well.
Instead of letting TempFile::keep just fail, we should let it go ahead
and try renaming the file.
This fixes an issue we are seeing when running clang-cl.exe through the
Incredibuild distributed build system. We're seeing that renaming
temporary object files would fail here:
5c1f7b296a/clang/lib/Frontend/CompilerInstance.cpp (L789)
Reviewed By: mstorsjo
Differential Revision: https://reviews.llvm.org/D118212
Extract the `readNativeFile()` loop from
`MemoryBuffer::getMemoryBufferForStream()` into `readNativeFileToEOF()`
to allow reuse. The chunk size is configurable; the default of `4*4096`
is exposed as `sys::fs::DefaultReadChunkSize` to allow sizing of
SmallVectors.
There's somewhere I'd like to read a usually-small file without overhead
of a MemoryBuffer; extracting existing logic rather than duplicating it.
Differential Revision: https://reviews.llvm.org/D115397
Stop using `SmallVector::set_size()` in sys::path APIs. In both cases,
use `truncate()` instead.
Differential Revision: https://reviews.llvm.org/D115391
Default to preferring forward slashes when built for MinGW, as
many usecases, when e.g. Clang is used as a drop-in replacement
for GCC, requires the compiler to output paths with forward slashes.
Not all tests pass yet, if configuring to prefer forward slashes though.
Differential Revision: https://reviews.llvm.org/D112787
This behaves just like the regular Windows style, with both separator
forms accepted, but with get_separator() returning forward slashes.
Add a more descriptive name for the existing style, keeping the old
name around as an alias initially.
Add a new function `make_preferred()` (like the C++17
`std::filesystem::path` function with the same name), which converts
windows paths to the preferred separator form (while this one works on
any platform and takes a `path::Style` argument).
Contrary to `native()` (just like `make_preferred()` in `std::filesystem`),
this doesn't do anything at all on Posix, it doesn't try to reinterpret
backslashes into forward slashes there.
Differential Revision: https://reviews.llvm.org/D111879
This takes care of cleaning up the temp files on crashes. It doesn't
handle cleanup when explicitly killed though.
Differential Revision: https://reviews.llvm.org/D112710
Use the new sys::path::is_style_posix() and is_style_windows() in a few
places that need to detect the system's native path style.
In llvm/lib/Support/Path.cpp, this patch removes most uses of the
private `real_style()`, where is_style_posix() and is_style_windows()
are just a little tidier.
Elsewhere, this removes `_WIN32` macro checks. Added a FIXME to a
FileManagerTest that seemed fishy, but maintained the existing
behaviour.
Differential Revision: https://reviews.llvm.org/D112289
Expose three helpers in namespace llvm::sys::path to detect the
path rules followed by sys::path::Style.
- is_style_posix()
- is_style_windows()
- is_style_native()
This are constexpr functions that that will allow a bunch of
path-related code to stop checking `_WIN32`.
Originally I looked at adding system_style(), analogous to
sys::endian::system_endianness(), but future patches (from others) will
add more Windows style variants for slash preferences. These helpers
should be resilient to that change, allowing callers to detect basic
path rules.
Differential Revision: https://reviews.llvm.org/D112288
Since D81803 / 79657e2339, temp files
created on network shares don't set "Disposition.DeleteFile = true".
This flag normally takes care of removing the temp file both if the
process exits abnormally (either crashing or killed externally), and
when the file is closed cleanly.
For network shares, we voluntarily choose to not set the flag, and
if the operation to inspect the file handle (as a prerequisite to
setting the flag since 79657e2339)
fails we also error out. In both of these cases, we can at least make
sure to remove the temp files when they are closed cleanly.
Adjust the semantics of "OF_Delete" to not set the delete
disposition, but only set the access mode for allowing deletion.
Move the call to setDeleteDisposition into TempFile::create,
where we can check if it failed, and if it did, set a flag noting
that the file should be removed manually at the end.
This does leak files on crash, but at least doesn't leak files
in regular successful runs. (Technically, the alternative codepath
could use the RemoveFileOnSignal function, but that might complicate
the TempFile implementation further.)
This fixes https://github.com/mstorsjo/llvm-mingw/issues/233 and
https://bugs.llvm.org/show_bug.cgi?id=52080.
Differential Revision: https://reviews.llvm.org/D111875
After 8fc7a907b9, this loop does
the same as a plain `std::replace`.
Also clarify the comment about what this function does.
Differential Revision: https://reviews.llvm.org/D111730
This patch https://reviews.llvm.org/D102876 caused some lit regressions on z/OS because tmp files were no longer being opened based on binary/text mode. This patch passes OpenFlags when creating tmp files so we can open files in different modes.
Reviewed By: amccarth
Differential Revision: https://reviews.llvm.org/D103806
incorrect std::string use. (Also remove redundant call to
RemoveFileOnSignal.)
Clang writes object files by first writing to a .tmp file and then
renaming to the final .obj name. On Windows, if a compile is killed
partway through the .tmp files don't get deleted.
Currently it seems like RemoveFileOnSignal takes care of deleting the
tmp files on Linux, but on Windows we need to call
setDeleteDisposition on tmp files so that they are deleted when
closed.
This patch switches to using TempFile to create the .tmp files we write
when creating object files, since it uses setDeleteDisposition on Windows.
This change applies to both Linux and Windows for consistency.
Differential Revision: https://reviews.llvm.org/D102876
This reverts commit 20797b129f.
Clang writes object files by first writing to a .tmp file and then
renaming to the final .obj name. On Windows, if a compile is killed
partway through the .tmp files don't get deleted.
Currently it seems like RemoveFileOnSignal takes care of deleting the
tmp files on Linux, but on Windows we need to call
setDeleteDisposition on tmp files so that they are deleted when
closed.
This patch switches to using TempFile to create the .tmp files we write
when creating object files, since it uses setDeleteDisposition on Windows.
This change applies to both Linux and Windows for consistency.
Differential Revision: https://reviews.llvm.org/D102876
This patch consists of the initial changes to help distinguish between text and binary content correctly on z/OS. I would like to get feedback from Windows users on setting OF_None for all ToolOutputFiles. This seems to have been done as an optimization to prevent CRLF translation on Windows in the past.
Reviewed By: zibi
Differential Revision: https://reviews.llvm.org/D97785
Measure amount of high-level or fixed-cost operations performed during
building/loading modules and during header search. High-level operations
like building a module or processing a .pcm file are motivated by
previous issues where clang was re-building modules or re-reading .pcm
files unnecessarily. Fixed-cost operations like `stat` calls are tracked
because clang cannot change how long each operation takes but it can
perform fewer of such operations to improve the compile time.
Also tracking such stats over time can help us detect compile-time
regressions. Added stats are more stable than the actual measured
compilation time, so expect the detected regressions to be less noisy.
On relanding drop stats in MemoryBuffer.cpp as their value is pretty low
but affects a lot of clients and many of those aren't interested in
modules and header search.
rdar://problem/55715134
Reviewed By: aprantl, bruno
Differential Revision: https://reviews.llvm.org/D86895
This reverts commit c4bacc3c9b.
Test "LLVM :: ThinLTO/X86/funcimport-stats.ll" is failing. Reverting now
and will recommit after making the test not fail with the added stats.
Measure amount of high-level or fixed-cost operations performed during
building/loading modules and during header search. High-level operations
like building a module or processing a .pcm file are motivated by
previous issues where clang was re-building modules or re-reading .pcm
files unnecessarily. Fixed-cost operations like `stat` calls are tracked
because clang cannot change how long each operation takes but it can
perform fewer of such operations to improve the compile time.
Also tracking such stats over time can help us detect compile-time
regressions. Added stats are more stable than the actual measured
compilation time, so expect the detected regressions to be less noisy.
rdar://problem/55715134
Reviewed By: aprantl, bruno
Differential Revision: https://reviews.llvm.org/D86895
Implements IS_ABSOLUTE_PATH from GNU tools.
C++17 is_absolute behavior is different the from the behavior defined by GNU
tools.
According to cppreference.com, C++17 states: "An absolute path is a path
that unambiguously identifies the location of a file without reference
to an additional starting location."
In other words, the rules are:
1. POSIX style paths with nonempty root directory are absolute.
2. Windows style paths with nonempty root name and root directory are
absolute.
3. No other paths are absolute.
GNU rules are:
1. Paths starting with a path separator are absolute.
2. Windows style paths are also absolute if they start with a character
followed by ':'.
3. No other paths are absolute.
On Windows style the path "C:\Users\Default" has "C:" as root name and "\"
as root directory.
Hence "C:" on Windows is absolute under GNU rules and not absolute under
C++17 because it has no root directory. Likewise "/" and "\" on Windows are
absolute under GNU and are not absolute under C++17 due to empty root name.
Related to PR46368.
Differential Revision: https://reviews.llvm.org/D87667
In D49466, sys::path::replace_path_prefix was used instead startswith for -f[macro/debug/file]-prefix-map options.
However those were reverted later (commit rG3bb24bf25767ef5bbcef958b484e7a06d8689204) due to broken Windows tests.
This patch restores those replace_path_prefix calls.
It also modifies the prefix matching to be case-insensitive under Windows.
Differential Revision : https://reviews.llvm.org/D76869
Currently, normalize() for posix replaces backslashes to slashes, except
that two backslashes in sequence are kept as-is.
clang calls normalize() to convert \ to / is microsoft compat mode. This
generally works well, but a path like "c:\\foo\\bar.h" with two
backslashes doesn't work due to the exception in normalize().
These paths happen naturally on Windows hosts with e.g.
`#include __FILE__`, and them not working on other hosts makes it
more difficult to write tests for this case.
The special case has been around without justification since this code
was added in r203611 (since then moved around in r215241 r215243). No
integration tests fail if I remove it.
Try removing the special case.
Differential Revision: https://reviews.llvm.org/D79265
This reverts commit fb5fd74685.
Re-instates commit 53913a65b4
The fix is to trim off trailing separators, as in `/foo/bar/` and
produce `/foo/bar`. VFS tests rely on this. I added unit tests for
remove_dots.
LLD calls this on every source file string in every object file when
writing PDBs, so it is somewhat hot.
Avoid rewriting paths that do not contain path traversal components
(./..). Use find_first_not_of(separators) directly instead of using the
path iterators. The path component iterators appear to be slow, and
directly searching for slashes makes it easier to find double separators
that need to be canonicalized.
I discovered that the VFS relies on remote_dots to not canonicalize
early slashes (/foo or C:/foo) on Windows, so I had to leave that
behavior behind with unit tests for it. This is undesirable, but I claim
that my change is NFC.
Added unit tests for 2 scenarios that were failing.
Made replace_path_prefix back to 3 parameters instead of 5, simplifying the implementation. The other 2 were always used with the default value.
This commit is intended to be the first of 3:
1) simplify/fix replace_path_prefix.
2) use it in the context of -fdebug-prefix-map and -fmacro-prefix-map (see D76869).
3) Make Windows version of replace_path_prefix insensitive to both case and separators (slash vs backslash).
Differential Revision: https://reviews.llvm.org/D77223
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.
This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.
This doesn't actually modify StringRef yet, I'll do that in a follow-up.
GCC 8 implements -fmacro-prefix-map. Like -fdebug-prefix-map, it replaces a string prefix for the __FILE__ macro.
-ffile-prefix-map is the union of -fdebug-prefix-map and -fmacro-prefix-map
Reviewed By: rnk, Lekensteyn, maskray
Differential Revision: https://reviews.llvm.org/D49466
This fixes a bug for making path with a //net style root absolute. I
discovered the bug while writing a test case for the VFS, which uses
these paths because they're both legal absolute paths on Windows and
Unix.
Differential revision: https://reviews.llvm.org/D65675
llvm-svn: 368053
When moving a temp file, explicitly set the file descriptor to -1 so we
can never accidentally close the moved-from TempFile.
Differential revision: https://reviews.llvm.org/D63087
llvm-svn: 363083
I added a diagnostic along the lines of `-Wpessimizing-move` to detect `return x = y` suppressing copy elision, but I don't know if the diagnostic is really worth it. Anyway, here are the places where my diagnostic reported that copy elision would have been possible if not for the assignment.
P1155R1 in the post-San-Diego WG21 (C++ committee) mailing discusses whether WG21 should fix this pitfall by just changing the core language to permit copy elision in cases like these.
(Kona update: The bulk of P1155 is proceeding to CWG review, but specifically *not* the parts that explored the notion of permitting copy-elision in these specific cases.)
Reviewed By: dblaikie
Author: Arthur O'Dwyer
Differential Revision: https://reviews.llvm.org/D54885
llvm-svn: 359236
It turns out that I mesread the man page and fcopyfile(3) does not
actually support COPYFILE_CLONE for files.
<rdar://problem/50148757>
llvm-svn: 359127
This change makes linking into .build-id atomic and safe to use.
Some users under particular workflows are reporting that this races
more than half the time under particular conditions.
llvm-svn: 356404
Moved the remove of the temporary file to after the close to avoid
remove failures caused by ETXTBSY errors.
This issue was seen when FileOutputBuffer falls back to an in memory
buffer due to the inability to mmap the on disk file. This occurred when
running LLD on an Ubuntu VM in VirtualBox on a Windows host attempting
to write the output to a VirtualBox shared folder.
Differential Revision: https://reviews.llvm.org/D57960
llvm-svn: 354017
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
Summary:
The version of make_absolute which accepted a specific directory to use
as the "base" for the computation could never fail, even though it
returned a std::error_code. The reason for that seems to be historical
-- the CWD flavour (which can fail due to failure to retrieve CWD) was
there first, and the new version was implemented by extending that.
This removes the error return value from the non-CWD overload and
reimplements the CWD version on top of that. This enables us to remove
some dead code where people were pessimistically trying to handle the
errors returned from this function.
Reviewers: zturner, sammccall
Subscribers: hiraditya, kristina, llvm-commits
Differential Revision: https://reviews.llvm.org/D56599
llvm-svn: 351317
This is available on most platforms (Linux/Mac/Win/BSD) with no extra syscalls.
On other platforms (e.g. Solaris) we stat() if this information is requested.
This will allow switching clang's VFS to efficiently expose (path, type) when
traversing a directory. Currently it exposes an entire Status, but does so by
calling fs::status() on all platforms.
Almost all callers only need the path, and all callers only need (path, type).
Patch by sammccall (Sam McCall)
Differential Revision: https://reviews.llvm.org/D51918
llvm-svn: 342089
In r338216 / D49860 TempFile::keep was extended to allow keeping across
filesystems. The aim on Windows was to have this happen in rename_internal
using the existing system API. However, to fix an issue and preserve the
idea of "renaming" not being a move, put Windows keep-across-filesystem in
TempFile::keep.
Differential Revision: https://reviews.llvm.org/D50048
llvm-svn: 338841
Summary:
On Windows, TempFile::create() was prone to failing with permission
denied errors when a process created many tempfiles without providing
a model large enough to accommodate them. There was also a problem
with createUniqueEntity getting into an infinite loop when all names
permitted by the model are in use. This change fixes both of these
problems and adds a unit test for them.
Reviewers: pcc, rnk, zturner
Reviewed By: zturner
Subscribers: inglorion, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D50126
llvm-svn: 338745
serge-sans-paille