On Windows we've observed that if you open a file, write to it, map it into
memory and close the file handle, the contents of the memory mapping can
sometimes be incorrect. That was what we did when adding an entry to the
ThinLTO cache using the TempFile and MemoryBuffer classes, and it was causing
intermittent build failures on Chromium's ThinLTO bots on Windows. More
details are in the associated Chromium bug (crbug.com/786127).
We can prevent this from happening by keeping a handle to the file open while
the mapping is active. So this patch changes the mapped_file_region class to
duplicate the file handle when mapping the file and close it upon unmapping it.
One gotcha is that the file handle that we keep open must not have been
created with FILE_FLAG_DELETE_ON_CLOSE, as otherwise the operating system
will prevent other processes from opening the file. We can achieve this
by avoiding the use of FILE_FLAG_DELETE_ON_CLOSE altogether. Instead,
we use SetFileInformationByHandle with FileDispositionInfo to manage the
delete-on-close bit. This lets us remove the hack that we used to use to
clear the delete-on-close bit on a file opened with FILE_FLAG_DELETE_ON_CLOSE.
A downside of using SetFileInformationByHandle/FileDispositionInfo as
opposed to FILE_FLAG_DELETE_ON_CLOSE is that it prevents us from using
CreateFile to open the file while the flag is set, even within the same
process. This doesn't seem to matter for almost every client of TempFile,
except for LockFileManager, which calls sys::fs::create_link to create a
hard link from the lock file, and in the process of doing so tries to open
the file. To prevent this change from breaking LockFileManager I changed it
to stop using TempFile by effectively reverting r318550.
Differential Revision: https://reviews.llvm.org/D48051
llvm-svn: 334630
Summary:
With D43396, no clients use the Path parameter anymore.
Depends on D43396.
Reviewers: pcc
Subscribers: mehdi_amini, inglorion, llvm-commits
Differential Revision: https://reviews.llvm.org/D43400
llvm-svn: 325619
This requires a small change to TempFile: allowing a discard after a
failed keep.
With this the cache now handles signals and reuses a fd instead of
reopening the file.
llvm-svn: 318322
Summary:
LTO/Caching.cpp uses file rename to atomically set the value for a
cache key. On Windows, this fails when the destination file already
exists. Previously, LLVM would report_fatal_error in such
cases. However, because the old and the new value for the cache key
are supposed to be equivalent, it actually doesn't matter which one we
keep. This change makes it so that failing the rename when an openable
file with the desired name already exists causes us to report success
instead of fataling.
Reviewers: pcc, hans
Subscribers: mehdi_amini, llvm-commits, hiraditya
Differential Revision: https://reviews.llvm.org/D39874
llvm-svn: 317899
Move error handling code next to the code that returns the error,
and change the error message in order to distinguish it from a similar
error message elsewhere in this file.
llvm-svn: 314745
I empirically verified that open files can in fact be renamed on
Windows with sys::fs::rename, so remove the incorrect code and comment.
llvm-svn: 312683
This is a safeguard against data loss if the user specifies a directory
that is not a cache directory. Teach the existing cache pruning clients
to create files with appropriate names.
Differential Revision: https://reviews.llvm.org/D31109
llvm-svn: 298271
After the call to sys::fs::exists succeeds, indicating a cache hit, we call
AddFile and the client will open the file using the supplied path. If the
client is using cache pruning, there is a potential race between the pruner
and the client. To avoid this, change the caching API so that it provides
a MemoryBuffer to the client, and have clients use that MemoryBuffer where
possible.
This scheme won't work with the gold plugin because the plugin API expects a
file path. So we have the gold plugin use the buffer identifier as a path and
live with the race for now. (Note that the gold plugin isn't actually affected
by the problem at the moment because it doesn't support cache pruning.)
This effectively reverts r279883 modulo the change to use the existing path
in the gold plugin.
Differential Revision: https://reviews.llvm.org/D31063
llvm-svn: 298020
This fixes a race condition where another linker process can observe a
partially written file if we copy it from another file system, and allows
the link to be independent of the amount of free disk space in $TMPDIR.
Differential Revision: https://reviews.llvm.org/D31045
llvm-svn: 297970
The NativeObjectOutput class has a design problem: it mixes up the caching
policy with the interface for output streams, which makes the client-side
code hard to follow and would for example make it harder to replace the
cache implementation in an arbitrary client.
This change separates the two aspects by moving the caching policy
to a separate field in Config, replacing NativeObjectOutput with a
NativeObjectStream class which only deals with streams and does not need to
be overridden by most clients and introducing an AddFile callback for adding
files (e.g. from the cache) to the link.
Differential Revision: https://reviews.llvm.org/D24622
llvm-svn: 282299
Summary:
Have the cache pass back the path to the cache entry when it
is ready to be loaded, instead of a buffer.
For gold-plugin we can simply pass this file back to gold directly,
which avoids expensive writing of a separate tmp file. Ensure
the cache entry is not deleted on cleanup by adjusting the setting
of the IsTemporary flags.
Moved the loading of the buffer into llvm-lto2 to maintain current
behavior.
Reviewers: mehdi_amini
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D23946
llvm-svn: 279883
Add the ability to plug a cache on the LTO API.
I tried to write such that a linker implementation can
control the cache backend. This is intrusive and I'm
not totally happy with it, but I can't figure out a
better design right now.
Differential Revision: https://reviews.llvm.org/D23599
llvm-svn: 279576
Andrew Ng