r327219 added wrappers to std::sort which randomly shuffle the container before
sorting. This will help in uncovering non-determinism caused due to undefined
sorting order of objects having the same key.
To make use of that infrastructure we need to invoke llvm::sort instead of
std::sort.
llvm-svn: 328636
Summary:
In https://reviews.llvm.org/D39572 , I added support for specifying
`Type` when invoking `InMemoryFileSystem::addFile()`.
However, I didn't account for the fact that when `Type` is
`directory_file`, we need to construct an `InMemoryDirectory`, not an
`InMemoryFile`, or else clients cannot create files inside that
directory.
This diff fixes the bug and adds a test.
Test Plan: New test added. Ran test with:
% make -j12 check-clang-tools
Reviewers: bkramer, hokein
Reviewed By: bkramer
Subscribers: cfe-commits
Differential Revision: https://reviews.llvm.org/D40140
llvm-svn: 318445
This allows clients to avoid an unnecessary fs::status() call on each
directory entry. Because the information returned by FindFirstFileEx
is a subset of the information returned by a regular status() call,
I needed to extract a base class from file_status that contains only
that information.
On my machine, this reduces the time required to enumerate a ThinLTO
cache directory containing 520k files from almost 4 minutes to less
than 2 seconds.
Differential Revision: https://reviews.llvm.org/D38716
llvm-svn: 315378
Modified the tests to accept any iteration order, to run only on Unix, and added
additional error reporting to investigate SystemZ bot issue.
The VFS directory iterator and recursive directory iterator behave differently
from the LLVM counterparts. Once the VFS iterators hit a broken symlink they
immediately abort. The LLVM counterparts don't stat entries unless they have to
descend into the next directory, which allows to recover from this issue by
clearing the error code and skipping to the next entry.
This change adds similar behavior to the VFS iterators. There should be no
change in current behavior in the current CLANG source base, because all
clients have loop exit conditions that also check the error code.
This fixes rdar://problem/30934619.
Differential Revision: https://reviews.llvm.org/D30768
llvm-svn: 297693
Modified the tests to accept any iteration order.
The VFS directory iterator and recursive directory iterator behave differently
from the LLVM counterparts. Once the VFS iterators hit a broken symlink they
immediately abort. The LLVM counterparts allow to recover from this issue by
clearing the error code and skipping to the next entry.
This change adds the same functionality to the VFS iterators. There should be
no change in current behavior in the current CLANG source base, because all
clients have loop exit conditions that also check the error code.
This fixes rdar://problem/30934619.
Differential Revision: https://reviews.llvm.org/D30768
llvm-svn: 297528
The VFS directory iterator and recursive directory iterator behave differently
from the LLVM counterparts. Once the VFS iterators hit a broken symlink they
immediately abort. The LLVM counterparts allow to recover from this issue by
clearing the error code and skipping to the next entry.
This change adds the same functionality to the VFS iterators. There should be
no change in current behavior in the current CLANG source base, because all
clients have loop exit conditions that also check the error code.
This fixes rdar://problem/30934619.
Differential Revision: https://reviews.llvm.org/D30768
llvm-svn: 297510
Merge all VFS mapped files inside -ivfsoverlay inputs into the vfs
overlay provided by the crash reproducer. This is the last missing piece
to allow crash reproducers to fully work with user frameworks; when
combined with headermaps, it allows clang to find additional frameworks.
rdar://problem/27913709
llvm-svn: 290326
Reapply r278457 with test fixed to not abouse fs case sensitivity.
When the VFS uses a YAML file, the real file path for a
virtual file is described in the "external-contents" field. Example:
...
{
'type': 'file',
'name': 'a.h',
'external-contents': '/a/b/c/a.h'
}
Currently, when parsing umbrella directories, we use
vfs::recursive_directory_iterator to gather the header files to generate the
equivalent modules for. If the external contents for a header does not exist,
we currently are unable to build a module, since the VFS
vfs::recursive_directory_iterator will fail when it finds an entry without a
reliable real path.
Since the YAML file could be prepared ahead of time and shared among
different compiler invocations, an entry might not yet have a reliable
path in 'external-contents', breaking the iteration.
Give the VFS the capability to skip such entries whenever
'ignore-non-existent-contents' property is set in the YAML file.
rdar://problem/27531549
llvm-svn: 278543
When the VFS uses a YAML file, the real file path for a
virtual file is described in the "external-contents" field. Example:
...
{
'type': 'file',
'name': 'a.h',
'external-contents': '/a/b/c/a.h'
}
Currently, when parsing umbrella directories, we use
vfs::recursive_directory_iterator to gather the header files to generate the
equivalent modules for. If the external contents for a header does not exist,
we currently are unable to build a module, since the VFS
vfs::recursive_directory_iterator will fail when it finds an entry without a
reliable real path.
Since the YAML file could be prepared ahead of time and shared among
different compiler invocations, an entry might not yet have a reliable
path in 'external-contents', breaking the iteration.
Give the VFS the capability to skip such entries whenever
'ignore-non-existent-contents' property is set in the YAML file.
rdar://problem/27531549
llvm-svn: 278457
Add 'ignore-non-existent-contents' to tell the VFS whether an invalid path
obtained via 'external-contents' should cause iteration on the VFS to stop.
If 'true', the VFS should ignore the entry and continue with the next. Allows
YAML files to be shared across multiple compiler invocations regardless of
prior existent paths in 'external-contents'. This global value is overridable
on a per-file basis.
This adds the parsing and write test part, but use by VFS comes next.
Differential Revision: https://reviews.llvm.org/D23422
rdar://problem/27531549
llvm-svn: 278456
Differential Revision: http://reviews.llvm.org/D19843
Corresponding LLVM change: http://reviews.llvm.org/D19842
Re-commit after addressing issues with of generating too many warnings for Windows and asan test failures.
Patch by Eric Niebler
llvm-svn: 272562
Reapply r269100 and r269270, reverted due to
https://llvm.org/bugs/show_bug.cgi?id=27725. Isolate the testcase that
corresponds to the new feature side of this commit and skip it on
windows hosts until we find why it does not work on these platforms.
Original commit message:
The way we currently build the internal VFS overlay representation leads
to inefficient path search and might yield wrong answers when asked for
recursive or regular directory iteration.
Currently, when reading an YAML file, each YAML root entry is placed
inside a new root in the filesystem overlay. In the crash reproducer, a
simple "@import Foundation" currently maps to 43 roots, and when looking
up paths, we traverse a directory tree for each of these different
roots, until we find a match (or don't). This has two consequences:
- It's slow.
- Directory iteration gives incomplete results since it only return
results within one root - since contents of the same directory can be
declared inside different roots, the result isn't accurate.
This is in part fault of the way we currently write out the YAML file
when emitting the crash reproducer - we could generate only one root and
that would make it fast and correct again. However, we should not rely
on how the client writes the YAML, but provide a good internal
representation regardless.
Build a proper virtual directory tree out of the YAML representation,
allowing faster search and proper iteration. Besides the crash
reproducer, this potentially benefits other VFS clients.
llvm-svn: 269327
The way we currently build the internal VFS overlay representation leads
to inefficient path search and might yield wrong answers when asked for
recursive or regular directory iteration.
Currently, when reading an YAML file, each YAML root entry is placed
inside a new root in the filesystem overlay. In the crash reproducer, a
simple "@import Foundation" currently maps to 43 roots, and when looking
up paths, we traverse a directory tree for each of these different
roots, until we find a match (or don't). This has two consequences:
- It's slow.
- Directory iteration gives incomplete results since it only return
results within one root - since contents of the same directory can be
declared inside different roots, the result isn't accurate.
This is in part fault of the way we currently write out the YAML file
when emitting the crash reproducer - we could generate only one root and
that would make it fast and correct again. However, we should not rely
on how the client writes the YAML, but provide a good internal
representation regardless.
This patch builds a proper virtual directory tree out of the YAML
representation, allowing faster search and proper iteration. Besides the
crash reproducer, this potentially benefits other VFS clients.
llvm-svn: 269270
Bruno made a couple valiant attempts but the bot is still red.
This reverts r269100 (primary commit), r269108 (fix attempt), r269133
(fix attempt).
llvm-svn: 269160
The way we currently build the internal VFS overlay representation leads
to inefficient path search and might yield wrong answers when asked for
recursive or regular directory iteration.
Currently, when reading an YAML file, each YAML root entry is placed
inside a new root in the filesystem overlay. In the crash reproducer, a
simple "@import Foundation" currently maps to 43 roots, and when looking
up paths, we traverse a directory tree for each of these different
roots, until we find a match (or don't). This has two consequences:
- It's slow.
- Directory iteration gives incomplete results since it only return
results within one root - since contents of the same directory can be
declared inside different roots, the result isn't accurate.
This is in part fault of the way we currently write out the YAML file
when emitting the crash reproducer - we could generate only one root and
that would make it fast and correct again. However, we should not rely
on how the client writes the YAML, but provide a good internal
representation regardless.
This patch builds a proper virtual directory tree out of the YAML
representation, allowing faster search and proper iteration. Besides the
crash reproducer, this potentially benefits other VFS clients.
llvm-svn: 269100
Hide the real paths when rebuilding from VFS by setting up the crash
reproducer to use 'use-external-names' = false. This way we avoid
module redifinition errors and consistently use the same paths against
all modules.
With this change on Darwin we are able to simulate a crash for a simple
application using "Foundation/Foundation.h" (which relies on a bunch of
different frameworks and headers) and successfully rebuild all the
modules by relying solely at the VFS overlay.
llvm-svn: 266234
The VFS YAML files contain empty directory entries to describe that it's
returning from a subdirectory before describing new files in the parent.
In the future, we should properly sort and write YAML files avoiding
such empty dirs and mitigate the extra recurson cost. However, since
this is used by previous existing YAMLs, make the traversal work in
their presence.
rdar://problem/24499339
llvm-svn: 264970
The FileSystem::makeAbsolute function has been calculating the current
working directory unconditionally, even when it is not needed. This calls
down to llvm::sys::fs::current_path, which is relatively expensive
because it stats two directories, regardless of whether those paths are
already in the stat cache. The net effect is that when using the
VFS, every stat during header search turns into three stats. With this
change, we get back to a single stat for absolute directory paths.
llvm-svn: 264519
This reapplies r261552 and r263748. Fixed testcase to reapply.
The VFS overlay mapping between virtual paths and real paths is done through
the 'external-contents' entries in YAML files, which contains hardcoded paths
to the real files.
When a module compilation crashes, headers are dumped into <name>.cache/vfs
directory and are mapped via the <name>.cache/vfs/vfs.yaml. The script
generated for reproduction uses -ivfsoverlay pointing to file to gather the
mapping between virtual paths and files inside <name>.cache/vfs. Currently, we
are only capable of reproducing such crashes in the same machine as they
happen, because of the hardcoded paths in 'external-contents'.
To be able to reproduce a crash in another machine, this patch introduces a new
option in the VFS yaml file called 'overlay-relative'. When it's equal to
'true' it means that the provided path to the YAML file through the
-ivfsoverlay option should also be used to prefix the final path for every
'external-contents'.
Example, given the invocation snippet "... -ivfsoverlay
<name>.cache/vfs/vfs.yaml" and the following entry in the yaml file:
"overlay-relative": "true",
"roots": [
...
"type": "directory",
"name": "/usr/include",
"contents": [
{
"type": "file",
"name": "stdio.h",
"external-contents": "/usr/include/stdio.h"
},
...
Here, a file manager request for virtual "/usr/include/stdio.h", that will map
into real path "/<absolute_path_to>/<name>.cache/vfs/usr/include/stdio.h.
This is a useful feature for debugging module crashes in machines other than
the one where the error happened.
Differential Revision: http://reviews.llvm.org/D17457
rdar://problem/24499339
llvm-svn: 263893
This reapplies r261552.
The VFS overlay mapping between virtual paths and real paths is done through
the 'external-contents' entries in YAML files, which contains hardcoded paths
to the real files.
When a module compilation crashes, headers are dumped into <name>.cache/vfs
directory and are mapped via the <name>.cache/vfs/vfs.yaml. The script
generated for reproduction uses -ivfsoverlay pointing to file to gather the
mapping between virtual paths and files inside <name>.cache/vfs. Currently, we
are only capable of reproducing such crashes in the same machine as they
happen, because of the hardcoded paths in 'external-contents'.
To be able to reproduce a crash in another machine, this patch introduces a new
option in the VFS yaml file called 'overlay-relative'. When it's equal to
'true' it means that the provided path to the YAML file through the
-ivfsoverlay option should also be used to prefix the final path for every
'external-contents'.
Example, given the invocation snippet "... -ivfsoverlay
<name>.cache/vfs/vfs.yaml" and the following entry in the yaml file:
"overlay-relative": "true",
"roots": [
...
"type": "directory",
"name": "/usr/include",
"contents": [
{
"type": "file",
"name": "stdio.h",
"external-contents": "/usr/include/stdio.h"
},
...
Here, a file manager request for virtual "/usr/include/stdio.h", that will map
into real path "/<absolute_path_to>/<name>.cache/vfs/usr/include/stdio.h.
This is a useful feature for debugging module crashes in machines other than
the one where the error happened.
Differential Revision: http://reviews.llvm.org/D17457
rdar://problem/24499339
llvm-svn: 263748
This was applied twice r261551 and 263617 and later reverted because:
(1) Windows bot failing on unittests. Change the current behavior to do
not handle path traversals on windows.
(2) Windows bot failed to include llvm/Config/config.h in order to use
HAVE_REALPATH. Use LLVM_ON_UNIX instead, as done in lib/Basic/FileManager.cpp.
Handle ".", ".." and "./" with trailing slashes while collecting files
to be dumped into the vfs overlay directory.
Include the support for symlinks into components. Given the path:
/install-dir/bin/../lib/clang/3.8.0/include/altivec.h, if "bin"
component is a symlink, it's not safe to use `path::remove_dots` here,
and `realpath` is used to get the right answer. Since `realpath`
is expensive, we only do it at collecting time (which only happens
during the crash reproducer) and cache the base directory for fast lookups.
Overall, this makes the input to the VFS YAML file to be canonicalized
to never contain traversal components.
Differential Revision: http://reviews.llvm.org/D17104
rdar://problem/24499339
llvm-svn: 263686
This is originally r261551, reverted because of windows bots failing on
unittests. Change the current behavior to do not handle path traversals
on windows.
Handle ".", ".." and "./" with trailing slashes while collecting files
to be dumped into the vfs overlay directory.
Include the support for symlinks into components. Given the path:
/install-dir/bin/../lib/clang/3.8.0/include/altivec.h, if "bin"
component is a symlink, it's not safe to use `path::remove_dots` here,
and `realpath` is used to get the right answer. Since `realpath`
is expensive, we only do it at collecting time (which only happens
during the crash reproducer) and cache the base directory for fast lookups.
Overall, this makes the input to the VFS YAML file to be canonicalized
to never contain traversal components.
Differential Revision: http://reviews.llvm.org/D17104
rdar://problem/24499339
llvm-svn: 263617
The SafelyCloseFileDescriptor machinery does the right thing in the face
of signals while close will do something platform specific which results
in the FD potentially getting leaked.
llvm-svn: 262687
The VFS overlay mapping between virtual paths and real paths is done through
the 'external-contents' entries in YAML files, which contains hardcoded paths
to the real files.
When a module compilation crashes, headers are dumped into <name>.cache/vfs
directory and are mapped via the <name>.cache/vfs/vfs.yaml. The script
generated for reproduction uses -ivfsoverlay pointing to file to gather the
mapping between virtual paths and files inside <name>.cache/vfs. Currently, we
are only capable of reproducing such crashes in the same machine as they
happen, because of the hardcoded paths in 'external-contents'.
To be able to reproduce a crash in another machine, this patch introduces a new
option in the VFS yaml file called 'overlay-relative'. When it's equal to
'true' it means that the provided path to the YAML file through the
-ivfsoverlay option should also be used to prefix the final path for every
'external-contents'.
Example, given the invocation snippet "... -ivfsoverlay
<name>.cache/vfs/vfs.yaml" and the following entry in the yaml file:
"overlay-relative": "true",
"roots": [
...
"type": "directory",
"name": "/usr/include",
"contents": [
{
"type": "file",
"name": "stdio.h",
"external-contents": "/usr/include/stdio.h"
},
...
Here, a file manager request for virtual "/usr/include/stdio.h", that will map
into real path "/<absolute_path_to>/<name>.cache/vfs/usr/include/stdio.h.
This is a useful feature for debugging module crashes in machines other than
the one where the error happened.
Differential Revision: http://reviews.llvm.org/D17457
rdar://problem/24499339
llvm-svn: 261552
Handle ".", ".." and "./" with trailing slashes while collecting files
to be dumped into the vfs overlay directory.
Include the support for symlinks into components. Given the path:
/install-dir/bin/../lib/clang/3.8.0/include/altivec.h, if "bin"
component is a symlink, it's not safe to use `path::remove_dots` here,
and `realpath` is used to get the right answer. Since `realpath`
is expensive, we only do it at collecting time (which only happens
during the crash reproducer) and cache the base directory for fast lookups.
Overall, this makes the input to the VFS YAML file to be canonicalized
to never contain traversal components.
Differential Revision: http://reviews.llvm.org/D17104
rdar://problem/24499339
llvm-svn: 261551
FixedCompilationDatabase sets the working dir to "." by default. For
chdir(".") this is a noop but this lead to InMemoryFileSystem to create
bogus paths. Fixes PR25327.
llvm-svn: 257260
Make RedirectedFileSystem::openFilForRead(path)->status() the same as
RedirectedFileSystem::status(path). Previously we would just get the
status of the underlying real file, which would not have the IsVFSMapped
bit set.
This fixes rebuilding a module that has an include that is relative to
the includer where we will lookup the real path of that file before we
lookup the VFS location.
rdar://problem/23640339
llvm-svn: 255312
Mandeep Singh Grang