Disable or canonicalize compiler options that are not relevant in
explicit module builds, similar to what we already did for the modules
cache path. This reduces uninteresting differences between
command-lines, which is particularly useful if there is a tool that can
cache the compilations.
Differential Revision: https://reviews.llvm.org/D127883
In D92191, a bunch of test cases were added to check `clang-scan-deps` works in `clang-cl` mode as well.
We don't need to duplicate all test cases, though. Testing the few special cases we have in `clang-scan-deps` for `clang-cl` should be good enough:
1. Deducing output path (and therefore target name in our make output).
2. Ignoring `-Xclang` arguments in step 1.
3. Deducing resource directory by invoking the compiler executuable.
This test de-duplicates the extra clang-cl test cases.
Reviewed By: dexonsmith, saudi
Differential Revision: https://reviews.llvm.org/D121812
Dependency scanner test for resource directory deduction doesn't account for LLVM builds with custom `CLANG_RESOURCE_DIR`.
This patch ensures we don't hardcode the default behavior into the test and take into account the actual value. This is done by running `%clang -print-resource-dir` and using that as the expected value in test assertions.
New comment also clarifies this is different from running that command as part of the dependency scan.
Reviewed By: mgorny
Differential Revision: https://reviews.llvm.org/D115628
Some command-line codegen arguments are likely to differ between identical modules discovered from different translation units. This patch removes them to make builds deterministic and/or reduce the number of built modules.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D112923
The #pragma directives push_macro/pop_macro and include_alias may influence the #include / import directives encountered by dependency scanning tools like clang-scan-deps.
This patch ensures that those directives are not removed during source code minimization.
Differential Revision: https://reviews.llvm.org/D112088
The `clang-scan-deps` CLI tool invokes the compiler with `-print-resource-dir` in case the `-resource-dir` argument is missing from the compilation command line. This is to enable running the tool on compilation databases that use compiler from a different toolchain than `clang-scan-deps` itself. While this doesn't make sense when scanning modular builds (due to the `-cc1` arguments the tool generates), the tool can can be used to efficiently scan for file dependencies of non-modular builds too.
This patch stops deducing the resource directory by invoking the compiler by default. This mode can still be enabled by invoking `clang-scan-deps` with `--resource-dir-recipe invoke-compiler`. The new default is `--resource-dir-recipe modify-compiler-path` which relies on the resource directory deduction taking place in `Driver::Driver` which is based on the compiler path. This makes the default more aligned with the intended usage of the tool while still allowing it to serve other use-cases.
Note that this functionality was also influenced by D108979, where the dependency scanner stopped going through `ClangTool::run`. The function tried to deduce the resource directory based on the current executable path, which might not be what the users expect when invoked from within a shared library.
Depends on D108979.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D108366
One of main goals of the dependency scanner is to be strict about module compatibility. This is achieved through strict context hash. This patch ensures that strict context hash is enabled not only during the scan itself (and its minimized implicit build), but also when actually reporting the dependency.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D111720
To reduce the number of explicit builds of a single module, we can try to squash multiple occurrences of the module with different command-lines (and context hashes) by removing benign command-line options. The greatest contributors to benign differences between command-lines are the header search paths.
In this patch, the lookup cache in `HeaderSearch` is used to identify paths that were actually used when implicitly building the module during scanning. This information is serialized into the unhashed control block of the implicitly-built PCM. The dependency scanner then loads this and may use it to prune the header search paths before computing the context hash of the module and generating the command-line.
We could also prune the header search paths when serializing `HeaderSearchOptions` into the PCM. That way, we could do it only once instead of every load of the PCM file by dependency scanner. However, that would result in a PCM file whose contents don't produce the same context hash as the original build, which is probably highly surprising.
There is an alternative approach to storing extra information into the PCM: wire up preprocessor callbacks to capture the used header search paths on-the-fly during preprocessing of modularized headers (similar to what we currently do for the main source file and textual headers). Right now, that's not compatible with the fact that we do an actual implicit build producing PCM files during dependency scanning. The second run of dependency scanner loads the PCM from the first run, skipping the preprocessing altogether, which would result in different results between runs. We can revisit this approach when we stop building implicitly during dependency scanning.
Depends on D102923.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D102488
module lookup by name alone
This removes the need to create a fake source file that imports a
module.
rdar://64538073
Differential Revision: https://reviews.llvm.org/D109485
There are a number of language and preprocessor options that are reset in the `CompilerInvocation` that describes the build of an implicit module. This patch uses the logic for explicit modules as well.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D108710
The `ASTReader` populates `Module::PresumedModuleMapFile` only for top-level modules, not submodules. To avoid generating empty `-fmodule-map-file=` arguments, make discovered modules depend on top-level precompiled modules. The granularity of submodules is not important here.
The documentation of `Module::PresumedModuleMapFile` says this field is non-empty only when building from preprocessed source. This means there can still be cases where the dependency scanner generates empty `-fmodule-map-file=` arguments. That's being addressed in separate patch: D108544.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D108647
In this patch, the dependency scanner starts collecting precompiled dependencies from all encountered submodules, not only from top-level modules.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D108540
Whenever -fmodule-name=top_level_module name is parsed, and clang actually tries to
import top_level_module, the headers are imported textually and the module isn't actually
built. However, the dependency scanner could still record it as a potential dependency
if the module was reimported and thus recorded by the preprocessor callbacks.
This change avoids collecting this kind of module as a dependency by verifying that we don't
collect top level modules without actual PCM files.
Differential Revision: https://reviews.llvm.org/D106100
One of the goals of the dependency scanner is to generate command-lines that can be used to explicitly build modular dependencies of a translation unit. The only modifications to these command-lines should be for the purposes of explicit modular build.
However, the current version of dependency scanner leaks its implementation details into the command-lines.
The first problem is that the `clang-scan-deps` tool adjusts the original textual command-line (adding `-Eonly -M -MT <target> -sys-header-deps -Wno-error -o /dev/null `, removing `--serialize-diagnostics`) in order to set up the `DependencyScanning` library. This has been addressed in D103461, D104012, D104030, D104031, D104033. With these patches, the `DependencyScanning` library receives the unmodified `CompilerInvocation`, sets it up and uses it for the implicit modular build.
Finally, to prevent leaking the implementation details to the resulting command-lines, this patch generates them from the **original** unmodified `CompilerInvocation` rather than from the one that drives the implicit build.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D104036
To prevent the creation of diagnostics file, `clang-scan-deps` strips the corresponding command-line argument. This behavior is useful even when using the C++ `DependencyScanner` library.
This patch transforms stripping of command-line in `clang-scan-deps` into stripping of `CompilerInvocation` in `DependencyScanning`.
AFAIK, the `clang-cl` driver doesn't even accept `--serialize-diagnostics`, so I've removed the test. (It would fail with an unknown command-line argument otherwise.)
Note: Since we're generating command-lines for modular dependencies from `CompilerInvocation`, the `--serialize-diagnostics` will be dropped. This was already happening in `clang-scan-deps` before this patch, but it will now happen also when using `DependencyScanning` library directly. This is resolved in D104036.
Reviewed By: dexonsmith, arphaman
Differential Revision: https://reviews.llvm.org/D104012
When a translation unit uses a PCH and imports the same modules as the PCH, we'd prefer to resolve to those modules instead of inventing new modules and reporting them as modular dependencies. Since the PCH modules have already been built nudge the compiler to reuse them when deciding whether to build a new module and don't report them as regular modular dependencies.
Depends on D103524 & D103802.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D103526
When a project uses PCH with explicit modules, the build will look like this:
1. scan PCH dependencies
2. explicitly build PCH
3. scan TU dependencies
4. explicitly build TU
Step 2 produces an object file for the PCH, which the dependency scanner needs to read in step 3. This patch adds support for this.
The `clang-scan-deps` invocation in the attached test would fail without this change.
Depends on D103516.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D103519
This reapplies commit 95033eb3 that reverted commit 1d9e8e13.
The tests were failing on Windows due to spaces and backslashes in paths not being handled carefully.
This patch adds support for inferred modules to the dependency scanner.
Effectively a cherry-pick of https://github.com/apple/llvm-project/pull/699 authored by @Bigcheese with libclang and other changes omitted.
Contains following changes:
1. [Clang][ScanDeps] Ignore __inferred_module.map dependency.
* This shows up with inferred modules, but it doesn't exist on disk, so don't report it as a dependency.
2. [Clang][ScanDeps] Use the module map a module was inferred from for inferred modules.
Also includes a smoke test that uses clang-scan-deps output to perform an explicit build. There's no intention to duplicate whatever `test/Modules` contains, just to verify the produced command-line does "work" (with very loose definition of work).
Split from D100934.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D102495
This reverts commit 199c397482.
This time, clang-scan-deps's search for output argument in clang-cl command line will now ignore arguments preceded by "-Xclang".
That way, it won't detect a /o argument in "-Xclang -ivfsoverlay -Xclang /opt/subpath"
Initial patch description:
clang-scan-deps contains some command line parsing and modifications.
This patch adds support for clang-cl command options.
Differential Revision: https://reviews.llvm.org/D92191
clang-scan-deps contains some command line parsing and modifications.
This patch adds support for clang-cl command options.
Differential Revision: https://reviews.llvm.org/D92191
This fixes argument injection in clang command lines, by adding them before "--".
Previously, the arguments were injected at the end of the command line and could be added after "--", which would be wrongly interpreted as input file paths.
This fix is needed for a subsequent patch, see D92191.
Differential Revision: https://reviews.llvm.org/D95099
clang-scan-deps contains some command line parsing and modifications.
This patch adds support for clang-cl command options.
Differential Revision: https://reviews.llvm.org/D92191
Differential Revision: https://reviews.llvm.org/D70268
This is a recommit of f978ea4983 with a fix for the PowerPC failure.
The issue was that:
* `CompilerInstance::ExecuteAction` calls
`getTarget().adjust(getLangOpts());`.
* `PPCTargetInfo::adjust` changes `LangOptions::HasAltivec`.
* This happens after the first few calls to `getModuleHash`.
There’s even a FIXME saying:
```
// FIXME: We shouldn't need to do this, the target should be immutable once
// created. This complexity should be lifted elsewhere.
```
This only showed up on PowerPC because it's one of the few targets that
almost always changes a hashed langopt.
I looked into addressing the fixme, but that would be a much larger
change, and it's not the only thing that happens in `ExecuteAction` that
can change the module context hash. Instead I changed the code to not
call `getModuleHash` until after it has been modified in `ExecuteAction`.
This patch makes the minimizer more conservative to avoid missing dependency files that are brought in by __has_include
PP expressions that occur in a condition of an #if/#elif that was previously skipped. The __has_include PP expressions
can be used in an #if/#elif either directly, or through macro expansion, so we can't detect them at the time of minimization.
Differential Revision: https://reviews.llvm.org/D70936
Previously, given a CompilationDatabase with two commands for the same
source file we would report that file twice with the union of the
dependencies for each command both times.
This was due to the way `ClangTool` runs actions given an input source
file (see the comment in `DependencyScanningTool.cpp`). This commit adds
a `SingleCommandCompilationDatabase` that is created with each
`CompileCommand` in the original CDB, which is then used for each
`ClangTool` invocation. This gives us a single run of
`DependencyScanningAction` per `CompileCommand`.
I looked at using `AllTUsToolExecutor` which is a parallel tool
executor, but I'm not sure it's suitable for `clang-scan-deps` as it
does a lot more sharing of state than `AllTUsToolExecutor` expects.
Differential Revision: https://reviews.llvm.org/D69643
This fixes two issues that prevent simple uses of modules from working.
* We would previously minimize _every_ file opened by clang, even module maps
and module pcm files. Now we only minimize files with known extensions. It
would be better if we knew which files clang intended to open as a source
file, but this works for now.
* We previously cached every lookup, even failed lookups. This is a problem
because clang stats the module cache directory before building a module and
creating that directory. If we cache that failure then the subsequent pcm
load doesn't see the module cache and fails.
Overall this still leaves us building minmized modules on disk during scanning.
This will need to be improved eventually for performance, but this is correct,
and works for now.
Differential Revision: https://reviews.llvm.org/D68835
This test is failing on Windows bots, revert for now (will check the right fix and retry the patch).
Summary: This reverts commit 962ca076e5.
Reviewers: Bigcheese, jkorous, arphaman
Subscribers: dexonsmith, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D69079
llvm-svn: 375079
Summary:
Clang's -M mode includes these extra dependencies in its output and clang-scan-deps
should have equivalent behavior, so adding these extradeps to output just like
how its being done for ".d" file generation mode.
Reviewers: arphaman, dexonsmith, Bigcheese, jkorous
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D69017
llvm-svn: 375074
The goal is to have 100% fidelity in clang-scan-deps behavior when
--analyze is present in compilation command.
At the same time I don't want to break clang-tidy which expects
__static_analyzer__ macro defined as built-in.
I introduce new cc1 options (-setup-static-analyzer) that controls
the macro definition and is conditionally set in driver.
Differential Revision: https://reviews.llvm.org/D68093
llvm-svn: 374815
Summary:
It seems that when the CachingFileSystem is first given a file to open that is actually a directory, it incorrectly
caches that path to be errenous and throws an error when subsequently a directory open call is made for the same
path.
This change makes it so that we do NOT cache a path if it turns out we asked for a file when its a directory.
Reviewers: arphaman
Subscribers: dexonsmith, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D68193
llvm-svn: 374366
Scan deps tool crashes when called on a C++ file, containing an include
that has the same name as a directory.
The tool crashes since it finds foo/dir and tries to read that as a file and fails.
Patch by: kousikk (Kousik Kumar)
Differential Revision: https://reviews.llvm.org/D67091
llvm-svn: 371903
This resolves differences observed on LLVM + Clang when running the comparison between canonical
dependencies (full preprocessing, no file manager reused), and dependencies obtained
when the file manager was reused between the full preprocessing invocations.
llvm-svn: 371751
Ben Langmuir