maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity
llvm-project/clang/unittests/Serialization/InMemoryModuleCacheTest.cpp

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

120 lines
3.9 KiB
C++
Raw Normal View History

Modules: Rename MemoryBufferCache to InMemoryModuleCache Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to Serialization. Another patch will start using it to manage module build more explicitly, but this is split out because it's mostly mechanical. Because of the move to Serialization we can no longer abuse the Preprocessor to forward it to the ASTReader. Besides the rename and file move, that means Preprocessor::Preprocessor has one fewer parameter and ASTReader::ASTReader has one more. llvm-svn: 355777
2019-03-10 01:33:56 +08:00
//===- InMemoryModuleCacheTest.cpp - InMemoryModuleCache tests ------------===//
Reapply "Modules: Cache PCMs in memory and avoid a use-after-free" This reverts commit r298185, effectively reapplying r298165, after fixing the new unit tests (PR32338). The memory buffer generator doesn't null-terminate the MemoryBuffer it creates; this version of the commit informs getMemBuffer about that to avoid the assert. Original commit message follows: ---- Clang's internal build system for implicit modules uses lock files to ensure that after a process writes a PCM it will read the same one back in (without contention from other -cc1 commands). Since PCMs are read from disk repeatedly while invalidating, building, and importing, the lock is not released quickly. Furthermore, the LockFileManager is not robust in every environment. Other -cc1 commands can stall until timeout (after about eight minutes). This commit changes the lock file from being necessary for correctness to a (possibly dubious) performance hack. The remaining benefit is to reduce duplicate work in competing -cc1 commands which depend on the same module. Follow-up commits will change the internal build system to continue after a timeout, and reduce the timeout. Perhaps we should reconsider blocking at all. This also fixes a use-after-free, when one part of a compilation validates a PCM and starts using it, and another tries to swap out the PCM for something new. The PCMCache is a new type called MemoryBufferCache, which saves memory buffers based on their filename. Its ownership is shared by the CompilerInstance and ModuleManager. - The ModuleManager stores PCMs there that it loads from disk, never touching the disk if the cache is hot. - When modules fail to validate, they're removed from the cache. - When a CompilerInstance is spawned to build a new module, each already-loaded PCM is assumed to be valid, and is frozen to avoid the use-after-free. - Any newly-built module is written directly to the cache to avoid the round-trip to the filesystem, making lock files unnecessary for correctness. Original patch by Manman Ren; most testcases by Adrian Prantl! llvm-svn: 298278
2017-03-21 01:58:26 +08:00
//
Update the file headers across all of the LLVM projects in the monorepo to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
2019-01-19 16:50:56 +08:00
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
Reapply "Modules: Cache PCMs in memory and avoid a use-after-free" This reverts commit r298185, effectively reapplying r298165, after fixing the new unit tests (PR32338). The memory buffer generator doesn't null-terminate the MemoryBuffer it creates; this version of the commit informs getMemBuffer about that to avoid the assert. Original commit message follows: ---- Clang's internal build system for implicit modules uses lock files to ensure that after a process writes a PCM it will read the same one back in (without contention from other -cc1 commands). Since PCMs are read from disk repeatedly while invalidating, building, and importing, the lock is not released quickly. Furthermore, the LockFileManager is not robust in every environment. Other -cc1 commands can stall until timeout (after about eight minutes). This commit changes the lock file from being necessary for correctness to a (possibly dubious) performance hack. The remaining benefit is to reduce duplicate work in competing -cc1 commands which depend on the same module. Follow-up commits will change the internal build system to continue after a timeout, and reduce the timeout. Perhaps we should reconsider blocking at all. This also fixes a use-after-free, when one part of a compilation validates a PCM and starts using it, and another tries to swap out the PCM for something new. The PCMCache is a new type called MemoryBufferCache, which saves memory buffers based on their filename. Its ownership is shared by the CompilerInstance and ModuleManager. - The ModuleManager stores PCMs there that it loads from disk, never touching the disk if the cache is hot. - When modules fail to validate, they're removed from the cache. - When a CompilerInstance is spawned to build a new module, each already-loaded PCM is assumed to be valid, and is frozen to avoid the use-after-free. - Any newly-built module is written directly to the cache to avoid the round-trip to the filesystem, making lock files unnecessary for correctness. Original patch by Manman Ren; most testcases by Adrian Prantl! llvm-svn: 298278
2017-03-21 01:58:26 +08:00
//
//===----------------------------------------------------------------------===//
Modules: Rename MemoryBufferCache to InMemoryModuleCache Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to Serialization. Another patch will start using it to manage module build more explicitly, but this is split out because it's mostly mechanical. Because of the move to Serialization we can no longer abuse the Preprocessor to forward it to the ASTReader. Besides the rename and file move, that means Preprocessor::Preprocessor has one fewer parameter and ASTReader::ASTReader has one more. llvm-svn: 355777
2019-03-10 01:33:56 +08:00
#include "clang/Serialization/InMemoryModuleCache.h"
Reapply "Modules: Cache PCMs in memory and avoid a use-after-free" This reverts commit r298185, effectively reapplying r298165, after fixing the new unit tests (PR32338). The memory buffer generator doesn't null-terminate the MemoryBuffer it creates; this version of the commit informs getMemBuffer about that to avoid the assert. Original commit message follows: ---- Clang's internal build system for implicit modules uses lock files to ensure that after a process writes a PCM it will read the same one back in (without contention from other -cc1 commands). Since PCMs are read from disk repeatedly while invalidating, building, and importing, the lock is not released quickly. Furthermore, the LockFileManager is not robust in every environment. Other -cc1 commands can stall until timeout (after about eight minutes). This commit changes the lock file from being necessary for correctness to a (possibly dubious) performance hack. The remaining benefit is to reduce duplicate work in competing -cc1 commands which depend on the same module. Follow-up commits will change the internal build system to continue after a timeout, and reduce the timeout. Perhaps we should reconsider blocking at all. This also fixes a use-after-free, when one part of a compilation validates a PCM and starts using it, and another tries to swap out the PCM for something new. The PCMCache is a new type called MemoryBufferCache, which saves memory buffers based on their filename. Its ownership is shared by the CompilerInstance and ModuleManager. - The ModuleManager stores PCMs there that it loads from disk, never touching the disk if the cache is hot. - When modules fail to validate, they're removed from the cache. - When a CompilerInstance is spawned to build a new module, each already-loaded PCM is assumed to be valid, and is frozen to avoid the use-after-free. - Any newly-built module is written directly to the cache to avoid the round-trip to the filesystem, making lock files unnecessary for correctness. Original patch by Manman Ren; most testcases by Adrian Prantl! llvm-svn: 298278
2017-03-21 01:58:26 +08:00
#include "llvm/Support/MemoryBuffer.h"
#include "gtest/gtest.h"
using namespace llvm;
using namespace clang;
namespace {
std::unique_ptr<MemoryBuffer> getBuffer(int I) {
SmallVector<char, 8> Bytes;
raw_svector_ostream(Bytes) << "data:" << I;
return MemoryBuffer::getMemBuffer(StringRef(Bytes.data(), Bytes.size()), "",
/* RequiresNullTerminator = */ false);
}
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
TEST(InMemoryModuleCacheTest, initialState) {
Modules: Rename MemoryBufferCache to InMemoryModuleCache Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to Serialization. Another patch will start using it to manage module build more explicitly, but this is split out because it's mostly mechanical. Because of the move to Serialization we can no longer abuse the Preprocessor to forward it to the ASTReader. Besides the rename and file move, that means Preprocessor::Preprocessor has one fewer parameter and ASTReader::ASTReader has one more. llvm-svn: 355777
2019-03-10 01:33:56 +08:00
InMemoryModuleCache Cache;
Revert "[modules] Do not cache invalid state for modules that we attempted to load." As per comment on https://reviews.llvm.org/D72860, it is suggested to revert this change in the meantime, since it has introduced regression. This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
2020-03-11 01:59:26 +08:00
EXPECT_EQ(InMemoryModuleCache::Unknown, Cache.getPCMState("B"));
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
EXPECT_FALSE(Cache.isPCMFinal("B"));
Revert "[modules] Do not cache invalid state for modules that we attempted to load." As per comment on https://reviews.llvm.org/D72860, it is suggested to revert this change in the meantime, since it has introduced regression. This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
2020-03-11 01:59:26 +08:00
EXPECT_FALSE(Cache.shouldBuildPCM("B"));
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
#if !defined(NDEBUG) && GTEST_HAS_DEATH_TEST
Revert "[modules] Do not cache invalid state for modules that we attempted to load." As per comment on https://reviews.llvm.org/D72860, it is suggested to revert this change in the meantime, since it has introduced regression. This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
2020-03-11 01:59:26 +08:00
EXPECT_DEATH(Cache.tryToDropPCM("B"), "PCM to remove is unknown");
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
EXPECT_DEATH(Cache.finalizePCM("B"), "PCM to finalize is unknown");
#endif
Reapply "Modules: Cache PCMs in memory and avoid a use-after-free" This reverts commit r298185, effectively reapplying r298165, after fixing the new unit tests (PR32338). The memory buffer generator doesn't null-terminate the MemoryBuffer it creates; this version of the commit informs getMemBuffer about that to avoid the assert. Original commit message follows: ---- Clang's internal build system for implicit modules uses lock files to ensure that after a process writes a PCM it will read the same one back in (without contention from other -cc1 commands). Since PCMs are read from disk repeatedly while invalidating, building, and importing, the lock is not released quickly. Furthermore, the LockFileManager is not robust in every environment. Other -cc1 commands can stall until timeout (after about eight minutes). This commit changes the lock file from being necessary for correctness to a (possibly dubious) performance hack. The remaining benefit is to reduce duplicate work in competing -cc1 commands which depend on the same module. Follow-up commits will change the internal build system to continue after a timeout, and reduce the timeout. Perhaps we should reconsider blocking at all. This also fixes a use-after-free, when one part of a compilation validates a PCM and starts using it, and another tries to swap out the PCM for something new. The PCMCache is a new type called MemoryBufferCache, which saves memory buffers based on their filename. Its ownership is shared by the CompilerInstance and ModuleManager. - The ModuleManager stores PCMs there that it loads from disk, never touching the disk if the cache is hot. - When modules fail to validate, they're removed from the cache. - When a CompilerInstance is spawned to build a new module, each already-loaded PCM is assumed to be valid, and is frozen to avoid the use-after-free. - Any newly-built module is written directly to the cache to avoid the round-trip to the filesystem, making lock files unnecessary for correctness. Original patch by Manman Ren; most testcases by Adrian Prantl! llvm-svn: 298278
2017-03-21 01:58:26 +08:00
}
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
TEST(InMemoryModuleCacheTest, addPCM) {
auto B = getBuffer(1);
auto *RawB = B.get();
Modules: Rename MemoryBufferCache to InMemoryModuleCache Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to Serialization. Another patch will start using it to manage module build more explicitly, but this is split out because it's mostly mechanical. Because of the move to Serialization we can no longer abuse the Preprocessor to forward it to the ASTReader. Besides the rename and file move, that means Preprocessor::Preprocessor has one fewer parameter and ASTReader::ASTReader has one more. llvm-svn: 355777
2019-03-10 01:33:56 +08:00
InMemoryModuleCache Cache;
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
EXPECT_EQ(RawB, &Cache.addPCM("B", std::move(B)));
Revert "[modules] Do not cache invalid state for modules that we attempted to load." As per comment on https://reviews.llvm.org/D72860, it is suggested to revert this change in the meantime, since it has introduced regression. This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
2020-03-11 01:59:26 +08:00
EXPECT_EQ(InMemoryModuleCache::Tentative, Cache.getPCMState("B"));
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
EXPECT_EQ(RawB, Cache.lookupPCM("B"));
EXPECT_FALSE(Cache.isPCMFinal("B"));
Revert "[modules] Do not cache invalid state for modules that we attempted to load." As per comment on https://reviews.llvm.org/D72860, it is suggested to revert this change in the meantime, since it has introduced regression. This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
2020-03-11 01:59:26 +08:00
EXPECT_FALSE(Cache.shouldBuildPCM("B"));
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
#if !defined(NDEBUG) && GTEST_HAS_DEATH_TEST
EXPECT_DEATH(Cache.addPCM("B", getBuffer(2)), "Already has a PCM");
Revert "[modules] Do not cache invalid state for modules that we attempted to load." As per comment on https://reviews.llvm.org/D72860, it is suggested to revert this change in the meantime, since it has introduced regression. This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
2020-03-11 01:59:26 +08:00
EXPECT_DEATH(Cache.addBuiltPCM("B", getBuffer(2)),
"Trying to override tentative PCM");
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
#endif
}
Revert "[modules] Do not cache invalid state for modules that we attempted to load." As per comment on https://reviews.llvm.org/D72860, it is suggested to revert this change in the meantime, since it has introduced regression. This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
2020-03-11 01:59:26 +08:00
TEST(InMemoryModuleCacheTest, addBuiltPCM) {
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
auto B = getBuffer(1);
auto *RawB = B.get();
InMemoryModuleCache Cache;
Revert "[modules] Do not cache invalid state for modules that we attempted to load." As per comment on https://reviews.llvm.org/D72860, it is suggested to revert this change in the meantime, since it has introduced regression. This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
2020-03-11 01:59:26 +08:00
EXPECT_EQ(RawB, &Cache.addBuiltPCM("B", std::move(B)));
EXPECT_EQ(InMemoryModuleCache::Final, Cache.getPCMState("B"));
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
EXPECT_EQ(RawB, Cache.lookupPCM("B"));
EXPECT_TRUE(Cache.isPCMFinal("B"));
Revert "[modules] Do not cache invalid state for modules that we attempted to load." As per comment on https://reviews.llvm.org/D72860, it is suggested to revert this change in the meantime, since it has introduced regression. This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
2020-03-11 01:59:26 +08:00
EXPECT_FALSE(Cache.shouldBuildPCM("B"));
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
#if !defined(NDEBUG) && GTEST_HAS_DEATH_TEST
EXPECT_DEATH(Cache.addPCM("B", getBuffer(2)), "Already has a PCM");
Revert "[modules] Do not cache invalid state for modules that we attempted to load." As per comment on https://reviews.llvm.org/D72860, it is suggested to revert this change in the meantime, since it has introduced regression. This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
2020-03-11 01:59:26 +08:00
EXPECT_DEATH(Cache.addBuiltPCM("B", getBuffer(2)),
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
"Trying to override finalized PCM");
#endif
}
Revert "[modules] Do not cache invalid state for modules that we attempted to load." As per comment on https://reviews.llvm.org/D72860, it is suggested to revert this change in the meantime, since it has introduced regression. This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
2020-03-11 01:59:26 +08:00
TEST(InMemoryModuleCacheTest, tryToDropPCM) {
Stop relying on allocator behaviour in modules unit test Another fixup for r355778 for Windows bots, this time to stop accidentally relying on allocator behaviour for the test to pass. llvm-svn: 355780
2019-03-10 04:15:01 +08:00
auto B1 = getBuffer(1);
auto B2 = getBuffer(2);
auto *RawB1 = B1.get();
auto *RawB2 = B2.get();
ASSERT_NE(RawB1, RawB2);
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
InMemoryModuleCache Cache;
Revert "[modules] Do not cache invalid state for modules that we attempted to load." As per comment on https://reviews.llvm.org/D72860, it is suggested to revert this change in the meantime, since it has introduced regression. This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
2020-03-11 01:59:26 +08:00
EXPECT_EQ(InMemoryModuleCache::Unknown, Cache.getPCMState("B"));
Stop relying on allocator behaviour in modules unit test Another fixup for r355778 for Windows bots, this time to stop accidentally relying on allocator behaviour for the test to pass. llvm-svn: 355780
2019-03-10 04:15:01 +08:00
EXPECT_EQ(RawB1, &Cache.addPCM("B", std::move(B1)));
Revert "[modules] Do not cache invalid state for modules that we attempted to load." As per comment on https://reviews.llvm.org/D72860, it is suggested to revert this change in the meantime, since it has introduced regression. This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
2020-03-11 01:59:26 +08:00
EXPECT_FALSE(Cache.tryToDropPCM("B"));
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
EXPECT_EQ(nullptr, Cache.lookupPCM("B"));
Revert "[modules] Do not cache invalid state for modules that we attempted to load." As per comment on https://reviews.llvm.org/D72860, it is suggested to revert this change in the meantime, since it has introduced regression. This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
2020-03-11 01:59:26 +08:00
EXPECT_EQ(InMemoryModuleCache::ToBuild, Cache.getPCMState("B"));
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
EXPECT_FALSE(Cache.isPCMFinal("B"));
Revert "[modules] Do not cache invalid state for modules that we attempted to load." As per comment on https://reviews.llvm.org/D72860, it is suggested to revert this change in the meantime, since it has introduced regression. This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
2020-03-11 01:59:26 +08:00
EXPECT_TRUE(Cache.shouldBuildPCM("B"));
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
#if !defined(NDEBUG) && GTEST_HAS_DEATH_TEST
Revert "[modules] Do not cache invalid state for modules that we attempted to load." As per comment on https://reviews.llvm.org/D72860, it is suggested to revert this change in the meantime, since it has introduced regression. This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
2020-03-11 01:59:26 +08:00
EXPECT_DEATH(Cache.addPCM("B", getBuffer(2)), "Already has a PCM");
EXPECT_DEATH(Cache.tryToDropPCM("B"),
"PCM to remove is scheduled to be built");
EXPECT_DEATH(Cache.finalizePCM("B"), "Trying to finalize a dropped PCM");
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
#endif
// Add a new one.
Revert "[modules] Do not cache invalid state for modules that we attempted to load." As per comment on https://reviews.llvm.org/D72860, it is suggested to revert this change in the meantime, since it has introduced regression. This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
2020-03-11 01:59:26 +08:00
EXPECT_EQ(RawB2, &Cache.addBuiltPCM("B", std::move(B2)));
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
EXPECT_TRUE(Cache.isPCMFinal("B"));
// Can try to drop again, but this should error and do nothing.
Revert "[modules] Do not cache invalid state for modules that we attempted to load." As per comment on https://reviews.llvm.org/D72860, it is suggested to revert this change in the meantime, since it has introduced regression. This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
2020-03-11 01:59:26 +08:00
EXPECT_TRUE(Cache.tryToDropPCM("B"));
Stop relying on allocator behaviour in modules unit test Another fixup for r355778 for Windows bots, this time to stop accidentally relying on allocator behaviour for the test to pass. llvm-svn: 355780
2019-03-10 04:15:01 +08:00
EXPECT_EQ(RawB2, Cache.lookupPCM("B"));
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
}
TEST(InMemoryModuleCacheTest, finalizePCM) {
auto B = getBuffer(1);
auto *RawB = B.get();
InMemoryModuleCache Cache;
Revert "[modules] Do not cache invalid state for modules that we attempted to load." As per comment on https://reviews.llvm.org/D72860, it is suggested to revert this change in the meantime, since it has introduced regression. This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
2020-03-11 01:59:26 +08:00
EXPECT_EQ(InMemoryModuleCache::Unknown, Cache.getPCMState("B"));
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
EXPECT_EQ(RawB, &Cache.addPCM("B", std::move(B)));
// Call finalize.
Cache.finalizePCM("B");
Revert "[modules] Do not cache invalid state for modules that we attempted to load." As per comment on https://reviews.llvm.org/D72860, it is suggested to revert this change in the meantime, since it has introduced regression. This reverts commit 83f4c3af021cd5322ea10fd1c4e839874c1dae49.
2020-03-11 01:59:26 +08:00
EXPECT_EQ(InMemoryModuleCache::Final, Cache.getPCMState("B"));
Modules: Invalidate out-of-date PCMs as they're discovered Leverage the InMemoryModuleCache to invalidate a module the first time it fails to import (and to lock a module as soon as it's built or imported successfully). For implicit module builds, this optimizes importing deep graphs where the leaf module is out-of-date; see example near the end of the commit message. Previously the cache finalized ("locked in") all modules imported so far when starting a new module build. This was sufficient to prevent loading two versions of the same module, but was somewhat arbitrary and hard to reason about. Now the cache explicitly tracks module state, where each module must be one of: - Unknown: module not in the cache (yet). - Tentative: module in the cache, but not yet fully imported. - ToBuild: module found on disk could not be imported; need to build. - Final: module in the cache has been successfully built or imported. Preventing repeated failed imports avoids variation in builds based on shifting filesystem state. Now it's guaranteed that a module is loaded from disk exactly once. It now seems safe to remove FileManager::invalidateCache, but I'm leaving that for a later commit. The new, precise logic uncovered a pre-existing problem in the cache: the map key is the module filename, and different contexts use different filenames for the same PCM file. (In particular, the test Modules/relative-import-path.c does not build without this commit. r223577 started using a relative path to describe a module's base directory when importing it within another module. As a result, the module cache sees an absolute path when (a) building the module or importing it at the top-level, and a relative path when (b) importing the module underneath another one.) The "obvious" fix is to resolve paths using FileManager::getVirtualFile and change the map key for the cache to a FileEntry, but some contexts (particularly related to ASTUnit) have a shorter lifetime for their FileManager than the InMemoryModuleCache. This is worth pursuing further in a later commit; perhaps by tying together the FileManager and InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a VFS layer. For now, use the PCM's base directory as-written for constructing the filename to check the ModuleCache. Example ======= To understand the build optimization, first consider the build of a module graph TU -> A -> B -> C -> D with an empty cache: TU builds A' A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' If we build TU again, where A, B, C, and D are in the cache and D is out-of-date, we would previously get this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' A' imports B imports C imports D (out-of-date) builds B' B' imports C imports D (out-of-date) builds C' C' imports D (out-of-date) builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' After this commit, we'll immediateley invalidate A, B, C, and D when we first observe that D is out-of-date, giving this build: TU imports A imports B imports C imports D (out-of-date) TU builds A' // The same graph as an empty cache. A' builds B' B' builds C' C' builds D' imports D' B' imports C' imports D' A' imports B' imports C' imports D' TU imports A' imports B' imports C' imports D' The new build matches what we'd naively expect, pretty closely matching the original build with the empty cache. rdar://problem/48545366 llvm-svn: 355778
2019-03-10 01:44:01 +08:00
EXPECT_TRUE(Cache.isPCMFinal("B"));
Reapply "Modules: Cache PCMs in memory and avoid a use-after-free" This reverts commit r298185, effectively reapplying r298165, after fixing the new unit tests (PR32338). The memory buffer generator doesn't null-terminate the MemoryBuffer it creates; this version of the commit informs getMemBuffer about that to avoid the assert. Original commit message follows: ---- Clang's internal build system for implicit modules uses lock files to ensure that after a process writes a PCM it will read the same one back in (without contention from other -cc1 commands). Since PCMs are read from disk repeatedly while invalidating, building, and importing, the lock is not released quickly. Furthermore, the LockFileManager is not robust in every environment. Other -cc1 commands can stall until timeout (after about eight minutes). This commit changes the lock file from being necessary for correctness to a (possibly dubious) performance hack. The remaining benefit is to reduce duplicate work in competing -cc1 commands which depend on the same module. Follow-up commits will change the internal build system to continue after a timeout, and reduce the timeout. Perhaps we should reconsider blocking at all. This also fixes a use-after-free, when one part of a compilation validates a PCM and starts using it, and another tries to swap out the PCM for something new. The PCMCache is a new type called MemoryBufferCache, which saves memory buffers based on their filename. Its ownership is shared by the CompilerInstance and ModuleManager. - The ModuleManager stores PCMs there that it loads from disk, never touching the disk if the cache is hot. - When modules fail to validate, they're removed from the cache. - When a CompilerInstance is spawned to build a new module, each already-loaded PCM is assumed to be valid, and is frozen to avoid the use-after-free. - Any newly-built module is written directly to the cache to avoid the round-trip to the filesystem, making lock files unnecessary for correctness. Original patch by Manman Ren; most testcases by Adrian Prantl! llvm-svn: 298278
2017-03-21 01:58:26 +08:00
}
} // namespace