This method is very hot, it is called when emitting diagnostics, in -E mode
and for many #pragma handlers. It scans through the whole source file to
count newlines, records and caches them in a vector.
The speedup from vectorization isn't very large, as we fall back to bytewise
scanning when we hit a newline. There might be a way to avoid leaving the sse
loop but everything I tried didn't work out because a call to push_back
clobbers xmm registers.
About 2% speedup on average on "clang -E > /dev/null" of all .cpp files in
clang's lib/Sema.
llvm-svn: 154204
from the one stored in the PCH/AST, while trying to load a SLocEntry.
We verify that all files of the PCH did not change before loading it but this is not enough because:
- The AST may have been 1) kept around, 2) to do queries on it.
- We may have 1) verified the PCH and 2) started parsing.
Between 1) and 2) files may change and we are going to have crashes because the rest of clang
cannot deal with the ASTReader failing to read a SLocEntry.
Handle this by recovering gracefully in such a case, by initializing the SLocEntry
with the info from the PCH/AST as well as reporting failure by the ASTReader.
rdar://10888929
llvm-svn: 151004
file in the source manager. This allows us to properly create and use
modules described by module map files without umbrella headers (or
with incompletely umbrella headers). More generally, we can actually
build a PCH file that makes use of file -> buffer remappings, which
could be useful in libclang in the future.
llvm-svn: 144830
of a ContentCache, since multiple FileIDs can have the same ContentCache
but the expanded macro arguments locations will be different.
llvm-svn: 140521
check whether the requested location points inside the precompiled preamble,
in which case the returned source location will be a "loaded" one.
llvm-svn: 140060
to increased calls to SourceManager::getFileID. (rdar://9992664)
Use a slightly different approach that is more efficient both in terms of speed
(no extra getFileID calls) and in SLocEntries reduction.
Comparing pre-r138129 and this patch we get:
For compiling SemaExpr.cpp reduction of SLocEntries by 26%.
For the boost enum library:
-SLocEntries -34% (note that this was -5% for r138129)
-Memory consumption -50%
-PCH size -31%
Reduced SLocEntries also benefit the hot function SourceManager::getFileID,
evident by the reduced "FileID scans".
llvm-svn: 138380
Currently getMacroArgExpandedLocation is very inefficient and for the case
of a location pointing at the main file it will end up checking almost all of
the SLocEntries. Make it faster:
-Use a map of macro argument chunks to their expanded source location. The map
is for a single source file, it's stored in the file's ContentCache and lazily
computed, like the source lines cache.
-In SLocEntry's FileInfo add an 'unsigned NumCreatedFIDs' field that keeps track
of the number of FileIDs (files and macros) that were created during preprocessing
of that particular file SLocEntry. This is useful when computing the macro argument
map in skipping included files while scanning for macro arg FileIDs that lexed from
a specific source file. Due to padding, the new field does not increase the size
of SLocEntry.
llvm-svn: 138225
If we pass it a source location that points inside a function macro argument,
the returned location will be the macro location in which the argument was expanded.
If a macro argument is used multiple times, the expanded location will
be at the first expansion of the argument.
e.g.
MY_MACRO(foo);
^
Passing a file location pointing at 'foo', will yield a macro location
where 'foo' was expanded into.
Make SourceManager::getLocation call getMacroArgExpandedLocation as well.
llvm-svn: 137794
etc. With this I think essentially all of the SourceManager APIs are
converted. Comments and random other bits of cleanup should be all thats
left.
llvm-svn: 136057
and various other 'expansion' based terms. I've tried to reformat where
appropriate and catch as many references in comments but I'm going to do
several more passes. Also I've tried to expand parameter names to be
more clear where appropriate.
llvm-svn: 136056
source locations from source locations loaded from an AST/PCH file.
Previously, loading an AST/PCH file involved carefully pre-allocating
space at the beginning of the source manager for the source locations
and FileIDs that correspond to the prefix, and then appending the
source locations/FileIDs used for parsing the remaining translation
unit. This design forced us into loading PCH files early, as a prefix,
whic has become a rather significant limitation.
This patch splits the SourceManager space into two parts: for source
location "addresses", the lower values (growing upward) are used to
describe parsed code, while upper values (growing downward) are used
for source locations loaded from AST/PCH files. Similarly, positive
FileIDs are used to describe parsed code while negative FileIDs are
used to file/macro locations loaded from AST/PCH files. As a result,
we can load PCH/AST files even during parsing, making various
improvemnts in the future possible, e.g., teaching #include <foo.h> to
look for and load <foo.h.gch> if it happens to be already available.
This patch was originally written by Sebastian Redl, then brought
forward to the modern age by Jonathan Turner, and finally
polished/finished by me to be committed.
llvm-svn: 135484
instantiation and improve diagnostics which are stem from macro
arguments to trace the argument itself back through the layers of macro
expansion.
This requires some tricky handling of the source locations, as the
argument appears to be expanded in the opposite direction from the
surrounding macro. This patch provides helper routines that encapsulate
the logic and explain the reasoning behind how we step through macros
during diagnostic printing.
This fixes the rest of the test cases originially in PR9279, and later
split out into PR10214 and PR10215.
There is still some more work we can do here to improve the macro
backtrace, but those will follow as separate patches.
llvm-svn: 134660
When a macro instantiation occurs, reserve a SLocEntry chunk with length the
full length of the macro definition source. Set the spelling location of this chunk
to point to the start of the macro definition and any tokens that are lexed directly
from the macro definition will get a location from this chunk with the appropriate offset.
For any tokens that come from argument expansion, '##' paste operator, etc. have their
instantiation location point at the appropriate place in the instantiated macro definition
(the argument identifier and the '##' token respectively).
This improves macro instantiation diagnostics:
Before:
t.c:5:9: error: invalid operands to binary expression ('struct S' and 'int')
int y = M(/);
^~~~
t.c:5:11: note: instantiated from:
int y = M(/);
^
After:
t.c:5:9: error: invalid operands to binary expression ('struct S' and 'int')
int y = M(/);
^~~~
t.c:3:20: note: instantiated from:
\#define M(op) (foo op 3);
~~~ ^ ~
t.c:5:11: note: instantiated from:
int y = M(/);
^
The memory savings for a candidate boost library that abuses the preprocessor are:
- 32% less SLocEntries (37M -> 25M)
- 30% reduction in PCH file size (900M -> 635M)
- 50% reduction in memory usage for the SLocEntry table (1.6G -> 800M)
llvm-svn: 134587
It would add up relative (decomposed) offsets like in getDecomposedSpellingLocSlowCase, but while
it makes sense to preserve the offset among lexed spelling locations, it doesn't make
sense to add anything to the offset of the instantiation location. The instantiation
location will be the same regardless of the relative offset in the tokens that were
instantiated.
This bug didn't actually affect anything because, currently, in practice we never create macro
locations with relative offset greater than 0.
llvm-svn: 134586
The small number of elements was determined by taking the median
file length in clang+llvm and /usr/include on OS X with xcode installed.
llvm-svn: 134496
during deserialization from a precompiled header, and update all of
its callers to note when this problem occurs and recover (more)
gracefully. Fixes <rdar://problem/9119249>.
llvm-svn: 129839
should report the original file name for contents of files that were overriden by other files,
otherwise it should report the name of the new file. Default is true.
Also add similar field in PreprocessorOptions and pass similar parameter in ASTUnit::LoadFromCommandLine.
llvm-svn: 127289
Allow remapping a file by specifying another filename whose contents should be loaded if the original
file gets loaded. This allows to override files without having to create & load buffers in advance.
llvm-svn: 127052
whose inode has changed since the file was first created and that is
being seen through a different path name (e.g., due to symlinks or
relative path elements), such that its FileEntry pointer doesn't match
a known FileEntry pointer. Since this requires a system call (to
stat()), we only perform this deeper checking if we can't find the
file by comparing FileEntry pointers.
Also, add a micro-optimization where we don't bother to compute line
numbers when given the location (1, 1). This improves the
efficiency of clang_getLocationForOffset().
llvm-svn: 124800
FileSystemOpts through a ton of apis, simplifying a lot of code.
This also fixes a latent bug in ASTUnit where it would invoke
methods on FileManager without creating one in some code paths
in cindextext.
llvm-svn: 120010
This patch completely defeated the "passing in a prestat'd size
to MemoryBuffer" optimization, leading to an extra fstat call for
every buffer opened, in order to find out if the datestamp and size
of the file on disk matches what is in the stat cache.
I fully admit that I don't completely understand what is going on here:
why punish code when a stat cache isn't in use? what is the point of a
stat cache if you have to turn around and stat stuff to validate it?
To resolve both these issues, just drop the modtime check and check the
file size, which is the important thing anyway. This should also resolve
PR6812, because presumably windows is stable when it comes to file sizes.
If the modtime is actually important, we should get it and keep it on the
first stat.
This eliminates 833 fstat syscalls when processing Cocoa.h, speeding up
system time on -Eonly Cocoa.h from 0.041 to 0.038s.
llvm-svn: 120001
-Move the stuff of Diagnostic related to creating/querying diagnostic IDs into a new DiagnosticIDs class.
-DiagnosticIDs can be shared among multiple Diagnostics for multiple translation units.
-The rest of the state in Diagnostic object is considered related and tied to one translation unit.
-Have Diagnostic point to the SourceManager that is related with. Diagnostic can now accept just a
SourceLocation instead of a FullSourceLoc.
-Reflect the changes to various interfaces.
llvm-svn: 119730
When -working-directory is passed in command line, file paths are resolved relative to the specified directory.
This helps both when using libclang (where we can't require the user to actually change the working directory)
and to help reproduce test cases when the reproduction work comes along.
--FileSystemOptions is introduced which controls how file system operations are performed (currently it just contains
the working directory value if set).
--FileSystemOptions are passed around to various interfaces that perform file operations.
--Opening & reading the content of files should be done only through FileManager. This is useful in general since
file operations will be abstracted in the future for the reproduction mechanism.
FileSystemOptions is independent of FileManager so that we can have multiple translation units sharing the same
FileManager but with different FileSystemOptions.
Addresses rdar://8583824.
llvm-svn: 118203
reparsing an ASTUnit. When saving a preamble, create a buffer larger
than the actual file we're working with but fill everything from the
end of the preamble to the end of the file with spaces (so the lexer
will quickly skip them). When we load the file, create a buffer of the
same size, filling it with the file and then spaces. Then, instruct
the lexer to start lexing after the preamble, therefore continuing the
parse from the spot where the preamble left off.
It's now possible to perform a simple preamble build + parse (+
reparse) with ASTUnit. However, one has to disable a bunch of checking
in the PCH reader to do so. That part isn't committed; it will likely
be handled with some other kind of flag (e.g., -fno-validate-pch).
As part of this, fix some issues with null termination of the memory
buffers created for the preamble; we were trying to explicitly
NULL-terminate them, even though they were also getting implicitly
NULL terminated, leading to excess warnings about NULL characters in
source files.
llvm-svn: 109445
to be algorithmically faster and avoid an std::map. This routine
basically boils down to finding the nearest common ancestor in a
tree, and we (implicitly) have information about nesting depth,
use it!
This wraps up rdar://7948633 - SourceManager::isBeforeInTranslationUnit has poor performance
llvm-svn: 103239
method to be correct. Right now it correctly computes the cache, then
goes ahead and computes the result the hard way, then asserts that they
match. Next I'll actually turn it on.
llvm-svn: 103231
method will sometimes return different results for the same input SourceLocations. I haven't
unraveled this method completely yet, so this truly is a workaround until a better fix comes
along.
llvm-svn: 103143
about it instead of producing tons of garbage from the lexer.
It would be even better for sourcemgr to dynamically transcode (e.g.
from UTF16 -> UTF8).
llvm-svn: 101924
precompiled headers and/or when reading the contents of the file into
memory. These checks seem to be causing spurious regression-test
failures on Windows.
llvm-svn: 100866
Diagnostic subsystem, which is used in the rare case where we find a
serious problem (i.e., an inconsistency in the file system) while
we're busy formatting another diagnostic. In this case, the delayed
diagnostic will be emitted after we're done with the other
diagnostic. This is only to be used for fatal conditions detected at
very inconvenient times, where we can neither stop the current
diagnostic in flight nor can we suppress the second error.
llvm-svn: 99175
deserialization of precompiled headers, where the deserialization of
the source location entry for a buffer (e.g., macro instantiation
scratch space) would overwrite a one-element FileID cache in the
source manager. When tickled at the wrong time, we would return the
wrong decomposed source location and eventually cause c-index-test to
crash.
Found by dumb luck. It's amazing this hasn't shown up before.
llvm-svn: 98940
buffer was invalid when it was created, and use that bit to always set
the "Invalid" flag according to whether the buffer is invalid. This
ensures that all accesses to an invalid buffer are marked invalid,
improving recovery.
llvm-svn: 98690
SourceManager's getBuffer() and, therefore, could fail, along with
Preprocessor::getSpelling(). Use the Invalid parameters in the literal
parsers (string, floating point, integral, character) to make them
robust against errors that stem from, e.g., PCH files that are not
consistent with the underlying file system.
I still need to audit every use caller to all of these routines, to
determine which ones need specific handling of error conditions.
llvm-svn: 98608
Benjamin Kramer