entities generated directly by the preprocessor from those loaded from
the external source (e.g., the ASTReader). By separating these two
sets of entities into different vectors, we allow both to grow
independently, and eliminate the need for preallocating all of the
loaded preprocessing entities. This is similar to the way the recent
SourceManager refactoring treats FileIDs and the source location
address space.
As part of this, switch over to building a continuous range map to
track preprocessing entities.
llvm-svn: 135646
source locations from source locations loaded from an AST/PCH file.
Previously, loading an AST/PCH file involved carefully pre-allocating
space at the beginning of the source manager for the source locations
and FileIDs that correspond to the prefix, and then appending the
source locations/FileIDs used for parsing the remaining translation
unit. This design forced us into loading PCH files early, as a prefix,
whic has become a rather significant limitation.
This patch splits the SourceManager space into two parts: for source
location "addresses", the lower values (growing upward) are used to
describe parsed code, while upper values (growing downward) are used
for source locations loaded from AST/PCH files. Similarly, positive
FileIDs are used to describe parsed code while negative FileIDs are
used to file/macro locations loaded from AST/PCH files. As a result,
we can load PCH/AST files even during parsing, making various
improvemnts in the future possible, e.g., teaching #include <foo.h> to
look for and load <foo.h.gch> if it happens to be already available.
This patch was originally written by Sebastian Redl, then brought
forward to the modern age by Jonathan Turner, and finally
polished/finished by me to be committed.
llvm-svn: 135484
This is switches all the interfaces points (and most of the commenst
/ local variables I saw on my way through) regarding the
NestedMacroInstantiations bit.
The libclang enums corresponding to this state were renamed, but
a legacy enum was added with the old name, and the same value to keep
existing clients working. I've added a documentation blurb for it, but
let me know if there is a canonical way to document legacy elemenst of
the libclang interface.
No functionality changed here, even in tests.
llvm-svn: 135141
clang_codeCompleteGetContexts(), that provides the client with
information about the context in which code completion has occurred
and what kinds of entities make sense as completions at that
point. Patch by Connor Wakamo!
llvm-svn: 134615
CompilerInvocation on the stack, because other objects (e.g., the
CompilerInstance) maintain an intrusive reference-counted pointer to
the CompilerInvocation. This doesn't matter in the normal case,
because we take back the CompilerInvocation. However, during crash
recovery, this leads to us trying to free an object on the stack, and
hilarity ensues. Fixes <rdar://problem/9652540>.
llvm-svn: 134245
CXTranslationUnit_NestedMacroInstantiations, which indicates whether
we want to see "nested" macro instantiations (e.g., those that occur
inside other macro instantiations) within the detailed preprocessing
record. Many clients (e.g., those that only care about visible tokens)
don't care about this information, and in code that uses preprocessor
metaprogramming, this information can have a very high cost.
Addresses <rdar://problem/9389320>.
llvm-svn: 130990
This change requires making a bunch of fundamental Clang structures (optionally) reference counted to allow correct
ownership semantics of these objects (e.g., ASTContext) to play out between an active ASTUnit and CompilerInstance
object.
llvm-svn: 128011
should report the original file name for contents of files that were overriden by other files,
otherwise it should report the name of the new file. Default is true.
Also add similar field in PreprocessorOptions and pass similar parameter in ASTUnit::LoadFromCommandLine.
llvm-svn: 127289
Allow remapping a file by specifying another filename whose contents should be loaded if the original
file gets loaded. This allows to override files without having to create & load buffers in advance.
llvm-svn: 127052
enumeration type, prioritize the enumeration constants and don't
provide completions for any other expressions. Fixes <rdar://problem/7283668>.
llvm-svn: 125991
code-completion results accessed via libclang, to extend the lifetime
of the allocator used for cached global code-completion results at
least until these completion results are destroyed. Fixes
<rdar://problem/8997369>.
llvm-svn: 125678
completions. We now compute a hash of the names of all top-level
declarations and macro definitions, and invalidate the cache when the
hash value changes.
llvm-svn: 125670
Store in PCH the directory that the PCH was originally created in.
If a header file is not found at the path that we expect it to be and the PCH file
was moved from its original location, try to resolve the file by assuming that
header+PCH were moved together and the header is in the same place relative to the PCH.
llvm-svn: 125576
overridden via remapping. Thus, when we create a "virtual" file in the
file manager, we still stat() the real file that lives behind it so
that we can provide proper uniquing based on inodes. This helps keep
the file manager much more consistent.
To take advantage of this when reparsing files in libclang, we disable
the use of the stat() cache when reparsing or performing code
completion, since the stat() cache is very likely to be out of date in
this use case.
llvm-svn: 124971
BumpPtrAllocator, rather than manually new/delete'ing them. This
optimization also allows us to avoid allocating memory for and copying
constant strings (e.g., "return", "class").
This also required embedding the priority and availability of results
within the code completion string, to avoid extra memory allocation
within libclang.
llvm-svn: 124673
trap the serialized preprocessing records (macro definitions, macro
instantiations, macro definitions) from the generation of the
precompiled preamble, then replay those when walking the list of
preprocessed entities. This eliminates a bug where clang_getCursor()
wasn't able to find preprocessed-entity cursors in the preamble.
llvm-svn: 120396
two copies, since they are fundamentally different
operations and the StringRef one should go away
(it shouldn't be part of FileManager at least).
Remove some dead arguments.
llvm-svn: 120013
FileSystemOpts through a ton of apis, simplifying a lot of code.
This also fixes a latent bug in ASTUnit where it would invoke
methods on FileManager without creating one in some code paths
in cindextext.
llvm-svn: 120010
This patch completely defeated the "passing in a prestat'd size
to MemoryBuffer" optimization, leading to an extra fstat call for
every buffer opened, in order to find out if the datestamp and size
of the file on disk matches what is in the stat cache.
I fully admit that I don't completely understand what is going on here:
why punish code when a stat cache isn't in use? what is the point of a
stat cache if you have to turn around and stat stuff to validate it?
To resolve both these issues, just drop the modtime check and check the
file size, which is the important thing anyway. This should also resolve
PR6812, because presumably windows is stable when it comes to file sizes.
If the modtime is actually important, we should get it and keep it on the
first stat.
This eliminates 833 fstat syscalls when processing Cocoa.h, speeding up
system time on -Eonly Cocoa.h from 0.041 to 0.038s.
llvm-svn: 120001
we were just getting a range covering only the property name, which is
certainly not correct (and broke token annotation, among other
things).
Also, teach libclang about the relationship between
@synthesize/@dynamic and @property, so we get property name and
cursor-reference information for @synthesize and @dynamic.
llvm-svn: 119409
caching global code-completion results. In particular, don't perform
either operation the first time we parse, but do both after the first
reparse.
llvm-svn: 119285
diagnostic-capturing client lives as long as the ASTUnit itself
does. Otherwise, we can end up with crashes when we get a diagnostic
outside of parsing/code completion. The circumstances under which this
happen are really hard to reproduce, because a file needs to change
from under us.
llvm-svn: 118751
When -working-directory is passed in command line, file paths are resolved relative to the specified directory.
This helps both when using libclang (where we can't require the user to actually change the working directory)
and to help reproduce test cases when the reproduction work comes along.
--FileSystemOptions is introduced which controls how file system operations are performed (currently it just contains
the working directory value if set).
--FileSystemOptions are passed around to various interfaces that perform file operations.
--Opening & reading the content of files should be done only through FileManager. This is useful in general since
file operations will be abstracted in the future for the reproduction mechanism.
FileSystemOptions is independent of FileManager so that we can have multiple translation units sharing the same
FileManager but with different FileSystemOptions.
Addresses rdar://8583824.
llvm-svn: 118203
timers to be dumped whenever the ASTUnit is destroyed. Instead, just
print the time elapsed for each operation after we perform the
operation.
llvm-svn: 117550
diagnostics produced by the driver itself. Previously, we were
allowing these to either be dropped or to slip through to stderr.
Fixes <rdar://problem/7595339>.
llvm-svn: 116285
clang_codeCompleteAt(). This uncovered a few issues with the latter:
- ASTUnit wasn't saving/restoring diagnostic state appropriately between
reparses and code completions.
- "Overload" completions weren't being passed through to the client
llvm-svn: 116241
preambles:
- When we rebuild a precompiled preamble, make sure to disable
skipping anything in the main file; we may have had leftover
preamble-skipping values in the lexer, which leads to very empty
preamble. This is a correctness issue.
- When we rebuild a precompiled preamble, clear out any prior state
in the Diagnostic object. Otherwise, we might think that there
were errors when we were building the preamble itself, and
therefore reject the resulting preamble. This is mainly a
performance issue.
- Don't remove old remappings when digging out the remapping for the
main file. Having the old mappings around does not hurt in the
common case (later remappings will just overwrite them), and is
important when we fail to find a preamble: we don't want to have
removed the remapping, because we'll need it later.
llvm-svn: 116041
identifier, we may have a Sema object but no translation unit scope
(because parsing is finished). In this case, we still need to update
the IdResolver, which might still be used when writing a PCH
containing another PCH (without chaining). This bug manifested as a
failure with precompiled preambles.
Also, add a little environment-variable-sensitive logging for
libclang.
llvm-svn: 114774
provided when the optimization is disabled. In particular, split
the completion context CCC_Other into two contexts: CCC_Other, which
means that it's an undisclosed context for which any other results are
unwelcome, and CCC_Recovery, which is used in recovery cases.
Since we're now using the completion context within the completion
results builder, make sure that it's always set to something.
Fixes <rdar://problem/8470644>.
llvm-svn: 114704
- In Objective-C, we prefer BOOL to bool for historic reasons;
slightly penalize "bool".
- Treat Nil macro as a NULL pointer constant.
- Treat YES, NO, true, and false macros as constants.
- Treat the bool macro as a type.
llvm-svn: 114356
expression, e.g., after the '(' that could also be a type cast. Here,
we provide types as code-completion results in C/Objective-C (C++
already had them), although we wouldn't in a normal expression context.
llvm-svn: 113904
preambles end up leaving the precompiled preambles around. This is by
design, since we do minimal cleanup during crash recovery. However,
it's unfortunate for testing, so introduce a hook that allows these
two tests to put the precompiled preamble somewhere where we can
delete them after testing.
llvm-svn: 113698
just abort creation of the precompiled preamble rather than doing
silly things. This is the second part of the fix for the weird
preamble-related failures on Windows. Big thanks to Francois Pichet
for the great detective work!
llvm-svn: 113697
Windows GetTempPath() function, and be sure to create the directory in
which the precompiled preamble will reside before creating the
temporary file itself.
llvm-svn: 113695
into the clients, e.g., the printing code-completion consumer and
c-index-test. Clients may want to re-sort the results anyway.
Provide a libclang function that sorts the results.
3rd try. How embarrassing.
llvm-svn: 112180
into the clients, e.g., the printing code-completion consumer and
c-index-test. Clients may want to re-sort the results anyway.
Provide a libclang function that sorts the results.
llvm-svn: 112149
code-completion results cached by ASTUnit, sort the resulting result
set. This makes testing far, far easier, so this commit also includes
tests for the previous few fixes.
llvm-svn: 112070
declarator. Here, we can only see a few things (e.g., cvr-qualifiers,
nested name specifiers) and we do not want to provide other non-macro
completions. Previously, we would end up in recovery mode and would
provide a large number of non-relevant completions.
llvm-svn: 111818
duplication between the constant and non-constant paths in all of this.
Implement ARM ABI semantics for member pointer constants and conversion.
llvm-svn: 111772
sure to (1) actually use the remapped files we were given rather
than old data, and (2) keep the remapped files alive until the
code-completion results are destroyed. Big thanks to Daniel for the
test case.
llvm-svn: 111597
and reenable crash recovery test.
- Reparsing is still very crashy / weird, so I had to sprinkle random code into
the remapped input to get it to do what I want (i.e., crash!).
llvm-svn: 111550
false (not true), so that the CompilerInstance will actually free data
structures when it's done. This fixes a major leak with libclang's
in-process code completion.
llvm-svn: 111457
translation unit, refresh code-completion results because they've
probably changed. However, enforce a cooldown period between
refreshes, to avoid thrashing.
llvm-svn: 111218
nested-name-specifiers. Also includes fixes to the generation of
nested-name-specifier result in the non-cached case; we were producing
lame results for namespaces and namespace aliases, which (1) didn't
always have nested-name-specifiers when we want them, and (2) did not
have the necessary "::" as part of the completion.
llvm-svn: 111203
the usage type of each declaration result, then compare those types to
the preferred type of the completion. This provides parity in the
priority calculation between the code-completion results produced
directly from Sema and those cached by ASTUnit.
For the standard Cocoa.h (+ others) example, there's a penalty of 3-4
hundredeths of a second when caching the global results (for ~31,000
results), because we need an ASTContext-agnostic representation of
types for the comparison, and therefore we use... strings. Eventually,
we'd like to implement a more efficient ASTContext-agnostic encoding
of types.
llvm-svn: 111165
type class, so that we can adjust priorities appropriately when the
preferred type for the context and the actual type of the completion
are similar.
This gets us one step closer to parity of the cached completion
results with the non-cached completion results.
llvm-svn: 111139
declarations (in addition to macros). Each kind of declaration maps to
a certain set of completion contexts, and the ASTUnit completion logic
introduces the completion strings for those declarations if the actual
code-completion occurs in one of the contexts where it matters.
There are a few new code-completion-context kinds. Without these,
certain completions (e.g., after "using namespace") would need to
suppress all global completions, which would be unfortunate.
Note that we don't get the priorities right for global completions,
because we don't have enough type information. We'll need a way to
compare types in an ASTContext-agnostic way before this can be
implemented.
llvm-svn: 111093
when the CXTranslationUnit_CacheCompletionResults option is given to
clang_parseTranslationUnit(). Essentially, we compute code-completion
results for macro definitions after we have parsed the file, then
store an ASTContext-agnostic version of those results (completion
string, cursor kind, priority, and active contexts) in the
ASTUnit. When performing code completion in that ASTUnit, we splice
the macro definition results into the results provided by the actual
code-completion (which has had macros turned off) before libclang gets
those results. We use completion context information to only splice in
those results that make sense for that context.
With a completion involving all of the macros from Cocoa.h and a few other
system libraries (totally ~8500 macro definitions) living in a
precompiled header, we get about a 9% performance improvement from
code completion, since we no longer have to deserialize all of the
macro definitions from the precompiled header.
Note that macro definitions are merely the canary; the cache is
designed to also support other top-level declarations, which should be
a bigger performance win. That optimization will be next.
Note also that there is no mechanism for determining when to throw
away the cache and recompute its contents.
llvm-svn: 111051
used when parsing (or re-parsing) a file. Also, when loading a
precompiled header into ASTUnit, create a Sema object that holds onto
semantic-analysis information.
llvm-svn: 111003
and create separate decl nodes for forward declarations and the
definition," which appears to be causing significant Objective-C
breakage.
llvm-svn: 110803
- Eagerly create ObjCInterfaceTypes for declarations.
- The two above changes lead to a 0.5% increase in memory use and no speed regression when parsing Cocoa.h. On the other hand, now chained PCH works when there's a forward declaration in one PCH and the interface definition in another.
- Add HandleInterestingDecl to ASTConsumer. PCHReader passes the "interesting" decls it finds to this function instead of HandleTopLevelDecl. The default implementation forwards to HandleTopLevelDecl, but ASTUnit's handler for example ignores them. This fixes a potential crash when lazy loading of PCH data would cause ASTUnit's "top level" declaration collection to change while being iterated.
llvm-svn: 110610
flags enumeration + default-generating function that allows
code-completion to be customized via the libclang API.
Plus, turn on spell-checking when performing code completion.
llvm-svn: 110319
completion within the translation unit using the same command-line
arguments for parsing the translation unit. Eventually, we'll reuse
the precompiled preamble to improve code-completion performance, and
this also gives us a place to cache results.
Expose this function via the new libclang function
clang_codeCompleteAt(), which performs the code completion within a
CXTranslationUnit. The completion occurs in-process
(clang_codeCompletion() runs code completion out-of-process).
llvm-svn: 110210
short "cooling off" period (defaulting to 5 reparses) before trying to
build a precompiled preamble again. Previously, if we failed to build
the precompiled preamble at any time, we just gave up the whole
charade any never tried again.
llvm-svn: 110187
declarations that we saw when creating the precompiled preamble, and
provide those declarations in addition to the declarations parsed in
the main source file when traversing top-level declarations. This
makes the use of precompiled preambles a pure optimization, rather
than changing the semantics of the parsed translation unit.
llvm-svn: 110131
creating the preamble and "replay" them when reusing the
preamble. Also, fix a thinko in the copying of the preamble when
building the precompiled preamble.
llvm-svn: 110061
of the usual consistency checks used to determine when a precompiled
header is incompatible with the translation unit it's being loaded
into.
Enable this option when loading a precompiled preamble, because the
preamble loader will be performing all of this checking itself. Enable
the preamble-based test now that it's working.
This option is also useful for debugging Clang's PCH
(<rdar://problem/7532213>).
llvm-svn: 109475
reparsing an ASTUnit. When saving a preamble, create a buffer larger
than the actual file we're working with but fill everything from the
end of the preamble to the end of the file with spaces (so the lexer
will quickly skip them). When we load the file, create a buffer of the
same size, filling it with the file and then spaces. Then, instruct
the lexer to start lexing after the preamble, therefore continuing the
parse from the spot where the preamble left off.
It's now possible to perform a simple preamble build + parse (+
reparse) with ASTUnit. However, one has to disable a bunch of checking
in the PCH reader to do so. That part isn't committed; it will likely
be handled with some other kind of flag (e.g., -fno-validate-pch).
As part of this, fix some issues with null termination of the memory
buffers created for the preamble; we were trying to explicitly
NULL-terminate them, even though they were also getting implicitly
NULL terminated, leading to excess warnings about NULL characters in
source files.
llvm-svn: 109445
appropriately-padded main file buffer (that has spaces in the extra
"reserved" space) and thread that buffer through to the parsing
function. This still does nothing.
llvm-svn: 109299
that preamble (the preamble text, preamble file, reserved main file
size). Check these details when we try to rebuild the precompiled
preamble, and when nothing has changed, re-use the precompiled
preamble.
This code is still very much a WIP, and can't even properly be tested
because we have no way to use the precompiled preamble yet. "Trust me"
llvm-svn: 109294
reparses an already-parsed translation unit. At the moment it's just a
convenience function, but we hope to use it for performance
optimizations.
llvm-svn: 108756
taking it in pieces.
- Fixes a problem where the Clang executable path was not initialized properly
on Win32, because sys::Path::getBasename() doesn't do what I always think it
does. Imagine that, a sys::Path interface that is confusing!
llvm-svn: 108667
- These inputs follow an abbreviated execution path, but are still worth handling by FrontendAction so they reuse all the other clang -cc1 features.
llvm-svn: 105582
the DeclContext for the translation unit. This is to workaround a fundamental issue in how
ObjC decls (within an @implementation) are parsed before the ObjCContainerDecl is available.
llvm-svn: 102944
ASTUnit. Previously, we would end up with use-after-free errors
because the Diagnostic object would be creating in one place (say,
CIndex) and its ownership would not be transferred into the
ASTUnit. Fixes <rdar://problem/7818608>.
llvm-svn: 100464
- Rename "Diagnostics" and related to "StoredDiagnostics", to better
capture what we're actually storing.
- Move SourceManager and FileManager to the heap.
llvm-svn: 100441
the macro definitions and macro instantiations that are found
during preprocessing. Preprocessing records are *not* generated by
default; rather, we provide a PPCallbacks subclass that hooks into the
existing callback mechanism to record this activity.
The only client of preprocessing records is CIndex, which keeps track
of macro definitions and instantations so that they can be exposed via
cursors. At present, only token annotation uses these facilities, and
only for macro instantiations; both will change in the near
future. However, with this change, token annotation properly annotates
macro instantiations that do not produce any tokens and instantiations
of macros that are later undef'd, improving our consistency.
Preprocessing directives that are not macro definitions are still
handled by clang_annotateTokens() via re-lexing, so that we don't have
to track every preprocessing directive in the preprocessing record.
Performance impact of preprocessing records is still TBD, although it
is limited to CIndex and therefore out of the path of the main compiler.
llvm-svn: 98836
copy the source buffers provided rather than referencing them
directly, so that the caller can free those buffers immediately after
calling clang_createTranslationUnitFromSourceFile(). Otherwise, we
risk hitting those buffers later (when building source ranges, forming
diagnostics, etc.).
llvm-svn: 97296
we attach diagnostics to translation units and code-completion
results, so they can be queried at any time.
To facilitate this, the new StoredDiagnostic class stores a diagnostic
in a serializable/deserializable form, and ASTUnit knows how to
capture diagnostics in this stored form. CIndex's CXDiagnostic is a
thin wrapper around StoredDiagnostic, providing a C interface to
stored or de-serialized diagnostics.
I've XFAIL'd one test case temporarily, because currently we end up
storing diagnostics in an ASTUnit that's never returned to the user
(because it contains errors). I'll introduce a temporary fix for this
soon; the real fix will be to allow us to return and query invalid ASTs.
llvm-svn: 96592
live as long as the ASTUnit. This is useful for clients which want to maintain
pointers to the LangOptions object which ultimately lives in the
CompilerInvocation, although it would be nice to make all of this ownership
stuff more explicit and obvious.
llvm-svn: 94924
from a source file.
- This allows CIndex to avoid iterating over all the top-level decls when using
a PCH, which means we deserialize far fewer decls.
llvm-svn: 90559
(clang/driver) command line arguments (including the source file).
- The arguments are expected to include the source file.
- The idea is that even though this is a somewhat odd API, its the form which
many tools can most easily use (for example, by interposing with the compiler).
Also, switch index-test's -ast-from-source to use this entry point, and provide
a -arg command line argument which can be used to test that the command line
arguments are handled correctly.
llvm-svn: 90288
Jonathan D. Turner