method. This lets us clean up the interface and make it more obvious that
this method is *really really* _Pragma specific.
Note that _Pragma handling uglifies the Lexer in the critical path. It would
be very interesting to consider making _Pragma remapping be a new special
lexer class of its own.
llvm-svn: 62425
"FileID" a concept that is now enforced by the compiler's type checker
instead of yet-another-random-unsigned floating around.
This is an important distinction from the "FileID" currently tracked by
SourceLocation. *That* FileID may refer to the start of a file or to a
chunk within it. The new FileID *only* refers to the file (and its
#include stack and eventually #line data), it cannot refer to a chunk.
FileID is a completely opaque datatype to all clients, only SourceManager
is allowed to poke and prod it.
llvm-svn: 62407
if warnings in system headers are disabled. isIdenticalTo can end up
calling the expensive getSpelling method, and other bad stuff and is
completely unneeded if the warning will be discarded anyway. rdar://6502956
llvm-svn: 62347
the "physical" location of tokens, refer to the "spelling" location.
This is more concrete and useful, tokens aren't really physical objects!
llvm-svn: 62309
- IdentifierInfo can now (optionally) have its string data not be
co-located with itself. This is for use with PTH. This aspect is a
little gross, as getName() and getLength() now make assumptions
about a possible alternate representation of IdentifierInfo.
Perhaps we should make IdentifierInfo have virtual methods?
IdentifierTable:
- Added class "IdentifierInfoLookup" that can be used by
IdentifierTable to perform "string -> IdentifierInfo" lookups using
an auxilliary data structure. This is used by PTH.
- Perform tests show that IdentifierTable::get() does not slow down
because of the extra check for the IdentiferInfoLookup object (the
regular StringMap lookup does enough work to mitigate the impact of
an extra null pointer check).
- The upshot is that now that some IdentifierInfo objects might be
owned by the IdentiferInfoLookup object. This should be reviewed.
PTH:
- Modified PTHManager::GetIdentifierInfo to *not* insert entries in
IdentifierTable's string map, and instead create IdentifierInfo
objects on the fly when mapping from persistent IDs to
IdentifierInfos. This saves a ton of work with string copies,
hashing, and StringMap lookup and resizing. This change was
motivated because when processing source files in the PTH cache we
don't need to do any string -> IdentifierInfo lookups.
- PTHManager now subclasses IdentifierInfoLookup, allowing clients of
IdentifierTable to transparently use IdentifierInfo objects managed
by the PTH file. PTHManager resolves "string -> IdentifierInfo"
queries by doing a binary search over a sorted table of identifier
strings in the PTH file (the exact algorithm we use can be changed
as needed).
These changes lead to the following performance changes when using PTH on Cocoa.h:
- fsyntax-only: 10% performance improvement
- Eonly: 30% performance improvement
llvm-svn: 62273
lexical order of the corresponding identifier strings. This will be used for a
forthcoming optimization. This slows down PTH generation time by 7%. We can
revert this change if the optimization proves to not be valuable.
llvm-svn: 62248
- Use canonical FileID when using getSpelling() caching. This
addresses some cache misses we were seeing with -fsyntax-only on
Cocoa.h
- Added Preprocessor::getPhysicalCharacterAt() utility method for
clients to grab the first character at a specified sourcelocation.
This uses the PTH spelling cache.
- Modified Sema::ActOnNumericConstant() to use
Preprocessor::getPhysicalCharacterAt() instead of
SourceManager::getCharacterData() (to get PTH hits).
These changes cause -fsyntax-only to not page in any sources from
Cocoa.h. We see a speedup of 27%.
llvm-svn: 62193
- Refactor caching logic into a helper class PTHSpellingSearch
- Allow "random accesses" in the spelling cache, thus catching the remaining
cases where 'getSpelling' wasn't hitting the PTH cache
For -Eonly, PTH, Cocoa.h:
- This reduces wall time by 3% (user time unchanged, sys time reduced)
- This reduces the amount of paged source by 1112K.
The remaining 1112K still being paged in is from somewhere else
(investigating).
llvm-svn: 62009
performance gain. Here's what we see for -Eonly on Cocoa.h (using PTH):
- wall time decreases by 21% (26% speedup overall)
- system time decreases by 35%
- user time decreases by 6%
These reductions are due to not paging source files just to get spellings for
literals. The solution in place doesn't appear to be 100% yet, as we still see
some of the pages for source files getting mapped in. Using -print-stats, we see
that SourceManager maps in 7179K less bytes of source text (reduction of 75%).
Will investigate why the remaining 25% are getting paged in.
With these changes, here's how PTH compares to non-PTH on Cocoa.h:
-Eonly: PTH takes 64% of the time as non-PTH (54% speedup)
-fsyntax-only: PTH takes 89% of the time as non-PTH (11% speedup)
llvm-svn: 61913
- Added stub PTHLexer::getSpelling() that will be used for fetching cached
spellings from the PTH file. This doesn't do anything yet.
- Added a hook in Preprocessor::getSpelling() to call PTHLexer::getSpelling()
when using a PTHLexer.
- Updated PTHLexer to read the offsets of spelling tables in the PTH file.
llvm-svn: 61911
- Encode the token length with 2 bytes instead of 4.
- This reduces the size of the .pth file for Cocoa.h by 12%.
- This speeds up PTH time (-Eonly) on Cocoa.h by 1.6%.
llvm-svn: 61364
- In PTHLexer::Lex read all of the token data from PTH file before
constructing the token. The idea is to enhance locality.
- Do not use Read8/Read32 in PTHLexer::Lex. Inline these operations manually.
- Change PTHManager::ReadIdentifierInfo() to PTHManager::GetIdentifierInfo().
They are functionally the same except that PTHLexer::Lex() reads the
persistent id.
These changes result in a 3.3% speedup for PTH on Cocoa.h (-Eonly).
llvm-svn: 61363
- Embed 'eom' tokens in PTH file.
- Use embedded 'eom' tokens to not lazily generate them in the PTHLexer.
This means that PTHLexer can always advance to the next token after
reading a token (instead of buffering tokens using a copy).
- Moved logic of 'ReadToken' into Lex. GetToken & ReadToken no longer exist.
- These changes result in a 3.3% speedup (-Eonly) on Cocoa.h.
- The code is a little gross. Many cleanups are possible and should be done.
llvm-svn: 61360
- Added a side-table per each token-cached file with the preprocessor conditional stack. This tracks what #if's are matched with what #endifs and where their respective tokens are in the PTH file. This will allow for quick skipping of excluded conditional branches in the Preprocessor.
- Performance testing shows the addition of this information (without actually utilizing it) leads to no performance regressions.
llvm-svn: 60911
- Added virtual method 'getSourceLocation()' (no arguments) that gets the location of the next "observable" location (e.g., next character, next token).
PPLexerChange.cpp:
- Implemented FIXME by using PreprocessorLexer::getSourceLocation() to get the location in the file we are returning to after lexing a #included file. This appears to be slightly faster than having the branch (i.e., 'if(CurLexer)'). It's also not a really hot part of the Preprocessor.
llvm-svn: 60860
- Added method "setPTHManager" that will be called by the driver to install
a PTHManager for the Preprocessor.
- Fixed some comments.
- Added EnterSourceFileWithPTH to mirror EnterSourceFileWithLexer.
llvm-svn: 60437
with implicit quotes around them. This has a bunch of follow-on
effects and requires tweaking to a whole lot of code. This causes
a regression in two tests (xfailed) by causing it to emit things like:
Line 10: duplicate interface declaration for category 'MyClass1' ('Category1')
instead of:
Line 10: duplicate interface declaration for category 'MyClass1(Category1)'
I will fix this in a follow-up commit.
As part of this, I had to start switching stuff to use ->getDeclName() instead
of Decl::getName() for consistency. This is good, but I was planning to do this
as an independent patch. There will be several follow-on patches
to clean up some of the mess, but this patch is already too big.
llvm-svn: 59917
would not eat the "-1" in "0x0p-1", but LiteralSupport would accept
it when extensions are on. This caused strangeness and failures
when hexfloats were properly treated as an extension (not error)
in LiteralSupport.
llvm-svn: 59865
its call sites. This makes it more explicit when the hasError flag is
getting set and removes a confusing difference in behavior between
PP.Diag and Diag in this code.
llvm-svn: 59863
(and carefully calculated) effect of allowing the compiler to reason
about the aliasing properties of DiagnosticBuilder object better,
allowing the whole thing to be promoted to registers instead of
resulting in a ton of stack traffic.
While I'm not very concerned about the performance of the Diag() method
invocations, I *am* more concerned about their code size and impact on the
non-diagnostic code. This patch shrinks the clang executable (in
release-asserts mode with gcc-4.2) from 14523980 to 14519816 bytes. This
isn't much, but it shrinks the lexer from 38192 to 37776, PPDirectives.o
from 31116 to 28868 bytes, etc.
llvm-svn: 59862
one for building up the diagnostic that is in flight (DiagnosticBuilder)
and one for pulling structured information out of the diagnostic when
formatting and presenting it.
There is no functionality change with this patch.
llvm-svn: 59849
- Move out logic for handling the end-of-file to LexEndOfFile (to match the Lexer) class. The logic now mirrors the Lexer class more, which allows us to pass most of the Preprocessor test cases.
llvm-svn: 59768
- Rename 'CurToken' and 'LastToken' to 'CurTokenIdx' and 'LastTokenIdx'
respectively.
- Add helper methods GetToken(), AdvanceToken(), AtLastToken() to abstract away
details of the token stream. This also allows us to easily replace their
implementation later.
llvm-svn: 59733
(temporary hack) to test the PTHLexer is that whenever we would create a Lexer
object we instead raw lex a memory buffer first and then use the PTHLexer. This
logic exists only to driver the PTHLexer and will be removed/changed in the
future. Note that the regular path using normal Lexer objects is what is used by
default.
llvm-svn: 59723
- Add variants of IsNonPragmaNonMacroLexer to accept an IncludeMacroStack entry
(simplifies some uses).
- Use IsNonPragmaNonMacroLexer in Preprocessor::LookupFile.
- Add 'FileID' to PreprocessorLexer, and have Preprocessor query this fileid
when looking up the FileEntry for a file
Performance testing of -Eonly on Cocoa.h shows no performance regression because
of this patch.
llvm-svn: 59666
- Add variants of IsNonPragmaNonMacroLexer to accept an IncludeMacroStack entry
(simplifies some uses).
- Use IsNonPragmaNonMacroLexer in Preprocessor::LookupFile.
Performance testing of -Eonly on Cocoa.h shows no performance regression because
of this patch.
llvm-svn: 59574
are formed. In particular, a diagnostic with all its strings and ranges is now
packaged up and sent to DiagnosticClients as a DiagnosticInfo instead of as a
ton of random stuff. This has the benefit of simplifying the interface, making
it more extensible, and allowing us to do more checking for things like access
past the end of the various arrays passed in.
In addition to introducing DiagnosticInfo, this also substantially changes how
Diagnostic::Report works. Instead of being passed in all of the info required
to issue a diagnostic, Report now takes only the required info (a location and
ID) and returns a fresh DiagnosticInfo *by value*. The caller is then free to
stuff strings and ranges into the DiagnosticInfo with the << operator. When
the dtor runs on the DiagnosticInfo object (which should happen at the end of
the statement), the diagnostic is actually emitted with all of the accumulated
information. This is a somewhat tricky dance, but it means that the
accumulated DiagnosticInfo is allowed to keep pointers to other expression
temporaries without those pointers getting invalidated.
This is just the minimal change to get this stuff working, but this will allow
us to eliminate the zillions of variant "Diag" methods scattered throughout
(e.g.) sema. For example, instead of calling:
Diag(BuiltinLoc, diag::err_overload_no_match, typeNames,
SourceRange(BuiltinLoc, RParenLoc));
We will soon be able to just do:
Diag(BuiltinLoc, diag::err_overload_no_match)
<< typeNames << SourceRange(BuiltinLoc, RParenLoc));
This scales better to support arbitrary types being passed in (not just
strings) in a type-safe way. Go operator overloading?!
llvm-svn: 59502
strings instead of array of strings. This reduces string copying
in some not-very-important cases, but paves the way for future
improvements.
llvm-svn: 59494
- Add static method to test if the current lexer is a non-macro/non-pragma
lexer.
- Refactor some code in PPLexerChange to use this static method.
- No performance change.
llvm-svn: 59486
This is because the PTHLexer will not support this method. Performance testing
on preprocessing Cocoa.h shows that this results in a negligible performance
difference (less than 1%).
I tried making Lexer::SetCommentRetentionState() an out-of-line function (a
precursor to making it a virtual function in PreprocessorLexer) and noticed a 1%
decrease in speed (it is called in a hot part of the Preprocessor).
llvm-svn: 59477
alias for the current PreprocessorLexer. No functionality change. Performance
testing shows this results in no performance degradation when preprocessing
Cocoa.h.
llvm-svn: 59474
PreprocessorLexer, which will either be a 'Lexer' or 'PTHLexer'.
- Added stub field 'CurPTHLexer' to keep track of the current PTHLexer.
- Modified IncludeStackInfo to track both the current PTHLexer and
current PreprocessorLexer.
llvm-svn: 59472
PreprocessorLexer now has a virtual method "IndirectLex" which allows it to call the lex method of its subclasses. This is not for performance intensive operations.
llvm-svn: 59185
etc more generic. For some targets, long may not be equal to pointer size. For
example: PIC16 has int as i16, ptr as i16 but long as i32.
Also fixed a few build warnings in assert() functions in CFRefCount.cpp,
CGDecl.cpp, SemaDeclCXX.cpp and ParseDeclCXX.cpp.
llvm-svn: 58501
even whitespace, as tokens from the file. This is enabled with
L->SetKeepWhitespaceMode(true) on a raw lexer. In this mode, you too
can use clang as a really complex version of 'cat' with code like this:
Lexer RawLex(SourceLocation::getFileLoc(SM.getMainFileID(), 0),
PP.getLangOptions(), File.first, File.second);
RawLex.SetKeepWhitespaceMode(true);
Token RawTok;
RawLex.LexFromRawLexer(RawTok);
while (RawTok.isNot(tok::eof)) {
std::cout << PP.getSpelling(RawTok);
RawLex.LexFromRawLexer(RawTok);
}
This will emit exactly the input file, with no canonicalization or other
translation. Realistic clients actually do something with the tokens of
course :)
llvm-svn: 57401
same we we do an unterminated string or character literal. This makes
it so we can guarantee that the lexer never calls into the
preprocessor (which would be suicide for a raw lexer).
llvm-svn: 57395
using LexRawToken, create one and use LexFromRawLexer. This avoids
twiddling the RawLexer flag around and simplifies some code (even
speeding raw lexing up a tiny bit).
This change also improves the token paster to use a Lexer on the stack
instead of new/deleting it.
llvm-svn: 57393
target indep code.
Note that this changes functionality on PIC16: it defines __INT_MAX__
correctly for it, and it changes sizeof(long) to 16-bits (to match
the size of pointer).
llvm-svn: 57132
NumericLiteral parser is not careful about overrun because
it should never be possible. It implicitly expects that its
input matched the regex for pp-constant. Because of this, it
knows it can't be pointing to a prefix of something that
looks like a number. This is all fine, except that __LINE__
does not prevent implicit concatenation from happening. Fix
__LINE__ to not do this.
llvm-svn: 56818
to whether the fileid is a 'extern c system header' in addition to whether it
is a system header, most of this is spreading plumbing around. Once we have that,
PPLexerChange bases its "file enter/exit" notifications to PPCallbacks to
base the system header state on FileIDInfo instead of HeaderSearch. Finally,
in Preprocessor::HandleIncludeDirective, mirror logic in GCC: the system headerness
of a file being entered can be set due to the #includer or the #includee.
llvm-svn: 56688
- Replace FIXME in Preprocessor::HandleIdentifier() with a check that avoids diagnosing extension tokens that originate from macro definitions.
llvm-svn: 55639
Note that this isn't really a complete fix; I think there are other
potential overrun situations. I don't really know what the best
systematic fix is, though.
llvm-svn: 55622
- Added as private members for each because it is not clear where to
put the common definition. Perhaps the IdentifierInfos all of these
"pseudo-keywords" should be collected into one place (this would
KnownFunctionIDs and Objective-C property IDs, for example).
Remove Token::isNamedIdentifier.
- There isn't a good reason to use strcmp when we have interned
strings, and there isn't a good reason to encourage clients to do
so.
llvm-svn: 54794
- Kill unnecessary #includes in .cpp files. This is an automatic
sweep so some things removed are actually used, but happen to be
included by a previous header. I tried to get rid of the obvious
examples and this was the easiest way to trim the #includes in one
fell swoop.
- We now return to regularly scheduled development.
llvm-svn: 54632
* Move FormatError() from TextDiagnostic up to DiagClient, remove now
empty class TextDiagnostic
* Make DiagClient optional for Diagnostic
This fixes the following problems:
* -html-diags (and probably others) does now output the same set of
warnings as console clang does
* nothing crashes if one forgets to call setHeaderSearch() on
TextDiagnostic
* some code duplication is removed
llvm-svn: 54620
1) New public methods added:
-EnableBacktrackAtThisPos
-DisableBacktrack
-Backtrack
-isBacktrackEnabled
2) LookAhead() implementation is replaced with a more efficient one.
3) LookNext() is removed.
llvm-svn: 54611
t2.c:1:17: warning: hexadecimal floating constants are a C99 feature
long double d = 0x0.0000003ffffffff00000p-16357L;
^
instead of emitting a weird error message that doesn't make sense:
t2.c:1:41: error: hexadecimal floating constants require an exponent
long double d = 0x0.0000003ffffffff00000p-16357L;
^
rdar://6096838
llvm-svn: 54035
related to pp-expressions. Doing so is pretty simple and this
patch implements it, yielding nice diagnostics like:
t.c:2:7: error: division by zero in preprocessor expression
#if 1 / (0 + 0)
~ ^ ~~~~~~~
t.c:5:14: error: expected ')' in preprocessor expression
#if (412 + 42
~~~~~~~~^
t.c:5:5: error: to match this '('
#if (412 + 42
^
t.c:10:10: warning: left side of operator converted from negative value to unsigned: -42 to 18446744073709551574
#if (-42 + 0U) / -2
~~~ ^ ~~
t.c:10:16: warning: right side of operator converted from negative value to unsigned: -2 to 18446744073709551614
#if (-42 + 0U) / -2
~~~~~~~~~~ ^ ~~
5 diagnostics generated.
llvm-svn: 50638
a) correct rejection of ',' in pp expressions.
b) the precedence of ',' was wrong w.r.t. ?:.
Thanks again to Neil for finding these and providing testcases.
llvm-svn: 50625
literal, indicate what character the error is about or where
it would be. For example:
pr2241.c:1:17: error: exponent has no digits
float f[] = { 1e , 1ee0 };
^
llvm-svn: 49996
clang.cpp: InitializePreprocessor now makes a copy of the contents of PredefinesBuffer and
passes it to the preprocessor object.
clang.cpp: DriverPreprocessorFactory now calls "InitializePreprocessor" instead of this being done in main().
html::HighlightMacros() now takes a PreprocessorFactory, allowing it to conjure up a new
Preprocessor to highlight macros.
class HTMLDiagnostics now takes a PreprocessorFactory* that it can use for html::HighlightMacros().
Updated clients of HTMLDiagnostics to use this new interface.
llvm-svn: 49875
lib dir and move all the libraries into it. This follows the main
llvm tree, and allows the libraries to be built in parallel. The
top level now enforces that all the libs are built before Driver,
but we don't care what order the libs are built in. This speeds
up parallel builds, particularly incremental ones.
llvm-svn: 48402
Chris Lattner