.def file for each library. This means that adding a diagnostic
to sema doesn't require all the other libraries to be rebuilt.
Patch by Anders Johnsen!
llvm-svn: 63111
Token now has a class of kinds for "literals", which include
numeric constants, strings, etc. These tokens can optionally have
a pointer to the start of the token in the lexer buffer. This
makes it faster to get spelling and do other gymnastics, because we
don't have to go through source locations.
This change is performance neutral, but will make other changes
more feasible down the road.
llvm-svn: 63028
ground work for implementing #line, and fixes the "out of macro ID's"
problem.
There is nothing particularly tricky about the code, other than the
very performance sensitive SourceManager::getFileID() method.
llvm-svn: 62978
Refactor how the preprocessor changes a token from being an tok::identifier to a
keyword (e.g. tok::kw_for). Instead of doing this in HandleIdentifier, hoist this
common case out into the caller, so that every keyword doesn't have to go through
HandleIdentifier. This drops time in HandleIdentifier from 1.25ms to .62ms, and
speeds up clang -Eonly with PTH by about 1%.
llvm-svn: 62855
tells us whether Preprocessor::HandleIdentifier needs to be called.
Because this method is only rarely needed, this saves a call and a
bunch of random checks. This drops the time in HandleIdentifier
from 3.52ms to .98ms on cocoa.h on my machine.
llvm-svn: 62675
the chunk ID not the file ID. This exposes problems in
TextDiagnosticPrinter where it should have been using the canonical
file ID but wasn't. Fix these along the way.
llvm-svn: 62427
method. This lets us clean up the interface and make it more obvious that
this method is *really really* _Pragma specific.
Note that _Pragma handling uglifies the Lexer in the critical path. It would
be very interesting to consider making _Pragma remapping be a new special
lexer class of its own.
llvm-svn: 62425
"FileID" a concept that is now enforced by the compiler's type checker
instead of yet-another-random-unsigned floating around.
This is an important distinction from the "FileID" currently tracked by
SourceLocation. *That* FileID may refer to the start of a file or to a
chunk within it. The new FileID *only* refers to the file (and its
#include stack and eventually #line data), it cannot refer to a chunk.
FileID is a completely opaque datatype to all clients, only SourceManager
is allowed to poke and prod it.
llvm-svn: 62407
the "physical" location of tokens, refer to the "spelling" location.
This is more concrete and useful, tokens aren't really physical objects!
llvm-svn: 62309
would not eat the "-1" in "0x0p-1", but LiteralSupport would accept
it when extensions are on. This caused strangeness and failures
when hexfloats were properly treated as an extension (not error)
in LiteralSupport.
llvm-svn: 59865
one for building up the diagnostic that is in flight (DiagnosticBuilder)
and one for pulling structured information out of the diagnostic when
formatting and presenting it.
There is no functionality change with this patch.
llvm-svn: 59849
- Add variants of IsNonPragmaNonMacroLexer to accept an IncludeMacroStack entry
(simplifies some uses).
- Use IsNonPragmaNonMacroLexer in Preprocessor::LookupFile.
- Add 'FileID' to PreprocessorLexer, and have Preprocessor query this fileid
when looking up the FileEntry for a file
Performance testing of -Eonly on Cocoa.h shows no performance regression because
of this patch.
llvm-svn: 59666
even whitespace, as tokens from the file. This is enabled with
L->SetKeepWhitespaceMode(true) on a raw lexer. In this mode, you too
can use clang as a really complex version of 'cat' with code like this:
Lexer RawLex(SourceLocation::getFileLoc(SM.getMainFileID(), 0),
PP.getLangOptions(), File.first, File.second);
RawLex.SetKeepWhitespaceMode(true);
Token RawTok;
RawLex.LexFromRawLexer(RawTok);
while (RawTok.isNot(tok::eof)) {
std::cout << PP.getSpelling(RawTok);
RawLex.LexFromRawLexer(RawTok);
}
This will emit exactly the input file, with no canonicalization or other
translation. Realistic clients actually do something with the tokens of
course :)
llvm-svn: 57401
same we we do an unterminated string or character literal. This makes
it so we can guarantee that the lexer never calls into the
preprocessor (which would be suicide for a raw lexer).
llvm-svn: 57395
using LexRawToken, create one and use LexFromRawLexer. This avoids
twiddling the RawLexer flag around and simplifies some code (even
speeding raw lexing up a tiny bit).
This change also improves the token paster to use a Lexer on the stack
instead of new/deleting it.
llvm-svn: 57393
- Added as private members for each because it is not clear where to
put the common definition. Perhaps the IdentifierInfos all of these
"pseudo-keywords" should be collected into one place (this would
KnownFunctionIDs and Objective-C property IDs, for example).
Remove Token::isNamedIdentifier.
- There isn't a good reason to use strcmp when we have interned
strings, and there isn't a good reason to encourage clients to do
so.
llvm-svn: 54794
Chris Lattner