Commit Graph

160 Commits

Author SHA1 Message Date
Benjamin Kramer 17ff23c708 Speed up BCPL comment lexing by looking aggressively for newlines and then scannig backwards to see if the newline is escaped.
3% speedup in preprocessing all of clang with -Eonly. Also includes a small testcase for coverage.

llvm-svn: 139116
2011-09-05 07:19:39 +00:00
Benjamin Kramer dbfb18a0a9 Use the Lexer's definition of whitespace here.
llvm-svn: 139115
2011-09-05 07:19:35 +00:00
Argyrios Kyrtzidis 5cec2aea3f Support code-completion for C++ inline methods and ObjC buffering methods.
Previously we would cut off the source file buffer at the code-completion
point; this impeded code-completion inside C++ inline methods and,
recently, with buffering ObjC methods.

Have the code-completion inserted into the source buffer so that it can
be buffered along with a method body. When we actually hit the code-completion
point the cut-off lexing or parsing.

Fixes rdar://10056932&8319466

llvm-svn: 139086
2011-09-04 03:32:15 +00:00
Argyrios Kyrtzidis a3deaeeb52 Fix Lexer::ComputePreamble when MaxLines parameter is non-zero.
The function was only counting lines that included tokens and not empty lines,
but MaxLines (mainly initiated to the line where the code-completion point resides)
is a count of overall lines (even empty ones).

llvm-svn: 139085
2011-09-04 03:32:04 +00:00
Douglas Gregor 081425343b Introduce support for a simple module import declaration, which
loads the named module. The syntax itself is intentionally hideous and
will be replaced at some later point with something more
palatable. For now, we're focusing on the semantics:
  - Module imports are handled first by the preprocessor (to get macro
  definitions) and then the same tokens are also handled by the parser
  (to get declarations). If both happen (as in normal compilation),
  the second one is redundant, because we currently have no way to
  hide macros or declarations when loading a module. Chris gets credit
  for this mad-but-workable scheme.
  - The Preprocessor now holds on to a reference to a module loader,
  which is responsible for loading named modules. CompilerInstance is
  the only important module loader: it now knows how to create and
  wire up an AST reader on demand to actually perform the module load.
  - We search for modules in the include path, using the module name
  with the suffix ".pcm" (precompiled module) for the file name. This
  is a temporary hack; we hope to improve the situation in the
  future.

llvm-svn: 138679
2011-08-26 23:56:07 +00:00
Argyrios Kyrtzidis 7aecbc7661 Make Lexer::ComputePreamble accept a LangOptions parameter, otherwise it may be
out-of-sync how a file is compiled. Patch by Matthias Kleine!

llvm-svn: 138580
2011-08-25 20:39:19 +00:00
Argyrios Kyrtzidis f6a3b0ca4b In Lexer::isAtEndOfMacroExpansion use SourceManager::isInFileID and avoid
the extra SourceManager::getFileID call.

llvm-svn: 138376
2011-08-23 21:02:30 +00:00
Argyrios Kyrtzidis 161868db4c Make Lexer::GetBeginningOfToken able to handle macro arg expansion locations.
llvm-svn: 137795
2011-08-17 00:31:23 +00:00
Craig Topper 54edccafc5 Add support for C++0x raw string literals.
llvm-svn: 137298
2011-08-11 04:06:15 +00:00
Anna Zaks 59a3c80717 Add a utility function to the Lexer, which makes it easier to find a token after the given location. (It is a generalized version of trans::findLocationAfterSemi from ArcMigrate, which will be changed to use the Lexer utility).
llvm-svn: 136268
2011-07-27 21:43:43 +00:00
Douglas Gregor fb65e592e0 Add support for C++0x unicode string and character literals, from Craig Topper!
llvm-svn: 136210
2011-07-27 05:40:30 +00:00
Chandler Carruth ee4c1d1298 Migrate 'Instantiation' data and API bits of SLocEntry to 'Expansion'
etc. With this I think essentially all of the SourceManager APIs are
converted. Comments and random other bits of cleanup should be all thats
left.

llvm-svn: 136057
2011-07-26 04:56:51 +00:00
Chandler Carruth 73ee5d7fae Convert InstantiationInfo and much of the related code to ExpansionInfo
and various other 'expansion' based terms. I've tried to reformat where
appropriate and catch as many references in comments but I'm going to do
several more passes. Also I've tried to expand parameter names to be
more clear where appropriate.

llvm-svn: 136056
2011-07-26 04:41:47 +00:00
Chandler Carruth 115b077f30 Rename create(MacroArg)InstantiationLoc to create(MacroArg)ExpansionLoc.
llvm-svn: 136054
2011-07-26 03:03:05 +00:00
Chandler Carruth ca757587a3 Rename SourceManager::getImmediateInstantiationRange to
getImmediateExpansionRange.

llvm-svn: 135960
2011-07-25 20:52:21 +00:00
Chandler Carruth 6d28d7f2a3 Rename SourceManager::getInstantiationRange to getExpansionRange.
llvm-svn: 135915
2011-07-25 16:56:02 +00:00
Chandler Carruth 35f5320d8e Mechanically rename SourceManager::getInstantiationLoc and
FullSourceLoc::getInstantiationLoc to ...::getExpansionLoc. This is part
of the API and documentation update from 'instantiation' as the term for
macros to 'expansion'.

llvm-svn: 135914
2011-07-25 16:49:02 +00:00
Chris Lattner 0e62c1cc0b remove unneeded llvm:: namespace qualifiers on some core types now that LLVM.h imports
them into the clang namespace.

llvm-svn: 135852
2011-07-23 10:55:15 +00:00
Joerg Sonnenberger da5d2b761a Spelling
llvm-svn: 135545
2011-07-20 00:14:37 +00:00
Douglas Gregor 925296b4c2 Revamp the SourceManager to separate the representation of parsed
source locations from source locations loaded from an AST/PCH file.

Previously, loading an AST/PCH file involved carefully pre-allocating
space at the beginning of the source manager for the source locations
and FileIDs that correspond to the prefix, and then appending the
source locations/FileIDs used for parsing the remaining translation
unit. This design forced us into loading PCH files early, as a prefix,
whic has become a rather significant limitation.

This patch splits the SourceManager space into two parts: for source
location "addresses", the lower values (growing upward) are used to
describe parsed code, while upper values (growing downward) are used
for source locations loaded from AST/PCH files. Similarly, positive
FileIDs are used to describe parsed code while negative FileIDs are
used to file/macro locations loaded from AST/PCH files. As a result,
we can load PCH/AST files even during parsing, making various
improvemnts in the future possible, e.g., teaching #include <foo.h> to
look for and load <foo.h.gch> if it happens to be already available.

This patch was originally written by Sebastian Redl, then brought
forward to the modern age by Jonathan Turner, and finally
polished/finished by me to be committed.

llvm-svn: 135484
2011-07-19 16:10:42 +00:00
Chandler Carruth e2c09ebcaa Convert terminology in the Lexer from 'instantiate' and variants to
'expand'. Also update the public API it provides to the new term, and
propagate that update to the various clients.

No functionality changed.

llvm-svn: 135138
2011-07-14 08:20:40 +00:00
Argyrios Kyrtzidis 61c58f7f43 Move SourceManager::isAt[Start/End]OfMacroInstantiation functions to the Lexer, since they depend on it now.
llvm-svn: 134644
2011-07-07 21:54:45 +00:00
Argyrios Kyrtzidis 41fb2d95a3 Make the Preprocessor more memory efficient and improve macro instantiation diagnostics.
When a macro instantiation occurs, reserve a SLocEntry chunk with length the
full length of the macro definition source. Set the spelling location of this chunk
to point to the start of the macro definition and any tokens that are lexed directly
from the macro definition will get a location from this chunk with the appropriate offset.

For any tokens that come from argument expansion, '##' paste operator, etc. have their
instantiation location point at the appropriate place in the instantiated macro definition
(the argument identifier and the '##' token respectively).
This improves macro instantiation diagnostics:

Before:

t.c:5:9: error: invalid operands to binary expression ('struct S' and 'int')
int y = M(/);
        ^~~~
t.c:5:11: note: instantiated from:
int y = M(/);
          ^

After:

t.c:5:9: error: invalid operands to binary expression ('struct S' and 'int')
int y = M(/);
        ^~~~
t.c:3:20: note: instantiated from:
\#define M(op) (foo op 3);
                ~~~ ^  ~
t.c:5:11: note: instantiated from:
int y = M(/);
          ^

The memory savings for a candidate boost library that abuses the preprocessor are:

- 32% less SLocEntries (37M -> 25M)
- 30% reduction in PCH file size (900M -> 635M)
- 50% reduction in memory usage for the SLocEntry table (1.6G -> 800M)

llvm-svn: 134587
2011-07-07 03:40:34 +00:00
Argyrios Kyrtzidis 2cfce18645 Allow Lexer::getLocForEndOfToken to return the location just passed the macro instantiation
if the location given points at the last token of the macro instantiation.

Fixes rdar://9045701.

llvm-svn: 133804
2011-06-24 17:58:59 +00:00
Eli Friedman 86a5101c27 Don't strlen() every file before parsing it.
llvm-svn: 131132
2011-05-10 17:11:21 +00:00
Chris Lattner 57540c5be0 fix a bunch of comment typos found by codespell. Patch by
Luis Felipe Strano Moraes!

llvm-svn: 129559
2011-04-15 05:22:18 +00:00
Richard Smith f7b6202e6c Implement C++0x [lex.pptoken]p3's handling of <::.
llvm-svn: 129525
2011-04-14 18:36:27 +00:00
Eric Christopher 7f36a79ee9 Eat the UTF-8 BOM at the beginning of a file since it's ignored anyhow.
Nom Nom Nom.

Patch by Anton Korobeynikov!

llvm-svn: 129174
2011-04-09 00:01:04 +00:00
John McCall 75ca6d72c2 Fix getLocForEndOfToken to not double-count spurious internal characters
within a token, like trigraphs and escaped newlines.               
Patch by Marcin Kowalczyk!

llvm-svn: 128978
2011-04-06 01:50:22 +00:00
Daniel Dunbar 1057f86d0e Lexer: Add extremely limited support for -traditional-cpp, ignoring BCPL
comments.

llvm-svn: 127910
2011-03-18 21:23:38 +00:00
John McCall 462c055d85 Fix my earlier commit to work with escaped newlines and leave breadcrumbs
in case we want to make a world where we can check intermediate instantiations
for this kind of breadcrumb.

llvm-svn: 127221
2011-03-08 07:59:04 +00:00
Peter Collingbourne 2f1e36bfd0 Rename tok::eom to tok::eod.
The previous name was inaccurate as this token in fact appears at
the end of every preprocessing directive, not just macro definitions.
No functionality change, except for a diagnostic tweak.

llvm-svn: 126631
2011-02-28 02:37:51 +00:00
Argyrios Kyrtzidis c541ade850 Warn for missing terminating " or ' instead of error for gcc compatibility. Fixed rdar://8914293.
llvm-svn: 125616
2011-02-15 23:45:31 +00:00
Peter Collingbourne c1270f51fa Lexer: add CUDA kernel call tokens
llvm-svn: 125218
2011-02-09 21:08:21 +00:00
Douglas Gregor 86af98444f Harden Lexer::GetBeginningOfToken() against bogus source locations and
the disappearance/alteration of files.

llvm-svn: 124616
2011-01-31 22:42:36 +00:00
Abramo Bagnara ea4f7c7761 Introduced raw_identifier token kind.
llvm-svn: 122394
2010-12-22 08:23:18 +00:00
Chris Lattner 39720111e0 move getSpelling from Preprocessor to Lexer, which it is more conceptually related to.
llvm-svn: 119479
2010-11-17 07:26:20 +00:00
Chris Lattner 2a6ee91619 move AdvanceToTokenCharacter and getLocForEndOfToken from
Preprocessor to Lexer where they make more sense.

llvm-svn: 119474
2010-11-17 07:05:50 +00:00
Chandler Carruth c3ce5840af Update remaining attribute macros to new style.
llvm-svn: 117204
2010-10-23 08:44:57 +00:00
Sebastian Redl 517523014d In MeasureTokenLength, the FileLoc supplied to the lexer must point to the start of the buffer, or we risk overflow.
llvm-svn: 115117
2010-09-30 01:03:03 +00:00
Chris Lattner 0f0492e69c improve isHexaLiteral to work with escaped newlines and trigraphs,
patch by Francois Pichet!

llvm-svn: 112602
2010-08-31 16:42:00 +00:00
Chris Lattner dec7334218 silence a warning
llvm-svn: 112549
2010-08-30 23:11:03 +00:00
Alexis Hunt 3b7918625c Revert my user-defined literal commits - r1124{58,60,67} pending
some issues being sorted out.

llvm-svn: 112493
2010-08-30 17:47:05 +00:00
Chris Lattner 5f183aa592 add a fixme.
llvm-svn: 112491
2010-08-30 17:11:14 +00:00
Chris Lattner 7a9e9e7d76 use 'features' instead of 'PP->getLangOptions'.
llvm-svn: 112490
2010-08-30 17:09:08 +00:00
Douglas Gregor 759ef23bb8 In Microsoft compatibility mode, don't parse the exponent as part of
the pp-number in a hexadecimal floating point literal, from Francois
Pichet! Fixes PR7968.

llvm-svn: 112481
2010-08-30 14:50:47 +00:00
Alexis Hunt 79eb5469e0 Implement C++0x user-defined string literals.
The extra data stored on user-defined literal Tokens is stored in extra
allocated memory, which is managed by the PreprocessorLexer because there isn't
a better place to put it that makes sure it gets deallocated, but only after
it's used up. My testing has shown no significant slowdown as a result, but
independent testing would be appreciated.

llvm-svn: 112458
2010-08-29 21:26:48 +00:00
Douglas Gregor 115837041e Introduce a preprocessor code-completion hook for contexts where we
expect "natural" language and should not provide any completions,
e.g., comments, string literals, #error.

llvm-svn: 112054
2010-08-25 17:04:25 +00:00
Douglas Gregor 3a7ad25eb6 Introduce basic code-completion support for preprocessor directives,
e.g., after a "#" we'll suggest #if, #ifdef, etc.

llvm-svn: 111943
2010-08-24 19:08:16 +00:00
Douglas Gregor 02690ba643 Don't emit end-of-file diagnostics like "unterminated conditional" or
"unterminated string" when we're performing code completion.

llvm-svn: 110933
2010-08-12 17:04:55 +00:00