Commit Graph

255 Commits

Author SHA1 Message Date
Richard Smith 8b7258bdb3 PR18855: Add support for UCNs and UTF-8 encoding within ud-suffixes.
llvm-svn: 201532
2014-02-17 21:52:30 +00:00
Alp Toker bfa3934f27 Rename language option MicrosoftMode to MSVCCompat
There's been long-standing confusion over the role of these two options. This
commit makes the necessary changes to differentiate them clearly, following up
from r198936.

MicrosoftExt (aka. fms-extensions):
 Enable largely unobjectionable Microsoft language extensions to ease
 portability. This mode, also supported by gcc, is used for building software
 like FreeBSD and Linux kernel extensions that share code with Windows drivers.

MSVCCompat (aka. -fms-compatibility, formerly MicrosoftMode):
 Turn on a special mode supporting 'heinous' extensions for drop-in
 compatibility with the Microsoft Visual C++ product. Standards-compilant C and
 C++ code isn't guaranteed to work in this mode. Implies MicrosoftExt.

Note that full -fms-compatibility mode is currently enabled by default on the
Windows target, which may need tuning to serve as a reasonable default.

See cfe-commits for the full discourse, thread 'r198497 - Move MS predefined
type_info out of InitializePredefinedMacros'

No change in behaviour.

llvm-svn: 199209
2014-01-14 12:51:41 +00:00
Chandler Carruth 5553d0d4ca Sort all the #include lines with LLVM's utils/sort_includes.py which
encodes the canonical rules for LLVM's style. I noticed this had drifted
quite a bit when cleaning up LLVM, so wanted to clean up Clang as well.

llvm-svn: 198686
2014-01-07 11:51:46 +00:00
Alp Toker 6de6da603e Lexer: Issue -Wbackslash-newline-escape for line comments
The warning for backslash and newline separated by whitespace was missed in
this code path.

backslash<whitespace><newline> is handled differently from compiler to compiler
so it's important to warn consistently where there's ambiguity.

Matches similar handling of block comments and non-comment lines.

llvm-svn: 197331
2013-12-14 23:32:31 +00:00
Alp Toker 08c2500f9c Fix raw lex crash and -frewrite-includes noeol-at-eof failure
Raw lexers don't have a preprocessor so we need to null check.

llvm-svn: 197245
2013-12-13 17:04:55 +00:00
Justin Bogner 5353513058 Lex: Don't restrict legal UCNs when preprocessing assembly
The C and C++ standards disallow using universal character names to
refer to some characters, such as basic ascii and control characters,
so we reject these sequences in the lexer. However, when the
preprocessor isn't being used on C or C++, it doesn't make sense to
apply these restrictions.

Notably, accepting these characters avoids issues with unicode escapes
when GHC uses the compiler as a preprocessor on haskell sources.

Fixes rdar://problem/14742289

llvm-svn: 193067
2013-10-21 05:02:28 +00:00
Richard Smith 7f2707a7f4 Per updates to D3781, allow underscore under ' in a pp-number, and allow ' in a #line directive.
llvm-svn: 191443
2013-09-26 18:13:20 +00:00
Richard Smith fde9485297 Implement C++1y digit separator proposal (' as a digit separator). This is not
yet approved by full committee, but was unanimously supported by EWG.

llvm-svn: 191417
2013-09-26 03:33:06 +00:00
Richard Smith 5acb759f39 Avoid a signed/unsigned comparison warning with compilers that don't know how
to handle constant expressions.

llvm-svn: 191336
2013-09-24 22:13:21 +00:00
Richard Smith 2a98862be2 Handle standard libraries that miss out the space when defining the standard
literal operators. Also, for now, allow the proposed C++1y "il", "i", and "if"
suffixes too. (Will revert the latter if LWG decides not to go ahead with that
change after all.)

llvm-svn: 191274
2013-09-24 04:06:10 +00:00
Eli Friedman 29749d2e3b Fix use-after-free in r190980.
llvm-svn: 190984
2013-09-19 01:51:23 +00:00
Eli Friedman 0834a4b901 Make Preprocessor::Lex non-recursive.
Before this patch, Lex() would recurse whenever the current lexer changed (e.g.
upon entry into a macro). This patch turns the recursion into a loop: the
various lex routines now don't return a token when the current lexer changes,
and at the top level Preprocessor::Lex() now loops until it finds a token.
Normally, the recursion wouldn't end up being very deep, but the recursion depth
can explode in edge cases like a bunch of consecutive macros which expand to
nothing (like in the testcase test/Preprocessor/macro_expand_empty.c in this
patch).

<rdar://problem/14569770>

llvm-svn: 190980
2013-09-19 00:41:32 +00:00
Alexander Kornienko 37d6b18633 Use new UnicodeCharSet interface.
Summary: This is a Clang part of http://llvm-reviews.chandlerc.com/D1534

Reviewers: jordan_rose, klimek, rsmith

Reviewed By: rsmith

CC: cfe-commits

Differential Revision: http://llvm-reviews.chandlerc.com/D1535

llvm-svn: 189583
2013-08-29 12:12:31 +00:00
Eli Friedman cefc7eafc6 Fix "//" comments with -traditional-cpp in C++.
Apparently, gcc's -traditional-cpp behaves slightly differently in C++ mode;
specifically, it discards "//" comments.  Match gcc's behavior.

<rdar://problem/14808126>

llvm-svn: 189515
2013-08-28 20:53:32 +00:00
Jordan Rose 4c55d45b13 Respect -Wnewline-eof even in C++11 mode.
If the user has requested this warning, we should emit it, even if it's not
an extension in the current language mode. However, being an extension is
more important, so prefer the pedantic warning or the pedantic-compatibility
warning if those are enabled.

<rdar://problem/12922063>

llvm-svn: 189110
2013-08-23 15:42:01 +00:00
Fariborz Jahanian d38ad47cfa ObjectiveC migrator: More work towards
insertion of ObjC audit pragmas.

llvm-svn: 188733
2013-08-20 00:07:23 +00:00
Richard Smith f4198b7598 C++1y literal suffix support:
* Allow ns, us, ms, s, min, h as numeric ud-suffixes
 * Allow s as string ud-suffix

llvm-svn: 186933
2013-07-23 08:14:48 +00:00
Michael J. Spencer 8c39840087 Replace Count{Leading,Trailing}Zeros_{32,64} with count{Leading,Trailing}Zeros.
llvm-svn: 182675
2013-05-24 21:42:04 +00:00
Argyrios Kyrtzidis dc9fdaf217 [modules] If we hit a failure while loading a PCH/module, abort parsing instead of trying to continue in an invalid state.
Also don't let libclang create a PCH with such an error.

Fixes rdar://13953768

llvm-svn: 182629
2013-05-24 05:44:08 +00:00
Argyrios Kyrtzidis 065d720c31 [Lexer] Improve Lexer::getSourceText() when the given range deals with function macro arguments.
This is a modified version of a patch by Manuel Klimek.

llvm-svn: 182055
2013-05-16 21:37:39 +00:00
Richard Smith 0f7f6f1abc Typo and misc comment fix.
llvm-svn: 181583
2013-05-10 02:36:35 +00:00
Argyrios Kyrtzidis 0903f8dac5 [libclang] Make sure the preable does not truncate comments.
rdar://13647445

llvm-svn: 179907
2013-04-19 23:24:25 +00:00
Richard Smith 06d274fdb7 Add -Wc99-compat warning for C11 unicode string and character literals.
llvm-svn: 176817
2013-03-11 18:01:42 +00:00
Richard Smith 9b36209e31 When lexing in C11 mode, accept unicode character and string literals, per C11
6.4.4.4/1 and 6.4.5/1.

llvm-svn: 176780
2013-03-09 23:56:02 +00:00
Jordan Rose 864b810739 Preprocessor: don't consider // to be a line comment in -E -std=c89 mode.
It's beneficial when compiling to treat // as the start of a line
comment even in -std=c89 mode, since it's not valid C code (with a few
rare exceptions) and is usually intended as such. We emit a pedantic
warning and then continue on as if line comments were enabled.
This has been our behavior for quite some time.

However, people use the preprocessor for things besides C source files.
In today's prompting example, the input contains (unquoted) URLs, which
contain // but should still be preserved.

This change instructs the lexer to treat // as a plain token if Clang is
in C90 mode and generating preprocessed output rather than actually compiling.

<rdar://problem/13338743>

llvm-svn: 176526
2013-03-05 22:51:04 +00:00
Jordan Rose cb8a1aca35 Preprocessor: preserve whitespace in -traditional-cpp mode.
Note that unlike GNU cpp we currently do not preserve whitespace in macros
(even in -traditional-cpp mode).

<rdar://problem/12897179>

llvm-svn: 175778
2013-02-21 18:53:19 +00:00
Jordan Rose 58c61e006f Properly validate UCNs for C99 and C++03 (both more restrictive than C(++)11).
Add warnings under -Wc++11-compat, -Wc++98-compat, and -Wc99-compat when a
particular UCN is incompatible with a different standard, and -Wunicode when
a UCN refers to a surrogate character in C++03.

llvm-svn: 174788
2013-02-09 01:10:25 +00:00
Jordan Rose a2100d755a Pull Lexer's CharInfo table out for general use throughout Clang.
Rewriting the same predicates over and over again is bad for code size and
code maintainence. Using the functions in <ctype.h> is generally unsafe
unless they are specified to be locale-independent (i.e. only isdigit and
isxdigit).

The next commit will try to clean up uses of <ctype.h> functions within Clang.

llvm-svn: 174765
2013-02-08 22:30:22 +00:00
Jordan Rose cc538345be Lexer: Don't warn about Unicode in preprocessor directives.
This allows people to use Unicode in their #pragma mark and in macros
that exist only to be string-ized.

<rdar://problem/13107323&13121362>

llvm-svn: 174081
2013-01-31 19:48:48 +00:00
Jordan Rose f649795f84 Fix r173881 to properly skip invalid UTF-8 characters in raw lexing and -E.
This caused hangs as we processed the same invalid byte over and over.

<rdar://problem/13115651>

llvm-svn: 173959
2013-01-30 19:21:12 +00:00
Dmitri Gribenko 9feeef40f5 Move UTF conversion routines from clang/lib/Basic to llvm/lib/Support
This is required to use them in TableGen.

llvm-svn: 173924
2013-01-30 12:06:08 +00:00
Jordan Rose 17441589c3 Don't warn about Unicode characters in -E mode.
People use the C preprocessor for things other than C files. Some of them
have Unicode characters. We shouldn't warn about Unicode characters
appearing outside of identifiers in this case.

There's not currently a way for the preprocessor to tell if it's in -E mode,
so I added a new flag, derived from the PreprocessorOutputOptions. This is
only used by the Unicode warnings for now, but could conceivably be used by
other warnings or even behavioral differences later.

<rdar://problem/13107323>

llvm-svn: 173881
2013-01-30 01:52:57 +00:00
Jordan Rose cccbdbf0db PR15067 (again): Don't warn about UCNs in C90 if we're raw-lexing.
Fixes a crash. Thanks, Richard.

llvm-svn: 173701
2013-01-28 17:49:02 +00:00
Jordan Rose c0cba27230 PR15067: Don't assert when a UCN appears in a C90 file.
Unfortunately, we can't accept the UCN as an extension because we're
required to treat it as two tokens for preprocessing purposes.

llvm-svn: 173622
2013-01-27 20:12:04 +00:00
NAKAMURA Takumi e8f83dbbd8 Lexer.cpp: Fix a warning with ptrdiff_t on i686. [-Wsign-compare]
llvm-svn: 173447
2013-01-25 14:57:21 +00:00
Jordan Rose 8b4af2ae88 Clarify comment: "diagnose" is better than "warn" when emitting an error.
Thanks, Dmitri.

llvm-svn: 173400
2013-01-25 00:20:28 +00:00
Jordan Rose 62db5066e9 Add a fixit for \U1234 -> \u1234.
llvm-svn: 173371
2013-01-24 20:50:52 +00:00
Jordan Rose 4246ae0089 As an extension, treat Unicode whitespace characters as whitespace.
llvm-svn: 173370
2013-01-24 20:50:50 +00:00
Jordan Rose 7f43dddae0 Handle universal character names and Unicode characters outside of literals.
This is a missing piece for C99 conformance.

This patch handles UCNs by adding a '\\' case to LexTokenInternal and
LexIdentifier -- if we see a backslash, we tentatively try to read in a UCN.
If the UCN is not syntactically well-formed, we fall back to the old
treatment: a backslash followed by an identifier beginning with 'u' (or 'U').

Because the spelling of an identifier with UCNs still has the UCN in it, we
need to convert that to UTF-8 in Preprocessor::LookUpIdentifierInfo.

Of course, valid code that does *not* use UCNs will see only a very minimal
performance hit (checks after each identifier for non-ASCII characters,
checks when converting raw_identifiers to identifiers that they do not
contain UCNs, and checks when getting the spelling of an identifier that it
does not contain a UCN).

This patch also adds basic support for actual UTF-8 in the source. This is
treated almost exactly the same as UCNs except that we consider stray
Unicode characters to be mistakes and offer a fixit to remove them.

llvm-svn: 173369
2013-01-24 20:50:46 +00:00
Dmitri Gribenko f857950d39 Remove useless 'llvm::' qualifier from names like StringRef and others that are
brought into 'clang' namespace by clang/Basic/LLVM.h

llvm-svn: 172323
2013-01-12 19:30:44 +00:00
Argyrios Kyrtzidis 86f1a935dc Pull the bulk of Lexer::MeasureTokenLength() out into a new function,
Lexer::getRawToken().

No functionality change.

llvm-svn: 171771
2013-01-07 19:16:18 +00:00
Richard Smith 2bf7fdb723 s/CPlusPlus0x/CPlusPlus11/g
llvm-svn: 171367
2013-01-02 11:42:31 +00:00
Chandler Carruth 3a02247dc9 Sort all of Clang's files under 'lib', and fix up the broken headers
uncovered.

This required manually correcting all of the incorrect main-module
headers I could find, and running the new llvm/utils/sort_includes.py
script over the files.

I also manually added quite a few missing headers that were uncovered by
shuffling the order or moving headers up to be main-module-headers.

llvm-svn: 169237
2012-12-04 09:13:33 +00:00
Richard Smith 9a67f47882 Teach Lexer::getSpelling about raw string literals. Specifically, if a raw
string literal needs cleaning (because it contains line-splicing in the
encoding prefix or in the ud-suffix), do not clean the section between the
double-quotes -- that's the "raw" bit!

llvm-svn: 168776
2012-11-28 07:29:00 +00:00
Nico Weber 4e270380c1 Fix crash on end-of-file after \ in a char literal, fixes PR14369.
This makes LexCharConstant() look more like LexStringLiteral(), which doesn't
have this bug. Add tests for eof after \ for several other cases.

llvm-svn: 168269
2012-11-17 20:25:54 +00:00
Eli Friedman b699e619fe Fix an assertion failure printing the unused-label fixit in files using CRLF line endings. <rdar://problem/12639047>.
llvm-svn: 167900
2012-11-14 01:28:38 +00:00
Daniel Dunbar cf3f2c49ea Revert r167801, "[preprocessor] When #including something that contributes no
tokens at all,". This change broke External/Nurbs in LLVM test-suite.

llvm-svn: 167858
2012-11-13 19:12:37 +00:00
Nico Weber 7cc28804e2 UCNs in char literals are done (in LiteralSupport), remove FIXME. Expand UCN FIXME in LexNumericConstant.
llvm-svn: 167818
2012-11-13 06:25:15 +00:00
Argyrios Kyrtzidis 4f10a3e9f0 [preprocessor] When #including something that contributes no tokens at all,
don't recursively continue lexing.

This avoids a stack overflow with a sequence of many empty #includes.
rdar://11988695

llvm-svn: 167801
2012-11-13 01:03:15 +00:00
Argyrios Kyrtzidis 36675b75fb In Lexer::LexTokenInternal, avoid code duplication; no functionality change.
llvm-svn: 167800
2012-11-13 01:02:40 +00:00