Summary:
The cache recorded the wrong expansion location for all but the first
stringization. It seems uncommon to stringize the same macro argument
multiple times, so this cache doesn't seem that important.
Fixes PR39942
Reviewers: vsk, rsmith
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D65428
llvm-svn: 367337
Summary:
By adding a hook to consume all tokens produced by the preprocessor.
The intention of this change is to make it possible to consume the
expanded tokens without re-runnig the preprocessor with minimal changes
to the preprocessor and minimal performance penalty when preprocessing
without recording the tokens.
The added hook is very low-level and reconstructing the expanded token
stream requires more work in the client code, the actual algorithm to
collect the tokens using this hook can be found in the follow-up change.
Reviewers: rsmith
Reviewed By: rsmith
Subscribers: eraman, nemanjai, kbarton, jsji, riccibruno, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D59885
llvm-svn: 361007
* __VA_OPT__ is expanded if the *expanded* __VA_ARGS__ is non-empty,
not if the original argument contained no tokens.
* Placemarkers at the start and end of __VA_OPT__ are retained just
long enough to paste them with adjacent ## operators. We never paste
"across" a discarded placemarker.
llvm-svn: 359964
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
This is similar to the LLVM change https://reviews.llvm.org/D46290.
We've been running doxygen with the autobrief option for a couple of
years now. This makes the \brief markers into our comments
redundant. Since they are a visual distraction and we don't want to
encourage more \brief markers in new code either, this patch removes
them all.
Patch produced by
for i in $(git grep -l '\@brief'); do perl -pi -e 's/\@brief //g' $i & done
for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done
Differential Revision: https://reviews.llvm.org/D46320
llvm-svn: 331834
When a '>>' token is split into two '>' tokens (in C++11 onwards), or (as an
extension) when we do the same for other tokens starting with a '>', we can't
just use a location pointing to the first '>' as the location of the split
token, because that would result in our miscomputing the length and spelling
for the token. As a consequence, for example, a refactoring replacing 'A<X>'
with something else would sometimes replace one character too many, and
similarly diagnostics highlighting a template-id source range would highlight
one character too many.
Fix this by creating an expansion range covering the first character of the
'>>' token, whose spelling is '>'. For this to work, we generalize the
expansion range of a macro FileID to be either a token range (the common case)
or a character range (used in this new case).
llvm-svn: 331155
Found via codespell -q 3 -I ../clang-whitelist.txt
Where whitelist consists of:
archtype
cas
classs
checkk
compres
definit
frome
iff
inteval
ith
lod
methode
nd
optin
ot
pres
statics
te
thru
Patch by luzpaz! (This is a subset of D44188 that applies cleanly with a few
files that have dubious fixes reverted.)
Differential revision: https://reviews.llvm.org/D44188
llvm-svn: 329399
This patch implements an extension to the preprocessor:
__VA_OPT__(contents) --> which expands into its contents if variadic arguments are supplied to the parent macro, or behaves as an empty token if none.
- Currently this feature is only enabled for C++2a (this could be enabled, with some careful tweaks, for other dialects with the appropriate extension or compatibility warnings)
- The patch was reviewed here: https://reviews.llvm.org/D35782 and asides from the above (and moving some of the definition and expansion recognition logic into the corresponding state machines), I believe I incorporated all of Richard's suggestions.
A few technicalities (most of which were clarified through private correspondence between rsmith, hubert and thomas) are worth mentioning. Given:
#define F(a,...) a #__VA_OPT__(a ## a) a ## __VA_OPT__(__VA_ARGS__)
- The call F(,) Does not supply any tokens for the variadic arguments and hence VA_OPT behaves as a placeholder.
- When expanding VA_OPT (for e.g. F(,1) token pasting occurs eagerly within its contents if the contents need to be stringified.
- A hash or a hashhash prior to VA_OPT does not inhibit expansion of arguments if they are the first token within VA_OPT.
- When a variadic argument is supplied, argument substitution occurs within the contents as does stringification - and these resulting tokens are inserted back into the macro expansions token stream just prior to the entire stream being rescanned and concatenated.
See wg21.link/P0306 for further details on the feature.
Acknowledgment: This patch would have been poorer if not for Richard Smith's usual thoughtful analysis and feedback.
llvm-svn: 315840
- it made the bots v angry!
I'm not exactly sure why the assertion doesn't hold - if anyone has any insight - would appreciate it.
Thanks!
llvm-svn: 314748
In passing:
- change the name of the function to pasteTokens c/w coding standards
- rename CurToken to CurTokenIdx (since it is not the token, but the index)
- add doxygen comments to document some of pasteTokens' functionality
- use parameter names different from the data member names.
This will be useful for implementing __VA_OPT__ (https://reviews.llvm.org/D35782#inline-322587)
llvm-svn: 314747
- MacroArgs already knows the maximum number of arguments that can be supplied to the macro. No need to pass MacroInfo (information about the macro definition) to the call to getPreExpArgument (which by the way might benefit from being called getExpandedArgument() ?) for it to compute the number of arguments.
llvm-svn: 314593
- Extracted the reading of the tokens out into a separate function.
- Replace 'Argument' with 'Parameter' when referring to the identifiers of the macro definition (as opposed to the supplied arguments - MacroArgs - during the macro invocation).
This is in preparation for submitting patches for review to implement __VA_OPT__ which will otherwise just keep lengthening the HandleDefineDirective function and making it less comprehensible.
I will also directly update some extra clang tooling that is broken by the change from Argument to Parameter.
Hopefully the bots will stay appeased.
Thanks!
llvm-svn: 308190
- Extracted the reading of the tokens out into a separate function.
- Replace 'Argument' with 'Parameter' when referring to the identifiers of the macro definition (as opposed to the supplied arguments - MacroArgs - during the macro invocation).
This is in preparation for submitting patches for review to implement __VA_OPT__ which will otherwise just keep lengthening the HandleDefineDirective function and making it less comprehensible.
Thanks!
llvm-svn: 308157
The whitespace should come from the argument name in the macro
expansion, rather than from the token passed to the macro (same as it
does when not pasting).
Added a new test case for the change in behavior to stringize_space.c.
FileCheck'ized macro_paste_commaext.c, tweaked the test case, and
added a comment; no behavioral change to this test.
Differential Revision: https://reviews.llvm.org/D30427
llvm-svn: 302195
While in the area, also change some unsigned variables to size_t, and
introduce an LLVM_FALLTHROUGH instead of a comment stating that.
Differential Revision: http://reviews.llvm.org/D25982
llvm-svn: 285193
The lexer sets the end location of macro arguments incorrectly *if*,
while merging consecutive args to fit into a single SLocEntry, it finds
args which come from different macro files.
Fix the issue by using separate SLocEntries in this situation.
This fixes a code coverage crasher (rdar://problem/26181005). Because
the lexer reported end locations for certain macro args incorrectly, we
would generate bogus coverage mappings with negative line offsets.
Reviewed-by: akyrtzi
Differential Revision: http://reviews.llvm.org/D20401
llvm-svn: 270160
Summary:
This fixes PR25875. When the trailing comma in a macro argument list is
elided, we need to treat it similarly to the case where a variadic macro
misses one actual argument.
Reviewers: rnk, rsmith
Subscribers: cfe-commits
Differential Revision: http://reviews.llvm.org/D15670
llvm-svn: 258530
first token of the expansion, don't forget to copy the "is at the start of a
line" token (which is always false, as newlines cannot appear within a macro
body); otherwise, stringizing the result can insert spurious whitespace.
llvm-svn: 257863
In public MS headers for XAudio, clang would fail to generate a valid UUID due to the UUID components being combined with the '-' UUID separators. Clang would attempting to recover but would preserve the leading whitespace from the tokens after each failed paste leading to spaces creeping into the UUID and causing an error in the __declspace(uuid()) parsing.
Reference: Microsoft DirectX SDK (June 2010)\Include\XAudio2.h(51)
Resolves http://llvm.org/pr23071
llvm-svn: 235186
Now that SmallString is a first-class citizen, most SmallString::str()
calls are not required. This patch removes a whole bunch of them, yet
there are lots more.
There are two use cases where str() is really needed:
1) To use one of StringRef member functions which is not available in
SmallString.
2) To convert to std::string, as StringRef implicitly converts while
SmallString do not. We may wish to change this, but it may introduce
ambiguity.
llvm-svn: 232622
Clang should form a wide string literal from L#macro_arg in a function-like macro in -fms-compatibility mode.
Fix for http://llvm.org/PR9984.
Differential Revision: http://reviews.llvm.org/D6604
llvm-svn: 224228
We would crash because we used operator[] to access past the end of a
SmallString. This occured because our token had length zero.
Instead, form the pointer using .data() and arithmetic. This is safe
because this forms a one-past-the-end pointer and it is only used to
compare with another one-past-the-end pointer.
This fixes PR21379.
llvm-svn: 220614
In TokenLexer::ExpandFunctionArguments(), CurTok.hasLeadingSpace() is
checked in multiple locations, each time subtly differently. Checking it
early, when the token is seen, and using NextTokGetsSpace exclusively
after that makes the code simpler.
No change in behaviour is intended.
Patch by Harald van Dijk!
llvm-svn: 200788
When a macro expansion does not result in any tokens, and the macro name
is preceded by whitespace, the whitespace should be passed to the first
token that follows the macro expansion. Similarly when a macro expansion
ends with a placemarker token, and that placemarker token is preceded by
whitespace. This worked already for top-level macro expansions, but is
now extended to also work for nested macro expansions.
Patch by Harald van Dijk!
llvm-svn: 200787
When a function-like macro definition ends with one of the macro's
parameters, and the argument is empty, any whitespace before the
parameter name in the macro definition needs to be preserved. Promoting
the existing NextTokGetsSpace to a preserved bit-field and checking it
at the end of the macro expansion allows it to be moved to the first
token following the macro expansion result.
Patch by Harald van Dijk!
llvm-svn: 200786
In x ## y, where x and y are regular tokens, whitespace between x and ##
is ignored, and whitespace between ## and y is also ignored. When either
x or y is a function argument, whitespace was preserved, but it should
not be. This patch removes the checks for whitespace before ## and
before y, and in the special case where x is an empty macro argument and
y is a regular token, actively removes whitespace before y.
One existing test is affected by that change, but as clang's output now
matches the standard's requirements and that of GCC, I've tweaked the
testcase.
Patch by Harald van Dijk!
llvm-svn: 200785
There's been long-standing confusion over the role of these two options. This
commit makes the necessary changes to differentiate them clearly, following up
from r198936.
MicrosoftExt (aka. fms-extensions):
Enable largely unobjectionable Microsoft language extensions to ease
portability. This mode, also supported by gcc, is used for building software
like FreeBSD and Linux kernel extensions that share code with Windows drivers.
MSVCCompat (aka. -fms-compatibility, formerly MicrosoftMode):
Turn on a special mode supporting 'heinous' extensions for drop-in
compatibility with the Microsoft Visual C++ product. Standards-compilant C and
C++ code isn't guaranteed to work in this mode. Implies MicrosoftExt.
Note that full -fms-compatibility mode is currently enabled by default on the
Windows target, which may need tuning to serve as a reasonable default.
See cfe-commits for the full discourse, thread 'r198497 - Move MS predefined
type_info out of InitializePredefinedMacros'
No change in behaviour.
llvm-svn: 199209
encodes the canonical rules for LLVM's style. I noticed this had drifted
quite a bit when cleaning up LLVM, so wanted to clean up Clang as well.
llvm-svn: 198686
Before this patch, Lex() would recurse whenever the current lexer changed (e.g.
upon entry into a macro). This patch turns the recursion into a loop: the
various lex routines now don't return a token when the current lexer changes,
and at the top level Preprocessor::Lex() now loops until it finds a token.
Normally, the recursion wouldn't end up being very deep, but the recursion depth
can explode in edge cases like a bunch of consecutive macros which expand to
nothing (like in the testcase test/Preprocessor/macro_expand_empty.c in this
patch).
<rdar://problem/14569770>
llvm-svn: 190980
This allows clang to parse the type_traits header in Visual Studio 2012,
which is included widely in practice.
This is a rework of r163022 by João Matos. The original patch broke
preprocessing of gtest headers, which this patch addresses.
Patch by Will Wilson!
llvm-svn: 184968
When x is empty, x ## is suppressed, and when y gets expanded, the fact that it follows ## is not
available in the macro expansion result. The macro definition can be checked instead, the ## will
be available there regardless of what x expands to.
Fixes http://llvm.org/PR12767
Patch by Harald van Dijk!
llvm-svn: 182699
Reid Kleckner