maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity
llvm-project/clang/unittests/Lex/LexerTest.cpp

539 lines
20 KiB
C++
Raw Normal View History

Sort the #include lines for unittests/... I've tried to place sensible headers at the top as main-module headers. llvm-svn: 169243
2012-12-04 17:45:34 +08:00
//===- unittests/Lex/LexerTest.cpp ------ Lexer tests ---------------------===//
Add unit testing for Lexer. llvm-svn: 148478
2012-01-19 23:59:01 +08:00
//
Update the file headers across all of the LLVM projects in the monorepo to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
2019-01-19 16:50:56 +08:00
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
Add unit testing for Lexer. llvm-svn: 148478
2012-01-19 23:59:01 +08:00
//
//===----------------------------------------------------------------------===//
Sort the #include lines for unittests/... I've tried to place sensible headers at the top as main-module headers. llvm-svn: 169243
2012-12-04 17:45:34 +08:00
#include "clang/Lex/Lexer.h"
Add unit testing for Lexer. llvm-svn: 148478
2012-01-19 23:59:01 +08:00
#include "clang/Basic/Diagnostic.h"
More unit-test fixes llvm-svn: 166511
2012-10-24 06:38:58 +08:00
#include "clang/Basic/DiagnosticOptions.h"
Sort the #include lines for unittests/... I've tried to place sensible headers at the top as main-module headers. llvm-svn: 169243
2012-12-04 17:45:34 +08:00
#include "clang/Basic/FileManager.h"
Add unit testing for Lexer. llvm-svn: 148478
2012-01-19 23:59:01 +08:00
#include "clang/Basic/LangOptions.h"
Really sort the #include lines in unittests/... I forgot to re-sort after fixing main module headers. llvm-svn: 169244
2012-12-04 17:53:37 +08:00
#include "clang/Basic/SourceManager.h"
Add unit testing for Lexer. llvm-svn: 148478
2012-01-19 23:59:01 +08:00
#include "clang/Basic/TargetInfo.h"
Sort the #include lines for unittests/... I've tried to place sensible headers at the top as main-module headers. llvm-svn: 169243
2012-12-04 17:45:34 +08:00
#include "clang/Basic/TargetOptions.h"
Add unit testing for Lexer. llvm-svn: 148478
2012-01-19 23:59:01 +08:00
#include "clang/Lex/HeaderSearch.h"
Update unit tests for HeaderSearch change llvm-svn: 166584
2012-10-25 00:24:38 +08:00
#include "clang/Lex/HeaderSearchOptions.h"
[Preprocessor]Correct Macro-Arg allocation of StringifiedArguments, correct getNumArguments StringifiedArguments is allocated (resized) based on the size the getNumArguments function. However, this function ACTUALLY currently returns the amount of total UnexpArgTokens which is minimum the same as the new implementation of getNumMacroArguments, since empty/omitted arguments result in 1 UnexpArgToken, and included ones at minimum include 2 (1 for the arg itself, 1 for eof). This patch renames the otherwise unused getNumArguments to be more clear that it is the number of arguments that the Macro expects, and thus the maximum number that can be stringified. This patch also replaces the explicit memset (which results in value instantiation of the new tokens, PLUS clearing the memory) with brace initialization. Differential Revision: https://reviews.llvm.org/D32046 llvm-svn: 305425
2017-06-15 07:09:01 +08:00
#include "clang/Lex/MacroArgs.h"
#include "clang/Lex/MacroInfo.h"
Sort the #include lines for unittests/... I've tried to place sensible headers at the top as main-module headers. llvm-svn: 169243
2012-12-04 17:45:34 +08:00
#include "clang/Lex/ModuleLoader.h"
Add unit testing for Lexer. llvm-svn: 148478
2012-01-19 23:59:01 +08:00
#include "clang/Lex/Preprocessor.h"
Teach the preprocessor to hold onto the preprocessor options. llvm-svn: 166599
2012-10-25 01:46:57 +08:00
#include "clang/Lex/PreprocessorOptions.h"
Add unit testing for Lexer. llvm-svn: 148478
2012-01-19 23:59:01 +08:00
#include "gtest/gtest.h"
using namespace clang;
namespace {
// The test fixture.
class LexerTest : public ::testing::Test {
protected:
LexerTest()
: FileMgr(FileMgrOpts),
DiagID(new DiagnosticIDs()),
Fixup unit tests for DiagnosticOptions change llvm-svn: 166509
2012-10-24 06:31:51 +08:00
Diags(DiagID, new DiagnosticOptions, new IgnoringDiagConsumer()),
Fix the handling of target options in our unit tests. llvm-svn: 166079
2012-10-17 08:11:35 +08:00
SourceMgr(Diags, FileMgr),
Stringizing raw string literals containing newline Summary: This patch implements 4.3 of http://open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4220.pdf. If a raw string contains a newline character, replace each newline character with the \n escape code. Without this patch, included test case (macro_raw_string.cpp) results compilation failure. Reviewers: rsmith, doug.gregor, jkorous-apple Reviewed By: jkorous-apple Subscribers: jkorous-apple, vsapsai, cfe-commits Differential Revision: https://reviews.llvm.org/D39279 llvm-svn: 319904
2017-12-07 01:00:53 +08:00
TargetOpts(new TargetOptions)
Fix the handling of target options in our unit tests. llvm-svn: 166079
2012-10-17 08:11:35 +08:00
{
TargetOpts->Triple = "x86_64-apple-darwin11.1.0";
Use non-intrusive refcounting for TargetOptions llvm-svn: 212388
2014-07-06 13:26:44 +08:00
Target = TargetInfo::CreateTargetInfo(Diags, TargetOpts);
Add unit testing for Lexer. llvm-svn: 148478
2012-01-19 23:59:01 +08:00
}
[Preprocessor]Correct Macro-Arg allocation of StringifiedArguments, correct getNumArguments StringifiedArguments is allocated (resized) based on the size the getNumArguments function. However, this function ACTUALLY currently returns the amount of total UnexpArgTokens which is minimum the same as the new implementation of getNumMacroArguments, since empty/omitted arguments result in 1 UnexpArgToken, and included ones at minimum include 2 (1 for the arg itself, 1 for eof). This patch renames the otherwise unused getNumArguments to be more clear that it is the number of arguments that the Macro expects, and thus the maximum number that can be stringified. This patch also replaces the explicit memset (which results in value instantiation of the new tokens, PLUS clearing the memory) with brace initialization. Differential Revision: https://reviews.llvm.org/D32046 llvm-svn: 305425
2017-06-15 07:09:01 +08:00
std::unique_ptr<Preprocessor> CreatePP(StringRef Source,
TrivialModuleLoader &ModLoader) {
Make TargetInfo store an actual DataLayout instead of a string. Use it to calculate UserLabelPrefix, instead of specifying it (often incorrectly). Note that the *actual* user label prefix has always come from the DataLayout, and is handled within LLVM. The main thing clang's TargetInfo::UserLabelPrefix did was to set the #define value. Having these be different from each-other is just silly. Differential Revision: http://reviews.llvm.org/D17183 llvm-svn: 262737
2016-03-05 03:00:41 +08:00
std::unique_ptr<llvm::MemoryBuffer> Buf =
llvm::MemoryBuffer::getMemBuffer(Source);
unique_ptrify SourceManager::createFileID llvm-svn: 216715
2014-08-29 15:59:55 +08:00
SourceMgr.setMainFileID(SourceMgr.createFileID(std::move(Buf)));
[Lexer] Improve Lexer::getSourceText() when the given range deals with function macro arguments. This is a modified version of a patch by Manuel Klimek. llvm-svn: 182055
2013-05-17 05:37:39 +08:00
shared_ptrify (from InclusiveRefCntPtr) HeaderSearchOptions llvm-svn: 291202
2017-01-06 09:04:46 +08:00
HeaderSearch HeaderInfo(std::make_shared<HeaderSearchOptions>(), SourceMgr,
Diags, LangOpts, Target.get());
[Clang] Migrate llvm::make_unique to std::make_unique Now that we've moved to C++14, we no longer need the llvm::make_unique implementation from STLExtras.h. This patch is a mechanical replacement of (hopefully) all the llvm::make_unique instances across the monorepo. Differential revision: https://reviews.llvm.org/D66259 llvm-svn: 368942
2019-08-15 07:04:18 +08:00
std::unique_ptr<Preprocessor> PP = std::make_unique<Preprocessor>(
[Preprocessor]Correct Macro-Arg allocation of StringifiedArguments, correct getNumArguments StringifiedArguments is allocated (resized) based on the size the getNumArguments function. However, this function ACTUALLY currently returns the amount of total UnexpArgTokens which is minimum the same as the new implementation of getNumMacroArguments, since empty/omitted arguments result in 1 UnexpArgToken, and included ones at minimum include 2 (1 for the arg itself, 1 for eof). This patch renames the otherwise unused getNumArguments to be more clear that it is the number of arguments that the Macro expects, and thus the maximum number that can be stringified. This patch also replaces the explicit memset (which results in value instantiation of the new tokens, PLUS clearing the memory) with brace initialization. Differential Revision: https://reviews.llvm.org/D32046 llvm-svn: 305425
2017-06-15 07:09:01 +08:00
std::make_shared<PreprocessorOptions>(), Diags, LangOpts, SourceMgr,
Modules: Rename MemoryBufferCache to InMemoryModuleCache Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to Serialization. Another patch will start using it to manage module build more explicitly, but this is split out because it's mostly mechanical. Because of the move to Serialization we can no longer abuse the Preprocessor to forward it to the ASTReader. Besides the rename and file move, that means Preprocessor::Preprocessor has one fewer parameter and ASTReader::ASTReader has one more. llvm-svn: 355777
2019-03-10 01:33:56 +08:00
HeaderInfo, ModLoader,
[Preprocessor]Correct Macro-Arg allocation of StringifiedArguments, correct getNumArguments StringifiedArguments is allocated (resized) based on the size the getNumArguments function. However, this function ACTUALLY currently returns the amount of total UnexpArgTokens which is minimum the same as the new implementation of getNumMacroArguments, since empty/omitted arguments result in 1 UnexpArgToken, and included ones at minimum include 2 (1 for the arg itself, 1 for eof). This patch renames the otherwise unused getNumArguments to be more clear that it is the number of arguments that the Macro expects, and thus the maximum number that can be stringified. This patch also replaces the explicit memset (which results in value instantiation of the new tokens, PLUS clearing the memory) with brace initialization. Differential Revision: https://reviews.llvm.org/D32046 llvm-svn: 305425
2017-06-15 07:09:01 +08:00
/*IILookup =*/nullptr,
/*OwnsHeaderSearch =*/false);
PP->Initialize(*Target);
PP->EnterMainSourceFile();
return PP;
}
std::vector<Token> Lex(StringRef Source) {
TrivialModuleLoader ModLoader;
auto PP = CreatePP(Source, ModLoader);
[Lexer] Improve Lexer::getSourceText() when the given range deals with function macro arguments. This is a modified version of a patch by Manuel Klimek. llvm-svn: 182055
2013-05-17 05:37:39 +08:00
std::vector<Token> toks;
while (1) {
Token tok;
[Preprocessor]Correct Macro-Arg allocation of StringifiedArguments, correct getNumArguments StringifiedArguments is allocated (resized) based on the size the getNumArguments function. However, this function ACTUALLY currently returns the amount of total UnexpArgTokens which is minimum the same as the new implementation of getNumMacroArguments, since empty/omitted arguments result in 1 UnexpArgToken, and included ones at minimum include 2 (1 for the arg itself, 1 for eof). This patch renames the otherwise unused getNumArguments to be more clear that it is the number of arguments that the Macro expects, and thus the maximum number that can be stringified. This patch also replaces the explicit memset (which results in value instantiation of the new tokens, PLUS clearing the memory) with brace initialization. Differential Revision: https://reviews.llvm.org/D32046 llvm-svn: 305425
2017-06-15 07:09:01 +08:00
PP->Lex(tok);
[Lexer] Improve Lexer::getSourceText() when the given range deals with function macro arguments. This is a modified version of a patch by Manuel Klimek. llvm-svn: 182055
2013-05-17 05:37:39 +08:00
if (tok.is(tok::eof))
break;
toks.push_back(tok);
}
[Lexer] Don't merge macro args from different macro files The lexer sets the end location of macro arguments incorrectly *if*, while merging consecutive args to fit into a single SLocEntry, it finds args which come from different macro files. Fix the issue by using separate SLocEntries in this situation. This fixes a code coverage crasher (rdar://problem/26181005). Because the lexer reported end locations for certain macro args incorrectly, we would generate bogus coverage mappings with negative line offsets. Reviewed-by: akyrtzi Differential Revision: http://reviews.llvm.org/D20401 llvm-svn: 270160
2016-05-20 07:44:02 +08:00
return toks;
}
std::vector<Token> CheckLex(StringRef Source,
ArrayRef<tok::TokenKind> ExpectedTokens) {
auto toks = Lex(Source);
[Lexer] Improve Lexer::getSourceText() when the given range deals with function macro arguments. This is a modified version of a patch by Manuel Klimek. llvm-svn: 182055
2013-05-17 05:37:39 +08:00
EXPECT_EQ(ExpectedTokens.size(), toks.size());
for (unsigned i = 0, e = ExpectedTokens.size(); i != e; ++i) {
EXPECT_EQ(ExpectedTokens[i], toks[i].getKind());
}
return toks;
}
std::string getSourceText(Token Begin, Token End) {
bool Invalid;
StringRef Str =
Lexer::getSourceText(CharSourceRange::getTokenRange(SourceRange(
Begin.getLocation(), End.getLocation())),
SourceMgr, LangOpts, &Invalid);
if (Invalid)
return "<INVALID>";
return Str;
}
Add unit testing for Lexer. llvm-svn: 148478
2012-01-19 23:59:01 +08:00
FileSystemOptions FileMgrOpts;
FileManager FileMgr;
Basic: import IntrusiveRefCntPtr<> into clang namespace The class name is long enough without the llvm:: added. Also bring in RefCountedBase and RefCountedBaseVPTR. llvm-svn: 150958
2012-02-20 22:00:23 +08:00
IntrusiveRefCntPtr<DiagnosticIDs> DiagID;
Add unit testing for Lexer. llvm-svn: 148478
2012-01-19 23:59:01 +08:00
DiagnosticsEngine Diags;
SourceManager SourceMgr;
LangOptions LangOpts;
Use non-intrusive refcounting for TargetOptions llvm-svn: 212388
2014-07-06 13:26:44 +08:00
std::shared_ptr<TargetOptions> TargetOpts;
Basic: import IntrusiveRefCntPtr<> into clang namespace The class name is long enough without the llvm:: added. Also bring in RefCountedBase and RefCountedBaseVPTR. llvm-svn: 150958
2012-02-20 22:00:23 +08:00
IntrusiveRefCntPtr<TargetInfo> Target;
Add unit testing for Lexer. llvm-svn: 148478
2012-01-19 23:59:01 +08:00
};
[Lexer] Improve Lexer::getSourceText() when the given range deals with function macro arguments. This is a modified version of a patch by Manuel Klimek. llvm-svn: 182055
2013-05-17 05:37:39 +08:00
TEST_F(LexerTest, GetSourceTextExpandsToMaximumInMacroArgument) {
std::vector<tok::TokenKind> ExpectedTokens;
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::l_paren);
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::r_paren);
clang/unittests: Fixup corresponding to r172290. llvm-svn: 172295
2013-01-12 10:16:29 +08:00
[Lexer] Improve Lexer::getSourceText() when the given range deals with function macro arguments. This is a modified version of a patch by Manuel Klimek. llvm-svn: 182055
2013-05-17 05:37:39 +08:00
std::vector<Token> toks = CheckLex("#define M(x) x\n"
"M(f(M(i)))",
ExpectedTokens);
EXPECT_EQ("M(i)", getSourceText(toks[2], toks[2]));
}
TEST_F(LexerTest, GetSourceTextExpandsToMaximumInMacroArgumentForEndOfMacro) {
std::vector<tok::TokenKind> ExpectedTokens;
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::identifier);
std::vector<Token> toks = CheckLex("#define M(x) x\n"
"M(M(i) c)",
ExpectedTokens);
EXPECT_EQ("M(i)", getSourceText(toks[0], toks[0]));
}
TEST_F(LexerTest, GetSourceTextExpandsInMacroArgumentForBeginOfMacro) {
std::vector<tok::TokenKind> ExpectedTokens;
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::identifier);
std::vector<Token> toks = CheckLex("#define M(x) x\n"
"M(c c M(i))",
ExpectedTokens);
EXPECT_EQ("c M(i)", getSourceText(toks[1], toks[2]));
}
TEST_F(LexerTest, GetSourceTextExpandsInMacroArgumentForEndOfMacro) {
std::vector<tok::TokenKind> ExpectedTokens;
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::identifier);
std::vector<Token> toks = CheckLex("#define M(x) x\n"
"M(M(i) c c)",
ExpectedTokens);
EXPECT_EQ("M(i) c", getSourceText(toks[0], toks[1]));
}
TEST_F(LexerTest, GetSourceTextInSeparateFnMacros) {
std::vector<tok::TokenKind> ExpectedTokens;
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::identifier);
std::vector<Token> toks = CheckLex("#define M(x) x\n"
"M(c M(i)) M(M(i) c)",
ExpectedTokens);
EXPECT_EQ("<INVALID>", getSourceText(toks[1], toks[2]));
}
TEST_F(LexerTest, GetSourceTextWorksAcrossTokenPastes) {
std::vector<tok::TokenKind> ExpectedTokens;
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::l_paren);
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::r_paren);
std::vector<Token> toks = CheckLex("#define M(x) x\n"
"#define C(x) M(x##c)\n"
"M(f(C(i)))",
ExpectedTokens);
EXPECT_EQ("C(i)", getSourceText(toks[2], toks[2]));
}
TEST_F(LexerTest, GetSourceTextExpandsAcrossMultipleMacroCalls) {
std::vector<tok::TokenKind> ExpectedTokens;
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::l_paren);
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::r_paren);
std::vector<Token> toks = CheckLex("#define M(x) x\n"
"f(M(M(i)))",
ExpectedTokens);
EXPECT_EQ("M(M(i))", getSourceText(toks[2], toks[2]));
}
TEST_F(LexerTest, GetSourceTextInMiddleOfMacroArgument) {
std::vector<tok::TokenKind> ExpectedTokens;
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::l_paren);
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::r_paren);
std::vector<Token> toks = CheckLex("#define M(x) x\n"
"M(f(i))",
ExpectedTokens);
EXPECT_EQ("i", getSourceText(toks[2], toks[2]));
}
TEST_F(LexerTest, GetSourceTextExpandsAroundDifferentMacroCalls) {
std::vector<tok::TokenKind> ExpectedTokens;
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::l_paren);
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::r_paren);
std::vector<Token> toks = CheckLex("#define M(x) x\n"
"#define C(x) x\n"
"f(C(M(i)))",
ExpectedTokens);
EXPECT_EQ("C(M(i))", getSourceText(toks[2], toks[2]));
}
TEST_F(LexerTest, GetSourceTextOnlyExpandsIfFirstTokenInMacro) {
std::vector<tok::TokenKind> ExpectedTokens;
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::l_paren);
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::r_paren);
std::vector<Token> toks = CheckLex("#define M(x) x\n"
"#define C(x) c x\n"
"f(C(M(i)))",
ExpectedTokens);
EXPECT_EQ("M(i)", getSourceText(toks[3], toks[3]));
}
TEST_F(LexerTest, GetSourceTextExpandsRecursively) {
std::vector<tok::TokenKind> ExpectedTokens;
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::l_paren);
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::r_paren);
std::vector<Token> toks = CheckLex("#define M(x) x\n"
"#define C(x) c M(x)\n"
"C(f(M(i)))",
ExpectedTokens);
EXPECT_EQ("M(i)", getSourceText(toks[3], toks[3]));
}
Add unit testing for Lexer. llvm-svn: 148478
2012-01-19 23:59:01 +08:00
TEST_F(LexerTest, LexAPI) {
[Lexer] Improve Lexer::getSourceText() when the given range deals with function macro arguments. This is a modified version of a patch by Manuel Klimek. llvm-svn: 182055
2013-05-17 05:37:39 +08:00
std::vector<tok::TokenKind> ExpectedTokens;
ExpectedTokens.push_back(tok::l_square);
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::r_square);
ExpectedTokens.push_back(tok::l_square);
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::r_square);
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::identifier);
ExpectedTokens.push_back(tok::identifier);
std::vector<Token> toks = CheckLex("#define M(x) [x]\n"
"#define N(x) x\n"
"#define INN(x) x\n"
"#define NOF1 INN(val)\n"
"#define NOF2 val\n"
"M(foo) N([bar])\n"
"N(INN(val)) N(NOF1) N(NOF2) N(val)",
ExpectedTokens);
Add unit testing for Lexer. llvm-svn: 148478
2012-01-19 23:59:01 +08:00
SourceLocation lsqrLoc = toks[0].getLocation();
SourceLocation idLoc = toks[1].getLocation();
SourceLocation rsqrLoc = toks[2].getLocation();
PR37189 Fix incorrect end source location and spelling for a split '>>' token. When a '>>' token is split into two '>' tokens (in C++11 onwards), or (as an extension) when we do the same for other tokens starting with a '>', we can't just use a location pointing to the first '>' as the location of the split token, because that would result in our miscomputing the length and spelling for the token. As a consequence, for example, a refactoring replacing 'A<X>' with something else would sometimes replace one character too many, and similarly diagnostics highlighting a template-id source range would highlight one character too many. Fix this by creating an expansion range covering the first character of the '>>' token, whose spelling is '>'. For this to work, we generalize the expansion range of a macro FileID to be either a token range (the common case) or a character range (used in this new case). llvm-svn: 331155
2018-04-30 13:25:48 +08:00
CharSourceRange macroRange = SourceMgr.getExpansionRange(lsqrLoc);
Introduce Lexer::makeFileCharRange() that accepts a token source range and returns a character range with file locations. llvm-svn: 148480
2012-01-19 23:59:14 +08:00
For Lexer's isAt[Start/End]OfMacroExpansion add an out parameter for the macro start/end location. It is commonly needed after calling the function; with this way we avoid recalculating it. llvm-svn: 148479
2012-01-19 23:59:08 +08:00
SourceLocation Loc;
EXPECT_TRUE(Lexer::isAtStartOfMacroExpansion(lsqrLoc, SourceMgr, LangOpts, &Loc));
Introduce Lexer::makeFileCharRange() that accepts a token source range and returns a character range with file locations. llvm-svn: 148480
2012-01-19 23:59:14 +08:00
EXPECT_EQ(Loc, macroRange.getBegin());
Add unit testing for Lexer. llvm-svn: 148478
2012-01-19 23:59:01 +08:00
EXPECT_FALSE(Lexer::isAtStartOfMacroExpansion(idLoc, SourceMgr, LangOpts));
EXPECT_FALSE(Lexer::isAtEndOfMacroExpansion(idLoc, SourceMgr, LangOpts));
For Lexer's isAt[Start/End]OfMacroExpansion add an out parameter for the macro start/end location. It is commonly needed after calling the function; with this way we avoid recalculating it. llvm-svn: 148479
2012-01-19 23:59:08 +08:00
EXPECT_TRUE(Lexer::isAtEndOfMacroExpansion(rsqrLoc, SourceMgr, LangOpts, &Loc));
Introduce Lexer::makeFileCharRange() that accepts a token source range and returns a character range with file locations. llvm-svn: 148480
2012-01-19 23:59:14 +08:00
EXPECT_EQ(Loc, macroRange.getEnd());
PR37189 Fix incorrect end source location and spelling for a split '>>' token. When a '>>' token is split into two '>' tokens (in C++11 onwards), or (as an extension) when we do the same for other tokens starting with a '>', we can't just use a location pointing to the first '>' as the location of the split token, because that would result in our miscomputing the length and spelling for the token. As a consequence, for example, a refactoring replacing 'A<X>' with something else would sometimes replace one character too many, and similarly diagnostics highlighting a template-id source range would highlight one character too many. Fix this by creating an expansion range covering the first character of the '>>' token, whose spelling is '>'. For this to work, we generalize the expansion range of a macro FileID to be either a token range (the common case) or a character range (used in this new case). llvm-svn: 331155
2018-04-30 13:25:48 +08:00
EXPECT_TRUE(macroRange.isTokenRange());
Introduce Lexer::makeFileCharRange() that accepts a token source range and returns a character range with file locations. llvm-svn: 148480
2012-01-19 23:59:14 +08:00
Change Lexer::makeFileCharRange() to have it accept a CharSourceRange instead of a SourceRange, and handle the case where the range is a char (not token) range. llvm-svn: 149677
2012-02-03 13:58:29 +08:00
CharSourceRange range = Lexer::makeFileCharRange(
CharSourceRange::getTokenRange(lsqrLoc, idLoc), SourceMgr, LangOpts);
Introduce Lexer::makeFileCharRange() that accepts a token source range and returns a character range with file locations. llvm-svn: 148480
2012-01-19 23:59:14 +08:00
EXPECT_TRUE(range.isInvalid());
Change Lexer::makeFileCharRange() to have it accept a CharSourceRange instead of a SourceRange, and handle the case where the range is a char (not token) range. llvm-svn: 149677
2012-02-03 13:58:29 +08:00
range = Lexer::makeFileCharRange(CharSourceRange::getTokenRange(idLoc, rsqrLoc),
Introduce Lexer::makeFileCharRange() that accepts a token source range and returns a character range with file locations. llvm-svn: 148480
2012-01-19 23:59:14 +08:00
SourceMgr, LangOpts);
EXPECT_TRUE(range.isInvalid());
Change Lexer::makeFileCharRange() to have it accept a CharSourceRange instead of a SourceRange, and handle the case where the range is a char (not token) range. llvm-svn: 149677
2012-02-03 13:58:29 +08:00
range = Lexer::makeFileCharRange(CharSourceRange::getTokenRange(lsqrLoc, rsqrLoc),
Introduce Lexer::makeFileCharRange() that accepts a token source range and returns a character range with file locations. llvm-svn: 148480
2012-01-19 23:59:14 +08:00
SourceMgr, LangOpts);
EXPECT_TRUE(!range.isTokenRange());
EXPECT_EQ(range.getAsRange(),
SourceRange(macroRange.getBegin(),
macroRange.getEnd().getLocWithOffset(1)));
Introduce Lexer::getSourceText() that returns a string for the source that the given source range encompasses. llvm-svn: 148481
2012-01-19 23:59:19 +08:00
StringRef text = Lexer::getSourceText(
Change Lexer::makeFileCharRange() to have it accept a CharSourceRange instead of a SourceRange, and handle the case where the range is a char (not token) range. llvm-svn: 149677
2012-02-03 13:58:29 +08:00
CharSourceRange::getTokenRange(lsqrLoc, rsqrLoc),
SourceMgr, LangOpts);
Introduce Lexer::getSourceText() that returns a string for the source that the given source range encompasses. llvm-svn: 148481
2012-01-19 23:59:19 +08:00
EXPECT_EQ(text, "M(foo)");
Enhance Lexer::makeFileCharRange to check for ranges inside a macro argument expansion, in which case it returns a file range in the location where the argument was spelled. llvm-svn: 148551
2012-01-21 00:52:43 +08:00
SourceLocation macroLsqrLoc = toks[3].getLocation();
SourceLocation macroIdLoc = toks[4].getLocation();
SourceLocation macroRsqrLoc = toks[5].getLocation();
SourceLocation fileLsqrLoc = SourceMgr.getSpellingLoc(macroLsqrLoc);
SourceLocation fileIdLoc = SourceMgr.getSpellingLoc(macroIdLoc);
SourceLocation fileRsqrLoc = SourceMgr.getSpellingLoc(macroRsqrLoc);
Change Lexer::makeFileCharRange() to have it accept a CharSourceRange instead of a SourceRange, and handle the case where the range is a char (not token) range. llvm-svn: 149677
2012-02-03 13:58:29 +08:00
range = Lexer::makeFileCharRange(
CharSourceRange::getTokenRange(macroLsqrLoc, macroIdLoc),
SourceMgr, LangOpts);
Enhance Lexer::makeFileCharRange to check for ranges inside a macro argument expansion, in which case it returns a file range in the location where the argument was spelled. llvm-svn: 148551
2012-01-21 00:52:43 +08:00
EXPECT_EQ(SourceRange(fileLsqrLoc, fileIdLoc.getLocWithOffset(3)),
range.getAsRange());
Change Lexer::makeFileCharRange() to have it accept a CharSourceRange instead of a SourceRange, and handle the case where the range is a char (not token) range. llvm-svn: 149677
2012-02-03 13:58:29 +08:00
range = Lexer::makeFileCharRange(CharSourceRange::getTokenRange(macroIdLoc, macroRsqrLoc),
Enhance Lexer::makeFileCharRange to check for ranges inside a macro argument expansion, in which case it returns a file range in the location where the argument was spelled. llvm-svn: 148551
2012-01-21 00:52:43 +08:00
SourceMgr, LangOpts);
EXPECT_EQ(SourceRange(fileIdLoc, fileRsqrLoc.getLocWithOffset(1)),
range.getAsRange());
PR37189 Fix incorrect end source location and spelling for a split '>>' token. When a '>>' token is split into two '>' tokens (in C++11 onwards), or (as an extension) when we do the same for other tokens starting with a '>', we can't just use a location pointing to the first '>' as the location of the split token, because that would result in our miscomputing the length and spelling for the token. As a consequence, for example, a refactoring replacing 'A<X>' with something else would sometimes replace one character too many, and similarly diagnostics highlighting a template-id source range would highlight one character too many. Fix this by creating an expansion range covering the first character of the '>>' token, whose spelling is '>'. For this to work, we generalize the expansion range of a macro FileID to be either a token range (the common case) or a character range (used in this new case). llvm-svn: 331155
2018-04-30 13:25:48 +08:00
macroRange = SourceMgr.getExpansionRange(macroLsqrLoc);
Change Lexer::makeFileCharRange() to have it accept a CharSourceRange instead of a SourceRange, and handle the case where the range is a char (not token) range. llvm-svn: 149677
2012-02-03 13:58:29 +08:00
range = Lexer::makeFileCharRange(
CharSourceRange::getTokenRange(macroLsqrLoc, macroRsqrLoc),
SourceMgr, LangOpts);
PR37189 Fix incorrect end source location and spelling for a split '>>' token. When a '>>' token is split into two '>' tokens (in C++11 onwards), or (as an extension) when we do the same for other tokens starting with a '>', we can't just use a location pointing to the first '>' as the location of the split token, because that would result in our miscomputing the length and spelling for the token. As a consequence, for example, a refactoring replacing 'A<X>' with something else would sometimes replace one character too many, and similarly diagnostics highlighting a template-id source range would highlight one character too many. Fix this by creating an expansion range covering the first character of the '>>' token, whose spelling is '>'. For this to work, we generalize the expansion range of a macro FileID to be either a token range (the common case) or a character range (used in this new case). llvm-svn: 331155
2018-04-30 13:25:48 +08:00
EXPECT_EQ(SourceRange(macroRange.getBegin(), macroRange.getEnd().getLocWithOffset(1)),
Enhance Lexer::makeFileCharRange to check for ranges inside a macro argument expansion, in which case it returns a file range in the location where the argument was spelled. llvm-svn: 148551
2012-01-21 00:52:43 +08:00
range.getAsRange());
text = Lexer::getSourceText(
CharSourceRange::getTokenRange(SourceRange(macroLsqrLoc, macroIdLoc)),
SourceMgr, LangOpts);
EXPECT_EQ(text, "[bar");
Improve Lexer::getImmediateMacroName to take into account inner macros of macro arguments. For "MAC1( MAC2(foo) )" and location of 'foo' token it would return "MAC1" instead of "MAC2". llvm-svn: 148704
2012-01-24 00:58:33 +08:00
SourceLocation idLoc1 = toks[6].getLocation();
SourceLocation idLoc2 = toks[7].getLocation();
SourceLocation idLoc3 = toks[8].getLocation();
SourceLocation idLoc4 = toks[9].getLocation();
EXPECT_EQ("INN", Lexer::getImmediateMacroName(idLoc1, SourceMgr, LangOpts));
EXPECT_EQ("INN", Lexer::getImmediateMacroName(idLoc2, SourceMgr, LangOpts));
EXPECT_EQ("NOF2", Lexer::getImmediateMacroName(idLoc3, SourceMgr, LangOpts));
EXPECT_EQ("N", Lexer::getImmediateMacroName(idLoc4, SourceMgr, LangOpts));
Add unit testing for Lexer. llvm-svn: 148478
2012-01-19 23:59:01 +08:00
}
[Lexer] Don't merge macro args from different macro files The lexer sets the end location of macro arguments incorrectly *if*, while merging consecutive args to fit into a single SLocEntry, it finds args which come from different macro files. Fix the issue by using separate SLocEntries in this situation. This fixes a code coverage crasher (rdar://problem/26181005). Because the lexer reported end locations for certain macro args incorrectly, we would generate bogus coverage mappings with negative line offsets. Reviewed-by: akyrtzi Differential Revision: http://reviews.llvm.org/D20401 llvm-svn: 270160
2016-05-20 07:44:02 +08:00
TEST_F(LexerTest, DontMergeMacroArgsFromDifferentMacroFiles) {
std::vector<Token> toks =
Lex("#define helper1 0\n"
"void helper2(const char *, ...);\n"
"#define M1(a, ...) helper2(a, ##__VA_ARGS__)\n"
"#define M2(a, ...) M1(a, helper1, ##__VA_ARGS__)\n"
"void f1() { M2(\"a\", \"b\"); }");
// Check the file corresponding to the "helper1" macro arg in M2.
//
// The lexer used to report its size as 31, meaning that the end of the
// expansion would be on the *next line* (just past `M2("a", "b")`). Make
// sure that we get the correct end location (the comma after "helper1").
SourceLocation helper1ArgLoc = toks[20].getLocation();
EXPECT_EQ(SourceMgr.getFileIDSize(SourceMgr.getFileID(helper1ArgLoc)), 8U);
}
[Preprocessor]Correct Macro-Arg allocation of StringifiedArguments, correct getNumArguments StringifiedArguments is allocated (resized) based on the size the getNumArguments function. However, this function ACTUALLY currently returns the amount of total UnexpArgTokens which is minimum the same as the new implementation of getNumMacroArguments, since empty/omitted arguments result in 1 UnexpArgToken, and included ones at minimum include 2 (1 for the arg itself, 1 for eof). This patch renames the otherwise unused getNumArguments to be more clear that it is the number of arguments that the Macro expects, and thus the maximum number that can be stringified. This patch also replaces the explicit memset (which results in value instantiation of the new tokens, PLUS clearing the memory) with brace initialization. Differential Revision: https://reviews.llvm.org/D32046 llvm-svn: 305425
2017-06-15 07:09:01 +08:00
TEST_F(LexerTest, DontOverallocateStringifyArgs) {
TrivialModuleLoader ModLoader;
auto PP = CreatePP("\"StrArg\", 5, 'C'", ModLoader);
llvm::BumpPtrAllocator Allocator;
[NFC] Refactor the Preprocessor function that handles Macro definitions and rename Arguments to Parameters in Macro Definitions. - Extracted the reading of the tokens out into a separate function. - Replace 'Argument' with 'Parameter' when referring to the identifiers of the macro definition (as opposed to the supplied arguments - MacroArgs - during the macro invocation). This is in preparation for submitting patches for review to implement __VA_OPT__ which will otherwise just keep lengthening the HandleDefineDirective function and making it less comprehensible. I will also directly update some extra clang tooling that is broken by the change from Argument to Parameter. Hopefully the bots will stay appeased. Thanks! llvm-svn: 308190
2017-07-18 01:18:43 +08:00
std::array<IdentifierInfo *, 3> ParamList;
[Preprocessor]Correct Macro-Arg allocation of StringifiedArguments, correct getNumArguments StringifiedArguments is allocated (resized) based on the size the getNumArguments function. However, this function ACTUALLY currently returns the amount of total UnexpArgTokens which is minimum the same as the new implementation of getNumMacroArguments, since empty/omitted arguments result in 1 UnexpArgToken, and included ones at minimum include 2 (1 for the arg itself, 1 for eof). This patch renames the otherwise unused getNumArguments to be more clear that it is the number of arguments that the Macro expects, and thus the maximum number that can be stringified. This patch also replaces the explicit memset (which results in value instantiation of the new tokens, PLUS clearing the memory) with brace initialization. Differential Revision: https://reviews.llvm.org/D32046 llvm-svn: 305425
2017-06-15 07:09:01 +08:00
MacroInfo *MI = PP->AllocateMacroInfo({});
MI->setIsFunctionLike();
[NFC] Refactor the Preprocessor function that handles Macro definitions and rename Arguments to Parameters in Macro Definitions. - Extracted the reading of the tokens out into a separate function. - Replace 'Argument' with 'Parameter' when referring to the identifiers of the macro definition (as opposed to the supplied arguments - MacroArgs - during the macro invocation). This is in preparation for submitting patches for review to implement __VA_OPT__ which will otherwise just keep lengthening the HandleDefineDirective function and making it less comprehensible. I will also directly update some extra clang tooling that is broken by the change from Argument to Parameter. Hopefully the bots will stay appeased. Thanks! llvm-svn: 308190
2017-07-18 01:18:43 +08:00
MI->setParameterList(ParamList, Allocator);
EXPECT_EQ(3u, MI->getNumParams());
[Preprocessor]Correct Macro-Arg allocation of StringifiedArguments, correct getNumArguments StringifiedArguments is allocated (resized) based on the size the getNumArguments function. However, this function ACTUALLY currently returns the amount of total UnexpArgTokens which is minimum the same as the new implementation of getNumMacroArguments, since empty/omitted arguments result in 1 UnexpArgToken, and included ones at minimum include 2 (1 for the arg itself, 1 for eof). This patch renames the otherwise unused getNumArguments to be more clear that it is the number of arguments that the Macro expects, and thus the maximum number that can be stringified. This patch also replaces the explicit memset (which results in value instantiation of the new tokens, PLUS clearing the memory) with brace initialization. Differential Revision: https://reviews.llvm.org/D32046 llvm-svn: 305425
2017-06-15 07:09:01 +08:00
EXPECT_TRUE(MI->isFunctionLike());
Token Eof;
Eof.setKind(tok::eof);
std::vector<Token> ArgTokens;
while (1) {
Token tok;
PP->Lex(tok);
if (tok.is(tok::eof)) {
ArgTokens.push_back(Eof);
break;
}
if (tok.is(tok::comma))
ArgTokens.push_back(Eof);
else
ArgTokens.push_back(tok);
}
LexerTest memory leak fix- A new LexerTest unittest introduced a memory leak. This patch uses a unique_ptr with a custom deleter to ensure it is properly deleted. llvm-svn: 305491
2017-06-16 02:34:47 +08:00
auto MacroArgsDeleter = [&PP](MacroArgs *M) { M->destroy(*PP); };
std::unique_ptr<MacroArgs, decltype(MacroArgsDeleter)> MA(
MacroArgs::create(MI, ArgTokens, false, *PP), MacroArgsDeleter);
Remove cache for macro arg stringization Summary: The cache recorded the wrong expansion location for all but the first stringization. It seems uncommon to stringize the same macro argument multiple times, so this cache doesn't seem that important. Fixes PR39942 Reviewers: vsk, rsmith Subscribers: cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D65428 llvm-svn: 367337
2019-07-31 01:58:22 +08:00
auto StringifyArg = [&](int ArgNo) {
return MA->StringifyArgument(MA->getUnexpArgument(ArgNo), *PP,
/*Charify=*/false, {}, {});
};
Token Result = StringifyArg(0);
[Preprocessor]Correct Macro-Arg allocation of StringifiedArguments, correct getNumArguments StringifiedArguments is allocated (resized) based on the size the getNumArguments function. However, this function ACTUALLY currently returns the amount of total UnexpArgTokens which is minimum the same as the new implementation of getNumMacroArguments, since empty/omitted arguments result in 1 UnexpArgToken, and included ones at minimum include 2 (1 for the arg itself, 1 for eof). This patch renames the otherwise unused getNumArguments to be more clear that it is the number of arguments that the Macro expects, and thus the maximum number that can be stringified. This patch also replaces the explicit memset (which results in value instantiation of the new tokens, PLUS clearing the memory) with brace initialization. Differential Revision: https://reviews.llvm.org/D32046 llvm-svn: 305425
2017-06-15 07:09:01 +08:00
EXPECT_EQ(tok::string_literal, Result.getKind());
EXPECT_STREQ("\"\\\"StrArg\\\"\"", Result.getLiteralData());
Remove cache for macro arg stringization Summary: The cache recorded the wrong expansion location for all but the first stringization. It seems uncommon to stringize the same macro argument multiple times, so this cache doesn't seem that important. Fixes PR39942 Reviewers: vsk, rsmith Subscribers: cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D65428 llvm-svn: 367337
2019-07-31 01:58:22 +08:00
Result = StringifyArg(1);
[Preprocessor]Correct Macro-Arg allocation of StringifiedArguments, correct getNumArguments StringifiedArguments is allocated (resized) based on the size the getNumArguments function. However, this function ACTUALLY currently returns the amount of total UnexpArgTokens which is minimum the same as the new implementation of getNumMacroArguments, since empty/omitted arguments result in 1 UnexpArgToken, and included ones at minimum include 2 (1 for the arg itself, 1 for eof). This patch renames the otherwise unused getNumArguments to be more clear that it is the number of arguments that the Macro expects, and thus the maximum number that can be stringified. This patch also replaces the explicit memset (which results in value instantiation of the new tokens, PLUS clearing the memory) with brace initialization. Differential Revision: https://reviews.llvm.org/D32046 llvm-svn: 305425
2017-06-15 07:09:01 +08:00
EXPECT_EQ(tok::string_literal, Result.getKind());
EXPECT_STREQ("\"5\"", Result.getLiteralData());
Remove cache for macro arg stringization Summary: The cache recorded the wrong expansion location for all but the first stringization. It seems uncommon to stringize the same macro argument multiple times, so this cache doesn't seem that important. Fixes PR39942 Reviewers: vsk, rsmith Subscribers: cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D65428 llvm-svn: 367337
2019-07-31 01:58:22 +08:00
Result = StringifyArg(2);
[Preprocessor]Correct Macro-Arg allocation of StringifiedArguments, correct getNumArguments StringifiedArguments is allocated (resized) based on the size the getNumArguments function. However, this function ACTUALLY currently returns the amount of total UnexpArgTokens which is minimum the same as the new implementation of getNumMacroArguments, since empty/omitted arguments result in 1 UnexpArgToken, and included ones at minimum include 2 (1 for the arg itself, 1 for eof). This patch renames the otherwise unused getNumArguments to be more clear that it is the number of arguments that the Macro expects, and thus the maximum number that can be stringified. This patch also replaces the explicit memset (which results in value instantiation of the new tokens, PLUS clearing the memory) with brace initialization. Differential Revision: https://reviews.llvm.org/D32046 llvm-svn: 305425
2017-06-15 07:09:01 +08:00
EXPECT_EQ(tok::string_literal, Result.getKind());
EXPECT_STREQ("\"'C'\"", Result.getLiteralData());
#if !defined(NDEBUG) && GTEST_HAS_DEATH_TEST
Remove cache for macro arg stringization Summary: The cache recorded the wrong expansion location for all but the first stringization. It seems uncommon to stringize the same macro argument multiple times, so this cache doesn't seem that important. Fixes PR39942 Reviewers: vsk, rsmith Subscribers: cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D65428 llvm-svn: 367337
2019-07-31 01:58:22 +08:00
EXPECT_DEATH(StringifyArg(3), "Invalid arg #");
[Preprocessor]Correct Macro-Arg allocation of StringifiedArguments, correct getNumArguments StringifiedArguments is allocated (resized) based on the size the getNumArguments function. However, this function ACTUALLY currently returns the amount of total UnexpArgTokens which is minimum the same as the new implementation of getNumMacroArguments, since empty/omitted arguments result in 1 UnexpArgToken, and included ones at minimum include 2 (1 for the arg itself, 1 for eof). This patch renames the otherwise unused getNumArguments to be more clear that it is the number of arguments that the Macro expects, and thus the maximum number that can be stringified. This patch also replaces the explicit memset (which results in value instantiation of the new tokens, PLUS clearing the memory) with brace initialization. Differential Revision: https://reviews.llvm.org/D32046 llvm-svn: 305425
2017-06-15 07:09:01 +08:00
#endif
}
[Lexer] Finding beginning of token with escaped new line Summary: Lexer::GetBeginningOfToken produced invalid location when backtracking across escaped new lines. This fixes PR26228 Reviewers: akyrtzi, alexfh, rsmith, doug.gregor Reviewed By: alexfh Subscribers: alexfh, cfe-commits Patch by Paweł Żukowski! Differential Revision: https://reviews.llvm.org/D30748 llvm-svn: 310576
2017-08-10 18:06:16 +08:00
TEST_F(LexerTest, IsNewLineEscapedValid) {
auto hasNewLineEscaped = [](const char *S) {
return Lexer::isNewLineEscaped(S, S + strlen(S) - 1);
};
EXPECT_TRUE(hasNewLineEscaped("\\\r"));
EXPECT_TRUE(hasNewLineEscaped("\\\n"));
EXPECT_TRUE(hasNewLineEscaped("\\\r\n"));
EXPECT_TRUE(hasNewLineEscaped("\\\n\r"));
EXPECT_TRUE(hasNewLineEscaped("\\ \t\v\f\r"));
EXPECT_TRUE(hasNewLineEscaped("\\ \t\v\f\r\n"));
EXPECT_FALSE(hasNewLineEscaped("\\\r\r"));
EXPECT_FALSE(hasNewLineEscaped("\\\r\r\n"));
EXPECT_FALSE(hasNewLineEscaped("\\\n\n"));
EXPECT_FALSE(hasNewLineEscaped("\r"));
EXPECT_FALSE(hasNewLineEscaped("\n"));
EXPECT_FALSE(hasNewLineEscaped("\r\n"));
EXPECT_FALSE(hasNewLineEscaped("\n\r"));
EXPECT_FALSE(hasNewLineEscaped("\r\r"));
EXPECT_FALSE(hasNewLineEscaped("\n\n"));
}
TEST_F(LexerTest, GetBeginningOfTokenWithEscapedNewLine) {
// Each line should have the same length for
// further offset calculation to be more straightforward.
const unsigned IdentifierLength = 8;
std::string TextToLex = "rabarbar\n"
"foo\\\nbar\n"
"foo\\\rbar\n"
"fo\\\r\nbar\n"
"foo\\\n\rba\n";
std::vector<tok::TokenKind> ExpectedTokens{5, tok::identifier};
std::vector<Token> LexedTokens = CheckLex(TextToLex, ExpectedTokens);
for (const Token &Tok : LexedTokens) {
std::pair<FileID, unsigned> OriginalLocation =
SourceMgr.getDecomposedLoc(Tok.getLocation());
for (unsigned Offset = 0; Offset < IdentifierLength; ++Offset) {
SourceLocation LookupLocation =
Tok.getLocation().getLocWithOffset(Offset);
std::pair<FileID, unsigned> FoundLocation =
SourceMgr.getDecomposedExpansionLoc(
Lexer::GetBeginningOfToken(LookupLocation, SourceMgr, LangOpts));
// Check that location returned by the GetBeginningOfToken
// is the same as original token location reported by Lexer.
EXPECT_EQ(FoundLocation.second, OriginalLocation.second);
}
}
}
[Lex] Avoid out-of-bounds dereference in SkipLineComment Credit to OSS-Fuzz for discovery: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=3145 rdar://34526482 llvm-svn: 315785
2017-10-14 09:18:30 +08:00
TEST_F(LexerTest, AvoidPastEndOfStringDereference) {
[Lex] Inline a variable in test in preparation for more similar tests. NFC. llvm-svn: 322240
2018-01-11 07:37:29 +08:00
EXPECT_TRUE(Lex(" // \\\n").empty());
[Lex] Avoid out-of-bounds dereference in LexAngledStringLiteral. Fix makes the loop in LexAngledStringLiteral more like the loops in LexStringLiteral, LexCharConstant. When we skip a character after backslash, we need to check if we reached the end of the file instead of reading the next character unconditionally. Discovered by OSS-Fuzz: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=3832 rdar://problem/35572754 Reviewers: arphaman, kcc, rsmith, dexonsmith Reviewed By: rsmith, dexonsmith Subscribers: cfe-commits, rsmith, dexonsmith Differential Revision: https://reviews.llvm.org/D41423 llvm-svn: 322390
2018-01-13 02:54:35 +08:00
EXPECT_TRUE(Lex("#include <\\\\").empty());
EXPECT_TRUE(Lex("#include <\\\\\n").empty());
[Lex] Avoid out-of-bounds dereference in SkipLineComment Credit to OSS-Fuzz for discovery: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=3145 rdar://34526482 llvm-svn: 315785
2017-10-14 09:18:30 +08:00
}
Stringizing raw string literals containing newline Summary: This patch implements 4.3 of http://open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4220.pdf. If a raw string contains a newline character, replace each newline character with the \n escape code. Without this patch, included test case (macro_raw_string.cpp) results compilation failure. Reviewers: rsmith, doug.gregor, jkorous-apple Reviewed By: jkorous-apple Subscribers: jkorous-apple, vsapsai, cfe-commits Differential Revision: https://reviews.llvm.org/D39279 llvm-svn: 319904
2017-12-07 01:00:53 +08:00
TEST_F(LexerTest, StringizingRasString) {
// For "std::string Lexer::Stringify(StringRef Str, bool Charify)".
std::string String1 = R"(foo
{"bar":[]}
baz)";
// For "void Lexer::Stringify(SmallVectorImpl<char> &Str)".
SmallString<128> String2;
String2 += String1.c_str();
// Corner cases.
std::string String3 = R"(\
\n
\\n
\\)";
SmallString<128> String4;
String4 += String3.c_str();
std::string String5 = R"(a\
\\b)";
SmallString<128> String6;
String6 += String5.c_str();
String1 = Lexer::Stringify(StringRef(String1));
Lexer::Stringify(String2);
String3 = Lexer::Stringify(StringRef(String3));
Lexer::Stringify(String4);
String5 = Lexer::Stringify(StringRef(String5));
Lexer::Stringify(String6);
EXPECT_EQ(String1, R"(foo\n {\"bar\":[]}\n baz)");
EXPECT_EQ(String2, R"(foo\n {\"bar\":[]}\n baz)");
EXPECT_EQ(String3, R"(\\\n \\n\n \\\\n\n \\\\)");
EXPECT_EQ(String4, R"(\\\n \\n\n \\\\n\n \\\\)");
EXPECT_EQ(String5, R"(a\\\n\n\n \\\\b)");
EXPECT_EQ(String6, R"(a\\\n\n\n \\\\b)");
}
Re-apply r357823 "[Lexer] NFC: Fix an off-by-one bug in getAsCharRange()." It now comes with a follow-up fix for the clients of this API in clangd and clang-tidy. Differential Revision: https://reviews.llvm.org/D59977 llvm-svn: 359035
2019-04-24 05:15:26 +08:00
TEST_F(LexerTest, CharRangeOffByOne) {
std::vector<Token> toks = Lex(R"(#define MOO 1
void foo() { MOO; })");
const Token &moo = toks[5];
EXPECT_EQ(getSourceText(moo, moo), "MOO");
SourceRange R{moo.getLocation(), moo.getLocation()};
EXPECT_TRUE(
Lexer::isAtStartOfMacroExpansion(R.getBegin(), SourceMgr, LangOpts));
EXPECT_TRUE(
Lexer::isAtEndOfMacroExpansion(R.getEnd(), SourceMgr, LangOpts));
CharSourceRange CR = Lexer::getAsCharRange(R, SourceMgr, LangOpts);
EXPECT_EQ(Lexer::getSourceText(CR, SourceMgr, LangOpts), "MOO"); // Was "MO".
}
Add unit testing for Lexer. llvm-svn: 148478
2012-01-19 23:59:01 +08:00
} // anonymous namespace