Last year I was working at Swift to add support for [Localization of Compiler Diagnostic Messages](https://forums.swift.org/t/localization-of-compiler-diagnostic-messages/36412/41). We are currently using YAML as the new diagnostic format. The LLVM::YAMLParser didn't have a support for multiline string literal folding and it's crucial to have that for the diagnostic message to help us keep up with the 80 columns rule. Therefore, I decided to add a multiline string literal folding support to the YAML parser.
Patch By: @HassanElDesouky (Hassan ElDesouky)
Differential Revision: https://reviews.llvm.org/D102590
The cleanup was manual, but assisted by "include-what-you-use". It consists in
1. Removing unused forward declaration. No impact expected.
2. Removing unused headers in .cpp files. No impact expected.
3. Removing unused headers in .h files. This removes implicit dependencies and
is generally considered a good thing, but this may break downstream builds.
I've updated llvm, clang, lld, lldb and mlir deps, and included a list of the
modification in the second part of the commit.
4. Replacing header inclusion by forward declaration. This has the same impact
as 3.
Notable changes:
- llvm/Support/TargetParser.h no longer includes llvm/Support/AArch64TargetParser.h nor llvm/Support/ARMTargetParser.h
- llvm/Support/TypeSize.h no longer includes llvm/Support/WithColor.h
- llvm/Support/YAMLTraits.h no longer includes llvm/Support/Regex.h
- llvm/ADT/SmallVector.h no longer includes llvm/Support/MemAlloc.h nor llvm/Support/ErrorHandling.h
You may need to add some of these headers in your compilation units, if needs be.
As an hint to the impact of the cleanup, running
clang++ -E -Iinclude -I../llvm/include ../llvm/lib/Support/*.cpp -std=c++14 -fno-rtti -fno-exceptions | wc -l
before: 8000919 lines
after: 7917500 lines
Reduced dependencies also helps incremental rebuilds and is more ccache
friendly, something not shown by the above metric :-)
Discourse thread on the topic: https://llvm.discourse.group/t/include-what-you-use-include-cleanup/5831
This reverts commit fd4808887e.
This patch causes gcc to issue a lot of warnings like:
warning: base class ‘class llvm::MCParsedAsmOperand’ should be
explicitly initialized in the copy constructor [-Wextra]
Currently unknown keys when inputting mapping traits have the location set to the Value.
Example:
```
YAML:1:14: error: unknown key 'UnknownKey'
{UnknownKey: SomeValue}
^~~~~~~~~
```
This is unhelpful for a user as it draws them to fix the wrong item.
Reviewed By: silvas
Differential Revision: https://reviews.llvm.org/D93037
In some places the parser guards against dereferencing `End`, while in
others it relies on the presence of a trailing `'\0'` to elide checks.
Add the remaining guards needed to ensure the parser never attempts to
dereference `End`, making it safe to not require a null-terminated input
buffer.
Update the parser fuzzer harness so that it tests with buffers that are
guaranteed to be non-null-terminated, null-terminated, and 1-terminated,
additionally ensuring the result of the parse is the same in each case.
Some of the regression tests were written by inspection, and some are
cases caught by the fuzzer which required additional fixes in the
parser.
Differential Revision: https://reviews.llvm.org/D84050
The `Range` of an alias/anchor token includes the leading `&` or `*`,
but it is skipped while parsing the name. The check for an empty name
fails to account for the skipped leading character and so the error is
never hit.
Fix the off-by-one and add a couple regression tests.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D91462
Currently newer clang-format options cannot be included in .clang-format files, if not all users can be forced to use an updated version.
This patch tries to solve this by adding an option to clang-format, enabling to ignore unknown (newer) options.
Differential Revision: https://reviews.llvm.org/D86137
As detailed on PR45043, static analysis was warning that the StringRef::iterator Position argument was being ignored and the function was hardwired to use the Current iterator.
This patch ensures we use the provided iterator and removes the (barely necessary) setError wrapper that always used Current.
Differential Revision: https://reviews.llvm.org/D76512
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.
This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.
This doesn't actually modify StringRef yet, I'll do that in a follow-up.
Summary: The attached test case replicates a null dereference crash in
`yaml::Document::skip()`. This was fixed by adding a check and early
return in the method.
Reviewers: Bigcheese, hintonda, beanz
Reviewed By: hintonda
Subscribers: hiraditya, dexonsmith, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D69974
Summary: This patch fixes a number of bugs found in the YAML parser
through fuzzing. In general, this makes the parser more robust against
malformed inputs.
The fixes are mostly improved null checking and returning errors in
more cases. In some cases, asserts were changed to regular errors,
this provides the same robustness but also protects release builds
from the triggering conditions. This also improves the fuzzability of
the YAML parser since asserts can act as a roadblock to further
fuzzing once they're hit.
Each fix has a corresponding test case:
- TestAnchorMapError - Added proper null pointer handling in
`Stream::printError` if N is null and `KeyValueNode::getValue` if
getKey returns null, `Input::createHNodes` `dyn_casts` changed to
`dyn_cast_or_null` so the null pointer checks are actually able to
fail
- TestFlowSequenceTokenErrors - Added case in
`Document::parseBlockNode` for FlowMappingEnd, FlowSequenceEnd, or
FlowEntry tokens outside of mappings or sequences
- TestDirectiveMappingNoValue - Changed assert to regular error
return in `Scanner::scanValue`
- TestUnescapeInfiniteLoop - Fixed infinite loop in
`ScalarNode::unescapeDoubleQuoted` by returning an error for
unrecognized escape codes
- TestScannerUnexpectedCharacter - Changed asserts to regular error
returns in `Scanner::consume`
- TestUnknownDirective - For both of the inputs the stream doesn't
fail and correctly returns TK_Error, but there is no valid root
node for the document. There's no reasonable way to make the
scanner fail for unknown directives without breaking the YAML spec
(see spec-07-01.test). I think the assert is unnecessary given
that an error is still generated for this case.
The `SimpleKeys.clear()` line fixes a bug found by AddressSanitizer
triggered by multiple test cases - when TokenQueue is cleared
SimpleKeys is still holding dangling pointers into it, so SimpleKeys
should be cleared as well.
Patch by Thomas Finch!
Reviewers: chandlerc, Bigcheese, hintonda
Reviewed By: Bigcheese, hintonda
Subscribers: hintonda, kristina, beanz, dexonsmith, hiraditya, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D61608
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
This is a follow-up to r331272.
We've been running doxygen with the autobrief option for a couple of
years now. This makes the \brief markers into our comments
redundant. Since they are a visual distraction and we don't want to
encourage more \brief markers in new code either, this patch removes
them all.
Patch produced by
for i in $(git grep -l '\@brief'); do perl -pi -e 's/\@brief //g' $i & done
https://reviews.llvm.org/D46290
llvm-svn: 331275
The existing YAML Output::scalarString code path includes a partial and
incorrect implementation of YAML escaping logic. In particular, the logic put
in place in rL321283 escapes non-printable bytes only if they are not part of a
multibyte UTF8 sequence; implicitly this means that all multibyte UTF8
sequences -- printable and non -- are passed through verbatim.
The simplest solution to this is to direct the Output::scalarString method to
use the standalone yaml::escape function, and this _almost_ works, except that
the existing code in that function _over_ escapes: any multibyte UTF8 sequence
is escaped, even printable ones. While this is permitted for YAML, it is also
more aggressive (and hard to read for non-English locales) than necessary,
and the entire point of rL321283 was to back off such aggressive over-escaping.
So in this change, I have both redirected Output::scalarString to use
yaml::escape _and_ modified yaml::escape to optionally restrict its escaping to
non-printables. This preserves behaviour of any existing clients while giving
them a path to more moderate escaping should they desire.
Reviewers: JDevlieghere, thegameg, MatzeB, vladimir.plyashkun
Reviewed By: thegameg
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D44863
llvm-svn: 328661
I did this a long time ago with a janky python script, but now
clang-format has built-in support for this. I fed clang-format every
line with a #include and let it re-sort things according to the precise
LLVM rules for include ordering baked into clang-format these days.
I've reverted a number of files where the results of sorting includes
isn't healthy. Either places where we have legacy code relying on
particular include ordering (where possible, I'll fix these separately)
or where we have particular formatting around #include lines that
I didn't want to disturb in this patch.
This patch is *entirely* mechanical. If you get merge conflicts or
anything, just ignore the changes in this patch and run clang-format
over your #include lines in the files.
Sorry for any noise here, but it is important to keep these things
stable. I was seeing an increasing number of patches with irrelevant
re-ordering of #include lines because clang-format was used. This patch
at least isolates that churn, makes it easy to skip when resolving
conflicts, and gets us to a clean baseline (again).
llvm-svn: 304787
Some scanner errors were not checked and reported by the parser.
Fix PR30934. Recommit r288014 after fixing unittest.
Patch by: Serge Guelton <serge.guelton@telecom-bretagne.eu>
Differential Revision: https://reviews.llvm.org/D26419
llvm-svn: 288071
- Add AllocatorList, a non-intrusive list that owns an LLVM-style
allocator and provides a std::list-like interface (trivially built on
top of simple_ilist),
- add a typedef (and unit tests) for BumpPtrList, and
- use BumpPtrList for the list of llvm::yaml::Token (i.e., TokenQueueT).
TokenQueueT has no need for the complexity of an intrusive list. The
only reason to inherit from ilist was to customize the allocator.
TokenQueueT was the only example in-tree of using ilist<> in a truly
non-intrusive way.
Moreover, this removes the final use of the non-intrusive
ilist_traits<>::createNode (after r280573, r281177, and r281181). I
have a WIP patch that removes this customization point (and the API that
relies on it) that I plan to commit soon.
Note: AllocatorList owns the allocator, which limits the viable API
(e.g., splicing must be on the same list). For now I've left out
any problematic API. It wouldn't be hard to split AllocatorList into
two layers: an Impl class that calls DerivedT::getAlloc (via CRTP), and
derived classes that handle Allocator ownership/reference/etc semantics;
and then implement splice with appropriate assertions; but TBH we should
probably just customize the std::list allocators at that point.
llvm-svn: 281182
Many lists want to override only allocation semantics, or callbacks for
iplist. Split these up to prevent code duplication.
- Specialize ilist_alloc_traits to change the implementations of
deleteNode() and createNode().
- One common desire is to do nothing deleteNode() and disable
createNode(). Specialize ilist_alloc_traits to inherit from
ilist_noalloc_traits for that behaviour.
- Specialize ilist_callback_traits to use the addNodeToList(),
removeNodeFromList(), and transferNodesFromList() callbacks.
As a drive-by, add some coverage to the callback-related unit tests.
llvm-svn: 280128
Remove all the dead code around ilist_*sentinel_traits. This is a
follow-up to gutting them as part of r279314 (originally r278974),
staged to prevent broken builds in sub-projects.
Uses were removed from clang in r279457 and lld in r279458.
llvm-svn: 279473
Share code for the (mostly problematic) embedded sentinel traits.
- Move the LLVM_NO_SANITIZE("object-size") attribute to
ilist_half_embedded_sentinel_traits and ilist_embedded_sentinel_traits
(previously it spread throughout the code duplication).
- Add an ilist_full_embedded_sentinel_traits which has no UB (but has
the downside of storing the complete node).
- Replace all the custom sentinel traits in LLVM with a declaration of
ilist_sentinel_traits that inherits from one of the embedded sentinel
traits classes.
There are still custom sentinel traits in other LLVM subprojects. I'll
remove those in a follow-up.
Nothing at all should be changing here, this is just rearranging code.
Note that the final goal here is to remove the sentinel traits
altogether, settling on the memory layout of
ilist_half_embedded_sentinel_traits without the UB. This intermediate
step moves the logic into ilist.h.
llvm-svn: 278513
r260925 introduced a version of the *trim methods which is preferable
when trimming a single kind of character. Update all users in llvm.
llvm-svn: 260926
Stop relying on ilist implicit conversions from `value_type&` to
`iterator` in YAMLParser.cpp.
I eventually want to outlaw this entirely. It encourages
`getNextNode()` and `getPrevNode()` in iterator logic, which is
extremely fragile (and relies on them never returning `nullptr`).
FTR, there's nothing nefarious going on in this case, it was just easy
to clean up since the callers really wanted iterators to begin with.
llvm-svn: 249767
The patch is generated using this command:
tools/clang/tools/extra/clang-tidy/tool/run-clang-tidy.py -fix \
-checks=-*,llvm-namespace-comment -header-filter='llvm/.*|clang/.*' \
llvm/lib/
Thanks to Eugene Kosov for the original patch!
llvm-svn: 240137
The commit null terminates the string value in the `yaml::BlockScalarNode`
class.
This change is motivated by the initial MIR serialization commit (r237708)
that I reverted in r237730 because the LLVM IR source from the block
scalar node wasn't terminated by a null character and thus the buildbots
failed on one testcase sometimes. This change enables me to recommit
the reverted commit.
llvm-svn: 237942
The commit r237314 that implements YAML block parsing
introduced a leak that was caught by the ASAN linux buildbot.
YAML Parser stores its tokens in an ilist, and allocates
tokens using a BumpPtrAllocator, but doesn't call the
destructor for the allocated tokens. R237314 added an
std::string field to a Token which leaked as the Token's
destructor wasn't called. This commit fixes this leak
by calling the Token's destructor when a Token is being
removed from an ilist of tokens.
llvm-svn: 237389
This commit implements the parsing of YAML block scalars.
Some code existed for it before, but it couldn't parse block
scalars.
This commit adds a new yaml node type to represent the block
scalar values.
This commit also deletes the 'spec-09-27' and 'spec-09-28' tests
as they are identical to the test file 'spec-09-26'.
This commit introduces 3 new utility functions to the YAML scanner
class: `skip_s_space`, `advanceWhile` and `consumeLineBreakIfPresent`.
Reviewers: Duncan P. N. Exon Smith
Differential Revision: http://reviews.llvm.org/D9503
llvm-svn: 237314
This commit enables the tests located in test/YAMLParser directory.
Those tests were never actually enabled, as llvm-lit didn't pick up the
files with the 'data' extension. The commit renames those test files to files
with the 'test' extension so that llvm-lit would find them.
This commit also modifies yaml-bench so that it returns an error status
if an error occurred during parsing. It also adds the '-use-color'
command line option to yaml-bench (to make sure that file check matches
the error messages in the output stream).
This commit modifies some of the renamed tests so that they wouldn't
fail. It gets rid of XFAILs and uses the 'not' command instead for
some of the tests that have to fail during parsing. This commit
also adds some 'FIXME' comments to a couple of tests that are
supposed to fail but currently pass because of various bugs
in the implementation of the yaml parser.
Reviewers: Justin Bogner
Differential Revision: http://reviews.llvm.org/D9448
llvm-svn: 236754
This commit extracts the code that skips over a YAML comment from
the 'scanToNextToken' method into a separate 'skipComment' method.
This refactoring is motivated by a patch that implements parsing
of YAML block scalars (http://reviews.llvm.org/D9503), as the
method that parses a block scalar reuses the 'skipComment' method.
llvm-svn: 236663
Kazu Hirata