In some cases it is desirable to match the same pattern repeatedly
many times. Currently the only way to do it is to copy the same
check pattern as many times as needed. And that gets pretty unwieldy
when its more than count is big.
Introducing CHECK-COUNT-<num> directive which acts like a plain CHECK
directive yet matches the same pattern exactly <num> times.
Extended FileCheckType to a struct to add Count there.
Changed some parsing routines to handle non-fixed length of directive
(all currently existing directives were fixed-length).
The code is generic enough to allow future support for COUNT in more
than just PlainCheck directives.
See motivating example for this feature in reviews.llvm.org/D54223.
Reviewed By: chandlerc, dblaikie
Differential Revision: https://reviews.llvm.org/D54336
llvm-svn: 346722
This feature makes it easy to tune FileCheck diagnostic output when
running the test suite via ninja, a bot, or an IDE. For example:
```
$ FILECHECK_OPTS='-color -v -dump-input-on-failure' \
LIT_FILTER='OpenMP/for_codegen.cpp' ninja check-clang \
| less -R
```
Reviewed By: probinson
Differential Revision: https://reviews.llvm.org/D53517
llvm-svn: 346272
(Relands r344930, reverted in r344935, and now hopefully fixed for
Windows.)
While this change specifically targets FileCheck, it affects any tool
using the same SourceMgr facilities.
Previously, -color was documented in FileCheck's -help output, but
-color had no effect. Now, -color obeys its documentation: it forces
colors to be used in FileCheck diagnostics even when stderr is not a
terminal.
-color is especially helpful when combined with FileCheck's -v, which
can produce a long series of diagnostics that you might wish to pipe
to a pager, such as less -R. The WithColor extensions here will also
help to clean up color usage in FileCheck's annotated dump of input,
which is proposed in D52999.
Reviewed By: JDevlieghere, zturner
Differential Revision: https://reviews.llvm.org/D53419
llvm-svn: 345202
While this change specifically targets FileCheck, it affects any tool
using the same SourceMgr facilities.
Previously, -color was documented in FileCheck's -help output, but
-color had no effect. Now, -color obeys its documentation: it forces
colors to be used in FileCheck diagnostics even when stderr is not a
terminal.
-color is especially helpful when combined with FileCheck's -v, which
can produce a long series of diagnostics that you might wish to pipe
to a pager, such as less -R. The WithColor extensions here will also
help to clean up color usage in FileCheck's annotated dump of input,
which is proposed in D52999.
Reviewed By: JDevlieghere
Differential Revision: https://reviews.llvm.org/D53419
llvm-svn: 344930
A DAG-NOT-DAG is a CHECK-DAG group, X, followed by a CHECK-NOT group,
N, followed by a CHECK-DAG group, Y. Let y be the initial directive
of Y. This patch makes the following changes to the behavior:
1. Directives in N can no longer match within part of Y's match
range just because y happens not to be the earliest match from
Y. Specifically, this patch withdraws N's search range end
from y's match range start to Y's match range start.
2. y can no longer match within X's match range, where a y match
produced a reordering complaint, which is thus no longer
possible. Specifically, this patch withdraws y's search range
start from X's permitted range start to X's match range end,
which was already the search range start for other members of
Y.
Both of these changes can only increase the number of test passes: #1
constrains the ability of CHECK-NOTs to match, and #2 expands the
ability of CHECK-DAGs to match without complaints.
These changes are based on discussions at:
<http://lists.llvm.org/pipermail/llvm-dev/2018-May/123550.html>
<https://reviews.llvm.org/D47106>
which conclude that:
1. These changes simplify the FileCheck conceptual model. First,
it makes search ranges for DAG-NOT-DAG more consistent with
other cases. Second, it was confusing that y was treated
differently from the rest of Y.
2. These changes add theoretical use cases for DAG-NOT-DAG that
had no obvious means to be expressed otherwise. We can justify
the first half of this assertion with the observation that
these changes can only increase the number of test passes.
3. Reordering detection for DAG-NOT-DAG had no obvious real
benefit.
We don't have evidence from real uses cases to help us debate
conclusions #2 and #3, but #1 at least seems intuitive.
Reviewed By: probinson
Differential Revision: https://reviews.llvm.org/D48986
llvm-svn: 337605
-v prints all directive pattern matches.
-vv additionally prints info that might be noise to users but that can
be helpful to FileCheck developers.
To maximize code reuse and to make diagnostics more consistent, this
patch also adjusts and extends some of the existing diagnostics.
CHECK-NOT failures now report variables uses. Many more diagnostics
now report the check prefix and kind of directive.
Reviewed By: probinson
Differential Revision: https://reviews.llvm.org/D47114
llvm-svn: 336967
That is, make CHECK-DAG skip matches that overlap the matches of any
preceding consecutive CHECK-DAG directives. This change makes
CHECK-DAG more consistent with other directives, and there is evidence
it makes CHECK-DAG more intuitive and less error-prone. See the RFC
discussion starting at:
http://lists.llvm.org/pipermail/llvm-dev/2018-May/123010.html
Moreover, this behavior enables CHECK-DAG groups for unordered,
non-unique strings or patterns. For example, it is useful for
verifying output or logs from a parallel program, such as the OpenMP
runtime.
This patch also implements the command-line option
-allow-deprecated-dag-overlap, which reverts CHECK-DAG to the old
overlapping behavior. This option should not be used in new tests.
It is meant only for the existing tests that are broken by this change
and that need time to update.
See the following bugzilla issue for tracking of such tests:
https://bugs.llvm.org/show_bug.cgi?id=37532
Patches to add -allow-deprecated-dag-overlap to those tests will
follow immediately.
Reviewed By: probinson
Differential Revision: https://reviews.llvm.org/D47106
llvm-svn: 336847
That is, make CHECK-DAG skip matches that overlap the matches of any
preceding consecutive CHECK-DAG directives. This change makes
CHECK-DAG more consistent with other directives, and there is evidence
it makes CHECK-DAG more intuitive and less error-prone. See the RFC
discussion starting at:
http://lists.llvm.org/pipermail/llvm-dev/2018-May/123010.html
Moreover, this behavior enables CHECK-DAG groups for unordered,
non-unique strings or patterns. For example, it is useful for
verifying output or logs from a parallel program, such as the OpenMP
runtime.
This patch also implements the command-line option
-allow-deprecated-dag-overlap, which reverts CHECK-DAG to the old
overlapping behavior. This option should not be used in new tests.
It is meant only for the existing tests that are broken by this change
and that need time to update.
See the following bugzilla issue for tracking of such tests:
https://bugs.llvm.org/show_bug.cgi?id=37532
Patches to add -allow-deprecated-dag-overlap to those tests will
follow immediately.
Reviewed By: probinson
Differential Revision: https://reviews.llvm.org/D47106
llvm-svn: 336830
Prior to this change, there was no clean way of getting FileCheck to
check that a line is completely empty. The expected way of using
"CHECK: {{^$}}" does not work because the '^' matches the end of the
previous match (this behaviour may be desirable in certain instances).
For the same reason, "CHECK-NEXT: {{^$}}" will fail when the previous
match was at the end of the line, as the pattern will match there.
Using the recommended [[:space:]] to match an explicit new line could
also match a space, and thus is not always desired. Literal '\n'
matches also do not work. A workaround was suggested in the review, but
it is a little clunky.
This change adds a new directive that behaves the same as CHECK-NEXT,
except that it only matches against empty lines (nothing, not even
whitespace, is allowed). As with CHECK-NEXT, it will fail if more than
one newline occurs before the next blank line. Example usage:
; test.txt
foo
bar
; CHECK: foo
; CHECK-EMPTY:
; CHECK-NEXT: bar
Differential Revision: https://reviews.llvm.org/D28896
Reviewed by: probinson
llvm-svn: 335613
FileCheck tool crashes when trying to parse --check-prefix argument if there is no any
data after it.
For example test like following would crash if there are no symbols and no EOL mark after `boom`:
# REQUIRES: x86
# RUN: <skipped few lines>
# RUN: llvm-readobj -t %t | FileCheck %s --check-prefix=boom
Patch fixes the issue.
Differential revision: https://reviews.llvm.org/D42057
llvm-svn: 322536
Summary:
This makes it very easy to test files that only differ in a constant
value somewhere in the test case.
Reviewers: jlebar, hfinkel, chandlerc, probinson
Reviewed By: probinson
Subscribers: probinson, llvm-commits
Differential Revision: https://reviews.llvm.org/D39629
llvm-svn: 317572
If there's enough data in fron of it the skipped region would just
become arbitrarily large, and we scan for the CHECK-NOT everywhere.
llvm-svn: 304900
If `--enable-var-scope` is in effect, variables with names that
start with `$` are considered to be global. All other variables are
local. All local variables get undefined at the beginning of each
CHECK-LABEL block. Global variables are not affected by CHECK-LABEL.
This makes it easier to ensure that individual tests are not affected
by variables set in preceding tests.
Differential Revision: https://reviews.llvm.org/D30749
llvm-svn: 297396
I unfortunately neglected to add it in r260540, but it has been
sitting in my working dir ever since. D'oh.
Modified to work with r290069, which made the CHECK patterns
themselves whitespace-sensitive as well, and remove the test added
then, as this tests both strict and non-strict modes.
llvm-svn: 291499
Summary:
This new alias takes a comma separated list of prefixes which allows
'--check-prefix=A --check-prefix=B --check-prefix=C' to be written as
'--check-prefixes=A,B,C'.
Reviewers: probinson
Subscribers: probinson, llvm-commits, dsanders
Differential Revision: http://reviews.llvm.org/D21293
llvm-svn: 272670
Combinations of suffixes that look useful are actually ignored;
complaining about them will avoid mistakes.
Differential Revision: http://reviews.llvm.org/D17587
llvm-svn: 262263
Combinations of suffixes that look useful actually are ignored;
complaining about them will avoid mistakes.
Differential Revision: http://reviews.llvm.org/D17587
llvm-svn: 262092
Add `CHECK-SAME`, which requires that the pattern matches on the *same*
line as the previous `CHECK`/`CHECK-NEXT` -- in other words, no newline
is allowed in the skipped region. This is similar to `CHECK-NEXT`,
which requires exactly 1 newline in the skipped region.
My motivation is to simplify checking the long lines of LLVM assembly
for the new debug info hierarchy. This allows CHECK sequences like the
following:
CHECK: ![[REF]] = !SomeMDNode(
CHECK-SAME: file: ![[FILE:[0-9]+]]
CHECK-SAME: otherField: 93{{[,)]}}
which is equivalent to:
CHECK: ![[REF]] = !SomeMDNode({{.*}}file: ![[FILE:[0-9]+]]{{.*}}otherField: 93{{[,)]}}
While this example just has two fields, many nodes in debug info have
more than that. `CHECK-SAME` will keep the logic easy to follow.
Morever, it enables interleaving `CHECK-NOT`s without allowing newlines.
Consider the following:
CHECK: ![[REF]] = !SomeMDNode(
CHECK-SAME: file: ![[FILE:[0-9]+]]
CHECK-NOT: unexpectedField:
CHECK-SAME: otherField: 93{{[,)]}}
CHECK-NOT: otherUnexpectedField:
CHECK-SAME: )
which doesn't seem to have an equivalent `CHECK` line.
llvm-svn: 230612
Currently FileCheck errors out on empty input. This is usually the
right thing to do, but makes testing things like "this command does
not emit some error message" hard to test. This usually leads to
people using "command 2>&1 | count 0" instead, and then the bots that
use guard malloc fail a few hours later.
By adding a flag to FileCheck that allows empty inputs, we can make
tests that consist entirely of "CHECK-NOT" lines feasible.
llvm-svn: 215127
Summary:
Add FileCheck -implicit-check-not option which allows specifying a
pattern that should only occur in the input when explicitly matched by a
positive check. This feature allows checking tool diagnostics in a way
clang -verify does it for compiler diagnostics.
The option has been tested on a number of clang-tidy checks, I'll post a link to
the clang-tidy patch to this thread.
Once there's an agreement on the general direction, I can add tests and
documentation.
Reviewers: djasper, bkramer
Reviewed By: bkramer
Subscribers: llvm-commits
Differential Revision: http://reviews.llvm.org/D4462
llvm-svn: 212810
Summary:
Directives are being ignored, when they occur between a partial-word false
match and any match on another prefix.
For example, with FOO and BAR prefixes:
_FOO
FOO: foo
BAR: bar
FileCheck incorrectly matches:
fog
bar
This happens because FOO falsely matched as a partial word at '_FOO' and was
ignored while BAR matched at 'BAR:'. The match of BAR is incorrectly returned
as the 'first match' causing the FOO directive to be discarded.
Fixed this the same way as r194565 (D2166) did for a similar test case.
The partial-word false match should be counted as a match for the purposes of
finding the first match of a prefix, but should be returned as a false match
using CheckTy::CheckNone so that it isn't treated as a directive.
Fixes PR17995
Reviewers: samsonov, arsenm
Reviewed By: samsonov
CC: llvm-commits
Differential Revision: http://llvm-reviews.chandlerc.com/D2228
llvm-svn: 195248
Summary:
Fix a case when "FileCheck --check-prefix=CHECK --check-prefix=CHECKER"
would silently ignore check-lines of the form:
CHECKER: foo
Reviewers: dsanders
Reviewed By: dsanders
CC: llvm-commits
Differential Revision: http://llvm-reviews.chandlerc.com/D2168
llvm-svn: 194577
Summary:
This fixes a subtle bug in new FileCheck feature added
in r194343. When we search for the first satisfying check-prefix,
we should actually return the first encounter of some check-prefix as a
substring, even if it's not a part of valid check-line. Otherwise
"FileCheck --check-prefix=FOO --check-prefix=BAR" with check file:
FOO not a vaild check-line
FOO: foo
BAR: bar
incorrectly accepted file:
fog
bar
as it skipped the first two encounters of FOO, matching only BAR: line.
Reviewers: arsenm, dsanders
Reviewed By: dsanders
CC: llvm-commits
Differential Revision: http://llvm-reviews.chandlerc.com/D2166
llvm-svn: 194565
- Instead of setting the suffixes in a bunch of places, just set one master
list in the top-level config. We now only modify the suffix list in a few
suites that have one particular unique suffix (.ml, .mc, .yaml, .td, .py).
- Aside from removing the need for a bunch of lit.local.cfg files, this enables
4 tests that were inadvertently being skipped (one in
Transforms/BranchFolding, a .s file each in DebugInfo/AArch64 and
CodeGen/PowerPC, and one in CodeGen/SI which is now failing and has been
XFAILED).
- This commit also fixes a bunch of config files to use config.root instead of
older copy-pasted code.
llvm-svn: 188513
FileCheck should check to make sure the prefix was found, and not a word
containing it (e.g -check-prefix=BASEREL shouldn't match NOBASEREL).
Patch by Ron Ofir.
llvm-svn: 188221
CHECK-LABEL is meant to be used in place on CHECK on lines containing identifiers or other unique labels (they need not actually be labels in the source or output language, though.) This is used to break up the input stream into separate blocks delineated by CHECK-LABEL lines, each of which is checked independently. This greatly improves the accuracy of errors and fix-it hints in many cases, and allows for FileCheck to recover from errors in one block by continuing to subsequent blocks.
Some tests will be converted to use this new directive in forthcoming patches.
llvm-svn: 186162
Pattern has source location by itself. After adding a trivial method to
retrieve it, it's unnecessary to pair a source location for CHECK-NOT patterns.
One thing revised after this is the diagnostic info is more accurate by
pointing to the start of the CHECK-NOT pattern instead of the end of the
CHECK-NOT pattern. E.g. diagnostic message previously looks like
<stdin>:1:1: error: CHECK-NOT: string occurred!
test
^
test.txt:1:16: note: CHECK-NOT: pattern specified here
CHECK-NOT: test
^
is changed to
<stdin>:1:1: error: CHECK-NOT: string occurred!
test
^
test.txt:1:12: note: CHECK-NOT: pattern specified here
CHECK-NOT: test
^
llvm-svn: 180578
; CHECK: [[VAR:[a-z]]]
The problem was that to find the end of the regex var definition, it was
simplistically looking for the next ]] and finding the incorrect one. A
better approach is to count nesting of brackets (taking escaping into
account). This way the brackets that are part of the regex can be discovered
and skipped properly, and the ]] ending is detected in the right place.
llvm-svn: 169109
Fedor Sergeev