Commit Graph

165 Commits

Author SHA1 Message Date
David Bozier 6e01cd6795 Improve error message of FileCheck when stdin is empty
Summary: Replace '-' in the error message with <stdin>. This is also consistent with another error message in the code.

Reviewers: jhenderson, probinson, jdenny, grimar, arichardson

Reviewed By: jhenderson

Subscribers: thopre, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D73793
2020-02-04 11:14:55 +00:00
Joel E. Denny fdde18a7c3 [FileCheck] Given multiple -dump-input, prefer most verbose
Problem: `FILECHECK_OPTS` was implemented so that a test runner, such
as a bot, can specify FileCheck debugging options, such as
`-dump-input=fail`.  However, some existing test suites have FileCheck
calls that already specify `-dump-input=fail` or `-dump-input=always`.
Without this patch, such tests fail under such a test runner because
FileCheck doesn't accept multiple occurrences of `-dump-input`.

Solution: This patch permits multiple occurrences of `-dump-input` by
assigning precedence to its values in the following descending order:
`help`, `always`, `fail`, and `never`.  That is, any occurrence of
`help` always obtains help, and otherwise the behavior is similar to
`-v` vs. `-vv` in that the option specifying the greatest verbosity
has precedence.

Rationale: My justification for the new behavior is as follows.  I
have not experienced use cases where, either as a test runner or as a
test author, I want to **limit** the permitted debugging verbosity
(except as a test author in FileCheck's or lit's test suites where the
FileCheck debugging output itself is under test, but the solution
there is `env FILECHECK_OPTS=`, and I imagine we should use the same
solution anywhere else this need might occur).  Of course, as either a
test runner or test author, it is useful to **increase** debugging
verbosity.

Reviewed By: probinson

Differential Revision: https://reviews.llvm.org/D70784
2019-12-03 14:21:13 -05:00
Kai Nacke 5b5b2fd2b8 [FileCheck] Implement --ignore-case option.
The FileCheck utility is enhanced to support a `--ignore-case`
option. This is useful in cases where the output of Unix tools
differs in case (e.g. case not specified by Posix).

Reviewers: Bigcheese, jakehehrlich, rupprecht, espindola, alexshap, jhenderson, MaskRay

Differential Revision: https://reviews.llvm.org/D68146

llvm-svn: 374538
2019-10-11 11:59:14 +00:00
Dmitri Gribenko d3aed7fc79 Revert "[FileCheck] Implement --ignore-case option."
This reverts commit r374339. It broke tests:
http://lab.llvm.org:8011/builders/clang-x86_64-debian-fast/builds/19066

llvm-svn: 374359
2019-10-10 14:27:14 +00:00
Kai Nacke dfd2b6f07f [FileCheck] Implement --ignore-case option.
The FileCheck utility is enhanced to support a `--ignore-case`
option. This is useful in cases where the output of Unix tools
differs in case (e.g. case not specified by Posix).

Reviewers: Bigcheese, jakehehrlich, rupprecht, espindola, alexshap, jhenderson, MaskRay

Differential Revision: https://reviews.llvm.org/D68146

llvm-svn: 374339
2019-10-10 13:15:41 +00:00
Thomas Preud'homme 02ada9bd2b [FileCheck] Remove implementation types from API
Summary:
Remove use of FileCheckPatternContext and FileCheckString concrete types
from FileCheck API to allow moving it and the other implementation only
only declarations into a private header file.

Reviewers: jhenderson, chandlerc, jdenny, probinson, grimar, arichardson, rnk

Subscribers: hiraditya, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D68186

llvm-svn: 373211
2019-09-30 14:12:03 +00:00
Joel E. Denny dbb757f462 [FileCheck] Document FILECHECK_OPTS in -help
Reviewed By: thopre

Differential Revision: https://reviews.llvm.org/D65707

llvm-svn: 368787
2019-08-14 02:56:20 +00:00
Joel E. Denny 608f2bfd65 [FileCheck] Move -dump-input diagnostic to first line
Without this patch, `-dump-input` prints a diagnostic at the end of
its marker range.  For example:

```
         1: Start.
check:1     ^~~~~~
         2: Bad.
next:2      X~~~
         3: Many lines
next:2      ~~~~~~~~~~
         4: of input.
next:2      ~~~~~~~~~
         5: End.
next:2      ~~~~ error: no match found
```

This patch moves it to the beginning like this:

```
         1: Start.
check:1     ^~~~~~
         2: Bad.
next:2      X~~~ error: no match found
         3: Many lines
next:2      ~~~~~~~~~~
         4: of input.
next:2      ~~~~~~~~~
         5: End.
next:2      ~~~~
```

The former somehow looks nicer because the diagnostic doesn't appear
to be somewhere within the marker range.  However, the latter is more
practical, especially when the marker range includes the remainder of
a very long dump.  First, in the case of an error, this patch enables
me to search the dump for `error:` and usually immediately land where
the detected error began.  Second, when trying to follow FileCheck's
logic, it's best to read top down, so this patch enables me to see
each diagnostic as soon as I encounter its marker.

Reviewed By: thopre

Differential Revision: https://reviews.llvm.org/D65702

llvm-svn: 368786
2019-08-14 02:56:09 +00:00
Rui Ueyama 4d41c332ef Revert r367649: Improve raw_ostream so that you can "write" colors using operator<<
This reverts commit r367649 in an attempt to unbreak Windows bots.

llvm-svn: 367658
2019-08-02 07:22:34 +00:00
Rui Ueyama a52f982f1c Improve raw_ostream so that you can "write" colors using operator<<
1. raw_ostream supports ANSI colors so that you can write messages to
the termina with colors. Previously, in order to change and reset
color, you had to call `changeColor` and `resetColor` functions,
respectively.

So, if you print out "error: " in red, for example, you had to do
something like this:

  OS.changeColor(raw_ostream::RED);
  OS << "error: ";
  OS.resetColor();

With this patch, you can write the same code as follows:

  OS << raw_ostream::RED << "error: " << raw_ostream::RESET;

2. Add a boolean flag to raw_ostream so that you can disable colored
output. If you disable colors, changeColor, operator<<(Color),
resetColor and other color-related functions have no effect.

Most LLVM tools automatically prints out messages using colors, and
you can disable it by passing a flag such as `--disable-colors`.
This new flag makes it easy to write code that works that way.

Differential Revision: https://reviews.llvm.org/D65564

llvm-svn: 367649
2019-08-02 04:48:30 +00:00
Michal Gorny ffc722a358 [llvm] [FileCheck] Use FILECHECK_DUMP_INPUT_ON_FAILURE only when non-empty
Enable dumping output only if FILECHECK_DUMP_INPUT_ON_FAILURE is set to
a non-empty value.  This is necessary to support disabling it via
POSIX-compliant env(1) that does not support '-u' argument,
and therefore fix regression caused by r366980.

Differential Revision: https://reviews.llvm.org/D65334

llvm-svn: 367122
2019-07-26 15:38:57 +00:00
Thomas Preud'homme 1a944d27b2 FileCheck: Improve FileCheck variable terminology
Summary:
Terminology introduced by [[#]] blocks is confusing and does not
integrate well with existing terminology.

First, variables referred by [[]] blocks are called "pattern variables"
while the text a CHECK directive needs to match is called a "CHECK
pattern". This is inconsistent with variables in [[#]] blocks since
[[#]] blocks are also found in CHECK pattern yet those variables are
called "numeric variable".

Second, the replacing of both [[]] and [[#]] blocks by the value of the
variable or expression they contain is represented by a
FileCheckPatternSubstitution class. The naming refers to being a
substitution in a CHECK pattern but could be wrongly understood as being
a substitution of a pattern variable.

Third and lastly, comments use "numeric expression" to refer both to the
[[#]] blocks as well as to the numeric expressions these blocks contain
which get evaluated at match time.

This patch solves these confusions by
- calling variables in [[]] and [[#]] blocks as string and numeric
  variables respectively;
- referring to [[]] and [[#]] as substitution *blocks*, with the former
  being a string substitution block and the latter a numeric
  substitution block;
- calling [[]] and [[#]] blocks to be replaced by the value of a
  variable or expression they contain a substitution (as opposed to
  definition when these blocks are used to defined a variable), with the
  former being a string substitution and the latter a numeric
  substitution;
- renaming the FileCheckPatternSubstitution as a FileCheckSubstitution
  class with FileCheckStringSubstitution and
  FileCheckNumericSubstitution subclasses;
- restricting the use of "numeric expression" to refer to the expression
  that is evaluated in a numeric substitution.

While numeric substitution blocks only support numeric substitutions of
numeric expressions at the moment there are plans to augment numeric
substitution blocks to support numeric definitions as well as both a
numeric definition and numeric substitution in the same numeric
substitution block.

Reviewers: jhenderson, jdenny, probinson, arichardson

Subscribers: hiraditya, arichardson, probinson, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D62146

llvm-svn: 361445
2019-05-23 00:10:14 +00:00
Rainer Orth 010982f750 [FileCheck] Fix FileCheck.cpp compilation on Solaris
Both LLVM 8.0.0 and current trunk fail to compile on Solaris with GCC 8.1.0:

  /vol/llvm/src/llvm/dist/utils/FileCheck/FileCheck.cpp: In function ‘void DumpAnnotatedInput(llvm::raw_ostream&, const llvm::FileCheckRequest&, llvm::StringRef, std::vector<InputAnnotation>&, unsigned int)’:
  /vol/llvm/src/llvm/dist/utils/FileCheck/FileCheck.cpp:408:41: error: call of overloaded ‘log10(unsigned int&)’ is ambiguous
     unsigned LineNoWidth = log10(LineCount) + 1;
                                           ^
  In file included from /vol/gcc-8/lib/gcc/i386-pc-solaris2.11/8.1.0/include-fixed/math.h:24,
                   from /vol/gcc-8/include/c++/8.1.0/cmath:45,
                   from /vol/llvm/src/llvm/dist/include/llvm-c/DataTypes.h:28,
                   from /vol/llvm/src/llvm/dist/include/llvm/Support/DataTypes.h:16,
                   from /vol/llvm/src/llvm/dist/include/llvm/ADT/Hashing.h:47,
                   from /vol/llvm/src/llvm/dist/include/llvm/ADT/ArrayRef.h:12,
                   from /vol/llvm/src/llvm/dist/include/llvm/Support/CommandLine.h:22,
                   from /vol/llvm/src/llvm/dist/utils/FileCheck/FileCheck.cpp:18:
  /vol/gcc-8/lib/gcc/i386-pc-solaris2.11/8.1.0/include-fixed/iso/math_iso.h:209:21: note: candidate: ‘long double std::log10(long double)’
    inline long double log10(long double __X) { return __log10l(__X); }
                       ^~~~~
  /vol/gcc-8/lib/gcc/i386-pc-solaris2.11/8.1.0/include-fixed/iso/math_iso.h:170:15: note: candidate: ‘float std::log10(float)’
    inline float log10(float __X) { return __log10f(__X); }
                 ^~~~~
  /vol/gcc-8/lib/gcc/i386-pc-solaris2.11/8.1.0/include-fixed/iso/math_iso.h:70:15: note: candidate: ‘double std::log10(double)’
   extern double log10 __P((double));
                 ^~~~~

Fixed by using std::log10 instead, which allowed the compilation on i386-pc-solaris2.11
and sparc-sun-solaris2.11 to continue.

Differential Revision: https://reviews.llvm.org/D60043

llvm-svn: 357509
2019-04-02 18:38:23 +00:00
Thomas Preud'homme a5e233bf79 Recommit: Detect incorrect FileCheck variable CLI definition
Summary:
While the backend code of FileCheck relies on definition of variable
from the command-line to have an equal sign '=' and a variable name
before that, the frontend does not actually enforce it. This leads to
FileCheck crashing when invoked with invalid syntax for the -D option.

This patch adds the missing validation in the frontend. It also makes
the -D option an AlwaysPrefix option to be able to detect -D=FOO as
being a define without variable and -D as missing its value.

Copyright:
- Linaro (changes in version 2 of revision D55940)
- GraphCore (changes in later versions)

Reviewers: jdenny

Subscribers: JonChesterfield, hiraditya, kristina, probinson,
llvm-commits

Differential Revision: https://reviews.llvm.org/D55940

llvm-svn: 353173
2019-02-05 14:17:28 +00:00
Thomas Preud'homme 447abc57c5 Revert "Detect incorrect FileCheck variable CLI definition"
This reverts commit r351039.

llvm-svn: 352309
2019-01-27 09:02:19 +00:00
Joel E. Denny 352695c336 [FileCheck] Suppress old -v/-vv diags if dumping input
The old diagnostic form of the trace produced by -v and -vv looks
like:

```
check1:1:8: remark: CHECK: expected string found in input
CHECK: abc
       ^
<stdin>:1:3: note: found here
; abc def
  ^~~
```

When dumping annotated input is requested (via -dump-input), I find
that this old trace is not useful and is sometimes harmful:

1. The old trace is mostly redundant because the same basic
   information also appears in the input dump's annotations.

2. The old trace buries any error diagnostic between it and the input
   dump, but I find it useful to see any error diagnostic up front.

3. FILECHECK_OPTS=-dump-input=fail requests annotated input dumps only
   for failed FileCheck calls.  However, I have to also add -v or -vv
   to get a full set of annotations, and that can produce massive
   output from all FileCheck calls in all tests.  That's a real
   problem when I run this in the IDE I use, which grinds to a halt as
   it tries to capture all that output.

When -dump-input=fail|always, this patch suppresses the old trace from
-v or -vv.  Error diagnostics still print as usual.  If you want the
old trace, perhaps to see variable expansions, you can set
-dump-input=none (the default).

Reviewed By: probinson

Differential Revision: https://reviews.llvm.org/D55825

llvm-svn: 351881
2019-01-22 21:41:42 +00:00
Chandler Carruth 2946cd7010 Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636
2019-01-19 08:50:56 +00:00
Thomas Preud'homme 84f4ff5119 Detect incorrect FileCheck variable CLI definition
Summary:
While the backend code of FileCheck relies on definition of variable
from the command-line to have an equal sign '=' and a variable name
before that, the frontend does not actually enforce it. This leads to
FileCheck crashing when invoked with invalid syntax for the -D option.

This patch adds the missing validation in the frontend. It also makes
the -D option an AlwaysPrefix option to be able to detect -D=FOO as
being a define without variable and -D as missing its value.

Copyright:
- Linaro (changes in version 2 of revision D55940)
- GraphCore (changes in later versions)

Reviewers: jdenny

Subscribers: JonChesterfield, hiraditya, kristina, probinson,
llvm-commits

Differential Revision: https://reviews.llvm.org/D55940

llvm-svn: 351039
2019-01-14 09:29:10 +00:00
Joel E. Denny e2afb61499 [FileCheck] Annotate input dump (final tweaks)
Apply final suggestions from probinson for this patch series plus a
few more tweaks:

* Improve various docs, for MatchType in particular.

* Rename some members of MatchType.  The main problem was that the
  term "final match" became a misnomer when CHECK-COUNT-<N> was
  created.

* Split InputStartLine, etc. declarations into multiple lines.

Differential Revision: https://reviews.llvm.org/D55738

Reviewed By: probinson

llvm-svn: 349425
2018-12-18 00:03:51 +00:00
Joel E. Denny 96f0e84ccf [FileCheck] Annotate input dump (7/7)
This patch implements annotations for diagnostics reporting CHECK-NOT
failed matches.  These diagnostics are enabled by -vv.  As for
diagnostics reporting failed matches for other directives, these
annotations mark the search ranges using `X~~`.  The difference here
is that failed matches for CHECK-NOT are successes not errors, so they
are green not red when colors are enabled.

For example:

```
$ FileCheck -dump-input=help
The following description was requested by -dump-input=help to
explain the input annotations printed by -dump-input=always and
-dump-input=fail:

  - L:     labels line number L of the input file
  - T:L    labels the only match result for a pattern of type T from line L of
           the check file
  - T:L'N  labels the Nth match result for a pattern of type T from line L of
           the check file
  - ^~~    marks good match (reported if -v)
  - !~~    marks bad match, such as:
           - CHECK-NEXT on same line as previous match (error)
           - CHECK-NOT found (error)
           - CHECK-DAG overlapping match (discarded, reported if -vv)
  - X~~    marks search range when no match is found, such as:
           - CHECK-NEXT not found (error)
           - CHECK-NOT not found (success, reported if -vv)
           - CHECK-DAG not found after discarded matches (error)
  - ?      marks fuzzy match when no match is found
  - colors success, error, fuzzy match, discarded match, unmatched input

If you are not seeing color above or in input dumps, try: -color

$ FileCheck -vv -dump-input=always check5 < input5 |& sed -n '/^<<<</,$p'
<<<<<<
         1: abcdef
check:1     ^~~
not:2          X~~
         2: ghijkl
not:2       ~~~
check:3        ^~~
         3: mnopqr
not:4       X~~~~~
         4: stuvwx
not:4       ~~~~~~
         5:
eof:4       ^
>>>>>>

$ cat check5
CHECK: abc
CHECK-NOT: foobar
CHECK: jkl
CHECK-NOT: foobar

$ cat input5
abcdef
ghijkl
mnopqr
stuvwx
```

Reviewed By: george.karpenkov, probinson

Differential Revision: https://reviews.llvm.org/D53899

llvm-svn: 349424
2018-12-18 00:03:36 +00:00
Joel E. Denny f7c1c4d8a4 [FileCheck] Annotate input dump (6/7)
This patch implements input annotations for diagnostics reporting
CHECK-DAG discarded matches.  These diagnostics are enabled by -vv.
These annotations mark discarded match ranges using `!~~` because they
are bad matches even though they are not errors.

CHECK-DAG discarded matches create another case where there can be
multiple match results for the same directive.

For example:

```
$ FileCheck -dump-input=help
The following description was requested by -dump-input=help to
explain the input annotations printed by -dump-input=always and
-dump-input=fail:

  - L:     labels line number L of the input file
  - T:L    labels the only match result for a pattern of type T from line L of
           the check file
  - T:L'N  labels the Nth match result for a pattern of type T from line L of
           the check file
  - ^~~    marks good match (reported if -v)
  - !~~    marks bad match, such as:
           - CHECK-NEXT on same line as previous match (error)
           - CHECK-NOT found (error)
           - CHECK-DAG overlapping match (discarded, reported if -vv)
  - X~~    marks search range when no match is found, such as:
           - CHECK-NEXT not found (error)
           - CHECK-DAG not found after discarded matches (error)
  - ?      marks fuzzy match when no match is found
  - colors success, error, fuzzy match, discarded match, unmatched input

If you are not seeing color above or in input dumps, try: -color

$ FileCheck -vv -dump-input=always check4 < input4 |& sed -n '/^<<<</,$p'
<<<<<<
         1: abcdef
dag:1       ^~~~
dag:2'0       !~~~ discard: overlaps earlier match
         2: cdefgh
dag:2'1     ^~~~
check:3         X~ error: no match found
>>>>>>

$ cat check4
CHECK-DAG: abcd
CHECK-DAG: cdef
CHECK: efgh

$ cat input4
abcdef
cdefgh
```

This shows that the line 3 CHECK fails to match even though its
pattern appears in the input because its search range starts after the
line 2 CHECK-DAG's match range.  The trouble might be that the line 2
CHECK-DAG's match range is later than expected because its first match
range overlaps with the line 1 CHECK-DAG match range and thus is
discarded.

Because `!~~` for CHECK-DAG does not indicate an error, it is not
colored red.  Instead, when colors are enabled, it is colored cyan,
which suggests a match that went cold.

Reviewed By: george.karpenkov, probinson

Differential Revision: https://reviews.llvm.org/D53898

llvm-svn: 349423
2018-12-18 00:03:19 +00:00
Joel E. Denny 7df86967b4 [FileCheck] Annotate input dump (5/7)
This patch implements input annotations for diagnostics enabled by -v,
which report good matches for directives.  These annotations mark
match ranges using `^~~`.

For example:

```
$ FileCheck -dump-input=help
The following description was requested by -dump-input=help to
explain the input annotations printed by -dump-input=always and
-dump-input=fail:

  - L:     labels line number L of the input file
  - T:L    labels the only match result for a pattern of type T from line L of
           the check file
  - T:L'N  labels the Nth match result for a pattern of type T from line L of
           the check file
  - ^~~    marks good match (reported if -v)
  - !~~    marks bad match, such as:
           - CHECK-NEXT on same line as previous match (error)
           - CHECK-NOT found (error)
  - X~~    marks search range when no match is found, such as:
           - CHECK-NEXT not found (error)
  - ?      marks fuzzy match when no match is found
  - colors success, error, fuzzy match, unmatched input

If you are not seeing color above or in input dumps, try: -color

$ FileCheck -v -dump-input=always check3 < input3 |& sed -n '/^<<<</,$p'
<<<<<<
         1: abc foobar def
check:1     ^~~
not:2           !~~~~~     error: no match expected
check:3                ^~~
>>>>>>

$ cat check3
CHECK:     abc
CHECK-NOT: foobar
CHECK:     def

$ cat input3
abc foobar def
```

-vv enables these annotations for FileCheck's implicit EOF patterns as
well.  For an example where EOF patterns become relevant, see patch 7
in this series.

If colors are enabled, `^~~` is green to suggest success.

-v plus color enables highlighting of input text that has no final
match for any expected pattern.  The highlight uses a cyan background
to suggest a cold section.  This highlighting can make it easier to
spot text that was intended to be matched but that failed to be
matched in a long series of good matches.

CHECK-COUNT-<num> good matches are another case where there can be
multiple match results for the same directive.

Reviewed By: george.karpenkov, probinson

Differential Revision: https://reviews.llvm.org/D53897

llvm-svn: 349422
2018-12-18 00:03:03 +00:00
Joel E. Denny 0e7e3fa0e9 [FileCheck] Annotate input dump (4/7)
This patch implements input annotations for diagnostics that report
unexpected matches for CHECK-NOT.  Like wrong-line matches for
CHECK-NEXT, CHECK-SAME, and CHECK-EMPTY, these annotations mark match
ranges using red `!~~` to indicate bad matches that are errors.

For example:

```
$ FileCheck -dump-input=help
The following description was requested by -dump-input=help to
explain the input annotations printed by -dump-input=always and
-dump-input=fail:

  - L:     labels line number L of the input file
  - T:L    labels the only match result for a pattern of type T from line L of
           the check file
  - T:L'N  labels the Nth match result for a pattern of type T from line L of
           the check file
  - !~~    marks bad match, such as:
           - CHECK-NEXT on same line as previous match (error)
           - CHECK-NOT found (error)
  - X~~    marks search range when no match is found, such as:
           - CHECK-NEXT not found (error)
  - ?      marks fuzzy match when no match is found
  - colors error, fuzzy match

If you are not seeing color above or in input dumps, try: -color

$ FileCheck -v -dump-input=always check3 < input3 |& sed -n '/^<<<</,$p'
<<<<<<
       1: abc foobar def
not:2         !~~~~~     error: no match expected
>>>>>>

$ cat check3
CHECK:     abc
CHECK-NOT: foobar
CHECK:     def

$ cat input3
abc foobar def
```

Reviewed By: george.karpenkov, probinson

Differential Revision: https://reviews.llvm.org/D53896

llvm-svn: 349421
2018-12-18 00:02:47 +00:00
Joel E. Denny cadfcef493 [FileCheck] Annotate input dump (3/7)
This patch implements input annotations for diagnostics that report
wrong-line matches for the directives CHECK-NEXT, CHECK-SAME, and
CHECK-EMPTY.  Instead of the usual `^~~`, which is used by later
patches for good matches, these annotations use `!~~` to mark the bad
match ranges so that this category of errors is visually distinct.
Because such matches are errors, these annotates are red when colors
are enabled.

For example:

```
$ FileCheck -dump-input=help
The following description was requested by -dump-input=help to
explain the input annotations printed by -dump-input=always and
-dump-input=fail:

  - L:     labels line number L of the input file
  - T:L    labels the only match result for a pattern of type T from line L of
           the check file
  - T:L'N  labels the Nth match result for a pattern of type T from line L of
           the check file
  - !~~    marks bad match, such as:
           - CHECK-NEXT on same line as previous match (error)
  - X~~    marks search range when no match is found, such as:
           - CHECK-NEXT not found (error)
  - ?      marks fuzzy match when no match is found
  - colors error, fuzzy match

If you are not seeing color above or in input dumps, try: -color

$ FileCheck -v -dump-input=always check2 < input2 |& sed -n '/^<<<</,$p'
<<<<<<
        1: foo bar
next:2         !~~ error: match on wrong line
>>>>>>

$ cat check2
CHECK: foo
CHECK-NEXT: bar

$ cat input2
foo bar
```

Reviewed By: george.karpenkov, probinson

Differential Revision: https://reviews.llvm.org/D53894

llvm-svn: 349420
2018-12-18 00:02:22 +00:00
Joel E. Denny 2c007c807d [FileCheck] Annotate input dump (2/7)
This patch implements input annotations for diagnostics that suggest
fuzzy matches for directives for which no matches were found.  Instead
of using the usual `^~~`, which is used by later patches for good
matches, these annotations use `?` so that fuzzy matches are visually
distinct.  No tildes are included as these diagnostics (independently
of this patch) currently identify only the start of the match.

For example:

```
$ FileCheck -dump-input=help
The following description was requested by -dump-input=help to
explain the input annotations printed by -dump-input=always and
-dump-input=fail:

  - L:     labels line number L of the input file
  - T:L    labels the only match result for a pattern of type T from line L of
           the check file
  - T:L'N  labels the Nth match result for a pattern of type T from line L of
           the check file
  - X~~    marks search range when no match is found
  - ?      marks fuzzy match when no match is found
  - colors error, fuzzy match

If you are not seeing color above or in input dumps, try: -color

$ FileCheck -v -dump-input=always check1 < input1 |& sed -n '/^<<<</,$p'
<<<<<<
          1: ; abc def
          2: ; ghI jkl
next:3'0     X~~~~~~~~ error: no match found
next:3'1       ?       possible intended match
>>>>>>

$ cat check1
CHECK: abc
CHECK-SAME: def
CHECK-NEXT: ghi
CHECK-SAME: jkl

$ cat input1
; abc def
; ghI jkl
```

This patch introduces the concept of multiple "match results" per
directive.  In the above example, the first match result for the
CHECK-NEXT directive is the failed match, for which the annotation
shows the search range.  The second match result is the fuzzy match.
Later patches will introduce other cases of multiple match results per
directive.

When colors are enabled, `?` is colored magenta.  That is, it doesn't
indicate the actual error, which a red `X~~` marker indicates, but its
color suggests it's closely related.

Reviewed By: george.karpenkov, probinson

Differential Revision: https://reviews.llvm.org/D53893

llvm-svn: 349419
2018-12-18 00:02:04 +00:00
Joel E. Denny 3c5d267eb7 [FileCheck] Annotate input dump (1/7)
Extend FileCheck to dump its input annotated with FileCheck's
diagnostics: errors, good matches if -v, and additional information if
-vv.  The goal is to make it easier to visualize FileCheck's matching
behavior when debugging.

Each patch in this series implements input annotations for a
particular category of FileCheck diagnostics.  While the first few
patches alone are somewhat useful, the annotations become much more
useful as later patches implement annotations for -v and -vv
diagnostics, which show the matching behavior leading up to the error.

This first patch implements boilerplate plus input annotations for
error diagnostics reporting that no matches were found for a
directive.  These annotations mark the search ranges of the failed
directives.  Instead of using the usual `^~~`, which is used by later
patches for good matches, these annotations use `X~~` so that this
category of errors is visually distinct.

For example:

```
$ FileCheck -dump-input=help
The following description was requested by -dump-input=help to
explain the input annotations printed by -dump-input=always and
-dump-input=fail:

  - L:     labels line number L of the input file
  - T:L    labels the match result for a pattern of type T from line L of
           the check file
  - X~~    marks search range when no match is found
  - colors error

If you are not seeing color above or in input dumps, try: -color

$ FileCheck -v -dump-input=always check1 < input1 |& sed -n '/^Input file/,$p'
Input file: <stdin>
Check file: check1

-dump-input=help describes the format of the following dump.

Full input was:
<<<<<<
        1: ; abc def
        2: ; ghI jkl
next:3     X~~~~~~~~ error: no match found
>>>>>>

$ cat check1
CHECK: abc
CHECK-SAME: def
CHECK-NEXT: ghi
CHECK-SAME: jkl

$ cat input1
; abc def
; ghI jkl
```

Some additional details related to the boilerplate:

* Enabling: The annotated input dump is enabled by `-dump-input`,
  which can also be set via the `FILECHECK_OPTS` environment variable.
  Accepted values are `help`, `always`, `fail`, or `never`.  As shown
  above, `help` describes the format of the dump.  `always` is helpful
  when you want to investigate a successful FileCheck run, perhaps for
  an unexpected pass. `-dump-input-on-failure` and
  `FILECHECK_DUMP_INPUT_ON_FAILURE` remain as a deprecated alias for
  `-dump-input=fail`.

* Diagnostics: The usual diagnostics are not suppressed in this mode
  and are printed first.  For brevity in the example above, I've
  omitted them using a sed command.  Sometimes they're perfectly
  sufficient, and then they make debugging quicker than if you were
  forced to hunt through a dump of long input looking for the error.
  If you think they'll get in the way sometimes, keep in mind that
  it's pretty easy to grep for the start of the input dump, which is
  `<<<`.

* Colored Annotations: The annotated input is colored if colors are
  enabled (enabling colors can be forced using -color).  For example,
  errors are red.  However, as in the above example, colors are not
  vital to reading the annotations.

I don't know how to test color in the output, so any hints here would
be appreciated.

Reviewed By: george.karpenkov, zturner, probinson

Differential Revision: https://reviews.llvm.org/D52999

llvm-svn: 349418
2018-12-18 00:01:39 +00:00
Joel E. Denny 24994d77b8 [FileCheck] Parse command-line options from FILECHECK_OPTS
This feature makes it easy to tune FileCheck diagnostic output when
running the test suite via ninja, a bot, or an IDE.  For example:

```
$ FILECHECK_OPTS='-color -v -dump-input-on-failure' \
  LIT_FILTER='OpenMP/for_codegen.cpp' ninja check-clang \
  | less -R
```

Reviewed By: probinson

Differential Revision: https://reviews.llvm.org/D53517

llvm-svn: 346272
2018-11-06 22:07:03 +00:00
Joel E. Denny 3e66509f6c [SourceMgr][FileCheck] Obey -color by extending WithColor
(Relands r344930, reverted in r344935, and now hopefully fixed for
Windows.)

While this change specifically targets FileCheck, it affects any tool
using the same SourceMgr facilities.

Previously, -color was documented in FileCheck's -help output, but
-color had no effect.  Now, -color obeys its documentation: it forces
colors to be used in FileCheck diagnostics even when stderr is not a
terminal.

-color is especially helpful when combined with FileCheck's -v, which
can produce a long series of diagnostics that you might wish to pipe
to a pager, such as less -R.  The WithColor extensions here will also
help to clean up color usage in FileCheck's annotated dump of input,
which is proposed in D52999.

Reviewed By: JDevlieghere, zturner

Differential Revision: https://reviews.llvm.org/D53419

llvm-svn: 345202
2018-10-24 21:46:42 +00:00
Aditya Nandakumar ffa9d2e404 Refactor FileCheck to make it usable as an API
https://reviews.llvm.org/D50283
reviewed by bogner

This patch refactors FileCheck's implementation into support so it can
be used from C++ in other places (Unit tests).

llvm-svn: 339192
2018-08-07 21:58:49 +00:00
George Karpenkov 346dfbe2bc [FileCheck] Provide an option for FileCheck to dump original input to stderr on failure
The option can be either set using environment variable (e.g. env
FILECHECK_DUMP_INPUT_ON_FAILURE=1 ninja check-fuzzer) or with a
FileCheck flag.

This can be extremely useful for debugging, cf.
https://groups.google.com/forum/#!topic/llvm-dev/kLrzg8OM_h8 for
discussion.

Differential Revision: https://reviews.llvm.org/D49328

llvm-svn: 337609
2018-07-20 20:21:57 +00:00
Joel E. Denny 6fc21c2522 [FileCheck] Fix search ranges for DAG-NOT-DAG
A DAG-NOT-DAG is a CHECK-DAG group, X, followed by a CHECK-NOT group,
N, followed by a CHECK-DAG group, Y.  Let y be the initial directive
of Y.  This patch makes the following changes to the behavior:

    1. Directives in N can no longer match within part of Y's match
       range just because y happens not to be the earliest match from
       Y.  Specifically, this patch withdraws N's search range end
       from y's match range start to Y's match range start.

    2. y can no longer match within X's match range, where a y match
       produced a reordering complaint, which is thus no longer
       possible.  Specifically, this patch withdraws y's search range
       start from X's permitted range start to X's match range end,
       which was already the search range start for other members of
       Y.

Both of these changes can only increase the number of test passes: #1
constrains the ability of CHECK-NOTs to match, and #2 expands the
ability of CHECK-DAGs to match without complaints.

These changes are based on discussions at:

   <http://lists.llvm.org/pipermail/llvm-dev/2018-May/123550.html>
   <https://reviews.llvm.org/D47106>

which conclude that:

    1. These changes simplify the FileCheck conceptual model.  First,
       it makes search ranges for DAG-NOT-DAG more consistent with
       other cases.  Second, it was confusing that y was treated
       differently from the rest of Y.

    2. These changes add theoretical use cases for DAG-NOT-DAG that
       had no obvious means to be expressed otherwise.  We can justify
       the first half of this assertion with the observation that
       these changes can only increase the number of test passes.

    3. Reordering detection for DAG-NOT-DAG had no obvious real
       benefit.

We don't have evidence from real uses cases to help us debate
conclusions #2 and #3, but #1 at least seems intuitive.

Reviewed By: probinson

Differential Revision: https://reviews.llvm.org/D48986

llvm-svn: 337605
2018-07-20 20:09:56 +00:00
Joel E. Denny dc5ba317b1 [FileCheck] Implement -v and -vv for tracing matches
-v prints all directive pattern matches.

-vv additionally prints info that might be noise to users but that can
be helpful to FileCheck developers.

To maximize code reuse and to make diagnostics more consistent, this
patch also adjusts and extends some of the existing diagnostics.
CHECK-NOT failures now report variables uses.  Many more diagnostics
now report the check prefix and kind of directive.

Reviewed By: probinson

Differential Revision: https://reviews.llvm.org/D47114

llvm-svn: 336967
2018-07-13 03:08:23 +00:00
Joel E. Denny bcf5b441d8 [FileCheck] Don't permit overlapping CHECK-DAG
That is, make CHECK-DAG skip matches that overlap the matches of any
preceding consecutive CHECK-DAG directives.  This change makes
CHECK-DAG more consistent with other directives, and there is evidence
it makes CHECK-DAG more intuitive and less error-prone.  See the RFC
discussion starting at:

  http://lists.llvm.org/pipermail/llvm-dev/2018-May/123010.html

Moreover, this behavior enables CHECK-DAG groups for unordered,
non-unique strings or patterns.  For example, it is useful for
verifying output or logs from a parallel program, such as the OpenMP
runtime.

This patch also implements the command-line option
-allow-deprecated-dag-overlap, which reverts CHECK-DAG to the old
overlapping behavior.  This option should not be used in new tests.
It is meant only for the existing tests that are broken by this change
and that need time to update.

See the following bugzilla issue for tracking of such tests:

  https://bugs.llvm.org/show_bug.cgi?id=37532

Patches to add -allow-deprecated-dag-overlap to those tests will
follow immediately.

Reviewed By: probinson

Differential Revision: https://reviews.llvm.org/D47106

llvm-svn: 336847
2018-07-11 20:27:27 +00:00
Joel E. Denny 614c986175 Revert r336830: [FileCheck] Don't permit overlapping CHECK-DAG
Companion patches are failing to commit, and this patch alone breaks
many tests.

llvm-svn: 336833
2018-07-11 19:03:00 +00:00
Joel E. Denny edf338856c [FileCheck] Don't permit overlapping CHECK-DAG
That is, make CHECK-DAG skip matches that overlap the matches of any
preceding consecutive CHECK-DAG directives.  This change makes
CHECK-DAG more consistent with other directives, and there is evidence
it makes CHECK-DAG more intuitive and less error-prone.  See the RFC
discussion starting at:

  http://lists.llvm.org/pipermail/llvm-dev/2018-May/123010.html

Moreover, this behavior enables CHECK-DAG groups for unordered,
non-unique strings or patterns.  For example, it is useful for
verifying output or logs from a parallel program, such as the OpenMP
runtime.

This patch also implements the command-line option
-allow-deprecated-dag-overlap, which reverts CHECK-DAG to the old
overlapping behavior.  This option should not be used in new tests.
It is meant only for the existing tests that are broken by this change
and that need time to update.

See the following bugzilla issue for tracking of such tests:

  https://bugs.llvm.org/show_bug.cgi?id=37532

Patches to add -allow-deprecated-dag-overlap to those tests will
follow immediately.

Reviewed By: probinson

Differential Revision: https://reviews.llvm.org/D47106

llvm-svn: 336830
2018-07-11 18:42:58 +00:00
James Henderson 5507f6688d [FileCheck] Add CHECK-EMPTY directive for checking for blank lines
Prior to this change, there was no clean way of getting FileCheck to
check that a line is completely empty. The expected way of using
"CHECK: {{^$}}" does not work because the '^' matches the end of the
previous match (this behaviour may be desirable in certain instances).
For the same reason, "CHECK-NEXT: {{^$}}" will fail when the previous
match was at the end of the line, as the pattern will match there.
Using the recommended [[:space:]] to match an explicit new line could
also match a space, and thus is not always desired. Literal '\n'
matches also do not work. A workaround was suggested in the review, but
it is a little clunky.

This change adds a new directive that behaves the same as CHECK-NEXT,
except that it only matches against empty lines (nothing, not even
whitespace, is allowed). As with CHECK-NEXT, it will fail if more than
one newline occurs before the next blank line. Example usage:
; test.txt
foo

bar
; CHECK: foo
; CHECK-EMPTY:
; CHECK-NEXT: bar

Differential Revision: https://reviews.llvm.org/D28896

Reviewed by: probinson

llvm-svn: 335613
2018-06-26 15:15:45 +00:00
Rui Ueyama 197194b6c9 Define InitLLVM to do common initialization all at once.
We have a few functions that virtually all command wants to run on
process startup/shutdown. This patch adds InitLLVM class to do that
all at once, so that we don't need to copy-n-paste boilerplate code
to each llvm command's main() function.

Differential Revision: https://reviews.llvm.org/D45602

llvm-svn: 330046
2018-04-13 18:26:06 +00:00
George Rimar 83e63d96a9 [FileCheck] - Fix possible buffer out of bounds access when parsing --check-prefix.
FileCheck tool crashes when trying to parse --check-prefix argument if there is no any
data after it.

For example test like following would crash if there are no symbols and no EOL mark after `boom`:

# REQUIRES: x86
# RUN: <skipped few lines>
# RUN: llvm-readobj -t %t | FileCheck %s --check-prefix=boom

Patch fixes the issue.

Differential revision: https://reviews.llvm.org/D42057

llvm-svn: 322536
2018-01-16 08:09:24 +00:00
Alexander Richardson 46e1fd6102 Add a -D flag to FileCheck to define variables
Summary:
This makes it very easy to test files that only differ in a constant
value somewhere in the test case.

Reviewers: jlebar, hfinkel, chandlerc, probinson

Reviewed By: probinson

Subscribers: probinson, llvm-commits

Differential Revision: https://reviews.llvm.org/D39629

llvm-svn: 317572
2017-11-07 13:24:44 +00:00
Benjamin Kramer cf60ab313a [FileCheck] Don't scan past the closing CHECK-DAG for CHECK-NOT inside CHECK-DAG
If there's enough data in fron of it the skipped region would just
become arbitrarily large, and we scan for the CHECK-NOT everywhere.

llvm-svn: 304900
2017-06-07 12:06:45 +00:00
James Henderson b5ecceff0c Test commit.
llvm-svn: 297731
2017-03-14 10:51:14 +00:00
Artem Belevich f55e72a5a0 [FileCheck] Added --enable-var-scope option to enable scope for regex variables.
If `--enable-var-scope` is in effect, variables with names that
start with `$` are considered to be global. All other variables are
local. All local variables get undefined at the beginning of each
CHECK-LABEL block. Global variables are not affected by CHECK-LABEL.
This makes it easier to ensure that individual tests are not affected
by variables set in preceding tests.

Differential Revision: https://reviews.llvm.org/D30749

llvm-svn: 297396
2017-03-09 17:59:04 +00:00
Tom de Vries 1714676ae0 [FileCheck] Fix --strict-whitespace --match-full-lines
Make sure FileCheck --strict-whitespace --match-full-lines translates
'CHECK: bla ' into pattern '^ bla $' instead of pattern '^bla$'.

llvm-svn: 290069
2016-12-18 20:45:59 +00:00
Tom de Vries a26bc91456 [FileCheck] Fix comment in ReadCheckFile
The comment in ReadCheckFile claims that both leading and trailing whitespace
are removed, but the associated statement only removes leading whitespace.

llvm-svn: 290061
2016-12-18 09:41:20 +00:00
Chandler Carruth 726774cbf8 [FileCheck] Re-implement the logic to find each check prefix in the
check file to not be unreasonably slow in the face of multiple check
prefixes.

The previous logic would repeatedly scan potentially large portions of
the check file looking for alternative prefixes. In the worst case this
would scan most of the file looking for a rare prefix between every
single occurance of a common prefix. Even if we bounded the scan, this
would do bad things if the order of the prefixes was "unlucky" and the
distant prefix was scanned for first.

None of this is necessary. It is straightforward to build a state
machine that recognizes the first, longest of the set of alternative
prefixes. That is in fact exactly whan a regular expression does.

This patch builds a regular expression once for the set of prefixes and
then uses it to search incrementally for the next prefix. This requires
some threading of state but actually makes the code dramatically
simpler. I've also added a big comment describing the algorithm as it
was not at all obvious to me when I started.

With this patch, several previously pathological test cases in
test/CodeGen/X86 are 5x and more faster. Overall, running all tests
under test/CodeGen/X86 uses 10% less CPU after this, and because all the
slowest tests were hitting this, finishes in 40% less wall time on my
system (going from just over 5.38s to just over 3.23s) on a release
build! This patch substantially improves the time of all 7 X86 tests
that were in the top 20 reported by --time-tests, 5 of them are
completely off the list and the remaining 2 are much lower. (Sadly, the
new tests on the list include 2 new X86 ones that are slow for unrelated
reasons, so the count stays at 4 of the top 20.)

It isn't clear how much this helps debug builds in aggregate in part
because of the noise, but it again makes mane of the slowest x86 tests
significantly faster (10% or more improvement).

llvm-svn: 289382
2016-12-11 12:49:05 +00:00
Chandler Carruth b03c166a6c [FileCheck] Remove a parameter that was simply always set to
a commandline flag and test the flag directly. NFC.

If we ever need this generality it can be added back.

llvm-svn: 289381
2016-12-11 10:22:17 +00:00
Chandler Carruth 4dabac20ad [FileCheck] Clean up doxygen comments throughout. NFC.
llvm-svn: 289380
2016-12-11 10:16:21 +00:00
Chandler Carruth e8f2fb2061 [FileCheck] Run clang-format over this code. NFC.
This fixes one formatting goof I left in my previous commit and *many*
other inconsistencies.

I'm planning to make substantial changes here and so wanted to get to
a clean baseline.

llvm-svn: 289379
2016-12-11 09:54:36 +00:00
Chandler Carruth 20247900d7 Refactor FileCheck some to reduce memory allocation and copying. Also
make some readability improvements.

Both the check file and input file have to be fully buffered to
normalize their whitespace. But previously this would be done in a stack
SmallString and then copied into a heap allocated MemoryBuffer. That
seems pretty wasteful, especially for something like FileCheck where
there are only ever two such entities.

This just rearranges the code so that we can keep the canonicalized
buffers on the stack of the main function, use reasonably large stack
buffers to reduce allocation. A rough estimate seems to show that about
80% of LLVM's .ll and .s files will fit into a 4k buffer, so this should
completely avoid heap allocation for the buffer in those cases. My
system's malloc is fast enough that the allocations don't directly show
up in timings. However, on some very slow test cases, this saves 1% - 2%
by avoiding the copy into the heap allocated buffer.

This also splits out the code which checks the input into a helper much
like the code to build the checks as that made the code much more
readable to me. Nit picks and suggestions welcome here. It has really
exposed a *bunch* of stuff that could be cleaned up though, so I'm
probably going to go and spring clean all of this code as I have more
changes coming to speed things up.

llvm-svn: 289378
2016-12-11 09:50:05 +00:00
Saleem Abdulrasool d1e020f7ee FileCheck: Minor cleanup of the class Pattern
1. Add the "explicit" specifier to the single-argument constructor of Pattern
2. Reorder the fields to remove excessive padding (8 bytes).

Patch by Alexander Shaposhnikov!

llvm-svn: 279832
2016-08-26 16:18:40 +00:00