Before the patch `SATest compare`, produced quite obscure results
when something about the diagnostic have changed (i.e. its description
or the name of the corresponding checker) because it was simply two
lists of warnings, ADDED and REMOVED. It was up to the developer
to match those warnings, understand that they are essentially the
same, and figure out what caused the difference.
This patch introduces another category of results: MODIFIED.
It tries to match new warnings against the old ones and prints out
clues on what is different between two builds.
Differential Revision: https://reviews.llvm.org/D85311
Summary:
Not all projects in the project map file might have newer results
for updating, we should handle this situation gracefully.
Additionally, not every user of the test system would want storing
reference results in git. For this reason, git functionality is now
optional.
Differential Revision: https://reviews.llvm.org/D84303
Summary:
This commit includes a couple of changes:
* Benchmark selected projects by analyzing them multiple times
* Compare two benchmarking results and visualizing them on one chart
* Organize project build logging, so we can use the same code
in benchmarks
Differential Revision: https://reviews.llvm.org/D83539
Summary:
It generalizes the way the output looks across any -jN.
Additionally it solves the buffering problems.
Differential Revision: https://reviews.llvm.org/D81601
Summary:
Another possible difference between various users of the
testing system might be a change in dependencies installed on the
container. This commit tries to prevent any problem related to
different versions of the libraries/headers used and fixes them to
currently installed versions.
Differential Revision: https://reviews.llvm.org/D81600
Summary:
Docker on its own has a pretty convenient way to run shell.
This method, however, requires target container to be currently running,
which is not a usual scenario for the test system. For this purpose,
it is better to have a simple way to run the container, shell it, and
clean up at the end of it all. New option `--shell` does exactly this.
Differential Revision: https://reviews.llvm.org/D81598
Summary:
If the user has only python2 installed and wants to use
the dockerized testing system, it is now totally OK.
Differential Revision: https://reviews.llvm.org/D81595
Summary:
Forward results of every command executed in docker. The actual commands
and their error codes are more informative than python stacktraces.
Differential Revision: https://reviews.llvm.org/D81593
Summary:
It provides a simpler interface for testing within docker.
This way the user is not required to no how to use `docker run` and
its options.
Differential Revision: https://reviews.llvm.org/D81572
Summary:
Static analysis is very sensitive to environment.
OS and libraries installed can affect the results. This fact makes
it extremely hard to have a regression testing system that will
produce stable results.
For this very reason, this commit introduces a new dockerized testing
environment, so that every analyzer developer can check their changes
against previous analysis results.
Differential Revision: https://reviews.llvm.org/D81571
Summary:
Handle `\l` separately because a string literal can be in code like "string\\literal" with the `\l` inside. Also on Windows macros __FILE__ produces specific delimiters `\` and a directory or file may starts with the letter `l`.
Fix:
Use regex for replacing all `\l` (like `,\l`, `}\l`, `[\l`) except `\\l`, because a literal as a rule contains multiple `\` before `\l`.
Differential Revision: https://reviews.llvm.org/D82092
Summary:
It makes it much harder to use from other modules when one of the
parameters is an argparse Namespace. This commit makes it easier
to use CmpRuns programmatically.
Differential Revision: https://reviews.llvm.org/D81566
Summary:
JSON format is a bit more verbose and easier to reason about
and extend. For this reason, before extending SATestBuild
functionality it is better to refactor the part of how we
configure the whole system.
Differential Revision: https://reviews.llvm.org/D81563
Summary:
Tasks can crash with many different exceptions including SystemExit.
Bare except still causes a warning, so let's use BaseException instead.
Differential Revision: https://reviews.llvm.org/D80443
Summary:
SATest scripts should be more python-style than they are now.
This includes better architecture, type annotations, naming
convesions, and up-to-date language features. This commit starts
with two scripts SATestBuild and SATestAdd.
Differential Revision: https://reviews.llvm.org/D80423
Summary:
Fix read/write in binary format, which crashes Python 3.
Additionally, clean up redundant (as for Python 3) code and
fix a handful of flake8 warnings.
Differential Revision: https://reviews.llvm.org/D79932
ExplodedGraph nodes will now have a numeric identifier stored in them
which will keep track of the order in which the nodes were created
and it will be fully deterministic both accross runs and across machines.
This is extremely useful for debugging as it allows reliably setting
conditional breakpoints by node IDs.
llvm-svn: 375186
Because cast expressions have their own hierarchy, it's extremely useful
to have some information about what kind of casts are we dealing with.
llvm-svn: 375185
They're useful when trying to understand what's going on
inside your LazyCompoundValues.
Differential Revision: https://reviews.llvm.org/D65427
llvm-svn: 368769
When -trim-egraph is unavailable (say, when you're debugging a crash on
a real-world code that takes too long to reduce), it makes sense to view
the untrimmed graph up to the crashing node's predecessor, then dump the ID
(or a pointer) of the node in the attached debugger, and then trim
the dumped graph in order to keep only paths from the root to the node.
The newly added --to flag does exactly that:
$ exploded-graph-rewriter.py ExprEngine.dot --to 0x12229acd0
Multiple nodes can be specified. Stable IDs of nodes can be used
instead of pointers.
Differential Revision: https://reviews.llvm.org/D65345
llvm-svn: 368768
Explorers aren't the right abstraction. For the purposes of displaying svg files
we don't care in which order do we explore the nodes. We may care about this for
other analyses, but we're not there yet.
The function of cutting out chunks of the graph is performed poorly by
the explorers, because querying predecessors/successors on the explored nodes
yields original successors/predecessors even if they aren't being explored.
Introduce a new entity, "trimmers", that do one thing but to it right: cut out
chunks of the graph. Trimmers mutate the graph, so stale edges aren't even
visible to their consumers in the pipeline. Additionally, trimmers are
intrinsically composable: multiple trimmers can be applied to the graph
sequentially.
Refactor the single-path explorer into the single-path trimmer.
Rename the test file for consistency.
Differential Revision: https://reviews.llvm.org/D65344
llvm-svn: 368767
Change the default behavior: the tool no longer dumps the rewritten .dot file
to stdout, but instead it automatically converts it into an .html file
(which essentially wraps an .svg file) and immediately opens it with
the default web browser.
This means that the tool should now be fairly easy to use:
$ exploded-graph-rewriter.py /tmp/ExprEngine.dot
The benefits of wrapping the .svg file into an .html file are:
- It'll open in a web browser, which is the intended behavior.
An .svg file would be open with an image viewer/editor instead.
- It avoids the white background around the otherwise dark svg area
in dark mode.
The feature can be turned off by passing a flag '--rewrite-only'.
The LIT substitution is updated to enforce the old mode because
we don't want web browsers opening on our buildbots.
Differential Revision: https://reviews.llvm.org/D65250
llvm-svn: 368766
- Correctly display macro expansion and spelling locations.
- Use the same procedure to display location context call site locations.
- Display statement IDs for program points.
llvm-svn: 365861
Valeriy Savchenko