Allow users to use a non-system version of perl, python and awk, which is useful
in certain package managers.
Reviewed By: JDevlieghere, MaskRay
Differential Revision: https://reviews.llvm.org/D95119
When newer build has duplicate issues the script tried to
remove it from the list more than once. The new approach
changes the way we filter out matching issues.
Differential Revision: https://reviews.llvm.org/D96611
Summary:
Whith the number of projects growing, it is important to be able to
filter them in a more convenient way than by names. It is especially
important for benchmarks, when it is not viable to analyze big
projects 20 or 50 times in a row.
Because of this reason, this commit adds a notion of sizes and a
filtering interface that puts a limit on a maximum size of the project
to analyze or benchmark.
Sizes assigned to the projects in this commit, do not directly
correspond to the number of lines or files in the project. The key
factor that is important for the developers of the analyzer is the
time it takes to analyze the project. And for this very reason,
"size" basically helps to cluster projects based on their analysis
time.
Differential Revision: https://reviews.llvm.org/D83942
Before the patch `SATest compare`, produced quite obscure results
when something about the diagnostic have changed (i.e. its description
or the name of the corresponding checker) because it was simply two
lists of warnings, ADDED and REMOVED. It was up to the developer
to match those warnings, understand that they are essentially the
same, and figure out what caused the difference.
This patch introduces another category of results: MODIFIED.
It tries to match new warnings against the old ones and prints out
clues on what is different between two builds.
Differential Revision: https://reviews.llvm.org/D85311
Summary:
Not all projects in the project map file might have newer results
for updating, we should handle this situation gracefully.
Additionally, not every user of the test system would want storing
reference results in git. For this reason, git functionality is now
optional.
Differential Revision: https://reviews.llvm.org/D84303
Summary:
This commit includes a couple of changes:
* Benchmark selected projects by analyzing them multiple times
* Compare two benchmarking results and visualizing them on one chart
* Organize project build logging, so we can use the same code
in benchmarks
Differential Revision: https://reviews.llvm.org/D83539
Summary:
It generalizes the way the output looks across any -jN.
Additionally it solves the buffering problems.
Differential Revision: https://reviews.llvm.org/D81601
Summary:
Another possible difference between various users of the
testing system might be a change in dependencies installed on the
container. This commit tries to prevent any problem related to
different versions of the libraries/headers used and fixes them to
currently installed versions.
Differential Revision: https://reviews.llvm.org/D81600
Summary:
Docker on its own has a pretty convenient way to run shell.
This method, however, requires target container to be currently running,
which is not a usual scenario for the test system. For this purpose,
it is better to have a simple way to run the container, shell it, and
clean up at the end of it all. New option `--shell` does exactly this.
Differential Revision: https://reviews.llvm.org/D81598
Summary:
If the user has only python2 installed and wants to use
the dockerized testing system, it is now totally OK.
Differential Revision: https://reviews.llvm.org/D81595
Summary:
Forward results of every command executed in docker. The actual commands
and their error codes are more informative than python stacktraces.
Differential Revision: https://reviews.llvm.org/D81593
Summary:
It provides a simpler interface for testing within docker.
This way the user is not required to no how to use `docker run` and
its options.
Differential Revision: https://reviews.llvm.org/D81572
Summary:
Static analysis is very sensitive to environment.
OS and libraries installed can affect the results. This fact makes
it extremely hard to have a regression testing system that will
produce stable results.
For this very reason, this commit introduces a new dockerized testing
environment, so that every analyzer developer can check their changes
against previous analysis results.
Differential Revision: https://reviews.llvm.org/D81571
Summary:
Handle `\l` separately because a string literal can be in code like "string\\literal" with the `\l` inside. Also on Windows macros __FILE__ produces specific delimiters `\` and a directory or file may starts with the letter `l`.
Fix:
Use regex for replacing all `\l` (like `,\l`, `}\l`, `[\l`) except `\\l`, because a literal as a rule contains multiple `\` before `\l`.
Differential Revision: https://reviews.llvm.org/D82092
Summary:
It makes it much harder to use from other modules when one of the
parameters is an argparse Namespace. This commit makes it easier
to use CmpRuns programmatically.
Differential Revision: https://reviews.llvm.org/D81566
Summary:
JSON format is a bit more verbose and easier to reason about
and extend. For this reason, before extending SATestBuild
functionality it is better to refactor the part of how we
configure the whole system.
Differential Revision: https://reviews.llvm.org/D81563
Summary:
Tasks can crash with many different exceptions including SystemExit.
Bare except still causes a warning, so let's use BaseException instead.
Differential Revision: https://reviews.llvm.org/D80443
Summary:
SATest scripts should be more python-style than they are now.
This includes better architecture, type annotations, naming
convesions, and up-to-date language features. This commit starts
with two scripts SATestBuild and SATestAdd.
Differential Revision: https://reviews.llvm.org/D80423
Summary:
Fix read/write in binary format, which crashes Python 3.
Additionally, clean up redundant (as for Python 3) code and
fix a handful of flake8 warnings.
Differential Revision: https://reviews.llvm.org/D79932
ExplodedGraph nodes will now have a numeric identifier stored in them
which will keep track of the order in which the nodes were created
and it will be fully deterministic both accross runs and across machines.
This is extremely useful for debugging as it allows reliably setting
conditional breakpoints by node IDs.
llvm-svn: 375186
Because cast expressions have their own hierarchy, it's extremely useful
to have some information about what kind of casts are we dealing with.
llvm-svn: 375185
They're useful when trying to understand what's going on
inside your LazyCompoundValues.
Differential Revision: https://reviews.llvm.org/D65427
llvm-svn: 368769
When -trim-egraph is unavailable (say, when you're debugging a crash on
a real-world code that takes too long to reduce), it makes sense to view
the untrimmed graph up to the crashing node's predecessor, then dump the ID
(or a pointer) of the node in the attached debugger, and then trim
the dumped graph in order to keep only paths from the root to the node.
The newly added --to flag does exactly that:
$ exploded-graph-rewriter.py ExprEngine.dot --to 0x12229acd0
Multiple nodes can be specified. Stable IDs of nodes can be used
instead of pointers.
Differential Revision: https://reviews.llvm.org/D65345
llvm-svn: 368768
Harmen Stoppels