Commit Graph

89 Commits

Author SHA1 Message Date
Artem Dergachev e8b29c00dd [analyzer] SATestBuild.py: Use driver for analyzing single-file tests.
Don't bother coming up with a -cc1 run-line ourselves.

This, in particular, gets rid of a macOS-specific code path.

llvm-svn: 362009
2019-05-29 18:49:31 +00:00
Serge Guelton d458974c45 Portable Python script across Python version
In Python3, dict.items, dict.keys, dict.values, zip, map and filter no longer return lists, they create generator instead.

The portability patch consists in forcing an extra `list` call if the result is actually used as a list.
`map` are replaced by list comprehension and `filter` by filtered list comprehension.

Differential Revision: https://reviews.llvm.org/D55197

llvm-svn: 349501
2018-12-18 16:04:21 +00:00
Serge Guelton c0ebe773cd Portable Python script across Python version
Using from __future__ import print_function it is possible to have a compatible behavior of `print(...)` across Python version.

Differential Revision: https://reviews.llvm.org/D55213

llvm-svn: 349454
2018-12-18 08:36:33 +00:00
Serge Guelton 1f88dc5aee Portable Python script across Python version
Queue module as been renamed into queue in Python3

Differential Revision: https://reviews.llvm.org/D55202

llvm-svn: 349009
2018-12-13 07:44:19 +00:00
Serge Guelton 3de410848c Portable Python script across Python version
Python2 supports the two following equivalent construct

	raise ExceptionType, exception_value
and
	raise ExceptionType(exception_value)

Only the later is supported by Python3.

Differential Revision: https://reviews.llvm.org/D55195

llvm-svn: 348126
2018-12-03 12:11:21 +00:00
George Karpenkov d3b0846e7a [analyzer] [tests] Hotfix: missing space
llvm-svn: 343643
2018-10-02 22:31:44 +00:00
George Karpenkov ac98683423 [analyzer] [tests] Allow specifying entire -analyze-config on the command line, make sure it's always propagated
Differential Revision: https://reviews.llvm.org/D52801

llvm-svn: 343636
2018-10-02 21:19:23 +00:00
George Karpenkov 47e5493b1d [analyzer] [testing] Pass through an extra argument for specifying extra analyzer options
Differential Revision: https://reviews.llvm.org/D52585

llvm-svn: 343158
2018-09-27 01:10:59 +00:00
George Karpenkov 13d3748e7c [analyzer] [tests] Style fixes for testing harness.
llvm-svn: 338322
2018-07-30 23:01:20 +00:00
George Karpenkov 6e4ddf49d8 [analyzer] [tests] Pass clang executable path to prefix-less executor scripts.
llvm-svn: 336124
2018-07-02 17:10:40 +00:00
George Karpenkov 5c23d6ab3c [analyzer] [tests] Allow the tested project to specify it's own analyzer wrapper
llvm-svn: 336023
2018-06-29 22:05:32 +00:00
George Karpenkov a262cf3146 [analyzer] [tests] Fix 80 column violation in SATestBuild.py
llvm-svn: 336022
2018-06-29 22:05:13 +00:00
Mikhail R. Gadelha afc62b7032 [analyzer] Fix string not being formatted with extra arguments
Signed-off-by: Mikhail Ramalho <mikhail.ramalho@gmail.com>
llvm-svn: 335739
2018-06-27 14:39:41 +00:00
George Karpenkov 30130b78c5 [analyzer] [tests] Include statistics in tests.
llvm-svn: 335685
2018-06-26 23:17:35 +00:00
George Karpenkov 711058165c [analyzer] [testing] Be less verbose by default in integration testing.
llvm-svn: 328752
2018-03-29 01:23:54 +00:00
George Karpenkov b7120c944b [analyzer] [tests] Update CmpRuns to write to stdout correctly in multithreaded environment
llvm-svn: 325070
2018-02-13 23:36:01 +00:00
George Karpenkov 192d9a186f [analyzer] [tests] Fix a typo in analyzer testing script.
Incorrect option instance construction.

llvm-svn: 324946
2018-02-12 22:13:01 +00:00
George Karpenkov fc782a341a [analyzer] [tests] [NFC] Remove a fragile tightly-coupled component emulating parser output
...when we can just use the real parser instead.

Differential Revision: https://reviews.llvm.org/D43098

llvm-svn: 324759
2018-02-09 18:39:47 +00:00
George Karpenkov f37d3a5f84 [analyzer] [tests] Test different projects concurrently
Differential Revision: https://reviews.llvm.org/D43031

llvm-svn: 324652
2018-02-08 21:22:42 +00:00
George Karpenkov 9ed25edc66 [analyzer] Use stable filenames in analyzer testing infrastructure
Makes finding the right file in test results easier.

Differential Revision: https://reviews.llvm.org/D42445

llvm-svn: 323697
2018-01-29 21:45:07 +00:00
George Karpenkov 3c128cb1e6 [analyzer] [tests] Remove empty folders in reference results, do not store diffs.txt
Storing diffs.txt is now redundant, as we simply dump the CmpRuns output
to stdout (it is saved in CI and tends to be small).
Not generating those files enables us to remove empty folders, which
confuse git, as it would not add them with reference results.

llvm-svn: 316948
2017-10-30 19:40:33 +00:00
George Karpenkov 0a6dba792f [Analyzer] [Tests] Fixing typo from the previous commit.
Can not open a non-existent file with r+.

llvm-svn: 316808
2017-10-27 22:52:36 +00:00
George Karpenkov e58044d432 [Analyzer] [Tests] Dump the output of scan-build to stdout on failure.
Eliminates extra lookup step during debugging.

llvm-svn: 316806
2017-10-27 22:39:54 +00:00
George Karpenkov ff555ce798 [Analyzer] [Tests] Write analyzers crashes to stdout, and not to a separate file
With this change it would be sufficient to look at CI console to see the
failure.

llvm-svn: 316687
2017-10-26 19:00:22 +00:00
George Karpenkov 65839bd429 [Analyzer] [Tests] Consistently use exit codes. Use code=42 to signify different results
llvm-svn: 316632
2017-10-26 01:13:22 +00:00
George Karpenkov a932c8745c [Analyzer] [Tests] Do not discard output from CmpRuns.py when running integration tests
Contrary to the deleted comment, in most cases CmpRuns.py produces a
fairly small amount of output, which is useful to see straight away to
see what has changed when executing the integration tests.

llvm-svn: 316618
2017-10-25 21:49:46 +00:00
George Karpenkov bf92c44616 [Analyzer] [Tests] Minor refactor of testing infrastructure:
Move utilities functions into a separate file to make comprehension
easier.

llvm-svn: 316535
2017-10-24 23:52:48 +00:00
George Karpenkov 318cd1f263 [Analyzer] [Tests] Remove temporary fields from generated reference results.
Pointer to HTML diagnostics is removed (as it is not stored) as well as
the version (as it would be available from the commit message).

llvm-svn: 316534
2017-10-24 23:52:46 +00:00
George Karpenkov 43f683cb33 [Analyzer] Fix bug in testing scripts, which always marked result as failure.
llvm-svn: 316522
2017-10-24 22:24:13 +00:00
George Karpenkov 7d36cd7223 [Analyzer] Remove dead code from testing scripts
Differential Revision: https://reviews.llvm.org/D38488

llvm-svn: 315489
2017-10-11 18:42:39 +00:00
George Karpenkov 1b51cbddf4 [Analyzer Tests] Run static analyzer integration tests until the end,
Do not stop at the first failure.

Differential Revision: https://reviews.llvm.org/D38589

llvm-svn: 314992
2017-10-05 17:32:06 +00:00
George Karpenkov a807660918 [Analyzer] Make testing scripts flake8 compliant
Differential Review: https://reviews.llvm.org/D38213

llvm-svn: 314692
2017-10-02 17:59:12 +00:00
George Karpenkov af76b4a553 [Analyzer] Add nullability to the list of tested checkers in SATestBuild
Differential Revision: https://reviews.llvm.org/D38162

llvm-svn: 314572
2017-09-30 00:05:24 +00:00
George Karpenkov 439a95bde7 [Analyzer] Fix minor errors in python scripts.
llvm-svn: 314042
2017-09-22 22:58:46 +00:00
George Karpenkov 3abfc3b0d7 [Analyzer] Refactor analyzer testing scripts.
- Exporting needed function for future reuse.
 - Idiomatic python: using with `file as f` instead of `try/finally`.
 - Fixing some indentation issues.
 - No need to reinvent python `multiprocessing.getCPUCount()`
 - Removing a function parameter which is always the same under all invocations.
 - Adding some docstrings.

Differential Revision: https://reviews.llvm.org/D38156

llvm-svn: 313949
2017-09-22 01:41:16 +00:00
George Karpenkov be6c329515 [Analyzer] Use CC environment variable to select analyzer path in SATestBuild.
This change is required to easily test the given checkout of the analyzer,
rather than the one bundled with a system compiler.

Differential Revision: https://reviews.llvm.org/D38155

llvm-svn: 313927
2017-09-21 22:12:49 +00:00
Devin Coughlin 0dfc6f0809 [analyzer] SATestBuild.py: Treat '#' as comment in projectMap.csv
Treat lines in projectMap.csv that start with '#' as comments. This enables a
workflow where projects can be temporarily disabled with a comment describing
when they should be turned back on.

Differential Revision: https://reviews.llvm.org/D24709

llvm-svn: 281880
2016-09-19 01:36:40 +00:00
Devin Coughlin 9ea8033d1b [analyzer] SATestBuild.py: Remove html and log when producing reference results.
The html reports are huge -- every issue in a given file results in a separate
copy of the source code, in HTML form, for the file. This gets very large
quickly and it doesn't make sense to check this into a reference repository.
Also remove the log when generating reference results because it can leak
absolute path names. We still keep both the html and the log around when
producing non-reference results.

llvm-svn: 258594
2016-01-23 01:09:07 +00:00
Devin Coughlin 86f61a9a62 [analyzer] SATestBuild.py: Handle spaces in path passed to --use-analyzer.
I missed this one in r258493.

llvm-svn: 258517
2016-01-22 18:45:22 +00:00
Devin Coughlin ab95cd2794 [analyzer] Update SATestBuild.py to handle spaces in paths.
The Jenkins workspace on the new Green Dragon builder for the static analyzer has spaces
in its path.

llvm-svn: 258493
2016-01-22 07:08:06 +00:00
Devin Coughlin 2cb767d32b [analyzer] Update SATestBuild.py to enable a 'download and patch' model for projects.
Currently the SATestBuild.py and SATestAdd.py buildbot scripts expect project
sources to be checked into the project repository. This commit changes these
scripts to additionally support a model where project sources are downloaded
rather than checked into the repository. Sometimes projects may need to be
modified (for example, to support a newer versions of clang), so the updated scripts
also allow for an optional patch file that will be applied to the downloaded
project source before analysis.

To support this workflow, this commit changes the expected layout of
a project in the repository. The project-specific helper scripts will stay
in the root of each project directory, but the benchmark source itself (if
checked into the repo) should now be stored in a subdirectory named
'CachedSource':

  project_name/
    cleanup_run_static_analyzer.sh [optional]
    run_static_analyzer.cmd [required]
    download_project.sh [optional]
    CachedSource/ [optional]
    changes_for_analyzer.patch [optional]

If the 'CachedSource' source directory is not present, the download script will
be executed. This script should download the project source into 'CachedSource'.
Then, if 'changes_for_analyzer.patch' is present its changes will
be applied to a copy of 'CachedSource' before analysis.

Differential Revision: http://reviews.llvm.org/D14345

llvm-svn: 252410
2015-11-07 18:27:35 +00:00
Devin Coughlin f3695c87cd [analyzer] SATestBuild.py: Move additional checkers logic so SATestAdd.py can use it as well.
Move the logic looking for additional checkers in the SA_ADDITIONAL_CHECKERS
environmental variable from SATestBuild's main() to runScanBuild(). This allows
SATestAdd.py to use the variable as well. Without it, we won't include
additional checkers when building reference results for the build bot.

Differential Revision: http://reviews.llvm.org/D12891

llvm-svn: 247767
2015-09-16 01:52:32 +00:00
Devin Coughlin bace032eb8 [analyzer] Update SATestBuild.py to set -isysroot for preprocessed files
Update the static analyzer buildbot script to set -isysroot to the OS X SDK path
when analyzing preprocessed files on OS X.

Differential Revision: http://reviews.llvm.org/D12769

llvm-svn: 247617
2015-09-14 21:22:24 +00:00
Ted Kremenek 3a0678e33c [analyzer] Apply whitespace cleanups by Honggyu Kim.
llvm-svn: 246978
2015-09-08 03:50:52 +00:00
Gabor Horvath da32a86341 [Static Analyzer] Made it easier to test new checkers using the test suite.
llvm-svn: 245637
2015-08-20 22:59:49 +00:00
Gabor Horvath c3177f2ed4 [Static Analyzer] Make the paths relative from the project root when generating reference results in the test suite.
llvm-svn: 241706
2015-07-08 18:39:31 +00:00
Gabor Horvath 2ed9456015 [Static Analyzer] No longer include the full path of the source file in the single file tests' bug identifiers.
llvm-svn: 241292
2015-07-02 19:20:46 +00:00
Gabor Horvath 93fde94d81 [Static Analyzer] Minor improvements to SATest.
Differential Revision: http://reviews.llvm.org/D10812

llvm-svn: 241073
2015-06-30 15:31:17 +00:00
Anna Zaks 8a020310b8 [analyzer] Rename NewDeleteLeaks checker in the test script.
Fixup to r220289.

llvm-svn: 220976
2014-10-31 17:40:14 +00:00
Alp Toker d473363876 Correct hyphenations in comments and assert messages
This patch tries to avoid unrelated changes other than fixing a few
hyphen-related ambiguities in nearby lines.

llvm-svn: 196466
2013-12-05 04:47:09 +00:00