Commit Graph

110 Commits

Author SHA1 Message Date
George Karpenkov 7d36cd7223 [Analyzer] Remove dead code from testing scripts
Differential Revision: https://reviews.llvm.org/D38488

llvm-svn: 315489
2017-10-11 18:42:39 +00:00
George Karpenkov 1b51cbddf4 [Analyzer Tests] Run static analyzer integration tests until the end,
Do not stop at the first failure.

Differential Revision: https://reviews.llvm.org/D38589

llvm-svn: 314992
2017-10-05 17:32:06 +00:00
George Karpenkov a807660918 [Analyzer] Make testing scripts flake8 compliant
Differential Review: https://reviews.llvm.org/D38213

llvm-svn: 314692
2017-10-02 17:59:12 +00:00
George Karpenkov af76b4a553 [Analyzer] Add nullability to the list of tested checkers in SATestBuild
Differential Revision: https://reviews.llvm.org/D38162

llvm-svn: 314572
2017-09-30 00:05:24 +00:00
George Karpenkov 439a95bde7 [Analyzer] Fix minor errors in python scripts.
llvm-svn: 314042
2017-09-22 22:58:46 +00:00
George Karpenkov 3abfc3b0d7 [Analyzer] Refactor analyzer testing scripts.
- Exporting needed function for future reuse.
 - Idiomatic python: using with `file as f` instead of `try/finally`.
 - Fixing some indentation issues.
 - No need to reinvent python `multiprocessing.getCPUCount()`
 - Removing a function parameter which is always the same under all invocations.
 - Adding some docstrings.

Differential Revision: https://reviews.llvm.org/D38156

llvm-svn: 313949
2017-09-22 01:41:16 +00:00
George Karpenkov be6c329515 [Analyzer] Use CC environment variable to select analyzer path in SATestBuild.
This change is required to easily test the given checkout of the analyzer,
rather than the one bundled with a system compiler.

Differential Revision: https://reviews.llvm.org/D38155

llvm-svn: 313927
2017-09-21 22:12:49 +00:00
Devin Coughlin 0dfc6f0809 [analyzer] SATestBuild.py: Treat '#' as comment in projectMap.csv
Treat lines in projectMap.csv that start with '#' as comments. This enables a
workflow where projects can be temporarily disabled with a comment describing
when they should be turned back on.

Differential Revision: https://reviews.llvm.org/D24709

llvm-svn: 281880
2016-09-19 01:36:40 +00:00
Devin Coughlin 9ea8033d1b [analyzer] SATestBuild.py: Remove html and log when producing reference results.
The html reports are huge -- every issue in a given file results in a separate
copy of the source code, in HTML form, for the file. This gets very large
quickly and it doesn't make sense to check this into a reference repository.
Also remove the log when generating reference results because it can leak
absolute path names. We still keep both the html and the log around when
producing non-reference results.

llvm-svn: 258594
2016-01-23 01:09:07 +00:00
Devin Coughlin 86f61a9a62 [analyzer] SATestBuild.py: Handle spaces in path passed to --use-analyzer.
I missed this one in r258493.

llvm-svn: 258517
2016-01-22 18:45:22 +00:00
Devin Coughlin ab95cd2794 [analyzer] Update SATestBuild.py to handle spaces in paths.
The Jenkins workspace on the new Green Dragon builder for the static analyzer has spaces
in its path.

llvm-svn: 258493
2016-01-22 07:08:06 +00:00
Devin Coughlin 2cb767d32b [analyzer] Update SATestBuild.py to enable a 'download and patch' model for projects.
Currently the SATestBuild.py and SATestAdd.py buildbot scripts expect project
sources to be checked into the project repository. This commit changes these
scripts to additionally support a model where project sources are downloaded
rather than checked into the repository. Sometimes projects may need to be
modified (for example, to support a newer versions of clang), so the updated scripts
also allow for an optional patch file that will be applied to the downloaded
project source before analysis.

To support this workflow, this commit changes the expected layout of
a project in the repository. The project-specific helper scripts will stay
in the root of each project directory, but the benchmark source itself (if
checked into the repo) should now be stored in a subdirectory named
'CachedSource':

  project_name/
    cleanup_run_static_analyzer.sh [optional]
    run_static_analyzer.cmd [required]
    download_project.sh [optional]
    CachedSource/ [optional]
    changes_for_analyzer.patch [optional]

If the 'CachedSource' source directory is not present, the download script will
be executed. This script should download the project source into 'CachedSource'.
Then, if 'changes_for_analyzer.patch' is present its changes will
be applied to a copy of 'CachedSource' before analysis.

Differential Revision: http://reviews.llvm.org/D14345

llvm-svn: 252410
2015-11-07 18:27:35 +00:00
Devin Coughlin f3695c87cd [analyzer] SATestBuild.py: Move additional checkers logic so SATestAdd.py can use it as well.
Move the logic looking for additional checkers in the SA_ADDITIONAL_CHECKERS
environmental variable from SATestBuild's main() to runScanBuild(). This allows
SATestAdd.py to use the variable as well. Without it, we won't include
additional checkers when building reference results for the build bot.

Differential Revision: http://reviews.llvm.org/D12891

llvm-svn: 247767
2015-09-16 01:52:32 +00:00
Devin Coughlin bace032eb8 [analyzer] Update SATestBuild.py to set -isysroot for preprocessed files
Update the static analyzer buildbot script to set -isysroot to the OS X SDK path
when analyzing preprocessed files on OS X.

Differential Revision: http://reviews.llvm.org/D12769

llvm-svn: 247617
2015-09-14 21:22:24 +00:00
Ted Kremenek 3a0678e33c [analyzer] Apply whitespace cleanups by Honggyu Kim.
llvm-svn: 246978
2015-09-08 03:50:52 +00:00
Gabor Horvath da32a86341 [Static Analyzer] Made it easier to test new checkers using the test suite.
llvm-svn: 245637
2015-08-20 22:59:49 +00:00
Gabor Horvath c3177f2ed4 [Static Analyzer] Make the paths relative from the project root when generating reference results in the test suite.
llvm-svn: 241706
2015-07-08 18:39:31 +00:00
Gabor Horvath 2ed9456015 [Static Analyzer] No longer include the full path of the source file in the single file tests' bug identifiers.
llvm-svn: 241292
2015-07-02 19:20:46 +00:00
Gabor Horvath 93fde94d81 [Static Analyzer] Minor improvements to SATest.
Differential Revision: http://reviews.llvm.org/D10812

llvm-svn: 241073
2015-06-30 15:31:17 +00:00
Anna Zaks 8a020310b8 [analyzer] Rename NewDeleteLeaks checker in the test script.
Fixup to r220289.

llvm-svn: 220976
2014-10-31 17:40:14 +00:00
Alp Toker d473363876 Correct hyphenations in comments and assert messages
This patch tries to avoid unrelated changes other than fixing a few
hyphen-related ambiguities in nearby lines.

llvm-svn: 196466
2013-12-05 04:47:09 +00:00
Jordan Rose 7bd9186cdd [analyzer] SATestBuild: strip trailing newlines from command file.
When running a make-based command, SATestBuild tries to append a -jN flag
with an appropriate N to run the build in parallel. However, it failed
to take into account that each line read includes a trailing newline
(unless it is the last line of a file without a trailing newline), which
resulted in the "-jN" appearing on a line on its own.

llvm-svn: 190164
2013-09-06 16:12:41 +00:00
Jordan Rose c7b992edcf [analyzer] SATestBuild: Don't require reference results to have logs.
The Logs directory isn't used for testing, so it's filtered out ahead of
time. However, there's then no reason to include it in version control at
all. Don't error if it's not present.

llvm-svn: 183689
2013-06-10 19:34:30 +00:00
Anna Zaks 7b4f8a4181 [analyzer] Always use ccc-analyzer when running scan-build on buildbot
llvm-svn: 182982
2013-05-31 02:31:09 +00:00
Jordan Rose 10ad081fc6 [analyzer] Re-enable cplusplus.NewDelete (but not NewDeleteLeaks).
As mentioned in the previous commit message, the use-after-free and
double-free warnings for 'delete' are worth enabling even while the
leak warnings still have false positives.

llvm-svn: 178891
2013-04-05 17:55:07 +00:00
Anton Yartsev 01acbcebbb [analyzer] Moving cplusplus.NewDelete to alpha.* for now.
llvm-svn: 178529
2013-04-02 05:59:24 +00:00
Anton Yartsev 13df03624b [analyzer] Adds cplusplus.NewDelete checker that check for memory leaks, double free, and use-after-free problems of memory managed by new/delete.
llvm-svn: 177849
2013-03-25 01:35:45 +00:00
Jordan Rose b18179da33 scan-build: Add a --keep-empty option for better testing.
SATestBuild expects to compare output directories for each invocation of
scan-build that it runs, but scan-build clears out empty directories by
default. We were coincidentally not getting that behavior until r173294.

llvm-svn: 173383
2013-01-24 23:07:59 +00:00
Jordan Rose 64e4cf001b [analyzer] SATestBuild.py: allow make builds to disable parallelization
Before, SATestBuild unilaterally added '-j<n>' to every project built with
'make'. Now, we check and see if there's a -j option already specified, which
allows a project to explicitly be marked '-j1'.

llvm-svn: 168603
2012-11-26 19:59:57 +00:00
Jordan Rose 8832924acb [analyzer] SATestBuild.py: fix broken string concatenation
This has been broken for a while, but the branch was never being taken.
(We were trying to do 'str + floatVal'; now we do 'str % intVal' and use
the '%d' format.)

llvm-svn: 168174
2012-11-16 17:41:21 +00:00
Jordan Rose f477505d03 [analyzer] SATestBuild.py: correctly add -j option for projects using make.
llvm-svn: 168099
2012-11-15 22:03:53 +00:00
Anna Zaks 69e2bf9e0c [analyzer] Test SimpleStream on the buildbot.
llvm-svn: 167317
2012-11-02 21:30:07 +00:00
Anna Zaks a2f970baa6 [analyzer] testing: add a build mode to allow C++11 testing.
llvm-svn: 163357
2012-09-06 23:30:27 +00:00
Jordan Rose 9858b12381 Revert "[analyzer] Disable diagnostic checking on the buildbot."
This reverts commit befc0f648c83f65e1f3567f2058c7c18b4bc3ad8 now that
plist-html diagnostics are working again.

llvm-svn: 162966
2012-08-31 00:36:30 +00:00
Anna Zaks 5e2f0b2151 [analyzer] Disable diagnostic checking on the buildbot.
Due to recent diagnostic changes, we may generate malformed diagnostics.

llvm-svn: 162940
2012-08-30 21:04:09 +00:00
Anna Zaks 07a821fb17 [analyzer] Fixup 162863.
Thanks Jordan.

llvm-svn: 162875
2012-08-29 23:23:39 +00:00
Anna Zaks f329823dc0 [analyzer]Fix typo, unbreak static analyzer BuildBot (at least
partially).

(Addresses a regression from r162790.)

llvm-svn: 162864
2012-08-29 21:22:40 +00:00
Ted Kremenek f9a539d858 Automatically use more cores when the test build uses 'make'.
llvm-svn: 162790
2012-08-28 20:40:04 +00:00
Ted Kremenek 42c14428d2 Consolidate helper methods and configuration logic. No functionality change.
llvm-svn: 162789
2012-08-28 20:40:02 +00:00
Ted Kremenek 6cb2080e5e Pass --use-analyzer to scan-build when running within the test harness.
llvm-svn: 162783
2012-08-28 20:20:52 +00:00
Ted Kremenek 4220144acb experimental.security.taint is now alpha.security.taint
llvm-svn: 162775
2012-08-28 18:45:22 +00:00
Jordan Rose c2a3223da9 SATestBuild should not ask for non-existent checkers.
- "cocoa" was moved to "osx.cocoa" a long time ago.
- "cplusplus" would be a valid package except we don't have any C++ checkers.

llvm-svn: 160821
2012-07-26 22:31:59 +00:00
Anna Zaks 9b7d7140ef [analyzer] Make CmpRuns external-user friendly.
CmpRuns can be used for static analyzer bug report comparison. However,
we want to make sure external users do not rely on the way bugs are
represented (plist files). Make sure that we have a user
friendly/documented API for CmpRuns script.

llvm-svn: 160314
2012-07-16 20:21:42 +00:00
Jordan Rose dc191a184f [analyzer] SATestBuild should execute SVN updates even if Verbose is off.
Also, re-use glob results when looking for failure logs.

llvm-svn: 157817
2012-06-01 16:24:43 +00:00
Jordan Rose 01ac57244c [analyzer] Fix SATestAdd to work with SATestBuild.
Also, eliminate global 'IsReferenceBuild' in SATestBuild. It doesn't get
passed around that much.

llvm-svn: 157816
2012-06-01 16:24:38 +00:00
Anna Zaks 6ecb6a6b7f [analyzer] Testing: These checkers are not experimental anymore.
llvm-svn: 150989
2012-02-20 21:10:40 +00:00
Anna Zaks a432feaf26 [analyzer] Add the Malloc checker to the list of tested checkers.
llvm-svn: 150535
2012-02-15 00:19:07 +00:00
Anna Zaks 4c1ef976a8 [analyzer] Testing: add automated reference results reset.
llvm-svn: 149682
2012-02-03 06:35:23 +00:00
Anna Zaks 97981bdeb0 [analyzer] Testing: add 2 new checkers to the buildbot script.
llvm-svn: 149514
2012-02-01 16:46:57 +00:00
Anna Zaks b3d896d333 [analyzer] Add the HTML file to the SATest diagnostic diff.
(Uses the functionality which has been in CmpRuns long before.)

llvm-svn: 148868
2012-01-24 21:57:35 +00:00
Anna Zaks a08c6b7efc [analyzer] Unbreak the static analyzer bot. Regression from r148558.
When we build a single source file, we call clang directly (not though scan-build).

llvm-svn: 148612
2012-01-21 01:11:35 +00:00
Anna Zaks 7a75cf32a2 [analyzer] Tweak the buildbot script to include the experimental taint checker and correct the checker enabling setting.
llvm-svn: 148558
2012-01-20 19:10:59 +00:00
Anna Zaks de1f7f8bfb [analyzer] Testing infrastructure: Flush output after every print
statement.

llvm-svn: 147862
2012-01-10 18:10:25 +00:00
Anna Zaks 5acd9603da [analyzer] Correct the error message in the testing infrastructure.
llvm-svn: 147568
2012-01-04 23:53:50 +00:00
Anna Zaks 26573c5585 [analyzer] Testing: test with all the non-experimental checker packages.
llvm-svn: 144142
2011-11-08 22:41:25 +00:00
Anna Zaks 767d356f87 [analyzer] Testing: Display the number of reported differences in the main log.
llvm-svn: 144113
2011-11-08 19:56:31 +00:00
Anna Zaks ae9d54bcf3 [analyzer] Testing infrastructure: flush the output between projects.
llvm-svn: 144051
2011-11-08 00:27:56 +00:00
Anna Zaks 4720a730f6 [analyzer] Add support for testing with individual preprocessed files.
llvm-svn: 143786
2011-11-05 05:20:48 +00:00
Anna Zaks 42a4463091 [analyzer] Tweak the test script
Rename the pre_run_static_analyzer script into cleanup_run_static_analyzer
and use it to clean up after the reference build - which is done as part of
adding a new project to the system.

llvm-svn: 143568
2011-11-02 20:46:50 +00:00
Anna Zaks f0c4116202 [analyzer] Static Analyzer Qualification Infrastructure: Scripts to support basic testing of the analyzer on external projects. This can be used as a basis for setting up a buildbot.
llvm-svn: 141337
2011-10-06 23:26:27 +00:00