Commit Graph

108 Commits

Author SHA1 Message Date
Johnny Chen c5cc625c79 Test driver should also report skipped tests because there were cases when tests were skipped
due to incorrect skip-logic.

llvm-svn: 137678
2011-08-15 23:09:08 +00:00
Johnny Chen 60726fab78 Fix a test driver bug where multiple filter specs (through the -f TestClassName.test_method_name
command line option) belonging to the same Test*.py file are ignored, except for the first one.

llvm-svn: 137540
2011-08-12 23:55:07 +00:00
Johnny Chen 70cb84a0f6 When running dotest.py under pdb (the python debugger), define DOTEST_PDB=YES and
DOTEST_SCRIPT_DIR=/path/leading/to/your/dotest/script/directory.

llvm-svn: 137492
2011-08-12 18:54:11 +00:00
Johnny Chen 19bfde34ed Add a test to check the integrity of the lldb public api directory containing SB*.h headers.
There should be nothing unwanted there and a simpe main.cpp (generated from main.cpp.template)
which includes SB*.h should compile and link with the LLDB framework.

llvm-svn: 136894
2011-08-04 18:17:16 +00:00
Johnny Chen 5ccbccfce0 Add a @benchmarks_test decorator for test method we want to categorize as benchmarks test.
The test driver now takes an option "+b" which enables to run just the benchmarks tests.
By default, tests decorated with the @benchmarks_test decorator do not get run.

Add an example benchmarks test directory which contains nothing for the time being,
just to demonstrate the @benchmarks_test concept.

For example,

$ ./dotest.py -v benchmarks

...

----------------------------------------------------------------------
Collected 2 tests

1: test_with_gdb (TestRepeatedExprs.RepeatedExprssCase)
   Test repeated expressions with gdb. ... skipped 'benchmarks tests'
2: test_with_lldb (TestRepeatedExprs.RepeatedExprssCase)
   Test repeated expressions with lldb. ... skipped 'benchmarks tests'

----------------------------------------------------------------------
Ran 2 tests in 0.047s

OK (skipped=2)
$ ./dotest.py -v +b benchmarks

...

----------------------------------------------------------------------
Collected 2 tests

1: test_with_gdb (TestRepeatedExprs.RepeatedExprssCase)
   Test repeated expressions with gdb. ... running test_with_gdb
benchmarks result for test_with_gdb
ok
2: test_with_lldb (TestRepeatedExprs.RepeatedExprssCase)
   Test repeated expressions with lldb. ... running test_with_lldb
benchmarks result for test_with_lldb
ok

----------------------------------------------------------------------
Ran 2 tests in 0.270s

OK

Also mark some Python API tests which are missing the @python_api_test decorator.

llvm-svn: 136553
2011-07-30 01:39:58 +00:00
Johnny Chen 4a57d122fe Add a redo.py script which takes a session directory name as arg and digs into the directory
to find out the tests which failed/errored and need re-running.  The dotest.py test driver
script is modified to allow specifying multiple -f testclass.testmethod in the command line
to accommodate the redo functionality.

An example,

 $ ./redo.py -n 2011-07-29-11_50_14
adding filterspec: TargetAPITestCase.test_find_global_variables_with_dwarf
adding filterspec: DisasmAPITestCase.test_with_dsym
Running ./dotest.py -v  -f TargetAPITestCase.test_find_global_variables_with_dwarf -f DisasmAPITestCase.test_with_dsym

...

----------------------------------------------------------------------
Collected 2 tests

1: test_with_dsym (TestDisasmAPI.DisasmAPITestCase)
   Exercise getting SBAddress objects, disassembly, and SBAddress APIs. ... ok
2: test_find_global_variables_with_dwarf (TestTargetAPI.TargetAPITestCase)
   Exercise SBTarget.FindGlobalVariables() API. ... ok

----------------------------------------------------------------------
Ran 2 tests in 15.328s

OK

llvm-svn: 136533
2011-07-29 22:54:56 +00:00
Johnny Chen cbdfe81171 Compute svn_info for git-svn bridge, too.
llvm-svn: 133836
2011-06-24 22:52:05 +00:00
Johnny Chen d0abdab8d1 Fix the log channel for gdb-remote. It should be "gdb-remote" now, instead of "process.gdb-remote".
llvm-svn: 133534
2011-06-21 19:25:45 +00:00
Peter Collingbourne 6f2b85221d Set LLDB_SESSION_DIRNAME to absolute path
This is to ensure that session files are written to the correct
directory if the -r option is given.

llvm-svn: 133483
2011-06-20 23:55:53 +00:00
Peter Collingbourne d6824de00e Modify TestHelp to search for LLDB-Info.plist in the correct place
This is required to run the test suite outside of the source tree
or on a case sensitive file system.

llvm-svn: 133460
2011-06-20 19:06:45 +00:00
Peter Collingbourne 28b5eb14c7 Have dotest.py search for test cases in its own directory by default,
rather than the current directory.

This is required to run the test suite outside of the source tree.

llvm-svn: 133458
2011-06-20 19:06:29 +00:00
Peter Collingbourne d736854419 Fix typo: curret -> current
llvm-svn: 132969
2011-06-14 03:55:45 +00:00
Johnny Chen 81a9df5b22 Let's also stick the svn info for the base directory into a file under
the session directory, which also includes the command line used to invoke
the test suite.

llvm-svn: 131501
2011-05-17 22:58:50 +00:00
Johnny Chen 44932b6805 For a test with unexpected success status, we also dump its session info into a unique file.
llvm-svn: 131011
2011-05-06 20:30:22 +00:00
Johnny Chen 0a8a54c117 Docstring.
llvm-svn: 130254
2011-04-26 23:10:51 +00:00
Johnny Chen 4fdcebda1d Docstring changees.
llvm-svn: 130251
2011-04-26 22:53:38 +00:00
Johnny Chen 489b52572c Simplify the parsing of ARCH and Compiler specs.
llvm-svn: 130240
2011-04-26 20:45:00 +00:00
Johnny Chen 71aaf75e3d Use self.TraceOn() API.
llvm-svn: 129936
2011-04-21 20:48:32 +00:00
Jim Ingham 5d3e822926 Have "-h" print just the command arguments, and "-h -v" print the examples as well.
llvm-svn: 129470
2011-04-13 21:11:41 +00:00
Greg Clayton 8b82f087a0 Moved the execution context that was in the Debugger into
the CommandInterpreter where it was always being used.

Make sure that Modules can track their object file offsets correctly to
allow opening of sub object files (like the "__commpage" on darwin).

Modified the Platforms to be able to launch processes. The first part of this
move is the platform soon will become the entity that launches your program
and when it does, it uses a new ProcessLaunchInfo class which encapsulates
all process launching settings. This simplifies the internal APIs needed for
launching. I want to slowly phase out process launching from the process
classes, so for now we can still launch just as we used to, but eventually
the platform is the object that should do the launching.

Modified the Host::LaunchProcess in the MacOSX Host.mm to correctly be able
to launch processes with all of the new eLaunchFlag settings. Modified any
code that was manually launching processes to use the Host::LaunchProcess
functions.

Fixed an issue where lldb_private::Args had implicitly defined copy 
constructors that could do the wrong thing. This has now been fixed by adding
an appropriate copy constructor and assignment operator.

Make sure we don't add empty ModuleSP entries to a module list.

Fixed the commpage module creation on MacOSX, but we still need to train
the MacOSX dynamic loader to not get rid of it when it doesn't have an entry
in the all image infos.

Abstracted many more calls from in ProcessGDBRemote down into the 
GDBRemoteCommunicationClient subclass to make the classes cleaner and more
efficient.

Fixed the default iOS ARM register context to be correct and also added support
for targets that don't support the qThreadStopInfo packet by selecting the
current thread (only if needed) and then sending a stop reply packet.

Debugserver can now start up with a --unix-socket (-u for short) and can 
then bind to port zero and send the port it bound to to a listening process
on the other end. This allows the GDB remote platform to spawn new GDB server
instances (debugserver) to allow platform debugging.

llvm-svn: 129351
2011-04-12 05:54:46 +00:00
Johnny Chen dbfc256eec As a simple measure, output the lldb version and the svn info for
which the testsuite is run against.

llvm-svn: 127782
2011-03-17 00:38:22 +00:00
Johnny Chen b7cfba4cb1 Add pexpect-2.4 (a pure Python module for controlling and automating other programs) to the test directory.
http://pypi.python.org/pypi/pexpect/

llvm-svn: 127484
2011-03-11 20:13:06 +00:00
Johnny Chen cd29b050b7 Add a mechanism to discover the full path of the 'lldb' program into the test driver.
This provides a way to potentially provide conversational interactions with 'lldb' in the test suite.

llvm-svn: 127479
2011-03-11 19:47:23 +00:00
Johnny Chen e0ec9ea5ce Add the ability for the test suite to specify a list of compilers and a list of architectures
on the command line.  For example, use '-A x86_64^i386' to launch the inferior use both x86_64
and i386.

This is an example of building the debuggee using both clang and gcc compiers:

[17:30:46] johnny:/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -C clang^gcc -v -f SourceManagerTestCase.test_modify_source_file_while_debugging

Session logs for test failures/errors will go into directory '2011-03-03-17_31_39'
Command invoked: python ./dotest.py -C clang^gcc -v -f SourceManagerTestCase.test_modify_source_file_while_debugging

Configuration:  compiler=clang
----------------------------------------------------------------------
Collected 1 test

1: test_modify_source_file_while_debugging (TestSourceManager.SourceManagerTestCase)
   Modify a source file while debugging the executable. ... Command 'run' failed!

original content: #include <stdio.h>

int main(int argc, char const *argv[]) {
    printf("Hello world.\n"); // Set break point at this line.
    return 0;
}

new content: #include <stdio.h>

int main(int argc, char const *argv[]) {
    printf("Hello lldb.\n"); // Set break point at this line.
    return 0;
}

os.path.getmtime() after writing new content: 1299202305.0
content restored to: #include <stdio.h>

int main(int argc, char const *argv[]) {
    printf("Hello world.\n"); // Set break point at this line.
    return 0;
}

os.path.getmtime() after restore: 1299202307.0
ok

----------------------------------------------------------------------
Ran 1 test in 8.259s

OK

Configuration:  compiler=gcc
----------------------------------------------------------------------
Collected 1 test

1: test_modify_source_file_while_debugging (TestSourceManager.SourceManagerTestCase)
   Modify a source file while debugging the executable. ... original content: #include <stdio.h>

int main(int argc, char const *argv[]) {
    printf("Hello world.\n"); // Set break point at this line.
    return 0;
}

new content: #include <stdio.h>

int main(int argc, char const *argv[]) {
    printf("Hello lldb.\n"); // Set break point at this line.
    return 0;
}

os.path.getmtime() after writing new content: 1299202307.0
content restored to: #include <stdio.h>

int main(int argc, char const *argv[]) {
    printf("Hello world.\n"); // Set break point at this line.
    return 0;
}

os.path.getmtime() after restore: 1299202309.0
ok

----------------------------------------------------------------------
Ran 1 test in 2.301s

OK
[17:31:49] johnny:/Volumes/data/lldb/svn/trunk/test $ 

llvm-svn: 126979
2011-03-04 01:35:22 +00:00
Greg Clayton 64195a2c8b Abtracted all mach-o and ELF out of ArchSpec. This patch is a modified form
of Stephen Wilson's idea (thanks for the input Stephen!). What I ended up
doing was:
- Got rid of ArchSpec::CPU (which was a generic CPU enumeration that mimics
  the contents of llvm::Triple::ArchType). We now rely upon the llvm::Triple 
  to give us the machine type from llvm::Triple::ArchType.
- There is a new ArchSpec::Core definition which further qualifies the CPU
  core we are dealing with into a single enumeration. If you need support for
  a new Core and want to debug it in LLDB, it must be added to this list. In
  the future we can allow for dynamic core registration, but for now it is
  hard coded.
- The ArchSpec can now be initialized with a llvm::Triple or with a C string
  that represents the triple (it can just be an arch still like "i386").
- The ArchSpec can still initialize itself with a architecture type -- mach-o
  with cpu type and subtype, or ELF with e_machine + e_flags -- and this will
  then get translated into the internal llvm::Triple::ArchSpec + ArchSpec::Core.
  The mach-o cpu type and subtype can be accessed using the getter functions:
  
  uint32_t
  ArchSpec::GetMachOCPUType () const;

  uint32_t
  ArchSpec::GetMachOCPUSubType () const;
  
  But these functions are just converting out internal llvm::Triple::ArchSpec 
  + ArchSpec::Core back into mach-o. Same goes for ELF.

All code has been updated to deal with the changes.

This should abstract us until later when the llvm::TargetSpec stuff gets
finalized and we can then adopt it.

llvm-svn: 126278
2011-02-23 00:35:02 +00:00
Greg Clayton 1971900fdc Don't enable thread safe logging as it currently deadlocks logging.
llvm-svn: 126260
2011-02-22 23:08:31 +00:00
Greg Clayton 76b436e21d Make logs threadsafe (add the -t option) when logging API stuff.
llvm-svn: 126257
2011-02-22 22:59:30 +00:00
Johnny Chen ffb3b51ba8 Refactored the test driver to abstract out the Python sys.path specification for
different build configurations.

llvm-svn: 125584
2011-02-15 18:50:19 +00:00
Greg Clayton c2448f6175 Added support for Xcode 4 build directories.
llvm-svn: 125522
2011-02-14 21:17:06 +00:00
Johnny Chen 74d53758c5 Added comment.
llvm-svn: 124517
2011-01-29 01:21:04 +00:00
Johnny Chen 6573818e38 Add a "-D" option to the test driver which dumps the Python sys.path variable
to be used for the test run.  Could be useful for debugging the setup of the
test environment.

llvm-svn: 124516
2011-01-29 01:16:52 +00:00
Johnny Chen 4f7370f7ae Add "-A arch_spec" and "-C compiler_spec" options to the test driver. Example:
$ ./dotest.py -A i386 -C clang -v -w -t -p TestObjCMethods.py
$ ./dotest.py -A x86_64 -C gcc -v -w types

llvm-svn: 124303
2011-01-26 19:07:42 +00:00
Johnny Chen f249090d7b Print out the command line used to invoke the test suite so I don't get confused
since different options can affect the run time.

llvm-svn: 123843
2011-01-19 19:31:46 +00:00
Johnny Chen 30775d0107 Added special logic to faciliate profiling the test suite run with the cProfile.py
module.

On my MBP running SnowLeopard:

$ DOTEST_PROFILE=YES DOTEST_SCRIPT_DIR=/Volumes/data/lldb/svn/trunk/test /System/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/cProfile.py -o my.profile ./dotest.py -v -w 2> ~/Developer/Log/lldbtest.log

After that, I used the pstats.py module to browse the statistics recorded in the my.profile file. 

llvm-svn: 123807
2011-01-19 02:10:40 +00:00
Johnny Chen 64e3868cf0 Properly indent the short description of the test case to make it align with the
previously added ordinal number of the currently running test case.

llvm-svn: 122922
2011-01-05 22:50:11 +00:00
Johnny Chen 77c81d3ee7 Enhance the test framework to be able to emit a counter value in verbose mode
describing the ordinal number of the currently running test case.

llvm-svn: 122901
2011-01-05 20:24:11 +00:00
Johnny Chen 240f8c7dd1 Do not specify both '-a' and '+a' at the same time.
llvm-svn: 121501
2010-12-10 19:02:23 +00:00
Johnny Chen f3e22ac3c7 Add a '+a' command line option to the test driver to run only the Python API tests.
Add an attribute __python_api_test__ (set to True) to the @python_api_test decorated
test method to distinguish them from the lldb command line tests.

llvm-svn: 121500
2010-12-10 18:52:10 +00:00
Johnny Chen 7c7d936e48 Add an infrastructure to mark the Python APIs only test using a decorator.
Example:

    @python_api_test
    def test_evaluate_expression_python(self):
        """Test SBFrame.EvaluateExpression() API for evaluating an expression."""
    ...

The opposite of Python APIs only test is an lldb command line test, which sends
commands to the lldb command interpreter.  Add a '-a' option to the test driver
to skip Python API only tests.

Modify TestExprs.py to mark a test as @python_api_test and remove an @expectedFailure
decorator as the bug has been fixed.

llvm-svn: 121442
2010-12-10 00:51:23 +00:00
Johnny Chen 025d1b8f95 For SBTarget.Launch()/LaunchProcess(), there's no need to pass an empty string
as the args and the envs to the launched process.

o lldbtest.py:

  Forgot to check in some assertion messages changes for lldbtest.py.

o dotest.py:

  Also add "api" category to the default lldb log option list.

llvm-svn: 121220
2010-12-08 01:25:21 +00:00
Johnny Chen 729c339a1c Add the '-F' option to the test driver, which makes the running of test suite failfast.
As soon as an error or a failure is encountered, it stops the test suite.

llvm-svn: 120837
2010-12-03 19:59:35 +00:00
Johnny Chen eb3411dc41 Log just the thread name, not process/thread id for easier human reading.
llvm-svn: 120751
2010-12-02 23:31:02 +00:00
Johnny Chen 7528af2f86 When logging, enable process and thread ID recordings. Also add 'state' to the lldb
log-categories, so it now becomes "event process expr state".

llvm-svn: 120745
2010-12-02 22:47:52 +00:00
Johnny Chen 20d0b2198b For lldb loggings, prepend log lines with a timestamp. And change the default log options:
o lldb => event process expr
o gdb-remote => packets process

llvm-svn: 120725
2010-12-02 18:35:13 +00:00
Johnny Chen 75739045d3 Add a '-b blacklistFile' option to the test driver to take a file specifying the
test classes or test cases to be excludued from the test suite.

Check in an example blacklist file: blacklist.py:

"""
'blacklist' is a Python dictionary, it stores the mapping of a string describing
either a testclass or a testcase, i.e, testclass.testmethod, to the reason (a
string) it is blacklisted.

Following is an example which states that test class IntegerTypesExprTestCase
should be skipped because 'This test class crashed' and the test case
FoundationTestCase.test_data_type_and_expr_with_dsym should be skipped because
it is 'Temporarily disabled'.

blacklist = {'IntegerTypesExprTestCase': 'This test class crashed',
             'FoundationTestCase.test_data_type_and_expr_with_dsym': 'Temporarily disabled'
             }
"""

blacklist = {}

An example of invoking the test driver and specifying a blacklist file:

./dotest.py -b blacklist.py -v types

This runs the tests under 'types' directory but excludes the tests specified in
balcklist.py.

llvm-svn: 120620
2010-12-01 22:47:54 +00:00
Johnny Chen bbda8ed39a Add comment.
llvm-svn: 120295
2010-11-29 17:52:43 +00:00
Johnny Chen 46389e9ddf When looping through the test suite more than once (repeat count > 1), we should
ignore and not enforce the signleton pattern for the LLDBTestResult class.

llvm-svn: 120294
2010-11-29 17:50:10 +00:00
Johnny Chen ecb072d840 Add an option '-# count' to run the specified test suite for a said number of times.
This is not to be used during normal test suite run, but to be used to stress test
specific test sequences repeatedly.

Example:

./dotest.py -# 3 -v breakpoint_conditions

will repeat the test suite 3 times for tests under the breakpoint_conditions directory.

llvm-svn: 119399
2010-11-16 22:42:58 +00:00
Johnny Chen 5f98696f51 Add help message for the scenario of using lldb.py module distributed by Xcode4
to invoke the test driver.

llvm-svn: 118847
2010-11-11 22:14:56 +00:00
Johnny Chen cc78b0b67f Add more comments on LLDBTestResult class.
llvm-svn: 118655
2010-11-09 23:56:14 +00:00