Commit Graph

185 Commits

Author SHA1 Message Date
Johnny Chen b127f00b51 Make the default architectures to run as both 64 and 32-bit.
rdar://problem/11031264

llvm-svn: 154573
2012-04-12 00:55:57 +00:00
Johnny Chen f1548d4f74 Add a new option to the test driver, -N dsym or -N dwarf, in order to exclude tests decorated with
either @dsym_test or @dwarf_test to be executed during the testsuite run.  There are still lots of
Test*.py files which have not been decorated with the new decorator.

An example:

# From TestMyFirstWatchpoint.py ->
class HelloWatchpointTestCase(TestBase):

    mydir = os.path.join("functionalities", "watchpoint", "hello_watchpoint")

    @dsym_test
    def test_hello_watchpoint_with_dsym_using_watchpoint_set(self):
        """Test a simple sequence of watchpoint creation and watchpoint hit."""
        self.buildDsym(dictionary=self.d)
        self.setTearDownCleanup(dictionary=self.d)
        self.hello_watchpoint()

    @dwarf_test
    def test_hello_watchpoint_with_dwarf_using_watchpoint_set(self):
        """Test a simple sequence of watchpoint creation and watchpoint hit."""
        self.buildDwarf(dictionary=self.d)
        self.setTearDownCleanup(dictionary=self.d)
        self.hello_watchpoint()


# Invocation ->
[17:50:14] johnny:/Volumes/data/lldb/svn/ToT/test $ ./dotest.py -N dsym -v -p TestMyFirstWatchpoint.py
LLDB build dir: /Volumes/data/lldb/svn/ToT/build/Debug
LLDB-137
Path: /Volumes/data/lldb/svn/ToT
URL: https://johnny@llvm.org/svn/llvm-project/lldb/trunk
Repository Root: https://johnny@llvm.org/svn/llvm-project
Repository UUID: 91177308-0d34-0410-b5e6-96231b3b80d8
Revision: 154133
Node Kind: directory
Schedule: normal
Last Changed Author: gclayton
Last Changed Rev: 154109
Last Changed Date: 2012-04-05 10:43:02 -0700 (Thu, 05 Apr 2012)



Session logs for test failures/errors/unexpected successes will go into directory '2012-04-05-17_50_49'
Command invoked: python ./dotest.py -N dsym -v -p TestMyFirstWatchpoint.py
compilers=['clang']

Configuration: arch=x86_64 compiler=clang
----------------------------------------------------------------------
Collected 2 tests

1: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
   Test a simple sequence of watchpoint creation and watchpoint hit. ... skipped 'dsym tests'
2: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
   Test a simple sequence of watchpoint creation and watchpoint hit. ... ok

----------------------------------------------------------------------
Ran 2 tests in 1.138s

OK (skipped=1)
Session logs for test failures/errors/unexpected successes can be found in directory '2012-04-05-17_50_49'
[17:50:50] johnny:/Volumes/data/lldb/svn/ToT/test $ 

llvm-svn: 154154
2012-04-06 00:56:05 +00:00
Johnny Chen e344486e63 Add a '-E' option to the test driver for the purpose of specifying some extra CFLAGS
to pass to the toolchain in order to build the inferior programs to be run/debugged
duirng the test suite.  The architecture might dictate some special CFLAGS which are
more easily specified in a central place (like the command line) instead of inside
make rules.

For Example,

./dotest.py -v -r /shared/phone -A armv7 -E "-isysroot your_sdk_root" functionalities/watchpoint/hello_watchpoint

will relocate the particular test directory ('functionalities/watchpoint/hello_watchpoint' in this case) to a
new directory named '/shared/phone'.  The particular incarnation of the architecture-compiler combination of the
test support files are therefore to be found under:

/shared/phone.arch=armv7-compiler=clang/functionalities/watchpoint/hello_watchpoint

The building of the inferior programs under testing is now working.

The actual launching/debugging of the inferior programs are not yet working,
neither is the setting of a watchpoint on the phone.

llvm-svn: 153070
2012-03-20 00:33:51 +00:00
Sean Callanan ddcda28fee Disabled the i386 tests pending fixes in the JIT.
llvm-svn: 152568
2012-03-12 18:54:10 +00:00
Johnny Chen 934c05d284 Add the capability on OS X to utilize 'xcrun' to locate the compilers used for building the inferior programs
to be debugged while running the test suite.  By default, compilers is set to ['clang'] and can be overridden
using the "-C compilerA^compilerB" option.

llvm-svn: 152367
2012-03-09 02:11:37 +00:00
Johnny Chen 5a9a9883be Change the test driver so that, by default, it takes into consideration of both 'x86_64' and 'i386' architectures
when building the inferior programs.

Example:

/Volumes/data/lldb/svn/ToT/test $ ./dotest.py -v functionalities/watchpoint
LLDB build dir: /Volumes/data/lldb/svn/ToT/build/Debug
LLDB-123
Path: /Volumes/data/lldb/svn/ToT
URL: https://johnny@llvm.org/svn/llvm-project/lldb/trunk
Repository Root: https://johnny@llvm.org/svn/llvm-project
Repository UUID: 91177308-0d34-0410-b5e6-96231b3b80d8
Revision: 152244
Node Kind: directory
Schedule: normal
Last Changed Author: gclayton
Last Changed Rev: 152244
Last Changed Date: 2012-03-07 13:03:09 -0800 (Wed, 07 Mar 2012)



Session logs for test failures/errors/unexpected successes will go into directory '2012-03-08-16_43_51'
Command invoked: python ./dotest.py -v functionalities/watchpoint

Configuration: arch=x86_64
----------------------------------------------------------------------
Collected 21 tests

 1: test_hello_watchlocation_with_dsym (TestWatchLocation.HelloWatchLocationTestCase)
    Test watching a location with '-x size' option. ... ok
 2: test_hello_watchlocation_with_dwarf (TestWatchLocation.HelloWatchLocationTestCase)
    Test watching a location with '-x size' option. ... ok
 3: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
    Test a simple sequence of watchpoint creation and watchpoint hit. ... ok
 4: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
    Test a simple sequence of watchpoint creation and watchpoint hit. ... ok
 5: test_watchpoint_multiple_threads_with_dsym (TestWatchpointMultipleThreads.WatchpointForMultipleThreadsTestCase)
    Test that lldb watchpoint works for multiple threads. ... ok
 6: test_watchpoint_multiple_threads_with_dwarf (TestWatchpointMultipleThreads.WatchpointForMultipleThreadsTestCase)
    Test that lldb watchpoint works for multiple threads. ... ok
 7: test_rw_disable_after_first_stop__with_dwarf (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint but disable it after the first stop. ... ok
 8: test_rw_disable_after_first_stop_with_dsym (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint but disable it after the first stop. ... ok
 9: test_rw_disable_then_enable_with_dsym (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint, disable initially, then enable it. ... ok
10: test_rw_disable_then_enable_with_dwarf (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint, disable initially, then enable it. ... ok
11: test_rw_watchpoint_delete_with_dsym (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test delete watchpoint and expect not to stop for watchpoint. ... ok
12: test_rw_watchpoint_delete_with_dwarf (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test delete watchpoint and expect not to stop for watchpoint. ... ok
13: test_rw_watchpoint_set_ignore_count_with_dsym (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test watchpoint ignore count and expect to not to stop at all. ... ok
14: test_rw_watchpoint_set_ignore_count_with_dwarf (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test watchpoint ignore count and expect to not to stop at all. ... ok
15: test_rw_watchpoint_with_dsym (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint and expect to stop two times. ... ok
16: test_rw_watchpoint_with_dwarf (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint and expect to stop two times. ... ok
17: test_watchpoint_cond_with_dsym (TestWatchpointConditionCmd.WatchpointConditionCmdTestCase)
    Test watchpoint condition. ... ok
18: test_watchpoint_cond_with_dwarf (TestWatchpointConditionCmd.WatchpointConditionCmdTestCase)
    Test watchpoint condition. ... ok
19: test_watchlocation_with_dsym_using_watchpoint_set (TestWatchLocationWithWatchSet.WatchLocationUsingWatchpointSetTestCase)
    Test watching a location with 'watchpoint set expression -w write -x size' option. ... ok
20: test_watchlocation_with_dwarf_using_watchpoint_set (TestWatchLocationWithWatchSet.WatchLocationUsingWatchpointSetTestCase)
    Test watching a location with 'watchpoint set expression -w write -x size' option. ... ok
21: test_error_cases_with_watchpoint_set (TestWatchpointSetErrorCases.WatchpointSetErrorTestCase)
    Test error cases with the 'watchpoint set' command. ... ok

----------------------------------------------------------------------
Ran 21 tests in 74.590s

OK

Configuration: arch=i386
----------------------------------------------------------------------
Collected 21 tests

 1: test_hello_watchlocation_with_dsym (TestWatchLocation.HelloWatchLocationTestCase)
    Test watching a location with '-x size' option. ... ok
 2: test_hello_watchlocation_with_dwarf (TestWatchLocation.HelloWatchLocationTestCase)
    Test watching a location with '-x size' option. ... ok
 3: test_hello_watchpoint_with_dsym_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
    Test a simple sequence of watchpoint creation and watchpoint hit. ... ok
 4: test_hello_watchpoint_with_dwarf_using_watchpoint_set (TestMyFirstWatchpoint.HelloWatchpointTestCase)
    Test a simple sequence of watchpoint creation and watchpoint hit. ... ok
 5: test_watchpoint_multiple_threads_with_dsym (TestWatchpointMultipleThreads.WatchpointForMultipleThreadsTestCase)
    Test that lldb watchpoint works for multiple threads. ... ok
 6: test_watchpoint_multiple_threads_with_dwarf (TestWatchpointMultipleThreads.WatchpointForMultipleThreadsTestCase)
    Test that lldb watchpoint works for multiple threads. ... ok
 7: test_rw_disable_after_first_stop__with_dwarf (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint but disable it after the first stop. ... ok
 8: test_rw_disable_after_first_stop_with_dsym (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint but disable it after the first stop. ... ok
 9: test_rw_disable_then_enable_with_dsym (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint, disable initially, then enable it. ... ok
10: test_rw_disable_then_enable_with_dwarf (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint, disable initially, then enable it. ... ok
11: test_rw_watchpoint_delete_with_dsym (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test delete watchpoint and expect not to stop for watchpoint. ... ok
12: test_rw_watchpoint_delete_with_dwarf (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test delete watchpoint and expect not to stop for watchpoint. ... ok
13: test_rw_watchpoint_set_ignore_count_with_dsym (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test watchpoint ignore count and expect to not to stop at all. ... ok
14: test_rw_watchpoint_set_ignore_count_with_dwarf (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test watchpoint ignore count and expect to not to stop at all. ... ok
15: test_rw_watchpoint_with_dsym (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint and expect to stop two times. ... ok
16: test_rw_watchpoint_with_dwarf (TestWatchpointCommands.WatchpointCommandsTestCase)
    Test read_write watchpoint and expect to stop two times. ... ok
17: test_watchpoint_cond_with_dsym (TestWatchpointConditionCmd.WatchpointConditionCmdTestCase)
    Test watchpoint condition. ... ok
18: test_watchpoint_cond_with_dwarf (TestWatchpointConditionCmd.WatchpointConditionCmdTestCase)
    Test watchpoint condition. ... ok
19: test_watchlocation_with_dsym_using_watchpoint_set (TestWatchLocationWithWatchSet.WatchLocationUsingWatchpointSetTestCase)
    Test watching a location with 'watchpoint set expression -w write -x size' option. ... ok
20: test_watchlocation_with_dwarf_using_watchpoint_set (TestWatchLocationWithWatchSet.WatchLocationUsingWatchpointSetTestCase)
    Test watching a location with 'watchpoint set expression -w write -x size' option. ... ok
21: test_error_cases_with_watchpoint_set (TestWatchpointSetErrorCases.WatchpointSetErrorTestCase)
    Test error cases with the 'watchpoint set' command. ... ok

----------------------------------------------------------------------
Ran 21 tests in 67.059s

OK

llvm-svn: 152357
2012-03-09 00:44:59 +00:00
Johnny Chen 5d0d607b6b Add a period.
llvm-svn: 149305
2012-01-31 00:48:02 +00:00
Johnny Chen 8f8a2a7774 Add a '-u ENV_VAR_NAME' option to the test driver, whose purpose is to unset the said
environment variable before starting the test runner which executes the test cases and
may spawn child processes.  An example:

    ./dotest.py -u MY_ENV1 -u MY_ENV2 -v -p TestWatchLocationWithWatchSet.py

llvm-svn: 149304
2012-01-31 00:38:03 +00:00
Johnny Chen bc774c1156 Add a '-X excluded-dir' option to the test driver to exclude a directory from consideration during test discovery. For example:
./dotest.py -X types -v

from the test dir will ignore test cases under test/types.

llvm-svn: 148379
2012-01-18 05:15:00 +00:00
Johnny Chen 2e275c9e6a Add usage example of specifying the full path to the compiler used while building the debuggees for running the test suite.
llvm-svn: 148273
2012-01-17 01:26:06 +00:00
Johnny Chen 316651ab21 Add a simple progress bar when neither '-v' nor '-t' is specified.
llvm-svn: 144940
2011-11-18 00:19:29 +00:00
Johnny Chen 0fddfb2ceb Add an option '-S' to skip the build and cleanup while running the test.
Use this option with care as you would need to build the inferior(s) by hand
and build the executable(s) with the correct name(s).  This option can be used
with '-# n' to stress test certain test cases for n number of times.

An example:

[11:55:11] johnny:/Volumes/data/lldb/svn/trunk/test/python_api/value $ ls
Makefile		TestValueAPI.pyc	linked_list
TestValueAPI.py		change_values		main.c
[11:55:14] johnny:/Volumes/data/lldb/svn/trunk/test/python_api/value $ make EXE=test_with_dsym
clang -gdwarf-2 -O0  -arch x86_64   -c -o main.o main.c
clang -gdwarf-2 -O0  -arch x86_64   main.o -o "test_with_dsym"
/usr/bin/dsymutil  -o "test_with_dsym.dSYM" "test_with_dsym"
[11:55:20] johnny:/Volumes/data/lldb/svn/trunk/test/python_api/value $ cd ../..
[11:55:24] johnny:/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -v -# 10 -S -f ValueAPITestCase.test_with_dsym
LLDB build dir: /Volumes/data/lldb/svn/trunk/build/Debug
LLDB-89
Path: /Volumes/data/lldb/svn/trunk
URL: https://johnny@llvm.org/svn/llvm-project/lldb/trunk
Repository Root: https://johnny@llvm.org/svn/llvm-project
Repository UUID: 91177308-0d34-0410-b5e6-96231b3b80d8
Revision: 144914
Node Kind: directory
Schedule: normal
Last Changed Author: gclayton
Last Changed Rev: 144911
Last Changed Date: 2011-11-17 09:22:31 -0800 (Thu, 17 Nov 2011)



Session logs for test failures/errors/unexpected successes will go into directory '2011-11-17-11_55_29'
Command invoked: python ./dotest.py -v -# 10 -S -f ValueAPITestCase.test_with_dsym
----------------------------------------------------------------------
Collected 1 test

1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 1.163s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.200s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.198s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.199s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.239s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 1.215s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.105s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.098s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 0.195s

OK
1: test_with_dsym (TestValueAPI.ValueAPITestCase)
   Exercise some SBValue APIs. ... ok

----------------------------------------------------------------------
Ran 1 test in 1.197s

OK
[11:55:34] johnny:/Volumes/data/lldb/svn/trunk/test $ 

llvm-svn: 144919
2011-11-17 19:57:27 +00:00
Daniel Dunbar 16d88ff045 tests: Improve Makefile/dotest to properly set LLDB_HERE variable used in some
tests.

llvm-svn: 143394
2011-10-31 23:27:06 +00:00
Johnny Chen 6c75b61f08 Fix a typo.
llvm-svn: 143207
2011-10-28 17:56:02 +00:00
Johnny Chen d9809f542b Print out the version of the locally built 'lldb' binary, not the one found in your PATH env variable.
llvm-svn: 143170
2011-10-28 00:59:00 +00:00
Johnny Chen fc9e79fb95 Benchmark the turnaround time starting a debugger and run to the breakpoint with lldb vs. gdb.
An example (with /Developer/usr/bin/lldb vs. /usr/bin/gdb):

[13:05:04] johnny:/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -v +b -n -p TestCompileRunToBreakpointTurnaround.py
1: test_run_lldb_then_gdb (TestCompileRunToBreakpointTurnaround.CompileRunToBreakpointBench)
   Benchmark turnaround time with lldb vs. gdb. ... 
lldb turnaround benchmark: Avg: 4.574600 (Laps: 3, Total Elapsed Time: 13.723799)
gdb turnaround benchmark: Avg: 7.966713 (Laps: 3, Total Elapsed Time: 23.900139)
lldb_avg/gdb_avg: 0.574214
ok

----------------------------------------------------------------------
Ran 1 test in 55.462s

OK

llvm-svn: 142949
2011-10-25 20:08:03 +00:00
Johnny Chen b8da426285 Add bench.py as a driver script to run some benchmarks on lldb.
Add benchmarks for expression evaluations (TestExpressionCmd.py) and disassembly (TestDoAttachThenDisassembly.py).

An example:
[17:45:55] johnny:/Volumes/data/lldb/svn/trunk/test $ ./bench.py 2>&1 | grep -P '^lldb.*benchmark:'
lldb startup delay (create fresh target) benchmark: Avg: 0.104274 (Laps: 30, Total Elapsed Time: 3.128214)
lldb startup delay (set first breakpoint) benchmark: Avg: 0.102216 (Laps: 30, Total Elapsed Time: 3.066470)
lldb frame variable benchmark: Avg: 1.649162 (Laps: 20, Total Elapsed Time: 32.983245)
lldb stepping benchmark: Avg: 0.104409 (Laps: 50, Total Elapsed Time: 5.220461)
lldb expr cmd benchmark: Avg: 0.206774 (Laps: 25, Total Elapsed Time: 5.169350)
lldb disassembly benchmark: Avg: 0.089086 (Laps: 10, Total Elapsed Time: 0.890859)

llvm-svn: 142708
2011-10-22 00:57:05 +00:00
Johnny Chen 4d4363ba56 Add a '-n' option to turn off printings of build dir, lldb version, svn info, and other headers
which happen before the listingings of test cases.

llvm-svn: 142668
2011-10-21 18:33:27 +00:00
Johnny Chen ff7fc9cfa4 Breakpoint specification can have the form '-n main', so it's not a good idea to
check that the option arg in '-x opt_arg' does not start with a '-' char.

llvm-svn: 142625
2011-10-20 22:16:24 +00:00
Johnny Chen 38f9daa303 Parameterize the iteration count used when running benchmarks, instead of hard-coded inside the test case.
Add a '-y count' option to the test driver for this purpose.  An example:

 $  ./dotest.py -v -y 25 +b -p TestDisassembly.py

...

----------------------------------------------------------------------
Collected 2 tests

1: test_run_gdb_then_lldb (TestDisassembly.DisassembleDriverMainLoop)
   Test disassembly on a large function with lldb vs. gdb. ... 
gdb benchmark: Avg: 0.226305 (Laps: 25, Total Elapsed Time: 5.657614)
lldb benchmark: Avg: 0.113864 (Laps: 25, Total Elapsed Time: 2.846606)
lldb_avg/gdb_avg: 0.503146
ok
2: test_run_lldb_then_gdb (TestDisassembly.DisassembleDriverMainLoop)
   Test disassembly on a large function with lldb vs. gdb. ... 
lldb benchmark: Avg: 0.113008 (Laps: 25, Total Elapsed Time: 2.825201)
gdb benchmark: Avg: 0.225240 (Laps: 25, Total Elapsed Time: 5.631001)
lldb_avg/gdb_avg: 0.501723
ok

----------------------------------------------------------------------
Ran 2 tests in 41.346s

OK

llvm-svn: 142598
2011-10-20 18:43:28 +00:00
Johnny Chen 999f7ab9bd Add '-k' option to the test driver to be able to specify a runhook, which is an lldb command
for the debugger to execute for certain kind of tests (for example, a benchmark).
A list of runhooks can be used to steer the debugger into the desired state before more
actions can be performed.

llvm-svn: 141626
2011-10-11 01:30:27 +00:00
Johnny Chen a10e6c1052 Add '-e' and '-x' options to the test driver to be able to specify an executable (full path)
and the breakpoint specification for the benchmark purpose.  This is used by TestSteppingSpeed.py
to benchmark the lldb stepping speed.  Without '-e' and 'x' specified, the test defaults to
run the built lldb against itself and stopped on Driver::MainLoop, then stepping for 50 times.

rdar://problem/7511193

llvm-svn: 141584
2011-10-10 22:03:44 +00:00
Johnny Chen 9846019114 Minor change for output message (less is better).
llvm-svn: 139924
2011-09-16 18:09:45 +00:00
Johnny Chen 9c0a94ad64 To be more paranoid, check cmd_output before searching in it.
llvm-svn: 139920
2011-09-16 18:03:19 +00:00
Johnny Chen 095ff6fdf6 Rephrase the checkDsymForUUIDIsNotOn() a bit so that in the normal case where automatic
lookup and cacheing of dSYMs is not enabled, the 'defaults read com.apple.DebugSymbols'
output is not shown.

llvm-svn: 139914
2011-09-16 17:50:44 +00:00
Johnny Chen 3cc78e608e Add a check for the test driver to make sure that, on Mac OS X, the automatic lookup and caching
of dSYMs is not turned on before running the test suite.

llvm-svn: 139896
2011-09-16 01:04:26 +00:00
Johnny Chen d890bfc962 Add a new attribute self.lldbHere, representing the fullpath to the 'lldb' executable
built locally from the source tree.  This is distinguished from self.lldbExec, which
can be used by test/benchmarks to measure the performances against other debuggers.

You can use environment variable LLDB_EXEC to specify self.lldbExec to the dotest.py
test driver, otherwise it is going to be populated with self.lldbHere.

Modify the regular tests under test dir, i.e., not test/benchmarks, to use self.lldbHere.
Also modify the benchmarks tests to use self.lldbHere when it needs an 'lldb' executable
with debug info to do the performance measurements.

llvm-svn: 138608
2011-08-26 00:00:01 +00:00
Johnny Chen c5cc625c79 Test driver should also report skipped tests because there were cases when tests were skipped
due to incorrect skip-logic.

llvm-svn: 137678
2011-08-15 23:09:08 +00:00
Johnny Chen 60726fab78 Fix a test driver bug where multiple filter specs (through the -f TestClassName.test_method_name
command line option) belonging to the same Test*.py file are ignored, except for the first one.

llvm-svn: 137540
2011-08-12 23:55:07 +00:00
Johnny Chen 70cb84a0f6 When running dotest.py under pdb (the python debugger), define DOTEST_PDB=YES and
DOTEST_SCRIPT_DIR=/path/leading/to/your/dotest/script/directory.

llvm-svn: 137492
2011-08-12 18:54:11 +00:00
Johnny Chen 19bfde34ed Add a test to check the integrity of the lldb public api directory containing SB*.h headers.
There should be nothing unwanted there and a simpe main.cpp (generated from main.cpp.template)
which includes SB*.h should compile and link with the LLDB framework.

llvm-svn: 136894
2011-08-04 18:17:16 +00:00
Johnny Chen 5ccbccfce0 Add a @benchmarks_test decorator for test method we want to categorize as benchmarks test.
The test driver now takes an option "+b" which enables to run just the benchmarks tests.
By default, tests decorated with the @benchmarks_test decorator do not get run.

Add an example benchmarks test directory which contains nothing for the time being,
just to demonstrate the @benchmarks_test concept.

For example,

$ ./dotest.py -v benchmarks

...

----------------------------------------------------------------------
Collected 2 tests

1: test_with_gdb (TestRepeatedExprs.RepeatedExprssCase)
   Test repeated expressions with gdb. ... skipped 'benchmarks tests'
2: test_with_lldb (TestRepeatedExprs.RepeatedExprssCase)
   Test repeated expressions with lldb. ... skipped 'benchmarks tests'

----------------------------------------------------------------------
Ran 2 tests in 0.047s

OK (skipped=2)
$ ./dotest.py -v +b benchmarks

...

----------------------------------------------------------------------
Collected 2 tests

1: test_with_gdb (TestRepeatedExprs.RepeatedExprssCase)
   Test repeated expressions with gdb. ... running test_with_gdb
benchmarks result for test_with_gdb
ok
2: test_with_lldb (TestRepeatedExprs.RepeatedExprssCase)
   Test repeated expressions with lldb. ... running test_with_lldb
benchmarks result for test_with_lldb
ok

----------------------------------------------------------------------
Ran 2 tests in 0.270s

OK

Also mark some Python API tests which are missing the @python_api_test decorator.

llvm-svn: 136553
2011-07-30 01:39:58 +00:00
Johnny Chen 4a57d122fe Add a redo.py script which takes a session directory name as arg and digs into the directory
to find out the tests which failed/errored and need re-running.  The dotest.py test driver
script is modified to allow specifying multiple -f testclass.testmethod in the command line
to accommodate the redo functionality.

An example,

 $ ./redo.py -n 2011-07-29-11_50_14
adding filterspec: TargetAPITestCase.test_find_global_variables_with_dwarf
adding filterspec: DisasmAPITestCase.test_with_dsym
Running ./dotest.py -v  -f TargetAPITestCase.test_find_global_variables_with_dwarf -f DisasmAPITestCase.test_with_dsym

...

----------------------------------------------------------------------
Collected 2 tests

1: test_with_dsym (TestDisasmAPI.DisasmAPITestCase)
   Exercise getting SBAddress objects, disassembly, and SBAddress APIs. ... ok
2: test_find_global_variables_with_dwarf (TestTargetAPI.TargetAPITestCase)
   Exercise SBTarget.FindGlobalVariables() API. ... ok

----------------------------------------------------------------------
Ran 2 tests in 15.328s

OK

llvm-svn: 136533
2011-07-29 22:54:56 +00:00
Johnny Chen cbdfe81171 Compute svn_info for git-svn bridge, too.
llvm-svn: 133836
2011-06-24 22:52:05 +00:00
Johnny Chen d0abdab8d1 Fix the log channel for gdb-remote. It should be "gdb-remote" now, instead of "process.gdb-remote".
llvm-svn: 133534
2011-06-21 19:25:45 +00:00
Peter Collingbourne 6f2b85221d Set LLDB_SESSION_DIRNAME to absolute path
This is to ensure that session files are written to the correct
directory if the -r option is given.

llvm-svn: 133483
2011-06-20 23:55:53 +00:00
Peter Collingbourne d6824de00e Modify TestHelp to search for LLDB-Info.plist in the correct place
This is required to run the test suite outside of the source tree
or on a case sensitive file system.

llvm-svn: 133460
2011-06-20 19:06:45 +00:00
Peter Collingbourne 28b5eb14c7 Have dotest.py search for test cases in its own directory by default,
rather than the current directory.

This is required to run the test suite outside of the source tree.

llvm-svn: 133458
2011-06-20 19:06:29 +00:00
Peter Collingbourne d736854419 Fix typo: curret -> current
llvm-svn: 132969
2011-06-14 03:55:45 +00:00
Johnny Chen 81a9df5b22 Let's also stick the svn info for the base directory into a file under
the session directory, which also includes the command line used to invoke
the test suite.

llvm-svn: 131501
2011-05-17 22:58:50 +00:00
Johnny Chen 44932b6805 For a test with unexpected success status, we also dump its session info into a unique file.
llvm-svn: 131011
2011-05-06 20:30:22 +00:00
Johnny Chen 0a8a54c117 Docstring.
llvm-svn: 130254
2011-04-26 23:10:51 +00:00
Johnny Chen 4fdcebda1d Docstring changees.
llvm-svn: 130251
2011-04-26 22:53:38 +00:00
Johnny Chen 489b52572c Simplify the parsing of ARCH and Compiler specs.
llvm-svn: 130240
2011-04-26 20:45:00 +00:00
Johnny Chen 71aaf75e3d Use self.TraceOn() API.
llvm-svn: 129936
2011-04-21 20:48:32 +00:00
Jim Ingham 5d3e822926 Have "-h" print just the command arguments, and "-h -v" print the examples as well.
llvm-svn: 129470
2011-04-13 21:11:41 +00:00
Greg Clayton 8b82f087a0 Moved the execution context that was in the Debugger into
the CommandInterpreter where it was always being used.

Make sure that Modules can track their object file offsets correctly to
allow opening of sub object files (like the "__commpage" on darwin).

Modified the Platforms to be able to launch processes. The first part of this
move is the platform soon will become the entity that launches your program
and when it does, it uses a new ProcessLaunchInfo class which encapsulates
all process launching settings. This simplifies the internal APIs needed for
launching. I want to slowly phase out process launching from the process
classes, so for now we can still launch just as we used to, but eventually
the platform is the object that should do the launching.

Modified the Host::LaunchProcess in the MacOSX Host.mm to correctly be able
to launch processes with all of the new eLaunchFlag settings. Modified any
code that was manually launching processes to use the Host::LaunchProcess
functions.

Fixed an issue where lldb_private::Args had implicitly defined copy 
constructors that could do the wrong thing. This has now been fixed by adding
an appropriate copy constructor and assignment operator.

Make sure we don't add empty ModuleSP entries to a module list.

Fixed the commpage module creation on MacOSX, but we still need to train
the MacOSX dynamic loader to not get rid of it when it doesn't have an entry
in the all image infos.

Abstracted many more calls from in ProcessGDBRemote down into the 
GDBRemoteCommunicationClient subclass to make the classes cleaner and more
efficient.

Fixed the default iOS ARM register context to be correct and also added support
for targets that don't support the qThreadStopInfo packet by selecting the
current thread (only if needed) and then sending a stop reply packet.

Debugserver can now start up with a --unix-socket (-u for short) and can 
then bind to port zero and send the port it bound to to a listening process
on the other end. This allows the GDB remote platform to spawn new GDB server
instances (debugserver) to allow platform debugging.

llvm-svn: 129351
2011-04-12 05:54:46 +00:00
Johnny Chen dbfc256eec As a simple measure, output the lldb version and the svn info for
which the testsuite is run against.

llvm-svn: 127782
2011-03-17 00:38:22 +00:00
Johnny Chen b7cfba4cb1 Add pexpect-2.4 (a pure Python module for controlling and automating other programs) to the test directory.
http://pypi.python.org/pypi/pexpect/

llvm-svn: 127484
2011-03-11 20:13:06 +00:00
Johnny Chen cd29b050b7 Add a mechanism to discover the full path of the 'lldb' program into the test driver.
This provides a way to potentially provide conversational interactions with 'lldb' in the test suite.

llvm-svn: 127479
2011-03-11 19:47:23 +00:00
Johnny Chen e0ec9ea5ce Add the ability for the test suite to specify a list of compilers and a list of architectures
on the command line.  For example, use '-A x86_64^i386' to launch the inferior use both x86_64
and i386.

This is an example of building the debuggee using both clang and gcc compiers:

[17:30:46] johnny:/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -C clang^gcc -v -f SourceManagerTestCase.test_modify_source_file_while_debugging

Session logs for test failures/errors will go into directory '2011-03-03-17_31_39'
Command invoked: python ./dotest.py -C clang^gcc -v -f SourceManagerTestCase.test_modify_source_file_while_debugging

Configuration:  compiler=clang
----------------------------------------------------------------------
Collected 1 test

1: test_modify_source_file_while_debugging (TestSourceManager.SourceManagerTestCase)
   Modify a source file while debugging the executable. ... Command 'run' failed!

original content: #include <stdio.h>

int main(int argc, char const *argv[]) {
    printf("Hello world.\n"); // Set break point at this line.
    return 0;
}

new content: #include <stdio.h>

int main(int argc, char const *argv[]) {
    printf("Hello lldb.\n"); // Set break point at this line.
    return 0;
}

os.path.getmtime() after writing new content: 1299202305.0
content restored to: #include <stdio.h>

int main(int argc, char const *argv[]) {
    printf("Hello world.\n"); // Set break point at this line.
    return 0;
}

os.path.getmtime() after restore: 1299202307.0
ok

----------------------------------------------------------------------
Ran 1 test in 8.259s

OK

Configuration:  compiler=gcc
----------------------------------------------------------------------
Collected 1 test

1: test_modify_source_file_while_debugging (TestSourceManager.SourceManagerTestCase)
   Modify a source file while debugging the executable. ... original content: #include <stdio.h>

int main(int argc, char const *argv[]) {
    printf("Hello world.\n"); // Set break point at this line.
    return 0;
}

new content: #include <stdio.h>

int main(int argc, char const *argv[]) {
    printf("Hello lldb.\n"); // Set break point at this line.
    return 0;
}

os.path.getmtime() after writing new content: 1299202307.0
content restored to: #include <stdio.h>

int main(int argc, char const *argv[]) {
    printf("Hello world.\n"); // Set break point at this line.
    return 0;
}

os.path.getmtime() after restore: 1299202309.0
ok

----------------------------------------------------------------------
Ran 1 test in 2.301s

OK
[17:31:49] johnny:/Volumes/data/lldb/svn/trunk/test $ 

llvm-svn: 126979
2011-03-04 01:35:22 +00:00
Greg Clayton 64195a2c8b Abtracted all mach-o and ELF out of ArchSpec. This patch is a modified form
of Stephen Wilson's idea (thanks for the input Stephen!). What I ended up
doing was:
- Got rid of ArchSpec::CPU (which was a generic CPU enumeration that mimics
  the contents of llvm::Triple::ArchType). We now rely upon the llvm::Triple 
  to give us the machine type from llvm::Triple::ArchType.
- There is a new ArchSpec::Core definition which further qualifies the CPU
  core we are dealing with into a single enumeration. If you need support for
  a new Core and want to debug it in LLDB, it must be added to this list. In
  the future we can allow for dynamic core registration, but for now it is
  hard coded.
- The ArchSpec can now be initialized with a llvm::Triple or with a C string
  that represents the triple (it can just be an arch still like "i386").
- The ArchSpec can still initialize itself with a architecture type -- mach-o
  with cpu type and subtype, or ELF with e_machine + e_flags -- and this will
  then get translated into the internal llvm::Triple::ArchSpec + ArchSpec::Core.
  The mach-o cpu type and subtype can be accessed using the getter functions:
  
  uint32_t
  ArchSpec::GetMachOCPUType () const;

  uint32_t
  ArchSpec::GetMachOCPUSubType () const;
  
  But these functions are just converting out internal llvm::Triple::ArchSpec 
  + ArchSpec::Core back into mach-o. Same goes for ELF.

All code has been updated to deal with the changes.

This should abstract us until later when the llvm::TargetSpec stuff gets
finalized and we can then adopt it.

llvm-svn: 126278
2011-02-23 00:35:02 +00:00
Greg Clayton 1971900fdc Don't enable thread safe logging as it currently deadlocks logging.
llvm-svn: 126260
2011-02-22 23:08:31 +00:00
Greg Clayton 76b436e21d Make logs threadsafe (add the -t option) when logging API stuff.
llvm-svn: 126257
2011-02-22 22:59:30 +00:00
Johnny Chen ffb3b51ba8 Refactored the test driver to abstract out the Python sys.path specification for
different build configurations.

llvm-svn: 125584
2011-02-15 18:50:19 +00:00
Greg Clayton c2448f6175 Added support for Xcode 4 build directories.
llvm-svn: 125522
2011-02-14 21:17:06 +00:00
Johnny Chen 74d53758c5 Added comment.
llvm-svn: 124517
2011-01-29 01:21:04 +00:00
Johnny Chen 6573818e38 Add a "-D" option to the test driver which dumps the Python sys.path variable
to be used for the test run.  Could be useful for debugging the setup of the
test environment.

llvm-svn: 124516
2011-01-29 01:16:52 +00:00
Johnny Chen 4f7370f7ae Add "-A arch_spec" and "-C compiler_spec" options to the test driver. Example:
$ ./dotest.py -A i386 -C clang -v -w -t -p TestObjCMethods.py
$ ./dotest.py -A x86_64 -C gcc -v -w types

llvm-svn: 124303
2011-01-26 19:07:42 +00:00
Johnny Chen f249090d7b Print out the command line used to invoke the test suite so I don't get confused
since different options can affect the run time.

llvm-svn: 123843
2011-01-19 19:31:46 +00:00
Johnny Chen 30775d0107 Added special logic to faciliate profiling the test suite run with the cProfile.py
module.

On my MBP running SnowLeopard:

$ DOTEST_PROFILE=YES DOTEST_SCRIPT_DIR=/Volumes/data/lldb/svn/trunk/test /System/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/cProfile.py -o my.profile ./dotest.py -v -w 2> ~/Developer/Log/lldbtest.log

After that, I used the pstats.py module to browse the statistics recorded in the my.profile file. 

llvm-svn: 123807
2011-01-19 02:10:40 +00:00
Johnny Chen 64e3868cf0 Properly indent the short description of the test case to make it align with the
previously added ordinal number of the currently running test case.

llvm-svn: 122922
2011-01-05 22:50:11 +00:00
Johnny Chen 77c81d3ee7 Enhance the test framework to be able to emit a counter value in verbose mode
describing the ordinal number of the currently running test case.

llvm-svn: 122901
2011-01-05 20:24:11 +00:00
Johnny Chen 240f8c7dd1 Do not specify both '-a' and '+a' at the same time.
llvm-svn: 121501
2010-12-10 19:02:23 +00:00
Johnny Chen f3e22ac3c7 Add a '+a' command line option to the test driver to run only the Python API tests.
Add an attribute __python_api_test__ (set to True) to the @python_api_test decorated
test method to distinguish them from the lldb command line tests.

llvm-svn: 121500
2010-12-10 18:52:10 +00:00
Johnny Chen 7c7d936e48 Add an infrastructure to mark the Python APIs only test using a decorator.
Example:

    @python_api_test
    def test_evaluate_expression_python(self):
        """Test SBFrame.EvaluateExpression() API for evaluating an expression."""
    ...

The opposite of Python APIs only test is an lldb command line test, which sends
commands to the lldb command interpreter.  Add a '-a' option to the test driver
to skip Python API only tests.

Modify TestExprs.py to mark a test as @python_api_test and remove an @expectedFailure
decorator as the bug has been fixed.

llvm-svn: 121442
2010-12-10 00:51:23 +00:00
Johnny Chen 025d1b8f95 For SBTarget.Launch()/LaunchProcess(), there's no need to pass an empty string
as the args and the envs to the launched process.

o lldbtest.py:

  Forgot to check in some assertion messages changes for lldbtest.py.

o dotest.py:

  Also add "api" category to the default lldb log option list.

llvm-svn: 121220
2010-12-08 01:25:21 +00:00
Johnny Chen 729c339a1c Add the '-F' option to the test driver, which makes the running of test suite failfast.
As soon as an error or a failure is encountered, it stops the test suite.

llvm-svn: 120837
2010-12-03 19:59:35 +00:00
Johnny Chen eb3411dc41 Log just the thread name, not process/thread id for easier human reading.
llvm-svn: 120751
2010-12-02 23:31:02 +00:00
Johnny Chen 7528af2f86 When logging, enable process and thread ID recordings. Also add 'state' to the lldb
log-categories, so it now becomes "event process expr state".

llvm-svn: 120745
2010-12-02 22:47:52 +00:00
Johnny Chen 20d0b2198b For lldb loggings, prepend log lines with a timestamp. And change the default log options:
o lldb => event process expr
o gdb-remote => packets process

llvm-svn: 120725
2010-12-02 18:35:13 +00:00
Johnny Chen 75739045d3 Add a '-b blacklistFile' option to the test driver to take a file specifying the
test classes or test cases to be excludued from the test suite.

Check in an example blacklist file: blacklist.py:

"""
'blacklist' is a Python dictionary, it stores the mapping of a string describing
either a testclass or a testcase, i.e, testclass.testmethod, to the reason (a
string) it is blacklisted.

Following is an example which states that test class IntegerTypesExprTestCase
should be skipped because 'This test class crashed' and the test case
FoundationTestCase.test_data_type_and_expr_with_dsym should be skipped because
it is 'Temporarily disabled'.

blacklist = {'IntegerTypesExprTestCase': 'This test class crashed',
             'FoundationTestCase.test_data_type_and_expr_with_dsym': 'Temporarily disabled'
             }
"""

blacklist = {}

An example of invoking the test driver and specifying a blacklist file:

./dotest.py -b blacklist.py -v types

This runs the tests under 'types' directory but excludes the tests specified in
balcklist.py.

llvm-svn: 120620
2010-12-01 22:47:54 +00:00
Johnny Chen bbda8ed39a Add comment.
llvm-svn: 120295
2010-11-29 17:52:43 +00:00
Johnny Chen 46389e9ddf When looping through the test suite more than once (repeat count > 1), we should
ignore and not enforce the signleton pattern for the LLDBTestResult class.

llvm-svn: 120294
2010-11-29 17:50:10 +00:00
Johnny Chen ecb072d840 Add an option '-# count' to run the specified test suite for a said number of times.
This is not to be used during normal test suite run, but to be used to stress test
specific test sequences repeatedly.

Example:

./dotest.py -# 3 -v breakpoint_conditions

will repeat the test suite 3 times for tests under the breakpoint_conditions directory.

llvm-svn: 119399
2010-11-16 22:42:58 +00:00
Johnny Chen 5f98696f51 Add help message for the scenario of using lldb.py module distributed by Xcode4
to invoke the test driver.

llvm-svn: 118847
2010-11-11 22:14:56 +00:00
Johnny Chen cc78b0b67f Add more comments on LLDBTestResult class.
llvm-svn: 118655
2010-11-09 23:56:14 +00:00
Johnny Chen bb4f8e6f7b Rephrase the output message about session information; make it both "session logs".
llvm-svn: 118650
2010-11-09 23:42:00 +00:00
Johnny Chen 27f391be61 Better wording of the help text for the '-f testclass.testmethod' option usage.
llvm-svn: 118438
2010-11-08 20:17:04 +00:00
Johnny Chen c06f422d73 Let's also emit the informational message for where to find the session logs for
failed/errored tests at the start of the test run, in case the test run crashes
for any reason.  That way, it is easy to locate the session logs for accumulated
failures/errors.

llvm-svn: 118427
2010-11-08 19:03:37 +00:00
Johnny Chen fa491601b5 Change the test driver so that simply specifying '-f testclass.testmethod' means that
we want to run just the instance of testclass-testmethod combination and nothing else.

Specifying '-g' now will admit the whole .py test module if it does not contain a matching
testclass-testmethod combination at all.

This option arrangement adheres to the RISC principle of making the common cases fast. :-)

rdar://problem/8584914 Can the default for dotest.py's "-g" flag be switched?

llvm-svn: 118392
2010-11-08 01:21:03 +00:00
Johnny Chen 9a6706bca4 Add comment about iterating over possible architecture and compiler combinations.
llvm-svn: 118279
2010-11-05 17:30:53 +00:00
Johnny Chen 110bd0f8cf Now that the test suite has very few expected failures remaining (2), it is a good
idea to also dump the session information for expected failures in addition to
failures and errors (tests with unexpected exceptions).

Modify the LLDBTestResult class which derives from unittest2.TextTestResult to
intercept the addExpectedFailure() method in order to mark the test instance as
needing its session information dumped.

llvm-svn: 118185
2010-11-03 18:17:03 +00:00
Johnny Chen 721f298a60 Add some comment about the default session dir name which is a timestamp.
llvm-svn: 117730
2010-10-29 22:26:38 +00:00
Johnny Chen 8ff5d56c20 The informational message for where to find the session logs for failed/errored
tests should be near the test summary message, instead of way up in the beginning.

llvm-svn: 117726
2010-10-29 22:20:36 +00:00
Johnny Chen bdc5c27894 Fix the test driver as part of the fix for http://llvm.org/bugs/show_bug.cgi?id=8493
File name "2010-10-19-14:10:49.059609" is non-portable.

Do not use ':' in the pathname as it is not permitted on windows platforms.

llvm-svn: 117556
2010-10-28 16:32:13 +00:00
Johnny Chen 8c251f4209 Add some more usage text.
llvm-svn: 117129
2010-10-22 19:00:18 +00:00
Johnny Chen c14e08f437 Rewording of output message.
llvm-svn: 117030
2010-10-21 17:00:35 +00:00
Johnny Chen 096011eebf Add an option '-s session-dir-name' to overwrite the default timestamp-named
directory used to dump the session info for test failures/errors.

Example:

/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -s jason -v array_types

Session info for test errors or failures will go into directory jason
----------------------------------------------------------------------
Collected 4 tests

test_with_dsym_and_python_api (TestArrayTypes.ArrayTypesTestCase)
Use Python APIs to inspect variables with array types. ... ok
test_with_dsym_and_run_command (TestArrayTypes.ArrayTypesTestCase)
Test 'frame variable var_name' on some variables with array types. ... ok
test_with_dwarf_and_python_api (TestArrayTypes.ArrayTypesTestCase)
Use Python APIs to inspect variables with array types. ... ok
test_with_dwarf_and_run_command (TestArrayTypes.ArrayTypesTestCase)
Test 'frame variable var_name' on some variables with array types. ... FAIL

======================================================================
FAIL: test_with_dwarf_and_run_command (TestArrayTypes.ArrayTypesTestCase)
Test 'frame variable var_name' on some variables with array types.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Volumes/data/lldb/svn/trunk/test/array_types/TestArrayTypes.py", line 27, in test_with_dwarf_and_run_command
    self.array_types()
  File "/Volumes/data/lldb/svn/trunk/test/array_types/TestArrayTypes.py", line 62, in array_types
    'stop reason = breakpoint'])
  File "/Volumes/data/lldb/svn/trunk/test/lldbtest.py", line 594, in expect
    self.runCmd(str, trace = (True if trace else False), check = not error)
  File "/Volumes/data/lldb/svn/trunk/test/lldbtest.py", line 564, in runCmd
    msg if msg else CMD_MSG(cmd, True))
AssertionError: False is not True : Command 'thread list' returns successfully

----------------------------------------------------------------------
Ran 4 tests in 3.086s

FAILED (failures=1)
/Volumes/data/lldb/svn/trunk/test $ ls jason
TestArrayTypes.ArrayTypesTestCase.test_with_dwarf_and_run_command.log
/Volumes/data/lldb/svn/trunk/test $ head -10 jason/TestArrayTypes.ArrayTypesTestCase.test_with_dwarf_and_run_command.log 
Session info generated @ Thu Oct 21 09:54:15 2010

os command: [['/bin/sh', '-c', 'make clean; make MAKE_DSYM=NO']]
stdout: rm -rf "a.out" "a.out.dSYM"  main.o main.d
cc -arch x86_64 -gdwarf-2 -O0   -c -o main.o main.c
cc -arch x86_64 -gdwarf-2 -O0  main.o -o "a.out"

stderr: None
retcode: 0

/Volumes/data/lldb/svn/trunk/test $ 

llvm-svn: 117028
2010-10-21 16:55:35 +00:00
Johnny Chen 5858df1315 Get rid of the microsecond field in the timestamp directory name used to dump the
session information files to.  This makes the directory name less intimidating.
Currently, the directory only gets created if there are failures/errors while
running the test suite.

llvm-svn: 116982
2010-10-21 01:03:38 +00:00
Johnny Chen 12672250c1 Add an example of option combination for running a single test method to the help text.
llvm-svn: 116980
2010-10-21 00:47:52 +00:00
Johnny Chen 04d2c5cb0e Modify the test driver and lldbtest.TestBase so that the dumping of session info
now goes into a timestamp-specific directory instead of the previous .session-*
files.

[17:24:34] johnny:/Volumes/data/lldb/svn/trunk $ ls -l test/2010-10-18-16:56:12.935342
total 48
-rw-r--r--  1 johnny  admin  1695 Oct 18 16:56 TestArrayTypes.ArrayTypesTestCase.test_with_dsym_and_run_command.log
-rw-r--r--  1 johnny  admin  1652 Oct 18 16:56 TestArrayTypes.ArrayTypesTestCase.test_with_dwarf_and_run_command.log
-rw-r--r--  1 johnny  admin  2967 Oct 18 16:56 TestBreakpointCommand.BreakpointCommandTestCase.test_with_dsym.log
-rw-r--r--  1 johnny  admin  1648 Oct 18 16:56 TestClassTypes.ClassTypesTestCase.test_with_dwarf_and_run_command.log
-rw-r--r--  1 johnny  admin  1665 Oct 18 16:56 TestClassTypesDisassembly.IterateFrameAndDisassembleTestCase.test_with_dsym_and_python_api.log
-rw-r--r--  1 johnny  admin  3873 Oct 18 16:58 TestFloatTypesExpr.FloatTypesTestCase.test_float_types_with_dsym.log
[17:24:37] johnny:/Volumes/data/lldb/svn/trunk $ 

Also, the dumping happens when a test errored in additioned to when it failed.

llvm-svn: 116778
2010-10-19 00:25:01 +00:00
Johnny Chen 56f7939cdc Simply use the TestBase.markFailure() callback method to set the __failed__ flag.
llvm-svn: 116559
2010-10-15 02:28:13 +00:00
Johnny Chen 150c3cc825 This is an initial version of test driver enhanceent to be able to dump the
session info after a test case failure, allowing more direct inspection of
debugger session which leads to the test failure.

For a simple usage scenario:

[18:06:26] johnny:/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -v . 2> ~/Developer/Log/lldbtest.log 

...

[18:14:43] johnny:/Volumes/data/lldb/svn/trunk/test $ ls -l .session-*
-rw-r--r--  1 johnny  admin  1359 Oct 14 18:06 .session-TestArrayTypes.ArrayTypesTestCase.test_with_dwarf_and_run_command
-rw-r--r--  1 johnny  admin  2054 Oct 14 18:07 .session-TestClassTypes.ClassTypesTestCase.test_with_dsym_and_expr_parser
-rw-r--r--  1 johnny  admin  2055 Oct 14 18:07 .session-TestClassTypes.ClassTypesTestCase.test_with_dwarf_and_expr_parser
-rw-r--r--  1 johnny  admin  1351 Oct 14 17:57 .session-TestClassTypes.ClassTypesTestCase.test_with_dwarf_and_run_command
[18:14:51] johnny:/Volumes/data/lldb/svn/trunk/test $ 

The test case which failed will have its recorded session info dumped to a
.session-* file in the current working directory.  For test suite using
relocated directory, expect to find the .session-* files there.

In this checkin, I also add @skip decorator to the two test methods in
test/foundation/TestObjCMethods.py as it looks like the test suite is
deadlocking when running the tests.  More investigations are needed.

llvm-svn: 116552
2010-10-15 01:18:29 +00:00
Johnny Chen cd8b2bbed2 Change the call within lldb.py to 'SBDebugger.Initialize()' from 'lldb.SBDebugger.Initialize()'.
Inside the lldb module, there's no need (and as a matter of fact, incorrect) to specify the 'lldb'
module name.

Comment out the call to lldb.SBDebugger.Initialize() within the test driver itself, since it is
already done when we import the lldb.py module.

llvm-svn: 116485
2010-10-14 16:36:49 +00:00
Johnny Chen 370799c272 Fix an obvious cut-and-paste error.
llvm-svn: 116343
2010-10-12 21:50:36 +00:00
Johnny Chen 10093aa534 Added the capability for the test driver to split the sys.stderr/sys.stdout into
different configuration-based files using the config file.  For example:

    sys.stderr = open("/tmp/lldbtest-stderr", "w")
    sys.stdout = open("/tmp/lldbtest-stdout", "w")
    compilers = ["gcc", "llvm-gcc"]
    archs = ["x86_64", "i386"]
    split_stderr = True # This will split the stderr into configuration-specific file
    split_stdout = True # This will split the stdout into configuration-specific file

will produce:

/tmp/lldbtest-stderr
/tmp/lldbtest-stderr.arch=i386-compiler=gcc
/tmp/lldbtest-stderr.arch=i386-compiler=llvm-gcc
/tmp/lldbtest-stderr.arch=x86_64-compiler=gcc
/tmp/lldbtest-stderr.arch=x86_64-compiler=llvm-gcc
/tmp/lldbtest-stdout
/tmp/lldbtest-stdout.arch=i386-compiler=gcc
/tmp/lldbtest-stdout.arch=i386-compiler=llvm-gcc
/tmp/lldbtest-stdout.arch=x86_64-compiler=gcc
/tmp/lldbtest-stdout.arch=x86_64-compiler=llvm-gcc

as a result of splitting stderr and stdout.  In addition, each configuration can have
its individual top level relocated directory to house the test files as well as the
intermediate files by using '-r dir' to relocate the tests into a new relocated directory
instead of running the tests in place.

llvm-svn: 116341
2010-10-12 21:35:54 +00:00
Johnny Chen ae19e46ca5 Update comments.
llvm-svn: 116309
2010-10-12 15:53:22 +00:00
Johnny Chen 707b3c9c73 Added the capability for the test driver to relocate the tests and the intermediate
files to a different top level directory than those specified on the command line.

When relocated, the test clanups normally performed afterwards after each test method
and after each test class will not be exercised at all.  This allows for an easier
postmortem analysis of test failures.

Example:

./dotest.py -v -t -r /tmp/lldbtest types

will create a /tmp/lldbtest directory which houses the types directory and its supported
files.

Files modified:

o dotest.py, lldbtest.py:

  Add logic to process '-r dir' option to support relocating the tests to a different
  top level directory instead of exected in place.

o darwin.py, test/types/Makefile:

  The 'make clean' should only clean the minimum .o and .d files.

llvm-svn: 116255
2010-10-11 22:25:46 +00:00
Johnny Chen afef640a9a Help text modification.
llvm-svn: 116206
2010-10-11 16:19:48 +00:00