Commit Graph

37 Commits

Author SHA1 Message Date
Enrico Granata 7a2421095c Add a benchmark test case that shows how much slower repeat 'continue' commands are than going through the SB API directly
llvm-svn: 226852
2015-01-22 19:33:53 +00:00
Zachary Turner 0c426319f6 XFAIL pexpect tests on Windows.
At some point we will need to either provide a pexpect equivalent
on Windows, or provide some other method of doing out-of-process
tests.

Even with a pexpect replacement, it may be worth re-evaluating
some of these tests to see if they would be better served as
in-process tests.  The larger issue of coming up with a pexpect
replacement on Windows is tracked in http://llvm.org/pr22274.

llvm-svn: 226614
2015-01-20 22:36:03 +00:00
Zachary Turner 045fde58d1 Fixes a number of issue related to test portability on Windows.
99% of this CL is simply moving calls to "import pexpect" to a more
narrow scope - i.e. the function that actually runs a particular
test.  This way the test suite can run on Windows, which doesn't have
pexpect, and the individual tests that use pexpect can be disabled on
a platform-specific basis.

Additionally, this CL fixes a few other cases of non-portability.
Notably, using "ps" to get the command line, and os.uname() to
determine the architecture don't work on Windows.  Finally, this
also adds a stubbed out builder_win32 module.

The full test suite runs correctly on Windows after this CL, although
there is still some work remaining on the C++ side to fix one-shot
script commands from LLDB (e.g. script print "foo"), which currently
deadlock.

Reviewed by: Todd Fiala

Differential Revision: http://reviews.llvm.org/D4573

llvm-svn: 213343
2014-07-18 01:02:02 +00:00
Greg Clayton 4570d3eba0 Massive test suite cleanup to stop everyone from manually having to compute "mydir" inside each test case.
This has led to many test suite failures because of copy and paste where new test cases were based off of other test cases and the "mydir" variable wasn't updated.

Now you can call your superclasses "compute_mydir()" function with "__file__" as the sole argument and the relative path will be computed for you. 

llvm-svn: 196985
2013-12-10 23:19:29 +00:00
Johnny Chen 7debd16c95 Test file renaming.
llvm-svn: 155369
2012-04-23 17:54:40 +00:00
Johnny Chen a9e77784e0 Move some print stmts to the test method, where they get printed only if the test is qualified to run
under the current test driver run configuration.

llvm-svn: 146320
2011-12-10 07:18:11 +00:00
Johnny Chen 1669f67776 Modified the script to have the flexibility of specifying the gdb executable path
for use in the benchmark against lldb's disassembly speed.  Note that the lldb
executable path can already be specified using the LLDB_EXEC env variable.

rdar://problem/7511194

llvm-svn: 146050
2011-12-07 19:27:06 +00:00
Johnny Chen c2d6974702 This benchmark is meant to run the locally built 'lldb' binary, not the binary on the PATH env variable.
llvm-svn: 143169
2011-10-28 00:46:47 +00:00
Johnny Chen 66d362e9bf Establish a baseline for bench.py score by using a fixed lldb executable as the
inferior program for the lldb debugger to operate on.  The fixed lldb executable
corresponds to r142902.

Plus some minor modifications to the test benchmark to conform to way bench.py
is meant to be invoked.

llvm-svn: 143075
2011-10-26 22:58:02 +00:00
Johnny Chen 6e5b330fb4 Add another metric for startup delay -- run to breakpoint, which measures the time from issuing
the run command till the first breakpoint hit.  Example:

 $ ./dotest.py -v +b -n -x '-F Driver::MainLoop()' -p TestStartupDelays.py
1: test_startup_delay (TestStartupDelays.StartupDelaysBench)
   Test start up delays creating a target and setting a breakpoint. ... 
lldb startup delay (create fresh target) benchmark: Avg: 0.124496 (Laps: 30, Total Elapsed Time: 3.734883)
lldb startup delay (set first breakpoint) benchmark: Avg: 0.220828 (Laps: 30, Total Elapsed Time: 6.624847)
lldb startup delay (run to breakpoint) benchmark: Avg: 0.478159 (Laps: 30, Total Elapsed Time: 14.344774)
ok

llvm-svn: 142993
2011-10-26 00:35:10 +00:00
Johnny Chen fc9e79fb95 Benchmark the turnaround time starting a debugger and run to the breakpoint with lldb vs. gdb.
An example (with /Developer/usr/bin/lldb vs. /usr/bin/gdb):

[13:05:04] johnny:/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -v +b -n -p TestCompileRunToBreakpointTurnaround.py
1: test_run_lldb_then_gdb (TestCompileRunToBreakpointTurnaround.CompileRunToBreakpointBench)
   Benchmark turnaround time with lldb vs. gdb. ... 
lldb turnaround benchmark: Avg: 4.574600 (Laps: 3, Total Elapsed Time: 13.723799)
gdb turnaround benchmark: Avg: 7.966713 (Laps: 3, Total Elapsed Time: 23.900139)
lldb_avg/gdb_avg: 0.574214
ok

----------------------------------------------------------------------
Ran 1 test in 55.462s

OK

llvm-svn: 142949
2011-10-25 20:08:03 +00:00
Johnny Chen b8da426285 Add bench.py as a driver script to run some benchmarks on lldb.
Add benchmarks for expression evaluations (TestExpressionCmd.py) and disassembly (TestDoAttachThenDisassembly.py).

An example:
[17:45:55] johnny:/Volumes/data/lldb/svn/trunk/test $ ./bench.py 2>&1 | grep -P '^lldb.*benchmark:'
lldb startup delay (create fresh target) benchmark: Avg: 0.104274 (Laps: 30, Total Elapsed Time: 3.128214)
lldb startup delay (set first breakpoint) benchmark: Avg: 0.102216 (Laps: 30, Total Elapsed Time: 3.066470)
lldb frame variable benchmark: Avg: 1.649162 (Laps: 20, Total Elapsed Time: 32.983245)
lldb stepping benchmark: Avg: 0.104409 (Laps: 50, Total Elapsed Time: 5.220461)
lldb expr cmd benchmark: Avg: 0.206774 (Laps: 25, Total Elapsed Time: 5.169350)
lldb disassembly benchmark: Avg: 0.089086 (Laps: 10, Total Elapsed Time: 0.890859)

llvm-svn: 142708
2011-10-22 00:57:05 +00:00
Johnny Chen 31fdfb1f33 Add a benchmark for measuring the response time of the 'frame variable' command.
Example (start the lldb inferior, break at the Driver::MainLoop() function, and
issue 'frame variable'):

 $ ./dotest.py -v +b -x '-F Driver::MainLoop()' -n -p TestFrameVariableResponse.py
----------------------------------------------------------------------
Collected 1 test

1: test_startup_delay (TestFrameVariableResponse.FrameVariableResponseBench)
   Test response time for the 'frame variable' command. ... 
lldb frame variable benchmark: Avg: 1.636897 (Laps: 20, Total Elapsed Time: 32.737944)
ok

----------------------------------------------------------------------
Ran 1 test in 65.105s

OK

llvm-svn: 142678
2011-10-21 20:19:51 +00:00
Johnny Chen 99904b33b7 Rephrase benchmark output display.
llvm-svn: 142676
2011-10-21 20:11:40 +00:00
Johnny Chen edb677c3c7 Fix wrong directory name.
llvm-svn: 142629
2011-10-21 01:09:29 +00:00
Johnny Chen 7f63fdaa74 Add a benchmark for measuring start up delays of lldb, including:
o create a fresh target; and
o set the first breakpoint

Example (using lldb to set a breakpoint on lldb's Driver::MainLoop function):

./dotest.py -v +b -x '-F Driver::MainLoop()' -p TestStartupDelays.py

...

1: test_startup_delay (TestStartupDelays.StartupDelaysBench)
   Test start up delays creating a target and setting a breakpoint. ... 
lldb startup delays benchmark:
create fresh target: Avg: 0.106732 (Laps: 15, Total Elapsed Time: 1.600985)
set first breakpoint: Avg: 0.102589 (Laps: 15, Total Elapsed Time: 1.538832)
ok

llvm-svn: 142628
2011-10-20 22:37:45 +00:00
Johnny Chen 8241d378f8 Directory renaming: example -> expression.
llvm-svn: 142602
2011-10-20 18:58:28 +00:00
Johnny Chen f4ca4b5f57 Directory renaming: example -> expression.
llvm-svn: 142601
2011-10-20 18:57:04 +00:00
Johnny Chen 38f9daa303 Parameterize the iteration count used when running benchmarks, instead of hard-coded inside the test case.
Add a '-y count' option to the test driver for this purpose.  An example:

 $  ./dotest.py -v -y 25 +b -p TestDisassembly.py

...

----------------------------------------------------------------------
Collected 2 tests

1: test_run_gdb_then_lldb (TestDisassembly.DisassembleDriverMainLoop)
   Test disassembly on a large function with lldb vs. gdb. ... 
gdb benchmark: Avg: 0.226305 (Laps: 25, Total Elapsed Time: 5.657614)
lldb benchmark: Avg: 0.113864 (Laps: 25, Total Elapsed Time: 2.846606)
lldb_avg/gdb_avg: 0.503146
ok
2: test_run_lldb_then_gdb (TestDisassembly.DisassembleDriverMainLoop)
   Test disassembly on a large function with lldb vs. gdb. ... 
lldb benchmark: Avg: 0.113008 (Laps: 25, Total Elapsed Time: 2.825201)
gdb benchmark: Avg: 0.225240 (Laps: 25, Total Elapsed Time: 5.631001)
lldb_avg/gdb_avg: 0.501723
ok

----------------------------------------------------------------------
Ran 2 tests in 41.346s

OK

llvm-svn: 142598
2011-10-20 18:43:28 +00:00
Johnny Chen 5a328cc8c6 Remove stale code.
llvm-svn: 142595
2011-10-20 17:49:44 +00:00
Johnny Chen 29b3f6b616 Remove stale code.
llvm-svn: 142594
2011-10-20 17:45:39 +00:00
Johnny Chen 2a80858903 Modify lldbtest.Base.runHooks() to now take the following keyword arguments:
child=None, child_prompt=None, use_cmd_api=False

By default, expect a pexpect spawned child and child prompt to be
supplied (use_cmd_api=False).  If use_cmd_api is true, ignore the child
and child prompt and use self.runCmd() to run the hooks one by one.

Modify existing client to reflect the change.

llvm-svn: 142532
2011-10-19 16:48:07 +00:00
Johnny Chen a737ba55af Extract the run hooks functionality into the base class lldbtest.Base.
llvm-svn: 142469
2011-10-19 01:06:21 +00:00
Johnny Chen 8a7f3f5bac Add a more generic stepping benchmark, which uses the '-k' option of the test driver
to be able to specify the runhook(s) to bring the debug session to a certain state
before running the benchmarking logic.  An example,

./dotest.py -v -t +b -k 'process attach -n Mail' -k 'thread backtrace all' -p TestRunHooksThenSteppings.py

spawns lldb, attaches to the 'Mail' application, does a backtrace for all threads, and then
runs the benchmark to step the inferior multiple times.

llvm-svn: 141740
2011-10-11 23:22:02 +00:00
Johnny Chen a10e6c1052 Add '-e' and '-x' options to the test driver to be able to specify an executable (full path)
and the breakpoint specification for the benchmark purpose.  This is used by TestSteppingSpeed.py
to benchmark the lldb stepping speed.  Without '-e' and 'x' specified, the test defaults to
run the built lldb against itself and stopped on Driver::MainLoop, then stepping for 50 times.

rdar://problem/7511193

llvm-svn: 141584
2011-10-10 22:03:44 +00:00
Johnny Chen ebe51726b8 If we spawn an lldb process for test (via pexpect), do not load the init file unless told otherwise.
Set up self.lldbOption to be "--no-lldbibit" unless env variable NO_LLDBIBIT is defined and equals "NO".
Also add "-nx" to gdb spawned.

llvm-svn: 141384
2011-10-07 19:21:09 +00:00
Johnny Chen d890bfc962 Add a new attribute self.lldbHere, representing the fullpath to the 'lldb' executable
built locally from the source tree.  This is distinguished from self.lldbExec, which
can be used by test/benchmarks to measure the performances against other debuggers.

You can use environment variable LLDB_EXEC to specify self.lldbExec to the dotest.py
test driver, otherwise it is going to be populated with self.lldbHere.

Modify the regular tests under test dir, i.e., not test/benchmarks, to use self.lldbHere.
Also modify the benchmarks tests to use self.lldbHere when it needs an 'lldb' executable
with debug info to do the performance measurements.

llvm-svn: 138608
2011-08-26 00:00:01 +00:00
Johnny Chen 250a501702 Check in a customized benchmark which compares the Xcode 4.1 vs. Xcode 4.2's gdb disassembly speed
on lldb's Driver::MainLoop function which is ~1190 lines of x86 assembly code.  This file is not
exercised during the normal test suite run, i.e., no +b option specified.  So it should be ok.

The following is the benchmark result on my MBP running OSX Lion:

[17:38:46] johnny:/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -v +b -p TestFlintVsSlate
/Volumes/data/lldb/svn/trunk/build/Debug
LLDB-71
Path: /Volumes/data/lldb/svn/trunk
URL: https://johnny@llvm.org/svn/llvm-project/lldb/trunk
Repository Root: https://johnny@llvm.org/svn/llvm-project
Repository UUID: 91177308-0d34-0410-b5e6-96231b3b80d8
Revision: 137008
Node Kind: directory
Schedule: normal
Last Changed Author: gclayton
Last Changed Rev: 137008
Last Changed Date: 2011-08-05 17:50:36 -0700 (Fri, 05 Aug 2011)



Session logs for test failures/errors/unexpected successes will go into directory '2011-08-08-17_38_52'
Command invoked: python ./dotest.py -v +b -p TestFlintVsSlate
----------------------------------------------------------------------
Collected 2 tests

1: test_run_41_then_42 (TestFlintVsSlateGDBDisassembly.FlintVsSlateGDBDisassembly)
   Test disassembly on a large function with 4.1 vs. 4.2's gdb. ... 
4.1 gdb benchmark: Avg: 0.205623 (Laps: 5, Total Elapsed Time: 1.028113)
4.2 gdb benchmark: Avg: 0.201970 (Laps: 5, Total Elapsed Time: 1.009849)
gdb_42_avg/gdb_41_avg: 0.982236
ok
2: test_run_42_then_41 (TestFlintVsSlateGDBDisassembly.FlintVsSlateGDBDisassembly)
   Test disassembly on a large function with 4.1 vs. 4.2's gdb. ... 
4.2 gdb benchmark: Avg: 0.202602 (Laps: 5, Total Elapsed Time: 1.013012)
4.1 gdb benchmark: Avg: 0.204418 (Laps: 5, Total Elapsed Time: 1.022089)
gdb_42_avg/gdb_41_avg: 0.991119
ok

----------------------------------------------------------------------
Ran 2 tests in 15.688s

OK

llvm-svn: 137092
2011-08-09 00:56:07 +00:00
Johnny Chen 0affe0213d Print out the stopwatch (which contains laps, total elapsed time, and average)
instead of just the average.

llvm-svn: 136932
2011-08-04 22:05:36 +00:00
Johnny Chen e464282f29 Add a benchmark comparing lldb vs. gdb with disassembly on a large function (lldb's Driver::MainLoop()).
Sample run on my OSX Lion (MacBook Pro):

1: test_run_gdb_then_lldb (TestDisassembly.DisassembleDriverMainLoop)
   Test disassembly on a large function with lldb vs. gdb. ... 
gdb benchmark: Avg: 0.201802 (Laps: 5, Total Elapsed Time: 1.009008)
lldb benchmark: Avg: 0.109569 (Laps: 5, Total Elapsed Time: 0.547843)
lldb_avg/gdb_avg: 0.542952
ok
2: test_run_lldb_then_gdb (TestDisassembly.DisassembleDriverMainLoop)
   Test disassembly on a large function with lldb vs. gdb. ... 
lldb benchmark: Avg: 0.109580 (Laps: 5, Total Elapsed Time: 0.547902)
gdb benchmark: Avg: 0.201587 (Laps: 5, Total Elapsed Time: 1.007936)
lldb_avg/gdb_avg: 0.543588
ok

llvm-svn: 136931
2011-08-04 22:01:21 +00:00
Johnny Chen c99dd97fc5 Fix typos.
llvm-svn: 136809
2011-08-03 20:40:35 +00:00
Johnny Chen a8d4df429a Add license header comment.
llvm-svn: 136808
2011-08-03 20:39:26 +00:00
Johnny Chen aaa82ff9ad Add the real benchmarks comparing lldb against gdb for repeated expression evaluations.
Modify lldbbench.py so that lldbtest.line_number() utility function is available to
BenchBase client as just line_number(), and modify lldbtest.py so that self.lldbExec
(the full path for the 'lldb' executable) is available to BenchBase client as well.

An example run of the test case on my MacBook Pro running Lion:

1: test_compare_lldb_to_gdb (TestRepeatedExprs.RepeatedExprsCase)
   Test repeated expressions with lldb vs. gdb. ... 
lldb_avg: 0.204339
gdb_avg: 0.205721
lldb_avg/gdb_avg: 0.993284
ok

llvm-svn: 136740
2011-08-02 22:54:37 +00:00
Johnny Chen da0384c90e Simple renaming: self.swatch -> self.stopwatch.
llvm-svn: 136666
2011-08-02 00:50:55 +00:00
Johnny Chen 2352af85f9 Add a Stopwatch utility class to lldbench.py module and initialize an instance of
Stopwatch (self.swatch) within the BenchBase's setUp() instance method to be available
to all the child classes.

Use self.swatch to measure elapsed time in TestRepeatedExprs.py, which needs to be
modified later on to actually measure repeated expression evaluations within the
context of lldb as well as gdb.

llvm-svn: 136664
2011-08-02 00:43:09 +00:00
Johnny Chen 985e740cd9 Add an abstract base class called BenchBase to be inherited by benchmark tests.
Modify the example TestRepeatedExprs.py to use BenchBase, instead.

llvm-svn: 136649
2011-08-01 21:13:26 +00:00
Johnny Chen 5ccbccfce0 Add a @benchmarks_test decorator for test method we want to categorize as benchmarks test.
The test driver now takes an option "+b" which enables to run just the benchmarks tests.
By default, tests decorated with the @benchmarks_test decorator do not get run.

Add an example benchmarks test directory which contains nothing for the time being,
just to demonstrate the @benchmarks_test concept.

For example,

$ ./dotest.py -v benchmarks

...

----------------------------------------------------------------------
Collected 2 tests

1: test_with_gdb (TestRepeatedExprs.RepeatedExprssCase)
   Test repeated expressions with gdb. ... skipped 'benchmarks tests'
2: test_with_lldb (TestRepeatedExprs.RepeatedExprssCase)
   Test repeated expressions with lldb. ... skipped 'benchmarks tests'

----------------------------------------------------------------------
Ran 2 tests in 0.047s

OK (skipped=2)
$ ./dotest.py -v +b benchmarks

...

----------------------------------------------------------------------
Collected 2 tests

1: test_with_gdb (TestRepeatedExprs.RepeatedExprssCase)
   Test repeated expressions with gdb. ... running test_with_gdb
benchmarks result for test_with_gdb
ok
2: test_with_lldb (TestRepeatedExprs.RepeatedExprssCase)
   Test repeated expressions with lldb. ... running test_with_lldb
benchmarks result for test_with_lldb
ok

----------------------------------------------------------------------
Ran 2 tests in 0.270s

OK

Also mark some Python API tests which are missing the @python_api_test decorator.

llvm-svn: 136553
2011-07-30 01:39:58 +00:00