forked from OSchip/llvm-project
275 lines
13 KiB
Plaintext
275 lines
13 KiB
Plaintext
Let's pick test/settings/TestSettings.py as our example. First, notice the file
|
|
name "TestSettings.py", the Test*.py pattern is the default mechanism that the
|
|
test driver uses for discovery of tests. As to TestSettings.py, it defines a
|
|
class:
|
|
|
|
class SettingsCommandTestCase(TestBase):
|
|
|
|
derived from TestBase, which is defined in test/lldbtest.py and is itself
|
|
derived from Python's unittest framework's TestCase class. See also
|
|
http://docs.python.org/library/unittest.html for more details.
|
|
|
|
To just run the TestSettings.py test, chdir to the lldb test directory, and then
|
|
type the following command:
|
|
|
|
/Volumes/data/lldb/svn/trunk/test $ ./dotest.py settings
|
|
----------------------------------------------------------------------
|
|
Collected 6 tests
|
|
|
|
----------------------------------------------------------------------
|
|
Ran 6 tests in 8.699s
|
|
|
|
OK (expected failures=1)
|
|
/Volumes/data/lldb/svn/trunk/test $
|
|
|
|
Pass '-v' option to the test driver to also output verbose descriptions of the
|
|
individual test cases and their test status:
|
|
|
|
/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -v settings
|
|
----------------------------------------------------------------------
|
|
Collected 6 tests
|
|
|
|
test_set_auto_confirm (TestSettings.SettingsCommandTestCase)
|
|
Test that after 'set auto-confirm true', manual confirmation should not kick in. ... ok
|
|
test_set_output_path (TestSettings.SettingsCommandTestCase)
|
|
Test that setting target.process.output-path for the launched process works. ... expected failure
|
|
test_set_prompt (TestSettings.SettingsCommandTestCase)
|
|
Test that 'set prompt' actually changes the prompt. ... ok
|
|
test_set_term_width (TestSettings.SettingsCommandTestCase)
|
|
Test that 'set term-width' actually changes the term-width. ... ok
|
|
test_with_dsym (TestSettings.SettingsCommandTestCase)
|
|
Test that run-args and env-vars are passed to the launched process. ... ok
|
|
test_with_dwarf (TestSettings.SettingsCommandTestCase)
|
|
Test that run-args and env-vars are passed to the launched process. ... ok
|
|
|
|
----------------------------------------------------------------------
|
|
Ran 6 tests in 5.735s
|
|
|
|
OK (expected failures=1)
|
|
/Volumes/data/lldb/svn/trunk/test $
|
|
|
|
Underneath, the '-v' option passes keyword argument verbosity=2 to the
|
|
Python's unittest.TextTestRunner (see also
|
|
http://docs.python.org/library/unittest.html#unittest.TextTestRunner). For very
|
|
detailed descriptions about what's going on during the test, pass '-t' to the
|
|
test driver, which asks the test driver to trace the commands executed and to
|
|
display their output. For brevity, the '-t' output is not included here.
|
|
|
|
Notice the 'expected failures=1' message at the end of the run. This is because
|
|
of a bug currently in lldb such that setting target.process.output-path to
|
|
'stdout.txt' does not have any effect on the redirection of the standard output
|
|
of the subsequent launched process. We are using unittest2 (a backport of new
|
|
unittest features for Python 2.4-2.6) to decorate (mark) the particular test
|
|
method as such:
|
|
|
|
@unittest2.expectedFailure
|
|
# rdar://problem/8435794
|
|
# settings set target.process.output-path does not seem to work
|
|
def test_set_output_path(self):
|
|
|
|
See http://pypi.python.org/pypi/unittest2 for more details.
|
|
|
|
Now let's look inside the test method:
|
|
|
|
def test_set_output_path(self):
|
|
"""Test that setting target.process.output-path for the launched process works."""
|
|
self.build()
|
|
|
|
exe = os.path.join(os.getcwd(), "a.out")
|
|
self.runCmd("file " + exe, CURRENT_EXECUTABLE_SET)
|
|
|
|
# Set the output-path and verify it is set.
|
|
self.runCmd("settings set target.process.output-path 'stdout.txt'")
|
|
self.expect("settings show target.process.output-path",
|
|
startstr = "target.process.output-path (string) = 'stdout.txt'")
|
|
|
|
self.runCmd("run", RUN_SUCCEEDED)
|
|
|
|
# The 'stdout.txt' file should now exist.
|
|
self.assertTrue(os.path.isfile("stdout.txt"),
|
|
"'stdout.txt' exists due to target.process.output-path.")
|
|
|
|
# Read the output file produced by running the program.
|
|
with open('stdout.txt', 'r') as f:
|
|
output = f.read()
|
|
|
|
self.expect(output, exe=False,
|
|
startstr = "This message should go to standard out.")
|
|
|
|
The self.build() statement is used to build a binary for this
|
|
test instance. This will build the binary for the current debug info format. If
|
|
we wanted to avoid running the test for every supported debug info format we
|
|
could annotate it with @no_debug_info_test. The test would then only be run for
|
|
the default format. The logic for building a test binary resides in the builder
|
|
modules (packages/Python/lldbsuite/test/builders/builder.py)
|
|
|
|
After the binary is built, it is time to specify the file to be used as the main
|
|
executable by lldb:
|
|
|
|
# Construct the path to a file "a.out" inside the test's build folder.
|
|
exe = self.getBuildArtifact("a.out")
|
|
self.runCmd("file " + exe, CURRENT_EXECUTABLE_SET)
|
|
|
|
This is where the attribute assignment:
|
|
|
|
class SettingsCommandTestCase(TestBase):
|
|
|
|
mydir = "settings"
|
|
|
|
which happens right after the SettingsCommandTestCase class declaration comes
|
|
into place. It specifies the relative directory to the top level 'test' so that
|
|
the test harness can change its working directory in order to find the
|
|
executable as well as the source code files. The runCmd() method is defined in
|
|
the TestBase base class (within test/lldbtest.py) and its purpose is to pass the
|
|
specified command to the lldb command interpreter. It's like you're typing the
|
|
command within an interactive lldb session.
|
|
|
|
The CURRENT_EXECUTABLE_SET is an assert message defined in the lldbtest module
|
|
so that it can be reused from other test modules.
|
|
|
|
By default, the runCmd() is going to check the return status of the command
|
|
execution and fails the test if it is not a success. The assert message, in our
|
|
case CURRENT_EXECUTABLE_SET, is used in the exception printout if this happens.
|
|
|
|
There are cases when we don't care about the return status from the command
|
|
execution. This can be accomplished by passing the keyword argument pair
|
|
'check=False' to the method.
|
|
|
|
After the current executable is set, we'll then execute two more commands:
|
|
|
|
# Set the output-path and verify it is set.
|
|
stdout = self.getBuildArtifact('stdout.txt')
|
|
self.runCmd("settings set target.process.output-path '%s'" %stdout)
|
|
self.expect("settings show target.process.output-path",
|
|
SETTING_MSG("target.process.output-path"),
|
|
startstr = "target.process.output-path (string) = '.*stdout.txt'")
|
|
|
|
The first uses the 'settings set' command to set the static setting
|
|
target.process.output-path to be 'stdout.txt', instead of the default
|
|
'/dev/stdout'. We then immediately issue a 'settings show' command to check
|
|
that, indeed, the setting did take place. Notice that we use a new method
|
|
expect() to accomplish the task, which in effect issues a runCmd() behind the
|
|
door and grabs the output from the command execution and expects to match the
|
|
start string of the output against what we pass in as the value of the keyword
|
|
argument pair:
|
|
|
|
startstr = "target.process.output-path (string) = '%s'" %stdout
|
|
|
|
Take a look at TestBase.expect() within lldbtest.py for more details. Among
|
|
other things, it can also match against a list of regexp patterns as well as a
|
|
list of sub strings. And it can also perform negative matching, i.e., instead
|
|
of expecting something from the output of command execution, it can perform the
|
|
action of 'not expecting' something.
|
|
|
|
This will launch/run the program:
|
|
|
|
self.runCmd("run", RUN_SUCCEEDED)
|
|
|
|
And this asserts that the file 'stdout.txt' should be present after running the
|
|
program.
|
|
|
|
# The 'stdout.txt' file should now exist.
|
|
self.assertTrue(os.path.isfile(stdout),
|
|
"stdout.txt' exists due to target.process.output-path.")
|
|
|
|
Also take a look at main.cpp which emits some message to the stdout. Now, if we
|
|
pass this assertion, it's time to examine the contents of the file to make sure
|
|
it contains the same message as programmed in main.cpp:
|
|
|
|
# Read the output file produced by running the program.
|
|
with open(stdout, 'r') as f:
|
|
output = f.read()
|
|
|
|
self.expect(output, exe=False,
|
|
startstr = "This message should go to standard out.")
|
|
|
|
We open the file and read its contents into output, then issue an expect()
|
|
method. The 'exe=False' keyword argument pair tells expect() that don't try to
|
|
execute the first arg as a command at all. Instead, treat it as a string to
|
|
match against whatever is thrown in as keyword argument pairs!
|
|
|
|
There are also other test methods present in the TestSettings.py mode:
|
|
test_set_prompt(), test_set_term_width(), test_set_auto_confirm(),
|
|
test_with_dsym(), and test_with_dwarf(). We are using the default test loader
|
|
from unittest framework, which uses the 'test' method name prefix to identify
|
|
test methods automatically.
|
|
|
|
This finishes the walkthrough of the test method test_set_output_path(self).
|
|
Before we say goodbye, notice the little method definition at the top of the
|
|
file:
|
|
|
|
@classmethod
|
|
def classCleanup(cls):
|
|
system(["/bin/sh", "-c", "rm -f "+self.getBuildArtifact("output.txt")])
|
|
system(["/bin/sh", "-c", "rm -f "+self.getBuildArtifact("stdout.txt")])
|
|
|
|
This is a classmethod (as shown by the @classmethod decorator) which allows the
|
|
individual test class to perform cleanup actions after the test harness finishes
|
|
with the particular test class. This is part of the so-called test fixture in
|
|
the unittest framework. From http://docs.python.org/library/unittest.html:
|
|
|
|
A test fixture represents the preparation needed to perform one or more tests,
|
|
and any associate cleanup actions. This may involve, for example, creating
|
|
temporary or proxy databases, directories, or starting a server process.
|
|
|
|
The TestBase class uses such fixture with setUp(self), tearDown(self),
|
|
setUpClass(cls), and tearDownClass(cls). And within teraDownClass(cls), it
|
|
checks whether the current class has an attribute named 'classCleanup', and
|
|
executes as a method if present. In this particular case, the classCleanup()
|
|
calls a utility function system() defined in lldbtest.py in order to remove the
|
|
files created by running the program as the tests are executed.
|
|
|
|
This system() function uses the Python subprocess module to spawn the process
|
|
and to retrieve its results. If the test instance passes the keyword argument
|
|
pair 'sender=self', the detailed command execution through the operating system
|
|
also gets recorded in a session object. If the test instance fails or errors,
|
|
the session info automatically gets dumped to a file grouped under a directory
|
|
named after the timestamp of the particular test suite run.
|
|
|
|
For simple cases, look for the timestamp directory in the same directory of the
|
|
test driver program dotest.py. For example, if we comment out the
|
|
@expectedFailure decorator for TestSettings.py, and then run the test module:
|
|
|
|
/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -v settings
|
|
----------------------------------------------------------------------
|
|
Collected 6 tests
|
|
|
|
test_set_auto_confirm (TestSettings.SettingsCommandTestCase)
|
|
Test that after 'set auto-confirm true', manual confirmation should not kick in. ... ok
|
|
test_set_output_path (TestSettings.SettingsCommandTestCase)
|
|
Test that setting target.process.output-path for the launched process works. ... FAIL
|
|
test_set_prompt (TestSettings.SettingsCommandTestCase)
|
|
Test that 'set prompt' actually changes the prompt. ... ok
|
|
test_set_term_width (TestSettings.SettingsCommandTestCase)
|
|
Test that 'set term-width' actually changes the term-width. ... ok
|
|
test_with_dsym (TestSettings.SettingsCommandTestCase)
|
|
Test that run-args and env-vars are passed to the launched process. ... ok
|
|
test_with_dwarf (TestSettings.SettingsCommandTestCase)
|
|
Test that run-args and env-vars are passed to the launched process. ... ok
|
|
|
|
======================================================================
|
|
FAIL: test_set_output_path (TestSettings.SettingsCommandTestCase)
|
|
Test that setting target.process.output-path for the launched process works.
|
|
----------------------------------------------------------------------
|
|
Traceback (most recent call last):
|
|
File "/Volumes/data/lldb/svn/trunk/test/settings/TestSettings.py", line 125, in test_set_output_path
|
|
"'stdout.txt' exists due to target.process.output-path.")
|
|
AssertionError: False is not True : 'stdout.txt' exists due to target.process.output-path.
|
|
|
|
----------------------------------------------------------------------
|
|
Ran 6 tests in 8.219s
|
|
|
|
FAILED (failures=1)
|
|
/Volumes/data/lldb/svn/trunk/test $ ls 2010-10-19-14:10:49.059609
|
|
|
|
NOTE: This directory name has been changed to not contain the ':' character
|
|
which is not allowed in windows platforms. We'll change the ':' to '_'
|
|
and get rid of the microsecond resolution by modifying the test driver.
|
|
|
|
TestSettings.SettingsCommandTestCase.test_set_output_path.log
|
|
/Volumes/data/lldb/svn/trunk/test $
|
|
|
|
We get one failure and a timestamp directory 2010-10-19-14:10:49.059609.
|
|
For education purposes, the directory and its contents are reproduced here in
|
|
the same directory as the current file.
|