Finish the first draft of a detailed walkthrough of an lldb test method.

llvm-svn: 116865

lldb/docs/testsuite/2010-10-19-14:10:49.059609/TestSettings.SettingsCommandTestCase.test_set_output_path.log

@@ -0,0 +1,43 @@
+
+os command: [['/bin/sh', '-c', 'make clean; make']]
+stdout: rm -rf "a.out" "a.out.dSYM"  main.o main.d
+g++ -arch x86_64 -gdwarf-2 -O0   -c -o main.o main.cpp
+g++ -arch x86_64 -gdwarf-2 -O0  main.o -o "a.out"
+/usr/bin/dsymutil  -o "a.out.dSYM" "a.out"
+
+stderr: None
+retcode: 0
+
+
+runCmd: file /Volumes/data/lldb/svn/trunk/test/settings/a.out
+output: Current executable set to '/Volumes/data/lldb/svn/trunk/test/settings/a.out' (x86_64).
+
+
+runCmd: settings set target.process.output-path 'stdout.txt'
+output: 
+
+runCmd: settings show target.process.output-path
+output: target.process.output-path (string) = 'stdout.txt'
+
+
+Expecting start string: target.process.output-path (string) = 'stdout.txt'
+Matched
+
+runCmd: run
+output: Process 43533 launched: '/Volumes/data/lldb/svn/trunk/test/settings/a.out' (x86_64)
+
+
+FAIL
+
+runCmd: process kill
+check of return status not required
+runCmd failed!
+error: Process must be launched.
+
+
+Traceback (most recent call last):
+  File "/Volumes/data/lldb/svn/trunk/test/settings/TestSettings.py", line 125, in test_set_output_path
+    "'stdout.txt' exists due to target.process.output-path.")
+AssertionError: False is not True : 'stdout.txt' exists due to target.process.output-path.
+
+

lldb/docs/testsuite/a-detailed-walkthrough.txt

@@ -161,4 +161,144 @@ the specified command to the lldb command interpreter.  It's like you're typing
 the command within an interactive lldb session.
 
 The CURRENT_EXECUTABLE_SET is an assert message defined in the lldbtest module
-so that it can reused from other test modules.
+so that it can be reused from other test modules.
+
+By default, the runCmd() is going to check the return status of the command
+execution and fails the test if it is not a success.  The assert message, in our
+case CURRENT_EXECUTABLE_SET, is used in the exception printout if this happens.
+
+There are cases when we don't care about the return status from the command
+execution.  This can be accomplished by passing the keyword argument pair
+'check=False' to the method.
+
+After the current executable is set, we'll then execute two more commands:
+
+        # Set the output-path and verify it is set.
+        self.runCmd("settings set target.process.output-path 'stdout.txt'")
+        self.expect("settings show target.process.output-path",
+                    SETTING_MSG("target.process.output-path"),
+            startstr = "target.process.output-path (string) = 'stdout.txt'")
+
+The first uses the 'settings set' command to set the static setting
+target.process.output-path to be 'stdout.txt', instead of the default
+'/dev/stdout'.  We then immediately issue a 'settings show' command to check
+that, indeed, the setting did take place.  Notice that we use a new method
+expect() to accomplish the task, which in effect issues a runCmd() behind the
+door and grabs the output from the command execution and expects to match the
+start string of the output against what we pass in as the value of the keyword
+argument pair:
+
+            startstr = "target.process.output-path (string) = 'stdout.txt'"
+
+Take a look at TestBase.expect() within lldbtest.py for more details.  Among
+other things, it can also match against a list of regexp patterns as well as a
+list of sub strings.  And it can also perform negative matching, i.e., instead
+of expecting something from the output of command execution, it can perform the
+action of 'not expecting' something.
+
+This will launch/run the program:
+
+        self.runCmd("run", RUN_SUCCEEDED)
+
+And this asserts that the file 'stdout.txt' should be present after running the
+program.
+
+        # The 'stdout.txt' file should now exist.
+        self.assertTrue(os.path.isfile("stdout.txt"),
+                        "'stdout.txt' exists due to target.process.output-path.")
+
+Also take a look at main.cpp which emits some message to the stdout.  Now, if we
+pass this assertion, it's time to examine the contents of the file to make sure
+it contains the same message as programmed in main.cpp:
+
+        # Read the output file produced by running the program.
+        with open('stdout.txt', 'r') as f:
+            output = f.read()
+
+        self.expect(output, exe=False,
+            startstr = "This message should go to standard out.")
+
+We open the file and read its contents into output, then issue an expect()
+method.  The 'exe=False' keyword argument pair tells expect() that don't try to
+execute the first arg as a command at all.  Instead, treat it as a string to
+match against whatever is thrown in as keyword argument pairs!
+
+There are also other test methods present in the TestSettings.py mode:
+test_set_prompt(), test_set_term_width(), test_set_auto_confirm(),
+test_with_dsym(), and test_with_dwarf().  We are using the default test loader
+from unittest framework, which uses the 'test' method name prefix to identify
+test methods automatically.
+
+This finishes the walkthrough of the test method test_set_output_path(self).
+Before we say goodbye, notice the little method definition at the top of the
+file:
+
+    @classmethod
+    def classCleanup(cls):
+        system(["/bin/sh", "-c", "rm -f output.txt"])
+        system(["/bin/sh", "-c", "rm -f stdout.txt"])
+
+This is a classmethod (as shown by the @classmethod decorator) which allows the
+individual test class to perform cleanup actions after the test harness finishes
+with the particular test class.  This is part of the so-called test fixture in
+the unittest framework.  From http://docs.python.org/library/unittest.html:
+
+A test fixture represents the preparation needed to perform one or more tests,
+and any associate cleanup actions. This may involve, for example, creating
+temporary or proxy databases, directories, or starting a server process.
+
+The TestBase class uses such fixture with setUp(self), tearDown(self),
+setUpClass(cls), and tearDownClass(cls).  And within teraDownClass(cls), it
+checks whether the current class has an attribute named 'classCleanup', and
+executes as a method if present.  In this particular case, the classCleanup()
+calls a utility function system() defined in lldbtest.py in order to remove the
+files created by running the program as the tests are executed.
+
+This system() function uses the Python subprocess module to spawn the process
+and to retrieve its results.  If the test instance passes the keyword argument
+pair 'sender=self', the detailed command execution through the operating system
+also gets recorded in a session object.  If the test instance fails or errors,
+the session info automatically gets dumped to a file grouped under a directory
+named after the timestamp of the particular test suite run.
+
+For simple cases, look for the timestamp directory in the same directory of the
+test driver program dotest.py.  For example, if we comment out the
+@expectedFailure decorator for TestSettings.py, and then run the test module:
+
+/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -v settings
+----------------------------------------------------------------------
+Collected 6 tests
+
+test_set_auto_confirm (TestSettings.SettingsCommandTestCase)
+Test that after 'set auto-confirm true', manual confirmation should not kick in. ... ok
+test_set_output_path (TestSettings.SettingsCommandTestCase)
+Test that setting target.process.output-path for the launched process works. ... FAIL
+test_set_prompt (TestSettings.SettingsCommandTestCase)
+Test that 'set prompt' actually changes the prompt. ... ok
+test_set_term_width (TestSettings.SettingsCommandTestCase)
+Test that 'set term-width' actually changes the term-width. ... ok
+test_with_dsym (TestSettings.SettingsCommandTestCase)
+Test that run-args and env-vars are passed to the launched process. ... ok
+test_with_dwarf (TestSettings.SettingsCommandTestCase)
+Test that run-args and env-vars are passed to the launched process. ... ok
+
+======================================================================
+FAIL: test_set_output_path (TestSettings.SettingsCommandTestCase)
+Test that setting target.process.output-path for the launched process works.
+----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "/Volumes/data/lldb/svn/trunk/test/settings/TestSettings.py", line 125, in test_set_output_path
+    "'stdout.txt' exists due to target.process.output-path.")
+AssertionError: False is not True : 'stdout.txt' exists due to target.process.output-path.
+
+----------------------------------------------------------------------
+Ran 6 tests in 8.219s
+
+FAILED (failures=1)
+/Volumes/data/lldb/svn/trunk/test $ ls 2010-10-19-14:10:49.059609
+TestSettings.SettingsCommandTestCase.test_set_output_path.log
+/Volumes/data/lldb/svn/trunk/test $ 
+
+We get one failure and a timestamp directory 2010-10-19-14:10:49.059609.
+For education purposes, the directory and its contents are reproduced here in
+the same directory as the current file.