llvm-project/openmp/testsuite/README

 ============================================================================
|                   OpenMP Validation Suite  V 3.0                           |
|              High Performance Computing Center, Stuttgart                  |
|       High Performance Computing and Tools, University of Houston          |
|                              Jan. 2012                                     |
 ============================================================================


TABLE OF CONTENTS

I    	INTRODUCTION
I.1.	  Aims and general function
I.2. 	  Files and directories

II		USAGE
II.1.	  First run with make
II.2	  Where to search for results
II.3.	  Using the runtest script

III		Adding and modifying tests
III.1.	  The template structure

IV		Known Issues and Workaround

V       Contact and Support

------------------------------------------------------------------------------

I.  INTRODUCTION
==============================================================================

I.1.  Aims and general function
--------------------------------

The OpenMP validation suite is designed to verify the correct implementation
of OpenMP in compilers. It is capable of checking Fortran as well as c
compiler.

Testing the implementation is based on statistics. Each directive is tested
by computing and verifying the result with the already known value. In most
cases a wrong implementation can result in the right values. So the tests are
run several times.

Additionally, the validation suite creates so called crosstests for each
directive. These are tests in which either the directive is missing or used
with different arguments. If this so called crosstest fails, this indicates
strongly that the previous test is capable of testing the directive.

Lastly, an orphaned test is also run to determine if the directive being
tested is able to correctly run when 'orphaned' from the main function.
Essentially, the directive's code is placed into its own function which is
called during execution of the main function and often inside a parallel
region.


I.2.  Files and directories
----------------------------


d c                         directory containing the templates for the c tests
d fortran                   directory containing the templates for the Fortran
                            tests
  Makefile                  Makefile containing options for compilation
  common_utility.f
  omp_my_sleep.h            thread save sleep function
  omp_testsuite.f           Fortran header file
  omp_testsuite.h           autogenerated c-header file
  ompts-c.conf              configuration file for the c tests about how often
                            the tests shall be executed or how large the loop size
                            is
  ompts_makeHeader.pl	    perl module for automatically generation of an up
                            to date header file
  ompts_parserFunctions.pm	perl module containing general functions for
				            the parser.pl script
  ompts_parser.pl	        script for generating the source code out of the templates
  ompts_standaloneProc.c	framework for the c tests
  ompts_standaloneProc.f	framework for the Fortran tests
  README		            the README file you've already found ;-)
  LICENSE                   contains license information
  runtest.pl		        the frame program of the test suite
  testlist-f.txt	        test list containing the available tests for Fortran
  testlist-c.txt	        test list containing the available tests for c


------------------------------------------------------------------------------

II.  USAGE
==============================================================================


II.1. First run with make
--------------------------


You can do a first simple run of the testsuite only after one step of
configuration:

1) Modify the ompts.conf and ompts-c.conf file, change the number of threads
and number of repetitions of each test.

2) Modify the Makefile, uncommenting the CC/FC and CFLAGS/FFLAGS variables for
the compiler of your choice.

And now you can run the testsuite either for a C compiler with:

>	make ctest

or for a Fortran compiler with:

>	make ftest


II.2. Running custom tests
--------------------------


In order to run single tests or custom groups of tests, two make commmands
are defined: make stest and make fstest. These two command reference the file
customtest.txt when looking for a testlist to use. Simply edit customtest.txt
to include the desired test or tests. If customtest.txt contains c tests,

>      make stest

or for fortran tests

>      make fstest

In order to change the number of threads used in the tests, simply edit the
Makefile variables MINTHREADS and MAXTHREADS. By default, they are configured
to use 2 threads. To change the number of times each test is run, for c tests
edit the REPETITIONS variable in the file ompts-c.conf. The LOOPCOUNT and
SLEEPTIME variables can also be changed here. For fortran tests, edit the file
omp_testsuite.f to change both the LOOPCOUNT and the number of times each test
is run.



II.3. Understanding the results
---------------------------------


When running the testsuite the results will be shown on the screen.

If you need the results for further purpose you can use the results.txt, which
is a simple list containing the results for each directive
in a single line. Each line starts with the name of the directive. Then follows
the result of the test given in the percentage of the passed tests. If 100% of
the tests passed successfully, the second number gives the result of the
corresponding crosstest. Crosstests will only be run if the normal test passes
with 100% accuracy. If a crosstest was not run or a test does not exist, it is
denotated by a "-".
After the results of the normal tests, there follow a series of tests in the
orphaned mode. If there were no orphaned tests available this is shown by a "-".

If you run the testsuite with different numbers of threads (e.g. using the
runtest.pl script) the results are shown in blocks of 4 columns for each number
of threads.

If a test fails you can find more detailed information in the ompts.log,
bin/c/*.out and *.log files. While the ompts.log file contains all compiler
error messages for all tests, the *.out and *.log files contain detailed inforamtion
on the execution process of the single tests.
In the *.out files there are listed all the results of the single executions of
the tests. In the *.log files there are error messages of the tests itself.


II.4. Cleaning Up
-----------------


Because many files are generated for each tested directive, it is often necessary
to clean the main directory after a battery of tests. To clean all generated files
in the main directory including the results and log files,

>     make clean


To clean only the logs and out files,

>     make cleanlogs

To clean only the results,

>     make cleanresults



II.4. Using the runtest script
-------------------------------


So for special purpose you can use the runtest.pl script, which allows a lot
more options for the execution process than the execution with make.

Using the runtest.pl script is rather easy. You can use the the test suite only
after two steps of modifications:

1.) Modify the Makefile to your wishes choosing your compiler and the necessary
    compiler flags.
2.) If necessary edit one of the test lists (testlist-c.txt) and comment out the
    tests you do not want to run using # at the beginning of a line. Testlists for Fortran end
    with -f.txt while test lists for c with -c.txt.

And now you can run the test suite either for Fortran using
	#	>  ./runtest.pl -lang=fortran -d=fortran testlist-f.txt
or for c
	#	>  ./runtest.pl -lang=c -d=c testlist-c.txt

With the --help option you can show the complete list of options and their
explanations.

The test results are summarized in cresults or fresults.txt while *.log keep
details for individual tests. There is also a file (ompts.log) keeping
compilation information. (see section II.2 )

If you don't want to test the directives in orphaned mode you can use the
-norphan option. You also can use the runtest.pl script either to compile all
tests or run compiled tests e.g. for cross compilation on other platforms. For
this there are the options -norun and -nocompile.

Happy testing!


------------------------------------------------------------------------------

III.	How to add new tests / The structure of test templates
==============================================================================

III.1  The template structure
------------------------------

The test suite is based on templates so that you only have one file for test,
crosstest and the orphaned versions of them.

	A) Description of the template structure

The syntax of the templates is much like xml. So each test begins with
'<ompts:test>' and ends with '</ompts:test>'.

In between there are several other blocks holding information:

- <ompts:testdescription> </ompts:testdescription> In between this tag you can
  give a description on what the test checks and how it works.

- <ompts:ompversion> </ompts:version> This tag is used to specify the
  OpenMP-version which includes the tested directive.

- <ompts:directive> </ompts:directive> Used to specify the directive how it is
  called in the programming language.

- <ompts:dependences> </ompts:dependences> With this tag you can specify other
  omp directives which are necessary for the correct execution of your test.
The directives have to be listed by their directive names as it is called in
the programming language. Multiple directives are separated by ','.

- <ompts:testcode> </ompts:testcode> In this tag stands the whole source code
for the test / crosstest.  Each test has to be written as a function. The
syntax of the functions differs between C and Fortran: In C it has to take a
file pointer 'FILE * logFile' and return an int. If the test has been passed
successful it has to return a value unequal to 0. The file pointer can be used
to write information into a log file.  In Fortran the function takes no
argument and the function name must not exceed XX characters.  The return value
has to be specified using the '<testfunctionname>' tags. It has to be 1 if the
test succeeded and 0 if the test failed. For details see the example.

To tell the test suite the name of your test function you have to enclose it
into the '<ompts:testcode:functionname> </ompts:testcode:functionname>' tag.

If there are differences between test and crosstest you can use the
<ompts:check> </ompts:check> and <ompts:crosscheck> </ompts:crosscheck> tag.
When generating the test the parser will use the code enclosed in
<ompts:check> tags and cut out the code written in <ompts:crosscheck> tags. So
you have two possibilities to write your template for test and crosstest: The
first way you can write the complete code is to write the test in one
<ompts:check> tag and later the crosstest in one <ompts:crosscheck> tag.  The
second way is to write both tests only by enclosing differing parts in
corresponding tags.

The first method should be preferred if test and crosstest differ much from
each other. The second e.g. if you only want to change a few options like
replacing an omp singleprivate clause by an omp private clause or to cut out
a directive like omp flush.  When you use the first way you have to take care
of the function name! You have to declare it twice with
<ompts:testcode:functionname>!

- <ompts:orphan> </ompts:orphan> This tag can be used if you want to enable
  your test to check the directive in orphan regions, too.  The code enclosed
in this part will be written to a separate function which will be called
instead.  If you have variables which are used outside this region you have to
declare them as global variables enclosed in an <ompts:orphan:vars> tag. For
further information see the description of the <ompts:orphan:vars> tag.

- <ompts:orphan:vars> </ompts:orphan:vars> This tag is used to specify global
  variables for an orphan region which allow the exchange of values between
the main program and the orphaned functions. The usage differs between C and
Fortran.  In C you have to use a single declaration for each variable. You can
either declare all variables in one single or in several different regions. You
must not initialize the variables inside!  In Fortran you have to put all
declarations in one single tag. Because there exist no global variables as in
C you have to use common blocks. For further information see the examples.

III.2.  Adding tests to the test lists
--------------------------------------

After you have created a new test you have to add them to a testlist. Simply
add the function name in a new Line into a file.



------------------------------------------------------------------------------


IV.   Known Issues and Workaround
==============================================================================

The Sun OS has a problem with the -maxdepth option on the 'make cleanall'
command. This prevents the tests from being removed from the working directory
and can cause problems with future tests. To remedy, edit the makefile line
under the clean command:

     -rm [cf]test*.[cf] [cf]crosstest*.[cf] [cf]ctest*.[cf] [cf]orphan*.[cf]

Change to:

     -rm [cf]test* [cf]crosstest* [cf]ctest* [cf]orphan*



------------------------------------------------------------------------------

V.   Contact and Support
==============================================================================

Contact: http://www.cs.uh.edu/~hpctools