forked from OSchip/llvm-project
335 lines
13 KiB
Plaintext
335 lines
13 KiB
Plaintext
============================================================================
|
|
| OpenMP Validation Suite V 3.0 |
|
|
| High Performance Computing Center, Stuttgart |
|
|
| High Performance Computing and Tools, University of Houston |
|
|
| Jan. 2012 |
|
|
============================================================================
|
|
|
|
|
|
TABLE OF CONTENTS
|
|
|
|
I INTRODUCTION
|
|
I.1. Aims and general function
|
|
I.2. Files and directories
|
|
|
|
II USAGE
|
|
II.1. First run with make
|
|
II.2 Where to search for results
|
|
II.3. Using the runtest script
|
|
|
|
III Adding and modifying tests
|
|
III.1. The template structure
|
|
|
|
IV Known Issues and Workaround
|
|
|
|
V Contact and Support
|
|
|
|
------------------------------------------------------------------------------
|
|
|
|
I. INTRODUCTION
|
|
==============================================================================
|
|
|
|
I.1. Aims and general function
|
|
--------------------------------
|
|
|
|
The OpenMP validation suite is designed to verify the correct implementation
|
|
of OpenMP in compilers. It is capable of checking Fortran as well as c
|
|
compiler.
|
|
|
|
Testing the implementation is based on statistics. Each directive is tested
|
|
by computing and verifying the result with the already known value. In most
|
|
cases a wrong implementation can result in the right values. So the tests are
|
|
run several times.
|
|
|
|
Additionally, the validation suite creates so called crosstests for each
|
|
directive. These are tests in which either the directive is missing or used
|
|
with different arguments. If this so called crosstest fails, this indicates
|
|
strongly that the previous test is capable of testing the directive.
|
|
|
|
Lastly, an orphaned test is also run to determine if the directive being
|
|
tested is able to correctly run when 'orphaned' from the main function.
|
|
Essentially, the directive's code is placed into its own function which is
|
|
called during execution of the main function and often inside a parallel
|
|
region.
|
|
|
|
|
|
I.2. Files and directories
|
|
----------------------------
|
|
|
|
|
|
d c directory containing the templates for the c tests
|
|
d fortran directory containing the templates for the Fortran
|
|
tests
|
|
Makefile Makefile containing options for compilation
|
|
common_utility.f
|
|
omp_my_sleep.h thread save sleep function
|
|
omp_testsuite.f Fortran header file
|
|
omp_testsuite.h autogenerated c-header file
|
|
ompts-c.conf configuration file for the c tests about how often
|
|
the tests shall be executed or how large the loop size
|
|
is
|
|
ompts_makeHeader.pl perl module for automatically generation of an up
|
|
to date header file
|
|
ompts_parserFunctions.pm perl module containing general functions for
|
|
the parser.pl script
|
|
ompts_parser.pl script for generating the source code out of the templates
|
|
ompts_standaloneProc.c framework for the c tests
|
|
ompts_standaloneProc.f framework for the Fortran tests
|
|
README the README file you've already found ;-)
|
|
LICENSE contains license information
|
|
runtest.pl the frame program of the test suite
|
|
testlist-f.txt test list containing the available tests for Fortran
|
|
testlist-c.txt test list containing the available tests for c
|
|
|
|
|
|
------------------------------------------------------------------------------
|
|
|
|
II. USAGE
|
|
==============================================================================
|
|
|
|
|
|
II.1. First run with make
|
|
--------------------------
|
|
|
|
|
|
You can do a first simple run of the testsuite only after one step of
|
|
configuration:
|
|
|
|
1) Modify the ompts.conf and ompts-c.conf file, change the number of threads
|
|
and number of repetitions of each test.
|
|
|
|
2) Modify the Makefile, uncommenting the CC/FC and CFLAGS/FFLAGS variables for
|
|
the compiler of your choice.
|
|
|
|
And now you can run the testsuite either for a C compiler with:
|
|
|
|
> make ctest
|
|
|
|
or for a Fortran compiler with:
|
|
|
|
> make ftest
|
|
|
|
|
|
II.2. Running custom tests
|
|
--------------------------
|
|
|
|
|
|
In order to run single tests or custom groups of tests, two make commmands
|
|
are defined: make stest and make fstest. These two command reference the file
|
|
customtest.txt when looking for a testlist to use. Simply edit customtest.txt
|
|
to include the desired test or tests. If customtest.txt contains c tests,
|
|
|
|
> make stest
|
|
|
|
or for fortran tests
|
|
|
|
> make fstest
|
|
|
|
In order to change the number of threads used in the tests, simply edit the
|
|
Makefile variables MINTHREADS and MAXTHREADS. By default, they are configured
|
|
to use 2 threads. To change the number of times each test is run, for c tests
|
|
edit the REPETITIONS variable in the file ompts-c.conf. The LOOPCOUNT and
|
|
SLEEPTIME variables can also be changed here. For fortran tests, edit the file
|
|
omp_testsuite.f to change both the LOOPCOUNT and the number of times each test
|
|
is run.
|
|
|
|
|
|
|
|
II.3. Understanding the results
|
|
---------------------------------
|
|
|
|
|
|
When running the testsuite the results will be shown on the screen.
|
|
|
|
If you need the results for further purpose you can use the results.txt, which
|
|
is a simple list containing the results for each directive
|
|
in a single line. Each line starts with the name of the directive. Then follows
|
|
the result of the test given in the percentage of the passed tests. If 100% of
|
|
the tests passed successfully, the second number gives the result of the
|
|
corresponding crosstest. Crosstests will only be run if the normal test passes
|
|
with 100% accuracy. If a crosstest was not run or a test does not exist, it is
|
|
denotated by a "-".
|
|
After the results of the normal tests, there follow a series of tests in the
|
|
orphaned mode. If there were no orphaned tests available this is shown by a "-".
|
|
|
|
If you run the testsuite with different numbers of threads (e.g. using the
|
|
runtest.pl script) the results are shown in blocks of 4 columns for each number
|
|
of threads.
|
|
|
|
If a test fails you can find more detailed information in the ompts.log,
|
|
bin/c/*.out and *.log files. While the ompts.log file contains all compiler
|
|
error messages for all tests, the *.out and *.log files contain detailed inforamtion
|
|
on the execution process of the single tests.
|
|
In the *.out files there are listed all the results of the single executions of
|
|
the tests. In the *.log files there are error messages of the tests itself.
|
|
|
|
|
|
II.4. Cleaning Up
|
|
-----------------
|
|
|
|
|
|
Because many files are generated for each tested directive, it is often necessary
|
|
to clean the main directory after a battery of tests. To clean all generated files
|
|
in the main directory including the results and log files,
|
|
|
|
> make clean
|
|
|
|
|
|
To clean only the logs and out files,
|
|
|
|
> make cleanlogs
|
|
|
|
To clean only the results,
|
|
|
|
> make cleanresults
|
|
|
|
|
|
|
|
II.4. Using the runtest script
|
|
-------------------------------
|
|
|
|
|
|
So for special purpose you can use the runtest.pl script, which allows a lot
|
|
more options for the execution process than the execution with make.
|
|
|
|
Using the runtest.pl script is rather easy. You can use the the test suite only
|
|
after two steps of modifications:
|
|
|
|
1.) Modify the Makefile to your wishes choosing your compiler and the necessary
|
|
compiler flags.
|
|
2.) If necessary edit one of the test lists (testlist-c.txt) and comment out the
|
|
tests you do not want to run using # at the beginning of a line. Testlists for Fortran end
|
|
with -f.txt while test lists for c with -c.txt.
|
|
|
|
And now you can run the test suite either for Fortran using
|
|
# > ./runtest.pl -lang=fortran -d=fortran testlist-f.txt
|
|
or for c
|
|
# > ./runtest.pl -lang=c -d=c testlist-c.txt
|
|
|
|
With the --help option you can show the complete list of options and their
|
|
explanations.
|
|
|
|
The test results are summarized in cresults or fresults.txt while *.log keep
|
|
details for individual tests. There is also a file (ompts.log) keeping
|
|
compilation information. (see section II.2 )
|
|
|
|
If you don't want to test the directives in orphaned mode you can use the
|
|
-norphan option. You also can use the runtest.pl script either to compile all
|
|
tests or run compiled tests e.g. for cross compilation on other platforms. For
|
|
this there are the options -norun and -nocompile.
|
|
|
|
Happy testing!
|
|
|
|
|
|
------------------------------------------------------------------------------
|
|
|
|
III. How to add new tests / The structure of test templates
|
|
==============================================================================
|
|
|
|
III.1 The template structure
|
|
------------------------------
|
|
|
|
The test suite is based on templates so that you only have one file for test,
|
|
crosstest and the orphaned versions of them.
|
|
|
|
A) Description of the template structure
|
|
|
|
The syntax of the templates is much like xml. So each test begins with
|
|
'<ompts:test>' and ends with '</ompts:test>'.
|
|
|
|
In between there are several other blocks holding information:
|
|
|
|
- <ompts:testdescription> </ompts:testdescription> In between this tag you can
|
|
give a description on what the test checks and how it works.
|
|
|
|
- <ompts:ompversion> </ompts:version> This tag is used to specify the
|
|
OpenMP-version which includes the tested directive.
|
|
|
|
- <ompts:directive> </ompts:directive> Used to specify the directive how it is
|
|
called in the programming language.
|
|
|
|
- <ompts:dependences> </ompts:dependences> With this tag you can specify other
|
|
omp directives which are necessary for the correct execution of your test.
|
|
The directives have to be listed by their directive names as it is called in
|
|
the programming language. Multiple directives are separated by ','.
|
|
|
|
- <ompts:testcode> </ompts:testcode> In this tag stands the whole source code
|
|
for the test / crosstest. Each test has to be written as a function. The
|
|
syntax of the functions differs between C and Fortran: In C it has to take a
|
|
file pointer 'FILE * logFile' and return an int. If the test has been passed
|
|
successful it has to return a value unequal to 0. The file pointer can be used
|
|
to write information into a log file. In Fortran the function takes no
|
|
argument and the function name must not exceed XX characters. The return value
|
|
has to be specified using the '<testfunctionname>' tags. It has to be 1 if the
|
|
test succeeded and 0 if the test failed. For details see the example.
|
|
|
|
To tell the test suite the name of your test function you have to enclose it
|
|
into the '<ompts:testcode:functionname> </ompts:testcode:functionname>' tag.
|
|
|
|
If there are differences between test and crosstest you can use the
|
|
<ompts:check> </ompts:check> and <ompts:crosscheck> </ompts:crosscheck> tag.
|
|
When generating the test the parser will use the code enclosed in
|
|
<ompts:check> tags and cut out the code written in <ompts:crosscheck> tags. So
|
|
you have two possibilities to write your template for test and crosstest: The
|
|
first way you can write the complete code is to write the test in one
|
|
<ompts:check> tag and later the crosstest in one <ompts:crosscheck> tag. The
|
|
second way is to write both tests only by enclosing differing parts in
|
|
corresponding tags.
|
|
|
|
The first method should be preferred if test and crosstest differ much from
|
|
each other. The second e.g. if you only want to change a few options like
|
|
replacing an omp singleprivate clause by an omp private clause or to cut out
|
|
a directive like omp flush. When you use the first way you have to take care
|
|
of the function name! You have to declare it twice with
|
|
<ompts:testcode:functionname>!
|
|
|
|
- <ompts:orphan> </ompts:orphan> This tag can be used if you want to enable
|
|
your test to check the directive in orphan regions, too. The code enclosed
|
|
in this part will be written to a separate function which will be called
|
|
instead. If you have variables which are used outside this region you have to
|
|
declare them as global variables enclosed in an <ompts:orphan:vars> tag. For
|
|
further information see the description of the <ompts:orphan:vars> tag.
|
|
|
|
- <ompts:orphan:vars> </ompts:orphan:vars> This tag is used to specify global
|
|
variables for an orphan region which allow the exchange of values between
|
|
the main program and the orphaned functions. The usage differs between C and
|
|
Fortran. In C you have to use a single declaration for each variable. You can
|
|
either declare all variables in one single or in several different regions. You
|
|
must not initialize the variables inside! In Fortran you have to put all
|
|
declarations in one single tag. Because there exist no global variables as in
|
|
C you have to use common blocks. For further information see the examples.
|
|
|
|
III.2. Adding tests to the test lists
|
|
--------------------------------------
|
|
|
|
After you have created a new test you have to add them to a testlist. Simply
|
|
add the function name in a new Line into a file.
|
|
|
|
|
|
|
|
------------------------------------------------------------------------------
|
|
|
|
|
|
IV. Known Issues and Workaround
|
|
==============================================================================
|
|
|
|
The Sun OS has a problem with the -maxdepth option on the 'make cleanall'
|
|
command. This prevents the tests from being removed from the working directory
|
|
and can cause problems with future tests. To remedy, edit the makefile line
|
|
under the clean command:
|
|
|
|
-rm [cf]test*.[cf] [cf]crosstest*.[cf] [cf]ctest*.[cf] [cf]orphan*.[cf]
|
|
|
|
Change to:
|
|
|
|
-rm [cf]test* [cf]crosstest* [cf]ctest* [cf]orphan*
|
|
|
|
|
|
|
|
------------------------------------------------------------------------------
|
|
|
|
V. Contact and Support
|
|
==============================================================================
|
|
|
|
Contact: http://www.cs.uh.edu/~hpctools
|