forked from OSchip/llvm-project
211 lines
7.3 KiB
ReStructuredText
211 lines
7.3 KiB
ReStructuredText
=============================
|
|
How To Validate a New Release
|
|
=============================
|
|
|
|
.. contents::
|
|
:local:
|
|
:depth: 1
|
|
|
|
Introduction
|
|
============
|
|
|
|
This document contains information about testing the release candidates that will
|
|
ultimately be the next LLVM release. For more information on how to manage the
|
|
actual release, please refer to :doc:`HowToReleaseLLVM`.
|
|
|
|
Overview of the Release Process
|
|
-------------------------------
|
|
|
|
Once the release process starts, the Release Manager will ask for volunteers,
|
|
and it'll be the role of each volunteer to:
|
|
|
|
* Test and benchmark the previous release
|
|
|
|
* Test and benchmark each release candidate, comparing to the previous release and candidates
|
|
|
|
* Identify, reduce and report every regression found during tests and benchmarks
|
|
|
|
* Make sure the critical bugs get fixed and merged to the next release candidate
|
|
|
|
Not all bugs or regressions are show-stoppers and it's a bit of a grey area what
|
|
should be fixed before the next candidate and what can wait until the next release.
|
|
|
|
It'll depend on:
|
|
|
|
* The severity of the bug, how many people it affects and if it's a regression or a
|
|
known bug. Known bugs are "unsupported features" and some bugs can be disabled if
|
|
they have been implemented recently.
|
|
|
|
* The stage in the release. Less critical bugs should be considered to be fixed between
|
|
RC1 and RC2, but not so much at the end of it.
|
|
|
|
* If it's a correctness or a performance regression. Performance regression tends to be
|
|
taken more lightly than correctness.
|
|
|
|
.. _scripts:
|
|
|
|
Scripts
|
|
=======
|
|
|
|
The scripts are in the ``utils/release`` directory.
|
|
|
|
test-release.sh
|
|
---------------
|
|
|
|
This script will check-out, configure and compile LLVM+Clang (+ most add-ons, like ``compiler-rt``,
|
|
``libcxx``, ``libomp`` and ``clang-extra-tools``) in three stages, and will test the final stage.
|
|
It'll have installed the final binaries on the Phase3/Releasei(+Asserts) directory, and
|
|
that's the one you should use for the test-suite and other external tests.
|
|
|
|
To run the script on a specific release candidate run::
|
|
|
|
./test-release.sh \
|
|
-release 3.3 \
|
|
-rc 1 \
|
|
-no-64bit \
|
|
-test-asserts \
|
|
-no-compare-files
|
|
|
|
Each system will require different options. For instance, x86_64 will obviously not need
|
|
``-no-64bit`` while 32-bit systems will, or the script will fail.
|
|
|
|
The important flags to get right are:
|
|
|
|
* On the pre-release, you should change ``-rc 1`` to ``-final``. On RC2, change it to ``-rc 2`` and so on.
|
|
|
|
* On non-release testing, you can use ``-final`` in conjunction with ``-no-checkout``, but you'll have to
|
|
create the ``final`` directory by hand and link the correct source dir to ``final/llvm.src``.
|
|
|
|
* For release candidates, you need ``-test-asserts``, or it won't create a "Release+Asserts" directory,
|
|
which is needed for release testing and benchmarking. This will take twice as long.
|
|
|
|
* On the final candidate you just need Release builds, and that's the binary directory you'll have to pack.
|
|
|
|
This script builds three phases of Clang+LLVM twice each (Release and Release+Asserts), so use
|
|
screen or nohup to avoid headaches, since it'll take a long time.
|
|
|
|
Use the ``--help`` option to see all the options and chose it according to your needs.
|
|
|
|
|
|
findRegressions-nightly.py
|
|
--------------------------
|
|
|
|
TODO
|
|
|
|
.. _test-suite:
|
|
|
|
Test Suite
|
|
==========
|
|
|
|
.. contents::
|
|
:local:
|
|
|
|
Follow the `LNT Quick Start Guide <http://llvm.org/docs/lnt/quickstart.html>`__ link on how to set-up the test-suite
|
|
|
|
The binary location you'll have to use for testing is inside the ``rcN/Phase3/Release+Asserts/llvmCore-REL-RC.install``.
|
|
Link that directory to an easier location and run the test-suite.
|
|
|
|
An example on the run command line, assuming you created a link from the correct
|
|
install directory to ``~/devel/llvm/install``::
|
|
|
|
./sandbox/bin/python sandbox/bin/lnt runtest \
|
|
nt \
|
|
-j4 \
|
|
--sandbox sandbox \
|
|
--test-suite ~/devel/llvm/test/test-suite \
|
|
--cc ~/devel/llvm/install/bin/clang \
|
|
--cxx ~/devel/llvm/install/bin/clang++
|
|
|
|
It should have no new regressions, compared to the previous release or release candidate. You don't need to fix
|
|
all the bugs in the test-suite, since they're not necessarily meant to pass on all architectures all the time. This is
|
|
due to the nature of the result checking, which relies on direct comparison, and most of the time, the failures are
|
|
related to bad output checking, rather than bad code generation.
|
|
|
|
If the errors are in LLVM itself, please report every single regression found as blocker, and all the other bugs
|
|
as important, but not necessarily blocking the release to proceed. They can be set as "known failures" and to be
|
|
fix on a future date.
|
|
|
|
.. _pre-release-process:
|
|
|
|
Pre-Release Process
|
|
===================
|
|
|
|
.. contents::
|
|
:local:
|
|
|
|
When the release process is announced on the mailing list, you should prepare
|
|
for the testing, by applying the same testing you'll do on the release candidates,
|
|
on the previous release.
|
|
|
|
You should:
|
|
|
|
* Download the previous release sources from http://llvm.org/releases/download.html.
|
|
|
|
* Run the test-release.sh script on ``final`` mode (change ``-rc 1`` to ``-final``).
|
|
|
|
* Once all three stages are done, it'll test the final stage.
|
|
|
|
* Using the ``Phase3/Release+Asserts/llvmCore-MAJ.MIN-final.install`` base, run the test-suite.
|
|
|
|
If the final phase's ``make check-all`` failed, it's a good idea to also test the
|
|
intermediate stages by going on the obj directory and running ``make check-all`` to find
|
|
if there's at least one stage that passes (helps when reducing the error for bug report
|
|
purposes).
|
|
|
|
.. _release-process:
|
|
|
|
Release Process
|
|
===============
|
|
|
|
.. contents::
|
|
:local:
|
|
|
|
When the Release Manager sends you the release candidate, download all sources,
|
|
unzip on the same directory (there will be sym-links from the appropriate places
|
|
to them), and run the release test as above.
|
|
|
|
You should:
|
|
|
|
* Download the current candidate sources from where the release manager points you
|
|
(ex. http://llvm.org/pre-releases/3.3/rc1/).
|
|
|
|
* Repeat the steps above with ``-rc 1``, ``-rc 2`` etc modes and run the test-suite
|
|
the same way.
|
|
|
|
* Compare the results, report all errors on Bugzilla and publish the binary blob
|
|
where the release manager can grab it.
|
|
|
|
Once the release manages announces that the latest candidate is the good one, you
|
|
have to pack the ``Release`` (no Asserts) install directory on ``Phase3`` and that
|
|
will be the official binary.
|
|
|
|
* Rename (or link) ``clang+llvm-REL-ARCH-ENV`` to the .install directory
|
|
|
|
* Tar that into the same name with ``.tar.gz`` extensioan from outside the directory
|
|
|
|
* Make it available for the release manager to download
|
|
|
|
.. _bug-reporting:
|
|
|
|
Bug Reporting Process
|
|
=====================
|
|
|
|
.. contents::
|
|
:local:
|
|
|
|
If you found regressions or failures when comparing a release candidate with the
|
|
previous release, follow the rules below:
|
|
|
|
* Critical bugs on compilation should be fixed as soon as possible, possibly before
|
|
releasing the binary blobs.
|
|
|
|
* Check-all tests should be fixed before the next release candidate, but can wait
|
|
until the test-suite run is finished.
|
|
|
|
* Bugs in the test suite or unimportant check-all tests can be fixed in between
|
|
release candidates.
|
|
|
|
* New features or recent big changes, when close to the release, should have done
|
|
in a way that it's easy to disable. If they misbehave, prefer disabling them than
|
|
releasing an unstable (but untested) binary package.
|