2012-09-14 07:51:08 +08:00
|
|
|
===================================================================
|
|
|
|
How To Add Your Build Configuration To LLVM Buildbot Infrastructure
|
|
|
|
===================================================================
|
|
|
|
|
|
|
|
Introduction
|
|
|
|
============
|
|
|
|
|
|
|
|
This document contains information about adding a build configuration and
|
2020-10-20 18:42:43 +08:00
|
|
|
buildbot-worker to private worker builder to LLVM Buildbot Infrastructure.
|
2012-09-14 07:51:08 +08:00
|
|
|
|
2017-02-06 05:21:23 +08:00
|
|
|
Buildmasters
|
|
|
|
============
|
|
|
|
|
|
|
|
There are two buildmasters running.
|
|
|
|
|
2021-10-29 05:50:11 +08:00
|
|
|
* The main buildmaster at `<https://lab.llvm.org/buildbot>`_. All builders
|
|
|
|
attached to this machine will notify commit authors every time they break
|
|
|
|
the build.
|
|
|
|
* The staging buildmaster at `<https://lab.llvm.org/staging>`_. All builders
|
|
|
|
attached to this machine will be completely silent by default when the build
|
|
|
|
is broken.
|
|
|
|
|
|
|
|
In order to remain connected to the main buildmaster (and thus notify
|
|
|
|
developers of failures), a builbot must:
|
|
|
|
|
|
|
|
* Be building a supported configuration. Builders for experimental backends
|
|
|
|
should generally be attached to staging buildmaster.
|
|
|
|
* Be able to keep up with new commits to the main branch, or at a minimum
|
|
|
|
recover to tip of tree within a couple of days of falling behind.
|
|
|
|
|
|
|
|
Additionally, we encourage all bot owners to point their bots towards the
|
|
|
|
staging master during maintenance windows, instability troubleshooting, and
|
|
|
|
such.
|
|
|
|
|
|
|
|
Roles & Expectations
|
|
|
|
====================
|
|
|
|
|
|
|
|
Each buildbot has an owner who is the responsible party for addressing problems
|
|
|
|
which arise with said buildbot. We generally expect the bot owner to be
|
|
|
|
reasonably responsive.
|
|
|
|
|
|
|
|
For some bots, the ownership responsibility is split between a "resource owner"
|
|
|
|
who provides the underlying machine resource, and a "configuration owner" who
|
|
|
|
maintains the build configuration. Generally, operational responsibility lies
|
|
|
|
with the "config owner". We do expect "resource owners" - who are generally
|
|
|
|
the contact listed in a workers attributes - to proxy requests to the relevant
|
|
|
|
"config owner" in a timely manner.
|
|
|
|
|
|
|
|
Most issues with a buildbot should be addressed directly with a bot owner
|
|
|
|
via email. Please CC `Galina Kistanova <mailto:gkistanova@gmail.com>`_.
|
2012-09-14 07:51:08 +08:00
|
|
|
|
|
|
|
Steps To Add Builder To LLVM Buildbot
|
|
|
|
=====================================
|
2020-10-20 18:42:43 +08:00
|
|
|
Volunteers can provide their build machines to work as build workers to
|
2012-09-14 07:51:08 +08:00
|
|
|
public LLVM Buildbot.
|
|
|
|
|
|
|
|
Here are the steps you can follow to do so:
|
|
|
|
|
|
|
|
#. Check the existing build configurations to make sure the one you are
|
|
|
|
interested in is not covered yet or gets built on your computer much
|
|
|
|
faster than on the existing one. We prefer faster builds so developers
|
|
|
|
will get feedback sooner after changes get committed.
|
|
|
|
|
|
|
|
#. The computer you will be registering with the LLVM buildbot
|
|
|
|
infrastructure should have all dependencies installed and you can
|
|
|
|
actually build your configuration successfully. Please check what degree
|
|
|
|
of parallelism (-j param) would give the fastest build. You can build
|
|
|
|
multiple configurations on one computer.
|
|
|
|
|
2020-10-20 18:42:43 +08:00
|
|
|
#. Install buildbot-worker (currently we are using buildbot version 2.8.5).
|
|
|
|
Depending on the platform, buildbot-worker could be available to download and
|
2016-09-29 20:51:26 +08:00
|
|
|
install with your package manager, or you can download it directly from
|
2012-09-14 07:51:08 +08:00
|
|
|
`<http://trac.buildbot.net>`_ and install it manually.
|
|
|
|
|
2020-10-20 18:42:43 +08:00
|
|
|
#. Create a designated user account, your buildbot-worker will be running under,
|
2012-09-14 07:51:08 +08:00
|
|
|
and set appropriate permissions.
|
|
|
|
|
2020-10-20 18:42:43 +08:00
|
|
|
#. Choose the buildbot-worker root directory (all builds will be placed under
|
|
|
|
it), buildbot-worker access name and password the build master will be using
|
|
|
|
to authenticate your buildbot-worker.
|
2012-09-14 07:51:08 +08:00
|
|
|
|
2020-10-20 18:42:43 +08:00
|
|
|
#. Create a buildbot-worker in context of that buildbot-worker account. Point it
|
|
|
|
to the **lab.llvm.org** port **9990** (see `Buildbot documentation,
|
|
|
|
Creating a worker
|
|
|
|
<http://docs.buildbot.net/current/tutorial/firstrun.html#creating-a-worker>`_
|
2012-09-14 07:51:08 +08:00
|
|
|
for more details) by running the following command:
|
|
|
|
|
|
|
|
.. code-block:: bash
|
|
|
|
|
2020-10-20 18:42:43 +08:00
|
|
|
$ buildbot-worker create-worker <buildbot-worker-root-directory> \
|
2012-09-14 07:51:08 +08:00
|
|
|
lab.llvm.org:9990 \
|
2020-10-20 18:42:43 +08:00
|
|
|
<buildbot-worker-access-name> \
|
|
|
|
<buildbot-worker-access-password>
|
2012-09-14 07:51:08 +08:00
|
|
|
|
2020-10-20 18:42:43 +08:00
|
|
|
To point a worker to silent master please use lab.llvm.org:9994 instead
|
2017-07-06 04:45:44 +08:00
|
|
|
of lab.llvm.org:9990.
|
|
|
|
|
2020-10-20 18:42:43 +08:00
|
|
|
#. Fill the buildbot-worker description and admin name/e-mail. Here is an
|
|
|
|
example of the buildbot-worker description::
|
2012-09-14 07:51:08 +08:00
|
|
|
|
|
|
|
Windows 7 x64
|
|
|
|
Core i7 (2.66GHz), 16GB of RAM
|
|
|
|
|
|
|
|
g++.exe (TDM-1 mingw32) 4.4.0
|
|
|
|
GNU Binutils 2.19.1
|
|
|
|
cmake version 2.8.4
|
|
|
|
Microsoft(R) 32-bit C/C++ Optimizing Compiler Version 16.00.40219.01 for 80x86
|
|
|
|
|
2020-10-20 18:42:43 +08:00
|
|
|
#. Make sure you can actually start the buildbot-worker successfully. Then set
|
|
|
|
up your buildbot-worker to start automatically at the start up time. See the
|
2012-09-14 07:51:08 +08:00
|
|
|
buildbot documentation for help. You may want to restart your computer
|
|
|
|
to see if it works.
|
|
|
|
|
2020-10-20 18:42:43 +08:00
|
|
|
#. Send a patch which adds your build worker and your builder to
|
2021-11-15 09:17:08 +08:00
|
|
|
`zorg <https://github.com/llvm/llvm-zorg>`_. Use the typical LLVM
|
2020-06-03 08:48:02 +08:00
|
|
|
`workflow <https://llvm.org/docs/Contributing.html#how-to-submit-a-patch>`_.
|
2012-09-14 07:51:08 +08:00
|
|
|
|
2021-01-22 16:18:09 +08:00
|
|
|
* workers are added to ``buildbot/osuosl/master/config/workers.py``
|
2012-09-14 07:51:08 +08:00
|
|
|
* builders are added to ``buildbot/osuosl/master/config/builders.py``
|
|
|
|
|
2019-12-07 06:28:02 +08:00
|
|
|
Please make sure your builder name and its builddir are unique through the
|
|
|
|
file.
|
|
|
|
|
2021-11-05 00:36:07 +08:00
|
|
|
All new builders should default to using the "'collapseRequests': False"
|
|
|
|
configuration. This causes the builder to build each commit individually
|
|
|
|
and not merge build requests. To maximize quality of feedback to developers,
|
|
|
|
we *strongly prefer* builders to be configured not to collapse requests.
|
|
|
|
This flag should be removed only after all reasonable efforts have been
|
|
|
|
exhausted to improve build times such that the builder can keep up with
|
|
|
|
commit flow.
|
|
|
|
|
2020-06-20 15:51:18 +08:00
|
|
|
It is possible to allow email addresses to unconditionally receive
|
2019-12-07 06:28:02 +08:00
|
|
|
notifications on build failure; for this you'll need to add an
|
|
|
|
``InformativeMailNotifier`` to ``buildbot/osuosl/master/config/status.py``.
|
|
|
|
This is particularly useful for the staging buildmaster which is silent
|
|
|
|
otherwise.
|
2017-02-06 05:21:23 +08:00
|
|
|
|
2020-10-20 18:42:43 +08:00
|
|
|
#. Send the buildbot-worker access name and the access password directly to
|
2012-09-14 07:51:08 +08:00
|
|
|
`Galina Kistanova <mailto:gkistanova@gmail.com>`_, and wait till she
|
|
|
|
will let you know that your changes are applied and buildmaster is
|
|
|
|
reconfigured.
|
|
|
|
|
2020-10-20 18:42:43 +08:00
|
|
|
#. Check the status of your buildbot-worker on the `Waterfall Display
|
2021-10-29 09:45:54 +08:00
|
|
|
<http://lab.llvm.org/buildbot/#/waterfall>`_ to make sure it is connected,
|
|
|
|
and the `Workers Display <http://lab.llvm.org/buildbot/#/workers>`_ to see if
|
|
|
|
administrator contact and worker information are correct.
|
2012-09-14 07:51:08 +08:00
|
|
|
|
|
|
|
#. Wait for the first build to succeed and enjoy.
|
2021-11-22 00:00:34 +08:00
|
|
|
|
|
|
|
|
|
|
|
Best Practices for Configuring a Fast Builder
|
|
|
|
=============================================
|
|
|
|
|
|
|
|
As mentioned above, we generally have a strong preference for
|
|
|
|
builders which can build every commit as they come in. This section
|
|
|
|
includes best practices and some recommendations as to how to achieve
|
|
|
|
that end.
|
|
|
|
|
|
|
|
The goal
|
|
|
|
In 2020, the monorepo had just under 35 thousand commits. This works
|
|
|
|
out to an average of 4 commits per hour. Already, we can see that a
|
|
|
|
builder must cycle in less than 15 minutes to have a hope of being
|
|
|
|
useful. However, those commits are not uniformly distributed. They
|
|
|
|
tend to cluster strongly during US working hours. Looking at a couple
|
|
|
|
of recent (Nov 2021) working days, we routinely see ~10 commits per
|
|
|
|
hour during peek times, with occasional spikes as high as ~15 commits
|
|
|
|
per hour. Thus, as a rule of thumb, we should plan for our builder to
|
|
|
|
complete ~10-15 builds an hour.
|
|
|
|
|
|
|
|
Resource Appropriately
|
|
|
|
At 10-15 builds per hour, we need to complete a new build on average every
|
|
|
|
4 to 6 minutes. For anything except the fastest of hardware/build configs,
|
|
|
|
this is going to be well beyond the ability of a single machine. In buildbot
|
|
|
|
terms, we likely going to need multiple workers to build requests in parallel
|
|
|
|
under a single builder configuration. For some rough back of the envelope
|
|
|
|
numbers, if your build config takes e.g. 30 minutes, you will need something
|
|
|
|
on the order of 5-8 workers. If your build config takes ~2 hours, you'll
|
|
|
|
need something on the order of 20-30 workers. The rest of this section
|
|
|
|
focuses on how to reduce cycle times.
|
|
|
|
|
|
|
|
Restrict what you build and test
|
|
|
|
Think hard about why you're setting up a bot, and restrict your build
|
|
|
|
configuration as much as you can. Basic functionality is probably
|
|
|
|
already covered by other bots, and you don't need to duplicate that
|
|
|
|
testing. You only need to be building and testing the *unique* parts
|
|
|
|
of the configuration. (e.g. For a multi-stage clang builder, you probably
|
|
|
|
don't need to be enabling every target or building all the various utilities.)
|
|
|
|
|
|
|
|
It can sometimes be worthwhile splitting a single builder into two or more,
|
|
|
|
if you have multiple distinct purposes for the same builder. As an example,
|
|
|
|
if you want to both a) confirm that all of LLVM builds with your host
|
|
|
|
compiler, and b) want to do a multi-stage clang build on your target, you
|
|
|
|
may be better off with two separate bots. Splitting increases resource
|
2021-11-23 03:00:53 +08:00
|
|
|
consumption, but makes it easy for each bot to keep up with commit flow.
|
|
|
|
Additionally, splitting bots may assist in triage by narrowing attention to
|
|
|
|
relevant parts of the failing configuration.
|
2021-11-22 00:00:34 +08:00
|
|
|
|
|
|
|
In general, we recommend Release build types with Assertions enabled. This
|
|
|
|
generally provides a good balance between build times and bug detection for
|
2021-11-23 03:00:53 +08:00
|
|
|
most buildbots. There may be room for including some debug info (e.g. with
|
|
|
|
`-gmlt`), but in general the balance between debug info quality and build
|
|
|
|
times is a delicate one.
|
2021-11-22 00:00:34 +08:00
|
|
|
|
|
|
|
Use Ninja & LLD
|
|
|
|
Ninja really does help build times over Make, particularly for highly
|
2021-11-23 03:00:53 +08:00
|
|
|
parallel builds. LLD helps to reduce both link times and memory usage
|
|
|
|
during linking significantly. With a build machine with sufficient
|
|
|
|
parallism, link times tend to dominate critical path of the build, and are
|
|
|
|
thus worth optimizing.
|
2021-11-22 00:00:34 +08:00
|
|
|
|
|
|
|
Use CCache and NOT incremental builds
|
|
|
|
Using ccache materially improves average build times. Incremental builds
|
|
|
|
can be slightly faster, but introduce the risk of build corruption due to
|
|
|
|
e.g. state changes, etc... At this point, the recommendation is not to
|
|
|
|
use incremental builds and instead use ccache as the latter captures the
|
|
|
|
majority of the benefit with less risk of false positives.
|
|
|
|
|
|
|
|
One of the non-obvious benefits of using ccache is that it makes the
|
|
|
|
builder less sensitive to which projects are being monitored vs built.
|
|
|
|
If a change triggers a build request, but doesn't change the build output
|
|
|
|
(e.g. doc changes, python utility changes, etc..), the build will entirely
|
|
|
|
hit in cache and the build request will complete in just the testing time.
|
|
|
|
|
|
|
|
With multiple workers, it is tempting to try to configure a shared cache
|
|
|
|
between the workers. Experience to date indicates this is difficult to
|
|
|
|
well, and that having local per-worker caches gets most of the benefit
|
|
|
|
anyways. We don't currently recommend shared caches.
|
|
|
|
|
2021-11-23 03:00:53 +08:00
|
|
|
CCache does depend on the builder hardware having sufficient IO to access
|
|
|
|
the cache with reasonable access times - i.e. a fast disk, or enough memory
|
|
|
|
for a RAM cache, etc.. For builders without, incremental may be your best
|
|
|
|
option, but is likely to require higher ongoing involvement from the
|
|
|
|
sponsor.
|
|
|
|
|
2021-11-22 00:00:34 +08:00
|
|
|
Enable batch builds
|
|
|
|
As a last resort, you can configure your builder to batch build requests.
|
|
|
|
This makes the build failure notifications markedly less actionable, and
|
|
|
|
should only be done once all other reasonable measures have been taken.
|
|
|
|
|
|
|
|
Leave it on the staging buildmaster
|
|
|
|
While most of this section has been biased towards builders intended for
|
|
|
|
the main buildmaster, it is worth highlighting that builders can run
|
|
|
|
indefinitely on the staging buildmaster. Such a builder may still be
|
|
|
|
useful for the sponsoring organization, without concern of negatively
|
|
|
|
impacting the broader community. The sponsoring organization simply
|
|
|
|
has to take on the responsibility of all bisection and triage.
|
|
|
|
|
|
|
|
|