… the CI.
There are 2 finale steps before finale binary distribution on Windows.
We must compile the GSettings XML schema files and register GdkPixbuf
loaders (for file format support in the GUI).
I used to provide a wrapper to be run inside Windows before first GIMP
run. Never did I realize that I can compile the distributed GSettings
schemas with the native `glib-compile-schemas` (works fine in my tests).
As for the GdkPixbuf loaders, we inspect DLL libraries, hence we do
require the target `gdk-pixbuf-query-loaders` which is unfortunately a
Windows executable. Yet it seems to work fine with Wine, so let's be
done with it in the CI instead of requiring manual steps from testers of
the CI builds. Then a few `sed` calls are enough to make the path in the
produced text file relative instead of absolute (which works fine, again
in my tests at least).
This means that I don't have to distribute the 2 binaries and the DLLs
they depend on anymore. Moreover let's remove the wrapper (but still
generate one which just calls GIMP so that we call it from the tree
root, where it's much less messy).
Note: I failed to install wine32 (32-bit Wine) on the Gitlab runner.
After following all instructions, I encountered weird errors. So
instead, I just make the win32-nightly job depend on win64-nightly and
copy `loaders.cache` from one to another, as it is a
platform-independent text file (as long as we provide the same GdkPixbuf
loaders on both of course, which we do).
The main purpose of these jobs is to only package the strict necessary
for a working GIMP under Windows, i.e. getting rid of all unnecessary
executables, and inspecting binary dependencies recursively to only
package used DLLs.
The dll_link.py script is taken from Siril codebase (see commit a86e82a8
on Siril repository, by FlorianBen). This was a very nice idea, and
makes for much smaller test archive (Siril is also GPLv3 so licensing is
ok for the reuse, also anyway it's just a small independent build
script).
Moreover having it as a separate job allows to have artifacts with only
the finale distribution (artifacts on the build job also have the build
directory and the whole prefix, which we want to keep in order to debug
when needed).
Hopefully I am not missing anything. Siril seems to package more, like
various gdk-pixbuf-*.exe, gspawn-*.exe and gdbus.exe. I am wondering if
these are actually necessary. I could run GIMP fine without these in
quick tests, but I guess I'll have to investigate a bit more to figure
this out. That's what nightly builds are for, after all, so hopefully
people will report if we miss some runtime dependencies.
I don't add this at the end of the distcheck job to make the interface
clearer, and also because the distcheck job will have full build
artifacts (allowing to debug a failing distcheck if necessary), whereas
the `sources` job will just publish tarballs and SHA256 sums generated
from these tarballs. Simple, clean.
The Linux CI job names are too long and are not recognizable on the web
GUI unless you hover the widgets with the mouse to read tooltips. Remove
the "/testing" part (if people want to know exactly which Debian we use
for our builds, they can always look at the script) and move left the
differenciating parts (i.e. autotools/meson/clang/distcheck) so that
these are visible in a glance, even when ellipsing long job names.
Problem with the CI is that the source is straight in our base directory
and therefore installed dependency files as well as cache or built files
are in subfolders.
Let's try to ignore as much as I can see. This should avoid a bunch of
warning and errors during report generation.
Seems that msys2 packages can sometimes be compressed as `.tar.zst`
instead of `.tar.xz`. I updated crossroad to handle this case. It now
requires the additional pypi `zstandard` package.
This is a new feature I implemented in the crossroad cross-compilation
tool. Msys2 repository has more packages and they are more up-to-date
compared to Fedora and Suse cross-built packages (the 2 other available
sources for pre-built Windows packages).
This allows to simplify a lot the dependency preparation for the Windows
CI, and speed things up.
`gegl` binary is being run for icon generations on gimp-2-10.
(cherry picked from commit 97549081fd)
Note: this commit is so far not needed on master as we don't call gegl
during build time unlike in gimp-2-10. But we could at some point.
Better to be thorough.
Also install missing xauth (required by xvfb-run).
`make check` was broken on CI, but got fixed by commits 707d3c6f64,
f6e9c6ee6f, 2c1efdedf0 and of course the present one.
Dependencies must be reinstalled on different jobs, even when related.
Fixes these CI errors on unit testing:
> GEGL-Message: 22:30:35.867: Module '/builds/GNOME/gimp/_install/lib/x86_64-linux-gnu/gegl-0.4/matting-levin.so' load error: libumfpack.so.5: cannot open shared object file: No such file or directory
> GEGL-MESSAGE: Module '/builds/GNOME/gimp/_install/lib/x86_64-linux-gnu/gegl-0.4/raw-load.so' load error: libraw.so.19: cannot open shared object file: No such file or directory
(cherry picked from commit a90d547cc0)
Not because they are working better than the ones in autotools, but
rather because it seems they are simply barely implemented! So of
course, the few tests currently there work.
The old custom ArchLinux got broken (apparently by some package
signature verification which fails, and obviously we don't want to
bypass these for security reasons).
I took the opportunity to port to Debian testing because this is GIMP's
base distribution for support (basically dependency versions must be in
Debian testing) so it makes sense that our CI is based off it as well.
Note though that I am not against additional CI tests so if someone
absolutely wants to get the Archlinux-based CI back and thinks it gets
us some additional worthy test, feel free to fix whatever was broken
then we may add it back (having both Debian testing and Archlinux CI).
Some issues may happen only for 32-bit builds. So it may be worth make a
Windows 32-bit build too.
Issue #2794 triggered this reasonning, though in this issue's case,
simply the build would not have discovered the bug since it was only a
build warning + run-time bug. Still long-term if we manage to get a
no-warning build, it could become relevant to discover more errors at
build time. So preparing the ground work here.
The GNU/Linux builds should start as soon as the Linux dependencies are
built. There is no need to wait for the Windows dependencies (and
reciprocally of course).
This should make for much faster CI total duration (with current
configuration).
Note: this "needs" keyword is quite a recent feature since gitlab 12.2,
3 months ago: https://docs.gitlab.com/ee/ci/yaml/#needs
A few commands need to be performed the first time for glib to work
properly, and gdk-pixbuf loaders to be found. I add them in a wrapper
script so that it's easy to ask people to test the dev builds (even
though it's not necessary to run these commands each time, but who
cares!).
Rather than having the whole Win32 cross-build into the 'gimp' stage,
break the dependencies and GIMP-only builds in 2 stages.
Since apparently we need to keep the same structure for the native and
cross build (otherwise we don't get parallel builds; in other words, I
didn't find the possibility to set separate pipelines up), I move babl
and GEGL into the same 'dependencies' stage.
Finally I remove the -base rules extended into actual jobs, except for
`.gimp-base` (this is the only which makes sense as it is actually
common to the meson and autotools build).
We don't need to pull 5 commits of history. Only the HEAD of the
selected branch is needed.
Also define it globally rather than re-defining it in every job to the
same value.
First replace the "when: on_failure" rule by a "when: always". We indeed
always want to get log artifact so that we can study a build if
necessary (neither only on failure nor on success; really on all cases,
since even an apparently successful build may have issues we might want
to diagnose).
Also expire all artifacts at 1 week (it seems the default on GNOME's
Gitlab is 4 weeks; we don't need to keep these so long. Even a few days
might be enough actually).
As for the artifacts contents, keep the build dirs rather than the
install dirs. Build dirs allow to check configuration logs and other
kind of logs which are the most useful when diagnosing a failed build.
Now install dirs are also interesting. Maybe we should provide them
again at some point. We'll see. For now I comment them out.
Still keep the install dir for dependencies though, since it seems this
is how data are passed from one job to another.
Note that ideally we would like to provide different artifacts depending
on failure or success but apparently this is currently not possible.
See: https://gitlab.com/gitlab-org/gitlab/issues/18744
Also not sure why for GIMP, the CI was only keeping the build app/tests/
directory. We should really keep the whole dir.
It was not necessary when I was only running the cross-build job. Not
sure why it is needed now. What do the parallel jobs share exactly in
this CI system? Anyway…
It looks like Arch does not have mingw64 cross-compilers in core package
repository. It does have some package in the user repository (AUR), but
I assume that such a repository cannot be deemed as safe.
Anyway I still tried, but apparently these AUR packages have to be built
and when I tried, I got this error:
> ERROR: Running makepkg as root is not allowed as it can cause
> permanent, catastrophic damage to your system.
Anyway it's all a big mess. Then I tried to move the cross-CI to Debian
testing, which is anyway our base compatibility system. Unfortunately I
encountered like what looked like some glibc++ macro problem on some
packages (most likely because the pre-built packages I use are Fedora
ones which likely uses a cross-compiler differently built from the
Debian one).
So in the end, for simplicity, I use a Fedora image, then I am sure to
get a good match between the system cross-compiler and the pre-built
dependencies.
This is the base system for our CI, and the former mypaint-brushes
package got bumped to v2, which broke the build. Instead of rolling
back, they added a mypaint-brushes1 package. Let's rely on it instead of
installing it ourselves now.
If I get it correctly, archlinux bumped the mypaint-brushes package to
v2, which broke our CI. The v1 and v2 brushes are not the same, and they
are not even compatible (GIMP only supports v1 brushes so far). These
should be different packages, hence the incremented major versionning.
For the archlinux change, see also:
https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/mypaint-brushes&id=dfd032b3aab40a46c751bd47713fb82bf11bd984
Let's just install the brushes ourselves from the correct branch.
With this kind of changes, as well as the weird patches they do when
they rename the pkg-config file of libmypaint, I think we should
consider not rely on this distribution for our CI.
We don't actually need any of the Python dependencies at runtime. We
only check for these to make packagers aware of the runtime need. Force
the build to go forward even with these deps.
Jehan