Our file classification is not exactly in large quantities. Fedora's
kernel-debuginfo-common has circa 25000 files, and classifying them
serially takes about a minute on my rusty old T520. With parallelization
this goes down to ~24s.
It's all remarkably simple, except for the fact that libmagic is not
thread-safe so we need separate magic handles for each of our threads.
This will leak those libmagic handles on error situations, I don't see
any obvious, nice way to handle that.
It's worth noting that this really is walking on thin ice as only few
parts of rpm are thread-protected. The spec is entirely unprotected so
must be accessed only for read-only purposes from parallel jobs (and
we should work towards enforcing that via const-correctness and other
protection as needed), and similarly the package struct and headers
are unprotected so they can only be manipulated within a single thread.
Based on initial work by Alexander Kanavin in PR #226
Enable OpenMP use in librpmbuild and set the number of OMP threads
from rpm config after spec parsing. The place matters as we want to
allow individual specs to control and disable parallel builds.
Now that we can, split the "lets run something on generated packages"
check out of packageBinaries(), it doesn't belong there at all. No
functional changes other fix attempt to check non-built packages
which return with no filename but RPMRC_OK from packageBinary().
These are clearly separate operations that deserve functions of their
own, and this makes memory management nicer. In addition, in case
the directory creation fails we now actually error out instead
of trying to continue, and take care not to fail in case somebody
created it behind our back. No other functional changes though.
Commit 6dafb24684 introduced a regression
where patch/source (and likely buildroot and docdir too) values
end up leaking to macros of other tags, due to missing reinitialization
to NULL in the code.
This only shows up in specs ordered in a specific way which is why
our test-suite failed to trip it, but it caused eg Fedora attr.spec and
kernel.spec to fail parse. Hijack the otherwise unused foo.spec to
create a test-case for this.
The ordering of conditionals is tested according of data
in structure branchTypes. It makes the test well-arranged.
Moreover the error for using %else after %else is added.
When adding update elements, we set the erase element to depend on the
install element, but if an API user adds the same erasure manually
after adding the update, we know its a duplicate erasure and filter
it out, BUT we zero out the dependent element in the process. And
if installing the update now fails, we end up removing the whole package
due to that missing dependent element.
This never happens with rpm itself so we can't easily test it, but is
100% reproducable with dnf (at least dnf 3-4). Apparently it adds all
erasures by itself (which is kind of understandable I guess, perhaps
we should better allow this in the API)
This introduces two new spec sections, %patchlist and %sourcelist,
which can be used for adding patches and sources by just listing their
names instead of all the tedious Patch[n]: etc boilerplate, you can
just copy-paste file names into the list, the entries are always
autonumbered, eg:
Patch0: popt-1.16-pkgconfig.patch
Patch1: popt-1.16-execfail.patch
Patch2: popt-1.16-man-page.patch
...
can now be replaced with
%patchlist
popt-1.16-pkgconfig.patch
popt-1.16-execfail.patch
popt-1.16-man-page.patch
Typical packages have far fewer sources than patches, so %sourcelist
is not as immediately useful but added anyway for symmetry and because
its so easy.
As of this commit, there can be multiple instances of both kinds of
lists because there's no technical reason to limit it, new lists
just add on the existing entries.
Makes it possible to feed patch and source names directly to
addSource() using autonumbering without looking at an existing
spec Patch[n] or Source[n] line. In addition, make addSource()
available within librpmbuild.
No functional changes here, but needed for the next step.
Unlike typical fooFree() functions in rpm, Fclose() doesn't set the
pointer to NULL so there's a use-after-free in checking for Ferror()
that segfaults and stuff. Delay Fclose() until the end so we actually
catch io errors too, that was another thing that went missing in
commit 0f21bdd0d7 (although it would've
probably caused an error via null digest instead)
We need to expand these things once per file attribute, not per file.
It saves a huge amount of huffing and puffing about, but doesn't
show up on wall clock due to the generator script interaction being
so dumb and slow. However, now we have the operation arranged in a
way that would make smarter generator script running actually possible.
Compile the exclude-regexes once per dependency type instead of per
every file. Saves a huge amount of huffing and puffing about, but
doesn't really show on wall clock as the time goes to even bigger
stupidities.
This alleged solution to the problem of not all scriptlet errors being
reflected in our exit code is not a solution at all, it just pushes
the problem to somebody elses lap and introduces inconsistency and
an untracked incompatibility as well.
This reverts commit 5455f02523.
Since we don't have proper user/group info inside the user namespace we
set up, avoid going there unless we have to. Fixes a regression introduced
in commit b4c832caed where non-root,
non-chroot verification shows user and group differing on all files.
Back in the turn of the century somebody thought it was a neat idea
to completely compromise system security to improve program start-up
start-up times a wee bit. Since then, people have thankfully started
coming to their senses and removed prelink from distros entirely.
Lets stop papering over the security disaster: we obviously cannot
stop people from using prelink, but instead of trying to undo the
damage for verification purposes, we'll now report such a system as
compromised. Which is how it should be, IMNSHO.
This eliminates a whole lot of extra junk from each and every file
digest calculation that we do, so it might even show up on somebodys
performance charts. It also gets rid of libelf dependency outside
librpmbuild, which is a nice little bonus.
Inspired by a patch to eliminate a rendundant double open of regular
files in rpmDoDigest() from Denys Vlasenko, taken a little further...
There are numerous tags that have deeper internal logic, can appear
multiple times and where the line of last occurrence doesn't mean
anything at all, and is just confusing. Witness RhBug:555926 and
ticket #689.
Drop the macro creation for all dependency tags, build architecture-
related bits and things where it just doesn't make any sense, like
NoPatch/NoSource and AutoReq/Prov.
In theory this could break somebody's spec somewhere out there but
those usages would be better served by using custom macros instead.
Closes: #689
Not all the world is autotools based, mercifully. Let the other build
systems benefit from the central rpm macros as well by splitting
the XXFLAGS setting out of %configure.
Based on Florian Weimer's work in Fedoras redhat-rpm-config macros.
%{optflags} has been the catchall for all compiler options but this
is quite limiting as there's no way to add for example C++ specific
options distro-wide. This adds separate %build_cflags, %build_cxxflags,
%build_fflags for the gcc-supported languages, and additionally
%build_ldflags for distro-wide LDFLAGS setting.
Based on Florian Weimer's work in Fedoras redhat-rpm-config macros.
When called in spec context, the package structs are properly freed
but in rpmdeps context, commit 49f2bb7d8f
only added a "dirty kludgery" to make it not blow up. This causes
the rpmds structures created in rpmfcApplyInternal() to leak memory.
Make freePackage() internally available and use it for freeing the
dummy struct too to fix.
Problem was in a buffer over-read of the memory right after
the end of the allocated area when expanding of an unfinished macro
starting "%{" that can contain only exclamation marks and question
marks. Like "%{", or "%{!", or "%{??!".
Similar problem as in commit 54f24ec548
with the difference that this problem was not found by a memory
sanitizer.
This is a good opportunity to refactor the corresponding code
(setting variables according to the number of exclamation marks and
question mark after % an before the macro).
The change made in fc2c986 can break for large values of %_smp_build_ncpus as
this many processes are able to overflow the following pipe.
Thanks to Denys Vlasenko for testing this.
This change solves this problem by running a whole processing pileline for each
parallel (file) process. This has also the benefit of running at least some
stip commands in parallel.
The -n param fro xargs was increased to 32 to further reduce the over head of
spawing the helpers as they are now needed for each run of the file command.
If multiple arguments are passed to rpm2archive, convert them all to
tgz. (Previous behavior was to convert only the first one and
silently ignore the rest.)
Signed-off-by: Robbie Harwood <rharwood@redhat.com>
Co-authored-by: Florian Festi <ffesti@redhat.com>
The split is causing more and more issues over the years. It simply
doesn't make much sense to split them. When using python-rpm, it doesn't
matter whether it brings some additional tools like gpg because you
already have python installed.
Signed-off-by: Igor Gnatenko <i.gnatenko.brain@gmail.com>
Makes the code much easier to follow and paves way for the next steps.
Also eliminates nasty tricks where we modify and restore characters
directly in the spec line buffer. No functional changes.
Return a signed integer instead of unsigned one so its clear the
return code is just that, and the actual number is via the pointer
retval. Also having -1 as an error code is more obvious than 1.
No functional changes.
Now that we easily can (because the on-disk path is in the source struct),
refactor the download attempt into a separate helper and do it as the
very last thing. Makes the code much clearer and actually fixes a
memory leak on failed download as well.
Icons are entirely different beasts from sources and patches and even
stored in different places, handling them in the same function makes
no sense at all. No functional changes, just refactoring to cleanup.
Panu Matilainen