Run the file installation in multiple stages:
1) gather intel
2) unpack the archive to temporary files
3) set file metadatas
4) commit files to final location
5) mop up leftovers on failure
This means we no longer leave behind a trail of untracked, potentially
harmful junk on installation failure.
If commit step fails the package can still be left in an inconsistent stage,
this could be further improved by first backing up old files to temporary
location to allow undo on failure, but leaving that for some other day.
Also unowned directories will still be left behind.
And yes, this is a somewhat scary change as it's the biggest ever change
to how rpm lays down files on install. Adopt the hardlink test spec
over to install tests and add some more tests for the new behavior.
Fixes: #967 (+ multiple reports over the years)
We only use a temporary suffix for regular files that we are actually
creating, skipped and touched files should not have it. Add XFA_CREATING()
macro to accomppany XFA_SKIPPING() to easily check whether the file
is being created or something else.
No functional changes but makes the logic clearer.
We only use a temporary suffix for regular files that we are actually
creating, skipped and touched files should not have it. Add XFA_CREATING()
macro to accomppany XFA_SKIPPING() to easily check whether the file
is being created or something else.
No functional changes but makes the logic clearer.
Move setmeta into the struct too (we'll want this later anyhow),
and now we only need to pass the struct to fsmMkfile(). One less
argument to pass around, it has way too many still.
No functional changes.
Collect the common state info into a struct shared by both file install
and remove, update code accordingly. The change looks much more drastic
than it is - it's just adding fp-> prefix to a lot of places.
While we're at it, remember the state data throughout the operation.
No functional changes here, just paving way for the next steps which
will look clearer with these pre-requisites in place.
Handle rpmfiNext() in the while-condition directly to make it more like
similar other constructs elsewhere, adjust for the end of iteration
code after the loop. Also take the file index from rpmfiNext() so
we don't need multiple calls to rpmfiFX() later.
More and more macros, scriptlets and other functionality has been getting
built around Lua, to the point that it has in practice been required for
several years now.
Maintaining the pretense of being optional comes at a cost in holding
back developments and having to check for that theoretical special
case. Lets make it a hard requirement and embrace it some more!
These knobs and tunables were only ever used by the BDB backend which
exposed all manner of internals to macro configuration. With that gone,
we don't needs these struct members either.
Back in 2013, the Berkeley DB license was changed in a way that prevented
most of open-source world to go along, rpm was no different. We now have
other options and a standalone migration path from BDB for those that
haven't yet done so.
Whatever else might be said about this partnership, it has been a long one.
Now's the time to part ways.
Since it's introduction, rpmtsInitDB() has passed the second argument
directly to rpmdbInit() as permission bits. However commit
81fef98480 incorrectly documented this
as being related to the db mode read/write *mode*, and also used it
that way in the python bindings.
Previously, trying to run rpmbuild on a M1 machine returns "error: No compatible architectures found for build".
Since RPM detects its architecture on macOS just by looking at what architecture flags are defined at compile time, this adds a clause to detect aarch64.
Since commit c5f82d3f62 we've blocked
most signals during transactions, which makes sense to rpm itself but
the signal mask is inherited to childs and carried even across exec(),
so all scriptlets are executing with those signals blocked as well.
Which in turn does not make sense, the scriptlets could run stuff that
actually depends on signal delivery (such as SIGALARM in RhBug:1913765).
Unblock all signals for forked scriptlet execution (Lua scriptlets are
totally different as they execute in-process for now)
There is no harm in allowing read access in this case. We still
error out in the database rebuild case, just to be on the safe
side. We now have the following logic:
_db_backend unset:
* error out for rebuilddb or read-write access
* use detected backend and print a debug message
_db_backend unknown:
* error out for rebuilddb or read-write access
* use detected backend and print a warning message
_db_backend set:
* use detected backend and print a warning message if it
does not match the configured backend
If somebody is opening a database in a read-only mode (eg for query)
then we really have no business to *create* such database if not existing.
Update all backends to error out instead of trying to create the database
or its directory when opening read-only, add a test for this specific
behavior.
This implicit discard of read-only flag is such a long standing behavior
that this is likely to break stuff here and there.
When we do lazy rpmdb open in rpmtsInitIterator(), we also do a lazy
keyring open. Except that since the keyring typically lives in the rpmdb,
we PROBABLY should try open the database first. One of those "WTF I've
been smoking" moments, lol.
Prevents an ugly if mostly harmless double error anything we can't open
the database for one reason or another.
We only actually used gethostbyname() for canonicalizing buildhost,
convert that to use getaddrinfo() instead, which actually has an
option for retrieving exactly what we want.
The other "use" was to initialize name services, but as we don't need
or use hostnames for any operation, we can just as well drop it. User
and group names are what we care about.
Curiously, sqlite allocates resources that need freeing even in case
of failure to open. Quoting from https://www.sqlite.org/c3ref/open.html:
> Whether or not an error occurs when it is opened, resources associated
> with the database connection handle should be released by passing it to
> sqlite3_close() when it is no longer required.
I disagree, but as it's documented behavior there's no point filing a
bug. So lets close the non-open connection and chug on.
There's no point or need to do all this fluff on library initialization,
we can just as well do it when we're told to use a chroot by
calling rpmChrootSet(), at which time we're still on familiar ground.
Eliminating unused cruft from initialization can't hurt our start-up
times either.
As of Doxygen >= 1.8.20 it started complaining about anything marked
as @retval being undocumented. As this is widely used in rpm...
Mass-replace all @retval uses with @param[out] to silence. Some of
these are actually in/out parameters but combing through all of them
is a bit too much...
Also escape <CR><LF> in rpmpgp.h to shut up yet another new warning.
Fixes compilation on musl, otherwise it fails with undefined references
to various O_* symbols as mentioned here:
https://www.man7.org/linux/man-pages/man0/fcntl.h.0p.html
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
The code is more obvious and there's a whole lot less of it, we get
rid of a klunky global table, allow Lua scriptlets to process arguments
using native facilities for variadic functions, all in a backwards
compatible manner. What's not to like?
As a side-effect, the package count arguments now appear as integers
instead of floats, which is much saner in the context. Lua will
automatically convert numbers as necessary so this should not break
anything.
Add support for passing getopt options and arguments to natively to Lua
scriptlets via the internal API. The processed opts and args are stored
in two chunk local two tables, options keyed by the letter and arguments
by their numeric index.
Update call sites accordingly, but no actual functionality changes here,
just pre-requisites for next steps.
Add API to get and set callback style, defaulting to the good ole
header first style and add a new style where a transaction element
is passed in place of the header, allowing a much saner interaction.
This is doubly so in Python, where the old style callback is a
braindamaged mess. In the new mode the "key" is not passed as separate
argument at all, it can be just as well retrieved via te.Key() for
the callbacks needing it.
All this time we've been logging file and transfile triggers with output
identical to that of regular triggers, confusing people for no good reason.
We know which of the three it is when creating so add a matching prefix
to the output at that point. All we need to do is move the '%' from
the scriptlet table to the printf() format string to make room for
the description prefix. Doh.
So now we get output using things with their proper names, such as:
D: running post-transaction scripts
D: %transfiletriggerin(glibc-common-2.31-4.fc32.x86_64): scriptlet start
Add support for an optional callback hook for reveiving notifications
about added and deleted transaction elements. This lets API clients
easily track which elements get created and/or replaced by a single
rpmtsAddFoo() call and adjust their own book-keeping accordingly.
Also makes for an easy place to set application private data pointers
for newly added elements.
Currently there's no way to delete individual transaction elements from
the API but that can happen via rpmtsEmpty(), which we do send notifications
for. rpmtsFree() does NOT send notification though, it would be precarious
and most likely totally pointless to do so when the ts is being torn down.
The "key" passed to rpmtsAddInstallEleemnt() and associated with transaction
elements is sometimes mistaken for user data, but it is not, as rpm relies
on it having a very specific meaning and is passed around with that very
assumption.
API users have all sorts of (legit!) needs, lets add a proper application
private user data member to transaction elements.
The Python bindings to this are kinda dangerous if you liberally mix Python
and C as we can't know whether the pointer is an actual Python object or not
so we can only assume it is and hope for the best. Of course nobody in their
right mind should be setting user data from one language and accessing it
from another, really...
Contrary to what the TODO comment said, we cannot rely on chroot cwd
because an API user could've changed to some other directory in the
meanwhile. For Lua scripts, we simply need to save whatever the directory
we are in and return to that, to avoid breaking API users.
Should've been reverted as a part of reverting commits
0da3c50d1f and
8ab279ae6b but got left behind and Fedora
has been carrying this reversion as a patch since then (and other
distros carrying others). This equally broken for everybody -state of
ARM is good for nobody. Lets revert back to pre 4.15 state and back to
the drawing board.
This reverts commit 8c3a7b8fa9.
When %_minimize_writes is enabled, we determine unchanged files during
fingerprinting and only update their metadata (FA_TOUCH) instead of
always recreating from scratch (FA_CREATE) during install. However
package scriptlets (and administrators) can and will do arbitrary stuff
in the meanwhile, such as rm -f their own files in %pre, hoping to
get a fresh copy of contents no matter what. Or something.
Now, if the file was determined to not need changing by rpm, this will
just fail with chown & friends trying to touch non-existent file.
One can consider this a case of package shooting itself in the foot, but
when a package update fails or succeeds depending on %_minimize_writes this
to me suggests the feature is at fault as much as the package.
Do fsmVerify() on all files to be FA_TOUCH'ed to detect files whose
type changed or were removed since fingerprinting. This still doesn't
ensure correctness if something tampers with the contents in the meanwhile,
(for that we'd need to run the file through the whole machinery again,
checksumming and all) but covers the most glaring cases.
RhBug:1872141 (PR #1347) demonstrates how brittle and dangerous this
thing is. Until we can (re-)verify (digest and all) files not needing
recreating from fsm, this thing should be considered experimental and
too dangerous to automatically enable for casual users.
This generates signatures for all files in the archive, then picks up
ghost files from the header metadata and generates signatures for them
as well. It finally submits them to RPMTAG_VERITYSIGNATURES in header
file order as we cannot rely on archive order and header order being
the same.
Signed-off-by: Jes Sorensen <jsorensen@fb.com>
This adds the array of verity signatures, and a signature length
header. We use 4K block for the Merkle tree, and rely on the kernel
doing the right thing.
Signed-off-by: Jes Sorensen <jsorensen@fb.com>
This will convert a tag of base64 strings to a binary array, similar
to how hex2bin() works. It supports variable sized strings, and will
determine the maximum string length and build the output array based
on that.
Signed-off-by: Jes Sorensen <jsorensen@fb.com>
The rpmdb is our most precious piece of data, don't make assumptions on
invalid configuration. Together with our crazy create-db-on-read behavior,
total database loss is just one 'rpmdb --rebuilddb' away in some scenarios
with the former behavior: access an sqlite/ndb database with older
version not supporting those, silently fallback to creating empty bdb,
and if db is now rebuilt, poof the data is gone.
Detect and warn on unknown/invalid %_db_backend configuration and fall
back to using dummy backend where no damage can occur. Doesn't help with
the old versions out there, but lets at least be saner going forward.
A 2016 change (57a96d2486) apparently
changed tagsByName from dynamic allocation to being static, so that
Valgrind would not complain about lost memory. The definition is:
static headerTagTableEntry tagsByName[TABLESIZE];
But a comparison was left of `tagsByName == NULL` in lib/tagname.c
and compiling with clang gives a warning, saying it is never NULL.
The existing FSM code doesn't correctly handle FA_TOUCH on hardlinked
file sets, which causes the hardlink set to break on at least some
upgrade scenarios when minimize_writes is enabled.
Only enable FA_TOUCH on non-hardlinked files to work around the issue
for now. While at it, rearrange the conditionals around min_writes to make
it a bit clearer. Testcase adopted from original reproducer by
Fabian Vogt, thanks!
Fixes: #1278
The prefix search was so wrong it's a small miracle it ever did anything
at all. What have I been thinking? Well, I do remember thinking this
prefix stuff looks kinda fishy but then it seems to work so...
The prefix search belongs to the keyed iterator fetch case of course,
not the case where we're otherwise iterating over all keys.
Fixes: #1260
Commit 47e2463d8a added logic to enable
%_flush_io automatically on non-rotational disks to avoid trashing system
caches and IO peaks on the grounds that this isn't so expensive on SSD,
but real world experience suggests otherwise. Install times go from
seconds to minutes which might not matter for the random individual system
but for build systems and the like churning away continuously...
Since commits [Place file signatures into the signature header where they
belong][1] applied, run `rpm -Kv **.rpm' failed if signature header
is larger than 64KB. Here are steps:
1) A unsigned rpm package, the size is 227560 bytes
$ ls -al xz-src-5.2.5-r0.corei7_64.rpm
-rw-------. 1 mockbuild 1000 227560 Jun 3 09:59
2) Sign the rpm package
$ rpmsign --addsign ... xz-src-5.2.5-r0.corei7_64.rpm
3) The size of signed rpm is 312208 bytes
$ ls -al xz-src-5.2.5-r0.corei7_64.rpm
-rw-------. 1 mockbuild 1000 312208 Jun 3 09:48
4) Run `rpm -Kv' failed with signature hdr data out of range
$ rpm -Kv xz-src-5.2.5-r0.corei7_64.rpm
xz-src-5.2.5-r0.corei7_64.rpm:
error: xz-src-5.2.5-r0.corei7_64.rpm: signature hdr data: BAD, no. of
bytes(88864) out of range
From 1) and 3), the size of signed rpm package increased
312208 - 227560 = 84648, so the check of dl_max (64KB,65536)
is not enough.
As [1] said:
This also means the signature header can be MUCH bigger than ever
before,so bump up the limit (to 64MB, arbitrary something for now)
So [1] missed to multiply by 1024.
[1] f558e88605
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
For now we simply check whether the removed packages exist at all,
ie if somebody removed the package in the time between adding the
package to the transaction and running the transaction. One could argue
that whoever removed it already did us a favor so why bother, but then
that's not in the spirit of transactions.
RPMPROB_PKG_INSTALLED is (ab)used for the purpose as it's related
to the install status anyhow, and we can trivially adjust the message
as per the instance number.
NSS is a behemoth of a library which drags in a whole runtime subsystem
of its own which is often at odds with normal Unix system behavior
(hello SIGPIPE). Now that we have nicer alternatives available there's
little reason to lug this baggage along. NSS was deprecated in rpm 4.16
(commit 0b9efb93fb).
It's more than a little hysterical that rpm hasn't had a meaningful
public API for parsing and comparing version strings. The API may seem
kinda overkill for some things but then this does give us a place to
check for invalid version strings, which is a thing we never had before
(although the current code doesn't do much in the way of checking)
Fixes: #561
Epoch promotion was a thing around and before the turn of the millenium,
we really shouldn't be carrying that cruft around anymore.
Remove all traces that we can without too much guilt about breaking
ABI/API and not bumping soname which we dont want to do for this
stupid old thing: all the symbols are left in place, they just don't
work anymore. Nobody should notice as nobody can have been using this
stuff in the last 15+ years.
No functional changes, just lose extra baggage that was needed to append
initialize and append rpm.vercmp() from librpm side when the rest is in
librpmio. We might need to bring it back some day, but given the history
here it doesn't seem all that likely.
Adding a new header just for this seems a bit much but we'll be adding
stuff there shortly.
No functional changes as such, this is prerequisite for supporting
version comparison in expressions.
size_t is unsigned, so returning -1 is not going to have the expected
behavior. Fix it to return ssize_t.
Signed-off-by: Jes Sorensen <jsorensen@fb.com>
Fixes regressions from commit 75ec16e660e784d7897b37cac1a2b9b135825f25:
the newly added provides of to-be-built packages were being used for
dependency resolution, such as spec satifying its own buildrequires,
and matched against conflicts in installed packages.
Source packages cannot obsolete anything or provide capabilities or files
to transactions, don't add them to rpmal at all. Explicitly skip checks
against source provides, similarly to what we already did with obsoletes.
Fixes: #1189
This is normally unneeded because an index sync is already done
after each package is added to the database. But rebuilding an
empty database lead to no index sync being done and a mismatch
between the generation count. This in turn will trigger a
index regeneration.
Found by using ndb when running the test suite.
Somehow I was under the impression that mapped regions are unmapped when
the corresponding file descriptor is closed, but that's not the case
for POSIX systems.
Instead just map it read-write if one of the functions request
exclusive access. We keep track of the number of exclusive
locks and fall back to read-only mapping if the count reaches
zero again.
Pass the flags to rpmidxOpenXdb and use read only mode if the
user specified O_RDONLY. We already did that for rpmidxOpen in
the past but we always used read-write mode when using the Xdb.
We still open the Xdb itself in read-write mode so that we can
regenerate missing index databases.
We forgot to copy the mapped pointer if a slot was kept when
re-syncing after a generation mismatch. This led to the mapped
pointer being zero even if the map callback was set.
Also add an extra test for the map callback before setting the
protection flag to PROT_READ.
Found by testing the code with different mapping protection flags.
See the next commits.
Lots of cruft here - the build-aux move related changes from commit
cd6e4eb9e0, and all manner of historical
cruft that hasn't existed in over a decade. While at it, consolidate
it all to the toplevel .gitignore.
dirent.h and struct dirent are actually standard on this millenium, the
only thing that isn't is d_type member for which we have and retain
a specific test in configure.ac. Include <dirent.h> where needed,
relatively few places do which makes it even a bigger insult to have
this included from system.h.
Unlike that multiple statfs() variants, statvfs() is actually in
POSIX 1-2001 already and covers everything we need so there's little
point mucking with anything else. statvfs() is what Linux has been
using all along anyway this means no change on the primary platform.
If this actually regresses something, adding back OS-specific bits
is not a problem, but at least we'll get a clean start with that.
In the three years that LMDB support has been in the tree, and four
since upstream promised 1.0.0 in a couple of months, there have been
no upstream changes towards eliminating the key size limitations that
we need. And in the meanwhile it has become clearer that LMDB is not
the promised land it seemed on the outset, instead it has issues
like requiring the database size to be pre-determined (#902).
Drop support for LMDB, there's active development going on in the area
of database backends and we cannot afford to drag along an experimental
backend that is blocked on upstream design limitations and shows no signs
of moving forward. We can always bring it back if the upstream situation
changes.
In some scenarios we previously only created some of the indexes only
lazy db init through query. Partially initialized databases don't make
sense and are only asking for trouble, in particular this was causing
issues with sqlite backend which is stricter about readonly-mode.
Except for the special case of reading a potentially damaged database
for rebuilding, always open all the indexes from openDatabase().
Try the configured backend first, and only if that fails try autodetection.
The former logic did not anticipate multiple backends handling same
files and gets mightily confused when both bdb and bdb-ro are enabled,
causing half the test-suite failing in "make check".
Also emit a different message when database rebuild is in progress,
the old message is quite confusing in that case.
Loosely based on a similar patch by Michael Schroeder.
"RPM v3 is dead" has been repeated over and over in various forums for
the last 15+ years but as long as rpm itself silently accepts it, most
users will have no clue. Log a deprecation warning on V3 retrofits,
in anticipation of eventual removal.
Related: #1107
Add a new "meta" qualifier for expressing dependencies that are not
concrete install- or run-time dependencies, and thus should not take
part in install ordering.
There are quite a lot of such dependencies in the wild, for example
versioned sub-package cross-dependencies are typically of this type
and a common source of unnecessary dependency loops. Another common
case are dependencies of meta-packages.
Centralize the logic in one place to make it easier to add new qualifiers
These relations are only relevant during installation and are ignored at some
places e.g. when processing already installed packages. They are also not added
to the rpmdb indexes.
The way the qualifiers work is a bit peculiar as no qualifier means the
relation is always relevant. Adding any qualifier restricts the the relation
to this particular place and time. But adding more qulifiers do not further
restrict the validity of the relation but adds more relevant cases.
So here there are three cases:
No qualifiers, only install only qualifiers, install only qualifiers that
get overruled by others
SQL LIKE is a quirky thing when you only really want exact matches
on a substring. SUBSTR() looks promising until somebody points out
that it counts unicode characters, whereas rpm always deals with bytes.
Implement a custom MATCH() operator which operates on arbitrary bytes
to handle it the way other backends do.
This can be used to recover as much data as possibly from a
terminally broken database. It works by scanning the database
file for entries that are not corrupt.
The rpmalCreate() function is called in two places, in both of
them the arguments are all read from the transaction set.
So simplify the code by passing the ts to rpmalCreate() and
getting the values in the constructor.
If the entry/subdirid parts of the fingerprint are equal to the
ones of the last added fingerprint, we know that calling
fpLookupSubdir() will not modify anything. So skip the call in
that case.
Try to auto-detect non-rotational disks on start of transactions,
and if all involved disks are non-rotational enable optimizations:
%_minimize_io to conserve writes, and %_flush_io to avoid trashing
the system caches.
The configuration logic relies on always pushing a new definition
for these macros so we can reliably undo at setSSD(): ensureMacro()
pushes the current value again if one exists, otherwise the supplied
default is used.
Update macro documentation as per the new behavior.
Fix the logic for enabling flush_io and minimize_writes -behaviors,
-1 generally means automatic, 0 disabled and 1 enabled. But these
are by no means mandatory macros so comment out the default setting
for minimize_writes as well.
This will detect some "shouldn't happen" cases such as rpm indexes
pointing to non-existent packages, which is far from covering full
database consistency but much more than at least BDB/LMDB can do.
PRAGMA integrity_check returns a single "ok" line if everything checks
out but otherwise error messages get returned, so we need to actually
check the returned values.
Also change the way return code is constructed in preparation for the
next commit.
When we encounter a symlink, we have to restart the function from
the beginning. Instead of duplicating the setup code, add a new loop
and make the code do a iteration for each symlink.
fpLookupSubdir() purpose is just to patch the fingerprint to
account for symlinks. It's much cleaner to set the hash after
the call to fpLookupSubdir(), and we can also skip the call
if no symlinks were found.
Parsing a spec, even unsuccessfully, will affect the global macro
state in any number of ways that may affect the following operations
in unpredictable ways. Lacking any saner way to do this, reset the
entire global macro state after each spec parse in rpmspec and spec
query code (rpmbuild already does this) while maintaining possible
cli-specified target and rcfile.
isUnorderedReq() returned True as soon as any qualifier that does not
require ordering is passed. But some qulifiers - basically the scriptlets
run during installation and erasure of the package - may still require
the dependency to be taken into account during ordering.
Now isUnorderedReq() returns 0 if any of those are also set.
Resolves: #1030
This commit implements a read-only backend that allows accessing
of BerkeleyDB databases without using the BerkeleyDB library.
The code supports btree version 9-10 and hash version 8-10.
There are two use cases for this:
1) Conversion of an existing BerkeleyDB to a different
backend.
2) Allowing package scriptlets to do database queries while
in a transaction that replaced rpm with a version that
no longer links against BerkeleyDB.
If both BerkeleyDB and the read-only backend are enabled, rpm will
default to BerkeleyDB.
We compare the user generation stored in the index database with
the generation count from the package database. In case there
is a mismatch, we delete all the index databases and rely on the
already existing missing index creation code to rebuild the
database.
This will delete all blobs (i.e. index databases) from the
ndb master index database. We do this kind of low-level on purpose,
so that it even works if the index database is corrupt.
This will be used in the next commit which implements automatic
index regeneration if the index is out of sync.
This seems to saner as the somewhat unreliable current time.
This element is not used in normal database operation, it is
for repair tools that need to decide which blob to take if two
blobs have the same pkgid.
The generic case was reported in #270 and fixed quite a while ago in
commit 34c2ba3c6a, but signing uses a
different code path and require the same treatment.
Fixes: #1002
This is the recommended way in the manpage; if there is
a lingering error from an unrelated dl*() call that was
never obtained via dlerror(), it needs to be cleared
prior to calling dlsym().
Spinning new cursors for each key value is just mind-bogglingly stupid
(if simple), use a sub-cursor with key binding instead. Which isn't any
more complicated, really.
Caching statements is not stupid but when it makes us go slower...
Also without the cache we have a better handle on locks and all,
revisit the caching issue later.
Rpm can reuse an iterator cursor for a write when rewriting an existing
header (such as when updating file states of already installed package),
which in sqlite terms turns a SELECT into INSERT, destroying the
iteration in progress.
Detect the condition and use a local cursor as necessary, this also means
we don't need to track the query fmt string in cursors.
In normal transactions this is but a drop in the ocean. However on
rpm -Va, the hashes get rebuilt from scratch on every single package
which on my moderate rpmdb (~2800 packages) testcase results in
103418347 data values fetched and added to dbi sets that are only
thrown away.
With bdb and lmdb this is only a minor optimization but for ndb and sqlite
which can retrieve keys independently, this is a much bigger win. In case
of sqlite, it's a massive one.
The regular index iterator grabs the associated data too, which we
don't always need. The data associated with indexes is relatively
lightweight, but as with everything, it adds up if in the millions scale.
Update all backends to allow for NULL set in the index retrieve to
signal key-only retrieval. Ndb actually had an earlier, abandoned
implementation of the same idea under slightly different API, lets
reuse the code-block.
Postpone sql index creation until closing time on database rebuilds,
this is more than 50% improvement on rebuilddb time on both SSD and
traditional disks.
As of sqlite 3.22.0, a database in WAL mode can be opened readonly
if one or more of the following is true:
1) The -shm and -wal files already exists and are readable
2) There is write permission on the directory containing the database
so that the -shm and -wal files can be created.
3) The database connection is opened using the immutable query parameter.
Regular users running queries cannot have permission to create, and
immutable databases can only exist on readonly media (because all locking
is disabled) so there's no choice but to leave the -shm and -wal files
around at all times, a little ugly as it might be.
Handle read-only database separately for table and index initialization
as they are different: if table cannot be created we're screwed and
should return an error, but if an index is missing it'll still work.
Also only return an allocated dbi if everything went okay.
While flushing IO constantly is horrible for performance on spinning
disks, it's beneficial on SSDs. So if %_flush_io is enabled, let sqlite
do its regular WAL auto-checkpointing. Pure DB install time on SSD
improves by some 25% here with %_flush_io, at cost of removal speed but
for overall sanity and smaller WAL file.
This is an enormous performance boost to almost all operation modes,
in particular spinning disks - on my old laptop, time to erase ~3300
packages comes down from half an hour to under three minutes.
By default sqlite likes to keep the WAL file relatively slow to keep
reader performance up, but as writes are rare in rpm, we let the WAL
grow unlimited during transactions and only flush it at database
close. This way transactions are fast and at the cost of read
performance during transactions, which is perfectly okay. To do this
we need to move statement cache free earlier to drop any locks that
might be lurking there.
Special handling of RPMDB_FLAG_REBUILD is dropped here as unlike other
modes, WAL is persistent and has interactions with EXCLUSIVE locking_mode
among other things that seem non-trivial to sort out. This has some
performance cost to --rebuilddb but it's a rare operation and normal
transaction performance is far more important. We can always tweak
it more later on...
Only create key indexes for strings, those are the ones that are
performance critical and anything else is just unnecessary weight.
Additionally create indexes on hnum for better performance especially
on erasure, but only bother for array type data where the amounts
get large enough for indexes to matter.
As we now cache the statements, not reseting the statements kept most
of the database locked for the whole transactions, preventing concurrent
access.
Logic seems to suggest that we could then not bother reseting on fetch
from cache, but that causes failures so there's something fishy still.
sqlite_init() relies on db_dbenv pointing to NULL to properly initialize,
not sure if we can actually end up reusing the same handle but just in
case...
Upstream sqlite defaults to secure_delete off, but its default value
is also a compile-time option which some distros (at least Fedora)
change to have it enabled by default. There's never any sensitive data
in the rpmdb, don't bother zeroing it out. Improves performance somewhat.
The initial implementation could recycle the topmost header numbers which
breaks libsolv caching as it assumes the traditional rpm behavior where
header numbers are not recycled.
Sqlite primary integer key with autoincrement behaves exactly the way rpm
database has traditionally allocated new "instance" numbers, and does not
seem to have any performance impact in this usage as there's always a
limited number of such increments anyway. Let it do the job instead.
Use sqlite3_table_column_metadata() to detect whether a table exists.
This will allow newly added rpm indexes to be automatically generated
and doesn't require running sql queries to figure out whether the
database was just created or not (and get it wrong because an empty
database would appear as newly created)
Previously, we assumed a backslash character would always be followed by
a character to be escaped, and advanced our "start" pointer by two
places before the next iteration. However, this assumption breaks if
the lonely backslash happens to be the last character in the query
string, in which case we would end up pointing beyond the \0 and let the
parser wander into the unknown, possibly crashing later.
This commit ensures we detect this corner case and error out gracefully
with a message.
LGTM flags localtime() as a "dangerous" function, which seems a bit
over the top to me, but as we're flirting with threads, it certainly
is not thread-safe.
We only permit comments at beginning of line in specs and macro
files too, of all things file manifests don't need anything fancier.
Resolves the oldest rpm bug in RH bugzilla, only took 16 years...
Now that we can, set db_descr centrally on database open instead of
relying on backends to do it (and forget, or leak memory, as has been
the case). Also don't bother mallocing, the name of the backend is
quite enough.
With backends knowing their own names we could probably eliminate db_descr
entirely but leaving that for another rainy day, it's possible there
are code paths that assume it being set to something.
Preparing SQL statements from textual representation is relatively
expensive and we do it a lot. Add a refcounted hash table for
reusing the statements, which roughly halves the time of rpm -Va,
other operations will naturally benefit too.
db_cache should really be a private member someplace in the sqlite
backend but there's no place to hang it, so without bigger changes
that are out of scope here, the only option is adding a new member
to the rpmdb struct itself.
The rest of rpm expects to get a pointer to a temporary data area
containing the header blob, which is then copied on header import.
Seems my recollection of this was the exact opposite, so the sqlite
backend was pushing back malloced copies of the blobs that nobody would
then free. Oops, leaks, leaks, leaks, sunk.
There's no need for sqlite to be different in this regard, but during
Packages iteration the implementation used a temporary cursor for
the actual header, causing the data to go away before reaching the
upper layers which necessiated dup'ing the data, that nobody would free.
Refactor to eliminate the temporary cursor from the picture to fix the
mismatch of expectations and thus the leak.
RPMDBI_PACKAGES differs from the indexes in how record searches behave,
indexes return NULL for non-existent values but with RPMDBI_PACKAGES
(and the pseudo index LABEL) we always get an iterator which we need to
walk to determine whether there's data or not. Use the newly introduced
helper to get the query return right.
One could argue that this should be done in the database code, but then
the behavior is kinda consistent other iteration over RPMDBI_PACKAGES:
you don't know how many, if any, matches there are until you walk it.
During build, also calculate a digest for the uncompressed payload data
and add to packages. On verify side this is equivalent to the existing
payload digest (through sharing the same disabler), which allows
either one to be used for verification. This means deltarpm and similar
don't need to recompress the data which is both expensive and error-prone
due to minor differences in compressed stream despite the actual data
being identical.
Add a testcase for the basic behavior and update other test output
expectations where necessary.
rpmdbStat() can be used without an open rpmdb handle similar to stat()
and rpmdbFStat() takes an open rpmdb handle, akin fstat().
The rationale for these calls is to give API users a backend-independent
way of obtaining data about the database's existence, timestamps etc.
Refactor the hand-written separate tests into an array walk now that we can.
No functional changes, except that there's no more special fallback for BDB.
Found by torture-testing the ndb database.
* Usedslots was allocated with the wrong size
This did not matter for rpm because rpm uses only a small
number of index databases
* The addslotpage function did not enqueue the new free slots correctly
It never gets called in rpm
* The protection bits were not set if moveblobto needed to map a blob
This can happen if it is called from the moveblobstofront() function,
it will just not shrink the database in the current call.
On sqlite, deleting doesn't require painstakingly recreating the same
exact data that was used for putting, we can just sweep out everything
with the given header num at once. Cuts roughly 1/4 out of erase time.
Let the backend see update of a single index as a whole instead of
single items fed by the upper layer. This allows backends to optimize
these operations in various ways that haven't been possible previously.
Pass header number to pkgdbPut() as a pointer so we can return values too.
Having allocation as a separate call unnecessarily dictates details
about backend implemenation, and this makes it all just that little
bit simpler. No functional changes.
All normal functionality is expected to work. Automatic generation
of missing index tables is missing, but that's not relevant at this time.
Going forward, we'll also want some sort of compatibility tracking
for the sql schema.
The database scheme basically just mirrors what BDB does, using strings
for strings and blobs for everything else due to the way integers are
handled in the sqlite C API, for now at least. Some amount of schema
changes are to be expected before this is considered final.
Performance is similar or better with BDB in the current unsafe CDB
model, but sqlite uses proper database transactions so this is expected
to be an order of magnitude more robust.
Many things are stupid and/or kind of backwards here due to the internal
API, which I've avoided changing in order to keep it backportable for the
time being. https://github.com/rpm-software-management/rpm/pull/836 is
needed but otherwise this should drop quite trivially into 4.14.x too.
However as we're planning for a longer term future here, it would be dumb
to limit ourselves by what's possible with an internal BDB-oriented API,
so I've fairly major changes planned in that direction.
ndb doesn't implement a specific verify option, but that doesn't mean
the data within should be considered invalid. This causes ndb to fail
the test-suite on "rpmdb --rebuilddb and verify empty database" for no
good reason. Similar arguments could be made for dummydb although it
matters much less there.
This is part II of commit 6d610e9b9a which
missed one but common case where the element with matches gets passed
to the callback instead of the owning one, as pointed out in RhBug:1724779.
rpmlua.h was originally written in a way that allows it to be included
regardless of whether Lua is actually enabled in rpm or not, or where
Lua headers are, specifically to isolate the rest of rpm from these
details. That was changed in commit 62bd62286a
when <lauxlib.h> started getting included in rpmlua.h, which leaks to
places like librpmbuild which do not directly use Lua.
The way Lua typedef's the luaL_Reg struct to itself defies my C fu for
for handling this in some nicer typesafe way, fix this all by just using
a void pointer instead, this is just an internal API where buyer can be
expected to beware.
Fixes#888
These messages have been an endless source of confusion and complaint
throughout their existence, to the point that various programs have added
their own "translation" for these messages. Changing the message is likely
to break those (regex-based) translations but then hopefully the
translations will not be needed after this.
Fixes#879 (and at least a dozen bugs in various distro bugzillas
whose numbers I'm too lazy to dig up)
headerExport() is the only thing that needs offset sorting, there's
no point messing up with the header itself for this when we can just
sort a copy instead. No visible functional changes, but makes
headerExport() a read-only operation to the header, as it should be.
This also makes h->sorted a straightforward boolean indicating whether
the index is currently sorted or not instead of the strange tri-state
thing it has been (see commit da3a3a14e7)
No functional changes, this is just to create a barrier between
access to the header structure and what is actually needed from it,
ie just the index. This makes the next step much more obvious.
The primary motivation here is to consolidate all database writes
(open, write, close) on one side of the chroot, currently it happens on
both sides of the border causing all sorts of issues and limitations (such
as preventing more advanced modes of BDB, not to mention other databases).
As a positive side-effect, the sections where we potentially run
inside chroot are more easily identifiable.
Consolidating on the outside may seem counter-productive, to improve
security it seems you'd want to spend as much time *in* as possible,
including database accesses. Unfortunately due to rpm's access patterns
and API promises, that's not really achievable (tried several approaches,
run into as many dead-ends).
Technically we could localize the chroot placement much further, but
doing so would change the side for transaction callbacks, which could
cause nasty breakage for our API users as various clients use those
callback slots to update their own databases and logs. So the chroot
spots here are selected to cover minimum possible code while preserving
the chroot side of callbacks and plugin slots: RPMCALLBACK_INST_OPEN/CLOSE,
ELEM_PROGRESS and VERIFY_* occur outside the chroot, everything else inside.
Of plugin slots, init/cleanup and tsm_pre/post occur outside, everything
else inside.
hdrblobGet() introduced in commits acfde0d0e8
and 9821de1881 has an off-by-one thinko
(perhaps the idea was to skip the first, region tag) which causes
the last entry to be unreachable. In typical packages, that is
RPMSIG_PAYLOADSIZE which is not used at all in this context so it doesn't
matter, but in large packages use RPMSIG_LONGARCHIVESIZE which has a lower
tag number and leaves either RPMSIGTAG_MD5 or RPMSIGTAG_GPG last,
unreachable and thus unverifiable. Oops.
This fixes the regression introduced in rpm 4.14, affecting verification
of large packages (ie having RPMSIG_LONGARCHIVESIZE)
Rpm used to rely on the "replacepkgs hack" to get rid of the old
header entry when reinstalling a package. This has a number of
problems when the headers are not identical or different
install flags were used.
To mitigate this, a '--reinstall' option was added that made rpm
use an erase element in this case.
This commit takes this one step further by changing the code to also
use an erase element in the --upgrade case. The code is mostly simpler,
but we need a different implementation for commit fd40d58efa, as we now
have erase elements both for --reinstall and --upgrade. Thus we
need to store the addop in the transaction element.
The commit does not change the behaviour of 'rpm --install'.
Demi M. Obenour