- Nothing outside rpmte.c should be able to muck around with the
file info set associated with the ts element. Except to free
memory, which is the only thing rpmteSetFI() is now being used for.
- Grab files and file states from the main loop and pass as arguments
to skipInstallFiles() and skipEraseFiles(), neither actually
needs the transaction element otherwise.
- Use local iterators in both skip*Files(), not that it actually
matters here other than eliminating rpmteFI() uses
- All the called functions expect rpmfiles anyway so might as well
grab that from the start. We could probably do some of this with
iterators now but as everything here is done with the random-access
functions, switching would take quite a bit of work...
- These all operate on rpmfiles, not rpmfi, now so make the point
clearer. All internal stuff so we're free to mess around.
- No functional changes, only a straightforward perl-assisted rename...
- The self-iterator in rpmfi prevents all sorts of sane uses of
file set iteration. Split the actual data into a separate data
type, changing the internal random-access functions to use the
new rpmfiles datatype directly and update internal callers minimally.
This should be entirely transparent to public API consumers who still
only see the braindamaged self-iterating version.
- Internal consumers dont directly benefit (yet), this is just an
early step on the road to rpmfi sanity. Much of the API and variables
will need renaming etc but avoiding that here to keep the changes
to minimum, this is a rather huge commit as it is.
- The DB backend is supposed to take care of syncing when necessary,
syncing all indexes "just in case" whether they were written to
doesn't make a whole lot of sense.
- Previously %pretrans and %posttrans were tied to --nopre and --nopost
disablers (since commit 0b2d7775c5)
because back then, there was no room new disablers in rpmtransFlags
bitfield. This is no longer the case as of rpm >= 4.9.x where
a bunch of obsolete flags were axed, so we can now add specific
--nopretrans and --noposttrans switches + corresponding flags.
- This is obviously a behavior change as --nopre and --nopost no
longer affect %pretrans and %posttrans, but --noscripts behavior
remains the same.
- Lazily initialize plugins when adding install or erase elements
to the transaction already. Since this ensures the setup will
always be performed, there's no need to do this from rpmtsRun()
anymore, empty transaction would not get that far anyway.
- This will allow adding hooks to the places where package header
is still available (which is required for various things) and
in general, before transaction is actually executed.
- The downside here is that adding or removing packages to a transaction
can now fail due to plugin initialization failure where its not
easily distinguishable from other errors, but then they can also
fail due to rpmdb open failure which kinda similar "shouldn't be
here" issue. The alternative would be initializing the plugins
in rpmtsCreate() but as that's not actually permitted to fail at
all, its not an option really.
- We hereby unceremoniously declare that from now on, SELinux and
other similar mechanisms are to live in plugins, rpm doesn't need
to know about every possible security etc mechanism there might be.
Its a big commit but as its really just removals...
- We need to disable sepolicy collection plugin for now as it relies on
built-in support for selinux, this not only makes no sense but
wont work now that there is no built-in support. The sepolicy
plugin needs to be merged into the selinux base plugin now.
- Another "breakage" is that --rebuilddb no longer relabels the database,
this needs plugins to called in a place where they currently cannot
be called. This needs to be resolved before next actual release.
- For now this doesn't do anything particularly interesting, it
just replaces the transaction lock handle we use internally.
We want to export this interface but that needs further changes.
- Explicitly test for shared file/dir being ghost when deciding whether
to skip create: we must never skip the creation of a real file/dir
because of a ghost as the ghosts do not get created.
- This fixes the shared ghost testcase, ie the thing that commit
a712252392 was supposed to fix,
without side-effects that I can see.
- Reverts the bad half of commit a712252392eca75443ca45c10a72873cabec7963:
treating files that wont be installed as skipped seems logical enough,
but rpm disagrees... making %ghosts FA_SKIP on install causes
upgrades on %ghost %config files to miss the necessary FA_SKIP getting
set on the erase half via handleInstInstalledFile() and
rpmfiDecideFate(), causing %ghost %config files to be nuked. Which isn't
very nice when your /var/lib/rpm/ contents are packaged as
%ghost %config...
- This fixes the %ghost %config testcase but breaks the shared %ghost
case in turn.
- Only skip shared file/dir creation if its actually being created
by another package. Previously we could've decided to skip an entry
where the other file was also being skipped for some other reason.
- Ensure %ghost entries always have FA_SKIP on install, previously
they often were FA_CREATE which makes no sense for %ghost. We dont
encounter %ghosts at all during install in the FSM as they dont
exist in the payload, but the file disposition calculations need
to take them into account now that we're avoiding redundant creates.
- Fixes a regression from commit f7f5f88f9f
which unearthed the %ghost FA_CREATE issue, reported here:
http://lists.fedoraproject.org/pipermail/buildsys/2013-January/004047.html
- Previously the pool would only get allocated on successfull addition
of install or erasure elements, causing assert() failures on
operations on empty transaction set, which should be just a no-op,
not an error.
- Make rpmtsPool() create the pool if it doesn't exist, update relevant
users to call rpmtsPool() instead of directly accessing tsmem->pool,
this avoids having to worry about pool existence in all the various cases.
- Also fix up the pool-related comment on rpmtsEmpty(): pools does not
and can not support emptying as it could break references to its
contents. Per-string refcount would be needed for emptying support.
-All post hooks take an additional rc argument that indicates general return code from operation.
-All post hooks are always called if correspoding pre hook has been called.
-The return value from post hooks is curently ignored
Signed-off-by: Panu Matilainen <pmatilai@redhat.com>
- Since commit f7f5f88f9f, only the first
instance of shared files is created, the rest are FA_SKIP which
consume no disk space and are not counted either.
- When updating packages, we first create them with a temporary names
and only after all files from payload have been created this way,
the files are renamed to the final target. This means that performing
an update temporarily requires roughly twice the disk space (and inodes)
compared to the final result on per-package level. Which matters
when space is tight, such as presumably in RhBug:872314.
- Simulate what happens on upgrades by adding block and inode delta
to the equation: installing a file always consumes an inode and
the specified amount of disk space. But when replacing files,
reduce the size-delta from disk consumption *after* checking for
problems in a given DSI.
- Also fixes inode accounting which has been broken for forever (since
commit a9a1fd866c more exactly)
This change adds a new type of the rpm plugin, called transaction plugin
and a set of initial hooks for this plugin. The hooks are:
PLUGINHOOK_TSM_PRE
Pre-transaction hook that is called before an rpm transaction begins
PLUGINHOOK_TSM_POST
Post-transaction hook that is called after an rpm transaction ends
PLUGINHOOK_PSM_PRE
Pre-transaction-element hook that is called before an rpm
transaction-element is processed
PLUGINHOOK_PSM_POST
Post-transaction-element hook that is called after an rpm
transaction-element is processed
PLUGINHOOK_SCRIPT_SETUP
Per-script hook that is called once for each rpm mainainers script
that is present in the package
Each hook is called for every plugin that have this hook registered.
The avaliable transaction plugins can be specified in macros.in via
transaction_plugins element.
Signed-off-by: Panu Matilainen <pmatilai@redhat.com>
- In the package/transaction related things the strpool is more of
an internal implementation detail than an end-goal in itself, move
string pool related interfaces of rpmts, rpmfi and rpmds to
internal-only APIs for now. The kind interfaces we'll want to eventually
export a) dont exist yet and b) are likely to be something very different.
- The string pool itself remains exported however, its a handy data
structure for all sorts of things and both librpm and librpmbuild
heavily use it already.
- Transaction preparation fingerprinting now shares all the strings from
rpmfi's etc. Rpmdb basename iterator still needs a private pool
as there's nowhere to get a pool handle (for now at least).
- We no longer need to (and actually we couldn't) freeze the
pool before fingerprinting.
- rpmfi cannot know anything about the storage, so rpmfiFpxIndex()
cannot be... change it to rpmfiFps() which only returns the pointer
we got from fpLookupList()
- Change fpCacheGetByFp() to assume it gets passed an array of fps,
and take an additional index argument. Return the fingerprint
pointer on success, NULL on not found to allow further operations
on the fp without knowing its internals.
- If the fingerprint pointer passed to it is NULL then allocate space
for a new fingerprint, otherwise reuse the previous space. This should
allow optimizing the case where repeatedly calling and directory
doesn't change inside fpc so callers dont need special-case code
for this. For now, we dont care about optimizations, other than
making it possible later.
- Only disk-space calculations need the actual entry contents, add
getter for dir name and device. We're passing the cache to these
getters too: its not currently unusedd but will be needed for
directory name pool id->string translation once we get there...
- Rename addFingerprints() to fpCachePopulate() and move into fprint.c.
This doesn't really belong here as it requires fprint becoming aware
of transactions and all, but at least these are all controlled API
accesses unlike where in transaction.c this was messing with somebody
elses data structures directly.
- Move the by-fingerprint creation to fpCachePopulate() so it gets
lazily done as needed and copy the original hash-size heuristics
back here.
- For now, always create the by-fingerprint hash although rpmdb usage
doesn't need it. Next steps will fix...
- Add wrapper API for retrieving the records, adjust callers
- No functional changes, at least intended ones... just first steps
towards eliminating the hash-jungle and forcing a single API
through which this stuff gets handled
- Instead of adding three more pool-aware versions of the old API's,
convert the main callers to the newew more flexible API. As a
"minor side-effect" these now use the transaction string-pool as well,
so ALL our pre-transaction dependency sets are now using the global
pool.
- The dependency sets created from installed headers during rpmtsCheck()
were using a private pool and thus ids not matching with the ones
in the global pool. Oops. Somehow none of our test-suite cases
caught this, looks like we'll need more tests... Also the safe-guard
assert()'s are in all the wrong places for catching this particular
problem. Doh :)
- There's a chicken-and-egg situation involved: in order to do this,
the global pool needs to be in unfrozen state during rpmtsCheck(),
which was not possible before switching rpmal provides (and files)
to pool ids. Now that it *is* using pool id's, move the freeze-point
to rpmtsPrepare() as the fingerprinting has similar issues with
moving strings.
- Add a pool pointer to to ts members struct and a getter function
- Grab the global pool for rpmte dependency- and file info creation,
if its NULL then the sets will use private pools of their own.
- Add the (currently) required magic voodoo rain-dance to freeze and
unfreeze the pool as necessary wrt new element additions: for
current rpmal and fingerprinting to work, the string pointers
must be immovable.
- This is infrastructure only: nothing creates the global pool yet,
so everything is still using private pools.
- Returning structs by value is a bit icky, pass in a fp pointer
for fpLookup() to fill in instead. This leaves the actual return code
free for handling errors (but ignoring that for now as we always have)
The other option would be always mallocing the return, and we dont
want to do that...
- Shouldn't change any actual functionality.
- Missed opportunity in commit 1a3a4089def9b00790eeebd6f931c99a03a3d44b:
removed packages have already gotten fingerprinted so there's no need
to redo that here.
- Speedup depends on the transaction and is by no means enormous,
but on my testcase of a largish erasure transaction this shaves
off circa four percent of the cycles spent in (re)hashing the
basenames.
- We've been calling matchNetsharedpath() for every single file in
the transaction regardless of whether %{_netsharedpath} is set or
not (and almost always it is not), meaning lots of pointless
strlen() calls and in case of erasure, rpmfi iterations performed.
Not exactly a massive speedup but a speedup nevertheless.
- Avoid repeatedly calling rpmteGetFileStates() for every file processed
- Avoid rpmfi iteration, use a good ole for-loop and index-accessors
- Dont bother looking up symlinks for skipped files
- Eliminate rpmfi from the latter loop, its not used for anything there
- Not that this is going to show on wall-clock times, the cycles saved here
are going to get very much lost in the noise of more expensive things.
- When we get rpmdb hits on files from packages that are to be removed
in the same transaction, we can use its existing file info set
to grab base- and directory names to avoid bunch of headerGet()'s
and consecutive rpmtd manipulation. In theory this should speed up
transactions where lots of packages get removed, in practise not
really - the big cost here is in loading the headers from db in the
first place, despite not being really needed.
- When we get rpmdb hits on files from packages that are to be removed
in the same transaction, we can use the exisiting file info set
to avoid constructing a new temporary one. Might be measurable
in large updates.
Panu Matilainen