Bug 1648721 - Install ordering should always prioritize scriptlet deps over runtime deps, but it seems it may not
Summary: Install ordering should always prioritize scriptlet deps over runtime deps, b...
Keywords:
Status: POST
Alias: None
Product: Fedora
Classification: Fedora
Component: rpm
Version: 31
Hardware: All
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Packaging Maintenance Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 1655995
TreeView+ depends on / blocked
 
Reported: 2018-11-11 21:39 UTC by Adam Williamson
Modified: 2020-03-02 08:38 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1647172 'unspecified' 'CLOSED' 'dbus-daemon not enabled on live images due to package install ordering problems caused by huge dependency loops' 2019-11-21 09:48:49 UTC

Internal Links: 1647172

Description Adam Williamson 2018-11-11 21:39:36 UTC
This bug is essentially a continuation of the parts of https://bugzilla.redhat.com/show_bug.cgi?id=1647172 that are about RPM package install ordering decisions.

We've solved all the current known major problems in Rawhide with a variety of hacks, but we'd still like to discuss RPM's behaviour here.

Here is a key quote from Panu in that bug:

"Right, it's the gigantic 147 member SCC (strongly connected component == loop, yes) where things blow up. Scriptlet dependencies (Requires(pre) etc) are tracked with => in the output whereas regular dependencies are tracked with ->, and yes they are considered when considering the loop break points, but loops this big just wont work, been there... The practical limit of rpm's loop breaking ability seems to be somewhere around 60-80 member SCC's, anything over 100 is almost certainly going to go wrong somewhere."

The understanding Stephen Gallagher and I had is that, when choosing the ordering for an install transaction like this, rpm should *always* treat scriptlet dependencies as the highest priority. That is, dependencies like Requires(pre): and Requires(post): . So, what we expect is that if we're installing 500 packages and there are all kinds of 'loops' in the regular, non-scriptlet dependencies, but there is a clearly correct, non-ambiguous solution to the *scriptlet* dependencies, then *that solution should always be respected in the order RPM decides on* - however it decides to 'break the loops' in the regular dependencies, it should *NOT* break the correct ordering for the scriptlet dependencies.

However, what Panu describes seems to be a different behaviour, where RPM kinda figures out the whole dependency situation, then tries to use the scriptlet deps as "hints" for how to resolve it, but if the overall situation is sufficiently complex, it might not ultimately decide on an ordering which reflects the scriptlet deps. And our recent experience seems to confirm this: if you run a Rawhide Fedora Server install with fedora-release-30-0.12, where fedora-release-server provides 'system-release' and has normal (non-scriptlet) requires for systemd, cockpit, and openssh-server, the install fails on a scriptlet error in crypto-policies scriptlets. If you run the same install with fedora-release-30-0.13, with those non-scriptlet requires dropped, the install succeeds...

We have concerns about that design. We think it would be better if the scriptlet deps were more strongly prioritized and, so long as they themselves have a clearly 'correct' solution within the transaction, that should *always* be respected in the final ordering.

Is our understanding of the current situation correct? And if so, would the RPM devs consider changing the implementation to respect scriptlet deps more strongly?

Comment 1 Stephen Gallagher 2018-11-11 22:57:31 UTC
As an addendum, I’d also like to ask if there is a mechanism to have a Requirement with *no* ordering. Phrased differently, I’d like to be able to require that a package is installed as part of the same transaction, but put no constraint on *when* in the transaction it should occur. The real-world example is that I want Fedora-release-Server to pull in Cockpit, but I only care that it’s there when the transaction concludes.

Comment 2 satellitgo 2018-11-11 23:19:43 UTC
Fedora-KDE-Live-x86_64-Rawhide-20181110.n.0
 "you need to load the kernel"

Fedora-Workstation-Live-x86_64-Rawhide-20181110.n.0.iso
 installs correctly

Comment 3 Adam Williamson 2018-11-12 00:35:36 UTC
satellit: please file that somewhere else, this is about the general issue in RPM, not about specific bugs. Also are you sure you tested right? The KDE live passed openQA testing.

Comment 4 Adam Williamson 2018-11-12 17:34:28 UTC
From the other bug, Panu says:

"There's a major misunderstanding here. Dependencies need to be always installed first, otherwise just about nothing would work. Requires(pre/post) etc only differ from "plain" Requires when there are dependency loops, in which case they're used as hints for loop breaking at the least harmful spots."

That's...interesting. I think Stephen makes an important point above, which is that we really don't know this, about any particular dependency. "Foo requires bar" is just not really sufficient information: that *might* be an install-time dependency, it might be a run-time dependency, or it might be something like the fedora-release-server case where there's not exactly any kind of actual code execution "dependency" in play *at all*, it's effectively just a way to state that "by policy, in Fedora, if this package is installed this other package must also be installed".

Of course in an ideal world it'd be nice to have syntax to indicate the nature of the dependency more precisely, but even if that existed and we could ignore all the 'but does it work in RHEL?' fun that comes along with new RPM features, who's going to go and retrofit the new syntax to however many tens of thousands of dependencies we already have in the distro? (Still, if we're going to think about this, perhaps take a hint from the way systemd does it: dependencies and ordering are separate, in systemd, you have things like 'Requires:' to indicate dependencies, and 'After:' and 'Before:' to handle ordering...perhaps for RPM, 'After:' and 'Before:' tags could be added, and given higher priority when they're present, or whatever).

Given that, I think it's still reasonable to argue that things should work as I suggested above - because a Requires(pre) or Requires(post) really *is* a very concrete statement. It means exactly what it says on the tin: for this package's %pre or %post to work, this other package MUST be installed first. There's no ambiguity about it, in the way there's an ambiguity about what a plain 'Requires' means.

Since we are armed with that knowledge, it seems like RPM should *first* ensure all those requirements are met (assuming none of them are contradictory). *Then* it should consider Requires: as ordering hints to further refine the ordering, without breaking the (pre) and (post) ordering. Because we know *for sure* that if the (pre) and (post) ordering get broken, we will have problems. We don't know *for sure* whether any given plain 'Requires' dependency really means anything for install ordering.

Comment 5 Panu Matilainen 2018-11-13 11:48:04 UTC
> I think Stephen makes an important point above, which is that we really don't 
> know this, about any particular dependency. "Foo requires bar" is just not 
> really sufficient information: that *might* be an install-time dependency, it 
> might be a run-time dependency, or it might be something like the 
> fedora-release-server case where there's not exactly any kind of actual code
> execution "dependency" in play *at all*, it's effectively just a way to state
> that "by policy, in Fedora, if this package is installed this other package
> must also be installed".

Sure we know: Requires(pre/post/etc) are install-time dependencies, Requires are run-time dependencies. And they must be installed before said package can be considered functional, ie usable by other packages Requires(pre) etc. As far as rpm is concerned, "by policy" dependencies don't exist at all, and of course that's exactly why they are problematic when expressed as either of the other types. Maybe there should be, I'm not denying such a case existing. 

Come to think of it, you can sort of achieve a "policy" dependency with by existing means (2. will work in any old rpm, 1. requires a relatively modern rpm (wont work in rhel-6 but rhel-7 is ok):

1) Requires(posttrans) dependencies are ignored during ordering because by definition, posttrans scripts occur after the entire transaction has completed and thus ordering is no more relevant. The potential downside (depending on the case) is that since it's technically an install-time dependency, it's possible to remove such a dependency after the installation

2) Requires(preun/postun) will affect *erase* ordering, but not that of installation, and unlike 1) such dependencies cannot be removed before the package itself has been removed.

Back to the actual issue, perhaps the biggest problem with rpm's ordering is that it doesn't detect and error out in impossible situations, namely those conflicting install-time dependencies. That makes spotting the actual loops / errors much more difficult than it should be.

Another factor is that since recently, rpm also takes weak dependencies into account when ordering but doesn't priorize them differently from hard dependencies, which wont help.

@ffesti, thoughts?

Comment 6 Adam Williamson 2018-11-13 23:50:23 UTC
True, yeah. I did realize while writing that you have problems when, e.g., 'foo' Requires(post) 'bar' and bar then requires a bunch of other stuff (including, in the worst case, foo...), because you have to assume that for 'bar' to do its job in 'foo''s %post, all its deps have to be installed first. Maybe that's not actually true for all of them, but it's definitely a sensible thing to believe...

There are problems aside from the 'policy' case, of course. For instance, the dbus/systemd complex is a good one. systemd requires dbus for *normal* operation. However, during dbus-daemon %post we need to run 'systemctl preset', which works *without* dbus installed...and is part of systemd. So we wind up with systemd "Requires" dbus, dbus "Requires" dbus-daemon (this part is just an implementation detail, there could just as well only be 'systemd' and 'dbus' packages), and dbus-daemon "Requires(post)" systemd...and none of those dependencies is actually wrong. I don't really see a good way to 'fix' that...do you?

Using Requires(postun) as a hack to achieve a 'policy' dependency sounds pretty ugly, but it may not be any more ugly than any other choice...:/

Comment 7 Adam Williamson 2018-11-13 23:51:44 UTC
btw, one thing we could do is patch all the compose tools to run with the deploop debug flag, so we get the dependency loops captured in the output of those tools...that way we'd at least be able to look up what loops we have there, any time we wanted to...

Comment 8 Panu Matilainen 2018-11-14 09:32:54 UTC
The sort of solution that can be expressed to package management is that systemd upstream and packaging needs to provide the preset-functionality without dragging in all of systemd and its dependencies. Installing systemd into an empty chroot pulls in 157 packages (including weak deps and their deps, 100 without). As a pre-requisite for installing *any* package with services, that is just insane. In contrast, in rhel-6 just 17 packages were required for roughly the same task (chkconfig).

Also makes me wonder if that functionality couldn't be moved to some post-transaction file trigger to reduce the number of scriptlets and thus scriptlet pre-dependencies.

As for composes with deploop flag, absolutely. Having them always available makes tracing down problems much much easier.

Comment 9 Panu Matilainen 2018-11-14 13:04:46 UTC
FWIW, technically implementing a new dependency qualifier that behaves in the "policy" sense as discussed here would be truly trivial. The question is what to call it: "noorder" would be technically correct but clunky-ugly and unlike the other qualifiers, I'd like to find something that somehow conveys the semantics but doesn't involve "order" at all.

"policy" seems too vague, "trans" would be sort of in line with the pre/post etc qualifiers, but that'd seem to indicate it only affects the transaction (so could be removed afterwards) which you can already achieve with Requires(posttrans). Ideas welcome, native English speakers will have a vocabulary advantage here :)

Comment 10 Stephen Gallagher 2018-11-14 13:19:42 UTC
(In reply to Panu Matilainen from comment #8)
> Also makes me wonder if that functionality couldn't be moved to some
> post-transaction file trigger to reduce the number of scriptlets and thus
> scriptlet pre-dependencies.

This is something that I'm actively looking into. I'll get back to you on that.


(In reply to Panu Matilainen from comment #9)
> FWIW, technically implementing a new dependency qualifier that behaves in
> the "policy" sense as discussed here would be truly trivial. The question is
> what to call it: "noorder" would be technically correct but clunky-ugly and
> unlike the other qualifiers, I'd like to find something that somehow conveys
> the semantics but doesn't involve "order" at all.
> 
> "policy" seems too vague, "trans" would be sort of in line with the pre/post
> etc qualifiers, but that'd seem to indicate it only affects the transaction
> (so could be removed afterwards) which you can already achieve with
> Requires(posttrans). Ideas welcome, native English speakers will have a
> vocabulary advantage here :)

Couldn't we just add a new directive called RequiresUnordered? Existing "Requires" would remain asserting that they need to be installed ahead of time, RequiresUnordered would assert that they just need to exist once the transaction is complete.

Question: how does Recommends work in this situation? Would we need to add RecommendsUnordered and SupplementsUnordered as well?

Comment 11 Panu Matilainen 2018-11-14 13:37:38 UTC
That question is exactly the reason I think it needs to be a qualifier instead of a new tag, so its applicable to all the dependency tags. We certainly don't want five new tags just for this.

Comment 12 Stephen Gallagher 2018-11-14 13:52:23 UTC
(In reply to Panu Matilainen from comment #11)
> That question is exactly the reason I think it needs to be a qualifier
> instead of a new tag, so its applicable to all the dependency tags. We
> certainly don't want five new tags just for this.

Can you explain what a "qualifier" is?

Is it just a parenthesized suffix to the tag? e.g. is "Requires(post)" a tag with a qualifier or another tag?

Requires(unordered): would be fine, if that's what you're getting at.

Comment 13 Panu Matilainen 2018-11-14 14:18:30 UTC
Yeah, the part in parenthesis is what I call qualifiers, in practise they end up in <dependency>FLAGS tag bitfield.

"unordered" is certainly less ugly than "noorder" but like said in comment #9, I'd prefer something that doesn't involve the word "order" at all.

Comment 14 Stephen Gallagher 2018-11-14 14:21:01 UTC
(In reply to Panu Matilainen from comment #13)
> Yeah, the part in parenthesis is what I call qualifiers, in practise they
> end up in <dependency>FLAGS tag bitfield.
> 
> "unordered" is certainly less ugly than "noorder" but like said in comment
> #9, I'd prefer something that doesn't involve the word "order" at all.

Why avoid the word order? It seems like the most exact word for this situation. Maybe you can explain what semantics you mean to convey, if not the transaction ordering? I might be missing something obvious.

Comment 15 Panu Matilainen 2018-11-14 14:44:44 UTC
For one, it's not the only dependency type that doesn't affect ordering, and it doesn't really fit with the other qualifiers, which describe the context of that dependency more than how it behaves. So from the context point of view, something in lines of Requires(installed) might be closer: it indicates that such a dependency just needs to be installed but no other promises need to be made.

Comment 16 Adam Williamson 2018-11-14 16:44:27 UTC
"The sort of solution that can be expressed to package management is that systemd upstream and packaging needs to provide the preset-functionality without dragging in all of systemd and its dependencies."

That's tricky, because systemctl does lots of things, not just presets, and most of them probably *do* require all the things systemd normally requires. Like a message bus...

In fact there's another possible RPM feature there, something you could call 'capabilities': the package could provide different things *with different dependencies*. So the systemd package could provide 'systemd' with normal dependencies (so anything that required systemd would require everything else systemd usually requires), and 'systemctl-preset' (or whatever) with *different* dependencies (so anything that required specifically 'systemctl-preset' would not pull in the rest of systemd's deps to that chain). This is a pretty...uh...'targeted' feature, though :)

"Installing systemd into an empty chroot pulls in 157 packages (including weak deps and their deps, 100 without)."

Oh god shhhhhhhhhhh, you know the anti-systemd trolls can *smell* this kind of post, right? :P

Whether or not we could reasonably trim systemd's deps, I don't think you're ever going to be able to trim dbus out of them, so the loop I showed would remain.

Comment 17 Adam Williamson 2018-11-14 16:47:59 UTC
"Also makes me wonder if that functionality couldn't be moved to some post-transaction file trigger to reduce the number of scriptlets and thus scriptlet pre-dependencies."

This is *kinda* an option, I think - but I think the 'standard' deps for anything with a service include 'Requires: systemd' anyway, so we'd still have a 'normal' ordered dependency there, if not a 'Requires(post)'...

Comment 18 Stephen Gallagher 2018-11-14 18:16:07 UTC
Panu: What about `Requires(metapackage):`?

The most common use for this is going to be a metapackage who exists solely to make sure certain other packages are installed, after all...

Comment 19 Panu Matilainen 2018-11-15 11:54:41 UTC
Hmm, meta(package) seems quite fitting, certainly best of the suggestions so far.

Comment 20 Stephen Gallagher 2019-01-02 16:31:33 UTC
It's been a month-and-a-half: if there are no better names than "Requires(metapackage):", can we run with that?

Comment 21 Panu Matilainen 2019-01-03 10:42:54 UTC
Yeah I've long since more or less settled on "meta" (without package) as the name, but this isn't something that would be immediately backported so there's no rush and I prefer thinking more than twice for something that uses up a precious bit in a rather full 32bit field for doing almost nothing.

Comment 22 Florian Festi 2019-01-27 10:23:40 UTC
What about Requires(posttrans)?

That should already exist and not be used for ordering. Have not tested it, though. But that's what it looks like in the code.

Comment 23 Panu Matilainen 2019-02-01 08:42:03 UTC
See comments #5-#9 - Requires(posttrans) dependencies can be removed after the installation has taken place, which is not desired here.

Comment 24 Ben Cotton 2019-08-13 16:58:58 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to '31'.

Comment 25 Ben Cotton 2019-08-13 19:35:56 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to 31.

Comment 26 Stephen Gallagher 2019-11-13 01:16:48 UTC
We just passed the anniversary of this BZ. Could we get a response on whether RPM will be adding this feature at any point?

Comment 27 Panu Matilainen 2020-01-30 15:11:53 UTC
Just submitted this upstream: https://github.com/rpm-software-management/rpm/pull/1028, lets see what happens.

Comment 28 Florian Festi 2020-03-02 08:38:34 UTC
This feature just made it into the upstream repository and is expected to make it into the next major RPM release:  https://github.com/rpm-software-management/rpm/commit/b4a3f6c624edb0175855a925f3e0c38e907cb8a9


Note You need to log in before you can comment on or make changes to this bug.