Bug 1648721

As an addendum, I’d also like to ask if there is a mechanism to have a Requirement with *no* ordering. Phrased differently, I’d like to be able to require that a package is installed as part of the same transaction, but put no constraint on *when* in the transaction it should occur. The real-world example is that I want Fedora-release-Server to pull in Cockpit, but I only care that it’s there when the transaction concludes.

Fedora-KDE-Live-x86_64-Rawhide-20181110.n.0
 "you need to load the kernel"

Fedora-Workstation-Live-x86_64-Rawhide-20181110.n.0.iso
 installs correctly

satellit: please file that somewhere else, this is about the general issue in RPM, not about specific bugs. Also are you sure you tested right? The KDE live passed openQA testing.

From the other bug, Panu says:

"There's a major misunderstanding here. Dependencies need to be always installed first, otherwise just about nothing would work. Requires(pre/post) etc only differ from "plain" Requires when there are dependency loops, in which case they're used as hints for loop breaking at the least harmful spots."

That's...interesting. I think Stephen makes an important point above, which is that we really don't know this, about any particular dependency. "Foo requires bar" is just not really sufficient information: that *might* be an install-time dependency, it might be a run-time dependency, or it might be something like the fedora-release-server case where there's not exactly any kind of actual code execution "dependency" in play *at all*, it's effectively just a way to state that "by policy, in Fedora, if this package is installed this other package must also be installed".

Of course in an ideal world it'd be nice to have syntax to indicate the nature of the dependency more precisely, but even if that existed and we could ignore all the 'but does it work in RHEL?' fun that comes along with new RPM features, who's going to go and retrofit the new syntax to however many tens of thousands of dependencies we already have in the distro? (Still, if we're going to think about this, perhaps take a hint from the way systemd does it: dependencies and ordering are separate, in systemd, you have things like 'Requires:' to indicate dependencies, and 'After:' and 'Before:' to handle ordering...perhaps for RPM, 'After:' and 'Before:' tags could be added, and given higher priority when they're present, or whatever).

Given that, I think it's still reasonable to argue that things should work as I suggested above - because a Requires(pre) or Requires(post) really *is* a very concrete statement. It means exactly what it says on the tin: for this package's %pre or %post to work, this other package MUST be installed first. There's no ambiguity about it, in the way there's an ambiguity about what a plain 'Requires' means.

Since we are armed with that knowledge, it seems like RPM should *first* ensure all those requirements are met (assuming none of them are contradictory). *Then* it should consider Requires: as ordering hints to further refine the ordering, without breaking the (pre) and (post) ordering. Because we know *for sure* that if the (pre) and (post) ordering get broken, we will have problems. We don't know *for sure* whether any given plain 'Requires' dependency really means anything for install ordering.

> I think Stephen makes an important point above, which is that we really don't 
> know this, about any particular dependency. "Foo requires bar" is just not 
> really sufficient information: that *might* be an install-time dependency, it 
> might be a run-time dependency, or it might be something like the 
> fedora-release-server case where there's not exactly any kind of actual code
> execution "dependency" in play *at all*, it's effectively just a way to state
> that "by policy, in Fedora, if this package is installed this other package
> must also be installed".

Sure we know: Requires(pre/post/etc) are install-time dependencies, Requires are run-time dependencies. And they must be installed before said package can be considered functional, ie usable by other packages Requires(pre) etc. As far as rpm is concerned, "by policy" dependencies don't exist at all, and of course that's exactly why they are problematic when expressed as either of the other types. Maybe there should be, I'm not denying such a case existing. 

Come to think of it, you can sort of achieve a "policy" dependency with by existing means (2. will work in any old rpm, 1. requires a relatively modern rpm (wont work in rhel-6 but rhel-7 is ok):

1) Requires(posttrans) dependencies are ignored during ordering because by definition, posttrans scripts occur after the entire transaction has completed and thus ordering is no more relevant. The potential downside (depending on the case) is that since it's technically an install-time dependency, it's possible to remove such a dependency after the installation

2) Requires(preun/postun) will affect *erase* ordering, but not that of installation, and unlike 1) such dependencies cannot be removed before the package itself has been removed.

Back to the actual issue, perhaps the biggest problem with rpm's ordering is that it doesn't detect and error out in impossible situations, namely those conflicting install-time dependencies. That makes spotting the actual loops / errors much more difficult than it should be.

Another factor is that since recently, rpm also takes weak dependencies into account when ordering but doesn't priorize them differently from hard dependencies, which wont help.

@ffesti, thoughts?

True, yeah. I did realize while writing that you have problems when, e.g., 'foo' Requires(post) 'bar' and bar then requires a bunch of other stuff (including, in the worst case, foo...), because you have to assume that for 'bar' to do its job in 'foo''s %post, all its deps have to be installed first. Maybe that's not actually true for all of them, but it's definitely a sensible thing to believe...

There are problems aside from the 'policy' case, of course. For instance, the dbus/systemd complex is a good one. systemd requires dbus for *normal* operation. However, during dbus-daemon %post we need to run 'systemctl preset', which works *without* dbus installed...and is part of systemd. So we wind up with systemd "Requires" dbus, dbus "Requires" dbus-daemon (this part is just an implementation detail, there could just as well only be 'systemd' and 'dbus' packages), and dbus-daemon "Requires(post)" systemd...and none of those dependencies is actually wrong. I don't really see a good way to 'fix' that...do you?

Using Requires(postun) as a hack to achieve a 'policy' dependency sounds pretty ugly, but it may not be any more ugly than any other choice...:/

btw, one thing we could do is patch all the compose tools to run with the deploop debug flag, so we get the dependency loops captured in the output of those tools...that way we'd at least be able to look up what loops we have there, any time we wanted to...

The sort of solution that can be expressed to package management is that systemd upstream and packaging needs to provide the preset-functionality without dragging in all of systemd and its dependencies. Installing systemd into an empty chroot pulls in 157 packages (including weak deps and their deps, 100 without). As a pre-requisite for installing *any* package with services, that is just insane. In contrast, in rhel-6 just 17 packages were required for roughly the same task (chkconfig).

Also makes me wonder if that functionality couldn't be moved to some post-transaction file trigger to reduce the number of scriptlets and thus scriptlet pre-dependencies.

As for composes with deploop flag, absolutely. Having them always available makes tracing down problems much much easier.

FWIW, technically implementing a new dependency qualifier that behaves in the "policy" sense as discussed here would be truly trivial. The question is what to call it: "noorder" would be technically correct but clunky-ugly and unlike the other qualifiers, I'd like to find something that somehow conveys the semantics but doesn't involve "order" at all.

"policy" seems too vague, "trans" would be sort of in line with the pre/post etc qualifiers, but that'd seem to indicate it only affects the transaction (so could be removed afterwards) which you can already achieve with Requires(posttrans). Ideas welcome, native English speakers will have a vocabulary advantage here :)

(In reply to Panu Matilainen from comment #8)
> Also makes me wonder if that functionality couldn't be moved to some
> post-transaction file trigger to reduce the number of scriptlets and thus
> scriptlet pre-dependencies.

This is something that I'm actively looking into. I'll get back to you on that.


(In reply to Panu Matilainen from comment #9)
> FWIW, technically implementing a new dependency qualifier that behaves in
> the "policy" sense as discussed here would be truly trivial. The question is
> what to call it: "noorder" would be technically correct but clunky-ugly and
> unlike the other qualifiers, I'd like to find something that somehow conveys
> the semantics but doesn't involve "order" at all.
> 
> "policy" seems too vague, "trans" would be sort of in line with the pre/post
> etc qualifiers, but that'd seem to indicate it only affects the transaction
> (so could be removed afterwards) which you can already achieve with
> Requires(posttrans). Ideas welcome, native English speakers will have a
> vocabulary advantage here :)

Couldn't we just add a new directive called RequiresUnordered? Existing "Requires" would remain asserting that they need to be installed ahead of time, RequiresUnordered would assert that they just need to exist once the transaction is complete.

Question: how does Recommends work in this situation? Would we need to add RecommendsUnordered and SupplementsUnordered as well?

That question is exactly the reason I think it needs to be a qualifier instead of a new tag, so its applicable to all the dependency tags. We certainly don't want five new tags just for this.

(In reply to Panu Matilainen from comment #11)
> That question is exactly the reason I think it needs to be a qualifier
> instead of a new tag, so its applicable to all the dependency tags. We
> certainly don't want five new tags just for this.

Can you explain what a "qualifier" is?

Is it just a parenthesized suffix to the tag? e.g. is "Requires(post)" a tag with a qualifier or another tag?

Requires(unordered): would be fine, if that's what you're getting at.

Yeah, the part in parenthesis is what I call qualifiers, in practise they end up in <dependency>FLAGS tag bitfield.

"unordered" is certainly less ugly than "noorder" but like said in comment #9, I'd prefer something that doesn't involve the word "order" at all.

(In reply to Panu Matilainen from comment #13)
> Yeah, the part in parenthesis is what I call qualifiers, in practise they
> end up in <dependency>FLAGS tag bitfield.
> 
> "unordered" is certainly less ugly than "noorder" but like said in comment
> #9, I'd prefer something that doesn't involve the word "order" at all.

Why avoid the word order? It seems like the most exact word for this situation. Maybe you can explain what semantics you mean to convey, if not the transaction ordering? I might be missing something obvious.

For one, it's not the only dependency type that doesn't affect ordering, and it doesn't really fit with the other qualifiers, which describe the context of that dependency more than how it behaves. So from the context point of view, something in lines of Requires(installed) might be closer: it indicates that such a dependency just needs to be installed but no other promises need to be made.

"The sort of solution that can be expressed to package management is that systemd upstream and packaging needs to provide the preset-functionality without dragging in all of systemd and its dependencies."

That's tricky, because systemctl does lots of things, not just presets, and most of them probably *do* require all the things systemd normally requires. Like a message bus...

In fact there's another possible RPM feature there, something you could call 'capabilities': the package could provide different things *with different dependencies*. So the systemd package could provide 'systemd' with normal dependencies (so anything that required systemd would require everything else systemd usually requires), and 'systemctl-preset' (or whatever) with *different* dependencies (so anything that required specifically 'systemctl-preset' would not pull in the rest of systemd's deps to that chain). This is a pretty...uh...'targeted' feature, though :)

"Installing systemd into an empty chroot pulls in 157 packages (including weak deps and their deps, 100 without)."

Oh god shhhhhhhhhhh, you know the anti-systemd trolls can *smell* this kind of post, right? :P

Whether or not we could reasonably trim systemd's deps, I don't think you're ever going to be able to trim dbus out of them, so the loop I showed would remain.

"Also makes me wonder if that functionality couldn't be moved to some post-transaction file trigger to reduce the number of scriptlets and thus scriptlet pre-dependencies."

This is *kinda* an option, I think - but I think the 'standard' deps for anything with a service include 'Requires: systemd' anyway, so we'd still have a 'normal' ordered dependency there, if not a 'Requires(post)'...

Panu: What about `Requires(metapackage):`?

The most common use for this is going to be a metapackage who exists solely to make sure certain other packages are installed, after all...

Hmm, meta(package) seems quite fitting, certainly best of the suggestions so far.

It's been a month-and-a-half: if there are no better names than "Requires(metapackage):", can we run with that?

Yeah I've long since more or less settled on "meta" (without package) as the name, but this isn't something that would be immediately backported so there's no rush and I prefer thinking more than twice for something that uses up a precious bit in a rather full 32bit field for doing almost nothing.

What about Requires(posttrans)?

That should already exist and not be used for ordering. Have not tested it, though. But that's what it looks like in the code.

See comments #5-#9 - Requires(posttrans) dependencies can be removed after the installation has taken place, which is not desired here.

This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to '31'.

This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to 31.

We just passed the anniversary of this BZ. Could we get a response on whether RPM will be adding this feature at any point?

Just submitted this upstream: https://github.com/rpm-software-management/rpm/pull/1028, lets see what happens.

This feature just made it into the upstream repository and is expected to make it into the next major RPM release:  https://github.com/rpm-software-management/rpm/commit/b4a3f6c624edb0175855a925f3e0c38e907cb8a9

This message is a reminder that Fedora 31 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 31 on 2020-11-24.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '31'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 31 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 31 changed to end-of-life (EOL) status on 2020-11-24. Fedora 31 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.