I'm not sure how exactly the rawhide image is put together, i.e. what packages are
*supposed* to be installed, but in general this not an error.

Apparently, systemd-standalone-sysusers is installed in the image.
To replace that with normal systemd, add --allowerasing.

If it is expect that the base image contains systemd, or that systemd can
be installed without --allowerasing, then I think you should report this against
the image definition. Please put me in CC if you do. I'm interesting in
understanding if there is a problem. But we don't track image definitions
in bugzilla.

JFTR: we encounter this (or similar) issue both working with F42 and Rawhide containers:
```
    Problem: problem with installed package
      - installed package systemd-standalone-sysusers-257.2-17.fc42.x86_64 conflicts with systemd provided by systemd-257.2-17.fc42.i686 from fedora
      - package systemd-257.2-17.fc42.i686 from fedora conflicts with systemd-standalone-sysusers provided by systemd-standalone-sysusers-257.2-17.fc42.x86_64 from fedora
      - package openssh-server-9.9p1-7.fc42.x86_64 from fedora requires systemd, but none of the providers can be installed
      - installed package systemd-standalone-sysusers-257.2-17.fc42.x86_64 conflicts with systemd provided by systemd-257.2-17.fc42.x86_64 from fedora
      - package systemd-257.2-17.fc42.x86_64 from fedora conflicts with systemd-standalone-sysusers provided by systemd-standalone-sysusers-257.2-17.fc42.x86_64 from fedora
```

This is probably somehow related to the sysusers merge. Rpm gained a dependency
on /usr/bin/systemd-sysusers, which is provided by either systemd and systemd-standalone-sysusers,
and some image definitions were changed to prefer systemd-standalone-sysusers.
But the packaging itself is correct. The question of whether some image is suitable
for what you want to use it for cannot be answered here.

I fail to see why systemd / systemd-standalone-sysusers are packaged in such a way that upgrading from systemd-standalone-sysusers to full systemd is not supported without the nuclear option of --allowerasing.

IIRC, product Fedora Container Images and component base (https://bugzilla.redhat.com/buglist.cgi?classification=Fedora&component=base&product=Fedora%20Container%20Images) is where Fedora container images are tracked.

(In reply to Jan Pazdziora from comment #4)
> I fail to see why systemd / systemd-standalone-sysusers are packaged in such
> a way that upgrading from systemd-standalone-sysusers to full systemd is not
> supported without the nuclear option of --allowerasing.

The standalone implementations provide an alternative that has the same file names.
The fact that --allowerasing is requires is a general property of how
dnf works… Those subpackages have used this form since they were introduced
in 2020.

> IIRC, product Fedora Container Images and component base
> (https://bugzilla.redhat.com/buglist.
> cgi?classification=Fedora&component=base&product=Fedora%20Container%20Images)
> is where Fedora container images are tracked.

Ah, OK.

(In reply to Zbigniew Jędrzejewski-Szmek from comment #5)
> 
> The standalone implementations provide an alternative that has the same file
> names.
> The fact that --allowerasing is requires is a general property of how
> dnf works…

To me it is a symptom of some Provides not being in place.

> Those subpackages have used this form since they were introduced
> in 2020.

Had I known about that, I would have considered it unfortunate already in 2020.

Let me reopen this bugzilla to specifically track finding a better way of packaging this so that upgrades are less painful, now that the issue keeps hitting people who build Fedora-based container images. The --allowerasing can hide other issues that people would like to know about, especially in their CI setups.

This problem has broken libvirt's upstream CI too, and once F42 is released it'll break QEMU too. Both try to install device-mapper-devel and that depends on device-mapper which depends on systemd which now crashes & burns because systemd-standalone-sysusers is pre-installed and conflicts with systemd.

IMHO expecting users to add --allowerasing in this scenario is unreasonable, as that effectively has become "--make-it-work" which is what users expect to just happen by default.

Is it possible to spin off the real /usr/bin/systemd-sysusers binary into a separate package for cases where a slimmer install footprint is needed, rather than having 2 parallel impls of systemd-sysusers which must conflict ?

> Attempt to install systemd in registry.fedoraproject.org/fedora:rawhide container fails.
> 
> podman run --rm registry.fedoraproject.org/fedora:rawhide dnf install -y systemd

Note while this the root cause, the actuall impact is far wider than this. It is relatively atypical for users to want to directly install 'systemd' in a container, but plenty of things they want will pull in 'systemd' indirectly. Perhaps the poster child for containerization, apache httpd, cannot be installed as it depends on systemd:

$ podman run -it fedora:42 dnf -y install httpd
Updating and loading repositories:
 Fedora 42 openh264 (From Cisco) - x86_64                                                                                                         100% |   3.8 KiB/s |   4.8 KiB |  00m01s
 Fedora 42 - x86_64                                                                                                                               100% |  19.3 MiB/s |  35.2 MiB |  00m02s
 Fedora 42 - x86_64 - Test Updates                                                                                                                100% |  35.4 KiB/s |  21.9 KiB |  00m01s
 Fedora 42 - x86_64 - Updates                                                                                                                     100% |  26.3 KiB/s |  25.6 KiB |  00m01s
Repositories loaded.
Failed to resolve the transaction:
Problem: systemd-257.2-17.fc42.i686 from fedora has inferior architecture
  - package httpd-2.4.62-6.fc42.x86_64 from fedora requires systemd, but none of the providers can be installed
  - problem with installed package
  - installed package systemd-standalone-sysusers-257.2-17.fc42.x86_64 conflicts with systemd provided by systemd-257.2-17.fc42.x86_64 from fedora
  - package systemd-257.2-17.fc42.x86_64 from fedora conflicts with systemd-standalone-sysusers provided by systemd-standalone-sysusers-257.2-17.fc42.x86_64 from fedora
  - conflicting requests
You can try to add to command line:
  --allowerasing to allow removing of installed packages to resolve problems
  --skip-broken to skip uninstallable packages

1st level direct deps come to about 700 packages, second level indirect deps expands this to almost 2500 that can no longer be easily installed.


> Is it possible to spin off the real /usr/bin/systemd-sysusers binary into a separate package for cases where a slimmer install footprint is needed, rather than having 2 parallel impls of systemd-sysusers which must conflict ?

The other classical solution is to use update-alternatives, to allow both packages to co-exist.

Proposed as a Blocker for 42-beta by Fedora user berrange using the blocker tracking app because:

 Many 1000's of packages are uninstallable in the Fedora 42 container images, without passing '--allowerasing' to dnf, due to existence of a conflict between systemd and systemd-standalone-sysusers which has been pulled in by RPM as a new dep.

I'm suggesting this violates the basic release criteria https://fedoraproject.org/wiki/Basic_Release_Criteria#Installing,_removing_and_updating_software

"The installed system must be able appropriately to install, remove, and update software with the default console tool for the relevant software type (e.g. default console package manager). This includes downloading of packages to be installed/updated. "

as expecting widespread changes in user behaviour/Dockerfiles to add '--allowerasing' to container usage is not "appropriate" IMHO

rpm-libs has Requires:/usr/bin/systemd-sysusers.

systemd-sysusers-standalone is 0.28 MB and has few dependencies.

We could split out /usr/bin/systemd-sysusers (the normal version) to a subpackage.
It links to /usr/lib64/systemd/libsystemd-shared-257.2-14.fc42.so, which
would need to be split out of the main systemd package too.
systemd-sysusers+libsystemd-shared.so is 4.8MB, but libsystemd-shared.so
also pulls in a bunch of libraries. But some of those would be pulled in by
dnf + rpm anyway, so I'm not sure what the actual overhead would be.

Is this worth pursuing?

Also, where are the contents of the fedora:rawhide image defined?

Container images are built in Kiwi, so their contents are defined in https://pagure.io/fedora-kiwi-descriptions . Specifically, in https://pagure.io/fedora-kiwi-descriptions/blob/rawhide/f/teams/cloud/container.xml .

Thanks! (Aside: trying to find this without knowing exactly where to look seems impossible.
There is a whole graveyard of dead projects, dead hosting sites, outdated documentation,
defunct SIGs, etc., all advertising themselves as _the_ place for Fedora container definitions.
There is even a bunch of tickets from various years asking for things to be marked as
deprecated, but nobody ever seems to do that ;( )

I was looking for this:
https://pagure.io/fedora-kiwi-descriptions/c/1f35331b84bd7e5d3d71e98479a735bb325a2979
This would need to be changed…

> Is this worth pursuing?

Please give me at least a provisional ack before start work on implementing this.

It makes sense to do this, but we need it for all the -standalone binaries, because this problem will crop up again with tmpfiles and other things too.

> Aside: trying to find this without knowing exactly where to look seems impossible. There is a whole graveyard of dead projects, dead hosting sites, outdated documentation, defunct SIGs, etc., all advertising themselves as _the_ place for Fedora container definitions.

What projects, sites, documentation and SIGs do you mean specifically? I can fix up any I have commit rights on.

https://github.com/container-images
https://pagure.io/ContainerSIG/container-sig/
https://pagure.io/ContainerSIG/container-sig/issue/13
https://docs.fedoraproject.org/en-US/containers/
https://lists.fedoraproject.org/archives/list/container-sig@lists.fedorahosted.org/
https://src.fedoraproject.org/projects/container/*

(In reply to Zbigniew Jędrzejewski-Szmek from comment #12)
> (Aside: trying to find this without knowing exactly where to look
> seems impossible.

I can reopen bug 2151849 if folks hit similar issue like I do quite often, having a Fedora image with some behaviour but not having clear pointers to where it came from. We should be able to have links to exact sources and to the job that produced a specific file in the image labels.

I think that'd make a lot of sense.
I'd like to for two extensions to #2151849:
- a link to the repo with the definition, not just the build pipeline,
- it would be great if this info could also be visible from inside when running the image too.

$ skopeo inspect docker://registry.fedoraproject.org/fedora:rawhide | jq .RepoTags -c
["34-aarch64","34-ppc64le","34-x86_64","34","34-s390x","34-armhfp","33-armhfp","32-armhfp","35-aarch64","35-ppc64le","35-s390x","35-x86_64","35","35-armhfp","36-aarch64","36-armhfp","36-ppc64le","36-s390x","36-x86_64","30-aarch64","30-ppc64le","30-s390x","30-x86_64","30","latest","rawhide","30-armhfp","36","31-aarch64","31-x86_64","31","31-armhfp","31-s390x","31-ppc64le","32-aarch64","32-ppc64le","32-s390x","32-x86_64","32","33-aarch64","33-ppc64le","33-s390x","33-x86_64","33","37-aarch64","37-ppc64le","37-s390x","37-x86_64","37","38-aarch64","38-ppc64le","38-s390x","38-x86_64","38","39-aarch64","39-s390x","39-x86_64","39","39-ppc64le","40-aarch64","40-ppc64le","40-s390x","40-x86_64","40","41-aarch64","41-ppc64le","41-s390x","41-x86_64","41","42-aarch64","42-ppc64le","42-s390x","42-x86_64","42","43-aarch64","43-ppc64le","43-s390x","43-x86_64","43"]

That looks suspicious :(

How do you mean, "looks suspicious"?

AFAIK, what that output means is just "these are all the tags that *at least one* image in this repository has.

Are you worried that older ones haven't been garbage collected?

I'd think that this would be tags attached to 'registry.fedoraproject.org/fedora:rawhide'.
But if you think it's good, then I'm fine with that.

Maybe I'm just confused, sorry - but are you saying you think "a link to the repo with the definition, not just the build pipeline" should be in the *tags*? I don't think that'd be usual - the tags aren't meant for that purpose. But as Jan says, we could potentially put that information in the *labels*:

[adamw@toolbx os-autoinst-distri-fedora (main %)]$ skopeo inspect docker://registry.fedoraproject.org/fedora:rawhide | jq .Labels -c
{"io.buildah.version":"1.39.0","license":"MIT","name":"fedora","org.opencontainers.image.license":"MIT","org.opencontainers.image.name":"fedora","org.opencontainers.image.url":"https://fedoraproject.org/","org.opencontainers.image.vendor":"Fedora Project","org.opencontainers.image.version":"rawhide","vendor":"Fedora Project","version":"rawhide"}

that's the intended place for this sort of metadata, AIUI. Right now we don't have the bits you want in there, adding them does seem like a good idea.

https://pagure.io/fedora-kiwi-descriptions/pull-request/141
https://koji.fedoraproject.org/koji/buildinfo?buildID=2659939
https://koji.fedoraproject.org/koji/buildinfo?buildID=2659938

FEDORA-2025-a42c6884b0 (systemd-257.3-6.fc43) has been submitted as an update to Fedora 43.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-a42c6884b0

FEDORA-2025-a42c6884b0 (systemd-257.3-6.fc43) has been pushed to the Fedora 43 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2025-751940c072 (systemd-257.3-6.fc42) has been submitted as an update to Fedora 42.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-751940c072

FEDORA-2025-751940c072 (systemd-257.3-6.fc42) has been pushed to the Fedora 42 stable repository.
If problem still persists, please make note of it in this bug report.

This supposed fix has actually broken ELN Content Resolver for anything with a systemd dependency, where it was working just fine before, e.g.:

```
 Problem: package chrony-4.6.1-2.fc42.aarch64 from Rawhide requires systemd, but none of the providers can be installed
  - package systemd-standalone-sysusers-257.3-6.fc43.aarch64 from Rawhide conflicts with systemd provided by systemd-257.3-6.eln146.aarch64 from BaseOS
  - package systemd-standalone-sysusers-257.3-6.fc43.aarch64 from Rawhide conflicts with systemd provided by systemd-257.3-6.eln146.aarch64 from buildroot
  - problem with installed package systemd-standalone-sysusers-257.3-6.eln146.aarch64
  - package systemd-standalone-sysusers-257.3-6.eln146.aarch64 from buildroot conflicts with systemd provided by systemd-257.3-6.eln146.aarch64 from BaseOS
  - package systemd-standalone-sysusers-257.3-6.eln146.aarch64 from buildroot conflicts with systemd provided by systemd-257.3-6.eln146.aarch64 from buildroot
  - conflicting requests
  - package chrony-4.6.1-2.eln145.aarch64 from buildroot requires systemd, but none of the providers can be installed
  - package systemd-standalone-sysusers-257.3-6.eln146.aarch64 from @System conflicts with systemd provided by systemd-257.3-6.fc43.aarch64 from Rawhide
  - package systemd-standalone-sysusers-257.3-6.eln146.aarch64 from CRB conflicts with systemd provided by systemd-257.3-6.fc43.aarch64 from Rawhide
  - package systemd-standalone-sysusers-257.3-6.eln146.aarch64 from buildroot conflicts with systemd provided by systemd-257.3-6.fc43.aarch64 from Rawhide
  - package systemd-standalone-sysusers-257.3-6.fc43.aarch64 from Rawhide conflicts with systemd provided by systemd-257.3-6.fc43.aarch64 from Rawhide
  - package systemd-standalone-sysusers-257.3-6.eln146.aarch64 from CRB conflicts with systemd provided by systemd-257.3-6.eln146.aarch64 from BaseOS
  - package systemd-standalone-sysusers-257.3-6.eln146.aarch64 from CRB conflicts with systemd provided by systemd-257.3-6.eln146.aarch64 from buildroot
  - package systemd-standalone-sysusers-257.3-6.eln146.aarch64 from @System conflicts with systemd provided by systemd-257.3-6.eln146.aarch64 from BaseOS
  - package systemd-standalone-sysusers-257.3-6.eln146.aarch64 from @System conflicts with systemd provided by systemd-257.3-6.eln146.aarch64 from buildroot
  - package chrony-4.6.1-2.eln145.aarch64 from BaseOS requires systemd, but none of the providers can be installed
```

Furthermore, I don't see that it solves the OP problem, as seen in both rawhide and ELN containers (pulled just now):

```
[root@e60b3bcc5935 /]# dnf install httpd
Updating and loading repositories:
Repositories loaded.
Failed to resolve the transaction:
Problem: systemd-257.3-6.fc43.i686 from rawhide has inferior architecture
  - package httpd-2.4.62-6.fc42.x86_64 from rawhide requires systemd, but none of the providers can be installed
  - problem with installed package
  - installed package systemd-standalone-sysusers-257.3-6.fc43.x86_64 conflicts with systemd provided by systemd-257.3-6.fc43.x86_64 from rawhide
  - package systemd-standalone-sysusers-257.3-6.fc43.x86_64 from rawhide conflicts with systemd provided by systemd-257.3-6.fc43.x86_64 from rawhide
  - conflicting requests

```

IMO the actual fix for this would be for things to (almost?) never have dependencies on systemd (or related packages such as (systemd-)udev or other system services).  Either we're on a bootable system (or an init container) where systemd can be assumed to be present, or we're in a userspace container, toolbox, or flatpak where systemd will never be present (and whatever unit files or the like that are the reason for the systemd dep will never be used).  Unless the package is completely nonfunctional without systemd **in the same layer** (and hence unusable on a non-bootable system), then what purpose does the dependency serve?

However, that would mean a change to packaging guidelines and changes to a lot of packages to fully implement; while worthwhile in the long run, a stop-gap solution is likely needed, but it doesn't seem like 257.3-6 is it.

>   - problem with installed package systemd-standalone-sysusers-257.3-6.eln146.aarch64
This is the source of the problem. systemd-standalone-sysusers is installed, but
we need systemd-sysusers (the newly added subpackage) to be installed instead.
With the new package split, it should be possible to install systemd
without any conflicts on top of container-base.

Ah, right, https://pagure.io/fedora-kiwi-descriptions/pull-request/141 still
isn't merged. So that failure is expected.

I don't know why chrony requires systemd. We changed the Packaging Guidelines to
specifically say that services *should not* require systemd, because that makes
construction of certain container types harder. So that's something to fix independently.

I've proposed an alternative PR here to fix both fedora and eln: https://pagure.io/fedora-kiwi-descriptions/pull-request/147

Discussed on 2025-02-24 in a blocker review meeting [1]:

!agreed 2344322 - punt - We're not sure whether this is a bug, or just something to be documented (and improved in certain packages). A discussion with FESCo would be very helpful. But first let's try whether this issue is still valid or not, after latest package changes.

[1] https://meetbot.fedoraproject.org/blocker-review_matrix_fedoraproject-org/2025-02-24/f42-blocker-review.2025-02-24-17.01.log.html

The original issue:
> installed package systemd-standalone-sysusers-257.2-17.fc42.x86_64 conflicts with systemd provided by systemd-257.2-17.fc42.x86_64 from rawhide

does NOT reproduce anymore. It *is* now possible to install either systemd + systemd-sysusers
or systemd + systemd-standalone-sysusers.

is that also the case for F42?

Yes, the builds of systemd are identical in f42 and rawhide so far.