Bug 2124887

# podman info
host:
  arch: amd64
  buildahVersion: 1.27.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpuUtilization:
    idlePercent: 89.02
    systemPercent: 3.48
    userPercent: 7.5
  cpus: 8
  distribution:
    distribution: fedora
    variant: workstation
    version: "36"
  eventLogger: journald
  hostname: greebo
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.19.6-200.fc36.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 23288221696
  memTotal: 33579565056
  networkBackend: cni
  ociRuntime:
    name: crun
    package: crun-1.5-1.fc36.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.5
      commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 11811151872
  swapTotal: 11811151872
  uptime: 0h 60m 59.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 10
    paused: 0
    running: 3
    stopped: 7
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 154386116608
  graphRootUsed: 86245064704
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 6
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.2.0
  Built: 1660228937
  BuiltTime: Thu Aug 11 16:42:17 2022
  GitCommit: ""
  GoVersion: go1.18.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.2.0

This looks like a bug, but If you can I would recommend you remove CNI and use netavark.

podman system reset

Should remove all containers and images and allow you to switch to Netavark.

Hi Alex,

I tried with an initial installation of containernetworking-plugins + podman v4.1.1 and then ran a dnf update to v4.2.0 but containernetworking-plugins didn't get removed.

Did you perhaps have any modifications to dnf options, like uninstall recommended or suggested packages by any chance?

Note, netavark is a `Requires` on podman, while containernetworking-plugins is a `Suggests`.

containernetworking-plugins was not removed, it just was never there before.

I see, it's strange that it would complain about cni-plugins if it was never there there, because we haven't encountered this issue before for fresh installs.

Could you please tell us more about the background of this environment?

Was this a fresh Fedora 36 with a first time podman install?

Did you by any chance also have buildah or skopeo (or anything that depends on containers-common) installed prior to podman installation?

This was my desktop machine with F36, which has been incrementally updated many times, so far from a fresh install.

However, I should note that we saw the same issue in the automotive project where we're building an image that just adds the "podman" package. It used to work, but once 4.2 hit the repos it produced an image without containernetworking-plugins where podman failed to start. Changing the manifest to explicitly add the containernetworking-plugins rpm fixed this. Of course, this is a centos stream 9 image, and it's build with ignore-weak-deps, so its a different situation.

Are you making any config changes to force CNI? Because Stream 9 should always be on Netavark without intervention.

No, there was no specific configuration for CNI, just regular install of podman.
Here is the image manifest:

 https://gitlab.com/CentOS/automotive/sample-images/-/blob/main/osbuild-manifests/images/soafee.mpp.yml

It now lists containernetworking-plugins in the package list, but before this merge request: https://gitlab.com/CentOS/automotive/sample-images/-/merge_requests/126 it didn't, and it worked fine with podman 4.1. However once 4.2 was in the repo it created a broken image, which was just noticed when we tried to do that change.

Just to add on Alex's comments, since the image is built form scratch, "podman system reset" did not do anything to solve the issue. 
In addition, please note that in the image case, the issue reproduces only when running podman as root - with sudo or as a systemd service (without --user). Running rootless worked just fine

I see what looks like it might be a modified storage.conf being installed. Is that true, and if so what changes are being made?

I believe https://github.com/containers/common/pull/1148 might fix this issue.

https://gitlab.com/CentOS/automotive/sample-images/-/blob/main/osbuild-manifests/files/storage.conf this is the custom storage.conf. It is primarily setting up additionalimagestores.

I don't seem to have a custom storage.conf on my system though. Although I did at some point (for setting additionalimagestores).

(I didn't try reseting though).

Patch from Dan will definitely fix us, then - our upgrade detection logic was broken when additional image stores were in use.

I've tested an image without modifying storage.conf and I can confirm that both rootless and rootful is working with the default configuration. So, yes, it is related to the storage.conf file that Alex linked.

Alex, Ygal, does this still occur for you?

This message is a reminder that Fedora Linux 36 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 36 on 2023-05-16.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '36'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 36 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Fedora Linux 36 entered end-of-life (EOL) status on 2023-05-16.

Fedora Linux 36 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora Linux
please feel free to reopen this bug against that version. Note that the version
field may be hidden. Click the "Show advanced fields" button if you do not see
the version field.

If you are unable to reopen this bug, please file a new report against an
active release.

Thank you for reporting this bug and we are sorry it could not be fixed.

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days