Bug 2123210

Summary: podman network backend does not switch to netavark when embedding container in image
Product: Red Hat Enterprise Linux 9 Reporter: Xiaofeng Wang <xiaofwan>
Component: osbuild-composerAssignee: Achilleas Koutsou <akoutsou>
Status: CLOSED ERRATA QA Contact: Xiaofeng Wang <xiaofwan>
Severity: high Docs Contact:
Priority: high    
Version: 9.1CC: akoutsou, mcattamo, obudai, pvlasin, qzhang, thozza, yih
Target Milestone: rcKeywords: Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2125249 (view as bug list) Environment:
Last Closed: 2022-11-15 09:50:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2125249    

Description Xiaofeng Wang 2022-09-01 04:30:16 UTC
Description of problem:
Enable embedding container in image, then install the image in to VM. Connect to VM, then run "sudo podman run ubi8-minimal:latest cat /etc/redhat-release" failed with the following error:

WARN[0005] Failed to load cached network config: network podman not found in CNI cache, falling back to loading network podman from disk
WARN[0005] 1 error occurred:
    * plugin type="tuning" failed (delete): failed to find plugin "tuning" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin]

Error: plugin type="bridge" failed (add): failed to find plugin "bridge" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin]

But run "podman run ubi8-minimal:latest cat /etc/redhat-release" without any error.

Get network backend info:
podman info -> networkBackend: netavark
sudo podman info -> networkBackend: cni
In root mode, the podman keeps using cni not netavark.

Do not have this issue if embedding container does not enabled.

Version-Release number of selected component (if applicable):
podman version
Client:       Podman Engine
Version:      4.2.0
API Version:  4.2.0
Go Version:   go1.18.4
Built:        Mon Aug 22 08:37:10 2022
OS/Arch:      linux/amd64

How reproducible:

Steps to Reproduce:
1. git clone https://github.com/virt-s1/rhel-edge.git
2. cd rhel-edge
3. ./ostree.sh

Actual results:
run "sudo podman run ubi8-minimal:latest cat /etc/redhat-release" failed

Expected results:
no error found

Additional info:

Comment 1 Achilleas Koutsou 2022-09-01 10:42:29 UTC
In podman 4.2.0 the default network is netavark but it falls back to CNI in some cases

From the default config file (/usr/share/containers/containers.conf)

> # The default value is empty which means that it will automatically choose CNI or netavark. If there are
> # already containers/images or CNI networks preset it will choose CNI.

When we embed a container in the image, podman uses CNI, but this requires containernetworking-plugins which we do not install in our images.

We should add containernetworking-plugins whenever we add podman to cover all use cases.

Comment 2 Achilleas Koutsou 2022-09-01 10:47:44 UTC
(In reply to Achilleas Koutsou from comment #1)
> In podman 4.2.0 the default network is netavark but it falls back to CNI in
> some cases

Correction: this change happened in v4.0.0 https://github.com/containers/podman/releases/tag/v4.0.0

Comment 12 Xiaofeng Wang 2022-09-21 08:57:33 UTC
Verified on osbuild-composer-62.1-1.el9.x86_64.rpm

Comment 14 errata-xmlrpc 2022-11-15 09:50:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: Image Builder security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7950