Bug 2150692
| Summary: | podman can't create network with error "creating locker directory: mkdir /etc/containers/networks: permission denied" | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Xiaofeng Wang <xiaofwan> |
| Component: | container-selinux | Assignee: | Daniel Walsh <dwalsh> |
| Status: | CLOSED MIGRATED | QA Contact: | atomic-bugs <atomic-bugs> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 9.2 | CC: | bbaude, dwalsh, elpereir, gscrivan, jnovy, lsm5, lvrabec, mboddu, mheon, mmalik, pthomas, tsweeney, umohnani, zpytela |
| Target Milestone: | rc | Keywords: | MigratedToJIRA |
| Target Release: | 9.2 | Flags: | pm-rhel:
mirror+
|
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-09-11 19:08:29 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Xiaofeng Wang
2022-12-05 02:33:12 UTC
@gscrivan could you please take a look? the issue is not related to crun, "network create" doesn't use the OCI runtime. Why is /etc read only? If you create a network you need to be able to write to /etc No, /etc is not read only, it's writable. And compose RHEL-9.2.0-20221129.2 and before does not have this issue. could you try disabling temporarily selinux? Yeah, "setenforce 0" fixed this issue. AVCs please? ausearch -m avc -ts recent I've re-assigned to container-selinux since it doesn't depend from the Podman upgrade (In reply to Daniel Walsh from comment #7) > AVCs please? > > ausearch -m avc -ts recent sudo ausearch -m avc -ts recent <no matches> Did you o this right after a failure? Right. + sudo podman --log-level debug network inspect edge + sudo podman --log-level debug network create --driver=bridge --subnet=192.168.200.0/24 --gateway=192.168.200.254 edge time="2022-12-05T11:56:39-05:00" level=info msg="podman filtering at log level debug" time="2022-12-05T11:56:39-05:00" level=debug msg="Called create.PersistentPreRunE(podman --log-level debug network create --driver=bridge --subnet=192.168.200.0/24 --gateway=192.168.200.254 edge)" time="2022-12-05T11:56:39-05:00" level=debug msg="Merged system config \"/usr/share/containers/containers.conf\"" time="2022-12-05T11:56:39-05:00" level=debug msg="Using conmon: \"/usr/bin/conmon\"" time="2022-12-05T11:56:39-05:00" level=debug msg="Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db" time="2022-12-05T11:56:39-05:00" level=debug msg="Using graph driver overlay" time="2022-12-05T11:56:39-05:00" level=debug msg="Using graph root /var/lib/containers/storage" time="2022-12-05T11:56:39-05:00" level=debug msg="Using run root /run/containers/storage" time="2022-12-05T11:56:39-05:00" level=debug msg="Using static dir /var/lib/containers/storage/libpod" time="2022-12-05T11:56:39-05:00" level=debug msg="Using tmp dir /run/libpod" time="2022-12-05T11:56:39-05:00" level=debug msg="Using volume path /var/lib/containers/storage/volumes" time="2022-12-05T11:56:39-05:00" level=debug msg="Set libpod namespace to \"\"" time="2022-12-05T11:56:39-05:00" level=debug msg="[graphdriver] trying provided driver \"overlay\"" time="2022-12-05T11:56:39-05:00" level=debug msg="Cached value indicated that overlay is supported" time="2022-12-05T11:56:39-05:00" level=debug msg="Cached value indicated that overlay is supported" time="2022-12-05T11:56:39-05:00" level=debug msg="Cached value indicated that metacopy is being used" time="2022-12-05T11:56:39-05:00" level=debug msg="Cached value indicated that native-diff is not being used" time="2022-12-05T11:56:39-05:00" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled" time="2022-12-05T11:56:39-05:00" level=debug msg="backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true" time="2022-12-05T11:56:39-05:00" level=debug msg="Initializing event backend file" time="2022-12-05T11:56:39-05:00" level=debug msg="Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument" time="2022-12-05T11:56:39-05:00" level=debug msg="Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument" time="2022-12-05T11:56:39-05:00" level=debug msg="Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument" time="2022-12-05T11:56:39-05:00" level=debug msg="Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument" time="2022-12-05T11:56:39-05:00" level=debug msg="Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument" time="2022-12-05T11:56:39-05:00" level=debug msg="Using OCI runtime \"/usr/bin/crun\"" Error: creating locker directory: mkdir /etc/containers/networks: permission denied + true + sudo ausearch -m avc -ts recent <no matches> could you run "semodule -DB" first then repeat the podman command and "sudo ausearch -m avc -ts recent"? Still <no matches> + sudo semodule -DB + sudo podman --log-level debug network inspect edge + sudo podman --log-level debug network create --driver=bridge --subnet=192.168.200.0/24 --gateway=192.168.200.254 edge time="2022-12-06T04:23:20-05:00" level=info msg="podman filtering at log level debug" time="2022-12-06T04:23:20-05:00" level=debug msg="Called create.PersistentPreRunE(podman --log-level debug network create --driver=bridge --subnet=192.168.200.0/24 --gateway=192.168.200.254 edge)" time="2022-12-06T04:23:20-05:00" level=debug msg="Merged system config \"/usr/share/containers/containers.conf\"" time="2022-12-06T04:23:20-05:00" level=debug msg="Using conmon: \"/usr/bin/conmon\"" time="2022-12-06T04:23:20-05:00" level=debug msg="Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db" time="2022-12-06T04:23:20-05:00" level=debug msg="Using graph driver overlay" time="2022-12-06T04:23:20-05:00" level=debug msg="Using graph root /var/lib/containers/storage" time="2022-12-06T04:23:20-05:00" level=debug msg="Using run root /run/containers/storage" time="2022-12-06T04:23:20-05:00" level=debug msg="Using static dir /var/lib/containers/storage/libpod" time="2022-12-06T04:23:20-05:00" level=debug msg="Using tmp dir /run/libpod" time="2022-12-06T04:23:20-05:00" level=debug msg="Using volume path /var/lib/containers/storage/volumes" time="2022-12-06T04:23:20-05:00" level=debug msg="Set libpod namespace to \"\"" time="2022-12-06T04:23:20-05:00" level=debug msg="[graphdriver] trying provided driver \"overlay\"" time="2022-12-06T04:23:20-05:00" level=debug msg="Cached value indicated that overlay is supported" time="2022-12-06T04:23:20-05:00" level=debug msg="Cached value indicated that overlay is supported" time="2022-12-06T04:23:20-05:00" level=debug msg="Cached value indicated that metacopy is being used" time="2022-12-06T04:23:20-05:00" level=debug msg="Cached value indicated that native-diff is not being used" time="2022-12-06T04:23:20-05:00" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled" time="2022-12-06T04:23:20-05:00" level=debug msg="backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true" time="2022-12-06T04:23:20-05:00" level=debug msg="Initializing event backend file" time="2022-12-06T04:23:20-05:00" level=debug msg="Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument" time="2022-12-06T04:23:20-05:00" level=debug msg="Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument" time="2022-12-06T04:23:20-05:00" level=debug msg="Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument" time="2022-12-06T04:23:20-05:00" level=debug msg="Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument" time="2022-12-06T04:23:20-05:00" level=debug msg="Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument" time="2022-12-06T04:23:20-05:00" level=debug msg="Using OCI runtime \"/usr/bin/crun\"" Error: creating locker directory: mkdir /etc/containers/networks: permission denied + true + sudo ausearch -m avc -ts recent <no matches> Looks like the auditing subsystem is not working if you got no avcs Could you run `jounalctl -b | grep -i avc` + journalctl -b + grep -i avc Dec 06 22:18:22 runner-rhos-01-rhel-9-2-large-4330 dbus-broker-launch[721]: avc: op=load_policy lsm=selinux seqno=2 res=1 Dec 06 22:18:50 runner-rhos-01-rhel-9-2-large-4330 dbus-broker-launch[721]: avc: op=load_policy lsm=selinux seqno=5 res=1 Dec 06 22:19:07 runner-rhos-01-rhel-9-2-large-4330 dbus-broker-launch[721]: avc: op=load_policy lsm=selinux seqno=6 res=1 This system is not looking like an SELinux system or something is very wrong. Are you sure this works if you set `sudo setenforce 0` Yes. And the test details can be found from https://github.com/virt-s1/rhel-edge/actions/runs/3619140872/jobs/6099842294#step:4:3938 + sudo setenforce 0 + sudo podman --log-level debug network inspect edge + sudo podman --log-level debug network create --driver=bridge --subnet=192.168.200.0/24 --gateway=192.168.200.254 edge time="2022-12-05T05:41:14-05:00" level=info msg="podman filtering at log level debug" time="2022-12-05T05:41:14-05:00" level=debug msg="Called create.PersistentPreRunE(podman --log-level debug network create --driver=bridge --subnet=192.168.200.0/24 --gateway=192.168.200.254 edge)" time="2022-12-05T05:41:14-05:00" level=debug msg="Merged system config \"/usr/share/containers/containers.conf\"" time="2022-12-05T05:41:14-05:00" level=debug msg="Using conmon: \"/usr/bin/conmon\"" time="2022-12-05T05:41:14-05:00" level=debug msg="Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db" time="2022-12-05T05:41:14-05:00" level=debug msg="Using graph driver overlay" time="2022-12-05T05:41:14-05:00" level=debug msg="Using graph root /var/lib/containers/storage" time="2022-12-05T05:41:14-05:00" level=debug msg="Using run root /run/containers/storage" time="2022-12-05T05:41:14-05:00" level=debug msg="Using static dir /var/lib/containers/storage/libpod" time="2022-12-05T05:41:14-05:00" level=debug msg="Using tmp dir /run/libpod" time="2022-12-05T05:41:14-05:00" level=debug msg="Using volume path /var/lib/containers/storage/volumes" time="2022-12-05T05:41:14-05:00" level=debug msg="Set libpod namespace to \"\"" time="2022-12-05T05:41:14-05:00" level=debug msg="[graphdriver] trying provided driver \"overlay\"" time="2022-12-05T05:41:14-05:00" level=debug msg="Cached value indicated that overlay is supported" time="2022-12-05T05:41:14-05:00" level=debug msg="Cached value indicated that overlay is supported" time="2022-12-05T05:41:14-05:00" level=debug msg="Cached value indicated that metacopy is being used" edge time="2022-12-05T05:41:14-05:00" level=debug msg="Cached value indicated that native-diff is not being used" time="2022-12-05T05:41:14-05:00" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled" time="2022-12-05T05:41:14-05:00" level=debug msg="backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true" time="2022-12-05T05:41:14-05:00" level=debug msg="Initializing event backend journald" time="2022-12-05T05:41:14-05:00" level=debug msg="Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument" time="2022-12-05T05:41:14-05:00" level=debug msg="Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument" time="2022-12-05T05:41:14-05:00" level=debug msg="Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument" time="2022-12-05T05:41:14-05:00" level=debug msg="Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument" time="2022-12-05T05:41:14-05:00" level=debug msg="Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument" time="2022-12-05T05:41:14-05:00" level=debug msg="Using OCI runtime \"/usr/bin/crun\"" time="2022-12-05T05:41:14-05:00" level=info msg="Setting parallel job count to 13" time="2022-12-05T05:41:14-05:00" level=debug msg="Successfully loaded 1 networks" time="2022-12-05T05:41:14-05:00" level=debug msg="found free device name podman1" time="2022-12-05T05:41:14-05:00" level=debug msg="Called create.PersistentPostRunE(podman --log-level debug network create --driver=bridge --subnet=192.168.200.0/24 --gateway=192.168.200.254 edge)" ----
type=PROCTITLE msg=audit(12/09/2022 11:04:03.319:1096) : proctitle=/usr/bin/podman images --all --no-trunc --digests
type=PATH msg=audit(12/09/2022 11:04:03.319:1096) : item=1 name=/etc/containers/networks nametype=CREATE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(12/09/2022 11:04:03.319:1096) : item=0 name=/etc/containers/ inode=8390229 dev=fd:01 mode=dir,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:etc_t:s0 nametype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(12/09/2022 11:04:03.319:1096) : cwd=/
type=SYSCALL msg=audit(12/09/2022 11:04:03.319:1096) : arch=x86_64 syscall=mkdirat success=no exit=EACCES(Permission denied) a0=AT_FDCWD a1=0xc0003a9540 a2=0700 a3=0x0 items=2 ppid=25857 pid=25858 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=podman exe=/usr/bin/podman subj=system_u:system_r:container_runtime_t:s0 key=(null)
type=AVC msg=audit(12/09/2022 11:04:03.319:1096) : avc: denied { create } for pid=25858 comm=podman name=networks scontext=system_u:system_r:container_runtime_t:s0 tcontext=system_u:object_r:etc_t:s0 tclass=dir permissive=0
----
Reproducible on 1MT-RHEL-9.2.0-20221205.2 machines with the right set of installed packages:
# rpm -qa selinux\* container\* insights-client\* | sort
containers-common-1-46.el9.x86_64
container-selinux-2.193.0-1.el9.noarch
insights-client-3.1.7-8.el9.noarch
selinux-policy-38.1.2-1.el9.noarch
selinux-policy-targeted-38.1.2-1.el9.noarch
#
Triggered by the insights-client service which performed a data collection activity on the machines.
# grep -i networks /var/log/insights-client/* | grep -v '{'
/var/log/insights-client/insights-client.log: "RPC call Host.setupNetworks failed",
/var/log/insights-client/insights-client.log.1: "RPC call Host.setupNetworks failed",
/var/log/insights-client/insights-client.log.1:2022-12-09 11:04:01,211 DEBUG insights.core.plugins (125, ['timeout', '-s', '9', '120', '/usr/bin/docker', 'ps', '--all', '--no-trunc'], b'Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.\nError: creating locker directory: mkdir /etc/containers/networks: permission denied\n')
/var/log/insights-client/insights-client.log.1:2022-12-09 11:04:01,320 DEBUG insights.core.plugins (125, ['timeout', '-s', '9', '120', '/usr/bin/podman', 'ps', '--all', '--no-trunc'], b'Error: creating locker directory: mkdir /etc/containers/networks: permission denied\n')
/var/log/insights-client/insights-client.log.2: "RPC call Host.setupNetworks failed",
/var/log/insights-client/insights-client.log.3: "RPC call Host.setupNetworks failed",
#
This issue does not exist since compose RHEL-9.2.0-20221209.2. Does the container-selinux-3:2.193.0-2.el9.noarch fixed this issue? Or container-selinux-3:2.193.0-2.el9.noarch is built against selinux-policy-38.1.2-1.el9.noarch? Thanks. It was not a container-selinux fix but a package fix. rpm -q container-selinux --requires | grep selinux-policy Should show you the selinux-policy package that is required to be installed before container-selinux. Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug. This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there. Due to differences in account names between systems, some fields were not replicated. Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information. To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer. You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like: "Bugzilla Bug" = 1234567 In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information. |