Bug 2106396

Example log: https://cockpit-logs.us-east-1.linodeobjects.com/pull-17500-20220712-134830-92565072-centos-8-stream/log.html

(In reply to Martin Pitt from comment #1)
> Example log:
> https://cockpit-logs.us-east-1.linodeobjects.com/pull-17500-20220712-134830-
> 92565072-centos-8-stream/log.html

I am unable to get any useful information off the logs.
Have you managed to get a reliable reproducer?
Were you able to nail down where it started to happen? It is very unlikely this actually was a regression in selinux-policy, so I'd look for some other component change.

The original logs from Fedora bug 2031022 are long gone (that image refresh had a version delta). For us it started to fail in RHEL 9.1 (bug 2031022) in May, but that's also too long ago to still have the logs from the image refresh, which would have the package version delta. Unfortunately this didn't coincide with a VM refresh in CentOS 8 stream for us, otherwise we'd have a package delta.

Our tracker at https://github.com/cockpit-project/bots/issues/2787 still sees this all the time, on CentOS 8 stream and Fedora CoreOS - the latter is where we run the most containers. It seems that pretty much every time a different test is affected, this seems to be some race condition between iptables (firewalld? isn't that using nftables these days?) and starting a container. I'd probably call this a regression in iptables or firewalld or so -- what business do they have to poke around in podman's overlays?

Nevertheless, this is hard to reproduce as it's a race condition. Every cockpit test starts ~ 300 VMs and even more podman containers, and it hits one or two of these. I'm running one test in a loop on my system, but it didn't bite so far. What we can do is to enable some debugging options to get more detailed logs, let them run for a while, and send the updated logs when it happens again.

What I tried:

   for i in `seq 1000`; do podman run -p 9090:9090 -d --rm quay.io/cockpit/ws ; sleep 1; podman rm -flt1; done
   journalctl -f | grep avc

Not quite that easy, I'm afraid.

Lokesh,

In this bz and also in Bug #2089257 and Bug #2031022 (RHEL 9, Fedora) we see quite a strange issue which is hard to reproduce.
It also probably is a result of a yet unknown component update.

Do you know what is going on here?
Do you have any idea how to trobleshoot further?

Copying Dan Walsh..

This looks like a leaked file descriptor?

I do not believe that iptables would be doing an ioctl on the merge directory.

For now this AVC is can most likely be ignored.

Reproduced (only F37), although still not understood. The denial with full auditing:

----
type=PROCTITLE msg=audit(09/13/2022 08:44:00.074:28547) : proctitle=iptables -t nat -C OUTPUT -j NETAVARK-HOSTPORT-DNAT -m addrtype --dst-type LOCAL --wait
type=PATH msg=audit(09/13/2022 08:44:00.074:28547) : item=1 name=/lib64/ld-linux-x86-64.so.2 inode=137936 dev=fc:01 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:ld_so_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(09/13/2022 08:44:00.074:28547) : item=0 name=/usr/sbin/iptables inode=158128 dev=fc:01 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:iptables_exec_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(09/13/2022 08:44:00.074:28547) : cwd=/root
type=EXECVE msg=audit(09/13/2022 08:44:00.074:28547) : argc=12 a0=iptables a1=-t a2=nat a3=-C a4=OUTPUT a5=-j a6=NETAVARK-HOSTPORT-DNAT a7=-m a8=addrtype a9=--dst-type a10=LOCAL a11=--wait
type=SYSCALL msg=audit(09/13/2022 08:44:00.074:28547) : arch=x86_64 syscall=execve success=yes exit=0 a0=0x7fae866b4d90 a1=0x55a49c0edd10 a2=0x7ffc81d2a1b8 a3=0x8 items=2 ppid=151721 pid=151776 auid=root uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=1 comm=iptables exe=/usr/sbin/xtables-nft-multi subj=unconfined_u:system_r:iptables_t:s0-s0:c0.c1023 key=(null)
type=AVC msg=audit(09/13/2022 08:44:00.074:28547) : avc:  denied  { ioctl } for  pid=151776 comm=iptables path=/var/lib/containers/storage/overlay/ec00a0ddc348f7f6ec69426049d6e14a20ff9dd655b07494f41f0d80a2fe2bc9/merged dev="overlay" ino=399476 scontext=unconfined_u:system_r:iptables_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0:c121,c900 tclass=dir permissive=0
----

happens on execve and with success=yes, so likely a leaked descriptor.

Switching the component, see
https://bugzilla.redhat.com/show_bug.cgi?id=2031022

Fixed in container-selinux-2.191.0-1

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2758

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2758