RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2106396 - avc: denied { ioctl } for pid=510216 comm="iptables" path="/var/lib/containers/storage/overlay/7d65c03c0ff08daf6366d735723151aa1f2cf165d51be30f62bded9ed586b838/merged" dev="overlay" ino=42308193 scontext=unconfined_u:system_r:iptables_t:s0-s0:c0.c1023
Summary: avc: denied { ioctl } for pid=510216 comm="iptables" path="/var/lib/contai...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: container-selinux
Version: CentOS Stream
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: rc
: ---
Assignee: Jindrich Novy
QA Contact: Edward Shen
URL:
Whiteboard: CockpitTest
Depends On: 2031022 2089257
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-07-12 14:41 UTC by Martin Pitt
Modified: 2023-05-16 09:05 UTC (History)
26 users (show)

Fixed In Version: container-selinux-2.191.0-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 2089257
Environment:
Last Closed: 2023-05-16 08:20:33 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-127508 0 None None None 2022-07-12 14:54:27 UTC
Red Hat Product Errata RHSA-2023:2758 0 None None None 2023-05-16 08:39:32 UTC

Description Martin Pitt 2022-07-12 14:41:54 UTC 
+++ This bug was initially created as a clone of Bug #2089257 +++

+++ This bug was initially created as a clone of Bug #2031022 +++

Description of problem:
During CKI podman test [1] we've hit the following issue:

avc:  denied  { ioctl } for  pid=510216 comm="iptables" path="/var/lib/containers/storage/overlay/7d65c03c0ff08daf6366d735723151aa1f2cf165d51be30f62bded9ed586b838/merged" dev="overlay" ino=42308193 scontext=unconfined_u:system_r:iptables_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0:c878,c982 tclass=dir permissive=0

Version-Release number of selected component (if applicable):
selinux-policy-35.6-1.fc36.noarch

How reproducible:
It seems easily reproducible with podman test

Steps to Reproduce:
1. Run test [1]


Additional info:
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33
selinux-policy-35.6-1.fc36.noarch

[1] https://gitlab.com/cki-project/kernel-tests/-/tree/main/container/podman

--- Additional comment from Bruno Goncalves on 2021-12-10 11:19:12 UTC ---

beaker job: https://beaker.engineering.redhat.com/recipes/11120125#task136635605

--- Additional comment from Zdenek Pytela on 2021-12-10 11:37:43 UTC ---

Switching the component, there are already some rules in container-selinux:

f35# sesearch -A -s iptables_t -t container_file_t -c dir,file
allow application_domain_type logfile:file { append getattr ioctl lock };
allow domain file_type:file map; [ domain_can_mmap_files ]:True
allow iptables_t container_file_t:dir { getattr open search };
allow iptables_t container_file_t:file open;

--- Additional comment from Martin Pitt on 2022-01-10 07:36:05 UTC ---

In Cockpit we started to see these as well now, in our Fedora CoreOS CI: for example, [1]

audit: type=1400 audit(1641799023.984:237): avc:  denied  { ioctl } for  pid=1161 comm="iptables" path="/var/lib/containers/storage/overlay/0fac9b410d57d0f8ae6fa8f042e8672ae70ddbeb4e25845223e35a3b2260c169/merged" dev="overlay" ino=17950249 scontext=unconfined_u:system_r:iptables_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=0

Nothing in that test fiddles with iptables, that's somehow internal to podman. But lots of tests start a cockpit/ws container there, so this just feels random. E.g. here [2] and here[3] it hit two different, and completely unrelated tests. We haven't seen this before, and haven't refreshed our CoreOS image in a whole week. Was this some sort of a time bomb? (But I can't imagine SELinux rules being time dependent)

[1] https://logs.cockpit-project.org/logs/pull-16798-20220110-064948-be591291-fedora-coreos/log.html#232
[2] https://logs.cockpit-project.org/logs/pull-2784-20220110-024544-7afed88e-fedora-coreos-cockpit-project-cockpit/log.html#152
[3] https://logs.cockpit-project.org/logs/pull-2784-20220110-063907-7afed88e-fedora-coreos-cockpit-project-cockpit/log.html#241

--- Additional comment from Ben Cotton on 2022-02-08 21:15:40 UTC ---

This bug appears to have been reported against 'rawhide' during the Fedora 36 development cycle.
Changing version to 36.

--- Additional comment from Marius Vollmer on 2022-05-23 10:01:29 UTC ---

This now turns up on RHEL 9.1 as well.

selinux-policy-34.1.32-1.el9.noarch

https://cockpit-logs.us-east-1.linodeobjects.com/pull-3398-20220523-090548-c9f0c810-rhel-9-1-cockpit-project-cockpit/log.html#51



This now trickled into CentOS 8 stream as well, so most likely RHEL 8 soon as well. It's a little saddening to see regressions land in stable releases which have been known for 7 months (!).
Comment 1 Martin Pitt 2022-07-12 14:42:22 UTC 
Example log: https://cockpit-logs.us-east-1.linodeobjects.com/pull-17500-20220712-134830-92565072-centos-8-stream/log.html
Comment 2 Zdenek Pytela 2022-08-10 11:55:13 UTC 
(In reply to Martin Pitt from comment #1)
> Example log:
> https://cockpit-logs.us-east-1.linodeobjects.com/pull-17500-20220712-134830-
> 92565072-centos-8-stream/log.html

I am unable to get any useful information off the logs.
Have you managed to get a reliable reproducer?
Were you able to nail down where it started to happen? It is very unlikely this actually was a regression in selinux-policy, so I'd look for some other component change.
Comment 3 Martin Pitt 2022-08-11 04:45:48 UTC 
The original logs from Fedora bug 2031022 are long gone (that image refresh had a version delta). For us it started to fail in RHEL 9.1 (bug 2031022) in May, but that's also too long ago to still have the logs from the image refresh, which would have the package version delta. Unfortunately this didn't coincide with a VM refresh in CentOS 8 stream for us, otherwise we'd have a package delta.

Our tracker at https://github.com/cockpit-project/bots/issues/2787 still sees this all the time, on CentOS 8 stream and Fedora CoreOS - the latter is where we run the most containers. It seems that pretty much every time a different test is affected, this seems to be some race condition between iptables (firewalld? isn't that using nftables these days?) and starting a container. I'd probably call this a regression in iptables or firewalld or so -- what business do they have to poke around in podman's overlays?

Nevertheless, this is hard to reproduce as it's a race condition. Every cockpit test starts ~ 300 VMs and even more podman containers, and it hits one or two of these. I'm running one test in a loop on my system, but it didn't bite so far. What we can do is to enable some debugging options to get more detailed logs, let them run for a while, and send the updated logs when it happens again.
Comment 4 Martin Pitt 2022-08-11 04:56:38 UTC 
What I tried:

   for i in `seq 1000`; do podman run -p 9090:9090 -d --rm quay.io/cockpit/ws ; sleep 1; podman rm -flt1; done
   journalctl -f | grep avc

Not quite that easy, I'm afraid.
Comment 5 Zdenek Pytela 2022-08-11 08:25:14 UTC 
Lokesh,

In this bz and also in Bug #2089257 and Bug #2031022 (RHEL 9, Fedora) we see quite a strange issue which is hard to reproduce.
It also probably is a result of a yet unknown component update.

Do you know what is going on here?
Do you have any idea how to trobleshoot further?
Comment 6 Lokesh Mandvekar 2022-08-11 13:29:16 UTC 
Copying Dan Walsh..
Comment 7 Daniel Walsh 2022-08-11 15:19:35 UTC 
This looks like a leaked file descriptor?

I do not believe that iptables would be doing an ioctl on the merge directory.

For now this AVC is can most likely be ignored.
Comment 8 Zdenek Pytela 2022-09-13 14:22:51 UTC 
Reproduced (only F37), although still not understood. The denial with full auditing:

----
type=PROCTITLE msg=audit(09/13/2022 08:44:00.074:28547) : proctitle=iptables -t nat -C OUTPUT -j NETAVARK-HOSTPORT-DNAT -m addrtype --dst-type LOCAL --wait
type=PATH msg=audit(09/13/2022 08:44:00.074:28547) : item=1 name=/lib64/ld-linux-x86-64.so.2 inode=137936 dev=fc:01 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:ld_so_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(09/13/2022 08:44:00.074:28547) : item=0 name=/usr/sbin/iptables inode=158128 dev=fc:01 mode=file,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:iptables_exec_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(09/13/2022 08:44:00.074:28547) : cwd=/root
type=EXECVE msg=audit(09/13/2022 08:44:00.074:28547) : argc=12 a0=iptables a1=-t a2=nat a3=-C a4=OUTPUT a5=-j a6=NETAVARK-HOSTPORT-DNAT a7=-m a8=addrtype a9=--dst-type a10=LOCAL a11=--wait
type=SYSCALL msg=audit(09/13/2022 08:44:00.074:28547) : arch=x86_64 syscall=execve success=yes exit=0 a0=0x7fae866b4d90 a1=0x55a49c0edd10 a2=0x7ffc81d2a1b8 a3=0x8 items=2 ppid=151721 pid=151776 auid=root uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=1 comm=iptables exe=/usr/sbin/xtables-nft-multi subj=unconfined_u:system_r:iptables_t:s0-s0:c0.c1023 key=(null)
type=AVC msg=audit(09/13/2022 08:44:00.074:28547) : avc:  denied  { ioctl } for  pid=151776 comm=iptables path=/var/lib/containers/storage/overlay/ec00a0ddc348f7f6ec69426049d6e14a20ff9dd655b07494f41f0d80a2fe2bc9/merged dev="overlay" ino=399476 scontext=unconfined_u:system_r:iptables_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0:c121,c900 tclass=dir permissive=0
----

happens on execve and with success=yes, so likely a leaked descriptor.
Comment 9 Zdenek Pytela 2022-09-19 14:21:30 UTC 
Switching the component, see
https://bugzilla.redhat.com/show_bug.cgi?id=2031022
Comment 10 Daniel Walsh 2022-12-01 00:38:49 UTC 
Fixed in container-selinux-2.191.0-1
Comment 17 errata-xmlrpc 2023-05-16 08:20:33 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2758
Comment 18 errata-xmlrpc 2023-05-16 08:39:24 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2758
Note You need to log in before you can comment on or make changes to this bug.