Description of problem: You can not ping through a specific IP, [root@podaxxx /]# ping 172.20.32.1 -I 172.20.32.234 bind: Permission denied Version-Release number of selected component (if applicable): while you can via a specific interface. [root@podaxxx/]# ping 172.20.32.1 -I ext0 PING 172.20.32.1 (172.20.32.1) from 172.20.32.234 ext0: 56(84) bytes of data. 64 bytes from 172.20.32.1: icmp_seq=1 ttl=64 time=0.962 ms 64 bytes from 172.20.32.1: icmp_seq=2 ttl=64 time=1.25 ms ^C This test is done on ubi7. pods with privileged have the same problem. How reproducible: Always
This might be caused by removal of the NET_RAW capability. Is the host a RHEL7 machine?
Seunghwan, Can you give a little more description on what you did to setup the container so we can reproduce it? As Dan asked, and operating system version would be a good start along with what ever commands that you ran before you hit the error.
Tom, It is Openshift 4.6, RHCOS kernel version is 4.18.0-193.37.1.el8_2.x86_64. Please ignore Comment 1, I can reproduce it with ubi8 also. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ $ cat test.yaml apiVersion: v1 kind: Pod metadata: name: test spec: containers: - name: test securityContext: privileged: true image: ubi8 lifecycle: postStart: exec: command: - sh - -c - yum install iputils iproute -y command: ['sh', '-c', 'echo "Hello, world!" && sleep 3600'] $ oc rsh test sh-4.4# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: eth0@if4938: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default link/ether 0a:58:0a:80:02:42 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.128.2.66/23 brd 10.128.3.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::8079:40ff:fef7:3cb0/64 scope link valid_lft forever preferred_lft forever sh-4.4# ping 8.8.8.8 -I eth0 PING 8.8.8.8 (8.8.8.8) from 10.128.2.66 eth0: 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=113 time=20.10 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=113 time=8.31 ms 64 bytes from 8.8.8.8: icmp_seq=3 ttl=113 time=8.30 ms ^C --- 8.8.8.8 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 5ms rtt min/avg/max/mdev = 8.302/12.532/20.986/5.977 ms sh-4.4# ping 8.8.8.8 -I eth0@if4938 ping: SO_BINDTODEVICE: No such device sh-4.4# ping 8.8.8.8 -I 10.128.2.66 bind: Permission denied <======================= ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I think this one is more targeted to the Node team. Holler if I'm mistaken.
I think Dan is correct, try apiVersion: v1 kind: Pod metadata: name: test spec: containers: - name: test securityContext: privileged: true capabilities: add: ["NET_RAW"] image: ubi8 lifecycle: postStart: exec: command: - sh - -c - yum install iputils iproute -y command: ['sh', '-c', 'echo "Hello, world!" && sleep 3600']
Hi peter, just tested #6, still the same.
(In reply to Harshal Patil from comment #12) > I am using standard RHCOS system that's used with OpenShift 4.7. > > I am moving this bug to RHCOS team to check if unconfined module is disabled > by default on RHCOS in OpenShift 4.7. On a RHCOS 4.8 system (haven't checked 4.7 yet) ``` [core@cosa-devsh ~]$ rpm-ostree status State: idle Deployments: * ostree://c2432414d5020c3bf05c9a7836b91e90a258c9d0b25d8ea13ef1607694ec3f90 Version: 48.83.202103112135-0 (2021-03-11T21:38:14Z) [core@cosa-devsh ~]$ sudo semodule -l | grep unconfined unconfined unconfineduser ``` We only turn on a few SELinux booleans during RHCOS build time: ``` $ git grep semanage manifest.yaml:119: semanage boolean --modify --on container_use_cephfs # RHBZ#1694045 manifest.yaml:120: semanage boolean --modify --on virt_use_samba # RHBZ#1754825 ``` There should be no other modification to the SELinux policy that comes from RHEL's `selinux-policy` and `selinux-policy-targeted`
No difference on RHCOS 4.7 in terms of SELinux modules: ``` sh-4.4# rpm-ostree status State: idle Deployments: * pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b2c764f69eb4663efb2954e74d0c235b5edcb429fd9d66f151dc666be03f63c CustomOrigin: Managed by machine-config-operator Version: 47.83.202103051045-0 (2021-03-05T10:48:29Z) ostree://646a9832dd0dc9fe174a2fc005863a9582186518a5476522a0e9bdccc0e5252a Version: 47.83.202102090044-0 (2021-02-09T00:47:36Z) sh-4.4# semodule -l | grep unconfined unconfined unconfineduser ``` Using the YAML from comment #6 and Dan's input from comment #11, I can both reproduce the error and see that the transition is not allowed: ``` $ cat ping.yaml apiVersion: v1 kind: Pod metadata: name: ping-test spec: containers: - name: ping securityContext: privileged: true capabilities: add: ["NET_RAW"] image: registry.access.redhat.com/ubi8/ubi lifecycle: postStart: exec: command: - sh - -c - yum install iputils iproute -y command: ['sh', '-c', 'echo "Hello, world!" && sleep 3600'] $ oc create -f ping.yaml pod/ping-test created $ oc rsh ping-test sh-4.4# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: eth0@if48: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1410 qdisc noqueue state UP group default link/ether 0a:58:0a:83:00:2b brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.131.0.43/23 brd 10.131.1.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::ecf4:a7ff:fec2:f60d/64 scope link valid_lft forever preferred_lft forever sh-4.4# ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=114 time=1.61 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=114 time=0.927 ms ^C sh-4.4# ping 1.1.1.1 -I eth0 PING 1.1.1.1 (1.1.1.1) from 10.131.0.43 eth0: 56(84) bytes of data. 64 bytes from 1.1.1.1: icmp_seq=1 ttl=52 time=14.0 ms 64 bytes from 1.1.1.1: icmp_seq=2 ttl=52 time=13.4 ms 64 bytes from 1.1.1.1: icmp_seq=3 ttl=52 time=13.3 ms ^C --- 1.1.1.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 7ms rtt min/avg/max/mdev = 13.293/13.577/14.030/0.337 ms sh-4.4# ping 1.1.1.1 -I 10.131.0.43 bind: Permission denied sh-4.4# exit exit $ oc debug node/ci-ln-57jmm3k-f76d1-gbljv-worker-c-9xxf2 Starting pod/ci-ln-57jmm3k-f76d1-gbljv-worker-c-9xxf2-debug ... To use host binaries, run `chroot /host` Pod IP: 10.0.32.3 If you don't see a command prompt, try pressing enter. sh-4.4# chroot /host sh-4.4# ausearch -m AVC -ts recent ---- time->Tue Mar 16 18:13:57 2021 type=PROCTITLE msg=audit(1615918437.492:134): proctitle=70696E6700312E312E312E31002D490031302E3133312E302E3433 type=SYSCALL msg=audit(1615918437.492:134): arch=c000003e syscall=49 success=no exit=-13 a0=3 a1=5615e8494070 a2=10 a3=7fa8ebda39e0 items=0 ppid=132277 pid=133682 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=4294967295 comm="ping" exe="/usr/bin/ping" subj=system_u:system_r:spc_ t:s0 key=(null) type=AVC msg=audit(1615918437.492:134): avc: denied { node_bind } for pid=133682 comm="ping" saddr=10.131.0.43 scontext=system_u:system_r:spc_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=0 sh-4.4# ausearch -m AVC -ts recent > /tmp/t sh-4.4# audit2allow -i /tmp/t #============= spc_t ============== allow spc_t node_t:icmp_socket node_bind; sh-4.4# exit exit sh-4.4# exit exit Removing debug pod ... ``` @Dan what other info can I gather?
Dan Walsh, any thoughts on the selinux issue in this BZ?
#============= spc_t ============== #!!!! This avc is allowed in the current policy allow spc_t node_t:icmp_socket node_bind; storage (selinux) $ rpm -q container-selinux container-selinux-2.160.0-2.fc34.noarch So this is allowed in the current container-selinux package. Looks like RHCOS needs newer packages.
I am not an SELinux expert, so I may be looking for the wrong thing, but it wasn't obvious which rule in the `container-selinux` policy would apply here. I did find this BZ and PR which looks somewhat related, but the `scontext` in the error message was `unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023` rather than `system_u:system_r:spc_t:s0` https://bugzilla.redhat.com/show_bug.cgi?id=1848929 https://github.com/fedora-selinux/selinux-policy/pull/403/ Anyways, since Dan says this is in `container-selinux` I am going to send it over to the Containers team for evaluation. If the policy can be adjusted in RHEL8, the RHCOS team would like to consume any fixed builds which are produced. (Note, this BZ is reported against RHCOS 4.6, which is using RHEL 8.2 EUS...but since `container-selinux` is part of the `container-tools` module, we could theoretically consume a newer `container-selinux` in RHCOS outside of the module.)
Yes you need to consume a newer container-selinux.
container-selinux-2.162.0-1.module+el8.4.0+11012+a2ae5674 should be current in RHEL8.4 and should have the fix.
RHCOS 4.8 is currently using RHEL 8.4.0 content. Looking at the `container-tools` modules that were released with RHEL 8.4.0 shows that `container-selinux-2:2.158.0-1.module+el8.4.0+10607+f4da7515.noarch` was released in the `rhel8` and `3.0` streams. It appears `container-selinux-2:2.162.0-1.module+el8.4.0+11012+a2ae5674.noarch` is going to be released as part of the first 8.4.z release in the `rhel8` stream. So RHCOS/OCP would likely include this new version as part of the first 4.8.z release.
Jindrich, Could you please confirm (or deny) that `container-selinux-2:2.162.0-1.module+el8.4.0+11012+a2ae5674.noarch` will be part of RHEL 8.4.0.1 or 8.4.0.2 or somewhere else please?
Yes Tom, there will be at least container-selinux-2.162.0 in 8.4.0.1.
I would prefer to just install the older version of the software. I am wondering why this has not come up in the past.
I am fine with back porting all of container-selinux.
Are you sure this is being blocked by SELinux or is it being blocked because of a missing NET_RAW Capability/missing sysctl?
Are you seeing AVCs?
So this looks like the container-selinux never got updated.
Jindrich seems people want the current RHEL container-selinux back ported to older RHEL 8.2 and RHEL8.4 releases. Do we still support RHEL8.2? I believe this is no longer supported, so I am not sure we can back port to it.
Thanks, switching this bug to MODIFIED then.
OpenShift engineering has decided to NOT ship 4.8.6 on 8/23 due to the following issue. https://bugzilla.redhat.com/show_bug.cgi?id=1995785 All the fixes part will be now included in 4.8.7 on 8/30.
Yes and it is in the current container-selinux Cat the AVC to a file /tmp/t for example and then run # audit2allow -i /tmp/t #============= spc_t ============== #!!!! This avc is allowed in the current policy allow spc_t node_t:icmp_socket node_bind; This should tell you if the current SELinux policy allows the access.
Yes
Nope just an updated container-selinux package.
Dan, assuming this is fixed by container-selinux-2.170.0 which is about to land in RHAOS4.6+ soon?
The AVC has been fixed. I have no new information and no idea if IPv6 ping works or not. but if SELinux is blocking it with this fix, then I need to see the new AVC.
@jiandr
If I pipe that AVC through to my container-selinux policy in Fedora I see $ cat /tmp/t type=AVC msg=audit(1642536873.951:211): avc: denied { node_bind } for pid=17312 comm="ping6" saddr=e80::3c57:23ff:fe90:f3d0 scontext=system_u:system_r:container_t:s0:c355,c405 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=0 type=SYSCALL msg=audit(1642536873.951:211): arch=c000003e syscall=49 success=no exit=-13 a0=3 a1=55b23457b150 a2=1c a3=0 items=0 ppid=17163 pid=17312 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="ping6" exe="/usr/bin/ping" subj=system_u:system_r:container_t:s0:c355,c405 key=(null)ARCH=x86_64 SYSCALL=bind AUID="cloud-user" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root" type=PROCTITLE msg=audit(1642536873.951:211): proctitle=70696E6736002D49006538303A3A336335373A323366663A666539303A66336430006 66538303A3A373463643A383366663A666530363A63383330 $ audit2allow -i /tmp/t #============= container_t ============== #!!!! This avc is allowed in the current policy allow container_t node_t:icmp_socket node_bind; Slightly newer policy but no change for this fix recently.
Jindrich, can you add a requres for the version of selinux-policy or newer, since I think the issue, is that container-selinux needs a the latest selinux-policy when it is installed to work properly.
Great.
Assigning to Jindrich for any further packaging and bz needs.
It appears 4.8.24 included selinux-policy-3.14.3-67.el8_4.2 and 4.8.25 included an updated selinux-policy-3.14.3-67.el8_4.4 release. Checking the difference between these two selinux-policy releases: $ rhpkg -q clone selinux-policy && cd selinux-policy/ $ git log $(rhpkg gitbuildhash selinux-policy-3.14.3-67.el8_4.2)..$(rhpkg gitbuildhash selinux-policy-3.14.3-67.el8_4.4) commit 4fe1918ca7a57c485e8496f32c746d5ac6df3514 (origin/rhel-8.4.0) Author: Zdenek Pytela <zpytela> Date: Fri Dec 10 13:42:03 2021 +0100 * Fri Dec 10 2021 Zdenek Pytela <zpytela> - 3.14.3-67.4 - Allow unconfined_t to node_bind icmp_sockets in node_t domain Resolves: rhbz#2029802 commit b07c56f9d4210a009b19e7847fa6aa6541fcb44e Author: Zdenek Pytela <zpytela> Date: Wed Nov 10 11:26:17 2021 +0100 * Wed Nov 10 2021 Zdenek Pytela <zpytela> - 3.14.3-67.3 - Allow unconfined domains to bpf all other domains Resolves: rhbz#2015845 commit 4fe1918ca7a57c485e8496f32c746d5ac6df3514 seems to be related to https://github.com/fedora-selinux/selinux-policy/pull/403/ that Micah found earlier (comment#23). Despite Dan's comment#21 I don't think there's an OCP bug here. With the existing container-selinux-2.170.0-2.rhaos4.8 and updated selinux-policy-3.14.3-67.el8_4.4 it seems this issue is resolved.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.8.31 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:0484
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days