Description of problem: Using the latest OpenShift 3.11.248 (as a part of Redhat Openshift Kubernetes Service on IBM Cloud, ROKS 3.11) we are seeing many cases where liveness and readiness probes are failing for pods. The probes that are failing at ones that try to contact a port that is listening on localhost of a hostNetwork pod (calico-node is one example). We have tracked down what we believe to be the cause, and it is a combination of: 1. New rule in iptables filter table, which was added to address CVE-2020-8558 that is dropping these liveness/readiness check packets. Here is the rule/chain on a node experiencing this problem (with many dropped packets): ``` Chain KUBE-FIREWALL (2 references) pkts bytes target prot opt in out source destination 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000 25377 2426K DROP all -- * * !127.0.0.0/8 127.0.0.0/8 /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT ``` And here is the rule on a node that is NOT experiencing this problem (no DROPs): ``` Chain KUBE-FIREWALL (2 references) pkts bytes target prot opt in out source destination 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000 0 0 DROP all -- * * !127.0.0.0/8 127.0.0.0/8 /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT ``` 2. On the RHEL nodes that are hitting this problem, when I run curl to try to connect to a liveness/readiness check port from the local node doing exactly what kubelet is doing, something like: `curl http://localhost:9099/` it will timeout, and in tcpdump I can see that the source IP of the packets is NOT a localhost IP (i.e. not in 127.0.0.0/8), instead it is for some reason using the private IP of the node, 10.138.184.112, and that the SYN packets are not getting a response because they are being dropped by the iptables rule shown above. ``` [root@kube-bskvlvjs041nuk5r1pb0-kubee2epvgx-default-00000144 ~]# tcpdump -lnei lo port 9099 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes 10:02:43.732740 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 74: 10.138.184.112.36788 > 127.0.0.1.9099: Flags [S], seq 3331120788, win 65535, options [mss 65495,sackOK,TS val 52407966 ecr 0,nop,wscale 9], length 0 10:02:44.733879 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 74: 10.138.184.112.36788 > 127.0.0.1.9099: Flags [S], seq 3331120788, win 65535, options [mss 65495,sackOK,TS val 52408968 ecr 0,nop,wscale 9], length 0 10:02:46.737949 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 74: 10.138.184.112.36788 > 127.0.0.1.9099: Flags [S], seq 3331120788, win 65535, options [mss 65495,sackOK,TS val 52410972 ecr 0,nop,wscale 9], length 0 10:02:50.741899 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 74: 10.138.184.112.36788 > 127.0.0.1.9099: Flags [S], seq 3331120788, win 65535, options [mss 65495,sackOK,TS val 52414976 ecr 0,nop,wscale 9], length 0 10:07:25.852721 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 74: 10.138.184.112.41694 > 127.0.0.1.9099: Flags [S], seq 3430041411, win 65535, options [mss 65495,sackOK,TS val 52690086 ecr 0,nop,wscale 9], length 0 ``` on the node that is working, tcpdump shows the source IP of this traffic as 127.0.0.1 (as expected), and that the traffic is therefore not being dropped: ``` [root@kube-bskvlvjs041nuk5r1pb0-kubee2epvgx-default-00000210 ~]# tcpdump -lnei lo port 9099 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes 10:05:02.824523 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 74: 127.0.0.1.19572 > 127.0.0.1.9099: Flags [S], seq 9456684, win 65535, options [mss 65495,sackOK,TS val 52541273 ecr 0,nop,wscale 9], length 0 10:05:02.824575 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 74: 127.0.0.1.9099 > 127.0.0.1.19572: Flags [S.], seq 2108984837, ack 9456685, win 65535, options [mss 65495,sackOK,TS val 52541274 ecr 52541273,nop,wscale 9], length 0 10:05:02.824598 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 66: 127.0.0.1.19572 > 127.0.0.1.9099: Flags [.], ack 1, win 256, options [nop,nop,TS val 52541274 ecr 52541274], length 0 10:05:02.824909 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 170: 127.0.0.1.19572 > 127.0.0.1.9099: Flags [P.], seq 1:105, ack 1, win 256, options [nop,nop,TS val 52541274 ecr 52541274], length 104 10:05:02.824934 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 66: 127.0.0.1.9099 > 127.0.0.1.19572: Flags [.], ack 105, win 256, options [nop,nop,TS val 52541274 ecr 52541274], length 0 10:05:02.825221 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 130: 127.0.0.1.9099 > 127.0.0.1.19572: Flags [P.], seq 1:65, ack 105, win 256, options [nop,nop,TS val 52541274 ecr 52541274], length 64 10:05:02.825242 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 66: 127.0.0.1.19572 > 127.0.0.1.9099: Flags [.], ack 65, win 256, options [nop,nop,TS val 52541274 ecr 52541274], length 0 10:05:02.826434 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 66: 127.0.0.1.19572 > 127.0.0.1.9099: Flags [F.], seq 105, ack 65, win 256, options [nop,nop,TS val 52541275 ecr 52541274], length 0 10:05:02.826556 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 66: 127.0.0.1.9099 > 127.0.0.1.19572: Flags [F.], seq 65, ack 106, win 256, options [nop,nop,TS val 52541276 ecr 52541275], length 0 10:05:02.826573 00:00:00:00:00:00 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 66: 127.0.0.1.19572 > 127.0.0.1.9099: Flags [.], ack 66, win 256, options [nop,nop,TS val 52541276 ecr 52541276], length 0 ``` My guess is that this source IP inconsistency/bug has been around for a while, but wasn't a problem with Openshift until this security fix. Version-Release number of selected component (if applicable): OpenShift 3.11.248 How reproducible: Not exactly sure. We see it in our regression testing in about 25% of the nodes we deploy. Steps to Reproduce: 1. Deploy latest Openshift 3.11 with calico-node pod (or some other pod with a liveness probe that curls a hostNetwork localhost port) 2. Keep adding worker nodes until one of them has the pod keep failing the liveness probe 3. Actual results: On certain nodes, source IP not being in 127.0.0.0/8 causes liveness probe packets to be dropped Expected results: Liveness probes to localhost ports from kubelet should succeed (or at least not be blocked by k8s iptables rule listed above) Additional info:
This was fixed in origin by https://github.com/openshift/origin/pull/25141/files for the bug https://bugzilla.redhat.com/show_bug.cgi?id=1849175. But that's just a backport of Casey's https://github.com/kubernetes/kubernetes/pull/91569 PR.
Just to clarify, I don't believe this bug has been fixed yet. The PRs mentioned above solve CVE-2020-8558, but I believe they CAUSE the case I detailed above. This bug is about how the fix for CVE-2020-8558 seems to have blocked liveness/readiness probes that use network calls to localhost from the local kubelet.
I was able to narrow this down even further. It appears like this problem appears only on nodes that have had a HostPort pod run on them. It appears running a HostPort pod on a node causes the following iptables MASQUERADE rule to be added to the node at the end of the POSTROUTING chain in the nat table: ``` Chain POSTROUTING (policy ACCEPT 55 packets, 3828 bytes) pkts bytes target prot opt in out source destination 18772 1304K cali-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:O3lYWMrLQYEMJtB5 */ 17249 1218K KUBE-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes postrouting rules */ 0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0 975 107K MASQUERADE all -- * lo 127.0.0.0/8 0.0.0.0/0 /* SNAT for localhost access to hostports */ ``` This `/* SNAT for localhost access to hostports */` rule is what is causing the behavior I described initially in this ticket. When I remove this rule manually, the behavior goes back to normal and the liveness probe succeeds. I think it is something in the Openshift components that adds this rule and implements HostPorts in general, but I'm not sure about that. I've looked at the "portmap" CNI plugin which is used to implement HostPort on base k8s clusters, and they both use more specific iptables rules so that only traffic bound for a HostPort is MASQ'd. So I think something similar would fix this issue.
Brad, Thanks for the excellent analysis. I think you're exactly right.
*** Bug 1866132 has been marked as a duplicate of this bug. ***
Update: we need to backport https://github.com/kubernetes/kubernetes/pull/80591 to CRI-O.
For bureaucratic reasons, marking this as a duplicate of bug 1866132. Your analysis was spot-on, and it turns out we already have a fix that needs to be backported. *** This bug has been marked as a duplicate of bug 1866132 ***