Red Hat Bugzilla – Bug 1378000
[RHEL73] The Pod with QoS setting cannot reach outside network on RHEL7.3
Last modified: 2016-10-27 11:42:23 EDT
Description of problem: The Pod with QoS setting cannot reach outside network on RHEL7.3 Version-Release number of selected component (if applicable): oc v3.3.0.31 kubernetes v1.3.0+52492b4 ovs-vsctl (Open vSwitch) 2.4.0 Compiled Mar 22 2016 08:42:47 DB Schema 7.12.1 Red Hat Enterprise Linux Server release 7.3 Beta (Maipo) How reproducible: always Steps to Reproduce: 1. create iperf pod with QoS setting (see Additional info for iperf.yaml file) 2. oc rsh iperf 3. ifconfig eth0 and ping 10.1.0.1 Actual results: / # ifconfig eth0 eth0 Link encap:Ethernet HWaddr 02:42:0A:01:00:05 inet addr:10.1.0.5 Bcast:0.0.0.0 Mask:255.255.255.0 inet6 addr: fe80::42:aff:fe01:5/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1410 Metric:1 RX packets:2 errors:0 dropped:0 overruns:0 frame:0 TX packets:26 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:180 (180.0 B) TX bytes:1404 (1.3 KiB) / # / # ping 10.1.0.1 PING 10.1.0.1 (10.1.0.1): 56 data bytes ^C --- 10.1.0.1 ping statistics --- 7 packets transmitted, 0 packets received, 100% packet loss Expected results: The Pod with QoS setting can reach outside network Additional info: 1. no issue found if create pod without QoS setting 2. iperf.yaml apiVersion: v1 kind: Pod metadata: name: iperf annotations: kubernetes.io/ingress-bandwidth: 3M kubernetes.io/egress-bandwidth: 2M spec: containers: - name: iperf image: yadu/hello-openshift-iperf
The OpenShift code hasn't changed since this feature was added and it worked at that time, so confirming that it worked on 7.2 would help narrow down the cause. Can we confirm whether or not this worked on RHEL 7.2?
It works fine on RHEL 7.2.
Root caused the problem, and it's a result of kernel changes due to https://bugzilla.redhat.com/show_bug.cgi?id=1152231. Pushed an origin PR to fix: https://github.com/openshift/origin/pull/11126
@dcbw please make sure this is also patched in ose-3.3. Not just origin/master. Thank you Dan!
(In reply to hongli from comment #0) > / # ifconfig eth0 > eth0 Link encap:Ethernet HWaddr 02:42:0A:01:00:05 > inet addr:10.1.0.5 Bcast:0.0.0.0 Mask:255.255.255.0 > inet6 addr: fe80::42:aff:fe01:5/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1410 Metric:1 > RX packets:2 errors:0 dropped:0 overruns:0 frame:0 > TX packets:26 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 ^^^^^^^^^^^^ > RX bytes:180 (180.0 B) TX bytes:1404 (1.3 KiB) (Thanks for providing ifconfig output) The important info from ifconfig is the txqueuelen==0. I guess the eth0 part of a veth pair. It is not clear what kernel qdisc is getting used?
The short answer is, userspace MUST set tx_queue_len on an interface with QoS/qdisc, thus DCBW's commit/fix is the right solution: https://github.com/openshift/origin/pull/11126/commits/8a2fbcf4fd7530d79e2 I recommend to use that fix, to make openshift set a queue length. It have *always* been a misconfiguration to add a qdisc to an interface with txqueuelen==0 (ifconfig syntax). Some qdisc's (htb, fifo, gred, plug, sfb) the kernel had workarounds setting txqueuelen=1 (2 for HTB). These workarounds were actually quite bad, as things seems to work, but work poorly, because a queue of 1 packet is not sufficient. I'm actually happy that we backported the fix which removed[1] these workarounds because it exposed a problem like this, instead of having a semi-working solution. Thus, setting queue len is actually a fix for OpenShift regardless of the kernel used.
According to the PR: https://github.com/openshift/origin/pull/11126 I changed the openshift-sdn-ovs manually, and the Pod with QoS settings works well now.
test env for 3.3.1 is not ready yet, will verify it ASAP if env ready.
verified in openshift 3.3.1.1 with RHEL 7.3 Beta, the bug has been fixed
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:2084