Bug 1519991 - The unidler fails when there is no network plugin
Summary: The unidler fails when there is no network plugin
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 3.9.0
Assignee: Ben Bennett
QA Contact: Hongan Li
URL:
Whiteboard:
Depends On: 1529473
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-12-01 21:05 UTC by Ben Bennett
Modified: 2018-12-28 03:05 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: There was a race in the startup code that relied on the node setup setting a field that the userspace proxy needed. When the network plugin was not used (or if it was fast) the userspace proxy setup ran sooner and would read a nil value for the ip address of the node. Then when the proxy was enabled (or the unidler which uses it) and tried to use the nil value, it would crash. Consequence: The userspace proxy would not work, and the unidler (which uses it) would fail, so would never unidle the service. Fix: Add a retry loop to wait for the value to be set. Result: The userspace proxy and unidler work as expected.
Clone Of:
Environment:
Last Closed: 2018-03-28 14:14:24 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Origin (Github) 17564 0 None None None 2017-12-01 21:10:51 UTC
Red Hat Product Errata RHBA-2018:0489 0 None None None 2018-03-28 14:14:59 UTC

Description Ben Bennett 2017-12-01 21:05:43 UTC
Description of problem:

When starting openshift with no network plugin, if you ever idle a pod, it will never unidle.


Version-Release number of selected component (if applicable):

3.7.0


How reproducible:

100%


Steps to Reproduce:
1. hack/dind-cluster.sh start -r -n none -N 1
2. create a dc for a hello-openshift pod
3. oc expose dc hello-openshift
4. oc idle hello-openshift
5. curl ...


Actual results:

Nothing happens when you curl.  If you look at the iptables rules on the node after step 4 you see that they are not correct... there are none pointing at the userspace proxy.

Expected results:

The curl should unidle the pod and it should return "Hello Openshift!"


Additional info:

It turns out that the problem is actually with the way the userspace proxy is set up.  It probably happens when that is enabled and idling is out of the picture.  The proxy setup races with the node registration, and the proxy tries to read the node record to retrieve the address, and fails if the registration has not happened yet, but proceeds despite the failure.

Comment 2 Hongan Li 2018-01-29 07:53:28 UTC
git clone latest origin build and verified in dind cluster openshift v3.9.0-alpha.3+4f709b4-198, the issue has been fixed.

the userspace proxy set up correctly after idling and pod can be unidled by curl as below:
[root@openshift-node-1 /]# iptables-save | grep unsecure
-A KUBE-PORTALS-CONTAINER -d 172.30.51.61/32 -p tcp -m comment --comment "default/service-unsecure:http" -m tcp --dport 27017 -j DNAT --to-destination 172.17.0.3:42459
-A KUBE-PORTALS-HOST -d 172.30.51.61/32 -p tcp -m comment --comment "default/service-unsecure:http" -m tcp --dport 27017 -j DNAT --to-destination 172.17.0.3:42459


[root@openshift-node-1 /]# curl 172.30.51.61:27017
Hello-OpenShift-1 http-8080

Comment 5 errata-xmlrpc 2018-03-28 14:14:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489


Note You need to log in before you can comment on or make changes to this bug.