Description of problem: When a NodePort type Service object is created in Openshift, kuryr-controller crashes. [openshift@master-0 ~]$ oc new-project test Now using project "test" on server "https://master-0.openshift.example.com:8443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git to build a new example application in Ruby. [openshift@master-0 ~]$ oc run --image=kuryr/demo demo deploymentconfig "demo" created [openshift@master-0 ~]$ oc get pod NAME READY STATUS RESTARTS AGE demo-1-v465w 1/1 Running 0 41s [openshift@master-0 ~]$ oc get pod --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE default router-1-jpgwz 1/1 Running 0 1h 192.168.99.6 infra-node-0.openshift.example.com kube-system master-api-master-0.openshift.example.com 1/1 Running 4 1h 192.168.99.14 master-0.openshift.example.com kube-system master-controllers-master-0.openshift.example.com 1/1 Running 0 1h 192.168.99.14 master-0.openshift.example.com kube-system master-etcd-master-0.openshift.example.com 1/1 Running 1 1h 192.168.99.14 master-0.openshift.example.com openshift-infra bootstrap-autoapprover-0 1/1 Running 0 1h 10.11.0.40 master-0.openshift.example.com openshift-infra kuryr-cni-ds-74n6j 1/1 Running 0 1h 192.168.99.14 master-0.openshift.example.com openshift-infra kuryr-cni-ds-jfm72 1/1 Running 0 1h 192.168.99.12 app-node-1.openshift.example.com openshift-infra kuryr-cni-ds-k565x 1/1 Running 0 1h 192.168.99.9 app-node-0.openshift.example.com openshift-infra kuryr-cni-ds-wfxmb 1/1 Running 0 1h 192.168.99.6 infra-node-0.openshift.example.com openshift-infra kuryr-controller-65c98f7444-nn48r 1/1 Running 0 1h 192.168.99.12 app-node-1.openshift.example.com openshift-node sync-dwd2r 1/1 Running 1 1h 192.168.99.9 app-node-0.openshift.example.com openshift-node sync-mkdrk 1/1 Running 1 1h 192.168.99.6 infra-node-0.openshift.example.com openshift-node sync-rlsz6 1/1 Running 1 1h 192.168.99.12 app-node-1.openshift.example.com openshift-node sync-tmlnv 1/1 Running 1 1h 192.168.99.14 master-0.openshift.example.com test demo-1-v465w 1/1 Running 0 2m 10.11.0.16 app-node-0.openshift.example.com [openshift@master-0 ~]$ oc get service No resources found. [openshift@master-0 ~]$ oc expose dc/demo --port 80 --target-port 8080 --type=NodePort service "demo" exposed [openshift@master-0 ~]$ oc get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE demo NodePort 172.30.5.144 <none> 80:31575/TCP 4s [openshift@master-0 ~]$ oc describe svc demo Name: demo Namespace: test Labels: run=demo Annotations: <none> Selector: run=demo Type: NodePort IP: 172.30.5.144 Port: <unset> 80/TCP TargetPort: 8080/TCP NodePort: <unset> 31575/TCP Endpoints: 10.11.0.16:8080 Session Affinity: None External Traffic Policy: Cluster Events: <none> [openshift@master-0 ~]$ oc-n openshift-infra get pod NAME READY STATUS RESTARTS AGE bootstrap-autoapprover-0 1/1 Running 0 2h kuryr-cni-ds-74n6j 1/1 Running 0 2h kuryr-cni-ds-jfm72 1/1 Running 0 2h kuryr-cni-ds-k565x 1/1 Running 0 2h kuryr-cni-ds-wfxmb 1/1 Running 0 2h kuryr-controller-65c98f7444-nn48r 0/1 CrashLoopBackOff 15 2h [openshift@master-0 ~]$ oci logs kuryr-controller-65c98f7444-nn48r 2018-06-01 12:48:12.991 1 INFO kuryr_kubernetes.config [-] Logging enabled! 2018-06-01 12:48:12.991 1 INFO kuryr_kubernetes.config [-] /usr/bin/kuryr-k8s-controller version 0.4.3 2018-06-01 12:48:13.806 1 INFO os_vif [-] Loaded VIF plugins: noop, ovs, linux_bridge 2018-06-01 12:48:13.860 1 INFO kuryr_kubernetes.controller.service [-] Service 'KuryrK8sService' stopped 2018-06-01 12:48:13.866 1 INFO kuryr_kubernetes.controller.service [-] Service 'KuryrK8sService' starting 2018-06-01 12:48:13.867 1 INFO kuryr_kubernetes.controller.managers.health [-] Starting health check server. 2018-06-01 12:48:13.868 1 INFO werkzeug [-] * Running on http://localhost:8082/ 2018-06-01 12:48:13.871 1 INFO kuryr_kubernetes.watcher [-] Started watching '/api/v1/pods' 2018-06-01 12:48:13.873 1 INFO kuryr_kubernetes.watcher [-] Started watching '/api/v1/services' 2018-06-01 12:48:13.875 1 INFO kuryr_kubernetes.watcher [-] Started watching '/api/v1/endpoints' 2018-06-01 12:48:14.339 1 INFO kuryr_kubernetes.controller.drivers.vif_pool [-] PORTS POOL: pools updated with pre-created ports 2018-06-01 12:48:14.406 1 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {u'object': {u'status': {u'loadBalancer': {}}, u'kind': u'Service', u'spec': {u'selector': {u'run': u'demo'}, u'extern alTrafficPolicy': u'Cluster', u'clusterIP': u'172.30.5.144', u'sessionAffinity': u'None', u'type': u'NodePort', u'ports': [{u'targetPort': 8080, u'protocol': u'TCP', u'port': 80, u'nodePort': 31575}]}, u'apiVers ion': u'v1', u'metadata': {u'name': u'demo', u'labels': {u'run': u'demo'}, u'namespace': u'test2', u'resourceVersion': u'11026', u'creationTimestamp': u'2018-06-01T12:16:48Z', u'selfLink': u'/api/v1/namespaces/t est2/services/demo', u'uid': u'a8321f35-6595-11e8-8811-fa163ede3cba'}}, u'type': u'ADDED'}: AttributeError: 'NoneType' object has no attribute 'ports' 2018-06-01 12:48:14.406 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last): 2018-06-01 12:48:14.406 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/logging.py", line 37, in __call__ 2018-06-01 12:48:14.406 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event) 2018-06-01 12:48:14.406 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/retry.py", line 55, in __call__ 2018-06-01 12:48:14.406 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event) 2018-06-01 12:48:14.406 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 67, in __call__ 2018-06-01 12:48:14.406 1 ERROR kuryr_kubernetes.handlers.logging self.on_present(obj) 2018-06-01 12:48:14.406 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 56, in on_present 2018-06-01 12:48:14.406 1 ERROR kuryr_kubernetes.handlers.logging if self._has_lbaas_spec_changes(service, lbaas_spec): 2018-06-01 12:48:14.406 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 116, in _has_lbaas_spec_changes 2018-06-01 12:48:14.406 1 ERROR kuryr_kubernetes.handlers.logging self._has_port_changes(service, lbaas_spec)) 2018-06-01 12:48:14.406 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/lbaas.py", line 131, in _has_port_changes 2018-06-01 12:48:14.406 1 ERROR kuryr_kubernetes.handlers.logging for port in lbaas_spec.ports} 2018-06-01 12:48:14.406 1 ERROR kuryr_kubernetes.handlers.logging AttributeError: 'NoneType' object has no attribute 'ports' 2018-06-01 12:48:14.406 1 ERROR kuryr_kubernetes.handlers.logging 2018-06-01 12:48:24.132 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified. 2018-06-01 12:48:24.133 1 INFO werkzeug [-] 192.168.99.12 - - [01/Jun/2018 12:48:24] "GET /ready HTTP/1.1" 200 - 2018-06-01 12:48:31.835 1 INFO werkzeug [-] 192.168.99.12 - - [01/Jun/2018 12:48:31] "GET /alive HTTP/1.1" 500 - 2018-06-01 12:48:34.206 1 INFO kuryr_kubernetes.controller.managers.health [-] Kuryr Controller readiness verified. 2018-06-01 12:48:34.207 1 INFO werkzeug [-] 192.168.99.12 - - [01/Jun/2018 12:48:34] "GET /ready HTTP/1.1" 200 - 2018-06-01 12:48:41.835 1 INFO werkzeug [-] 192.168.99.12 - - [01/Jun/2018 12:48:41] "GET /alive HTTP/1.1" 500 - 2018-06-01 12:48:42.141 1 INFO kuryr_kubernetes.controller.service [-] Service 'KuryrK8sService' stopping 2018-06-01 12:48:42.142 1 INFO kuryr_kubernetes.watcher [-] Stopped watching '/api/v1/pods' 2018-06-01 12:48:42.142 1 INFO kuryr_kubernetes.watcher [-] Stopped watching '/api/v1/services' 2018-06-01 12:48:42.143 1 INFO kuryr_kubernetes.watcher [-] Stopped watching '/api/v1/endpoints' 2018-06-01 12:48:42.143 1 INFO kuryr_kubernetes.controller.service [-] Service 'KuryrK8sService' stopped Version-Release number of selected component (if applicable): openstack-kuryr-kubernetes-common-0.4.3-1.el7ost.noarch openstack-kuryr-kubernetes-controller-0.4.3-1.el7ost.noarch How reproducible: always Steps to Reproduce: 1. Deploy OCP on OSP with kuryr 2. From master node: oc new-project test oc run --image=kuryr/demo demo oc expose dc/demo --port 80 --target-port 8080 --type=NodePort Actual results: kuryr-controller crashes and remains in CrashLoopBackOff status Expected results: kuryr-controller should not crash
Fixed upstream: https://review.openstack.org/#/c/571719/ And backported to Queens: https://review.openstack.org/#/c/572024/
This bug is marked for inclusion in the errata but does not currently contain draft documentation text. To ensure the timely release of this advisory please provide draft documentation text for this bug as soon as possible. If you do not think this bug requires errata documentation, set the requires_doc_text flag to "-". To add draft documentation text: * Select the documentation type from the "Doc Type" drop down field. * A template will be provided in the "Doc Text" field based on the "Doc Type" value selected. Enter draft text in the "Doc Text" field.
Verified on OSP 13 2018-08-16.1 puddle / openshift-ansible-3.10.34-1, and: openstack-kuryr-kubernetes-cni-0.4.3-2.el7ost.noarch openstack-kuryr-kubernetes-controller-0.4.3-2.el7ost.noarch Verification steps: 1. Deploy OCP on OSP with kuryr 2. From master node: $ oc new-project test $ oc run --image=kuryr/demo demo $ oc get pods NAME READY STATUS RESTARTS AGE demo-1-5pgdh 1/1 Running 0 1m $oc -n openshift-infra get pods NAME READY STATUS RESTARTS AGE bootstrap-autoapprover-0 1/1 Running 0 1h kuryr-cni-ds-7klr7 2/2 Running 0 1h kuryr-cni-ds-dfr9j 2/2 Running 0 1h kuryr-cni-ds-qm6tp 2/2 Running 0 1h kuryr-cni-ds-vmhrr 2/2 Running 0 1h kuryr-controller-59fc7f478b-cbqjf 1/1 Running 0 1h $ oc expose dc/demo --port 80 --target-port 8080 --type=NodePort $ oc get pods NAME READY STATUS RESTARTS AGE demo-1-5pgdh 1/1 Running 0 2m $ oc -n openshift-infra get pods NAME READY STATUS RESTARTS AGE bootstrap-autoapprover-0 1/1 Running 0 1h kuryr-cni-ds-7klr7 2/2 Running 0 1h kuryr-cni-ds-dfr9j 2/2 Running 0 1h kuryr-cni-ds-qm6tp 2/2 Running 0 1h kuryr-cni-ds-vmhrr 2/2 Running 0 1h kuryr-controller-59fc7f478b-cbqjf 1/1 Running 0 1h kuryr-controller pod does not crash/restart, it remains in Running status, and does not show any error message on its logs. 3. Delete the project: $ oc delete project test
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2573