Bug 1781198
| Summary: | Pod is stuck in container creating after installation | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Itzik Brown <itbrown> | |
| Component: | Networking | Assignee: | Luis Tomas Bolivar <ltomasbo> | |
| Networking sub component: | kuryr | QA Contact: | Jon Uriarte <juriarte> | |
| Status: | CLOSED ERRATA | Docs Contact: | ||
| Severity: | high | |||
| Priority: | medium | CC: | bbennett, juriarte, ltomasbo, nagrawal, oblaut | |
| Version: | 4.3.0 | Keywords: | Triaged | |
| Target Milestone: | --- | |||
| Target Release: | 4.4.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1784469 (view as bug list) | Environment: | ||
| Last Closed: | 2020-05-13 21:54:18 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1784469 | |||
Verified in 4.4.0-0.nightly-2019-12-20-210709 on 2019-12-13.1 OSP 13 puddle. The OCP installer finishes successfully: $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.4.0-0.nightly-2019-12-20-210709 True False 14h Cluster version is 4.4.0-0.nightly-2019-12-20-210709 New pods can be created successfully: $ oc create -f https://raw.githubusercontent.com/openshift/origin/master/examples/hello-openshift/hello-pod.json pod/hello-openshift created $ oc get pods NAME READY STATUS RESTARTS AGE hello-openshift 1/1 Running 0 2m $ oc delete -f https://raw.githubusercontent.com/openshift/origin/master/examples/hello-openshift/hello-pod.json pod "hello-openshift" deleted $ oc get pods No resources found in default namespace. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581 |
Description of problem: After Openshift installation is done a pod that is created is stuck in ContainerCreating status. Kuryr-controller log: 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {'type': 'ADDED', 'object': {'kind': 'Pod', 'apiVersion': 'v1', 'metadata': {'name': 'hello-openshift', 'namespace': 'default', 'selfLink': '/api/v1/namespaces/default/pods/hello-openshift', 'uid': '823208a3-c160-4f95-81a8-62585c25319d', 'resourceVersion': '90953', 'creationTimestamp': '2019-12-09T08:09:56Z', 'labels': {'name': 'hello-openshift'}, 'annotations': {'k8s.v1.cni.cncf.io/networks-status': ''}}, 'spec': {'volumes': [{'name': 'tmp', 'emptyDir': {}}, {'name': 'default-token-mhnnn', 'secret': {'secretName': 'default-token-mhnnn', 'defaultMode': 420}}], 'containers': [{'name': 'hello-openshift', 'image': 'openshift/hello-openshift', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'resources': {}, 'volumeMounts': [{'name': 'tmp', 'mountPath': '/tmp'}, {'name': 'default-token-mhnnn', 'readOnly': True, 'mountPath': '/var/run/secrets/kubernetes.io/serviceaccount'}], 'terminationMessagePath': '/dev/termination-log', 'terminationMessagePolicy': 'File', 'imagePullPolicy': 'IfNotPresent', 'securityContext': {'capabilities': {}, 'privileged': False}}], 'restartPolicy': 'Always', 'terminationGracePeriodSeconds': 30, 'dnsPolicy': 'ClusterFirst', 'serviceAccountName': 'default', 'serviceAccount': 'default', 'nodeName': 'ostest-tw7wb-worker-pxpjq', 'securityContext': {}, 'imagePullSecrets': [{'name': 'default-dockercfg-nt77f'}], 'schedulerName': 'default-scheduler', 'tolerations': [{'key': 'node.kubernetes.io/not-ready', 'operator': 'Exists', 'effect': 'NoExecute', 'tolerationSeconds': 300}, {'key': 'node.kubernetes.io/unreachable', 'operator': 'Exists', 'effect': 'NoExecute', 'tolerationSeconds': 300}], 'priority': 0, 'dnsConfig': {'options': [{'name': 'use-vc', 'value': ''}]}, 'enableServiceLinks': True}, 'status': {'phase': 'Pending', 'conditions': [{'type': 'Initialized', 'status': 'True', 'lastProbeTime': None, 'lastTransitionTime': '2019-12-09T08:09:57Z'}, {'type': 'Ready', 'status': 'False', 'lastProbeTime': None, 'lastTransitionTime': '2019-12-09T08:09:57Z', 'reason': 'ContainersNotReady', 'message': 'containers with unready status: [hello-openshift]'}, {'type': 'ContainersReady', 'status': 'False', 'lastProbeTime': None, 'lastTransitionTime': '2019-12-09T08:09:57Z', 'reason': 'ContainersNotReady', 'message': 'containers with unready status: [hello-openshift]'}, {'type': 'PodScheduled', 'status': 'True', 'lastProbeTime': None, 'lastTransitionTime': '2019-12-09T08:09:56Z'}], 'hostIP': '10.196.0.15', 'startTime': '2019-12-09T08:09:57Z', 'containerStatuses': [{'name': 'hello-openshift', 'state': {'waiting': {'reason': 'ContainerCreating'}}, 'lastState': {}, 'ready': False, 'restartCount': 0, 'image': 'openshift/hello-openshift', 'imageID': '', 'started': False}], 'qosClass': 'BestEffort'}}}: openstack.exceptions.ResourceNotFound: No Subnet found for 902e690f-c0b1-4ed4-b022-c9e5d0d1c64d: Client Error for url: http://10.46.22.208:9696/v2.0/subnets/902e690f-c0b1-4ed4-b022-c9e5d0d1c64d, Subnet 902e690f-c0b1-4ed4-b022-c9e5d0d1c64d could not be found. 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last): 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/logging.py", line 37, in __call__ 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event) 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/retry.py", line 78, in __call__ 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event) 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 75, in __call__ 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging self.on_present(obj) 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/handlers/vif.py", line 96, in on_present 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging subnets = self._drv_subnets.get_subnets(pod, project_id) 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/namespace_subnet.py", line 47, in get_subnets 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging return self.get_namespace_subnet(pod_namespace) 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/namespace_subnet.py", line 52, in get_namespace_subnet 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging return {subnet_id: utils.get_subnet(subnet_id)} 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 1270, in decorate 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging should_cache_fn) 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 864, in get_or_create 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging async_creator) as value: 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 186, in __enter__ 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging return self._enter() 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 93, in _enter 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging generated = self._enter_create(value, createdtime) 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 179, in _enter_create 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging return self.creator() 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 831, in gen_value 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging created_value = creator() 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 1266, in creator 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging return fn(*arg, **kw) 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/utils.py", line 183, in get_subnet 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging n_subnet = os_net.get_subnet(subnet_id) 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/openstack/network/v2/_proxy.py", line 3432, in get_subnet 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging return self._get(_subnet.Subnet, subnet) 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/openstack/proxy.py", line 46, in check 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging return method(self, expected, actual, *args, **kwargs) 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/openstack/proxy.py", line 447, in _get 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging resource_type=resource_type.__name__, value=value)) 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/openstack/resource.py", line 1318, in fetch 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging self._translate_response(response, **kwargs) 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/openstack/resource.py", line 1107, in _translate_response 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging exceptions.raise_from_response(response, error_message=error_message) 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/openstack/exceptions.py", line 229, in raise_from_response 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging http_status=http_status, request_id=request_id 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging openstack.exceptions.ResourceNotFound: No Subnet found for 902e690f-c0b1-4ed4-b022-c9e5d0d1c64d: Client Error for url: http://10.46.22.208:9696/v2.0/subnets/902e690f-c0b1-4ed4-b022-c9e5d0d1c64d, Subnet 902e690f-c0b1-4ed4-b022-c9e5d0d1c64d could not be found. 2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging 2019-12-09 12:26:00.180 1 INFO kuryr_kubernetes.controller.drivers.vif_pool [-] PORTS POOL: pools updated with pre-created ports Version-Release number of selected component (if applicable): OSP13 OVN/DVR 4.3.0-0.nightly-2019-12-09-035405 How reproducible: Steps to Reproduce: 1. As described 2. 3. Actual results: Expected results: Additional info: