Bug 1781198 - Pod is stuck in container creating after installation
Summary: Pod is stuck in container creating after installation
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: 4.4.0
Assignee: Luis Tomas Bolivar
QA Contact: Jon Uriarte
URL:
Whiteboard:
Depends On:
Blocks: 1784469
TreeView+ depends on / blocked
 
Reported: 2019-12-09 14:21 UTC by Itzik Brown
Modified: 2020-05-13 21:54 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1784469 (view as bug list)
Environment:
Last Closed: 2020-05-13 21:54:18 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift kuryr-kubernetes pull 113 None closed Bug 1781198: Ensure namespace are recreated upon a k8s client failure 2020-10-20 18:17:58 UTC
OpenStack gerrit 699201 None MERGED Ensure namespace are recreated upon a k8s client failure 2020-04-27 08:44:24 UTC
Red Hat Product Errata RHBA-2020:0581 None None None 2020-05-13 21:54:22 UTC

Description Itzik Brown 2019-12-09 14:21:22 UTC
Description of problem:
After Openshift installation is done a pod that is created is stuck in ContainerCreating status.

Kuryr-controller log:
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {'type': 'ADDED', 'object': {'kind': 'Pod', 'apiVersion': 'v1', 'metadata': {'name': 'hello-openshift', 'namespace': 'default', 'selfLink': '/api/v1/namespaces/default/pods/hello-openshift', 'uid': '823208a3-c160-4f95-81a8-62585c25319d', 'resourceVersion': '90953', 'creationTimestamp': '2019-12-09T08:09:56Z', 'labels': {'name': 'hello-openshift'}, 'annotations': {'k8s.v1.cni.cncf.io/networks-status': ''}}, 'spec': {'volumes': [{'name': 'tmp', 'emptyDir': {}}, {'name': 'default-token-mhnnn', 'secret': {'secretName': 'default-token-mhnnn', 'defaultMode': 420}}], 'containers': [{'name': 'hello-openshift', 'image': 'openshift/hello-openshift', 'ports': [{'containerPort': 8080, 'protocol': 'TCP'}], 'resources': {}, 'volumeMounts': [{'name': 'tmp', 'mountPath': '/tmp'}, {'name': 'default-token-mhnnn', 'readOnly': True, 'mountPath': '/var/run/secrets/kubernetes.io/serviceaccount'}], 'terminationMessagePath': '/dev/termination-log', 'terminationMessagePolicy': 'File', 'imagePullPolicy': 'IfNotPresent', 'securityContext': {'capabilities': {}, 'privileged': False}}], 'restartPolicy': 'Always', 'terminationGracePeriodSeconds': 30, 'dnsPolicy': 'ClusterFirst', 'serviceAccountName': 'default', 'serviceAccount': 'default', 'nodeName': 'ostest-tw7wb-worker-pxpjq', 'securityContext': {}, 'imagePullSecrets': [{'name': 'default-dockercfg-nt77f'}], 'schedulerName': 'default-scheduler', 'tolerations': [{'key': 'node.kubernetes.io/not-ready', 'operator': 'Exists', 'effect': 'NoExecute', 'tolerationSeconds': 300}, {'key': 'node.kubernetes.io/unreachable', 'operator': 'Exists', 'effect': 'NoExecute', 'tolerationSeconds': 300}], 'priority': 0, 'dnsConfig': {'options': [{'name': 'use-vc', 'value': ''}]}, 'enableServiceLinks': True}, 'status': {'phase': 'Pending', 'conditions': [{'type': 'Initialized', 'status': 'True', 'lastProbeTime': None, 'lastTransitionTime': '2019-12-09T08:09:57Z'}, {'type': 'Ready', 'status': 'False', 'lastProbeTime': None, 'lastTransitionTime': '2019-12-09T08:09:57Z', 'reason': 'ContainersNotReady', 'message': 'containers with unready status: [hello-openshift]'}, {'type': 'ContainersReady', 'status': 'False', 'lastProbeTime': None, 'lastTransitionTime': '2019-12-09T08:09:57Z', 'reason': 'ContainersNotReady', 'message': 'containers with unready status: [hello-openshift]'}, {'type': 'PodScheduled', 'status': 'True', 'lastProbeTime': None, 'lastTransitionTime': '2019-12-09T08:09:56Z'}], 'hostIP': '10.196.0.15', 'startTime': '2019-12-09T08:09:57Z', 'containerStatuses': [{'name': 'hello-openshift', 'state': {'waiting': {'reason': 'ContainerCreating'}}, 'lastState': {}, 'ready': False, 'restartCount': 0, 'image': 'openshift/hello-openshift', 'imageID': '', 'started': False}], 'qosClass': 'BestEffort'}}}: openstack.exceptions.ResourceNotFound: No Subnet found for 902e690f-c0b1-4ed4-b022-c9e5d0d1c64d: Client Error for url: http://10.46.22.208:9696/v2.0/subnets/902e690f-c0b1-4ed4-b022-c9e5d0d1c64d, Subnet 902e690f-c0b1-4ed4-b022-c9e5d0d1c64d could not be found.
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/logging.py", line 37, in __call__
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging     self._handler(event)
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/retry.py", line 78, in __call__
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging     self._handler(event)
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 75, in __call__
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging     self.on_present(obj)
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/handlers/vif.py", line 96, in on_present
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging     subnets = self._drv_subnets.get_subnets(pod, project_id)
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/namespace_subnet.py", line 47, in get_subnets
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging     return self.get_namespace_subnet(pod_namespace)
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/namespace_subnet.py", line 52, in get_namespace_subnet
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging     return {subnet_id: utils.get_subnet(subnet_id)}
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 1270, in decorate
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging     should_cache_fn)
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 864, in get_or_create
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging     async_creator) as value:
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 186, in __enter__
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging     return self._enter()
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 93, in _enter
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging     generated = self._enter_create(value, createdtime)
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/dogpile/lock.py", line 179, in _enter_create
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging     return self.creator()
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 831, in gen_value
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging     created_value = creator()
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 1266, in creator
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging     return fn(*arg, **kw)
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/utils.py", line 183, in get_subnet
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging     n_subnet = os_net.get_subnet(subnet_id)
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/openstack/network/v2/_proxy.py", line 3432, in get_subnet
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging     return self._get(_subnet.Subnet, subnet)
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/openstack/proxy.py", line 46, in check
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging     return method(self, expected, actual, *args, **kwargs)
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/openstack/proxy.py", line 447, in _get
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging     resource_type=resource_type.__name__, value=value))
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/openstack/resource.py", line 1318, in fetch
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging     self._translate_response(response, **kwargs)
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/openstack/resource.py", line 1107, in _translate_response
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging     exceptions.raise_from_response(response, error_message=error_message)
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging   File "/usr/lib/python3.6/site-packages/openstack/exceptions.py", line 229, in raise_from_response
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging     http_status=http_status, request_id=request_id
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging openstack.exceptions.ResourceNotFound: No Subnet found for 902e690f-c0b1-4ed4-b022-c9e5d0d1c64d: Client Error for url: http://10.46.22.208:9696/v2.0/subnets/902e690f-c0b1-4ed4-b022-c9e5d0d1c64d, Subnet 902e690f-c0b1-4ed4-b022-c9e5d0d1c64d could not be found.
2019-12-09 12:25:39.581 1 ERROR kuryr_kubernetes.handlers.logging 
2019-12-09 12:26:00.180 1 INFO kuryr_kubernetes.controller.drivers.vif_pool [-] PORTS POOL: pools updated with pre-created ports


Version-Release number of selected component (if applicable):
OSP13 OVN/DVR
4.3.0-0.nightly-2019-12-09-035405

How reproducible:


Steps to Reproduce:
1. As described 
2.
3.

Actual results:


Expected results:


Additional info:

Comment 5 Jon Uriarte 2019-12-24 10:03:31 UTC
Verified in 4.4.0-0.nightly-2019-12-20-210709 on 2019-12-13.1 OSP 13 puddle.

The OCP installer finishes successfully:

 $ oc get clusterversion
 NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
 version   4.4.0-0.nightly-2019-12-20-210709   True        False         14h     Cluster version is 4.4.0-0.nightly-2019-12-20-210709

New pods can be created successfully:

 $ oc create -f https://raw.githubusercontent.com/openshift/origin/master/examples/hello-openshift/hello-pod.json
  pod/hello-openshift created
 
 $ oc get pods
 NAME              READY   STATUS    RESTARTS   AGE
 hello-openshift   1/1     Running   0          2m

 $ oc delete -f https://raw.githubusercontent.com/openshift/origin/master/examples/hello-openshift/hello-pod.json
 pod "hello-openshift" deleted

 $ oc get pods
 No resources found in default namespace.

Comment 7 errata-xmlrpc 2020-05-13 21:54:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.