Description of problem: Upgrade failed at TASK [openshift_node : Wait for node to be ready] Version-Release number of the following components: openshift-ansible-3.9.0-0.36.0.git.0.da68f13.el7.noarch ansible-2.4.2.0-2.el7.noarch # ansible --version ansible 2.4.2.0 config file = /etc/ansible/ansible.cfg configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible python version = 2.7.5 (default, May 3 2017, 07:55:04) [GCC 4.8.5 20150623 (Red Hat 4.8.5-14)] How reproducible: Steps to Reproduce: 1. setup containerised OCP v3.7 cluster on RHEL 2. upgrade containerized OCP on RHEL from v3.7.27 to v3.9.0-0.36.0 # ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade_control_plane.yml Actual results: TASK [openshift_node : Wait for node to be ready] task path: /usr/share/ansible/openshift-ansible/roles/openshift_node/tasks/upgrade.yml:39 Using module file /usr/share/ansible/openshift-ansible/roles/lib_openshift/library/oc_obj.py <***.redhat.com> ESTABLISH SSH CONNECTION FOR USER: root fatal: [***.redhat.com -> ***.redhat.com]: FAILED! => { "attempts": 24, "changed": false, "invocation": { "module_args": { "all_namespaces": null, "content": null, "debug": false, "delete_after": false, "files": null, "force": false, "kind": "node", "kubeconfig": "/etc/origin/master/admin.kubeconfig", "name": "172.16.120.12", "namespace": "default", "selector": null, "state": "list" } }, "results": { "cmd": "/usr/local/bin/oc get node 172.16.120.12 -o json -n default", "results": [ { "apiVersion": "v1", "kind": "Node", "metadata": { "annotations": { "volumes.kubernetes.io/controller-managed-attach-detach": "true" }, "creationTimestamp": "2018-02-02T14:35:53Z", "labels": { "beta.kubernetes.io/arch": "amd64", "beta.kubernetes.io/os": "linux", "failure-domain.beta.kubernetes.io/region": "regionOne", "failure-domain.beta.kubernetes.io/zone": "nova", "kubernetes.io/hostname": "172.16.120.12", "openshift-infra": "apiserver", "role": "node" }, "name": "172.16.120.12", "resourceVersion": "172361", "selfLink": "/api/v1/nodes/172.16.120.12", "uid": "5f44264b-0826-11e8-b1cd-fa163ecaae05" }, "spec": { "externalID": "890823be-6c70-4f28-8815-1bfb6ef19ceb", "providerID": "openstack:///890823be-6c70-4f28-8815-1bfb6ef19ceb", "unschedulable": true }, "status": { "addresses": [ { "address": "172.16.120.12", "type": "InternalIP" }, { "address": "10.8.244.76", "type": "ExternalIP" }, { "address": "172.16.120.12", "type": "Hostname" } ], "allocatable": { "cpu": "4", "memory": "7907784Ki", "pods": "250" }, "capacity": { "cpu": "4", "memory": "8010184Ki", "pods": "250" }, "conditions": [ { "lastHeartbeatTime": "2018-02-03T02:53:43Z", "lastTransitionTime": "2018-02-02T14:35:53Z", "message": "kubelet has sufficient disk space available", "reason": "KubeletHasSufficientDisk", "status": "False", "type": "OutOfDisk" }, { "lastHeartbeatTime": "2018-02-03T02:53:43Z", "lastTransitionTime": "2018-02-03T02:54:24Z", "message": "Kubelet stopped posting node status.", "reason": "NodeStatusUnknown", "status": "Unknown", "type": "MemoryPressure" }, { "lastHeartbeatTime": "2018-02-03T02:53:43Z", "lastTransitionTime": "2018-02-03T02:54:24Z", "message": "Kubelet stopped posting node status.", "reason": "NodeStatusUnknown", "status": "Unknown", "type": "DiskPressure" }, { "lastHeartbeatTime": "2018-02-03T02:53:43Z", "lastTransitionTime": "2018-02-03T02:54:24Z", "message": "Kubelet stopped posting node status.", "reason": "NodeStatusUnknown", "status": "Unknown", "type": "Ready" } ], "daemonEndpoints": { "kubeletEndpoint": { "Port": 10250 } }, "images": [ { "names": [ "registry.reg-aws.openshift.com:443/openshift3/openvswitch@sha256:df58cbaa300e2f7ae9a3dd2fc81446b46123bbc1ed8808ba46e1300808a37b8d", "registry.reg-aws.openshift.com:443/openshift3/openvswitch:v3.7.27" ], "sizeBytes": 1305163439 }, { "names": [ "registry.reg-aws.openshift.com:443/openshift3/node@sha256:4f564ab5cba3954f7f51c414f05b002f8e826bb181b4bf6f0629e0a4e0c883ba", "registry.reg-aws.openshift.com:443/openshift3/node:v3.7.27" ], "sizeBytes": 1303475182 }, { "names": [ "registry.reg-aws.openshift.com:443/openshift3/ose@sha256:8c0438e6ab79560743f7b2d901a5f45102ada1ee581793a9cb6deef9316352ad", "registry.reg-aws.openshift.com:443/openshift3/ose:v3.9.0" ], "sizeBytes": 1251727128 }, { "names": [ "registry.reg-aws.openshift.com:443/openshift3/ose@sha256:6994509c83bb74144dcfa1173f5603eea1b5a749f205b63501e4b2057b941370", "registry.reg-aws.openshift.com:443/openshift3/ose:v3.7.27" ], "sizeBytes": 1088450502 }, { "names": [ "registry.reg-aws.openshift.com:443/openshift3/ose@sha256:5ae218f6cb9989288d43c7c0d263b1dfa2ab192a3864d68e7c38514b1bc21b83", "registry.reg-aws.openshift.com:443/openshift3/ose:v3.7" ], "sizeBytes": 1088442337 }, { "names": [ "registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog@sha256:084db4e96739d66dd29a32d84a466ea8255b1ae3d74b0f439cb59227c34bef27", "registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog:v3.7" ], "sizeBytes": 268799200 }, { "names": [ "registry.access.redhat.com/rhel7/etcd@sha256:173ef5745a2b1e4d641ad2cbd351c01817a227e95957c890c55243069b3af42e", "registry.access.redhat.com/rhel7/etcd:latest" ], "sizeBytes": 250073248 }, { "names": [ "registry.reg-aws.openshift.com:443/openshift3/ose-pod@sha256:4fe4b799c1cba759188a1a4335b65df42e617181c891d1b3d04ea6cc4ff25a72", "registry.reg-aws.openshift.com:443/openshift3/ose-pod:v3.7.27" ], "sizeBytes": 208859057 } ], "nodeInfo": { "architecture": "amd64", "bootID": "280b740e-8790-455d-9c1d-a0164d9e17df", "containerRuntimeVersion": "docker://1.12.6", "kernelVersion": "3.10.0-685.el7.x86_64", "kubeProxyVersion": "v1.7.6+a08f5eeb62", "kubeletVersion": "v1.7.6+a08f5eeb62", "machineID": "a8bca9dc431c40ecbe4e679d318363fc", "operatingSystem": "linux", "osImage": "Red Hat Enterprise Linux Server 7.4 (Maipo)", "systemUUID": "890823BE-6C70-4F28-8815-1BFB6EF19CEB" } } } ], "returncode": 0 }, "state": "list" } NO MORE HOSTS LEFT ************************************************************************************************************************************************************************************************ NO MORE HOSTS LEFT ************************************************************************************************************************************************************************************************ to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade_control_plane.retry Failure summary: Hosts: xxx.redhat.com Play: Drain and upgrade master nodes Task: Wait for node to be ready Message: Failed without returning a message. Expected results: Upgrade succeeds
Env info before upgrade "atomic-openshift version: v3.7.27", "Operation System: Red Hat Enterprise Linux Server release 7.4 Beta (Maipo)", "Cluster Install Method: docker container", "Docker Version: docker-1.12.6-71.git3e8e77d.el7.x86_64", "Docker Storage Driver: overlay2", "OpenvSwitch Version: openvswitch-2.7.3-2.git20171010.el7fdp.x86_64", "etcd Version: etcd-3.2.11-1.el7.x86_64", "Network Plugin: redhat/openshift-ovs-subnet", "Auth Method: allowall", "Registry Deployment Method: deploymentconfig", "Secure Registry: True", "Registry Backend Storage: swift", "Load Balancer: Haproxy", "Docker System Container: False", "CRI-O Enable: False", "Firewall Service: iptables"
Containerised OCP cluster upgrade is blocked by this bug.
Failure due to: ImportError: No module named ipaddress *** This bug has been marked as a duplicate of bug 1540537 ***