Description of problem: During OCP 4.6.16 (with CNV 2.5.3) upgrade to OCP 4.7.0-rc.1, one node becomes NotReady. The upgrade is not progressing: version 4.6.16 True True 143m Unable to apply 4.7.0-rc.1: an unknown error has occurred: MultipleErrors Version-Release number of selected component (if applicable): OCP 4.6.16 (with CNV 2.5.3) upgrade to OCP 4.7.0-rc.1 How reproducible: Observed 4 times, but before thought it was caused by bug 1918609 Steps to Reproduce: 1. OCP 4.6.16 with CNV 2.5.3 2. Run 3 VMs (2 can be live migrated, 1 cannot) 3. Start an upgrade to OCP 4.6.16 (with CNV 2.5.3) Actual results: One node becomes NotReady. The upgrade is not progressing. This node had running workloads on it before the upgrade. A 3 workload is not live migrated. Expected results: Upgrade should complete successfully. Additional info: =============================================== $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.6.16 True True 143m Unable to apply 4.7.0-rc.1: an unknown error has occurred: MultipleErrors =============================================== $ oc get node NAME STATUS ROLES AGE VERSION ssp04-rvkqg-master-0 Ready master 23h v1.20.0+ba45583 ssp04-rvkqg-master-1 Ready master 23h v1.20.0+ba45583 ssp04-rvkqg-master-2 Ready master 23h v1.20.0+ba45583 ssp04-rvkqg-worker-0-9knk9 Ready worker 22h v1.20.0+ba45583 ssp04-rvkqg-worker-0-ns68h Ready,SchedulingDisabled worker 22h v1.19.0+e49167a ssp04-rvkqg-worker-0-zvfkw NotReady worker 22h v1.19.0+e49167a =============================================== $ oc get vmi NAME AGE PHASE IP NODENAME fedora33-hpp 90m Scheduling rhel8-nfs 178m Running 10.131.0.47 ssp04-rvkqg-worker-0-zvfkw win10-ocs 166m Running 10.131.0.48 ssp04-rvkqg-worker-0-zvfkw =============================================== $ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.7.0-rc.1 True False False 26s baremetal 4.7.0-rc.1 True False False 124m cloud-credential 4.7.0-rc.1 True False False 23h cluster-autoscaler 4.7.0-rc.1 True False False 23h config-operator 4.7.0-rc.1 True False False 23h console 4.7.0-rc.1 True False False 79m csi-snapshot-controller 4.7.0-rc.1 True False False 86m dns 4.7.0-rc.1 True False False 23h etcd 4.7.0-rc.1 True False False 23h image-registry 4.7.0-rc.1 True False False 87m ingress 4.7.0-rc.1 True False True 22h insights 4.7.0-rc.1 True False True 23h kube-apiserver 4.7.0-rc.1 True False False 23h kube-controller-manager 4.7.0-rc.1 True False False 23h kube-scheduler 4.7.0-rc.1 True False False 23h kube-storage-version-migrator 4.7.0-rc.1 True False False 85m machine-api 4.7.0-rc.1 True False False 23h machine-approver 4.7.0-rc.1 True False False 23h machine-config 4.6.16 False True True 100m marketplace 4.7.0-rc.1 True False False 78m monitoring 4.7.0-rc.1 False True True 86m network 4.7.0-rc.1 True True True 100m node-tuning 4.7.0-rc.1 True False False 121m openshift-apiserver 4.7.0-rc.1 True False False 72m openshift-controller-manager 4.7.0-rc.1 True False False 23h openshift-samples 4.7.0-rc.1 True False False 121m operator-lifecycle-manager 4.7.0-rc.1 True False False 23h operator-lifecycle-manager-catalog 4.7.0-rc.1 True False False 23h operator-lifecycle-manager-packageserver 4.7.0-rc.1 True False False 79m service-ca 4.7.0-rc.1 True False False 23h storage 4.7.0-rc.1 True True False 79m =============================================== $ oc describe node ssp04-rvkqg-worker-0-zvfkw Name: ssp04-rvkqg-worker-0-zvfkw Roles: worker Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/instance-type=ci.nested.virt.m1.xlarge beta.kubernetes.io/os=linux cluster.ocs.openshift.io/openshift-storage= cpumanager=true failure-domain.beta.kubernetes.io/zone=nova feature.node.kubernetes.io/cpu-feature-aes=true feature.node.kubernetes.io/cpu-feature-avx=true feature.node.kubernetes.io/cpu-feature-avx2=true feature.node.kubernetes.io/cpu-feature-bmi1=true feature.node.kubernetes.io/cpu-feature-bmi2=true feature.node.kubernetes.io/cpu-feature-erms=true feature.node.kubernetes.io/cpu-feature-f16c=true feature.node.kubernetes.io/cpu-feature-fma=true feature.node.kubernetes.io/cpu-feature-fsgsbase=true feature.node.kubernetes.io/cpu-feature-invpcid=true feature.node.kubernetes.io/cpu-feature-movbe=true feature.node.kubernetes.io/cpu-feature-pcid=true feature.node.kubernetes.io/cpu-feature-pclmuldq=true feature.node.kubernetes.io/cpu-feature-popcnt=true feature.node.kubernetes.io/cpu-feature-rdrand=true feature.node.kubernetes.io/cpu-feature-rdtscp=true feature.node.kubernetes.io/cpu-feature-smep=true feature.node.kubernetes.io/cpu-feature-spec-ctrl=true feature.node.kubernetes.io/cpu-feature-sse4.2=true feature.node.kubernetes.io/cpu-feature-svm=true feature.node.kubernetes.io/cpu-feature-tsc-deadline=true feature.node.kubernetes.io/cpu-feature-vme=true feature.node.kubernetes.io/cpu-feature-x2apic=true feature.node.kubernetes.io/cpu-feature-xsave=true feature.node.kubernetes.io/cpu-model-Haswell-noTSX=true feature.node.kubernetes.io/cpu-model-Haswell-noTSX-IBRS=true feature.node.kubernetes.io/cpu-model-IvyBridge=true feature.node.kubernetes.io/cpu-model-IvyBridge-IBRS=true feature.node.kubernetes.io/cpu-model-Nehalem=true feature.node.kubernetes.io/cpu-model-Nehalem-IBRS=true feature.node.kubernetes.io/cpu-model-Opteron_G1=true feature.node.kubernetes.io/cpu-model-Opteron_G2=true feature.node.kubernetes.io/cpu-model-Penryn=true feature.node.kubernetes.io/cpu-model-SandyBridge=true feature.node.kubernetes.io/cpu-model-SandyBridge-IBRS=true feature.node.kubernetes.io/cpu-model-Westmere=true feature.node.kubernetes.io/cpu-model-Westmere-IBRS=true feature.node.kubernetes.io/cpu-model-kvm32=true feature.node.kubernetes.io/cpu-model-kvm64=true feature.node.kubernetes.io/cpu-model-qemu32=true feature.node.kubernetes.io/cpu-model-qemu64=true feature.node.kubernetes.io/kvm-info-cap-hyperv-base=true feature.node.kubernetes.io/kvm-info-cap-hyperv-frequencies=true feature.node.kubernetes.io/kvm-info-cap-hyperv-ipi=true feature.node.kubernetes.io/kvm-info-cap-hyperv-reenlightenment=true feature.node.kubernetes.io/kvm-info-cap-hyperv-reset=true feature.node.kubernetes.io/kvm-info-cap-hyperv-runtime=true feature.node.kubernetes.io/kvm-info-cap-hyperv-synic=true feature.node.kubernetes.io/kvm-info-cap-hyperv-synic2=true feature.node.kubernetes.io/kvm-info-cap-hyperv-synictimer=true feature.node.kubernetes.io/kvm-info-cap-hyperv-time=true feature.node.kubernetes.io/kvm-info-cap-hyperv-tlbflush=true feature.node.kubernetes.io/kvm-info-cap-hyperv-vpindex=true kubernetes.io/arch=amd64 kubernetes.io/hostname=ssp04-rvkqg-worker-0-zvfkw kubernetes.io/os=linux kubevirt.io/schedulable=false node-role.kubernetes.io/worker= node.kubernetes.io/instance-type=ci.nested.virt.m1.xlarge node.openshift.io/os_id=rhcos topology.cinder.csi.openstack.org/zone=nova topology.kubernetes.io/zone=nova topology.rook.io/rack=rack2 Annotations: csi.volume.kubernetes.io/nodeid: {"cinder.csi.openstack.org":"885b9301-803e-43b9-a6c8-c07324f015e6","manila.csi.openstack.org":"ssp04-rvkqg-worker-0-zvfkw","openshift-stor... kubevirt.io/heartbeat: 2021-02-11T14:46:59Z machine.openshift.io/machine: openshift-machine-api/ssp04-rvkqg-worker-0-zvfkw machineconfiguration.openshift.io/currentConfig: rendered-worker-e32d0c5fbb975fcd1cd2ff67ba0f0965 machineconfiguration.openshift.io/desiredConfig: rendered-worker-e32d0c5fbb975fcd1cd2ff67ba0f0965 machineconfiguration.openshift.io/reason: machineconfiguration.openshift.io/state: Done node-labeller-feature.node.kubernetes.io/cpu-feature-aes: true node-labeller-feature.node.kubernetes.io/cpu-feature-avx: true node-labeller-feature.node.kubernetes.io/cpu-feature-avx2: true node-labeller-feature.node.kubernetes.io/cpu-feature-bmi1: true node-labeller-feature.node.kubernetes.io/cpu-feature-bmi2: true node-labeller-feature.node.kubernetes.io/cpu-feature-erms: true node-labeller-feature.node.kubernetes.io/cpu-feature-f16c: true node-labeller-feature.node.kubernetes.io/cpu-feature-fma: true node-labeller-feature.node.kubernetes.io/cpu-feature-fsgsbase: true node-labeller-feature.node.kubernetes.io/cpu-feature-invpcid: true node-labeller-feature.node.kubernetes.io/cpu-feature-movbe: true node-labeller-feature.node.kubernetes.io/cpu-feature-pcid: true node-labeller-feature.node.kubernetes.io/cpu-feature-pclmuldq: true node-labeller-feature.node.kubernetes.io/cpu-feature-popcnt: true node-labeller-feature.node.kubernetes.io/cpu-feature-rdrand: true node-labeller-feature.node.kubernetes.io/cpu-feature-rdtscp: true node-labeller-feature.node.kubernetes.io/cpu-feature-smep: true node-labeller-feature.node.kubernetes.io/cpu-feature-spec-ctrl: true node-labeller-feature.node.kubernetes.io/cpu-feature-sse4.2: true node-labeller-feature.node.kubernetes.io/cpu-feature-svm: true node-labeller-feature.node.kubernetes.io/cpu-feature-tsc-deadline: true node-labeller-feature.node.kubernetes.io/cpu-feature-vme: true node-labeller-feature.node.kubernetes.io/cpu-feature-x2apic: true node-labeller-feature.node.kubernetes.io/cpu-feature-xsave: true node-labeller-feature.node.kubernetes.io/cpu-model-Haswell-noTSX: true node-labeller-feature.node.kubernetes.io/cpu-model-Haswell-noTSX-IBRS: true node-labeller-feature.node.kubernetes.io/cpu-model-IvyBridge: true node-labeller-feature.node.kubernetes.io/cpu-model-IvyBridge-IBRS: true node-labeller-feature.node.kubernetes.io/cpu-model-Nehalem: true node-labeller-feature.node.kubernetes.io/cpu-model-Nehalem-IBRS: true node-labeller-feature.node.kubernetes.io/cpu-model-Opteron_G1: true node-labeller-feature.node.kubernetes.io/cpu-model-Opteron_G2: true node-labeller-feature.node.kubernetes.io/cpu-model-Penryn: true node-labeller-feature.node.kubernetes.io/cpu-model-SandyBridge: true node-labeller-feature.node.kubernetes.io/cpu-model-SandyBridge-IBRS: true node-labeller-feature.node.kubernetes.io/cpu-model-Westmere: true node-labeller-feature.node.kubernetes.io/cpu-model-Westmere-IBRS: true node-labeller-feature.node.kubernetes.io/cpu-model-kvm32: true node-labeller-feature.node.kubernetes.io/cpu-model-kvm64: true node-labeller-feature.node.kubernetes.io/cpu-model-qemu32: true node-labeller-feature.node.kubernetes.io/cpu-model-qemu64: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-base: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-frequencies: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-ipi: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-reenlightenment: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-reset: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-runtime: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-synic: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-synic2: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-synictimer: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-time: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-tlbflush: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-vpindex: true volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Wed, 10 Feb 2021 17:12:28 +0000 Taints: node.kubernetes.io/unreachable:NoExecute node.kubernetes.io/unreachable:NoSchedule Unschedulable: false Lease: HolderIdentity: ssp04-rvkqg-worker-0-zvfkw AcquireTime: <unset> RenewTime: Thu, 11 Feb 2021 14:47:18 +0000 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure Unknown Thu, 11 Feb 2021 14:47:20 +0000 Thu, 11 Feb 2021 14:48:01 +0000 NodeStatusUnknown Kubelet stopped posting node status. DiskPressure Unknown Thu, 11 Feb 2021 14:47:20 +0000 Thu, 11 Feb 2021 14:48:01 +0000 NodeStatusUnknown Kubelet stopped posting node status. PIDPressure Unknown Thu, 11 Feb 2021 14:47:20 +0000 Thu, 11 Feb 2021 14:48:01 +0000 NodeStatusUnknown Kubelet stopped posting node status. Ready Unknown Thu, 11 Feb 2021 14:47:20 +0000 Thu, 11 Feb 2021 14:48:01 +0000 NodeStatusUnknown Kubelet stopped posting node status. Addresses: InternalIP: 192.168.1.21 Hostname: ssp04-rvkqg-worker-0-zvfkw Capacity: attachable-volumes-cinder: 256 cpu: 8 devices.kubevirt.io/kvm: 110 devices.kubevirt.io/tun: 110 devices.kubevirt.io/vhost-net: 110 ephemeral-storage: 41391084Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 16418260Ki ovs-cni.network.kubevirt.io/br0: 1k pods: 250 Allocatable: attachable-volumes-cinder: 256 cpu: 7500m devices.kubevirt.io/kvm: 110 devices.kubevirt.io/tun: 110 devices.kubevirt.io/vhost-net: 110 ephemeral-storage: 37072281128 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 15267284Ki ovs-cni.network.kubevirt.io/br0: 1k pods: 250 System Info: Machine ID: 885b9301803e43b9a6c8c07324f015e6 System UUID: 885b9301-803e-43b9-a6c8-c07324f015e6 Boot ID: 8e64b1bd-f422-411c-a633-c451e40469ec Kernel Version: 4.18.0-193.41.1.el8_2.x86_64 OS Image: Red Hat Enterprise Linux CoreOS 46.82.202101301821-0 (Ootpa) Operating System: linux Architecture: amd64 Container Runtime Version: cri-o://1.19.1-7.rhaos4.6.git6377f68.el8 Kubelet Version: v1.19.0+e49167a Kube-Proxy Version: v1.19.0+e49167a ProviderID: openstack:///885b9301-803e-43b9-a6c8-c07324f015e6 Non-terminated Pods: (69 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- b4-upgarde virt-launcher-rhel8-nfs-rr9tx 100m (1%) 100m (1%) 1709Mi (11%) 40M (0%) 175m b4-upgarde virt-launcher-win10-ocs-2kx2v 100m (1%) 100m (1%) 4481613825 (28%) 40M (0%) 163m openshift-cluster-csi-drivers openstack-cinder-csi-driver-node-h4pp8 20m (0%) 0 (0%) 100Mi (0%) 0 (0%) 121m openshift-cluster-node-tuning-operator tuned-7ll2f 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 120m openshift-cnv bridge-marker-n9pqq 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-cnv cdi-apiserver-fd7f4fb6-q2b8v 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-cnv cdi-deployment-65c8dbfdc7-hxms2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-cnv cdi-uploadproxy-c9ff9fc78-qp99p 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-cnv cluster-network-addons-operator-69dd97cb44-l26h9 0 (0%) 0 (0%) 0 (0%) 0 (0%) 94m openshift-cnv hostpath-provisioner-operator-55d7b8b595-zhbk9 0 (0%) 0 (0%) 0 (0%) 0 (0%) 94m openshift-cnv hostpath-provisioner-vhvr7 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-cnv kube-cni-linux-bridge-plugin-zvvcf 60m (0%) 0 (0%) 30Mi (0%) 0 (0%) 21h openshift-cnv kubevirt-node-labeller-qmmb6 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-cnv nmstate-handler-ssqsc 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-cnv ovs-cni-amd64-t8cbz 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-cnv virt-api-6d765b6dd5-t2v5l 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-cnv virt-controller-68985f6974-mv2vb 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-cnv virt-handler-mzfgg 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-cnv virt-operator-7d57664b89-nzjcl 0 (0%) 0 (0%) 0 (0%) 0 (0%) 94m openshift-cnv virt-template-validator-65dbfc87d8-56dks 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-cnv virt-template-validator-65dbfc87d8-89rvn 0 (0%) 0 (0%) 0 (0%) 0 (0%) 87m openshift-cnv vm-import-controller-74d785b999-4vrkx 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-cnv vm-import-operator-7d856d4f4b-kdzh6 0 (0%) 0 (0%) 0 (0%) 0 (0%) 87m openshift-dns dns-default-pn9k5 65m (0%) 0 (0%) 110Mi (0%) 512Mi (3%) 100m openshift-image-registry node-ca-lmnqz 10m (0%) 0 (0%) 10Mi (0%) 0 (0%) 120m openshift-ingress-canary ingress-canary-gn4fs 10m (0%) 0 (0%) 20Mi (0%) 0 (0%) 122m openshift-ingress router-default-6c5c8d967b-vm8qb 100m (1%) 0 (0%) 256Mi (1%) 0 (0%) 94m openshift-local-storage local-block-local-diskmaker-8zkvh 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-local-storage local-block-local-provisioner-phdhq 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-machine-config-operator machine-config-daemon-sqjbv 40m (0%) 0 (0%) 100Mi (0%) 0 (0%) 98m openshift-manila-csi-driver csi-nodeplugin-nfsplugin-frgkn 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 121m openshift-manila-csi-driver openstack-manila-csi-nodeplugin-v9889 15m (0%) 0 (0%) 70Mi (0%) 0 (0%) 121m openshift-marketplace certified-operators-5ghdn 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 94m openshift-marketplace hco-catalogsource-s5p9d 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 21h openshift-marketplace ocs-catalogsource-m5974 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 87m openshift-marketplace redhat-marketplace-qfmcp 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 87m openshift-marketplace redhat-operators-vhszj 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 94m openshift-monitoring alertmanager-main-0 8m (0%) 0 (0%) 270Mi (1%) 0 (0%) 94m openshift-monitoring alertmanager-main-1 8m (0%) 0 (0%) 270Mi (1%) 0 (0%) 121m openshift-monitoring kube-state-metrics-6c798c69f5-sdkph 4m (0%) 0 (0%) 120Mi (0%) 0 (0%) 87m openshift-monitoring node-exporter-x6lq2 9m (0%) 0 (0%) 210Mi (1%) 0 (0%) 121m openshift-monitoring openshift-state-metrics-6bd7d6f65-b7c9x 3m (0%) 0 (0%) 190Mi (1%) 0 (0%) 94m openshift-monitoring prometheus-adapter-6575b658bf-qnxj6 1m (0%) 0 (0%) 25Mi (0%) 0 (0%) 94m openshift-monitoring prometheus-adapter-6575b658bf-rzxht 1m (0%) 0 (0%) 25Mi (0%) 0 (0%) 87m openshift-monitoring prometheus-k8s-0 76m (1%) 0 (0%) 1204Mi (8%) 0 (0%) 87m openshift-monitoring prometheus-k8s-1 76m (1%) 0 (0%) 1204Mi (8%) 0 (0%) 121m openshift-monitoring thanos-querier-565fc8859b-pkzx2 9m (0%) 0 (0%) 92Mi (0%) 0 (0%) 94m openshift-multus multus-4w72h 10m (0%) 0 (0%) 150Mi (1%) 0 (0%) 104m openshift-multus network-metrics-daemon-nm5zq 20m (0%) 0 (0%) 120Mi (0%) 0 (0%) 105m openshift-network-diagnostics network-check-target-4bsnh 10m (0%) 0 (0%) 15Mi (0%) 0 (0%) 107m openshift-openstack-infra coredns-ssp04-rvkqg-worker-0-zvfkw 100m (1%) 0 (0%) 200Mi (1%) 0 (0%) 23h openshift-openstack-infra keepalived-ssp04-rvkqg-worker-0-zvfkw 200m (2%) 0 (0%) 400Mi (2%) 0 (0%) 23h openshift-openstack-infra mdns-publisher-ssp04-rvkqg-worker-0-zvfkw 100m (1%) 0 (0%) 200Mi (1%) 0 (0%) 22h openshift-sdn ovs-k77kz 15m (0%) 0 (0%) 400Mi (2%) 0 (0%) 103m openshift-sdn sdn-srksb 110m (1%) 0 (0%) 220Mi (1%) 0 (0%) 107m openshift-storage csi-cephfsplugin-provisioner-6bc7cbf6f-88vxh 0 (0%) 0 (0%) 0 (0%) 0 (0%) 94m openshift-storage csi-cephfsplugin-xjcf6 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-storage csi-rbdplugin-8rtwc 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-storage csi-rbdplugin-provisioner-7699b8c4b8-vtf9b 0 (0%) 0 (0%) 0 (0%) 0 (0%) 94m openshift-storage noobaa-endpoint-7584c5969f-rwn7z 0 (0%) 0 (0%) 0 (0%) 0 (0%) 87m openshift-storage noobaa-operator-5855b5688-cfn67 0 (0%) 0 (0%) 0 (0%) 0 (0%) 94m openshift-storage ocs-metrics-exporter-5d66d5fc59-fbqm6 0 (0%) 0 (0%) 0 (0%) 0 (0%) 87m openshift-storage ocs-operator-cd5b866f5-29q72 0 (0%) 0 (0%) 0 (0%) 0 (0%) 87m openshift-storage rook-ceph-crashcollector-ssp04-rvkqg-worker-0-zvfkw-666867qr7vg 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-storage rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-786bdbf7qv4qj 0 (0%) 0 (0%) 0 (0%) 0 (0%) 94m openshift-storage rook-ceph-mgr-a-5576d7458b-sdsz2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 94m openshift-storage rook-ceph-operator-6cfc658d4b-prp6k 0 (0%) 0 (0%) 0 (0%) 0 (0%) 94m openshift-storage rook-ceph-osd-1-5946d84494-rkmbx 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-storage rook-ceph-rgw-ocs-storagecluster-cephobjectstore-b-8b6b579tmgtf 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 1350m (18%) 200m (2%) memory 12943622145 (82%) 616870912 (3%) ephemeral-storage 0 (0%) 0 (0%) hugepages-1Gi 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) attachable-volumes-cinder 0 0 devices.kubevirt.io/kvm 2 2 devices.kubevirt.io/tun 2 2 devices.kubevirt.io/vhost-net 1 1 ovs-cni.network.kubevirt.io/br0 0 0 Events: <none> =============================================== $ oc describe node ssp04-rvkqg-worker-0-ns68h Name: ssp04-rvkqg-worker-0-ns68h Roles: worker Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/instance-type=ci.nested.virt.m1.xlarge beta.kubernetes.io/os=linux cluster.ocs.openshift.io/openshift-storage= cpumanager=true failure-domain.beta.kubernetes.io/zone=nova feature.node.kubernetes.io/cpu-feature-aes=true feature.node.kubernetes.io/cpu-feature-avx=true feature.node.kubernetes.io/cpu-feature-avx2=true feature.node.kubernetes.io/cpu-feature-bmi1=true feature.node.kubernetes.io/cpu-feature-bmi2=true feature.node.kubernetes.io/cpu-feature-erms=true feature.node.kubernetes.io/cpu-feature-f16c=true feature.node.kubernetes.io/cpu-feature-fma=true feature.node.kubernetes.io/cpu-feature-fsgsbase=true feature.node.kubernetes.io/cpu-feature-invpcid=true feature.node.kubernetes.io/cpu-feature-movbe=true feature.node.kubernetes.io/cpu-feature-pcid=true feature.node.kubernetes.io/cpu-feature-pclmuldq=true feature.node.kubernetes.io/cpu-feature-popcnt=true feature.node.kubernetes.io/cpu-feature-rdrand=true feature.node.kubernetes.io/cpu-feature-rdtscp=true feature.node.kubernetes.io/cpu-feature-smep=true feature.node.kubernetes.io/cpu-feature-spec-ctrl=true feature.node.kubernetes.io/cpu-feature-sse4.2=true feature.node.kubernetes.io/cpu-feature-svm=true feature.node.kubernetes.io/cpu-feature-tsc-deadline=true feature.node.kubernetes.io/cpu-feature-vme=true feature.node.kubernetes.io/cpu-feature-x2apic=true feature.node.kubernetes.io/cpu-feature-xsave=true feature.node.kubernetes.io/cpu-model-Haswell-noTSX=true feature.node.kubernetes.io/cpu-model-Haswell-noTSX-IBRS=true feature.node.kubernetes.io/cpu-model-IvyBridge=true feature.node.kubernetes.io/cpu-model-IvyBridge-IBRS=true feature.node.kubernetes.io/cpu-model-Nehalem=true feature.node.kubernetes.io/cpu-model-Nehalem-IBRS=true feature.node.kubernetes.io/cpu-model-Opteron_G1=true feature.node.kubernetes.io/cpu-model-Opteron_G2=true feature.node.kubernetes.io/cpu-model-Penryn=true feature.node.kubernetes.io/cpu-model-SandyBridge=true feature.node.kubernetes.io/cpu-model-SandyBridge-IBRS=true feature.node.kubernetes.io/cpu-model-Westmere=true feature.node.kubernetes.io/cpu-model-Westmere-IBRS=true feature.node.kubernetes.io/cpu-model-kvm32=true feature.node.kubernetes.io/cpu-model-kvm64=true feature.node.kubernetes.io/cpu-model-qemu32=true feature.node.kubernetes.io/cpu-model-qemu64=true feature.node.kubernetes.io/kvm-info-cap-hyperv-base=true feature.node.kubernetes.io/kvm-info-cap-hyperv-frequencies=true feature.node.kubernetes.io/kvm-info-cap-hyperv-ipi=true feature.node.kubernetes.io/kvm-info-cap-hyperv-reenlightenment=true feature.node.kubernetes.io/kvm-info-cap-hyperv-reset=true feature.node.kubernetes.io/kvm-info-cap-hyperv-runtime=true feature.node.kubernetes.io/kvm-info-cap-hyperv-synic=true feature.node.kubernetes.io/kvm-info-cap-hyperv-synic2=true feature.node.kubernetes.io/kvm-info-cap-hyperv-synictimer=true feature.node.kubernetes.io/kvm-info-cap-hyperv-time=true feature.node.kubernetes.io/kvm-info-cap-hyperv-tlbflush=true feature.node.kubernetes.io/kvm-info-cap-hyperv-vpindex=true kubernetes.io/arch=amd64 kubernetes.io/hostname=ssp04-rvkqg-worker-0-ns68h kubernetes.io/os=linux kubevirt.io/schedulable=true node-role.kubernetes.io/worker= node.kubernetes.io/instance-type=ci.nested.virt.m1.xlarge node.openshift.io/os_id=rhcos topology.cinder.csi.openstack.org/zone=nova topology.kubernetes.io/zone=nova topology.rook.io/rack=rack1 Annotations: csi.volume.kubernetes.io/nodeid: {"cinder.csi.openstack.org":"b171cd7d-095d-448c-b602-19d849256e73","manila.csi.openstack.org":"ssp04-rvkqg-worker-0-ns68h","openshift-stor... kubevirt.io/heartbeat: 2021-02-11T16:13:43Z machine.openshift.io/machine: openshift-machine-api/ssp04-rvkqg-worker-0-ns68h machineconfiguration.openshift.io/currentConfig: rendered-worker-e32d0c5fbb975fcd1cd2ff67ba0f0965 machineconfiguration.openshift.io/desiredConfig: rendered-worker-5cfc02ef8ec6a59bf10e98d7ebc772b6 machineconfiguration.openshift.io/reason: failed to drain node (5 tries): timed out waiting for the condition: error when evicting pods/"rook-ceph-osd-2-5786c9fb5f-vf97v" -n "opens... machineconfiguration.openshift.io/state: Degraded node-labeller-feature.node.kubernetes.io/cpu-feature-aes: true node-labeller-feature.node.kubernetes.io/cpu-feature-avx: true node-labeller-feature.node.kubernetes.io/cpu-feature-avx2: true node-labeller-feature.node.kubernetes.io/cpu-feature-bmi1: true node-labeller-feature.node.kubernetes.io/cpu-feature-bmi2: true node-labeller-feature.node.kubernetes.io/cpu-feature-erms: true node-labeller-feature.node.kubernetes.io/cpu-feature-f16c: true node-labeller-feature.node.kubernetes.io/cpu-feature-fma: true node-labeller-feature.node.kubernetes.io/cpu-feature-fsgsbase: true node-labeller-feature.node.kubernetes.io/cpu-feature-invpcid: true node-labeller-feature.node.kubernetes.io/cpu-feature-movbe: true node-labeller-feature.node.kubernetes.io/cpu-feature-pcid: true node-labeller-feature.node.kubernetes.io/cpu-feature-pclmuldq: true node-labeller-feature.node.kubernetes.io/cpu-feature-popcnt: true node-labeller-feature.node.kubernetes.io/cpu-feature-rdrand: true node-labeller-feature.node.kubernetes.io/cpu-feature-rdtscp: true node-labeller-feature.node.kubernetes.io/cpu-feature-smep: true node-labeller-feature.node.kubernetes.io/cpu-feature-spec-ctrl: true node-labeller-feature.node.kubernetes.io/cpu-feature-sse4.2: true node-labeller-feature.node.kubernetes.io/cpu-feature-svm: true node-labeller-feature.node.kubernetes.io/cpu-feature-tsc-deadline: true node-labeller-feature.node.kubernetes.io/cpu-feature-vme: true node-labeller-feature.node.kubernetes.io/cpu-feature-x2apic: true node-labeller-feature.node.kubernetes.io/cpu-feature-xsave: true node-labeller-feature.node.kubernetes.io/cpu-model-Haswell-noTSX: true node-labeller-feature.node.kubernetes.io/cpu-model-Haswell-noTSX-IBRS: true node-labeller-feature.node.kubernetes.io/cpu-model-IvyBridge: true node-labeller-feature.node.kubernetes.io/cpu-model-IvyBridge-IBRS: true node-labeller-feature.node.kubernetes.io/cpu-model-Nehalem: true node-labeller-feature.node.kubernetes.io/cpu-model-Nehalem-IBRS: true node-labeller-feature.node.kubernetes.io/cpu-model-Opteron_G1: true node-labeller-feature.node.kubernetes.io/cpu-model-Opteron_G2: true node-labeller-feature.node.kubernetes.io/cpu-model-Penryn: true node-labeller-feature.node.kubernetes.io/cpu-model-SandyBridge: true node-labeller-feature.node.kubernetes.io/cpu-model-SandyBridge-IBRS: true node-labeller-feature.node.kubernetes.io/cpu-model-Westmere: true node-labeller-feature.node.kubernetes.io/cpu-model-Westmere-IBRS: true node-labeller-feature.node.kubernetes.io/cpu-model-kvm32: true node-labeller-feature.node.kubernetes.io/cpu-model-kvm64: true node-labeller-feature.node.kubernetes.io/cpu-model-qemu32: true node-labeller-feature.node.kubernetes.io/cpu-model-qemu64: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-base: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-frequencies: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-ipi: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-reenlightenment: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-reset: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-runtime: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-synic: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-synic2: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-synictimer: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-time: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-tlbflush: true node-labeller-feature.node.kubernetes.io/kvm-info-cap-hyperv-vpindex: true volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Wed, 10 Feb 2021 17:12:34 +0000 Taints: node.kubernetes.io/unschedulable:NoSchedule Unschedulable: true Lease: HolderIdentity: ssp04-rvkqg-worker-0-ns68h AcquireTime: <unset> RenewTime: Thu, 11 Feb 2021 16:13:59 +0000 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Thu, 11 Feb 2021 16:11:19 +0000 Wed, 10 Feb 2021 18:47:25 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Thu, 11 Feb 2021 16:11:19 +0000 Wed, 10 Feb 2021 18:47:25 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Thu, 11 Feb 2021 16:11:19 +0000 Wed, 10 Feb 2021 18:47:25 +0000 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Thu, 11 Feb 2021 16:11:19 +0000 Wed, 10 Feb 2021 18:47:25 +0000 KubeletReady kubelet is posting ready status Addresses: InternalIP: 192.168.0.70 Hostname: ssp04-rvkqg-worker-0-ns68h Capacity: attachable-volumes-cinder: 256 cpu: 8 devices.kubevirt.io/kvm: 110 devices.kubevirt.io/tun: 110 devices.kubevirt.io/vhost-net: 110 ephemeral-storage: 41391084Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 16418260Ki ovs-cni.network.kubevirt.io/br0: 1k pods: 250 Allocatable: attachable-volumes-cinder: 256 cpu: 7500m devices.kubevirt.io/kvm: 110 devices.kubevirt.io/tun: 110 devices.kubevirt.io/vhost-net: 110 ephemeral-storage: 37072281128 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 15267284Ki ovs-cni.network.kubevirt.io/br0: 1k pods: 250 System Info: Machine ID: b171cd7d095d448cb60219d849256e73 System UUID: b171cd7d-095d-448c-b602-19d849256e73 Boot ID: a28163e5-e265-4b5d-99e4-f6986b939664 Kernel Version: 4.18.0-193.41.1.el8_2.x86_64 OS Image: Red Hat Enterprise Linux CoreOS 46.82.202101301821-0 (Ootpa) Operating System: linux Architecture: amd64 Container Runtime Version: cri-o://1.19.1-7.rhaos4.6.git6377f68.el8 Kubelet Version: v1.19.0+e49167a Kube-Proxy Version: v1.19.0+e49167a ProviderID: openstack:///b171cd7d-095d-448c-b602-19d849256e73 Non-terminated Pods: (29 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- openshift-cluster-csi-drivers openstack-cinder-csi-driver-node-d62s2 20m (0%) 0 (0%) 100Mi (0%) 0 (0%) 122m openshift-cluster-node-tuning-operator tuned-ncbs8 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 121m openshift-cnv bridge-marker-jpjh7 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-cnv hostpath-provisioner-5pmkg 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-cnv kube-cni-linux-bridge-plugin-7vd5p 60m (0%) 0 (0%) 30Mi (0%) 0 (0%) 21h openshift-cnv kubevirt-node-labeller-wx6s2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-cnv nmstate-handler-knshm 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-cnv ovs-cni-amd64-z4qgg 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-cnv virt-handler-v9m9k 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-dns dns-default-lnh4c 65m (0%) 0 (0%) 110Mi (0%) 512Mi (3%) 99m openshift-image-registry node-ca-vs7cs 10m (0%) 0 (0%) 10Mi (0%) 0 (0%) 121m openshift-ingress-canary ingress-canary-fxkr9 10m (0%) 0 (0%) 20Mi (0%) 0 (0%) 122m openshift-local-storage local-block-local-diskmaker-lsfjd 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-local-storage local-block-local-provisioner-6zhx6 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-machine-config-operator machine-config-daemon-fbtcb 40m (0%) 0 (0%) 100Mi (0%) 0 (0%) 97m openshift-manila-csi-driver csi-nodeplugin-nfsplugin-kn4rg 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 121m openshift-manila-csi-driver openstack-manila-csi-nodeplugin-gf45p 15m (0%) 0 (0%) 70Mi (0%) 0 (0%) 122m openshift-monitoring node-exporter-pdqdr 9m (0%) 0 (0%) 210Mi (1%) 0 (0%) 122m openshift-multus multus-v52tr 10m (0%) 0 (0%) 150Mi (1%) 0 (0%) 102m openshift-multus network-metrics-daemon-fnrvf 20m (0%) 0 (0%) 120Mi (0%) 0 (0%) 106m openshift-network-diagnostics network-check-target-lsmpc 10m (0%) 0 (0%) 15Mi (0%) 0 (0%) 108m openshift-openstack-infra coredns-ssp04-rvkqg-worker-0-ns68h 100m (1%) 0 (0%) 200Mi (1%) 0 (0%) 22h openshift-openstack-infra keepalived-ssp04-rvkqg-worker-0-ns68h 200m (2%) 0 (0%) 400Mi (2%) 0 (0%) 23h openshift-openstack-infra mdns-publisher-ssp04-rvkqg-worker-0-ns68h 100m (1%) 0 (0%) 200Mi (1%) 0 (0%) 22h openshift-sdn ovs-7mqt8 15m (0%) 0 (0%) 400Mi (2%) 0 (0%) 107m openshift-sdn sdn-kzs6k 110m (1%) 0 (0%) 220Mi (1%) 0 (0%) 106m openshift-storage csi-cephfsplugin-4szch 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-storage csi-rbdplugin-tc7rf 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h openshift-storage rook-ceph-osd-2-5786c9fb5f-vf97v 0 (0%) 0 (0%) 0 (0%) 0 (0%) 21h Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 814m (10%) 0 (0%) memory 2455Mi (16%) 512Mi (3%) ephemeral-storage 0 (0%) 0 (0%) hugepages-1Gi 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) attachable-volumes-cinder 0 0 devices.kubevirt.io/kvm 0 0 devices.kubevirt.io/tun 0 0 devices.kubevirt.io/vhost-net 0 0 ovs-cni.network.kubevirt.io/br0 0 0 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal NodeNotSchedulable 88m (x3 over 21h) kubelet Node ssp04-rvkqg-worker-0-ns68h status is now: NodeNotSchedulable =============================================== $ oc get vm rhel8-nfs -oyaml apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: annotations: kubevirt.io/latest-observed-api-version: v1alpha3 kubevirt.io/storage-observed-api-version: v1alpha3 creationTimestamp: "2021-02-11T13:04:48Z" generation: 1 labels: app: rhel8-nfs vm.kubevirt.io/template: rhel8-server-tiny-v0.11.3 vm.kubevirt.io/template.revision: "1" vm.kubevirt.io/template.version: v0.12.4 managedFields: - apiVersion: kubevirt.io/v1alpha3 fieldsType: FieldsV1 fieldsV1: f:metadata: f:labels: .: {} f:app: {} f:vm.kubevirt.io/template: {} f:vm.kubevirt.io/template.revision: {} f:vm.kubevirt.io/template.version: {} f:spec: .: {} f:runStrategy: {} f:template: .: {} f:metadata: .: {} f:labels: .: {} f:kubevirt.io/domain: {} f:kubevirt.io/size: {} f:spec: .: {} f:domain: .: {} f:cpu: .: {} f:cores: {} f:sockets: {} f:threads: {} f:devices: .: {} f:disks: {} f:interfaces: {} f:networkInterfaceMultiqueue: {} f:rng: {} f:machine: .: {} f:type: {} f:resources: .: {} f:requests: .: {} f:memory: {} f:evictionStrategy: {} f:networks: {} f:terminationGracePeriodSeconds: {} f:volumes: {} manager: kubectl-create operation: Update time: "2021-02-11T13:04:48Z" - apiVersion: kubevirt.io/v1alpha3 fieldsType: FieldsV1 fieldsV1: f:status: .: {} f:conditions: {} f:created: {} manager: virt-api operation: Update time: "2021-02-11T13:17:48Z" - apiVersion: kubevirt.io/v1alpha3 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:kubevirt.io/latest-observed-api-version: {} f:kubevirt.io/storage-observed-api-version: {} f:status: f:conditions: {} f:created: {} manager: virt-controller operation: Update time: "2021-02-11T13:17:56Z" name: rhel8-nfs namespace: b4-upgarde resourceVersion: "1053441" selfLink: /apis/kubevirt.io/v1alpha3/namespaces/b4-upgarde/virtualmachines/rhel8-nfs uid: 96660cdf-d2f0-44a4-b814-fcddedb41596 spec: runStrategy: Manual template: metadata: creationTimestamp: null labels: kubevirt.io/domain: rhel8-nfs kubevirt.io/size: tiny spec: domain: cpu: cores: 1 sockets: 1 threads: 1 devices: disks: - disk: bus: virtio name: rootdisk - disk: bus: virtio name: cloudinitdisk interfaces: - masquerade: {} name: default networkInterfaceMultiqueue: true rng: {} machine: type: pc-q35-rhel8.2.0 resources: requests: memory: 1536Mi evictionStrategy: LiveMigrate networks: - name: default pod: {} terminationGracePeriodSeconds: 180 volumes: - name: rootdisk persistentVolumeClaim: claimName: rhel8 - cloudInitNoCloud: userData: |- #cloud-config user: cloud-user password: redhat chpasswd: { expire: False } name: cloudinitdisk status: conditions: - lastProbeTime: null lastTransitionTime: "2021-02-11T13:17:53Z" status: "True" type: Ready created: true
what's the output of `oc adm node-logs ssp04-rvkqg-worker-0-zvfkw`
$ oc adm node-logs ssp04-rvkqg-worker-0-zvfkw error: error trying to reach service: dial tcp 192.168.1.21:10250: i/o timeout
are you able to access via ssh? if not, you may need to reboot the node via a cloud console to access. I believe we'll need the node logs to make any forward progress
this looks like a DNS issue: running `ssh core@ssp04-rvkqg-worker-0-zvfkw` fails but running `ssh core.1.21` (IP address gotten through `oc get nodes -o wide`) succeeds. On the node, the kubelet is complaining a bunch about ``` Feb 11 21:46:50 ssp04-rvkqg-worker-0-zvfkw hyperkube[2292]: E0211 21:46:50.878835 2292 kubelet.go:2190] node "ssp04-rvkqg-worker-0-zvfkw" not found ``` and trying with ping from another host shows: ``` [cnv-qe-jenkins@ssp04-rvkqg-executor ~]$ ping 192.168.1.21 PING 192.168.1.21 (192.168.1.21) 56(84) bytes of data. 64 bytes from 192.168.1.21: icmp_seq=1 ttl=64 time=1.24 ms 64 bytes from 192.168.1.21: icmp_seq=2 ttl=64 time=0.665 ms 64 bytes from 192.168.1.21: icmp_seq=3 ttl=64 time=0.595 ms 64 bytes from 192.168.1.21: icmp_seq=4 ttl=64 time=0.714 ms 64 bytes from 192.168.1.21: icmp_seq=5 ttl=64 time=0.654 ms 64 bytes from 192.168.1.21: icmp_seq=6 ttl=64 time=0.798 ms 64 bytes from 192.168.1.21: icmp_seq=7 ttl=64 time=0.658 ms ^C --- 192.168.1.21 ping statistics --- 7 packets transmitted, 7 received, 0% packet loss, time 111ms rtt min/avg/max/mdev = 0.595/0.761/1.244/0.206 ms [cnv-qe-jenkins@ssp04-rvkqg-executor ~]$ ping ssp04-rvkqg-worker-0-zvfkw ping: ssp04-rvkqg-worker-0-zvfkw: Name or service not known ```
From within the cluster it works, from one of the masters: [core@ssp04-rvkqg-master-0 ~]$ ping ssp04-rvkqg-worker-0-zvfkw PING ssp04-rvkqg-worker-0-zvfkw.ssp04.cnv-qe.rhcloud.com (192.168.1.21) 56(84) bytes of data. 64 bytes from host-192-168-1-21.openstacklocal (192.168.1.21): icmp_seq=1 ttl=64 time=1.45 ms 64 bytes from host-192-168-1-21.openstacklocal (192.168.1.21): icmp_seq=2 ttl=64 time=0.928 ms 64 bytes from host-192-168-1-21.openstacklocal (192.168.1.21): icmp_seq=3 ttl=64 time=0.615 ms 64 bytes from host-192-168-1-21.openstacklocal (192.168.1.21): icmp_seq=4 ttl=64 time=0.609 ms ^C --- ssp04-rvkqg-worker-0-zvfkw.ssp04.cnv-qe.rhcloud.com ping statistics --- (the access to the cluster is done via a separate vm, which does not have DNS)
*** This bug has been marked as a duplicate of bug 1913532 ***