Description of problem: QE upgrade testing failed in v4.9 upgrading testing with DaemonSet "openshift-network-diagnostics/network-check-target" is not available (awaiting 1 nodes) Version-Release number of selected component (if applicable): 4.8.0-0.nightly-2022-01-24-131630 - > 4.9.0-0.nightly-2022-01-24-212243 4.9.0-0.nightly-arm64-2022-01-24-192540 - > 4.9.0-0.nightly-arm64-2022-01-24-205435 How reproducible: Failed in QE CI testing in both SDN and OVN cluster Steps to Reproduce: #### All nodes are ready: 01-26 00:59:34.280 NAME STATUS ROLES AGE VERSION 01-26 00:59:34.280 machineset-clone-41175-b9hnz Ready worker 3h6m v1.22.3+e790d7f 01-26 00:59:34.280 machineset-clone-41803-dtcr8 Ready worker 178m v1.22.3+e790d7f 01-26 00:59:34.280 machineset-clone-41804-p5z4v Ready worker 169m v1.22.3+e790d7f 01-26 00:59:34.280 newugd-4031-k6lmz-master-0 Ready master 4h2m v1.22.3+e790d7f 01-26 00:59:34.280 newugd-4031-k6lmz-master-1 Ready master 4h2m v1.22.3+e790d7f 01-26 00:59:34.280 newugd-4031-k6lmz-master-2 Ready master 4h1m v1.22.3+e790d7f 01-26 00:59:34.280 newugd-4031-k6lmz-worker-0-rc5hs Ready worker 3h49m v1.22.3+e790d7f 01-26 00:59:34.280 newugd-4031-k6lmz-worker-0-wmfpc Ready worker 3h48m v1.22.3+e790d7f 01-26 00:59:34.280 newugd-4031-k6lmz-worker-0-zvdvp Ready worker 3h48m v1.22.3+e790d7f #### But DaemonSet "openshift-network-diagnostics/network-check-target" is not available (awaiting 1 nodes) 01-26 00:59:34.825 clusteroperators: 01-26 00:59:34.825 NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE 01-26 00:59:34.825 authentication 4.9.0-0.nightly-2022-01-24-212243 True False False 3m10s 01-26 00:59:34.825 baremetal 4.9.0-0.nightly-2022-01-24-212243 True False False 4h 01-26 00:59:34.825 cloud-controller-manager 4.9.0-0.nightly-2022-01-24-212243 True False False 110m 01-26 00:59:34.825 cloud-credential 4.9.0-0.nightly-2022-01-24-212243 True False False 4h2m 01-26 00:59:34.825 cluster-autoscaler 4.9.0-0.nightly-2022-01-24-212243 True False False 3h59m 01-26 00:59:34.825 config-operator 4.9.0-0.nightly-2022-01-24-212243 True False False 4h 01-26 00:59:34.825 console 4.9.0-0.nightly-2022-01-24-212243 True False False 16m 01-26 00:59:34.825 csi-snapshot-controller 4.9.0-0.nightly-2022-01-24-212243 True False False 4h 01-26 00:59:34.825 dns 4.9.0-0.nightly-2022-01-24-212243 True False False 4h 01-26 00:59:34.825 etcd 4.9.0-0.nightly-2022-01-24-212243 True False False 3h57m 01-26 00:59:34.825 image-registry 4.9.0-0.nightly-2022-01-24-212243 True False False 3h48m 01-26 00:59:34.825 ingress 4.9.0-0.nightly-2022-01-24-212243 True False False 3m7s 01-26 00:59:34.825 insights 4.9.0-0.nightly-2022-01-24-212243 True False False 3h52m 01-26 00:59:34.825 kube-apiserver 4.9.0-0.nightly-2022-01-24-212243 True False False 3h48m 01-26 00:59:34.825 kube-controller-manager 4.9.0-0.nightly-2022-01-24-212243 True False False 3h58m 01-26 00:59:34.825 kube-scheduler 4.9.0-0.nightly-2022-01-24-212243 True False False 3h58m 01-26 00:59:34.825 kube-storage-version-migrator 4.9.0-0.nightly-2022-01-24-212243 True False False 3m23s 01-26 00:59:34.825 machine-api 4.9.0-0.nightly-2022-01-24-212243 True False False 3h56m 01-26 00:59:34.825 machine-approver 4.9.0-0.nightly-2022-01-24-212243 True False False 3h59m 01-26 00:59:34.825 machine-config 4.9.0-0.nightly-2022-01-24-212243 True False False 4m28s 01-26 00:59:34.825 marketplace 4.9.0-0.nightly-2022-01-24-212243 True False False 3h59m 01-26 00:59:34.825 monitoring 4.9.0-0.nightly-2022-01-24-212243 True False False 95m 01-26 00:59:34.825 network 4.9.0-0.nightly-2022-01-24-212243 True True False 4h1m DaemonSet "openshift-network-diagnostics/network-check-target" is not available (awaiting 1 nodes) Actual results: DaemonSet "openshift-network-diagnostics/network-check-target" is not available (awaiting 1 nodes) Expected results: Upgrade should pass Additional info:
Right now, the QE CI testing cluster is destroyed and can not collect must-gather QE will try to run similar upgrade testing locally to see if we can reproduce this issue.
Hey Anurag, is this observed on ovn-k or sdn clusters? This bz's sub-component is sdn though I know Weibin usually works on ovn-k. If this is indeed ovn-k then its possible this is a dupe of the bug you linked. @trozet said he will backport this to 4.9 as a part of his fix for https://bugzilla.redhat.com/show_bug.cgi?id=1973286. So if this bz is in ovn-k, you could mark is as duplicate of whichever 4.9.z backport bug Tim opens.
*** This bug has been marked as a duplicate of bug 1973286 ***
This was submitted for some failed QE CI upgrade testing cases, after manually re running the upgrading test locally several times, the upgrading successfully passed from old build to new build. Even I still can see the below warning message during the upgrading process "openshift-network-diagnostics/network-check-target" is not available (awaiting 1 nodes) but eventually, all the cluster operators get upgraded. I think this may be a automation issue, not a production issue. https://issues.redhat.com/browse/OCPQE-8489 is reported for QE CI issue.
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days