test: [cli] oc adm must-gather [Top Level] [cli] oc adm must-gather runs successfully [Suite:openshift/conformance/parallel] failed, see job: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-4.4/1545 Error: [skipping gathering routes.route.openshift.io/oauth-openshift due to error: a resource cannot be retrieved by name across all namespaces, skipping gathering services/oauth-openshift due to error: a resource cannot be retrieved by name across all namespaces, skipping gathering machineautoscalers.machine.openshift.io due to error: the server doesn't have a resource type "machineautoscalers", skipping gathering clusterautoscalers.machine.openshift.io due to error: the server doesn't have a resource type "clusterautoscalers", skipping gathering roles.rbac.authorization.k8s.io/cloud-provider-config-reader due to error: roles.rbac.authorization.k8s.io "cloud-provider-config-reader" not found, skipping gathering all/openshift-monitoring due to error: the server doesn't have a resource type "all", skipping gathering namespaces/openshift-service-catalog-apiserver due to error: namespaces "openshift-service-catalog-apiserver" not found, skipping gathering apiservices.apiregistration.k8s.io/v1beta1.servicecatalog.k8s.io due to error: apiservices.apiregistration.k8s.io "v1beta1.servicecatalog.k8s.io" not found, skipping gathering namespaces/openshift-service-catalog-controller-manager due to error: namespaces "openshift-service-catalog-controller-manager" not found]
The actual bug is: fail [github.com/openshift/origin/test/extended/cli/mustgather.go:150]: Expected <string>: etadata","auditID":"f938e80b-ed9e-4083-8dea-071fd3795fd6","stage":"ResponseComplete","requestURI":"/apis/route.openshift.io/v1/namespaces/openshift-console/routes/downloads","verb":"get","user":{"username":"system:serviceaccount:openshift-console-operator:console-operator","groups":["system:serviceaccounts","system:serviceaccounts:openshift-console-operator","system:authenticated"]},"sourceIPs":["10.0.0.4"],"userAgent":"console/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"routes","namespace":"openshift-console","name":"downloads","apiGroup":"route.openshift.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2020-04-28T22:59:16.746721Z","stageTimestamp":"2020-04-28T22:59:16.749569Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"console-operator/openshift-console\" of Role \"console-operator\" to ServiceAccount \"console-operator/openshift-console-operator\""}} to have prefix <string>: {"kind":"Event", The problem is with audit files loosing some of the data. I'll sync with api&auth team about this one.
fairly frequent per search results: https://search-clayton-ci-search.apps.build01.ci.devcluster.openshift.com/?search=+oc+adm+must-gather+runs+successfully+&maxAge=336h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job sippy analysis say this is failing 5% of the time.
Jan will look into it later this week.
This still requires further investigation and as such won't make 4.5 cut, moving to 4.6.
Checking https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-4.4/1545/artifacts/e2e-gcp/must-gather.tar, quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-6326b78d6c05d9925576ae8c6768f7ee846b8e99618da670b5e7b698e1dd433a/audit_logs/openshift-apiserver/ci-op-bhrb9-m-0.c.openshift-gce-devel-ci.internal-audit.log.gz contains the following line ``` etadata","auditID":"f938e80b-ed9e-4083-8dea-071fd3795fd6","stage":"ResponseComplete","requestURI":"/apis/route.openshift.io/v1/namespaces/openshift-console/routes/downloads","verb":"get","user":{"username":"system:serviceaccount:openshift-console-operator:console-operator","groups":["system:serviceaccounts","system:serviceaccounts:openshift-console-operator","system:authenticated"]},"sourceIPs":["10.0.0.4"],"userAgent":"console/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"routes","namespace":"openshift-console","name":"downloads","apiGroup":"route.openshift.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2020-04-28T22:59:16.746721Z","stageTimestamp":"2020-04-28T22:59:16.749569Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"console-operator/openshift-console\" of Role \"console-operator\" to ServiceAccount \"console-operator/openshift-console-operator\""}} ``` Although the must-gather payload from must-gather.tar is different from the one collect in "[cli] oc adm must-gather [Top Level] [cli] oc adm must-gather runs successfully", the same line and the same master correlate.
*** Bug 1844089 has been marked as a duplicate of this bug. ***
More time is requires to properly investigate the issue.
*** Bug 1852860 has been marked as a duplicate of this bug. ***
This test is consistently failing in aws jobs: https://search.apps.build01.ci.devcluster.openshift.com/?maxAge=168h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job&search=%5C%5Bsig-cli%5C%5D+oc+adm+must-gather+runs+successfully [sig-cli] oc adm must-gather runs successfully [Suite:openshift/conformance/parallel] expand_less Run #0: Failed expand_less 3m36s fail [github.com/openshift/origin/test/extended/cli/mustgather.go:128]: Expected <string>: /tmp/test.oc-adm-must-gather.925469404/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-3cf9e8cdd293580d40c6146362a81ff5c16173d838295ce00df905be55dfce7c/namespaces/openshift-kube-apiserver/core/configmaps.yaml to exist open stdoutopen_in_new Run #1: Failed expand_less 1m32s fail [github.com/openshift/origin/test/extended/cli/mustgather.go:128]: Expected <string>: /tmp/test.oc-adm-must-gather.381660880/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-3cf9e8cdd293580d40c6146362a81ff5c16173d838295ce00df905be55dfce7c/namespaces/openshift-kube-apiserver/core/configmaps.yaml to exist
Wasn't this failure blocking 4.6 release promotion [1]? I'm not clear on how to square "4.6 promotion blocker" with "medium priority and continued UpcomingSprint punts". Moving to urgent to match bug 1852860's urgent/urgent born-as values, and folks who feel like that's inappropriate for a 4.6 promotion blocker can motivate a return to medium if they feel that's needed ;). [1]: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.6-blocking#release-openshift-ocp-installer-e2e-aws-4.6
$ w3m -dump -cols 200 'https://search.svc.ci.openshift.org/?search=oc+adm+must-gather+runs+successfully&maxAge=24h&context=1&type=junit&name=release-openshift-ocp&groupBy=job' | grep 'failures match' release-openshift-ocp-installer-e2e-ovirt-4.6 - 9 runs, 100% failed, 33% of failures match release-openshift-ocp-installer-e2e-aws-4.6 - 32 runs, 100% failed, 44% of failures match release-openshift-ocp-installer-e2e-azure-4.6 - 20 runs, 80% failed, 38% of failures match release-openshift-ocp-installer-e2e-aws-ovn-4.6 - 2 runs, 100% failed, 50% of failures match release-openshift-ocp-installer-e2e-azure-ovn-4.6 - 2 runs, 100% failed, 50% of failures match release-openshift-ocp-installer-e2e-gcp-ovn-4.3 - 4 runs, 100% failed, 25% of failures match release-openshift-ocp-installer-e2e-ovirt-4.5 - 16 runs, 13% failed, 50% of failures match release-openshift-ocp-installer-e2e-gcp-4.6 - 4 runs, 100% failed, 25% of failures match release-openshift-ocp-installer-e2e-openstack-4.2 - 2 runs, 100% failed, 50% of failures match release-openshift-ocp-installer-e2e-metal-4.6 - 4 runs, 100% failed, 25% of failures match release-openshift-ocp-installer-e2e-aws-ovn-4.5 - 13 runs, 15% failed, 50% of failures match release-openshift-ocp-installer-e2e-metal-4.5 - 14 runs, 43% failed, 17% of failures match
TestGrid gives us broad brackets (unfortunately there seems to have been an overlapping '503 Service Unavailable' install error issue, e.g. [1]). Not sure if this is all that much help for narrowing down the culprit, but in case it is: $ diff -U0 <(curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.6/1276364681870577664/artifacts/release-images-latest/release-images-latest | jq -r '[.spec.tags[] | .name + " " + .annotations["io.openshift.build.source-location"] + "/commit/" + .annotations["io.openshift.build.commit.id"]] | sort[]') <(curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.6/1277756091807043584/artifacts/release-images-latest/release-images-latest | jq -r '[.spec.tags[] | .name + " " + .annotations["io.openshift.build.source-location"] + "/commit/" + .annotations["io.openshift.build.commit.id"]] | sort[]') --- /dev/fd/63 2020-07-02 14:40:03.520471477 -0700 +++ /dev/fd/62 2020-07-02 14:40:03.520471477 -0700 @@ -4 +4 @@ -baremetal-installer https://github.com/openshift/installer/commit/9c2cd4d3bcba017cd26cf31676546cec4741479b +baremetal-installer https://github.com/openshift/installer/commit/dd27e4cc4e2b433216dee8a0e9121c635a824de3 @@ -6 +6 @@ -baremetal-operator https://github.com/openshift/baremetal-operator/commit/68685ef951215c98a7cc781fc435b2b51ca86881 +baremetal-operator https://github.com/openshift/baremetal-operator/commit/5337e644bf969f2b2d1d4bf030df7f207eeef33b @@ -8,4 +8,4 @@ -cli https://github.com/openshift/oc/commit/7762806d2191ee4880307cf5973a1b7fe916d6d9 -cli-artifacts https://github.com/openshift/oc/commit/7762806d2191ee4880307cf5973a1b7fe916d6d9 -cloud-credential-operator https://github.com/openshift/cloud-credential-operator/commit/c2bec26d3734e444666024858e76d4ca80dd31cf -cluster-authentication-operator https://github.com/openshift/cluster-authentication-operator/commit/852cd7313ec9c4e8e1ba7b0353903a4fd9dd485f +cli https://github.com/openshift/oc/commit/ad8b00fb56b2e0263a08404df252ba9a87b0b872 +cli-artifacts https://github.com/openshift/oc/commit/ad8b00fb56b2e0263a08404df252ba9a87b0b872 +cloud-credential-operator https://github.com/openshift/cloud-credential-operator/commit/1f9a14c54d8f30fe3ecdc72a402c290dacace9f4 +cluster-authentication-operator https://github.com/openshift/cluster-authentication-operator/commit/feb8128aabc586d9b93ec5dd112209d421f822d4 @@ -20,3 +20,3 @@ -cluster-ingress-operator https://github.com/openshift/cluster-ingress-operator/commit/423217e7fe8e6002c81ad72295f392efcf108aa2 -cluster-kube-apiserver-operator https://github.com/openshift/cluster-kube-apiserver-operator/commit/f73bebb6361c3649dab5305d8c7d1cd9753e61aa -cluster-kube-controller-manager-operator https://github.com/openshift/cluster-kube-controller-manager-operator/commit/b5238d21b496f9026c7a9d0cc2bdf2ba2bc91962 +cluster-ingress-operator https://github.com/openshift/cluster-ingress-operator/commit/07b4a561e2abb6bbaa6f56750706168092403319 +cluster-kube-apiserver-operator https://github.com/openshift/cluster-kube-apiserver-operator/commit/0c87c5b74a7e96c93e2507848d8e704ffbeb48ef +cluster-kube-controller-manager-operator https://github.com/openshift/cluster-kube-controller-manager-operator/commit/436ac26c50264ff58b79163e2da32a24bd4007bd @@ -26,2 +26,2 @@ -cluster-monitoring-operator https://github.com/openshift/cluster-monitoring-operator/commit/d8c6e7756bf06955dd9b54c3e4540828aca2c98c -cluster-network-operator https://github.com/openshift/cluster-network-operator/commit/d4cccbfd530b3fb1e3c2a7ee81a286bb4fad5836 +cluster-monitoring-operator https://github.com/openshift/cluster-monitoring-operator/commit/3d25c0daaa9238cc351b71c15e5e6e6c565282ca +cluster-network-operator https://github.com/openshift/cluster-network-operator/commit/2f04083382a5a824aeca0adfc2b5a58bd34cbb80 @@ -29 +29 @@ -cluster-node-tuning-operator https://github.com/openshift/cluster-node-tuning-operator/commit/e030681be13ddeb36af00d8b75bb9b4de7303182 +cluster-node-tuning-operator https://github.com/openshift/cluster-node-tuning-operator/commit/96aed05be8384d7b65245bc7236052d2423294f3 @@ -33 +33 @@ -cluster-samples-operator https://github.com/openshift/cluster-samples-operator/commit/0038cc3bb2b67731b801c4514d4fadf3ecb23679 +cluster-samples-operator https://github.com/openshift/cluster-samples-operator/commit/d43c691d1a868d91a3a55e8670d21056fb46f02d @@ -38 +38 @@ -cluster-version-operator https://github.com/openshift/cluster-version-operator/commit/2b849292e49f1365ce1c84541194b736b76d7b18 +cluster-version-operator https://github.com/openshift/cluster-version-operator/commit/692a8a37b52c2bc00c45fb84622ba2d5ed3125bb @@ -40,2 +40,2 @@ -console https://github.com/openshift/console/commit/8ab4b7105152b9ede4ec0ebff722a4203c39c461 -console-operator https://github.com/openshift/console-operator/commit/1aba599397284e7fb79c30c3ddf3b2c44e01b4c0 +console https://github.com/openshift/console/commit/8c4d53107d8b3ab5bc39af575d04e923ab9fe769 +console-operator https://github.com/openshift/console-operator/commit/c47a8e386ed04b86531b8ed81e36e01f2c9728f9 @@ -45 +45 @@ -deployer https://github.com/openshift/oc/commit/7762806d2191ee4880307cf5973a1b7fe916d6d9 +deployer https://github.com/openshift/oc/commit/ad8b00fb56b2e0263a08404df252ba9a87b0b872 @@ -50 +50 @@ -grafana https://github.com/openshift/grafana/commit/4862ca28766c278d0294e2939351ec409dcca3d9 +grafana https://github.com/openshift/grafana/commit/3093e9a4c8dc74448141b18b57d2fbe3af564800 @@ -52,4 +52,4 @@ -hyperkube https://github.com/openshift/origin/commit/8871b3ddf225da6d41014674149d0c6623d29f55 -insights-operator https://github.com/openshift/insights-operator/commit/c6c067ac63ea2742dafd99b5572c218fafb69794 -installer https://github.com/openshift/installer/commit/9c2cd4d3bcba017cd26cf31676546cec4741479b -installer-artifacts https://github.com/openshift/installer/commit/9c2cd4d3bcba017cd26cf31676546cec4741479b +hyperkube https://github.com/openshift/origin/commit/5d8c7f115968f4e3cdc44e047be2585bec6f1e7e +insights-operator https://github.com/openshift/insights-operator/commit/2370641886fedf044dcd5c97959bd3946c75c13c +installer https://github.com/openshift/installer/commit/dd27e4cc4e2b433216dee8a0e9121c635a824de3 +installer-artifacts https://github.com/openshift/installer/commit/dd27e4cc4e2b433216dee8a0e9121c635a824de3 @@ -60 +60 @@ -ironic-machine-os-downloader https://github.com/openshift/ironic-rhcos-downloader/commit/561c7d4cb21fc2e5137f5919e5931f302a3cd383 +ironic-machine-os-downloader https://github.com/openshift/ironic-rhcos-downloader/commit/dfca19599cb3c5c4206613233cc7bc8667f90e74 @@ -62,3 +62,3 @@ -jenkins https://github.com/openshift/jenkins/commit/4b41eb3113c2da4904643454e2c3b87fad739493 -jenkins-agent-maven https://github.com/openshift/jenkins/commit/4b41eb3113c2da4904643454e2c3b87fad739493 -jenkins-agent-nodejs https://github.com/openshift/jenkins/commit/4b41eb3113c2da4904643454e2c3b87fad739493 +jenkins https://github.com/openshift/jenkins/commit/6d8fcdf822f8f5ed07ef50d5fb4c6198024d6be2 +jenkins-agent-maven https://github.com/openshift/jenkins/commit/6d8fcdf822f8f5ed07ef50d5fb4c6198024d6be2 +jenkins-agent-nodejs https://github.com/openshift/jenkins/commit/6d8fcdf822f8f5ed07ef50d5fb4c6198024d6be2 @@ -66 +66 @@ -keepalived-ipfailover https://github.com/openshift/images/commit/6cdb5d360768d8f87a615286180e46784ae7d28f +keepalived-ipfailover https://github.com/openshift/images/commit/98fd27ea391c2d0df19d238ca000f75980aea2aa @@ -73,2 +73,2 @@ -kuryr-cni https://github.com/openshift/kuryr-kubernetes/commit/03413eb0c4e06d357ee9c0c75387285383e3c9ec -kuryr-controller https://github.com/openshift/kuryr-kubernetes/commit/03413eb0c4e06d357ee9c0c75387285383e3c9ec +kuryr-cni https://github.com/openshift/kuryr-kubernetes/commit/cef2a957a2d364214a3c3364b9417d89f1cbb607 +kuryr-controller https://github.com/openshift/kuryr-kubernetes/commit/cef2a957a2d364214a3c3364b9417d89f1cbb607 @@ -77,2 +77,2 @@ -machine-api-operator https://github.com/openshift/machine-api-operator/commit/1e858a423db8b5ff829b5e7b8e54be1717330e1b -machine-config-operator https://github.com/openshift/machine-config-operator/commit/0f33cc64182f6cb69609064e235235214411b648 +machine-api-operator https://github.com/openshift/machine-api-operator/commit/168f5015e9b88dc9dbcdee1e9f6180c766470e9a +machine-config-operator https://github.com/openshift/machine-config-operator/commit/915ca7a11fead41d75ffa0ed7e310f7284645a13 @@ -86 +86 @@ -network-metrics-daemon https://github.com/openshift/network-metrics-daemon/commit/9aa9aa5d67711761706f8222c9a5bbbfdecab155 +network-metrics-daemon https://github.com/openshift/network-metrics-daemon/commit/f61fbf6b1e82becf35f003fee6973a982d21c8fb @@ -92,2 +92,2 @@ -openstack-machine-controllers https://github.com/openshift/cluster-api-provider-openstack/commit/97593603aaf290e000e08df1c641a69a109ccda0 -operator-lifecycle-manager https://github.com/operator-framework/operator-lifecycle-manager/commit/a90b83a4bd74bd6e7a135d66b8e6474788057744 +openstack-machine-controllers https://github.com/openshift/cluster-api-provider-openstack/commit/e517210af2932f052d97ef9eb9f50d06150c3deb +operator-lifecycle-manager https://github.com/operator-framework/operator-lifecycle-manager/commit/afd23480a756d8db5e7ce08869ae882dcdba0ed8 @@ -95 +95 @@ -operator-registry https://github.com/operator-framework/operator-registry/commit/275301b779f8c059c733afd8e6a01c071aee9c22 +operator-registry https://github.com/operator-framework/operator-registry/commit/7d251ac37b0959b609487a8f58cdc3fb81e2316f @@ -97,2 +97,2 @@ -ovn-kubernetes https://github.com/openshift/ovn-kubernetes/commit/1997ea6b71af51d167797533eff8413e060a9dfd -pod https://github.com/openshift/images/commit/6cdb5d360768d8f87a615286180e46784ae7d28f +ovn-kubernetes https://github.com/openshift/ovn-kubernetes/commit/d41bfd48e768fac2ea32ae2711999ffa2b01d5be +pod https://github.com/openshift/images/commit/98fd27ea391c2d0df19d238ca000f75980aea2aa @@ -100 +100 @@ -prometheus https://github.com/openshift/prometheus/commit/b95ba6250ba46710f9d5e2c4aae7721c75ee2fc3 +prometheus https://github.com/openshift/prometheus/commit/8a0dc90e5a01eb1aa719d4645961c90b8d2828e6 @@ -105 +105 @@ -sdn https://github.com/openshift/sdn/commit/37b23906d5b479f288cc881a8468d6bc5c164d78 +sdn https://github.com/openshift/sdn/commit/d6dea6f36106b9cebe1c8fb3d512b1706787a527 @@ -107,2 +107,2 @@ -telemeter https://github.com/openshift/telemeter/commit/e801ab5a64b15c42f8a72e16f43a6b4fa654fd4a -tests https://github.com/openshift/origin/commit/8871b3ddf225da6d41014674149d0c6623d29f55 +telemeter https://github.com/openshift/telemeter/commit/7413e5f8141ba1580f70130046388ffafee1f28e +tests https://github.com/openshift/origin/commit/5d8c7f115968f4e3cdc44e047be2585bec6f1e7e @@ -110 +110 @@ -tools https://github.com/openshift/oc/commit/7762806d2191ee4880307cf5973a1b7fe916d6d9 +tools https://github.com/openshift/oc/commit/ad8b00fb56b2e0263a08404df252ba9a87b0b872 [1]: https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.6/1277462551726133248
Maybe Azure can give tighter brackets [1]. $ diff -U0 <(curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-azure-4.6/1278504032784093184/artifacts/release-images-latest/release-images-latest | jq -r '[.spec.tags[] | .name + " " + .annotations["io.openshift.build.source-location"] + "/commit/" + .annotations["io.openshift.build.commit.id"]] | sort[]') <(curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-azure-4.6/1278539183564001280/artifacts/release-images-latest/release-images-latest | jq -r '[.spec.tags[] | .name + " " + .annotations["io.openshift.build.source-location"] + "/commit/" + .annotations["io.openshift.build.commit.id"]] | sort[]') --- /dev/fd/63 2020-07-02 14:47:07.632048475 -0700 +++ /dev/fd/62 2020-07-02 14:47:07.633048493 -0700 @@ -52 +52 @@ -hyperkube https://github.com/openshift/origin/commit/6cafe416a2d8dd0ffdd4b0bd7a37606194737e92 +hyperkube https://github.com/openshift/origin/commit/537bb810681838fb5dd20b4f80d028b7952c94ec @@ -78 +78 @@ -machine-config-operator https://github.com/openshift/machine-config-operator/commit/c38f3123e4ce12ec288a38b2dd47f261dcd9f694 +machine-config-operator https://github.com/openshift/machine-config-operator/commit/fd7594321058670dc9ec8fceac987b5c32b88e54 @@ -85 +85 @@ -must-gather https://github.com/openshift/must-gather/commit/0dde9773019a9155af657d45a3feb759e620df98 +must-gather https://github.com/openshift/must-gather/commit/34d2ef6c1f662921192dd6439211349f388e9a48 @@ -109 +109 @@ -tests https://github.com/openshift/origin/commit/6cafe416a2d8dd0ffdd4b0bd7a37606194737e92 +tests https://github.com/openshift/origin/commit/537bb810681838fb5dd20b4f80d028b7952c94ec [1]: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.6-informing#release-openshift-ocp-installer-e2e-azure-4.6
Azure diff looks completely irrelevant (only must-gather change is [1], and AWS brackets had no must-gather changes at all). Digging into [2], it fails on: fail [github.com/openshift/origin/test/extended/cli/mustgather.go:128]: Expected <string>: /tmp/test.oc-adm-must-gather.554005973/registry-svc-ci-openshift-org-ocp-4-6-2020-07-02-182149-sha256-73b71590b8ed9733a3e095805343a06d343738ce9614f7c9b61ddb46ceaedb15/namespaces/openshift-kube-apiserver/core/configmaps.yaml to exist like all of these. Looking at the post-test gather: $ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-azure-4.6/1278756056272474112/artifacts/e2e-azure/must-gather.tar | tar -tz | grep namespaces/openshift-kube-apiserver/core ./registry-svc-ci-openshift-org-ocp-4-6-2020-07-02-182149-sha256-73b71590b8ed9733a3e095805343a06d343738ce9614f7c9b61ddb46ceaedb15/namespaces/openshift-kube-apiserver/core/ ./registry-svc-ci-openshift-org-ocp-4-6-2020-07-02-182149-sha256-73b71590b8ed9733a3e095805343a06d343738ce9614f7c9b61ddb46ceaedb15/namespaces/openshift-kube-apiserver/core/pods.yaml ./registry-svc-ci-openshift-org-ocp-4-6-2020-07-02-182149-sha256-73b71590b8ed9733a3e095805343a06d343738ce9614f7c9b61ddb46ceaedb15/namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml ./registry-svc-ci-openshift-org-ocp-4-6-2020-07-02-182149-sha256-73b71590b8ed9733a3e095805343a06d343738ce9614f7c9b61ddb46ceaedb15/namespaces/openshift-kube-apiserver/core/services.yaml Huh. No configmaps.yaml there either. We do have some configmaps: $ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-azure-4.6/1278756056272474112/artifacts/e2e-azure/must-gather.tar | tar -tz | grep configmap ./registry-svc-ci-openshift-org-ocp-4-6-2020-07-02-182149-sha256-73b71590b8ed9733a3e095805343a06d343738ce9614f7c9b61ddb46ceaedb15/namespaces/openshift-config-managed/core/configmaps/ ./registry-svc-ci-openshift-org-ocp-4-6-2020-07-02-182149-sha256-73b71590b8ed9733a3e095805343a06d343738ce9614f7c9b61ddb46ceaedb15/namespaces/openshift-config-managed/core/configmaps/console-public.yaml ... Here's the log from the post-test gather: $ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-azure-4.6/1278756056272474112/artifacts/e2e-azure/must-gather.tar | tar -xOz ./must-gather.log | grep configmaps [must-gather-fp6bq] OUT namespaces/openshift-config-managed/core/configmaps/ [must-gather-fp6bq] OUT namespaces/openshift-config-managed/core/configmaps/console-public.yaml [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-0/rules-configmap-reloader/ [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-0/rules-configmap-reloader/rules-configmap-reloader/ [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-0/rules-configmap-reloader/rules-configmap-reloader/logs/ [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-0/rules-configmap-reloader/rules-configmap-reloader/logs/current.log [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-0/rules-configmap-reloader/rules-configmap-reloader/logs/previous.insecure.log [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-0/rules-configmap-reloader/rules-configmap-reloader/logs/previous.log [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-1/rules-configmap-reloader/ [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-1/rules-configmap-reloader/rules-configmap-reloader/ [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-1/rules-configmap-reloader/rules-configmap-reloader/logs/ [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-1/rules-configmap-reloader/rules-configmap-reloader/logs/current.log [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-1/rules-configmap-reloader/rules-configmap-reloader/logs/previous.insecure.log [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-1/rules-configmap-reloader/rules-configmap-reloader/logs/previous.log [must-gather-fp6bq] OUT namespaces/openshift-multus/core/configmaps/ [must-gather-fp6bq] OUT namespaces/openshift-multus/core/configmaps/cni-binary-copy-script.yaml [must-gather-fp6bq] OUT namespaces/openshift-network-operator/core/configmaps/ [must-gather-fp6bq] OUT namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml [must-gather-fp6bq] OUT namespaces/openshift-network-operator/core/configmaps/openshift-service-ca.yaml [must-gather-fp6bq] OUT namespaces/openshift-sdn/core/configmaps/ [must-gather-fp6bq] OUT namespaces/openshift-sdn/core/configmaps/sdn-config.yaml Is this as basic as "the test suite was looking at a namespace that never had any ConfigMaps, and now that we just collect specific maps and not a listing configmaps.yaml, the test is failing? [1]: https://github.com/openshift/must-gather/pull/157/files [2]: https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-azure-4.6/1278756056272474112
From before the break: $ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.6/1276364681870577664/artifacts/e2e-aws/must-gather.tar | tar -xOz ./quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-9237929065fb53cf031f2989e290bc34bb9b1695bbf4a2834d77da4b09c25afc/namespaces/openshift-kube-apiserver/core/configmaps.yaml | yaml2json | jq -r '.items[].metadata.name' aggregator-client-ca bound-sa-token-signing-certs bound-sa-token-signing-certs-2 ... So unless those ConfigMaps have moved elsewhere, this is really just "must-gather/oc broke their ConfigMap collection, and seriously hindering org progress in the meantime".
*** Bug 1853612 has been marked as a duplicate of this bug. ***
Checked from testGrid , no issue found from recently test: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.6-informing#release-openshift-ocp-installer-e2e-gcp-4.6
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196