Bug 1829243 - test: [cli] oc adm must-gather [Top Level] [cli] oc adm must-gather runs successfully [Suite:openshift/conformance/parallel]
Summary: test: [cli] oc adm must-gather [Top Level] [cli] oc adm must-gather runs succ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oc
Version: 4.4
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.6.0
Assignee: Jan Chaloupka
QA Contact: zhou ying
URL:
Whiteboard:
: 1844089 1852860 1853612 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-29 09:42 UTC by Periklis Tsirakidis
Modified: 2020-10-27 15:58 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
test: [cli] oc adm must-gather [Top Level] [cli] oc adm must-gather runs successfully [Suite:openshift/conformance/parallel] [sig-cli] oc adm must-gather runs successfully
Last Closed: 2020-10-27 15:58:32 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift oc pull 484 0 None closed bug 1829243: Revert "run inspect in parallel" 2021-01-17 09:00:24 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 15:58:56 UTC

Description Periklis Tsirakidis 2020-04-29 09:42:17 UTC
test: [cli] oc adm must-gather [Top Level] [cli] oc adm must-gather runs successfully [Suite:openshift/conformance/parallel] failed, see job: 

https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-4.4/1545


Error:
[skipping gathering routes.route.openshift.io/oauth-openshift due to error: a resource cannot be retrieved by name across all namespaces, skipping gathering services/oauth-openshift due to error: a resource cannot be retrieved by name across all namespaces, skipping gathering machineautoscalers.machine.openshift.io due to error: the server doesn't have a resource type "machineautoscalers", skipping gathering clusterautoscalers.machine.openshift.io due to error: the server doesn't have a resource type "clusterautoscalers", skipping gathering roles.rbac.authorization.k8s.io/cloud-provider-config-reader due to error: roles.rbac.authorization.k8s.io "cloud-provider-config-reader" not found, skipping gathering all/openshift-monitoring due to error: the server doesn't have a resource type "all", skipping gathering namespaces/openshift-service-catalog-apiserver due to error: namespaces "openshift-service-catalog-apiserver" not found, skipping gathering apiservices.apiregistration.k8s.io/v1beta1.servicecatalog.k8s.io due to error: apiservices.apiregistration.k8s.io "v1beta1.servicecatalog.k8s.io" not found, skipping gathering namespaces/openshift-service-catalog-controller-manager due to error: namespaces "openshift-service-catalog-controller-manager" not found]

Comment 1 Maciej Szulik 2020-04-29 11:45:42 UTC
The actual bug is:

fail [github.com/openshift/origin/test/extended/cli/mustgather.go:150]: Expected
    <string>: etadata","auditID":"f938e80b-ed9e-4083-8dea-071fd3795fd6","stage":"ResponseComplete","requestURI":"/apis/route.openshift.io/v1/namespaces/openshift-console/routes/downloads","verb":"get","user":{"username":"system:serviceaccount:openshift-console-operator:console-operator","groups":["system:serviceaccounts","system:serviceaccounts:openshift-console-operator","system:authenticated"]},"sourceIPs":["10.0.0.4"],"userAgent":"console/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"routes","namespace":"openshift-console","name":"downloads","apiGroup":"route.openshift.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2020-04-28T22:59:16.746721Z","stageTimestamp":"2020-04-28T22:59:16.749569Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"console-operator/openshift-console\" of Role \"console-operator\" to ServiceAccount \"console-operator/openshift-console-operator\""}}
to have prefix
    <string>: {"kind":"Event",

The problem is with audit files loosing some of the data. I'll sync with api&auth team about this one.

Comment 3 Maciej Szulik 2020-05-19 12:03:36 UTC
Jan will look into it later this week.

Comment 4 Maciej Szulik 2020-05-25 11:59:17 UTC
This still requires further investigation and as such won't make 4.5 cut, moving to 4.6.

Comment 5 Jan Chaloupka 2020-06-03 12:12:59 UTC
Checking https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-4.4/1545/artifacts/e2e-gcp/must-gather.tar, quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-6326b78d6c05d9925576ae8c6768f7ee846b8e99618da670b5e7b698e1dd433a/audit_logs/openshift-apiserver/ci-op-bhrb9-m-0.c.openshift-gce-devel-ci.internal-audit.log.gz contains the following line

```
etadata","auditID":"f938e80b-ed9e-4083-8dea-071fd3795fd6","stage":"ResponseComplete","requestURI":"/apis/route.openshift.io/v1/namespaces/openshift-console/routes/downloads","verb":"get","user":{"username":"system:serviceaccount:openshift-console-operator:console-operator","groups":["system:serviceaccounts","system:serviceaccounts:openshift-console-operator","system:authenticated"]},"sourceIPs":["10.0.0.4"],"userAgent":"console/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"routes","namespace":"openshift-console","name":"downloads","apiGroup":"route.openshift.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2020-04-28T22:59:16.746721Z","stageTimestamp":"2020-04-28T22:59:16.749569Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"console-operator/openshift-console\" of Role \"console-operator\" to ServiceAccount \"console-operator/openshift-console-operator\""}}
```

Although the must-gather payload from must-gather.tar is different from the one collect in "[cli] oc adm must-gather [Top Level] [cli] oc adm must-gather runs successfully", the same line and the same master correlate.

Comment 7 Maciej Szulik 2020-06-05 14:22:05 UTC
*** Bug 1844089 has been marked as a duplicate of this bug. ***

Comment 8 Jan Chaloupka 2020-06-18 10:22:08 UTC
More time is requires to properly investigate the issue.

Comment 9 Maciej Szulik 2020-07-02 08:18:21 UTC
*** Bug 1852860 has been marked as a duplicate of this bug. ***

Comment 10 Alberto 2020-07-02 08:37:40 UTC
This test is consistently failing in aws jobs:
https://search.apps.build01.ci.devcluster.openshift.com/?maxAge=168h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job&search=%5C%5Bsig-cli%5C%5D+oc+adm+must-gather+runs+successfully


[sig-cli] oc adm must-gather runs successfully [Suite:openshift/conformance/parallel] expand_less
Run #0: Failed expand_less	3m36s
fail [github.com/openshift/origin/test/extended/cli/mustgather.go:128]: Expected
    <string>: /tmp/test.oc-adm-must-gather.925469404/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-3cf9e8cdd293580d40c6146362a81ff5c16173d838295ce00df905be55dfce7c/namespaces/openshift-kube-apiserver/core/configmaps.yaml
to exist
open stdoutopen_in_new
Run #1: Failed expand_less	1m32s
fail [github.com/openshift/origin/test/extended/cli/mustgather.go:128]: Expected
    <string>: /tmp/test.oc-adm-must-gather.381660880/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-3cf9e8cdd293580d40c6146362a81ff5c16173d838295ce00df905be55dfce7c/namespaces/openshift-kube-apiserver/core/configmaps.yaml
to exist

Comment 11 W. Trevor King 2020-07-02 16:14:20 UTC
Wasn't this failure blocking 4.6 release promotion [1]?  I'm not clear on how to square "4.6 promotion blocker" with "medium priority and continued UpcomingSprint punts".  Moving to urgent to match bug 1852860's urgent/urgent born-as values, and folks who feel like that's inappropriate for a 4.6 promotion blocker can motivate a return to medium if they feel that's needed ;).

[1]: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.6-blocking#release-openshift-ocp-installer-e2e-aws-4.6

Comment 12 W. Trevor King 2020-07-02 21:34:27 UTC
$ w3m -dump -cols 200 'https://search.svc.ci.openshift.org/?search=oc+adm+must-gather+runs+successfully&maxAge=24h&context=1&type=junit&name=release-openshift-ocp&groupBy=job' | grep 'failures match'
release-openshift-ocp-installer-e2e-ovirt-4.6 - 9 runs, 100% failed, 33% of failures match
release-openshift-ocp-installer-e2e-aws-4.6 - 32 runs, 100% failed, 44% of failures match
release-openshift-ocp-installer-e2e-azure-4.6 - 20 runs, 80% failed, 38% of failures match
release-openshift-ocp-installer-e2e-aws-ovn-4.6 - 2 runs, 100% failed, 50% of failures match
release-openshift-ocp-installer-e2e-azure-ovn-4.6 - 2 runs, 100% failed, 50% of failures match
release-openshift-ocp-installer-e2e-gcp-ovn-4.3 - 4 runs, 100% failed, 25% of failures match
release-openshift-ocp-installer-e2e-ovirt-4.5 - 16 runs, 13% failed, 50% of failures match
release-openshift-ocp-installer-e2e-gcp-4.6 - 4 runs, 100% failed, 25% of failures match
release-openshift-ocp-installer-e2e-openstack-4.2 - 2 runs, 100% failed, 50% of failures match
release-openshift-ocp-installer-e2e-metal-4.6 - 4 runs, 100% failed, 25% of failures match
release-openshift-ocp-installer-e2e-aws-ovn-4.5 - 13 runs, 15% failed, 50% of failures match
release-openshift-ocp-installer-e2e-metal-4.5 - 14 runs, 43% failed, 17% of failures match

Comment 13 W. Trevor King 2020-07-02 21:42:25 UTC
TestGrid gives us broad brackets (unfortunately there seems to have been an overlapping '503 Service Unavailable' install error issue, e.g. [1]).  Not sure if this is all that much help for narrowing down the culprit, but in case it is:

$ diff -U0 <(curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.6/1276364681870577664/artifacts/release-images-latest/release-images-latest | jq -r '[.spec.tags[] | .name + " " + .annotations["io.openshift.build.source-location"] + "/commit/" + .annotations["io.openshift.build.commit.id"]] | sort[]') <(curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.6/1277756091807043584/artifacts/release-images-latest/release-images-latest | jq -r '[.spec.tags[] | .name + " " + .annotations["io.openshift.build.source-location"] + "/commit/" + .annotations["io.openshift.build.commit.id"]] | sort[]')
--- /dev/fd/63	2020-07-02 14:40:03.520471477 -0700
+++ /dev/fd/62	2020-07-02 14:40:03.520471477 -0700
@@ -4 +4 @@
-baremetal-installer https://github.com/openshift/installer/commit/9c2cd4d3bcba017cd26cf31676546cec4741479b
+baremetal-installer https://github.com/openshift/installer/commit/dd27e4cc4e2b433216dee8a0e9121c635a824de3
@@ -6 +6 @@
-baremetal-operator https://github.com/openshift/baremetal-operator/commit/68685ef951215c98a7cc781fc435b2b51ca86881
+baremetal-operator https://github.com/openshift/baremetal-operator/commit/5337e644bf969f2b2d1d4bf030df7f207eeef33b
@@ -8,4 +8,4 @@
-cli https://github.com/openshift/oc/commit/7762806d2191ee4880307cf5973a1b7fe916d6d9
-cli-artifacts https://github.com/openshift/oc/commit/7762806d2191ee4880307cf5973a1b7fe916d6d9
-cloud-credential-operator https://github.com/openshift/cloud-credential-operator/commit/c2bec26d3734e444666024858e76d4ca80dd31cf
-cluster-authentication-operator https://github.com/openshift/cluster-authentication-operator/commit/852cd7313ec9c4e8e1ba7b0353903a4fd9dd485f
+cli https://github.com/openshift/oc/commit/ad8b00fb56b2e0263a08404df252ba9a87b0b872
+cli-artifacts https://github.com/openshift/oc/commit/ad8b00fb56b2e0263a08404df252ba9a87b0b872
+cloud-credential-operator https://github.com/openshift/cloud-credential-operator/commit/1f9a14c54d8f30fe3ecdc72a402c290dacace9f4
+cluster-authentication-operator https://github.com/openshift/cluster-authentication-operator/commit/feb8128aabc586d9b93ec5dd112209d421f822d4
@@ -20,3 +20,3 @@
-cluster-ingress-operator https://github.com/openshift/cluster-ingress-operator/commit/423217e7fe8e6002c81ad72295f392efcf108aa2
-cluster-kube-apiserver-operator https://github.com/openshift/cluster-kube-apiserver-operator/commit/f73bebb6361c3649dab5305d8c7d1cd9753e61aa
-cluster-kube-controller-manager-operator https://github.com/openshift/cluster-kube-controller-manager-operator/commit/b5238d21b496f9026c7a9d0cc2bdf2ba2bc91962
+cluster-ingress-operator https://github.com/openshift/cluster-ingress-operator/commit/07b4a561e2abb6bbaa6f56750706168092403319
+cluster-kube-apiserver-operator https://github.com/openshift/cluster-kube-apiserver-operator/commit/0c87c5b74a7e96c93e2507848d8e704ffbeb48ef
+cluster-kube-controller-manager-operator https://github.com/openshift/cluster-kube-controller-manager-operator/commit/436ac26c50264ff58b79163e2da32a24bd4007bd
@@ -26,2 +26,2 @@
-cluster-monitoring-operator https://github.com/openshift/cluster-monitoring-operator/commit/d8c6e7756bf06955dd9b54c3e4540828aca2c98c
-cluster-network-operator https://github.com/openshift/cluster-network-operator/commit/d4cccbfd530b3fb1e3c2a7ee81a286bb4fad5836
+cluster-monitoring-operator https://github.com/openshift/cluster-monitoring-operator/commit/3d25c0daaa9238cc351b71c15e5e6e6c565282ca
+cluster-network-operator https://github.com/openshift/cluster-network-operator/commit/2f04083382a5a824aeca0adfc2b5a58bd34cbb80
@@ -29 +29 @@
-cluster-node-tuning-operator https://github.com/openshift/cluster-node-tuning-operator/commit/e030681be13ddeb36af00d8b75bb9b4de7303182
+cluster-node-tuning-operator https://github.com/openshift/cluster-node-tuning-operator/commit/96aed05be8384d7b65245bc7236052d2423294f3
@@ -33 +33 @@
-cluster-samples-operator https://github.com/openshift/cluster-samples-operator/commit/0038cc3bb2b67731b801c4514d4fadf3ecb23679
+cluster-samples-operator https://github.com/openshift/cluster-samples-operator/commit/d43c691d1a868d91a3a55e8670d21056fb46f02d
@@ -38 +38 @@
-cluster-version-operator https://github.com/openshift/cluster-version-operator/commit/2b849292e49f1365ce1c84541194b736b76d7b18
+cluster-version-operator https://github.com/openshift/cluster-version-operator/commit/692a8a37b52c2bc00c45fb84622ba2d5ed3125bb
@@ -40,2 +40,2 @@
-console https://github.com/openshift/console/commit/8ab4b7105152b9ede4ec0ebff722a4203c39c461
-console-operator https://github.com/openshift/console-operator/commit/1aba599397284e7fb79c30c3ddf3b2c44e01b4c0
+console https://github.com/openshift/console/commit/8c4d53107d8b3ab5bc39af575d04e923ab9fe769
+console-operator https://github.com/openshift/console-operator/commit/c47a8e386ed04b86531b8ed81e36e01f2c9728f9
@@ -45 +45 @@
-deployer https://github.com/openshift/oc/commit/7762806d2191ee4880307cf5973a1b7fe916d6d9
+deployer https://github.com/openshift/oc/commit/ad8b00fb56b2e0263a08404df252ba9a87b0b872
@@ -50 +50 @@
-grafana https://github.com/openshift/grafana/commit/4862ca28766c278d0294e2939351ec409dcca3d9
+grafana https://github.com/openshift/grafana/commit/3093e9a4c8dc74448141b18b57d2fbe3af564800
@@ -52,4 +52,4 @@
-hyperkube https://github.com/openshift/origin/commit/8871b3ddf225da6d41014674149d0c6623d29f55
-insights-operator https://github.com/openshift/insights-operator/commit/c6c067ac63ea2742dafd99b5572c218fafb69794
-installer https://github.com/openshift/installer/commit/9c2cd4d3bcba017cd26cf31676546cec4741479b
-installer-artifacts https://github.com/openshift/installer/commit/9c2cd4d3bcba017cd26cf31676546cec4741479b
+hyperkube https://github.com/openshift/origin/commit/5d8c7f115968f4e3cdc44e047be2585bec6f1e7e
+insights-operator https://github.com/openshift/insights-operator/commit/2370641886fedf044dcd5c97959bd3946c75c13c
+installer https://github.com/openshift/installer/commit/dd27e4cc4e2b433216dee8a0e9121c635a824de3
+installer-artifacts https://github.com/openshift/installer/commit/dd27e4cc4e2b433216dee8a0e9121c635a824de3
@@ -60 +60 @@
-ironic-machine-os-downloader https://github.com/openshift/ironic-rhcos-downloader/commit/561c7d4cb21fc2e5137f5919e5931f302a3cd383
+ironic-machine-os-downloader https://github.com/openshift/ironic-rhcos-downloader/commit/dfca19599cb3c5c4206613233cc7bc8667f90e74
@@ -62,3 +62,3 @@
-jenkins https://github.com/openshift/jenkins/commit/4b41eb3113c2da4904643454e2c3b87fad739493
-jenkins-agent-maven https://github.com/openshift/jenkins/commit/4b41eb3113c2da4904643454e2c3b87fad739493
-jenkins-agent-nodejs https://github.com/openshift/jenkins/commit/4b41eb3113c2da4904643454e2c3b87fad739493
+jenkins https://github.com/openshift/jenkins/commit/6d8fcdf822f8f5ed07ef50d5fb4c6198024d6be2
+jenkins-agent-maven https://github.com/openshift/jenkins/commit/6d8fcdf822f8f5ed07ef50d5fb4c6198024d6be2
+jenkins-agent-nodejs https://github.com/openshift/jenkins/commit/6d8fcdf822f8f5ed07ef50d5fb4c6198024d6be2
@@ -66 +66 @@
-keepalived-ipfailover https://github.com/openshift/images/commit/6cdb5d360768d8f87a615286180e46784ae7d28f
+keepalived-ipfailover https://github.com/openshift/images/commit/98fd27ea391c2d0df19d238ca000f75980aea2aa
@@ -73,2 +73,2 @@
-kuryr-cni https://github.com/openshift/kuryr-kubernetes/commit/03413eb0c4e06d357ee9c0c75387285383e3c9ec
-kuryr-controller https://github.com/openshift/kuryr-kubernetes/commit/03413eb0c4e06d357ee9c0c75387285383e3c9ec
+kuryr-cni https://github.com/openshift/kuryr-kubernetes/commit/cef2a957a2d364214a3c3364b9417d89f1cbb607
+kuryr-controller https://github.com/openshift/kuryr-kubernetes/commit/cef2a957a2d364214a3c3364b9417d89f1cbb607
@@ -77,2 +77,2 @@
-machine-api-operator https://github.com/openshift/machine-api-operator/commit/1e858a423db8b5ff829b5e7b8e54be1717330e1b
-machine-config-operator https://github.com/openshift/machine-config-operator/commit/0f33cc64182f6cb69609064e235235214411b648
+machine-api-operator https://github.com/openshift/machine-api-operator/commit/168f5015e9b88dc9dbcdee1e9f6180c766470e9a
+machine-config-operator https://github.com/openshift/machine-config-operator/commit/915ca7a11fead41d75ffa0ed7e310f7284645a13
@@ -86 +86 @@
-network-metrics-daemon https://github.com/openshift/network-metrics-daemon/commit/9aa9aa5d67711761706f8222c9a5bbbfdecab155
+network-metrics-daemon https://github.com/openshift/network-metrics-daemon/commit/f61fbf6b1e82becf35f003fee6973a982d21c8fb
@@ -92,2 +92,2 @@
-openstack-machine-controllers https://github.com/openshift/cluster-api-provider-openstack/commit/97593603aaf290e000e08df1c641a69a109ccda0
-operator-lifecycle-manager https://github.com/operator-framework/operator-lifecycle-manager/commit/a90b83a4bd74bd6e7a135d66b8e6474788057744
+openstack-machine-controllers https://github.com/openshift/cluster-api-provider-openstack/commit/e517210af2932f052d97ef9eb9f50d06150c3deb
+operator-lifecycle-manager https://github.com/operator-framework/operator-lifecycle-manager/commit/afd23480a756d8db5e7ce08869ae882dcdba0ed8
@@ -95 +95 @@
-operator-registry https://github.com/operator-framework/operator-registry/commit/275301b779f8c059c733afd8e6a01c071aee9c22
+operator-registry https://github.com/operator-framework/operator-registry/commit/7d251ac37b0959b609487a8f58cdc3fb81e2316f
@@ -97,2 +97,2 @@
-ovn-kubernetes https://github.com/openshift/ovn-kubernetes/commit/1997ea6b71af51d167797533eff8413e060a9dfd
-pod https://github.com/openshift/images/commit/6cdb5d360768d8f87a615286180e46784ae7d28f
+ovn-kubernetes https://github.com/openshift/ovn-kubernetes/commit/d41bfd48e768fac2ea32ae2711999ffa2b01d5be
+pod https://github.com/openshift/images/commit/98fd27ea391c2d0df19d238ca000f75980aea2aa
@@ -100 +100 @@
-prometheus https://github.com/openshift/prometheus/commit/b95ba6250ba46710f9d5e2c4aae7721c75ee2fc3
+prometheus https://github.com/openshift/prometheus/commit/8a0dc90e5a01eb1aa719d4645961c90b8d2828e6
@@ -105 +105 @@
-sdn https://github.com/openshift/sdn/commit/37b23906d5b479f288cc881a8468d6bc5c164d78
+sdn https://github.com/openshift/sdn/commit/d6dea6f36106b9cebe1c8fb3d512b1706787a527
@@ -107,2 +107,2 @@
-telemeter https://github.com/openshift/telemeter/commit/e801ab5a64b15c42f8a72e16f43a6b4fa654fd4a
-tests https://github.com/openshift/origin/commit/8871b3ddf225da6d41014674149d0c6623d29f55
+telemeter https://github.com/openshift/telemeter/commit/7413e5f8141ba1580f70130046388ffafee1f28e
+tests https://github.com/openshift/origin/commit/5d8c7f115968f4e3cdc44e047be2585bec6f1e7e
@@ -110 +110 @@
-tools https://github.com/openshift/oc/commit/7762806d2191ee4880307cf5973a1b7fe916d6d9
+tools https://github.com/openshift/oc/commit/ad8b00fb56b2e0263a08404df252ba9a87b0b872

[1]: https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.6/1277462551726133248

Comment 14 W. Trevor King 2020-07-02 21:47:35 UTC
Maybe Azure can give tighter brackets [1].

$ diff -U0 <(curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-azure-4.6/1278504032784093184/artifacts/release-images-latest/release-images-latest | jq -r '[.spec.tags[] | .name + " " + .annotations["io.openshift.build.source-location"] + "/commit/" + .annotations["io.openshift.build.commit.id"]] | sort[]') <(curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-azure-4.6/1278539183564001280/artifacts/release-images-latest/release-images-latest | jq -r '[.spec.tags[] | .name + " " + .annotations["io.openshift.build.source-location"] + "/commit/" + .annotations["io.openshift.build.commit.id"]] | sort[]')
--- /dev/fd/63	2020-07-02 14:47:07.632048475 -0700
+++ /dev/fd/62	2020-07-02 14:47:07.633048493 -0700
@@ -52 +52 @@
-hyperkube https://github.com/openshift/origin/commit/6cafe416a2d8dd0ffdd4b0bd7a37606194737e92
+hyperkube https://github.com/openshift/origin/commit/537bb810681838fb5dd20b4f80d028b7952c94ec
@@ -78 +78 @@
-machine-config-operator https://github.com/openshift/machine-config-operator/commit/c38f3123e4ce12ec288a38b2dd47f261dcd9f694
+machine-config-operator https://github.com/openshift/machine-config-operator/commit/fd7594321058670dc9ec8fceac987b5c32b88e54
@@ -85 +85 @@
-must-gather https://github.com/openshift/must-gather/commit/0dde9773019a9155af657d45a3feb759e620df98
+must-gather https://github.com/openshift/must-gather/commit/34d2ef6c1f662921192dd6439211349f388e9a48
@@ -109 +109 @@
-tests https://github.com/openshift/origin/commit/6cafe416a2d8dd0ffdd4b0bd7a37606194737e92
+tests https://github.com/openshift/origin/commit/537bb810681838fb5dd20b4f80d028b7952c94ec

[1]: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.6-informing#release-openshift-ocp-installer-e2e-azure-4.6

Comment 15 W. Trevor King 2020-07-02 22:02:40 UTC
Azure diff looks completely irrelevant (only must-gather change is [1], and AWS brackets had no must-gather changes at all).  Digging into [2], it fails on:

  fail [github.com/openshift/origin/test/extended/cli/mustgather.go:128]: Expected
    <string>: /tmp/test.oc-adm-must-gather.554005973/registry-svc-ci-openshift-org-ocp-4-6-2020-07-02-182149-sha256-73b71590b8ed9733a3e095805343a06d343738ce9614f7c9b61ddb46ceaedb15/namespaces/openshift-kube-apiserver/core/configmaps.yaml
  to exist

like all of these.  Looking at the post-test gather:

  $ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-azure-4.6/1278756056272474112/artifacts/e2e-azure/must-gather.tar | tar -tz | grep namespaces/openshift-kube-apiserver/core
  ./registry-svc-ci-openshift-org-ocp-4-6-2020-07-02-182149-sha256-73b71590b8ed9733a3e095805343a06d343738ce9614f7c9b61ddb46ceaedb15/namespaces/openshift-kube-apiserver/core/
  ./registry-svc-ci-openshift-org-ocp-4-6-2020-07-02-182149-sha256-73b71590b8ed9733a3e095805343a06d343738ce9614f7c9b61ddb46ceaedb15/namespaces/openshift-kube-apiserver/core/pods.yaml
  ./registry-svc-ci-openshift-org-ocp-4-6-2020-07-02-182149-sha256-73b71590b8ed9733a3e095805343a06d343738ce9614f7c9b61ddb46ceaedb15/namespaces/openshift-kube-apiserver/core/replicationcontrollers.yaml
  ./registry-svc-ci-openshift-org-ocp-4-6-2020-07-02-182149-sha256-73b71590b8ed9733a3e095805343a06d343738ce9614f7c9b61ddb46ceaedb15/namespaces/openshift-kube-apiserver/core/services.yaml

Huh.  No configmaps.yaml there either.  We do have some configmaps:

  $ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-azure-4.6/1278756056272474112/artifacts/e2e-azure/must-gather.tar | tar -tz | grep configmap
  ./registry-svc-ci-openshift-org-ocp-4-6-2020-07-02-182149-sha256-73b71590b8ed9733a3e095805343a06d343738ce9614f7c9b61ddb46ceaedb15/namespaces/openshift-config-managed/core/configmaps/
  ./registry-svc-ci-openshift-org-ocp-4-6-2020-07-02-182149-sha256-73b71590b8ed9733a3e095805343a06d343738ce9614f7c9b61ddb46ceaedb15/namespaces/openshift-config-managed/core/configmaps/console-public.yaml
  ...

Here's the log from the post-test gather:

  $ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-azure-4.6/1278756056272474112/artifacts/e2e-azure/must-gather.tar | tar -xOz ./must-gather.log | grep configmaps
  [must-gather-fp6bq] OUT namespaces/openshift-config-managed/core/configmaps/
  [must-gather-fp6bq] OUT namespaces/openshift-config-managed/core/configmaps/console-public.yaml
  [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-0/rules-configmap-reloader/
  [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-0/rules-configmap-reloader/rules-configmap-reloader/
  [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-0/rules-configmap-reloader/rules-configmap-reloader/logs/
  [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-0/rules-configmap-reloader/rules-configmap-reloader/logs/current.log
  [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-0/rules-configmap-reloader/rules-configmap-reloader/logs/previous.insecure.log
  [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-0/rules-configmap-reloader/rules-configmap-reloader/logs/previous.log
  [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-1/rules-configmap-reloader/
  [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-1/rules-configmap-reloader/rules-configmap-reloader/
  [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-1/rules-configmap-reloader/rules-configmap-reloader/logs/
  [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-1/rules-configmap-reloader/rules-configmap-reloader/logs/current.log
  [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-1/rules-configmap-reloader/rules-configmap-reloader/logs/previous.insecure.log
  [must-gather-fp6bq] OUT namespaces/openshift-monitoring/pods/prometheus-k8s-1/rules-configmap-reloader/rules-configmap-reloader/logs/previous.log
  [must-gather-fp6bq] OUT namespaces/openshift-multus/core/configmaps/
  [must-gather-fp6bq] OUT namespaces/openshift-multus/core/configmaps/cni-binary-copy-script.yaml
  [must-gather-fp6bq] OUT namespaces/openshift-network-operator/core/configmaps/
  [must-gather-fp6bq] OUT namespaces/openshift-network-operator/core/configmaps/applied-cluster.yaml
  [must-gather-fp6bq] OUT namespaces/openshift-network-operator/core/configmaps/openshift-service-ca.yaml
  [must-gather-fp6bq] OUT namespaces/openshift-sdn/core/configmaps/
  [must-gather-fp6bq] OUT namespaces/openshift-sdn/core/configmaps/sdn-config.yaml

Is this as basic as "the test suite was looking at a namespace that never had any ConfigMaps, and now that we just collect specific maps and not a listing configmaps.yaml, the test is failing?


[1]: https://github.com/openshift/must-gather/pull/157/files
[2]: https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-azure-4.6/1278756056272474112

Comment 16 W. Trevor King 2020-07-02 22:06:36 UTC
From before the break:

  $ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.6/1276364681870577664/artifacts/e2e-aws/must-gather.tar | tar -xOz ./quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-9237929065fb53cf031f2989e290bc34bb9b1695bbf4a2834d77da4b09c25afc/namespaces/openshift-kube-apiserver/core/configmaps.yaml | yaml2json | jq -r '.items[].metadata.name'
  aggregator-client-ca
  bound-sa-token-signing-certs
  bound-sa-token-signing-certs-2
  ...

So unless those ConfigMaps have moved elsewhere, this is really just "must-gather/oc broke their ConfigMap collection, and seriously hindering org progress in the meantime".

Comment 17 Maciej Szulik 2020-07-06 10:58:10 UTC
*** Bug 1853612 has been marked as a duplicate of this bug. ***

Comment 20 zhou ying 2020-07-13 08:16:34 UTC
Checked from testGrid , no issue found from recently test:

https://testgrid.k8s.io/redhat-openshift-ocp-release-4.6-informing#release-openshift-ocp-installer-e2e-gcp-4.6

Comment 22 errata-xmlrpc 2020-10-27 15:58:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.