Description of problem: Creating CRDs hangs there for hours. $ wget https://raw.githubusercontent.com/operator-framework/operator-lifecycle-manager/master/deploy/ocp/manifests/0.15.1/0000_50_olm_00-clusterserviceversions.crd.yaml $ ./oc --loglevel 8 apply -f 0000_50_olm_00-clusterserviceversions.crd.yaml ... I0628 04:54:34.485840 1730 request.go:1068] Response Body: {"kind":"CustomResourceDefinition","apiVersion":"apiextensions.k8s.io/v1","metadata":{"name":"clusterserviceversions.operators.coreos.com","selfLink":"/apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusterserviceversions.operators.coreos.com","uid":"37defd90-d137-43df-98cd-c1ca432c9186","resourceVersion":"30944","generation":1,"creationTimestamp":"2020-06-28T07:39:30Z","annotations":{"controller-gen.kubebuilder.io/version":"v0.3.0"},"managedFields":[{"manager":"kube-apiserver","operation":"Update","apiVersion":"apiextensions.k8s.io/v1","time":"2020-06-28T07:39:30Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:acceptedNames":{"f:categories":{},"f:kind":{},"f:listKind":{},"f:plural":{},"f:shortNames":{},"f:singular":{}},"f:conditions":{}}}},{"manager":"kubectl","operation":"Update","apiVersion":"apiextensions.k8s.io/v1","time":"2020-06-28T07:39:30Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:controller-gen.kubebuilder.io/version":{}}},"f:spec":{"f:conversion":{". [truncated 221575 chars] Warning: oc apply should be used on resource created by either oc create --save-config or oc apply Version-Release number of selected component (if applicable): [root@preserve-olm-env data]# ./oc version Client Version: 4.5.0-rc.4 Server Version: 4.5.0-0.nightly-2020-06-26-215024 Kubernetes Version: v1.18.3+f291db1 How reproducible: always Steps to Reproduce: 1. Install the OCP 4.5. This clusterserviceversions CRD already exist in the cluster as default. [root@preserve-olm-env data]# ./oc get crd|grep clusterserviceversions clusterserviceversions.operators.coreos.com 2020-06-28T09:15:22Z 2. Install this clusterserviceversions CRD again. $ wget https://raw.githubusercontent.com/operator-framework/operator-lifecycle-manager/master/deploy/ocp/manifests/0.15.1/0000_50_olm_00-clusterserviceversions.crd.yaml Actual results: It hangs there for a long time. $ ./oc --loglevel 8 apply -f 0000_50_olm_00-clusterserviceversions.crd.yaml ... I0628 04:54:34.485840 1730 request.go:1068] Response Body: {"kind":"CustomResourceDefinition","apiVersion":"apiextensions.k8s.io/v1","metadata":{"name":"clusterserviceversions.operators.coreos.com","selfLink":"/apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusterserviceversions.operators.coreos.com","uid":"37defd90-d137-43df-98cd-c1ca432c9186","resourceVersion":"30944","generation":1,"creationTimestamp":"2020-06-28T07:39:30Z","annotations":{"controller-gen.kubebuilder.io/version":"v0.3.0"},"managedFields":[{"manager":"kube-apiserver","operation":"Update","apiVersion":"apiextensions.k8s.io/v1","time":"2020-06-28T07:39:30Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:acceptedNames":{"f:categories":{},"f:kind":{},"f:listKind":{},"f:plural":{},"f:shortNames":{},"f:singular":{}},"f:conditions":{}}}},{"manager":"kubectl","operation":"Update","apiVersion":"apiextensions.k8s.io/v1","time":"2020-06-28T07:39:30Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:controller-gen.kubebuilder.io/version":{}}},"f:spec":{"f:conversion":{". [truncated 221575 chars] Warning: oc apply should be used on resource created by either oc create --save-config or oc apply Expected results: The `oc apply` should return errors immediately if anything wrong instead of hanging there. For example, # ./oc create -f 0000_50_olm_00-clusterserviceversions.crd.yaml Error from server (AlreadyExists): error when creating "0000_50_olm_00-clusterserviceversions.crd.yaml": customresourcedefinitions.apiextensions.k8s.io "clusterserviceversions.operators.coreos.com" already exists Additional info: 1, The `oc create` works well. 2, The `oc apply` works well after removing the exist clusterserviceversions CRD 3, Check the kube-controller-manager logs but didn't get any errors: [root@preserve-olm-env data]# oc logs kube-controller-manager-ip-10-0-186-87.us-east-2.compute.internal -n openshift-kube-controller-manager -c kube-controller-manager |grep clusterserviceversions.operators.coreos.com I0628 07:23:25.734631 1 resource_quota_monitor.go:228] QuotaMonitor created object count evaluator for clusterserviceversions.operators.coreos.com ... [], removed: [operators.coreos.com/v1, Resource=operatorgroups operators.coreos.com/v1alpha1, Resource=catalogsources operators.coreos.com/v1alpha1, Resource=clusterserviceversions operators.coreos.com/v1alpha1, Resource=installplans operators.coreos.com/v1alpha1, Resource=subscriptions] I0628 07:39:29.660946 1 garbagecollector.go:204] syncing garbage collector with updated resources from discovery (attempt 1): added: [], removed: [operators.coreos.com/v1, Resource=operatorgroups operators.coreos.com/v1alpha1, Resource=catalogsources operators.coreos.com/v1alpha1, Resource=clusterserviceversions operators.coreos.com/v1alpha1, Resource=installplans operators.coreos.com/v1alpha1, Resource=subscriptions] I0628 07:39:59.524013 1 resource_quota_controller.go:434] syncing resource quota controller with updated resources from discovery: added: [operators.coreos.com/v1, Resource=operatorgroups operators.coreos.com/v1alpha1, Resource=catalogsources operators.coreos.com/v1alpha1, Resource=clusterserviceversions operators.coreos.com/v1alpha1, Resource=installplans operators.coreos.com/v1alpha1, Resource=subscriptions], removed: [] I0628 07:39:59.684951 1 garbagecollector.go:204] syncing garbage collector with updated resources from discovery (attempt 1): added: [operators.coreos.com/v1, Resource=operatorgroups operators.coreos.com/v1alpha1, Resource=catalogsources operators.coreos.com/v1alpha1, Resource=clusterserviceversions operators.coreos.com/v1alpha1, Resource=installplans operators.coreos.com/v1alpha1, Resource=subscriptions], removed: []
From my quick debugging session it looks like github.com/evanphx/json-patch library is stuck when creating a merge patch on a CRD specifically. I have a few possible suspicions: - a bug in a library, we might need to upgrade it - the size of data which might be causing the merge patch calculation complicated - semi-related, the managed fields are affecting the previous
I’m adding UpcomingSprint, because I was occupied by fixing bugs with higher priority/severity, developing new features with higher priority, or developing new features to improve stability at a macro level. I will revisit this bug next sprint.
Actually https://github.com/kubernetes/kubernetes/pull/92069 will be part of the next k8s bump so we should get it then.
This was fixed in https://github.com/openshift/oc/pull/491
Below is the oc version with which the bug was verified. [ramakasturinarra@dhcp35-60 ~]$ oc version -o yaml clientVersion: buildDate: "2020-08-21T02:37:08Z" compiler: gc gitCommit: ea0d54068621ec0f95973068729f739f3dacfef7 gitTreeState: clean gitVersion: 4.6.0-202008210209.p0-ea0d540 goVersion: go1.14.4 major: "" minor: "" platform: linux/amd64 openshiftVersion: 4.6.0-0.nightly-2020-08-23-214712 serverVersion: buildDate: "2020-08-20T16:46:57Z" compiler: gc gitCommit: 3e083ac29409923906267ebcc5f8e0aa13072c72 gitTreeState: dirty gitVersion: v1.19.0-rc.2+3e083ac-dirty goVersion: go1.14.4 major: "1" minor: 19+ platform: linux/amd64
With $ oc version Client Version: 4.6.0-0.ci-2020-08-24-152549 Server Version: 4.6.0-0.ci-2020-08-24-152549 Kubernetes Version: v1.19.0-rc.2.467+3e083ac2940992-dirty this is working as expected, w/o any errors or delays.
From my tests it looks like only this particular CRD is having issues, I'm pushing this over to 4.7
I've re-checked this bug again just now. Tested both below versions - original and latest available: - https://raw.githubusercontent.com/operator-framework/operator-lifecycle-manager/master/deploy/ocp/manifests/0.15.1/0000_50_olm_00-clusterserviceversions.crd.yaml - https://raw.githubusercontent.com/operator-framework/operator-lifecycle-manager/master/deploy/ocp/manifests/0.17.0/0000_50_olm_00-clusterserviceversions.crd.yaml I've tried running multiple times each and works as expected. Moving to qa for verification.
Verified bug with the payload below and i see that oc apply does not hang anymore when run more than once. [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-08-225909]$ ./oc version Client Version: 4.7.0-0.nightly-2020-11-08-225909 Server Version: 4.7.0-0.nightly-2020-10-27-051128 Kubernetes Version: v1.19.0+e67f5dc Below are the steps followed to verify: ===================================== 1) Install 4.7 cluster 2) Now run the command below which says Already Exists as expected [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-08-225909]$ ./oc create -f 0000_50_olm_00-clusterserviceversions.crd.yaml Error from server (AlreadyExists): error when creating "0000_50_olm_00-clusterserviceversions.crd.yaml": customresourcedefinitions.apiextensions.k8s.io "clusterserviceversions.operators.coreos.com" already exists 3) Now run the command oc apply thirce and i did not any hang as reported in the description. [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-08-225909]$ ./oc --loglevel 8 apply -f 0000_50_olm_00-clusterserviceversions.crd.yaml I1109 19:11:33.990035 12019 loader.go:375] Config loaded from file: /home/knarra/Downloads/kubeconfig_47 I1109 19:11:33.990652 12019 round_trippers.go:420] GET https://api.knarra1109.qe.devcluster.openshift.com:6443/openapi/v2?timeout=32s I1109 19:11:33.990658 12019 round_trippers.go:427] Request Headers: I1109 19:11:33.990661 12019 round_trippers.go:431] Accept: application/com.github.proto-openapi.spec.v2+protobuf I1109 19:11:33.990664 12019 round_trippers.go:431] User-Agent: oc/4.7.0 (linux/amd64) kubernetes/9f54c1d I1109 19:11:33.990667 12019 round_trippers.go:431] Authorization: Bearer <masked> I1109 19:11:35.894003 12019 round_trippers.go:446] Response Status: 200 OK in 1903 milliseconds I1109 19:11:35.894035 12019 round_trippers.go:449] Response Headers: I1109 19:11:35.894040 12019 round_trippers.go:452] Audit-Id: 618d5d2d-0d48-4b34-9509-0c2ec0c64f10 I1109 19:11:35.894043 12019 round_trippers.go:452] Cache-Control: no-cache, private I1109 19:11:35.894059 12019 round_trippers.go:452] Content-Type: application/octet-stream I1109 19:11:35.894062 12019 round_trippers.go:452] X-Kubernetes-Pf-Prioritylevel-Uid: 1517604c-7276-4328-ad28-1565abb05931 I1109 19:11:35.894065 12019 round_trippers.go:452] Date: Mon, 09 Nov 2020 13:41:35 GMT I1109 19:11:35.894067 12019 round_trippers.go:452] Accept-Ranges: bytes I1109 19:11:35.894070 12019 round_trippers.go:452] X-Varied-Accept: application/com.github.proto-openapi.spec.v2+protobuf I1109 19:11:35.894073 12019 round_trippers.go:452] Etag: "C6440351D4A49BB350D54AEED20945D882975D7BC61165A5A2A43E38AD2A9AA7F8937FB5CF25A11A222B178AEB8C5B3B2221164CFBE6295408A3EDB11CDFD14C" I1109 19:11:35.894075 12019 round_trippers.go:452] Last-Modified: Mon, 09 Nov 2020 10:34:38 GMT I1109 19:11:35.894078 12019 round_trippers.go:452] Vary: Accept-Encoding I1109 19:11:35.894085 12019 round_trippers.go:452] Vary: Accept I1109 19:11:35.894089 12019 round_trippers.go:452] X-Kubernetes-Pf-Flowschema-Uid: 202c5c05-54a2-4b02-ade2-8f30ecb59d3e I1109 19:11:40.958146 12019 request.go:1095] Response Body: 00000000 0a 03 32 2e 30 12 1d 0a 0a 4b 75 62 65 72 6e 65 |..2.0....Kuberne| 00000010 74 65 73 12 0f 76 31 2e 31 39 2e 30 2b 65 36 37 |tes..v1.19.0+e67| 00000020 66 35 64 63 42 b7 a3 bc 02 12 fa 55 0a 2c 2f 61 |f5dcB......U.,/a| 00000030 70 69 73 2f 63 6f 6e 66 69 67 2e 6f 70 65 6e 73 |pis/config.opens| 00000040 68 69 66 74 2e 69 6f 2f 76 31 2f 69 6e 66 72 61 |hift.io/v1/infra| 00000050 73 74 72 75 63 74 75 72 65 73 12 c9 55 12 d3 25 |structures..U..%| 00000060 0a 14 63 6f 6e 66 69 67 4f 70 65 6e 73 68 69 66 |..configOpenshif| 00000070 74 49 6f 5f 76 31 1a 23 6c 69 73 74 20 6f 62 6a |tIo_v1.#list obj| 00000080 65 63 74 73 20 6f 66 20 6b 69 6e 64 20 49 6e 66 |ects of kind Inf| 00000090 72 61 73 74 72 75 63 74 75 72 65 2a 25 6c 69 73 |rastructure*%lis| 000000a0 74 43 6f 6e 66 69 67 4f 70 65 6e 73 68 69 66 74 |tConfigOpenshift| 000000b0 49 6f 56 31 49 6e 66 72 61 73 74 72 75 63 74 75 |IoV1Infrastructu| 000000c0 72 65 32 10 61 70 70 6c 69 63 61 74 69 6f 6e 2f |re2.application [truncated 37022160 chars] I1109 19:11:41.074655 12019 round_trippers.go:420] GET https://api.knarra1109.qe.devcluster.openshift.com:6443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusterserviceversions.operators.coreos.com I1109 19:11:41.074670 12019 round_trippers.go:427] Request Headers: I1109 19:11:41.074693 12019 round_trippers.go:431] Accept: application/json I1109 19:11:41.074698 12019 round_trippers.go:431] Authorization: Bearer <masked> I1109 19:11:41.074702 12019 round_trippers.go:431] User-Agent: oc/4.7.0 (linux/amd64) kubernetes/9f54c1d I1109 19:11:41.627646 12019 round_trippers.go:446] Response Status: 200 OK in 552 milliseconds I1109 19:11:41.627659 12019 round_trippers.go:449] Response Headers: I1109 19:11:41.627663 12019 round_trippers.go:452] Cache-Control: no-cache, private I1109 19:11:41.627666 12019 round_trippers.go:452] Content-Type: application/json I1109 19:11:41.627669 12019 round_trippers.go:452] Vary: Accept-Encoding I1109 19:11:41.627672 12019 round_trippers.go:452] X-Kubernetes-Pf-Flowschema-Uid: 202c5c05-54a2-4b02-ade2-8f30ecb59d3e I1109 19:11:41.627677 12019 round_trippers.go:452] X-Kubernetes-Pf-Prioritylevel-Uid: 1517604c-7276-4328-ad28-1565abb05931 I1109 19:11:41.627681 12019 round_trippers.go:452] Date: Mon, 09 Nov 2020 13:41:41 GMT I1109 19:11:41.627684 12019 round_trippers.go:452] Audit-Id: ec36d70e-ec26-44e9-a497-7315bb1b0459 I1109 19:11:41.629078 12019 request.go:1097] Response Body: {"kind":"CustomResourceDefinition","apiVersion":"apiextensions.k8s.io/v1","metadata":{"name":"clusterserviceversions.operators.coreos.com","selfLink":"/apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusterserviceversions.operators.coreos.com","uid":"0bc8b63f-7424-4650-bf93-b3c9103e8bec","resourceVersion":"898","generation":1,"creationTimestamp":"2020-11-09T10:11:23Z","annotations":{"controller-gen.kubebuilder.io/version":"v0.3.0"},"managedFields":[{"manager":"cluster-version-operator","operation":"Update","apiVersion":"apiextensions.k8s.io/v1","time":"2020-11-09T10:11:23Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:controller-gen.kubebuilder.io/version":{}}},"f:spec":{"f:conversion":{".":{},"f:strategy":{}},"f:group":{},"f:names":{"f:categories":{},"f:kind":{},"f:listKind":{},"f:plural":{},"f:shortNames":{},"f:singular":{}},"f:scope":{},"f:versions":{}},"f:status":{"f:storedVersions":{}}}},{"manager":"kube-apiserver","operation":"Update","apiVersion":"apiextensions.k [truncated 221908 chars] Warning: oc apply should be used on resource created by either oc create --save-config or oc apply I1109 19:11:41.741674 12019 request.go:1097] Request Body: {"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{\"controller-gen.kubebuilder.io/version\":\"v0.3.0\"},\"creationTimestamp\":null,\"name\":\"clusterserviceversions.operators.coreos.com\"},\"spec\":{\"group\":\"operators.coreos.com\",\"names\":{\"categories\":[\"olm\"],\"kind\":\"ClusterServiceVersion\",\"listKind\":\"ClusterServiceVersionList\",\"plural\":\"clusterserviceversions\",\"shortNames\":[\"csv\",\"csvs\"],\"singular\":\"clusterserviceversion\"},\"scope\":\"Namespaced\",\"versions\":[{\"additionalPrinterColumns\":[{\"description\":\"The name of the CSV\",\"jsonPath\":\".spec.displayName\",\"name\":\"Display\",\"type\":\"string\"},{\"description\":\"The version of the CSV\",\"jsonPath\":\".spec.version\",\"name\":\"Version\",\"type\":\"string\"},{\"description\":\"The name of a CSV that this one replaces\",\"jsonPath\":\".spec.replaces\",\"name\":\"Replaces\", [truncated 455864 chars] I1109 19:11:41.741734 12019 round_trippers.go:420] PATCH https://api.knarra1109.qe.devcluster.openshift.com:6443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusterserviceversions.operators.coreos.com?fieldManager=kubectl-client-side-apply I1109 19:11:41.741738 12019 round_trippers.go:427] Request Headers: I1109 19:11:41.741741 12019 round_trippers.go:431] Content-Type: application/merge-patch+json I1109 19:11:41.741746 12019 round_trippers.go:431] User-Agent: oc/4.7.0 (linux/amd64) kubernetes/9f54c1d I1109 19:11:41.741750 12019 round_trippers.go:431] Authorization: Bearer <masked> I1109 19:11:41.741753 12019 round_trippers.go:431] Accept: application/json I1109 19:11:48.127176 12019 round_trippers.go:446] Response Status: 200 OK in 6385 milliseconds I1109 19:11:48.127195 12019 round_trippers.go:449] Response Headers: I1109 19:11:48.127200 12019 round_trippers.go:452] Content-Type: application/json I1109 19:11:48.127222 12019 round_trippers.go:452] Vary: Accept-Encoding I1109 19:11:48.127227 12019 round_trippers.go:452] X-Kubernetes-Pf-Flowschema-Uid: 202c5c05-54a2-4b02-ade2-8f30ecb59d3e I1109 19:11:48.127246 12019 round_trippers.go:452] X-Kubernetes-Pf-Prioritylevel-Uid: 1517604c-7276-4328-ad28-1565abb05931 I1109 19:11:48.127251 12019 round_trippers.go:452] Date: Mon, 09 Nov 2020 13:41:47 GMT I1109 19:11:48.127256 12019 round_trippers.go:452] Audit-Id: 2524efe4-e7d1-4abb-9c30-d915ea5d9e41 I1109 19:11:48.127261 12019 round_trippers.go:452] Cache-Control: no-cache, private I1109 19:11:49.207545 12019 request.go:1097] Response Body: {"kind":"CustomResourceDefinition","apiVersion":"apiextensions.k8s.io/v1","metadata":{"name":"clusterserviceversions.operators.coreos.com","selfLink":"/apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusterserviceversions.operators.coreos.com","uid":"0bc8b63f-7424-4650-bf93-b3c9103e8bec","resourceVersion":"78886","generation":2,"creationTimestamp":"2020-11-09T10:11:23Z","annotations":{"controller-gen.kubebuilder.io/version":"v0.3.0","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{\"controller-gen.kubebuilder.io/version\":\"v0.3.0\"},\"creationTimestamp\":null,\"name\":\"clusterserviceversions.operators.coreos.com\"},\"spec\":{\"group\":\"operators.coreos.com\",\"names\":{\"categories\":[\"olm\"],\"kind\":\"ClusterServiceVersion\",\"listKind\":\"ClusterServiceVersionList\",\"plural\":\"clusterserviceversions\",\"shortNames\":[\"csv\",\"csvs\"],\"singular\":\"clusterserviceversion\"},\"sc [truncated 458122 chars] customresourcedefinition.apiextensions.k8s.io/clusterserviceversions.operators.coreos.com configured I1109 19:11:49.214419 12019 apply.go:390] Running apply post-processor function [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-08-225909]$ ./oc --loglevel 8 apply -f 0000_50_olm_00-clusterserviceversions.crd.yaml I1109 19:11:51.461551 12045 loader.go:375] Config loaded from file: /home/knarra/Downloads/kubeconfig_47 I1109 19:11:51.462176 12045 round_trippers.go:420] GET https://api.knarra1109.qe.devcluster.openshift.com:6443/openapi/v2?timeout=32s I1109 19:11:51.462181 12045 round_trippers.go:427] Request Headers: I1109 19:11:51.462184 12045 round_trippers.go:431] Accept: application/com.github.proto-openapi.spec.v2+protobuf I1109 19:11:51.462187 12045 round_trippers.go:431] User-Agent: oc/4.7.0 (linux/amd64) kubernetes/9f54c1d I1109 19:11:51.462190 12045 round_trippers.go:431] Authorization: Bearer <masked> I1109 19:11:57.351712 12045 round_trippers.go:446] Response Status: 200 OK in 5889 milliseconds I1109 19:11:57.351728 12045 round_trippers.go:449] Response Headers: I1109 19:11:57.351734 12045 round_trippers.go:452] X-Varied-Accept: application/com.github.proto-openapi.spec.v2+protobuf I1109 19:11:57.351738 12045 round_trippers.go:452] Accept-Ranges: bytes I1109 19:11:57.351742 12045 round_trippers.go:452] Content-Type: application/octet-stream I1109 19:11:57.351746 12045 round_trippers.go:452] Etag: "A85F76D2788113642E47D3786CBC0A9D007BF06CE007706AA0C4FE14A8DD9C8728A17057ABFFA813614446BF24F5468C30A4C9F85422948D00A0474E8621D54D" I1109 19:11:57.351751 12045 round_trippers.go:452] Last-Modified: Mon, 09 Nov 2020 13:41:52 GMT I1109 19:11:57.351756 12045 round_trippers.go:452] X-Kubernetes-Pf-Flowschema-Uid: 202c5c05-54a2-4b02-ade2-8f30ecb59d3e I1109 19:11:57.351760 12045 round_trippers.go:452] X-Kubernetes-Pf-Prioritylevel-Uid: 1517604c-7276-4328-ad28-1565abb05931 I1109 19:11:57.351764 12045 round_trippers.go:452] Audit-Id: e4ff55e2-5386-4c12-b925-8823710d52a5 I1109 19:11:57.351769 12045 round_trippers.go:452] Cache-Control: no-cache, private I1109 19:11:57.351772 12045 round_trippers.go:452] Vary: Accept-Encoding I1109 19:11:57.351776 12045 round_trippers.go:452] Vary: Accept I1109 19:11:57.351779 12045 round_trippers.go:452] Date: Mon, 09 Nov 2020 13:41:56 GMT E1109 19:12:23.462574 12045 request.go:1001] Unexpected error when reading response body: context deadline exceeded (Client.Timeout or context cancellation while reading body) I1109 19:12:23.492906 12045 round_trippers.go:420] GET https://api.knarra1109.qe.devcluster.openshift.com:6443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusterserviceversions.operators.coreos.com I1109 19:12:23.492923 12045 round_trippers.go:427] Request Headers: I1109 19:12:23.492927 12045 round_trippers.go:431] Authorization: Bearer <masked> I1109 19:12:23.492930 12045 round_trippers.go:431] Accept: application/json I1109 19:12:23.492932 12045 round_trippers.go:431] User-Agent: oc/4.7.0 (linux/amd64) kubernetes/9f54c1d I1109 19:12:26.830988 12045 round_trippers.go:446] Response Status: 200 OK in 3338 milliseconds I1109 19:12:26.831004 12045 round_trippers.go:449] Response Headers: I1109 19:12:26.831008 12045 round_trippers.go:452] X-Kubernetes-Pf-Flowschema-Uid: 202c5c05-54a2-4b02-ade2-8f30ecb59d3e I1109 19:12:26.831012 12045 round_trippers.go:452] X-Kubernetes-Pf-Prioritylevel-Uid: 1517604c-7276-4328-ad28-1565abb05931 I1109 19:12:26.831014 12045 round_trippers.go:452] Date: Mon, 09 Nov 2020 13:42:23 GMT I1109 19:12:26.831018 12045 round_trippers.go:452] Audit-Id: 35ddb16e-052e-45d9-a5e8-75a6c9b18bc1 I1109 19:12:26.831021 12045 round_trippers.go:452] Cache-Control: no-cache, private I1109 19:12:26.831024 12045 round_trippers.go:452] Content-Type: application/json I1109 19:12:26.831027 12045 round_trippers.go:452] Vary: Accept-Encoding I1109 19:12:30.372487 12045 request.go:1097] Response Body: {"kind":"CustomResourceDefinition","apiVersion":"apiextensions.k8s.io/v1","metadata":{"name":"clusterserviceversions.operators.coreos.com","selfLink":"/apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusterserviceversions.operators.coreos.com","uid":"0bc8b63f-7424-4650-bf93-b3c9103e8bec","resourceVersion":"79023","generation":3,"creationTimestamp":"2020-11-09T10:11:23Z","annotations":{"controller-gen.kubebuilder.io/version":"v0.3.0","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{\"controller-gen.kubebuilder.io/version\":\"v0.3.0\"},\"creationTimestamp\":null,\"name\":\"clusterserviceversions.operators.coreos.com\"},\"spec\":{\"group\":\"operators.coreos.com\",\"names\":{\"categories\":[\"olm\"],\"kind\":\"ClusterServiceVersion\",\"listKind\":\"ClusterServiceVersionList\",\"plural\":\"clusterserviceversions\",\"shortNames\":[\"csv\",\"csvs\"],\"singular\":\"clusterserviceversion\"},\"sc [truncated 458429 chars] I1109 19:12:30.476594 12045 request.go:1097] Request Body: {"spec":{"versions":[{"additionalPrinterColumns":[{"description":"The name of the CSV","jsonPath":".spec.displayName","name":"Display","type":"string"},{"description":"The version of the CSV","jsonPath":".spec.version","name":"Version","type":"string"},{"description":"The name of a CSV that this one replaces","jsonPath":".spec.replaces","name":"Replaces","type":"string"},{"jsonPath":".status.phase","name":"Phase","type":"string"}],"name":"v1alpha1","schema":{"openAPIV3Schema":{"description":"ClusterServiceVersion is a Custom Resource of type `ClusterServiceVersionSpec`.","properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object repres [truncated 219580 chars] I1109 19:12:30.476661 12045 round_trippers.go:420] PATCH https://api.knarra1109.qe.devcluster.openshift.com:6443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusterserviceversions.operators.coreos.com?fieldManager=kubectl-client-side-apply I1109 19:12:30.476666 12045 round_trippers.go:427] Request Headers: I1109 19:12:30.476671 12045 round_trippers.go:431] Authorization: Bearer <masked> I1109 19:12:30.476675 12045 round_trippers.go:431] Accept: application/json I1109 19:12:30.476696 12045 round_trippers.go:431] Content-Type: application/merge-patch+json I1109 19:12:30.476700 12045 round_trippers.go:431] User-Agent: oc/4.7.0 (linux/amd64) kubernetes/9f54c1d I1109 19:12:35.810922 12045 round_trippers.go:446] Response Status: 200 OK in 5334 milliseconds I1109 19:12:35.810936 12045 round_trippers.go:449] Response Headers: I1109 19:12:35.810940 12045 round_trippers.go:452] X-Kubernetes-Pf-Flowschema-Uid: 202c5c05-54a2-4b02-ade2-8f30ecb59d3e I1109 19:12:35.810943 12045 round_trippers.go:452] X-Kubernetes-Pf-Prioritylevel-Uid: 1517604c-7276-4328-ad28-1565abb05931 I1109 19:12:35.810945 12045 round_trippers.go:452] Date: Mon, 09 Nov 2020 13:42:35 GMT I1109 19:12:35.810947 12045 round_trippers.go:452] Audit-Id: 38687b5c-1b1b-483a-91be-f029a5311035 I1109 19:12:35.810950 12045 round_trippers.go:452] Cache-Control: no-cache, private I1109 19:12:35.810952 12045 round_trippers.go:452] Content-Type: application/json I1109 19:12:35.810954 12045 round_trippers.go:452] Vary: Accept-Encoding I1109 19:12:46.633881 12045 request.go:1097] Response Body: {"kind":"CustomResourceDefinition","apiVersion":"apiextensions.k8s.io/v1","metadata":{"name":"clusterserviceversions.operators.coreos.com","selfLink":"/apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusterserviceversions.operators.coreos.com","uid":"0bc8b63f-7424-4650-bf93-b3c9103e8bec","resourceVersion":"79122","generation":4,"creationTimestamp":"2020-11-09T10:11:23Z","annotations":{"controller-gen.kubebuilder.io/version":"v0.3.0","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{\"controller-gen.kubebuilder.io/version\":\"v0.3.0\"},\"creationTimestamp\":null,\"name\":\"clusterserviceversions.operators.coreos.com\"},\"spec\":{\"group\":\"operators.coreos.com\",\"names\":{\"categories\":[\"olm\"],\"kind\":\"ClusterServiceVersion\",\"listKind\":\"ClusterServiceVersionList\",\"plural\":\"clusterserviceversions\",\"shortNames\":[\"csv\",\"csvs\"],\"singular\":\"clusterserviceversion\"},\"sc [truncated 458122 chars] customresourcedefinition.apiextensions.k8s.io/clusterserviceversions.operators.coreos.com configured I1109 19:12:46.639320 12045 apply.go:390] Running apply post-processor function [knarra@knarra openshift-client-linux-4.7.0-0.nightly-2020-11-08-225909]$ ./oc --loglevel 8 apply -f 0000_50_olm_00-clusterserviceversions.crd.yaml I1109 19:12:55.600182 12114 loader.go:375] Config loaded from file: /home/knarra/Downloads/kubeconfig_47 I1109 19:12:55.600774 12114 round_trippers.go:420] GET https://api.knarra1109.qe.devcluster.openshift.com:6443/openapi/v2?timeout=32s I1109 19:12:55.600780 12114 round_trippers.go:427] Request Headers: I1109 19:12:55.600782 12114 round_trippers.go:431] Accept: application/com.github.proto-openapi.spec.v2+protobuf I1109 19:12:55.600785 12114 round_trippers.go:431] User-Agent: oc/4.7.0 (linux/amd64) kubernetes/9f54c1d I1109 19:12:55.600788 12114 round_trippers.go:431] Authorization: Bearer <masked> I1109 19:12:57.219528 12114 round_trippers.go:446] Response Status: 200 OK in 1618 milliseconds I1109 19:12:57.219544 12114 round_trippers.go:449] Response Headers: I1109 19:12:57.219548 12114 round_trippers.go:452] Accept-Ranges: bytes I1109 19:12:57.219551 12114 round_trippers.go:452] Cache-Control: no-cache, private I1109 19:12:57.219554 12114 round_trippers.go:452] Content-Type: application/octet-stream I1109 19:12:57.219556 12114 round_trippers.go:452] X-Kubernetes-Pf-Flowschema-Uid: 202c5c05-54a2-4b02-ade2-8f30ecb59d3e I1109 19:12:57.219559 12114 round_trippers.go:452] X-Kubernetes-Pf-Prioritylevel-Uid: 1517604c-7276-4328-ad28-1565abb05931 I1109 19:12:57.219562 12114 round_trippers.go:452] Audit-Id: d3876b69-bee6-42c9-b87c-9112444afa0a I1109 19:12:57.219564 12114 round_trippers.go:452] Etag: "31BCD824419712B67F395B83AABF97584195BE57C27D7206DD012381242A141840AF8D52FECE53BA31D6B4DB3F29FFE1ECB848972F6BE0686D3B9C4A53F1A80A" I1109 19:12:57.219569 12114 round_trippers.go:452] Last-Modified: Mon, 09 Nov 2020 13:42:39 GMT I1109 19:12:57.219571 12114 round_trippers.go:452] Vary: Accept-Encoding I1109 19:12:57.219574 12114 round_trippers.go:452] Vary: Accept I1109 19:12:57.219578 12114 round_trippers.go:452] Date: Mon, 09 Nov 2020 13:42:57 GMT I1109 19:12:57.219580 12114 round_trippers.go:452] X-Varied-Accept: application/com.github.proto-openapi.spec.v2+protobuf I1109 19:13:02.234474 12114 request.go:1095] Response Body: 00000000 0a 03 32 2e 30 12 1d 0a 0a 4b 75 62 65 72 6e 65 |..2.0....Kuberne| 00000010 74 65 73 12 0f 76 31 2e 31 39 2e 30 2b 65 36 37 |tes..v1.19.0+e67| 00000020 66 35 64 63 42 b7 a3 bc 02 12 80 03 0a 1c 2f 61 |f5dcB........./a| 00000030 70 69 73 2f 6f 61 75 74 68 2e 6f 70 65 6e 73 68 |pis/oauth.opensh| 00000040 69 66 74 2e 69 6f 2f 76 31 2f 12 df 02 12 dc 02 |ift.io/v1/......| 00000050 0a 13 6f 61 75 74 68 4f 70 65 6e 73 68 69 66 74 |..oauthOpenshift| 00000060 49 6f 5f 76 31 1a 17 67 65 74 20 61 76 61 69 6c |Io_v1..get avail| 00000070 61 62 6c 65 20 72 65 73 6f 75 72 63 65 73 2a 21 |able resources*!| 00000080 67 65 74 4f 61 75 74 68 4f 70 65 6e 73 68 69 66 |getOauthOpenshif| 00000090 74 49 6f 56 31 41 50 49 52 65 73 6f 75 72 63 65 |tIoV1APIResource| 000000a0 73 32 10 61 70 70 6c 69 63 61 74 69 6f 6e 2f 6a |s2.application/j| 000000b0 73 6f 6e 32 10 61 70 70 6c 69 63 61 74 69 6f 6e |son2.application| 000000c0 2f 79 61 6d 6c 32 23 61 70 70 6c 69 63 61 74 69 |/yaml2#applicat [truncated 37021055 chars] I1109 19:13:02.346865 12114 round_trippers.go:420] GET https://api.knarra1109.qe.devcluster.openshift.com:6443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusterserviceversions.operators.coreos.com I1109 19:13:02.346881 12114 round_trippers.go:427] Request Headers: I1109 19:13:02.346884 12114 round_trippers.go:431] Accept: application/json I1109 19:13:02.346887 12114 round_trippers.go:431] User-Agent: oc/4.7.0 (linux/amd64) kubernetes/9f54c1d I1109 19:13:02.346890 12114 round_trippers.go:431] Authorization: Bearer <masked> I1109 19:13:02.756675 12114 round_trippers.go:446] Response Status: 200 OK in 409 milliseconds I1109 19:13:02.756691 12114 round_trippers.go:449] Response Headers: I1109 19:13:02.756697 12114 round_trippers.go:452] Vary: Accept-Encoding I1109 19:13:02.756701 12114 round_trippers.go:452] X-Kubernetes-Pf-Flowschema-Uid: 202c5c05-54a2-4b02-ade2-8f30ecb59d3e I1109 19:13:02.756704 12114 round_trippers.go:452] X-Kubernetes-Pf-Prioritylevel-Uid: 1517604c-7276-4328-ad28-1565abb05931 I1109 19:13:02.756706 12114 round_trippers.go:452] Date: Mon, 09 Nov 2020 13:43:02 GMT I1109 19:13:02.756709 12114 round_trippers.go:452] Audit-Id: 203ea072-29df-4c9a-b17e-800ea9890f25 I1109 19:13:02.756712 12114 round_trippers.go:452] Cache-Control: no-cache, private I1109 19:13:02.756714 12114 round_trippers.go:452] Content-Type: application/json I1109 19:13:03.140004 12114 request.go:1097] Response Body: {"kind":"CustomResourceDefinition","apiVersion":"apiextensions.k8s.io/v1","metadata":{"name":"clusterserviceversions.operators.coreos.com","selfLink":"/apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusterserviceversions.operators.coreos.com","uid":"0bc8b63f-7424-4650-bf93-b3c9103e8bec","resourceVersion":"79122","generation":4,"creationTimestamp":"2020-11-09T10:11:23Z","annotations":{"controller-gen.kubebuilder.io/version":"v0.3.0","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{\"controller-gen.kubebuilder.io/version\":\"v0.3.0\"},\"creationTimestamp\":null,\"name\":\"clusterserviceversions.operators.coreos.com\"},\"spec\":{\"group\":\"operators.coreos.com\",\"names\":{\"categories\":[\"olm\"],\"kind\":\"ClusterServiceVersion\",\"listKind\":\"ClusterServiceVersionList\",\"plural\":\"clusterserviceversions\",\"shortNames\":[\"csv\",\"csvs\"],\"singular\":\"clusterserviceversion\"},\"sc [truncated 458122 chars] I1109 19:13:03.164265 12114 request.go:1097] Request Body: {"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}} I1109 19:13:03.164312 12114 round_trippers.go:420] PATCH https://api.knarra1109.qe.devcluster.openshift.com:6443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusterserviceversions.operators.coreos.com?fieldManager=kubectl-client-side-apply I1109 19:13:03.164316 12114 round_trippers.go:427] Request Headers: I1109 19:13:03.164319 12114 round_trippers.go:431] Accept: application/json I1109 19:13:03.164323 12114 round_trippers.go:431] Content-Type: application/merge-patch+json I1109 19:13:03.164346 12114 round_trippers.go:431] User-Agent: oc/4.7.0 (linux/amd64) kubernetes/9f54c1d I1109 19:13:03.164351 12114 round_trippers.go:431] Authorization: Bearer <masked> I1109 19:13:03.678476 12114 round_trippers.go:446] Response Status: 200 OK in 514 milliseconds I1109 19:13:03.678495 12114 round_trippers.go:449] Response Headers: I1109 19:13:03.678498 12114 round_trippers.go:452] Date: Mon, 09 Nov 2020 13:43:03 GMT I1109 19:13:03.678503 12114 round_trippers.go:452] Audit-Id: 72832cf9-3670-4e10-ae06-d65bdb689563 I1109 19:13:03.678508 12114 round_trippers.go:452] Cache-Control: no-cache, private I1109 19:13:03.678510 12114 round_trippers.go:452] Content-Type: application/json I1109 19:13:03.678513 12114 round_trippers.go:452] Vary: Accept-Encoding I1109 19:13:03.678516 12114 round_trippers.go:452] X-Kubernetes-Pf-Flowschema-Uid: 202c5c05-54a2-4b02-ade2-8f30ecb59d3e I1109 19:13:03.678520 12114 round_trippers.go:452] X-Kubernetes-Pf-Prioritylevel-Uid: 1517604c-7276-4328-ad28-1565abb05931 I1109 19:13:03.698983 12114 request.go:1097] Response Body: {"kind":"CustomResourceDefinition","apiVersion":"apiextensions.k8s.io/v1","metadata":{"name":"clusterserviceversions.operators.coreos.com","selfLink":"/apis/apiextensions.k8s.io/v1/customresourcedefinitions/clusterserviceversions.operators.coreos.com","uid":"0bc8b63f-7424-4650-bf93-b3c9103e8bec","resourceVersion":"79231","generation":4,"creationTimestamp":"2020-11-09T10:11:23Z","annotations":{"controller-gen.kubebuilder.io/version":"v0.3.0","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{\"controller-gen.kubebuilder.io/version\":\"v0.3.0\"},\"creationTimestamp\":null,\"name\":\"clusterserviceversions.operators.coreos.com\"},\"spec\":{\"group\":\"operators.coreos.com\",\"names\":{\"categories\":[\"olm\"],\"kind\":\"ClusterServiceVersion\",\"listKind\":\"ClusterServiceVersionList\",\"plural\":\"clusterserviceversions\",\"shortNames\":[\"csv\",\"csvs\"],\"singular\":\"clusterserviceversion\"},\"sc [truncated 458122 chars] customresourcedefinition.apiextensions.k8s.io/clusterserviceversions.operators.coreos.com configured I1109 19:13:03.707079 12114 apply.go:390] Running apply post-processor function Based on the above moving bug to verified state.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633