Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1752591

Summary: [GCP] e2e failure: [Feature:Image][triggers] Image change build triggers TestMultipleImageChangeBuildTriggers [Suite:openshift/conformance/parallel]
Product: OpenShift Container Platform Reporter: David Eads <deads>
Component: kube-controller-managerAssignee: Maciej Szulik <maszulik>
Status: CLOSED DUPLICATE QA Contact: zhou ying <yinzhou>
Severity: high Docs Contact:
Priority: high    
Version: 4.2.0CC: adam.kaplan, aos-bugs, mfojtik, tnozicka, wzheng
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-09-17 08:54:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
healthy-build-gc
none
hanging build gc none

Description David Eads 2019-09-16 17:48:50 UTC 
5/20 flake on GCP

https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/canary-openshift-ocp-installer-e2e-gcp-4.2/282 shows it flaking.
Comment 1 Adam Kaplan 2019-09-16 18:27:17 UTC 
Created attachment 1615603 [details]
healthy-build-gc

Healthy garbage collection of namespace, builds, and owned objects.
Comment 2 Adam Kaplan 2019-09-16 18:28:16 UTC 
Created attachment 1615604 [details]
hanging build gc

Logs where build garbage collection appears to hang deleting ConfigMaps.
Comment 4 Tomáš Nožička 2019-09-17 08:54:19 UTC 
I don't think the configmap is related or blocks deletion.

The reason the test failed is:

Couldn't delete ns: "e2e-test-image-change-build-triggger-57b94": namespace e2e-test-image-change-build-triggger-57b94 was not deleted with limit: timed out waiting for the condition, namespace is empty but is not yet removed (&errors.errorString{s:"namespace e2e-test-image-change-build-triggger-57b94 was not deleted with limit: timed out waiting for the condition, namespace is empty but is not yet removed"})

which in your logs would correlate with a lot of 

2019-09-16T12:38:28.888664448Z E0916 12:38:28.888583       1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods
2019-09-16T12:38:29.536340483Z E0916 12:38:29.536263       1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods
2019-09-16T12:38:30.322487437Z E0916 12:38:30.322410       1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods
2019-09-16T12:38:31.112118795Z E0916 12:38:31.112010       1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods
2019-09-16T12:38:31.889845681Z E0916 12:38:31.889741       1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods
2019-09-16T12:38:32.551497298Z E0916 12:38:32.551227       1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods
2019-09-16T12:38:33.422561224Z E0916 12:38:33.422372       1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods
2019-09-16T12:38:34.376367228Z E0916 12:38:34.376289       1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods
2019-09-16T12:38:35.70376082Z E0916 12:38:35.703643       1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods
2019-09-16T12:38:37.82946912Z E0916 12:38:37.829329       1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods
2019-09-16T12:38:41.366488381Z E0916 12:38:41.365910       1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods
2019-09-16T12:38:47.761612419Z E0916 12:38:47.761518       1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods
2019-09-16T12:38:58.801942068Z E0916 12:38:58.801879       1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods
2019-09-16T12:39:20.477961663Z E0916 12:39:20.477885       1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods
2019-09-16T12:40:02.883683418Z E0916 12:40:02.883610       1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods
2019-09-16T12:41:25.592953724Z E0916 12:41:25.592868       1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods
2019-09-16T12:43:28.949574721Z E0916 12:43:28.949509       1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods

which is caused by slow pod deletion either by kubelet or container runtime.

*** This bug has been marked as a duplicate of bug 1727090 ***
Comment 5 Adam Kaplan 2019-09-17 13:15:36 UTC 
@Tomas wouldn't slow deletion of the ConfigMaps get in the way of "slow pod deletion"? After all each build pod has 3 ConfigMaps which owner-ref it.