5/20 flake on GCP https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/canary-openshift-ocp-installer-e2e-gcp-4.2/282 shows it flaking.
Created attachment 1615603 [details] healthy-build-gc Healthy garbage collection of namespace, builds, and owned objects.
Created attachment 1615604 [details] hanging build gc Logs where build garbage collection appears to hang deleting ConfigMaps.
I don't think the configmap is related or blocks deletion. The reason the test failed is: Couldn't delete ns: "e2e-test-image-change-build-triggger-57b94": namespace e2e-test-image-change-build-triggger-57b94 was not deleted with limit: timed out waiting for the condition, namespace is empty but is not yet removed (&errors.errorString{s:"namespace e2e-test-image-change-build-triggger-57b94 was not deleted with limit: timed out waiting for the condition, namespace is empty but is not yet removed"}) which in your logs would correlate with a lot of 2019-09-16T12:38:28.888664448Z E0916 12:38:28.888583 1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods 2019-09-16T12:38:29.536340483Z E0916 12:38:29.536263 1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods 2019-09-16T12:38:30.322487437Z E0916 12:38:30.322410 1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods 2019-09-16T12:38:31.112118795Z E0916 12:38:31.112010 1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods 2019-09-16T12:38:31.889845681Z E0916 12:38:31.889741 1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods 2019-09-16T12:38:32.551497298Z E0916 12:38:32.551227 1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods 2019-09-16T12:38:33.422561224Z E0916 12:38:33.422372 1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods 2019-09-16T12:38:34.376367228Z E0916 12:38:34.376289 1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods 2019-09-16T12:38:35.70376082Z E0916 12:38:35.703643 1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods 2019-09-16T12:38:37.82946912Z E0916 12:38:37.829329 1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods 2019-09-16T12:38:41.366488381Z E0916 12:38:41.365910 1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods 2019-09-16T12:38:47.761612419Z E0916 12:38:47.761518 1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods 2019-09-16T12:38:58.801942068Z E0916 12:38:58.801879 1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods 2019-09-16T12:39:20.477961663Z E0916 12:39:20.477885 1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods 2019-09-16T12:40:02.883683418Z E0916 12:40:02.883610 1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods 2019-09-16T12:41:25.592953724Z E0916 12:41:25.592868 1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods 2019-09-16T12:43:28.949574721Z E0916 12:43:28.949509 1 namespace_controller.go:148] unexpected items still remain in namespace: e2e-test-image-change-build-triggger-57b94 for gvr: /v1, Resource=pods which is caused by slow pod deletion either by kubelet or container runtime. *** This bug has been marked as a duplicate of bug 1727090 ***
@Tomas wouldn't slow deletion of the ConfigMaps get in the way of "slow pod deletion"? After all each build pod has 3 ConfigMaps which owner-ref it.