Description of problem: The DPTP ipi-deprovisioner tool that runs openshift-install destroy cluster [1] gets stuck on deleting a network, accompanied by the following messages: level=debug msg="Networks: failed to delete network ci-op-sq9x1it6-0df6f-kdt74-network with error: RESOURCE_IN_USE_BY_ANOTHER_RESOURCE: The network resource 'projects/openshift-gce-devel-ci/global/networks/ci-op-sq9x1it6-0df6f-kdt74-network' is already being used by 'projects/openshift-gce-devel-ci/global/firewalls/k8s-a091b5cea9ce44d1589ce122fe0b62bb-http-hc'" [1] https://github.com/openshift/ci-tools/blob/f977bb476cfacf74b8ecea1df1178a13cfa7a3e3/cmd/ipi-deprovision/ipi-deprovision.sh#L29-L60 Example occurrence: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ipi-deprovision/1301547877104881664#1:build-log.txt%3A444 How reproducible: ~cca 1-2x per week our CI produces something like this and it needs manual intervention
The health checks are created with random names, and the only way installer can associate them is to lookup which LB -> which machines -> which cluster. So if the machines are gone there is not way for us to re-associate. Secondly the de-provision script is running on the same cluster multiple times with previously _deleted / left around_ clusters which makes this problem more apparent. There is not good way to circumvent this unless we involve upstream to tag them appropriately. Will need a lot more work and planning, moving to 4.7
*** Bug 1801968 has been marked as a duplicate of this bug. ***
https://bugzilla.redhat.com/show_bug.cgi?id=1801968 was closed as duplicate of this.
https://issues.redhat.com/browse/CORS-1573 should be good enough to also include this fix.
Thanks. We'll track the work for this in Jira.
*** Bug 1906172 has been marked as a duplicate of this bug. ***