Bug 1719792

Description of problem:
Sometimes SA token generation takes more time after a project is created. When running a new build (i.e. a new-app), build fails because SA token is not present. But the problem is that build still fails, there is no retry to get the builder token

# oc describe builds/testapp-1 -n ns
Name:		testapp-1
Namespace:	ns
Created:	33 minutes ago
Labels:		app=testapp
		buildconfig=testapp
		openshift.io/build-config.name=testapp
		openshift.io/build.start-policy=Serial
Annotations:	openshift.io/build-config.name=testapp
		openshift.io/build.number=1

Status:		New (Unable to look up the service account secrets for this build.)
Duration:	waiting for 33m4s


Background:
BZ 1462542 "API token is not present at build time":
If new-app is run immediately after a project is created, it races with service account token generation. This has been this way since 3.0 (c.f. https://bugzilla.redhat.com/show_bug.cgi?id=1318917#c5)

The time it takes for the service account and tokens to be provisioned has been greatly improved, but the fundamental race still exists.


Version-Release number of selected component (if applicable):
3.11.104

How reproducible:
Run a loop to create a new project and get the SA token builder

Steps to Reproduce:
1. Run a loop to create a new project
for i in {1..200}; do echo "Test: $i" >> test-sa-debug && date 2>&1 >> test-sa-debug && oc new-project appu-1640 >> test-sa-debug 2>&1 && oc serviceaccounts get-token builder -n appu-1640 --loglevel 10 >> test-sa-debug 2>&1 && oc delete project appu-1640 --loglevel 10 >> test-sa-debug 2>&1; done


Actual results:
In HA environments, it will fail to get the SA token after some loops