Description of problem:
Deploying one specific application in one existing project the customer sees the push to the docker registry fail. The docker registry logs point to [0], but the build logs simply look like [1].

[0] time="<DATE_TIME_TIMEZONE>" level=error msg="error canceling upload after error: filesystem: remove /registry/docker/registry/v2/repositories/devopslab/welcome/_uploads/76502937-3d53-4c34-aaf5-b620169435c6/hashstates/sha256: directory not empty"

[1] I0921 <TIME>       1 sti.go:268] Using provided push secret for pushing <IP>:5000/devopslab/welcome:latest image
I0921 <TIME>       1 sti.go:272] Pushing <IP>:5000/devopslab/welcome:latest image ...
I0921 <TIME>       1 sti.go:277] Registry server Address: 
I0921 <TIME>       1 sti.go:278] Registry server User Name: serviceaccount
I0921 <TIME>       1 sti.go:279] Registry server Email: serviceaccount
I0921 <TIME>       1 sti.go:284] Registry server Password: <<non-empty>>
F0921 <TIME>       1 builder.go:204] Error: build error: Failed to push image. Response from registry is: Received unexpected HTTP status: 500 Internal Server Error

When using this same application in a different project, it works just fine, and if I understood correctly, they were able to do other pushes in that same project.

Additional info:
GlusterFS is the storage used for the persistent storage for the docker registry.

Other than that last bit of behavior (only the one application causing the issue) the problem seems very similar to the github issue with docker here [2], and specifically sounds like the behavior described in this one comment of that issue [3].

[2] https://github.com/docker/docker/issues/15833
[3] https://github.com/docker/docker/issues/15833#issuecomment-153029208