Bug 1380422

Summary: 3.2 cluster upgraded from 3.1 with gluster for registry storage has issues pushing a certain application image
Product: OpenShift Container Platform Reporter: Eric Jones <erjones>
Component: Image RegistryAssignee: Alexey Gladkov <agladkov>
Status: CLOSED CANTFIX QA Contact: Wei Sun <wsun>
Severity: low Docs Contact:
Priority: low    
Version: 3.2.1CC: aos-bugs, mfojtik
Target Milestone: ---Keywords: Unconfirmed
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
OpenShift 3.2.1 upgraded from 3.1.1.6
Last Closed: 2017-08-23 08:53:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Eric Jones 2016-09-29 14:40:23 UTC 
Description of problem:
Deploying one specific application in one existing project the customer sees the push to the docker registry fail. The docker registry logs point to [0], but the build logs simply look like [1].

[0] time="<DATE_TIME_TIMEZONE>" level=error msg="error canceling upload after error: filesystem: remove /registry/docker/registry/v2/repositories/devopslab/welcome/_uploads/76502937-3d53-4c34-aaf5-b620169435c6/hashstates/sha256: directory not empty"

[1] I0921 <TIME>       1 sti.go:268] Using provided push secret for pushing <IP>:5000/devopslab/welcome:latest image
I0921 <TIME>       1 sti.go:272] Pushing <IP>:5000/devopslab/welcome:latest image ...
I0921 <TIME>       1 sti.go:277] Registry server Address: 
I0921 <TIME>       1 sti.go:278] Registry server User Name: serviceaccount
I0921 <TIME>       1 sti.go:279] Registry server Email: serviceaccount
I0921 <TIME>       1 sti.go:284] Registry server Password: <<non-empty>>
F0921 <TIME>       1 builder.go:204] Error: build error: Failed to push image. Response from registry is: Received unexpected HTTP status: 500 Internal Server Error

When using this same application in a different project, it works just fine, and if I understood correctly, they were able to do other pushes in that same project.

Additional info:
GlusterFS is the storage used for the persistent storage for the docker registry.

Other than that last bit of behavior (only the one application causing the issue) the problem seems very similar to the github issue with docker here [2], and specifically sounds like the behavior described in this one comment of that issue [3].

[2] https://github.com/docker/docker/issues/15833
[3] https://github.com/docker/docker/issues/15833#issuecomment-153029208