Bug 1318939

Summary: Push shared layer images to docker-registry with diff tags will meet time-out error.
Product: OpenShift Online Reporter: Bing Li <bingli>
Component: Image RegistryAssignee: Maciej Szulik <maszulik>
Status: CLOSED CURRENTRELEASE QA Contact: Wei Sun <wsun>
Severity: low Docs Contact:
Priority: low    
Version: 3.xCC: aos-bugs, bingli, maszulik, pweil, yinzhou
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1382790 (view as bug list) Environment:
Last Closed: 2016-10-04 13:08:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1382790    
Attachments:
Description Flags
Log of Docker registry, filtered by project name: binglitest0331
none
s3 push none

Comment 1 Michal Minar 2016-03-21 09:53:01 UTC 
Bing Li,

could you please also upload registry logs from the time of the failed push?

Anyway, this doesn't look like a registry issue. It's rather an issue with remote s3 bucket.
Comment 2 Bing Li 2016-03-31 07:50:37 UTC 
Created attachment 1142091 [details]
Log of Docker registry, filtered by project name: binglitest0331
Comment 3 Michail Kargakis 2016-04-04 15:59:51 UTC 
Created attachment 1143385 [details]
s3 push
Comment 4 Michail Kargakis 2016-04-04 16:03:29 UTC 
I cannot reproduce this either locally or using s3. (See attachment above and pastebin here: http://pastebin.com/y3Lg3v7P). Not sure if it will be of any help but can you post your s3 configuration?
Comment 5 Paul Weil 2016-04-04 19:53:11 UTC 
Is this happening consistently in your environment and if so can Michail have access to it?
Comment 6 Bing Li 2016-04-05 07:48:30 UTC 
When I tried to push images from my computer in beijing, it would fail frequently, like this: http://pastebin.com/M4M8z81Z. Until after I pushed 5 times, then it succeeded.

But when I tried to push the images from an AWS instance, it always succeeded. This issue never happened again.

So maybe it's not a registry issue.
Comment 7 Maciej Szulik 2016-04-05 09:09:05 UTC 
I'm inclined to say this is similar to bug 1321968, which might be periodic AWS problems. I've went through the registry logs and I can't see any anomalies there, all the s3-related operations are quite instantaneous. I'd propose lowering prio of this bug but keep it open until it appears again. There seems to be some problems and we need to track them down, definitely.

@Bing Li: next time when attaching docker registry logs, please attach full log from the time the timeouts happen +/- 5 mins. It might be that grepping per project might cut some important messages we're missing here.
Comment 8 Maciej Szulik 2016-06-28 09:40:57 UTC 
With the new docker registry [1] comes different s3 driver, specifically it's using AWS driver instead of external one. According to tests done by Michal the performance should be greatly improved. I'm moving this bug to QA, similarly to bug 1314381, comment 24.

[1] https://github.com/openshift/origin/pull/8938
Comment 10 zhou ying 2016-09-01 08:08:37 UTC 
confirmed with fork_ami_openshift3_miminar_295, and can not reproduce this issue