Bug 1445469

Summary: [starter][starter-us-east-1] 404 error while pushing image
Product: OpenShift Online Reporter: Vikas Laad <vlaad>
Component: Image RegistryAssignee: Michal Minar <miminar>
Status: CLOSED WORKSFORME QA Contact: Mike Fiedler <mifiedle>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.xCC: abhgupta, aos-bugs, dakini, haowang, jeder, mifiedle, miminar, pweil, somalley, vlaad
Target Milestone: ---Keywords: OnlineStarter
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-09-11 14:13:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
registry 1
none
registry 3 none

Description Vikas Laad 2017-04-25 17:50:13 UTC 
Created attachment 1273985 [details]
registry 1

Description of problem:
We are doing performance testing in starter-us-east-1 cluster, here is how cluster looks like

281 compute nodes
3 masters
3 infra nodes

I created 2810 build configs on the cluster (10 bc/node). Around 25 builds failed due to following error

Cloning "https://github.com/redhat-performance/cakephp-ex.git" ...
        Commit: 0014ddebb91bc7dff3a1dabfbd7b51da762a6677 (made changes to enable database example)
        Author: ofthecure <robdean.smith>
        Date:   Mon Apr 25 14:33:06 2016 -0400
Pulling image "registry.access.redhat.com/rhscl/php-56-rhel7@sha256:9304f5063c8aa209c5f8e04226c57c2d7635be6a911d3614eb066875d09885a4" ...
Pulling image "registry.access.redhat.com/rhscl/php-56-rhel7@sha256:9304f5063c8aa209c5f8e04226c57c2d7635be6a911d3614eb066875d09885a4" ...
DEPRECATED: Use .s2i/bin instead of .sti/bin
---> Installing application source...

Pushing image 172.30.208.107:5000/svt-build417/cakephp-mysql-example:latest ...
Registry server Address:
Registry server User Name: serviceaccount
Registry server Email: serviceaccount
Registry server Password: <<non-empty>>
error: build error: Failed to push image: Error: Status 404 trying to push repository svt-build417/cakephp-mysql-example: "404 page not found\n"                                                                                                                                                                                                                                                                                        

Version-Release number of selected component (if applicable):
Server https://internal.api.starter-us-east-1.openshift.com:443
openshift v3.5.5.9
kubernetes v1.5.2+43a9be4

Actual results:
25 builds failed

Expected results:
Builds should not fail

Additional info:
Please see all 3 registry logs and build pod logs.
Comment 1 Vikas Laad 2017-04-25 17:51:24 UTC 
Created attachment 1273986 [details]
registry 3
Comment 6 Abhishek Gupta 2017-04-27 14:20:05 UTC 
This was working for individual builds repeatedly outside of performance tests. Sally and I both conducted tests for this. I am inclined to push this bug out of the blocker list for the Summit launch as well.
Comment 7 Michal Minar 2017-05-02 14:03:33 UTC 
Could you please provide build logs?

I don't see svt-build417/cakephp-mysql-example in any registry log provided. They seem to be incomplete.

Nevertheless "404 page not found\n" suggests that the proxy didn't find the registry running. It may had crashed. We'd know for sure from the output of `oc get pods` and complete registry logs.
Comment 8 Vikas Laad 2017-05-02 14:13:21 UTC 
Michal, I do not have access to this environment anymore. All the logs were captured after the issue happened. I guess since there was 404 for this build while pushing the image that is why we don't see this in registry logs.

you want output of all 3 registries right ?

oc logs <registry-pod>

Stefanie, could you please provide complete registry logs to Michal.
Comment 15 Abhishek Gupta 2017-09-08 17:19:27 UTC 
Looking at the comments above, I would suggest that QE try and reproduce this bug again and capture the requested information. It doesn't seem likely that any progress can be made on this bug in its current state.
Comment 16 ge liu 2017-09-11 07:59:56 UTC 
hello, @mifiedle, Could you help to take a look for this performance issue? thanks.
Comment 17 Mike Fiedler 2017-09-11 14:13:57 UTC 
We no longer have access to this environment to do system testing.  If the users of the production starter-us-east-1 environment are not reporting this issue, we will close this as WORKSFORME.