Bug 1385488

Summary: [ocp-on-osp] Failed to push image due to docker-registry is re-created
Product: OpenShift Container Platform Reporter: Gan Huang <ghuang>
Component: InstallerAssignee: scollier
Status: CLOSED INSUFFICIENT_DATA QA Contact: Gan Huang <ghuang>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.3.0CC: aos-bugs, jokerman, mmccomas
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-03 10:19:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
Part logs of ansible none

Description Gan Huang 2016-10-17 07:54:58 UTC
Description of problem:
After the heat stack was created, S2I build was failed due to the docker-registry IP had been changed. And the changes was done by extra playbook, not openshift-ansible. See https://bugzilla.redhat.com/show_bug.cgi?id=1192257

Version-Release number of selected component (if applicable):
v0.9.2

How reproducible:
Sometimes

Steps to Reproduce:
1.Create a heat stack
2.Do S2I build after the stack was completed
3.

Actual results:
1. docker-registry and router were created successfully, but found that docker-registry was re-deployed.
# oc get po -n default
NAME                      READY     STATUS    RESTARTS   AGE
docker-registry-2-8pk96   1/1       Running   0          45m
router-1-n8qva            1/1       Running   0          45m
router-1-qnnnb            1/1       Running   0          44m

# oc get svc -n default
NAME              CLUSTER-IP       EXTERNAL-IP   PORT(S)                   AGE
docker-registry   172.30.138.108   <none>        5000/TCP                  50m
kubernetes        172.30.0.1       <none>        443/TCP,53/UDP,53/TCP     1h
router            172.30.29.192    <none>        80/TCP,443/TCP,1936/TCP   50m


2. S2I build failed due to docker registry ip changed.
I1017 03:14:08.420080       1 sti.go:334] Successfully built test/cakephp-example-1:4ddc039c
I1017 03:14:08.662623       1 cleanup.go:23] Removing temporary directory /tmp/s2i-build260843205
I1017 03:14:08.662694       1 fs.go:156] Removing directory '/tmp/s2i-build260843205'
I1017 03:14:08.721730       1 sti.go:268] Using provided push secret for pushing 172.30.138.108:5000/test/cakephp-example:latest image
I1017 03:14:08.721752       1 sti.go:272] Pushing 172.30.138.108:5000/test/cakephp-example:latest image ...
I1017 03:14:15.908876       1 sti.go:277] Registry server Address: 
I1017 03:14:15.908922       1 sti.go:278] Registry server User Name: serviceaccount
I1017 03:14:15.908929       1 sti.go:279] Registry server Email: serviceaccount
I1017 03:14:15.908935       1 sti.go:284] Registry server Password: <<non-empty>>
F1017 03:14:15.908943       1 builder.go:204] Error: build error: Failed to push image. Response from registry is: Error: Status 404 trying to push repository test/cakephp-example: "404 page not found\n"


Expected results:
S2I build successed after the stack was created.

Additional info:

Seen from the ansible logs

The docker-registry ip might be changed in this situation:

After running BYO playbook (openshift-ansible), the task "Restart node service" was exected in extra playbook, this task *might* destroy the docker-registry deployment, then trigger a new deployment for docker-registry.

Comment 1 Gan Huang 2016-10-17 08:00:51 UTC
Created attachment 1211239 [details]
Part logs of ansible

Comment 2 Gan Huang 2016-11-03 10:19:36 UTC
Close it as  it can't be reproduced any more.