Bug 1250452

Summary:

Failed to jenkins build on STG

Product:

OpenShift Online

Reporter:

zhaozhanqi <zzhao>

Component:

Deployments

Assignee:

Miciah Dashiel Butler Masters <mmasters>

Status:

CLOSED CURRENTRELEASE

QA Contact:

Yan Du <yadu>

Severity:

high

Docs Contact:

Priority:

high

Version:

2.x

CC:

bmeng, libra-bugs, tiwillia, wsun

Target Milestone:

---

Keywords:

TestBlocker

Target Release:

---

Hardware:

All

OS:

All

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2015-09-08 20:14:55 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
log	none

Description zhaozhanqi 2015-08-05 11:51:14 UTC

Description of problem:
Create one scale/non-scale app. Add hot_deploy marker and git push. During the push, always show message "
Failed to report deployment to broker. This will be corrected on the next git push. Message: Connection timed out - connect(2)"

Version-Release number of selected component (if applicable):
STG(devenv_stage_1174)

How reproducible:
always

Steps to Reproduce:
1. Create one scale app
2. Add hot_deploy marker
touch .openshift/markers/hot_deploy
3. git push

Actual results:
Writing objects: 100% (4/4), 400 bytes | 0 bytes/s, done.
Total 4 (delta 1), reused 0 (delta 0)
remote: Not stopping cartridge php because hot deploy is enabled
remote: Syncing git content to other proxy gears
remote: Building git ref 'master', commit db68884
remote: Checking .openshift/pear.txt for PEAR dependency...
remote: Preparing build for deployment
remote: Deployment id is 4716296e
remote: Activating deployment
remote: Not starting cartridge haproxy because hot deploy is enabled
remote: Not starting cartridge php because hot deploy is enabled
remote: Failed to report deployment to broker. This will be corrected on the next git push. Message: Connection timed out - connect(2)
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Activation status: success
remote: Deployment completed with status: success
To ssh://55c1f21864480ff2270001b0.rhcloud.com/~/git/phps.git/
66bd4a2..db68884 master -> master

Expected results:
should be successful

Additional info:
no this error message on devenv_stage_1174

Comment 1 zhaozhanqi 2015-08-05 12:07:59 UTC

Seems this issue it not related to hot deploy. 
At the beginning, I thought it is cases by hot_deploy, when I delete the marker, still has this issue.

Comment 2 Timothy Williams 2015-08-18 21:45:19 UTC

I am unable to reproduce this with or without the hot_deploy marker in a recent devenv:

remote: Not stopping cartridge php because hot deploy is enabled
remote: Syncing git content to other proxy gears
remote: Building git ref 'master', commit 12f7325
remote: Checking .openshift/pear.txt for PEAR dependency...
remote: Preparing build for deployment
remote: Deployment id is 45cd4a52
remote: Activating deployment
remote: Not starting cartridge haproxy because hot deploy is enabled
remote: Not starting cartridge php because hot deploy is enabled
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Activation status: success
remote: Deployment completed with status: success
To ssh://55d3a00bdfcacb3a3b000001.rhcloud.com/~/git/scaletest.git/
   df82c36..12f7325  master -> master

I am also unable to reproduce this issue in the devenv_stage_1174 ami:

Delta compression using up to 2 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 411 bytes, done.
Total 4 (delta 1), reused 0 (delta 0)
remote: Not stopping cartridge php because hot deploy is enabled
remote: Syncing git content to other proxy gears
remote: Building git ref 'master', commit 19dbc69
remote: Checking .openshift/pear.txt for PEAR dependency...
remote: Preparing build for deployment
remote: Deployment id is ab79f5d9
remote: Activating deployment
remote: Not starting cartridge haproxy because hot deploy is enabled
remote: Not starting cartridge php because hot deploy is enabled
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Activation status: success
remote: Deployment completed with status: success
To ssh://55d3a6e38b95c7dc09000009.rhcloud.com/~/git/phptest.git/
   3636985..19dbc69  master -> master

QE, could you please verify this?

Comment 3 zhaozhanqi 2015-08-19 02:17:51 UTC

this issue only happened on STG env. yes, on the latest devenv and devenv_stsge, it works well

Comment 4 Meng Bo 2015-08-21 09:28:12 UTC

Looks like there are some networking issues when the gear trying to talk with broker via API.

Jenkins build will always fail due to the same timeout problem. As logs below:

Aug 21, 2015 4:51:10 AM hudson.plugins.openshift.OpenShiftCloud getOpenShiftConnection
INFO: Initiating Java Client Service - Configured for OpenShift Server https://stg.openshift.redhat.com
Aug 21, 2015 4:51:10 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://stg.openshift.redhat.com/broker/rest/api
Aug 21, 2015 4:52:13 AM hudson.plugins.openshift.OpenShiftCloud provision
WARNING: Caught java.io.IOException: com.openshift.client.OpenShiftEndpointException: Could not request https://stg.openshift.redhat.com/broker/rest/api: java.net.ConnectException: Connection timed out. Will retry 0 more times before canceling build.
Aug 21, 2015 4:52:18 AM hudson.plugins.openshift.OpenShiftCloud provision
WARNING: Cancelling build due to earlier exceptions
java.io.IOException: com.openshift.client.OpenShiftEndpointException: Could not request https://stg.openshift.redhat.com/broker/rest/api: java.net.ConnectException: Connection timed out



And both above issues cannot be reproduced on devenv-stage_1175 with multi-node env.

Comment 5 Yan Du 2015-08-25 02:16:57 UTC

Issue still could be reproduced on devenv_stage-1177

Comment 6 Yan Du 2015-08-25 02:27:01 UTC

(In reply to Yan Du from comment #5)
> Issue still could be reproduced on devenv_stage-1177

I mean issue still could be reproduced on STG(devenv_stage-1177)

Comment 7 Miciah Dashiel Butler Masters 2015-08-25 17:05:19 UTC

Ops modified the security-groups rules for STG, and git push now succeeds when I try it, so I am hopeful that the issue is resolved.

Comment 8 Yan Du 2015-08-26 05:07:24 UTC

Test on latest env on STG

1. Normal build is successful now.

remote: Stopping PHP 5.4 cartridge (Apache+mod_php)
remote: Waiting for stop to finish
remote: Waiting for stop to finish
remote: Building git ref 'master', commit c2b53b0
remote: Checking .openshift/pear.txt for PEAR dependency...
remote: Preparing build for deployment
remote: Deployment id is d7341ded
remote: Activating deployment
remote: Starting PHP 5.4 cartridge (Apache+mod_php)
remote: Application directory "/" selected as DocumentRoot
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Activation status: success
remote: Deployment completed with status: success


2. Jenkins build still failed, and got the same error as #Comment 4, please check the log in attachment for detail.

Comment 9 Yan Du 2015-08-26 05:07:53 UTC

Created attachment 1067140 [details]
log

Comment 10 Miciah Dashiel Butler Masters 2015-08-27 03:34:10 UTC

Ops has performed further changes that should resolve the issue.  STG is now using internal.stg.openshift.com, which is a newly, properly configured ELB, whereas stg.openshift.redhat.com is the old ELB that is causing problems.

If you are using an existing Jenkins instance, you will probably need to reconfigure it to use internal.stg.openshift.com instead of stg.openshift.redhat.com.  It may be easiest to delete and recreate any old Jenkins instance you may be using.

Comment 11 Yan Du 2015-08-27 08:31:31 UTC

Test on latest STG env, both jenkins build and normal build work well.


remote: Executing Jenkins build.
remote: 
remote: You can track your build at https://jenkins-111.stg.rhcloud.com/job/app1-build
remote: 
remote: Waiting for build to schedule.....	............................................................Done
remote: Waiting for job to complete........................Done
remote: SUCCESS
remote: New build has been deployed.
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Deployment completed with status: success
To ssh://55dea59bc0c5a7f7ee000086.rhcloud.com/~/git/app1.git/
   5054752..e393d29  master -> master


Move to verified.