Bug 1250452 - Failed to jenkins build on STG
Failed to jenkins build on STG
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Deployments (Show other bugs)
2.x
All All
high Severity high
: ---
: ---
Assigned To: Miciah Dashiel Butler Masters
Yan Du
: TestBlocker
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-05 07:51 EDT by zhaozhanqi
Modified: 2015-09-08 16:14 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-09-08 16:14:55 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
log (9.19 KB, text/plain)
2015-08-26 01:07 EDT, Yan Du
no flags Details

  None (edit)
Description zhaozhanqi 2015-08-05 07:51:14 EDT
Description of problem:
Create one scale/non-scale app. Add hot_deploy marker and git push. During the push, always show message "
Failed to report deployment to broker.  This will be corrected on the next git push. Message: Connection timed out - connect(2)"

Version-Release number of selected component (if applicable):
STG(devenv_stage_1174)

How reproducible:
always

Steps to Reproduce:
1. Create one scale app
2. Add hot_deploy marker
   touch .openshift/markers/hot_deploy
3. git push

Actual results:
Writing objects: 100% (4/4), 400 bytes | 0 bytes/s, done.
Total 4 (delta 1), reused 0 (delta 0)
remote: Not stopping cartridge php because hot deploy is enabled
remote: Syncing git content to other proxy gears
remote: Building git ref 'master', commit db68884
remote: Checking .openshift/pear.txt for PEAR dependency...
remote: Preparing build for deployment
remote: Deployment id is 4716296e
remote: Activating deployment
remote: Not starting cartridge haproxy because hot deploy is enabled
remote: Not starting cartridge php because hot deploy is enabled
remote: Failed to report deployment to broker.  This will be corrected on the next git push. Message: Connection timed out - connect(2)
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Activation status: success
remote: Deployment completed with status: success
To ssh://55c1f21864480ff2270001b0@phps-zqd2.stg.rhcloud.com/~/git/phps.git/
   66bd4a2..db68884  master -> master


Expected results:
should be successful

Additional info:
no this error message on devenv_stage_1174
Comment 1 zhaozhanqi 2015-08-05 08:07:59 EDT
Seems this issue it not related to hot deploy. 
At the beginning, I thought it is cases by hot_deploy, when I delete the marker, still has this issue.
Comment 2 Timothy Williams 2015-08-18 17:45:19 EDT
I am unable to reproduce this with or without the hot_deploy marker in a recent devenv:

remote: Not stopping cartridge php because hot deploy is enabled
remote: Syncing git content to other proxy gears
remote: Building git ref 'master', commit 12f7325
remote: Checking .openshift/pear.txt for PEAR dependency...
remote: Preparing build for deployment
remote: Deployment id is 45cd4a52
remote: Activating deployment
remote: Not starting cartridge haproxy because hot deploy is enabled
remote: Not starting cartridge php because hot deploy is enabled
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Activation status: success
remote: Deployment completed with status: success
To ssh://55d3a00bdfcacb3a3b000001@scaletest-admin.dev.rhcloud.com/~/git/scaletest.git/
   df82c36..12f7325  master -> master

I am also unable to reproduce this issue in the devenv_stage_1174 ami:

Delta compression using up to 2 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 411 bytes, done.
Total 4 (delta 1), reused 0 (delta 0)
remote: Not stopping cartridge php because hot deploy is enabled
remote: Syncing git content to other proxy gears
remote: Building git ref 'master', commit 19dbc69
remote: Checking .openshift/pear.txt for PEAR dependency...
remote: Preparing build for deployment
remote: Deployment id is ab79f5d9
remote: Activating deployment
remote: Not starting cartridge haproxy because hot deploy is enabled
remote: Not starting cartridge php because hot deploy is enabled
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Activation status: success
remote: Deployment completed with status: success
To ssh://55d3a6e38b95c7dc09000009@phptest-admin.dev.rhcloud.com/~/git/phptest.git/
   3636985..19dbc69  master -> master

QE, could you please verify this?
Comment 3 zhaozhanqi 2015-08-18 22:17:51 EDT
this issue only happened on STG env. yes, on the latest devenv and devenv_stsge, it works well
Comment 4 Meng Bo 2015-08-21 05:28:12 EDT
Looks like there are some networking issues when the gear trying to talk with broker via API.

Jenkins build will always fail due to the same timeout problem. As logs below:

Aug 21, 2015 4:51:10 AM hudson.plugins.openshift.OpenShiftCloud getOpenShiftConnection
INFO: Initiating Java Client Service - Configured for OpenShift Server https://stg.openshift.redhat.com
Aug 21, 2015 4:51:10 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://stg.openshift.redhat.com/broker/rest/api
Aug 21, 2015 4:52:13 AM hudson.plugins.openshift.OpenShiftCloud provision
WARNING: Caught java.io.IOException: com.openshift.client.OpenShiftEndpointException: Could not request https://stg.openshift.redhat.com/broker/rest/api: java.net.ConnectException: Connection timed out. Will retry 0 more times before canceling build.
Aug 21, 2015 4:52:18 AM hudson.plugins.openshift.OpenShiftCloud provision
WARNING: Cancelling build due to earlier exceptions
java.io.IOException: com.openshift.client.OpenShiftEndpointException: Could not request https://stg.openshift.redhat.com/broker/rest/api: java.net.ConnectException: Connection timed out



And both above issues cannot be reproduced on devenv-stage_1175 with multi-node env.
Comment 5 Yan Du 2015-08-24 22:16:57 EDT
Issue still could be reproduced on devenv_stage-1177
Comment 6 Yan Du 2015-08-24 22:27:01 EDT
(In reply to Yan Du from comment #5)
> Issue still could be reproduced on devenv_stage-1177

I mean issue still could be reproduced on STG(devenv_stage-1177)
Comment 7 Miciah Dashiel Butler Masters 2015-08-25 13:05:19 EDT
Ops modified the security-groups rules for STG, and git push now succeeds when I try it, so I am hopeful that the issue is resolved.
Comment 8 Yan Du 2015-08-26 01:07:24 EDT
Test on latest env on STG

1. Normal build is successful now.

remote: Stopping PHP 5.4 cartridge (Apache+mod_php)
remote: Waiting for stop to finish
remote: Waiting for stop to finish
remote: Building git ref 'master', commit c2b53b0
remote: Checking .openshift/pear.txt for PEAR dependency...
remote: Preparing build for deployment
remote: Deployment id is d7341ded
remote: Activating deployment
remote: Starting PHP 5.4 cartridge (Apache+mod_php)
remote: Application directory "/" selected as DocumentRoot
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Activation status: success
remote: Deployment completed with status: success


2. Jenkins build still failed, and got the same error as #Comment 4, please check the log in attachment for detail.
Comment 9 Yan Du 2015-08-26 01:07:53 EDT
Created attachment 1067140 [details]
log
Comment 10 Miciah Dashiel Butler Masters 2015-08-26 23:34:10 EDT
Ops has performed further changes that should resolve the issue.  STG is now using internal.stg.openshift.com, which is a newly, properly configured ELB, whereas stg.openshift.redhat.com is the old ELB that is causing problems.

If you are using an existing Jenkins instance, you will probably need to reconfigure it to use internal.stg.openshift.com instead of stg.openshift.redhat.com.  It may be easiest to delete and recreate any old Jenkins instance you may be using.
Comment 11 Yan Du 2015-08-27 04:31:31 EDT
Test on latest STG env, both jenkins build and normal build work well.


remote: Executing Jenkins build.
remote: 
remote: You can track your build at https://jenkins-111.stg.rhcloud.com/job/app1-build
remote: 
remote: Waiting for build to schedule.....	............................................................Done
remote: Waiting for job to complete........................Done
remote: SUCCESS
remote: New build has been deployed.
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Deployment completed with status: success
To ssh://55dea59bc0c5a7f7ee000086@app1-111.stg.rhcloud.com/~/git/app1.git/
   5054752..e393d29  master -> master


Move to verified.

Note You need to log in before you can comment on or make changes to this bug.