Bug 1013421

Summary:	[deploy] Jenkins builds always fail for all cartridges
Product:	OpenShift Online	Reporter:	Zhe Wang <zhewang>
Component:	Containers	Assignee:	Paul Morie <pmorie>
Status:	CLOSED CURRENTRELEASE	QA Contact:	libra bugs <libra-bugs>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	2.x	CC:	agoldste, bmeng, lzhang, pmorie, xtian
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-10-17 13:32:51 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Zhe Wang 2013-09-30 02:48:27 UTC

Description of problem:
Given a JBoss apps (JBossEWS 1.0, 2.0, JBossEAP, JBossAS) with Jenkins-client added, its deployment always fails with the following error:

remote: rsync: mkdir "/var/lib/openshift/5248dd10c667f6beca000007/jbossews//webapps" failed: File exists (17)
remote: rsync error: error in file IO (code 11) at main.c(576) [receiver=3.0.6]
remote: rsync: connection unexpectedly closed (9 bytes received so far) [sender]
remote: rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
remote: An error occurred executing 'gear postreceive' (exit code: 12)
remote: Error message: Failed to execute: 'control deploy' for /var/lib/openshift/5248dd10c667f6beca000007/jbossews


Version-Release number of selected component (if applicable):
fork_ami_deploy_873

How reproducible:
always

Steps to Reproduce:
1. create a JBoss app (JBossEWS 1.0, 2.0, JBossEAP, JBossAS) WITHOUT Jenkins-client
2. make some change and trigger a deployment
3. add a jenkins-client to this app
4. trigger another deployment

Actual results:
The first build without jenkins-client succeeded, but the one after adding the Jenins-client failed with the error in the build log:

remote: + rsync -r --delete /var/lib/openshift/5248dd10c667f6beca000007/app-root/runtime/repo//webapps/ /var/lib/openshift/5248dd10c667f6beca000007/jbossews//webapps/
remote: rsync: mkdir "/var/lib/openshift/5248dd10c667f6beca000007/jbossews//webapps" failed: File exists (17)
remote: rsync error: error in file IO (code 11) at main.c(576) [receiver=3.0.6]
remote: rsync: connection unexpectedly closed (9 bytes received so far) [sender]
remote: rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
remote: An error occurred executing 'gear postreceive' (exit code: 12)
remote: Error message: Failed to execute: 'control deploy' for /var/lib/openshift/5248dd10c667f6beca000007/jbossews
remote: 
remote: For more details about the problem, try running the command again with the '--trace' option.
To ssh://5248dd10c667f6beca000007.rhcloud.com/~/git/ews2.git/


Expected results:
The Jenkins build of the JBoss app should be successful.

Additional info:

Comment 1 Zhe Wang 2013-09-30 02:57:18 UTC

I also found this problem in ruby-1.9 app:

+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/5248e7fcc667f6beca000218/app-root/runtime/build-dependencies/ '5248e7a9c667f6beca0001f6.rhcloud.com:$OPENSHIFT_BUILD_DEPENDENCIES_DIR'

rsync: mkdir "/var/lib/openshift/5248e7a9c667f6beca0001f6/app-root/runtime/build-dependencies" failed: File exists (17)
rsync error: error in file IO (code 11) at main.c(576) [receiver=3.0.6]
rsync: connection unexpectedly closed (9 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
Build step 'Execute shell' marked build as failure
Archiving artifacts

Comment 2 Meng Bo 2013-09-30 09:55:02 UTC

This issue also happened for scalable apps,

Waiting for stop to finish
++ /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh 524948e547b357724400010f.rhcloud.com 'gear create_deployment_dir'
+ deployment_dir=2013-09-30_05-52-29.303
+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/524949a147b357724400014b/app-root/runtime/repo/ 524948e547b357724400010f.rhcloud.com:app-deployments/2013-09-30_05-52-29.303/repo/
+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/524949a147b357724400014b/app-root/runtime/build-dependencies/ '524948e547b357724400010f.rhcloud.com:$OPENSHIFT_BUILD_DEPENDENCIES_DIR'
rsync: mkdir "/var/lib/openshift/524948e547b357724400010f/app-root/runtime/build-dependencies" failed: File exists (17)
rsync error: error in file IO (code 11) at main.c(576) [receiver=3.0.6]
rsync: connection unexpectedly closed (9 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
Build step 'Execute shell' marked build as failure
Archiving artifacts
Finished: FAILURE

Comment 3 Paul Morie 2013-10-01 17:17:39 UTC

I have added a scenario locally to reproduce this scenario in scaling_func_test.  It wasn't captured previously because all tests were run with OPENSHIFT_KEEP_DEPLOYMENTS set to 3.  This appears only to manifest when KEEP_DEPLOYMENTS is 1

Comment 4 Lei Zhang 2013-10-09 03:16:59 UTC

Test on devenv_3874, Jenkins builds of all apps always fail

Comment 5 Lei Zhang 2013-10-09 04:42:18 UTC

[repo] $ /bin/sh -xe /tmp/hudson6941463946891258127.sh
+ alias 'rsync=rsync --delete-after -az -e '\''/usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh'\'''
+ upstream_ssh=5254dc5f469ff993b4000007.rhcloud.com
+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh '5254dc5f469ff993b4000007.rhcloud.com:$OPENSHIFT_BUILD_DEPENDENCIES_DIR' /var/lib/openshift/5254ddc7469ff993b4000055/app-root/runtime/build-dependencies/
+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh '5254dc5f469ff993b4000007.rhcloud.com:$OPENSHIFT_DEPENDENCIES_DIR' /var/lib/openshift/5254ddc7469ff993b4000055/app-root/runtime/dependencies/
+ gear build
Building PHP cartridge
+ /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh 5254dc5f469ff993b4000007.rhcloud.com 'gear stop --conditional --exclude-web-proxy'
Stopping gear...
Stopping PHP cartridge
Waiting for stop to finish
++ /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh 5254dc5f469ff993b4000007.rhcloud.com 'gear create_deployment_dir'
+ deployment_dir=2013-10-09_00-39-47.750
+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/5254ddc7469ff993b4000055/app-root/runtime/repo/ 5254dc5f469ff993b4000007.rhcloud.com:app-deployments/2013-10-09_00-39-47.750/repo/
+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/5254ddc7469ff993b4000055/app-root/runtime/build-dependencies/ '5254dc5f469ff993b4000007.rhcloud.com:$OPENSHIFT_BUILD_DEPENDENCIES_DIR'
rsync: mkdir "/var/lib/openshift/5254dc5f469ff993b4000007/app-root/runtime/build-dependencies" failed: File exists (17)
rsync error: error in file IO (code 11) at main.c(576) [receiver=3.0.6]
rsync: connection unexpectedly closed (9 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
Build step 'Execute shell' marked build as failure
Archiving artifacts
Finished: FAILURE

Comment 6 Andy Goldstein 2013-10-09 17:46:00 UTC

https://github.com/openshift/origin-server/pull/3819

Comment 7 Zhe Wang 2013-10-10 06:17:47 UTC

Will verify this bug after the fix is merged.

Thanks,
z.

Comment 8 Zhe Wang 2013-10-12 01:41:15 UTC

Not find this bug in the auto testing against devenv_3886, move this bug to VERIFIED.