Bug 1013421 - [deploy] Jenkins builds always fail for all cartridges
[deploy] Jenkins builds always fail for all cartridges
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Paul Morie
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-29 22:48 EDT by Zhe Wang
Modified: 2015-05-14 19:29 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-17 09:32:51 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Zhe Wang 2013-09-29 22:48:27 EDT
Description of problem:
Given a JBoss apps (JBossEWS 1.0, 2.0, JBossEAP, JBossAS) with Jenkins-client added, its deployment always fails with the following error:

remote: rsync: mkdir "/var/lib/openshift/5248dd10c667f6beca000007/jbossews//webapps" failed: File exists (17)
remote: rsync error: error in file IO (code 11) at main.c(576) [receiver=3.0.6]
remote: rsync: connection unexpectedly closed (9 bytes received so far) [sender]
remote: rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
remote: An error occurred executing 'gear postreceive' (exit code: 12)
remote: Error message: Failed to execute: 'control deploy' for /var/lib/openshift/5248dd10c667f6beca000007/jbossews


Version-Release number of selected component (if applicable):
fork_ami_deploy_873

How reproducible:
always

Steps to Reproduce:
1. create a JBoss app (JBossEWS 1.0, 2.0, JBossEAP, JBossAS) WITHOUT Jenkins-client
2. make some change and trigger a deployment
3. add a jenkins-client to this app
4. trigger another deployment

Actual results:
The first build without jenkins-client succeeded, but the one after adding the Jenins-client failed with the error in the build log:

remote: + rsync -r --delete /var/lib/openshift/5248dd10c667f6beca000007/app-root/runtime/repo//webapps/ /var/lib/openshift/5248dd10c667f6beca000007/jbossews//webapps/
remote: rsync: mkdir "/var/lib/openshift/5248dd10c667f6beca000007/jbossews//webapps" failed: File exists (17)
remote: rsync error: error in file IO (code 11) at main.c(576) [receiver=3.0.6]
remote: rsync: connection unexpectedly closed (9 bytes received so far) [sender]
remote: rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
remote: An error occurred executing 'gear postreceive' (exit code: 12)
remote: Error message: Failed to execute: 'control deploy' for /var/lib/openshift/5248dd10c667f6beca000007/jbossews
remote: 
remote: For more details about the problem, try running the command again with the '--trace' option.
To ssh://5248dd10c667f6beca000007@ews2-fork873tst.dev.rhcloud.com/~/git/ews2.git/


Expected results:
The Jenkins build of the JBoss app should be successful.

Additional info:
Comment 1 Zhe Wang 2013-09-29 22:57:18 EDT
I also found this problem in ruby-1.9 app:

+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/5248e7fcc667f6beca000218/app-root/runtime/build-dependencies/ '5248e7a9c667f6beca0001f6@rb9-fork873tst.dev.rhcloud.com:$OPENSHIFT_BUILD_DEPENDENCIES_DIR'

rsync: mkdir "/var/lib/openshift/5248e7a9c667f6beca0001f6/app-root/runtime/build-dependencies" failed: File exists (17)
rsync error: error in file IO (code 11) at main.c(576) [receiver=3.0.6]
rsync: connection unexpectedly closed (9 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
Build step 'Execute shell' marked build as failure
Archiving artifacts
Comment 2 Meng Bo 2013-09-30 05:55:02 EDT
This issue also happened for scalable apps,

Waiting for stop to finish
++ /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh 524948e547b357724400010f@perl1s-bmengdev.dev.rhcloud.com 'gear create_deployment_dir'
+ deployment_dir=2013-09-30_05-52-29.303
+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/524949a147b357724400014b/app-root/runtime/repo/ 524948e547b357724400010f@perl1s-bmengdev.dev.rhcloud.com:app-deployments/2013-09-30_05-52-29.303/repo/
+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/524949a147b357724400014b/app-root/runtime/build-dependencies/ '524948e547b357724400010f@perl1s-bmengdev.dev.rhcloud.com:$OPENSHIFT_BUILD_DEPENDENCIES_DIR'
rsync: mkdir "/var/lib/openshift/524948e547b357724400010f/app-root/runtime/build-dependencies" failed: File exists (17)
rsync error: error in file IO (code 11) at main.c(576) [receiver=3.0.6]
rsync: connection unexpectedly closed (9 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
Build step 'Execute shell' marked build as failure
Archiving artifacts
Finished: FAILURE
Comment 3 Paul Morie 2013-10-01 13:17:39 EDT
I have added a scenario locally to reproduce this scenario in scaling_func_test.  It wasn't captured previously because all tests were run with OPENSHIFT_KEEP_DEPLOYMENTS set to 3.  This appears only to manifest when KEEP_DEPLOYMENTS is 1
Comment 4 Lei Zhang 2013-10-08 23:16:59 EDT
Test on devenv_3874, Jenkins builds of all apps always fail
Comment 5 Lei Zhang 2013-10-09 00:42:18 EDT
[repo] $ /bin/sh -xe /tmp/hudson6941463946891258127.sh
+ alias 'rsync=rsync --delete-after -az -e '\''/usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh'\'''
+ upstream_ssh=5254dc5f469ff993b4000007@myphp53-chunchen.dev.rhcloud.com
+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh '5254dc5f469ff993b4000007@myphp53-chunchen.dev.rhcloud.com:$OPENSHIFT_BUILD_DEPENDENCIES_DIR' /var/lib/openshift/5254ddc7469ff993b4000055/app-root/runtime/build-dependencies/
+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh '5254dc5f469ff993b4000007@myphp53-chunchen.dev.rhcloud.com:$OPENSHIFT_DEPENDENCIES_DIR' /var/lib/openshift/5254ddc7469ff993b4000055/app-root/runtime/dependencies/
+ gear build
Building PHP cartridge
+ /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh 5254dc5f469ff993b4000007@myphp53-chunchen.dev.rhcloud.com 'gear stop --conditional --exclude-web-proxy'
Stopping gear...
Stopping PHP cartridge
Waiting for stop to finish
++ /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh 5254dc5f469ff993b4000007@myphp53-chunchen.dev.rhcloud.com 'gear create_deployment_dir'
+ deployment_dir=2013-10-09_00-39-47.750
+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/5254ddc7469ff993b4000055/app-root/runtime/repo/ 5254dc5f469ff993b4000007@myphp53-chunchen.dev.rhcloud.com:app-deployments/2013-10-09_00-39-47.750/repo/
+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/5254ddc7469ff993b4000055/app-root/runtime/build-dependencies/ '5254dc5f469ff993b4000007@myphp53-chunchen.dev.rhcloud.com:$OPENSHIFT_BUILD_DEPENDENCIES_DIR'
rsync: mkdir "/var/lib/openshift/5254dc5f469ff993b4000007/app-root/runtime/build-dependencies" failed: File exists (17)
rsync error: error in file IO (code 11) at main.c(576) [receiver=3.0.6]
rsync: connection unexpectedly closed (9 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
Build step 'Execute shell' marked build as failure
Archiving artifacts
Finished: FAILURE
Comment 6 Andy Goldstein 2013-10-09 13:46:00 EDT
https://github.com/openshift/origin-server/pull/3819
Comment 7 Zhe Wang 2013-10-10 02:17:47 EDT
Will verify this bug after the fix is merged.

Thanks,
z.
Comment 8 Zhe Wang 2013-10-11 21:41:15 EDT
Not find this bug in the auto testing against devenv_3886, move this bug to VERIFIED.

Note You need to log in before you can comment on or make changes to this bug.