Bug 1153557

Summary:	Moving the application with jenkins client make the project be changed to disabled in jenkins server console
Product:	OpenShift Online	Reporter:	Fulei Li <fli>
Component:	Image	Assignee:	Michal Fojtik <mfojtik>
Status:	CLOSED CURRENTRELEASE	QA Contact:	libra bugs <libra-bugs>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	2.x	CC:	bparees, decarr, jokerman, mmahut, mmccomas, wsun
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1155766 (view as bug list)		Environment:
Last Closed:	2015-02-18 16:52:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1155766

Description Fulei Li 2014-10-16 08:36:33 UTC

Description of problem:
Create a scalable application embeded jenkins client and move this application within district, after moving,failed to deploy change by jenkins build , check the jenkins server web console find the reason is that the project has been changed to disabled.

Version-Release number of selected component (if applicable):
devenv_5233

How reproducible:
always

Steps to Reproduce:
1.Create scalable application, and embed jenkins client to this application.
2.Move this application within district
3.After moving, do change deploy push

Actual results:
Check the jenkins server web console find that the project has been changed to disabled.
Could not deploy change successfully by jenkins build
$ git push
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 240 bytes | 0 bytes/s, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: Syncing git content to other proxy gears
remote: Executing Jenkins build.
remote:
remote: You can track your build at https://jenkinsci-fli.dev.rhcloud.com/job/ruby20s-build
remote:
remote: ERROR - Couldn't schedule job
remote: !!!!!!!!
remote: Deployment Halted!
remote: If the build failed before the deploy step, your previous
remote: build is still running. Otherwise, your application may be
remote: partially deployed or inaccessible.
remote: Fix the build and try again.
remote: !!!!!!!!
remote: An error occurred executing 'gear postreceive' (exit code: 1)
remote: Error message: CLIENT_ERROR: Failed to execute: 'control post-receive' for /var/lib/openshift/543faafb9c6aed38aa000175/jenkins-client
remote:
remote: For more details about the problem, try running the command again with the '--trace' option.
To ssh://543faafb9c6aed38aa000175.rhcloud.com/~/git/ruby20s.git/
bfe00bb..462b2c8 master -> master

Expected results:

Could deploy change successfully by jenkins build

$ git push
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 246 bytes | 0 bytes/s, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: Syncing git content to other proxy gears
remote: Executing Jenkins build.
remote:
remote: You can track your build at https://jenkinsci-fli.dev.rhcloud.com/job/ruby20s-build
remote:
remote: Waiting for build to schedule..............................Done
remote: Waiting for job to complete.............................Done
remote: SUCCESS
remote: New build has been deployed.
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Deployment completed with status: success
To ssh://543faafb9c6aed38aa000175.rhcloud.com/~/git/ruby20s.git/
3bdb399..bfe00bb master -> master

Additional info:

Comment 1 Derek Carr 2014-10-20 17:58:49 UTC

I was able to reproduce the issue, but still trying to debug the cause.

Comment 2 Derek Carr 2014-10-20 18:49:21 UTC

I reproduced the issue, and noticed after the gear was moved to another node, the Jenkins project appears disabled.  Enabling the project in Jenkins did not address my ability push code, and I continued to get the same error.

Moving to cartridge team per feedback from Dan McPherson.

Comment 3 Derek Carr 2014-10-20 20:09:42 UTC

I was able to create the same scenario when not creating a scalable app.

Comment 5 Marek Mahut 2014-10-21 15:24:48 UTC

Only the first push fails, restart of the builder gears helps as well.

WARNING: Publisher hudson.tasks.ArtifactArchiver aborted due to exception
/var/lib/openshift/544663f12587c8ec8900c5c9/app-root/runtime/repo does not exist.
        at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:483)
        at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:460)
        at hudson.tasks.ArtifactArchiver$ListFiles.invoke(ArtifactArchiver.java:181)
        at hudson.tasks.ArtifactArchiver$ListFiles.invoke(ArtifactArchiver.java:172)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
        at hudson.remoting.UserRequest.perform(UserRequest.java:118)
        at hudson.remoting.UserRequest.perform(UserRequest.java:48)
        at hudson.remoting.Request$2.run(Request.java:328)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)


Need further investigation.

Comment 6 Ben Parees 2014-10-21 20:03:46 UTC

As discussed on the scrum this morning, this might be caused the an ssh man in the middle attack warning, need to try not writing the known_hosts file.

Comment 7 Michal Fojtik 2014-10-22 18:32:51 UTC

The PR: https://github.com/openshift/origin-server/pull/5898

We are still not sure if this fixes the problem. I did verification on my devenv, but I will need Ops to verify this fix in INT (will do that tomorrow).

Comment 8 Fulei Li 2014-10-23 10:21:02 UTC

I have tested it after i modified the file " /var/lib/openshift/.cartridge_repository/redhat-jenkins/0.0.16/bin/git_ssh_wrapper.sh" depend on The PR content and restarted online-broker service. it works fine now for scalable and non-scalable app.

Comment 9 openshift-github-bot 2014-10-23 11:21:19 UTC

Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/86a838bfda61878067781e92a7a4428987b60cb9
Bug 1153557 - Disable strict known_hosts checking in Jenkins

Comment 10 Michal Fojtik 2014-10-23 11:50:51 UTC

Fulei Li: The PR is now merged

Comment 11 Fulei Li 2014-10-24 05:39:34 UTC

Checked on devenv_5262 

Jenkins build for both scale and non-scale apps are working fine now.

Move bug to verified.