+++ This bug was initially created as a clone of Bug #1153557 +++

Description of problem:
Create a scalable application embeded jenkins client and move this application within district, after moving,failed to deploy change by jenkins build , check the jenkins server web console find the reason is that the project has been changed to disabled.

Version-Release number of selected component (if applicable):
devenv_5233

How reproducible:
always

Steps to Reproduce:
1.Create scalable application, and embed jenkins client to this application.
2.Move this application within district
3.After moving, do change deploy push

Actual results:
Check the jenkins server web console find  that the project has been changed to disabled.
Could not deploy change successfully by jenkins build
$ git push 
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 240 bytes | 0 bytes/s, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: Syncing git content to other proxy gears
remote: Executing Jenkins build.
remote: 
remote: You can track your build at https://jenkinsci-fli.dev.rhcloud.com/job/ruby20s-build
remote: 
remote: ERROR - Couldn't schedule job
remote: !!!!!!!!
remote: Deployment Halted!
remote: If the build failed before the deploy step, your previous
remote: build is still running.  Otherwise, your application may be
remote: partially deployed or inaccessible.
remote: Fix the build and try again.
remote: !!!!!!!!
remote: An error occurred executing 'gear postreceive' (exit code: 1)
remote: Error message: CLIENT_ERROR: Failed to execute: 'control post-receive' for /var/lib/openshift/543faafb9c6aed38aa000175/jenkins-client
remote: 
remote: For more details about the problem, try running the command again with the '--trace' option.
To ssh://543faafb9c6aed38aa000175.rhcloud.com/~/git/ruby20s.git/
   bfe00bb..462b2c8  master -> master



Expected results:

Could deploy change successfully by jenkins build

$ git push 
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 246 bytes | 0 bytes/s, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: Syncing git content to other proxy gears
remote: Executing Jenkins build.
remote: 
remote: You can track your build at https://jenkinsci-fli.dev.rhcloud.com/job/ruby20s-build
remote: 
remote: Waiting for build to schedule..............................Done
remote: Waiting for job to complete.............................Done
remote: SUCCESS
remote: New build has been deployed.
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Deployment completed with status: success
To ssh://543faafb9c6aed38aa000175.rhcloud.com/~/git/ruby20s.git/
   3bdb399..bfe00bb  master -> master


Additional info:

--- Additional comment from Derek Carr on 2014-10-20 13:58:49 EDT ---

I was able to reproduce the issue, but still trying to debug the cause.

--- Additional comment from Derek Carr on 2014-10-20 14:49:21 EDT ---

I reproduced the issue, and noticed after the gear was moved to another node, the Jenkins project appears disabled.  Enabling the project in Jenkins did not address my ability push code, and I continued to get the same error.

Moving to cartridge team per feedback from Dan McPherson.

--- Additional comment from Derek Carr on 2014-10-20 16:09:42 EDT ---

I was able to create the same scenario when not creating a scalable app.

--- Additional comment from Michal Fojtik on 2014-10-21 08:54:48 EDT ---

Derek do you have the devenv running with the broken app and Jenkins? I'm trying to reproduce this to see what is wrong with the Jenkins. Maybe it needs to call some activation routine after move.

--- Additional comment from Marek Mahut on 2014-10-21 11:24:48 EDT ---

Only the first push fails, restart of the builder gears helps as well.

WARNING: Publisher hudson.tasks.ArtifactArchiver aborted due to exception
/var/lib/openshift/544663f12587c8ec8900c5c9/app-root/runtime/repo does not exist.
        at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:483)
        at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:460)
        at hudson.tasks.ArtifactArchiver$ListFiles.invoke(ArtifactArchiver.java:181)
        at hudson.tasks.ArtifactArchiver$ListFiles.invoke(ArtifactArchiver.java:172)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
        at hudson.remoting.UserRequest.perform(UserRequest.java:118)
        at hudson.remoting.UserRequest.perform(UserRequest.java:48)
        at hudson.remoting.Request$2.run(Request.java:328)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)


Need further investigation.

--- Additional comment from Ben Parees on 2014-10-21 16:03:46 EDT ---

As discussed on the scrum this morning, this might be caused the an ssh man in the middle attack warning, need to try not writing the known_hosts file.

--- Additional comment from Michal Fojtik on 2014-10-22 14:32:51 EDT ---

The PR: https://github.com/openshift/origin-server/pull/5898

We are still not sure if this fixes the problem. I did verification on my devenv, but I will need Ops to verify this fix in INT (will do that tomorrow).