Description of problem: Create a scalable application embeded jenkins client and move this application within district, after moving,failed to deploy change by jenkins build , check the jenkins server web console find the reason is that the project has been changed to disabled. Version-Release number of selected component (if applicable): devenv_5233 How reproducible: always Steps to Reproduce: 1.Create scalable application, and embed jenkins client to this application. 2.Move this application within district 3.After moving, do change deploy push Actual results: Check the jenkins server web console find that the project has been changed to disabled. Could not deploy change successfully by jenkins build $ git push Counting objects: 3, done. Delta compression using up to 4 threads. Compressing objects: 100% (2/2), done. Writing objects: 100% (2/2), 240 bytes | 0 bytes/s, done. Total 2 (delta 1), reused 0 (delta 0) remote: Syncing git content to other proxy gears remote: Executing Jenkins build. remote: remote: You can track your build at https://jenkinsci-fli.dev.rhcloud.com/job/ruby20s-build remote: remote: ERROR - Couldn't schedule job remote: !!!!!!!! remote: Deployment Halted! remote: If the build failed before the deploy step, your previous remote: build is still running. Otherwise, your application may be remote: partially deployed or inaccessible. remote: Fix the build and try again. remote: !!!!!!!! remote: An error occurred executing 'gear postreceive' (exit code: 1) remote: Error message: CLIENT_ERROR: Failed to execute: 'control post-receive' for /var/lib/openshift/543faafb9c6aed38aa000175/jenkins-client remote: remote: For more details about the problem, try running the command again with the '--trace' option. To ssh://543faafb9c6aed38aa000175.rhcloud.com/~/git/ruby20s.git/ bfe00bb..462b2c8 master -> master Expected results: Could deploy change successfully by jenkins build $ git push Counting objects: 3, done. Delta compression using up to 4 threads. Compressing objects: 100% (2/2), done. Writing objects: 100% (2/2), 246 bytes | 0 bytes/s, done. Total 2 (delta 1), reused 0 (delta 0) remote: Syncing git content to other proxy gears remote: Executing Jenkins build. remote: remote: You can track your build at https://jenkinsci-fli.dev.rhcloud.com/job/ruby20s-build remote: remote: Waiting for build to schedule..............................Done remote: Waiting for job to complete.............................Done remote: SUCCESS remote: New build has been deployed. remote: ------------------------- remote: Git Post-Receive Result: success remote: Deployment completed with status: success To ssh://543faafb9c6aed38aa000175.rhcloud.com/~/git/ruby20s.git/ 3bdb399..bfe00bb master -> master Additional info:
I was able to reproduce the issue, but still trying to debug the cause.
I reproduced the issue, and noticed after the gear was moved to another node, the Jenkins project appears disabled. Enabling the project in Jenkins did not address my ability push code, and I continued to get the same error. Moving to cartridge team per feedback from Dan McPherson.
I was able to create the same scenario when not creating a scalable app.
Only the first push fails, restart of the builder gears helps as well. WARNING: Publisher hudson.tasks.ArtifactArchiver aborted due to exception /var/lib/openshift/544663f12587c8ec8900c5c9/app-root/runtime/repo does not exist. at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:483) at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:460) at hudson.tasks.ArtifactArchiver$ListFiles.invoke(ArtifactArchiver.java:181) at hudson.tasks.ArtifactArchiver$ListFiles.invoke(ArtifactArchiver.java:172) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:328) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Need further investigation.
As discussed on the scrum this morning, this might be caused the an ssh man in the middle attack warning, need to try not writing the known_hosts file.
The PR: https://github.com/openshift/origin-server/pull/5898 We are still not sure if this fixes the problem. I did verification on my devenv, but I will need Ops to verify this fix in INT (will do that tomorrow).
I have tested it after i modified the file " /var/lib/openshift/.cartridge_repository/redhat-jenkins/0.0.16/bin/git_ssh_wrapper.sh" depend on The PR content and restarted online-broker service. it works fine now for scalable and non-scalable app.
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/86a838bfda61878067781e92a7a4428987b60cb9 Bug 1153557 - Disable strict known_hosts checking in Jenkins
Fulei Li: The PR is now merged
Checked on devenv_5262 Jenkins build for both scale and non-scale apps are working fine now. Move bug to verified.