Description of problem: Given one normal app embedded with jenkins-client created(e.g., myjbossews10), change something in local git repo, Build Failed. [rayzhang@rayzhang myjbossews10]$ git add -A && git commit -amp && git push [master 55c3055] p 1 file changed, 1 insertion(+), 1 deletion(-) warning: push.default is unset; its implicit value is changing in Git 2.0 from 'matching' to 'simple'. To squelch this message and maintain the current behavior after the default changes, use: git config --global push.default matching To squelch this message and adopt the new behavior now, use: git config --global push.default simple See 'git help config' and search for 'push.default' for further information. (the 'simple' mode was introduced in Git 1.7.11. Use the similar mode 'current' instead of 'simple' if you sometimes use older versions of Git) Counting objects: 11, done. Delta compression using up to 8 threads. Compressing objects: 100% (5/5), done. Writing objects: 100% (6/6), 434 bytes, done. Total 6 (delta 4), reused 0 (delta 0) remote: Executing Jenkins build. remote: remote: You can track your build at https://jenkins-rayzhang.int.rhcloud.com/job/myjbossews10-build remote: remote: Waiting for build to schedule....................... remote: **BUILD FAILED/CANCELLED** remote: Please see the Jenkins log for more details via 'rhc tail' remote: !!!!!!!! remote: Deployment Halted! remote: If the build failed before the deploy step, your previous remote: build is still running. Otherwise, your application may be remote: partially deployed or inaccessible. remote: Fix the build and try again. remote: !!!!!!!! remote: An error occurred executing 'gear postreceive' (exit code: 1) remote: Error message: Failed to execute: 'control post-receive' for /var/lib/openshift/520c2cb403ef6490b5000070/jenkins-client remote: remote: For more details about the problem, try running the command again with the '--trace' option. To ssh://520c2cb403ef6490b5000070.rhcloud.com/~/git/myjbossews10.git/ 00e58e5..55c3055 master -> master Version-Release number of selected component (if applicable): INT(devenv_3652) rhc-1.13.1 How reproducible: always Steps to Reproduce: 1.Create one app embedded with jenkins-client # rhc app create myjbossews10 jbossews-1.0 --enable-jenkins 2.change something in local git repo and git push # sed -i 's/Welcome/Hi/g' src/main/webapp/index.html && git add -A && git commit -amp && git push 3. Actual results: Build failed with "Error message: Failed to execute: 'control post-receive' for /var/lib/openshift/520c2f826cec0ed30f000095/jenkins-client" Expected results: App should build successfully embedded jenkins-client Additional info:
Here is the log in Jenkins: Aug 14, 2013 11:40:11 PM com.openshift.internal.client.RestService request INFO: Requesting GET on https://int.openshift.redhat.com/broker/rest/user Aug 14, 2013 11:40:11 PM hudson.plugins.openshift.OpenShiftCloud provision WARNING: Caught com.openshift.client.OpenShiftEndpointException: Could not request https://int.openshift.redhat.com/broker/rest/user: Operation failed with exit code "1".Reason given: "Unable to complete the requested operation due to: undefined method `__bson_dump__' for #<Domain:0x000000056ec4e0>. Reference ID: c3c7ffda964bddfd0b96e727f9c35563" . Will retry 0 more times before canceling build. com.openshift.client.OpenShiftEndpointException: Could not request https://int.openshift.redhat.com/broker/rest/user: Operation failed with exit code "1".Reason given: "Unable to complete the requested operation due to: undefined method `__bson_dump__' for #<Domain:0x000000056ec4e0>. Reference ID: c3c7ffda964bddfd0b96e727f9c35563" at com.openshift.internal.client.RestService.request(RestService.java:106) at com.openshift.internal.client.RestService.request(RestService.java:91) at com.openshift.internal.client.RestService.request(RestService.java:76) at com.openshift.internal.client.AbstractOpenShiftResource$ServiceRequest.execute(AbstractOpenShiftResource.java:124) at com.openshift.internal.client.APIResource$GetUserRequest.execute(APIResource.java:239) at com.openshift.internal.client.APIResource.getUser(APIResource.java:98) at hudson.plugins.openshift.OpenShiftCloud.getSlaves(OpenShiftCloud.java:859) at hudson.plugins.openshift.OpenShiftCloud.provisionSlave(OpenShiftCloud.java:436) at hudson.plugins.openshift.OpenShiftCloud.provision(OpenShiftCloud.java:401) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:264) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:347) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Caused by: com.openshift.internal.client.httpclient.HttpClientException: {"api_version":1.0,"data":null,"messages":[{"exit_code":1,"field":null,"severity":"error","text":"Unable to complete the requested operation due to: undefined method `__bson_dump__' for #<Domain:0x000000056ec4e0>.\nReference ID: c3c7ffda964bddfd0b96e727f9c35563"}],"status":"internal_server_error","supported_api_versions":[1.0,1.1,1.2,1.3,1.4,1.5,1.6],"type":null,"version":"1.0"} at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.createException(UrlConnectionHttpClient.java:193) at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.get(UrlConnectionHttpClient.java:102) at com.openshift.internal.client.RestService.request(RestService.java:144) at com.openshift.internal.client.RestService.request(RestService.java:98) ... 14 more Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: https://int.openshift.redhat.com/broker/rest/user at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1625) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254) at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.get(UrlConnectionHttpClient.java:100) ... 16 more On the other hand, it succeeded when I manually tried to call the User API with REST API 1.0. curl -k -H 'Accept: application/json, version=1.0' -u zhewang+1:redhat https://int.openshift.redhat.com/broker/rest/user | python -m json.tool
New changes to the repo are also not visible in the git push after a jenkins client cartridge has been attached. For example: 0. Create a jenkins app #rhc app create j14 jenkins-1.4 1. Create a ruby app #rhc app-create app1 ruby-1.8 2. Change something in the repo, do git push #cd app1 && touch thing1 #git add . ; git commit -am 'test' ; git push ***(change shows in remote)*** 3. Add a jenkins client to the app #rhc cartridge add jenkins-client-1.4 -a app1 4. Change something in the repo, do git push #touch thing2 #git add . ; git commit -am 'test' ; git push ***(change not visible in remote, and build fails)***
*** Bug 997366 has been marked as a duplicate of this bug. ***
*** Bug 997261 has been marked as a duplicate of this bug. ***
*** Bug 997343 has been marked as a duplicate of this bug. ***
Fixed in https://github.com/openshift/origin-server/pull/3368 (package installation issue is causing this merge to fail, once that is fixed it'll go in).
Retest on devenv_3658, those three bugs(#997366 #997261 #997343) can not be reproduced,but #997234 still exits. [rayzhang@rayzhang myruby18]$ git add -A && git commit -amp && git push [master c542e03] p 1 file changed, 1 insertion(+), 1 deletion(-) warning: push.default is unset; its implicit value is changing in Git 2.0 from 'matching' to 'simple'. To squelch this message and maintain the current behavior after the default changes, use: git config --global push.default matching To squelch this message and adopt the new behavior now, use: git config --global push.default simple See 'git help config' and search for 'push.default' for further information. (the 'simple' mode was introduced in Git 1.7.11. Use the similar mode 'current' instead of 'simple' if you sometimes use older versions of Git) Counting objects: 5, done. Delta compression using up to 8 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 280 bytes, done. Total 3 (delta 2), reused 0 (delta 0) remote: Executing Jenkins build. remote: remote: You can track your build at https://jenkins-rayzhang.dev.rhcloud.com/job/myruby18-build remote: remote: Waiting for build to schedule.............................. remote: **BUILD FAILED/CANCELLED** remote: Please see the Jenkins log for more details via 'rhc tail' remote: !!!!!!!! remote: Deployment Halted! remote: If the build failed before the deploy step, your previous remote: build is still running. Otherwise, your application may be remote: partially deployed or inaccessible. remote: Fix the build and try again. remote: !!!!!!!! remote: An error occurred executing 'gear postreceive' (exit code: 1) remote: Error message: Failed to execute: 'control post-receive' for /var/lib/openshift/abeb8518061211e3b30122000aa4690c/jenkins-client remote: remote: For more details about the problem, try running the command again with the '--trace' option. To ssh://abeb8518061211e3b30122000aa4690c.rhcloud.com/~/git/myruby18.git/ 6f6eb1a..c542e03 master -> master
This bug is reproducible in devenv_3660, and the errors in Jenkins logs are: Aug 16, 2013 4:18:26 AM hudson.plugins.openshift.OpenShiftCloud hasCapacity INFO: Checking capacity Aug 16, 2013 4:18:26 AM hudson.plugins.openshift.OpenShiftCloud hasCapacity INFO: User has consumed 5 of 100 gears. Aug 16, 2013 4:18:26 AM hudson.plugins.openshift.OpenShiftCloud reloadConfig INFO: Reloading configuration for phptst-build... Aug 16, 2013 4:18:26 AM hudson.plugins.openshift.OpenShiftCloud reloadConfig INFO: Retrieving config XML from http://127.0.252.129:8080/job/phptst-build/config.xml Aug 16, 2013 4:18:26 AM hudson.plugins.openshift.OpenShiftCloud createConnection INFO: Using credentials system_builder:Id3dELmcQkb- Aug 16, 2013 4:18:26 AM hudson.plugins.openshift.OpenShiftCloud reloadConfig INFO: Reloading config from XML: <?xml version="1.0" encoding="UTF-8"?><project> <actions/> <description/> <logRotator> <daysToKeep>-1</daysToKeep> <numToKeep>100</numToKeep> <artifactDaysToKeep>-1</artifactDaysToKeep> <artifactNumToKeep>10</artifactNumToKeep> </logRotator> <keepDependencies>false</keepDependencies> <properties> <hudson.plugins.openshift.OpenShiftBuilderSizeJobProperty> <builderSize>small</builderSize> </hudson.plugins.openshift.OpenShiftBuilderSizeJobProperty> <hudson.plugins.openshift.OpenShiftBuilderTypeJobProperty> <builderType>php-5.3</builderType> </hudson.plugins.openshift.OpenShiftBuilderTypeJobProperty> <hudson.plugins.openshift.OpenShiftBuilderTimeoutJobProperty> <builderTimeout>300000</builderTimeout> </hudson.plugins.openshift.OpenShiftBuilderTimeoutJobProperty> </properties> <scm class="hudson.plugins.git.GitSCM"> <configVersion>2</configVersion> <userRemoteConfigs> <hudson.plugins.git.UserRemoteConfig> <name/> <refspec/> <url>ssh://520dd18cfe36160aab000006.rhcloud.com/~/git/phptst.git</url> </hudson.plugins.git.UserRemoteConfig> </userRemoteConfigs> <branches> <hudson.plugins.git.BranchSpec> <name>**</name> </hudson.plugins.git.BranchSpec> </branches> <recursiveSubmodules>false</recursiveSubmodules> <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations> <authorOrCommitter>false</authorOrCommitter> <clean>false</clean> <wipeOutWorkspace>false</wipeOutWorkspace> <pruneBranches>false</pruneBranches> <remotePoll>false</remotePoll> <buildChooser class="hudson.plugins.git.util.DefaultBuildChooser"/> <gitTool>Default</gitTool> <submoduleCfg class="list"/> <relativeTargetDir/> <excludedRegions/> <excludedUsers/> <gitConfigName/> <gitConfigEmail/> <skipTag>false</skipTag> <scmName/> </scm> <assignedNode>phptst-build</assignedNode> <canRoam>false</canRoam> <disabled>false</disabled> <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding> <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding> <triggers class="vector"/> <concurrentBuild>false</concurrentBuild> <customWorkspace>$OPENSHIFT_REPO_DIR</customWorkspace> <builders> <hudson.tasks.Shell> <command> alias rsync="rsync --delete-after -azO -e '$GIT_SSH'" upstream_ssh="520dd18cfe36160aab000006@phptst-${OPENSHIFT_NAMESPACE}.dev.rhcloud.com" # Sync any libraries rsync $upstream_ssh:'${OPENSHIFT_PHP_DIR}phplib/' ${OPENSHIFT_PHP_DIR}phplib/ # Build/update libs and run user pre_build and build gear build # Run tests here # Deploy new build # Stop app $GIT_SSH $upstream_ssh 'gear stop --conditional' # Push content back to application rsync ${OPENSHIFT_PHP_DIR}phplib/ $upstream_ssh:'${OPENSHIFT_PHP_DIR}phplib/' rsync $WORKSPACE/ $upstream_ssh:'${OPENSHIFT_REPO_DIR}' # Configure / start app $GIT_SSH $upstream_ssh 'gear remotedeploy' </command> </hudson.tasks.Shell> </builders> <publishers> <hudson.tasks.ArtifactArchiver> <artifacts>**</artifacts> <latestOnly>false</latestOnly> </hudson.tasks.ArtifactArchiver> </publishers> <buildWrappers/> </project> Aug 16, 2013 4:18:26 AM hudson.plugins.openshift.OpenShiftCloud createConnection INFO: Using credentials system_builder:Id3dELmcQkb- Aug 16, 2013 4:18:26 AM hudson.plugins.openshift.OpenShiftCloud reloadConfig INFO: Reload ResponseCode: 200 Aug 16, 2013 4:18:26 AM hudson.plugins.openshift.OpenShiftCloud reloadConfig INFO: Config reload result: Aug 16, 2013 4:18:26 AM hudson.plugins.openshift.OpenShiftSlave <init> INFO: Creating slave with 15mins time-to-live Aug 16, 2013 4:18:26 AM hudson.plugins.openshift.OpenShiftSlave createApp INFO: Creating builder application php-5.3 phptstbldr dev3660tst of size small ... Aug 16, 2013 4:18:26 AM com.openshift.internal.client.RestService request INFO: Requesting POST on https://localhost/broker/rest/domains/dev3660tst/applications Aug 16, 2013 4:18:26 AM hudson.plugins.openshift.OpenShiftCloud provision WARNING: Caught com.openshift.client.OpenShiftEndpointException: Could not request https://localhost/broker/rest/domains/dev3660tst/applications: Operation failed.Reason given: "You are not permitted to perform this action with the scopes application/520ddf12fe361672040002fb/scale application/520ddf12fe361672040002fb/build (create_application on domain)" . Will retry 0 more times before canceling build. com.openshift.client.OpenShiftEndpointException: Could not request https://localhost/broker/rest/domains/dev3660tst/applications: Operation failed.Reason given: "You are not permitted to perform this action with the scopes application/520ddf12fe361672040002fb/scale application/520ddf12fe361672040002fb/build (create_application on domain)" at com.openshift.internal.client.RestService.request(RestService.java:106) at com.openshift.internal.client.RestService.request(RestService.java:91) at com.openshift.internal.client.RestService.request(RestService.java:76) at com.openshift.internal.client.AbstractOpenShiftResource$ServiceRequest.execute(AbstractOpenShiftResource.java:124) at com.openshift.internal.client.DomainResource$CreateApplicationRequest.execute(DomainResource.java:316) at com.openshift.internal.client.DomainResource$CreateApplicationRequest.execute(DomainResource.java:296) at com.openshift.internal.client.DomainResource.createApplication(DomainResource.java:132) at com.openshift.internal.client.DomainResource.createApplication(DomainResource.java:113) at hudson.plugins.openshift.OpenShiftSlave.createApp(OpenShiftSlave.java:247) at hudson.plugins.openshift.OpenShiftSlave.provision(OpenShiftSlave.java:225) at hudson.plugins.openshift.OpenShiftCloud.provisionSlave(OpenShiftCloud.java:470) at hudson.plugins.openshift.OpenShiftCloud.provision(OpenShiftCloud.java:401) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:264) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:347) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Caused by: com.openshift.internal.client.httpclient.HttpClientException: {"api_version":1.0,"data":null,"messages":[{"exit_code":null,"field":null,"severity":"error","text":"You are not permitted to perform this action with the scopes application/520ddf12fe361672040002fb/scale application/520ddf12fe361672040002fb/build (create_application on domain)"}],"status":"forbidden","supported_api_versions":[1.0,1.1,1.2,1.3,1.4,1.5,1.6],"type":null,"version":"1.0"} at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.createException(UrlConnectionHttpClient.java:193) at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.write(UrlConnectionHttpClient.java:165) at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.post(UrlConnectionHttpClient.java:135) at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.post(UrlConnectionHttpClient.java:131) at com.openshift.internal.client.RestService.request(RestService.java:146) at com.openshift.internal.client.RestService.request(RestService.java:98) ... 17 more Caused by: java.io.IOException: Server returned HTTP response code: 403 for URL: https://localhost/broker/rest/domains/dev3660tst/applications at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1625) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254) at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.write(UrlConnectionHttpClient.java:163) ... 21 more
I have detected many mismatch of the jenkins key from mongodb and on node Gear '520dc413b0a2a50b34000012' has key with name 'cjk' in mongo but not on the node. Gear '877415207527856149102592' has key with name 'jenkins' in mongo but not on the node. Gear '520dc278b0a2a50b34000007' has key with name 'cjk' in mongo but not on the node. Gear '520dcae1b0a2a57fbb000001' has key with name 'jenkins' in mongo but not on the node. Gear '295716904651374790180864' has key with name 'jenkins' in mongo but not on the node. Gear '520dd6cdb0a2a503d1000006' has key with name 'jenkins' in mongo but not on the node. Gear '314505224065557874081792' has key with name 'jenkins' in mongo but not on the node. Gear '766054384219261644046336' has key with name 'jenkins' in mongo but not on the node. Gear '520dedb1b0a2a54159000006' has key with name 'application-haproxy-7580c8e8064511e3bb8412313d1ad2e8' in mongo but not on the node. Gear '520dedb1b0a2a54159000006' has key with name 'jenkins' in mongo but not on the node. Gear '520dedb1b0a2a54159000006' has key with name 'default' in mongo but not on the node. The node expected to contain 1 gears wasn't returned from mcollective for the gear list In mongo: "system_ssh_keys" : [ { "_id" : ObjectId("520ddb72b0a2a5847d000051"), "type" : "ssh-rsa", "name" : "jenkins", "content" : "AAAAB3NzaC1yc2EAAAABIwAAAQEAp/sB3TBWe575aUtkiM+2iVssLt7kQvq64aTY3cpDtpSr+45JFVbgz4slyUsE0kr1RvAymv2BVkugzrvyG0F4lqMS9WbAPshlWNFCIoSqmjGG7tdkrzK7xPQ2aKS8Qw6b0fvdNvUaHIidWdQGQBV1IzXsLUo+DC+8c7r/EyzCYUFUd0DawSioYNmFDvuH8wbdsh4gpItMCG5rVls2g+GtrybIiTyIcIlXc85TMGjtGE7bIYjXiKUBl6vb4Jjewm5SsFH07T245XGWZkGtgOybGK/qiaL/qfpdMLEGdVAZte4+oJiPOJkLz+sRtwLYJwhRaVZdWeuN5qIcF1SAUVhn3w==", "component_id" : ObjectId("520ddb66b0a2a5847d00004c"), "_type" : "SystemSshKey" } ], On node: [root@ip-10-82-221-18 766054384219261644046336]# cat .ssh/authorized_keys command="/usr/bin/oo-trap-user",no-X11-forwarding ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvk6aI6mJplA2BqfLaf6XheT26VQvvVhToSWGEl09+tZ/pVLBiZf+tDX0duGj5y+ATlMKAZ/urahO4nliQ1Kjs8ZO/xHYiIrZPm9jTgLEOd5c/wQh8Nof4E/FVqQEpcqq3ZhIk3a0bODE5nHfYPUphWjgLawFqVr0eoEbXUSc4QZ8Pubet2km6/sGtaEusF4ThERBq9jLjA5q4Gt/856lCa1SXPKCBtMBRLvMt4H7RYWwLG7ahzzuxygVNUVrIdLzW+WEWWJdI5SSpBXCDsJbPwBxmJhRqkyNNzPbq5u8DY7g2eoAthtDS3wE3rWKFFNFMTScK9ANwvh3Tn+DEwBuX OPENSHIFT-766054384219261644046336-520dce44b0a2a5c0a8000075-default the key name with 'jenkins' is missing
Fix is in progress https://github.com/openshift/origin-server/pull/3382
Verified on devenv_3672 [lijun@rayzhang myjbossews20]$ git add -A && git commit -amp && git push [master f2451a9] p 1 file changed, 1 insertion(+), 1 deletion(-) warning: push.default is unset; its implicit value is changing in Git 2.0 from 'matching' to 'simple'. To squelch this message and maintain the current behavior after the default changes, use: git config --global push.default matching To squelch this message and adopt the new behavior now, use: git config --global push.default simple See 'git help config' and search for 'push.default' for further information. (the 'simple' mode was introduced in Git 1.7.11. Use the similar mode 'current' instead of 'simple' if you sometimes use older versions of Git) Counting objects: 11, done. Delta compression using up to 8 threads. Compressing objects: 100% (5/5), done. Writing objects: 100% (6/6), 438 bytes, done. Total 6 (delta 4), reused 0 (delta 0) remote: Executing Jenkins build. remote: remote: You can track your build at https://jenkins-chunchen.dev.rhcloud.com/job/myjbossews20-build remote: remote: Waiting for build to schedule..............................Done remote: Waiting for job to complete..................................................................................Done remote: SUCCESS remote: New build has been deployed. To ssh://f28f89a0087f11e3befd22000ab311b3.rhcloud.com/~/git/myjbossews20.git/ e3150ad..f2451a9 master -> master