Bug 804010 - jenkins push build will hang there after migration
jenkins push build will hang there after migration
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Pod (Show other bugs)
1.x
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Rob Millner
libra bugs
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-03-16 06:10 EDT by Meng Bo
Modified: 2013-11-17 19:38 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-03-19 14:22:52 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Meng Bo 2012-03-16 06:10:38 EDT
Description of problem:
after migration, try to push build for previous app with jenkins embedded, jenkins build will hang there.

Version-Release number of selected component (if applicable):
devenv_stage-144

How reproducible:
always

Steps to Reproduce:
1.start denenv_stage-143
2.create an app with jenkins embedded
3.do upgrade and migration
4.try to push changes for the exist app via jenkins.
  
Actual results:
jenkins build will hang there.

Jenkins Log:
INFO: Provisioning new node for workload = 1 and label = perl1-build
com.openshift.express.client.OpenShiftEndpointException: Could not configure application "pyp1bldr" at "https://localhost/broker/cartridge"
	at com.openshift.express.client.OpenShiftService.sendRequest(OpenShiftService.java:529)
	at com.openshift.express.client.OpenShiftService.requestApplicationAction(OpenShiftService.java:435)
	at com.openshift.express.client.OpenShiftService.createApplication(OpenShiftService.java:290)
	at hudson.plugins.openshift.OpenShiftSlave.createApp(OpenShiftSlave.java:225)
	at hudson.plugins.openshift.OpenShiftSlave.provision(OpenShiftSlave.java:194)
	at hudson.plugins.openshift.OpenShiftCloud$2.call(OpenShiftCloud.java:494)
	at hudson.plugins.openshift.OpenShiftCloud$2.call(OpenShiftCloud.java:488)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:679)
Caused by: com.openshift.express.internal.client.httpclient.BadRequestException: {"data":"","debug":"","api":"1.1.3","broker_c":["namespace","rhlogin","ssh","app_uuid","debug","alter","cartridge","cart_type","action","app_name","api"],"messages":"","api_c":["placeholder"],"result":"Invalid Profile: std.  Must be: (jumbo|exlarge|large|micro|std)","exit_code":1}
	at com.openshift.express.internal.client.httpclient.UrlConnectionHttpClient.createException(UrlConnectionHttpClient.java:109)
	at com.openshift.express.internal.client.httpclient.UrlConnectionHttpClient.post(UrlConnectionHttpClient.java:75)
	at com.openshift.express.client.OpenShiftService.sendRequest(OpenShiftService.java:518)


Expected results:
push build should successful.

Additional info:
during the migration period, jenkins app will got 'Node execution failure' first time, and will pass the migration after another try.

<--------->
Migrating app 'jenkins' with uuid '54076892d9844998ae57312ef99d6da4' on node 'ip-10-64-29-128' for user: bmeng@redhat.com
Failed to migrate with cmd: './migrate-2.0.7 --rhlogin 'bmeng@redhat.com' --migrate-app 'jenkins'' after 2 tries with exception: Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.
/var/www/stickshift/broker/lib/express/broker/application_container_proxy.rb:894:in `run_cartridge_command'./migrate-2.0.7:159:in `send'./migrate-2.0.7:159:in `redeploy_httpd_proxy'./migrate-2.0.7:49:in `migrate_app'/usr/lib/ruby/site_ruby/1.8/mcollective/rpc/client.rb:258:in `call'/usr/lib/ruby/site_ruby/1.8/mcollective/rpc/client.rb:258:in `custom_request'/usr/lib/ruby/site_ruby/1.8/mcollective/rpc/client.rb:511:in `call'/usr/lib/ruby/site_ruby/1.8/mcollective/rpc/client.rb:511:in `process_results_with_block'/usr/lib/ruby/site_ruby/1.8/mcollective/rpc/client.rb:450:in `call_agent'/usr/lib/ruby/site_ruby/1.8/mcollective/client.rb:129:in `req'/usr/lib/ruby/site_ruby/1.8/mcollective/client.rb:124:in `loop'/usr/lib/ruby/site_ruby/1.8/mcollective/client.rb:124:in `req'/usr/lib/ruby/1.8/timeout.rb:67:in `timeout'/usr/lib/ruby/site_ruby/1.8/mcollective/client.rb:123:in `req'/usr/lib/ruby/site_ruby/1.8/mcollective/rpc/client.rb:446:in `call_agent'/usr/lib/ruby/site_ruby/1.8/mcollective/rpc/client.rb:257:in `custom_request'/usr/lib/ruby/site_ruby/1.8/mcollective/rpc/client.rb:193:in `method_missing'./migrate-2.0.7:34:in `migrate_app'/var/www/stickshift/broker/lib/express/broker/application_container_proxy.rb:611:in `rpc_exec'./migrate-2.0.7:33:in `migrate_app'/usr/lib/ruby/1.8/timeout.rb:67:in `timeout'./migrate-2.0.7:32:in `migrate_app'./migrate-2.0.7:463:in `migrate_from_file'./migrate-2.0.7:461:in `each'./migrate-2.0.7:461:in `migrate_from_file'./migrate-2.0.7:573
Output:
Migrating app on node with: ./migrate-2.0.7 --rhlogin 'bmeng@redhat.com' --migrate-app 'jenkins'
<--------->
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Failures:
./migrate-2.0.7 --rhlogin 'bmeng@redhat.com' --migrate-app 'jenkins'
./migrate-2.0.7 --rhlogin 'bmeng+1@redhat.com' --migrate-app 'jk'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
<--------->
[root@ip-10-64-29-128 bin]# ./migrate-2.0.7 --rhlogin 'bmeng@redhat.com' --migrate-app 'jenkins'
Migrating app on node with: ./migrate-2.0.7 --rhlogin 'bmeng@redhat.com' --migrate-app 'jenkins'
Redeploying httpd proxy for 'jenkins' with uuid '54076892d9844998ae57312ef99d6da4' on node ip-10-64-29-128
Leaving app stopped 'jenkins' with uuid '54076892d9844998ae57312ef99d6da4' on node 'ip-10-64-29-128'
[root@ip-10-64-29-128 bin]# ./migrate-2.0.7 --rhlogin 'bmeng+1@redhat.com' --migrate-app 'jk'
Migrating app on node with: ./migrate-2.0.7 --rhlogin 'bmeng+1@redhat.com' --migrate-app 'jk'
Redeploying httpd proxy for 'jk' with uuid '30c1e14c15cd44ea9f134b82b2e761ae' on node ip-10-64-29-128
Leaving app stopped 'jk' with uuid '30c1e14c15cd44ea9f134b82b2e761ae' on node 'ip-10-64-29-128'
Comment 1 Dan McPherson 2012-03-16 10:01:06 EDT
Rob, can you take a look.  Looks like there are 2 issues.  1 the builder sizes aren't being migrated.  And 2 there is still an issue with redeploy httpd proxy after the folder renames.
Comment 2 Rob Millner 2012-03-16 18:32:46 EDT
Checkins 5ff56e99 and 5b8651e5 update builder sizes in the jenkins and build configurations respectively.
Comment 3 Rob Millner 2012-03-16 18:33:51 EDT
Correction, 5b8651e5 just updates a comment and is safe to either include or exclude from the release.
Comment 4 Rob Millner 2012-03-17 04:40:34 EDT
The following checkins address the (and related) issue:

5b8651e5   cleanup error message.
5ff56e993  Update jenkins config.xml and per-job config.xml to reflect new gear sizes.
252b2465   Dont need verbose output from the file edit.
fd3ad80bd  Had missed the existing jenkins block
2aa3651    missing variable migration
2a1570e    Add APP_NAME to the list of retained variables.
e87d1af    PERL has an environment variable screen ...


Migration works.  Jenkins builds do not get stuck.

The first time after migration Jenkins builds may fail.  After a few tries re-running the slave is rebuilt and they run properly.


Bug 804276 was created to track that issue.
Comment 5 Johnny Liu 2012-03-19 07:20:36 EDT
Verified this bug with devenv_stage_146 updated from devenv_stage_143, and PASS

Note You need to log in before you can comment on or make changes to this bug.