Bug 804010 - jenkins push build will hang there after migration
Summary: jenkins push build will hang there after migration
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OKD
Classification: Red Hat
Component: Pod
Version: 1.x
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Rob Millner
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-03-16 10:10 UTC by Meng Bo
Modified: 2013-11-18 00:38 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-03-19 18:22:52 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 804276 0 medium CLOSED Jenkins builds fail the first few times after migration 2021-02-22 00:41:40 UTC

Internal Links: 804276

Description Meng Bo 2012-03-16 10:10:38 UTC
Description of problem:
after migration, try to push build for previous app with jenkins embedded, jenkins build will hang there.

Version-Release number of selected component (if applicable):
devenv_stage-144

How reproducible:
always

Steps to Reproduce:
1.start denenv_stage-143
2.create an app with jenkins embedded
3.do upgrade and migration
4.try to push changes for the exist app via jenkins.
  
Actual results:
jenkins build will hang there.

Jenkins Log:
INFO: Provisioning new node for workload = 1 and label = perl1-build
com.openshift.express.client.OpenShiftEndpointException: Could not configure application "pyp1bldr" at "https://localhost/broker/cartridge"
	at com.openshift.express.client.OpenShiftService.sendRequest(OpenShiftService.java:529)
	at com.openshift.express.client.OpenShiftService.requestApplicationAction(OpenShiftService.java:435)
	at com.openshift.express.client.OpenShiftService.createApplication(OpenShiftService.java:290)
	at hudson.plugins.openshift.OpenShiftSlave.createApp(OpenShiftSlave.java:225)
	at hudson.plugins.openshift.OpenShiftSlave.provision(OpenShiftSlave.java:194)
	at hudson.plugins.openshift.OpenShiftCloud$2.call(OpenShiftCloud.java:494)
	at hudson.plugins.openshift.OpenShiftCloud$2.call(OpenShiftCloud.java:488)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:679)
Caused by: com.openshift.express.internal.client.httpclient.BadRequestException: {"data":"","debug":"","api":"1.1.3","broker_c":["namespace","rhlogin","ssh","app_uuid","debug","alter","cartridge","cart_type","action","app_name","api"],"messages":"","api_c":["placeholder"],"result":"Invalid Profile: std.  Must be: (jumbo|exlarge|large|micro|std)","exit_code":1}
	at com.openshift.express.internal.client.httpclient.UrlConnectionHttpClient.createException(UrlConnectionHttpClient.java:109)
	at com.openshift.express.internal.client.httpclient.UrlConnectionHttpClient.post(UrlConnectionHttpClient.java:75)
	at com.openshift.express.client.OpenShiftService.sendRequest(OpenShiftService.java:518)


Expected results:
push build should successful.

Additional info:
during the migration period, jenkins app will got 'Node execution failure' first time, and will pass the migration after another try.

<--------->
Migrating app 'jenkins' with uuid '54076892d9844998ae57312ef99d6da4' on node 'ip-10-64-29-128' for user: bmeng
Failed to migrate with cmd: './migrate-2.0.7 --rhlogin 'bmeng' --migrate-app 'jenkins'' after 2 tries with exception: Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.
/var/www/stickshift/broker/lib/express/broker/application_container_proxy.rb:894:in `run_cartridge_command'./migrate-2.0.7:159:in `send'./migrate-2.0.7:159:in `redeploy_httpd_proxy'./migrate-2.0.7:49:in `migrate_app'/usr/lib/ruby/site_ruby/1.8/mcollective/rpc/client.rb:258:in `call'/usr/lib/ruby/site_ruby/1.8/mcollective/rpc/client.rb:258:in `custom_request'/usr/lib/ruby/site_ruby/1.8/mcollective/rpc/client.rb:511:in `call'/usr/lib/ruby/site_ruby/1.8/mcollective/rpc/client.rb:511:in `process_results_with_block'/usr/lib/ruby/site_ruby/1.8/mcollective/rpc/client.rb:450:in `call_agent'/usr/lib/ruby/site_ruby/1.8/mcollective/client.rb:129:in `req'/usr/lib/ruby/site_ruby/1.8/mcollective/client.rb:124:in `loop'/usr/lib/ruby/site_ruby/1.8/mcollective/client.rb:124:in `req'/usr/lib/ruby/1.8/timeout.rb:67:in `timeout'/usr/lib/ruby/site_ruby/1.8/mcollective/client.rb:123:in `req'/usr/lib/ruby/site_ruby/1.8/mcollective/rpc/client.rb:446:in `call_agent'/usr/lib/ruby/site_ruby/1.8/mcollective/rpc/client.rb:257:in `custom_request'/usr/lib/ruby/site_ruby/1.8/mcollective/rpc/client.rb:193:in `method_missing'./migrate-2.0.7:34:in `migrate_app'/var/www/stickshift/broker/lib/express/broker/application_container_proxy.rb:611:in `rpc_exec'./migrate-2.0.7:33:in `migrate_app'/usr/lib/ruby/1.8/timeout.rb:67:in `timeout'./migrate-2.0.7:32:in `migrate_app'./migrate-2.0.7:463:in `migrate_from_file'./migrate-2.0.7:461:in `each'./migrate-2.0.7:461:in `migrate_from_file'./migrate-2.0.7:573
Output:
Migrating app on node with: ./migrate-2.0.7 --rhlogin 'bmeng' --migrate-app 'jenkins'
<--------->
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Failures:
./migrate-2.0.7 --rhlogin 'bmeng' --migrate-app 'jenkins'
./migrate-2.0.7 --rhlogin 'bmeng+1' --migrate-app 'jk'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
<--------->
[root@ip-10-64-29-128 bin]# ./migrate-2.0.7 --rhlogin 'bmeng' --migrate-app 'jenkins'
Migrating app on node with: ./migrate-2.0.7 --rhlogin 'bmeng' --migrate-app 'jenkins'
Redeploying httpd proxy for 'jenkins' with uuid '54076892d9844998ae57312ef99d6da4' on node ip-10-64-29-128
Leaving app stopped 'jenkins' with uuid '54076892d9844998ae57312ef99d6da4' on node 'ip-10-64-29-128'
[root@ip-10-64-29-128 bin]# ./migrate-2.0.7 --rhlogin 'bmeng+1' --migrate-app 'jk'
Migrating app on node with: ./migrate-2.0.7 --rhlogin 'bmeng+1' --migrate-app 'jk'
Redeploying httpd proxy for 'jk' with uuid '30c1e14c15cd44ea9f134b82b2e761ae' on node ip-10-64-29-128
Leaving app stopped 'jk' with uuid '30c1e14c15cd44ea9f134b82b2e761ae' on node 'ip-10-64-29-128'

Comment 1 Dan McPherson 2012-03-16 14:01:06 UTC
Rob, can you take a look.  Looks like there are 2 issues.  1 the builder sizes aren't being migrated.  And 2 there is still an issue with redeploy httpd proxy after the folder renames.

Comment 2 Rob Millner 2012-03-16 22:32:46 UTC
Checkins 5ff56e99 and 5b8651e5 update builder sizes in the jenkins and build configurations respectively.

Comment 3 Rob Millner 2012-03-16 22:33:51 UTC
Correction, 5b8651e5 just updates a comment and is safe to either include or exclude from the release.

Comment 4 Rob Millner 2012-03-17 08:40:34 UTC
The following checkins address the (and related) issue:

5b8651e5   cleanup error message.
5ff56e993  Update jenkins config.xml and per-job config.xml to reflect new gear sizes.
252b2465   Dont need verbose output from the file edit.
fd3ad80bd  Had missed the existing jenkins block
2aa3651    missing variable migration
2a1570e    Add APP_NAME to the list of retained variables.
e87d1af    PERL has an environment variable screen ...


Migration works.  Jenkins builds do not get stuck.

The first time after migration Jenkins builds may fail.  After a few tries re-running the slave is rebuilt and they run properly.


Bug 804276 was created to track that issue.

Comment 5 Johnny Liu 2012-03-19 11:20:36 UTC
Verified this bug with devenv_stage_146 updated from devenv_stage_143, and PASS


Note You need to log in before you can comment on or make changes to this bug.