Bug 802387

Summary:	Some default settings in stickshift-node.conf are improper to be used in devenv instance
Product:	OKD	Reporter:	Johnny Liu <jialiu>
Component:	Containers	Assignee:	Krishna Raman <kraman>
Status:	CLOSED CURRENTRELEASE	QA Contact:	libra bugs <libra-bugs>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	1.x	CC:	kraman, mfisher, mpatel, rmillner
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-03-19 18:22:37 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Johnny Liu 2012-03-12 12:54:40 UTC

Description of problem:
Trigger jenkins build and failed.
$ git commit -a -m "xx"; git push
[master 71b5e8f] xx
 1 files changed, 1 insertions(+), 1 deletions(-)
Failed to add the host to the list of known hosts (/dev/nul).
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 272 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Executing Jenkins build.
remote: 
remote: You can track your build at https://jenkins-jialiu.dev.rhcloud.com/job/phptest-build
remote: 
remote: Waiting for build to schedule.........
remote: **BUILD FAILED/CANCELLED**
remote: Please see the Jenkins log for more details via rhc-tail-files
remote: !!!!!!!!
remote: Deployment Halted!
remote: If the build failed before the deploy step, your previous
remote: build is still running.  Otherwise, your application may be
remote: partially deployed or inaccessible.
remote: Fix the build and try again.
remote: !!!!!!!!
To ssh://c1e92db8235d4005987009bf831a4c13.rhcloud.com/~/git/phptest.git/
   0afaf33..71b5e8f  master -> master


Version-Release number of selected component (if applicable):
rhc-cartridge-jenkins-1.4-0.88.3-1.el6_2.noarch
rhc-cartridge-jenkins-client-1.4-0.22.2-1.el6_2.noarch
jenkins-plugin-openshift-0.4.28-1.el6_2.x86_64
rubygem-stickshift-controller-0.6.5-1.el6_2.noarch
stickshift-abstract-0.6.5-1.el6_2.noarch
rubygem-stickshift-node-0.6.6-1.el6_2.noarch
rubygem-stickshift-common-0.6.4-1.el6_2.noarch
rhc-node-0.88.4-1.el6_2.x86_64


How reproducible:
Always

Steps to Reproduce:
1. Create an app with jenkins enabled.
2. Do some change, then trigger jenkins build.
3.
  
Actual results:
jenkins build failed.

jenkins log message:
<--snip-->
Mar 12, 2012 8:30:44 AM hudson.plugins.openshift.OpenShiftCloud provision
INFO: Provisioning new node for workload = 1 and label = phptest-build
com.openshift.express.client.OpenShiftEndpointException: Could not get user info for user "" at "https://openshift.redhat.com/broker/userinfo"
	at com.openshift.express.client.OpenShiftService.sendRequest(OpenShiftService.java:529)
	at com.openshift.express.client.OpenShiftService.getUserInfo(OpenShiftService.java:126)
	at hudson.plugins.openshift.OpenShiftCloud.getUserInfo(OpenShiftCloud.java:401)
	at hudson.plugins.openshift.OpenShiftCloud.getSlaves(OpenShiftCloud.java:618)
	at hudson.plugins.openshift.OpenShiftCloud.provision(OpenShiftCloud.java:439)
	at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:203)
	at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:54)
	at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:286)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
	at java.util.TimerThread.mainLoop(Timer.java:534)
	at java.util.TimerThread.run(Timer.java:484)
Caused by: com.openshift.express.internal.client.httpclient.InternalServerErrorException: {"debug":"padding check failed/var/www/libra/broker/lib/express/broker/auth_service.rb:109:in `private_decrypt'\n/var/www/libra/broker/lib/express/broker/auth_service.rb:109:in `check_broker_key'\n/var/www/libra/broker/lib/express/broker/auth_service.rb:50:in `login'\n/usr/lib/ruby/gems/1.8/gems/cloud-sdk-controller-0.5.10/lib/cloud-sdk-controller/app/controllers/legacy_broker_controller.rb:349:in `authenticate'\n/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:447:in `_run__688730835__process_action__927155398__callbacks'\n/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:410:in `send'","api":"1.1.2","data":"","api_c":["placeholder"],"broker_c":["namespace","rhlogin","ssh","app_uuid","debug","alter","cartridge","cart_type","action","app_name","api"],"exit_code":1,"messages":"","result":"padding check failed"}
	at com.openshift.express.internal.client.httpclient.UrlConnectionHttpClient.createException(UrlConnectionHttpClient.java:107)
	at com.openshift.express.internal.client.httpclient.UrlConnectionHttpClient.post(UrlConnectionHttpClient.java:75)
	at com.openshift.express.client.OpenShiftService.sendRequest(OpenShiftService.java:518)
	... 10 more
Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: https://openshift.redhat.com/broker/userinfo
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1403)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
	at com.openshift.express.internal.client.httpclient.UrlConnectionHttpClient.post(UrlConnectionHttpClient.java:70)
	... 11 more
<--snip-->

Expected results:
jenkins build should be succesfull.

Additional info:
Go to jenkins UI - > Manage Jenkins -> "OpenShift Broker Hostname", found jenkins job is using "openshift.redhat.com"

Check /etc/libra/node.conf:
# Libra server
libra_server='localhost'

Looks like jenkins server does not read libra_server value from  /etc/libra/node.conf,

Comment 1 Johnny Liu 2012-03-12 12:56:52 UTC

Refer to  https://bugzilla.redhat.com/show_bug.cgi?id=802258#c1 for workaround about app creation failure.

Comment 2 Johnny Liu 2012-03-13 06:31:18 UTC

Now all setting is located in /etc/stickshift/stickshift-node.conf, the root cause of this bug is improper default settings in stickshift-node.conf.

# cat /etc/stickshift/stickshift-node.conf
<--snip-->
CLOUD_DOMAIN="dev.rhcloud.com"                
BROKER_HOST="openshift.redhat.com"            
CREATE_APP_SYMLINKS=0                 
<--snip-->

BROKER_HOST setting is used by jenkins app, but for devenv environment, "localhost" is more proper here.

BTW, "CREATE_APP_SYMLINKS" should be set to "1" in devenv environment, that will be more convenient for debugging.

I know user can modify this config file manually, but tester must modify this file every time after launch a new instance.

If these two default settings are modified to be more proper for devenv environment, that will bring much convenience for QE's testing.

Because the root cause of this issue is incorrect setting in config file, low priority to "medium".

Comment 3 Johnny Liu 2012-03-14 07:13:16 UTC

Verified this bug with devenv_1661, and PASS.

# cat /etc/stickshift/stickshift-node.conf
<--snip-->               
BROKER_HOST="localhost"            
CREATE_APP_SYMLINKS=1                 
<--snip-->