Bug 864921

Summary: Exception is seen upon creating domain when no cartridge is installed in node.
Product: OpenShift Container Platform Reporter: Johnny Liu <jialiu>
Component: NodeAssignee: Luke Meyer <lmeyer>
Status: CLOSED ERRATA QA Contact: libra bugs <libra-bugs>
Severity: low Docs Contact:
Priority: low    
Version: 1.1.0CC: bleanhar, jkeck, xjia
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-01-31 20:31:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 888550    
Bug Blocks:    

Description Johnny Liu 2012-10-10 11:53:09 UTC
Description of problem:
There is two vms, one is broker, another is node.
On broker, install openshift-origin-broker, rubygem-openshift-origin-msg-broker-mcollective, rubygem-openshift-origin-auth-remote-user and rubygem-openshift-origin-dns-bind.

On node, install rubygem-openshift-origin-node and openshift-origin-msg-node-mcollective.

Then create domain via rest api.
# curl -k -X POST -H 'Accept: application/xml' --data-urlencode id=jialiu --user jialiu:passwd https://10.66.11.161/broker/rest/domains

The following error is seen:
<--snip-->
  <title>Action Controller: Exception caught</title>
<--snip-->
<h1>
  JSON::ParserError
    in DomainsController#create
</h1>
<pre>A JSON text must at least contain two octets!</pre>
<--snip-->


Check development.log, found broker was trying to get cartridge list when create domain for the first time.
# tailf /var/www/openshift/broker/log/development.log
Started POST "/broker/rest/domains" for 10.66.65.182 at Wed Oct 10 19:37:35 +0800 2012
  Processing by DomainsController#create as XML
  Parameters: {"id"=>"jialiu"}
MongoDataStore.find(CloudUser, jialiu, jialiu)

Creating domain with namespace jialiu
Checking to see if namesspace jialiu is available
Creating domain 5cd519f81a934f75abd18b46430fa79b with namespace jialiu for user jialiu
Attempting to add namespace 'jialiu'
notifying the domain observer of domain create
done notifying the domain observer
MongoDataStore.create(Domain, jialiu, 5cd519f81a934f75abd18b46430fa79b, #hidden)

CURRENT SERVER: node0
DEBUG: find_one_impl: current_server: node0
DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fb550e198f0>
DEBUG: rpc_client.custom_request('cartridge_do', {:args=>{"--with-descriptors"=>true, "--porcelain"=>true}, :cartridge=>"openshift-origin-node", :action=>"cartridge-list"}, @id, {'identity' => @id})
DEBUG: server results: No such file or directory - /usr/libexec/openshift/cartridges
Completed 500 Internal Server Error in 5325ms

JSON::ParserError (A JSON text must at least contain two octets!):
  

Rendered /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.13/lib/action_dispatch/middleware/templates/rescues/_trace.erb (1.4ms)
Rendered /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.13/lib/action_dispatch/middleware/templates/rescues/_request_and_response.erb (5.1ms)
Rendered /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.13/lib/action_dispatch/middleware/templates/rescues/diagnostics.erb within rescues/layout (11.2ms)


Actually domain is created successfully, I think it is not necessary to get cartridge list when creating domain.



Version-Release number of selected component (if applicable):
http://buildvm-devops.usersys.redhat.com/puddle/build/OpenShiftEnterprise/Beta/2012-10-09.4/DevOps/x86_64/os/

Following https://openshift.redhat.com/community/wiki/build-your-own to setup onpremise env using the above repo 

How reproducible:
Always

Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:
During creating domain, it should retrieve cartridge list, they should be separated.

Additional info:

Comment 2 Brenton Leanhardt 2012-12-04 19:51:53 UTC
We're investigating this sprint how to have rhc properly handle 500 errors from the server.

Comment 3 Luke Meyer 2012-12-18 20:39:23 UTC
The actual problem reported has gone away at some point. When the node has an empty cart list (or none), that's what's reported via the API.

You can still get an error in the same way, though, by having no nodes, or none that respond. The error bubbles up when doing a domain operation, including rhc setup (because a cartridge list is requested for reporting to the user; I don't think that's a problem).

rhc is being modified to improve its handling, but I still think it's improper for the broker to give a 500 error just because it can't reach nodes. Particularly during the early stages of an install, before the user has tried to do anything requiring a node.

Working on the fix.

Comment 4 Luke Meyer 2012-12-19 01:43:34 UTC
The exception is now caught and just logged, with an empty cartridge list returned. The empty list isn't cached, so once any node shows up, the user should get a fresh cartridge list.

So to test:
1. Set up a new broker & node
2. service mcollective stop on the node
3. (clear broker cache if it has been populated)
4. rhc setup a user (it shouldn't get any errors but have empty cartridge list)
5. service mcollective start on the node
6. rhc setup a user again (it should report the correct list of cartridges)

Fix is in rubygem-openshift-origin-controller-1.0.9-1.el6op

Comment 5 xjia 2012-12-19 11:05:42 UTC
[jason@jasontest ~]$ rhc setup
OpenShift Client Tools (RHC) Setup Wizard

This wizard will help you upload your SSH keys, set your application namespace, and check that other programs like Git are properly installed.

Login to broker.example.com: |jia| 
Password: ***

Configuration file /home/jason/.openshift/express.conf already exists, backing up to /home/jason/.openshift/express.conf.bak

Created local config file: /home/jason/.openshift/express.conf
The express.conf file contains user configuration, and can be transferred to different computers.

We will now check to see if you have the necessary client tools installed.

Checking for git ... found

Checking for your namespace ... found namespace:
    jia

Checking for applications ... none found

Run 'rhc app create' to create your first application.
Below is a list of the types of application you can create: 


Thank you for setting up your system.  You can rerun this at any time by calling 'rhc setup'.

[jason@jasontest ~]$ 


Version:
OpenShiftEnterprise/1.1.x/2012-12-18.1

Comment 7 errata-xmlrpc 2013-01-31 20:31:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0220.html