Bug 864921 - Exception is seen upon creating domain when no cartridge is installed in node.
Exception is seen upon creating domain when no cartridge is installed in node.
Product: OpenShift Container Platform
Classification: Red Hat
Component: Pod (Show other bugs)
Unspecified Unspecified
low Severity low
: ---
: ---
Assigned To: Luke Meyer
libra bugs
: Triaged
Depends On: 888550
  Show dependency treegraph
Reported: 2012-10-10 07:53 EDT by Johnny Liu
Modified: 2017-03-08 12 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2013-01-31 15:31:57 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Johnny Liu 2012-10-10 07:53:09 EDT
Description of problem:
There is two vms, one is broker, another is node.
On broker, install openshift-origin-broker, rubygem-openshift-origin-msg-broker-mcollective, rubygem-openshift-origin-auth-remote-user and rubygem-openshift-origin-dns-bind.

On node, install rubygem-openshift-origin-node and openshift-origin-msg-node-mcollective.

Then create domain via rest api.
# curl -k -X POST -H 'Accept: application/xml' --data-urlencode id=jialiu --user jialiu@redhat.com:passwd

The following error is seen:
  <title>Action Controller: Exception caught</title>
    in DomainsController#create
<pre>A JSON text must at least contain two octets!</pre>

Check development.log, found broker was trying to get cartridge list when create domain for the first time.
# tailf /var/www/openshift/broker/log/development.log
Started POST "/broker/rest/domains" for at Wed Oct 10 19:37:35 +0800 2012
  Processing by DomainsController#create as XML
  Parameters: {"id"=>"jialiu"}
MongoDataStore.find(CloudUser, jialiu@redhat.com, jialiu@redhat.com)

Creating domain with namespace jialiu
Checking to see if namesspace jialiu is available
Creating domain 5cd519f81a934f75abd18b46430fa79b with namespace jialiu for user jialiu@redhat.com
Attempting to add namespace 'jialiu'
notifying the domain observer of domain create
done notifying the domain observer
MongoDataStore.create(Domain, jialiu@redhat.com, 5cd519f81a934f75abd18b46430fa79b, #hidden)

DEBUG: find_one_impl: current_server: node0
DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fb550e198f0>
DEBUG: rpc_client.custom_request('cartridge_do', {:args=>{"--with-descriptors"=>true, "--porcelain"=>true}, :cartridge=>"openshift-origin-node", :action=>"cartridge-list"}, @id, {'identity' => @id})
DEBUG: server results: No such file or directory - /usr/libexec/openshift/cartridges
Completed 500 Internal Server Error in 5325ms

JSON::ParserError (A JSON text must at least contain two octets!):

Rendered /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.13/lib/action_dispatch/middleware/templates/rescues/_trace.erb (1.4ms)
Rendered /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.13/lib/action_dispatch/middleware/templates/rescues/_request_and_response.erb (5.1ms)
Rendered /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.13/lib/action_dispatch/middleware/templates/rescues/diagnostics.erb within rescues/layout (11.2ms)

Actually domain is created successfully, I think it is not necessary to get cartridge list when creating domain.

Version-Release number of selected component (if applicable):

Following https://openshift.redhat.com/community/wiki/build-your-own to setup onpremise env using the above repo 

How reproducible:

Steps to Reproduce:
Actual results:

Expected results:
During creating domain, it should retrieve cartridge list, they should be separated.

Additional info:
Comment 2 Brenton Leanhardt 2012-12-04 14:51:53 EST
We're investigating this sprint how to have rhc properly handle 500 errors from the server.
Comment 3 Luke Meyer 2012-12-18 15:39:23 EST
The actual problem reported has gone away at some point. When the node has an empty cart list (or none), that's what's reported via the API.

You can still get an error in the same way, though, by having no nodes, or none that respond. The error bubbles up when doing a domain operation, including rhc setup (because a cartridge list is requested for reporting to the user; I don't think that's a problem).

rhc is being modified to improve its handling, but I still think it's improper for the broker to give a 500 error just because it can't reach nodes. Particularly during the early stages of an install, before the user has tried to do anything requiring a node.

Working on the fix.
Comment 4 Luke Meyer 2012-12-18 20:43:34 EST
The exception is now caught and just logged, with an empty cartridge list returned. The empty list isn't cached, so once any node shows up, the user should get a fresh cartridge list.

So to test:
1. Set up a new broker & node
2. service mcollective stop on the node
3. (clear broker cache if it has been populated)
4. rhc setup a user (it shouldn't get any errors but have empty cartridge list)
5. service mcollective start on the node
6. rhc setup a user again (it should report the correct list of cartridges)

Fix is in rubygem-openshift-origin-controller-1.0.9-1.el6op
Comment 5 xjia 2012-12-19 06:05:42 EST
[jason@jasontest ~]$ rhc setup
OpenShift Client Tools (RHC) Setup Wizard

This wizard will help you upload your SSH keys, set your application namespace, and check that other programs like Git are properly installed.

Login to broker.example.com: |jia@redhat.com| 
Password: ***

Configuration file /home/jason/.openshift/express.conf already exists, backing up to /home/jason/.openshift/express.conf.bak

Created local config file: /home/jason/.openshift/express.conf
The express.conf file contains user configuration, and can be transferred to different computers.

We will now check to see if you have the necessary client tools installed.

Checking for git ... found

Checking for your namespace ... found namespace:

Checking for applications ... none found

Run 'rhc app create' to create your first application.
Below is a list of the types of application you can create: 

Thank you for setting up your system.  You can rerun this at any time by calling 'rhc setup'.

[jason@jasontest ~]$ 

Comment 7 errata-xmlrpc 2013-01-31 15:31:57 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.