Bug 864921 - Exception is seen upon creating domain when no cartridge is installed in node.
Summary: Exception is seen upon creating domain when no cartridge is installed in node.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 1.1.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: ---
Assignee: Luke Meyer
QA Contact: libra bugs
URL:
Whiteboard:
Depends On: 888550
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-10-10 11:53 UTC by Johnny Liu
Modified: 2017-03-08 17:34 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-01-31 20:31:57 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2013:0220 0 normal SHIPPED_LIVE Important: Red Hat OpenShift Enterprise 1.1 update 2013-02-01 01:23:24 UTC

Description Johnny Liu 2012-10-10 11:53:09 UTC
Description of problem:
There is two vms, one is broker, another is node.
On broker, install openshift-origin-broker, rubygem-openshift-origin-msg-broker-mcollective, rubygem-openshift-origin-auth-remote-user and rubygem-openshift-origin-dns-bind.

On node, install rubygem-openshift-origin-node and openshift-origin-msg-node-mcollective.

Then create domain via rest api.
# curl -k -X POST -H 'Accept: application/xml' --data-urlencode id=jialiu --user jialiu:passwd https://10.66.11.161/broker/rest/domains

The following error is seen:
<--snip-->
  <title>Action Controller: Exception caught</title>
<--snip-->
<h1>
  JSON::ParserError
    in DomainsController#create
</h1>
<pre>A JSON text must at least contain two octets!</pre>
<--snip-->


Check development.log, found broker was trying to get cartridge list when create domain for the first time.
# tailf /var/www/openshift/broker/log/development.log
Started POST "/broker/rest/domains" for 10.66.65.182 at Wed Oct 10 19:37:35 +0800 2012
  Processing by DomainsController#create as XML
  Parameters: {"id"=>"jialiu"}
MongoDataStore.find(CloudUser, jialiu, jialiu)

Creating domain with namespace jialiu
Checking to see if namesspace jialiu is available
Creating domain 5cd519f81a934f75abd18b46430fa79b with namespace jialiu for user jialiu
Attempting to add namespace 'jialiu'
notifying the domain observer of domain create
done notifying the domain observer
MongoDataStore.create(Domain, jialiu, 5cd519f81a934f75abd18b46430fa79b, #hidden)

CURRENT SERVER: node0
DEBUG: find_one_impl: current_server: node0
DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fb550e198f0>
DEBUG: rpc_client.custom_request('cartridge_do', {:args=>{"--with-descriptors"=>true, "--porcelain"=>true}, :cartridge=>"openshift-origin-node", :action=>"cartridge-list"}, @id, {'identity' => @id})
DEBUG: server results: No such file or directory - /usr/libexec/openshift/cartridges
Completed 500 Internal Server Error in 5325ms

JSON::ParserError (A JSON text must at least contain two octets!):
  

Rendered /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.13/lib/action_dispatch/middleware/templates/rescues/_trace.erb (1.4ms)
Rendered /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.13/lib/action_dispatch/middleware/templates/rescues/_request_and_response.erb (5.1ms)
Rendered /usr/lib/ruby/gems/1.8/gems/actionpack-3.0.13/lib/action_dispatch/middleware/templates/rescues/diagnostics.erb within rescues/layout (11.2ms)


Actually domain is created successfully, I think it is not necessary to get cartridge list when creating domain.



Version-Release number of selected component (if applicable):
http://buildvm-devops.usersys.redhat.com/puddle/build/OpenShiftEnterprise/Beta/2012-10-09.4/DevOps/x86_64/os/

Following https://openshift.redhat.com/community/wiki/build-your-own to setup onpremise env using the above repo 

How reproducible:
Always

Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:
During creating domain, it should retrieve cartridge list, they should be separated.

Additional info:

Comment 2 Brenton Leanhardt 2012-12-04 19:51:53 UTC
We're investigating this sprint how to have rhc properly handle 500 errors from the server.

Comment 3 Luke Meyer 2012-12-18 20:39:23 UTC
The actual problem reported has gone away at some point. When the node has an empty cart list (or none), that's what's reported via the API.

You can still get an error in the same way, though, by having no nodes, or none that respond. The error bubbles up when doing a domain operation, including rhc setup (because a cartridge list is requested for reporting to the user; I don't think that's a problem).

rhc is being modified to improve its handling, but I still think it's improper for the broker to give a 500 error just because it can't reach nodes. Particularly during the early stages of an install, before the user has tried to do anything requiring a node.

Working on the fix.

Comment 4 Luke Meyer 2012-12-19 01:43:34 UTC
The exception is now caught and just logged, with an empty cartridge list returned. The empty list isn't cached, so once any node shows up, the user should get a fresh cartridge list.

So to test:
1. Set up a new broker & node
2. service mcollective stop on the node
3. (clear broker cache if it has been populated)
4. rhc setup a user (it shouldn't get any errors but have empty cartridge list)
5. service mcollective start on the node
6. rhc setup a user again (it should report the correct list of cartridges)

Fix is in rubygem-openshift-origin-controller-1.0.9-1.el6op

Comment 5 xjia 2012-12-19 11:05:42 UTC
[jason@jasontest ~]$ rhc setup
OpenShift Client Tools (RHC) Setup Wizard

This wizard will help you upload your SSH keys, set your application namespace, and check that other programs like Git are properly installed.

Login to broker.example.com: |jia| 
Password: ***

Configuration file /home/jason/.openshift/express.conf already exists, backing up to /home/jason/.openshift/express.conf.bak

Created local config file: /home/jason/.openshift/express.conf
The express.conf file contains user configuration, and can be transferred to different computers.

We will now check to see if you have the necessary client tools installed.

Checking for git ... found

Checking for your namespace ... found namespace:
    jia

Checking for applications ... none found

Run 'rhc app create' to create your first application.
Below is a list of the types of application you can create: 


Thank you for setting up your system.  You can rerun this at any time by calling 'rhc setup'.

[jason@jasontest ~]$ 


Version:
OpenShiftEnterprise/1.1.x/2012-12-18.1

Comment 7 errata-xmlrpc 2013-01-31 20:31:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0220.html


Note You need to log in before you can comment on or make changes to this bug.