Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 914429

Summary: Can't embed cartridge into app if all the nodes are in districts and no district is "small" size.
Product: OpenShift Container Platform Reporter: xjia <xjia>
Component: NodeAssignee: Luke Meyer <lmeyer>
Status: CLOSED ERRATA QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 1.1.1CC: baulakh, bleanhar, bmoss, jialiu, libra-onpremise-devel, xtian
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rubygem-openshift-origin-controller-1.0.14-1.el6op Doc Type: Bug Fix
Doc Text:
When adding an embedded cartridge, the broker unnecessarily searched for a node with node_profile of "small". If no such node existed or it was not districted, the broker returned a "no nodes available" error. This bug fix updates the broker so that it no longer makes this check when embedding cartridges, allowing cartridges to be embedded successfully.
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-04-02 08:50:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Error log none

Description xjia 2013-02-22 10:15:54 UTC
Created attachment 701103 [details]
Error log

Description of problem:
Now I have three nodes. Two of them are "medium", one of them is "newsize", no one is "small". Then create two disctricts. One for "medium" nodes, one for "newsize" node. Create a medium app, and embeded cartridge to the app, it will say "No node is avaiable".


 Version-Release number of selected component (if applicable):
Openshift Enterprise puddle 1.1.z/2013-02-20.1

How reproducible:
Always

Steps to Reproduce:
1.Prepare this environment.
node1, node3 : node_profile is "medium"
node2: node_profile is "newsize"
2.Modify /etc/openshift/broker.conf
[root@broker ~]# grep newsize /etc/openshift/broker.conf 
VALID_GEAR_SIZES="small,medium,newsize"
DEFAULT_GEAR_CAPABILITIES="small,newsize"
3. Create two district
district medium: node1, node3
district newsize: node2
4. Create a medium php app 
5. Embeded mysql to the app

Actual results:
[root@broker ~]# rhc cartridge add mysql -a php1 -pjia
Adding 'mysql-5.1' to application 'php1' ... No nodes available.

Expected results:
Should succeed

Additional info:
If  node2 move out from district "newsize", then all is ok. In the other words, one node is not contained in any district, all will be ok.

Comment 2 Johnny Liu 2013-02-22 11:21:03 UTC
Dig code a little to find out the following logic:

emb_cart_controller.rb -> mcollective_application_container_proxy.rb -> district.rb -> mongo_data_store.rb

emb_cart_controller.rb:
<--snip-->
  def create
    <--snip-->
    begin
      container = OpenShift::ApplicationContainerProxy.find_available(nil)
      if not check_cartridge_type(name, container, "embedded")
        carts = Application.get_available_cartridges("embedded")
        return render_error(:bad_request, "Invalid cartridge. Valid values are (#{carts.join(', ')})",
                            109, "EMBED_CARTRIDGE", "cartridge")
      end
    rescue Exception => e
      return render_exception(e, "EMBED_CARTRIDGE")
<--snip-->


mongo_data_store.rb
<--snip-->
    def find_available_district(node_profile=nil)
      node_profile = node_profile ? node_profile : "small"
      MongoDataStore.rescue_con_failure do
        hash = district_collection.find(
          { "available_capacity" => { "$gt" => 0 },
            "active_server_identities_size" => { "$gt" => 0 },
            "node_profile" => node_profile}).sort(["available_capacity", "descending"]).limit(1).next
        hash_to_district_ret(hash)
      end
    end
<--snip-->


When no node_profile is specified, broker will try to find a "small" district, so "No nodes available" happened.

Comment 3 Luke Meyer 2013-02-22 21:08:38 UTC
Good find. Another Online assumption we need to fix.

Comment 4 Luke Meyer 2013-02-26 16:32:22 UTC
The check in emb_cart_controller.rb is rather spurious. 
Fixing with commit 216b3d32488bd74c085db65229bd2e94cdfb297f and another build of the controller.

This isn't a problem upstream; I tested, and once you get the app created (by specifying -g) it can add cartridges fine.

Comment 6 Luke Meyer 2013-02-26 19:08:40 UTC
Fixed with rubygem-openshift-origin-controller-1.0.14-1.el6op

Comment 7 xjia 2013-02-27 11:55:47 UTC
Version:
http://buildvm-devops.usersys.redhat.com/puddle/build/OpenShiftEnterprise/1.1.z/2013-02-26.1/

Verify:
[root@broker ~]# oo-admin-ctl-district


{"name"=>"newsize",
 "uuid"=>"91cf4df558724c75992e611d5341a819",
 "max_uid"=>6999,
 "externally_reserved_uids_size"=>0,
 "node_profile"=>"newsize",
 "server_identities"=>{"broker"=>{"active"=>true}},
 "active_server_identities_size"=>1,
 "creation_time"=>"2013-02-27T06:22:23-05:00",
 "max_capacity"=>6000,
 "available_uids"=>"<5998 uids hidden>",
 "available_capacity"=>5998}


{"name"=>"medium",
 "uuid"=>"884ea38ec8bd4157a665e4345208f61d",
 "max_uid"=>6999,
 "externally_reserved_uids_size"=>0,
 "node_profile"=>"medium",
 "server_identities"=>{"node1"=>{"active"=>true}},
 "active_server_identities_size"=>1,
 "creation_time"=>"2013-02-27T06:25:38-05:00",
 "max_capacity"=>6000,
 "available_uids"=>"<6000 uids hidden>",
 "available_capacity"=>6000}
[root@broker ~]# mco ping
broker                                   time=52.51 ms
node1                                    time=92.02 ms
^C

---- ping statistics ----
2 replies max: 92.02 min: 52.51 avg: 72.26
[root@broker ~]# rhc setup
OpenShift Client Tools (RHC) Setup Wizard

This wizard will help you upload your SSH keys, set your application namespace, and check that other programs like Git are properly installed.

Login to broker.test.com: |zsun| jia
Password: ***

Saving previous configuration to /root/.openshift/express.conf.bak

Creating /root/.openshift/express.conf to store your configuration

Checking for git ... found git version 1.7.1

Checking your namespace ... jia

Checking for applications ... none

Run 'rhc app create' to create your first application.

  Do-It-Yourself                            rhc app create <app name> diy-0.1
  JBoss Enterprise Application Platform 6.0 rhc app create <app name> jbosseap-6.0
  Jenkins Server 1.4                        rhc app create <app name> jenkins-1.4
  PHP 5.3                                   rhc app create <app name> php-5.3
  Perl 5.10                                 rhc app create <app name> perl-5.10
  Python 2.6                                rhc app create <app name> python-2.6
  Ruby 1.8                                  rhc app create <app name> ruby-1.8
  Ruby 1.9                                  rhc app create <app name> ruby-1.9
  Tomcat (JBoss Enterprise Web Server 1.0)  rhc app create <app name> jbossews-1.0

Your client tools are now configured.
[root@broker ~]# rhc app create php5 php -pjia -s
Using php-5.3 (PHP 5.3) for 'php'

Application Options
===================
  Namespace: jia
  Scaling:   yes
  Cartridge: php-5.3
  Gear Size: default

Creating application 'php5' ... done

Waiting for your DNS name to be available ... done

Downloading the application Git repository ...
Initialized empty Git repository in /root/php5/.git/
The authenticity of host 'php5-jia.test.com (10.4.59.196)' can't be established.
RSA key fingerprint is 77:09:84:7f:03:7b:85:3d:58:c7:cf:3b:eb:51:cf:43.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'php5-jia.test.com' (RSA) to the list of known hosts.

Your application code is now in 'php5'

php5 @ http://php5-jia.test.com/ (uuid: a11fa6efb7c349c5b6c153e9b3fb0cb4)
=========================================================================
  Created:   6:50 AM
  Gear Size: newsize
  Git URL:   ssh://a11fa6efb7c349c5b6c153e9b3fb0cb4.com/~/git/php5.git/
  SSH:       a11fa6efb7c349c5b6c153e9b3fb0cb4.com

  php-5.3 (PHP 5.3)
  =================
    Scaling: x2 (minimum: 2, maximum: available) on newsize gears

  haproxy-1.4 (HAProxy 1.4)
  =========================

RESULT:
Application php5 was created.

[root@broker ~]# rhc cartridge add mysql -a php5 -pjia
Adding 'mysql-5.1' to application 'php5' ... Success
mysql-5.1 (MySQL Database 5.1)
==============================
  Connection URL: mysql://f6d8294630-jia.test.com:38071/
  Database Name:  php5
  Password:       LfL8dfy4ks42
  Username:       admin

RESULT:
Added mysql-5.1 to application php5

MySQL 5.1 database added.  Please make note of these credentials:

   Root User: admin
   Root Password: LfL8dfy4ks42
   Database Name: php5

Connection URL: mysql://$OPENSHIFT_GEAR_DNS:$OPENSHIFT_MYSQL_DB_PROXY_PORT/
MySQL gear-local connection URL: mysql://$OPENSHIFT_MYSQL_DB_HOST:$OPENSHIFT_MYSQL_DB_PORT/

Comment 11 errata-xmlrpc 2013-04-02 08:50:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0694.html