Bug 987777 - Status of the added gear should be 'started' instead of 'new' for scalable app after scale up via REST API
Status of the added gear should be 'started' instead of 'new' for scalable ap...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Pod (Show other bugs)
2.x
Unspecified Linux
medium Severity medium
: ---
: ---
Assigned To: Rajat Chopra
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-24 03:10 EDT by Lei Zhang
Modified: 2015-05-14 20:19 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-08-07 18:56:18 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Lei Zhang 2013-07-24 03:10:12 EDT
Description of problem:
Given a scalable app(e.g myruby18s), scale up via REST API. The added gear can be accessed via ssh, but when check the status of app, the status of added gear always displaying 'new' insead of 'started'. when check the haproxy-status page, no added gear shows.
if access via HTTP(51ef68efda7b860a2a000004-ray.dev.rhcloud.com), the URL will redirect into (https://51ef68efda7b860a2a000004-ray.dev.rhcloud.com:8118/)


Version-Release number of selected component (if applicable):
devenv_3545
rhc-1.12.1

How reproducible:
always

Steps to Reproduce:
1. create one scalable app
rhc app create myruby18s ruby-1.8 -s 
2. scale up via REST API
 curl -k -H "Accept:application/xml" --user "lzhang@redhat.com:rayzhang13" https://ec2-23-22-26-133.compute-1.amazonaws.com/broker/rest/domains/ray/applications/myruby18s/events -d event=scale-up -X POST
3. check the status of gear
rhc app show --gear -a myruby18s

Actual results:
After scale up via REST API, the status of added gear displays 'new' instead of 'started'
[rayzhang@rayzhang Work]$ rhc app show --gear -a myruby18s
ID                       State   Cartridges           Size  SSH URL
------------------------ ------- -------------------- ----- ---------------------------------------------------------------------
51ef6879da7b860a2a000001 started ruby-1.8 haproxy-1.4 small 51ef6879da7b860a2a000001@myruby18s-ray.dev.rhcloud.com
51ef68efda7b860a2a000004 new     ruby-1.8 haproxy-1.4 small 51ef68efda7b860a2a000004@51ef68efda7b860a2a000004-ray.dev.rhcloud.com

Expected results:
After scale up via REST API, the status of added gear should display 'started' instead of 'new'

Additional info:

'use rhc command to scale up' works well.
 [rayzhang@rayzhang Work]$  rhc cartridge scale -a myruby18s -c ruby-1.8 --min 2
This operation will run until the application is at the minimum scale and may take several minutes.
Setting scale range for ruby-1.8 ... done
ruby-1.8 (Ruby 1.8)
-------------------
  Scaling: x2 (minimum: 2, maximum: available) on small gears
[rayzhang@rayzhang Work]$ rhc app show --gear -a myruby18s
ID                       State   Cartridges           Size  SSH URL
------------------------ ------- -------------------- ----- ---------------------------------------------------------------------
185733443926305487716352 started ruby-1.8 haproxy-1.4 small 185733443926305487716352@myruby18s-ray.dev.rhcloud.com
709458229968406684106752 started ruby-1.8 haproxy-1.4 small 709458229968406684106752@709458229968406684106752-ray.dev.rhcloud.com
Comment 1 Lei Zhang 2013-07-24 04:41:16 EDT
Auto scaling testing also failed, the status of added gear always displays 'new'. and check the haproxy/logs/scale_events.log in rhcsh, found exit when add gear.

I, [2013-07-24T04:20:41.956699 #21430]  INFO -- : GEAR_UP - capacity: 100.0% gear_count: 1 sessions: 16 up_thresh: 90.0%
I, [2013-07-24T04:20:44.529675 #2018]  INFO -- : GEAR_UP - capacity: 350.0% gear_count: 1 sessions: 56 up_thresh: 90.0%
I, [2013-07-24T04:21:42.099826 #7917]  INFO -- : Starting haproxy_ctld
D, [2013-07-24T04:21:42.101842 #7917] DEBUG -- : GEAR_INFO - capacity: 0.0% gear_count: 1 sessions: 0 up/remove_thresh: 90.0%/1.0% sec_left_til_remove: 120 gear_remove_thresh: 0/20
I, [2013-07-24T04:21:49.195646 #9136]  INFO -- : Starting haproxy_ctld
D, [2013-07-24T04:21:49.196859 #9136] DEBUG -- : GEAR_INFO - capacity: 0.0% gear_count: 1 sessions: 0 up/remove_thresh: 90.0%/1.0% sec_left_til_remove: 120 gear_remove_thresh: 0/20
D, [2013-07-24T04:21:49.900874 #2018] DEBUG -- : GEAR_UP - add-gear: exit: pid 5673 exit 0  stdout: The add-gear request failed with http_code: 503
Comment 2 Rajat Chopra 2013-07-24 14:51:06 EDT
While the CURL request was being processed, the output does show 'new'.  But after it is completed, the output shows 'started'. Could not reproduce this problem with php-5.3!


[root@ip-10-204-131-114 ~]# rhc app show --gear --app test2
ID                       State   Cartridges          Size  SSH URL
------------------------ ------- ------------------- ----- -----------------------------------------------------------------------
51f0208f207cc50993000008 started php-5.3 haproxy-1.4 small 51f0208f207cc50993000008@test2-matte.dev.rhcloud.com
646050197976223953453056 new     php-5.3 haproxy-1.4 small 646050197976223953453056@646050197976223953453056-matte.dev.rhcloud.com


[root@ip-10-204-131-114 ~]# rhc app show --gear --app test2
ID                       State   Cartridges          Size  SSH URL
------------------------ ------- ------------------- ----- -----------------------------------------------------------------------
51f0208f207cc50993000008 started php-5.3 haproxy-1.4 small 51f0208f207cc50993000008@test2-matte.dev.rhcloud.com
646050197976223953453056 started php-5.3 haproxy-1.4 small 646050197976223953453056@646050197976223953453056-matte.dev.rhcloud.com
Comment 3 Rajat Chopra 2013-07-24 14:55:05 EDT
Works for ruby-1.8 also. Please verify if the 'rhc app show --gear' command was run in parallel or not. It has to wait for the curl request to complete for the correct status to show up.
Comment 4 Rajat Chopra 2013-07-24 15:00:41 EDT
Maybe there was some unknown issue with the devenv. Just tested the auto-scaling, that works too.

From comment#1, status 503 suggests that the broker was not in good shape.
Comment 5 Lei Zhang 2013-07-26 00:56:26 EDT
Hi Rajat,

I retest on devenv-3563 rhc-1.12.2, this bug still exists. I found the key point is to do scale up. if scaled up, jenkins-client can be added. 

Step 1: create a scalable app 
#rhc app create myruby18s ruby-1.8 -s

[rayzhang@rayzhang Work]$ rhc app show myruby18s
myruby18s @ http://myruby18s-rayzhang.dev.rhcloud.com/ (uuid: 51f1fce9bcec436d1000000f)
---------------------------------------------------------------------------------------
  Created: 12:36 PM
  Gears:   2 (defaults to small)
  Git URL: ssh://51f1fce9bcec436d1000000f@myruby18s-rayzhang.dev.rhcloud.com/~/git/myruby18s.git/
  SSH:     51f1fce9bcec436d1000000f@myruby18s-rayzhang.dev.rhcloud.com

  ruby-1.8 (Ruby 1.8)
  -------------------
    Scaling: x2 (minimum: 1, maximum: available) on small gears

  haproxy-1.4 (OpenShift Web Balancer)
  ------------------------------------
    Gears: Located with ruby-1.8

Step 2: scale up via REST API
#curl -k -H "Accept:application/xml" --user "lzhang+1@redhat.com:rayzhang13" https://ec2-23-22-229-123.compute-1.amazonaws.com/broker/rest/domains/rayzhang/applications/myruby18s/events -d event=scale-up -X POST

Step 3: add jenkins-client
# rhc cartridge add -a myruby18s -c jenkins-client-1.4 

[rayzhang@rayzhang ~]$ rhc cartridge add -a myruby18s -c jenkins-client-1.4 
Adding jenkins-client-1.4 to application 'myruby18s' ... done

jenkins-client-1.4 (Jenkins Client 1.4)
---------------------------------------
  Gears: Located with ruby-1.8, haproxy-1.4
Comment 6 Lei Zhang 2013-07-26 01:03:37 EDT
#Comment 5 is invalid, should be like thest:
I retest on devenv-3563 rhc-1.12.2, this bug still exists.

Step 1: create a scalable app 
#rhc app create myruby18s ruby-1.8 -s

[rayzhang@rayzhang Work]$ rhc app show myruby18s
myruby18s @ http://myruby18s-rayzhang.dev.rhcloud.com/ (uuid: 51f1fce9bcec436d1000000f)
---------------------------------------------------------------------------------------
  Created: 12:36 PM
  Gears:   2 (defaults to small)
  Git URL: ssh://51f1fce9bcec436d1000000f@myruby18s-rayzhang.dev.rhcloud.com/~/git/myruby18s.git/
  SSH:     51f1fce9bcec436d1000000f@myruby18s-rayzhang.dev.rhcloud.com

  ruby-1.8 (Ruby 1.8)
  -------------------
    Scaling: x2 (minimum: 1, maximum: available) on small gears

  haproxy-1.4 (OpenShift Web Balancer)
  ------------------------------------
    Gears: Located with ruby-1.8

Step 2: scale up via REST API
#curl -k -H "Accept:application/xml" --user "lzhang+1@redhat.com:rayzhang13" https://ec2-23-22-229-123.compute-1.amazonaws.com/broker/rest/domains/rayzhang/applications/myruby18s/events -d event=scale-up -X POST

Step 3: check the status 
#rhc app show --gear -a myruby18s

[rayzhang@rayzhang Work]$ rhc app show --gear -a myruby18s 
ID                       State   Cartridges           Size  SSH URL
------------------------ ------- -------------------- ----- --------------------------------------------------------------------------
51f1fce9bcec436d1000000f started ruby-1.8 haproxy-1.4 small 51f1fce9bcec436d1000000f@myruby18s-rayzhang.dev.rhcloud.com
628894016789686173827072 new     ruby-1.8 haproxy-1.4 small 628894016789686173827072@628894016789686173827072-rayzhang.dev.rhcloud.com



Other point, After I scaled up at first time, waiting after auto-scaling down happend, I scale up again,  error happend 'Cannot scale up beyond maximum gear limit in app myruby18s.'

[rayzhang@rayzhang Work]$ curl -k -H "Accept:application/xml" --user "lzhang+1@redhat.com:rayzhang13" https://ec2-23-22-229-123.compute-1.amazonaws.com/broker/rest/domains/rayzhang/applications/myruby18s/events -d event=scale-up -X POST
<?xml version="1.0" encoding="UTF-8"?>
<response>
  <status>unprocessable_entity</status>
  <type nil="true"></type>
  <data>
    <datum nil="true"></datum>
  </data>
  <messages>
    <message>
      <severity>error</severity>
      <text>Cannot scale up beyond maximum gear limit in app myruby18s.</text>
      <exit-code>168</exit-code>
      <field nil="true"></field>
    </message>
  </messages>
  <version>1.6</version>
  <api-version>1.6</api-version>
  <supported-api-versions>
    <supported-api-version>1.0</supported-api-version>
    <supported-api-version>1.1</supported-api-version>
    <supported-api-version>1.2</supported-api-version>
    <supported-api-version>1.3</supported-api-version>
    <supported-api-version>1.4</supported-api-version>
    <supported-api-version>1.5</supported-api-version>
    <supported-api-version>1.6</supported-api-version>
  </supported-api-versions>
</response>
Comment 7 Rajat Chopra 2013-07-29 15:11:28 EDT
Fixed with rev#0bc92a9d4b75514bf667eacdf356581b055d6b6c
Comment 8 Lei Zhang 2013-07-29 22:16:50 EDT
Verified on INT(devenv_3578) and rhc-1.12.3, after scaled up via REST API, the status of gear is 'started'. 

[lijun@rayzhang Work]$ rhc app show --gear -a myruby18s
ID                       State   Cartridges           Size  SSH URL
------------------------ ------- -------------------- ----- --------------------------------------------------------------------------
51f71e2203ef64660e00029b started ruby-1.8 haproxy-1.4 small 51f71e2203ef64660e00029b@myruby18s-rayzhang.int.rhcloud.com
51f71ffa6cec0ea3f3000002 started ruby-1.8 haproxy-1.4 small 51f71ffa6cec0ea3f3000002@51f71ffa6cec0ea3f3000002-rayzhang.int.rhcloud.co

Note You need to log in before you can comment on or make changes to this bug.