Bug 1067404

Summary: rhc cartridge storage --remove not working on haproxy gear
Product: OpenShift Online Reporter: Aleksandar Kostadinov <akostadi>
Component: PodAssignee: Abhishek Gupta <abhgupta>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 2.xCC: abhgupta, bperkins, jhou, jokerman, libra-bugs, lnader, mmccomas, xiama
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1108575 (view as bug list) Environment:
Last Closed: 2014-07-15 10:27:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1108575    

Description Aleksandar Kostadinov 2014-02-20 12:13:08 UTC
In the below log you can see that adding storage to haproxy-1.4 cartridge works but then removing it does not work. It reports successful operation but storage keeps staying at 11GB. Doing remove on the php-5.3 cartridge does the right thing though.
I'm not sure why one would add/remove storage to/from the haproxy gear but since this is listed in the test plan I'm reporting it.

Tested with devenv_4399. I'm pretty sure I did the same with an earlier build a couple of days ago and it was working. But now it does not.

> [avalon@localhost ~]$ rhc cartridge storage -c haproxy-1.4 -a cphp0s0 --add 4gb
> Set storage on cartridge ... set to 7GB
> 
> Storage Info
> ------------
>   Base Gear Storage:       1GB
>   Additional Gear Storage: 7GB
> [avalon@localhost ~]$ rhc cartridge storage -c haproxy-1.4 -a cphp0s0 --add 4
> Set storage on cartridge ... set to 11GB
> 
> Storage Info
> ------------
>   Base Gear Storage:       1GB
>   Additional Gear Storage: 11GB
> [avalon@localhost ~]$ rhc cartridge storage -c haproxy-1.4 -a cphp0s0 --show
> RESULT:
> Web Load Balancer
> -----------------
>   Base Gear Storage:       1GB
>   Additional Gear Storage: 11GB
> 
> [avalon@localhost ~]$ rhc cartridge storage  -a cphp0s0
> RESULT:
> 
> Web Load Balancer
> -----------------
>   Base Gear Storage:       1GB
>   Additional Gear Storage: 11GB
> 
> PHP 5.3
> -------
>   Base Gear Storage:       1GB
>   Additional Gear Storage: 11GB
> 
> MongoDB 2.4
> -----------
>   Base Gear Storage:       1GB
>   Additional Gear Storage: 4GB
> [avalon@localhost ~]$ rhc cartridge storage -c haproxy-1.4 -a cphp0s0  --remove 1gb
> Set storage on cartridge ... set to 10GB
> 
> Storage Info
> ------------
>   Base Gear Storage:       1GB
>   Additional Gear Storage: 11GB
> [avalon@localhost ~]$ rhc cartridge storage -c haproxy-1.4 -a cphp0s0 --remove 1gb
> Set storage on cartridge ... set to 10GB
> 
> Storage Info
> ------------
>   Base Gear Storage:       1GB
>   Additional Gear Storage: 11GB
> [avalon@localhost ~]$ rhc cartridge storage -c haproxy-1.4 -a cphp0s0 --remove 1gb
> Set storage on cartridge ... set to 10GB
> 
> Storage Info
> ------------
>   Base Gear Storage:       1GB
>   Additional Gear Storage: 11GB
> 
> [avalon@localhost ~]$ rhc cartridge storage -c php-5.3 -a cphp0s0 --remove 1gb
> Set storage on cartridge ... set to 10GB
> 
> Storage Info
> ------------
>   Base Gear Storage:       1GB
>   Additional Gear Storage: 10GB

Comment 2 Rajat Chopra 2014-05-27 19:19:11 UTC
Broker does not seem to send the instructions to node in case of --remove.

Comment 3 Abhishek Gupta 2014-05-28 20:06:48 UTC
The workaround for this issue, as stated, is to add/remove the storage on the web_framework cartridge.


There are two alternative approaches:

1. Block sparse cartridges from setting additional storage.
 - This will need to be handled in case haproxy is in "standalone" mode (Windows application)
 - If a sparse cart is the ONLY cart within a group instance, an exception could be made to this rule

2. We could make storage part of component specific override as well.
 - This creates confusion when both sparse and non-sparse carts, within the same group instance, specify storage 


Given the issues with #2, I would be tempted to go with approach #1 and make it obvious to the user by specifying the exact reason for the failure to add storage to the haproxy cart. The response could include the cartridge that the user could specify the storage on.

Comment 4 Lili Nader 2014-05-28 20:33:05 UTC
Given that we are moving away from multiple carts residing on the same gear in 3.x I suggest we lower the priority on this bug.

Comment 5 Aleksandar Kostadinov 2014-05-29 07:54:19 UTC
Why not just allow for the cart to have quota changed? And I couldn't understand how "moving away from multiple carts residing on the same gear in 3.x" would change anything about priority of this bug. We won't need to support --remove for haproxy in this case? Could you elaborate?

Comment 6 Lili Nader 2014-05-29 20:26:32 UTC
In current scheme of things a happroxy and php-5.3 cartridge can reside on the same gear groups. When a storage quota is changed, it is set on the gear group and not on the cartridge itself. So when you set quota on haproxy or php-5.3 you are setting the quota on the gear group containing both cartridges. 

The happroxy is a sparse cartridge and will reside only on some of the gears within the gear group (actually one in case of haproxy but by definition a sparse cartridge can be on every nth gear). 

So you could say in the current schema we cannot support setting quota on a sparse cartridge. Because we have no way of setting the quota on specific gears (where the sparse cartridge resides).  We only have a away of setting a quota on a gear group (which may contain other cartridges).

We could go ahead and build this into 2.x but given that all this is going to change in 3.x.  It seems hardly worth it.

Alternatively, we could prevent both add/remove storage functionality for sparse cartridges but then we prevent users from setting quota on sparse cartridge when the sparse cartridge is the only cartridges (see examples given in comment 3 above)

The workaround for this issue is for the user to remove quota on web framework cartridge instead of the web proxy cartridge.

Comment 7 Aleksandar Kostadinov 2014-05-29 20:35:41 UTC
I see your point but the question I still can't understand is why --add storage to haproxy should work but remove not? IMO the simplest solution is to make --remove also work for haproxy. It has worked in the past, so also makes sense from backward compatibility point of view.
Moreover it's absolutely not acceptable for the command to exit with success status and quata to still be unchanged.

Any technical difficulties to make --remove work with haproxy given --add already works?

IMHO +1 for the above proposed complicated changes to wait for OSO 3.x

Comment 8 Lili Nader 2014-05-30 17:22:34 UTC
I agree with you that the command should not have a success status when it has not been successful in changing the storage.

Comment 10 Abhishek Gupta 2014-06-16 19:52:40 UTC
Fixed with --> https://github.com/openshift/origin-server/pull/5458

Comment 11 openshift-github-bot 2014-06-16 21:13:26 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/4de9bb52836a24d34832f61496186f5d753e048e
Bug 1067404: Handling additional storage correctly at the group level

Comment 12 Jianwei Hou 2014-06-17 02:44:23 UTC
Verified on devenv_4874, the cartridge storage remove command now works on haproxy-1.4 as well.

% rhc cartridge storage --add 8 -c haproxy-1.4 -a php1s
Set storage on cartridge ... set to 8GB

Storage Info
------------
  Base Gear Storage:       1GB
  Additional Gear Storage: 8GB

0 % rhc cartridge storage --show -c haproxy-1.4 -a php1s
RESULT:
Web Load Balancer
-----------------
  Base Gear Storage:       1GB
  Additional Gear Storage: 8GB

0 % rhc cartridge storage --remove 4 -c haproxy-1.4 -a php1s
Set storage on cartridge ... set to 4GB

Storage Info
------------
  Base Gear Storage:       1GB
  Additional Gear Storage: 4GB

0 % rhc cartridge storage --show -a php1s               
RESULT:

Web Load Balancer
-----------------
  Base Gear Storage:       1GB
  Additional Gear Storage: 4GB

PHP 5.3
-------
  Base Gear Storage:       1GB
  Additional Gear Storage: 4GB

0 % rhc cartridge storage --remove 4 -c haproxy-1.4 -a php1s
Set storage on cartridge ... set to 0GB

Storage Info
------------
  Base Gear Storage:       1GB
  Additional Gear Storage: None

hjw@hjw-sixmachine ~
0 % rhc cartridge storage --show -a php1s                   
RESULT:

Web Load Balancer
-----------------
  Base Gear Storage:       1GB
  Additional Gear Storage: None

PHP 5.3
-------
  Base Gear Storage:       1GB
  Additional Gear Storage: None