Bug 1361305

Summary:	gears exceeding quota cannot be stopped or idled
Product:	OpenShift Container Platform	Reporter:	Rory Thrasher <rthrashe>
Component:	Containers	Assignee:	Sally <somalley>
Status:	CLOSED ERRATA	QA Contact:	Gaoyun Pei <gpei>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	2.2.0	CC:	agrimm, aos-bugs, dma, gpei, jialiu, jokerman, mmccomas, rthrashe, somalley, sten, tiwillia
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	rubygem-openshift-origin-node-1.38.6.3-1.el6op, rubygem-openshift-origin-msg-broker-mcollective-1.36.2.2-1.el6op	Doc Type:	Bug Fix
Doc Text:	Cause: When unidling, upgrading, or moving a gear, a small amount of free space must be available on the gear for the operation. Consequence: Gears that are currently at or exeeding their filesystem usage quota are unable to be unidled, upgraded, or moved. Fix: A small amount of additional filesystem space is added to the gear's quota during these operations. Result: Gears at or exceeding their quota can successfully be unidled, upgraded, and moved.	Story Points:	---
Clone Of:	1292133	Environment:
Last Closed:	2016-08-24 19:46:53 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1292133
Bug Blocks:	1277547

Comment 3 Johnny Liu 2016-08-09 08:08:13 UTC

Verified this bug with 2.2/2016-08-08.1 puddle, and PASS.

In my gear:
> quota
Disk quotas for user jialiu-ruby18app-1 (uid 4286): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
     /dev/loop0 1048576*      0 1048576           79993       0   80000        

# repquota -a
*** Report for user quotas on device /dev/loop0
Block grace time: 7days; Inode grace time: 7days
                        Block limits                File limits
User            used    soft    hard  grace    used  soft  hard  grace
----------------------------------------------------------------------
root      --   19764       0       0           1420     0     0       
jialiu-ruby18app-1 -- 1048576       0 1048576          79993     0 80000       

stop/start the app successfully.
$ rhc app stop ruby18app
Warning: Gear jialiu-ruby18app-1 is using 100.0% of disk quota
Warning: Gear jialiu-ruby18app-1 is using 100.0% of inodes allowed

RESULT:
ruby18app stopped
[jialiu@jialiu-laptop ~]$ rhc app start ruby18app
Warning: Gear jialiu-ruby18app-1 is using 100.0% of disk quota
Warning: Gear jialiu-ruby18app-1 is using 100.0% of inodes allowed

RESULT:
ruby18app started


move the app successfully, and the quota is bump a little more.
# oo-admin-move --gear_uuid jialiu-ruby18app-1 -i node2.ose22-auto.com.cn
URL: http://ruby18app-jialiu.ose22-auto.com.cn
Login: jialiu
App UUID: 57a864f182611da20d0002b1
Gear UUID: 57a864f182611da20d0002b1
DEBUG: Source district uuid: 57a8243482611d52e5000001
DEBUG: Destination district uuid: 57a8243482611d52e5000001
WARNING: Quota exceeds 98% of target quota limits, using buffer.
QUOTA BUFFER: blocks max: 1258291 inodes max: 96000 allowed in increments.
DEBUG: Current blocks quota increased by 1% to 1059062.
DEBUG: Current inodes quota increased by 10 inodes to 80010.
DEBUG: Getting existing app 'ruby18app' status before moving
DEBUG: Gear component 'ruby-1.8' was running
DEBUG: Unpublishing routing information for gear 'jialiu-ruby18app-1'
DEBUG: Stopping existing app cartridge 'ruby-1.8' before moving
DEBUG: Force stopping existing app before moving
DEBUG: Gear platform is 'linux'
DEBUG: Creating new account for gear 'jialiu-ruby18app-1' on node2.ose22-auto.com.cn
DEBUG: Moving content for app 'ruby18app', gear 'jialiu-ruby18app-1' to node2.ose22-auto.com.cn
Agent pid 31379
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 31379 killed;
DEBUG: Moving system components for app 'ruby18app', gear 'jialiu-ruby18app-1' to node2.ose22-auto.com.cn
Agent pid 31925
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 31925 killed;
DEBUG: Starting cartridge 'ruby-1.8' in 'ruby18app' after move on node2.ose22-auto.com.cn
DEBUG: Fixing DNS and mongo for gear 'jialiu-ruby18app-1' after move
DEBUG: Changing server identity of 'jialiu-ruby18app-1' from 'node1.ose22-auto.com.cn' to 'node2.ose22-auto.com.cn'
DEBUG: Updating routing information for gear 'jialiu-ruby18app-1' after move
DEBUG: Deconfiguring old app 'ruby18app' on node1.ose22-auto.com.cn after move
Successfully moved gear with uuid 'jialiu-ruby18app-1' of app 'ruby18app' from 'node1.ose22-auto.com.cn' to 'node2.ose22-auto.com.cn'

After move, check the gear's quota.
> quota
Disk quotas for user jialiu-ruby18app-1 (uid 4286): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
     /dev/loop0    3344       0 1059061           79993       0   80010

Comment 5 errata-xmlrpc 2016-08-24 19:46:53 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-1773.html