Bug 1122084

Summary:	oo-admin-move should move gears to nodes with enough free space + buffer space
Product:	OpenShift Online	Reporter:	Matt Woodson <mwoodson>
Component:	Unknown	Assignee:	Sally <somalley>
Status:	CLOSED EOL	QA Contact:	zhaozhanqi <zzhao>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	2.x	CC:	aos-bugs, eparis, jgoulding, jokerman, mmccomas, tiwillia, xtian
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1362666 (view as bug list)		Environment:
Last Closed:	2018-09-11 02:14:11 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1277547, 1362666

Description Matt Woodson 2014-07-22 13:51:51 UTC

Description of problem:

Last night, oo-admin-move moved a gear that was 21GB to a node that had 21.1GB free.  This set off our paging alerts as the node was essentially full (only 100MB were free).  While there was technically enough space on the node, this makes the node unusable after the move.

When a gear gets moved, it needs to move to a host that has adequate free space + a buffer of space.  This could be a number of GB or a % of free space.


Version-Release number of selected component (if applicable):
openshift-origin-broker-util-1.26.3-1.el6oso.noarch

How reproducible:

I would assume very reproducible


Actual results:

Gear was moved to new node, but there was not enough disk space after the move.

Expected results:

Gear to be moved and have a buffer as well.

Comment 2 Sally 2016-06-09 21:00:33 UTC

https://github.com/openshift/origin-server/pull/6382 merged

Comment 3 zhaozhanqi 2016-11-09 10:38:00 UTC

Verified this bug on devenv_5830

with steps:

1) Launch ec2 instance and setup multi-node env
2) create distinct and add nodes to it
3) Create app 
4) Create big data file in gear on node1 and make the available disk to 1G using
  dd if=/dev/zero of=swapfile4 bs=1024 count=7262144
5) Create another app on other node2
6) Create big data file in this app about 1G
7) move this app to node1

# oo-admin-move --gear_uuid 58219aecc69622f2ad00024b -i ip-172-18-2-156
URL: http://app1-zzhao.dev.rhcloud.com
Login: zzhao
App UUID: 58219aecc69622f2ad00024b
Gear UUID: 58219aecc69622f2ad00024b
DEBUG: Source district uuid: 533183076375773558341632
DEBUG: Destination district uuid: 533183076375773558341632
DEBUG: Getting existing app 'app1' status before moving
DEBUG: Gear component 'php-5.4' was running
DEBUG: Stopping existing app cartridge 'php-5.4' before moving
DEBUG: Force stopping existing app before moving
DEBUG: Gear platform is 'linux'
DEBUG: Moving failed.  Rolling back gear '58219aecc69622f2ad00024b' in 'app1' with delete on 'ip-172-18-2-156'
Gear '58219aecc69622f2ad00024b' cannot be moved to 'ip-172-18-2-156'.  Not enough disk space, node would be > 95% full after move