Bug 1095947

Summary: oo-admin-move fails on scaled apps when one gear is at quota
Product: OpenShift Online Reporter: Sten Turpin <sten>
Component: MasterAssignee: Abhishek Gupta <abhgupta>
Status: CLOSED DUPLICATE QA Contact: weiwei jiang <wjiang>
Severity: low Docs Contact:
Priority: unspecified    
Version: 2.xCC: agrimm, aos-bugs, jokerman, mmccomas, somalley, wsun
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-02-02 17:00:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1277547    
Attachments:
Description Flags
redacted snippets from the broker log none

Description Sten Turpin 2014-05-08 21:55:54 UTC 
Description of problem: When one gear of a scaled app is at quota, none of the other gears can be oo-admin-moved


Version-Release number of selected component (if applicable): openshift-origin-broker-util-1.23.8-1.el6oso.noarch


How reproducible: Always


Steps to Reproduce:
1. Identify a scaled app with one member gear at quota
2. Attempt to oo-admin-move on any of the gears in the app

Actual results:
$ sudo oo-admin-move --gear_uuid 51b85b58e0b8cd1f75000046 --change_district -i ex-std-node424.prod.rhcloud.com
URL: http://[redacted].rhcloud.com
Login: [redacted]
App UUID: 51b8584c500446ecab00003c
Gear UUID: 51b85b58e0b8cd1f75000046
DEBUG: Source district uuid: 517be5425973ca09b4000001
DEBUG: Destination district uuid: 536930de5973ca5851000001
DEBUG: Getting existing app '[redacted]' status before moving
DEBUG: Gear component 'python-2.6' was running
DEBUG: Stopping existing app cartridge 'mysql-5.1' before moving
DEBUG: Reserved uid '5530' on district: '536930de5973ca5851000001'
DEBUG: Creating new account for gear '51b85b58e0b8cd1f75000046' on ex-std-node424.prod.rhcloud.com
DEBUG: Moving content for app '[redacted]', gear '51b85b58e0b8cd1f75000046' to ex-std-node424.prod.rhcloud.com
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Warning: Permanently added '10.91.29.188' (RSA) to the list of known hosts.
Agent pid 29310
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 29310 killed;
DEBUG: Moving system components for app '[redacted]', gear '51b85b58e0b8cd1f75000046' to ex-std-node424.prod.rhcloud.com
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 29381
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 29381 killed;
DEBUG: Starting cartridge 'mysql-5.1' in '[redacted]' after move on ex-std-node424.prod.rhcloud.com
DEBUG: Fixing DNS and mongo for gear '51b85b58e0b8cd1f75000046' after move
DEBUG: Changing server identity of '51b85b58e0b8cd1f75000046' from 'ex-std-node74.prod.rhcloud.com' to 'ex-std-node424.prod.rhcloud.com'
DEBUG: Moving failed.  Rolling back gear '51b85b58e0b8cd1f75000046' in 'teammixer' with delete on 'ex-std-node424.prod.rhcloud.com'
Failed to execute: 'control update-cluster' for /var/lib/openshift/51b8584c500446ecab00003c/haproxy

Expected results:
Gear should move successfully. 

Additional info:
It would be acceptible if oo-admin-move temporarily increased the quota on the full gear a small amount during the move.
Comment 1 Sten Turpin 2014-05-08 22:01:08 UTC 
Created attachment 893798 [details]
redacted snippets from the broker log
Comment 2 Sally 2016-02-02 17:00:25 UTC 
Closing/Marking as Duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1292133

Also, https://bugzilla.redhat.com/show_bug.cgi?id=1292133
has a workaround documented.

*** This bug has been marked as a duplicate of bug 1292133 ***