This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1012264 - [origin_broker_98]Only can remove one cartridge one time by 'oo-admin-repair --unresponsive-apps' for scalable app with multi DB cartridge while the node is down.
[origin_broker_98]Only can remove one cartridge one time by 'oo-admin-repair ...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Pod (Show other bugs)
2.x
All All
medium Severity medium
: ---
: ---
Assigned To: Ravi Sankar
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-26 02:46 EDT by zhaozhanqi
Modified: 2015-05-14 20:21 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-17 09:31:29 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description zhaozhanqi 2013-09-26 02:46:11 EDT
Description of problem:
Given one scalable app with mysql and mongodb cartridge, make sure mysql and mongodb gear on the node which is down and the head gear's node is normal. Execute 'oo-admin-repair --unresponsive-apps -v',it only remove one DB cartridge one time.

Version-Release number of selected component (if applicable):
devenv_3829

How reproducible:
always

Steps to Reproduce:
1.create multi-node env
2.create one scalable app with mysql and mongodb db
3.make the node(mysql gear and mongodb gear) down by killing the 'mcollectived' process
4.oo-admin-repair --unresponsive-apps -v

Actual results:

step 4: only mysql cartridge removed,did not remove mongodb cartridge.

[root@ip-10-203-26-188 broker]# oo-admin-repair --unresponsive-apps -v
Started at: 2013-09-26 02:10:47 -0400
Time to fetch mongo data: 0.11s
Total gears found in mongo: 5
Checking for unresponsive servers...	FAIL
Servers that are unresponsive:
	Server: ip-10-114-25-51 (district: dist1)
Check failed.
Some servers are unresponsive: ip-10-114-25-51


Do you want to delete unresponsive servers from their respective districts [yes/no]: no
Skipped deleting unresponsive servers from their respective districts.
Found 1 unresponsive scalable apps that can not be recovered.
zqpy33s (id: 5243b7ab4977d1cab7000020)
Do you want to delete all of them [yes/no]: yes
Application with id: 5243b7ab4977d1cab7000020 deleted.
Found 1 unresponsive scalable apps that are recoverable but some features/carts need to be removed.
zqpy27s (id: 5243c56b4977d1cab700005e)
Do you want to skip all of them [yes/no]:(Warning: entering 'no' will remove features from apps) no 
Removed features: mysql-5.1 from application with id: 5243c56b4977d1cab700005e.
Application with id: 5243c56b4977d1cab700005e fixed.
Finished fixing/deleting unresponsive scalable apps.


Total time: 180.681s
Finished at: 2013-09-26 02:13:48 -0400


Expected results:
Both mysql cartrdige and mongo cartridge should be removed.

Additional info:
Comment 1 Ravi Sankar 2013-09-26 17:18:33 EDT
Fixed in https://github.com/openshift/origin-server/pull/3716
Comment 2 openshift-github-bot 2013-09-27 00:37:23 EDT
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/4037d2fadb7ef3c3d84f748bb67ba9588be12158
Bug 1012264 - Pretty print usage duration in oo-admin-ctl-usage script
Comment 3 Ravi Sankar 2013-09-27 20:27:04 EDT
Fixed in https://github.com/openshift/origin-server/pull/3731
Comment 4 openshift-github-bot 2013-09-27 20:46:55 EDT
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/f8edd3fec7afa8268615973aa21b82365c5d29d4
Bug 1012264 - Remove all unresponsive db features + misc bug fixes
Comment 5 zhaozhanqi 2013-09-29 00:54:12 EDT
Tested this bug on devenv_3844
when deleting multi DB gears on node which is down,  it did have not give a prompt. 

1. create one scalable app with mongo mysql and postgresql db
2. make mysql and postgresql db gear down
3. run oo-admin-repair --removed-nodes

[root@ip-10-204-113-68 ~]# oo-admin-repair --removed-nodes
Started at: 2013-09-28 23:33:19 -0400
Time to fetch mongo data: 0.091s
Total gears found in mongo: 7
Servers that are unresponsive:
	Server: ip-10-100-211-42 (district: dist1), Confirm [yes/no]: 
yes
Check failed.
Some servers are unresponsive: ip-10-100-211-42


Do you want to delete unresponsive servers from their respective districts [yes/no]: no


Total time: 78.569s
Finished at: 2013-09-28 23:34:38 -0400

4. [zqzhao@dhcp-13-222 ~]$ rhc app show zqpy33s -g
ID                       State   Cartridges             Size  SSH URL
------------------------ ------- ---------------------- ----- ---------------------------------------------------------------------
52479b542d19cf5001000127 started python-3.3 haproxy-1.4 small 52479b542d19cf5001000127@zqpy33s-zqd.dev.rhcloud.com
52479ddf2d19cf0ab500002f started mongodb-2.2            small 52479ddf2d19cf0ab500002f@52479ddf2d19cf0ab500002f-zqd.dev.rhcloud.com
Comment 6 zhaozhanqi 2013-09-29 06:35:13 EDT
And show this app by 'rhc app show $app', it also can show mysql and postgresql, the gears show 0


[zqzhao@dhcp-13-222 ~]$ rhc app show phps2
phps2 @ http://phps2-zqd.dev.rhcloud.com/ (uuid: 5247fff62d19cf900200070c)
--------------------------------------------------------------------------
  Domain:  zqd
  Created: 6:24 AM
  Gears:   2 (defaults to small)
  Git URL: ssh://5247fff62d19cf900200070c@phps2-zqd.dev.rhcloud.com/~/git/phps2.git/
  SSH:     5247fff62d19cf900200070c@phps2-zqd.dev.rhcloud.com

  php-5.3 (PHP 5.3)
  -----------------
    Scaling: x1 (minimum: 1, maximum: available) on small gears

  haproxy-1.4 (Web Load Balancer)
  -------------------------------
    Scaling: x1 (minimum: 1, maximum: 9) on small gears

  mysql-5.1 (MySQL 5.1)
  ---------------------
    Gears:          0 small          ----------------------------------
    Connection URL: mysql://$OPENSHIFT_MYSQL_DB_HOST:$OPENSHIFT_MYSQL_DB_PORT/
    Database Name:  phps2
    Password:       77NAgSsWfXmd
    Username:       adminEwbX6WF

  mongodb-2.2 (MongoDB 2.2)
  -------------------------
    Gears:          1 small
    Connection URL: mongodb://$OPENSHIFT_MONGODB_DB_HOST:$OPENSHIFT_MONGODB_DB_PORT/
    Database Name:  phps2
    Password:       CbCBFMJK-K_P
    Username:       admin

  postgresql-8.4 (PostgreSQL 8.4)
  -------------------------------
    Gears:          0 small   -----------------------------------
    Connection URL: postgresql://$OPENSHIFT_POSTGRESQL_DB_HOST:$OPENSHIFT_POSTGRESQL_DB_PORT
    Database Name:  phps2
    Password:       NZ6wq8nMX7Sw
    Username:       adminrg4nl8y
Comment 7 Ravi Sankar 2013-09-30 15:32:15 EDT
Fixed in https://github.com/openshift/origin-server/pull/3742
Comment 8 openshift-github-bot 2013-10-01 19:53:41 EDT
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/61f244099c9909879b3d13cde4a8f66131b56fbb
Bug 1012264 : oo-admin-repair --removed-nodes fixes
Comment 9 zhaozhanqi 2013-10-08 01:59:45 EDT
Tested this issue on devenv_stage_488, it has been fixed.

Note You need to log in before you can comment on or make changes to this bug.