Hide Forgot
Description of problem: Given one scalable app with mysql and mongodb cartridge, make sure mysql and mongodb gear on the node which is down and the head gear's node is normal. Execute 'oo-admin-repair --unresponsive-apps -v',it only remove one DB cartridge one time. Version-Release number of selected component (if applicable): devenv_3829 How reproducible: always Steps to Reproduce: 1.create multi-node env 2.create one scalable app with mysql and mongodb db 3.make the node(mysql gear and mongodb gear) down by killing the 'mcollectived' process 4.oo-admin-repair --unresponsive-apps -v Actual results: step 4: only mysql cartridge removed,did not remove mongodb cartridge. [root@ip-10-203-26-188 broker]# oo-admin-repair --unresponsive-apps -v Started at: 2013-09-26 02:10:47 -0400 Time to fetch mongo data: 0.11s Total gears found in mongo: 5 Checking for unresponsive servers... FAIL Servers that are unresponsive: Server: ip-10-114-25-51 (district: dist1) Check failed. Some servers are unresponsive: ip-10-114-25-51 Do you want to delete unresponsive servers from their respective districts [yes/no]: no Skipped deleting unresponsive servers from their respective districts. Found 1 unresponsive scalable apps that can not be recovered. zqpy33s (id: 5243b7ab4977d1cab7000020) Do you want to delete all of them [yes/no]: yes Application with id: 5243b7ab4977d1cab7000020 deleted. Found 1 unresponsive scalable apps that are recoverable but some features/carts need to be removed. zqpy27s (id: 5243c56b4977d1cab700005e) Do you want to skip all of them [yes/no]:(Warning: entering 'no' will remove features from apps) no Removed features: mysql-5.1 from application with id: 5243c56b4977d1cab700005e. Application with id: 5243c56b4977d1cab700005e fixed. Finished fixing/deleting unresponsive scalable apps. Total time: 180.681s Finished at: 2013-09-26 02:13:48 -0400 Expected results: Both mysql cartrdige and mongo cartridge should be removed. Additional info:
Fixed in https://github.com/openshift/origin-server/pull/3716
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/4037d2fadb7ef3c3d84f748bb67ba9588be12158 Bug 1012264 - Pretty print usage duration in oo-admin-ctl-usage script
Fixed in https://github.com/openshift/origin-server/pull/3731
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/f8edd3fec7afa8268615973aa21b82365c5d29d4 Bug 1012264 - Remove all unresponsive db features + misc bug fixes
Tested this bug on devenv_3844 when deleting multi DB gears on node which is down, it did have not give a prompt. 1. create one scalable app with mongo mysql and postgresql db 2. make mysql and postgresql db gear down 3. run oo-admin-repair --removed-nodes [root@ip-10-204-113-68 ~]# oo-admin-repair --removed-nodes Started at: 2013-09-28 23:33:19 -0400 Time to fetch mongo data: 0.091s Total gears found in mongo: 7 Servers that are unresponsive: Server: ip-10-100-211-42 (district: dist1), Confirm [yes/no]: yes Check failed. Some servers are unresponsive: ip-10-100-211-42 Do you want to delete unresponsive servers from their respective districts [yes/no]: no Total time: 78.569s Finished at: 2013-09-28 23:34:38 -0400 4. [zqzhao@dhcp-13-222 ~]$ rhc app show zqpy33s -g ID State Cartridges Size SSH URL ------------------------ ------- ---------------------- ----- --------------------------------------------------------------------- 52479b542d19cf5001000127 started python-3.3 haproxy-1.4 small 52479b542d19cf5001000127.rhcloud.com 52479ddf2d19cf0ab500002f started mongodb-2.2 small 52479ddf2d19cf0ab500002f.rhcloud.com
And show this app by 'rhc app show $app', it also can show mysql and postgresql, the gears show 0 [zqzhao@dhcp-13-222 ~]$ rhc app show phps2 phps2 @ http://phps2-zqd.dev.rhcloud.com/ (uuid: 5247fff62d19cf900200070c) -------------------------------------------------------------------------- Domain: zqd Created: 6:24 AM Gears: 2 (defaults to small) Git URL: ssh://5247fff62d19cf900200070c.rhcloud.com/~/git/phps2.git/ SSH: 5247fff62d19cf900200070c.rhcloud.com php-5.3 (PHP 5.3) ----------------- Scaling: x1 (minimum: 1, maximum: available) on small gears haproxy-1.4 (Web Load Balancer) ------------------------------- Scaling: x1 (minimum: 1, maximum: 9) on small gears mysql-5.1 (MySQL 5.1) --------------------- Gears: 0 small ---------------------------------- Connection URL: mysql://$OPENSHIFT_MYSQL_DB_HOST:$OPENSHIFT_MYSQL_DB_PORT/ Database Name: phps2 Password: 77NAgSsWfXmd Username: adminEwbX6WF mongodb-2.2 (MongoDB 2.2) ------------------------- Gears: 1 small Connection URL: mongodb://$OPENSHIFT_MONGODB_DB_HOST:$OPENSHIFT_MONGODB_DB_PORT/ Database Name: phps2 Password: CbCBFMJK-K_P Username: admin postgresql-8.4 (PostgreSQL 8.4) ------------------------------- Gears: 0 small ----------------------------------- Connection URL: postgresql://$OPENSHIFT_POSTGRESQL_DB_HOST:$OPENSHIFT_POSTGRESQL_DB_PORT Database Name: phps2 Password: NZ6wq8nMX7Sw Username: adminrg4nl8y
Fixed in https://github.com/openshift/origin-server/pull/3742
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/61f244099c9909879b3d13cde4a8f66131b56fbb Bug 1012264 : oo-admin-repair --removed-nodes fixes
Tested this issue on devenv_stage_488, it has been fixed.