Bug 1008007
| Summary: | oo-admin-repair --ssh-keys crashes in production | ||
|---|---|---|---|
| Product: | OpenShift Online | Reporter: | Sten Turpin <sten> |
| Component: | Pod | Assignee: | Abhishek Gupta <abhgupta> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | libra bugs <libra-bugs> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 2.x | CC: | lxia, twiest, xtian |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2013-09-19 16:51:21 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This has been fixed and should be in the release 2.0.33. checked on devenv_3786 with following steps, no crash. But oo-admin-repair is using the wrong id(should be app_uuid, but it's using user_uuid) to fixed the ssh key mismatch, thus still not working. 1.Set up and create app 2.change the ssh-key content and canonical name to null in mongo 3.oo-admin-chk 4.oo-admin-repair 5.oo-admin-repair Step 5 still show ssh key mismatch as oo-admin-repair is using wrong id. [root@domU-12-31-39-09-15-D8 ~]# oo-admin-chk -l 1 -v Started at: 2013-09-16 23:00:28 -0400 Time to fetch mongo data: 0.023s Total gears found in mongo: 1 Checking consumed gear count for user user_with_multiple_gear_sizes... OK Checking consumed gear count for user user_with_extra_storage... OK Checking consumed gear count for user user_with_certificate_capabilities... OK Checking consumed gear count for user a... OK Time to get all gears from nodes: 21.371s Total gears found on the nodes: 1 Total nodes that responded : 1 Time to get all sshkeys for all gears from nodes: 20.057s Total gears found on the nodes: 1 Total nodes that responded : 1 Checking application gears and ssh keys on corresponding nodes: 5237c3c4f40cab048a00000a : String... OK Checking ssh keys for gear: 5237c3c4f40cab048a00000a... OK Checking node gears in application database: 5237c3c4f40cab048a00000a... OK Checking for application without any group instances in the database: Checking for application without any gears in the group instances in the database: Checking for application without the gears attribute in the group instances in the database: Checking for users with nil or empty or missing login in the database: Checking for unreserved UIDs in the district: Checking for unused UIDs in the district: Checking for gears with the same UID: Checking gears available in applications collection but not in usage_records and viceversa: OK Checking gears with additional storage in applications collection but not in usage_records and viceversa: OK Checking gears with premium cartridge in applications collection but not in usage_records and viceversa: OK Checking un-ended records in usage_records collection but not in usage collection and viceversa: OK Check failed. Application 'app' for domain '5237c047f40cab048a000006' has a denormalized domain_namespace of 'a' instead of the correct canonical namespace '' in mongo. Please refer to the oo-admin-repair tool to resolve some of these inconsistencies. Total time: 41.571s Finished at: 2013-09-16 23:01:10 -0400 [root@domU-12-31-39-09-15-D8 ~]# oo-admin-repair --ssh-keys --verbose Started at: 2013-09-16 23:03:28 -0400 Time to fetch mongo data: 0.101s Total gears found in mongo: 1 Time to get all sshkeys for all gears from nodes: 21.283s Total gears found on the nodes: 1 Total nodes that responded : 1 Checking application gears and ssh keys on corresponding nodes: Checking ssh keys for gear: 5237c3c4f40cab048a00000a... FAIL Check failed. Gear '5237c3c4f40cab048a00000a' has key with comment 'OPENSHIFT-5237c3c4f40cab048a00000a-application-5237c3c4f40cab048a00000a' on the node but not in mongo. Gear '5237c3c4f40cab048a00000a' has key with comment 'OPENSHIFT-5237c3c4f40cab048a00000a-5237c03ef40cab048a000001-adomU1231390915D' on the node but not in mongo. Total 1 applications have ssh key mismatches. Fixing ssh key inconsistencies for all affected applications: Fixing ssh keys for application: 5237c3c4f40cab0cdb000001... OK Application '5237c3c4f40cab0cdb000001' not found in the database Fixed ssh key mismatches for 1 applications. Total time: 21.398s Finished at: 2013-09-16 23:03:50 -0400 [root@domU-12-31-39-09-15-D8 ~]# oo-admin-repair --ssh-keys --verbose Started at: 2013-09-16 23:06:55 -0400 Time to fetch mongo data: 0.012s Total gears found in mongo: 1 Time to get all sshkeys for all gears from nodes: 21.352s Total gears found on the nodes: 1 Total nodes that responded : 1 Checking application gears and ssh keys on corresponding nodes: Checking ssh keys for gear: 5237c3c4f40cab048a00000a... FAIL Check failed. Gear '5237c3c4f40cab048a00000a' has key with comment 'OPENSHIFT-5237c3c4f40cab048a00000a-application-5237c3c4f40cab048a00000a' on the node but not in mongo. Gear '5237c3c4f40cab048a00000a' has key with comment 'OPENSHIFT-5237c3c4f40cab048a00000a-5237c03ef40cab048a000001-default' on the node but not in mongo. Gear '5237c3c4f40cab048a00000a' has key with comment 'OPENSHIFT-5237c3c4f40cab048a00000a-5237c03ef40cab048a000001-adomU1231390915D' on the node but not in mongo. Total 1 applications have ssh key mismatches. Fixing ssh key inconsistencies for all affected applications: Fixing ssh keys for application: 5237c3c4f40cab0cdb000001... OK Application '5237c3c4f40cab0cdb000001' not found in the database Fixed ssh key mismatches for 1 applications. Total time: 21.377s Finished at: 2013-09-16 23:07:16 -0400 Back to ON_QA to retest against devenv-stage_475 since the fix is even not merged in devenv_3786 yet. checked on devenv_stage_477, no crash now. Problems list below will be tracked by bug 1006526, thus moving this one to verified. Change the ssh-key content and canonical name to null in mongo, then run 'oo-admin-repair --ssh-keys --verbose' several times, ssh key mismatch and then be fixed show every time the command is run. # oo-admin-repair --ssh-keys --verbose Started at: 2013-09-17 02:37:20 -0400 Time to fetch mongo data: 0.172s Total gears found in mongo: 1 Time to get all sshkeys for all gears from nodes: 20.791s Total gears found on the nodes: 1 Total nodes that responded : 1 Checking application gears and ssh keys on corresponding nodes: Checking ssh keys for gear: 5237f5e3c727ff7544000007... FAIL Check failed. Gear '5237f5e3c727ff7544000007' has key with comment 'OPENSHIFT-5237f5e3c727ff7544000007-5237f59ac727ff7544000001-default' on the node but not in mongo. Total 1 applications have ssh key mismatches. Fixing ssh key inconsistencies for all affected applications: Fixing ssh keys for application: 5237f5e3c727ff7544000007... OK Fixed ssh key mismatches for 1 applications. Total time: 22.274s Finished at: 2013-09-17 02:37:42 -0400 # oo-admin-repair --ssh-keys --verbose Started at: 2013-09-17 02:38:41 -0400 Time to fetch mongo data: 0.035s Total gears found in mongo: 1 Time to get all sshkeys for all gears from nodes: 20.948s Total gears found on the nodes: 1 Total nodes that responded : 1 Checking application gears and ssh keys on corresponding nodes: Checking ssh keys for gear: 5237f5e3c727ff7544000007... FAIL Check failed. Gear '5237f5e3c727ff7544000007' has key with comment 'OPENSHIFT-5237f5e3c727ff7544000007-5237f59ac727ff7544000001-default' on the node but not in mongo. Total 1 applications have ssh key mismatches. Fixing ssh key inconsistencies for all affected applications: Fixing ssh keys for application: 5237f5e3c727ff7544000007... OK Fixed ssh key mismatches for 1 applications. Total time: 22.347s Finished at: 2013-09-17 02:39:03 -0400 # oo-admin-repair --ssh-keys --verbose Started at: 2013-09-17 02:39:37 -0400 Time to fetch mongo data: 0.04s Total gears found in mongo: 1 Time to get all sshkeys for all gears from nodes: 20.902s Total gears found on the nodes: 1 Total nodes that responded : 1 Checking application gears and ssh keys on corresponding nodes: Checking ssh keys for gear: 5237f5e3c727ff7544000007... FAIL Check failed. Gear '5237f5e3c727ff7544000007' has key with comment 'OPENSHIFT-5237f5e3c727ff7544000007-5237f59ac727ff7544000001-default' on the node but not in mongo. Total 1 applications have ssh key mismatches. Fixing ssh key inconsistencies for all affected applications: Fixing ssh keys for application: 5237f5e3c727ff7544000007... OK Fixed ssh key mismatches for 1 applications. Total time: 21.755s Finished at: 2013-09-17 02:39:59 -0400 |
Description of problem: oo-admin-repair --ssh-keys throws a stacktrace in production Version-Release number of selected component (if applicable): openshift-origin-broker-util-1.13.11-1.el6oso.noarch How reproducible: always Steps to Reproduce: 1. run oo-admin-repair --ssh-keys --verbose Actual results: $ sudo oo-admin-repair --ssh-keys --verbose Started at: 2013-09-13 14:36:19 -0400 /usr/sbin/oo-admin-repair:161:in `digest': can't convert nil into String (TypeError) from /usr/sbin/oo-admin-repair:161:in `hexdigest' from /usr/sbin/oo-admin-repair:161:in `block in get_user_info' from /usr/sbin/oo-admin-repair:161:in `each' from /usr/sbin/oo-admin-repair:161:in `get_user_info' from /usr/sbin/oo-admin-repair:206:in `block in <main>' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.13.12/lib/openshift/data_store.rb:27:in `block (2 levels) in find' from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/cursor.rb:286:in `each' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.13.12/lib/openshift/data_store.rb:26:in `block in find' from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/collection.rb:276:in `find' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.13.12/lib/openshift/data_store.rb:25:in `find' from /usr/sbin/oo-admin-repair:205:in `<main>' Expected results: oo-admin-repair repairs ssh keys Additional info: possibly related to https://bugzilla.redhat.com/show_bug.cgi?id=1006526