Hide Forgot
Description of problem: oo-admin-repair --ssh-keys throws a stacktrace in production Version-Release number of selected component (if applicable): openshift-origin-broker-util-1.13.11-1.el6oso.noarch How reproducible: always Steps to Reproduce: 1. run oo-admin-repair --ssh-keys --verbose Actual results: $ sudo oo-admin-repair --ssh-keys --verbose Started at: 2013-09-13 14:36:19 -0400 /usr/sbin/oo-admin-repair:161:in `digest': can't convert nil into String (TypeError) from /usr/sbin/oo-admin-repair:161:in `hexdigest' from /usr/sbin/oo-admin-repair:161:in `block in get_user_info' from /usr/sbin/oo-admin-repair:161:in `each' from /usr/sbin/oo-admin-repair:161:in `get_user_info' from /usr/sbin/oo-admin-repair:206:in `block in <main>' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.13.12/lib/openshift/data_store.rb:27:in `block (2 levels) in find' from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/cursor.rb:286:in `each' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.13.12/lib/openshift/data_store.rb:26:in `block in find' from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/collection.rb:276:in `find' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.13.12/lib/openshift/data_store.rb:25:in `find' from /usr/sbin/oo-admin-repair:205:in `<main>' Expected results: oo-admin-repair repairs ssh keys Additional info: possibly related to https://bugzilla.redhat.com/show_bug.cgi?id=1006526
This has been fixed and should be in the release 2.0.33.
checked on devenv_3786 with following steps, no crash. But oo-admin-repair is using the wrong id(should be app_uuid, but it's using user_uuid) to fixed the ssh key mismatch, thus still not working. 1.Set up and create app 2.change the ssh-key content and canonical name to null in mongo 3.oo-admin-chk 4.oo-admin-repair 5.oo-admin-repair Step 5 still show ssh key mismatch as oo-admin-repair is using wrong id. [root@domU-12-31-39-09-15-D8 ~]# oo-admin-chk -l 1 -v Started at: 2013-09-16 23:00:28 -0400 Time to fetch mongo data: 0.023s Total gears found in mongo: 1 Checking consumed gear count for user user_with_multiple_gear_sizes... OK Checking consumed gear count for user user_with_extra_storage... OK Checking consumed gear count for user user_with_certificate_capabilities... OK Checking consumed gear count for user a... OK Time to get all gears from nodes: 21.371s Total gears found on the nodes: 1 Total nodes that responded : 1 Time to get all sshkeys for all gears from nodes: 20.057s Total gears found on the nodes: 1 Total nodes that responded : 1 Checking application gears and ssh keys on corresponding nodes: 5237c3c4f40cab048a00000a : String... OK Checking ssh keys for gear: 5237c3c4f40cab048a00000a... OK Checking node gears in application database: 5237c3c4f40cab048a00000a... OK Checking for application without any group instances in the database: Checking for application without any gears in the group instances in the database: Checking for application without the gears attribute in the group instances in the database: Checking for users with nil or empty or missing login in the database: Checking for unreserved UIDs in the district: Checking for unused UIDs in the district: Checking for gears with the same UID: Checking gears available in applications collection but not in usage_records and viceversa: OK Checking gears with additional storage in applications collection but not in usage_records and viceversa: OK Checking gears with premium cartridge in applications collection but not in usage_records and viceversa: OK Checking un-ended records in usage_records collection but not in usage collection and viceversa: OK Check failed. Application 'app' for domain '5237c047f40cab048a000006' has a denormalized domain_namespace of 'a' instead of the correct canonical namespace '' in mongo. Please refer to the oo-admin-repair tool to resolve some of these inconsistencies. Total time: 41.571s Finished at: 2013-09-16 23:01:10 -0400 [root@domU-12-31-39-09-15-D8 ~]# oo-admin-repair --ssh-keys --verbose Started at: 2013-09-16 23:03:28 -0400 Time to fetch mongo data: 0.101s Total gears found in mongo: 1 Time to get all sshkeys for all gears from nodes: 21.283s Total gears found on the nodes: 1 Total nodes that responded : 1 Checking application gears and ssh keys on corresponding nodes: Checking ssh keys for gear: 5237c3c4f40cab048a00000a... FAIL Check failed. Gear '5237c3c4f40cab048a00000a' has key with comment 'OPENSHIFT-5237c3c4f40cab048a00000a-application-5237c3c4f40cab048a00000a' on the node but not in mongo. Gear '5237c3c4f40cab048a00000a' has key with comment 'OPENSHIFT-5237c3c4f40cab048a00000a-5237c03ef40cab048a000001-adomU1231390915D' on the node but not in mongo. Total 1 applications have ssh key mismatches. Fixing ssh key inconsistencies for all affected applications: Fixing ssh keys for application: 5237c3c4f40cab0cdb000001... OK Application '5237c3c4f40cab0cdb000001' not found in the database Fixed ssh key mismatches for 1 applications. Total time: 21.398s Finished at: 2013-09-16 23:03:50 -0400 [root@domU-12-31-39-09-15-D8 ~]# oo-admin-repair --ssh-keys --verbose Started at: 2013-09-16 23:06:55 -0400 Time to fetch mongo data: 0.012s Total gears found in mongo: 1 Time to get all sshkeys for all gears from nodes: 21.352s Total gears found on the nodes: 1 Total nodes that responded : 1 Checking application gears and ssh keys on corresponding nodes: Checking ssh keys for gear: 5237c3c4f40cab048a00000a... FAIL Check failed. Gear '5237c3c4f40cab048a00000a' has key with comment 'OPENSHIFT-5237c3c4f40cab048a00000a-application-5237c3c4f40cab048a00000a' on the node but not in mongo. Gear '5237c3c4f40cab048a00000a' has key with comment 'OPENSHIFT-5237c3c4f40cab048a00000a-5237c03ef40cab048a000001-default' on the node but not in mongo. Gear '5237c3c4f40cab048a00000a' has key with comment 'OPENSHIFT-5237c3c4f40cab048a00000a-5237c03ef40cab048a000001-adomU1231390915D' on the node but not in mongo. Total 1 applications have ssh key mismatches. Fixing ssh key inconsistencies for all affected applications: Fixing ssh keys for application: 5237c3c4f40cab0cdb000001... OK Application '5237c3c4f40cab0cdb000001' not found in the database Fixed ssh key mismatches for 1 applications. Total time: 21.377s Finished at: 2013-09-16 23:07:16 -0400
Back to ON_QA to retest against devenv-stage_475 since the fix is even not merged in devenv_3786 yet.
checked on devenv_stage_477, no crash now. Problems list below will be tracked by bug 1006526, thus moving this one to verified. Change the ssh-key content and canonical name to null in mongo, then run 'oo-admin-repair --ssh-keys --verbose' several times, ssh key mismatch and then be fixed show every time the command is run. # oo-admin-repair --ssh-keys --verbose Started at: 2013-09-17 02:37:20 -0400 Time to fetch mongo data: 0.172s Total gears found in mongo: 1 Time to get all sshkeys for all gears from nodes: 20.791s Total gears found on the nodes: 1 Total nodes that responded : 1 Checking application gears and ssh keys on corresponding nodes: Checking ssh keys for gear: 5237f5e3c727ff7544000007... FAIL Check failed. Gear '5237f5e3c727ff7544000007' has key with comment 'OPENSHIFT-5237f5e3c727ff7544000007-5237f59ac727ff7544000001-default' on the node but not in mongo. Total 1 applications have ssh key mismatches. Fixing ssh key inconsistencies for all affected applications: Fixing ssh keys for application: 5237f5e3c727ff7544000007... OK Fixed ssh key mismatches for 1 applications. Total time: 22.274s Finished at: 2013-09-17 02:37:42 -0400 # oo-admin-repair --ssh-keys --verbose Started at: 2013-09-17 02:38:41 -0400 Time to fetch mongo data: 0.035s Total gears found in mongo: 1 Time to get all sshkeys for all gears from nodes: 20.948s Total gears found on the nodes: 1 Total nodes that responded : 1 Checking application gears and ssh keys on corresponding nodes: Checking ssh keys for gear: 5237f5e3c727ff7544000007... FAIL Check failed. Gear '5237f5e3c727ff7544000007' has key with comment 'OPENSHIFT-5237f5e3c727ff7544000007-5237f59ac727ff7544000001-default' on the node but not in mongo. Total 1 applications have ssh key mismatches. Fixing ssh key inconsistencies for all affected applications: Fixing ssh keys for application: 5237f5e3c727ff7544000007... OK Fixed ssh key mismatches for 1 applications. Total time: 22.347s Finished at: 2013-09-17 02:39:03 -0400 # oo-admin-repair --ssh-keys --verbose Started at: 2013-09-17 02:39:37 -0400 Time to fetch mongo data: 0.04s Total gears found in mongo: 1 Time to get all sshkeys for all gears from nodes: 20.902s Total gears found on the nodes: 1 Total nodes that responded : 1 Checking application gears and ssh keys on corresponding nodes: Checking ssh keys for gear: 5237f5e3c727ff7544000007... FAIL Check failed. Gear '5237f5e3c727ff7544000007' has key with comment 'OPENSHIFT-5237f5e3c727ff7544000007-5237f59ac727ff7544000001-default' on the node but not in mongo. Total 1 applications have ssh key mismatches. Fixing ssh key inconsistencies for all affected applications: Fixing ssh keys for application: 5237f5e3c727ff7544000007... OK Fixed ssh key mismatches for 1 applications. Total time: 21.755s Finished at: 2013-09-17 02:39:59 -0400