Bug 1008007 - oo-admin-repair --ssh-keys crashes in production
oo-admin-repair --ssh-keys crashes in production
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Pod (Show other bugs)
2.x
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Abhishek Gupta
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-13 14:41 EDT by Sten Turpin
Modified: 2015-05-14 20:20 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-19 12:51:21 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Sten Turpin 2013-09-13 14:41:30 EDT
Description of problem: oo-admin-repair --ssh-keys throws a stacktrace in production


Version-Release number of selected component (if applicable):
openshift-origin-broker-util-1.13.11-1.el6oso.noarch

How reproducible: always


Steps to Reproduce:
1. run oo-admin-repair --ssh-keys --verbose

Actual results:
$ sudo oo-admin-repair --ssh-keys --verbose
Started at: 2013-09-13 14:36:19 -0400
/usr/sbin/oo-admin-repair:161:in `digest': can't convert nil into String (TypeError)
        from /usr/sbin/oo-admin-repair:161:in `hexdigest'
        from /usr/sbin/oo-admin-repair:161:in `block in get_user_info'
        from /usr/sbin/oo-admin-repair:161:in `each'
        from /usr/sbin/oo-admin-repair:161:in `get_user_info'
        from /usr/sbin/oo-admin-repair:206:in `block in <main>'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.13.12/lib/openshift/data_store.rb:27:in `block (2 levels) in find'
        from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/cursor.rb:286:in `each'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.13.12/lib/openshift/data_store.rb:26:in `block in find'
        from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/collection.rb:276:in `find'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.13.12/lib/openshift/data_store.rb:25:in `find'
        from /usr/sbin/oo-admin-repair:205:in `<main>'

Expected results:
oo-admin-repair repairs ssh keys

Additional info:
possibly related to https://bugzilla.redhat.com/show_bug.cgi?id=1006526
Comment 1 Abhishek Gupta 2013-09-16 12:55:39 EDT
This has been fixed and should be in the release 2.0.33.
Comment 2 Liang Xia 2013-09-16 23:22:40 EDT
checked on devenv_3786 with following steps, no crash.
But oo-admin-repair is using the wrong id(should be app_uuid, but it's using user_uuid) to fixed the ssh key mismatch, thus still not working.

1.Set up and create app
2.change the ssh-key content and canonical name to null in mongo
3.oo-admin-chk
4.oo-admin-repair
5.oo-admin-repair

Step 5 still show ssh key mismatch as oo-admin-repair is using wrong id.

[root@domU-12-31-39-09-15-D8 ~]# oo-admin-chk -l 1 -v
Started at: 2013-09-16 23:00:28 -0400
Time to fetch mongo data: 0.023s
Total gears found in mongo: 1
Checking consumed gear count for user user_with_multiple_gear_sizes@test.com...	OK
Checking consumed gear count for user user_with_extra_storage@test.com...	OK
Checking consumed gear count for user user_with_certificate_capabilities@test.com...	OK
Checking consumed gear count for user a@a.a...	OK
Time to get all gears from nodes: 21.371s
Total gears found on the nodes: 1
Total nodes that responded : 1
Time to get all sshkeys for all gears from nodes: 20.057s
Total gears found on the nodes: 1
Total nodes that responded : 1
Checking application gears and ssh keys on corresponding nodes:
5237c3c4f40cab048a00000a : String...	OK
Checking ssh keys for gear: 5237c3c4f40cab048a00000a...	OK
Checking node gears in application database:
5237c3c4f40cab048a00000a...	OK
Checking for application without any group instances in the database:
Checking for application without any gears in the group instances in the database:
Checking for application without the gears attribute in the group instances in the database:
Checking for users with nil or empty or missing login in the database:
Checking for unreserved UIDs in the district:
Checking for unused UIDs in the district:
Checking for gears with the same UID:
Checking gears available in applications collection but not in usage_records and viceversa: OK
Checking gears with additional storage in applications collection but not in usage_records and viceversa: OK
Checking gears with premium cartridge in applications collection but not in usage_records and viceversa: OK
Checking un-ended records in usage_records collection but not in usage collection and viceversa: OK
Check failed.
Application 'app' for domain '5237c047f40cab048a000006' has a denormalized domain_namespace of 'a' instead of the correct canonical namespace '' in mongo.
Please refer to the oo-admin-repair tool to resolve some of these inconsistencies.
Total time: 41.571s
Finished at: 2013-09-16 23:01:10 -0400
[root@domU-12-31-39-09-15-D8 ~]#  oo-admin-repair --ssh-keys --verbose
Started at: 2013-09-16 23:03:28 -0400
Time to fetch mongo data: 0.101s
Total gears found in mongo: 1
Time to get all sshkeys for all gears from nodes: 21.283s
Total gears found on the nodes: 1
Total nodes that responded : 1
Checking application gears and ssh keys on corresponding nodes:
Checking ssh keys for gear: 5237c3c4f40cab048a00000a...	FAIL
Check failed.
Gear '5237c3c4f40cab048a00000a' has  key with comment 'OPENSHIFT-5237c3c4f40cab048a00000a-application-5237c3c4f40cab048a00000a' on the node but not in mongo.
Gear '5237c3c4f40cab048a00000a' has  key with comment 'OPENSHIFT-5237c3c4f40cab048a00000a-5237c03ef40cab048a000001-adomU1231390915D' on the node but not in mongo.

Total 1 applications have ssh key mismatches.

Fixing ssh key inconsistencies for all affected applications:
Fixing ssh keys for application: 5237c3c4f40cab0cdb000001...	OK
Application '5237c3c4f40cab0cdb000001' not found in the database

Fixed ssh key mismatches for 1 applications.

Total time: 21.398s
Finished at: 2013-09-16 23:03:50 -0400
[root@domU-12-31-39-09-15-D8 ~]#  oo-admin-repair --ssh-keys --verbose
Started at: 2013-09-16 23:06:55 -0400
Time to fetch mongo data: 0.012s
Total gears found in mongo: 1
Time to get all sshkeys for all gears from nodes: 21.352s
Total gears found on the nodes: 1
Total nodes that responded : 1
Checking application gears and ssh keys on corresponding nodes:
Checking ssh keys for gear: 5237c3c4f40cab048a00000a...	FAIL
Check failed.
Gear '5237c3c4f40cab048a00000a' has  key with comment 'OPENSHIFT-5237c3c4f40cab048a00000a-application-5237c3c4f40cab048a00000a' on the node but not in mongo.
Gear '5237c3c4f40cab048a00000a' has  key with comment 'OPENSHIFT-5237c3c4f40cab048a00000a-5237c03ef40cab048a000001-default' on the node but not in mongo.
Gear '5237c3c4f40cab048a00000a' has  key with comment 'OPENSHIFT-5237c3c4f40cab048a00000a-5237c03ef40cab048a000001-adomU1231390915D' on the node but not in mongo.

Total 1 applications have ssh key mismatches.

Fixing ssh key inconsistencies for all affected applications:
Fixing ssh keys for application: 5237c3c4f40cab0cdb000001...	OK
Application '5237c3c4f40cab0cdb000001' not found in the database

Fixed ssh key mismatches for 1 applications.

Total time: 21.377s
Finished at: 2013-09-16 23:07:16 -0400
Comment 3 Xiaoli Tian 2013-09-17 00:40:20 EDT
Back to ON_QA to retest against devenv-stage_475 since the fix is even  not merged in devenv_3786 yet.
Comment 4 Liang Xia 2013-09-17 03:24:00 EDT
checked on devenv_stage_477, no crash now. Problems list below will be tracked by bug 1006526, thus moving this one to verified.

Change the ssh-key content and canonical name to null in mongo, then run 'oo-admin-repair --ssh-keys --verbose' several times, ssh key mismatch and then be fixed show every time the command is run.

# oo-admin-repair --ssh-keys --verbose
Started at: 2013-09-17 02:37:20 -0400
Time to fetch mongo data: 0.172s
Total gears found in mongo: 1
Time to get all sshkeys for all gears from nodes: 20.791s
Total gears found on the nodes: 1
Total nodes that responded : 1
Checking application gears and ssh keys on corresponding nodes:
Checking ssh keys for gear: 5237f5e3c727ff7544000007...	FAIL
Check failed.
Gear '5237f5e3c727ff7544000007' has  key with comment 'OPENSHIFT-5237f5e3c727ff7544000007-5237f59ac727ff7544000001-default' on the node but not in mongo.

Total 1 applications have ssh key mismatches.

Fixing ssh key inconsistencies for all affected applications:
Fixing ssh keys for application: 5237f5e3c727ff7544000007...	OK

Fixed ssh key mismatches for 1 applications.

Total time: 22.274s
Finished at: 2013-09-17 02:37:42 -0400



# oo-admin-repair --ssh-keys --verbose
Started at: 2013-09-17 02:38:41 -0400
Time to fetch mongo data: 0.035s
Total gears found in mongo: 1
Time to get all sshkeys for all gears from nodes: 20.948s
Total gears found on the nodes: 1
Total nodes that responded : 1
Checking application gears and ssh keys on corresponding nodes:
Checking ssh keys for gear: 5237f5e3c727ff7544000007...	FAIL
Check failed.
Gear '5237f5e3c727ff7544000007' has  key with comment 'OPENSHIFT-5237f5e3c727ff7544000007-5237f59ac727ff7544000001-default' on the node but not in mongo.

Total 1 applications have ssh key mismatches.

Fixing ssh key inconsistencies for all affected applications:
Fixing ssh keys for application: 5237f5e3c727ff7544000007...	OK

Fixed ssh key mismatches for 1 applications.

Total time: 22.347s
Finished at: 2013-09-17 02:39:03 -0400



# oo-admin-repair --ssh-keys --verbose
Started at: 2013-09-17 02:39:37 -0400
Time to fetch mongo data: 0.04s
Total gears found in mongo: 1
Time to get all sshkeys for all gears from nodes: 20.902s
Total gears found on the nodes: 1
Total nodes that responded : 1
Checking application gears and ssh keys on corresponding nodes:
Checking ssh keys for gear: 5237f5e3c727ff7544000007...	FAIL
Check failed.
Gear '5237f5e3c727ff7544000007' has  key with comment 'OPENSHIFT-5237f5e3c727ff7544000007-5237f59ac727ff7544000001-default' on the node but not in mongo.

Total 1 applications have ssh key mismatches.

Fixing ssh key inconsistencies for all affected applications:
Fixing ssh keys for application: 5237f5e3c727ff7544000007...	OK

Fixed ssh key mismatches for 1 applications.

Total time: 21.755s
Finished at: 2013-09-17 02:39:59 -0400

Note You need to log in before you can comment on or make changes to this bug.