Bug 1277261

Summary: Encountering "KeyError: 'idCert'" when getting a Response: status=202 during registration
Product: Red Hat Enterprise Linux 7 Reporter: John Sefler <jsefler>
Component: subscription-managerAssignee: candlepin-bugs
Status: CLOSED WONTFIX QA Contact: John Sefler <jsefler>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.2CC: bcourt
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-16 20:07:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description John Sefler 2015-11-02 20:27:52 UTC
Description of problem:
Following from Bug 1277256... I am also encountering "KeyError: 'idCert'" when getting a Response: status=202 from production candlepin environment during a registration.  Note that this bug was written to capture and fix the traceback on the client.  The bigger problem is that the server should not be returning a 202. 

Version-Release number of selected component (if applicable):
[root@jsefler-7 ~]# subscription-manager version
server type: This system is currently not registered.
subscription management server: 0.9.26.11-1
subscription management rules: 5.15
subscription-manager: 1.15.9-15.el7
python-rhsm: 1.15.4-5.el7

How reproducible:
I suspect this issue is caused by the aftermath of the weekend outage https://mojo.redhat.com/docs/DOC-1053566
This bug is normally not reproducible.

Steps to Reproduce:
[root@jsefler-7 ~]# subscription-manager register --serverurl=subscription.rhn.redhat.com:443/subscription --username=qa
Registering to: subscription.rhn.redhat.com:443/subscription
Password: 
Organization: foobar       <========= BANG! The qa account in production normally belongs to its own org 
'idCert'                   <========= BANG!
[root@jsefler-7 ~]# 
[root@jsefler-7 ~]# tail -50 /var/log/rhsm/rhsm.log
2015-11-02 15:18:51,030 [INFO] subscription-manager:22970 @connection.py:778 - Connection built: host=subscription.rhn.redhat.com port=443 handler=/subscription auth=none
2015-11-02 15:18:51,056 [DEBUG] subscription-manager:22970 @connection.py:475 - Loaded CA certificates from /etc/rhsm/ca/: jsefler-f22-candlepin.pem, jsefler-f14-candlepin.pem, jsefler-f22-7candlepin.pem, redhat-uep.pem, redhat-entitlement-authority.pem
2015-11-02 15:18:51,056 [DEBUG] subscription-manager:22970 @connection.py:523 - Making request: GET /subscription/
2015-11-02 15:18:51,519 [DEBUG] subscription-manager:22970 @connection.py:555 - Response: status=200
2015-11-02 15:18:51,522 [DEBUG] subscription-manager:22970 @connection.py:795 - Server supports the following resources: {'': '/', 'guestids': '/consumers/{consumer_uuid}/guestids', 'cdn': '/cdn', 'content_overrides': '/consumers/{consumer_uuid}/content_overrides', 'hypervisors': '/hypervisors', 'serials': '/serials', 'deleted_consumers': '/deleted_consumers', 'consumers': '/consumers', 'migrations': '/migrations', 'content': '/content', 'entitlements': '/entitlements', 'events': '/events', 'status': '/status', 'jobs': '/jobs', 'users': '/users', 'subscriptions': '/subscriptions', 'rules': '/rules', 'distributor_versions': '/distributor_versions', 'consumertypes': '/consumertypes', 'statistics/generate': '/statistics/generate', 'pools': '/pools', 'atom': '/atom', 'packages': '/consumers/{consumer_uuid}/packages', 'owners': '/owners', 'roles': '/roles', 'admin': '/admin', 'products': '/products', 'activation_keys': '/activation_keys', 'crl': '/crl'}
2015-11-02 15:18:51,527 [DEBUG] subscription-manager:22970 @connection.py:475 - Loaded CA certificates from /etc/rhsm/ca/: jsefler-f22-candlepin.pem, jsefler-f14-candlepin.pem, jsefler-f22-7candlepin.pem, redhat-uep.pem, redhat-entitlement-authority.pem
2015-11-02 15:18:51,528 [DEBUG] subscription-manager:22970 @connection.py:523 - Making request: GET /subscription/status
2015-11-02 15:18:52,002 [DEBUG] subscription-manager:22970 @connection.py:555 - Response: status=200
2015-11-02 15:18:52,006 [INFO] subscription-manager:22970 @managercli.py:363 - Server Versions: {'rules-version': '5.15', 'candlepin': '0.9.26.11-1', 'server-type': 'This system is currently not registered.'}
2015-11-02 15:18:52,008 [INFO] subscription-manager:22970 @managercli.py:352 - Client Versions: {'python-rhsm': '1.15.4-5.el7', 'subscription-manager': '1.15.9-15.el7'}
2015-11-02 15:18:52,008 [INFO] subscription-manager:22970 @managercli.py:328 - Consumer Identity name=None uuid=None
2015-11-02 15:18:52,056 [INFO] subscription-manager:22970 @managercli.py:328 - Consumer Identity name=None uuid=None
2015-11-02 15:19:04,013 [INFO] subscription-manager:22970 @connection.py:778 - Connection built: host=subscription.rhn.redhat.com port=443 handler=/subscription auth=basic username=qa
2015-11-02 15:19:04,067 [DEBUG] subscription-manager:22970 @hwprobe.py:554 - cpu info: {'cpu.cpu(s)': 2, 'cpu.core(s)_per_socket': 1, 'cpu.thread(s)_per_core': 1, 'cpu.topology_source': 'kernel /sys cpu sibling lists', 'cpu.cpu_socket(s)': 2}
2015-11-02 15:19:04,417 [DEBUG] subscription-manager:22970 @hwprobe.py:772 - Running 'virt-what'
2015-11-02 15:19:04,522 [DEBUG] subscription-manager:22970 @hwprobe.py:776 - virt-what stdout: kvm

2015-11-02 15:19:04,523 [DEBUG] subscription-manager:22970 @hwprobe.py:777 - virt-what stderr: 
2015-11-02 15:19:04,538 [INFO] subscription-manager:22970 @hwprobe.py:854 - collected virt facts: virt.is_guest=True, virt.host_type=kvm, virt.uuid=05c3b827-58a9-7a28-fd5b-ee639ccdda62
2015-11-02 15:19:04,540 [DEBUG] subscription-manager:22970 @connection.py:475 - Loaded CA certificates from /etc/rhsm/ca/: jsefler-f22-candlepin.pem, jsefler-f14-candlepin.pem, jsefler-f22-7candlepin.pem, redhat-uep.pem, redhat-entitlement-authority.pem
2015-11-02 15:19:04,541 [DEBUG] subscription-manager:22970 @connection.py:523 - Making request: GET /subscription/users/qa%40redhat.com/owners
2015-11-02 15:19:05,079 [DEBUG] subscription-manager:22970 @connection.py:555 - Response: status=202
2015-11-02 15:19:13,671 [DEBUG] subscription-manager:22970 @connection.py:475 - Loaded CA certificates from /etc/rhsm/ca/: jsefler-f22-candlepin.pem, jsefler-f14-candlepin.pem, jsefler-f22-7candlepin.pem, redhat-uep.pem, redhat-entitlement-authority.pem
2015-11-02 15:19:13,672 [DEBUG] subscription-manager:22970 @connection.py:523 - Making request: GET /subscription/
2015-11-02 15:19:14,134 [DEBUG] subscription-manager:22970 @connection.py:555 - Response: status=200
2015-11-02 15:19:14,135 [DEBUG] subscription-manager:22970 @connection.py:795 - Server supports the following resources: {'': '/', 'guestids': '/consumers/{consumer_uuid}/guestids', 'cdn': '/cdn', 'content_overrides': '/consumers/{consumer_uuid}/content_overrides', 'hypervisors': '/hypervisors', 'serials': '/serials', 'deleted_consumers': '/deleted_consumers', 'consumers': '/consumers', 'migrations': '/migrations', 'content': '/content', 'entitlements': '/entitlements', 'events': '/events', 'status': '/status', 'jobs': '/jobs', 'users': '/users', 'subscriptions': '/subscriptions', 'rules': '/rules', 'distributor_versions': '/distributor_versions', 'consumertypes': '/consumertypes', 'statistics/generate': '/statistics/generate', 'pools': '/pools', 'atom': '/atom', 'packages': '/consumers/{consumer_uuid}/packages', 'owners': '/owners', 'roles': '/roles', 'admin': '/admin', 'products': '/products', 'activation_keys': '/activation_keys', 'crl': '/crl'}
2015-11-02 15:19:14,138 [DEBUG] subscription-manager:22970 @connection.py:475 - Loaded CA certificates from /etc/rhsm/ca/: jsefler-f22-candlepin.pem, jsefler-f14-candlepin.pem, jsefler-f22-7candlepin.pem, redhat-uep.pem, redhat-entitlement-authority.pem
2015-11-02 15:19:14,138 [DEBUG] subscription-manager:22970 @connection.py:523 - Making request: POST /subscription/consumers?owner=foobar
2015-11-02 15:19:14,901 [DEBUG] subscription-manager:22970 @connection.py:555 - Response: status=202
2015-11-02 15:19:14,910 [DEBUG] subscription-manager:22970 @cache.py:110 - Wrote cache: /var/lib/rhsm/cache/installed_products.json
2015-11-02 15:19:14,912 [ERROR] subscription-manager:22970 @managercli.py:160 - exception caught in subscription-manager
2015-11-02 15:19:14,912 [ERROR] subscription-manager:22970 @managercli.py:161 - 'idCert'
Traceback (most recent call last):
  File "/usr/sbin/subscription-manager", line 86, in <module>
    sys.exit(abs(main() or 0))
  File "/usr/sbin/subscription-manager", line 77, in main
    return managercli.ManagerCLI().main()
  File "/usr/share/rhsm/subscription_manager/managercli.py", line 2629, in main
    return CLI.main(self)
  File "/usr/share/rhsm/subscription_manager/cli.py", line 159, in main
    return cmd.main()
  File "/usr/share/rhsm/subscription_manager/managercli.py", line 489, in main
    return_code = self._do_command()
  File "/usr/share/rhsm/subscription_manager/managercli.py", line 1081, in _do_command
    consumer_info = self._persist_identity_cert(consumer)
  File "/usr/share/rhsm/subscription_manager/managercli.py", line 1148, in _persist_identity_cert
    return managerlib.persist_consumer_cert(consumer)
  File "/usr/share/rhsm/subscription_manager/managerlib.py", line 70, in persist_consumer_cert
    consumer = identity.ConsumerIdentity(consumerinfo['idCert']['key'],
KeyError: 'idCert'

Comment 2 Barnaby Court 2015-11-16 20:07:38 UTC
Too low a priority to fix as it appears to be a gremlin related to a larger data center outage.