Bug 1478265

Summary: After deleting consumer host on Satellite6.1.12 server, no consumer.old folder is created in client.
Product: Red Hat Satellite Reporter: qianzhan
Component: CandlepinAssignee: Barnaby Court <bcourt>
Status: CLOSED WORKSFORME QA Contact: Katello QA List <katello-qa-list>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: UnspecifiedCC: bbuckingham, khowell, qianzhan
Target Milestone: Unspecified   
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-07 07:45:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description qianzhan 2017-08-04 07:00:29 UTC
Description of problem:
After deleting consumer host on Satellite6.1.12 server, no consumer.old folder is created in client.

Version-Release number of selected component (if applicable):


How reproducible:
always

Steps to Reproduce:
1. Register RHEL-7.4-20170727.1 against Satellite6.1.12.
[root@kvm-02-guest10 ~]# subscription-manager register
Registering to: bkr-hv03-guest39.dsal.lab.eng.bos.redhat.com:443/rhsm
Username: admin
Password: 
The system has been registered with ID: b4c4a052-c1a4-4c15-873e-d186e1d55ab1 

[root@kvm-02-guest10 ~]# subscription-manager identity
system identity: b4c4a052-c1a4-4c15-873e-d186e1d55ab1
name: kvm-02-guest10.rhts.eng.bos.redhat.com
org name: Default Organization
org ID: Default_Organization
environment name: Library

2. Log in the Satellite web UI, and delete the client.
3. Restart rhsmcertd service and wait 2 minutes:
[root@kvm-02-guest10 ~]# subscription-manager identity
Unit b4c4a052-c1a4-4c15-873e-d186e1d55ab1 has been deleted
[root@kvm-02-guest10 ~]# service rhsmcertd restart
Redirecting to /bin/systemctl restart rhsmcertd.service
[root@kvm-02-guest10 ~]# sleep 120

4. Check consumer.old folder:

[root@kvm-02-guest10 ~]# ls /etc/pki
CA        consumer     java   nss-legacy  product-default  rsyslog
ca-trust  entitlement  nssdb  product     rpm-gpg          tls
[root@kvm-02-guest10 ~]# ls /etc/pki/entitlement/
[root@kvm-02-guest10 ~]# ls /etc/pki/consumer/
cert.pem  key.pem


Actual results:
As step 4, the consumer.old folder is not created. And the consumer certs still exist in consumer folder.

Expected results:
After step 3, the consumer folder should be empty, and consumer certs should be moved in consumer.old folder.

Additional info:

Please refer the rhsm.log:

[root@kvm-02-guest10 ~]# cat /var/log/rhsm/rhsm.log 
2017-08-04 02:48:32,589 [INFO] rhsmd:23004:MainThread @rhsmd:263 - rhsmd started
2017-08-04 02:48:32,634 [INFO] subscription-manager:22998:MainThread @managercli.py:518 - X-Correlation-ID: 18f819b547564383809d5c147dc34f6f
2017-08-04 02:48:32,634 [INFO] subscription-manager:22998:MainThread @managercli.py:407 - Client Versions: {'python-rhsm': '1.19.9-1.el7', 'subscription-manager': '1.19.21-1.el7'}
2017-08-04 02:48:32,635 [INFO] subscription-manager:22998:MainThread @connection.py:822 - Connection built: host=bkr-hv03-guest39.dsal.lab.eng.bos.redhat.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2017-08-04 02:48:32,635 [INFO] subscription-manager:22998:MainThread @connection.py:822 - Connection built: host=bkr-hv03-guest39.dsal.lab.eng.bos.redhat.com port=443 handler=/rhsm auth=none
2017-08-04 02:48:32,635 [INFO] subscription-manager:22998:MainThread @managercli.py:407 - Client Versions: {'python-rhsm': '1.19.9-1.el7', 'subscription-manager': '1.19.21-1.el7'}
2017-08-04 02:48:32,647 [INFO] subscription-manager:22998:MainThread @managercli.py:382 - Consumer Identity name=None uuid=None
2017-08-04 02:48:32,647 [INFO] subscription-manager:22998:MainThread @managercli.py:382 - Consumer Identity name=None uuid=None
2017-08-04 02:48:36,172 [INFO] subscription-manager:22998:MainThread @connection.py:822 - Connection built: host=bkr-hv03-guest39.dsal.lab.eng.bos.redhat.com port=443 handler=/rhsm auth=basic username=admin
2017-08-04 02:48:36,248 [INFO] subscription-manager:22998:MainThread @dmiinfo.py:73 - Using dmidecode dump file: /dev/mem
2017-08-04 02:48:36,342 [INFO] subscription-manager:22998:MainThread @custom.py:85 - Loading custom facts from: /etc/rhsm/facts/custom.facts
2017-08-04 02:48:36,392 [INFO] subscription-manager:22998:MainThread @connection.py:552 - Response: status=200, request="GET /rhsm/users/admin/owners"
2017-08-04 02:48:36,428 [INFO] subscription-manager:22998:MainThread @connection.py:552 - Response: status=200, request="GET /rhsm/"
2017-08-04 02:48:36,475 [INFO] subscription-manager:22998:MainThread @connection.py:552 - Response: status=200, request="GET /rhsm/owners/Default_Organization/environments"
2017-08-04 02:48:38,056 [INFO] subscription-manager:22998:MainThread @connection.py:552 - Response: status=200, request="POST /rhsm/environments/1/consumers"
2017-08-04 02:48:38,059 [INFO] subscription-manager:22998:MainThread @managerlib.py:74 - Consumer created: {'consumer_name': u'kvm-02-guest10.rhts.eng.bos.redhat.com', 'uuid': 'b4c4a052-c1a4-4c15-873e-d186e1d55ab1'}
2017-08-04 02:48:38,060 [INFO] subscription-manager:22998:MainThread @connection.py:822 - Connection built: host=bkr-hv03-guest39.dsal.lab.eng.bos.redhat.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2017-08-04 02:48:38,095 [INFO] subscription-manager:22998:MainThread @connection.py:552 - Response: status=200, request="GET /rhsm/"
2017-08-04 02:48:38,187 [INFO] subscription-manager:22998:MainThread @connection.py:552 - Response: status=200, request="GET /rhsm/status"
2017-08-04 02:48:38,188 [INFO] subscription-manager:22998:MainThread @managercli.py:418 - Server Versions: {'rules-version': u'5.16', 'candlepin': u'2.2.0.95-Unknown', 'server-type': u'Red Hat Subscription Management'}
2017-08-04 02:48:38,225 [INFO] subscription-manager:22998:MainThread @connection.py:552 - Response: status=200, request="GET /rhsm/"
2017-08-04 02:48:39,491 [INFO] subscription-manager:22998:MainThread @connection.py:552 - Response: status=200, request="PUT /rhsm/consumers/b4c4a052-c1a4-4c15-873e-d186e1d55ab1/packages"
2017-08-04 02:48:39,727 [INFO] subscription-manager:22998:MainThread @connection.py:552 - Response: status=200, request="GET /rhsm/status"
2017-08-04 02:48:39,998 [INFO] subscription-manager:22998:MainThread @connection.py:552 - Response: status=200, request="GET /rhsm/consumers/b4c4a052-c1a4-4c15-873e-d186e1d55ab1/compliance"
2017-08-04 02:48:39,999 [INFO] subscription-manager:22998:MainThread @cert_sorter.py:204 - Product status: valid_products= partial_products= expired_products= unentitled_producs=69 future_products= valid_until=None
2017-08-04 02:48:40,044 [INFO] rhsmd:23004:MainThread @connection.py:822 - Connection built: host=bkr-hv03-guest39.dsal.lab.eng.bos.redhat.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2017-08-04 02:48:40,299 [INFO] rhsmd:23004:MainThread @connection.py:552 - Response: status=200, request="GET /rhsm/consumers/b4c4a052-c1a4-4c15-873e-d186e1d55ab1/compliance"
2017-08-04 02:48:40,300 [INFO] rhsmd:23004:MainThread @cert_sorter.py:204 - Product status: valid_products= partial_products= expired_products= unentitled_producs=69 future_products= valid_until=None
2017-08-04 02:48:44,251 [INFO] rhsmd:23055:MainThread @rhsmd:263 - rhsmd started
2017-08-04 02:48:44,296 [INFO] subscription-manager:23049:MainThread @managercli.py:518 - X-Correlation-ID: cb1f2668af584dfcb2b2d2176384431e
2017-08-04 02:48:44,296 [INFO] subscription-manager:23049:MainThread @managercli.py:407 - Client Versions: {'python-rhsm': '1.19.9-1.el7', 'subscription-manager': '1.19.21-1.el7'}
2017-08-04 02:48:44,297 [INFO] subscription-manager:23049:MainThread @connection.py:822 - Connection built: host=bkr-hv03-guest39.dsal.lab.eng.bos.redhat.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2017-08-04 02:48:44,297 [INFO] subscription-manager:23049:MainThread @connection.py:822 - Connection built: host=bkr-hv03-guest39.dsal.lab.eng.bos.redhat.com port=443 handler=/rhsm auth=none
2017-08-04 02:48:44,309 [INFO] subscription-manager:23049:MainThread @managercli.py:382 - Consumer Identity name=kvm-02-guest10.rhts.eng.bos.redhat.com uuid=b4c4a052-c1a4-4c15-873e-d186e1d55ab1
2017-08-04 02:48:44,388 [INFO] subscription-manager:23049:MainThread @connection.py:552 - Response: status=200, request="GET /rhsm/consumers/b4c4a052-c1a4-4c15-873e-d186e1d55ab1/owner"
2017-08-04 02:48:44,425 [INFO] subscription-manager:23049:MainThread @connection.py:552 - Response: status=200, request="GET /rhsm/"
2017-08-04 02:48:44,509 [INFO] subscription-manager:23049:MainThread @connection.py:552 - Response: status=200, request="GET /rhsm/consumers/b4c4a052-c1a4-4c15-873e-d186e1d55ab1"
2017-08-04 02:49:43,953 [INFO] subscription-manager:23063:MainThread @managercli.py:518 - X-Correlation-ID: d7c5f716b9cf4cb9a822de9f5a27eb01
2017-08-04 02:49:43,953 [INFO] subscription-manager:23063:MainThread @managercli.py:407 - Client Versions: {'python-rhsm': '1.19.9-1.el7', 'subscription-manager': '1.19.21-1.el7'}
2017-08-04 02:49:43,954 [INFO] subscription-manager:23063:MainThread @connection.py:822 - Connection built: host=bkr-hv03-guest39.dsal.lab.eng.bos.redhat.com port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2017-08-04 02:49:43,954 [INFO] subscription-manager:23063:MainThread @connection.py:822 - Connection built: host=bkr-hv03-guest39.dsal.lab.eng.bos.redhat.com port=443 handler=/rhsm auth=none
2017-08-04 02:49:43,965 [INFO] subscription-manager:23063:MainThread @managercli.py:382 - Consumer Identity name=kvm-02-guest10.rhts.eng.bos.redhat.com uuid=b4c4a052-c1a4-4c15-873e-d186e1d55ab1
2017-08-04 02:49:44,137 [INFO] subscription-manager:23063:MainThread @connection.py:552 - Response: status=410, request="GET /rhsm/consumers/b4c4a052-c1a4-4c15-873e-d186e1d55ab1/owner"
2017-08-04 02:49:44,137 [ERROR] subscription-manager:23063:MainThread @managercli.py:780 - Unit b4c4a052-c1a4-4c15-873e-d186e1d55ab1 has been deleted
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/subscription_manager/managercli.py", line 747, in _do_command
    owner = self.cp.getOwner(consumerid)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 1090, in getOwner
    return self.conn.request_get(method)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 646, in request_get
    return self._request("GET", method, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 672, in _request
    info=info, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 561, in _request
    self.validateResponse(result, request_type, handler)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 593, in validateResponse
    parsed['deletedId'])
GoneException: Unit b4c4a052-c1a4-4c15-873e-d186e1d55ab1 has been deleted
2017-08-04 02:49:44,138 [ERROR] subscription-manager:23063:MainThread @managercli.py:781 - Error: Unable to generate a new identity for the system: Unit b4c4a052-c1a4-4c15-873e-d186e1d55ab1 has been deleted

Comment 1 Kevin Howell 2017-08-04 13:58:55 UTC
qianzhan, 7.4 subscription-manager contains logic for maxSplayMinutes. I suspect this is the cause. With default settings in 7.4 (maxSplayMinutes=10), sleep 12 minutes should work. Alternatively, you can change maxSplayMinutes in your test. Please retest with maxSplayMinutes accounted for, and let us know the results.

Comment 2 qianzhan 2017-08-07 07:45:04 UTC
Hi, Kevin
The following are results you need:

1. Set maxSplayMinutes=0, the result is what we want:

1) Disable splay:
[root@bkr-hv03-guest32 ~]# subscription-manager  config | grep splay
   splay = 0

2) Register the rhel7.4 system against satellite6.1.12:
[root@bkr-hv03-guest32 ~]# subscription-manager  register
Registering to: hp-dl320egen8-01.rhts.eng.bos.redhat.com:443/rhsm
Username: admin
Password: 
The system has been registered with ID: 23fd76f4-fa03-4116-bf20-c7fa3f17c9f0 

3) Delete the system on satellite web UI:
[root@bkr-hv03-guest32 ~]# subscription-manager identity
Unit 23fd76f4-fa03-4116-bf20-c7fa3f17c9f0 has been deleted

4) Restart the rhsmcertd service and wait 2 minutes to check consumer folder and consumer.old folder:

[root@bkr-hv03-guest32 ~]# service rhsmcertd restart
Redirecting to /bin/systemctl restart rhsmcertd.service
[root@bkr-hv03-guest32 ~]# sleep 120
[root@bkr-hv03-guest32 ~]# ls /etc/pki/consumer
[root@bkr-hv03-guest32 ~]# ls /etc/pki/consumer.old/
cert.pem  key.pem


2. Use default maxSplayMinutes=10:

1) Set splay=1 in rhsm.conf:
[root@bkr-hv03-guest32 ~]# subscription-manager  config | grep splay
   splay = [1]

2) Register system against satellite6.1.12:
[root@bkr-hv03-guest32 ~]# subscription-manager  register
Registering to: hp-dl320egen8-01.rhts.eng.bos.redhat.com:443/rhsm
Username: admin   
Password: 
The system has been registered with ID: da12e861-f6fe-454d-b3a7-ddfadf71d52d 

3) Delete the system on satellite web UI:
[root@bkr-hv03-guest32 ~]# subscription-manager  identity
Unit da12e861-f6fe-454d-b3a7-ddfadf71d52d has been deleted

4) Restart the rhsmcertd service and wait 12 minutes to check consumer folder and consumer.old folder:
[root@bkr-hv03-guest32 ~]# service rhsmcertd restart
Redirecting to /bin/systemctl restart rhsmcertd.service

[root@bkr-hv03-guest32 ~]# sleep 720
[root@bkr-hv03-guest32 ~]# ls /etc/pki/consumer
[root@bkr-hv03-guest32 ~]# ls /etc/pki/consumer.old/
cert.pem  key.pem


Close the bug as WORKSFORME.