Bug 1566000

Summary: KVM hypervisor profile does not contain guests running on it in the webui and creates duplicate profile with virt-who-* prefix
Product: Red Hat Satellite Reporter: Michal Dekan <mdekan>
Component: CandlepinAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: jcallaha
Severity: urgent Docs Contact:
Priority: high    
Version: 6.3.0CC: bhoefer, egolov, ehelms, fgarciad, hsun, jbhatia, khowell, ktordeur, mdekan, mmccune, rproffit, wpoteat
Target Milestone: 6.5.0Keywords: Triaged
Target Release: UnusedFlags: jcallaha: needinfo? (mmccune)
Hardware: x86_64   
OS: Linux   
Fixed In Version: candlepin-2.3.5 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-14 12:37:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Michal Dekan 2018-04-11 10:12:58 UTC
Description of problem:

Version-Release number of selected component (if applicable):

Satellite 6.3 + KVM hypervisor with guests + virt-who (tested with virt-who-0.19-2.el7sat.noarch.rpm and virt-who-0.21.5-1.el7.noarch.rpm)

How reproducible:

Register KVM hypervisor with few guests against Satellite 6.3.0

Steps to Reproduce:
1. KVM hypervisor with few VM guests running on it.
2. virt-who running on that KVM hypervisor in libvirt mode.
3. virt-who reporting guest-host mapping to Satellite 6.3.

Actual results:

2 hypervisors profiles are created on Content Hosts page:

Name virt-who-satotest.gsslab.brq.redhat.com-10
UUID 09e46059-da11-4fdf-ae3e-8d33884b55ed
Virtual Guests
    2 Content Hosts 
Registered Through
Subscription Status
    Unsubscribed hypervisor 

Name satotest.gsslab.brq2.redhat.com
UUID 266121ea-1dc8-4219-ae49-facc1f44c752 ---> Actual uuid from sub-man identity
Virtual Guests
    0 Content Hosts ---------------> we should see Content Hosts here

[0] 09:47 root satotest.brq ~ # subscription-manager identity
system identity: 266121ea-1dc8-4219-ae49-facc1f44c752
name: satotest.gsslab.brq.redhat.com
org name: mdekan-org1
org ID: mdekan-org1
environment name: Library

This makes impossible to subscribe hypervisor + its guests with "RHEL 1-2 Sockets, Unlimited guests with Smart Management" subscription.

Expected results:

No hypervisor profile with prefix virt-who- created. There was no such profile in Satellite 6.2 for KVM hypervisor.

Comment 2 William Poteat 2018-04-11 14:14:11 UTC
Try commenting out 


and running again. It should mgirate the guests to the second instance. You should be able to remove the first one once that has happened.

Comment 3 Michal Dekan 2018-04-11 15:25:54 UTC
It created the task conflict and it did not migrate any guests:

[0] 17:08 root satotest.brq ~ # virt-who -o -d

        "guestId": "ff86e5ac-37ef-4da4-b78e-df0e708431fa", 
        "state": 5, 
        "attributes": {
            "active": 0, 
            "virtWhoType": "libvirt"
2018-04-11 17:08:51,595 [rhsm.connection INFO] MainProcess(4697):MainThread @connection.py:_request:551 - Response: status=200, request="GET /rhsm/"
2018-04-11 17:08:51,596 [rhsm.connection DEBUG] MainProcess(4697):MainThread @connection.py:_load_supported_resources:838 - Server supports the following resources: {u'available_releases': u'/rhsm/consumers/:id/available_releases', u'status': u'/rhsm/status', u'guestids': u'/rhsm/consumers/:id/guestids', u'content_overrides': u'/rhsm/consumers/:id/content_overrides', u'environments': u'/rhsm/owners/:organization_id/environments', u'hypervisors': u'/rhsm/hypervisors', u'owner': u'/rhsm/consumers/:id/owner', u'certificates': u'/rhsm/consumers/:consumer_id/certificates', u'servicelevels': u'/rhsm/owners/:organization_id/servicelevels', u'serials': u'/rhsm/consumers/:id/certificates/serials', u'deleted_consumers': u'/rhsm/deleted_consumers', u'consumers': u'/rhsm/environments/:environment_id/consumers', u'entitlements': u'/rhsm/entitlements', u'profile': u'/rhsm/consumers/:id/profile', u'dry-run': u'/rhsm/consumers/:id/entitlements/dry-run', u'subscriptions': u'/rhsm/subscriptions', u'checkin': u'/rhsm/consumers/:id/checkin', u'deletionrecord': u'/rhsm/consumers/:id/deletionrecord', u'release': u'/rhsm/consumers/:id/release', u':jobId': u'/rhsm/jobs/:jobId', u'packages': u'/rhsm/consumers/:id/packages', u'owners': u'/rhsm/users/:login/owners', u'compliance': u'/rhsm/consumers/:id/compliance', u':owner': u'/rhsm/hypervisors/:owner', u'enabled_repos': u'/rhsm/systems/:id/enabled_repos', u'pools': u'/rhsm/pools', u'tracer': u'/rhsm/consumers/:id/tracer'}
2018-04-11 17:08:51,598 [rhsm.connection DEBUG] MainProcess(4697):MainThread @connection.py:_request:515 - Making request: PUT /rhsm/consumers/266121ea-1dc8-4219-ae49-facc1f44c752
2018-04-11 17:08:51,969 [virtwho.virt-who-config-1 DEBUG] Libvirtd-1(4706):MainThread @virt.py:run:400 - Virt backend 'virt-who-config-1' stopped after sending one report
2018-04-11 17:08:53,203 [rhsm.connection INFO] MainProcess(4697):MainThread @connection.py:_request:551 - Response: status=500, request="PUT /rhsm/consumers/266121ea-1dc8-4219-ae49-facc1f44c752"
2018-04-11 17:08:53,204 [virtwho.main ERROR] MainProcess(4697):MainThread @executor.py:send:161 - Error in communication with subscription manager:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/virtwho/executor.py", line 138, in send
  File "/usr/lib/python2.7/site-packages/virtwho/executor.py", line 167, in _sendGuestList
    manager.sendVirtGuests(report, self.options)
  File "/usr/lib/python2.7/site-packages/virtwho/manager/subscriptionmanager/subscriptionmanager.py", line 158, in sendVirtGuests
    self.connection.updateConsumer(self.uuid(), guest_uuids=serialized_guests, hypervisor_id=report.hypervisor_id)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 1001, in updateConsumer
    ret = self.conn.request_put(method, params)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 654, in request_put
    return self._request("PUT", method, params, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 671, in _request
    info=info, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 560, in _request
    self.validateResponse(result, request_type, handler)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 609, in validateResponse
    raise RestlibException(response['status'], error_msg, response.get('headers'))
RestlibException: Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.

Required lock: update
Conflicts with tasks:
- https://provisioning.sysmgmt.lan/foreman_tasks/tasks/005ef543-d81f-41d0-8336-639b464a16fd
2018-04-11 17:08:53,224 [virtwho.main DEBUG] MainProcess(4697):MainThread @executor.py:send_report:109 - Report from "virt-who-config-1" failed to sent
2018-04-11 17:08:53,225 [virtwho.main DEBUG] MainProcess(4697):MainThread @__main__.py:main:23 - virt-who terminated
2018-04-11 17:08:53,225 [virtwho.main DEBUG] MainProcess(4697):MainThread @executor.py:terminate:308 - virt-who is shutting down

However removing virt-who-satotest.gsslab.brq.redhat.com-10 profile and run virt-who -od again helped.

Now there is no virt-who-satotest.gsslab.brq.redhat.com-10 and guests are reported under satotest.gsslab.brq2.redhat.com:

Name satotest.gsslab.brq2.redhat.com
UUID 266121ea-1dc8-4219-ae49-facc1f44c752 ---> Actual uuid from sub-man identity
Virtual Guests
    2 Content Hosts ---------------> we should see Content Hosts here

2018-04-11 17:16:13,685 [rhsm.connection DEBUG] MainProcess(5651):MainThread @connection.py:_request:515 - Making request: PUT /rhsm/consumers/266121ea-1dc8-4219-ae49-facc1f44c752
2018-04-11 17:16:14,349 [virtwho.virt-who-config-1 DEBUG] Libvirtd-1(5668):MainThread @virt.py:run:400 - Virt backend 'virt-who-config-1' stopped after sending one report
2018-04-11 17:16:14,626 [rhsm.connection INFO] MainProcess(5651):MainThread @connection.py:_request:551 - Response: status=200, request="PUT /rhsm/consumers/266121ea-1dc8-4219-ae49-facc1f44c752"
2018-04-11 17:16:14,627 [virtwho.main DEBUG] MainProcess(5651):MainThread @executor.py:send_report:102 - Report for config "virt-who-config-1" sent
2018-04-11 17:16:14,639 [virtwho.main DEBUG] MainProcess(5651):MainThread @__main__.py:main:23 - virt-who terminated
2018-04-11 17:16:14,639 [virtwho.main DEBUG] MainProcess(5651):MainThread @executor.py:terminate:308 - virt-who is shutting down

Well, obviously there had to be some change in the code, as this was working with server= line in Satellite 6.2 ....

Comment 5 William Poteat 2018-04-13 15:26:44 UTC
This needs to go to the Sat6 team to determine the differences (6.2 vs. 6.3) that led to the change in behavior in Comment 3.

Comment 10 William Poteat 2018-04-19 19:11:17 UTC
Moving to candlepin pile to get determination of behavior change.

Sat 6.2 is 0.9.54
Sat 6.3 is 2.1.14

Comment 11 Kevin Howell 2018-04-23 14:46:27 UTC
Will, please let us know what you find wrt whether the issue is in Candlepin.

Comment 12 William Poteat 2018-04-26 17:36:20 UTC
Would you please separate out the remaining issue as you have suggested. Thank you.

Comment 14 William Poteat 2018-05-30 14:53:02 UTC
Removing need info. The fix for this will be part of a larger solution.

Comment 15 Kevin Howell 2018-06-04 14:38:00 UTC
Can you please add related bugs to "See Also"?

Comment 18 Eric Helms 2018-12-10 16:41:07 UTC
Can you set the fixed in to the package and version that fixes this BZ?

Comment 21 William Poteat 2019-02-26 17:04:21 UTC
The fix for this issue requires a Candlepin version of 2.5.5+ and a virt-who version of 0.22.2+ or 0.23.0+.

It will not fix old entries but will server to avoid the scenario to happen going forward. It will take care of any of the fabrics where the consumer checks in and the consumer is also a hypervisor that virt-who is reporting on. Setting up this scenario will be the best way to see that the fix is complete.

Comment 23 jcallaha 2019-02-28 14:08:53 UTC
Mike, based on Eko's comment, I'd like to tentatively dual propose this to 6.4.z. 
I'll keep it out of 6.3.z, as I don't see us bringing this that far back. 
If you disagree, feel free to remove the flag.

Comment 26 errata-xmlrpc 2019-05-14 12:37:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.