Bug 1566000 - KVM hypervisor profile does not contain guests running on it in the webui and creates duplicate profile with virt-who-* prefix [NEEDINFO]
Summary: KVM hypervisor profile does not contain guests running on it in the webui and...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Candlepin
Version: 6.3.0
Hardware: x86_64
OS: Linux
high
urgent
Target Milestone: 6.5.0
Assignee: satellite6-bugs
QA Contact: jcallaha
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-11 10:12 UTC by Michal Dekan
Modified: 2019-11-05 22:27 UTC (History)
12 users (show)

Fixed In Version: candlepin-2.3.5
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-14 12:37:00 UTC
Target Upstream Version:
jcallaha: needinfo? (mmccune)


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github candlepin virt-who pull 149 None closed ENT-554 Host reports for libvirt and rhevm include the system hardware uuid 2020-06-04 12:36:33 UTC
Red Hat Bugzilla 1314902 medium CLOSED 2 virt-who reporting on the same hypervisor using local libvirt and remote libvirt methods creates duplicate systems 2020-10-14 00:28:05 UTC
Red Hat Bugzilla 1410601 medium CLOSED [RFE] virt-who need to make sure there is only one entry in satellite content host for the same hypervisor when configu... 2020-10-14 00:28:05 UTC
Red Hat Bugzilla 1576110 high CLOSED Org admins not able to view system 2020-10-14 00:28:05 UTC
Red Hat Product Errata RHSA-2019:1222 None None None 2019-05-14 12:37:12 UTC

Internal Links: 1314902 1410601 1576110

Description Michal Dekan 2018-04-11 10:12:58 UTC
Description of problem:


Version-Release number of selected component (if applicable):

Satellite 6.3 + KVM hypervisor with guests + virt-who (tested with virt-who-0.19-2.el7sat.noarch.rpm and virt-who-0.21.5-1.el7.noarch.rpm)

How reproducible:

Register KVM hypervisor with few guests against Satellite 6.3.0

Steps to Reproduce:
1. KVM hypervisor with few VM guests running on it.
2. virt-who running on that KVM hypervisor in libvirt mode.
3. virt-who reporting guest-host mapping to Satellite 6.3.

Actual results:

2 hypervisors profiles are created on Content Hosts page:

===
Name virt-who-satotest.gsslab.brq.redhat.com-10
UUID 09e46059-da11-4fdf-ae3e-8d33884b55ed
Virtual Guests
    2 Content Hosts 
Registered Through
    Unknown
Subscription Status
    Unsubscribed hypervisor 
===

===
Name satotest.gsslab.brq2.redhat.com
UUID 266121ea-1dc8-4219-ae49-facc1f44c752 ---> Actual uuid from sub-man identity
Virtual Guests
    0 Content Hosts ---------------> we should see Content Hosts here
===

[0] 09:47 root satotest.brq ~ # subscription-manager identity
system identity: 266121ea-1dc8-4219-ae49-facc1f44c752
name: satotest.gsslab.brq.redhat.com
org name: mdekan-org1
org ID: mdekan-org1
environment name: Library

This makes impossible to subscribe hypervisor + its guests with "RHEL 1-2 Sockets, Unlimited guests with Smart Management" subscription.

Expected results:

No hypervisor profile with prefix virt-who- created. There was no such profile in Satellite 6.2 for KVM hypervisor.

Comment 2 William Poteat 2018-04-11 14:14:11 UTC
Try commenting out 

server=qemu:///system

and running again. It should mgirate the guests to the second instance. You should be able to remove the first one once that has happened.

Comment 3 Michal Dekan 2018-04-11 15:25:54 UTC
It created the task conflict and it did not migrate any guests:

[0] 17:08 root satotest.brq ~ # virt-who -o -d
...
...
...

  }, 
    {
        "guestId": "ff86e5ac-37ef-4da4-b78e-df0e708431fa", 
        "state": 5, 
        "attributes": {
            "active": 0, 
            "virtWhoType": "libvirt"
        }
    }
]
2018-04-11 17:08:51,595 [rhsm.connection INFO] MainProcess(4697):MainThread @connection.py:_request:551 - Response: status=200, request="GET /rhsm/"
2018-04-11 17:08:51,596 [rhsm.connection DEBUG] MainProcess(4697):MainThread @connection.py:_load_supported_resources:838 - Server supports the following resources: {u'available_releases': u'/rhsm/consumers/:id/available_releases', u'status': u'/rhsm/status', u'guestids': u'/rhsm/consumers/:id/guestids', u'content_overrides': u'/rhsm/consumers/:id/content_overrides', u'environments': u'/rhsm/owners/:organization_id/environments', u'hypervisors': u'/rhsm/hypervisors', u'owner': u'/rhsm/consumers/:id/owner', u'certificates': u'/rhsm/consumers/:consumer_id/certificates', u'servicelevels': u'/rhsm/owners/:organization_id/servicelevels', u'serials': u'/rhsm/consumers/:id/certificates/serials', u'deleted_consumers': u'/rhsm/deleted_consumers', u'consumers': u'/rhsm/environments/:environment_id/consumers', u'entitlements': u'/rhsm/entitlements', u'profile': u'/rhsm/consumers/:id/profile', u'dry-run': u'/rhsm/consumers/:id/entitlements/dry-run', u'subscriptions': u'/rhsm/subscriptions', u'checkin': u'/rhsm/consumers/:id/checkin', u'deletionrecord': u'/rhsm/consumers/:id/deletionrecord', u'release': u'/rhsm/consumers/:id/release', u':jobId': u'/rhsm/jobs/:jobId', u'packages': u'/rhsm/consumers/:id/packages', u'owners': u'/rhsm/users/:login/owners', u'compliance': u'/rhsm/consumers/:id/compliance', u':owner': u'/rhsm/hypervisors/:owner', u'enabled_repos': u'/rhsm/systems/:id/enabled_repos', u'pools': u'/rhsm/pools', u'tracer': u'/rhsm/consumers/:id/tracer'}
2018-04-11 17:08:51,598 [rhsm.connection DEBUG] MainProcess(4697):MainThread @connection.py:_request:515 - Making request: PUT /rhsm/consumers/266121ea-1dc8-4219-ae49-facc1f44c752
2018-04-11 17:08:51,969 [virtwho.virt-who-config-1 DEBUG] Libvirtd-1(4706):MainThread @virt.py:run:400 - Virt backend 'virt-who-config-1' stopped after sending one report
2018-04-11 17:08:53,203 [rhsm.connection INFO] MainProcess(4697):MainThread @connection.py:_request:551 - Response: status=500, request="PUT /rhsm/consumers/266121ea-1dc8-4219-ae49-facc1f44c752"
2018-04-11 17:08:53,204 [virtwho.main ERROR] MainProcess(4697):MainThread @executor.py:send:161 - Error in communication with subscription manager:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/virtwho/executor.py", line 138, in send
    self._sendGuestList(report)
  File "/usr/lib/python2.7/site-packages/virtwho/executor.py", line 167, in _sendGuestList
    manager.sendVirtGuests(report, self.options)
  File "/usr/lib/python2.7/site-packages/virtwho/manager/subscriptionmanager/subscriptionmanager.py", line 158, in sendVirtGuests
    self.connection.updateConsumer(self.uuid(), guest_uuids=serialized_guests, hypervisor_id=report.hypervisor_id)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 1001, in updateConsumer
    ret = self.conn.request_put(method, params)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 654, in request_put
    return self._request("PUT", method, params, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 671, in _request
    info=info, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 560, in _request
    self.validateResponse(result, request_type, handler)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 609, in validateResponse
    raise RestlibException(response['status'], error_msg, response.get('headers'))
RestlibException: Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.

Required lock: update
Conflicts with tasks:
- https://provisioning.sysmgmt.lan/foreman_tasks/tasks/005ef543-d81f-41d0-8336-639b464a16fd
2018-04-11 17:08:53,224 [virtwho.main DEBUG] MainProcess(4697):MainThread @executor.py:send_report:109 - Report from "virt-who-config-1" failed to sent
2018-04-11 17:08:53,225 [virtwho.main DEBUG] MainProcess(4697):MainThread @__main__.py:main:23 - virt-who terminated
2018-04-11 17:08:53,225 [virtwho.main DEBUG] MainProcess(4697):MainThread @executor.py:terminate:308 - virt-who is shutting down

However removing virt-who-satotest.gsslab.brq.redhat.com-10 profile and run virt-who -od again helped.

Now there is no virt-who-satotest.gsslab.brq.redhat.com-10 and guests are reported under satotest.gsslab.brq2.redhat.com:

===
Name satotest.gsslab.brq2.redhat.com
UUID 266121ea-1dc8-4219-ae49-facc1f44c752 ---> Actual uuid from sub-man identity
Virtual Guests
    2 Content Hosts ---------------> we should see Content Hosts here
===


2018-04-11 17:16:13,685 [rhsm.connection DEBUG] MainProcess(5651):MainThread @connection.py:_request:515 - Making request: PUT /rhsm/consumers/266121ea-1dc8-4219-ae49-facc1f44c752
2018-04-11 17:16:14,349 [virtwho.virt-who-config-1 DEBUG] Libvirtd-1(5668):MainThread @virt.py:run:400 - Virt backend 'virt-who-config-1' stopped after sending one report
2018-04-11 17:16:14,626 [rhsm.connection INFO] MainProcess(5651):MainThread @connection.py:_request:551 - Response: status=200, request="PUT /rhsm/consumers/266121ea-1dc8-4219-ae49-facc1f44c752"
2018-04-11 17:16:14,627 [virtwho.main DEBUG] MainProcess(5651):MainThread @executor.py:send_report:102 - Report for config "virt-who-config-1" sent
2018-04-11 17:16:14,639 [virtwho.main DEBUG] MainProcess(5651):MainThread @__main__.py:main:23 - virt-who terminated
2018-04-11 17:16:14,639 [virtwho.main DEBUG] MainProcess(5651):MainThread @executor.py:terminate:308 - virt-who is shutting down

Well, obviously there had to be some change in the code, as this was working with server= line in Satellite 6.2 ....

Comment 5 William Poteat 2018-04-13 15:26:44 UTC
This needs to go to the Sat6 team to determine the differences (6.2 vs. 6.3) that led to the change in behavior in Comment 3.

Comment 10 William Poteat 2018-04-19 19:11:17 UTC
Moving to candlepin pile to get determination of behavior change.

Sat 6.2 is 0.9.54
Sat 6.3 is 2.1.14

Comment 11 Kevin Howell 2018-04-23 14:46:27 UTC
Will, please let us know what you find wrt whether the issue is in Candlepin.

Comment 12 William Poteat 2018-04-26 17:36:20 UTC
Would you please separate out the remaining issue as you have suggested. Thank you.

Comment 14 William Poteat 2018-05-30 14:53:02 UTC
Removing need info. The fix for this will be part of a larger solution.

Comment 15 Kevin Howell 2018-06-04 14:38:00 UTC
Can you please add related bugs to "See Also"?

Comment 18 Eric Helms 2018-12-10 16:41:07 UTC
Can you set the fixed in to the package and version that fixes this BZ?

Comment 21 William Poteat 2019-02-26 17:04:21 UTC
The fix for this issue requires a Candlepin version of 2.5.5+ and a virt-who version of 0.22.2+ or 0.23.0+.

It will not fix old entries but will server to avoid the scenario to happen going forward. It will take care of any of the fabrics where the consumer checks in and the consumer is also a hypervisor that virt-who is reporting on. Setting up this scenario will be the best way to see that the fix is complete.

Comment 23 jcallaha 2019-02-28 14:08:53 UTC
Mike, based on Eko's comment, I'd like to tentatively dual propose this to 6.4.z. 
I'll keep it out of 6.3.z, as I don't see us bringing this that far back. 
If you disagree, feel free to remove the flag.

Comment 26 errata-xmlrpc 2019-05-14 12:37:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222


Note You need to log in before you can comment on or make changes to this bug.