Bug 1566000

Try commenting out 

server=qemu:///system

and running again. It should mgirate the guests to the second instance. You should be able to remove the first one once that has happened.

It created the task conflict and it did not migrate any guests:

[0] 17:08 root satotest.brq ~ # virt-who -o -d
...
...
...

  }, 
    {
        "guestId": "ff86e5ac-37ef-4da4-b78e-df0e708431fa", 
        "state": 5, 
        "attributes": {
            "active": 0, 
            "virtWhoType": "libvirt"
        }
    }
]
2018-04-11 17:08:51,595 [rhsm.connection INFO] MainProcess(4697):MainThread @connection.py:_request:551 - Response: status=200, request="GET /rhsm/"
2018-04-11 17:08:51,596 [rhsm.connection DEBUG] MainProcess(4697):MainThread @connection.py:_load_supported_resources:838 - Server supports the following resources: {u'available_releases': u'/rhsm/consumers/:id/available_releases', u'status': u'/rhsm/status', u'guestids': u'/rhsm/consumers/:id/guestids', u'content_overrides': u'/rhsm/consumers/:id/content_overrides', u'environments': u'/rhsm/owners/:organization_id/environments', u'hypervisors': u'/rhsm/hypervisors', u'owner': u'/rhsm/consumers/:id/owner', u'certificates': u'/rhsm/consumers/:consumer_id/certificates', u'servicelevels': u'/rhsm/owners/:organization_id/servicelevels', u'serials': u'/rhsm/consumers/:id/certificates/serials', u'deleted_consumers': u'/rhsm/deleted_consumers', u'consumers': u'/rhsm/environments/:environment_id/consumers', u'entitlements': u'/rhsm/entitlements', u'profile': u'/rhsm/consumers/:id/profile', u'dry-run': u'/rhsm/consumers/:id/entitlements/dry-run', u'subscriptions': u'/rhsm/subscriptions', u'checkin': u'/rhsm/consumers/:id/checkin', u'deletionrecord': u'/rhsm/consumers/:id/deletionrecord', u'release': u'/rhsm/consumers/:id/release', u':jobId': u'/rhsm/jobs/:jobId', u'packages': u'/rhsm/consumers/:id/packages', u'owners': u'/rhsm/users/:login/owners', u'compliance': u'/rhsm/consumers/:id/compliance', u':owner': u'/rhsm/hypervisors/:owner', u'enabled_repos': u'/rhsm/systems/:id/enabled_repos', u'pools': u'/rhsm/pools', u'tracer': u'/rhsm/consumers/:id/tracer'}
2018-04-11 17:08:51,598 [rhsm.connection DEBUG] MainProcess(4697):MainThread @connection.py:_request:515 - Making request: PUT /rhsm/consumers/266121ea-1dc8-4219-ae49-facc1f44c752
2018-04-11 17:08:51,969 [virtwho.virt-who-config-1 DEBUG] Libvirtd-1(4706):MainThread @virt.py:run:400 - Virt backend 'virt-who-config-1' stopped after sending one report
2018-04-11 17:08:53,203 [rhsm.connection INFO] MainProcess(4697):MainThread @connection.py:_request:551 - Response: status=500, request="PUT /rhsm/consumers/266121ea-1dc8-4219-ae49-facc1f44c752"
2018-04-11 17:08:53,204 [virtwho.main ERROR] MainProcess(4697):MainThread @executor.py:send:161 - Error in communication with subscription manager:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/virtwho/executor.py", line 138, in send
    self._sendGuestList(report)
  File "/usr/lib/python2.7/site-packages/virtwho/executor.py", line 167, in _sendGuestList
    manager.sendVirtGuests(report, self.options)
  File "/usr/lib/python2.7/site-packages/virtwho/manager/subscriptionmanager/subscriptionmanager.py", line 158, in sendVirtGuests
    self.connection.updateConsumer(self.uuid(), guest_uuids=serialized_guests, hypervisor_id=report.hypervisor_id)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 1001, in updateConsumer
    ret = self.conn.request_put(method, params)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 654, in request_put
    return self._request("PUT", method, params, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 671, in _request
    info=info, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 560, in _request
    self.validateResponse(result, request_type, handler)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 609, in validateResponse
    raise RestlibException(response['status'], error_msg, response.get('headers'))
RestlibException: Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.

Required lock: update
Conflicts with tasks:
- https://provisioning.sysmgmt.lan/foreman_tasks/tasks/005ef543-d81f-41d0-8336-639b464a16fd
2018-04-11 17:08:53,224 [virtwho.main DEBUG] MainProcess(4697):MainThread @executor.py:send_report:109 - Report from "virt-who-config-1" failed to sent
2018-04-11 17:08:53,225 [virtwho.main DEBUG] MainProcess(4697):MainThread @__main__.py:main:23 - virt-who terminated
2018-04-11 17:08:53,225 [virtwho.main DEBUG] MainProcess(4697):MainThread @executor.py:terminate:308 - virt-who is shutting down

However removing virt-who-satotest.gsslab.brq.redhat.com-10 profile and run virt-who -od again helped.

Now there is no virt-who-satotest.gsslab.brq.redhat.com-10 and guests are reported under satotest.gsslab.brq2.redhat.com:

===
Name satotest.gsslab.brq2.redhat.com
UUID 266121ea-1dc8-4219-ae49-facc1f44c752 ---> Actual uuid from sub-man identity
Virtual Guests
    2 Content Hosts ---------------> we should see Content Hosts here
===


2018-04-11 17:16:13,685 [rhsm.connection DEBUG] MainProcess(5651):MainThread @connection.py:_request:515 - Making request: PUT /rhsm/consumers/266121ea-1dc8-4219-ae49-facc1f44c752
2018-04-11 17:16:14,349 [virtwho.virt-who-config-1 DEBUG] Libvirtd-1(5668):MainThread @virt.py:run:400 - Virt backend 'virt-who-config-1' stopped after sending one report
2018-04-11 17:16:14,626 [rhsm.connection INFO] MainProcess(5651):MainThread @connection.py:_request:551 - Response: status=200, request="PUT /rhsm/consumers/266121ea-1dc8-4219-ae49-facc1f44c752"
2018-04-11 17:16:14,627 [virtwho.main DEBUG] MainProcess(5651):MainThread @executor.py:send_report:102 - Report for config "virt-who-config-1" sent
2018-04-11 17:16:14,639 [virtwho.main DEBUG] MainProcess(5651):MainThread @__main__.py:main:23 - virt-who terminated
2018-04-11 17:16:14,639 [virtwho.main DEBUG] MainProcess(5651):MainThread @executor.py:terminate:308 - virt-who is shutting down

Well, obviously there had to be some change in the code, as this was working with server= line in Satellite 6.2 ....

This needs to go to the Sat6 team to determine the differences (6.2 vs. 6.3) that led to the change in behavior in Comment 3.

Moving to candlepin pile to get determination of behavior change.

Sat 6.2 is 0.9.54
Sat 6.3 is 2.1.14

Will, please let us know what you find wrt whether the issue is in Candlepin.

Would you please separate out the remaining issue as you have suggested. Thank you.

Removing need info. The fix for this will be part of a larger solution.

Can you please add related bugs to "See Also"?

Can you set the fixed in to the package and version that fixes this BZ?

The fix for this issue requires a Candlepin version of 2.5.5+ and a virt-who version of 0.22.2+ or 0.23.0+.

It will not fix old entries but will server to avoid the scenario to happen going forward. It will take care of any of the fabrics where the consumer checks in and the consumer is also a hypervisor that virt-who is reporting on. Setting up this scenario will be the best way to see that the fix is complete.

Mike, based on Eko's comment, I'd like to tentatively dual propose this to 6.4.z. 
I'll keep it out of 6.3.z, as I don't see us bringing this that far back. 
If you disagree, feel free to remove the flag.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222