Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1220620 - Failed to migrate guest as required lock is already taken by "Update"
Summary: Failed to migrate guest as required lock is already taken by "Update"
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Uncategorized
Version: Unspecified
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: 6.4.0
Assignee: satellite6-bugs
QA Contact: Perry Gagne
URL:
Whiteboard:
: 1232078 (view as bug list)
Depends On: 1592281
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-05-12 02:58 UTC by Liushihui
Modified: 2021-06-10 10:54 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-16 18:57:06 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
locked task (98.24 KB, image/png)
2015-05-12 02:58 UTC, Liushihui 		no flags Details
Lock log on Satellite 6.1.0-20150818.1 (3.84 KB, application/x-gzip)
2015-08-20 02:44 UTC, Liushihui 		no flags Details
Lock log on Satellite-6.1.0-RHEL-6-20160321.0 (1.03 MB, application/x-gzip)
2016-04-05 06:33 UTC, Liushihui 		no flags Details
View All

Description Liushihui 2015-05-12 02:58:26 UTC 
Description of problem:
When migrate guest from orig host to dest host, it will show "RestlibException: Required lock is already taken by other running tasks" in the log file, this task has conflicted with "update" task. Failed to migrate guest.

Version-Release number of selected component (if applicable):
virt-who-0.12-7.el6.noarch
subscription-manager-1.14.6-1.el6.x86_64
python-rhsm-1.14.2-1.el6.x86_64
candlepin-0.9.49.0-1.el7.noarch
katello-2.2.0.8-1.el7sat.noarch
foreman-1.7.2.19-1.el7sat.noarch
Satellite-6.1.0-RHEL-7-20150506.0

How reproducible:
50%

Steps to Reproduce:
Precondition: Two hosts.Orig host with guest1.
1.Register two hosts and guest1 to satellite
2.Start virt-who service, make it run at libvirt mode
3.Migrate guest1 from orig host to dest host
[root@hp-z220-05 libvirt-test-API]# virsh migrate --live 6.5_Server_x86_64 qemu+ssh://10.66.100.111/system
root.100.111's password: 
4.During migration, check the rhsm log in the dest host.
[root@hp-z220-06 ~]# tail -f /var/log/rhsm/rhsm.log
2015-05-12 09:54:46,087 [DEBUG] rhsmd:18238 @identity.py:131 - Loading consumer info from identity certificates.
2015-05-12 09:54:46,092 [DEBUG] rhsmd:18238 @rhsmd:148 - D-Bus signal com.redhat.SubscriptionManager.EntitlementStatus.entitlement_status_changed emitted
2015-05-12 10:36:17,658 [DEBUG]  @libvirtd.py:200 - Libvirt domains found: []
2015-05-12 10:36:17,673 [DEBUG]  @subscriptionmanager.py:75 - Authenticating with certificate: /etc/pki/consumer/cert.pem
2015-05-12 10:36:21,818 [INFO]  @subscriptionmanager.py:116 - Sending domain info: []
2015-05-12 10:36:24,724 [DEBUG]  @libvirtd.py:190 - Virtual machine found: 6.5_Server_x86_64: 3257cbbc-9396-1e11-0868-e2302568f2cc
2015-05-12 10:36:24,727 [DEBUG]  @libvirtd.py:200 - Libvirt domains found: [{'guestId': '3257cbbc-9396-1e11-0868-e2302568f2cc', 'attributes': {'active': 1, 'virtWhoType': 'libvirt', 'hypervisorType': 'QEMU'}, 'state': 3}]
2015-05-12 10:36:27,677 [ERROR]  @virtwho.py:115 - Error in communication with subscription manager:
Traceback (most recent call last):
  File "/usr/share/virt-who/virtwho.py", line 102, in send
    self._sendGuestList(report)
  File "/usr/share/virt-who/virtwho.py", line 121, in _sendGuestList
    manager.sendVirtGuests(report.guests)
  File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 119, in sendVirtGuests
    self.connection.updateConsumer(self.uuid(), guest_uuids=domains)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 858, in updateConsumer
    ret = self.conn.request_put(method, params)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 611, in request_put
    return self._request("PUT", method, params)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 530, in _request
    self.validateResponse(result, request_type, handler)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 572, in validateResponse
    raise RestlibException(response['status'], error_msg)
RestlibException: Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.

Required lock: read
Conflicts with tasks:
- https://cisco-c210-01.rhts.eng.bos.redhat.com/foreman_tasks/tasks/b2a2dfc2-8582-4208-bca4-f340a544828d
5.Open the conflict task, it will error on "update" task, please see the screenshot in attachment.
 Id: b2a2dfc2-8582-4208-bca4-f340a544828d
Label: Actions::Katello::System::Update
Name: Update
Owner:
Started at: 2015-05-11 05:45:25 UTC
Ended at:
State: paused
Result: error
Params: system 'hp-z220-06.qe.lab.eng.nay.redhat.com'; organization 'Default Organization'

Actual results:
Failed to migrate guest from orig host to dest host. it will show "Required lock is already taken by other running tasks" on dest host.

Expected results:
It shouldn't show conflict task when do migration. Migration successfully, 

Additional info:
Comment 1 Liushihui 2015-05-12 02:58:56 UTC 
Created attachment 1024444 [details]
locked task
Comment 2 RHEL Program Management 2015-05-12 03:02:47 UTC 
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.
Comment 5 Radek Novacek 2015-05-14 05:40:59 UTC 
Mike,

why did you move this bug to virt-who? It looks more like server side issue from my point of view. Is virt-who doing something wrong?
Comment 6 Liushihui 2015-05-21 03:59:25 UTC 
When use virt-who-0.12-8.el6.noarch test Satellite-6.1.0-RHEL-7-20150513.0, when migrate back guest from dest host to orig host. it will not only lead to conflict task but also generate some "RestlibException" when communicate with subscription-manager. please see the detail as the following:
2015-05-21 11:15:42,420 [ERROR]  @virtwho.py:122 - Error in communication with subscription manager:
Traceback (most recent call last):
  File "/usr/share/virt-who/virtwho.py", line 102, in send
    self._sendGuestList(report)
  File "/usr/share/virt-who/virtwho.py", line 128, in _sendGuestList
    manager.sendVirtGuests(report.guests)
  File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 119, in sendVirtGuests
    self.connection.updateConsumer(self.uuid(), guest_uuids=domains)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 858, in updateConsumer
    ret = self.conn.request_put(method, params)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 611, in request_put
    return self._request("PUT", method, params)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 530, in _request
    self.validateResponse(result, request_type, handler)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 572, in validateResponse
    raise RestlibException(response['status'], error_msg)
RestlibException: Task e35de971-2e31-4e8b-907c-7206d6b9d426: Katello::Errors::CandlepinError: Runtime Error No entitlements for products: [] at org.candlepin.policy.js.autobind.AutobindRules.selectBestPools:79
Comment 7 Radek Novacek 2015-05-28 06:38:02 UTC 
Looks like some server side issue:

RestlibException: Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.

Moving back to candlepin.
Comment 8 RHEL Program Management 2015-05-28 06:52:22 UTC 
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.
Comment 10 Liushihui 2015-06-16 03:31:12 UTC 
*** Bug 1232078 has been marked as a duplicate of this bug. ***
Comment 11 Liushihui 2015-06-19 03:09:15 UTC 
When occurred this problem, virt-who will failed to send host/guest mapping to Satellite until the conflict task is resumed or destroyed.
Comment 12 Devan Goodwin 2015-08-13 12:25:40 UTC 
Original report references a lock that appears to be from foreman, don't think this one is candlepin related. 

The error in comment #6 is candlepin related and just a normal situation being incorrectly thrown as an error. This should actually be fixed aleady be fixed in 6.1.

But key issue looks to be foreman tasks of some kind.

Apologies for the component bouncing, and the fact I don't know what component this should be so setting to uncategorized.
Comment 13 Liushihui 2015-08-20 02:44:03 UTC 
It also reproduced on Satellite 6.1.0-20150818.1. please see the related log in the attachment lock.tar.gz
Comment 14 Liushihui 2015-08-20 02:44:49 UTC 
Created attachment 1065069 [details]
Lock log on Satellite 6.1.0-20150818.1
Comment 16 Liushihui 2016-04-05 06:32:54 UTC 
It still exist on Satellite-6.1.0-RHEL-6-20160321.0, When it occurred this issue, it will failed to list available pool. please see the candlepin log, foreman log, rhsm log in attachment "Lock log on Satellite-6.1.0-RHEL-6-20160321.0 .
# subscription-manager list --available
Task 64e2754b-6cd6-4f24-b246-0836f5067152: Katello::Errors::CandlepinError: Runtime Error ERROR: value too long for type character varying(255) at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,157
# tail -f /var/log/rhsm/rhsm.log
2016-04-04 23:40:30,376 [virtwho.main ERROR] MainProcess(8953):MainThread @virtwho.py:send:216 - Error in communication with subscription manager:
Traceback (most recent call last):
  File "/usr/share/virt-who/virtwho.py", line 197, in send
    self._sendGuestList(report)
  File "/usr/share/virt-who/virtwho.py", line 222, in _sendGuestList
    manager.sendVirtGuests(report, self.options)
  File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 152, in sendVirtGuests
    self.connection.updateConsumer(self.uuid(), guest_uuids=serialized_guests)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 995, in updateConsumer
    ret = self.conn.request_put(method, params)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 690, in request_put
    return self._request("PUT", method, params)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 598, in _request
    self.validateResponse(result, request_type, handler)
  File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 648, in validateResponse
    raise RestlibException(response['status'], error_msg, response.get('headers'))
RestlibException: Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.

Required lock: read
Conflicts with tasks:
- https://satellite-b0a66026-99ec-4c05-a983-5704d9e04e89.novalocal/foreman_tasks/tasks/64e2754b-6cd6-4f24-b246-0836f5067152

Reproduced version:
candlepin-0.9.49.12-1.el7.noarch
katello-2.2.0.19-1.el7sat.noarch
foreman-1.7.2.55-1.el7sat.noarch
subscription-manager-1.16.8-8.el6.x86_64
python-rhsm-1.16.6-1.el6.x86_64
Comment 17 Liushihui 2016-04-05 06:33:31 UTC 
Created attachment 1143676 [details]
Lock log on Satellite-6.1.0-RHEL-6-20160321.0
Comment 19 Bryan Kearney 2017-01-13 20:44:11 UTC 
This is an older bug which I do not envision being addressed in the near term. I am closing this out. If you believe doing so is an issue, please feel free to re-open and provide additional business information. Thank you.
Comment 26 Bryan Kearney 2018-10-16 18:57:06 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2927
Note You need to log in before you can comment on or make changes to this bug.