Bug 1220620
| Summary: | Failed to migrate guest as required lock is already taken by "Update" | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Liushihui <shihliu> | ||||||||
| Component: | Uncategorized | Assignee: | satellite6-bugs <satellite6-bugs> | ||||||||
| Status: | CLOSED ERRATA | QA Contact: | Perry Gagne <pgagne> | ||||||||
| Severity: | high | Docs Contact: | |||||||||
| Priority: | unspecified | ||||||||||
| Version: | Unspecified | CC: | bbuckingham, bkearney, cwelton, dgoodwin, ldai, mmccune, sgao, wpinheir | ||||||||
| Target Milestone: | 6.4.0 | Keywords: | Reopened, Triaged | ||||||||
| Target Release: | Unused | ||||||||||
| Hardware: | x86_64 | ||||||||||
| OS: | Linux | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2018-10-16 18:57:06 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Bug Depends On: | 1592281 | ||||||||||
| Bug Blocks: | |||||||||||
| Attachments: |
|
||||||||||
Created attachment 1024444 [details]
locked task
Since this issue was entered in Red Hat Bugzilla, the release flag has been set to ? to ensure that it is properly evaluated for this release. Mike, why did you move this bug to virt-who? It looks more like server side issue from my point of view. Is virt-who doing something wrong? When use virt-who-0.12-8.el6.noarch test Satellite-6.1.0-RHEL-7-20150513.0, when migrate back guest from dest host to orig host. it will not only lead to conflict task but also generate some "RestlibException" when communicate with subscription-manager. please see the detail as the following:
2015-05-21 11:15:42,420 [ERROR] @virtwho.py:122 - Error in communication with subscription manager:
Traceback (most recent call last):
File "/usr/share/virt-who/virtwho.py", line 102, in send
self._sendGuestList(report)
File "/usr/share/virt-who/virtwho.py", line 128, in _sendGuestList
manager.sendVirtGuests(report.guests)
File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 119, in sendVirtGuests
self.connection.updateConsumer(self.uuid(), guest_uuids=domains)
File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 858, in updateConsumer
ret = self.conn.request_put(method, params)
File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 611, in request_put
return self._request("PUT", method, params)
File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 530, in _request
self.validateResponse(result, request_type, handler)
File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 572, in validateResponse
raise RestlibException(response['status'], error_msg)
RestlibException: Task e35de971-2e31-4e8b-907c-7206d6b9d426: Katello::Errors::CandlepinError: Runtime Error No entitlements for products: [] at org.candlepin.policy.js.autobind.AutobindRules.selectBestPools:79
Looks like some server side issue: RestlibException: Required lock is already taken by other running tasks. Please inspect their state, fix their errors and resume them. Moving back to candlepin. Since this issue was entered in Red Hat Bugzilla, the release flag has been set to ? to ensure that it is properly evaluated for this release. *** Bug 1232078 has been marked as a duplicate of this bug. *** When occurred this problem, virt-who will failed to send host/guest mapping to Satellite until the conflict task is resumed or destroyed. Original report references a lock that appears to be from foreman, don't think this one is candlepin related. The error in comment #6 is candlepin related and just a normal situation being incorrectly thrown as an error. This should actually be fixed aleady be fixed in 6.1. But key issue looks to be foreman tasks of some kind. Apologies for the component bouncing, and the fact I don't know what component this should be so setting to uncategorized. It also reproduced on Satellite 6.1.0-20150818.1. please see the related log in the attachment lock.tar.gz Created attachment 1065069 [details]
Lock log on Satellite 6.1.0-20150818.1
It still exist on Satellite-6.1.0-RHEL-6-20160321.0, When it occurred this issue, it will failed to list available pool. please see the candlepin log, foreman log, rhsm log in attachment "Lock log on Satellite-6.1.0-RHEL-6-20160321.0 .
# subscription-manager list --available
Task 64e2754b-6cd6-4f24-b246-0836f5067152: Katello::Errors::CandlepinError: Runtime Error ERROR: value too long for type character varying(255) at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,157
# tail -f /var/log/rhsm/rhsm.log
2016-04-04 23:40:30,376 [virtwho.main ERROR] MainProcess(8953):MainThread @virtwho.py:send:216 - Error in communication with subscription manager:
Traceback (most recent call last):
File "/usr/share/virt-who/virtwho.py", line 197, in send
self._sendGuestList(report)
File "/usr/share/virt-who/virtwho.py", line 222, in _sendGuestList
manager.sendVirtGuests(report, self.options)
File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 152, in sendVirtGuests
self.connection.updateConsumer(self.uuid(), guest_uuids=serialized_guests)
File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 995, in updateConsumer
ret = self.conn.request_put(method, params)
File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 690, in request_put
return self._request("PUT", method, params)
File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 598, in _request
self.validateResponse(result, request_type, handler)
File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 648, in validateResponse
raise RestlibException(response['status'], error_msg, response.get('headers'))
RestlibException: Required lock is already taken by other running tasks.
Please inspect their state, fix their errors and resume them.
Required lock: read
Conflicts with tasks:
- https://satellite-b0a66026-99ec-4c05-a983-5704d9e04e89.novalocal/foreman_tasks/tasks/64e2754b-6cd6-4f24-b246-0836f5067152
Reproduced version:
candlepin-0.9.49.12-1.el7.noarch
katello-2.2.0.19-1.el7sat.noarch
foreman-1.7.2.55-1.el7sat.noarch
subscription-manager-1.16.8-8.el6.x86_64
python-rhsm-1.16.6-1.el6.x86_64
Created attachment 1143676 [details]
Lock log on Satellite-6.1.0-RHEL-6-20160321.0
This is an older bug which I do not envision being addressed in the near term. I am closing this out. If you believe doing so is an issue, please feel free to re-open and provide additional business information. Thank you. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2927 |
Description of problem: When migrate guest from orig host to dest host, it will show "RestlibException: Required lock is already taken by other running tasks" in the log file, this task has conflicted with "update" task. Failed to migrate guest. Version-Release number of selected component (if applicable): virt-who-0.12-7.el6.noarch subscription-manager-1.14.6-1.el6.x86_64 python-rhsm-1.14.2-1.el6.x86_64 candlepin-0.9.49.0-1.el7.noarch katello-2.2.0.8-1.el7sat.noarch foreman-1.7.2.19-1.el7sat.noarch Satellite-6.1.0-RHEL-7-20150506.0 How reproducible: 50% Steps to Reproduce: Precondition: Two hosts.Orig host with guest1. 1.Register two hosts and guest1 to satellite 2.Start virt-who service, make it run at libvirt mode 3.Migrate guest1 from orig host to dest host [root@hp-z220-05 libvirt-test-API]# virsh migrate --live 6.5_Server_x86_64 qemu+ssh://10.66.100.111/system root.100.111's password: 4.During migration, check the rhsm log in the dest host. [root@hp-z220-06 ~]# tail -f /var/log/rhsm/rhsm.log 2015-05-12 09:54:46,087 [DEBUG] rhsmd:18238 @identity.py:131 - Loading consumer info from identity certificates. 2015-05-12 09:54:46,092 [DEBUG] rhsmd:18238 @rhsmd:148 - D-Bus signal com.redhat.SubscriptionManager.EntitlementStatus.entitlement_status_changed emitted 2015-05-12 10:36:17,658 [DEBUG] @libvirtd.py:200 - Libvirt domains found: [] 2015-05-12 10:36:17,673 [DEBUG] @subscriptionmanager.py:75 - Authenticating with certificate: /etc/pki/consumer/cert.pem 2015-05-12 10:36:21,818 [INFO] @subscriptionmanager.py:116 - Sending domain info: [] 2015-05-12 10:36:24,724 [DEBUG] @libvirtd.py:190 - Virtual machine found: 6.5_Server_x86_64: 3257cbbc-9396-1e11-0868-e2302568f2cc 2015-05-12 10:36:24,727 [DEBUG] @libvirtd.py:200 - Libvirt domains found: [{'guestId': '3257cbbc-9396-1e11-0868-e2302568f2cc', 'attributes': {'active': 1, 'virtWhoType': 'libvirt', 'hypervisorType': 'QEMU'}, 'state': 3}] 2015-05-12 10:36:27,677 [ERROR] @virtwho.py:115 - Error in communication with subscription manager: Traceback (most recent call last): File "/usr/share/virt-who/virtwho.py", line 102, in send self._sendGuestList(report) File "/usr/share/virt-who/virtwho.py", line 121, in _sendGuestList manager.sendVirtGuests(report.guests) File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 119, in sendVirtGuests self.connection.updateConsumer(self.uuid(), guest_uuids=domains) File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 858, in updateConsumer ret = self.conn.request_put(method, params) File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 611, in request_put return self._request("PUT", method, params) File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 530, in _request self.validateResponse(result, request_type, handler) File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 572, in validateResponse raise RestlibException(response['status'], error_msg) RestlibException: Required lock is already taken by other running tasks. Please inspect their state, fix their errors and resume them. Required lock: read Conflicts with tasks: - https://cisco-c210-01.rhts.eng.bos.redhat.com/foreman_tasks/tasks/b2a2dfc2-8582-4208-bca4-f340a544828d 5.Open the conflict task, it will error on "update" task, please see the screenshot in attachment. Id: b2a2dfc2-8582-4208-bca4-f340a544828d Label: Actions::Katello::System::Update Name: Update Owner: Started at: 2015-05-11 05:45:25 UTC Ended at: State: paused Result: error Params: system 'hp-z220-06.qe.lab.eng.nay.redhat.com'; organization 'Default Organization' Actual results: Failed to migrate guest from orig host to dest host. it will show "Required lock is already taken by other running tasks" on dest host. Expected results: It shouldn't show conflict task when do migration. Migration successfully, Additional info: