Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1654804

Summary:	qemu-img: fails to get shared 'write' lock
Product:	Red Hat OpenStack	Reporter:	Siggy Sigwald <ssigwald>
Component:	openstack-nova	Assignee:	Kashyap Chamarthy <kchamart>
Status:	CLOSED DUPLICATE	QA Contact:	OSP DFG:Compute <osp-dfg-compute>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	10.0 (Newton)	CC:	chayang, coli, dasmith, eglynn, jhakimra, juzhang, kchamart, michen, ptoscano, rcernin, rjones, sbauza, sgordon, virt-maint, vromanso
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-12-05 08:21:43 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Siggy Sigwald 2018-11-29 17:26:38 UTC

Description of problem:

2018-11-26 07:43:09.946 682424 ERROR nova.compute.manager Command: /usr/bin/python2 -m oslo_concurrency.prlimit --as=1073741824 --cpu=8 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/1ec303ae-0a91-4df3-b6ee-8ab2a32cfcf2/disk
2018-11-26 07:43:09.946 682424 ERROR nova.compute.manager Exit code: 1
2018-11-26 07:43:09.946 682424 ERROR nova.compute.manager Stdout: u''
2018-11-26 07:43:09.946 682424 ERROR nova.compute.manager Stderr: u'qemu-img: Could not open \'/var/lib/nova/instances/1ec303ae-0a91-4df3-b6ee-8ab2a32cfcf2/disk\': Failed to get shared "write" lock\nIs another process using the image?\n'
2018-11-26 07:43:09.946 682424 ERROR nova.compute.manager
2018-11-26 07:44:10.789 682424 INFO nova.compute.resource_tracker [req-42cc1bab-1d28-4ffe-979c-e8a2980a9133 - - - - -] Auditing locally available compute resources for node povccosp01-compute-22.vcc.t-mobile.com
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager [req-42cc1bab-1d28-4ffe-979c-e8a2980a9133 - - - - -] Error updating resources for node povccosp01-compute-22.vcc.t-mobile.com.
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager Traceback (most recent call last):
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6460, in update_available_resource_for_node
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager     rt.update_available_resource(context)
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 511, in update_available_resource
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager     resources = self.driver.get_available_resource(self.nodename)
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5595, in get_available_resource
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager     disk_over_committed = self._get_disk_over_committed_size_total()
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7188, in _get_disk_over_committed_size_total
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager     block_device_info=block_device_info)
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7097, in _get_instance_disk_info
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager     dk_size = disk_api.get_allocated_disk_size(path)
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/disk/api.py", line 158, in get_allocated_disk_size
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager     return images.qemu_img_info(path).disk_size
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/virt/images.py", line 72, in qemu_img_info
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager     raise exception.InvalidDiskInfo(reason=msg)
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager InvalidDiskInfo: Disk info file is invalid: qemu-img failed to execute on /var/lib/nova/instances/1ec303ae-0a91-4df3-b6ee-8ab2a32cfcf2/disk : Unexpected error while running command.
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager Command: /usr/bin/python2 -m oslo_concurrency.prlimit --as=1073741824 --cpu=8 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/1ec303ae-0a91-4df3-b6ee-8ab2a32cfcf2/disk
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager Exit code: 1
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager Stdout: u''
2018-11-26 07:44:10.924 682424 ERROR nova.compute.manager Stderr: u'qemu-img: Could not open \'/var/lib/nova/instances/1ec303ae-0a91-4df3-b6ee-8ab2a32cfcf2/disk\': Failed to get shared "write" lock\nIs another process using the image?\n'
Version-Release number of selected component (if applicable):

According to the upstream bug attached it should be fixed in newer version (1.40) but customer has (1.38)

Comment 2 Tingting Mao 2018-11-30 05:58:20 UTC

According to the below comment, it seems that this bug should belong to libguestfs. Please correct me if I am wrong, thanks.

https://bugs.launchpad.net/qemu/+bug/1740364/comments/6

Comment 3 Richard W.M. Jones 2018-11-30 08:39:07 UTC

No this is a bug in nova.  It should be using the qemu-img -U option.

Comment 4 Richard W.M. Jones 2018-11-30 08:50:19 UTC

I would say that the fix required here is probably:

commit 807579755c4a116309eca5b2bcdbab9d1f393bab
Author: Matt Riedemann <mriedem.os>
Date:   Wed Sep 20 10:44:11 2017 -0400

    Support qemu >= 2.10
    
    Qemu 2.10 added the requirement of a --force-share flag to qemu-img
    info when reading information about a disk that is in use by a
    guest. We do this a lot in Nova for operations like gathering
    information before live migration.
    
    Up until this point all qemu/libvirt version matching has been solely
    inside the libvirt driver, however all the image manip code was moved
    out to nova.virt.images. We need the version of QEMU available there.
    
    This does it by initializing that version on driver init host. The net
    effect is also that broken libvirt connections are figured out
    earlier, as there is an active probe for this value.
    
    Co-Authored-By: Sean Dague <sean>
    
    Change-Id: Iae2962bb86100f03fd3ad9aac3767da876291e74
    Closes-Bug: #1718295

Comment 5 Richard W.M. Jones 2018-11-30 08:51:04 UTC

Upstream bug: https://bugs.launchpad.net/nova/+bug/1718295

Comment 6 Robin Cernin 2018-12-05 05:35:56 UTC

Backport request for that commit in https://bugzilla.redhat.com/show_bug.cgi?id=1533444

Comment 7 Kashyap Chamarthy 2018-12-05 08:21:43 UTC

(In reply to Robin Cernin from comment #6)
> Backport request for that commit in
> https://bugzilla.redhat.com/show_bug.cgi?id=1533444

We should close this bug (1654804) as a duplicate of the above (1533444).

*** This bug has been marked as a duplicate of bug 1533444 ***

Comment 9 Kashyap Chamarthy 2018-12-05 08:47:33 UTC

Actually closing this as the duplicate of bug 1646382, which has the Fixed-In-Version:  

openstack-nova-14.1.0-34.el7ost

*** This bug has been marked as a duplicate of bug 1646382 ***