Bug 919504

Summary: Crash changing CDROM device media
Product: Red Hat Enterprise Linux 6 Reporter: Chris Pelland <cpelland>
Component: libvirtAssignee: Osier Yang <jyang>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 6.4CC: acathrow, berrange, cpelland, dallan, dyasny, dyuan, gsun, jyang, mzhan, pkrempa, pm-eus, rwu, weizhan, whuang, zhpeng
Target Milestone: rcKeywords: Regression, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-0.10.2-18.el6_4.1 Doc Type: Bug Fix
Doc Text:
Cause: Dereference pointers (E.g. disk def) which might be already freed. Consequence: Regressions like libvirtd crashing Fix: Add various checking to avoid dereferencing NULL pointers. And also copy the disk def before changing CD-ROM or Floppy medium. Result: The crash is fixed.
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-03-21 14:03:13 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 908073    
Bug Blocks:    

Description Chris Pelland 2013-03-08 16:39:37 UTC
This bug has been copied from bug #908073 and has been proposed
to be backported to 6.4 z-stream (EUS).

Comment 8 weizhang 2013-03-18 03:48:09 UTC
Verify pass on libvirt-0.10.2-18.el6_4.1.x86_64

When start guest with 
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw' cache='none'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <shareable/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>

guest can be started successfully without crash


Bug can be reproduced on libvirt-0.10.2-18.el6.x86_64, when start guest, libvirtd will crashed 
# virsh start tt
error: Failed to start domain tt
error: End of file while reading data: Input/output error
error: Failed to reconnect to the hypervisor

# service libvirtd status
libvirtd dead but pid file exists


So verify pass.

Comment 9 weizhang 2013-03-19 08:46:06 UTC
When test on the following 2 scenarios I think libvirt still have some problem, but not sure if they are serious enough

Scenario 1
1. Start a guest with
<disk type='block' device='cdrom'>
<driver name='qemu' type='raw'/>
<target dev='sdc' bus='scsi'/>
<readonly/>
<shareable/>
</disk>

2. Update cdrom device with unmatched source dev
# cat cdrom.xml
<disk type='block' device='cdrom'>
<driver name='qemu' type='raw'/>
<source dev='/var/lib/libvirt/images/tt.img'/>
<target dev='sdc' bus='scsi'/>
<readonly/>
<shareable/>
</disk>
# virsh update-device guest cdrom.xml
Device updated successfully

3. Restart libvirt
4. Check the guest status
guest is destroyed, with error in libvirtd.log

2013-03-19 08:23:27.144+0000: 18490: error : qemuGetSharedDiskKey:765 : Unable to get minor number of device '/var/lib/libvirt/images/tt.img': Invalid argument
2013-03-19 08:23:27.433+0000: 18490: error : qemuGetSharedDiskKey:765 : Unable to get minor number of device '/var/lib/libvirt/images/tt.img': Invalid argument

Scenario 2
1. Start a guest with xml
<disk type='block' device='lun'>
<driver name='qemu' type='raw'/>
<source dev='/dev/sdb'/>
<shareable/>
<target dev='vdb' bus='virtio'/>
</disk>

2. Attach device with same source but different sgio
# cat disk.xml
<disk type='block' device='lun' sgio='unfiltered'>
<driver name='qemu' type='raw'/>
<source dev='/dev/sdb'/>
<shareable/>
<target dev='vdc' bus='virtio'/>
</disk>

# virsh attach-device guest disk.xml
error: Failed to attach device from disk.xml
error: Requested operation is not valid: sgio of shared disk '/dev/sdb' conflicts with other active domains

Then do attach-device again
# virsh attach-device guest disk.xml 
Device attached successfully

@osier, could you please have a look?

Comment 10 weizhang 2013-03-19 09:22:37 UTC
Another scenario is that if I start 2 guests with same xml except the domain name, also have same attached device
    <disk type='block' device='lun'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/sdb'/>
      <target dev='vdb' bus='virtio'/>
      <shareable/>
    </disk>

when do attach-device on second defined guest 2 times with xml
<disk type='block' device='lun' sgio='unfiltered'>
<driver name='qemu' type='raw'/>
<source dev='/dev/sdb'/>
<shareable/>
<target dev='vdc' bus='virtio'/>
</disk>

disk can be attached successfully, and after restart libvirtd, the first started guest will be destroyed without any error on libvirtd.log

Comment 11 Osier Yang 2013-03-19 16:11:37 UTC
(In reply to comment #9)
> When test on the following 2 scenarios I think libvirt still have some
> problem, but not sure if they are serious enough
> 
> Scenario 1
> 1. Start a guest with
> <disk type='block' device='cdrom'>
> <driver name='qemu' type='raw'/>
> <target dev='sdc' bus='scsi'/>
> <readonly/>
> <shareable/>
> </disk>
> 
> 2. Update cdrom device with unmatched source dev
> # cat cdrom.xml
> <disk type='block' device='cdrom'>
> <driver name='qemu' type='raw'/>
> <source dev='/var/lib/libvirt/images/tt.img'/>
> <target dev='sdc' bus='scsi'/>
> <readonly/>
> <shareable/>
> </disk>
> # virsh update-device guest cdrom.xml
> Device updated successfully

A patch posted in upstream:
https://www.redhat.com/archives/libvir-list/2013-March/msg01021.html

With the patch, for this case, update-device will fail with error like "Unable to get minor number of device ....". The error is somehow expected, as the disk type and disk source are mismatched. The error is a bit not clear though. We might need to validate whether disk type and disk source are matched somewhere in future though.

So there are two problems here. One is I forgot to add the new disk src
into shared disk table. The other is to validate the disk type and disk
source matching (not sure we should do the validation though).

Both are not regression, so please file new bugs for them.

> 
> 3. Restart libvirt
> 4. Check the guest status
> guest is destroyed, with error in libvirtd.log
> 
> 2013-03-19 08:23:27.144+0000: 18490: error : qemuGetSharedDiskKey:765 :
> Unable to get minor number of device '/var/lib/libvirt/images/tt.img':
> Invalid argument
> 2013-03-19 08:23:27.433+0000: 18490: error : qemuGetSharedDiskKey:765 :
> Unable to get minor number of device '/var/lib/libvirt/images/tt.img':
> Invalid argument
> 
> Scenario 2
> 1. Start a guest with xml
> <disk type='block' device='lun'>
> <driver name='qemu' type='raw'/>
> <source dev='/dev/sdb'/>
> <shareable/>
> <target dev='vdb' bus='virtio'/>
> </disk>
> 
> 2. Attach device with same source but different sgio
> # cat disk.xml
> <disk type='block' device='lun' sgio='unfiltered'>
> <driver name='qemu' type='raw'/>
> <source dev='/dev/sdb'/>
> <shareable/>
> <target dev='vdc' bus='virtio'/>
> </disk>

Hum, I think sharing same disk source in same guest doesn't make sense, but I'm not confident about it yet. If it doesn't make sense, we should prohibit it. However, I don't think one will do this in practice anyway.

> # virsh attach-device guest disk.xml
> error: Failed to attach device from disk.xml
> error: Requested operation is not valid: sgio of shared disk '/dev/sdb'
> conflicts with other active domains
> 
> Then do attach-device again
> # virsh attach-device guest disk.xml 
> Device attached successfully
> 
> @osier, could you please have a look?

The reason for second attaching succeeded is the hash entry for the old disk is removed from the shared disk table when first attaching fails for the conflict, as we always assume same guest won't share same disk source. So either we should
prohibit sharing same disk source in same guest, or we should improve the codes to work around it. I'm tending to prohibit it.

Not regression either, please file a new bug for it.

Comment 14 errata-xmlrpc 2013-03-21 14:03:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0664.html