RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1149669 - libvirt drops cdrom attached to local ISO in case of uid mismatch
Summary: libvirt drops cdrom attached to local ISO in case of uid mismatch
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.6
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: rc
: 6.6
Assignee: Michal Privoznik
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 1002699 1150611
TreeView+ depends on / blocked
 
Reported: 2014-10-06 12:41 UTC by Francesco Romani
Modified: 2014-12-11 16:13 UTC (History)
28 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 1145636
: 1150611 (view as bug list)
Environment:
Last Closed: 2014-10-20 11:37:04 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Francesco Romani 2014-10-06 12:41:05 UTC 
+++ This bug was initially created as a clone of Bug #1145636 +++


(In reply to Francesco Romani from comment #10)

[...]
But libvirt configuration is messed up. dst vdsm receives:

Thread-228::DEBUG::2014-10-01 18:11:33,438::vm::4817::vm.Vm::(_getUnderlyingVmInfo) vmId=`9bf662e3-9c72-450e-98e0-2a658b42c602`::VM device changed:
<devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk device="cdrom" type="file">
      <driver name="qemu" type="raw"/>
      <source startupPolicy="optional"/>
      <target bus="ide" dev="hdc"/>
      <readonly/>
      <serial/>
      <alias name="ide0-1-0"/>
      <address bus="1" controller="0" target="0" type="drive" unit="0"/>
    </disk>
    <disk device="cdrom" type="file">
      <driver name="qemu" type="raw"/>
      <source startupPolicy="optional"/>
      <target bus="ide" dev="hdd"/>
      <readonly/>
      <serial/>
      <alias name="ide0-1-1"/>
      <address bus="1" controller="0" target="0" type="drive" unit="1"/>
    </disk>

This comes from additional debug logging I added to VDSM, on domDependentInit (VDSM re-fetches XML domain data from libvirt once the VM started)
And this causes all the havoc we seen.

To wrap up:
- we though it happens on RHEL 6.5 also, but can't reproduce here
- seems libvirt issue
- seems RHEL 6.6 regression

--- Additional comment from Francesco Romani on 2014-10-02 04:26:38 EDT ---

snippet of VDSM logs of the first migration with more debug data, demonstrating the disappearance of the path.

--- Additional comment from Francesco Romani on 2014-10-02 04:34:36 EDT ---

Hi Michal,

is the strange behaviour documented in https://bugzilla.redhat.com/show_bug.cgi?id=1145636#c13

a libvirt regression?

--- Additional comment from Michal Privoznik on 2014-10-02 05:36:11 EDT ---

(In reply to Francesco Romani from comment #15)
> Hi Michal,
> 
> is the strange behaviour documented in
> https://bugzilla.redhat.com/show_bug.cgi?id=1145636#c13
> 
> a libvirt regression?

Well, you're setting cdroms to be optional, so if the source file doesn't exist it's dropped. Can you attach libvirt debug logs among with short description what you think is behaving wrongly? My brain's too small to parse vdsm debug logs, sorry.

--- Additional comment from Francesco Romani on 2014-10-02 07:29:31 EDT ---

(In reply to Michal Privoznik from comment #16)
> (In reply to Francesco Romani from comment #15)
> > Hi Michal,
> > 
> > is the strange behaviour documented in
> > https://bugzilla.redhat.com/show_bug.cgi?id=1145636#c13
> > 
> > a libvirt regression?
> 
> Well, you're setting cdroms to be optional, so if the source file doesn't
> exist it's dropped. Can you attach libvirt debug logs among with short
> description what you think is behaving wrongly? My brain's too small to
> parse vdsm debug logs, sorry.

What I was expecting here is the drive to be dropped altogether, not just the source path. I was most likely just remembering wrongly.

So, we have to dig deeper.
On a well-behaving RHEL 6.5 host:
$ ls -lhZ d9be4550-7af1-4580-9430-4bfdaa7d50cf.f648da39aae979cd5d799be77f172429.img 
-rw-r-----. vdsm qemu system_u:object_r:virt_content_t:s0 d9be4550-7af1-4580-9430-4bfdaa7d50cf.f648da39aae979cd5d799be77f172429.img

On the failing RHEL 6.6 host:
# ls -lhZ 9bf662e3-9c72-450e-98e0-2a658b42c602.0defd0b2d811c0e399a713a0b684ce36.img 
-rw-r-----. vdsm qemu unconfined_u:object_r:virt_var_run_t:s0 9bf662e3-9c72-450e-98e0-2a658b42c602.0defd0b2d811c0e399a713a0b684ce36.img

then is selinux labeling again.

--- Additional comment from Michal Privoznik on 2014-10-02 08:17:19 EDT ---

(In reply to Francesco Romani from comment #17)
> (In reply to Michal Privoznik from comment #16)
> > (In reply to Francesco Romani from comment #15)
> > > Hi Michal,
> > > 
> > > is the strange behaviour documented in
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1145636#c13
> > > 
> > > a libvirt regression?
> > 
> > Well, you're setting cdroms to be optional, so if the source file doesn't
> > exist it's dropped. Can you attach libvirt debug logs among with short
> > description what you think is behaving wrongly? My brain's too small to
> > parse vdsm debug logs, sorry.
> 
> What I was expecting here is the drive to be dropped altogether, not just
> the source path. I was most likely just remembering wrongly.

You can't do that. It would break machine ABI. We must keep the drive but with empty source path. It's like ripping CD off the tray (CD/floppies can disappear anytime in real HW too).

> 
> So, we have to dig deeper.
> On a well-behaving RHEL 6.5 host:
> $ ls -lhZ
> d9be4550-7af1-4580-9430-4bfdaa7d50cf.f648da39aae979cd5d799be77f172429.img 
> -rw-r-----. vdsm qemu system_u:object_r:virt_content_t:s0
> d9be4550-7af1-4580-9430-4bfdaa7d50cf.f648da39aae979cd5d799be77f172429.img
> 
> On the failing RHEL 6.6 host:
> # ls -lhZ
> 9bf662e3-9c72-450e-98e0-2a658b42c602.0defd0b2d811c0e399a713a0b684ce36.img 
> -rw-r-----. vdsm qemu unconfined_u:object_r:virt_var_run_t:s0
> 9bf662e3-9c72-450e-98e0-2a658b42c602.0defd0b2d811c0e399a713a0b684ce36.img
> 
> then is selinux labeling again.

Well, libvirt uses 'access($disk_src_path, F_OK) == 0' to determine if the path is accessible or not (this applies to local storage like in the case we're discussing).

Truth is, this check happens before libvirt even tries to relabel all the paths. However, libvirt is running as root:root so there's no label that would prevent libvirtd to touch.

Maybe there's a race somewhere? What if vdsm creates the file not upfront but somewhat later in the migration? For instance after the libvirtd has checked the disk presence?

Again, libvirt debug logs would shed more light.

--- Additional comment from Francesco Romani on 2014-10-03 09:16:04 EDT ---

managed to grab some debug logs, attached.

Highltighing relevant lines:

2014-10-03 12:45:31.102+0000: 59520: debug : qemuDomainCheckDiskPresence:2006 : Checking for disk presence
2014-10-03 12:45:31.107+0000: 59520: debug : virStorageFileGetMetadata:1006 : path=/var/run/vdsm/payload/f41aaf55-a521-4bee-9484-e5c2100599e7.f0066193f44d186b66f2dd5cac35309b.img format=1 uid=107 gid=107 probe=0
2014-10-03 12:45:31.107+0000: 59520: debug : virStorageFileGetMetadataRecurse:938 : path=/var/run/vdsm/payload/f41aaf55-a521-4bee-9484-e5c2100599e7.f0066193f44d186b66f2dd5cac35309b.img format=1 uid=107 gid=107 probe=0
2014-10-03 12:45:31.107+0000: 59520: error : virStorageFileGetMetadataRecurse:952 : Failed to open file '/var/run/vdsm/payload/f41aaf55-a521-4bee-9484-e5c2100599e7.f0066193f44d186b66f2dd5cac35309b.img': No such file or directory
2014-10-03 12:45:31.107+0000: 59520: debug : qemuDomainCheckRemoveOptionalDisk:1937 : Dropping disk 'hdd' on domain 'rhel-1' (UUID 'f41aaf55-a521-4bee-9484-e5c2100599e7') due to inaccessible source '/var/run/vdsm/payload/f41aaf55-a521-4bee-9484-e5c2100599e7.f0066193f44d186b66f2dd5cac35309b.img'


permissions on files:

  File: `/var/run/vdsm/payload/82604221-514a-4844-8f1e-767f397ebc03.4ca50fab90c0e9eebea1a52d6ff17e84.img'
  Size: 366592    	Blocks: 728        IO Block: 4096   regular file
Device: 803h/2051d	Inode: 23093288    Links: 1
Access: (0640/-rw-r-----)  Uid: (   36/    vdsm)   Gid: (  107/    qemu)
Access: 2014-10-02 15:01:09.000000000 +0300
Modify: 2014-10-02 15:01:09.000000000 +0300
Change: 2014-10-02 15:01:09.000000000 +0300

  File: `/var/run/vdsm/payload/db35b270-56c9-4304-b0f8-92e758d7b3b8.8329070093a08e7513cfe0b8e08d2eb1.img'
  Size: 366592    	Blocks: 728        IO Block: 4096   regular file
Device: 803h/2051d	Inode: 23093282    Links: 1
Access: (0640/-rw-r-----)  Uid: (   36/    vdsm)   Gid: (  107/    qemu)
Access: 2014-10-02 15:01:05.000000000 +0300
Modify: 2014-10-02 15:01:05.000000000 +0300
Change: 2014-10-02 15:01:05.000000000 +0300

  File: `/var/run/vdsm/payload/f41aaf55-a521-4bee-9484-e5c2100599e7.143152bdd9cf5372422f0669f030cda5.img'
  Size: 366592    	Blocks: 728        IO Block: 4096   regular file
Device: 803h/2051d	Inode: 23093281    Links: 1
Access: (0640/-rw-r-----)  Uid: (   36/    vdsm)   Gid: (  107/    qemu)
Access: 2014-10-03 15:58:01.000000000 +0300
Modify: 2014-10-03 15:58:01.000000000 +0300
Change: 2014-10-03 15:58:01.000000000 +0300

groups:

uid 36 is vdsm

qemu:x:107:107:qemu user:/:/sbin/nologin
qemu:x:107:vdsm,sanlock

--- Additional comment from Francesco Romani on 2014-10-03 09:17:27 EDT ---

uid mismatch. Apparently libvirt expects qemu/qemu, but vdsm sets vdsm/qemu.

--- Additional comment from Michal Privoznik on 2014-10-03 11:26:58 EDT ---

(In reply to Francesco Romani from comment #23)
> Created attachment 943715 [details]
> libvirt debug logs of a reproduction of the problem
> 
> uid mismatch. Apparently libvirt expects qemu/qemu, but vdsm sets vdsm/qemu.

Correct. In the version you are using libvirt checks (mistakenly) if disk is accessible as user/group from qemu.conf while domain may be running under different pair. This was fixed upstream by:

commit 9bf629ab60ec0f25be5426e8598e98c89c244c5a
Author:     Peter Krempa <pkrempa>
AuthorDate: Fri Feb 7 18:42:27 2014 +0100
Commit:     Peter Krempa <pkrempa>
CommitDate: Mon Feb 10 15:49:59 2014 +0100

    qemu: Use correct permissions when determining the image chain
    
    The code took into account only the global permissions. The domains now
    support per-vm DAC labels and per-image DAC labels. Use the most
    specific label available.

which is contained in v1.2.2-rc1~147
Comment 6 Michal Privoznik 2014-10-09 08:18:40 UTC 
Another try:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2014-October/msg00258.html
Comment 8 Francesco Romani 2014-10-14 16:37:31 UTC 
the provided RPM works for me, but even thepast snapshot (6.6.RC5) did, so to be 100% we need to test it on the boxes which made the issue manifest.
Comment 10 zhenfeng wang 2014-10-17 06:04:32 UTC 
I want to try the issue in libvirt side, I try serveral ways to let libvirt drop the iso abnormally, however every time the guest can drop the iso correctly, I doubt that what's situationw will cause the guest drop the iso abnormally, can you give me some advise? The following steps were my reproduce steps , my reproduce steps was little different with the original bug, maybe i miss something important, please help point out it , thanks

steps
Scenario 1
1.Two hosts registered in rhevm: host1(source)/host2(target)
2.The configuration in qemu.conf in two rhevm hosts like following
#cat /etc/libvirt/qemu.conf
auto_dump_path="/var/log/core"
dynamic_ownership = 0
lock_manager="sanlock"
remote_display_port_max=6923
remote_display_port_min=5900
save_image_format="lzop"
spice_tls=1
spice_tls_x509_cert_dir="/etc/pki/vdsm/libvirt-spice"

3.Prepare a nfs server, put the guest os into the share directoy, and set its permisson to  vdsm:kvm, then mount the nfs server on the two rhevm hosts
#mount $nfs_ip:/export /mnt
# ll /mnt/rhel6.img 
-rw-rw----. 1 vdsm kvm 3809017856 Oct 17 00:35 /mnt/rhel6.img

4.Create a guest in the source host1 with shareable os in nfs server
and two cdroms with unshareable iso which located in the host1
#virsh dumpxml rhel6
--
  <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/mnt/zhwang/img/rhel6.img' startupPolicy='optional'>
        <seclabel model='selinux' relabel='no'/>
      </source>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/cd1.iso' startupPolicy='optional'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <serial></serial>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/cd2.iso' startupPolicy='optional'/>
      <target dev='hdd' bus='ide'/>
      <readonly/>
      <serial></serial>
      <alias name='ide0-1-1'/>
      <address type='drive' controller='0' bus='1' target='0' unit='1'/>
    </disk>


5.check the iso's label
# ll /var/lib/libvirt/images/ -Z
-rw-r-----. vdsm qemu system_u:object_r:virt_content_t:s0 cd1.iso
-rw-r-----. vdsm qemu system_u:object_r:virt_content_t:s0 cd2.iso

6.Migrate the guest to the target
#virsh migrate --live rhel6 qemu+ssh://$host2/system 

7.After migrated ,check the iso in host2, found the two isos in the cdrom has been dropped 
 <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/mnt/zhwang/img/rhel6.img' startupPolicy='optional'>
        <seclabel model='selinux' relabel='no'/>
      </source>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source startupPolicy='optional'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <serial></serial>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source startupPolicy='optional'/>
      <target dev='hdd' bus='ide'/>
      <readonly/>
      <serial></serial>
      <alias name='ide0-1-1'/>
      <address type='drive' controller='0' bus='1' target='0' unit='1'/>
    </disk>
8.Migrate the guest back to the source, the guest migrate successfully

Scenario 2
The scenario2 was same with scenario1 except the following steps
1.pre-create the cd1.iso cd2.iso in the corresponding directory in the target host before do the migration from the source to 
the target


Results: the guest can migrate successfully to the target host, and the guest won't drop the iso, even the label on cd1.iso cd2.iso
on target host2 were different with the label on the source host1

2.Then change the label for cd1.iso cd2.iso in source host1, then migrate the guest from the target back to the source
host1# chcon unconfined_u:object_r:virt_image_t:s0 cd1.iso 
host1# chcon unconfined_u:object_r:virt_image_t:s0 cd2.iso 

Results: the guest can migrate back successfuly to the source host, and the guest won't drop the iso, after migration the label on cd1.iso
cd2.iso on the source host was likely following
# ll -Z
-rw-r-----. vdsm qemu system_u:object_r:virt_content_t:s0 cd1.iso
-rw-r-----. vdsm qemu system_u:object_r:virt_content_t:s0 cd2.iso

3.Re-migrate the guest to the target host2, migrate successfully, and the guest won't drop the iso

4.Re-migrate the guest back to the source host1 , migrate successfully, and the guest won't drop the iso
Comment 11 zhenfeng wang 2014-10-17 06:04:49 UTC 
I saw you add the per-image on the patch, how does it work ? I do the following research about it, but not sure how to use it,  will it use the per-image user's permission to open disk if the 
qemu process didn't have the perssion open it or will the libvirt relabel the disk with the per-image setting ?


1.Set dynamic_ownership=0 in qemu.conf
user=qemu
group=qemu
dynamic_ownership=0
#service libvirtd retart

2.Edit guest's xml, add the following content
#virsh dumpxml rhel6
--
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/rhel6.img'>
        <seclabel model='dac' relabel='yes'>
          <label>test1:test1</label>
        </seclabel>
      </source>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/test.img'>
        <seclabel model='dac' relabel='yes'>
          <label>test1:test1</label>
        </seclabel>
      </source>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </disk>
 
3.Change the image's owner/group to test1/test1
# ll /var/lib/libvirt/images
total 6833492
-rw-r-----. 1 test1 test1 3804430336 Oct 17 01:11 rhel6.img
-rw-r--r--. 1 test1 test1 1073741824 Oct 15 04:01 test.img

4.Start the guest, the guest will fail to start
# virsh start rhel6
error: Failed to start domain rhel6
error: internal error Process exited while reading console log output: char device redirected to /dev/pts/1
2014-10-17T05:15:57.586405Z qemu-kvm: -drive file=/var/lib/libvirt/images/rhel6.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: could not open disk image /var/lib/libvirt/images/rhel6.img: Permission denied
Comment 13 Michal Privoznik 2014-10-20 11:37:04 UTC 
Closing per https://bugzilla.redhat.com/show_bug.cgi?id=1145636#c35 comment
Note You need to log in before you can comment on or make changes to this bug.