Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 607919 - "Permission denied" is seen when trying to start vm with a nfs disk image after libivrtd restart.
"Permission denied" is seen when trying to start vm with a nfs disk image aft...
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt (Show other bugs)
6.0
All Linux
low Severity medium
: rc
: ---
Assigned To: Laine Stump
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-06-25 04:01 EDT by Johnny Liu
Modified: 2010-06-28 23:45 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-06-28 23:45:13 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
guest log (8.31 KB, text/plain)
2010-06-25 06:09 EDT, Johnny Liu
no flags Details
qemu.conf (7.31 KB, text/plain)
2010-06-25 06:10 EDT, Johnny Liu
no flags Details
log file for comment 8 (20.87 KB, text/plain)
2010-06-25 07:03 EDT, Johnny Liu
no flags Details
guest log - ver 2 (16.99 KB, text/plain)
2010-06-25 07:04 EDT, Johnny Liu
no flags Details
qemu.conf - ver 2 (7.31 KB, text/plain)
2010-06-25 07:04 EDT, Johnny Liu
no flags Details

  None (edit)
Description Johnny Liu 2010-06-25 04:01:53 EDT
Description of problem:
Restart libvirtd service, try to start a vm with disk image on a nfs share, the following error is seen:
# virsh start test
error: Failed to start domain test
error: internal error Process exited while reading console log output: char device redirected to /dev/pts/3
qemu: could not open disk image /mnt/jialiu/vm1.img: Permission denied


Version-Release number of selected component (if applicable):
libvirt-0.8.1-10.el6.x86_64
# uname -a
Linux dhcp-66-70-86.nay.redhat.com 2.6.32-37.el6.x86_64 #1 SMP Sun Jun 20 19:29:35 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
qemu-kvm-0.12.1.2-2.81.el6.x86_64
# rpm -qa|grep selinux
selinux-policy-targeted-3.7.19-27.el6.noarch
libselinux-2.0.94-1.el6.x86_64
libselinux-python-2.0.94-1.el6.x86_64
selinux-policy-3.7.19-27.el6.noarch
libselinux-utils-2.0.94-1.el6.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Install a fresh os from nightly build - RHEL6.0-20100622.n.0
2. mount nfs share
# setenforce 0
# getenforce 
Permissive
# iptables -F
# mount 10.66.90.113:/vol/xenimage /mnt (10.66.90.113 is a share nfs)
3. Define a vm, and start it.
# virsh dumpxml test
<domain type='kvm'>
  <name>test</name>
  <uuid>89731ff2-def2-9f1a-0708-5cac6b051fab</uuid>
  <memory>524288</memory>
  <currentMemory>524288</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' cache='none'/>
      <source file='/mnt/jialiu/vm1.img'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' unit='0'/>
    </disk>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:29:e3:76'/>
      <source network='default'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
  </devices>
</domain>
# virsh start test
Domain test started

# virsh destroy test
Domain test destroyed

4. restart libvirtd, and try to start the vm again:
# service libvirtd restart
Stopping libvirtd daemon:                                  [  OK  ]
Starting libvirtd daemon:                                  [  OK  ]
# virsh start test
error: Failed to start domain test
error: internal error Process exited while reading console log output: char device redirected to /dev/pts/3
qemu: could not open disk image /mnt/jialiu/vm1.img: Permission denied

Actual results:
Fail to start vm, and "Permission denied" is seen.

Expected results:
vm should be started successfully.

Additional info:
Comment 2 RHEL Product and Program Management 2010-06-25 04:22:54 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.
Comment 3 Daniel Berrange 2010-06-25 04:58:05 EDT
Can you attach the /etc/libvirt/qemu.conf file, /var/log/libvirt/qemu/$GUEST.log and provide output of

 ls -l /mnt/jialiu/vm1.img
 ls -dl /mnt/
 ls -dl /mnt/jialiu/
Comment 4 dyuan 2010-06-25 05:58:31 EDT
start domain with 'Permission denied' error both with enforcing and permissive mode. test version is the same as orignal description.

qemu: could not open disk image /mnt/test/dyuan/img/rhel5u4-x86_64.img: Permission denied

10.66.90.113:/vol/xenimage on /mnt/test type nfs (rw,addr=10.66.90.113)

# getsebool -a|grep virt_use_nfs
virt_use_nfs --> on

# ll -dZ /mnt/
drwxr-xr-x. root root system_u:object_r:mnt_t:s0       /mnt/
# ll -dZ /mnt/test/
drwx-----x. root root system_u:object_r:nfs_t:s0       /mnt/test/
# ll -dZ /mnt/test/dyuan/
drwxrwxr-x. qemu qemu system_u:object_r:nfs_t:s0       /mnt/test/dyuan/
# ll -dZ /mnt/test/dyuan/img/
drwxrwxr-x. qemu qemu system_u:object_r:nfs_t:s0       /mnt/test/dyuan/img/
# ll -dZ /mnt/test/dyuan/img/rhel5u4-x86_64.img 
-rw-r--r--. root root system_u:object_r:nfs_t:s0       /mnt/test/dyuan/img/rhel5u4-x86_64.img
Comment 5 Johnny Liu 2010-06-25 06:05:19 EDT
# ls -l /mnt/jialiu/vm1.img
-rw-------. 1 root root 5242880000 Jun 25 06:03 /mnt/jialiu/vm1.img
[root@dhcp-66-70-86 ~]# ls -dl /mnt/
drwx-----x. 17 root root 4096 Jun 24 10:20 /mnt/
[root@dhcp-66-70-86 ~]# ls -dl /mnt/jialiu/
drwxr-xr-x. 2 qemu qemu 4096 Jun 24 21:28 /mnt/jialiu/

The log file will be attached later.
Comment 6 Johnny Liu 2010-06-25 06:09:32 EDT
Created attachment 426831 [details]
guest log
Comment 7 Johnny Liu 2010-06-25 06:10:15 EDT
Created attachment 426832 [details]
qemu.conf
Comment 8 Daniel Berrange 2010-06-25 06:40:53 EDT
Can you set 

 log_filters="1:util 1:security 1:qemu"
 log_outputs="1:file:/var/log/libvirt/libvirtd.log"

in /etc/libvirt/libvirtd.conf and restart libvirtd and try the guest again, then attach that log file to this bug.
Comment 9 Johnny Liu 2010-06-25 07:02:23 EDT
Sorry for my mistake, the attachments in comment 6 and comment 7 are not correct ones.

So I attached them again and with the log file generated in comment 8.
Comment 10 Johnny Liu 2010-06-25 07:03:25 EDT
Created attachment 426845 [details]
log file for comment 8
Comment 11 Johnny Liu 2010-06-25 07:04:23 EDT
Created attachment 426847 [details]
guest log - ver 2
Comment 12 Johnny Liu 2010-06-25 07:04:59 EDT
Created attachment 426848 [details]
qemu.conf - ver 2
Comment 13 Hugh Brock 2010-06-28 12:13:53 EDT
Not sure if this is misconfiguration or for real, but in any case if it's NFS it's critical-path for VDSM and must be fixed.
Comment 14 Laine Stump 2010-06-28 23:45:13 EDT
In order to store the image files on a root-squashed NFS share, dynamic_ownership must be set to 0 in qemu.conf. This is not the case here.

When I set dynamic_ownership=1 in my own test setup, failure occurs as indicated here. When I set dynamic_ownership=0, it the domain can be successfully created and destroyed multiple times.

(this did point out a separate bug in in a patch I just pushed today, though, so it's actually good this bug was filed ;-)

Note You need to log in before you can comment on or make changes to this bug.