The second issue is that the operation should even fail because the directory is owned by vdsm:kvm (writable for kvm group) and we are trying to open the file as qemu:qemu while qemu user also belongs to kvm group. In other words, we don't initialize the supplementary groups for qemu user before trying to open the file.

This issue is fixed upstream by v0.9.1-230-g4dd9c16 and v0.9.1-231-g5e09aea:

commit 4dd9c16161ce0e73177a999bb26523a79c338cc2
Author: Jiri Denemark <jdenemar>
Date:   Sun May 22 17:16:44 2011 +0300

    util: Keep errno set to the root error after when returning from virSetUIDGID

commit 5e09aea7b07a683996863000bee0277f2e255f1c
Author: Jiri Denemark <jdenemar>
Date:   Sun May 22 17:05:07 2011 +0300

    Replace all remaining setgid/setuid calls with virSetUIDGID
    
    Two additional places need initgroups call to properly work in an
    environment where the UID is allowed to open/create stuff through its
    supplementary groups.

This should be fixed by the libvirt-0.9.2-1.el6 rebase

I tested it with libvirt-0.9.2-1.el6.x86_64 and vdsm-4.9-78.el6.x86_64, it doesn't work, I still get the Permission Denied error.

Thread-144::ERROR::2011-06-29 11:04:54,920::vm::230::vm.Vm::(run) vmId=`5cc7b29d-2e7f-494e-b3e7-b0239c0f8ec8`::Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 222, in run
    self._startUnderlyingMigration()
  File "/usr/share/vdsm/libvirtvm.py", line 252, in _startUnderlyingMigration
    self._vm._dom.save(fname)
  File "/usr/share/vdsm/libvirtvm.py", line 302, in f
    ret = attr(*args, **kwargs)
  File "/usr/share/vdsm/libvirtconnection.py", line 63, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 793, in save
    if ret == -1: raise libvirtError ('virDomainSave() failed', dom=self)
libvirtError: Error from child process creating '/rhev/data-center/8c2989d8-d8a1-495a-8c57-09b537a7a7b8/da6aa19a-03b0-4bd2-8aab-dabc924f8e3f/images/4ed9d520-92d3-4283-9c24-2c1bdb705265/3e8144f1-c153-432a-a92a-c94c6a690e02': Permission de
nied

Created attachment 510411 [details]
vdsm and libvirt logs

According to comment 5, the bug not fixed, move to assigned.

- What are the ownerships and permissions along the
/rhev/data-center/8c2989d8-d8a1-495a-8c57-09b537a7a7b8/da6aa19a-03b0-4bd2-8aab-dabc924f8e3f/images/4ed9d520-92d3-4283-9c24-2c1bdb705265/3e8144f1-c153-432a-a92a-c94c6a690e02 path?
- Who owns the associated qemu-kvm process and what does id command say about that owner?
- Can you also check that the operation wasn't denied by SELinux?

I reproduced it again, when trying to suspend the VM it changes to Paused, then in the next trial to suspend it, suspension succeed (it occurs alternately).

Ownership and permissions:
-rw-rw----. 1 vdsm kvm 0 Jul  3  2011 /rhev/data-center/8123336e-070d-44e8-bfc3-5c0159d10ee6/5060a943-7f74-40be-a0fa-dc2620f8f514/images/7f53d089-370e-4bdb-a7a5-c815f9d5e008/cbeca92e-56a9-4865-9fd3-366cf70a4306

SELinux:
It occurrs even without SELinux (I changed SELinux to Permissive using "setenforce 0")

Process' user:
[root@stone-vds1 ~]# ps -aux | grep 10263
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
qemu     10263 14.2  1.6 793308 262152 ?       Sl   12:22   0:22 /usr/libexec/qemu-kvm -S -M rhel6.0.0 -cpu Nehalem -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name nfs_vm -uuid 5f32f27e-8dcf-4bea-8cfe-3bc8870c0b08 -smbios type=1,manufacturer=Red Hat,product=RHEL,version=6Server-6.1.0.2.el6,serial=1383FF0A-EE0D-11DF-B57E-E41F13CC3514_E4:1F:13:CC:35:14,uuid=5f32f27e-8dcf-4bea-8cfe-3bc8870c0b08 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/nfs_vm.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2011-07-03T09:22:22 -boot c -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x4 -drive file=/rhev/data-center/8123336e-070d-44e8-bfc3-5c0159d10ee6/5060a943-7f74-40be-a0fa-dc2620f8f514/images/60f30e4c-d051-4ea7-ac41-a89650d8c8b1/1a37bdef-d5f1-44a7-98e5-1d93e0dd6ca4,if=none,id=drive-virtio-disk0,format=raw,serial=a7-ac41-a89650d8c8b1,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:99:42,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/nfs_vm.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -usb -device usb-tablet,id=input0 -vnc 0:0,password -k en-us -vga cirrus
root     10284  0.0  0.0      0     0 ?        S    12:22   0:00 [vhost-10263]
root     12818  0.0  0.0 103236   856 pts/8    S+   12:24   0:00 grep 10263
[root@stone-vds1 ~]# 
[root@stone-vds1 ~]# ps -Zww  grep 10263
LABEL                             PID TTY      STAT   TIME COMMAND
system_u:system_r:svirt_t:s0:c896,c1002 10263 ? Rl    0:20 /usr/libexec/qemu-kvm -S -M rhel6.0.0 -cpu Nehalem -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name nfs_vm -uuid 5f32f27e-8dcf-4bea-8cfe-3bc8870c0b08 -smbios type=1,manufacturer=Red Hat,product=RHEL,version=6Server-6.1.0.2.el6,serial=1383FF0A-EE0D-11DF-B57E-E41F13CC3514_E4:1F:13:CC:35:14,uuid=5f32f27e-8dcf-4bea-8cfe-3bc8870c0b08 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/nfs_vm.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2011-07-03T09:22:22 -boot c -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x4 -drive file=/rhev/data-center/8123336e-070d-44e8-bfc3-5c0159d10ee6/5060a943-7f74-40be-a0fa-dc2620f8f514/images/60f30e4c-d051-4ea7-ac41-a89650d8c8b1/1a37bdef-d5f1-44a7-98e5-1d93e0dd6ca4,if=none,id=drive-virtio-disk0,format=raw,serial=a7-ac41-a89650d8c8b1,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:99:42,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/nfs_vm.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -usb -device usb-tablet,id=input0 -vnc 0:0,password -k en-us -vga cirrus LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 12484 pts/8 R+   0:00 ps -Zww grep 10263 HOSTNAME=stone-vds1.qa.lab.tlv.redhat.com SELINUX_ROLE_REQUESTED= TERM=xterm SHELL=/bin/bash HISTSIZE=1000 SSH_CLIENT=10.35.3.121 55114 22 SELINUX_USE_CURRENT_RANGE= QTDIR=/usr/lib64/qt-3.3 QTINC=/usr/lib64/qt-3.3/include SSH_TTY=/dev/pts/8 USER=root LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36: MAIL=/var/spool/mail/root PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin PWD=/root XMODIFIERS=@im=none LANG=en_US.utf8 SELINUX_LEVEL_REQUESTED= HISTCONTROL=ignoredups SHLVL=1 HOME=/root LOGNAME=root QTLIB=/usr/lib64/qt-3.3/lib CVS_RSH=ssh SSH_CONNECTION=10.35.3.121 55114 10.35.115.14 22 LESSOPEN=|/usr/bin/lesspipe.sh %s G_BROKEN_FILENAMES=1 _=/bin/ps

This bug blocks our RHEVM suspend VM automation ( migration to file ).

I managed to reproduce this additional issue locally and prepared a patch which I just sent upstream for review: https://www.redhat.com/archives/libvir-list/2011-July/msg00429.html

Pushed upstream as v0.9.3-38-g2f4d249:

commit 2f4d2496a88055a8343b3efca618522da8715d92
Author: Jiri Denemark <jdenemar>
Date:   Fri Jul 8 14:11:01 2011 +0200

    util: Don't try to fchown files opened as non-root
    
    When virFileOpenAs is called with VIR_FILE_OPEN_AS_UID flag and uid/gid
    different from root/root while libvirtd is running as root, we fork a
    new child, change its effective UID/GID to uid/gid and run
    virFileOpenAsNoFork. It doesn't make any sense to fchown() the opened
    file in this case since we already know that uid/gid can access the file
    when open succeeds and one of the following situations may happen:
    
    - the file is already owned by uid/gid and we skip fchown even before
      this patch
    - the file is owned by uid but not gid because it was created in a
      directory with SETGID set, in which case it is desirable not to change
      the group
    - the file may be owned by a completely different user and/or group
      because it was created on a root-squashed or even all-squashed NFS
      filesystem, in which case fchown would most likely fail anyway

Tested on libvirt-0.9.3-2.el6.x86_64:

12:51:03.363: 573: debug : virDomainSave:2252 : dom=0x7fe8f411f2a0, (VM: name=NFS-FOR-TEMP, uuid=e1f626a1-3328-4d4e-be7d-0fdd036842ea), to=/rhev/data-center/400b08e6-8d9d-
4470-91b9-3b8076351314/65ff1e27-ef02-4714-9b86-cfdeee6a73c1/images/e2d8f7f1-dd84-40f7-846c-6074e79bc795/b570b726-fea8-43d1-94ff-e675f0675403
12:51:03.364: 573: error : virFileOpenAsNoFork:659 : failed to create file '/rhev/data-center/400b08e6-8d9d-4470-91b9-3b8076351314/65ff1e27-ef02-4714-9b86-cfdeee6a73c1/ima
ges/e2d8f7f1-dd84-40f7-846c-6074e79bc795/b570b726-fea8-43d1-94ff-e675f0675403': Permission denied
12:51:03.365: 573: debug : virStorageFileIsSharedFSType:980 : Check if path /rhev/data-center/400b08e6-8d9d-4470-91b9-3b8076351314/65ff1e27-ef02-4714-9b86-cfdeee6a73c1/ima
ges/e2d8f7f1-dd84-40f7-846c-6074e79bc795/b570b726-fea8-43d1-94ff-e675f0675403 with FS magic 26985 is shared
12:51:03.403: 573: error : virFileOpenAsNoFork:659 : failed to create file '/rhev/data-center/400b08e6-8d9d-4470-91b9-3b8076351314/65ff1e27-ef02-4714-9b86-cfdeee6a73c1/ima
ges/e2d8f7f1-dd84-40f7-846c-6074e79bc795/b570b726-fea8-43d1-94ff-e675f0675403': Permission denied
12:51:03.403: 573: error : qemudDomainSaveFlag:2259 : Error from child process creating '/rhev/data-center/400b08e6-8d9d-4470-91b9-3b8076351314/65ff1e27-ef02-4714-9b86-cfd
eee6a73c1/images/e2d8f7f1-dd84-40f7-846c-6074e79bc795/b570b726-fea8-43d1-94ff-e675f0675403': Permission denied

Final fix which also makes it working for David was sent upstream: https://www.redhat.com/archives/libvir-list/2011-July/msg00741.html

Okay commited upstream as 3e75c5ec85de98d92624f130148dd44a63ea27cb

Daniel

Verified on libvirt-0.9.3-3.el6.

According to comment 18, move this to VERIFIED.

tested with libvirt-0.9.4-0rc1.1.el6.x86_64
1. Suspending the VM by clicking suspend button in RHEVM.

2. See log in RHEL/VDSM.

# cat /var/log/vdsm/vdsm.log

All of the record for suspending vm are without error messages . So keep the bug status as VERIFIED

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
If NFS storage was configured to be only accessible by users from a supplementary group of the user whose identity is used to run QEMU processes (usually "qemu" user), libvirtd can fail to access or create files on that storage. With this update, libvirtd properly initializes supplementary groups when changing identity to qemu user and group which allows it to access/create such files.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1513.html