Bug 707257
| Summary: | libvirt fails to save VM into NFS | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Jiri Denemark <jdenemar> | ||||
| Component: | libvirt | Assignee: | Jiri Denemark <jdenemar> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 6.1 | CC: | abaron, cpelland, dallan, danken, dnaori, dyuan, iheim, mgoldboi, mprivozn, nzhang, oramraz, rvaknin, rwu, vbian, veillard, zpeng | ||||
| Target Milestone: | rc | Keywords: | Regression, TestBlocker | ||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | storage | ||||||
| Fixed In Version: | libvirt-0.9.3-3.el6 | Doc Type: | Bug Fix | ||||
| Doc Text: |
If NFS storage was configured to be only accessible by users from a supplementary group of the user whose identity is used to run QEMU processes (usually "qemu" user), libvirtd can fail to access or create files on that storage. With this update, libvirtd properly initializes supplementary groups when changing identity to qemu user and group which allows it to access/create such files.
|
Story Points: | --- | ||||
| Clone Of: | 702411 | Environment: | |||||
| Last Closed: | 2011-12-06 11:11:06 UTC | Type: | --- | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 702411 | ||||||
| Bug Blocks: | 565939 | ||||||
| Attachments: |
|
||||||
This should be fixed by the libvirt-0.9.2-1.el6 rebase I tested it with libvirt-0.9.2-1.el6.x86_64 and vdsm-4.9-78.el6.x86_64, it doesn't work, I still get the Permission Denied error.
Thread-144::ERROR::2011-06-29 11:04:54,920::vm::230::vm.Vm::(run) vmId=`5cc7b29d-2e7f-494e-b3e7-b0239c0f8ec8`::Traceback (most recent call last):
File "/usr/share/vdsm/vm.py", line 222, in run
self._startUnderlyingMigration()
File "/usr/share/vdsm/libvirtvm.py", line 252, in _startUnderlyingMigration
self._vm._dom.save(fname)
File "/usr/share/vdsm/libvirtvm.py", line 302, in f
ret = attr(*args, **kwargs)
File "/usr/share/vdsm/libvirtconnection.py", line 63, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib64/python2.6/site-packages/libvirt.py", line 793, in save
if ret == -1: raise libvirtError ('virDomainSave() failed', dom=self)
libvirtError: Error from child process creating '/rhev/data-center/8c2989d8-d8a1-495a-8c57-09b537a7a7b8/da6aa19a-03b0-4bd2-8aab-dabc924f8e3f/images/4ed9d520-92d3-4283-9c24-2c1bdb705265/3e8144f1-c153-432a-a92a-c94c6a690e02': Permission de
nied
Created attachment 510411 [details]
vdsm and libvirt logs
According to comment 5, the bug not fixed, move to assigned. - What are the ownerships and permissions along the /rhev/data-center/8c2989d8-d8a1-495a-8c57-09b537a7a7b8/da6aa19a-03b0-4bd2-8aab-dabc924f8e3f/images/4ed9d520-92d3-4283-9c24-2c1bdb705265/3e8144f1-c153-432a-a92a-c94c6a690e02 path? - Who owns the associated qemu-kvm process and what does id command say about that owner? - Can you also check that the operation wasn't denied by SELinux? I reproduced it again, when trying to suspend the VM it changes to Paused, then in the next trial to suspend it, suspension succeed (it occurs alternately). Ownership and permissions: -rw-rw----. 1 vdsm kvm 0 Jul 3 2011 /rhev/data-center/8123336e-070d-44e8-bfc3-5c0159d10ee6/5060a943-7f74-40be-a0fa-dc2620f8f514/images/7f53d089-370e-4bdb-a7a5-c815f9d5e008/cbeca92e-56a9-4865-9fd3-366cf70a4306 SELinux: It occurrs even without SELinux (I changed SELinux to Permissive using "setenforce 0") Process' user: [root@stone-vds1 ~]# ps -aux | grep 10263 Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ qemu 10263 14.2 1.6 793308 262152 ? Sl 12:22 0:22 /usr/libexec/qemu-kvm -S -M rhel6.0.0 -cpu Nehalem -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name nfs_vm -uuid 5f32f27e-8dcf-4bea-8cfe-3bc8870c0b08 -smbios type=1,manufacturer=Red Hat,product=RHEL,version=6Server-6.1.0.2.el6,serial=1383FF0A-EE0D-11DF-B57E-E41F13CC3514_E4:1F:13:CC:35:14,uuid=5f32f27e-8dcf-4bea-8cfe-3bc8870c0b08 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/nfs_vm.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2011-07-03T09:22:22 -boot c -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x4 -drive file=/rhev/data-center/8123336e-070d-44e8-bfc3-5c0159d10ee6/5060a943-7f74-40be-a0fa-dc2620f8f514/images/60f30e4c-d051-4ea7-ac41-a89650d8c8b1/1a37bdef-d5f1-44a7-98e5-1d93e0dd6ca4,if=none,id=drive-virtio-disk0,format=raw,serial=a7-ac41-a89650d8c8b1,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:99:42,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/nfs_vm.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -usb -device usb-tablet,id=input0 -vnc 0:0,password -k en-us -vga cirrus root 10284 0.0 0.0 0 0 ? S 12:22 0:00 [vhost-10263] root 12818 0.0 0.0 103236 856 pts/8 S+ 12:24 0:00 grep 10263 [root@stone-vds1 ~]# [root@stone-vds1 ~]# ps -Zww grep 10263 LABEL PID TTY STAT TIME COMMAND system_u:system_r:svirt_t:s0:c896,c1002 10263 ? Rl 0:20 /usr/libexec/qemu-kvm -S -M rhel6.0.0 -cpu Nehalem -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name nfs_vm -uuid 5f32f27e-8dcf-4bea-8cfe-3bc8870c0b08 -smbios type=1,manufacturer=Red Hat,product=RHEL,version=6Server-6.1.0.2.el6,serial=1383FF0A-EE0D-11DF-B57E-E41F13CC3514_E4:1F:13:CC:35:14,uuid=5f32f27e-8dcf-4bea-8cfe-3bc8870c0b08 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/nfs_vm.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2011-07-03T09:22:22 -boot c -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x4 -drive file=/rhev/data-center/8123336e-070d-44e8-bfc3-5c0159d10ee6/5060a943-7f74-40be-a0fa-dc2620f8f514/images/60f30e4c-d051-4ea7-ac41-a89650d8c8b1/1a37bdef-d5f1-44a7-98e5-1d93e0dd6ca4,if=none,id=drive-virtio-disk0,format=raw,serial=a7-ac41-a89650d8c8b1,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:99:42,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/nfs_vm.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -usb -device usb-tablet,id=input0 -vnc 0:0,password -k en-us -vga cirrus LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 12484 pts/8 R+ 0:00 ps -Zww grep 10263 HOSTNAME=stone-vds1.qa.lab.tlv.redhat.com SELINUX_ROLE_REQUESTED= TERM=xterm SHELL=/bin/bash HISTSIZE=1000 SSH_CLIENT=10.35.3.121 55114 22 SELINUX_USE_CURRENT_RANGE= QTDIR=/usr/lib64/qt-3.3 QTINC=/usr/lib64/qt-3.3/include SSH_TTY=/dev/pts/8 USER=root LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36: MAIL=/var/spool/mail/root PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin PWD=/root XMODIFIERS=@im=none LANG=en_US.utf8 SELINUX_LEVEL_REQUESTED= HISTCONTROL=ignoredups SHLVL=1 HOME=/root LOGNAME=root QTLIB=/usr/lib64/qt-3.3/lib CVS_RSH=ssh SSH_CONNECTION=10.35.3.121 55114 10.35.115.14 22 LESSOPEN=|/usr/bin/lesspipe.sh %s G_BROKEN_FILENAMES=1 _=/bin/ps This bug blocks our RHEVM suspend VM automation ( migration to file ). I managed to reproduce this additional issue locally and prepared a patch which I just sent upstream for review: https://www.redhat.com/archives/libvir-list/2011-July/msg00429.html Pushed upstream as v0.9.3-38-g2f4d249:
commit 2f4d2496a88055a8343b3efca618522da8715d92
Author: Jiri Denemark <jdenemar>
Date: Fri Jul 8 14:11:01 2011 +0200
util: Don't try to fchown files opened as non-root
When virFileOpenAs is called with VIR_FILE_OPEN_AS_UID flag and uid/gid
different from root/root while libvirtd is running as root, we fork a
new child, change its effective UID/GID to uid/gid and run
virFileOpenAsNoFork. It doesn't make any sense to fchown() the opened
file in this case since we already know that uid/gid can access the file
when open succeeds and one of the following situations may happen:
- the file is already owned by uid/gid and we skip fchown even before
this patch
- the file is owned by uid but not gid because it was created in a
directory with SETGID set, in which case it is desirable not to change
the group
- the file may be owned by a completely different user and/or group
because it was created on a root-squashed or even all-squashed NFS
filesystem, in which case fchown would most likely fail anyway
Tested on libvirt-0.9.3-2.el6.x86_64: 12:51:03.363: 573: debug : virDomainSave:2252 : dom=0x7fe8f411f2a0, (VM: name=NFS-FOR-TEMP, uuid=e1f626a1-3328-4d4e-be7d-0fdd036842ea), to=/rhev/data-center/400b08e6-8d9d- 4470-91b9-3b8076351314/65ff1e27-ef02-4714-9b86-cfdeee6a73c1/images/e2d8f7f1-dd84-40f7-846c-6074e79bc795/b570b726-fea8-43d1-94ff-e675f0675403 12:51:03.364: 573: error : virFileOpenAsNoFork:659 : failed to create file '/rhev/data-center/400b08e6-8d9d-4470-91b9-3b8076351314/65ff1e27-ef02-4714-9b86-cfdeee6a73c1/ima ges/e2d8f7f1-dd84-40f7-846c-6074e79bc795/b570b726-fea8-43d1-94ff-e675f0675403': Permission denied 12:51:03.365: 573: debug : virStorageFileIsSharedFSType:980 : Check if path /rhev/data-center/400b08e6-8d9d-4470-91b9-3b8076351314/65ff1e27-ef02-4714-9b86-cfdeee6a73c1/ima ges/e2d8f7f1-dd84-40f7-846c-6074e79bc795/b570b726-fea8-43d1-94ff-e675f0675403 with FS magic 26985 is shared 12:51:03.403: 573: error : virFileOpenAsNoFork:659 : failed to create file '/rhev/data-center/400b08e6-8d9d-4470-91b9-3b8076351314/65ff1e27-ef02-4714-9b86-cfdeee6a73c1/ima ges/e2d8f7f1-dd84-40f7-846c-6074e79bc795/b570b726-fea8-43d1-94ff-e675f0675403': Permission denied 12:51:03.403: 573: error : qemudDomainSaveFlag:2259 : Error from child process creating '/rhev/data-center/400b08e6-8d9d-4470-91b9-3b8076351314/65ff1e27-ef02-4714-9b86-cfd eee6a73c1/images/e2d8f7f1-dd84-40f7-846c-6074e79bc795/b570b726-fea8-43d1-94ff-e675f0675403': Permission denied Final fix which also makes it working for David was sent upstream: https://www.redhat.com/archives/libvir-list/2011-July/msg00741.html Okay commited upstream as 3e75c5ec85de98d92624f130148dd44a63ea27cb Daniel Verified on libvirt-0.9.3-3.el6. According to comment 18, move this to VERIFIED. tested with libvirt-0.9.4-0rc1.1.el6.x86_64 1. Suspending the VM by clicking suspend button in RHEVM. 2. See log in RHEL/VDSM. # cat /var/log/vdsm/vdsm.log All of the record for suspending vm are without error messages . So keep the bug status as VERIFIED
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
New Contents:
If NFS storage was configured to be only accessible by users from a supplementary group of the user whose identity is used to run QEMU processes (usually "qemu" user), libvirtd can fail to access or create files on that storage. With this update, libvirtd properly initializes supplementary groups when changing identity to qemu user and group which allows it to access/create such files.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2011-1513.html |
The second issue is that the operation should even fail because the directory is owned by vdsm:kvm (writable for kvm group) and we are trying to open the file as qemu:qemu while qemu user also belongs to kvm group. In other words, we don't initialize the supplementary groups for qemu user before trying to open the file. This issue is fixed upstream by v0.9.1-230-g4dd9c16 and v0.9.1-231-g5e09aea: commit 4dd9c16161ce0e73177a999bb26523a79c338cc2 Author: Jiri Denemark <jdenemar> Date: Sun May 22 17:16:44 2011 +0300 util: Keep errno set to the root error after when returning from virSetUIDGID commit 5e09aea7b07a683996863000bee0277f2e255f1c Author: Jiri Denemark <jdenemar> Date: Sun May 22 17:05:07 2011 +0300 Replace all remaining setgid/setuid calls with virSetUIDGID Two additional places need initgroups call to properly work in an environment where the UID is allowed to open/create stuff through its supplementary groups.