Red Hat Bugzilla – Bug 624447
[vdsm] [libvirt] permission error on run vm task when using NFS storage (libvirt log!)
Last modified: 2014-01-12 19:46:52 EST
Description of problem:
watching libvirt logs I see permission errors when running vm (on each run vm command scenarios).
currently. there is no actual affect on the system, though I would like to know why it occurs, if we need to deal with it, and if not, remove it from log.
16:50:58.366: warning : virDomainDiskDefForeachPath:7654 : Ignoring open failure on /rhev/data-center│pU
Th│96e5-6544ba38821e/6224acd4-d93a-435c-ba0c-c9ee4297add6: Permission denied │es
po│16:50:58.376: info : qemudDispatchSignalEvent:397 : Received unexpected signal 17 │}
Th│16:50:58.401: info : brProbeVnetHdr:449 : Enabling IFF_VNET_HDR │s:
(│16:50:58.411: info : qemudDispatchSignalEvent:397 : Received unexpected signal 17 │
s:│40c6ac8c::_readPauseCode unsupported by libvirt vm
1) run new vm
Please verify if the problem's gone with virt_use_nfs=on.
Since it's libvirt.log-related, why is it open on vdsm? I suspect that behavior will be the same if you use virsh directly.
no, still hit same behivior.
look at the attached log.
11:15:11.051: debug : qemuSetupDiskPathAllow:3539 : Process path /rhev/data-center/02f12ae5-7121-4e67-b19c-b2c293bc6206/0e3f4ad3-87fb-4c67-b660-50b9e4584216/images/a26c9ff2-24be-4cf9-b64e-84e3b0c0b9e9/9faac66d-5ca
2-4bad-8b2c-d5b4210adcb4 for disk
11:15:11.051: debug : qemuSetupDiskPathAllow:3545 : Ignoring EINVAL for /rhev/data-center/02f12ae5-7121-4e67-b19c-b2c293bc6206/0e3f4ad3-87fb-4c67-b660-50b9e4584216/images/a26c9ff2-24be-4cf9-b64e-84e3b0c0b9e9/9faac
11:15:11.052: warning : virDomainDiskDefForeachPath:7654 : Ignoring open failure on /rhev/data-center/02f12ae5-7121-4e67-b19c-b2c293bc6206/0e3f4ad3-87fb-4c67-b660-50b9e4584216/images/a26c9ff2-24be-4cf9-b64e-84e3b0
c0b9e9/9faac66d-5ca2-4bad-8b2c-d5b4210adcb4: Permission denied
virt_use_nfs --> on
[root@magenta-vdsc ~]# getenforce
thanks Haim, but what about my second question: why did you open this minor libvirt logging issue on vdsm? Without compelling explanation I would have to close this bug, or move it libvirt.
so please move to libvirt. it reproduces all time.
None of these logs are error logs; the one mentioned in the summary (and the one message common between both comments that list log messages) is warning level.
It is reporting that libvirt was unable to open a backing store file while trying to determine the volume type for informational purposes. This happens because that code runs as root, but the backing store is on a root-squashed disk owned by a different uid (not readable by root).
We kept these warning messages because while in VDSM usage they may be expected, in other usage scenarios they are a sign of real problems. Without the warning messages we'd not be able to identify real problems that arise in non-VDSM usage.
The packages of comment 0 are too old that i can't use them to register to the RHEVM.
But, i used newer packages:
There are no such errors like comment 0 and comment 3.
So i need your help to confirm whether issue still exists in your env
Note that it is important for the NFS share to be mounted with root_squash, and the qemu user (in /etc/libvirt/qemu.conf) to be set to something other than "root", in order to see this bug (presumably, the files on the share would be owned by the qemu user:group).
Although I can't confirm right now that the message is still output, I can confirm that the code that would cause those two messages (the
ignoring ..." messages) is still present in the source.
(zhpeng: never mind, I set up my own root-squash NFS system and saw that both messages still appear)
I recently posted patches upstream that eliminate the Warning message (but not the debug message, since it's only seen when the log level is set to debug, and fixing that to not fail wouldn't solve any problem anyway). It's still stuck in review, but will hopefully be in libvirt-0.9.10 so that it can be in the next rebase for RHEL.
A new version of the upstream patches, once again waiting for review/ACK:
Patches to eliminate the warning message have been pushed upstream, so they will be in 0.9.10:
Author: Laine Stump <email@example.com>
Date: Fri Jan 13 15:26:45 2012 -0500
util: refactor virFileOpenAs
virFileOpenAs previously would only try opening a file as the current
user, or as a different user, but wouldn't try both methods in a
single call. This made it cumbersome to use as a replacement for
open(2). Additionally, it had a lot of historical baggage that led to
it being difficult to understand.
This patch refactors virFileOpenAs in the following ways:
* reorganize the code so that everything dealing with both the parent
and child sides of the "fork+setuid+setgid+open" method are in a
separate function. This makes the public function easier to understand.
* Allow a single call to virFileOpenAs() to first attempt the open as
the current user, and if that fails to automatically re-try after
doing fork+setuid (if deemed appropriate, i.e. errno indicates it
would now be successful, and the file is on a networkFS). This makes
it possible (in many, but possibly not all, cases) to drop-in
virFileOpenAs() as a replacement for open(2).
(NB: currently qemuOpenFile() calls virFileOpenAs() twice, once
without forking, then again with forking. That unfortunately can't
be changed without at least some discussion of the ramifications,
because the requested file permissions are different in each case,
which is something that a single call to virFileOpenAs() can't deal
* Add a flag so that any fchown() of the file to a different uid:gid
is explicitly requested when the function is called, rather than it
being implied by the presence of the O_CREAT flag. This just makes
for less subtle surprises to consumers. (Commit
b1643dc15c5de886fefe56ad18608d65f1325a2c added the check for O_CREAT
before forcing ownership. This patch just makes that restriction
* If either the uid or gid is specified as "-1", virFileOpenAs will
interpret this to mean "the current [gu]id".
All current consumers of virFileOpenAs should retain their present
behavior (after a few minor changes to their setup code and
Author: Laine Stump <firstname.lastname@example.org>
Date: Thu Jan 12 13:24:45 2012 -0500
qemu: eliminate "Ignoring open failure" when using root-squash NFS
This eliminates the warning message reported in:
It was caused by a failure to open an image file that is not
accessible by root (the uid libvirtd is running as) because it's on a
root-squash NFS share, owned by a different user, with permissions of
660 (or maybe 600).
The solution is to use virFileOpenAs() rather than open(). The
codepath that generates the error is during qemuSetupDiskCGroup(), but
the actual open() is in a lower-level generic function called from
many places (virDomainDiskDefForeachPath), so some other pieces of the
code were touched just to add dummy (or possibly useful) uid and gid
Eliminating this warning message has the nice side effect that the
requested operation may even succeed (which in this case isn't
necessary, but shouldn't hurt anything either).
Verify this bug with :
1) update libvirt pkg to libvirt-0.9.10-1.el6.x86_64
2) restart libvirtd service
#initctl restart libvirtd
3) in the rhev-m start a NFS guest
4) check the libvirtd.log
#cat /var/log/libvirtd.log |grep -i permission
Note : there is not any error about permission
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
In some configurations, libvirt or RHEV users would see log messages such as this:
"warning : virDomainDiskDefForeachPath:7654 : Ignoring open
failure on xxx.xxx"
These messages were harmless, but sometimes caused concern. They should no longer occur unless there really is a problem that needs handling.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.