Bug 624447 - [vdsm] [libvirt] permission error on run vm task when using NFS storage (libvirt log!)
Summary: [vdsm] [libvirt] permission error on run vm task when using NFS storage (libv...
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.1
Hardware: All
OS: Linux
Target Milestone: rc
: ---
Assignee: Laine Stump
QA Contact: Virtualization Bugs
Depends On:
TreeView+ depends on / blocked
Reported: 2010-08-16 13:54 UTC by Haim
Modified: 2014-01-13 00:46 UTC (History)
14 users (show)

Fixed In Version: libvirt-0.9.10-1.el6
Doc Type: Bug Fix
Doc Text:
In some configurations, libvirt or RHEV users would see log messages such as this: "warning : virDomainDiskDefForeachPath:7654 : Ignoring open failure on xxx.xxx" These messages were harmless, but sometimes caused concern. They should no longer occur unless there really is a problem that needs handling.
Clone Of:
Last Closed: 2012-06-20 06:24:18 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2012:0748 0 normal SHIPPED_LIVE Low: libvirt security, bug fix, and enhancement update 2012-06-19 19:31:38 UTC

Description Haim 2010-08-16 13:54:30 UTC
Description of problem:

watching libvirt logs I see permission errors when running vm (on each run vm command scenarios). 
currently. there is no actual affect on the system, though I would like to know why it occurs, if we need to deal with it, and if not, remove it from log. 

16:50:58.366: warning : virDomainDiskDefForeachPath:7654 : Ignoring open failure on /rhev/data-center│pU
Th│96e5-6544ba38821e/6224acd4-d93a-435c-ba0c-c9ee4297add6: Permission denied                            │es
po│16:50:58.376: info : qemudDispatchSignalEvent:397 : Received unexpected signal 17                    │}
Th│16:50:58.401: info : brProbeVnetHdr:449 : Enabling IFF_VNET_HDR                                      │s:
 (│16:50:58.411: info : qemudDispatchSignalEvent:397 : Received unexpected signal 17                    │
Th│Thread-2962::WARNING::2010-08-16 16:50:58,672::libvirtvm::633::vds.vmlog.68fccbbf-cad6-4d94-aebc-f5fc│in
s:│40c6ac8c::_readPauseCode unsupported by libvirt vm


1) run new vm

Comment 2 Dan Kenigsberg 2010-09-06 12:13:46 UTC
Please verify if the problem's gone with virt_use_nfs=on.

Since it's libvirt.log-related, why is it open on vdsm? I suspect that behavior will be the same if you use virsh directly.

Comment 3 Haim 2010-09-20 08:50:25 UTC
no, still hit same behivior.
look at the attached log. 

11:15:11.051: debug : qemuSetupDiskPathAllow:3539 : Process path /rhev/data-center/02f12ae5-7121-4e67-b19c-b2c293bc6206/0e3f4ad3-87fb-4c67-b660-50b9e4584216/images/a26c9ff2-24be-4cf9-b64e-84e3b0c0b9e9/9faac66d-5ca
2-4bad-8b2c-d5b4210adcb4 for disk
11:15:11.051: debug : qemuSetupDiskPathAllow:3545 : Ignoring EINVAL for /rhev/data-center/02f12ae5-7121-4e67-b19c-b2c293bc6206/0e3f4ad3-87fb-4c67-b660-50b9e4584216/images/a26c9ff2-24be-4cf9-b64e-84e3b0c0b9e9/9faac
11:15:11.052: warning : virDomainDiskDefForeachPath:7654 : Ignoring open failure on /rhev/data-center/02f12ae5-7121-4e67-b19c-b2c293bc6206/0e3f4ad3-87fb-4c67-b660-50b9e4584216/images/a26c9ff2-24be-4cf9-b64e-84e3b0
c0b9e9/9faac66d-5ca2-4bad-8b2c-d5b4210adcb4: Permission denied

virt_use_nfs --> on

[root@magenta-vdsc ~]#  getenforce 

Comment 4 Dan Kenigsberg 2010-09-25 22:16:22 UTC
thanks Haim, but what about my second question: why did you open this minor libvirt logging issue on vdsm? Without compelling explanation I would have to close this bug, or move it libvirt.

Comment 5 Haim 2010-09-26 15:10:43 UTC
so please move to libvirt. it reproduces all time.

Comment 6 Laine Stump 2011-01-31 20:44:54 UTC
None of these logs are error logs; the one mentioned in the summary (and the one message common between both comments that list log messages) is warning level.

It is reporting that libvirt was unable to open a backing store file while trying to determine the volume type for informational purposes. This happens because that code runs as root, but the backing store is on a root-squashed disk owned by a different uid (not readable by root).

Comment 7 Daniel Berrangé 2011-02-01 10:16:06 UTC
We kept these warning messages because while in VDSM usage they may be expected, in other usage scenarios they are a sign of real problems. Without the warning messages we'd not be able to identify real problems that arise in non-VDSM usage.

Comment 13 zhpeng 2012-01-04 10:13:31 UTC
The packages of comment 0 are too old that i can't use them to register to the RHEVM.

But, i used newer packages:

with virt_use_nfs=on

There are no such errors like comment 0 and comment 3.

So i need your help to confirm whether issue still exists in your env


Comment 16 Laine Stump 2012-01-04 18:30:13 UTC
Note that it is important for the NFS share to be mounted with root_squash, and the qemu user (in /etc/libvirt/qemu.conf) to be set to something other than "root", in order to see this bug (presumably, the files on the share would be owned by the qemu user:group).

Although I can't confirm right now that the message is still output, I can confirm that the code that would cause those two messages (the 
ignoring ..." messages) is still present in the source.

Comment 20 Laine Stump 2012-01-25 18:07:01 UTC
(zhpeng: never mind, I set up my own root-squash NFS system and saw that both messages still appear)

I recently posted patches upstream that eliminate the Warning message (but not the debug message, since it's only seen when the log level is set to debug, and fixing that to not fail wouldn't solve any problem anyway). It's still stuck in review, but will hopefully be in libvirt-0.9.10 so that it can be in the next rebase for RHEL.


Comment 21 Laine Stump 2012-02-03 15:56:34 UTC
A new version of the upstream patches, once again waiting for review/ACK:


Comment 22 Laine Stump 2012-02-03 21:54:16 UTC
Patches to eliminate the warning message have been pushed upstream, so they will be in 0.9.10:

commit 90e4d681bc30074466912e9ae733b0bf0b9d5a03
Author: Laine Stump <laine@laine.org>
Date:   Fri Jan 13 15:26:45 2012 -0500

    util: refactor virFileOpenAs
    virFileOpenAs previously would only try opening a file as the current
    user, or as a different user, but wouldn't try both methods in a
    single call. This made it cumbersome to use as a replacement for
    open(2). Additionally, it had a lot of historical baggage that led to
    it being difficult to understand.
    This patch refactors virFileOpenAs in the following ways:
    * reorganize the code so that everything dealing with both the parent
      and child sides of the "fork+setuid+setgid+open" method are in a
      separate function. This makes the public function easier to understand.
    * Allow a single call to virFileOpenAs() to first attempt the open as
      the current user, and if that fails to automatically re-try after
      doing fork+setuid (if deemed appropriate, i.e. errno indicates it
      would now be successful, and the file is on a networkFS). This makes
      it possible (in many, but possibly not all, cases) to drop-in
      virFileOpenAs() as a replacement for open(2).
      (NB: currently qemuOpenFile() calls virFileOpenAs() twice, once
      without forking, then again with forking. That unfortunately can't
      be changed without at least some discussion of the ramifications,
      because the requested file permissions are different in each case,
      which is something that a single call to virFileOpenAs() can't deal
    * Add a flag so that any fchown() of the file to a different uid:gid
      is explicitly requested when the function is called, rather than it
      being implied by the presence of the O_CREAT flag. This just makes
      for less subtle surprises to consumers. (Commit
      b1643dc15c5de886fefe56ad18608d65f1325a2c added the check for O_CREAT
      before forcing ownership. This patch just makes that restriction
      more explicit.)
    * If either the uid or gid is specified as "-1", virFileOpenAs will
      interpret this to mean "the current [gu]id".
    All current consumers of virFileOpenAs should retain their present
    behavior (after a few minor changes to their setup code and

commit c18a88ac489c5356734bb7cff59f8fc73c9d1227
Author: Laine Stump <laine@laine.org>
Date:   Thu Jan 12 13:24:45 2012 -0500

    qemu: eliminate "Ignoring open failure" when using root-squash NFS
    This eliminates the warning message reported in:
    It was caused by a failure to open an image file that is not
    accessible by root (the uid libvirtd is running as) because it's on a
    root-squash NFS share, owned by a different user, with permissions of
    660 (or maybe 600).
    The solution is to use virFileOpenAs() rather than open(). The
    codepath that generates the error is during qemuSetupDiskCGroup(), but
    the actual open() is in a lower-level generic function called from
    many places (virDomainDiskDefForeachPath), so some other pieces of the
    code were touched just to add dummy (or possibly useful) uid and gid
    Eliminating this warning message has the nice side effect that the
    requested operation may even succeed (which in this case isn't
    necessary, but shouldn't hurt anything either).

Comment 25 Huang Wenlong 2012-02-14 08:32:15 UTC
Verify this bug with : 

1)  update libvirt pkg to libvirt-0.9.10-1.el6.x86_64

2)  restart libvirtd service 

#initctl restart libvirtd 

3)  in the rhev-m  start a NFS guest  

4) check the libvirtd.log 

#cat /var/log/libvirtd.log |grep -i permission  

Note : there is not any error about permission

Comment 26 Laine Stump 2012-05-08 19:24:10 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    New Contents:
In some configurations, libvirt or RHEV users would see log messages such as this:

 "warning : virDomainDiskDefForeachPath:7654 : Ignoring open
failure on xxx.xxx"

These messages were harmless, but sometimes caused concern. They should no longer occur unless there really is a problem that needs handling.

Comment 28 errata-xmlrpc 2012-06-20 06:24:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.