Thanks for the report. You probably are missing the --with-qemu-user/group libvirt configure options. This is what distribution packages use to tell libvirt to run qemu binaries as qemu:qemu, and change file ownership to match. Otherwise the default is root:root, which is kind of problematic because libvirt will drop privs for the emulator which causes issues.

$ ./configure --help | grep qemu
  --with-qemu             add QEMU/KVM support [default=yes]
  --with-qemu-user        username to run QEMU system instance as
  --with-qemu-group       groupname to run QEMU system instance as


So, pass: --with-qemu-user qemu --with-qemu-group qemu

Please reopen if I'm wrong

Thanks Cole for looking into this.

Actually libvirt is compiled with --with-qemu-user=nobody --with-qemu-group=kvm. So that should essentially do the same as qemu:qemu, don't you think?

Set it in /etc/libvirt/qemu.conf - that will override whatever is compiled in.

Sorry I didn't read closely enough. qemu:qemu is what fedora uses. So if you passed --with-qemu-user=nobody --with-qemu-group=kvm to ./configure, and that is what matches Arch linux package config, I think it should be working. Do you have any custom settings in /etc/libvirt/qemu.conf that might be changing behavior unexpectedly?

Hi, 
thanks for any help. 

Libvirt is configured with:

./configure --prefix=/usr --localstatedir=/var --sysconfdir=/etc \
			--with-openssl --without-sasl \
			--without-uml --without-vbox --without-lxc \
                        --without-esx  --without-vmware --without-openvz \
			--without-firewalld --without-network \
                        --with-interface --with-virtualport --with-macvtap \
			--disable-nls --without-avahi --without-test-suit \
                        --without-dbus \
			--with-qemu-user=nobody --with-qemu-group=kvm \
			--with-storage-dir --without-storage-fs \
                        --without-storage-lvm  --without-storage-iscsi \
			--without-storage-scsi --without-storage-mpath \
                        --without-storage-disk --without-storage-rbd \
                        --without-storage-sheepdog --without-storage-gluster \
                        --without-storage-zfs

The I posted the qemu.conf there:

http://nopaste.ipfire.org/view/4Aesq26x


Qemu runs with the right of nobody:kvm when it is started by libvirt. (htop says also that qemu runs as user nobody http://nopaste.ipfire.org/view/vCp1urKa) So the option works, but a qemu process with the rights of nobody:kvm can not access an image file which is owned by root:root.

On Arch Linux the ownership of the file is changed by libvirt to noboy:kvm on the system (IPFire) for it I compiled and packaged libvirt, libvirt seems to be unable to change the ownership of the image file and qemu failed with the error in my first post.

Regards Jonatan

Hi,
i uploaded the debug log to 

http://people.ipfire.org/~jschlag/1363864/1_libvirtd.log

The UID of the user nobody is 99, the GID of the group kvm is 1011.

Following the log the ownership is changed but why is the file still owned by root:root?

The daemon run as root and there are also no selinux which prevent the access to the file.

Regards Jonatan

Apparmor is also disabled or better not installed.

We have narrowed some things down here. We found out that virStorageFileSupportsSecurityDriver() returns false for our storage driver (which is storage dir) although we expect it to return true.

virStorageFileSupportsSecurityDriver goes to the storage backend and checks if storageFileChown is set. That is the case for all storage dir and storage fs backend types.

Could anyone confirm that this behaviour is correct here or if our assumption that the wrong value is returned here is correct.

> static int
> qemuSecurityChownCallback(virStorageSourcePtr src,
>                           uid_t uid,
>                           gid_t gid)
> {
>    struct stat sb;
>    int save_errno = 0;
>    int ret = -1;
>
>    if (!virStorageFileSupportsSecurityDriver(src))
>        return 0;

(In reply to Michael Tremer from comment #8)
> We have narrowed some things down here. We found out that
> virStorageFileSupportsSecurityDriver() returns false for our storage driver
> (which is storage dir) although we expect it to return true.

Our storage driver? Are you writing your own storage driver?

> 
> virStorageFileSupportsSecurityDriver goes to the storage backend and checks
> if storageFileChown is set. That is the case for all storage dir and storage
> fs backend types.
> 
> Could anyone confirm that this behaviour is correct here or if our
> assumption that the wrong value is returned here is correct.
> 
> > static int
> > qemuSecurityChownCallback(virStorageSourcePtr src,
> >                           uid_t uid,
> >                           gid_t gid)
> > {
> >    struct stat sb;
> >    int save_errno = 0;
> >    int ret = -1;
> >
> >    if (!virStorageFileSupportsSecurityDriver(src))
> >        return 0;


I can't confirm what the return value should be unless I know what src is. From the upstream discussion I've learned that you're using storage pool and in the domain XML there's storage volume from that pool. However, depending on what the /data/path/to/your/images is (e.g. is it a NFS mount?), what the storage pool XML and domain XML look like this might return true and false.

BTW: can you attach the debugger once again, try to step into virStorageFileSupportsSecurityDriver() and look what backend points to? I'd reproduce myself, but I don't have the configuration you have.

Hi,
thank you very much, Michal Privoznik.

I posted our domain xml file here:

http://nopaste.ipfire.org/view/QrUaJOks

the storage xml file is available here:

http://nopaste.ipfire.org/view/QVYDDjGV.

The storage pool is directly  located on a 1 TN hard drive we do not use NFS samba or other network protocols.

GDB output: http://nopaste.ipfire.org/view/6MhSKdPB

We found out that backend is actually not set and hence virStorageFileSupportsSecurityDriver is returning false.

> 2716 bool
> 2717 virStorageFileSupportsSecurityDriver(virStorageSourcePtr src)
> 2718 {
> 2719     int actualType;
> 2720     virStorageFileBackendPtr backend;
> 2721 
> 2722     if (!src)
> 2723         return false;
> 2724     actualType = virStorageSourceGetActualType(src);
> 2725 
> 2726     if (src->drv) {
> 2727         backend = src->drv->backend;
> 2728     } else {
> 2729         if (!(backend = virStorageFileBackendForTypeInternal(actualType,
> 2730                                                              src->protocol,
> 2731                                                              false)))
> 2732             return false;
> 2733     }
> 2734 
> 2735     return !!backend->storageFileChown;
> 2736 }

libvirtd jumps directly into line 2729 where actualType is set to 1 (which we assume is VIR_STORAGE_TYPE_FILE). According to the configuration this should actually be VIR_STORAGE_TYPE_DIR, we *think*.

Right here we are stuck again. Please let us know if this information helps you to get closer to the problem and if we could provide any further information if needed.

(In reply to Michael Tremer from comment #11)
> GDB output: http://nopaste.ipfire.org/view/6MhSKdPB
> 
> We found out that backend is actually not set and hence
> virStorageFileSupportsSecurityDriver is returning false.
> 
> > 2716 bool
> > 2717 virStorageFileSupportsSecurityDriver(virStorageSourcePtr src)
> > 2718 {
> > 2719     int actualType;
> > 2720     virStorageFileBackendPtr backend;
> > 2721 
> > 2722     if (!src)
> > 2723         return false;
> > 2724     actualType = virStorageSourceGetActualType(src);
> > 2725 
> > 2726     if (src->drv) {
> > 2727         backend = src->drv->backend;
> > 2728     } else {
> > 2729         if (!(backend = virStorageFileBackendForTypeInternal(actualType,
> > 2730                                                              src->protocol,
> > 2731                                                              false)))
> > 2732             return false;
> > 2733     }
> > 2734 
> > 2735     return !!backend->storageFileChown;
> > 2736 }
> 
> libvirtd jumps directly into line 2729 where actualType is set to 1 (which
> we assume is VIR_STORAGE_TYPE_FILE). According to the configuration this
> should actually be VIR_STORAGE_TYPE_DIR, we *think*.
> 
> Right here we are stuck again. Please let us know if this information helps
> you to get closer to the problem and if we could provide any further
> information if needed.

I believe this is a packaging issue. I mean, you say you are building the libvirt on your own and it does not work. Moreover, in comment 5 you give us the configure command line. Well, if you enable storage fs there, everything will work again:

./configure --with-storage-fs ...

But I'll rather wait for confirmation before closing this one.

The thing is, in the configure phase you are deliberately disabling VIR_STORAGE_TYPE_FILE backend to storage driver. Therefore, virStorageFileSupportsSecurityDriver() is not fiding any backend for the type of source configured in domain XML and thus do not relabel it.

Hi,
Sorry for the delay, but my last 2 weeks were full with other work.

If I enable storage fs everything works fine, thank you very much.

This solves my issue, but I think there is still a bug because libvirt do not log that it fails to change the ownership.

There should be something like

info: Unable to change the ownership because libvirt was compiled without storage fs.

In the momenthe log is:

2016-08-04 09:54:23.203+0000: 31063: debug : qemuProcessStart:4863 : Setting domain security labels
2016-08-04 09:54:23.203+0000: 31063: info : virSecurityDACSetOwnershipInternal:248 : Setting DAC user and group on '/data/hdd1/libvirt/iso/KNOPPIX_V7.6.1DVD-2016-01-16-DE.iso' to '99:1011'
2016-08-04 09:54:23.203+0000: 31063: info : virSecurityDACSetOwnershipInternal:248 : Setting DAC user and group on '/data/hdd1/libvirt/images/test.img' to '99:1011'
2016-08-04 09:54:23.203+0000: 31063: debug : qemuProcessStart:4892 : Labelling done, completing handshake to child

But this problem should be tracked in another bug report.

So Thank you very much for every help.

Regards Jonatan

Closing per comment 13.