Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2017-March/msg00528.html

To POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2017-March/msg00284.html

I can reproduce the bug with libvirt-3.1.0-2.el7.x86_64

Verify pass with libvirt-3.2.0-1.el7.x86_64.
Test steps:
1.touch /mnt/demo
2.touch /dev/demo
3.mount --bind /mnt/demo /dev/demo
4.Start a guest and the guest can start correctly:
  #virsh start rhel7.3
5.Compile the ns_exec.c in the attachment:
 #gcc -o ns_exec ns_exec.c
6.Create a script to check the files in the /dev dir:
 #vim check.sh
  #!/bin/bash
 ls -lRZ /dev
7.Check the /dev/demo is mounted correctly in the /dev of the qemu process's namespace:
 #./ns_exec /proc/`pidof qemu-kvm`/ns/mnt `pwd`/check.sh | grep -i demo
-rw-r--r--. root root unconfined_u:object_r:etc_runtime_t:s0 demo

Created attachment 1269577 [details]
ns_exec.c

Hi, Michal,

I found the guest failed to start when mounting a dir/file under preserved mount points. I am not sure whether it needs to be fixed. Would you help to check that please? Thanks a lot.

Test steps:
1.touch /mnt/demo
2.touch /dev/shm/demo
3.mount --bind /mnt/demo /dev/shm/demo
4.Start a guest:
#virsh start q35
error: Failed to start domain q35
error: internal error: Process exited prior to exec: libvirt: QEMU Driver error : Unable to stat: /dev/shm/test: No such file or directory
5.The guest can start successfully when disable creating namespaces for qemu process.

Oh, thank you for catching that. I've proposed the patches online:

https://www.redhat.com/archives/libvir-list/2017-June/msg00504.html

Hi, Michal,

When I attached a disk with source file in the /dev dir, it failed with error:
# virsh attach-device rhel7.3 disk.xml 
error: Failed to attach device from disk.xml
error: internal error: child reported: unable to set user and group to '107:107' on '/dev/test.img': No such file or directory

Would you help to check it please? Thanks a lot.

Test steps:
1.Prepare a img in the /dev:
#qemu-img create -f qcow2 /dev/test.img 10M

2.Prepare the disk xml:
#cat disk.xml
 <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/dev/test.img'/>
      <target dev='vdc' bus='virtio'/>
    </disk>

3.Attach the disk to a running guest:
#virsh attach-device full-73 disk.xml
error: Failed to attach device from disk.xml
error: internal error: child reported: unable to set user and group to '107:107' on '/dev/test.img': No such file or directory


4.The disk with source file in the /dev dir can be attached successfully if disabled namespace.

(In reply to yafu from comment #11)
> Hi, Michal,
> 
> When I attached a disk with source file in the /dev dir, it failed with
> error:
> # virsh attach-device rhel7.3 disk.xml 
> error: Failed to attach device from disk.xml
> error: internal error: child reported: unable to set user and group to
> '107:107' on '/dev/test.img': No such file or directory
> 
> Would you help to check it please? Thanks a lot.
> 
> Test steps:
> 1.Prepare a img in the /dev:
> #qemu-img create -f qcow2 /dev/test.img 10M
> 
> 2.Prepare the disk xml:
> #cat disk.xml
>  <disk type='file' device='disk'>
>       <driver name='qemu' type='qcow2'/>
>       <source file='/dev/test.img'/>
>       <target dev='vdc' bus='virtio'/>
>     </disk>
> 
> 3.Attach the disk to a running guest:
> #virsh attach-device full-73 disk.xml
> error: Failed to attach device from disk.xml
> error: internal error: child reported: unable to set user and group to
> '107:107' on '/dev/test.img': No such file or directory
> 
> 
> 4.The disk with source file in the /dev dir can be attached successfully if
> disabled namespace.

Yeah, this is a bug in our code. The namespace implementation expects that anything under /dev is a device not a regular file. So libvirt got confused here and didn't created the file in the namespace.

(In reply to Michal Privoznik from comment #12)
> (In reply to yafu from comment #11)
> > Hi, Michal,
> > 
> > When I attached a disk with source file in the /dev dir, it failed with
> > error:
> > # virsh attach-device rhel7.3 disk.xml 
> > error: Failed to attach device from disk.xml
> > error: internal error: child reported: unable to set user and group to
> > '107:107' on '/dev/test.img': No such file or directory
> > 
> > Would you help to check it please? Thanks a lot.
> > 
> > Test steps:
> > 1.Prepare a img in the /dev:
> > #qemu-img create -f qcow2 /dev/test.img 10M
> > 
> > 2.Prepare the disk xml:
> > #cat disk.xml
> >  <disk type='file' device='disk'>
> >       <driver name='qemu' type='qcow2'/>
> >       <source file='/dev/test.img'/>
> >       <target dev='vdc' bus='virtio'/>
> >     </disk>
> > 
> > 3.Attach the disk to a running guest:
> > #virsh attach-device full-73 disk.xml
> > error: Failed to attach device from disk.xml
> > error: internal error: child reported: unable to set user and group to
> > '107:107' on '/dev/test.img': No such file or directory
> > 
> > 
> > 4.The disk with source file in the /dev dir can be attached successfully if
> > disabled namespace.
> 
> Yeah, this is a bug in our code. The namespace implementation expects that
> anything under /dev is a device not a regular file. So libvirt got confused
> here and didn't created the file in the namespace.


Thanks Michal. File a bug to track this issue:
https://bugzilla.redhat.com/show_bug.cgi?id=1462060

I've just pushed the patches upstream:

commit 6451b55ec3d801bb03e912b0811408cf82cfc880 (HEAD -> master, origin/master, origin/HEAD, qemu_ns)
Author:     Michal Privoznik <mprivozn>
AuthorDate: Mon Jun 12 16:44:45 2017 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Jun 16 14:38:49 2017 +0200

    qemuDomainGetPreservedMounts: Fix suffixes for corner cases
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1431112
    
    Imagine a FS mounted on /dev/blah/blah2. Our process of creating
    suffix for temporary location where all the mounted filesystems
    are moved is very simplistic. We want:
    
    /var/run/libvirt/qemu/$domName.$suffix\
    
    were $suffix is just the mount point path stripped of the "/dev/"
    prefix. For instance:
    
    /var/run/libvirt/qemu/fedora.mqueue  for /dev/mqueue
    /var/run/libvirt/qemu/fedora.pts     for /dev/pts
    
    and so on. Now if we plug /dev/blah/blah2 into the example we see
    some misbehaviour:
    
    /var/run/libvirt/qemu/fedora.blah/blah2
    
    Well, misbehaviour if /dev/blah/blah2 is a file, because in that
    case we call virFileTouch() instead of virFileMakePath().
    The solution is to replace all the slashes in the suffix with say
    dots. That way we don't have to care about nested directories.
    IOW, the result we want for given example is:
    
    /var/run/libvirt/qemu/fedora.blah.blah2
    
    Signed-off-by: Michal Privoznik <mprivozn>
    Reviewed-by: John Ferlan <jferlan>

commit cdd9205dfffa3aaed935446a41f0d2dd1357c268
Author:     Michal Privoznik <mprivozn>
AuthorDate: Mon Jun 12 16:28:03 2017 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Jun 16 14:38:23 2017 +0200

    qemuDomainGetPreservedMounts: Prune nested mount points
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1431112
    
    There can be nested mount points. For instance /dev/shm/blah can
    be a mount point and /dev/shm too. It doesn't make much sense to
    return the former path because callers preserve the latter (and
    with that the former too). Therefore prune nested mount points.
    
    Signed-off-by: Michal Privoznik <mprivozn>
    Reviewed-by: John Ferlan <jferlan>

commit 6ab3e2f6c4c665efdddb313ac9ecd80bf9c67670
Author:     Michal Privoznik <mprivozn>
AuthorDate: Mon Jun 12 17:46:30 2017 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Jun 16 14:29:12 2017 +0200

    qemuDomainBuildNamespace: Clean up temp files
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1431112
    
    After 290a00e41d we know how to deal with file mount points.
    However, when cleaning up the temporary location for preserved
    mount points we are still calling rmdir(). This won't fly for
    files. We need to call unlink(). Now, since we don't really care
    if the cleanup succeeded or not (it's the best effort anyway), we
    can call both rmdir() and unlink() without need for
    differentiation between files and directories.
    
    Signed-off-by: Michal Privoznik <mprivozn>
    Reviewed-by: John Ferlan <jferlan>

v3.4.0-125-g6451b55ec

I just wanted to add that this happens also with overlayfs, like:

overlaid                 3.7G  1.6G  2.1G  44% /dev/shm/asd-antenore/home/antenore/lotus/notes/data

This mount point is created automatically by anything-sync-daemon

Verified pass with libvirt-3.9.0-6.el7.x86_64.

Test steps:
Scenario 1:Start a guest when  mounting a dir/file under preserved mount points:
1.#touch /tmp/demo
2.#touch /dev/shm/demo
3.#mount --bind /tmp/demo /dev/shm/demo
4.#virsh start avocado-vt-vm1
Domain avocado-vt-vm1 started


Scenario 2: Start a guest when /dev contains a mount point that is a file:
1.#touch /tmp/demo
2.#touch /dev/demo
3.#mount --bind /tmp/demo /dev/demo
4.#virsh start avocado-vt-vm1
Domain avocado-vt-vm1 started

Scenario 3: Start a guest when /dev/shm contains a overlaid mount point:
1.Install asd and edit /etc/asd.conf:
WHATTOSYNC=('/foo/bar')
VOLATILE="/dev/shm"
USE_OVERLAYFS="yes"

2.Create /foo/bar:
#mkdir -p /foo/bar
3.Restart asd service:
#systemctl restart asd
4.Check the mount point
#df -h
overlaid                            5.7G   30M  5.7G   1% /dev/shm/asd-root/foo/bar
5.Start the guest:
#virsh start avocado-vt-vm1
Domain avocado-vt-vm1 started

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0704