RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1471660 - error : Unable to move /dev/log mount to /var/run/libvirt/qemu/instance-00000002.log: No such file or directory
Summary: error : Unable to move /dev/log mount to /var/run/libvirt/qemu/instance-00000...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.4
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: rc
: ---
Assignee: Michal Privoznik
QA Contact: yafu
URL:
Whiteboard:
Depends On: 1459592
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-07-17 08:13 UTC by Oneata Mircea Teodor
Modified: 2017-09-22 06:04 UTC (History)
31 users (show)

Fixed In Version: libvirt-3.2.0-14.el7_4.2
Doc Type: Bug Fix
Doc Text:
Cause: For better security libvirt runs every qemu domain in separate namespace. However, this means libvirt has to prepare the namespace every time it wants to start a domain. This involves parsing the mount table and moving some mount points around for the time that the namespace is being built. The reason for that is to preserve the mount points. For instance, if there's /dev/blah mount point it's being moved to some temporary location, the namespace is built (all the devices under /dev are created, etc.) and then the mount point it moved back to its origin location so that qemu has access to it (should it need one). But a mount point doesn't necessarily have to be a directory (see bug 1431112). Consequence: Due to a bug libvirt accepted just directories and regular files as valid mount points and ignored the rest of file types leaving users unable to start any domain if they have a special type of file in the mount table. Fix: Libvirt was taught to recognize all the types of files when moving the mount points to temporary location and back. Result: Users can start new domains. Happily.
Clone Of: 1459592
Environment:
Last Closed: 2017-08-01 11:30:30 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:2334 0 normal SHIPPED_LIVE libvirt bug fix update 2017-08-14 22:56:05 UTC

Description Oneata Mircea Teodor 2017-07-17 08:13:57 UTC 
This bug has been copied from bug #1459592 and has been proposed to be backported to 7.4 z-stream (EUS).
Comment 3 Michal Privoznik 2017-07-17 12:57:12 UTC 
Moving to POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2017-July/msg00165.html

Scratch build to test can be found here:

https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=13666620
Comment 4 Michal Privoznik 2017-07-17 13:03:47 UTC 
Just a word of caution when trying to verify this bug. The steps for reproducing you'll find in the linked patch's commit message work. However, when trying this out on RHEL I've found that some messages are logged into /dev/log which in turn makes socat exit thus invalidating the bind mount point. Therefore, libvirt is unable to start the domain again. This is a broader issue and IMO affects everybody who parses the system mount table.
Comment 5 yafu 2017-07-18 07:43:07 UTC 
Hi,Michal,

It's still a little confused that libvirt can not start guest if the mounted file is deleted, even if the guest does not use the mount file at all. Is it possible that libvirt do something to ignore the invalid mount table?


Test steps:
1.# touch /tmp/testfile
2.# touch /dev/testfile
3.mount --bind /tmp/testfile /dev/testfile
4.rm -rf /tmp/testfile
5.virsh start rhel7.3
error: Failed to start domain rhel7.3
error: internal error: Process exited prior to exec: libvirt: QEMU Driver error : Unable to stat: /dev/testfile (deleted): No such file or directory
Comment 6 yafu 2017-07-18 10:14:44 UTC 
Test with the scratch build in comment 3, after mounting /dev/log more than one time, libvirt can not start guest correctly.

Test steps:
1.#mount --bind /dev/log /dev/log

2.#virsh start rhel7.3
Domain rhel7.3 started
  #virsh destroy rhel7.3
Domain rhel7.3 destroyed

3.#mount --bind /dev/log /dev/log

4.#mount | grep log
devtmpfs on /dev/log type devtmpfs (rw,nosuid,seclabel,size=5899976k,nr_inodes=1474994,mode=755)
devtmpfs on /dev/log type devtmpfs (rw,nosuid,seclabel,size=5899976k,nr_inodes=1474994,mode=755)
devtmpfs on /dev/log type devtmpfs (rw,nosuid,seclabel,size=5899976k,nr_inodes=1474994,mode=755)

5.# virsh start rhel7.3
error: Failed to start domain rhel7.3
error: internal error: Process exited prior to exec: libvirt: QEMU Driver error : Failed to create /var/run/libvirt/qemu/rhel7.3.log: No such device or address
Comment 7 Michal Privoznik 2017-07-18 11:48:21 UTC 
(In reply to yafu from comment #5)
> Hi,Michal,
> 
> It's still a little confused that libvirt can not start guest if the mounted
> file is deleted, even if the guest does not use the mount file at all. Is it
> possible that libvirt do something to ignore the invalid mount table?

Sure, libvirt can ignore it. However, that might mask some real problem. Moreover, even if not using namespaces, qemu will fail to start anyway (well, if configured to use the stale path). Then again - I've ran docker to see how they are dealing with this scenario and they just fail to start too. So I guess that's the consensus here.
 
> 
> Test steps:
> 1.# touch /tmp/testfile
> 2.# touch /dev/testfile
> 3.mount --bind /tmp/testfile /dev/testfile
> 4.rm -rf /tmp/testfile
> 5.virsh start rhel7.3
> error: Failed to start domain rhel7.3
> error: internal error: Process exited prior to exec: libvirt: QEMU Driver
> error : Unable to stat: /dev/testfile (deleted): No such file or directory

Yep. Based on what I've written above I think this is expected. Libvirt can't know if qemu is going to touch /dev/testfile. It can be accessed by qemu directly or by some other library that qemu links with (okay, in this scenario /dev/testfile is obviously not going to be touched, but in general we cannot assume that).

Long story short, if libvirt cannot create namespace exactly as the host, we must fail.
Comment 9 Michal Privoznik 2017-07-19 07:26:04 UTC 
(In reply to yafu from comment #6)
> Test with the scratch build in comment 3, after mounting /dev/log more than
> one time, libvirt can not start guest correctly.
> 
> Test steps:
> 1.#mount --bind /dev/log /dev/log
> 
> 2.#virsh start rhel7.3
> Domain rhel7.3 started
>   #virsh destroy rhel7.3
> Domain rhel7.3 destroyed
> 
> 3.#mount --bind /dev/log /dev/log
> 
> 4.#mount | grep log
> devtmpfs on /dev/log type devtmpfs
> (rw,nosuid,seclabel,size=5899976k,nr_inodes=1474994,mode=755)
> devtmpfs on /dev/log type devtmpfs
> (rw,nosuid,seclabel,size=5899976k,nr_inodes=1474994,mode=755)
> devtmpfs on /dev/log type devtmpfs
> (rw,nosuid,seclabel,size=5899976k,nr_inodes=1474994,mode=755)
> 
> 5.# virsh start rhel7.3
> error: Failed to start domain rhel7.3
> error: internal error: Process exited prior to exec: libvirt: QEMU Driver
> error : Failed to create /var/run/libvirt/qemu/rhel7.3.log: No such device
> or address

This is fixed upstream already by cdd9205dfffa3a and in 7.5 it'll be picked up by rebase. For this bug I think it is out of the scope.
Comment 10 yafu 2017-07-19 10:02:10 UTC 
Reproduced with libvirt-3.2.0-14.el7_4.1.x86_64

Reproduce steps:
1.mount --bind /dev/log /dev/log

2.mount | grep log
devtmpfs on /dev/log type devtmpfs (rw,nosuid,seclabel,size=5899976k,nr_inodes=1474994,mode=755)

3.Start a guest:
# virsh start rhel7.3
error: Failed to start domain rhel7.3
error: internal error: Process exited prior to exec: libvirt:  error : Unable to move /dev/log mount to /var/run/libvirt/qemu/rhel7.3.log: No such file or directory


Verified with libvirt-3.2.0-14.el7_4.2.x86_64.
Test steps:
1.Execute steps 1-2 in reproduce steps;

2.Start a guest:
#virsh start rhel7.3
Domain rhel7.3 started

3.Also start guest with mounted other types of file in the /dev, such as: dir,regular file,pipe,link,character file, all the guests can start correctly.
Comment 11 yafu 2017-07-19 10:09:15 UTC 
Hi,Alexander,

Would you help to test this bug from openstack with libvirt-3.2.0-14.el7_4.2.x86_64 please?

Thanks a lot.
Comment 12 Alexander Chuzhoy 2017-07-20 14:54:01 UTC 
Was able to launch an instance without using the w/a mentioned here: https://bugzilla.redhat.com/show_bug.cgi?id=1459592#c19
Comment 13 Alexander Chuzhoy 2017-07-20 14:55:40 UTC 
Environment (relevant for comment #12):
libvirt-daemon-driver-storage-rbd-3.2.0-14.el7.x86_64
libvirt-daemon-driver-network-3.2.0-14.el7.x86_64
libvirt-daemon-config-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-qemu-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-gluster-3.2.0-14.el7.x86_64
libvirt-daemon-driver-nwfilter-3.2.0-14.el7.x86_64
libvirt-daemon-driver-lxc-3.2.0-14.el7.x86_64
libvirt-libs-3.2.0-14.el7.x86_64
libvirt-daemon-driver-interface-3.2.0-14.el7.x86_64
libvirt-daemon-config-network-3.2.0-14.el7.x86_64
libvirt-client-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-core-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-logical-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7.x86_64
libvirt-daemon-kvm-3.2.0-14.el7.x86_64
libvirt-daemon-driver-nodedev-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-disk-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-3.2.0-14.el7.x86_64
libvirt-3.2.0-14.el7.x86_64
libvirt-daemon-driver-storage-scsi-3.2.0-14.el7.x86_64
libvirt-python-3.2.0-3.el7.x86_64
libvirt-daemon-driver-storage-mpath-3.2.0-14.el7.x86_64
libvirt-daemon-3.2.0-14.el7.x86_64
libvirt-daemon-driver-secret-3.2.0-14.el7.x86_64
Comment 14 yafu 2017-07-25 06:23:37 UTC 
According to comment 13, the bug can not be reproduced in openstack. So move the bug to verified according to comment 10.
Comment 15 errata-xmlrpc 2017-08-01 11:30:30 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2334
Note You need to log in before you can comment on or make changes to this bug.