Bug 1471660
| Summary: | error : Unable to move /dev/log mount to /var/run/libvirt/qemu/instance-00000002.log: No such file or directory | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Oneata Mircea Teodor <toneata> |
| Component: | libvirt | Assignee: | Michal Privoznik <mprivozn> |
| Status: | CLOSED ERRATA | QA Contact: | yafu <yafu> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | high | ||
| Version: | 7.4 | CC: | afazekas, berrange, chhu, crobinso, dasmith, dprince, dyuan, eglynn, fjin, jdenemar, jreznik, jsaucier, jschluet, jsuchane, kchamart, libvirt-maint, mprivozn, mtessun, ohochman, rbalakri, rbryant, salmy, sasha, sbauza, sferdjao, sgordon, srevivo, vromanso, xuzhang, yafu, zpeng |
| Target Milestone: | rc | Keywords: | Upstream, ZStream |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | libvirt-3.2.0-14.el7_4.2 | Doc Type: | Bug Fix |
| Doc Text: |
Cause:
For better security libvirt runs every qemu domain in separate namespace. However, this means libvirt has to prepare the namespace every time it wants to start a domain. This involves parsing the mount table and moving some mount points around for the time that the namespace is being built. The reason for that is to preserve the mount points. For instance, if there's /dev/blah mount point it's being moved to some temporary location, the namespace is built (all the devices under /dev are created, etc.) and then the mount point it moved back to its origin location so that qemu has access to it (should it need one). But a mount point doesn't necessarily have to be a directory (see bug 1431112).
Consequence:
Due to a bug libvirt accepted just directories and regular files as valid mount points and ignored the rest of file types leaving users unable to start any domain if they have a special type of file in the mount table.
Fix:
Libvirt was taught to recognize all the types of files when moving the mount points to temporary location and back.
Result:
Users can start new domains. Happily.
|
Story Points: | --- |
| Clone Of: | 1459592 | Environment: | |
| Last Closed: | 2017-08-01 11:30:30 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1459592 | ||
| Bug Blocks: | |||
|
Description
Oneata Mircea Teodor
2017-07-17 08:13:57 UTC
Moving to POST: http://post-office.corp.redhat.com/archives/rhvirt-patches/2017-July/msg00165.html Scratch build to test can be found here: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=13666620 Just a word of caution when trying to verify this bug. The steps for reproducing you'll find in the linked patch's commit message work. However, when trying this out on RHEL I've found that some messages are logged into /dev/log which in turn makes socat exit thus invalidating the bind mount point. Therefore, libvirt is unable to start the domain again. This is a broader issue and IMO affects everybody who parses the system mount table. Hi,Michal, It's still a little confused that libvirt can not start guest if the mounted file is deleted, even if the guest does not use the mount file at all. Is it possible that libvirt do something to ignore the invalid mount table? Test steps: 1.# touch /tmp/testfile 2.# touch /dev/testfile 3.mount --bind /tmp/testfile /dev/testfile 4.rm -rf /tmp/testfile 5.virsh start rhel7.3 error: Failed to start domain rhel7.3 error: internal error: Process exited prior to exec: libvirt: QEMU Driver error : Unable to stat: /dev/testfile (deleted): No such file or directory Test with the scratch build in comment 3, after mounting /dev/log more than one time, libvirt can not start guest correctly. Test steps: 1.#mount --bind /dev/log /dev/log 2.#virsh start rhel7.3 Domain rhel7.3 started #virsh destroy rhel7.3 Domain rhel7.3 destroyed 3.#mount --bind /dev/log /dev/log 4.#mount | grep log devtmpfs on /dev/log type devtmpfs (rw,nosuid,seclabel,size=5899976k,nr_inodes=1474994,mode=755) devtmpfs on /dev/log type devtmpfs (rw,nosuid,seclabel,size=5899976k,nr_inodes=1474994,mode=755) devtmpfs on /dev/log type devtmpfs (rw,nosuid,seclabel,size=5899976k,nr_inodes=1474994,mode=755) 5.# virsh start rhel7.3 error: Failed to start domain rhel7.3 error: internal error: Process exited prior to exec: libvirt: QEMU Driver error : Failed to create /var/run/libvirt/qemu/rhel7.3.log: No such device or address (In reply to yafu from comment #5) > Hi,Michal, > > It's still a little confused that libvirt can not start guest if the mounted > file is deleted, even if the guest does not use the mount file at all. Is it > possible that libvirt do something to ignore the invalid mount table? Sure, libvirt can ignore it. However, that might mask some real problem. Moreover, even if not using namespaces, qemu will fail to start anyway (well, if configured to use the stale path). Then again - I've ran docker to see how they are dealing with this scenario and they just fail to start too. So I guess that's the consensus here. > > Test steps: > 1.# touch /tmp/testfile > 2.# touch /dev/testfile > 3.mount --bind /tmp/testfile /dev/testfile > 4.rm -rf /tmp/testfile > 5.virsh start rhel7.3 > error: Failed to start domain rhel7.3 > error: internal error: Process exited prior to exec: libvirt: QEMU Driver > error : Unable to stat: /dev/testfile (deleted): No such file or directory Yep. Based on what I've written above I think this is expected. Libvirt can't know if qemu is going to touch /dev/testfile. It can be accessed by qemu directly or by some other library that qemu links with (okay, in this scenario /dev/testfile is obviously not going to be touched, but in general we cannot assume that). Long story short, if libvirt cannot create namespace exactly as the host, we must fail. (In reply to yafu from comment #6) > Test with the scratch build in comment 3, after mounting /dev/log more than > one time, libvirt can not start guest correctly. > > Test steps: > 1.#mount --bind /dev/log /dev/log > > 2.#virsh start rhel7.3 > Domain rhel7.3 started > #virsh destroy rhel7.3 > Domain rhel7.3 destroyed > > 3.#mount --bind /dev/log /dev/log > > 4.#mount | grep log > devtmpfs on /dev/log type devtmpfs > (rw,nosuid,seclabel,size=5899976k,nr_inodes=1474994,mode=755) > devtmpfs on /dev/log type devtmpfs > (rw,nosuid,seclabel,size=5899976k,nr_inodes=1474994,mode=755) > devtmpfs on /dev/log type devtmpfs > (rw,nosuid,seclabel,size=5899976k,nr_inodes=1474994,mode=755) > > 5.# virsh start rhel7.3 > error: Failed to start domain rhel7.3 > error: internal error: Process exited prior to exec: libvirt: QEMU Driver > error : Failed to create /var/run/libvirt/qemu/rhel7.3.log: No such device > or address This is fixed upstream already by cdd9205dfffa3a and in 7.5 it'll be picked up by rebase. For this bug I think it is out of the scope. Reproduced with libvirt-3.2.0-14.el7_4.1.x86_64 Reproduce steps: 1.mount --bind /dev/log /dev/log 2.mount | grep log devtmpfs on /dev/log type devtmpfs (rw,nosuid,seclabel,size=5899976k,nr_inodes=1474994,mode=755) 3.Start a guest: # virsh start rhel7.3 error: Failed to start domain rhel7.3 error: internal error: Process exited prior to exec: libvirt: error : Unable to move /dev/log mount to /var/run/libvirt/qemu/rhel7.3.log: No such file or directory Verified with libvirt-3.2.0-14.el7_4.2.x86_64. Test steps: 1.Execute steps 1-2 in reproduce steps; 2.Start a guest: #virsh start rhel7.3 Domain rhel7.3 started 3.Also start guest with mounted other types of file in the /dev, such as: dir,regular file,pipe,link,character file, all the guests can start correctly. Hi,Alexander, Would you help to test this bug from openstack with libvirt-3.2.0-14.el7_4.2.x86_64 please? Thanks a lot. Was able to launch an instance without using the w/a mentioned here: https://bugzilla.redhat.com/show_bug.cgi?id=1459592#c19 Environment (relevant for comment #12): libvirt-daemon-driver-storage-rbd-3.2.0-14.el7.x86_64 libvirt-daemon-driver-network-3.2.0-14.el7.x86_64 libvirt-daemon-config-nwfilter-3.2.0-14.el7.x86_64 libvirt-daemon-driver-qemu-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-gluster-3.2.0-14.el7.x86_64 libvirt-daemon-driver-nwfilter-3.2.0-14.el7.x86_64 libvirt-daemon-driver-lxc-3.2.0-14.el7.x86_64 libvirt-libs-3.2.0-14.el7.x86_64 libvirt-daemon-driver-interface-3.2.0-14.el7.x86_64 libvirt-daemon-config-network-3.2.0-14.el7.x86_64 libvirt-client-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-core-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-logical-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7.x86_64 libvirt-daemon-kvm-3.2.0-14.el7.x86_64 libvirt-daemon-driver-nodedev-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-disk-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-3.2.0-14.el7.x86_64 libvirt-3.2.0-14.el7.x86_64 libvirt-daemon-driver-storage-scsi-3.2.0-14.el7.x86_64 libvirt-python-3.2.0-3.el7.x86_64 libvirt-daemon-driver-storage-mpath-3.2.0-14.el7.x86_64 libvirt-daemon-3.2.0-14.el7.x86_64 libvirt-daemon-driver-secret-3.2.0-14.el7.x86_64 According to comment 13, the bug can not be reproduced in openstack. So move the bug to verified according to comment 10. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2334 |