Bug 1404990
| Summary: | udev rewrites permissions set by libvirt on block devices that are closed | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Virtualization Manager | Reporter: | Jaroslav Suchanek <jsuchane> |
| Component: | vdsm | Assignee: | Tomas Jelinek <tjelinek> |
| Status: | CLOSED ERRATA | QA Contact: | Israel Pinto <ipinto> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | unspecified | CC: | chorn, lsurette, mavital, michal.skrivanek, mprivozn, nsoffer, rbalakri, srevivo, tjelinek, virt-bugs, ycui, ykaul |
| Target Milestone: | ovirt-4.2.0 | Keywords: | TestOnly |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: |
undefined
|
Story Points: | --- |
| Clone Of: | 1404952 | Environment: | |
| Last Closed: | 2018-05-15 17:49:33 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | Virt | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1404952, 1404992 | ||
| Bug Blocks: | |||
|
Description
Jaroslav Suchanek
2016-12-15 10:19:36 UTC
I'm not sure it is relevant to RHV, Michal, can you please get back to me/us to see how does it fit with what RHV is doing (In reply to Michal Skrivanek from comment #1) > I'm not sure it is relevant to RHV, Michal, can you please get back to me/us > to see how does it fit with what RHV is doing Sure. The thing is: when libvirt starts a domain / attaches a device to an already running one, it also sets up security labels (uid:gid + SELinux label). However, there's a small window between libvirt setting it all up and qemu opening the device. Problem was that during this small window udev came and decided to process events. Which ended up in mangled security labels rendering qemu unable to open the device. The only way we could solve this problem was to take udev out of the picture and thus spawn qemu processes under their own mount namespace. Here /dev is a tmpfs managed by libvirt (entries are created on domain startup/hotplug) so udev no longer has ability to screw up things. Ideally, this change is transparent to upper layers. However, it may happen that in my implementation I've forgotten about some silly corner case. Therefore, what I'd advise you to do is to test device passthrough, various disk backends for your VMs, etc. Anything that has something to do with /dev/* files. Also, IIRC you guys have your own udev rules to address the issue I'm describing above, right? Well, you will not need them any longer :-) (In reply to Michal Privoznik from comment #2) > Also, IIRC you guys have your own udev rules to address the issue I'm > describing above, right? Well, you will not need them any longer :-) We do need our udev rules, this his how ovirt lvs get the vdsm:kvm owner. Without the udev rules we will not be able to manage these devices, regardless of what libvirt is doing. Note that we do not know anything about the libvirt private devices, so the paths that we are using will still be the same paths to /rhev/data-center/... or /run/vdms/storage/..., and the paths libvirt report back to vdsm must be the same. We are also accessing file based images while the vm is running, for example for checking current file size on NFS storage, so the real file and libvirt version of the file must be in sync. For block storage we extend and refresh the lvs while the vm is running, so the libvirt private device must work for this use case. After live merge, we delete the device that was removed from the chain, this again must work with libvirt private devices. (In reply to Nir Soffer from comment #4) > (In reply to Michal Privoznik from comment #2) > > Also, IIRC you guys have your own udev rules to address the issue I'm > > describing above, right? Well, you will not need them any longer :-) > > > Note that we do not know anything about the libvirt private devices And that's how it is supposed to be. We don't want anybody to mess with the namespace. It's libvirt internals. > so the > paths > that we are using will still be the same paths to /rhev/data-center/... or > /run/vdms/storage/..., and the paths libvirt report back to vdsm must be the > same. Sure. The reported paths (and also those which qemu sees) are not going to change. > > We are also accessing file based images while the vm is running, for example > for > checking current file size on NFS storage, so the real file and libvirt > version > of the file must be in sync. They are. It is the same device after all. And for the file based storage there is no change from namespaces POV. It's just /dev that is not shared with the parent namespace. Everything else is shared and thus the same. > > For block storage we extend and refresh the lvs while the vm is running, so > the > libvirt private device must work for this use case. It should. But you should also check if there no funny bug around. > > After live merge, we delete the device that was removed from the chain, this > again > must work with libvirt private devices. This is a bit tricky. Libvirt does not remove the device after a block job. However, if you lvremove it, the one that is in the namespace stops working. I mean, /dev/something entry is still going to be there, but upon any access kernel shall return an error as it is a non-existent device. (In reply to Michal Privoznik from comment #5) > > After live merge, we delete the device that was removed from the chain, this > > again > > must work with libvirt private devices. > > This is a bit tricky. Libvirt does not remove the device after a block job. > However, if you lvremove it, the one that is in the namespace stops working. > I mean, /dev/something entry is still going to be there, but upon any access > kernel shall return an error as it is a non-existent device. We typically only deactivate the lv, but from libvirt point of view, this is the same as removing the lv, the device does not exists on the host. After live merge the device is not part of the qcow chain, so libvirt or qemu are not expected to access is. Maybe libvirt should remove the private device after it was removed from the vm, in the same way it treat devices detached from the vm? Can we test this changes now? on Fedora? RHEL? (In reply to Nir Soffer from comment #6) > (In reply to Michal Privoznik from comment #5) > > > After live merge, we delete the device that was removed from the chain, this > > > again > > > must work with libvirt private devices. > > > > This is a bit tricky. Libvirt does not remove the device after a block job. > > However, if you lvremove it, the one that is in the namespace stops working. > > I mean, /dev/something entry is still going to be there, but upon any access > > kernel shall return an error as it is a non-existent device. > > We typically only deactivate the lv, but from libvirt point of view, this > is the same as removing the lv, the device does not exists on the host. > > After live merge the device is not part of the qcow chain, so libvirt or qemu > are not expected to access is. True. So no problem there. > > Maybe libvirt should remove the private device after it was removed from the > vm, in the same way it treat devices detached from the vm? It can't. The device might still be in use - as weirdly as it sounds - for instance as a second disk of the domain (in RO mode). Or it is in a backing chain of a different disk or something. That's why when it comes to disks, libvirt does not remove the device from the namespace. Had it been any other type of device, libvirt would remove it. > > Can we test this changes now? on Fedora? RHEL? Sure. There are still some bugs we are fixing upstream. Therefore, fedora build has this feature disabled, and RHEL build is waiting for the fixes. Stay tuned. Any updates? Yes, libvirt with namespaces enabled (and a tons of fixes) was build: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=542044 libvirt-3.1.0-1.el7 This is the build that is ready to be tested. This bug is in modified status but no patches are attached to it. Can you please check if the fix is included in latest build and move to QE if so? it depends on libvirt-3.1.0-1.el7 so it is safe to move it to on_qa Verify with: Software Version:4.2.0.2-0.1.el7 Migrate VM with block device. Migration pass Moving back to verify bot problem Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:1489 BZ<2>Jira Resync |