Red Hat Bugzilla – Bug 1470593
Don't use cgroup mount points from /proc/mounts that are hidden
Last modified: 2017-09-14 16:22:45 EDT
+++ This bug was initially created as a clone of Bug #1468214 +++
Description of problem:
When libvirt scans the /proc/mounts file in order to find the cgroup moint points it does so reading the lines of the file in order, from the first to the last, and picking only the first one that matches. But the order of the /proc/mounts file is the order of the mount operations. This means that a mount point in a line of that file may be hidden or replaced by another later line. So in the presence of duplicates libvirt should actually pick the *last* line of the file, not the first.
Version-Release number of selected component (if applicable):
This was detected in a quite complicated environment: Kubernetes+KubeVirt+libvirt. These are the relevant versions in the Kubernetes environment:
# cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)
# uname -a
Linux kubevirt.local 3.10.0-514.21.2.el7.x86_64 #1 SMP Tue Jun 20 12:24:47 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
# rpm -qa 'kube*' 'docker*' | sort
And these are the relevant versions in the container where libvirt is running:
# cat /etc/redhat-release
Fedora release 25 (Twenty Five)
# rpm -qa 'libvirt*' | sort
Steps to Reproduce:
1. Create the Kubernetes+KubeVirt environment described above.
2. Try to start a virtual machine.
The virtual machine won't start, and libvirt will generate the following error message:
2017-06-30 18:10:10.465+0000: 3585: error : virCgroupMakeGroup:1085 : Failed to create controller net_cls for group: No such file or directory
The virtual machine should start without problems.
This issue was initially reported in the KubeVirt project. See here for more details:
Libvirt fails when there are hidden cgroup mount points in `/proc/mounts`
The issue was also discussed in this mail thread:
Potential problem with /proc/mounts items that don't exist inside Kubernetes
--- Additional comment from Juan Hernández on 2017-07-06 07:09:06 EDT ---
I am taking the bug and I will try to fix it. Not sure if that is what you usually do in this project, so feel free to amend as required.
--- Additional comment from Cole Robinson on 2017-07-06 10:53:08 EDT ---
(In reply to Juan Hernández from comment #1)
> I am taking the bug and I will try to fix it. Not sure if that is what you
> usually do in this project, so feel free to amend as required.
We don't have too much process with the bug tracker so assigning it to yourself is perfectly fine, just be sure to send your patches to the mailing list!
--- Additional comment from Juan Hernández on 2017-07-06 11:04:49 EDT ---
Thanks Cole. Here is the patch:
--- Additional comment from Michal Privoznik on 2017-07-13 03:54:58 EDT ---
The patch has been merged upstream:
Author: Juan Hernandez <firstname.lastname@example.org>
AuthorDate: Thu Jul 6 17:03:31 2017 +0200
Commit: Martin Kletzander <email@example.com>
CommitDate: Thu Jul 13 09:37:52 2017 +0200
Avoid hidden cgroup mount points
Currently the scan of the /proc/mounts file used to find cgroup mount
points doesn't take into account that mount points may hidden by other
mount points. For, example in certain Kubernetes environments the
/proc/mounts contains the following lines:
cgroup /sys/fs/cgroup/net_prio,net_cls cgroup ...
tmpfs /sys/fs/cgroup tmpfs ...
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup ...
In this particular environment the first mount point is hidden by the
second one. The correct mount point is the third one, but libvirt will
never process it because it only checks the first mount point for each
controller (net_cls in this case). So libvirt will try to use the first
mount point, which doesn't actually exist, and the complete detection
process will fail.
To avoid that issue this patch changes the virCgroupDetectMountsFromFile
function so that when there are duplicates it takes the information from
the last line in /proc/mounts. This requires removing the previous
explicit condition to skip duplicates, and adding code to free the
memory used by the processing of duplicated lines.
Signed-off-by: Juan Hernandez <firstname.lastname@example.org>
This issue causes a problem in the KubeVirt project, which currently uses Fedodra 25. Will you consider back-porting the fix?
Yup I'll grab it for the next build
libvirt-2.2.1-3.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-226cbd995b
libvirt-2.2.1-3.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-226cbd995b
libvirt-3.2.1-5.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-3c92db10b8
libvirt-3.2.1-5.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
libvirt-2.2.1-3.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.