+++ This bug was initially created as a clone of Bug #1468214 +++ Description of problem: When libvirt scans the /proc/mounts file in order to find the cgroup moint points it does so reading the lines of the file in order, from the first to the last, and picking only the first one that matches. But the order of the /proc/mounts file is the order of the mount operations. This means that a mount point in a line of that file may be hidden or replaced by another later line. So in the presence of duplicates libvirt should actually pick the *last* line of the file, not the first. Version-Release number of selected component (if applicable): This was detected in a quite complicated environment: Kubernetes+KubeVirt+libvirt. These are the relevant versions in the Kubernetes environment: # cat /etc/redhat-release CentOS Linux release 7.3.1611 (Core) # uname -a Linux kubevirt.local 3.10.0-514.21.2.el7.x86_64 #1 SMP Tue Jun 20 12:24:47 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux # rpm -qa 'kube*' 'docker*' | sort docker-1.12.6-28.git1398f24.el7.centos.x86_64 docker-client-1.12.6-28.git1398f24.el7.centos.x86_64 docker-common-1.12.6-28.git1398f24.el7.centos.x86_64 kubeadm-1.6.6-0.x86_64 kubectl-1.6.6-0.x86_64 kubelet-1.6.6-0.x86_64 kubernetes-cni-0.5.1-0.x86_64 And these are the relevant versions in the container where libvirt is running: # cat /etc/redhat-release Fedora release 25 (Twenty Five) # rpm -qa 'libvirt*' | sort libvirt-client-2.2.0-2.fc25.x86_64 libvirt-daemon-2.2.0-2.fc25.x86_64 libvirt-daemon-driver-interface-2.2.0-2.fc25.x86_64 libvirt-daemon-driver-network-2.2.0-2.fc25.x86_64 libvirt-daemon-driver-nodedev-2.2.0-2.fc25.x86_64 libvirt-daemon-driver-nwfilter-2.2.0-2.fc25.x86_64 libvirt-daemon-driver-qemu-2.2.0-2.fc25.x86_64 libvirt-daemon-driver-secret-2.2.0-2.fc25.x86_64 libvirt-daemon-driver-storage-2.2.0-2.fc25.x86_64 libvirt-daemon-kvm-2.2.0-2.fc25.x86_64 libvirt-daemon-qemu-2.2.0-2.fc25.x86_64 libvirt-libs-2.2.0-2.fc25.x86_64 How reproducible: Always. Steps to Reproduce: 1. Create the Kubernetes+KubeVirt environment described above. 2. Try to start a virtual machine. Actual results: The virtual machine won't start, and libvirt will generate the following error message: 2017-06-30 18:10:10.465+0000: 3585: error : virCgroupMakeGroup:1085 : Failed to create controller net_cls for group: No such file or directory Expected results: The virtual machine should start without problems. Additional info: This issue was initially reported in the KubeVirt project. See here for more details: Libvirt fails when there are hidden cgroup mount points in `/proc/mounts` https://github.com/kubevirt/libvirt/issues/4 The issue was also discussed in this mail thread: Potential problem with /proc/mounts items that don't exist inside Kubernetes https://www.redhat.com/archives/libvir-list/2017-July/msg00120.html --- Additional comment from Juan Hernández on 2017-07-06 07:09:06 EDT --- I am taking the bug and I will try to fix it. Not sure if that is what you usually do in this project, so feel free to amend as required. --- Additional comment from Cole Robinson on 2017-07-06 10:53:08 EDT --- (In reply to Juan Hernández from comment #1) > I am taking the bug and I will try to fix it. Not sure if that is what you > usually do in this project, so feel free to amend as required. We don't have too much process with the bug tracker so assigning it to yourself is perfectly fine, just be sure to send your patches to the mailing list! --- Additional comment from Juan Hernández on 2017-07-06 11:04:49 EDT --- Thanks Cole. Here is the patch: https://www.redhat.com/archives/libvir-list/2017-July/msg00141.html --- Additional comment from Michal Privoznik on 2017-07-13 03:54:58 EDT --- The patch has been merged upstream: commit dacd160d7479e0ec2d8a63f102145fd30636a1c8 Author: Juan Hernandez <jhernand> AuthorDate: Thu Jul 6 17:03:31 2017 +0200 Commit: Martin Kletzander <mkletzan> CommitDate: Thu Jul 13 09:37:52 2017 +0200 Avoid hidden cgroup mount points Currently the scan of the /proc/mounts file used to find cgroup mount points doesn't take into account that mount points may hidden by other mount points. For, example in certain Kubernetes environments the /proc/mounts contains the following lines: cgroup /sys/fs/cgroup/net_prio,net_cls cgroup ... tmpfs /sys/fs/cgroup tmpfs ... cgroup /sys/fs/cgroup/net_cls,net_prio cgroup ... In this particular environment the first mount point is hidden by the second one. The correct mount point is the third one, but libvirt will never process it because it only checks the first mount point for each controller (net_cls in this case). So libvirt will try to use the first mount point, which doesn't actually exist, and the complete detection process will fail. To avoid that issue this patch changes the virCgroupDetectMountsFromFile function so that when there are duplicates it takes the information from the last line in /proc/mounts. This requires removing the previous explicit condition to skip duplicates, and adding code to free the memory used by the processing of duplicated lines. Related-To: https://bugzilla.redhat.com/1468214 Related-To: https://github.com/kubevirt/libvirt/issues/4 Signed-off-by: Juan Hernandez <jhernand> v3.5.0-78-g19335bc48
This issue causes a problem in the KubeVirt project, which currently uses Fedodra 25. Will you consider back-porting the fix?
Yup I'll grab it for the next build
libvirt-2.2.1-3.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-226cbd995b
libvirt-2.2.1-3.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-226cbd995b
libvirt-3.2.1-5.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-3c92db10b8
libvirt-3.2.1-5.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
libvirt-2.2.1-3.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.