+++ This bug was initially created as a clone of Bug #1468214 +++

Description of problem:

When libvirt scans the /proc/mounts file in order to find the cgroup moint points it does so reading the lines of the file in order, from the first to the last, and picking only the first one that matches. But the order of the /proc/mounts file is the order of the mount operations. This means that a mount point in a line of that file may be hidden or replaced by another later line. So in the presence of duplicates libvirt should actually pick the *last* line of the file, not the first.

Version-Release number of selected component (if applicable):

This was detected in a quite complicated environment: Kubernetes+KubeVirt+libvirt. These are the relevant versions in the Kubernetes environment:

# cat /etc/redhat-release 
CentOS Linux release 7.3.1611 (Core)

# uname -a
Linux kubevirt.local 3.10.0-514.21.2.el7.x86_64 #1 SMP Tue Jun 20 12:24:47 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

# rpm -qa 'kube*' 'docker*' | sort
docker-1.12.6-28.git1398f24.el7.centos.x86_64
docker-client-1.12.6-28.git1398f24.el7.centos.x86_64
docker-common-1.12.6-28.git1398f24.el7.centos.x86_64
kubeadm-1.6.6-0.x86_64
kubectl-1.6.6-0.x86_64
kubelet-1.6.6-0.x86_64
kubernetes-cni-0.5.1-0.x86_64

And these are the relevant versions in the container where libvirt is running:

# cat /etc/redhat-release 
Fedora release 25 (Twenty Five)

# rpm -qa 'libvirt*' | sort
libvirt-client-2.2.0-2.fc25.x86_64
libvirt-daemon-2.2.0-2.fc25.x86_64
libvirt-daemon-driver-interface-2.2.0-2.fc25.x86_64
libvirt-daemon-driver-network-2.2.0-2.fc25.x86_64
libvirt-daemon-driver-nodedev-2.2.0-2.fc25.x86_64
libvirt-daemon-driver-nwfilter-2.2.0-2.fc25.x86_64
libvirt-daemon-driver-qemu-2.2.0-2.fc25.x86_64
libvirt-daemon-driver-secret-2.2.0-2.fc25.x86_64
libvirt-daemon-driver-storage-2.2.0-2.fc25.x86_64
libvirt-daemon-kvm-2.2.0-2.fc25.x86_64
libvirt-daemon-qemu-2.2.0-2.fc25.x86_64
libvirt-libs-2.2.0-2.fc25.x86_64

How reproducible:

Always.


Steps to Reproduce:

1. Create the Kubernetes+KubeVirt environment described above.
2. Try to start a virtual machine.

Actual results:

The virtual machine won't start, and libvirt will generate the following error message:

2017-06-30 18:10:10.465+0000: 3585: error : virCgroupMakeGroup:1085 : Failed to create controller net_cls for group: No such file or directory

Expected results:

The virtual machine should start without problems.

Additional info:

This issue was initially reported in the KubeVirt project. See here for more details:

  Libvirt fails when there are hidden cgroup mount points in `/proc/mounts`
  https://github.com/kubevirt/libvirt/issues/4

The issue was also discussed in this mail thread:

  Potential problem with /proc/mounts items that don't exist inside Kubernetes
  https://www.redhat.com/archives/libvir-list/2017-July/msg00120.html

--- Additional comment from Juan Hernández on 2017-07-06 07:09:06 EDT ---

I am taking the bug and I will try to fix it. Not sure if that is what you usually do in this project, so feel free to amend as required.

--- Additional comment from Cole Robinson on 2017-07-06 10:53:08 EDT ---

(In reply to Juan Hernández from comment #1)
> I am taking the bug and I will try to fix it. Not sure if that is what you
> usually do in this project, so feel free to amend as required.

We don't have too much process with the bug tracker so assigning it to yourself is perfectly fine, just be sure to send your patches to the mailing list!

--- Additional comment from Juan Hernández on 2017-07-06 11:04:49 EDT ---

Thanks Cole. Here is the patch:

  https://www.redhat.com/archives/libvir-list/2017-July/msg00141.html

--- Additional comment from Michal Privoznik on 2017-07-13 03:54:58 EDT ---

The patch has been merged upstream:

commit dacd160d7479e0ec2d8a63f102145fd30636a1c8
Author:     Juan Hernandez <jhernand@redhat.com>
AuthorDate: Thu Jul 6 17:03:31 2017 +0200
Commit:     Martin Kletzander <mkletzan@redhat.com>
CommitDate: Thu Jul 13 09:37:52 2017 +0200

    Avoid hidden cgroup mount points
    
    Currently the scan of the /proc/mounts file used to find cgroup mount
    points doesn't take into account that mount points may hidden by other
    mount points. For, example in certain Kubernetes environments the
    /proc/mounts contains the following lines:
    
      cgroup /sys/fs/cgroup/net_prio,net_cls cgroup ...
      tmpfs /sys/fs/cgroup tmpfs ...
      cgroup /sys/fs/cgroup/net_cls,net_prio cgroup ...
    
    In this particular environment the first mount point is hidden by the
    second one. The correct mount point is the third one, but libvirt will
    never process it because it only checks the first mount point for each
    controller (net_cls in this case). So libvirt will try to use the first
    mount point, which doesn't actually exist, and the complete detection
    process will fail.
    
    To avoid that issue this patch changes the virCgroupDetectMountsFromFile
    function so that when there are duplicates it takes the information from
    the last line in /proc/mounts. This requires removing the previous
    explicit condition to skip duplicates, and adding code to free the
    memory used by the processing of duplicated lines.
    
    Related-To: https://bugzilla.redhat.com/1468214
    Related-To: https://github.com/kubevirt/libvirt/issues/4
    Signed-off-by: Juan Hernandez <jhernand@redhat.com>

v3.5.0-78-g19335bc48