Bug 1470593 - Don't use cgroup mount points from /proc/mounts that are hidden
Don't use cgroup mount points from /proc/mounts that are hidden
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: libvirt (Show other bugs)
25
x86_64 Linux
unspecified Severity unspecified
: ---
: ---
Assigned To: Libvirt Maintainers
Fedora Extras Quality Assurance
:
Depends On: 1468214
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-13 05:26 EDT by Juan Hernández
Modified: 2017-09-14 16:22 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1468214
Environment:
Last Closed: 2017-09-14 16:22:45 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Juan Hernández 2017-07-13 05:26:49 EDT
+++ This bug was initially created as a clone of Bug #1468214 +++

Description of problem:

When libvirt scans the /proc/mounts file in order to find the cgroup moint points it does so reading the lines of the file in order, from the first to the last, and picking only the first one that matches. But the order of the /proc/mounts file is the order of the mount operations. This means that a mount point in a line of that file may be hidden or replaced by another later line. So in the presence of duplicates libvirt should actually pick the *last* line of the file, not the first.

Version-Release number of selected component (if applicable):

This was detected in a quite complicated environment: Kubernetes+KubeVirt+libvirt. These are the relevant versions in the Kubernetes environment:

# cat /etc/redhat-release 
CentOS Linux release 7.3.1611 (Core)

# uname -a
Linux kubevirt.local 3.10.0-514.21.2.el7.x86_64 #1 SMP Tue Jun 20 12:24:47 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

# rpm -qa 'kube*' 'docker*' | sort
docker-1.12.6-28.git1398f24.el7.centos.x86_64
docker-client-1.12.6-28.git1398f24.el7.centos.x86_64
docker-common-1.12.6-28.git1398f24.el7.centos.x86_64
kubeadm-1.6.6-0.x86_64
kubectl-1.6.6-0.x86_64
kubelet-1.6.6-0.x86_64
kubernetes-cni-0.5.1-0.x86_64

And these are the relevant versions in the container where libvirt is running:

# cat /etc/redhat-release 
Fedora release 25 (Twenty Five)

# rpm -qa 'libvirt*' | sort
libvirt-client-2.2.0-2.fc25.x86_64
libvirt-daemon-2.2.0-2.fc25.x86_64
libvirt-daemon-driver-interface-2.2.0-2.fc25.x86_64
libvirt-daemon-driver-network-2.2.0-2.fc25.x86_64
libvirt-daemon-driver-nodedev-2.2.0-2.fc25.x86_64
libvirt-daemon-driver-nwfilter-2.2.0-2.fc25.x86_64
libvirt-daemon-driver-qemu-2.2.0-2.fc25.x86_64
libvirt-daemon-driver-secret-2.2.0-2.fc25.x86_64
libvirt-daemon-driver-storage-2.2.0-2.fc25.x86_64
libvirt-daemon-kvm-2.2.0-2.fc25.x86_64
libvirt-daemon-qemu-2.2.0-2.fc25.x86_64
libvirt-libs-2.2.0-2.fc25.x86_64

How reproducible:

Always.


Steps to Reproduce:

1. Create the Kubernetes+KubeVirt environment described above.
2. Try to start a virtual machine.

Actual results:

The virtual machine won't start, and libvirt will generate the following error message:

2017-06-30 18:10:10.465+0000: 3585: error : virCgroupMakeGroup:1085 : Failed to create controller net_cls for group: No such file or directory

Expected results:

The virtual machine should start without problems.

Additional info:

This issue was initially reported in the KubeVirt project. See here for more details:

  Libvirt fails when there are hidden cgroup mount points in `/proc/mounts`
  https://github.com/kubevirt/libvirt/issues/4

The issue was also discussed in this mail thread:

  Potential problem with /proc/mounts items that don't exist inside Kubernetes
  https://www.redhat.com/archives/libvir-list/2017-July/msg00120.html

--- Additional comment from Juan Hernández on 2017-07-06 07:09:06 EDT ---

I am taking the bug and I will try to fix it. Not sure if that is what you usually do in this project, so feel free to amend as required.

--- Additional comment from Cole Robinson on 2017-07-06 10:53:08 EDT ---

(In reply to Juan Hernández from comment #1)
> I am taking the bug and I will try to fix it. Not sure if that is what you
> usually do in this project, so feel free to amend as required.

We don't have too much process with the bug tracker so assigning it to yourself is perfectly fine, just be sure to send your patches to the mailing list!

--- Additional comment from Juan Hernández on 2017-07-06 11:04:49 EDT ---

Thanks Cole. Here is the patch:

  https://www.redhat.com/archives/libvir-list/2017-July/msg00141.html

--- Additional comment from Michal Privoznik on 2017-07-13 03:54:58 EDT ---

The patch has been merged upstream:

commit dacd160d7479e0ec2d8a63f102145fd30636a1c8
Author:     Juan Hernandez <jhernand@redhat.com>
AuthorDate: Thu Jul 6 17:03:31 2017 +0200
Commit:     Martin Kletzander <mkletzan@redhat.com>
CommitDate: Thu Jul 13 09:37:52 2017 +0200

    Avoid hidden cgroup mount points
    
    Currently the scan of the /proc/mounts file used to find cgroup mount
    points doesn't take into account that mount points may hidden by other
    mount points. For, example in certain Kubernetes environments the
    /proc/mounts contains the following lines:
    
      cgroup /sys/fs/cgroup/net_prio,net_cls cgroup ...
      tmpfs /sys/fs/cgroup tmpfs ...
      cgroup /sys/fs/cgroup/net_cls,net_prio cgroup ...
    
    In this particular environment the first mount point is hidden by the
    second one. The correct mount point is the third one, but libvirt will
    never process it because it only checks the first mount point for each
    controller (net_cls in this case). So libvirt will try to use the first
    mount point, which doesn't actually exist, and the complete detection
    process will fail.
    
    To avoid that issue this patch changes the virCgroupDetectMountsFromFile
    function so that when there are duplicates it takes the information from
    the last line in /proc/mounts. This requires removing the previous
    explicit condition to skip duplicates, and adding code to free the
    memory used by the processing of duplicated lines.
    
    Related-To: https://bugzilla.redhat.com/1468214
    Related-To: https://github.com/kubevirt/libvirt/issues/4
    Signed-off-by: Juan Hernandez <jhernand@redhat.com>

v3.5.0-78-g19335bc48
Comment 1 Juan Hernández 2017-07-13 05:31:12 EDT
This issue causes a problem in the KubeVirt project, which currently uses Fedodra 25. Will you consider back-porting the fix?
Comment 2 Cole Robinson 2017-07-13 08:02:36 EDT
Yup I'll grab it for the next build
Comment 3 Fedora Update System 2017-08-04 22:41:52 EDT
libvirt-2.2.1-3.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-226cbd995b
Comment 4 Fedora Update System 2017-08-07 04:19:37 EDT
libvirt-2.2.1-3.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-226cbd995b
Comment 5 Fedora Update System 2017-08-23 04:12:56 EDT
libvirt-3.2.1-5.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-3c92db10b8
Comment 6 Fedora Update System 2017-08-23 15:52:14 EDT
libvirt-3.2.1-5.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
Comment 7 Fedora Update System 2017-08-23 20:51:59 EDT
libvirt-2.2.1-3.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.