Bug 950436 - Could not access KVM kernel module: Permission denied
Summary: Could not access KVM kernel module: Permission denied
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: qemu
Version: 22
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Fedora Virtualization Maintainers
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Keywords: Reopened
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-04-10 08:56 UTC by Tim Waugh
Modified: 2016-06-17 16:11 UTC (History)
30 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2015-05-01 16:37:40 UTC


Attachments (Terms of Use)

Description Tim Waugh 2013-04-10 08:56:13 UTC
Description of problem:
I get this message when trying to start a guest:

Error starting domain: internal error Process exited while reading console log output: char device redirected to /dev/pts/1 (label charserial0)
Could not access KVM kernel module: Permission denied
failed to initialize KVM: Permission denied

Details:
Error starting domain: internal error Process exited while reading console log output: char device redirected to /dev/pts/1 (label charserial0)
Could not access KVM kernel module: Permission denied
failed to initialize KVM: Permission denied


Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 96, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 117, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1160, in startup
    self._backend.create()
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 692, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: internal error Process exited while reading console log output: char device redirected to /dev/pts/1 (label charserial0)
Could not access KVM kernel module: Permission denied
failed to initialize KVM: Permission denied

Version-Release number of selected component (if applicable):
libvirt-daemon-1.0.4-1.fc19.x86_64
qemu-kvm-1.4.0-11.fc19.x86_64
kernel-3.9.0-0.rc5.git1.301.fc19.x86_64

How reproducible:
Happens every time.

Comment 1 Cole Robinson 2013-04-13 23:06:54 UTC
Does selinux=permissive make any difference?
What's ls -l /dev/kvm ?

Comment 2 Tim Waugh 2013-04-15 11:15:18 UTC
I haven't seen this happen since I reported it.

Comment 3 Arnaud Lacombe 2013-07-02 18:59:37 UTC
Please re-open. I'm seeing this on F19 as of today (bringing up virtualization after re-install)

% ls -l /dev/kvm 
crw-rw-rw-+ 1 root kvm 10, 232 Jul  2 12:41 /dev/kvm

I'll follow up on the issue...

Comment 4 Arnaud Lacombe 2013-07-02 19:03:25 UTC
The SElinux trick does not change the outcome.

% setenforce 0
% getenforce
Permissive

But issue remains.

Comment 5 Arnaud Lacombe 2013-07-02 19:06:22 UTC
FWIW, the following did the trick so far:

# rmmod kvm_intel
# rmmod kvm
# modprobe kvm
# modprobe kvm_intel

Comment 6 Cole Robinson 2013-07-02 20:03:03 UTC
What is ls -l /dev/kvm for the working setup?

Comment 7 Arnaud Lacombe 2013-07-02 20:14:44 UTC
Sadly:

% ls -l /dev/kvm 
crw-rw-rw-+ 1 root kvm 10, 232 Jul  2 15:05 /dev/kvm

Comment 8 Cole Robinson 2013-07-03 14:53:33 UTC
Does this reproduce on reboot or anything? if not we are kind of stuck for finding the culprit

Comment 9 Dave Allan 2013-07-03 14:58:32 UTC
Does this happen only the first time after installing virt packages maybe?

Comment 10 Arnaud Lacombe 2013-07-03 15:04:47 UTC
Exact, this only seems to happen the first time after installing the @virtualization group of packages. Subsequent boots look fine.

Comment 11 Cole Robinson 2013-07-03 15:24:16 UTC
Yeah the confusing thing though is that I've tried to reproduce before and never have.

Comment 12 Jan Hutař 2013-08-19 08:06:55 UTC
After I have installed virt-manager, libvirt, qemu-kvm, I have also seen this issue. Workaround from comment #5 made the trick. Now I have:

# stat /dev/kvm
  File: ‘/dev/kvm’
  Size: 0         	Blocks: 0          IO Block: 4096   character special file
Device: 5h/5d	Inode: 204923      Links: 1     Device type: a,e8
Access: (0666/crw-rw-rw-)  Uid: (    0/    root)   Gid: (   36/     kvm)
Context: system_u:object_r:kvm_device_t:s0
Access: 2013-08-19 10:01:14.496898749 +0200
Modify: 2013-08-19 10:01:14.496898749 +0200
Change: 2013-08-19 10:01:14.497898746 +0200
 Birth: -

Unfortunately I do not have this before reloading these KVM related modules.

Comment 13 Martin Kletzander 2013-08-19 10:52:34 UTC
Just in case someone is looking for more data, this looks like a dup of Bug #911640, should be ok with 3.9 and 3.10 kernels.

Comment 14 Phil Anderson 2014-04-04 22:48:47 UTC
For what it's worth, I just had the same problem on Fedora 20 (3.13.7-200.fc20.x86_64) after the initial installation.  rmmod & insmod got it running.

Comment 15 James (purpleidea) 2014-05-13 03:49:38 UTC
I can reproduce this on a fresh install of Fedora 20.

excerpt of: cat /proc/cpuinfo

processor       : 23
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           E5645  @ 2.40GHz
stepping        : 2
microcode       : 0x15
cpu MHz         : 1596.000
cache size      : 12288 KB
physical id     : 1
siblings        : 12
core id         : 10
cpu cores       : 6
apicid          : 53
initial apicid  : 53
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm arat epb dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips        : 4799.88
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:


uname -a
Linux <fqdn> 3.14.2-200.fc20.x86_64 #1 SMP Mon Apr 28 14:40:57 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Let me know if you need other info. I'm reopening for now unless anyone has objections.

HTH

Comment 16 Martin Kletzander 2014-05-13 12:22:25 UTC
Moving to qemu per info in Bug 958860 (that one is qemu-kvm).

Comment 17 Cole Robinson 2014-05-19 21:03:07 UTC
Please provide:

ls -l /dev/kvm
getfacl /dev/kvm
dmesg | grep -i kvm

Does a reboot fix it?

Comment 18 James (purpleidea) 2014-05-20 00:14:14 UTC
(In reply to Cole Robinson from comment #17)
> Please provide:
> 
> ls -l /dev/kvm
> getfacl /dev/kvm
> dmesg | grep -i kvm

Don't have these because the machine is down right now. Can run them later this week. Do you still want these on the machine now that it's "fixed" (by the rmmod/modprobe) ?

> 
> Does a reboot fix it?

It was fixed by:

# rmmod kvm_intel
# rmmod kvm
# modprobe kvm
# modprobe kvm_intel

(from comment 5)

Comment 19 Cole Robinson 2014-05-20 13:27:43 UTC
(In reply to James from comment #18)
> (In reply to Cole Robinson from comment #17)
> > Please provide:
> > 
> > ls -l /dev/kvm
> > getfacl /dev/kvm
> > dmesg | grep -i kvm
> 
> Don't have these because the machine is down right now. Can run them later
> this week. Do you still want these on the machine now that it's "fixed" (by
> the rmmod/modprobe) ?

Nope, don't bother.

But another question: can you describe how you installed the system? Were virt packages installed during OS installation, or yum install'ed afterwards? Was the updates repo enabled during anaconda installation?

Comment 20 Cole Robinson 2014-05-20 13:28:18 UTC
Sorry, setting correct needinfo. James, please see comment #19

Comment 21 Martin Kletzander 2014-05-20 13:49:44 UTC
From Bug 958860 comment #11 from Miroslav:

This is caused by incorrect rules file location, udev rules has to be in /usr/lib/udev not in /usr/lib64/udev

I though the patch already exists since it is fixed in Bug 958860, but not back-ported to Fedora, that's why I re-assigned it with that info instead of marking it as a duplicate.  Shouldn't that be enough?  If not then I suggest asking Miroslav for further info.

Comment 22 Cole Robinson 2014-05-20 14:09:23 UTC
That equivalent fix is already in fedora. But it is in fact not enough: /dev/kvm is created by udev with root:root permissions, even before kvm is installed. When kvm is installed and the relaxed permissions are added by the udev rule, they aren't automatically triggered just on rules installation, so the spec file needs to invoke the proper udevadm trigger command. But apparently that _still_ isn't enough, since it doesn't reset /dev/kvm ACLs or something, I forget the details. However that should also be fixed in Fedora.

If there is still an issue, it's something new or undiscovered

Comment 23 Martin Kletzander 2014-05-20 14:18:52 UTC
(In reply to Cole Robinson from comment #22)
Oh, thanks for the explanation and sorry for my misunderstanding.

Comment 24 James (purpleidea) 2014-05-28 20:51:45 UTC
(In reply to Cole Robinson from comment #19)
> (In reply to James from comment #18)
> > (In reply to Cole Robinson from comment #17)
> > > Please provide:
> > > 
> > > ls -l /dev/kvm
> > > getfacl /dev/kvm
> > > dmesg | grep -i kvm
> > 
> > Don't have these because the machine is down right now. Can run them later
> > this week. Do you still want these on the machine now that it's "fixed" (by
> > the rmmod/modprobe) ?
> 
> Nope, don't bother.
> 
> But another question: can you describe how you installed the system? Were
> virt packages installed during OS installation, or yum install'ed
> afterwards? Was the updates repo enabled during anaconda installation?

Virt packages were installed after os installation with yum. I don't know about what happened with anaconda. Machine was built via satellite, but not my satellite.

Comment 25 Cole Robinson 2014-05-29 13:55:49 UTC
Well I've tried again and I can't reproduce, these issues are finicky and really the only thing I can think of is to poke at a machine while the issue is present. So just closing this as INSUFFICIENT_DATA. If someone reproduces later please reopen and provide explicit steps for how the machine was provisioned.

Comment 26 Bhaskar Chowdhury 2014-10-23 10:37:05 UTC
Yep it is still bugging me on f20:

Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory

LAP-02-1755_16:03:14_Thu Oct 23:~ # lsmod | grep kvm
kvm                   452677  0 


:/usr/lib/udev/rules.d # cat 80-kvm.rules 
KERNEL=="kvm", GROUP="kvm", MODE="0666"

O heck:

16:04:52_Thu Oct 23:~ # ls -al /dev/kvm
ls: cannot access /dev/kvm: No such file or directory

do I have to run mknod by hand??

Comment 27 Christophe Fergeau 2014-10-23 10:54:34 UTC
Is modprobe kvm_intel or modprobe kvm_amd working for you ? Do you get messages in dmesg after running these commands? Virtualization extensions (needed by kvm) tend to be disabled in the BIOS, so you need to enable them in the BIOS firrst.

Comment 28 Bhaskar Chowdhury 2014-10-23 11:00:40 UTC
LAP-02-1755_16:26:20_Thu Oct 23:~> sudo modprobe kvm_intel
modprobe: ERROR: could not insert 'kvm_intel': Operation not supported
bchowdhury@LAP-02-1755_16:26:30_Thu Oct 23:~> sudo modprobe kvm_amd
modprobe: ERROR: could not insert 'kvm_amd': Operation not supported

Heck!

[21203.361826] kvm: disabled by bios
[21223.198821] kvm: disabled by bios
[21524.429381] kvm: disabled by bios
[23848.046950] kvm: disabled by bios

@ Christophe Fergeau my mistake..I should have been more vigilent...I shall be careful next time steal your time..

Thanks a bunch......

Comment 29 Todd Warner 2015-02-16 15:19:20 UTC
Still an issue... In this case, F21.

Install Fedora 21.
Used if for months...

sudo yum install @virtualization
sudo systemctl start libvirtd
sudo systemctl status libvirtd

(everything *appeared* to work fine)

sudo virt-manager
New instance... 

And then I get that error described in the initial description.

Rebooted.
All works. Something is up with the kvm module load.

Have a nice day. :)

Comment 30 Christophe Fergeau 2015-02-16 15:45:38 UTC
Several comments have been asking for additional information, in particular https://bugzilla.redhat.com/show_bug.cgi?id=950436#c17

Comment 31 Todd Stansell 2015-02-21 04:38:30 UTC
I just ran into this on CentOS 7 ... reloading the kernel modules fixed the permissions for me. This was after a clean install of CentOS and then doing:

# yum install libvirt qemu-kvm qemu-img virt-install bridge-utils

Got the same permission denied errors as others and then found this bug.  Here are the command output requested before:

[root@hypervisor15 log]# uname -a
Linux hypervisor15 3.10.0-123.20.1.el7.x86_64 #1 SMP Thu Jan 29 18:05:33 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@hypervisor15 log]# cat /etc/redhat-release
CentOS Linux release 7.0.1406 (Core)
[root@hypervisor15 log]# ls -l /dev/kvm
crw-------. 1 root root 10, 232 Feb 20 22:40 /dev/kvm
[root@hypervisor15 log]# getfacl /dev/kvm
getfacl: Removing leading '/' from absolute path names
# file: dev/kvm
# owner: root
# group: root
user::rw-
group::---
other::---

[root@hypervisor15 log]# dmesg | grep -i kvm
[root@hypervisor15 log]# rmmod kvm_intel
[root@hypervisor15 log]# rmmod kvm
[root@hypervisor15 log]# modprobe kvm
[root@hypervisor15 log]# modprobe kvm_intel
[root@hypervisor15 log]# ls -l /dev/kvm
crw-rw-rw-. 1 root kvm 10, 232 Feb 21 04:07 /dev/kvm
[root@hypervisor15 log]# getfacl /dev/kvm
getfacl: Removing leading '/' from absolute path names
# file: dev/kvm
# owner: root
# group: kvm
user::rw-
group::rw-
other::rw-
[root@hypervisor15 log]# dmesg | grep -i kvm

Comment 32 Pádraig Brady 2015-02-24 15:20:31 UTC
Seeing this on centos 7 also with internal qemu-rhev package.
Machine wasn't rebooted since installing qemu-rhev

[root@c7 ~(keystone_admin)]# uname -a
Linux c7 3.10.0-123.8.1.el7.x86_64 #1 SMP Mon Sep 22 19:06:58 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

[root@c7 ~(keystone_admin)]# rpm -qa qemu* libvirt
qemu-img-rhev-1.5.3-60.el7_0.12.x86_64
libvirt-1.1.1-29.el7_0.7.x86_64
qemu-kvm-rhev-1.5.3-60.el7_0.12.x86_64
qemu-kvm-common-rhev-1.5.3-60.el7_0.12.x86_64


Notice the lack of group perms from:

# getfacl /dev/kvm
getfacl: Removing leading '/' from absolute path names
# file: dev/kvm
# owner: root
# group: kvm
user::rw-
user:padraig:rw-
group::---
mask::rw-
other::rw-


Now resetting the module, restores group perms:

[root@c7 ~(keystone_admin)]# ls -l /dev/kvm
crw-rw-rw-+ 1 root kvm 10, 232 Feb 23 14:34 /dev/kvm
[root@c7 ~(keystone_admin)]# dmesg | grep -i kvm
[root@c7 ~(keystone_admin)]# rmmod kvm_intel
[root@c7 ~(keystone_admin)]# rmmod kvm
[root@c7 ~(keystone_admin)]# modprobe kvm
[root@c7 ~(keystone_admin)]# modprobe kvm_intel
[root@c7 ~(keystone_admin)]# ls -l /dev/kvm
crw-rw-rw-+ 1 root kvm 10, 232 Feb 24 15:12 /dev/kvm
[root@c7 ~(keystone_admin)]# getfacl /dev/kvm
getfacl: Removing leading '/' from absolute path names
# file: dev/kvm
# owner: root
# group: kvm
user::rw-
user:padraig:rw-
group::rw-
mask::rw-
other::rw-

Comment 33 Yaniv Kaul 2015-03-29 15:32:55 UTC
Same issue with CentOS 7, after installing DevStack (OpenStack).
[root@lg858 qemu]# rpm -qa |grep qemu
qemu-common-2.0.0-1.el7.3.x86_64
qemu-img-1.5.3-60.el7_0.11.x86_64
ipxe-roms-qemu-20130517-5.gitc4bce43.el7.noarch
qemu-kvm-common-1.5.3-60.el7_0.11.x86_64
qemu-system-x86-2.0.0-1.el7.3.x86_64
libvirt-daemon-driver-qemu-1.1.1-29.el7_0.7.x86_64

[root@lg858 qemu]# getfacl /dev/kvm
getfacl: Removing leading '/' from absolute path names
# file: dev/kvm
# owner: root
# group: root
user::rw-
group::---
other::---

root@lg858 qemu]# rmmod kvm_intel
[root@lg858 qemu]# rmmod kvm
[root@lg858 qemu]# modprobe kvm
[root@lg858 qemu]# modprobe kvm_intel
[root@lg858 qemu]# getfacl /dev/kvm
getfacl: Removing leading '/' from absolute path names
# file: dev/kvm
# owner: root
# group: root
user::rw-
group::---
other::---

[root@lg858 qemu]# ls -l /dev/kvm
crw------- 1 root root 10, 232 Mar 29 18:30 /dev/kvm


I was under the impression libvirt needs the dev to be part of the 'root' group:
# The group for QEMU processes run by the system instance. It can be
# specified in a similar way to user.
#group = "root"

Or is it just for the qemu processes?

I'm quite sure the maching HAS been rebooted since installation.

Comment 34 Cole Robinson 2015-04-16 20:57:39 UTC
I just tried to reproduce with f21, f22, and centos 7.1. After installing qemu-kvm the /dev/kvm permissions are the expected 666. So I really don't know what the issue could be.

Maybe this was fixed in 7.1 since the above centos reports sound like 7.0 packages. And Comment #33 seems like it's referencing way outdates EPEL packages as well.

Comment #29 says this is still an issue on F21... but again I still can't reproduce so I'm not sure what more I can do.

This bug has gotten pretty long and convoluted. I realize it's probably one of the top google hits for this issue, but it's reached its limit of usefulness.

If anyone can still reproduce, please do the following:

- Document the exact steps to reproduce, starting with how you provisioned the machine.

- Please make a _second_ attempt to reproduce, installing a new machine/VM from scratch if need be. Even if you can't reproduce on the second attempt, it will be good to know whether this is deterministic. I have repeatedly been unable to reproduce so either I'm doing something wrong, or it's setup/hw dependent, or it's completely random, but so far I'm unclear on which category it is :)

- File a bug with the relevant distro. So if you are using centos or RHEL, please file a bug there and not in the Fedora tracker. If you hit this on Fedora again, please open a _new_ bug report

Comment 35 Cole Robinson 2015-04-16 21:00:15 UTC
I'm actually going to try one more thing in the spec file, just forcing 'chmod 666 /dev/kvm' on the initial package install. It's less than ideal because it duplicates 80-kvm.rules permissions, but it also eliminates udevadm and setfacl from the equation which maybe are sporadically failing

Comment 36 Fedora Update System 2015-04-27 20:30:48 UTC
qemu-2.3.0-1.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/qemu-2.3.0-1.fc22

Comment 37 Fedora Update System 2015-04-29 13:03:46 UTC
Package qemu-2.3.0-1.fc22:
* should fix your issue,
* was pushed to the Fedora 22 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing qemu-2.3.0-1.fc22'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-7131/qemu-2.3.0-1.fc22
then log in and leave karma (feedback).

Comment 38 Fedora Update System 2015-05-01 16:37:40 UTC
qemu-2.3.0-1.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 39 David Jones 2016-06-17 15:51:29 UTC
Still present in RHEL 7.2 with qemu-2.0.0

Comment 40 Richard W.M. Jones 2016-06-17 16:11:54 UTC
(In reply to David Jones from comment #39)
> Still present in RHEL 7.2 with qemu-2.0.0

There is no qemu-2.0.0 in RHEL 7.2.  This bug is for
Fedora and likely nothing to do with your problem.
If you have an issue with supported packages, then
please contact Red Hat's support.


Note You need to log in before you can comment on or make changes to this bug.