Bug 860658

Summary: nova boot fails because /dev/kvm is 0600 instead of 0666
Product: [Fedora] Fedora Reporter: Sandro Mathys <sandro>
Component: qemuAssignee: Fedora Virtualization Maintainers <virt-maint>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 18CC: akscram, alexander.sakhnov, amit.shah, apevec, apevec, asalkeld, berrange, bfilippov, breu, cfergeau, crobinso, dwmw2, itamar, Jan.van.Eldik, jonathansteffan, jose.castro.leon, knoel, markmc, matt_domsch, mlvov, pbonzini, p, rbryant, rjones, rkukura, scottt.tw, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 863374 (view as bug list) Environment:
Last Closed: 2012-09-28 15:48:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
compute.log
none
scheduler.log none

Description Sandro Mathys 2012-09-26 12:22:16 UTC 
Description of problem:
Following the test day instructions (with the quantum option) on F18 bare metal, nova boot ends with an ERROR state.

Version-Release number of selected component (if applicable):
openstack-nova-compute-2012.2-0.9.rc1.fc18.noarch

How reproducible:
Not sure, I think I've seen it twice (out of 2) and others suggested they've seen the same.

Steps to Reproduce:
https://fedoraproject.org/wiki/QA:Testcase_launch_an_instance_on_OpenStack
  
Actual results:
ERROR

Expected results:
ACTIVE

Additional info:
The logs (which I'll attach in a minute) suggested issues with libvirt/kvm. In #fedora-openstack, derekh suggested permissions on /dev/kvm could be wrong. It turned out they were 0600 while they should probably be 0666 (as on F17) instead. chmod 0666 /dev/kvm therefore fixed the problem. It's been suggested that for others a reboot (instead of the chmod) fixed things as well.

Not sure whether this should be considered a bug in udev (or ...) instead but I figure that change happened for a reason.

Also, it was suggested I run sudo su - qemu -s /bin/sh -c virt-host-validate:
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking for device /dev/kvm                                         : FAIL (Check that the 'kvm-intel' or 'kvm-amd' modules are loaded & the BIOS has enabled virtualization)
  QEMU: Checking for device /dev/vhost-net                                   : WARN (Load the 'vhost_net' module to improve performance of virtio networking)
  QEMU: Checking for device /dev/net/tun                                     : PASS
   LXC: Checking for Linux >= 2.6.26                                         : PASS
...but kvm and kvm-intel were loaded alright. After the chmod that test PASSes alright.
Comment 1 Sandro Mathys 2012-09-26 12:24:07 UTC 
Created attachment 617531 [details]
compute.log
Comment 2 Sandro Mathys 2012-09-26 12:25:34 UTC 
Created attachment 617534 [details]
scheduler.log
Comment 3 Sandro Mathys 2012-09-26 12:32:32 UTC 
Note regarding the logs: what I did was roughly:
- nova boot 4 systems
- restart scheduler (as suggested in similar bugs)
- nova boot 4 systems
- chmod 0666 /dev/kvm
- nova boot 2 systems
Comment 4 Mark McLoughlin 2012-09-26 12:39:45 UTC 
Cole - have you seen reports of bad /dev/kvm permissions in F18?
Comment 5 Sandro Mathys 2012-09-26 13:37:21 UTC 
FWIW:
/usr/lib/udev/rules.d/80-kvm.rules:KERNEL=="kvm", GROUP="kvm", MODE="0666"

Figure that explains why reboot helps. But the first time kvm is loaded (libvirtd is installed and started) it seems to get 0600 nevertheless - is /dev/kvm not created by udev in that situation?
Comment 6 Cole Robinson 2012-09-27 01:24:56 UTC 
Mark, haven't seen any bug reports go by about that besides this one. Is this consistently reproducible or a one off?
Comment 7 Sandro Mathys 2012-09-27 12:15:23 UTC 
Even on a fresh F18 Alpha minimal install (no updates, no openstack):
crw-------. 1 root root 10, 232 Sep 27 08:46 /dev/kvm
Comment 8 Jan van Eldik 2012-09-28 07:13:00 UTC 
Just to confirm Sandro's observation on a fresh F17 install, without
updates or openstack:

crw-------. 1 root root 10, 232 Sep 28 08:57 /dev/kvm
Comment 9 Pádraig Brady 2012-09-28 13:51:34 UTC 
Moving to qemu as it seems that package on install should do:
  chmod 666 /dev/kvm
or more generally
  udevadm trigger --action=change
Comment 10 Alan Pevec 2012-09-28 14:21:24 UTC 
%post in qemu-system-x86 runs /etc/sysconfig/modules/kvm.modules where kvm kernel module is modprobed. Why modprobe itself doesn't trigger udev?
Comment 11 Daniel Berrangé 2012-09-28 14:36:21 UTC 
The KVM modules are part of the main kernel RPM, so perhaps the module was loaded before the KVM RPM was installed. If this was the case it would have got the default permssiosn, since the udev rules would not be present yet. Then when %post ran module, nothing would be done since they were already loaded (albeit with wrong permissions).
Comment 12 Paolo Bonzini 2012-09-28 15:48:09 UTC 
Fixed in qemu-1.2.0-12.fc18 and qemu-1.2.0-12.fc19.
Comment 13 Pádraig Brady 2012-09-28 15:58:49 UTC 
For reference the change is:
http://pkgs.fedoraproject.org/cgit/qemu.git/commit/?id=8cc727f1

I was a bit surprised that --action=change wasn't specified,
but I now see that that's the default operation.

Also note this was reported as an issue against F17 as well as F18.

Thanks Paolo!
Comment 14 Fedora Update System 2012-10-08 23:55:49 UTC 
qemu-1.2.0-12.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/FEDORA-2012-15631/qemu-1.2.0-12.fc18