Bug 829567

Summary: RHEL 6.3 requires reboot to build rhevm/vsphere images, since qemu-kvm kernel module loading changed
Product: Red Hat Enterprise Linux 6 Reporter: Dave Johnson <dajohnso>
Component: qemu-kvmAssignee: Virtualization Maintenance <virt-maint>
Status: CLOSED DUPLICATE QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.3CC: acathrow, areis, bsarathy, cpelland, dyasny, eblake, gholms, jlaska, jrd, mitch, mkenneth, thomas.oulevey, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: RHEL63
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-07-11 17:08:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
'yum install qemu-kvm' output none

Description Dave Johnson 2012-06-07 03:34:46 UTC
Description of problem:
==============================
Fought this issue all day... at first it was thought that the issue was because we were mixing rhel 6.2/6.3 rpms.  Cleared that up and got our rhel 6.2 installs pulling dependencies from a rhel 6.2 repo, everything worked great, was able to build/push/deploy images to all providers.

However, as soon as I installed a RHEL 6.3 system with appropriate 6.3 repos, the problem reappeared.  Basically kvm kernel modules are not being loaded after after installing/configuring aeolus and rhevm/vsphere builds fail with "OzException: This host does not support virtualization type kvm or qemu".

Loading the appropriate kvm modules and restarting libvirtd and imagefactory did not help.  It was only after a reboot that building begain working.

Version-Release number of selected component (if applicable):
=================================================================
imagefactory-1.0.0rc11-1.el6.noarch


How reproducible:
====================
100%

Steps to Reproduce:
=======================
1.  install CFCE v1.0.1 on RHEL 6.3
2.  configure for rhevm and/or vsphere
3.  build image
  
Actual results:
===============
OzException

Expected results:
=================
built image

Comment 1 Steve Loranz 2012-06-07 21:06:08 UTC
This seems like something that should be logged against libvirt/kvm instead of imagefactory. I do not see imagefactory or Oz doing any kvm setup other than requiring libvirt as one of the rpm dependencies.

Comment 2 jrd 2012-06-08 15:35:26 UTC
Our current theory is that this has nothing to do with image factory, and is pointing to an underlying kvm or libvirt problem.  Dave?

Comment 3 Dave Allan 2012-06-08 17:03:36 UTC
This behavior really sounds like libvirt wasn't restarted after KVM was running properly.  Eric, can you work with Dave J. on this?

Comment 4 Eric Blake 2012-06-08 17:08:11 UTC
Are you able to easily reproduce the scenario?  Can you run 'virt-host-validate' both before and after each step that you take in your efforts to get things started (enabling the kvm module, restarting libvirtd, and finally rebooting), to see if it detects something obviously difference between each step?

Comment 5 Dave Johnson 2012-06-13 15:55:52 UTC
So basically this comes down to the loading of the kvm modules.  On 6.2 installs, this would happen during the install of imagefactory and its dependencies (libvirt/qemu/etc) but that doesn't seem to be the case in rhel 6.3.

The virt-host-validate command shows exactly that (on 6.3).  After installing aeolus and its components, in 6.2 the kvm modules are loaded, in 6.3 that is not the case until they are either manually loaded or a reboot is performed.

Any idea if that is expected?  We could get our aeolus-configure setup script to load these if this is now needed.

Comment 6 Eric Blake 2012-06-13 19:41:02 UTC
Fedora packaging policy is that installing an rpm must NOT start a service by default (instead, it can register the service to be auto-started on next reboot, or the user can manually start it after installation if they don't want to wait for reboot).  This is so that it is possible to install an rpm into a chroot jail, where starting a service makes no sense.  For a similar reason, I'm finding it hard to believe that installing any package should ever load a kernel module - I can see that the first time you USE a package (whether by manual start or by rebooting after the install registered the package for autostart) being the useful place to load a module.

I don't know if the change to not autoload the kvm module was intentional, or for that matter, even which package it was that in 6.2 was autoloading the kvm module.  But since I think the new 6.3 semantics of 'install does not autoload modules' seems to make sense, I think it may be an issue where aeolus-configure will need to load the kvm module rather than relying on installation doing the same trick.

Comment 10 Dave Johnson 2012-06-14 03:59:10 UTC
Created attachment 591725 [details]
'yum install qemu-kvm' output

qemu-kvm-0.12.1.2-2.209.el6.x86_64 is the one loading the kvm modules on its rpm install in rhel 6.2... for whatever reason it stopped doing this in 6.3 (which is installing qemu-kvm.x86_64 2:0.12.1.2-2.295.el6)

Comment 11 Eric Blake 2012-06-14 11:26:02 UTC
Reassigning to qemu-kvm (perhaps temporarily) to determine if the change in the rpm installation no longer loading the kernel module was intentional.  If so, we can reassign it back to imagefactory to deal with the fallout.

Comment 13 jrd 2012-06-14 13:33:57 UTC
(In reply to comment #11)
> Reassigning to qemu-kvm (perhaps temporarily) to determine if the change in
> the rpm installation no longer loading the kernel module was intentional. 
> If so, we can reassign it back to imagefactory to deal with the fallout.

Well, ok, so it sounds like where you're going with this is that if some random app (eg imagefactory) wants services like libvirtd, it's responsible for ensuring they're started?  That doesn't sound unreasonable, btw, just making sure I'm correctly parsing the intent...

Comment 14 Eric Blake 2012-06-14 13:46:21 UTC
(In reply to comment #13)

> Well, ok, so it sounds like where you're going with this is that if some
> random app (eg imagefactory) wants services like libvirtd, it's responsible
> for ensuring they're started?  That doesn't sound unreasonable, btw, just
> making sure I'm correctly parsing the intent...

Yes, if imagefactory needs to run a qemu-kvm guest and starts by installing software, then it is reasonable for imagefactory to ensure that after qemu and libvirt are installed, that the kvm kernel module loaded, and libvirtd started, all before trying to use them, since the install process alone should not be starting services or loading modules.

Comment 15 jrd 2012-06-14 17:11:01 UTC
Well, ok, but to my knowledge, imagefac doesn't believe it's installing software on the host at runtime.  IOW, we come up and try to start doing our work, and implicitly assume that the needed services are there.

Was the scenario which spawned this bz as follows?

1.  start from base image
2.  install software including imagefac, which by side-effect pulled in libvirt
3.  start imagefac

?

Even in that case, it may be that imagefac should expect to ensure, on its own, that right services are started.  But if, for example, we believe that this scenario should work without imagefac explicitly starting services...

1.  start from base image
2.  install software including imagefac, which by side-effect pulled in libvirt
3.  reboot
4.  start imagefac


... then I'd argue that this bug should be closed.  

Now, one could argue that imagefac should not be making assumptions, and even if installing code and rebooting the machine might be expected to start things, good practice says we should assume nothing, that's fine too.

Mostly just trying to get crisp about what expectations should be about services.

Comment 16 Ademar Reis 2012-06-18 21:33:41 UTC
Based on comment 15, it doesn't look like there's anything qemu-kvm can do, so I'm moving this bug back to imagefactory.

Comment 17 James Laska 2012-06-25 15:09:51 UTC
What is the next step for this issue?  Do the imagefactory SysV (and systemd) startup procedures need to check whether libvirtd is running *and* whether the kvm module(s) are loaded?

Comment 18 Thomas Oulevey 2012-06-29 07:56:08 UTC
Hi,

It seems there is a bug in qemu-kvm.

In the %post section of the spec file Redhat added a If statement that change the behavior of qemu-kvm package if qemu-kvm-agent is defined, then the sh %{_sysconfdir}/sysconfig/modules/kvm.modules script is not added to qemu-kvm package because another %post statement is added and change the logic.

Thomas.

Comment 19 Dave Johnson 2012-07-11 15:07:27 UTC
moving to qemu-kvm team based on comment 18

Comment 20 Thomas Oulevey 2012-07-11 15:14:09 UTC
BTW, private bugzilla #836498 has been opened by our support contact.

Comment 22 Ademar Reis 2012-07-11 17:08:22 UTC
Closing as dupe of Bug 836498, which has a customer portal case attached to it.

*** This bug has been marked as a duplicate of bug 836498 ***

Comment 23 Garrett Holmstrom 2012-07-11 18:44:45 UTC
Do I have any hope of being able to follow this bug in the future?  This affects Eucalyptus as well.

Comment 24 Ademar Reis 2012-07-11 19:46:28 UTC
Garret, the other bug is restricted at the moment (I'm checking if we can open the other bug to the public), but FYI, the patch is trivial:

--- qemu-kvm.spec.orig	2012-06-29 11:29:51.060386352 +0200
+++ qemu-kvm.spec	2012-06-29 11:30:25.141384511 +0200
@@ -8110,16 +8110,16 @@
 /sbin/chkconfig --add ksm
 /sbin/chkconfig --add ksmtuned
 
+# load kvm modules now, so we can make sure no reboot is needed.
+# If there's already a kvm module installed, we don't mess with it
+sh %{_sysconfdir}/sysconfig/modules/kvm.modules
+
 %if %{with guest_agent}
 %post -n qemu-guest-agent%{?pkgsuffix}
 /sbin/chkconfig --add qemu-ga
 /sbin/chkconfig qemu-ga off
 %endif # guest_agent
 
-# load kvm modules now, so we can make sure no reboot is needed.
-# If there's already a kvm module installed, we don't mess with it
-sh %{_sysconfdir}/sysconfig/modules/kvm.modules
-
 %preun
 if [ $1 -eq 0 ]; then
     /sbin/service ksmtuned stop &>/dev/null || :