Bug 692747

Summary: libvirtd error using oz v3
Product: [Retired] CloudForms Cloud Engine Reporter: Dave Johnson <dajohnso>
Component: imagefactoryAssignee: Chris Lalancette <clalance>
Status: CLOSED CURRENTRELEASE QA Contact: Dave Johnson <dajohnso>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 0.3.1CC: cpelland, deltacloud-maint, whayutin
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-04-01 02:56:26 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dave Johnson 2011-04-01 02:48:55 UTC 
Description of problem:

Running the latest aeolus rpms with oz v3 ran into libvirtd error (below).  I believe it is the same error as the one we were getting when running on hosts with VT on the cpu.  This is bare metal with vmx so no idea what the deal is.  Hit it three times on separate installs.



Version-Release number of selected component (if applicable):

[root@hp-dl120g6-01 log]# rpm -qa | egrep 'aeolus|oz'
aeolus-conductor-0.0.3-0.fc14.20110401012616git41f8ba8.x86_64
aeolus-conductor-daemons-0.0.3-0.fc14.20110401012616git41f8ba8.x86_64
oz-0.2.0-1.fc14.x86_64
aeolus-conductor-doc-0.0.3-0.fc14.20110401012616git41f8ba8.x86_64
aeolus-configure-2.0.0-5.fc14.20110331212647gitcd696c2.noarch


Additional info:

beaker box: 
https://beaker.engineering.redhat.com/view/hp-dl120g6-01.rhts.eng.bos.redhat.com

[root@hp-dl120g6-01 log]# cat /proc/cpuinfo | grep vmx
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm tpr_shadow vnmi flexpriority ept vpid


[root@hp-dl120g6-01 log]# tail -f imagefactory.log 
2011-03-31 22:23:12,927 DEBUG oz.Guest.FedoraGuest pid(3956) Message: Output ISO path: /var/tmp/empty2-url-oz.iso
2011-03-31 22:23:12,927 DEBUG oz.Guest.FedoraGuest pid(3956) Message: ISO content path: /var/lib/oz/isocontent/empty2-url
2011-03-31 22:23:12,928 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(3956) Message: build() called on FedoraBuilder...
2011-03-31 22:23:12,928 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(3956) Message: Building for target ec2 with warehouse config http://localhost:9090/
2011-03-31 22:23:12,928 DEBUG imagefactory.qmfagent.BuildAdaptor.BuildAdaptor pid(3956) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed status from NEW to BUILDING
2011-03-31 22:23:12,929 INFO oz.Guest.FedoraGuest pid(3956) Message: Cleaning up guest named empty2
2011-03-31 22:23:12,936 INFO oz.Guest.FedoraGuest pid(3956) Message: Generating install media
2011-03-31 22:23:12,936 INFO oz.Guest.FedoraGuest pid(3956) Message: Fetching the original media
2011-03-31 22:23:19,196 INFO oz.Guest.FedoraGuest pid(3956) Message: Fetching the original install media from ftp://gromit.redhat.com//pub/fedora/linux/releases/13/Fedora/x86_64/os/images/boot.iso
2011-03-31 22:23:19,717 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 0kB of 212992kB
2011-03-31 22:23:22,983 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 10241kB of 212992kB
2011-03-31 22:23:26,239 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 20481kB of 212992kB
2011-03-31 22:23:29,424 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 30721kB of 212992kB
2011-03-31 22:23:32,672 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 40961kB of 212992kB
2011-03-31 22:23:35,908 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 51200kB of 212992kB
2011-03-31 22:23:39,124 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 61442kB of 212992kB
2011-03-31 22:23:42,361 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 71680kB of 212992kB
2011-03-31 22:23:45,614 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 81920kB of 212992kB
2011-03-31 22:23:48,839 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 92161kB of 212992kB
2011-03-31 22:23:53,146 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 102400kB of 212992kB
2011-03-31 22:23:56,885 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 112641kB of 212992kB
2011-03-31 22:24:00,096 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 122880kB of 212992kB
2011-03-31 22:24:03,337 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 133121kB of 212992kB
2011-03-31 22:24:06,517 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 143360kB of 212992kB
2011-03-31 22:24:09,712 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 153600kB of 212992kB
2011-03-31 22:24:12,919 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 163840kB of 212992kB
2011-03-31 22:24:16,117 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 174080kB of 212992kB
2011-03-31 22:24:19,335 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 184320kB of 212992kB
2011-03-31 22:24:22,563 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 194561kB of 212992kB
2011-03-31 22:24:25,801 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 204800kB of 212992kB
2011-03-31 22:24:29,109 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 212992kB of 212992kB
2011-03-31 22:24:29,110 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 212992kB of 212992kB
2011-03-31 22:24:29,110 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 212992kB of 212992kB
2011-03-31 22:24:29,110 DEBUG oz.Guest.FedoraGuest pid(3956) Message: 212992kB of 212992kB
2011-03-31 22:24:29,132 INFO oz.Guest.FedoraGuest pid(3956) Message: Copying ISO contents for modification
2011-03-31 22:24:29,133 INFO oz.Guest.FedoraGuest pid(3956) Message: Setting up guestfs handle for empty2
2011-03-31 22:24:29,133 DEBUG oz.Guest.FedoraGuest pid(3956) Message: Adding ISO image /var/lib/oz/isos/Fedora13x86_64-url.iso
2011-03-31 22:24:29,133 DEBUG oz.Guest.FedoraGuest pid(3956) Message: Launching guestfs
2011-03-31 22:24:50,267 DEBUG oz.Guest.FedoraGuest pid(3956) Message: Mounting ISO
2011-03-31 22:24:50,282 DEBUG oz.Guest.FedoraGuest pid(3956) Message: Checking if there is enough space on the filesystem
2011-03-31 22:24:50,282 DEBUG oz.Guest.FedoraGuest pid(3956) Message: Extracting ISO contents
2011-03-31 22:24:52,142 DEBUG oz.Guest.FedoraGuest pid(3956) Message: Putting the kickstart in place
2011-03-31 22:24:52,143 DEBUG oz.Guest.FedoraGuest pid(3956) Message: Modifying the boot options
2011-03-31 22:24:52,145 DEBUG oz.Guest.FedoraGuest pid(3956) Message: Generating new ISO
2011-03-31 22:24:52,429 INFO oz.Guest.FedoraGuest pid(3956) Message: Cleaning up old ISO data
2011-03-31 22:24:54,959 DEBUG imagefactory.qmfagent.BuildAdaptor.BuildAdaptor pid(3956) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed percent complete from 0 to 10
2011-03-31 22:24:54,960 INFO oz.Guest.FedoraGuest pid(3956) Message: Generating 10GB diskimage with fake partition for empty2
2011-03-31 22:24:54,960 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(3956) Message: Doing base install via Oz
2011-03-31 22:24:54,960 INFO oz.Guest.FedoraGuest pid(3956) Message: Running install for empty2
2011-03-31 22:24:54,960 INFO oz.Guest.FedoraGuest pid(3956) Message: Generate XML for guest empty2 with bootdev cdrom
2011-03-31 22:24:54,961 DEBUG oz.Guest.FedoraGuest pid(3956) Message: Generated XML:
<?xml version="1.0"?>
<domain type="kvm">
  <name>empty2</name>
  <memory>1048576</memory>
  <currentMemory>1048576</currentMemory>
  <uuid>58c4a917-12e2-4f09-80b0-33b792235865</uuid>
  <clock offset="utc"/>
  <vcpu>1</vcpu>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <os>
    <type>hvm</type>
    <boot dev="cdrom"/>
  </os>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <console device="pty"/>
    <graphics port="-1" type="vnc"/>
    <interface type="bridge">
      <source bridge="virbr0"/>
      <mac address="52:54:00:c5:e9:bb"/>
      <model type="virtio"/>
    </interface>
    <input bus="ps2" type="mouse"/>
    <console type="pty">
      <target port="0"/>
    </console>
    <disk device="disk" type="file">
      <target dev="vda" bus="virtio"/>
      <source file="/var/tmp/base-image-52f7d71f-91c6-43ae-a5ad-4592725d0eba.dsk"/>
    </disk>
    <disk type="file" device="cdrom">
      <source file="/var/tmp/empty2-url-oz.iso"/>
      <target dev="hdc"/>
    </disk>
  </devices>
</domain>

2011-03-31 22:24:54,963 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(3956) Message: Unexpected error: (<class 'libvirt.libvirtError'>)
2011-03-31 22:24:54,963 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(3956) Message:              value: (unknown OS type hvm)
2011-03-31 22:24:54,963 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(3956) Message:          traceback:
2011-03-31 22:24:54,964 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(3956) Message:      File "/usr/lib/python2.7/site-packages/imagefactory/builders/FedoraBuilder.py", line 101, in build
    libvirt_xml = self.guest.install(self.app_config["timeout"])

2011-03-31 22:24:54,964 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(3956) Message:      File "/usr/lib/python2.7/site-packages/oz/Guest.py", line 895, in install
    dom = self.libvirt_conn.createXML(xml, 0)

2011-03-31 22:24:54,965 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(3956) Message:      File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1289, in createXML
    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)

2011-03-31 22:24:54,965 INFO oz.Guest.FedoraGuest pid(3956) Message: Cleaning up guest named empty2
2011-03-31 22:24:54,965 DEBUG imagefactory.qmfagent.BuildAdaptor.BuildAdaptor pid(3956) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed status from BUILDING to FAILED
2011-03-31 22:24:54,966 INFO oz.Guest.FedoraGuest pid(3956) Message: Cleaning up after install
Comment 1 Dave Johnson 2011-04-01 02:56:26 UTC 

*** This bug has been marked as a duplicate of bug 692190 ***
Comment 2 wes hayutin 2011-04-01 02:58:12 UTC 
same trace from https://bugzilla.redhat.com/show_bug.cgi?id=692190

cloud engine components are putting libvirt into a state where guests can not get networking.
Comment 3 wes hayutin 2011-12-08 14:17:01 UTC 
perm close