692190 – error while bulding simple f13 template, libvirt_xml = self.guest.install(self.app_config["timeout"])

Bug 692190 - error while bulding simple f13 template, libvirt_xml = self.guest.install(self.app_config["timeout"])

Summary: error while bulding simple f13 template, libvirt_xml = self.guest.install(sel...

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	CloudForms Cloud Engine
Classification:	Retired
Component:	aeolus-conductor
Sub Component:
Version:	0.3.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	rc
Assignee:	Chris Lalancette
QA Contact:	wes hayutin
Docs Contact:
URL:	http://dhcp231-29.rdu.redhat.com/cond...
Whiteboard:
Depends On:
Blocks:	ce-beta
TreeView+	depends on / blocked

Reported:	2011-03-30 16:21 UTC by wes hayutin
Modified:	2012-01-26 12:28 UTC (History)
CC List:	6 users (show)
Fixed In Version:
Clone Of:
Environment:
Last Closed:
Embargoed:

Attachments	(Terms of Use)

Description wes hayutin 2011-03-30 16:21:50 UTC

Description of problem:

Error while building simple F13 template w/ no packages..

[root@dhcp231-29 ~]# rpm -qa | grep aeolus
aeolus-conductor-0.0.3-0.fc14.20110330133246git37c7457.x86_64
aeolus-configure-2.0.0-5.fc14.2011033093316gitecf8f95.noarch
aeolus-conductor-daemons-0.0.3-0.fc14.20110330133246git37c7457.x86_64
aeolus-conductor-doc-0.0.3-0.fc14.20110330133246git37c7457.x86_64



[root@dhcp231-29 ~]# cat /var/log/imagefactory.log 
2011-03-30 12:11:00,311 INFO root pid(3305) Message: Launched as daemon...
2011-03-30 12:11:00,455 INFO imagefactory.qmfagent.ImageFactoryAgent.ImageFactoryAgent pid(3305) Message: image_factory has qmf/qpid address: redhat.com:imagefactory:7cc0ed88-bdda-4959-b783-d8968ab4c6e5:image_factory
2011-03-30 12:16:24,166 DEBUG imagefactory.qmfagent.ImageFactoryAgent.ImageFactoryAgent pid(3305) Message: Method called: name = image 
 args = {'target': 'ec2', 'template': 'http://localhost:9090/templates/ffdba400-5ae8-11e0-83fb-001e4fac24ec'} 
 handle = <cqmf2.AgentEvent; proxy of <Swig Object of type 'qmf::AgentEvent *' at 0x1e40240> > 
 addr = redhat.com:imagefactory:7cc0ed88-bdda-4959-b783-d8968ab4c6e5:image_factory 
 subtypes = {} 
 userId = anonymous
2011-03-30 12:16:24,219 DEBUG oz.Guest.FedoraGuest pid(3305) Message: Name: aasdf, UUID: abb53648-c04e-4d26-a127-552d8440333a
2011-03-30 12:16:24,219 DEBUG oz.Guest.FedoraGuest pid(3305) Message: MAC: 52:54:00:51:f8:56, distro: Fedora
2011-03-30 12:16:24,219 DEBUG oz.Guest.FedoraGuest pid(3305) Message: update: 13, arch: x86_64, diskimage: /var/tmp/aasdf.dsk
2011-03-30 12:16:24,219 DEBUG oz.Guest.FedoraGuest pid(3305) Message: host IP: 192.168.122.1, nicmodel: virtio, clockoffset: utc
2011-03-30 12:16:24,219 DEBUG oz.Guest.FedoraGuest pid(3305) Message: mousetype: ps2, disk_bus: virtio, disk_dev: vda
2011-03-30 12:16:24,219 DEBUG oz.Guest.FedoraGuest pid(3305) Message: icicletmp: /var/lib/oz/icicletmp/aasdf, listen_port: 52028
2011-03-30 12:16:24,220 DEBUG oz.Guest.FedoraGuest pid(3305) Message: Cache original media?: True
2011-03-30 12:16:24,220 DEBUG oz.Guest.FedoraGuest pid(3305) Message: Original ISO path: /var/lib/oz/isos/Fedora13x86_64-url.iso
2011-03-30 12:16:24,220 DEBUG oz.Guest.FedoraGuest pid(3305) Message: Modified ISO cache: /var/lib/oz/isos/Fedora13x86_64-url-oz.iso
2011-03-30 12:16:24,220 DEBUG oz.Guest.FedoraGuest pid(3305) Message: Output ISO path: /var/tmp/aasdf-url-oz.iso
2011-03-30 12:16:24,220 DEBUG oz.Guest.FedoraGuest pid(3305) Message: ISO content path: /var/lib/oz/isocontent/aasdf-url
2011-03-30 12:16:24,221 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(3305) Message: build() called on FedoraBuilder...
2011-03-30 12:16:24,221 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(3305) Message: Building for target ec2 with warehouse config http://localhost:9090/
2011-03-30 12:16:24,222 DEBUG imagefactory.qmfagent.BuildAdaptor.BuildAdaptor pid(3305) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed status from NEW to BUILDING
2011-03-30 12:16:24,222 INFO oz.Guest.FedoraGuest pid(3305) Message: Cleaning up guest named aasdf
2011-03-30 12:16:24,228 INFO oz.Guest.FedoraGuest pid(3305) Message: Generating install media
2011-03-30 12:16:24,228 INFO oz.Guest.FedoraGuest pid(3305) Message: Fetching the original media
2011-03-30 12:16:24,736 INFO oz.Guest.FedoraGuest pid(3305) Message: Fetching the original install media from ftp://gromit.redhat.com//pub/fedora/linux/releases/13/Fedora/x86_64/os/images/boot.iso
2011-03-30 12:16:24,808 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 0kB of 212992kB
2011-03-30 12:16:25,792 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 10240kB of 212992kB
2011-03-30 12:16:26,838 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 20481kB of 212992kB
2011-03-30 12:16:27,729 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 30720kB of 212992kB
2011-03-30 12:16:28,633 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 40961kB of 212992kB
2011-03-30 12:16:29,543 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 51201kB of 212992kB
2011-03-30 12:16:30,449 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 61440kB of 212992kB
2011-03-30 12:16:31,359 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 71681kB of 212992kB
2011-03-30 12:16:32,269 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 81920kB of 212992kB
2011-03-30 12:16:33,168 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 92161kB of 212992kB
2011-03-30 12:16:34,087 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 102400kB of 212992kB
2011-03-30 12:16:34,978 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 112640kB of 212992kB
2011-03-30 12:16:35,903 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 122880kB of 212992kB
2011-03-30 12:16:36,815 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 133120kB of 212992kB
2011-03-30 12:16:37,707 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 143360kB of 212992kB
2011-03-30 12:16:38,600 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 153600kB of 212992kB
2011-03-30 12:16:39,499 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 163840kB of 212992kB
2011-03-30 12:16:40,412 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 174081kB of 212992kB
2011-03-30 12:16:41,304 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 184320kB of 212992kB
2011-03-30 12:16:42,285 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 194560kB of 212992kB
2011-03-30 12:16:43,177 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 204800kB of 212992kB
2011-03-30 12:16:43,956 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 212992kB of 212992kB
2011-03-30 12:16:43,957 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 212992kB of 212992kB
2011-03-30 12:16:43,957 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 212992kB of 212992kB
2011-03-30 12:16:43,957 DEBUG oz.Guest.FedoraGuest pid(3305) Message: 212992kB of 212992kB
2011-03-30 12:16:43,957 INFO oz.Guest.FedoraGuest pid(3305) Message: Copying ISO contents for modification
2011-03-30 12:16:43,958 INFO oz.Guest.FedoraGuest pid(3305) Message: Setting up guestfs handle for aasdf
2011-03-30 12:16:43,958 DEBUG oz.Guest.FedoraGuest pid(3305) Message: Adding ISO image /var/lib/oz/isos/Fedora13x86_64-url.iso
2011-03-30 12:16:43,958 DEBUG oz.Guest.FedoraGuest pid(3305) Message: Launching guestfs
2011-03-30 12:17:16,687 DEBUG oz.Guest.FedoraGuest pid(3305) Message: Mounting ISO
2011-03-30 12:17:16,719 DEBUG oz.Guest.FedoraGuest pid(3305) Message: Checking if there is enough space on the filesystem
2011-03-30 12:17:16,720 DEBUG oz.Guest.FedoraGuest pid(3305) Message: Extracting ISO contents
2011-03-30 12:17:20,821 DEBUG oz.Guest.FedoraGuest pid(3305) Message: Putting the kickstart in place
2011-03-30 12:17:20,822 DEBUG oz.Guest.FedoraGuest pid(3305) Message: Modifying the boot options
2011-03-30 12:17:20,823 DEBUG oz.Guest.FedoraGuest pid(3305) Message: Generating new ISO
2011-03-30 12:17:21,472 INFO oz.Guest.FedoraGuest pid(3305) Message: Cleaning up old ISO data
2011-03-30 12:17:22,069 DEBUG imagefactory.qmfagent.BuildAdaptor.BuildAdaptor pid(3305) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed percent complete from 0 to 10
2011-03-30 12:17:22,070 INFO oz.Guest.FedoraGuest pid(3305) Message: Generating 10GB diskimage with fake partition for aasdf
2011-03-30 12:17:22,070 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(3305) Message: Doing base install via Oz
2011-03-30 12:17:22,070 INFO oz.Guest.FedoraGuest pid(3305) Message: Running install for aasdf
2011-03-30 12:17:22,071 INFO oz.Guest.FedoraGuest pid(3305) Message: Generate XML for guest aasdf with bootdev cdrom
2011-03-30 12:17:22,072 DEBUG oz.Guest.FedoraGuest pid(3305) Message: Generated XML:
<?xml version="1.0"?>
<domain type="kvm">
  <name>aasdf</name>
  <memory>1048576</memory>
  <currentMemory>1048576</currentMemory>
  <uuid>abb53648-c04e-4d26-a127-552d8440333a</uuid>
  <clock offset="utc"/>
  <vcpu>1</vcpu>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <os>
    <type>hvm</type>
    <boot dev="cdrom"/>
  </os>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <console device="pty"/>
    <graphics port="-1" type="vnc"/>
    <interface type="bridge">
      <source bridge="virbr0"/>
      <mac address="52:54:00:51:f8:56"/>
      <model type="virtio"/>
    </interface>
    <input bus="ps2" type="mouse"/>
    <console type="pty">
      <target port="0"/>
    </console>
    <disk device="disk" type="file">
      <target dev="vda" bus="virtio"/>
      <source file="/var/tmp/base-image-b1689f2a-af39-428a-a7a2-fa12259ac169.dsk"/>
    </disk>
    <disk type="file" device="cdrom">
      <source file="/var/tmp/aasdf-url-oz.iso"/>
      <target dev="hdc"/>
    </disk>
  </devices>
</domain>

2011-03-30 12:17:22,073 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(3305) Message: Unexpected error: (<class 'libvirt.libvirtError'>)
2011-03-30 12:17:22,073 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(3305) Message:              value: (unknown OS type hvm)
2011-03-30 12:17:22,074 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(3305) Message:          traceback:
2011-03-30 12:17:22,076 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(3305) Message:      File "/usr/lib/python2.7/site-packages/imagefactory/builders/FedoraBuilder.py", line 101, in build
    libvirt_xml = self.guest.install(self.app_config["timeout"])

2011-03-30 12:17:22,076 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(3305) Message:      File "/usr/lib/python2.7/site-packages/oz/Guest.py", line 895, in install
    dom = self.libvirt_conn.createXML(xml, 0)

2011-03-30 12:17:22,076 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(3305) Message:      File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1289, in createXML
    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)

2011-03-30 12:17:22,076 INFO oz.Guest.FedoraGuest pid(3305) Message: Cleaning up guest named aasdf
2011-03-30 12:17:22,077 DEBUG imagefactory.qmfagent.BuildAdaptor.BuildAdaptor pid(3305) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed status from BUILDING to FAILED
2011-03-30 12:17:22,078 INFO oz.Guest.FedoraGuest pid(3305) Message: Cleaning up after install
2011-03-30 12:17:22,179 DEBUG oz.Guest.FedoraGuest pid(3305) Message: Removed modified ISO

Comment 1 wes hayutin 2011-03-30 16:22:22 UTC

recreate:
1. install
2. create provider account
3. create f13 template w/o packages
4. build

Comment 2 Mike Orazi 2011-03-30 18:54:07 UTC

I believe this is slated to be addressed with a newer rev of Oz than we are currently carrying, but would like Chris to confirm.

Comment 3 Chris Lalancette 2011-03-30 22:08:55 UTC

I haven't done anything in particular to address this one.  While the error message is a little opaque, it actually shows what the problem is: the host that this was run on does not support HVM guests (i.e. KVM).  If this is a machine that doesn't support KVM at all, then it more-or-less can't be used to do builds.  If KVM is disabled in the BIOS, then it needs to be enabled.

Chris Lalancette

Comment 4 wes hayutin 2011-03-30 23:25:16 UTC

[root@dhcp231-29 ~]# cat /proc/cpuinfo | grep vmx
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca lahf_lm tpr_shadow vnmi flexpriority
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca lahf_lm tpr_shadow vnmi flexpriority
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca lahf_lm tpr_shadow vnmi flexpriority
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca lahf_lm tpr_shadow vnmi flexpriority
[root@dhcp231-29 ~]# 

While its possible that its turned off in the bios.. very true.. Going to try and install a virt guest by hand, which I have some several times on this hardware. Its possible some kind of config has changed on this hardware since then.

Thanks for looking into this bug.

Comment 5 wes hayutin 2011-03-30 23:38:49 UTC

[root@dhcp231-29 ~]# 
[root@dhcp231-29 ~]# # virt-install -n auto-rhq -r 2046 -f /var/lib/libvirt/images/auto-rhq.img -s 30 --vnc -l http://vault.rhndev.redhat.com/engarchive2/released/F-12/GOLD/Fedora/i386/os/ -x "noipv6 ks=http://whayutin.rdu.redhat.com/ks/ks.cfg" --bridge=virbr0

Fails w/ 

Mar 30 19:37:37 localhost libvirtd: 19:37:37.912: error : qemudDomainLookupByName:4403 : Domain not found: no domain with matching name 'auto-rhq'
Mar 30 19:37:37 localhost libvirtd: 19:37:37.918: error : storageVolumeLookupByPath:1241 : Storage volume not found: no storage vol with matching path
Mar 30 19:37:37 localhost libvirtd: 19:37:37.986: warning : qemudParsePCIDeviceStrs:1422 : Unexpected exit status '1', qemu probably failed
Mar 30 19:37:37 localhost libvirtd: 19:37:37.989: error : storageVolumeLookupByPath:1241 : Storage volume not found: no storage vol with matching path
Mar 30 19:37:37 localhost libvirtd: 19:37:37.992: error : storageVolumeLookupByPath:1241 : Storage volume not found: no storage vol with matching path
Mar 30 19:37:37 localhost libvirtd: 19:37:37.995: error : storageVolumeLookupByName:1151 : Storage volume not found: no storage vol with matching name 'auto-rhq.img'
Mar 30 19:37:38 localhost kernel: [16030.036493] [IPTABLES] INPUT : IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:ae:79:43:4f:08:00 SRC=10.11.228.35 DST=255.255.255.255 LEN=151 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=17500 DPT=17500 LEN=131 
Mar 30 19:37:58 localhost kernel: [16050.467380] [IPTABLES] INPUT : IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:a4:ba:db:87:98:ea:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=300 TOS=0x00 PREC=0x00 TTL=128 ID=50474 PROTO=UDP SPT=68 DPT=67 LEN=280 


so Chris is correct.. something is wrong... I'm not convinced its hardware yet though

Comment 6 wes hayutin 2011-03-31 12:54:57 UTC

k.. I've confirmed virt is enabled in the bios and libvirt was installed.

Some other things I noticed.
1. I was able to start virt install after rebooting using 

virt-install -n auto-rhq -r 2046 -f
/var/lib/libvirt/images/auto-rhq.img -s 30 --vnc -l
http://vault.rhndev.redhat.com/engarchive2/released/F-12/GOLD/Fedora/i386/os/
-x "noipv6 ks=http://whayutin.rdu.redhat.com/ks/ks.cfg" --bridge=virbr0

The install would hang when trying to pull something from the network.  Something must be wrong w/ the default network interface virb0.  On some hardware we create our own network bridge, and I wonder if there is a issue there.

2. virt was enabled in the bios


So w/ virt enabled in the bios, using a cpu w/ vmx flags and libvirt installed was not enough to successfully create a guest..  Not sure why yet.. that should be enough

Comment 7 wes hayutin 2011-03-31 15:44:06 UTC

OK.. Chris found the root cause here..
https://bugzilla.redhat.com/show_bug.cgi?id=692558

Comment 8 Dave Johnson 2011-04-01 02:56:26 UTC

*** Bug 692747 has been marked as a duplicate of this bug. ***

Comment 9 wes hayutin 2011-06-23 22:08:50 UTC

oot@ibm-x3950m2-01 noarch]# 
[root@ibm-x3950m2-01 noarch]# cat /var/log/aeolus-connector.log | grep -i complete
D, [2011-06-23T13:56:58.581322 #10459] DEBUG -- : GOT AN EVENT:  redhat.com:imagefactory:848fa44e-3eaf-4821-be1e-a351c2ddf8c1, percent_complete100eventPERCENTAGEaddr_object_namebuild_adaptor:build_image:8809de31-c008-457b-8578-f743948eac2c_agent_nameredhat.com:imagefactory:848fa44e-3eaf-4821-be1e-a351c2ddf8c1 at 1308851818579773033
D, [2011-06-23T13:56:58.591694 #10459] DEBUG -- : GOT AN EVENT:  redhat.com:imagefactory:848fa44e-3eaf-4821-be1e-a351c2ddf8c1, old_statusNEWeventSTATUSnew_statusCOMPLETEDaddr_object_namebuild_adaptor:build_image:8809de31-c008-457b-8578-f743948eac2c_agent_nameredhat.com:imagefactory:848fa44e-3eaf-4821-be1e-a351c2ddf8c1 at 1308851818585917097
D, [2011-06-23T13:56:58.632069 #10459] DEBUG -- : Data: #<FactoryRestHandler::EventData:0x7f2f6ebee818 @uuid="8809de31-c008-457b-8578-f743948eac2c", @obj="build_image", @value="completed", @event="STATUS">
D, [2011-06-23T13:57:44.712984 #10459] DEBUG -- : GOT AN EVENT:  redhat.com:imagefactory:848fa44e-3eaf-4821-be1e-a351c2ddf8c1, percent_complete100eventPERCENTAGEaddr_object_namebuild_adaptor:build_image:2da7c4ff-e68c-4fdb-8174-40f600bbcb6f_agent_nameredhat.com:imagefactory:848fa44e-3eaf-4821-be1e-a351c2ddf8c1 at 1308851864708022162
D, [2011-06-23T13:57:44.714291 #10459] DEBUG -- : GOT AN EVENT:  redhat.com:imagefactory:848fa44e-3eaf-4821-be1e-a351c2ddf8c1, old_statusNEWeventSTATUSnew_statusCOMPLETEDaddr_obje

[root@ibm-x3950m2-01 noarch]# rpm -qa | grep aeolus
aeolus-conductor-0.3.0-0.el6.20110623205403git551632a.noarch
aeolus-configure-2.0.1-0.el6.20110622123902gitdf4ae05.noarch
aeolus-conductor-doc-0.3.0-0.el6.20110623205403git551632a.noarch
aeolus-all-0.3.0-0.el6.20110623205403git551632a.noarch
rubygem-aeolus-cli-0.0.1-1.el6.20110623205403git551632a.noarch
aeolus-conductor-daemons-0.3.0-0.el6.20110623205403git551632a.noarch
aeolus-conductor-devel-0.3.0-0.el6.20110623205403git551632a.noarch

Comment 10 wes hayutin 2011-08-01 19:53:08 UTC

release pending...

Comment 11 wes hayutin 2011-08-01 19:55:30 UTC

release pending...

Comment 13 wes hayutin 2011-12-08 13:50:57 UTC

closing out old bugs

Comment 14 wes hayutin 2011-12-08 14:03:48 UTC

perm close

Note You need to log in before you can comment on or make changes to this bug.