Bug 758900

Summary: no KVM-related tests run because of this failure: - /distribution/virt/start
Product: [Retired] Beaker Reporter: Larry Troan <ltroan>
Component: testsAssignee: Jeff Burke <jburke>
Status: CLOSED NOTABUG QA Contact: tools-bugs <tools-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 0.6CC: bpeck, dcallagh, ichute, jburke, jstancek, junichi.nomura, juzhang, llim, michen, rmancy, stl, tatsu-ab1, xjia
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: Misc
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-04 13:35:56 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
receipe of kvm failure
none
Test log of /distribution/virt/install
none
rhel6_x86_64_install.log
none
TESTOUT.log of /distribution/virt/install (on RHEL6, beaker 0.9.0-7)
none
TESTOUT.log of /distribution/virt/install (on RHEL5, beaker 0.9.0-7) none

Description Larry Troan 2011-11-30 22:48:13 UTC 
Description of problem:

Bill Peck suggested Nomura-san forward this data to the kvm test engineer. Opening a bug with the data on his behalf and adding him to the cc list so he can add additional information.

-----
On 09/15/11 19:16, Jun'ichi Nomura wrote:
> And no KVM-related tests run because of this failure:
>>   - /distribution/virt/start
>
> It fails to find 'virtinst-vmlinuz.******'
> It's strange why it tries to access a file with such name
> while the installation has already finished.

With recent updates in Fuchu Beaker, all tests on KVM guest became
not working in our lab.
Simple example of failing job:
https://flh9aaf026.tky.mesh.ad.jp/bkr/jobs/1233
(The tests used to be working fine with older versions of /distribution/virt/*)

Since RHEL6.2 beta-cycle has already started, I hope this to be fixed soon.

Could you investigate this problem and/or send me information how to
run KVM testing in Beaker?

# Should I send this type of request to quality-engineering
# instead of normal e-mail?

Thanks,
-- 
Jun'ichi Nomura, NEC Corporation
Comment 1 Larry Troan 2011-12-13 17:57:51 UTC 
From Nomura-san (he apparently doesn't have write access to this bug):

Hi,

I'm trying to put a comment in the subject bug but rejected as:
"You are not permitted to edit bugs in product RHEL Tests. "

So I'm sending in e-mail instead:
-------------------------------------------------------------
Additional information about related packages:

$ ls /var/www/beaker/rpms/*virt*
/var/www/beaker/rpms/rh-tests-distribution-virtinstall-3.7-142.noarch.rpm
/var/www/beaker/rpms/rh-tests-distribution-virtinstall-3.7-153.noarch.rpm
/var/www/beaker/rpms/rh-tests-distribution-virt-start-2.0-18.noarch.rpm
/var/www/beaker/rpms/rh-tests-distribution-virt-start_stop-2.0-11.noarch.rpm
/var/www/beaker/rpms/rh-tests-distribution-virt-stop-2.0-13.noarch.rpm

$ rpm -q beaker
beaker-0.6.14-7.el5

$ ls /var/www/beaker/harness/RedHatEnterpriseLinux6/beah-*
/var/www/beaker/harness/RedHatEnterpriseLinux6/beah-0.6.25-1.git.5.8361de7.el6.noarch.rpm
/var/www/beaker/harness/RedHatEnterpriseLinux6/beah-0.6.26-1.git.1.0471125.el6eso.noarch.rpm
/var/www/beaker/harness/RedHatEnterpriseLinux6/beah-0.6.29-1.el6eso.noarch.rpm

$ ls /var/www/beaker/harness/RedHatEnterpriseLinuxServer5/beah-*
/var/www/beaker/harness/RedHatEnterpriseLinuxServer5/beah-0.6.25-1.git.5.8361de7.el5.noarch.rpm
/var/www/beaker/harness/RedHatEnterpriseLinuxServer5/beah-0.6.29-1.el5.noarch.rpm
-------------------------------------------------------------

Since this issue prevents us from running KVM-related tests
in Beaker, your help is appreciated.
Comment 2 Larry Troan 2012-05-13 01:16:15 UTC 
From Miya Chen who performs KVM testing for Red Hat in Beaker....

On 05/11/2012 01:40 AM, Miya Chen wrote:
> On 05/11/2012 05:23 AM, Larry Troan wrote:
>> Bill, Miya,
>>
>> Do you know if the KVM tests can now run in RHTS/Beaker?
>>
> Hi Larry,
> do you mean if we can run rhel testing over kvm guests in Beaker? if it 
> is, I think the answer is we can, the followed is a recent job that we 
> submitted in beaker to run kernel tier1 test cases in kvm guest, you can 
> have a check:
> https://beaker.engineering.redhat.com/jobs/227239
> 
> Correct me if I were wrong, thanks.
> 
> Regards,
> Miya
Comment 3 Larry Troan 2012-05-13 01:46:51 UTC 
> https://beaker.engineering.redhat.com/jobs/227239
In the subject job, 2 of 3 recipes ran successfully. Within the third failing recipt, there were 207 tests run with 110=pass and 97=fail.
Comment 4 Larry Troan 2012-05-24 06:43:45 UTC 
On 05/13/2012 08:17 PM, Jun'ichi Nomura wrote:
> So some jobs run fine in Red Hat lab.
> In Fuchu, all KVM job I run fail.
> 
> Attached XML is for the job mentioned in the initial
> comment of BZ#758900.
> It creates a KVM guest and runs a test on it.
> However, the guest creation fails in Fuchu beaker.
> 
> Could you try this XML in Red Hat lab?
> (You have to change "/examples/nec/distro/hackbench" to
>   a test available in Red Hat lab,
>   and "b120b.rhts.fuchu" to a target hostname.)
> 
> Or could you attach one of your job XML here so that I can
> try on Fuchu lab?
> 
> -- Jun'ichi Nomura, NEC Corporation
Comment 5 Larry Troan 2012-05-24 06:44:38 UTC 
<job retention_tag="scratch">
	<whiteboard>
		virt test
	</whiteboard>
	<recipeSet priority="Urgent">
		<recipe kernel_options="" kernel_options_post="" ks_meta="" role="None" whiteboard="">
			<guestrecipe guestargs="--os-variant=rhel6 --nographics --ram=2048 --vcpus=4 --file-size=20 --hvm --kvm" guestname="rhel6_x86_64" kernel_options="" kernel_options_post="" ks_meta="" role="None" whiteboard="">
				<autopick random="false"/>
				<watchdog/>
				<packages/>
				<ks_appends/>
				<repos/>
				<distroRequires>
					<and>
						<distro_name op="=" value="RHEL6-U1"/>
						<distro_arch op="=" value="x86_64"/>
					</and>
					<distro_virt op="=" value=""/>
				</distroRequires>
				<hostRequires>
					<and/>
				</hostRequires>
				<partitions/>
				<task name="/distribution/install" role="STANDALONE">
					<params/>
				</task>
				<task name="/examples/nec/distro/hackbench" role="STANDALONE">
					<params>
						<param name="TEST_TITLE" value="'RHEL6-U1/x86_64 on KVM RHEL6-U1'"/>
					</params>
				</task>
			</guestrecipe>
			<autopick random="false"/>
			<watchdog/>
			<packages/>
			<ks_appends/>
			<repos/>
			<distroRequires>
				<and>
					<distro_name op="=" value="RHEL6-U1"/>
					<distro_arch op="=" value="x86_64"/>
				</and>
				<distro_virt op="=" value=""/>
			</distroRequires>
			<hostRequires>
				<and>
					<hostname op="=" value="b120b.rhts.fuchu"/>
				</and>
			</hostRequires>
			<partitions/>
			<task name="/distribution/install" role="STANDALONE">
				<params/>
			</task>
			<task name="/distribution/virt/install" role="STANDALONE">
				<params/>
			</task>
			<task name="/distribution/virt/start" role="STANDALONE">
				<params/>
			</task>
		</recipe>
	</recipeSet>
</job>
Comment 6 Larry Troan 2012-05-24 06:48:21 UTC 
Created attachment 586563 [details]
receipe of kvm failure
Comment 7 Gurhan Ozen 2012-05-25 13:54:00 UTC 
Sorry, I can't see that url either. 
This is usually caused by libvirt not being able to start the guest and is usually an indication of something going awry with guest installation. Either the guest was corrupted or for whatever reason libvirt can't start the guest. All /distribution/virt/start test does is to issue virsh start $guestname command and either fail or pass based on the return code of that operation. 

Another thing that might come into play is the older version of /distribution/virt/start rpm they are using. Currently it's 2-21 in beaker, is it possible to have them upgrade these? Somewhat recently, we have changed the way beaker guests handle console logging etc and minor changes needed for some of those tests, so it'd be great if they could update all /distribution/virt/* rpms.
Comment 8 Jun'ichi NOMURA 2012-05-28 05:11:29 UTC 
Created attachment 587140 [details]
Test log of /distribution/virt/install

https://flh9aaf026.tky.mesh.ad.jp/bkr/logs/2011/09/12/1233/4310/37000//TESTOUT.log
Comment 9 Jun'ichi NOMURA 2012-05-28 05:13:21 UTC 
Created attachment 587141 [details]
rhel6_x86_64_install.log

https://flh9aaf026.tky.mesh.ad.jp/bkr/logs/2011/09/12/1233/4310/37000//rhel6_x86_64_install.log
Comment 10 Jun'ichi NOMURA 2012-05-28 05:23:56 UTC 
(In reply to comment #7)
> Sorry, I can't see that url either. 

Uploaded 2 of logs from /distribution/virt/install.
If you want other logs, please let me know.

> This is usually caused by libvirt not being able to start the guest and is
> usually an indication of something going awry with guest installation.

There are following messages in the logs:
  Tue, 27 Sep 2011 21:53:25 DEBUG    Started guest, connecting to console if requested
  Tue, 27 Sep 2011 21:53:25 DEBUG    Launching console callbackConnected to domain rhel6_x86_64
  error: internal error character device (null) is not using a PTY

  Tue, 27 Sep 2011 21:53:26 DEBUG    Removing /var/lib/libvirt/boot/virtinst-vmlinuz.1Mjul1
  Tue, 27 Sep 2011 21:53:26 DEBUG    Removing /var/lib/libvirt/boot/virtinst-initrd.img.BvEYdI
  Tue, 27 Sep 2011 21:53:28 DEBUG    Domain state after install: 1

3 seconds seem too short for installation.

> it'd be great if they could update all /distribution/virt/* rpms.

Bill, Raymond, could you do that?
Comment 11 Bill Peck 2012-05-29 18:07:58 UTC 
(In reply to comment #10)
> (In reply to comment #7)
> > Sorry, I can't see that url either. 
> 
> Uploaded 2 of logs from /distribution/virt/install.
> If you want other logs, please let me know.
> 
> > This is usually caused by libvirt not being able to start the guest and is
> > usually an indication of something going awry with guest installation.
> 
> There are following messages in the logs:
>   Tue, 27 Sep 2011 21:53:25 DEBUG    Started guest, connecting to console if
> requested
>   Tue, 27 Sep 2011 21:53:25 DEBUG    Launching console callbackConnected to
> domain rhel6_x86_64
>   error: internal error character device (null) is not using a PTY
> 
>   Tue, 27 Sep 2011 21:53:26 DEBUG    Removing
> /var/lib/libvirt/boot/virtinst-vmlinuz.1Mjul1
>   Tue, 27 Sep 2011 21:53:26 DEBUG    Removing
> /var/lib/libvirt/boot/virtinst-initrd.img.BvEYdI
>   Tue, 27 Sep 2011 21:53:28 DEBUG    Domain state after install: 1
> 
> 3 seconds seem too short for installation.
> 
> > it'd be great if they could update all /distribution/virt/* rpms.
> 
> Bill, Raymond, could you do that?

Hi Junichi,

I updated the virt rpms and kicked off a test job.

https://flh9aaf026.tky.mesh.ad.jp/bkr/jobs/2069
Comment 12 Jun'ichi NOMURA 2012-05-30 01:26:35 UTC 
(In reply to comment #11)
> I updated the virt rpms and kicked off a test job.
> 
> https://flh9aaf026.tky.mesh.ad.jp/bkr/jobs/2069

Thank you Bill. So now we have:
  distribution-distribution-virt-install-3.8-8.noarch.rpm
  rh-tests-distribution-virt-stop-2.0-15.noarch.rpm
  rh-tests-distribution-virt-start_stop-2.0-17.noarch.rpm
  rh-tests-distribution-virt-start-2.0-21.noarch.rpm

But the job 2069 failed in the same way as before...

Could it be possible for you to run a job, which is known to be
working in Red Hat site?
Then we could see whether my XML is bad or not.

I found the job failed differently for RHEL6 host and RHEL5 host.

For RHEL6 host:
  Creating domain...                                       |    0 B     00:00     
  Tue, 29 May 2012 14:23:21 DEBUG    Started guest, connecting to console if requested
  Tue, 29 May 2012 14:23:21 DEBUG    Launching console callbackConnected to domain rhel6_x86_64
  Escape character is ^]
  error: internal error character device (null) is not using a PTY

  Tue, 29 May 2012 14:23:22 DEBUG    Removing /var/lib/libvirt/boot/virtinst-vmlinuz.BVA77p
  Tue, 29 May 2012 14:23:22 DEBUG    Removing /var/lib/libvirt/boot/virtinst-initrd.img.GGCMnx
  Tue, 29 May 2012 14:23:24 DEBUG    Domain state after install: 1
  Domain installation still in progress. You can reconnect to 
  the console to complete the installation process.
  Install of rhel6_x86_64 is SUCCESS!! . 

For RHEL5 host:
  command: spawn virt-install  --name rhel6_x86_64 --location nfs://nfssv:/export/engineering/redhat/released/RHEL6/U1/x86_64/os --os-variant=rhel6 --nographics --ram=2048 --vcpus=4 --file-size=20 --hvm --debug --extra-args "ks=http://beaker.rhts.fuchu/cblr/svc/op/ks/system/virt-guest-28.rhts.fuchu console=tty0 console=ttyS0,115200" --prompt --network bridge:br0 --file /var/lib/libvirt/images/rhel6_x86_64.img --noreboot 
  Tue, 29 May 2012 18:58:44 DEBUG    Requesting libvirt URI default
  Tue, 29 May 2012 18:58:44 DEBUG    Received libvirt URI 'qemu:///system'
  Would you like to use KVM acceleration? (yes or no) installation timed out.
Comment 13 Gurhan Ozen 2012-05-30 18:32:32 UTC 
Nomura-san
Can I have an account to have access to the systems. For some reason something's trying to connect to the guests' serial console but I can't find out what. I'd like to get access so I can reserve some systems and debug this issue. This behavior doesn't happen in our beaker yet it does on yours.
Comment 14 Jun'ichi NOMURA 2012-06-14 10:38:16 UTC 
Ozen-san,
I've sent you an e-mail about an accout.
Please check it and reply in e-mail.
Comment 15 Jun'ichi NOMURA 2012-07-18 23:57:01 UTC 
On beaker 0.9, it still fails:
https://flh9aaf026.tky.mesh.ad.jp/bkr/jobs/2256

/distribution/virt/install package is:
distribution-distribution-virt-install-3.8-8.noarch.rpm

Logs for RHEL6 and RHEL5 are attached.
Could you check it?

Both seemed to access cobbler and failed to get ks for guest.
(Cobbler is no longer available in beaker 0.9)

Also it tried to install Xvfb and failed on RHEL6.

And rhts-reboot during /virt/install caused following error:
time->Wed Jul 18 05:56:32 2012
type=SYSCALL msg=audit(1342605392.823:65): arch=c000003e syscall=16 success=no exit=-19 a0=3 a1=89a3 a2=7fff7b31d7d0 a3=7fff7b31d520 items=0 ppid=7062 pid=7152 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="brctl" exe="/usr/sbin/brctl" subj=unconfined_u:system_r:brctl_t:s0 key=(null)
type=AVC msg=audit(1342605392.823:65): avc:  denied  { sys_module } for  pid=7152 comm="brctl" capability=16  scontext=unconfined_u:system_r:brctl_t:s0 tcontext=unconfined_u:system_r:brctl_t:s0 tclass=capability
Fail: AVC messages found.
Checking for errors...
Using stronger AVC checks.
	Define empty RHTS_OPTION_STRONGER_AVC parameter if this causes any problems.
Comment 16 Jun'ichi NOMURA 2012-07-18 23:59:35 UTC 
Created attachment 599018 [details]
TESTOUT.log of /distribution/virt/install (on RHEL6, beaker 0.9.0-7)
Comment 17 Jun'ichi NOMURA 2012-07-19 00:00:21 UTC 
Created attachment 599019 [details]
TESTOUT.log of /distribution/virt/install (on RHEL5, beaker 0.9.0-7)
Comment 18 Bill Peck 2012-07-19 01:11:33 UTC 
there is a new version of the task. Will update as soon as possible.
Comment 19 Gurhan Ozen 2012-08-31 14:22:59 UTC 
Hi, 
So, as I was debugging this, I was able to get past the "error: internal error character device (null) is not using a PTY" point by not providing the "--serial ..." argument and even then it won't work because the installation is unable to bring up the networking interface:



                       ┌───────┤ Network Error ├────────┐
                       │                                │
                       │ There was an error configuring │
                       │ your network interface.        │                       
                       │                                │                       
                       │           ┌───────┐            │                       
                       │           │ Retry │            │                       
                       │           └───────┘            │                       
                       │                                │
                       │                                │
                       └────────────────────────────────┘

Just do something like:
/usr/sbin/virt-install --name rhel6_x86_64 --location nfs://nfssv:/export/engineering/redhat/released/RHEL6/U1/x86_64/os/ --os-variant=rhel6 --nographics --ram=2048 --vcpus=4 --file-size=20 --hvm --debug --extra-args 
"ks=http://beaker-rhel6.rhts.fuchu/bkr/kickstart/1036 console=tty0 console=ttyS0,115200" --prompt --network bridge:br0 --file /var/lib/libvirt/images/rhel6_x86_64.img --noreboot

You can also do the same by providing "--virttest" argument in guestargs in the xml. 

I am guessing that there is something wrong with the dhcp server or something? I have also manually tried to install without giving any kickstart file, doing everything step by step, it still failed to bring up the network interface.
Comment 20 Jun'ichi NOMURA 2012-09-12 01:18:15 UTC 
Sorry for slow response..

(In reply to comment #19)
> I am guessing that there is something wrong with the dhcp server or
> something? I have also manually tried to install without giving any
> kickstart file, doing everything step by step, it still failed to bring up
> the network interface.

There are occasional switch problem and b120b (the machine you used) is
connected to the switch.

So I tried on other machine (r120a-2) which is safe from the issue
but hit the same network error for RHEL6 host:
https://flh9aaf026.tky.mesh.ad.jp/bkr/jobs/2390

It seems /virt/install using auto-generated MAC for guest like this:
https://flh9aaf026.tky.mesh.ad.jp/bkr/logs/2012/09/23/2390/15168/202237/rhel6_x86_64_install.log:
    <interface type='bridge'>
      <source bridge='br0'/>
      <mac address='52:54:00:e1:e4:5b'/>
      <model type='virtio'/>
    </interface>

However, in our lab, all virt-guest-xx have MAC address starting with 00:16:3e
(they were generated when xen was the primary virtualization)
and DHCP server does not give IP for unknown MACs.

So I think it causes the network configuration error.
How does the test decide what MAC address to use?
Old RHTS seemed to pass such parameters through 'GUESTS' env variable.
Comment 21 Dan Callaghan 2012-10-10 06:59:07 UTC 
The /distribution/virt/install task was recently changed to ignore the MAC address assigned by Beaker. This was to work around a problem where Beaker would pick a MAC address/IP address which was not in the right subnet (since Beaker is unaware of network topology within the lab).

For bug 655009 Beaker will no longer be assigning the MAC addresses to guests at all, instead they will get a default generated one from libvirt and we rely on the lab DHCP server to assign an address for it.

Jun'ichi, could you update your DHCP server configuration to allow a pool of addresses to be assigned to these auto-generated MAC addresses?
Comment 23 Jun'ichi NOMURA 2012-10-10 08:28:46 UTC 
(In reply to comment #21)
> The /distribution/virt/install task was recently changed to ignore the MAC
> address assigned by Beaker. This was to work around a problem where Beaker
> would pick a MAC address/IP address which was not in the right subnet (since
> Beaker is unaware of network topology within the lab).
> 
> For bug 655009 Beaker will no longer be assigning the MAC addresses to
> guests at all, instead they will get a default generated one from libvirt
> and we rely on the lab DHCP server to assign an address for it.

I'm afraid the auto generated MAC address could conflict with that on other hosts...
Though the probability might be low, once the conflict occurs, the test result would be very much unstable.
For the use with bridge mode, it's really safer to use beaker managed MAC addresses.
Could you reconsider it?

> Jun'ichi, could you update your DHCP server configuration to allow a pool of
> addresses to be assigned to these auto-generated MAC addresses?

That means we have to allow unknown clients to get IP from DHCP server.
I would like to avoid that from lab management point of view.


BTW, it seems /distribution/virt/install is partially "fixed" in 3.9-16.
RHEL6-on-RHEL6 test assigns expected MAC address to guest and works fine:
  https://flh9aaf026.tky.mesh.ad.jp/bkr/jobs/2708

However, RHEL5-on-RHEL5 test still does not work:
  https://flh9aaf026.tky.mesh.ad.jp/bkr/jobs/2707
it stopped in interactive mode waiting for user input:
> Kickstart URL for guest rhel5_x86_64 is: http://beaker-rhel6.rhts.fuchu/bkr/kickstart/7207
> command: spawn virt-install  --name rhel5_x86_64 --mac 00:16:3e:6c:72:2f --location nfs://nfssv:/export/engineering/redhat/released/RHEL-Server-5.8-GA/5/x86_64/os/ --os-variant=rhel5 --nographics --file-size=20 --hvm --ram=1024 --vcpus=1 --debug --extra-args "ks=http://beaker-rhel6.rhts.fuchu/bkr/kickstart/7207 console=tty0 console=ttyS0,115200" --prompt --network bridge:br0 --file /var/lib/libvirt/images/rhel5_x86_64.img --noreboot 
> spawn virt-install --name rhel5_x86_64 --mac 00:16:3e:6c:72:2f --location nfs://nfssv:/export/engineering/redhat/released/RHEL-Server-5.8-GA/5/x86_64/os/ --os-variant=rhel5 --nographics --file-size=20 --hvm --ram=1024 --vcpus=1 --debug --extra-args ks=http://beaker-rhel6.rhts.fuchu/bkr/kickstart/7207 console=tty0 console=ttyS0,115200 --prompt --network bridge:br0 --file /var/lib/libvirt/images/rhel5_x86_64.img --noreboot
> Fri, 05 Oct 2012 00:47:58 DEBUG    Requesting libvirt URI default
> Fri, 05 Oct 2012 00:47:58 DEBUG    Received libvirt URI 'qemu:///system'
> Would you like to use KVM acceleration? (yes or no)
Comment 24 Dan Callaghan 2012-10-11 21:59:02 UTC 
(In reply to comment #23)
> I'm afraid the auto generated MAC address could conflict with that on other
> hosts...
> Though the probability might be low, once the conflict occurs, the test
> result would be very much unstable.

You're right, a conflict is possible. I thought qemu/kvm might have some way of addressing this issue but it seems it does not. We could make the MAC address use a certain prefix + recipe ID or something like that, to make it deterministic and avoid collisions.

> For the use with bridge mode, it's really safer to use beaker managed MAC
> addresses.
> Could you reconsider it?

Unfortunately I can't think of any other way to solve bug 655009. Removing the Virtual system records also lets us fix up several other long-standing issues: by not scheduling the guests we can avoid pinning the recipeset to a suboptimal lab, it saves the scheduler a lot of work too, and users won't be confused about the purpose of the Virtual records and try to reserve them.

> > Jun'ichi, could you update your DHCP server configuration to allow a pool of
> > addresses to be assigned to these auto-generated MAC addresses?
> 
> That means we have to allow unknown clients to get IP from DHCP server.
> I would like to avoid that from lab management point of view.

Would it help if we used a fixed prefix + recipe ID? That reduces the possible range of MAC addresses at least (though unfortunately the range would grow unbounded...)

> BTW, it seems /distribution/virt/install is partially "fixed" in 3.9-16.
> RHEL6-on-RHEL6 test assigns expected MAC address to guest and works fine:
>   https://flh9aaf026.tky.mesh.ad.jp/bkr/jobs/2708

Yes, someone has reverted the change /distribution/virt/install though I'm not sure why. In any case it will no longer work in Beaker 0.10 since Beaker will not be passing a MAC address down.

> However, RHEL5-on-RHEL5 test still does not work:
>   https://flh9aaf026.tky.mesh.ad.jp/bkr/jobs/2707
> it stopped in interactive mode waiting for user input:
> > Kickstart URL for guest rhel5_x86_64 is: http://beaker-rhel6.rhts.fuchu/bkr/kickstart/7207
> > command: spawn virt-install  --name rhel5_x86_64 --mac 00:16:3e:6c:72:2f --location nfs://nfssv:/export/engineering/redhat/released/RHEL-Server-5.8-GA/5/x86_64/os/ --os-variant=rhel5 --nographics --file-size=20 --hvm --ram=1024 --vcpus=1 --debug --extra-args "ks=http://beaker-rhel6.rhts.fuchu/bkr/kickstart/7207 console=tty0 console=ttyS0,115200" --prompt --network bridge:br0 --file /var/lib/libvirt/images/rhel5_x86_64.img --noreboot 
> > spawn virt-install --name rhel5_x86_64 --mac 00:16:3e:6c:72:2f --location nfs://nfssv:/export/engineering/redhat/released/RHEL-Server-5.8-GA/5/x86_64/os/ --os-variant=rhel5 --nographics --file-size=20 --hvm --ram=1024 --vcpus=1 --debug --extra-args ks=http://beaker-rhel6.rhts.fuchu/bkr/kickstart/7207 console=tty0 console=ttyS0,115200 --prompt --network bridge:br0 --file /var/lib/libvirt/images/rhel5_x86_64.img --noreboot
> > Fri, 05 Oct 2012 00:47:58 DEBUG    Requesting libvirt URI default
> > Fri, 05 Oct 2012 00:47:58 DEBUG    Received libvirt URI 'qemu:///system'
> > Would you like to use KVM acceleration? (yes or no)

This seems like a separate issue that we will need to look into.
Comment 25 Jun'ichi NOMURA 2012-10-17 05:01:31 UTC 
> However, RHEL5-on-RHEL5 test still does not work:
>   https://flh9aaf026.tky.mesh.ad.jp/bkr/jobs/2707
> it stopped in interactive mode waiting for user input:
...
> > Would you like to use KVM acceleration? (yes or no)

For RHEL5 host, if I add "--accelerate" to guestargs, it works for both RHEL5 guest and RHEL6 guest.
https://flh9aaf026.tky.mesh.ad.jp/bkr/jobs/2851

With RHEL6.2 host, the installation completes fine. But I get this error:
----
time->Tue Oct 16 21:37:37 2012
type=SYSCALL msg=audit(1350437857.315:66): arch=c000003e syscall=16 success=no exit=-19 a0=3 a1=89f0 a2=7fff760d59e0 a3=7fff760d5730 items=0 ppid=7311 pid=7458 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="brctl" exe="/usr/sbin/brctl" subj=unconfined_u:system_r:brctl_t:s0 key=(null)
type=AVC msg=audit(1350437857.315:66): avc:  denied  { sys_module } for  pid=7458 comm="brctl" capability=16  scontext=unconfined_u:system_r:brctl_t:s0 tcontext=unconfined_u:system_r:brctl_t:s0 tclass=capability
----
time->Tue Oct 16 21:37:37 2012
type=SYSCALL msg=audit(1350437857.302:65): arch=c000003e syscall=16 success=no exit=-19 a0=3 a1=89a3 a2=7fff760d59e0 a3=7fff760d5730 items=0 ppid=7311 pid=7458 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="brctl" exe="/usr/sbin/brctl" subj=unconfined_u:system_r:brctl_t:s0 key=(null)
type=AVC msg=audit(1350437857.302:65): avc:  denied  { sys_module } for  pid=7458 comm="brctl" capability=16  scontext=unconfined_u:system_r:brctl_t:s0 tcontext=unconfined_u:system_r:brctl_t:s0 tclass=capability
Fail: AVC messages found.

It does not occur with RHEL6.3 host.
Comment 26 Raymond Mancy 2012-10-22 02:22:44 UTC 
(In reply to comment #25)

> With RHEL6.2 host, the installation completes fine. But I get this error:
> ----
> time->Tue Oct 16 21:37:37 2012
> type=SYSCALL msg=audit(1350437857.315:66): arch=c000003e syscall=16
> success=no exit=-19 a0=3 a1=89f0 a2=7fff760d59e0 a3=7fff760d5730 items=0
> ppid=7311 pid=7458 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0
> sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="brctl" exe="/usr/sbin/brctl"
> subj=unconfined_u:system_r:brctl_t:s0 key=(null)
> type=AVC msg=audit(1350437857.315:66): avc:  denied  { sys_module } for 
> pid=7458 comm="brctl" capability=16 
> scontext=unconfined_u:system_r:brctl_t:s0
> tcontext=unconfined_u:system_r:brctl_t:s0 tclass=capability
> ----
> time->Tue Oct 16 21:37:37 2012
> Fail: AVC messages found.
> type=SYSCALL msg=audit(1350437857.302:65): arch=c000003e syscall=16
> success=no exit=-19 a0=3 a1=89a3 a2=7fff760d59e0 a3=7fff760d5730 items=0
> ppid=7311 pid=7458 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0
> sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="brctl" exe="/usr/sbin/brctl"
> subj=unconfined_u:system_r:brctl_t:s0 key=(null)
> type=AVC msg=audit(1350437857.302:65): avc:  denied  { sys_module } for 
> pid=7458 comm="brctl" capability=16 
> scontext=unconfined_u:system_r:brctl_t:s0
> tcontext=unconfined_u:system_r:brctl_t:s0 tclass=capability
> 
> It does not occur with RHEL6.3 host.

This looks like a problem with the selinux-policy package.
Comment 27 Jun'ichi NOMURA 2012-10-24 05:46:41 UTC 
I've run /distribution/virt/install on combinations of
 host RHEL5.[45678] and RHEL6.[123]
 with RHEL5.8 guest and RHEL6.2 guest.

It works fine with RHEL5 hosts iff "--accelerate" is passed to guestargs.

With RHEL6 hosts, there are following issues:

RHEL6.1 host
  - Failure due to guest installation timeout.
    It seems guest does not start up after showing 'Restarting system'.
    Both with RHEL5.8 guest and with RHEL6.3 guest.
    Example:
      https://flh9aaf026.tky.mesh.ad.jp/bkr/recipes/29041

RHEL6.2 host
  - Failure due to audit error (related to brctl).
    Worked around by turning off selinux.
    Example:
      https://flh9aaf026.tky.mesh.ad.jp/bkr/recipes/28600

RHEL6.3 host
  - Warning due to 'size_limit'.
    It seems libvirtd_debug.log grown too large. (about ~200MB)
    Debug log increases by hosts' RHEL versions. (RHEL6.3 > 6.2 > 6.1)
    Log is much larger with RHEL5.8 guest than with RHEL6.2 guest.
      e.g. RHEL5.8 guest has ~200MB, RHEL6.2 guest has ~20MB.
    I guess it's due to the enhancement of libvirt about more detailed
    debug logs.
    Example:
      https://flh9aaf026.tky.mesh.ad.jp/bkr/recipes/29051

Do you have any idea about the timeout issue with RHEL6.1 host?
I would like to get it fixed because we might have to run a test for RHEL6.1.z stream.

The log size issue of RHEL6.3 host needs a fix, too.
Maybe beaker to allow larger logs or /distribution/virt/install to limit the log level?
Comment 28 Raymond Mancy 2013-02-14 01:01:06 UTC 
Hi,

I've been running some tests with Beaker 0.11 and virt-install 4.0-37.


(In reply to comment #27)
> I've run /distribution/virt/install on combinations of
>  host RHEL5.[45678] and RHEL6.[123]
>  with RHEL5.8 guest and RHEL6.2 guest.
> 
> It works fine with RHEL5 hosts iff "--accelerate" is passed to guestargs.
> 
> With RHEL6 hosts, there are following issues:
> 
> RHEL6.1 host
>   - Failure due to guest installation timeout.
>     It seems guest does not start up after showing 'Restarting system'.
>     Both with RHEL5.8 guest and with RHEL6.3 guest.
>     Example:
>       https://flh9aaf026.tky.mesh.ad.jp/bkr/recipes/29041
>

RHEL6.1 Host: Fails, selinux-policy error, see BZ#909763
 
  RHEL5.8 Guest: Passes, boots
  RHEL6.3 Guest: Passes, boots
  
> RHEL6.2 host
>   - Failure due to audit error (related to brctl).
>     Worked around by turning off selinux.
>     Example:
>       https://flh9aaf026.tky.mesh.ad.jp/bkr/recipes/28600
> 

RHEL6.2 Host: Fails, same selinux-policy error as 6.1


> RHEL6.3 host
>   - Warning due to 'size_limit'.
>     It seems libvirtd_debug.log grown too large. (about ~200MB)
>     Debug log increases by hosts' RHEL versions. (RHEL6.3 > 6.2 > 6.1)
>     Log is much larger with RHEL5.8 guest than with RHEL6.2 guest.
>       e.g. RHEL5.8 guest has ~200MB, RHEL6.2 guest has ~20MB.
>     I guess it's due to the enhancement of libvirt about more detailed
>     debug logs.
>     Example:
>       https://flh9aaf026.tky.mesh.ad.jp/bkr/recipes/29051
> 

RHEL6.3 Host: Passes
  
  RHEL6.2 Guest: Passes, boots
  RHEL5.8 Guest: Passes, boots

> Do you have any idea about the timeout issue with RHEL6.1 host?
> I would like to get it fixed because we might have to run a test for
> RHEL6.1.z stream.
> 
> The log size issue of RHEL6.3 host needs a fix, too.
> Maybe beaker to allow larger logs or /distribution/virt/install to limit the
> log level?
Comment 31 Jeff Burke 2013-11-04 13:35:56 UTC 
Ray,
 Gurhan has left Red Hat. Reading Comment#28 it looks like we can close this BZ.

Thanks,
Jeff