Bug 645719

NetApp: Since you are seeing this on both the RHEL 6 and RHEL 5 hypervisors, going to set this to RHEL 6, and if this needs to be ported to RHEL 5 we can decide that then.

NetApp: is this blocking any 5.6 certs? We are already pretty late in the 5.6 devel cycle.

Need the following info:
1. is the system set up with NAT?
   If yes how is dnsmasq configured?
   Pls supply output of
   ps -ef | grep dnsmasq
2. Does the problem happen in a transparent bridge setup
   (no NAT)?
3. please supply output of
   tcpdump -i tap0
   on host where tap0 is replaced with tap device for the guest

additionally please supply the output of
'brctl show' and  'brctl showstp  <bridgename>'
where <bridgename> is the bridge connected to the specific
guest.

Please answer requests made in comments 9 & 10.

Created attachment 461275 [details]
tcpdump of vnet0

(In reply to comment #9)
> Need the following info:
> 1. is the system set up with NAT?
>    If yes how is dnsmasq configured?
>    Pls supply output of
>    ps -ef | grep dnsmasq
NO, I am using bridged networking for the guests. I havent tried on NAT setup yet.
> 2. Does the problem happen in a transparent bridge setup
>    (no NAT)?
Yes.
> 3. please supply output of
>    tcpdump -i tap0
>    on host where tap0 is replaced with tap device for the guest
Since I am using Bridged networking, there is a corresponding device vnet<x> for the guest.
Please find attached the output of `tcpdump -i vnet0`

(In reply to comment #10)
> additionally please supply the output of
> 'brctl show' and  'brctl showstp  <bridgename>'
> where <bridgename> is the bridge connected to the specific
> guest.
Output of `brctl show`
# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.001a64e5fb98       no              eth0
                                                        vnet2
virbr0          8000.000000000000       yes

Output of `brctl showstp br0`
# brctl showstp br0
br0
 bridge id              8000.001a64e5fb98
 designated root        8000.001a64e5fb98
 root port                 0                    path cost                  0
 max age                  19.99                 bridge max age            19.99
 hello time                1.99                 bridge hello time          1.99
 forward delay            14.99                 bridge forward delay      14.99
 ageing time             299.95
 hello timer               0.75                 tcn timer                  0.00
 topology change timer     0.00                 gc timer                   1.75
 hash elasticity           4                    hash max                 512
 mc last member count      2                    mc init query count        2
 mc router                 1                    mc snooping                1
 mc last member timer      0.99                 mc membership timer      259.96
 mc querier timer        254.96                 mc query interval        124.98
 mc response interval      9.99                 mc init query interval    31.24
 flags


eth0 (0)
 port id                0000                    state                forwarding
 designated root        8000.001a64e5fb98       path cost                 19
 designated bridge      8000.001a64e5fb98       message age timer          0.00
 designated port        8001                    forward delay timer        0.00
 designated cost           0                    hold timer                 0.00
 mc router                 1
 flags

vnet2 (0)
 port id                0000                    state                forwarding
 designated root        8000.001a64e5fb98       path cost                100
 designated bridge      8000.001a64e5fb98       message age timer          0.00
 designated port        8004                    forward delay timer        0.00
 designated cost           0                    hold timer                 0.00
 mc router                 1
 flags

(In reply to comment #12)
> Please answer requests made in comments 9 & 10.

Done :)

Also, I would just like to mention it again that the issue is seen only when guest networking driver is configured as virtio.

I think I see the problem here:

You have enabled forwarding delay on the bridge, so it takes time
for the bridge to start forwarding packets.
Meanwhile iscsi times out.

On my setup created by libvirt I see:
 forward delay             0.00                 bridge forward delay       0.00

How did you create the bridge?

(In reply to comment #14)
> I think I see the problem here:
> 
> You have enabled forwarding delay on the bridge, so it takes time
> for the bridge to start forwarding packets.
> Meanwhile iscsi times out.
> 
> On my setup created by libvirt I see:
>  forward delay             0.00                 bridge forward delay       0.00
> 
> How did you create the bridge?

I created the bridge by editing the files under /etc/sysconfig/network-scripts/.
Is there any recommended way of creating a bridge?

But if forwarding delay on the bridge is the cause, then I should be seeing this issue independent of what network driver I use on the guest, please correct me if I am wrong.

> I created the bridge by editing the files under
/etc/sysconfig/network-scripts/.
> Is there any recommended way of creating a bridge?

Some hints on libvirt wiki:
http://wiki.libvirt.org/page/Networking#Bridged_networking_.28aka_.22shared_physical_device.22.29

> But if forwarding delay on the bridge is the cause, then I should be seeing
this issue independent of what network driver I use on the guest, please
correct me if I am wrong.

Some other issue could be masking this.
Could you try with DELAY=0 please?

Also let's verify initrd loads virtio.
Could you attach the initrd file that got generated please?

this is to verify we are
not seeing a duplicate of
https://bugzilla.redhat.com/show_bug.cgi?id=568325

(In reply to comment #16)
> > I created the bridge by editing the files under
> /etc/sysconfig/network-scripts/.
> > Is there any recommended way of creating a bridge?
> 
> Some hints on libvirt wiki:
> http://wiki.libvirt.org/page/Networking#Bridged_networking_.28aka_.22shared_physical_device.22.29
> 
> > But if forwarding delay on the bridge is the cause, then I should be seeing
> this issue independent of what network driver I use on the guest, please
> correct me if I am wrong.
> 
> Some other issue could be masking this.
> Could you try with DELAY=0 please?

I tried with DELAY=0, but I am hitting the issue again.

(In reply to comment #17)
> Also let's verify initrd loads virtio.
> Could you attach the initrd file that got generated please?

How to do this? 
The guest hangs while booting. So I cant access the initrd through guest.
Also the boot partition for the guest resides on a file in the hypervisor,
and since that file is partitioned from guest, I cant mount it on the hypervisor. And therefore I cant access the initrd through the hypervisor also.
Is there some to access the initrd?
If not I will install the guest OS again but this time without creating partition for /boot.

Created attachment 461978 [details]
initrd image of guest os

Please find attached the initrd image of guest operating system.

At this point in RHEL 5.6, this won't make it. Still debugging. 

NetApp: please keep up with narrowing this issue down in the meantime.

Yes, the initrd lacks virtio-pci.ko and virtio-ring.ko
So it's a duplicate of 
https://bugzilla.redhat.com/show_bug.cgi?id=568325
Closing.

*** This bug has been marked as a duplicate of bug 568325 ***