Bug 619151

Summary: networking issues with kvm during installation..
Product: Red Hat Enterprise Linux 6 Reporter: Gurhan Ozen <gozen>
Component: qemu-kvmAssignee: Amit Shah <amit.shah>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 6.0CC: agospoda, benl, jburke, lihuang, llim, mjenner, mkenneth, mst, tburke, virt-maint
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-08-10 20:31:15 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Debug log for libvirtd none

Description Gurhan Ozen 2010-07-28 17:26:34 UTC
Created attachment 435087 [details]
Debug log for libvirtd

Description of problem:

This has been happening on randomly with recent kernels. Randomly in the sense that it doesn't happen on every machine, but once it happens on a machine, it can be reproduced reliably in that particular machine. 

The problem is, while installing guests, the networking just dies and hence installation can't be completed. 

The set up for it was:
kernel-2.6.32-54.el6.x86_64
libvirt-0.8.1-13.el6.x86_64
qemu-kvm-0.12.1.2-2.90.el6.x86_64

Then tried the same with updated libvirt and qemu-kvm packages (libvirt-0.8.1-15.el6.x86_64 & qemu-kvm-0.12.1.2-2.96.el6.x86_64) but the result was the same. 

A machine where this issue can be reliably reproduced is ibm-hs22-04.lab.bos.redhat.com . ( https://beaker.engineering.redhat.com/view/ibm-hs22-04.lab.bos.redhat.com )

Bridged networking is used for guest networking and libvirtd debug logs will be attached. This used to work, and will try back with kernel-2.6.32-44.2 on once i get the machine loaned from beaker to see which kernel started to have issues.

The installation command was:
virt-install --name testguest --cdrom /var/lib/libvirt/images/guest-81-214.rhts.eng.bos.redhat.com.iso --file /var/lib/libvirt/images/testguest.img --ram=1024 --vcpus=2 --file-size=10 --hvm --debug --extra-args ks=http://lab2.rhts.eng.bos.redhat.com/cblr/svc/op/ks/system/guest-81-214.rhts.eng.bos.redhat.com --prompt --accelerate --os-variant=virtio26 --network bridge:br0   --noreboot --vnc

# brctl show
bridge name	bridge id		STP enabled	interfaces
br0		8000.00215e99f14c	no		eth0
							vnet0
virbr0		8000.000000000000	yes

# ifconfig -a
br0       Link encap:Ethernet  HWaddr 00:21:5E:99:F1:4C  
          inet addr:10.16.44.60  Bcast:10.16.47.255  Mask:255.255.248.0
          inet6 addr: fe80::221:5eff:fe99:f14c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1332601 errors:0 dropped:0 overruns:0 frame:0
          TX packets:246076 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:736492072 (702.3 MiB)  TX bytes:60834110 (58.0 MiB)

eth0      Link encap:Ethernet  HWaddr 00:21:5E:99:F1:4C  
          inet6 addr: fe80::221:5eff:fe99:f14c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2976182 errors:0 dropped:0 overruns:0 frame:0
          TX packets:415817 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1265398891 (1.1 GiB)  TX bytes:80143975 (76.4 MiB)
          Interrupt:30 Memory:fa000000-fa012800 

eth1      Link encap:Ethernet  HWaddr 00:21:5E:99:F1:4E  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:37 Memory:92000000-92012800 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:30825 errors:0 dropped:0 overruns:0 frame:0
          TX packets:30825 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:39663693 (37.8 MiB)  TX bytes:39663693 (37.8 MiB)

sit0      Link encap:IPv6-in-IPv4  
          NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

usb0      Link encap:Ethernet  HWaddr 02:21:5E:9D:F1:2F  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

virbr0    Link encap:Ethernet  HWaddr E6:8C:06:D2:75:DB  
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:370 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:22848 (22.3 KiB)

vnet0     Link encap:Ethernet  HWaddr 72:F3:83:73:4D:46  
          inet6 addr: fe80::70f3:83ff:fe73:4d46/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:61 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12891 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:6205 (6.0 KiB)  TX bytes:1836287 (1.7 MiB)

Additional testlogs can be seen at 
http://rhts.redhat.com/cgi-bin/rhts/test_log.cgi?id=15700613 .


Version-Release number of selected component (if applicable):


How reproducible:
When it happens on a machine, it happens all the time. So far, from talking to others, it only seems to happen on Intel boxen. 

Steps to Reproduce:
1. See above.
2.
3.

Comment 1 Martin Jenner 2010-07-28 18:54:22 UTC
I suggest trying the following blacklisting the vhost module so it is not loaded and see if that helps

create file /etc/modprobe.d/vhost-blacklist.conf

# cat <EOF > /etc/modprobe.d/vhost-blacklist.conf
install vhost_net /bin/true
EOF

reboot the system and then try installing the guest(s)

Comment 2 Gurhan Ozen 2010-07-28 19:01:47 UTC
(In reply to comment #1)
> I suggest trying the following blacklisting the vhost module so it is not
> loaded and see if that helps
> 
> create file /etc/modprobe.d/vhost-blacklist.conf
> 
> # cat <EOF > /etc/modprobe.d/vhost-blacklist.conf
> install vhost_net /bin/true
> EOF
> 
> reboot the system and then try installing the guest(s)    

I've tried that but have got the same issue..

Comment 8 Amit Shah 2010-07-30 07:53:36 UTC
Can you please try a few things:

- Check the guest logs (kernel, system) and upload them here
- Anything obviously wrong with guest networking seen in the logs?
- Jes mentions libvirtd starting could be an issue -- can you try a .ks file that removes libvirtd install in the guest?

Comment 20 Martin Jenner 2010-08-06 22:41:40 UTC
Today I installed on of the systems ibm-hs22-04.lab.bos.redhat.com that was hitting this issue 100% with tree RHEL6.0-20100805.0 the snapshot #10 candidate.

I was able to install multiple guests across various versions of RHEL and I did not see the issue.

[root@ibm-hs22-04 results]# uname -a
Linux ibm-hs22-04.lab.bos.redhat.com 2.6.32-59.el6.x86_64 #1 SMP Wed Aug 4 12:47:47 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

[root@ibm-hs22-04 results]# lsmod | grep vhost
vhost_net              23820  0 
macvtap                 7867  1 vhost_net
tun                    16583  3 vhost_net

[root@ibm-hs22-04 results]#  virsh hostname
viibm-hs22-04.lab.bos.redhat.com

[root@ibm-hs22-04 results]# virsh list --all
 Id Name                 State
----------------------------------
 20 kvm_rhel39_32_raw    running
  - kvm_rhel39_64_raw    shut off
  - kvm_rhel48_64_raw    shut off
  - kvm_rhel55_32_raw    shut off
  - kvm_rhel55_64_raw    shut off
  - kvm_rhel60_32_raw    shut off
  - kvm_rhel60_64_raw    shut off
  - vm1                  shut off

Comment 21 Dor Laor 2010-08-09 16:30:02 UTC
(In reply to comment #20)
> Today I installed on of the systems ibm-hs22-04.lab.bos.redhat.com that was
> hitting this issue 100% with tree RHEL6.0-20100805.0 the snapshot #10
> candidate.
> 
> I was able to install multiple guests across various versions of RHEL and I did
> not see the issue.

I'm not sure I follow, is there a problem in snap10 or not?

Comment 22 Martin Jenner 2010-08-10 13:26:41 UTC
No problem, I was not able to reproduce the problem with snap10.

Comment 23 Dor Laor 2010-08-10 20:31:15 UTC
I'll close this issue. Also note that spanning tree is not suitable for virt anyway.