Bug 619151 - networking issues with kvm during installation..
networking issues with kvm during installation..
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
6.0
All Linux
high Severity high
: rc
: ---
Assigned To: Amit Shah
Virtualization Bugs
: Regression
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-07-28 13:26 EDT by Gurhan Ozen
Modified: 2018-06-25 14:49 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-08-10 16:31:15 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Debug log for libvirtd (196.98 KB, text/plain)
2010-07-28 13:26 EDT, Gurhan Ozen
no flags Details

  None (edit)
Description Gurhan Ozen 2010-07-28 13:26:34 EDT
Created attachment 435087 [details]
Debug log for libvirtd

Description of problem:

This has been happening on randomly with recent kernels. Randomly in the sense that it doesn't happen on every machine, but once it happens on a machine, it can be reproduced reliably in that particular machine. 

The problem is, while installing guests, the networking just dies and hence installation can't be completed. 

The set up for it was:
kernel-2.6.32-54.el6.x86_64
libvirt-0.8.1-13.el6.x86_64
qemu-kvm-0.12.1.2-2.90.el6.x86_64

Then tried the same with updated libvirt and qemu-kvm packages (libvirt-0.8.1-15.el6.x86_64 & qemu-kvm-0.12.1.2-2.96.el6.x86_64) but the result was the same. 

A machine where this issue can be reliably reproduced is ibm-hs22-04.lab.bos.redhat.com . ( https://beaker.engineering.redhat.com/view/ibm-hs22-04.lab.bos.redhat.com )

Bridged networking is used for guest networking and libvirtd debug logs will be attached. This used to work, and will try back with kernel-2.6.32-44.2 on once i get the machine loaned from beaker to see which kernel started to have issues.

The installation command was:
virt-install --name testguest --cdrom /var/lib/libvirt/images/guest-81-214.rhts.eng.bos.redhat.com.iso --file /var/lib/libvirt/images/testguest.img --ram=1024 --vcpus=2 --file-size=10 --hvm --debug --extra-args ks=http://lab2.rhts.eng.bos.redhat.com/cblr/svc/op/ks/system/guest-81-214.rhts.eng.bos.redhat.com --prompt --accelerate --os-variant=virtio26 --network bridge:br0   --noreboot --vnc

# brctl show
bridge name	bridge id		STP enabled	interfaces
br0		8000.00215e99f14c	no		eth0
							vnet0
virbr0		8000.000000000000	yes

# ifconfig -a
br0       Link encap:Ethernet  HWaddr 00:21:5E:99:F1:4C  
          inet addr:10.16.44.60  Bcast:10.16.47.255  Mask:255.255.248.0
          inet6 addr: fe80::221:5eff:fe99:f14c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1332601 errors:0 dropped:0 overruns:0 frame:0
          TX packets:246076 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:736492072 (702.3 MiB)  TX bytes:60834110 (58.0 MiB)

eth0      Link encap:Ethernet  HWaddr 00:21:5E:99:F1:4C  
          inet6 addr: fe80::221:5eff:fe99:f14c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2976182 errors:0 dropped:0 overruns:0 frame:0
          TX packets:415817 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1265398891 (1.1 GiB)  TX bytes:80143975 (76.4 MiB)
          Interrupt:30 Memory:fa000000-fa012800 

eth1      Link encap:Ethernet  HWaddr 00:21:5E:99:F1:4E  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:37 Memory:92000000-92012800 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:30825 errors:0 dropped:0 overruns:0 frame:0
          TX packets:30825 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:39663693 (37.8 MiB)  TX bytes:39663693 (37.8 MiB)

sit0      Link encap:IPv6-in-IPv4  
          NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

usb0      Link encap:Ethernet  HWaddr 02:21:5E:9D:F1:2F  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

virbr0    Link encap:Ethernet  HWaddr E6:8C:06:D2:75:DB  
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:370 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:22848 (22.3 KiB)

vnet0     Link encap:Ethernet  HWaddr 72:F3:83:73:4D:46  
          inet6 addr: fe80::70f3:83ff:fe73:4d46/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:61 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12891 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:6205 (6.0 KiB)  TX bytes:1836287 (1.7 MiB)

Additional testlogs can be seen at 
http://rhts.redhat.com/cgi-bin/rhts/test_log.cgi?id=15700613 .


Version-Release number of selected component (if applicable):


How reproducible:
When it happens on a machine, it happens all the time. So far, from talking to others, it only seems to happen on Intel boxen. 

Steps to Reproduce:
1. See above.
2.
3.
Comment 1 Martin Jenner 2010-07-28 14:54:22 EDT
I suggest trying the following blacklisting the vhost module so it is not loaded and see if that helps

create file /etc/modprobe.d/vhost-blacklist.conf

# cat <EOF > /etc/modprobe.d/vhost-blacklist.conf
install vhost_net /bin/true
EOF

reboot the system and then try installing the guest(s)
Comment 2 Gurhan Ozen 2010-07-28 15:01:47 EDT
(In reply to comment #1)
> I suggest trying the following blacklisting the vhost module so it is not
> loaded and see if that helps
> 
> create file /etc/modprobe.d/vhost-blacklist.conf
> 
> # cat <EOF > /etc/modprobe.d/vhost-blacklist.conf
> install vhost_net /bin/true
> EOF
> 
> reboot the system and then try installing the guest(s)    

I've tried that but have got the same issue..
Comment 8 Amit Shah 2010-07-30 03:53:36 EDT
Can you please try a few things:

- Check the guest logs (kernel, system) and upload them here
- Anything obviously wrong with guest networking seen in the logs?
- Jes mentions libvirtd starting could be an issue -- can you try a .ks file that removes libvirtd install in the guest?
Comment 20 Martin Jenner 2010-08-06 18:41:40 EDT
Today I installed on of the systems ibm-hs22-04.lab.bos.redhat.com that was hitting this issue 100% with tree RHEL6.0-20100805.0 the snapshot #10 candidate.

I was able to install multiple guests across various versions of RHEL and I did not see the issue.

[root@ibm-hs22-04 results]# uname -a
Linux ibm-hs22-04.lab.bos.redhat.com 2.6.32-59.el6.x86_64 #1 SMP Wed Aug 4 12:47:47 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

[root@ibm-hs22-04 results]# lsmod | grep vhost
vhost_net              23820  0 
macvtap                 7867  1 vhost_net
tun                    16583  3 vhost_net

[root@ibm-hs22-04 results]#  virsh hostname
viibm-hs22-04.lab.bos.redhat.com

[root@ibm-hs22-04 results]# virsh list --all
 Id Name                 State
----------------------------------
 20 kvm_rhel39_32_raw    running
  - kvm_rhel39_64_raw    shut off
  - kvm_rhel48_64_raw    shut off
  - kvm_rhel55_32_raw    shut off
  - kvm_rhel55_64_raw    shut off
  - kvm_rhel60_32_raw    shut off
  - kvm_rhel60_64_raw    shut off
  - vm1                  shut off
Comment 21 Dor Laor 2010-08-09 12:30:02 EDT
(In reply to comment #20)
> Today I installed on of the systems ibm-hs22-04.lab.bos.redhat.com that was
> hitting this issue 100% with tree RHEL6.0-20100805.0 the snapshot #10
> candidate.
> 
> I was able to install multiple guests across various versions of RHEL and I did
> not see the issue.

I'm not sure I follow, is there a problem in snap10 or not?
Comment 22 Martin Jenner 2010-08-10 09:26:41 EDT
No problem, I was not able to reproduce the problem with snap10.
Comment 23 Dor Laor 2010-08-10 16:31:15 EDT
I'll close this issue. Also note that spanning tree is not suitable for virt anyway.

Note You need to log in before you can comment on or make changes to this bug.