Bug 626678

Summary: network in host become unavailable for few secs when guest quit
Product: Red Hat Enterprise Linux 6 Reporter: Shirley Zhou <szhou>
Component: kernelAssignee: Neil Horman <nhorman>
Status: CLOSED DUPLICATE QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: low    
Version: 6.0CC: herbert.xu, jolsa, mshao, mwagner, nhorman, tgraf
Target Milestone: rcKeywords: Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 650097 (view as bug list) Environment:
Last Closed: 2010-11-15 18:48:57 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 580953, 650097    

Description Shirley Zhou 2010-08-24 05:34:49 UTC 
Description of problem:
Transfer big file from host to guest, then quit guest, then network in host will become unavailable for a few seconds as following:

ping from external box to host:
[root@dhcp-91-145 ~]# ping 10.66.91.53
PING 10.66.91.53 (10.66.91.53) 56(84) bytes of data.
64 bytes from 10.66.91.53: icmp_seq=1 ttl=64 time=1.26 ms
64 bytes from 10.66.91.53: icmp_seq=2 ttl=64 time=0.215 ms
64 bytes from 10.66.91.53: icmp_seq=3 ttl=64 time=0.188 ms
64 bytes from 10.66.91.53: icmp_seq=4 ttl=64 time=0.193 ms
64 bytes from 10.66.91.53: icmp_seq=5 ttl=64 time=0.198 ms
64 bytes from 10.66.91.53: icmp_seq=6 ttl=64 time=0.196 ms
64 bytes from 10.66.91.53: icmp_seq=7 ttl=64 time=0.204 ms
64 bytes from 10.66.91.53: icmp_seq=50 ttl=64 time=1.28 ms
64 bytes from 10.66.91.53: icmp_seq=51 ttl=64 time=0.166 ms
64 bytes from 10.66.91.53: icmp_seq=52 ttl=64 time=0.164 ms
64 bytes from 10.66.91.53: icmp_seq=53 ttl=64 time=0.167 ms
^C
--- 10.66.91.53 ping statistics ---
53 packets transmitted, 11 received, 79% packet loss, time 52692ms
rtt min/avg/max/mdev = 0.164/0.385/1.282/0.420 ms

ping from host to external box:
[root@dhcp-91-53 Desktop]# ping 10.66.91.127
PING 10.66.91.127 (10.66.91.127) 56(84) bytes of data.
64 bytes from 10.66.91.127: icmp_seq=1 ttl=64 time=0.182 ms
64 bytes from 10.66.91.127: icmp_seq=2 ttl=64 time=0.176 ms
64 bytes from 10.66.91.127: icmp_seq=3 ttl=64 time=0.177 ms
64 bytes from 10.66.91.127: icmp_seq=9 ttl=64 time=1.20 ms
64 bytes from 10.66.91.127: icmp_seq=10 ttl=64 time=0.145 ms
64 bytes from 10.66.91.127: icmp_seq=11 ttl=64 time=0.147 ms
64 bytes from 10.66.91.127: icmp_seq=12 ttl=64 time=0.157 ms
64 bytes from 10.66.91.127: icmp_seq=13 ttl=64 time=0.142 ms
64 bytes from 10.66.91.127: icmp_seq=14 ttl=64 time=0.165 ms
^C
--- 10.66.91.127 ping statistics ---
14 packets transmitted, 9 received, 35% packet loss, time 13851ms
rtt min/avg/max/mdev = 0.142/0.277/1.209/0.330 ms

Version-Release number of selected component (if applicable):
qemu-kvm-0.12.1.2-2.112.el6.x86_64
kernel-2.6.32-66.el6.x86_64

How reproducible:
easy to reproduce when transfer big file from guest to host, or from host to guest.

Steps to Reproduce:
1.start rhel6 guest
/usr/libexec/qemu-kvm -M rhel6.0.0 -enable-kvm -m 2G -smp 2 -name src -uuid 4667028b-49fa-aba8-a5a4-868a8212342f -nodefconfig -nodefaults -monitor stdio -boot c -drive file=/home/rhel6.img,if=none,id=drive-virtio-disk0,boot=on,format=raw,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,id=hostnet1 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:12:a1:54,bus=pci.0,addr=0x3 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -device usb-tablet,id=input0 -vnc :12 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -qmp tcp:0:4446,server,nowait --no-kvm-pit-reinjection -rtc base=utc,clock=host,driftfix=slew
2.scp big file (4G) from host to above guest
3.after scp finished,quit guest through monitor

  
Actual results:
after step3, network in host become unavailable for few seconds,can not ping external, and external can not ping this host

Expected results:
Network in host should be smoothly. 

Additional info:
Comment 1 Shirley Zhou 2010-08-24 07:26:29 UTC 
Additional info:

this issue happens when shutdown vm in guest.
Comment 2 Mark Wagner 2010-08-27 14:03:40 UTC 
looks like you are running this on a "public" network. Please rerun on a private network where you can control all other activity on the network.

I would also disagree with the assumption that the network becomes unavailable.  In my lab a see a quick spike in latency but no packet loss. 
This is on a private network that we can control.

[root@perf10 ~]# ping 192.168.1.22
PING 192.168.1.22 (192.168.1.22) 56(84) bytes of data.
64 bytes from 192.168.1.22: icmp_seq=1 ttl=64 time=0.861 ms
64 bytes from 192.168.1.22: icmp_seq=2 ttl=64 time=0.053 ms
64 bytes from 192.168.1.22: icmp_seq=3 ttl=64 time=0.057 ms
64 bytes from 192.168.1.22: icmp_seq=4 ttl=64 time=0.053 ms
64 bytes from 192.168.1.22: icmp_seq=5 ttl=64 time=0.047 ms
64 bytes from 192.168.1.22: icmp_seq=6 ttl=64 time=0.049 ms
64 bytes from 192.168.1.22: icmp_seq=7 ttl=64 time=0.048 ms
64 bytes from 192.168.1.22: icmp_seq=8 ttl=64 time=0.050 ms
64 bytes from 192.168.1.22: icmp_seq=9 ttl=64 time=0.047 ms
64 bytes from 192.168.1.22: icmp_seq=10 ttl=64 time=0.046 ms
64 bytes from 192.168.1.22: icmp_seq=11 ttl=64 time=0.047 ms
64 bytes from 192.168.1.22: icmp_seq=12 ttl=64 time=0.073 ms
64 bytes from 192.168.1.22: icmp_seq=13 ttl=64 time=0.062 ms
64 bytes from 192.168.1.22: icmp_seq=14 ttl=64 time=0.047 ms
64 bytes from 192.168.1.22: icmp_seq=15 ttl=64 time=0.049 ms
64 bytes from 192.168.1.22: icmp_seq=16 ttl=64 time=0.043 ms
64 bytes from 192.168.1.22: icmp_seq=17 ttl=64 time=0.054 ms
64 bytes from 192.168.1.22: icmp_seq=18 ttl=64 time=0.038 ms
64 bytes from 192.168.1.22: icmp_seq=19 ttl=64 time=0.069 ms
64 bytes from 192.168.1.22: icmp_seq=20 ttl=64 time=0.067 ms
64 bytes from 192.168.1.22: icmp_seq=21 ttl=64 time=0.047 ms
64 bytes from 192.168.1.22: icmp_seq=22 ttl=64 time=0.047 ms
64 bytes from 192.168.1.22: icmp_seq=23 ttl=64 time=0.045 ms
64 bytes from 192.168.1.22: icmp_seq=24 ttl=64 time=0.047 ms
64 bytes from 192.168.1.22: icmp_seq=25 ttl=64 time=0.047 ms
64 bytes from 192.168.1.22: icmp_seq=26 ttl=64 time=0.048 ms
64 bytes from 192.168.1.22: icmp_seq=27 ttl=64 time=0.047 ms
64 bytes from 192.168.1.22: icmp_seq=28 ttl=64 time=0.047 ms
64 bytes from 192.168.1.22: icmp_seq=29 ttl=64 time=0.049 ms
64 bytes from 192.168.1.22: icmp_seq=30 ttl=64 time=0.046 ms
64 bytes from 192.168.1.22: icmp_seq=31 ttl=64 time=0.044 ms
64 bytes from 192.168.1.22: icmp_seq=32 ttl=64 time=0.046 ms
64 bytes from 192.168.1.22: icmp_seq=33 ttl=64 time=0.045 ms
64 bytes from 192.168.1.22: icmp_seq=34 ttl=64 time=0.046 ms
64 bytes from 192.168.1.22: icmp_seq=35 ttl=64 time=0.050 ms
64 bytes from 192.168.1.22: icmp_seq=36 ttl=64 time=0.047 ms
64 bytes from 192.168.1.22: icmp_seq=37 ttl=64 time=0.049 ms
64 bytes from 192.168.1.22: icmp_seq=38 ttl=64 time=0.046 ms
64 bytes from 192.168.1.22: icmp_seq=39 ttl=64 time=0.045 ms
64 bytes from 192.168.1.22: icmp_seq=40 ttl=64 time=0.047 ms
64 bytes from 192.168.1.22: icmp_seq=41 ttl=64 time=0.049 ms
64 bytes from 192.168.1.22: icmp_seq=42 ttl=64 time=0.048 ms
64 bytes from 192.168.1.22: icmp_seq=43 ttl=64 time=0.046 ms
Comment 3 Shirley Zhou 2010-08-30 02:14:37 UTC 
(In reply to comment #2)
I reproduce this bug on private network bridge virbr0 as following:

1. run two rhel6 vms , vm1 and vm2 on the same host with private network
2. scp a big file(4G) from host to vm1
3. ping vm2 from host
4. quit vm1

result of step3:
[root@dhcp-91-145 ~]# ping 192.168.122.59
PING 192.168.122.59 (192.168.122.59) 56(84) bytes of data.
64 bytes from 192.168.122.59: icmp_seq=1 ttl=64 time=0.532 ms
64 bytes from 192.168.122.59: icmp_seq=2 ttl=64 time=0.433 ms
64 bytes from 192.168.122.59: icmp_seq=3 ttl=64 time=0.434 ms
64 bytes from 192.168.122.59: icmp_seq=4 ttl=64 time=0.601 ms
64 bytes from 192.168.122.59: icmp_seq=5 ttl=64 time=0.451 ms
64 bytes from 192.168.122.59: icmp_seq=6 ttl=64 time=0.429 ms
64 bytes from 192.168.122.59: icmp_seq=7 ttl=64 time=0.428 ms
64 bytes from 192.168.122.59: icmp_seq=8 ttl=64 time=0.526 ms
64 bytes from 192.168.122.59: icmp_seq=33 ttl=64 time=0.399 ms
64 bytes from 192.168.122.59: icmp_seq=34 ttl=64 time=0.399 ms
64 bytes from 192.168.122.59: icmp_seq=35 ttl=64 time=0.448 ms
64 bytes from 192.168.122.59: icmp_seq=36 ttl=64 time=0.452 ms
64 bytes from 192.168.122.59: icmp_seq=37 ttl=64 time=0.296 ms
64 bytes from 192.168.122.59: icmp_seq=38 ttl=64 time=0.526 ms
64 bytes from 192.168.122.59: icmp_seq=39 ttl=64 time=0.348 ms

From above result, we can see about 25 packages lost when vm1 quit.
Comment 4 Neil Horman 2010-10-14 15:07:21 UTC 
can you please attach the hosts message log after you encounter this problem.  It would be interesting to see if a port change on the tap interface leads to some bridging issue.

My first guest is that you are running stp on vbir0, and the downing of veth0 on the bridge is seen as a topology change, leading to ports going back into learning mode or some such, during which time traffic is not forwarded.
Comment 5 Shirley Zhou 2010-10-15 02:21:38 UTC 
(In reply to comment #4)
> can you please attach the hosts message log after you encounter this problem. 
> It would be interesting to see if a port change on the tap interface leads to
> some bridging issue.
> 
> My first guest is that you are running stp on vbir0, and the downing of veth0
> on the bridge is seen as a topology change, leading to ports going back into
> learning mode or some such, during which time traffic is not forwarded.

reproduce this bug as scenario in comment3.
[root@dhcp-91-53 home]# ping 192.168.122.99
PING 192.168.122.99 (192.168.122.99) 56(84) bytes of data.
64 bytes from 192.168.122.99: icmp_seq=1 ttl=64 time=1.25 ms
64 bytes from 192.168.122.99: icmp_seq=2 ttl=64 time=0.175 ms
64 bytes from 192.168.122.99: icmp_seq=3 ttl=64 time=0.158 ms
64 bytes from 192.168.122.99: icmp_seq=10 ttl=64 time=0.369 ms
64 bytes from 192.168.122.99: icmp_seq=11 ttl=64 time=0.174 ms
64 bytes from 192.168.122.99: icmp_seq=12 ttl=64 time=0.171 ms
64 bytes from 192.168.122.99: icmp_seq=13 ttl=64 time=0.188 ms
64 bytes from 192.168.122.99: icmp_seq=14 ttl=64 time=0.163 ms
64 bytes from 192.168.122.99: icmp_seq=15 ttl=64 time=0.165 ms
64 bytes from 192.168.122.99: icmp_seq=16 ttl=64 time=0.174 ms
64 bytes from 192.168.122.99: icmp_seq=17 ttl=64 time=0.192 ms
64 bytes from 192.168.122.99: icmp_seq=18 ttl=64 time=0.169 ms
64 bytes from 192.168.122.99: icmp_seq=19 ttl=64 time=0.140 ms
64 bytes from 192.168.122.99: icmp_seq=20 ttl=64 time=0.129 ms
64 bytes from 192.168.122.99: icmp_seq=21 ttl=64 time=0.180 ms
^C
--- 192.168.122.99 ping statistics ---
21 packets transmitted, 15 received, 28% packet loss, time 20108ms
rtt min/avg/max/mdev = 0.129/0.253/1.250/0.271 ms

dmesg info:
virbr0: port 1(tap0) entering disabled state
device tap0 left promiscuous mode
virbr0: port 1(tap0) entering disabled state
device tap0 entered promiscuous mode
virbr0: topology change detected, propagating
virbr0: port 1(tap0) entering forwarding state
kvm: 26516: cpu0 unimplemented perfctr wrmsr: 0xc1 data 0x0
kvm: 26516: cpu0 unimplemented perfctr wrmsr: 0x186 data 0x130079
kvm: 26516: cpu0 unimplemented perfctr wrmsr: 0xc1 data 0xffd76974
kvm: 26516: cpu0 unimplemented perfctr wrmsr: 0x186 data 0x530079
kvm: 26516: cpu1 unimplemented perfctr wrmsr: 0xc1 data 0x0
kvm: 26516: cpu1 unimplemented perfctr wrmsr: 0x186 data 0x130079
kvm: 26516: cpu1 unimplemented perfctr wrmsr: 0xc1 data 0xffd76974
kvm: 26516: cpu1 unimplemented perfctr wrmsr: 0x186 data 0x530079
tap0: no IPv6 routers present
device tap1 entered promiscuous mode
virbr0: topology change detected, propagating
virbr0: port 2(tap1) entering forwarding state
kvm: 26674: cpu0 unimplemented perfctr wrmsr: 0xc1 data 0x0
kvm: 26674: cpu0 unimplemented perfctr wrmsr: 0x186 data 0x130079
kvm: 26674: cpu0 unimplemented perfctr wrmsr: 0xc1 data 0xffd76974
kvm: 26674: cpu0 unimplemented perfctr wrmsr: 0x186 data 0x530079
kvm: 26674: cpu1 unimplemented perfctr wrmsr: 0xc1 data 0x0
kvm: 26674: cpu1 unimplemented perfctr wrmsr: 0x186 data 0x130079
kvm: 26674: cpu1 unimplemented perfctr wrmsr: 0xc1 data 0xffd76974
kvm: 26674: cpu1 unimplemented perfctr wrmsr: 0x186 data 0x530079
tap1: no IPv6 routers present
virbr0: port 1(tap0) entering disabled state
device tap0 left promiscuous mode
virbr0: port 1(tap0) entering disabled state
device tap0 entered promiscuous mode
virbr0: topology change detected, propagating
virbr0: port 1(tap0) entering forwarding state
kvm: 27093: cpu0 unimplemented perfctr wrmsr: 0xc1 data 0x0
kvm: 27093: cpu0 unimplemented perfctr wrmsr: 0x186 data 0x130079
kvm: 27093: cpu0 unimplemented perfctr wrmsr: 0xc1 data 0xffd76974
kvm: 27093: cpu0 unimplemented perfctr wrmsr: 0x186 data 0x530079
kvm: 27093: cpu1 unimplemented perfctr wrmsr: 0xc1 data 0x0
kvm: 27093: cpu1 unimplemented perfctr wrmsr: 0x186 data 0x130079
kvm: 27093: cpu1 unimplemented perfctr wrmsr: 0xc1 data 0xffd76974
kvm: 27093: cpu1 unimplemented perfctr wrmsr: 0x186 data 0x530079
tap0: no IPv6 routers present
virbr0: port 1(tap0) entering disabled state
device tap0 left promiscuous mode
virbr0: port 1(tap0) entering disabled state
Comment 6 Neil Horman 2010-10-15 23:32:27 UTC 
well it definately seems as though your bridge topology is changing alot during shutdown.  Can you use brctl to disable stp on the bridge in question?  That should solve the problem.
Comment 7 Shirley Zhou 2010-10-18 02:29:08 UTC 
(In reply to comment #6)
> well it definately seems as though your bridge topology is changing alot during
> shutdown.  Can you use brctl to disable stp on the bridge in question?  That
> should solve the problem.

Hi, Neil

After disable stp on virbr0 bridge.
# brctl show
bridge name     bridge id               STP enabled     interfaces
breth0          8000.b8ac6f3b0c30       no              eth0
virbr0          8000.000000000000       no

Then do operations as comment4, this issue can be reproduce as following
# ping 192.168.122.99
PING 192.168.122.99 (192.168.122.99) 56(84) bytes of data.
64 bytes from 192.168.122.99: icmp_seq=1 ttl=64 time=0.178 ms
64 bytes from 192.168.122.99: icmp_seq=2 ttl=64 time=0.157 ms
64 bytes from 192.168.122.99: icmp_seq=3 ttl=64 time=0.148 ms
64 bytes from 192.168.122.99: icmp_seq=4 ttl=64 time=0.137 ms
64 bytes from 192.168.122.99: icmp_seq=5 ttl=64 time=0.139 ms
64 bytes from 192.168.122.99: icmp_seq=6 ttl=64 time=0.115 ms
^C
--- 192.168.122.99 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5054ms
rtt min/avg/max/mdev = 0.115/0.145/0.178/0.023 ms

And attach dmesg info as following:
[root@dhcp-91-53 home]# dmesg
virbr0: port 1(tap0) entering disabled state
device tap0 left promiscuous mode
virbr0: port 1(tap0) entering disabled state
device tap0 entered promiscuous mode
virbr0: port 1(tap0) entering forwarding state
__ratelimit: 6 callbacks suppressed
kvm: 32702: cpu0 unimplemented perfctr wrmsr: 0xc1 data 0x0
kvm: 32702: cpu0 unimplemented perfctr wrmsr: 0x186 data 0x130079
kvm: 32702: cpu0 unimplemented perfctr wrmsr: 0xc1 data 0xffd76974
kvm: 32702: cpu0 unimplemented perfctr wrmsr: 0x186 data 0x530079
kvm: 32702: cpu1 unimplemented perfctr wrmsr: 0xc1 data 0x0
kvm: 32702: cpu1 unimplemented perfctr wrmsr: 0x186 data 0x130079
kvm: 32702: cpu1 unimplemented perfctr wrmsr: 0xc1 data 0xffd76974
kvm: 32702: cpu1 unimplemented perfctr wrmsr: 0x186 data 0x530079
tap0: no IPv6 routers present
virbr0: port 1(tap0) entering disabled state
device tap0 left promiscuous mode
virbr0: port 1(tap0) entering disabled state

Any suggestion? Thanks.
Comment 8 Shirley Zhou 2010-11-04 07:14:22 UTC 
Reproduce this issue on kernel-2.6.32-71.8.1.el6.x86_64.

After set stp=off for virbr0
# brctl show
bridge name	bridge id		STP enabled	interfaces
breth0		8000.0023ae8dca1c	no		eth0
virbr0		8000.b2ab3616f407	no		tap1

then did operation as comment 4, this bug can reproduce as:
# ping 192.168.122.176
PING 192.168.122.176 (192.168.122.176) 56(84) bytes of data.
64 bytes from 192.168.122.176: icmp_seq=1 ttl=64 time=0.223 ms
64 bytes from 192.168.122.176: icmp_seq=2 ttl=64 time=0.187 ms
64 bytes from 192.168.122.176: icmp_seq=3 ttl=64 time=0.214 ms
64 bytes from 192.168.122.176: icmp_seq=4 ttl=64 time=0.177 ms
64 bytes from 192.168.122.176: icmp_seq=5 ttl=64 time=0.194 ms
64 bytes from 192.168.122.176: icmp_seq=6 ttl=64 time=0.125 ms
64 bytes from 192.168.122.176: icmp_seq=39 ttl=64 time=0.420 ms
64 bytes from 192.168.122.176: icmp_seq=40 ttl=64 time=0.112 ms
64 bytes from 192.168.122.176: icmp_seq=41 ttl=64 time=0.170 ms
^C
--- 192.168.122.176 ping statistics ---
41 packets transmitted, 9 received, 78% packet loss, time 40746ms
rtt min/avg/max/mdev = 0.112/0.202/0.420/0.085 ms
Comment 9 Shirley Zhou 2010-11-05 04:50:57 UTC 
It is easy to reproduce this bug on host with e1000e nic, like Optiplex 780, Optiplex 760.

00:19.0 Ethernet controller: Intel Corporation 82567LM-3 Gigabit Network Connection (rev 02)
	Subsystem: Dell Device 0276
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 106
	Region 0: Memory at fdfe0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fdfd9000 (32-bit, non-prefetchable) [size=4K]
	Region 2: I/O ports at ece0 [size=32]
	Capabilities: [c8] Power Management version 2
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee03000  Data: 406a
	Capabilities: [e0] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: e1000e
	Kernel modules: e1000e


And this bug also happens on RHEL5.6 host.
kernel-2.6.18-229.el5
Comment 10 Neil Horman 2010-11-10 14:22:11 UTC 

*** This bug has been marked as a duplicate of bug 650097 ***
Comment 11 Shirley Zhou 2010-11-10 14:26:29 UTC 
Re-open this bug, because bug 650097 is report on RHEL5.6, and this bug is open on RHEL6.0.
Comment 12 Neil Horman 2010-11-10 15:12:45 UTC 
yes, but both bugs are due to the same limitation in the virt guests, in that they have unstable MAC addresses, so the solution will be the same in both releases, and it doesn't involve a code change.  AS herbert indicated, it still doesn't explain the external loss of connectivity, so I'll leave this open, but I'm guessing if you assign a stable MAC that will no longer occur either, and we can close these either as dups or NOTABUGs
Comment 13 Shirley Zhou 2010-11-11 04:43:46 UTC 
(In reply to comment #12)
> yes, but both bugs are due to the same limitation in the virt guests, in that
> they have unstable MAC addresses, so the solution will be the same in both
> releases, and it doesn't involve a code change.  AS herbert indicated, it still
> doesn't explain the external loss of connectivity, so I'll leave this open, but
> I'm guessing if you assign a stable MAC that will no longer occur either, and
> we can close these either as dups or NOTABUGs

Hi, Neil

This bug also exist when using public bridge, please see https://bugzilla.redhat.com/show_bug.cgi?id=650097#c7.
Comment 14 Neil Horman 2010-11-11 12:28:45 UTC 
yeah, I see.  Its appearing from the trace to me that it looks like virbr0 might be dissappearing entirely, or the route table is changing such that traffic from the bare metal host is getting directed out the physical interface, rather than through the bridge after the guest exits.

Can you please provide here the ifconfig and ip route show output from this system before and after you shutdown the kvm guest?
Comment 15 Shirley Zhou 2010-11-15 08:48:35 UTC 
Hi, Neil, Herbert 

You are right, this issue does because of bridge mac change.

1. before kvm guest running 
[root@dhcp-91-8 home]# ifconfig
breth0    Link encap:Ethernet  HWaddr B8:AC:6F:19:E7:77  
          inet addr:10.66.91.8  Bcast:10.66.91.255  Mask:255.255.255.0
          inet6 addr: fe80::baac:6fff:fe19:e777/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:102998 errors:0 dropped:0 overruns:0 frame:0
          TX packets:104091 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:6065133 (5.7 MiB)  TX bytes:1533271546 (1.4 GiB)

eth0      Link encap:Ethernet  HWaddr B8:AC:6F:19:E7:77  
          inet6 addr: fe80::baac:6fff:fe19:e777/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:19205 errors:0 dropped:0 overruns:0 frame:0
          TX packets:25355 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1453946 (1.3 MiB)  TX bytes:35206101 (33.5 MiB)
          Memory:fdfe0000-fe000000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:4960 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4960 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:302924 (295.8 KiB)  TX bytes:302924 (295.8 KiB)

2.after kvm guest running
[root@dhcp-91-8 home]# ifconfig
breth0    Link encap:Ethernet  HWaddr AA:7C:C8:E8:4A:5A  
          inet addr:10.66.91.8  Bcast:10.66.91.255  Mask:255.255.255.0
          inet6 addr: fe80::baac:6fff:fe19:e777/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:103035 errors:0 dropped:0 overruns:0 frame:0
          TX packets:104106 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:6067873 (5.7 MiB)  TX bytes:1533275144 (1.4 GiB)

eth0      Link encap:Ethernet  HWaddr B8:AC:6F:19:E7:77  
          inet6 addr: fe80::baac:6fff:fe19:e777/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:19246 errors:0 dropped:0 overruns:0 frame:0
          TX packets:25370 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1457444 (1.3 MiB)  TX bytes:35209699 (33.5 MiB)
          Memory:fdfe0000-fe000000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:5000 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5000 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:305324 (298.1 KiB)  TX bytes:305324 (298.1 KiB)

tap0      Link encap:Ethernet  HWaddr AA:7C:C8:E8:4A:5A  
          inet6 addr: fe80::a87c:c8ff:fee8:4a5a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

3. after guest quit
[root@dhcp-91-8 ~]# ifconfig
breth0    Link encap:Ethernet  HWaddr B8:AC:6F:19:E7:77  
          inet addr:10.66.91.8  Bcast:10.66.91.255  Mask:255.255.255.0
          inet6 addr: fe80::baac:6fff:fe19:e777/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:103116 errors:0 dropped:0 overruns:0 frame:0
          TX packets:104127 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:6075998 (5.7 MiB)  TX bytes:1533279322 (1.4 GiB)

eth0      Link encap:Ethernet  HWaddr B8:AC:6F:19:E7:77  
          inet6 addr: fe80::baac:6fff:fe19:e777/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:19318 errors:0 dropped:0 overruns:0 frame:0
          TX packets:25410 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1463188 (1.3 MiB)  TX bytes:35218580 (33.5 MiB)
          Memory:fdfe0000-fe000000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:5088 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5088 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:310604 (303.3 KiB)  TX bytes:310604 (303.3 KiB)

So after guest quit, host bridge mac will return back. And this host network issue happens at that time.
Comment 16 Neil Horman 2010-11-15 18:48:57 UTC 
ok, so it seems we know both whats going on here, and what the fix is.  Theres general agreement it would seem that the fix needs to be done in libvirt by statically assigning a mac to the bridge.  The rhel6 version of this bug (bz 650097) has been closed as a dup of bz 609463.  I'm going to close this as a dup of the corresponding RHEL5 virt bug.

*** This bug has been marked as a duplicate of bug 560994 ***