Bug 846585 - [qemu-kvm] [hot-plug] qemu-process (RHEL6.3 guest) goes into D state during nic hot unplug (netdev_del hostnet1)
[qemu-kvm] [hot-plug] qemu-process (RHEL6.3 guest) goes into D state during n...
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel (Show other bugs)
x86_64 Linux
urgent Severity urgent
: rc
: ---
Assigned To: jason wang
: ZStream
: 851874 (view as bug list)
Depends On:
Blocks: 851444
  Show dependency treegraph
Reported: 2012-08-08 03:51 EDT by GenadiC
Modified: 2013-02-21 01:45 EST (History)
24 users (show)

See Also:
Fixed In Version: kernel-2.6.32-301.el6
Doc Type: Bug Fix
Doc Text:
If a mirror or redirection action is configured to cause packets to go to another device, the classifier holds a reference count. However, it was previously assuming that the administrator cleaned up all redirections before removing. Packets were therefore dropped if the mirrored device was not present, and connectivity to the host could be lost. To prevent such problems, a notifier and cleanup are now run during the unregister action. Packets are not dropped if the a mirrored device is not present.
Story Points: ---
Clone Of:
Last Closed: 2013-02-21 01:45:26 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
logs (15.27 MB, application/x-tar)
2012-08-08 03:58 EDT, GenadiC
no flags Details

  None (edit)
Description GenadiC 2012-08-08 03:51:02 EDT
Description of problem:

we have a case in which qemu process (guest, rhel6.3 latest) goes into Dl state after nic deactivation.
in some other cases, host looses network connectivity after same deactivation.

the following message start to appear on host console: 

Message from syslogd@orchid-vds1 at Aug  8 10:35:39 ...
kernel:unregister_netdevice: waiting for vnet1 to become free. Usage count = 2

we started systemtap debugger to capture the qemu-commands between libvirt and qemu-process (attached), the last command sent to the qemu-process is:

 219.898 > 0x7f57f8010fe0 {"execute":"device_del","arguments":{"id":"net2"},"id":"libvirt-70"}
219.899 < 0x7f57f8010fe0 {"return": {}, "id": "libvirt-70"}
219.899 > 0x7f57f8010fe0 {"execute":"netdev_del","arguments":{"id":"hostnet2"},"id":"libvirt-71"}
219.933 < 0x7f57f8010fe0 {"return": {}, "id": "libvirt-71"}
220.298 > 0x7f57f8010fe0 {"execute":"query-balloon","id":"libvirt-72"}
220.299 < 0x7f57f8010fe0 {"return": {"actual": 1073741824}, "id": "libvirt-72"}
220.347 > 0x7f57f8010fe0 {"execute":"device_del","arguments":{"id":"net1"},"id":"libvirt-73"}
220.348 < 0x7f57f8010fe0 {"return": {}, "id": "libvirt-73"}
220.348 > 0x7f57f8010fe0 {"execute":"netdev_del","arguments":{"id":"hostnet1"},"id":"libvirt-74"

process command-line:

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -S -M rhel6.3.0 -cpu Conroe -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name RHEL6_1 -uuid d41a5e62-582b-4d52-8326-8335a94ed77c -smbios type=1,manufacturer=Red Hat,product=RHEV Hypervisor,version=6Server-,serial=06577002-2B21-3E45-9926-E52BFFFF3659_00:14:5E:17:D5:B0,uuid=d41a5e62-582b-4d52-8326-8335a94ed77c -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/RHEL6_1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2012-08-07T19:08:27,driftfix=slew -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/47ee94d0-cfcd-44a6-9a8f-9cec41868ae5/85d077a4-1992-4030-9393-678e397a31e8/images/7759d507-99c0-43e0-8d52-e86af5327d0a/15b4cdcc-9668-45f9-b680-45a671dc45ad,if=none,id=drive-virtio-disk0,format=qcow2,serial=7759d507-99c0-43e0-8d52-e86af5327d0a,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:23:46:0a,bus=pci.0,addr=0x3 -netdev tap,fd=31,id=hostnet1,vhost=on,vhostfd=32 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=00:1a:4a:23:46:23,bus=pci.0,addr=0x4 -netdev tap,fd=33,id=hostnet2,vhost=on,vhostfd=34 -device virtio-net-pci,netdev=hostnet2,id=net2,mac=00:1a:4a:23:46:c5,bus=pci.0,addr=0x6 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/RHEL6_1.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -chardev pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0 -spice port=5900,tls-port=5901,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=inputs -k en-us -vga qxl -global qxl-vga.vram_size=67108864 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8
char device redirected to /dev/pts/2
Comment 1 GenadiC 2012-08-08 03:57:42 EDT



kernel: 2.6.32-279.el6.x86_64


[root@e ~]# lsmod | grep virtio
virtio_balloon          4856  0 
virtio_console         18027  0 
virtio_net             16760  0 
virtio_blk              7292  3 
virtio_pci              7113  0 
virtio_ring             7729  5 virtio_balloon,virtio_console,virtio_net,virtio_blk,virtio_pci
virtio                  4890  5 virtio_balloon,virtio_console,virtio_net,virtio_blk,virtio_pci

kernel: 2.6.32-279.el6.x86_64
Comment 2 GenadiC 2012-08-08 03:58:24 EDT
Created attachment 602959 [details]
Comment 3 Dor Laor 2012-08-08 05:15:38 EDT
- Does it happens each time or not that often?
- Can you please repeat the exact same scenario but w/ virsh commands?
  The KVM qe do these tests on a regular basis and we haven't got similar 
Comment 4 Haim 2012-08-08 11:47:36 EDT
(In reply to comment #3)
> - Does it happens each time or not that often?

it happens each time for specific guests (domains) managed by vdsm and libvirt.
> - Can you please repeat the exact same scenario but w/ virsh commands?

reproduction steps:

use the following network xml files:


<interface type='bridge'>
  <mac address='00:1a:4a:23:46:0a'/>
  <source bridge='rhevm'/>
  <target dev='vnet0'/>
  <model type='virtio'/>
  <alias name='net0'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

<interface type='bridge'>                                                                                                                                                                                            
  <mac address='00:1a:4a:23:46:23'/>                                                                                                                                                                                 
  <source bridge='VM_VLAN12'/>                                                                                                                                                                                       
  <target dev='vnet1'/>                                                                                                                                                                                              
  <model type='virtio'/>                                                                                                                                                                                             
  <alias name='net1'/>                                                                                                                                                                                               
  <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>                                                                                                                                        

<interface type='bridge'>                                                                                                                                                                                            
  <mac address='00:1a:4a:23:46:c5'/>                                                                                                                                                                                 
  <source bridge='VM_VLAN12'/>                                                                                                                                                                                       
  <target dev='vnet2'/>                                                                                                                                                                                              
  <model type='virtio'/>
  <alias name='net2'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

start vm using the following xml:

<domain type="kvm">
                <channel type="unix">
                        <target name="com.redhat.rhevm.vdsm" type="virtio"/>
                        <source mode="bind" path="/var/lib/libvirt/qemu/channels/RHEL6_1.com.redhat.rhevm.vdsm"/>
                <input bus="ps2" type="mouse"/>
                <channel type="spicevmc">
                        <target name="com.redhat.spice.0" type="virtio"/>
                <graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice">
                        <channel mode="secure" name="main"/>
                        <channel mode="secure" name="inputs"/>
                <console type="pty">
                        <target port="0" type="virtio"/>
                <controller type="usb">
                        <address  domain="0x0000"  function="0x2"  slot="0x01"  type="pci" bus="0x00"/>
                        <address  domain="0x0000"  function="0x0"  slot="0x02"  type="pci" bus="0x00"/>
                        <model heads="1" type="qxl" vram="65536"/>
                <interface type="bridge">
                        <mac address="00:1a:4a:23:46:0a"/>
                        <model type="virtio"/>
                        <source bridge="rhevm"/>
                <interface type="bridge">
                        <address  domain="0x0000"  function="0x0"  slot="0x04"  type="pci" bus="0x00"/>
                        <mac address="00:1a:4a:23:46:23"/>
                        <model type="virtio"/>
                        <source bridge="VM_VLAN12"/>
                <interface type="bridge">
                        <address  domain="0x0000"  function="0x0"  slot="0x06"  type="pci" bus="0x00"/>
                        <mac address="00:1a:4a:23:46:c5"/>
                        <model type="virtio"/>
                        <source bridge="VM_VLAN12"/>
                <memballoon model="virtio"/>
                <disk device="cdrom" snapshot="no" type="file">
                        <address  bus="1"  controller="0"  target="0"  type="drive" unit="0"/>
                        <source file="" startupPolicy="optional"/>
                        <target bus="ide" dev="hdc"/>
                <disk device="disk" snapshot="no" type="block">
                        <address  domain="0x0000"  function="0x0"  slot="0x05"  type="pci" bus="0x00"/>
                        <source dev="/rhev/data-center/47ee94d0-cfcd-44a6-9a8f-9cec41868ae5/85d077a4-1992-4030-9393-678e397a31e8/images/7759d507-99c0-43e0-8d52-e86af5327d0a/15b4cdcc-9668-45f9-b680-45a671dc45ad"/>
                        <target bus="virtio" dev="vda"/>
                        <boot order="1"/>
                        <driver cache="none" error_policy="stop" io="native" name="qemu" type="qcow2"/>
                <type arch="x86_64" machine="rhel6.3.0">hvm</type>
                <smbios mode="sysinfo"/>
        <sysinfo type="smbios">
                        <entry name="manufacturer">Red Hat</entry>
                        <entry name="product">RHEV Hypervisor</entry>
                        <entry name="version">6Server-</entry>
                        <entry name="serial">936B24D8-3EA3-3A6C-AD2E-84D35C84B839_00:14:5E:17:D0:38</entry>
                        <entry name="uuid">d41a5e62-582b-4d52-8326-8335a94ed77c</entry>
        <clock adjustment="-43200" offset="variable">
                <timer name="rtc" tickpolicy="catchup"/>
        <cpu match="exact">
                <topology cores="1" sockets="1" threads="1"/>

run the following detach command using libvirt (virsh):

virsh detach-device <domain> vnet0.xml

lost connectivity to host
Comment 18 Jarod Wilson 2012-08-24 15:27:52 EDT
Patch(es) available on kernel-2.6.32-301.el6
Comment 21 Meni Yakove 2012-08-26 07:54:41 EDT
Verified with kernel-2.6.32-301.el6.x86_64.
Comment 22 jason wang 2012-08-27 07:27:04 EDT
*** Bug 851874 has been marked as a duplicate of this bug. ***
Comment 24 errata-xmlrpc 2013-02-21 01:45:26 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.