Bug 1141621
| Summary: | libvirtd will crashed after hot-plug a virtual NIC to a guest which use qemu-attach connect to libvirtd | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Luyao Huang <lhuang> | ||||
| Component: | libvirt | Assignee: | John Ferlan <jferlan> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 7.1 | CC: | berrange, dyuan, jferlan, jiahu, mzhan, rbalakri, zhwang | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | libvirt-1.2.8-6.el7 | Doc Type: | Bug Fix | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2015-03-05 07:44:30 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
also can use detach-interface to make libvirtd crashed:
#/usr/libexec/qemu-kvm -m 512 -localtime -net nic,vlan=0,macaddr=52:54:00:12:34:01 -net tap,vlan=0,ifname=tap0,script=no -boot d -cdrom /tmp/test.iso -smp 2 -monitor unix:/tmp/demo,server,nowait -name sdsd2 -uuid cece4f9f-dff0-575d-0e8e-01fe380f12ef &
# virsh qemu-attach 32549
Domain sdsd2 attached to pid 32549
# virsh domiflist sdsd2
Interface Type Source Model MAC
-------------------------------------------------------
tap0 ethernet - rtl8139 52:54:00:12:34:01
#virsh dumpxml sdsd2
<domain type='kvm' id='4'>
<name>sdsd2</name>
<uuid>cece4f9f-dff0-575d-0e8e-01fe380f12ef</uuid>
<memory unit='KiB'>524288</memory>
<currentMemory unit='KiB'>524288</currentMemory>
<vcpu placement='static'>2</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
<boot dev='cdrom'/>
</os>
<features>
<acpi/>
<pae/>
</features>
<clock offset='localtime'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/tmp/test.iso'/>
<backingStore/>
<target dev='hdc' bus='ide'/>
<readonly/>
<address type='drive' controller='0' bus='1' target='0' unit='0'/>
</disk>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='usb' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<interface type='ethernet'>
<mac address='52:54:00:12:34:01'/>
<script path='no'/>
<target dev='tap0'/>
<model type='rtl8139'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='sdl' display='localhost:10.0'/>
<video>
<model type='cirrus' vram='9216' heads='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='none'/>
</devices>
<seclabel type='static' model='selinux' relabel='yes'>
<label>system_u:system_r:virtd_t:s0-s0:c0.c1023</label>
</seclabel>
<seclabel type='static' model='dac' relabel='yes'>
<label>system_u:system_r:virtd_t:s0-s0:c0.c1023</label>
</seclabel>
</domain>
# virsh detach-interface sdsd2 ethernet 52:54:00:12:34:01
error: Failed to detach interface
error: End of file while reading data: Input/output error
error: Failed to reconnect to the hypervisor
Verify this bug with libvirt-1.2.8-6.el7:
Steps:
1.# /usr/libexec/qemu-kvm -m 512 -localtime -net nic,vlan=0,macaddr=52:54:00:12:34:01 -net tap,vlan=0,ifname=tap0,script=no -boot d -cdrom /tmp/test.iso -smp 2 -monitor unix:/tmp/demo,server,nowait -name sdsd2 -uuid cece4f9f-dff0-575d-0e8e-01fe380f12ef &
[1] 21667
# VNC server running on `::1:5900'
2.# virsh qemu-attach 21667
Domain sdsd2 attached to pid 21667
3.# virsh list --all
Id Name State
----------------------------------------------------
73 sdsd2 running
4.# virsh domblklist sdsd2
Target Source
------------------------------------------------
hdc /tmp/test.iso
5.# virsh domiflist sdsd2
Interface Type Source Model MAC
-------------------------------------------------------
tap0 ethernet - rtl8139 52:54:00:12:34:01
6.# virsh detach-interface sdsd2 ethernet 52:54:00:12:34:01
error: Failed to detach interface
error: operation failed: detaching net0 device failed: Device 'net0' not found
7.# virsh dumpxml sdsd2
<domain type='kvm' id='73'>
<name>sdsd2</name>
<uuid>cece4f9f-dff0-575d-0e8e-01fe380f12ef</uuid>
<memory unit='KiB'>524288</memory>
<currentMemory unit='KiB'>524288</currentMemory>
<vcpu placement='static'>2</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-rhel7.1.0'>hvm</type>
<boot dev='cdrom'/>
</os>
<features>
<acpi/>
<pae/>
</features>
<clock offset='localtime'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/tmp/test.iso'/>
<backingStore/>
<target dev='hdc' bus='ide'/>
<readonly/>
<alias name='ide0-1-0'/>
<address type='drive' controller='0' bus='1' target='0' unit='0'/>
</disk>
<controller type='ide' index='0'>
<alias name='ide0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='usb' index='0'>
<alias name='usb0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<interface type='ethernet'>
<mac address='52:54:00:12:34:01'/>
<script path='no'/>
<target dev='tap0'/>
<model type='rtl8139'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='sdl' display=':0' xauth='/root/.xauthUk4TJ3'/>
<video>
<model type='cirrus' vram='9216' heads='1'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='none'>
<alias name='balloon0'/>
</memballoon>
</devices>
<seclabel type='static' model='selinux' relabel='yes'>
<label>system_u:system_r:virtd_t:s0-s0:c0.c1023</label>
</seclabel>
<seclabel type='static' model='dac' relabel='yes'>
<label>system_u:system_r:virtd_t:s0-s0:c0.c1023</label>
</seclabel>
</domain>
8.# vim domif1.xml
9.# virsh detach-device sdsd2 domif1.xml
error: Failed to detach device from domif1.xml
error: operation failed: detaching net0 device failed: Device 'net0' not found
10.# virsh attach-disk sdsd2 /var/lib/libvirt/images/test6.img vda
Disk attached successfully
11.# virsh attach-interface sdsd2 network default
Interface attached successfully
Hi John,
During i do some extend test for qemu-attach, i found another crash,
Should i open another bug or just fix it in this bug?thanks.
Steps:
# /usr/libexec/qemu-kvm -m 512 -hda /var/lib/libvirt/images/test6.img -net nic -net tap,vlan=0,ifname=tap0,script=no --daemonize -monitor unix:/tmp/demo,server,nowait -vnc 127.0.0.1:2 -name sdsd2
# virsh qemu-attach 22094
Domain sdsd2 attached to pid 22094
# virsh screenshot sdsd2
error: could not take a screenshot of sdsd2
error: End of file while reading data: Input/output error
error: Failed to reconnect to the hyperviso
(gdb) t a a bt
Thread 11 (Thread 0x7f114a34d700 (LWP 24159)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f115a39d2a6 in virCondWait (c=c@entry=0x7f115c2f5440,
m=m@entry=0x7f115c2f5418) at util/virthread.c:153
#2 0x00007f115a39d75b in virThreadPoolWorker (
opaque=opaque@entry=0x7f115c2d7a60) at util/virthreadpool.c:104
#3 0x00007f115a39d05e in virThreadHelper (data=<optimized out>)
at util/virthread.c:197
#4 0x00007f1157bf7df3 in start_thread (arg=0x7f114a34d700)
at pthread_create.c:308
#5 0x00007f115750e05d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 10 (Thread 0x7f115ae66880 (LWP 24155)):
#0 0x00007f1157503a8d in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007f115a3625a1 in poll (__timeout=5000, __nfds=13,
__fds=<optimized out>) at /usr/include/bits/poll2.h:46
#2 virEventPollRunOnce () at util/vireventpoll.c:643
#3 0x00007f115a361092 in virEventRunDefaultImpl () at util/virevent.c:308
#4 0x00007f115aee96ad in virNetServerRun (srv=srv@entry=0x7f115c2f52c0)
at rpc/virnetserver.c:1139
---Type <return> to continue, or q <return> to quit---
#5 0x00007f115aeb6548 in main (argc=<optimized out>, argv=<optimized out>)
at libvirtd.c:1507
Thread 9 (Thread 0x7f114ab4e700 (LWP 24158)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f115a39d2a6 in virCondWait (c=c@entry=0x7f115c2f5440,
m=m@entry=0x7f115c2f5418) at util/virthread.c:153
#2 0x00007f115a39d75b in virThreadPoolWorker (
opaque=opaque@entry=0x7f115c2d8090) at util/virthreadpool.c:104
#3 0x00007f115a39d05e in virThreadHelper (data=<optimized out>)
at util/virthread.c:197
#4 0x00007f1157bf7df3 in start_thread (arg=0x7f114ab4e700)
at pthread_create.c:308
#5 0x00007f115750e05d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 8 (Thread 0x7f1148349700 (LWP 24163)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f115a39d2a6 in virCondWait (c=c@entry=0x7f115c2f54d8,
m=m@entry=0x7f115c2f5418) at util/virthread.c:153
#2 0x00007f115a39d77b in virThreadPoolWorker (
---Type <return> to continue, or q <return> to quit---
opaque=opaque@entry=0x7f115c2d7a60) at util/virthreadpool.c:104
#3 0x00007f115a39d05e in virThreadHelper (data=<optimized out>)
at util/virthread.c:197
#4 0x00007f1157bf7df3 in start_thread (arg=0x7f1148349700)
at pthread_create.c:308
#5 0x00007f115750e05d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 7 (Thread 0x7f1146b46700 (LWP 24166)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f115a39d2a6 in virCondWait (c=c@entry=0x7f115c2f54d8,
m=m@entry=0x7f115c2f5418) at util/virthread.c:153
#2 0x00007f115a39d77b in virThreadPoolWorker (
opaque=opaque@entry=0x7f115c2d8090) at util/virthreadpool.c:104
#3 0x00007f115a39d05e in virThreadHelper (data=<optimized out>)
at util/virthread.c:197
#4 0x00007f1157bf7df3 in start_thread (arg=0x7f1146b46700)
at pthread_create.c:308
#5 0x00007f115750e05d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 6 (Thread 0x7f1147347700 (LWP 24165)):
---Type <return> to continue, or q <return> to quit---
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f115a39d2a6 in virCondWait (c=c@entry=0x7f115c2f54d8,
m=m@entry=0x7f115c2f5418) at util/virthread.c:153
#2 0x00007f115a39d77b in virThreadPoolWorker (
opaque=opaque@entry=0x7f115c2d7a60) at util/virthreadpool.c:104
#3 0x00007f115a39d05e in virThreadHelper (data=<optimized out>)
at util/virthread.c:197
#4 0x00007f1157bf7df3 in start_thread (arg=0x7f1147347700)
at pthread_create.c:308
#5 0x00007f115750e05d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 5 (Thread 0x7f114b34f700 (LWP 24157)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f115a39d2a6 in virCondWait (c=c@entry=0x7f115c2f5440,
m=m@entry=0x7f115c2f5418) at util/virthread.c:153
#2 0x00007f115a39d75b in virThreadPoolWorker (
opaque=opaque@entry=0x7f115c2d7a60) at util/virthreadpool.c:104
#3 0x00007f115a39d05e in virThreadHelper (data=<optimized out>)
at util/virthread.c:197
#4 0x00007f1157bf7df3 in start_thread (arg=0x7f114b34f700)
---Type <return> to continue, or q <return> to quit---
at pthread_create.c:308
#5 0x00007f115750e05d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 4 (Thread 0x7f1147b48700 (LWP 24164)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f115a39d2a6 in virCondWait (c=c@entry=0x7f115c2f54d8,
m=m@entry=0x7f115c2f5418) at util/virthread.c:153
#2 0x00007f115a39d77b in virThreadPoolWorker (
opaque=opaque@entry=0x7f115c2d8090) at util/virthreadpool.c:104
#3 0x00007f115a39d05e in virThreadHelper (data=<optimized out>)
at util/virthread.c:197
#4 0x00007f1157bf7df3 in start_thread (arg=0x7f1147b48700)
at pthread_create.c:308
#5 0x00007f115750e05d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 3 (Thread 0x7f1148b4a700 (LWP 24162)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f115a39d2a6 in virCondWait (c=c@entry=0x7f115c2f54d8,
m=m@entry=0x7f115c2f5418) at util/virthread.c:153
---Type <return> to continue, or q <return> to quit---
#2 0x00007f115a39d77b in virThreadPoolWorker (
opaque=opaque@entry=0x7f115c2d8090) at util/virthreadpool.c:104
#3 0x00007f115a39d05e in virThreadHelper (data=<optimized out>)
at util/virthread.c:197
#4 0x00007f1157bf7df3 in start_thread (arg=0x7f1148b4a700)
at pthread_create.c:308
#5 0x00007f115750e05d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 2 (Thread 0x7f114934b700 (LWP 24161)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f115a39d2a6 in virCondWait (c=c@entry=0x7f115c2f5440,
m=m@entry=0x7f115c2f5418) at util/virthread.c:153
#2 0x00007f115a39d75b in virThreadPoolWorker (
opaque=opaque@entry=0x7f115c2d7a60) at util/virthreadpool.c:104
#3 0x00007f115a39d05e in virThreadHelper (data=<optimized out>)
at util/virthread.c:197
#4 0x00007f1157bf7df3 in start_thread (arg=0x7f114934b700)
at pthread_create.c:308
#5 0x00007f115750e05d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
---Type <return> to continue, or q <return> to quit---
Thread 1 (Thread 0x7f1149b4c700 (LWP 24160)):
#0 __strcmp_sse42 () at ../sysdeps/x86_64/multiarch/strcmp-sse42.S:164
#1 0x00007f115a50d0c9 in virSecuritySELinuxSetFileconHelper (
path=0x7f1124000e50 "/var/cache/libvirt/qemu/qemu.screendump.m4QkFu",
tcon=0x0, optional=<optimized out>) at security/security_selinux.c:890
#2 0x00007f115a509513 in virSecurityManagerSetSavedStateLabel (
mgr=0x7f113c10d630, vm=vm@entry=0x7f1138000cf0,
savefile=savefile@entry=0x7f1124000e50 "/var/cache/libvirt/qemu/qemu.screendump.m4QkFu") at security/security_manager.c:547
#3 0x00007f115a506476 in virSecurityStackSetSavedStateLabel (
mgr=<optimized out>, vm=0x7f1138000cf0,
savefile=0x7f1124000e50 "/var/cache/libvirt/qemu/qemu.screendump.m4QkFu")
at security/security_stack.c:351
#4 0x00007f115a509513 in virSecurityManagerSetSavedStateLabel (
mgr=0x7f113c1680a0, vm=0x7f1138000cf0,
savefile=0x7f1124000e50 "/var/cache/libvirt/qemu/qemu.screendump.m4QkFu")
at security/security_manager.c:547
#5 0x00007f11432ff94f in qemuDomainScreenshot (dom=<optimized out>,
st=0x7f11240009f0, screen=<optimized out>, flags=<optimized out>)
at qemu/qemu_driver.c:3858
#6 0x00007f115a425b10 in virDomainScreenshot (
domain=domain@entry=0x7f1124000930, stream=stream@entry=0x7f11240009f0,
screen=0, flags=0) at libvirt.c:3171
---Type <return> to continue, or q <return> to quit---
#7 0x00007f115aec8833 in remoteDispatchDomainScreenshot (
server=<optimized out>, ret=0x7f11240008e0, args=0x7f1124000900,
rerr=0x7f1149b4bc80, msg=<optimized out>, client=0x7f115c2f6340)
at remote_dispatch.h:7412
#8 remoteDispatchDomainScreenshotHelper (server=<optimized out>,
client=0x7f115c2f6340, msg=<optimized out>, rerr=0x7f1149b4bc80,
args=0x7f1124000900, ret=0x7f11240008e0) at remote_dispatch.h:7379
#9 0x00007f115a498ff2 in virNetServerProgramDispatchCall (msg=0x7f115c303db0,
client=0x7f115c2f6340, server=0x7f115c2f52c0, prog=0x7f115c300d20)
at rpc/virnetserverprogram.c:437
#10 virNetServerProgramDispatch (prog=0x7f115c300d20,
server=server@entry=0x7f115c2f52c0, client=0x7f115c2f6340,
msg=0x7f115c303db0) at rpc/virnetserverprogram.c:307
#11 0x00007f115aee81fd in virNetServerProcessMsg (msg=<optimized out>,
prog=<optimized out>, client=<optimized out>, srv=0x7f115c2f52c0)
at rpc/virnetserver.c:172
#12 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7f115c2f52c0)
at rpc/virnetserver.c:193
#13 0x00007f115a39d6c5 in virThreadPoolWorker (
opaque=opaque@entry=0x7f115c2d8090) at util/virthreadpool.c:145
#14 0x00007f115a39d05e in virThreadHelper (data=<optimized out>)
at util/virthread.c:197
#15 0x00007f1157bf7df3 in start_thread (arg=0x7f1149b4c700)
---Type <return> to continue, or q <return> to quit---
at pthread_create.c:308
#16 0x00007f115750e05d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
I found the crash(comment 10) reason and it is a different issue with this bug. So filed another bug 1161831, sorry for the wrong needinfo, and remove it. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0323.html |
Created attachment 937427 [details] back trace Description of problem: libvirtd will crashed after hot-plug a virtual NIC to a guest which use qemu-attach connect to libvirtd Version-Release number of selected component (if applicable): libvirt-1.2.8-2.el7.x86_64 qemu-kvm-rhev-2.1.0-3.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1.[root@localhost ~]# /usr/libexec/qemu-kvm -cdrom /tmp/test.iso -monitor unix:/tmp/demo,server,nowait -name sdsd2 -uuid cece4f9f-dff0-575d-0e8e-01fe380f12ef & [3] 30013 [root@localhost ~]# VNC server running on `::1:5901' 2.[root@localhost ~]# virsh qemu-attach 30013 Domain sdsd2 attached to pid 30013 3.[root@localhost ~]# virsh attach-interface sdsd2 network default error: Failed to attach interface error: End of file while reading data: Input/output error error: Failed to reconnect to the hypervisor 4.check libvirtd pid changed Actual results: libvirtd crashed after hot-plug a virtual NC Expected results: report error or attach success Additional info: