Bug 1530577
| Summary: | [virtio-blk] Guest reboot automatically after hot unplug virtio-blk device | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Yongxue Hong <yhong> | ||||||
| Component: | qemu-kvm-rhev | Assignee: | Fam Zheng <famz> | ||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | aihua liang <aliang> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | high | ||||||||
| Version: | 7.5 | CC: | aliang, bugproxy, chayang, coli, famz, hannsj_uhl, junli, juzhang, knoel, lijin, michen, phou, qzhang, virt-maint, yilzhang | ||||||
| Target Milestone: | rc | ||||||||
| Target Release: | 7.5 | ||||||||
| Hardware: | All | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2018-02-01 00:43:33 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | 1536979 | ||||||||
| Bug Blocks: | 1399177, 1525303, 1536955 | ||||||||
| Attachments: |
|
||||||||
Additional info : Info from /var/crash: .... [ 74.198552] RTAS: event: 1, Type: Unknown, Severity: 1 [ 74.463578] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 74.463779] pci 0000:00:0a.0: reg 0x10: [io 0x10000-0x1003f] [ 74.463839] pci 0000:00:0a.0: reg 0x14: [mem 0x00000000-0x00000fff] [ 74.464059] pci 0000:00:0a.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref] [ 74.464832] iommu: Adding device 0000:00:0a.0 to group 0 [ 74.464883] pci 0000:00:0a.0: BAR 4: assigned [mem 0x210000004000-0x210000007fff 64bit pref] [ 74.464968] pci 0000:00:0a.0: BAR 1: assigned [mem 0x200080004000-0x200080004fff] [ 74.464994] pci 0000:00:0a.0: BAR 0: assigned [io 0x10080-0x100bf] [ 74.465142] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0003) [ 74.466150] virtio-pci 0000:00:0a.0: Using 64-bit direct DMA at offset 800000000000000 [ 77.639506] RTAS: event: 2, Type: Unknown, Severity: 1 [ 77.993408] iommu: Removing device 0000:00:0a.0 from group 0 [ 81.909332] Unable to handle kernel paging request for data at address 0x00000010 [ 81.909334] Faulting instruction address: 0xc0000000004e17ac [ 81.909377] Oops: Kernel access of bad area, sig: 11 [#1] [ 81.909380] SMP NR_CPUS=2048 NUMA pSeries [ 81.909415] Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables xfs libcrc32c virtio_blk virtio_net bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm virtio_pci virtio_ring i2c_core virtio dm_mirror dm_region_hash dm_log dm_mod [ 81.909445] CPU: 1 PID: 3367 Comm: kworker/1:1H Not tainted 3.10.0-826.el7.ppc64le #1 [ 81.909502] Workqueue: kblockd blk_mq_run_work_fn [ 81.909504] task: c0000003ee3f80c0 ti: c000000049408000 task.ti: c000000049408000 [ 81.909505] NIP: c0000000004e17ac LR: c0000000004d7598 CTR: c0000000004d74b0 [ 81.909506] REGS: c00000004940b920 TRAP: 0300 Not tainted (3.10.0-826.el7.ppc64le) [ 81.909507] MSR: 8000000100009033 <SF,EE,ME,IR,DR,RI,LE> CR: 240e2922 XER: 00000000 [ 81.909512] CFAR: 00003fffa4dcb564 DAR: 0000000000000010 DSISR: 40000000 SOFTE: 1 GPR00: c0000000004d7598 c00000004940bba0 c000000001273f00 c00000004b577800 GPR04: c00000004b577a40 c00000004b577a40 0000000b1b72bbf9 0000000000000001 GPR08: 0000000000000000 0000000000000000 c00000004b577808 0000000000990000 GPR12: c0000000004d74b0 c000000007b80900 c000000000123c68 c0000003f8c5bb40 GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 GPR20: 0000000000000001 c0000000011e1e70 0000000000000001 fffffffffffffef7 GPR24: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 GPR28: c0000003f0690000 c00000004b577808 c00000004940bbc0 c00000004b577800 [ 81.909527] NIP [c0000000004e17ac] blk_mq_sched_dispatch_requests+0x18c/0x260 [ 81.909528] LR [c0000000004d7598] blk_mq_run_work_fn+0xe8/0x110 [ 81.909529] Call Trace: [ 81.909531] [c00000004940bba0] [c00000004940bc10] 0xc00000004940bc10 (unreliable) [ 81.909533] [c00000004940bc00] [c0000000004d7598] blk_mq_run_work_fn+0xe8/0x110 [ 81.909546] [c00000004940bc40] [c00000000011784c] process_one_work+0x1dc/0x680 [ 81.909548] [c00000004940bce0] [c000000000117e90] worker_thread+0x1a0/0x520 [ 81.909549] [c00000004940bd80] [c000000000123d4c] kthread+0xec/0x100 [ 81.909552] [c00000004940be30] [c00000000000a4b8] ret_from_kernel_thread+0x5c/0xa4 [ 81.909553] Instruction dump: [ 81.909554] eba1ffe8 ebc1fff0 ebe1fff8 7c0803a6 4e800020 e93f0010 7fbd4800 409eff08 [ 81.909556] 2fbb0000 409effc4 e93c00d0 e9290008 <e9290010> 2fa90000 419e008c 7fe3fb78 [ 81.909560] ---[ end trace db2c3c7bf2349f7c ]--- [ 81.911315] [ 81.911319] Sending IPI to other CPUs [ 81.912327] IPI complete Test on x86, also have this issue:
kernel version:3.10.0-826.el7.x86_64
qemu version:qemu-img-rhev-2.10.0-14.el7.x86_64
Test Steps:
1.Start guest with cmds：
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1' \
-sandbox off \
-machine pc \
-nodefaults \
-vga cirrus \
-chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20171102-094326-DoIK24XP,server,nowait \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/rhel75-64-virtio-scsi.qcow2 \
-device virtio-blk-pci,id=image1,drive=drive_image1,bus=pci.0 \
-drive file=/home/test.img,snapshot=off,aio=native,cache=none,format=qcow2,id=drive_data1,if=none \
-device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 \
-device virtio-net-pci,mac=9a:12:13:14:15:17,id=id9HuGl4,vectors=4,netdev=idpS7PTn,bus=pci.0 \
-netdev tap,id=idpS7PTn,vhost=on \
-m 2048 \
-smp 2,maxcpus=2,cores=1,threads=1,sockets=2 \
-cpu 'Opteron_G2',+kvm_pv_unhalt \
-vnc :1 \
-rtc base=utc,clock=host,driftfix=slew \
-boot menu=off,strict=off,order=cdn,once=c \
-enable-kvm \
-monitor stdio \
2. Hot unplug data device
{"execute":"device_del","arguments":{"id":"data1"}}
Test Result:
Guest reboot with crash info in "/var/crash",crash info, see attachment.
Created attachment 1376670 [details]
vmcore-dmesg for x86
*** Bug 1528194 has been marked as a duplicate of this bug. *** Not a regression bug on x86.
Test on kernel:3.10.0-537.el7.x86_64 + qemu:qemu-kvm-rhev-2.9.0-1.el7.x86_64 also have this issue.
Can't reproduce it on qemu earlier versions, for can't get the source from brewweb.
Note:
Guest will reboot about 10s later after unplug the device.
And windows don't hit this issue.
(In reply to aihua liang from comment #7) > Not a regression bug on x86. > > Test on kernel:3.10.0-537.el7.x86_64 + > qemu:qemu-kvm-rhev-2.9.0-1.el7.x86_64 also have this issue. > > Can't reproduce it on qemu earlier versions, for can't get the source from > brewweb. > > Note: > Guest will reboot about 10s later after unplug the device. > And windows don't hit this issue. Removing Regression keyword then (but this bug is still a high priority given the severity). Yes, it's still 7.4 material. Oops, I meant 7.5. I cannot reproduce this. What data is on /home/test.img? What operation is the guest doing when you hot unplug? (In reply to aihua liang from comment #4) > Created attachment 1376670 [details] > vmcore-dmesg for x86 guest call trace info: [ 55.455992] Call Trace: [ 55.456210] [<ffffffff8911ff62>] __blk_mq_run_hw_queue+0xa2/0xb0 [ 55.456730] [<ffffffff891201c5>] blk_mq_run_work_fn+0x15/0x20 [ 55.457218] [<ffffffff88eaf89a>] process_one_work+0x17a/0x440 [ 55.457711] [<ffffffff88eb0566>] worker_thread+0x126/0x3c0 [ 55.458183] [<ffffffff88eb0440>] ? manage_workers.isra.24+0x2a0/0x2a0 [ 55.458734] [<ffffffff88eb783f>] kthread+0xcf/0xe0 [ 55.459151] [<ffffffff88eb7770>] ? insert_kthread_work+0x40/0x40 [ 55.459668] [<ffffffff89505c58>] ret_from_fork+0x58/0x90 [ 55.460120] [<ffffffff88eb7770>] ? insert_kthread_work+0x40/0x40 [ 55.460638] Code: 00 00 48 83 c4 18 5b 41 5c 41 5d 41 5e 41 5f 5d c3 4c 3b 6b 10 0f 85 43 ff ff ff 45 84 ff 75 c7 49 8b 86 d0 00 00 00 48 8b 40 08 <48> 83 78 10 00 74 47 48 89 df e8 27 f7 ff ff eb b3 0f 1f 44 00 [ 55.463169] RIP [<ffffffff89126b9a>] blk_mq_sched_dispatch_requests+0x14a/0x1c0 [ 55.463805] RSP <ffff9290fc0ebda0> [ 55.464107] CR2: 0000000000000010 (In reply to Fam Zheng from comment #14) > I cannot reproduce this. What data is on /home/test.img? What operation is > the guest doing when you hot unplug? Bug 1536979 might be a duplicate of this one. (In reply to Ademar Reis from comment #17) > (In reply to Fam Zheng from comment #14) > > I cannot reproduce this. What data is on /home/test.img? What operation is > > the guest doing when you hot unplug? > > Bug 1536979 might be a duplicate of this one. And Bug 1536955 might be related. Can't hit this issue on RHEL7.4 guest. Created attachment 1387641 [details]
The qemu log
Can't hit this issue when guest's kernel version in range [835, 842] So, it's ok on latest kernel version, do i need more debug on earlier versions(before 826), fam? Good, I think we can just close this bug. Thanks! |
Description of problem: Guest reboot automatically after hot unplug virtio-blk device Version-Release number of selected component (if applicable): host kernel : 3.10.0-826.el7.ppc64le guest kernel : 3.10.0-826.el7.ppc64le qemu : qemu-kvm-rhev-2.10.0-14.el7.ppc64le How reproducible: 100% Steps to Reproduce: 1.Boot a guest. eg: /usr/libexec/qemu-kvm \ -name 'guest' \ -machine pseries-rhel7.5.0 \ -m 16G \ -nodefaults \ -smp 8,cores=4,threads=2,sockets=1 \ -boot order=cdn,once=d,menu=off,strict=off \ -device nec-usb-xhci,id=xhci0 \ -device usb-tablet,id=usb-tablet0 \ -device usb-kbd,id=usb-kbd0 \ -device VGA,id=vga0 \ -chardev socket,id=qmp_id_qmpmonitor,path=/var/tmp/qmp-cmd-monitor-yhong,server,nowait \ -mon chardev=qmp_id_qmpmonitor,mode=control \ -enable-kvm \ -drive file=/home/yhong/images/rhel7-5-ppc64le-30G.qcow2,format=qcow2,aio=native,if=none,cache=none,media=disk,werror=stop,rerror=stop,id=drive_system \ -device virtio-blk-pci,bus=pci.0,drive=drive_system,id=system,addr=0x7,bootindex=0 \ -netdev tap,id=hostnet0,script=/etc/qemu-ifup \ -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=40:f2:e9:5d:9c:03 \ -qmp tcp:0:3000,server,nowait \ -chardev socket,id=serial_id_serial,path=/var/tmp/serial-yhong,server,nowait \ -device spapr-vty,reg=0x30000000,chardev=serial_id_serial \ -monitor stdio \ -vnc :30 2.Hot plug a virtio-blk device. eg: [root@ibm-p8-garrison-04 images]# nc -U /var/tmp/qmp-cmd-monitor-yhong {"QMP": {"version": {"qemu": {"micro": 0, "minor": 10, "major": 2}, "package": "(qemu-kvm-rhev-2.10.0-14.el7)"}, "capabilities": []}} {"execute":"qmp_capabilities"} {"return": {}} {"execute":"__com.redhat_drive_add", "arguments":{"file":"/home/yhong/images/data0-10G.qcow2","format":"qcow2","id":"drive-data-disk0"}} {"return": {}} {"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"drive-data-disk0","addr":"0x0a","id":"data-disk0"}} {"return": {}} 3.Hot unplug the above virtio-blk device. eg: {"execute":"device_del","arguments":{"id":"data-disk0"}} {"return": {}} {"timestamp": {"seconds": 1514978124, "microseconds": 223447}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/data-disk0/virtio-backend"}} {"timestamp": {"seconds": 1514978124, "microseconds": 232576}, "event": "DEVICE_DELETED", "data": {"device": "data-disk0", "path": "/machine/peripheral/data-disk0"}} {"timestamp": {"seconds": 1514978129, "microseconds": 149119}, "event": "RESET", "data": {"guest": true}} Actual results: Guest reboot automatically. Expected results: Guest should not reboot automatically. Additional info: