Description of problem: Description of problem: I'm unplugging multiple (20) virtio_scsi disks simultaneously over multiple (2) monitors while reading/writing randomly from/to those disks using stresser.sh script. Version-Release number of selected component (if applicable): qemu-kvm-1.4.2-15.fc19.x86_64 and upstream qemu-1.7.0 guest: F19 How reproducible: Always Steps to Reproduce: 1. start VM with 2 monitors, smp4 2. start stresser scripts 3. hotplug 20 virtio_scsi disks (using one monitor) 4. unplug disks using booth monitors Actual results: qemu-system-x86_64: /builddir/build/BUILD/qemu-1.4.2/hw/virtio-scsi.c:633: virtio_scsi_push_event: Assertion `event == 0' failed. and process termination with status 134 Expected results: All disks hot-plugged correctly (or at least qemu survives :D) LOG: 17:39:37 DEBUG| Context: Unplug and remove the devices 17:39:37 DEBUG| Th0: works with ['drive_stg0', 'stg0', 'drive_stg2', 'stg2', 'drive_stg4', 'stg4', 'drive_stg6', 'stg6', 'drive_stg8', 'stg8'] devices 17:39:37 DEBUG| Th1: works with ['drive_stg1', 'stg1', 'drive_stg3', 'stg3', 'drive_stg5', 'stg5', 'drive_stg7', 'stg7', 'drive_stg9', 'stg9'] devices 17:39:37 DEBUG| (monitor hmp1) Sending command 'device_del stg8' 17:39:37 DEBUG| Send command: device_del stg8 17:39:37 DEBUG| (monitor TestHMP1) Sending command 'device_del stg9' 17:39:37 DEBUG| Send command: device_del stg9 17:39:37 DEBUG| (monitor hmp1) Sending command 'device_del stg6' 17:39:37 DEBUG| Send command: device_del stg6 17:39:37 DEBUG| (monitor TestHMP1) Sending command 'device_del stg7' 17:39:37 DEBUG| Send command: device_del stg7 17:39:38 DEBUG| (monitor hmp1) Sending command 'device_del stg4' 17:39:38 DEBUG| Send command: device_del stg4 17:39:38 DEBUG| (monitor TestHMP1) Sending command 'device_del stg5' 17:39:38 DEBUG| Send command: device_del stg5 17:39:38 DEBUG| (monitor hmp1) Sending command 'device_del stg2' 17:39:38 DEBUG| Send command: device_del stg2 17:39:38 DEBUG| (monitor TestHMP1) Sending command 'device_del stg3' 17:39:38 DEBUG| Send command: device_del stg3 17:39:38 DEBUG| (monitor hmp1) Sending command 'device_del stg0' 17:39:38 DEBUG| Send command: device_del stg0 17:39:38 DEBUG| (monitor TestHMP1) Sending command 'device_del stg1' 17:39:38 DEBUG| Send command: device_del stg1 17:39:38 INFO | [qemu output] qemu-system-x86_64: /builddir/build/BUILD/qemu-1.4.2/hw/virtio-scsi.c:633: virtio_scsi_push_event: Assertion `event == 0' failed. 17:39:38 DEBUG| Unplug status: verified 5 17:39:38 DEBUG| Unplug status: verified 5 17:39:38 DEBUG| All threads finished. 17:39:38 DEBUG| Checking image file images/stg0 17:39:38 DEBUG| Running '/bin/qemu-img info images/stg0' 17:39:38 DEBUG| Running '/bin/qemu-img check images/stg0' 17:39:38 DEBUG| Removing image file images/stg0 17:39:38 DEBUG| Checking image file images/stg1 17:39:38 DEBUG| Running '/bin/qemu-img info images/stg1' 17:39:38 DEBUG| Running '/bin/qemu-img check images/stg1' 17:39:38 DEBUG| Removing image file images/stg1 17:39:38 DEBUG| Checking image file images/stg2 17:39:38 DEBUG| Running '/bin/qemu-img info images/stg2' 17:39:38 DEBUG| Running '/bin/qemu-img check images/stg2' 17:39:38 DEBUG| Removing image file images/stg2 17:39:38 DEBUG| Checking image file images/stg3 17:39:38 DEBUG| Running '/bin/qemu-img info images/stg3' 17:39:38 DEBUG| Running '/bin/qemu-img check images/stg3' 17:39:38 DEBUG| Removing image file images/stg3 17:39:38 DEBUG| Checking image file images/stg4 17:39:38 DEBUG| Running '/bin/qemu-img info images/stg4' 17:39:38 DEBUG| Running '/bin/qemu-img check images/stg4' 17:39:38 DEBUG| Checking image file images/stg5 17:39:38 DEBUG| Running '/bin/qemu-img info images/stg5' 17:39:38 DEBUG| Running '/bin/qemu-img check images/stg5' 17:39:38 DEBUG| Removing image file images/stg5 17:39:38 DEBUG| Checking image file images/stg6 17:39:38 DEBUG| Running '/bin/qemu-img info images/stg6' 17:39:38 DEBUG| Running '/bin/qemu-img check images/stg6' 17:39:38 DEBUG| Removing image file images/stg6 17:39:38 DEBUG| Checking image file images/stg7 17:39:38 DEBUG| Running '/bin/qemu-img info images/stg7' 17:39:38 DEBUG| Running '/bin/qemu-img check images/stg7' 17:39:38 DEBUG| Removing image file images/stg7 17:39:38 DEBUG| Checking image file images/stg8 17:39:38 DEBUG| Running '/bin/qemu-img info images/stg8' 17:39:38 DEBUG| Running '/bin/qemu-img check images/stg8' 17:39:38 DEBUG| Removing image file images/stg8 17:39:38 DEBUG| Checking image file images/stg9 17:39:38 DEBUG| Running '/bin/qemu-img info images/stg9' 17:39:39 DEBUG| Running '/bin/qemu-img check images/stg9' 17:39:39 DEBUG| Removing image file images/stg9 17:39:39 DEBUG| Context: Verify disks after unplug 17:39:42 INFO | [qemu output] /tmp/aexpect/KTbARGIs/aexpect-aRufRI.sh: line 1: 2381 Aborted (core dumped) MALLOC_PERTURB_=1 /bin/qemu-kvm -S -name 'virt-tests-vm1' -sandbox off -M pc -nodefaults -vga std -chardev socket,id=hmp_id_hmp1,path=/tmp/monitor-hmp1-20140110-173912-RifqXYSh,server,nowait -mon chardev=hmp_id_hmp1,mode=readline -chardev socket,id=hmp_id_TestHMP1,path=/tmp/monitor-TestHMP1-20140110-173912-RifqXYSh,server,nowait -mon chardev=hmp_id_TestHMP1,mode=readline -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20140110-173912-RifqXYSh,server,nowait -device isa-serial,chardev=serial_id_serial0 -chardev socket,id=seabioslog_id_20140110-173912-RifqXYSh,path=/tmp/seabios-20140110-173912-RifqXYSh,server,nowait -device isa-debugcon,chardev=seabioslog_id_20140110-173912-RifqXYSh,iobase=0x402 -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=03 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=04 -drive id=drive_image1,if=none,file=/home/medic/Work/Projekty/autotest/autotest-ldoktor/client/tests/virt/shared/data/images/f19-64.qcow2 -device scsi-hd,id=image1,drive=drive_image1 -device virtio-net-pci,mac=9a:b2:b3:b4:b5:b6,id=idITGGLz,netdev=idveGzWi,bus=pci.0,addr=05 -netdev tap,id=idveGzWi,fd=15 -m 1024 -smp 4,maxcpus=4,cores=1,threads=1,sockets=4 -cpu 'SandyBridge' -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :0 -rtc base=utc,clock=host,driftfix=none -boot order=cdn,once=c,menu=off -enable-kvm 17:39:42 INFO | [qemu output] (Process terminated with status 134) Version-Release number of selected component: qemu-system-x86-1.4.2-15.fc19 Additional info: reporter: libreport-2.1.10 backtrace_rating: 4 cmdline: /usr/bin/qemu-system-x86_64 -machine accel=kvm -S -name virt-tests-vm1 -sandbox off -M pc -nodefaults -vga std -chardev socket,id=hmp_id_hmp1,path=/tmp/monitor-hmp1-20140110-173912-RifqXYSh,server,nowait -mon chardev=hmp_id_hmp1,mode=readline -chardev socket,id=hmp_id_TestHMP1,path=/tmp/monitor-TestHMP1-20140110-173912-RifqXYSh,server,nowait -mon chardev=hmp_id_TestHMP1,mode=readline -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20140110-173912-RifqXYSh,server,nowait -device isa-serial,chardev=serial_id_serial0 -chardev socket,id=seabioslog_id_20140110-173912-RifqXYSh,path=/tmp/seabios-20140110-173912-RifqXYSh,server,nowait -device isa-debugcon,chardev=seabioslog_id_20140110-173912-RifqXYSh,iobase=0x402 -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=03 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=04 -drive id=drive_image1,if=none,file=/home/medic/Work/Projekty/autotest/autotest-ldoktor/client/tests/virt/shared/data/images/f19-64.qcow2 -device scsi-hd,id=image1,drive=drive_image1 -device virtio-net-pci,mac=9a:b2:b3:b4:b5:b6,id=idITGGLz,netdev=idveGzWi,bus=pci.0,addr=05 -netdev tap,id=idveGzWi,fd=15 -m 1024 -smp 4,maxcpus=4,cores=1,threads=1,sockets=4 -cpu SandyBridge -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :0 -rtc base=utc,clock=host,driftfix=none -boot order=cdn,once=c,menu=off -enable-kvm crash_function: virtio_scsi_push_event executable: /usr/bin/qemu-system-x86_64 kernel: 3.12.5-200.fc19.x86_64 runlevel: N 5 type: CCpp uid: 0 Truncated backtrace: Thread no. 1 (4 frames) #4 virtio_scsi_push_event at /usr/src/debug/qemu-1.4.2/hw/virtio-scsi.c:633 #5 qemu_iohandler_poll at iohandler.c:159 #6 main_loop_wait at main-loop.c:417 #7 main_loop at vl.c:2001
Created attachment 848311 [details] File: backtrace
Created attachment 848312 [details] File: cgroup
Created attachment 848313 [details] File: core_backtrace
Created attachment 848314 [details] File: dso_list
Created attachment 848315 [details] File: environ
Created attachment 848316 [details] File: limits
Created attachment 848317 [details] File: maps
Created attachment 848318 [details] File: open_fds
Created attachment 848319 [details] File: proc_pid_status
Created attachment 848320 [details] File: var_log_messages
Interesting. Does the script use libvirt? libguestfs also does some hotplugging tests on virtio-scsi. We're obviously not nearly stressing qemu enough! https://github.com/libguestfs/libguestfs/tree/master/tests/hotplug
Created attachment 848321 [details] Disk stresser which randomly read/writes on all disks (but the first one)
The same problem usually occurs during the hotplug, I can fill another bugzilla if you like. When I use only one monitor or don't start the stresser or use virtio_blk disks, everything works fine.
Paolo, any thoughts?
Should be fixed by http://permalink.gmane.org/gmane.comp.emulators.qemu/250817
Yep, it works fine now, thank you.
Well, qemu doesn't crash, but guest system is still failing and sometimes panics. Should I file another bug for kernel? You can try it yourself, the test is already in current autotest/master: ./run -t qemu -g Fedora.19.x86_64 --no-cleanup --no-downloads -k --keep-image-between-tests --keep-guest-running --smp=4 --machine-type=q35 --no-downloads --qemu-bin=/usr/local/bin/qemu-system-x86_64 --qemu_sandbox=off --nettype bridge --tests='multi_disk_random_hotplug.serial.single_type' -v --disk-bus=virtio_scsi log + serial output will follow.
Created attachment 860129 [details] debug log of multi_disk_random_hotplug.serial.single_type Test log of multi_disk_random_hotplug.serial.single_type: 1) add 20 virtio_scsi disks 2) remove 20 virtio_scsi disks 3) kernel panic Usually it survives all three rounds (with couple of "Buffer I/O error ..." and "end_request: I/O error" messages). This time guest kernel crashed during the first iteration.
Created attachment 860130 [details] serial line of the guest from multi_disk_random_hotplug.serial.single_type This is serial line output of the guest from the test run.
Host: Fedora 19, upstream qemu 8cfc114a2f293c40077d1bdb7500b29db359ca22 Guest: Fedora 19, kernel-3.14-0-0.rc1.git1.1.fc21.x86_64, smp4 Similar results were on kernel-3.12.9-201.fc19 and 3.1.1-200.fc19. Usually it only throws some I/O Errors, but sometimes guest kernel crashes. I haven't seen qemu crash.
Lukas, if there are additional kernel issues, please file a separate bug report. Since no end users have reported this, I'm not going to bother backporting to f19 since there aren't any other fixes pending there and it doesn't justify a build and update bandwidth. I'll backport this to F20 though
qemu-1.6.2-1.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/qemu-1.6.2-1.fc20
Package qemu-1.6.2-1.fc20: * should fix your issue, * was pushed to the Fedora 20 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing qemu-1.6.2-1.fc20' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2014-4134/qemu-1.6.2-1.fc20 then log in and leave karma (feedback).
qemu-1.6.2-1.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.
Hi, the problem still persist. I just tested it on Fedora 20: Host: kernel-3.13.6-200.fc20.x86_64 qemu-kvm-1.6.2-1.fc20.x86_64 and upstream qemu-2.0 (db237e33c08a279f0179f8f5128a6d10d9adc38a) Guest: Fedora 19 - kernel-3.13.5-101.fc19.x86_64 Fedora 20 - kernel-3.13.6-200.fc20.x86_64 Please see the details in the serial session logs
Created attachment 879521 [details] Serial log with qemu-kvm-1.6.2-1.fc20.x86_64 and guest kernel-3.13.6-200.fc20.x86_64
Created attachment 879522 [details] Serial log with qemu-2.0 and guest kernel-3.13.5-101.fc19.x86_64
Lukas, you overloaded this bug with two reported issues. The original report was qemu crashing. Paolo indicated patches fixed it, that's what is in the qemu-2.0.0 build. Are you still seeing _that_ issue? Comment 21 I requested that any other issues (like the ones you mentioned in Comment 17 - 20) be filed as a different bug report. It sounds like you are still seeing those issues with up2date packages. If that's the case (and the original issue is indeed fixed), please re-close this bug, open a new kernel bug, and CC me and paolo
OK, sorry, now I understand the previous message. I'll create a new bugzilla for the in-guest kernel crash. Up-to-date qemu survives without crash.
Thanks Lukas. Reclosing since the update is in stable now