Bug 1168213
| Summary: | main-loop: WARNING: I/O thread spun for 1000 iterations while doing stream block device. | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Yang Xue <yaxue> | ||||
| Component: | qemu-kvm-rhev | Assignee: | Virtualization Maintenance <virt-maint> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Gu Nini <ngu> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 7.1 | CC: | chayang, coli, hhuang, juzhang, lolyu, michen, ngu, virt-maint, xfu, xiagao, xuhan | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | qemu-kvm-rhev-2.12.0-5.el7 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-11-01 11:01:10 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Reproduce this problem again and 100% reproducible. This problem causes the vcpu not running. Met this in ppc host with ppc64le guest too: qemu-kvm related package: ipxe-roms-qemu-20130517-6.gitc4bce43.el7.noarch qemu-img-rhev-2.1.2-14.el7.ppc64 qemu-kvm-tools-rhev-2.1.2-14.el7.ppc64 libvirt-daemon-driver-qemu-1.2.8-9.el7.ppc64 qemu-kvm-common-rhev-2.1.2-14.el7.ppc64 qemu-kvm-rhev-2.1.2-14.el7.ppc64 qemu-kvm-rhev-debuginfo-2.1.2-14.el7.ppc64 host kernel verion:3.10.0-201.el7.ppc64 guest kernel version: 3.10.0-210 This problem in ppc host have some different results: Met this once the qemu process is terminated after 8 minutes that this message shows up. main-loop: WARNING: I/O thread spun for 1000 iterations qemu: terminating on signal 1 And normally it will blocked the guest boot up normally after the block stream guest shut down. Created attachment 1156629 [details]
iometer.icf
Found the same log in the bug on following sw versions with Win10.x86_64 guest started with aio=native disks; while for the same guest started with aio=threads disks, there is no the problem.
Host kernel: 3.10.0-370.el7.x86_64
Qemu-kvm: qemu-kvm-rhev-2.5.0-4.el7.x86_64
virtio-win: virtio-win-1.8.0-4
Steps:
1) Start an installed Win10.x86_64 guest with following qemu command:
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1' \
-sandbox off \
-machine pc \
-nodefaults \
-vga std \
-chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_1,server,nowait \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_2,server,nowait \
-mon chardev=qmp_id_catch_monitor,mode=control \
-device pvpanic,ioport=0x505,id=idZelHT0 \
-chardev socket,id=serial_id_serial0,path=/var/tmp/avocado_3,server,nowait \
-device isa-serial,chardev=serial_id_serial0 \
-chardev socket,id=seabioslog_id_20160511-031111-dD14QDp6,path=/var/tmp/avocado_4,server,nowait \
-device isa-debugcon,chardev=seabioslog_id_20160511-031111-dD14QDp6,iobase=0x402 \
-device ich9-usb-ehci1,id=usb1,addr=1d.7,multifunction=on,bus=pci.0 \
-device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=1d.0,firstport=0,bus=pci.0 \
-device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=1d.2,firstport=2,bus=pci.0 \
-device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=1d.4,firstport=4,bus=pci.0 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=03,disable-legacy=off,disable-modern=on \
-drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,file=/usr/share/avocado/data/avocado-vt/images/win10-64-virtio-scsi.qcow2 \
-device scsi-hd,id=image1,drive=drive_image1 \
-drive id=drive_disk1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,file=/usr/share/avocado/data/avocado-vt/images/storage.qcow2 \
-device scsi-hd,id=disk1,drive=drive_disk1 \
-device virtio-net-pci,mac=9a:1a:1b:1c:1d:1e,id=idTv7QY9,vectors=4,netdev=idgrlUJn,bus=pci.0,addr=04,disable-legacy=off,disable-modern=on \
-netdev tap,id=idgrlUJn,vhost=on \
-m 16384 \
-smp 8,maxcpus=8,cores=4,threads=1,sockets=2 \
-cpu 'Westmere',+kvm_pv_unhalt,hv_spinlocks=0x1fff,hv_vapic,hv_time \
-drive id=drive_cd1,if=none,snapshot=off,aio=native,media=cdrom,file=/usr/share/avocado/data/avocado-vt/isos/windows/winutils.iso \
-device scsi-cd,id=cd1,drive=drive_cd1 \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-vnc :0 \
-rtc base=localtime,clock=host,driftfix=slew \
-boot order=cdn,once=c,menu=off,strict=off \
-enable-kvm \
-monitor stdio
2). After the guest boots up, run iometer read/write operation on data drive drive_disk1 inside the guest with iometer.icf(attached).
Results:
Log 'main-loop: WARNING: I/O thread spun for 1000 iterations' was sent out in hmp, the guest hang AND the cpu usage rate of qemu-kvm process would increase to 300% in the host.
Jeff, Would you help to check if comment #8 is caused for the same reason of the bug? If not, I will report a new one. BTW, there is some discussion on the same log at https://bugs.launchpad.net/qemu/+bug/1481272 (In reply to Gu Nini from comment #9) > Jeff, > > Would you help to check if comment #8 is caused for the same reason of the > bug? If not, I will report a new one. > > BTW, there is some discussion on the same log at > https://bugs.launchpad.net/qemu/+bug/1481272 Could not reproduce comment #8 problem with following newest 7.3 sw: Host Kernel: 3.10.0-396.el7.x86_64 Qemu-kvm-rhev: qemu-kvm-rhev-2.6.0-1.el7.x86_64 virtio-win: virtio-win-1.8.0-4 So please ignore the needinfo in comment #9. Could reproduce the warning message with the following version selected: Host: kernel-3.10.0-693.el7.x86_64 qemu-kvm-rhev-2.9.0-17.el7a.x86_64 Guest: kernel-3.10.0-693.el7.x86_64 Commit upstream from Stefan that addresses this BZ: https://lists.gnu.org/archive/html/qemu-devel/2018-05/msg06889.html However, the dependency for this commit is non-trivial; it depends on changes made to thread lock optimizations for the BQL, in Pavel Dovgalyuk's "replay additions" patch series. This is probably too invasive to backport into 7.6, unless we just want to suppress the output anyways: https://lists.gnu.org/archive/html/qemu-devel/2018-02/msg06572.html Submitted backport to go ahead and suppress the output anyway; regardless of the BQL, it has never been a very useful debug message and produces a lot of spurious noise. Fix included in qemu-kvm-rhev-2.12.0-5.el7 Reproduce the bug on 'qemu-kvm-rhev-2.12.0-4.el7.x86_64' with the same steps as that in the bz description. Verify the bug on 'qemu-kvm-rhev-2.12.0-5.el7.x86_64', i.e. no 'main-loop: WARNING: I/O thread spun for 1000 iterations' is sent out from hmp. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3443 |
Description of problem: win2012r2 guest main-loop: WARNING: I/O thread spun for 1000 iterations ,while doing stream block device. Version-Release number of selected component (if applicable): Host: kernel-3.10.0-205.el7.x86_64 qemu-kvm-rhev-2.1.2-10.el7.x86_64 How reproducible: 50% Steps to Reproduce: 1.start win2012r2 guest /bin/qemu-kvm \ -S \ -name 'virt-tests-vm1' \ -sandbox off \ -M pc \ -nodefaults \ -vga qxl \ -global qxl-vga.vram_size=33554432 \ -device intel-hda,bus=pci.0,addr=03 \ -device hda-duplex \ -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20141122-031435-M0GYyrum,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20141122-031435-M0GYyrum,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -device virtio-serial-pci,id=virtio_serial_pci0,bus=pci.0,addr=04 \ -chardev socket,id=devvs,path=/tmp/virtio_port-vs-20141122-031435-M0GYyrum,server,nowait \ -device virtserialport,chardev=devvs,name=vs,id=vs,bus=virtio_serial_pci0.0 \ -chardev socket,id=seabioslog_id_20141122-031435-M0GYyrum,path=/tmp/seabios-20141122-031435-M0GYyrum,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20141122-031435-M0GYyrum,iobase=0x402 \ -device nec-usb-xhci,id=usb1,bus=pci.0,addr=05 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=06 \ -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,file=/home/autotest-devel/client/tests/virt/shared/data/images/win2012-64r2-virtio.qcow2 \ -device scsi-hd,id=image1,drive=drive_image1 \ -device virtio-net-pci,mac=9a:8e:8f:90:91:92,id=idxToyso,vectors=4,netdev=idU9oayT,bus=pci.0,addr=07 \ -netdev tap,id=idU9oayT,vhost=on,vhostfd=23,fd=22 \ -m 65536 \ -smp 8,maxcpus=8,cores=4,threads=1,sockets=2 \ -cpu 'Penryn',+kvm_pv_unhalt,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \ -drive id=drive_cd1,if=none,snapshot=off,aio=native,media=cdrom,file=/home/autotest-devel/client/tests/virt/shared/data/isos/windows/winutils.iso \ -device ide-cd,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -spice port=3000,password=123456,addr=0,tls-port=3200,x509-dir=/tmp/spice_x509d,tls-channel=main,tls-channel=inputs,image-compression=auto_glz,zlib-glz-wan-compression=auto,streaming-video=all,agent-mouse=on,playback-compression=on,ipv4 \ -rtc base=localtime,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off \ -enable-kvm 2.{'execute': 'blockdev-snapshot-sync', 'arguments': {'device': u'drive_image1', 'snapshot-file': '/home/autotest-devel/client/tests/virt/shared/data/images/sn1', 'format': 'qcow2'}, 'id': 'UdasHoGF'} 3.start to stream block device {'execute': 'block-stream', 'arguments': {'device': u'drive_image1', 'speed': 1048576}, 'id': 'RKxrPfkM'} 4.{"execute": "block-job-set-speed", "arguments": {"device": "drive_image1", "speed": 10485760}, "id": "m7HB8euI"} 5.{'execute': 'query-block-jobs', 'id': 'NaHuYYlt'} when block stream job is done. 6.reboot guest Actual results: [qemu output] main-loop: WARNING: I/O thread spun for 1000 iterations guest reboot and can't get IP address. Expected results: [qemu output] no warning guest reboot in normal status. Additional info: