Bug 1168213 - main-loop: WARNING: I/O thread spun for 1000 iterations while doing stream block device.
Summary: main-loop: WARNING: I/O thread spun for 1000 iterations while doing stream bl...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.1
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Virtualization Maintenance
QA Contact: Gu Nini
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-11-26 12:17 UTC by Yang Xue
Modified: 2018-11-01 11:01 UTC (History)
11 users (show)

Fixed In Version: qemu-kvm-rhev-2.12.0-5.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-01 11:01:10 UTC
Target Upstream Version:


Attachments (Terms of Use)
iometer.icf (9.01 KB, text/plain)
2016-05-12 10:27 UTC, Gu Nini
no flags Details

Description Yang Xue 2014-11-26 12:17:40 UTC
Description of problem:
win2012r2 guest main-loop: WARNING: I/O thread spun for 1000 iterations ,while doing stream block device.

Version-Release number of selected component (if applicable):
Host:
kernel-3.10.0-205.el7.x86_64
qemu-kvm-rhev-2.1.2-10.el7.x86_64

How reproducible:
50%

Steps to Reproduce:
1.start win2012r2 guest
/bin/qemu-kvm \
    -S  \
    -name 'virt-tests-vm1'  \
    -sandbox off  \
    -M pc  \
    -nodefaults  \
    -vga qxl  \
    -global qxl-vga.vram_size=33554432 \
    -device intel-hda,bus=pci.0,addr=03 \
    -device hda-duplex  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20141122-031435-M0GYyrum,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20141122-031435-M0GYyrum,server,nowait \
    -device isa-serial,chardev=serial_id_serial0 \
    -device virtio-serial-pci,id=virtio_serial_pci0,bus=pci.0,addr=04  \
    -chardev socket,id=devvs,path=/tmp/virtio_port-vs-20141122-031435-M0GYyrum,server,nowait \
    -device virtserialport,chardev=devvs,name=vs,id=vs,bus=virtio_serial_pci0.0  \
    -chardev socket,id=seabioslog_id_20141122-031435-M0GYyrum,path=/tmp/seabios-20141122-031435-M0GYyrum,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20141122-031435-M0GYyrum,iobase=0x402 \
    -device nec-usb-xhci,id=usb1,bus=pci.0,addr=05 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=06 \
    -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,file=/home/autotest-devel/client/tests/virt/shared/data/images/win2012-64r2-virtio.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device virtio-net-pci,mac=9a:8e:8f:90:91:92,id=idxToyso,vectors=4,netdev=idU9oayT,bus=pci.0,addr=07  \
    -netdev tap,id=idU9oayT,vhost=on,vhostfd=23,fd=22  \
    -m 65536  \
    -smp 8,maxcpus=8,cores=4,threads=1,sockets=2  \
    -cpu 'Penryn',+kvm_pv_unhalt,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \
    -drive id=drive_cd1,if=none,snapshot=off,aio=native,media=cdrom,file=/home/autotest-devel/client/tests/virt/shared/data/isos/windows/winutils.iso \
    -device ide-cd,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -spice port=3000,password=123456,addr=0,tls-port=3200,x509-dir=/tmp/spice_x509d,tls-channel=main,tls-channel=inputs,image-compression=auto_glz,zlib-glz-wan-compression=auto,streaming-video=all,agent-mouse=on,playback-compression=on,ipv4  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off \
    -enable-kvm
2.{'execute': 'blockdev-snapshot-sync', 'arguments': {'device': u'drive_image1', 'snapshot-file': '/home/autotest-devel/client/tests/virt/shared/data/images/sn1', 'format': 'qcow2'}, 'id': 'UdasHoGF'}
3.start to stream block device
{'execute': 'block-stream', 'arguments': {'device': u'drive_image1', 'speed': 1048576}, 'id': 'RKxrPfkM'}
4.{"execute": "block-job-set-speed", "arguments": {"device": "drive_image1", "speed": 10485760}, "id": "m7HB8euI"}
5.{'execute': 'query-block-jobs', 'id': 'NaHuYYlt'}
when block stream job is done.
6.reboot guest

Actual results:
[qemu output] main-loop: WARNING: I/O thread spun for 1000 iterations
guest reboot and can't get IP address.

Expected results:
[qemu output] no warning 
guest reboot in normal status.

Additional info:

Comment 2 Yang Xue 2014-11-27 02:05:50 UTC
Reproduce this problem again and 100% reproducible. 
This problem causes the vcpu not running.

Comment 3 Joy Pu 2014-12-10 08:50:57 UTC
Met this in ppc host with ppc64le guest too:
qemu-kvm related package:
ipxe-roms-qemu-20130517-6.gitc4bce43.el7.noarch
qemu-img-rhev-2.1.2-14.el7.ppc64
qemu-kvm-tools-rhev-2.1.2-14.el7.ppc64
libvirt-daemon-driver-qemu-1.2.8-9.el7.ppc64
qemu-kvm-common-rhev-2.1.2-14.el7.ppc64
qemu-kvm-rhev-2.1.2-14.el7.ppc64
qemu-kvm-rhev-debuginfo-2.1.2-14.el7.ppc64

host kernel verion:3.10.0-201.el7.ppc64
guest kernel version: 3.10.0-210

Comment 4 Joy Pu 2014-12-10 08:57:19 UTC
This problem in ppc host have some different results:

Met this once the qemu process is terminated after 8 minutes that this message shows up.
main-loop: WARNING: I/O thread spun for 1000 iterations
qemu: terminating on signal 1

And normally it will blocked the guest boot up normally after the block stream guest shut down.

Comment 8 Gu Nini 2016-05-12 10:27:23 UTC
Created attachment 1156629 [details]
iometer.icf

Found the same log in the bug on following sw versions with Win10.x86_64 guest started with aio=native disks; while for the same guest started with aio=threads disks, there is no the problem.

Host kernel: 3.10.0-370.el7.x86_64
Qemu-kvm: qemu-kvm-rhev-2.5.0-4.el7.x86_64
virtio-win: virtio-win-1.8.0-4


Steps:

1) Start an installed Win10.x86_64 guest with following qemu command:

/usr/libexec/qemu-kvm \
     -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults  \
    -vga std  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_1,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_2,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idZelHT0  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/avocado_3,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20160511-031111-dD14QDp6,path=/var/tmp/avocado_4,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20160511-031111-dD14QDp6,iobase=0x402 \
    -device ich9-usb-ehci1,id=usb1,addr=1d.7,multifunction=on,bus=pci.0 \
    -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=1d.0,firstport=0,bus=pci.0 \
    -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=1d.2,firstport=2,bus=pci.0 \
    -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=1d.4,firstport=4,bus=pci.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=03,disable-legacy=off,disable-modern=on \
    -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,file=/usr/share/avocado/data/avocado-vt/images/win10-64-virtio-scsi.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -drive id=drive_disk1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,file=/usr/share/avocado/data/avocado-vt/images/storage.qcow2 \
    -device scsi-hd,id=disk1,drive=drive_disk1 \
    -device virtio-net-pci,mac=9a:1a:1b:1c:1d:1e,id=idTv7QY9,vectors=4,netdev=idgrlUJn,bus=pci.0,addr=04,disable-legacy=off,disable-modern=on  \
    -netdev tap,id=idgrlUJn,vhost=on \
    -m 16384  \
    -smp 8,maxcpus=8,cores=4,threads=1,sockets=2  \
    -cpu 'Westmere',+kvm_pv_unhalt,hv_spinlocks=0x1fff,hv_vapic,hv_time \
    -drive id=drive_cd1,if=none,snapshot=off,aio=native,media=cdrom,file=/usr/share/avocado/data/avocado-vt/isos/windows/winutils.iso \
    -device scsi-cd,id=cd1,drive=drive_cd1 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -monitor stdio

2). After the guest boots up, run iometer read/write operation on data drive drive_disk1 inside the guest with iometer.icf(attached).


Results:
Log 'main-loop: WARNING: I/O thread spun for 1000 iterations' was sent out in hmp, the guest hang AND the cpu usage rate of qemu-kvm process would increase to 300% in the host.

Comment 9 Gu Nini 2016-05-12 10:33:16 UTC
Jeff,

Would you help to check if comment #8 is caused for the same reason of the bug? If not, I will report a new one.

BTW, there is some discussion on the same log at https://bugs.launchpad.net/qemu/+bug/1481272

Comment 10 Gu Nini 2016-05-13 05:55:50 UTC
(In reply to Gu Nini from comment #9)
> Jeff,
> 
> Would you help to check if comment #8 is caused for the same reason of the
> bug? If not, I will report a new one.
> 
> BTW, there is some discussion on the same log at
> https://bugs.launchpad.net/qemu/+bug/1481272

Could not reproduce comment #8 problem with following newest 7.3 sw:

Host Kernel: 3.10.0-396.el7.x86_64
Qemu-kvm-rhev: qemu-kvm-rhev-2.6.0-1.el7.x86_64
virtio-win: virtio-win-1.8.0-4

So please ignore the needinfo in comment #9.

Comment 12 Longxiang Lyu 2017-07-28 08:07:05 UTC
Could reproduce the warning message with the following version selected:
Host:
kernel-3.10.0-693.el7.x86_64
qemu-kvm-rhev-2.9.0-17.el7a.x86_64

Guest:
kernel-3.10.0-693.el7.x86_64

Comment 13 Jeff Cody 2018-05-30 20:20:18 UTC
Commit upstream from Stefan that addresses this BZ:  https://lists.gnu.org/archive/html/qemu-devel/2018-05/msg06889.html


However, the dependency for this commit is non-trivial; it depends on changes made to thread lock optimizations for the BQL, in Pavel Dovgalyuk's "replay additions" patch series.  This is probably too invasive to backport into 7.6, unless we just want to suppress the output anyways: https://lists.gnu.org/archive/html/qemu-devel/2018-02/msg06572.html

Comment 14 Jeff Cody 2018-06-13 16:21:47 UTC
Submitted backport to go ahead and suppress the output anyway; regardless of the BQL, it has never been a very useful debug message and produces a lot of spurious noise.

Comment 16 Miroslav Rezanina 2018-06-25 14:01:02 UTC
Fix included in qemu-kvm-rhev-2.12.0-5.el7

Comment 18 Gu Nini 2018-07-02 04:00:02 UTC
Reproduce the bug on 'qemu-kvm-rhev-2.12.0-4.el7.x86_64' with the same steps as that in the bz description.

Verify the bug on 'qemu-kvm-rhev-2.12.0-5.el7.x86_64', i.e. no 'main-loop: WARNING: I/O thread spun for 1000 iterations' is sent out from hmp.

Comment 21 errata-xmlrpc 2018-11-01 11:01:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3443


Note You need to log in before you can comment on or make changes to this bug.