Bug 880557 - Guest failed to reboot when throttling system disk as iops_wr=100
Guest failed to reboot when throttling system disk as iops_wr=100
Status: CLOSED WORKSFORME
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
7.0
Unspecified Unspecified
medium Severity medium
: rc
: ---
Assigned To: Fam Zheng
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-11-27 05:22 EST by juzhang
Modified: 2014-02-21 10:27 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-02-21 10:27:45 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
screenshot1 (42.41 KB, image/png)
2012-11-27 05:24 EST, juzhang
no flags Details
screenshot2 (43.67 KB, image/png)
2012-11-27 05:25 EST, juzhang
no flags Details

  None (edit)
Description juzhang 2012-11-27 05:22:26 EST
Description of problem:
Throttling system disk as iops_wr=100, the boot guest. Guest can be booted successfully, then reboot(reboot in guest or issue system_reset) the guest. Then the guest hang  in the loop  

Version-Release number of selected component (if applicable):
qemu-kvm-1.2.0-20.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Boot guest and Throttling system disk as iops_wr=100
#/usr/libexec/qemu-kvm -cpu Opteron_G3 -m 2048 -smp 2,sockets=1,cores=2,threads=1 -enable-kvm -name rhel64 -smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 -k en-us -rtc base=localtime,clock=host,driftfix=slew  -drive file=/root/zhangjunyi/cdrom.qcow2,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=3 -monitor stdio -qmp tcp:0:6666,server,nowait -boot menu=on -bios /usr/share/seabios/bios.bin -drive file=/home/rhel6u4_mazhang.qcow2,if=none,id=drive-scsi-disk,format=qcow2,cache=none,werror=stop,rerror=stop,iops_wr=100 -device virtio-scsi-pci,id=scsi0,addr=0x5 -device scsi-disk,drive=drive-scsi-disk,bus=scsi0.0,scsi-id=0,lun=0,id=scsi-disk,bootindex=1 -netdev tap,id=hostnet0,downscript=no -device e1000,netdev=hostnet0,id=net0,mac=00:1a:4a:2e:28:1a,bus=pci.0,addr=0x4,bootindex=2 -chardev socket,path=/tmp/isa-serial,server,nowait,id=isa1 -device isa-serial,chardev=isa1,id=isa-serial1 -vnc :10 -balloon virtio -smbios type=0,vendor=DW,version=0.1,date=2011-11-22,release=0.1 -drive file=/root/zhangjunyi/floopy.qcow2,if=none,id=drive-fdc0-0-0,format=qcow2,cache=none -global isa-fdc.driveA=drive-fdc0-0-0 -drive file=/root/zhangjunyi/cdrom_scsi.qcow2,if=none,media=cdrom,readonly=on,format=qcow2,id=cdrom1 -device scsi-cd,bus=scsi0.0,drive=cdrom1,id=scsi0-0 -device usb-ehci,id=ehci -drive file=/root/zhangjunyi/usb.qcow2,if=none,id=drive-usb-2-0,media=disk,format=qcow2,cache=none -device usb-storage,drive=drive-usb-2-0,id=usb-0-0,removable=on,bus=ehci.0,port=1 -drive file=/root/zhangjunyi/virtio.qcow2,if=none,id=block-ide,format=qcow2,werror=stop,rerror=stop,cache=none -device ide-drive,drive=block-ide,id=block-ide -drive file=/root/zhangjunyi/virtio.qcow2,format=qcow2,if=none,id=block-virtio,cache=none,werror=stop,rerror=stop  -device virtio-blk-pci,bus=pci.0,addr=0x8,drive=block-virtio,id=block-virtio -device sga -chardev socket,id=serial0,path=/var/test1,server,nowait -device isa-serial,chardev=serial0
2.reboot the guest
3.disable the throttling
#(qemu) block_set_io_throttle drive-scsi-disk 0 0 0 0 0 0
  
Actual results:
After step2, guest hang at "From "Press F12 to.." to "booting Readhat .. in 3 seconds" in the loop.

After step3, guest hang at "From "Press F12 to.." to "booting Readhat .. in 3 seconds" in the loop.

Expected results:
1. Guest works well after step2, Since most of iops should be read io during the booting guest, I think set iops_wr=100 should be enough for booting the guest.
2. At least, guest works well after step3.

Additional info:
Comment 1 juzhang 2012-11-27 05:24:12 EST
Created attachment 652565 [details]
screenshot1
Comment 2 juzhang 2012-11-27 05:25:12 EST
Created attachment 652566 [details]
screenshot2

Guest hang between screenshot1 and screenshot2 in the loop.
Comment 3 Stefan Hajnoczi 2013-03-26 10:37:33 EDT
Unable to reproduce on qemu-kvm-1.2.0-17.el7 and rhel7/master (60e3fdc327c6ee289faae766d282b3ea7a542c7e).

Do you have a link to RPMs that reproduce the bug for you?

I had to cut down the command-line because there were many CD-ROMs and disks (I didn't have enough image files laying around).  Can you reproduce it with a shorter command-line?

Here is my command-line:

/usr/libexec/qemu-kvm -cpu host -m 2048 -smp 2,sockets=1,cores=2,threads=1 -enable-kvm -name rhel64 -smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 -k en-us -rtc base=localtime,clock=host,driftfix=slew  -drive file=path/to/an.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=3 -monitor stdio -qmp tcp:0:6666,server,nowait -boot menu=on -drive file=rhel6u4.qcow2,if=none,id=drive-scsi-disk,format=qcow2,cache=none,werror=stop,rerror=stop,iops_wr=100 -device virtio-scsi-pci,id=scsi0,addr=0x5 -device scsi-disk,drive=drive-scsi-disk,bus=scsi0.0,scsi-id=0,lun=0,id=scsi-disk,bootindex=1 -netdev user,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=00:1a:4a:2e:28:1a,bus=pci.0,addr=0x4,bootindex=2 -chardev socket,path=/tmp/isa-serial,server,nowait,id=isa1 -device isa-serial,chardev=isa1,id=isa-serial1 -vnc :10 -balloon virtio -smbios type=0,vendor=DW,version=0.1,date=2011-11-22,release=0.1 -device usb-ehci,id=ehci -device sga -chardev socket,id=serial0,path=/tmp/test1,server,nowait -device isa-serial,chardev=serial0
Comment 4 FuXiangChun 2013-04-01 05:01:07 EDT
(In reply to comment #3)
> Unable to reproduce on qemu-kvm-1.2.0-17.el7 and rhel7/master
> (60e3fdc327c6ee289faae766d282b3ea7a542c7e).
> 
> Do you have a link to RPMs that reproduce the bug for you?
> 
> I had to cut down the command-line because there were many CD-ROMs and disks
> (I didn't have enough image files laying around).  Can you reproduce it with
> a shorter command-line?
> 
> Here is my command-line:
> 
> /usr/libexec/qemu-kvm -cpu host -m 2048 -smp 2,sockets=1,cores=2,threads=1
> -enable-kvm -name rhel64 -smbios type=1,manufacturer='Red Hat',product='RHEV
> Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-
> e63fcdb67170 -k en-us -rtc base=localtime,clock=host,driftfix=slew  -drive
> file=path/to/an.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,
> format=raw -device
> ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=3
> -monitor stdio -qmp tcp:0:6666,server,nowait -boot menu=on -drive
> file=rhel6u4.qcow2,if=none,id=drive-scsi-disk,format=qcow2,cache=none,
> werror=stop,rerror=stop,iops_wr=100 -device
> virtio-scsi-pci,id=scsi0,addr=0x5 -device
> scsi-disk,drive=drive-scsi-disk,bus=scsi0.0,scsi-id=0,lun=0,id=scsi-disk,
> bootindex=1 -netdev user,id=hostnet0 -device
> e1000,netdev=hostnet0,id=net0,mac=00:1a:4a:2e:28:1a,bus=pci.0,addr=0x4,
> bootindex=2 -chardev socket,path=/tmp/isa-serial,server,nowait,id=isa1
> -device isa-serial,chardev=isa1,id=isa-serial1 -vnc :10 -balloon virtio
> -smbios type=0,vendor=DW,version=0.1,date=2011-11-22,release=0.1 -device
> usb-ehci,id=ehci -device sga -chardev
> socket,id=serial0,path=/tmp/test1,server,nowait -device
> isa-serial,chardev=serial0

Stefan,
Can reproduce this issue with your command line.
steps:
1.boot rhel6.4-64 guest
/usr/libexec/qemu-kvm -cpu host -m 2048 -smp 2,sockets=1,cores=2,threads=1 -enable-kvm -name rhel64 -smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 -k en-us -rtc base=localtime,clock=host,driftfix=slew  -drive file=an.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=3 -monitor stdio -qmp tcp:0:6666,server,nowait -boot menu=on -drive file=/home/RHEL-Server-6.4-64-2.qcow2,if=none,id=drive-scsi-disk,format=qcow2,cache=none,werror=stop,rerror=stop,iops_wr=100 -device virtio-scsi-pci,id=scsi0,addr=0x5 -device scsi-disk,drive=drive-scsi-disk,bus=scsi0.0,scsi-id=0,lun=0,id=scsi-disk,bootindex=1 -netdev user,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=00:1a:4a:2e:28:1a,bus=pci.0,addr=0x4,bootindex=2 -chardev socket,path=/tmp/isa-serial,server,nowait,id=isa1 -device isa-serial,chardev=isa1,id=isa-serial1 -vnc :10 -balloon virtio -smbios type=0,vendor=DW,version=0.1,date=2011-11-22,release=0.1 -device usb-ehci,id=ehci -device sga -chardev socket,id=serial0,path=/tmp/test1,server,nowait -device isa-serial,chardev=serial0

2. reboot guest
Comment 5 Fam Zheng 2014-02-19 09:17:12 EST
When the guest is hung, could you see any change on block statistics (QMP query-blockstats)? (i.e. are the IO statistic numbers still going up?)

Thanks,
Fam
Comment 6 juzhang 2014-02-19 20:53:41 EST
Hi Xiangchun,

Could you add your comment since you tested this scenario in comment4?

Best Regards,
Junyi
Comment 7 FuXiangChun 2014-02-20 21:15:21 EST
Re-tested this bug with qemu-kvm-rhev-1.5.3-47.el7.x86_64 and kernel 3.10.0-85.el7.x86_64. and tested two linux guests RHEL6.4 and RHEL6.5.

According to comment 0 & comment 4. Both can not reproduce this bug. This problem goes away. I tested multiple times on AMD and Intel host. Guest works well.
Comment 8 Fam Zheng 2014-02-21 10:27:45 EST
Since this cannot be reproduced, close as WORKSFORME.

Note You need to log in before you can comment on or make changes to this bug.