Bug 710176
Summary: | [qemu-kvm] no enospace event when allocated space is consumed (Invalid argument (22) ) | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 6 | Reporter: | Haim <hateya> | ||||||
Component: | qemu-kvm | Assignee: | Kevin Wolf <kwolf> | ||||||
Status: | CLOSED WORKSFORME | QA Contact: | Virtualization Bugs <virt-bugs> | ||||||
Severity: | high | Docs Contact: | |||||||
Priority: | unspecified | ||||||||
Version: | 6.2 | CC: | chayang, ewarszaw, gcosta, juzhang, mgoldboi, mkenneth, tburke, virt-maint, yeylon, ykaul | ||||||
Target Milestone: | rc | ||||||||
Target Release: | --- | ||||||||
Hardware: | x86_64 | ||||||||
OS: | Linux | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2011-09-27 14:28:26 UTC | Type: | --- | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Attachments: |
|
Hi Kevin, I tried with qemu-kvm-0.12.1.2-2.164.el6.x86_64, but cannot reproduce issue mentioned in Comment #0, could you tell me what I am missing? CLI I use and error info I hit will be attached. Thanks, Created attachment 504408 [details]
CLI and error info
--snip of chayang test results logs-- 1. qemu control log (qemu) block I/O error in device 'drive-virtio0-0-0': No space left on device (28) block I/O error in device 'drive-virtio0-0-0': No space left on device (28) 2. qmp log {"timestamp": {"seconds": 1307956429, "microseconds": 514768}, "event": "STOP"} {"timestamp": {"seconds": 1307956429, "microseconds": 515037}, "event": "BLOCK_IO_ERROR", "data": {"device": "drive-virtio0-0-0", "__com.redhat_debug_info": {"message": "No space left on device", "errno": 28}, "__com.redhat_reason": "enospc", "operation": "write", "action": "stop"}} Hi,Haim Need to confirm to you, you report this issue because qemu-kvm report wrong errors, should be report enospc error,however,report eother error,right? any mistake,please fix me. According to chayangs' test, qemu-kvm can report right error message. Best Regards, Junyi I can't reproduce this either. Haim, is this reproducible? And if it is, is it reproducible even without VDSM? Do we have any log of the QMP traffic? I would like to see the real error reason, "eother" is rather unspecific, but unfortunately the VDSM log doesn't contain the QMP event that qemu sends. Tried again using CLI in Comment #3, this time with a 1G block, increase by 100M when hit ENOSPAC, still cannot reproduce even have increased to 5G. (In reply to comment #6) > I can't reproduce this either. > > Haim, is this reproducible? And if it is, is it reproducible even without VDSM? > > Do we have any log of the QMP traffic? I would like to see the real error > reason, "eother" is rather unspecific, but unfortunately the VDSM log doesn't > contain the QMP event that qemu sends. Kevin, hit it again using rhel6 server as guest. doing lots of sequential write, I happen to receive abnormal vm stop on eother. block I/O error in device 'drive-virtio-disk1': Invalid argument (22) handle_dev_input: stop block I/O error in device 'drive-virtio-disk1': Invalid argument (22) block I/O error in device 'drive-virtio-disk1': Invalid argument (22) block I/O error in device 'drive-virtio-disk1': Invalid argument (22) block I/O error in device 'drive-virtio-disk1': Invalid argument (22 qemu-kvm-0.12.1.2-2.166.el6.x86_64 I still can't reproduce it. I tried: 1. launch a guest directly using qemu-kvm, tested more than 10 times, don't find any eother event through QMP but only enospc event. 2. use RHEV-M to launch a RHEL6.0 guest, attach a qcow2 image which is thin-provisioned, 20G, allocated space, 1G. Tested for more than 6 times, sequential write to attached disk doesn't trigger this issue for me, don't see any eother event in vdsm log, attached disk size increases when allocated space is consumed. Guest is running well. My testing environment: # uname -r 2.6.32-168.el6.x86_64 # rpm -q qemu-kvm qemu-kvm-0.12.1.2-2.171.el6.x86_64 # rpm -q vdsm vdsm-4.9-79.el6.x86_64 # rpm -q libvirt libvirt-0.9.1-1.el6.x86_64 # lvscan ACTIVE '/dev/a38a0e73-cd4a-4a52-866b-fb7a10a248b7/e940502b-107a-4bee-89c8-8e679a904c8b' [4.00 GiB] inherit ACTIVE '/dev/a38a0e73-cd4a-4a52-866b-fb7a10a248b7/fc46a6ea-54d4-4454-b245-c9af56f39e6f' [21.00 GiB] inherit <--------this thin-provisioned attached disk size has been increased to 21G when allocated space is consumed Hi Haim, Do I have any chance to access your setup to reproduce this issue? If yes, could you let me know if the setup is available? Thanks, Haim, if we don't get more info very soon (development freeze is this week), I'll have to move this to 6.3. |
Created attachment 502570 [details] vdsm log Description of problem: no enospace event produced by qemu when allocated space is consumed on qcow disk. host - qemu-kvm-0.12.1.2-2.162.el6.x86_64 libvirt-0.9.1-1.el6.x86_64 kernel-2.6.32-152.el6.x86_64 scenario: booting from live CD, starting dd'ing disk (thin-provisioned, 20G, allocated space, 1G). on VDSM logs I don't see any 'enospace' event, but eother, since desired event is not fired, there is no lv extend, which leaves machine is pause state. also see the following errors: LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/root USER=root LOGNAME=root QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.0.0 -cpu Conroe -enable-kvm -m 3072 -smp 1,sockets=1,cores=1,threads=1 -name LVEX1 -uuid 2d201a12-adc5-4883-a00b-f33fcea80ff3 -smbios type=1,manufacturer=Red Hat,product=RHEL,version=6Server-6.1.0.1.el6,serial=38373035-3536-4247-3830-333334344130_78:E7:D1:E4:8C:58,uuid=2d201a12-adc5-4883-a00b-f33fcea80ff3 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/LVEX1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2011-06-02T14:20:55 -boot d -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x4 -drive file=/rhev/data-center/98a7b314-9395-4ab9-8c89-9186d4600d54/0959db5e-eb92-4b10-a476-2b983036eeb2/images/d0a50c68-d63b-4834-817e-60b5e646ba31/07a4073a-5408-43e8-9ff3-7b37c5a31bef,if=none,id=drive-virtio-disk0,format=qcow2,serial=34-817e-60b5e646ba31,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/rhev/data-center/98a7b314-9395-4ab9-8c89-9186d4600d54/0959db5e-eb92-4b10-a476-2b983036eeb2/images/3d6da9b8-8626-418d-957a-b5f9c420f64b/89a29e08-0422-4868-9b38-65279d504d2b,if=none,id=drive-ide0-0-0,format=qcow2,serial=8d-957a-b5f9c420f64b,cache=none,werror=stop,rerror=stop,aio=native -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/rhev/data-center/98a7b314-9395-4ab9-8c89-9186d4600d54/0959db5e-eb92-4b10-a476-2b983036eeb2/images/e8dcfd12-ab56-4172-a062-78cafb53c8b3/a839be2e-45a3-4bf4-b6c3-03a111316254,if=none,id=drive-virtio-disk1,format=qcow2,serial=72-a062-78cafb53c8b3,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/rhev/data-center/98a7b314-9395-4ab9-8c89-9186d4600d54/47172803-d3ee-421b-9f47-764d2580bd4e/images/11111111-1111-1111-1111-111111111111/Fedora-13-i686-Live.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:87:20,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/LVEX1.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -usb -device usb-tablet,id=input0 -vnc 0:0,password -k en-us -vga cirrus Using CPU model "Conroe" block I/O error in device 'drive-virtio-disk1': Invalid argument (22) block I/O error in device 'drive-virtio-disk1': Invalid argument (22) block I/O error in device 'drive-virtio-disk1': Invalid argument (22) block I/O error in device 'drive-virtio-disk1': Invalid argument (22) libvirtEventLoop::INFO::2011-06-02 17:25:27,549::libvirtvm::1067::vm.Vm::(_onAbnormalStop) vmId=`2d201a12-adc5-4883-a00b-f33fcea80ff3`::abnormal vm stop device virtio-disk1 error eother libvirtEventLoop::DEBUG::2011-06-02 17:25:27,550::libvirtvm::1230::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`2d201a12-adc5-4883-a00b-f33fcea80ff3`::event Suspend ed detail 2 opaque None libvirtEventLoop::INFO::2011-06-02 17:25:27,552::libvirtvm::1067::vm.Vm::(_onAbnormalStop) vmId=`2d201a12-adc5-4883-a00b-f33fcea80ff3`::abnormal vm stop device virtio-disk1 error eother libvirtEventLoop::INFO::2011-06-02 17:25:27,553::libvirtvm::1067::vm.Vm::(_onAbnormalStop) vmId=`2d201a12-adc5-4883-a00b-f33fcea80ff3`::abnormal vm stop device virtio-disk1 error eother libvirtEventLoop::INFO::2011-06-02 17:25:27,553::libvirtvm::1067::vm.Vm::(_onAbnormalStop) vmId=`2d201a12-adc5-4883-a00b-f33fcea80ff3`::abnormal vm stop device virtio-disk1 error eother Thread-2144::INFO::2011-06-02 17:25:27,590::vm::533::vm.Vm::(_onHighWrite) vmId=`2d201a12-adc5-4883-a00b-f33fcea80ff3`::_onHighWrite: write above watermark on hdb offset 1074994688