RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 710176 - [qemu-kvm] no enospace event when allocated space is consumed (Invalid argument (22) )
Summary: [qemu-kvm] no enospace event when allocated space is consumed (Invalid argume...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm
Version: 6.2
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: Kevin Wolf
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-06-02 15:33 UTC by Haim
Modified: 2018-12-04 14:25 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-09-27 14:28:26 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
vdsm log (1.25 MB, application/x-gzip)
2011-06-02 15:33 UTC, Haim 		no flags Details
CLI and error info (4.60 KB, text/plain)
2011-06-13 10:10 UTC, Chao Yang 		no flags Details
View All

Description Haim 2011-06-02 15:33:00 UTC 
Created attachment 502570 [details]
vdsm log

Description of problem:

no enospace event produced by qemu when allocated space is consumed on qcow disk.

host - qemu-kvm-0.12.1.2-2.162.el6.x86_64 
       libvirt-0.9.1-1.el6.x86_64 
       kernel-2.6.32-152.el6.x86_64
       
scenario: 

booting from live CD, starting dd'ing disk (thin-provisioned, 20G, allocated space, 1G). 
on VDSM logs I don't see any 'enospace' event, but eother, since desired event is not fired, there is no lv extend, which leaves machine is pause state. 
also see the following errors: 

LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/root USER=root LOGNAME=root QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.0.0 -cpu Conroe -enable-kvm -m 3072 -smp 1,sockets=1,cores=1,threads=1 -name LVEX1 -uuid 2d201a12-adc5-4883-a00b-f33fcea80ff3 -smbios type=1,manufacturer=Red Hat,product=RHEL,version=6Server-6.1.0.1.el6,serial=38373035-3536-4247-3830-333334344130_78:E7:D1:E4:8C:58,uuid=2d201a12-adc5-4883-a00b-f33fcea80ff3 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/LVEX1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2011-06-02T14:20:55 -boot d -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x4 -drive file=/rhev/data-center/98a7b314-9395-4ab9-8c89-9186d4600d54/0959db5e-eb92-4b10-a476-2b983036eeb2/images/d0a50c68-d63b-4834-817e-60b5e646ba31/07a4073a-5408-43e8-9ff3-7b37c5a31bef,if=none,id=drive-virtio-disk0,format=qcow2,serial=34-817e-60b5e646ba31,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/rhev/data-center/98a7b314-9395-4ab9-8c89-9186d4600d54/0959db5e-eb92-4b10-a476-2b983036eeb2/images/3d6da9b8-8626-418d-957a-b5f9c420f64b/89a29e08-0422-4868-9b38-65279d504d2b,if=none,id=drive-ide0-0-0,format=qcow2,serial=8d-957a-b5f9c420f64b,cache=none,werror=stop,rerror=stop,aio=native -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/rhev/data-center/98a7b314-9395-4ab9-8c89-9186d4600d54/0959db5e-eb92-4b10-a476-2b983036eeb2/images/e8dcfd12-ab56-4172-a062-78cafb53c8b3/a839be2e-45a3-4bf4-b6c3-03a111316254,if=none,id=drive-virtio-disk1,format=qcow2,serial=72-a062-78cafb53c8b3,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/rhev/data-center/98a7b314-9395-4ab9-8c89-9186d4600d54/47172803-d3ee-421b-9f47-764d2580bd4e/images/11111111-1111-1111-1111-111111111111/Fedora-13-i686-Live.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:87:20,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/LVEX1.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -usb -device usb-tablet,id=input0 -vnc 0:0,password -k en-us -vga cirrus
Using CPU model "Conroe"
block I/O error in device 'drive-virtio-disk1': Invalid argument (22)
block I/O error in device 'drive-virtio-disk1': Invalid argument (22)
block I/O error in device 'drive-virtio-disk1': Invalid argument (22)
block I/O error in device 'drive-virtio-disk1': Invalid argument (22)


libvirtEventLoop::INFO::2011-06-02 17:25:27,549::libvirtvm::1067::vm.Vm::(_onAbnormalStop) vmId=`2d201a12-adc5-4883-a00b-f33fcea80ff3`::abnormal vm stop device
 virtio-disk1 error eother
libvirtEventLoop::DEBUG::2011-06-02 17:25:27,550::libvirtvm::1230::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`2d201a12-adc5-4883-a00b-f33fcea80ff3`::event Suspend
ed detail 2 opaque None
libvirtEventLoop::INFO::2011-06-02 17:25:27,552::libvirtvm::1067::vm.Vm::(_onAbnormalStop) vmId=`2d201a12-adc5-4883-a00b-f33fcea80ff3`::abnormal vm stop device
 virtio-disk1 error eother
libvirtEventLoop::INFO::2011-06-02 17:25:27,553::libvirtvm::1067::vm.Vm::(_onAbnormalStop) vmId=`2d201a12-adc5-4883-a00b-f33fcea80ff3`::abnormal vm stop device virtio-disk1 error eother
libvirtEventLoop::INFO::2011-06-02 17:25:27,553::libvirtvm::1067::vm.Vm::(_onAbnormalStop) vmId=`2d201a12-adc5-4883-a00b-f33fcea80ff3`::abnormal vm stop device virtio-disk1 error eother
Thread-2144::INFO::2011-06-02 17:25:27,590::vm::533::vm.Vm::(_onHighWrite) vmId=`2d201a12-adc5-4883-a00b-f33fcea80ff3`::_onHighWrite: write above watermark on hdb offset 1074994688
Comment 3 Chao Yang 2011-06-13 10:06:07 UTC 
Hi Kevin,
 I tried with qemu-kvm-0.12.1.2-2.164.el6.x86_64, but cannot reproduce issue mentioned in Comment #0, could you tell me what I am missing？ CLI I use and error info I hit will be attached.

 Thanks,
Comment 4 Chao Yang 2011-06-13 10:10:03 UTC 
Created attachment 504408 [details]
CLI and error info
Comment 5 juzhang 2011-06-14 01:30:18 UTC 
--snip of chayang test results logs--

1. qemu control log
(qemu) block I/O error in device 'drive-virtio0-0-0': No space left on device (28)
block I/O error in device 'drive-virtio0-0-0': No space left on device (28)

2. qmp log
{"timestamp": {"seconds": 1307956429, "microseconds": 514768}, "event": "STOP"}
{"timestamp": {"seconds": 1307956429, "microseconds": 515037}, "event": "BLOCK_IO_ERROR", "data": {"device": "drive-virtio0-0-0", "__com.redhat_debug_info": {"message": "No space left on device", "errno": 28}, "__com.redhat_reason": "enospc", "operation": "write", "action": "stop"}}

Hi,Haim 

   Need to confirm to you, you report this issue because qemu-kvm report wrong errors, should be report enospc error,however,report eother error,right? any mistake,please fix me. According to chayangs' test, qemu-kvm can report right error message.

Best Regards,
Junyi
Comment 6 Kevin Wolf 2011-06-14 09:41:41 UTC 
I can't reproduce this either.

Haim, is this reproducible? And if it is, is it reproducible even without VDSM?

Do we have any log of the QMP traffic? I would like to see the real error reason, "eother" is rather unspecific, but unfortunately the VDSM log doesn't contain the QMP event that qemu sends.
Comment 7 Chao Yang 2011-06-20 04:46:06 UTC 
Tried again using CLI in Comment #3, this time with a 1G block, increase by 100M when hit ENOSPAC, still cannot reproduce even have increased to 5G.
Comment 8 Haim 2011-07-06 14:02:27 UTC 
(In reply to comment #6)
> I can't reproduce this either.
> 
> Haim, is this reproducible? And if it is, is it reproducible even without VDSM?
> 
> Do we have any log of the QMP traffic? I would like to see the real error
> reason, "eother" is rather unspecific, but unfortunately the VDSM log doesn't
> contain the QMP event that qemu sends.

Kevin, hit it again using rhel6 server as guest. doing lots of sequential write, I happen to receive abnormal vm stop on eother.  

block I/O error in device 'drive-virtio-disk1': Invalid argument (22)
handle_dev_input: stop
block I/O error in device 'drive-virtio-disk1': Invalid argument (22)
block I/O error in device 'drive-virtio-disk1': Invalid argument (22)
block I/O error in device 'drive-virtio-disk1': Invalid argument (22)
block I/O error in device 'drive-virtio-disk1': Invalid argument (22


qemu-kvm-0.12.1.2-2.166.el6.x86_64
Comment 9 Chao Yang 2011-07-19 01:53:36 UTC 
I still can't reproduce it. I tried:
1. launch a guest directly using qemu-kvm, tested more than 10 times, don't find any eother event through QMP but only enospc event.
2. use RHEV-M to launch a RHEL6.0 guest, attach a qcow2 image which is thin-provisioned, 20G, allocated space, 1G. Tested for more than 6 times, sequential write to attached disk doesn't trigger this issue for me, don't see any eother event in vdsm log, attached disk size increases when allocated space is consumed. Guest is running well.

My testing environment:
# uname -r 
2.6.32-168.el6.x86_64 
# rpm -q qemu-kvm 
qemu-kvm-0.12.1.2-2.171.el6.x86_64
# rpm -q vdsm 
vdsm-4.9-79.el6.x86_64 
# rpm -q libvirt 
libvirt-0.9.1-1.el6.x86_64
# lvscan 
ACTIVE            '/dev/a38a0e73-cd4a-4a52-866b-fb7a10a248b7/e940502b-107a-4bee-89c8-8e679a904c8b' [4.00 GiB] inherit 
ACTIVE            '/dev/a38a0e73-cd4a-4a52-866b-fb7a10a248b7/fc46a6ea-54d4-4454-b245-c9af56f39e6f' [21.00 GiB] inherit  <--------this thin-provisioned attached disk size has been increased to 21G when allocated space is consumed



Hi Haim,
 Do I have any chance to access your setup to reproduce this issue? If yes, could you let me know if the setup is available?

Thanks,
Comment 10 Kevin Wolf 2011-08-08 09:12:07 UTC 
Haim, if we don't get more info very soon (development freeze is this week), I'll have to move this to 6.3.
Note You need to log in before you can comment on or make changes to this bug.