Bug 658744

Summary: Formatting disk on nfs caused guest paused
Product: Red Hat Enterprise Linux 6 Reporter: Amos Kong <akong>
Component: kernelAssignee: Jeff Layton <jlayton>
Status: CLOSED DUPLICATE QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: medium Docs Contact:
Priority: high    
Version: 6.1CC: ailan, bfields, dhowells, jlayton, kwolf, mkenneth, riek, rwheeler, steved, virt-maint
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-12-02 12:40:53 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 580954    
Attachments:
Description Flags
strace output none

Description Amos Kong 2010-12-01 06:09:42 UTC 
Description of problem:

I tried to format a disk, which is on nfs server, format didn't complete and guest paused.

===> qemu output:
block I/O error in device 'drive-virtio-disk2': Cannot allocate memory (12)
handle_dev_input: stop
block I/O error in device 'drive-virtio-disk2': Cannot allocate memory (12)
block I/O error in device 'drive-virtio-disk2': Cannot allocate memory (12)
block I/O error in device 'drive-virtio-disk2': Cannot allocate memory (12)
handle_dev_input: start
block I/O error in device 'drive-virtio-disk2': Cannot allocate memory (12)
handle_dev_input: stop
block I/O error in device 'drive-virtio-disk2': Cannot allocate memory (12)
block I/O error in device 'drive-virtio-disk2': Cannot allocate memory (12)
block I/O error in device 'drive-virtio-disk2': Cannot allocate memory (12)

Version-Release number of selected component (if applicable):
host kernel: 2.6.32-84.el6.x86_64
guest kernel: 2.6.32-71.7.1.el6.i686
# rpm -qa |grep qemu
qemu-kvm-debuginfo-0.12.1.2-2.120.el6.x86_64
gpxe-roms-qemu-0.9.7-6.3.el6.noarch
qemu-kvm-0.12.1.2-2.120.el6.x86_64
qemu-kvm-tools-0.12.1.2-2.120.el6.x86_64
qemu-img-0.12.1.2-2.120.el6.x86_64


How reproducible:
always

Steps to Reproduce:
1. setup nfs on localhost
# mkdir -p /tmp/nfs_dir && mkdir -p images/nfs && service nfs restart && exportfs 127.0.0.1:/tmp/nfs_dir -o rw,no_root_squash && mount 127.0.0.1:/tmp/nfs_dir images/nfs -o rw,soft,timeo=1,retrans=1,vers=3
2. create a image on nfs dir
# qemu-img create -f qcow2 -opreallocation=metadata -ocluster_size=2M /root/autotest-devel/client/tests/kvm/images/nfs/storage.qcow2 10G
3. boot up guest with two images (one on localhost, one on nfs), werror=on
4. login guest and format second disk
guest) # mkfs vdb


Actual results:
format didn't complete, guest paused, qemu process outputted some error msg.

Expected results:
success to format disk on nfs

Additional info:

1. qemu commandline:
# qemu-kvm -name vm1 -chardev socket,id=human_monitor_fVCa,path=/tmp/monitor-humanmonitor1-20101201-125251-YB6k,server,nowait -mon chardev=human_monitor_fVCa,mode=readline -chardev socket,id=serial_wykW,path=/tmp/serial-20101201-125251-YB6k,server,nowait -device isa-serial,chardev=serial_wykW -drive file=/root/autotest-devel/client/tests/kvm/images/RHEL-Server-6.0-32-virtio.qcow2,index=0,if=none,id=drive-virtio-disk1,media=disk,cache=none,werror=stop,format=qcow2,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/root/autotest-devel/client/tests/kvm/images/nfs/storage.qcow2,if=none,id=drive-virtio-disk2,media=disk,cache=none,werror=stop,format=qcow2,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk2,id=virtio-disk2 -device virtio-net-pci,mac=9a:65:71:99:93:3a,netdev=idQpJUNB,id=ndev00idQpJUNB,bus=pci.0,addr=0x3 -netdev tap,id=idQpJUNB,vhost=on,ifname=t0-125251-YB6k,script=/root/autotest-devel/client/tests/kvm/scripts/qemu-ifup-switch,downscript=no,vhost=on -m 2048 -smp 2,cores=1,threads=1,sockets=2 -cpu cpu64-rhel6,+sse2,+x2apic -vnc :0 -spice port=8000,disable-ticketing -vga qxl -rtc base=utc,clock=host,driftfix=none -M rhel6.0.0 -usbdevice tablet -no-kvm-pit-reinjection -enable-kvm
Comment 4 chellwig@redhat.com 2010-12-01 13:42:06 UTC 
Please try aio=threads instead of aio=native and attach the strace -f output of the qemu binary, to figure out where the ENOMEM comes from.  I'm pretty sure it's the kernel and not qemu, but the strace output is required to verify that theory.
Comment 5 Amos Kong 2010-12-02 06:04:05 UTC 
Created attachment 464169 [details]
strace output

.......
2550  pread(13, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16384, 8355840) = 16384
2550  pread(13, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16384, 8372224) = 16384
2550  pwrite(13, "\0\1\0\1\0\1\0\1\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512, 6291456) = -1 ENOMEM (Cannot allocate memory)
2550  mprotect(0x7f8cec261000, 987136, PROT_READ|PROT_WRITE) = 0
2565  <... epoll_wait resumed> {}, 10, 10) = 0
2565  epoll_wait(22,  <unfinished ...>
2550  pwrite(13, "\0\1\0\1\0\1\0\1\0\1\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 512, 6291456) = -1 ENOMEM (Cannot allocate memory)
2550  madvise(0x7f8cec275000, 905216, MADV_DONTNEED) = 0
2550  write(2, "block I/O error in device 'drive"..., 76) = 76
2550  ioctl(10, 0x4008af30, 0x7f8cf34934b0) = 0
2550  ioctl(10, 0x4008af30, 0x7f8cf34934b0) = 0
.......
Comment 6 Jeff Layton 2010-12-02 12:40:53 UTC 
This is almost certainly the problem fixed by this patch:

    http://post-office.corp.redhat.com/archives/rhkernel-list/2010-November/msg00712.html

I'm going to close this as a duplicate of that bug. Please reopen if that patchset doesn't fix this.

*** This bug has been marked as a duplicate of bug 653066 ***