Machine log file hints to an I/O error when accessing the disk, same with the screenshot.

libvirt's machine log file:
2014-04-18 02:50:54.370+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name rhel7 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -cpu SandyBridge -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid b1410a75-f447-4dcc-bec6-a3b8b2e09539 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/rhel7.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot menu=on,strict=on -device pci-bridge,chassis_nr=1,id=pci.1,bus=pci.0,addr=0x9 -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0xa -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=nbd:10.66.106.26:10809,if=none,id=drive-scsi0-0-0-0,format=qcow2 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=25 -device virtio-net-pci,event_idx=off,netdev=hostnet0,id=net0,mac=52:54:00:95:79:af,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/rhel7.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -chardev pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0 -device usb-tablet,id=input0 -spice port=5900,addr=0.0.0.0,disable-ticketing,seamless-migration=on -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -chardev spicevmc,id=charredir2,name=usbredir -device usb-redir,chardev=charredir2,id=redir2 -chardev spicevmc,id=charredir3,name=usbredir -device usb-redir,chardev=charredir3,id=redir3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8
Warning: option deprecated, use lost_tick_policy property of kvm-pit instead.
char device redirected to /dev/pts/1 (label charconsole0)
main_channel_link: add main channel client
main_channel_handle_parsed: net test: latency 0.568000 ms, bitrate 454101995 bps (433.065410 Mbps)
red_dispatcher_set_cursor_peer: 
inputs_connect: inputs channel client create
block I/O error in device 'drive-scsi0-0-0-0': Invalid argument (22)
block I/O error in device 'drive-scsi0-0-0-0': Input/output error (5)
block I/O error in device 'drive-scsi0-0-0-0': Input/output error (5)
block I/O error in device 'drive-scsi0-0-0-0': Input/output error (5)
nbd.c:nbd_receive_reply():L746: read failed
nbd.c:nbd_receive_reply():L746: read failed
nbd.c:nbd_receive_reply():L746: read failed
nbd.c:nbd_receive_reply():L746: read failed
nbd.c:nbd_receive_reply():L746: read failed
nbd.c:nbd_receive_reply():L746: read failed
nbd.c:nbd_receive_reply():L746: read failed
nbd.c:nbd_receive_reply():L746: read failed
nbd.c:nbd_receive_reply():L746: read failed

Moving to qemu to investigate further.

Hi Sluo,

Can you have a try by using qemu-kvm CML directly?

Best Regards,
Junyi

(In reply to juzhang from comment #3)
> Hi Sluo,
> 
> Can you have a try by using qemu-kvm CML directly?
> 
Yes, hit this issue with qemu-kvm command line directly.

host info:
# uname -r && rpm -q qemu-kvm-rhev
3.10.0-121.el7.x86_64
qemu-kvm-rhev-1.5.3-60.el7ev.x86_64
# rpm -q nbd
nbd-2.9.20-3.el7.x86_64
guest info:
# uname -r
3.10.0-121.el7.x86_64

How reproducible:
3/5

Steps:
1. start nbd-server to export the system image with absolute path on the NBD server host.
# nbd-server 12345 /home/RHEL-7.0-20140409.0_Server_x86_64.qcow2bk

** (process:20912): WARNING **: Specifying an export on the command line is deprecated.

** (process:20912): WARNING **: Please use a configuration file instead.

2. launch a KVM guest with the exported the system image.
e.g:# /usr/libexec/qemu-kvm...-drive file=nbd:10.66.83.171:12345,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native -device virtio-scsi-pci,id=scsi0,addr=0x4,bus=pci.0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-system-disk,id=system-disk,bootindex=1

Results:
after step 2, KVM guest fail to boot up and HMP flooded with "nbd.c:nbd_receive_reply():L746: read failed".
(qemu) block I/O error in device 'drive-system-disk': Invalid argument (22)
block I/O error in device 'drive-system-disk': Input/output error (5)
block I/O error in device 'drive-system-disk': Input/output error (5)
block I/O error in device 'drive-system-disk': Input/output error (5)
nbd.c:nbd_receive_reply():L746: read failed
nbd.c:nbd_receive_reply():L746: read failed
nbd.c:nbd_receive_reply():L746: read failed
nbd.c:nbd_receive_reply():L746: read failed
nbd.c:nbd_receive_reply():L746: read failed
nbd.c:nbd_receive_reply():L746: read failed
nbd.c:nbd_receive_reply():L746: read failed
nbd.c:nbd_receive_reply():L746: read failed
nbd.c:nbd_receive_reply():L746: read failed
......

And guest kernel log output "XFS (dm-0): metadata I/O error: block 0xdf1b70 ("xfs_trans_read_buf_map") error", I will attach the guest kernel log later.

Best Regards,
sluo

Created attachment 889518 [details]
guest-kernel-log.txt

My qemu-kvm command line:
# /usr/libexec/qemu-kvm -S -M pc -cpu host -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -no-kvm-pit-reinjection -usb -device usb-tablet,id=input0 -name sluo -uuid 990ea161-6b67-41b2-b803-19fb01d30d30 -rtc base=localtime,clock=host,driftfix=slew -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=pci.0,addr=0x3 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port1 -chardev socket,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port2 -drive file=nbd:10.66.83.171:12345,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native -device virtio-scsi-pci,id=scsi0,addr=0x4,bus=pci.0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-system-disk,id=system-disk,bootindex=1 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=00:01:02:B6:40:11,bus=pci.0,addr=0x5 -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x6 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -serial unix:/tmp/ttyS0,server,nowait -k en-us -boot menu=on -qmp tcp:0:4444,server,nowait -vnc :1 -monitor stdio

Hi Sluo,

I get the same errors with upstream qemu when trying to boot from a non-preallocated qcow2 image. I don't know the internals of nbd-server, but from Wireshark and qemu's NBD debug output it appears to me that nbd-server does not allow writes beyond the end of the disk image (and stalls all other requests after such a write).

A non-preallocated qcow2 image of course may grow on write accesses, that's why as soon as a new cluster is allocated (at the end of the disk image) and data is written there, nbd-server returns an error and subsequently stalls. Converting the image to a preallocated one (qemu-img convert -O qcow2 -o preallocation=metadata rhel7.qcow2 rhel7-preallocated.qcow2) and then using that result works for me.

Could you try that, too? If it works, I guess this not really qemu's fault but rather nbd-server's which does not allow growing disk images. The best thing qemu could do is to try to sense whether the NBD server allows growing and if it doesn't, somehow pass that through to the image format drivers which must then prevent new allocation of clusters. But the result would practically be the same (EIO or ENOSPC on boot), therefore it's probably not worth it.

If this (preallocation) does not fix the problem for you, I'll try again with RHEL6's qemu.

Thank you,
Max

(In reply to Max Reitz from comment #8)
> Hi Sluo,
> 
> I get the same errors with upstream qemu when trying to boot from a
> non-preallocated qcow2 image. I don't know the internals of nbd-server, but
> from Wireshark and qemu's NBD debug output it appears to me that nbd-server
> does not allow writes beyond the end of the disk image (and stalls all other
> requests after such a write).
> 
> A non-preallocated qcow2 image of course may grow on write accesses, that's
> why as soon as a new cluster is allocated (at the end of the disk image) and
> data is written there, nbd-server returns an error and subsequently stalls.
> Converting the image to a preallocated one (qemu-img convert -O qcow2 -o
> preallocation=metadata rhel7.qcow2 rhel7-preallocated.qcow2) and then using
> that result works for me.
> 
> Could you try that, too? If it works, I guess this not really qemu's fault
> but rather nbd-server's which does not allow growing disk images. The best
> thing qemu could do is to try to sense whether the NBD server allows growing
> and if it doesn't, somehow pass that through to the image format drivers
> which must then prevent new allocation of clusters. But the result would
> practically be the same (EIO or ENOSPC on boot), therefore it's probably not
> worth it.
> 
> If this (preallocation) does not fix the problem for you, I'll try again
> with RHEL6's qemu.
> 
Thanks for your analysis，retried with your instruction which works well if specified the preallocation=metadata for disk image.

hostinfo:
3.10.0-123.4.2.el7.x86_64
qemu-kvm-rhev-1.5.3-60.el7ev.x86_64
nbd-2.9.20-3.el7.x86_64

# qemu-img create -f qcow2 -o preallocation=metadata metadata-data-disk.qcow2 10G
Formatting 'metadata-data-disk.qcow2', fmt=qcow2 size=10737418240 encryption=off cluster_size=65536 preallocation='metadata' lazy_refcounts=off 

# nbd-server 12345 /home/metadata-data-disk.qcow2

** (process:28742): WARNING **: Specifying an export on the command line is deprecated.

** (process:28742): WARNING **: Please use a configuration file instead.

e,g:...-drive file=nbd:10.66.104.53:12345,format=qcow2,if=none,id=drive-scsi-disk -device virtio-scsi-pci,id=scsi0,addr=0x8 -device scsi-hd,drive=drive-scsi-disk,bus=scsi0.0,scsi-id=0,lun=0,id=data-disk1

Best Regards,
sluo

(In reply to Max Reitz from comment #8)
> 
> If this (preallocation) does not fix the problem for you, I'll try again
> with RHEL6's qemu.
> 
rhel6: Bug 1111028 - QEMU fail to open NBD device image which exported by the nbd-server

Oh, I somehow thought this bug was about RHEL 6; sorry. Please read the "RHEL6" as "RHEL7", then.

Anyway, considering that my reasoning seems to be correct (nbd-server simply cannot grow image files), I think qemu cannot do anything to work around this behavior. The best thing we could do is to document it, but I don't know where to do so.

Therefore I think this is definitely not a bug in qemu and most probably not even a bug in nbd-server (only a kind of (perhaps even documented) behavior which makes it difficult to export non-raw images over NBD using nbd-server), but simply a problem inherent to NBD.

Maybe the best qemu could do is to emit a warning if a image format which needs its files to be able to grow (such as qcow2) is used non-read-only over NBD. I feel like this would not be completely trivial to implement, though, thus I'll close this BZ for now. If you feel like qemu should indeed emit a warning and it would significantly improve its behavior in this case, feel free to reopen it and I'll do my best to implement it (or if you have any other suggestion).

Max

(In reply to Max Reitz from comment #11)
> Oh, I somehow thought this bug was about RHEL 6; sorry. Please read the
> "RHEL6" as "RHEL7", then.
> 
> Anyway, considering that my reasoning seems to be correct (nbd-server simply
> cannot grow image files), I think qemu cannot do anything to work around
> this behavior. The best thing we could do is to document it, but I don't
> know where to do so.
> 
> Therefore I think this is definitely not a bug in qemu and most probably not
> even a bug in nbd-server (only a kind of (perhaps even documented) behavior
> which makes it difficult to export non-raw images over NBD using
> nbd-server), but simply a problem inherent to NBD.
> 
> Maybe the best qemu could do is to emit a warning if a image format which
> needs its files to be able to grow (such as qcow2) is used non-read-only
> over NBD. I feel like this would not be completely trivial to implement,
> though, thus I'll close this BZ for now. If you feel like qemu should indeed
> emit a warning and it would significantly improve its behavior in this case,
> feel free to reopen it and I'll do my best to implement it (or if you have
> any other suggestion).
> 
From KVM QE's POV we should add a friendly warning message prompt for KVM user about the reasion. It's same to bug 1111028#c4, please correct me if any mistake, thanks in advance.

Best Regards,
sluo

Okay, I'll take a look into it. I'll try to output a warning if an image format which may require growth on write accesses is used non-read-only over a non-growable protocol. If it's too ugly to implement, I probably won't do it, though.

(In reply to Max Reitz from comment #13)
> Okay, I'll take a look into it. I'll try to output a warning if an image
> format which may require growth on write accesses is used non-read-only over
> a non-growable protocol. If it's too ugly to implement, I probably won't do
> it, though.
Thanks a lot, i think you can let the QEMU quit with just a warning prompt error message for users instead of 'I/O error'. Just as Paolo Bonzini said in bug 1111028#c5: RHEL6 is not able to use good error messages because it only uses errno to generate errors, the RHEL7 changes to improve the error messages were huge and cannot be backported.

Best Regards,
sluo

As it turned out, fixing this bug goes along with the restructuring of (at least) some parts of the block layer which is currently taking place. Therefore, this will still take some time to be implemented even upstream, and I cannot make any promises on when it will be there. Due to how many dependencies the solution will probably have in the end, I will not bring this to RHEL (both 6 and 7), however. Therefore, closing this BZ as WONTFIX.

Max

(In reply to Max Reitz from comment #11)

> Anyway, considering that my reasoning seems to be correct (nbd-server simply
> cannot grow image files), I think qemu cannot do anything to work around
> this behavior. The best thing we could do is to document it, but I don't
> know where to do so.
> 
> Therefore I think this is definitely not a bug in qemu and most probably not
> even a bug in nbd-server (only a kind of (perhaps even documented) behavior
> which makes it difficult to export non-raw images over NBD using
> nbd-server), but simply a problem inherent to NBD.

There is a proposal to add NBD_CMD_RESIZE to the NBD spec; which WILL allow resizing non-raw images over NBD; but it will not land any sooner than upstream qemu 2.12.  Until that time, I agree that the inability to grow an image can lead to enough obscure errors that...

> 
> Maybe the best qemu could do is to emit a warning if a image format which
> needs its files to be able to grow (such as qcow2) is used non-read-only
> over NBD. I feel like this would not be completely trivial to implement,
> though, thus I'll close this BZ for now. If you feel like qemu should indeed
> emit a warning and it would significantly improve its behavior in this case,
> feel free to reopen it and I'll do my best to implement it (or if you have
> any other suggestion).

...issuing a warning about use of NBD with non-raw format may indeed be worthwhile.

(In reply to Eric Blake from comment #16)
> There is a proposal to add NBD_CMD_RESIZE to the NBD spec; which WILL allow
> resizing non-raw images over NBD; but it will not land any sooner than
> upstream qemu 2.12.  Until that time, I agree that the inability to grow an
> image can lead to enough obscure errors that...

But is there a point in exporting non-raw images over NBD?

> ...issuing a warning about use of NBD with non-raw format may indeed be
> worthwhile.

I think I once sent some patches for that.  The warning was to be emitted generally when you try to use a format which requires implicit growth of the underlying file, but one of the issues was the good old LVM thing (using qcow2 directly in LVM volumes and growing them manually when necessary).

Max