Created attachment 1005150 [details]
dmesg host

Created attachment 1005151 [details]
journal host

Created attachment 1005152 [details]
dmesg guest

Created attachment 1005153 [details]
journal guest

Created attachment 1005154 [details]
guest journal -k, sysrq t

Created attachment 1005155 [details]
host journal -k, sysrq t

I'm not sure the warnings on the KVM host are related to the block errors.  It might be more likely that there are some issues between the host and guest via virtio-blk.

Both host and guest messages are producible when regressing host kernel to:
3.19.2-200.fc21.x86_64
3.18.9-200.fc21.x86_64

Created attachment 1005456 [details]
dmesg guest ext4

This contains a little new information:
[  206.342302] WARNING: CPU: 7 PID: 2533 at fs/block_dev.c:57 __blkdev_put+0xc1/0x220()

OK this is some weird interaction between gnome-boxes and the host using Btrfs. The problem is not reproducible if host /home is XFS; or if host uses virt-manager on Btrfs.

virt-manager uses compat 1.1, where gnome-boxes uses compat 0.10. man qemu-img says compat 1.1 is the default, so I don't know why gnome-boxes is using 0.10.
###"compat=0.10" uses the traditional image format that can be read by any QEMU
since 0.10.  "compat=1.1" enables image format extensions that only QEMU 1.1 and newer understand (this is the default). Amongst others, this includes zero clusters, which allow efficient copy-on-read for sparse images.###

The gnome-boxes image is 10x more fragmented than the virt-manager's. qemu-img check finds no problem with either one. ext4 in the guest tolerates these errors more (retries?) as does Btrfs, and both complete an installation and then report no fs errors on the guest fs despite the write time i/o errors during installation. XFS on the other hand properly gets annoyed and gives up before installation completes. The resulting guest on either ext4 or Btrfs are erratic however, so some things probably weren't written even if the fs is consistent. So I'd consider it a catastrophic error.

Maybe the specific qemu command between virt-manager and gnome-boxes has meaningful differences so I'll incude those here too.

###This is for virt-manager, problem does not reproduce.

# ps -ww -F
qemu      3043     1 72 2153085 3808112 2 09:56 ?      Sl     8:49 /usr/bin/qemu-system-x86_64 -machine accel=kvm -name fedora22 -S -machine pc-i440fx-2.2,accel=kvm,usb=off -cpu SandyBridge -m 3584 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 39b5e059-4edd-4b36-b5d5-c048c7316669 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/fedora22.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-reboot -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/var/lib/libvirt/images/fedora21.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -drive file=/var/lib/libvirt/images/Fedora-Live-Workstation-x86_64-22_Beta-TC4.iso,if=none,id=drive-ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:84:59:29,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/fedora22.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on

# qemu-img info -f qcow2 /var/lib/libvirt/images/fedora21.qcow2 
image: /var/lib/libvirt/images/fedora21.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 3.8G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: true
    corrupt: false

# qemu-img check -f qcow2 /var/lib/libvirt/images/fedora21.qcow2 
No errors were found on the image.
327680/327680 = 100.00% allocated, 0.00% fragmented, 0.00% compressed clusters
Image end offset: 21478375424


[root@f22m images]# filefrag /var/lib/libvirt/images/fedora21.qcow2 
/var/lib/libvirt/images/fedora21.qcow2: 5772 extents found

--------------------------------------------------


###This is for gnome-boxes, problem does reproduce.

# ps -ww -F
chris     4235     1 99 1262500 840852 3 10:12 ?       Sl     1:03 /usr/bin/qemu-system-x86_64 -machine accel=kvm -name fedora-unkno -S -machine pc-i440fx-2.2,accel=kvm,usb=off -cpu SandyBridge -m 3619 -realtime mlock=off -smp 8,sockets=1,cores=4,threads=2 -uuid 15783bc9-e51e-455c-b3ad-eea981326225 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/home/chris/.config/libvirt/qemu/lib/fedora-unkno.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-reboot -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -device usb-ccid,id=ccid0 -drive file=/home/chris/.local/share/gnome-boxes/images/fedora-unkno,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -drive file=/var/lib/libvirt/images/Fedora-Live-Workstation-x86_64-22_Beta-TC4.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -netdev tap,fd=21,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:3a:68:f5,bus=pci.0,addr=0x3 -chardev spicevmc,id=charsmartcard0,name=smartcard -device ccid-card-passthru,chardev=charsmartcard0,id=smartcard0,bus=ccid0.0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -device usb-mouse,id=input1 -device usb-kbd,id=input2 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 -device AC97,id=sound0,bus=pci.0,addr=0x4 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -chardev spicevmc,id=charredir2,name=usbredir -device usb-redir,chardev=charredir2,id=redir2 -chardev spicevmc,id=charredir3,name=usbredir -device usb-redir,chardev=charredir3,id=redir3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on


# qemu-img info -f qcow2 /home/chris/.local/share/gnome-boxes/images/fedora-unkno 
image: /home/chris/.local/share/gnome-boxes/images/fedora-unkno
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 4.3G
cluster_size: 65536
Format specific information:
    compat: 0.10


# qemu-img check -f qcow2 /home/chris/.local/share/gnome-boxes/images/fedora-unkno 
No errors were found on the image.
69679/327680 = 21.26% allocated, 14.09% fragmented, 0.00% compressed clusters
Image end offset: 4568252416


[root@f22m images]# filefrag /home/chris/.local/share/gnome-boxes/images/fedora-unkno 
/home/chris/.local/share/gnome-boxes/images/fedora-unkno: 63345 extents found

FYI reference to fedora21.qcow2 is due to virt-manager OS preset not having a Fedora 22 option, so I picked Fedora 21 and this caused the qcow2 to be named fedora21.qcow2. But both on host and guest this is Fedora 22 Workstation (live) Beta TC4.

OK this problem is always reproducible when cache=none or cache=directsync, and the qcow2 is on Btrfs. It doesn't matter if the qcow2 is compat 1.1 or 0.10.

e.g.
/usr/bin/qemu-system-x86_64 -drive file=/var/lib/libvirt/images/fedora21.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none

By default virt-manager omits cache= entirely, and the problem doesn't happen. I have no idea what it's using when is not specified on the command line at all. I've tested the other cache policies, writethrough, writeback, and unsafe, and none of those have this problem.

See also:

Request gnome-boxes "Create qcow2 with compat 1.1"
https://bugzilla.gnome.org/show_bug.cgi?id=746660

Request gnome-boxes "do not use cache=none, causes problems on Btrfs"
and maybe it's not the best default anyway? virt-manager doesn't specify cache= at all and has no problem without it.
https://bugzilla.gnome.org/show_bug.cgi?id=746662

Setting it back to kernel since I'm not really sure if this is qemu, libvirt, or kvm territory.

I can say this is a regression, because I did a bunch of testing of the guest using ext4, XFS, and Btfs; with Raw and qcow2 files on Btrfs, using all of the cache methods including a lot of cache=none testing, maybe 2 years ago. But I have no idea of the interaction between kvm device caching and the file system the backing file is located on.

Created attachment 1006113 [details]
software versions, bug NOT reproducible

The bug is NOT reproducible with these versions from Fedora 20:
kernel
libvirt
qemu
+
qemu command line


[The problem has previously been reproduced with this same kernel, 3.18.9, therefore I don't think this is a kernel problem. It's probably in libvirt or qemu.]

Created attachment 1006114 [details]
software versions, bug IS reproducible

The bug is reproducible with these versions from Fedora 21:
kernel
libvirt
qemu
+
qemu command line

Upstream Btrfs looking into it on their end:
http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg42492.html
Emailed virt.org to see whether this may be libvirt or qemu related.

Created attachment 1007411 [details]
strace qemu

strace -e pwrite,pwritev -f -p qemuPID

The strace is started shortly after mkfs.xfs by Anaconda running in the guest, and is ~1 minute of capture while guest kernel reports I/O errors for /dev/vda, and then I issued poweroff -f in the guest.

All testing so far used Fedora 22 Workstation (live) Beta TC4 for guest. I just inadvertently tried Fedora 21 Workstation (live-final) in the guest, and the problem doesn't reproduce, even on hosts where F22ßTC4 does reproduce the problem. So there is something also about the guest that affects it, not only host libvirt/qemu versions.

Thanks for the strace, Chris. There is one single failing pwritev() call in it:

3091  <... pwritev resumed> )           = -1 EEXIST (File exists)

Now EEXIST is not an error code that makes any sense for a pwritev() call, but a
quick search on the internet suggests that btrfs does indeed use this error code
if something goes wrong internally. You should probably mention this error code
in your linux-btrfs thread, perhaps it gives the btrfs developers a hint.

Starting with Fedora 20 fully updated where the problem does not reproduce; and then selectively updating to Fedora 21 qemu parts, I end up in DepHell. This is fixed with --skip-broken but there are a pile of libraries not updated; and the problem is still not reproducible so for all I know it's one of those dependent libraries causing the problem. When I revert to the F20update base tree, and try to upgrade libvirt bits to F21 libvirt bits, the same thing happens but fails with --skip-broken. So I don't have an easy way to determine whether or which version of libvirt and qemu instigate the problem.

All I can say with certainty is the problem doesn't reproduce in Fedora 20 fully updated, but does reproduce in Fedora 21 clean installed only updating kernel (to same as Fedora 20 updated). That's about as close in versions as I can get them (same host hardware).

Yuck.  Can you find out where the EEXIST comes from?

At one point, something like this:

# for FUNCTION in `grep "t btrfs_" /proc/kallsyms | awk '{print $3}'`; do echo "r:ret_$FUNCTION $FUNCTION \$retval" >> /sys/kernel/debug/tracing/kprobe_events; done

# for ENABLE in /sys/kernel/debug/tracing/events/kprobes/ret_btrfs_*/enable; do echo 1 > $ENABLE; done

<run failing test>

# for ENABLE in /sys/kernel/debug/tracing/events/kprobes/ret_btrfs_*/enable; do echo 0 > $ENABLE; done

# cat /sys/kernel/debug/tracing/trace

and look for which function returned EEXIST...

(In reply to Eric Sandeen from comment #22)
> Yuck.  Can you find out where the EEXIST comes from?
1. Should I do this in the guest or the host?
2. If guest, the first command work, the 2nd doesn't.
# for ENABLE in /sys/kernel/debug/tracing/events/kprobes/ret_btrfs_*/enable; do echo 1 > $ENABLE; done
bash: /sys/kernel/debug/tracing/events/kprobes/ret_btrfs_*/enable: No such file or directory
3. If host, the first command doesn't work.
# for FUNCTION in `grep "t btrfs_" /proc/kallsyms | awk '{print $3}'`; do echo "r:ret_$FUNCTION $FUNCTION \$retval" >> /sys/kernel/debug/tracing/kprobe_events; done
bash: echo: write error: Invalid argument
bash: echo: write error: Invalid argument
bash: echo: write error: Invalid argument
bash: echo: write error: Invalid argument
<repeats apparently forever>

*** Bug 1220970 has been marked as a duplicate of this bug. ***

I seem to be facing something similar while trying to install F22 virt machine (on F21 host).

In fact for me this does not seem to be related to filesystem at all. I have exported a qemu disk into guest as virtIO disk and I can't even write a block to that device.

dd if=/dev/zero of=/dev/vda bs=4K count=1 fails. 

Jun 09 11:55:57 localhost kernel: blk_update_request: I/O error, dev vda, sector 0
Jun 09 11:55:57 localhost kernel: Buffer I/O error on dev vda, logical block 0, lost async page write


On host, I am using ext4 file system and I don't see any error messages on host.

Tried installing F21 guest (on F21 host) and faced same issue. So may be issue is somewhere on F21 host qemu/libvirt stuff.

(In reply to Vivek Goyal from comment #25)
> In fact for me this does not seem to be related to filesystem at all. I have
> exported a qemu disk into guest as virtIO disk and I can't even write a
> block to that device.

Then that's a different problem. Please file a separate BZ and provide the
strace output (options see above) for your case there.

(In reply to Kevin Wolf from comment #27)
> (In reply to Vivek Goyal from comment #25)
> > In fact for me this does not seem to be related to filesystem at all. I have
> > exported a qemu disk into guest as virtIO disk and I can't even write a
> > block to that device.
> 
> Then that's a different problem. Please file a separate BZ and provide the
> strace output (options see above) for your case there.

Kevin,

I opened bz https://bugzilla.redhat.com/show_bug.cgi?id=1230304

Description of problem:
i installed firefox on an old live disk while i was installing with anaconda

Version-Release number of selected component:
kernel

Additional info:
reporter:       libreport-2.5.1
cmdline:        BOOT_IMAGE=vmlinuz0 initrd=initrd0.img root=live:CDLABEL=Fedora-Live-Sec-i686-22-3 rootfstype=auto ro rd.live.image quiet  rhgb rd.luks=0 rd.md=0 rd.dm=0 nomodeset
kernel:         4.0.4-301.fc22.i686
runlevel:       N 5
type:           Kerneloops

Truncated backtrace:
WARNING: CPU: 0 PID: 2260 at fs/block_dev.c:57 __blkdev_put+0x98/0x1b0()
Modules linked in: fcoe libfcoe libfc scsi_transport_fc iscsi_ibft iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_crypt vfat fat dm_round_robin dm_multipath raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq raid1 raid0 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute bridge ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd iTCO_wdt
 gpio_ich iTCO_vendor_support ppdev parport_pc parport serio_raw lpc_ich i2c_i801 soundcore tpm_infineon tpm acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace isofs squashfs i2c_algo_bit drm_kms_helper 8021q garp stp llc ttm mrp tulip 3c59x ata_generic r8169 drm pata_acpi mii uas usb_storage scsi_dh_rdac scsi_dh_emc scsi_dh_alua sunrpc loop
CPU: 0 PID: 2260 Comm: e2fsck Not tainted 4.0.4-301.fc22.i686 #1
Hardware name:                  /D915PGN                        , BIOS CY91510A.86A.0035.2005.0429.1900 04/29/2005
 c0d1b947 753bded5 00000000 dd351ed4 c0a71a6e 00000000 dd351f04 c0458127
 c0c34ff4 00000000 000008d4 c0c58506 00000039 c05c9c78 c05c9c78 f30204e0
 f3020400 f302048c dd351f14 c0458232 00000009 00000000 dd351f34 c05c9c78
Call Trace:
 [<c0a71a6e>] dump_stack+0x41/0x52
 [<c0458127>] warn_slowpath_common+0x87/0xc0
 [<c05c9c78>] ? __blkdev_put+0x98/0x1b0
 [<c05c9c78>] ? __blkdev_put+0x98/0x1b0
 [<c0458232>] warn_slowpath_null+0x22/0x30
 [<c05c9c78>] __blkdev_put+0x98/0x1b0
 [<c05ca190>] blkdev_put+0x40/0x100
 [<c05ca26f>] blkdev_close+0x1f/0x30
 [<c0597248>] __fput+0xc8/0x1d0
 [<c059738d>] ____fput+0xd/0x10
 [<c04707c1>] task_work_run+0xb1/0xe0
 [<c0403225>] do_notify_resume+0x75/0x80
 [<c0a77225>] work_notifysig+0x30/0x37

*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 22 kernel bugs.

Fedora 22 has now been rebased to 4.2.3-200.fc22.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 23, and are still experiencing this issue, please change the version to Fedora 23.

If you experience different issues, please open a new bug report for those.

I can confirm this bug is still present in Fedora 22/23 with the latest kernel. Everytime when you copy a qcow2 image to a btrfs filesystem the system freezes with the error message described in the subject

(In reply to Aniruddha from comment #31)
> I can confirm this bug is still present in Fedora 22/23 with the latest
> kernel. Everytime when you copy a qcow2 image to a btrfs filesystem the
> system freezes with the error message described in the subject

That sounds completely different than the bug I reported: the errors happen in the guest OS. I've never had qcow2, or any other large file type, have problems copying to or from Btrfs on bare metal. I suggest opening a new bug with clear reproduce steps. I'd like to be cc'd on this so I can try to reproduce.

I'm not sure when this changed, but I can't reproduce this bug.

Host, any of:
kernel-4.4.0-1.fc24.x86_64
kernel-4.3.3-303.fc23.x86_64

And:
libvirt-daemon-kvm-1.2.18.2-1.fc23.x86_64
libvirt-daemon-driver-qemu-1.2.18.2-1.fc23.x86_64
libvirt-gobject-0.2.2-1.fc23.x86_64
libvirt-daemon-driver-nwfilter-1.2.18.2-1.fc23.x86_64
libvirt-daemon-driver-interface-1.2.18.2-1.fc23.x86_64
libvirt-daemon-driver-secret-1.2.18.2-1.fc23.x86_64
libvirt-client-1.2.18.2-1.fc23.x86_64
libvirt-glib-0.2.2-1.fc23.x86_64
libvirt-python-1.2.18-1.fc23.x86_64
libvirt-daemon-config-network-1.2.18.2-1.fc23.x86_64
libvirt-gconfig-0.2.2-1.fc23.x86_64
libvirt-daemon-driver-nodedev-1.2.18.2-1.fc23.x86_64
libvirt-daemon-driver-network-1.2.18.2-1.fc23.x86_64
libvirt-daemon-1.2.18.2-1.fc23.x86_64
libvirt-daemon-driver-storage-1.2.18.2-1.fc23.x86_64

Guest is Fedora-Live-Workstation-x86_64-23-10.iso

We'll close this out for now.