Description of problem: Guest is not paused in case of I/O errors despite error_policy=stop Version-Release number of selected component (if applicable): Host libvirt-daemon-6.0.0-15.module+el8.2.0+6106+b6345808.s390x qemu-kvm-4.2.0-17.module+el8.2.0+6129+b14d477b.s390x kernel-4.18.0-193.1.el8.s390x rpcbind-1.2.5-7.el8.s390x nfs-utils-2.3.3-31.el8.s390x Guest kernel-4.18.0-187.el8.s390x How reproducible: Always Preconditions 1. selinux permissive 2. attach virtio disk from img file; file is located in nfs, mounted on host (e.g. on /var/tmp/avocado_xzjhv4ny/mnt/libvirt-disk) 3. have img file with ext3 and partition /dev/vdb1 and mount it in guest 4. have a file 'some_file' in mounted /dev/vdb1 5. confirm 'some_file' can be written to (e.g. echo happily > <disk-mount>/some_file && cat <disk-mount>/some_file) Steps: 1. stop nfs server 2. have guest write to 'some_file' ==> Actual result 1. Guest keeps running 2. error messages [ 8012.326079] blk_update_request: I/O error, dev vdb, sector 55304 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0 [ 8012.327652] EXT4-fs warning (device vdb1): ext4_end_bio:323: I/O error 10 writing to inode 12 (offset 0 size 0 starting block 6914) [ 8012.327896] Buffer I/O error on device vdb1, logical block 6657 [ 8015.525815] JBD2: Detected IO errors while flushing file data on vdb1-8 [ 8018.534802] blk_update_request: I/O error, dev vdb, sector 6856 op 0x1:(WRITE) flags 0x800 phys_seg 5 prio class 0 [ 8018.535739] Aborting journal on device vdb1-8. [ 8021.573476] blk_update_request: I/O error, dev vdb, sector 6728 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0 [ 8021.574096] blk_update_request: I/O error, dev vdb, sector 6728 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0 [ 8021.574283] Buffer I/O error on dev vdb1, logical block 585, lost sync page write [ 8021.574521] JBD2: Error -5 detected when updating journal superblock for vdb1-8. Expected result: Guest is paused Additional info: 1. Can't reproduce on older version. More specifically, can't reproduce in the following environments: I) Host s390x libvirt-daemon-4.5.0-42.module+el8.2.0+6024+15a2423f.s390x qemu-kvm-2.12.0-99.module+el8.2.0+5827+8c39933c.s390x kernel-4.18.0-193.1.el8.s390x rpcbind-1.2.5-7.el8.s390x nfs-utils-2.3.3-31.el8.s390x Guest s390x kernel-4.18.0-187.el8.s390x II) Host x86_64 libvirt-daemon-6.0.0-15.module+el8.2.0+6106+b6345808.x86_64 qemu-kvm-4.2.0-17.module+el8.2.0+6129+b14d477b.x86_64 kernel-4.18.0-80.11.2.el8_0.x86_64 nfs-utils-2.3.3-14.el8_0.3.x86_64 rpcbind-1.2.5-3.el8.x86_64 Guest x86_64 kernel-4.13.9-300.fc27.x86_64 III) Host x86_64 libvirt-daemon-6.0.0-15.module+el8.2.0+6106+b6345808.x86_64 qemu-kvm-4.2.0-17.module+el8.2.0+6129+b14d477b.x86_64 kernel-4.18.0-80.11.2.el8_0.x86_64 nfs-utils-2.3.3-14.el8_0.3.x86_64 rpcbind-1.2.5-3.el8.x86_64 2. Can reproduce using both virtio and scsi, i.e. disk xml a. <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none' error_policy='stop'/> <source file='/var/tmp/avocado_xzjhv4ny/mnt/libvirt-disk' index='2'/> <backingStore/> <target dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/> <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0009'/> </disk> b. <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none' error_policy='stop'/> <source file='/var/tmp/avocado_xzjhv4ny/mnt/libvirt-disk' index='1'/> <backingStore/> <target dev='sdb' bus='scsi'/> <alias name='scsi0-0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> 3. qemu-kvm command in case of scsi: LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \ HOME=/var/lib/libvirt/qemu/domain-2-avocado-vt-vm1 \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-2-avocado-vt-vm1/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-2-avocado-vt-vm1/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-2-avocado-vt-vm1/.config \ QEMU_AUDIO_DRV=none \ /usr/libexec/qemu-kvm \ -name guest=avocado-vt-vm1,debug-threads=on \ -S \ -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-2-avocado-vt-vm1/master-key.aes \ -machine s390-ccw-virtio-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off \ -cpu z13.2-base,aen=on,aefsi=on,msa5=on,msa4=on,msa3=on,msa2=on,msa1=on,sthyi=on,edat=on,vx=on,ap=on,esop=on,apft=on,apqci=on,cte=on,bpb=on,ppa15=on,zpci=on,sea_esop2=on,te=on \ -m 1024 \ -overcommit mem-lock=off \ -smp 2,sockets=2,cores=1,threads=1 \ -uuid e8ab3fac-39bd-4fbe-9e36-fc9435770ed6 \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=37,server,nowait \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc \ -no-shutdown \ -boot strict=on \ -device virtio-scsi-ccw,id=scsi0,devno=fe.0.0003 \ -device virtio-serial-ccw,id=virtio-serial0,devno=fe.0.0004 \ -blockdev '{"driver":"file","filename":"/var/lib/avocado/data/avocado-vt/images/jeos-27-s390x.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":null}' \ -device virtio-blk-ccw,scsi=off,devno=fe.0.0000,drive=libvirt-2-format,id=virtio-disk0,bootindex=1 \ -blockdev '{"driver":"file","filename":"/var/tmp/avocado_xzjhv4ny/mnt/libvirt-disk","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \ -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,device_id=drive-scsi0-0-0-1,drive=libvirt-1-format,id=scsi0-0-0-1,write-cache=on \ -netdev tap,fd=39,id=hostnet0,vhost=on,vhostfd=40 \ -device virtio-net-ccw,netdev=hostnet0,id=net0,mac=52:54:00:93:44:a7,devno=fe.0.0001 \ -chardev pty,id=charserial0 \ -device sclpconsole,chardev=charserial0,id=serial0 \ -chardev pty,id=charconsole1 \ -device virtconsole,chardev=charconsole1,id=console1 \ -device virtio-keyboard-ccw,id=input0,devno=fe.0.0005 \ -device virtio-mouse-ccw,id=input1,devno=fe.0.0006 \ -vnc 127.0.0.1:0 \ -device virtio-gpu-ccw,id=video0,max_outputs=1,devno=fe.0.0002 \ -device virtio-balloon-ccw,id=balloon0,devno=fe.0.0007 \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -device vhost-vsock-ccw,id=vsock0,guest-cid=3,vhostfd=31,devno=fe.0.0008 \ -msg timestamp=on 4. I was not able to identify relevant log messages in libvirtd.log (debug level, filters 1:libvirt 1:qemu) nor /var/log/libvirt/qemu/<vm>.log
(In reply to smitterl from comment #0) > III) > Host x86_64 > libvirt-daemon-6.0.0-15.module+el8.2.0+6106+b6345808.x86_64 > qemu-kvm-4.2.0-17.module+el8.2.0+6129+b14d477b.x86_64 > kernel-4.18.0-80.11.2.el8_0.x86_64 > nfs-utils-2.3.3-14.el8_0.3.x86_64 > rpcbind-1.2.5-3.el8.x86_64 > ++ Guest x86_64 ++ kernel-4.18.0-187.el8.x86_64
As pointed out by Kevin Wolf, qemu command line seems to be missing werror/rerror option. On x86_64 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=libvirt-1-format,id=virtio-disk1,write-cache=on,werror=stop,rerror=stop Full: /usr/libexec/qemu-kvm -name guest=avocado-vt-vm1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-2-avocado-vt-vm1/master-key.aes -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu qemu64,x2apic=on,hypervisor=on,lahf-lm=on,svm=off -m 1024 -overcommit mem-lock=off -smp 3,maxcpus=8,sockets=8,cores=1,threads=1 -uuid 19a1bab2-b4b7-45b0-9847-ad7331e2cbfc -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=37,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -device usb-hub,id=hub0,bus=usb.0,port=1 -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/RHEL-8.2-x86_64-latest-03-11.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":null} -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=libvirt-2-format,id=virtio-disk0,bootindex=1 -blockdev {"driver":"file","filename":"/var/tmp/avocado_xzjhv4ny/mnt/libvirt-disk","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"} -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=libvirt-1-format,id=virtio-disk1,write-cache=on,werror=stop,rerror=stop -netdev tap,fd=39,id=hostnet0,vhost=on,vhostfd=40 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:6f:dc:90,bus=pci.0,addr=0x3 -add-fd set=3,fd=42 -chardev pty,id=charserial0,logfile=/dev/fdset/3,logappend=on -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0,bus=usb.0,port=2 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1.1 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=1.2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
Created attachment 1675153 [details] minimal xml for domxml-to-native
(In reply to smitterl from comment #4) > Created attachment 1675153 [details] > minimal xml for domxml-to-native I can confirm that with libvirt-daemon-6.0.0-15.module+el8.2.0+6106+b6345808 only on s390x the (r|w)error are missing, 'virsh domxml-to-native qemu-argv avocado-vt-vm1-minimal.xml|grep error'
Not reproducible for libvirt 5.0.0-15: # rpm -q libvirt-daemon libvirt-daemon-5.0.0-12.module+el8.0.1+3755+6782b0ed.s390x # virsh domxml-to-native qemu-argv vm.xml |grep error LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name guest=avocado-vt-vm5,debug-threads=on -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain--1-avocado-vt-vm5/master-key.aes -machine s390-ccw-virtio-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -m 1024 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid f566a90c-7a4f-4944-84f9-38697116d6e2 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain--1-avocado-vt-vm5/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -drive file=/var/lib/avocado/data/avocado-vt/images/jeos-27-s390x.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-ccw,scsi=off,devno=fe.0.0000,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/tmp/avocado_xzjhv4ny/mnt/libvirt-disk,format=raw,if=none,id=drive-virtio-disk1,werror=stop,rerror=stop,cache=none -device virtio-blk-ccw,scsi=off,devno=fe.0.0001,drive=drive-virtio-disk1,id=virtio-disk1,write-cache=on -chardev pty,id=charserial0 -device sclpconsole,chardev=charserial0,id=serial0 -device virtio-balloon-ccw,id=balloon0,devno=fe.0.0002 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
The commits moving the attributes to -device instead of -drive only do so when the 'werror' attribute is present on the 'usb-storage' device, because it was the last device to gain that capability: commit a087a8e60ad62a719165e3c2c9970480b9531062 qemu: capabilities: Add capability for werror/rerror for 'usb-device' frontend commit bb4f3543bbf3ebbffa833ae7df55c298920243eb qemu: command: Format rerror/werror with -device instead of -drive
Upstream patches: https://www.redhat.com/archives/libvir-list/2020-April/msg00000.html
Pushed upstream as: commit 29fc9e96a9db30000137e648b775b6b08e05d43b Author: Ján Tomko <jtomko> CommitDate: 2020-04-01 15:45:48 +0200 qemu: use QEMU_CAPS_STORAGE_WERROR for disk-error attributes When moving the formatting of this attributes from -drive to -device, the QEMU_CAPS_USB_STORAGE_WERROR capability was used, because usb-storage was the last device to gain this capability. However this lead to the assumption that QEMU binaries without the usb-storage device do not support this, leading to breakage on s390x with blockdev. Fixes: bb4f3543bbf3ebbffa833ae7df55c298920243eb https://bugzilla.redhat.com/show_bug.cgi?id=1819250 Signed-off-by: Ján Tomko <jtomko> Reviewed-by: Peter Krempa <pkrempa> commit 47e84b06ec8181cbdb00c87bcc67f20edd9e949b Author: Ján Tomko <jtomko> CommitDate: 2020-04-01 15:45:48 +0200 qemu: add QEMU_CAPS_STORAGE_WERROR Detect the werror property on SCSI and virtio disks. But clear it if the QEMU supports usb-storage device without it also supporting this option for usb-storage. Signed-off-by: Ján Tomko <jtomko> Reviewed-by: Peter Krempa <pkrempa> commit 248d205171c9927b6d0e0077d2f3d9996c5540d9 Author: Ján Tomko <jtomko> CommitDate: 2020-04-01 15:45:47 +0200 tests: qemu: add disk-error-policy tests for s390x To demonstrate the move of these attributes from -drive to -device. Signed-off-by: Ján Tomko <jtomko> Reviewed-by: Peter Krempa <pkrempa> git describe: v6.2.0-rc1-17-g29fc9e96a9
Verified with libvirt-6.0.0-18.module+el8.2.1+6456+a6d62e4e.s390x Steps: 1. Use attached vm.xml 2. Use virsh to create qemu-kvm native representation 3. Check for (r|w)error to be present on device ==> -blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' -device virtio-blk-ccw,scsi=off,devno=fe.0.0001,drive=libvirt-1-format,id=virtio-disk1,write-cache=on,werror=stop,rerror=stop Full: # rpm -q libvirt && virsh domxml-to-native qemu-argv vm.xml|grep error libvirt-6.0.0-18.module+el8.2.1+6456+a6d62e4e.s390x LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin HOME=/var/lib/libvirt/qemu/domain--1-avocado-vt-vm5 XDG_DATA_HOME=/var/lib/libvirt/qemu/domain--1-avocado-vt-vm5/.local/share XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain--1-avocado-vt-vm5/.cache XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain--1-avocado-vt-vm5/.config QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name guest=avocado-vt-vm5,debug-threads=on -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain--1-avocado-vt-vm5/master-key.aes -machine s390-ccw-virtio-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu z13.2-base,aen=on,aefsi=on,msa5=on,msa4=on,msa3=on,msa2=on,msa1=on,sthyi=on,edat=on,ri=on,edat2=on,vx=on,ipter=on,ap=on,esop=on,apqi=on,apft=on,apqci=on,cte=on,bpb=on,ppa15=on,zpci=on,sea_esop2=on,te=on,cmm=on -m 1024 -overcommit mem-lock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 7965bc53-ee45-49fc-a497-0b05b1a0a995 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain--1-avocado-vt-vm5/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -blockdev '{"driver":"file","filename":"/var/lib/avocado/data/avocado-vt/images/jeos-27-s390x.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' -blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage"}' -device virtio-blk-ccw,scsi=off,devno=fe.0.0000,drive=libvirt-2-format,id=virtio-disk0,bootindex=1 -blockdev '{"driver":"file","filename":"/var/tmp/avocado_xzjhv4ny/mnt/libvirt-disk","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' -blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' -device virtio-blk-ccw,scsi=off,devno=fe.0.0001,drive=libvirt-1-format,id=virtio-disk1,write-cache=on,werror=stop,rerror=stop -chardev pty,id=charserial0 -device sclpconsole,chardev=charserial0,id=serial0 -device virtio-balloon-ccw,id=balloon0,devno=fe.0.0002 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
Hi Jarda, As the ZTR is set to 8.2.0, but no z stream bug cloned till now, could you please clone it to 8.2.0.z? Or just clear the ZTR if it will not target to 8.2.0.z. Thank you!
Setting z-stream- Justification: RHEL 8.3 virt:rhel will be based on RHEL-AV 8.2.1, which has that fix already. Right now there is no way for IBM Z to consume RHEL-AV outside of "irregular" or non-standard shipment. With that having the fix in RHEL-AV 8.2.1 is sufficient.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3172