Bug 2041227

Summary: I/O error on virtual disk in LVM pool
Product: [Fedora] Fedora Reporter: Christoph Karl <pampelmuse>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED DUPLICATE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 35CC: acaringi, adscvr, airlied, alciregi, bskeggs, hdegoede, jarodwilson, jeremy, jglisse, jonathan, josef, kernel-maint, lgoncalv, linville, masami256, mchehab, ptalbert, steved
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-02-08 18:24:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Setup script for the VM
none
kickstart file none

Description Christoph Karl 2022-01-16 15:37:25 UTC 
Created attachment 1851132 [details]
Setup script for the VM

Description of problem:
There are lots if I/O errors in dmesg.
Sometimes the whole VM freezes.


Version-Release number of selected component (if applicable):
Very unsure about where the problem is.
I took kernel, because dmesg reports the problem.


How reproducible:
Not always, but will be triggered if the I/O load is high.


Steps to Reproduce:
1. Setup fresh VM (setup-script and kickstart file attached)
2. Generate I/O load: eg. dnf update --refresh
3. dmesg

Actual results:
[   86.056173] blk_update_request: I/O error, dev vda, sector 3463152 op 0x1:(WRITE) flags 0x0 phys_seg 2 prio class 0
[   86.056217] blk_update_request: I/O error, dev vda, sector 3463168 op 0x1:(WRITE) flags 0x4000 phys_seg 254 prio class 0
[   86.056228] blk_update_request: I/O error, dev vda, sector 3465200 op 0x1:(WRITE) flags 0x0 phys_seg 2 prio class 0
[   86.056229] EXT4-fs warning (device vda2): ext4_end_bio:342: I/O error 10 writing to inode 526364 starting block 433152)
and so on


Expected results:
No errors


Additional info:
The PV is an external (USB) SSD disk, host is a laptop with F35.
The problem is not reproducible if the virtual disk is NOT in a LVM pool, but is a qcow2 file.
There are no I/O errors (or others) on the host system.
I am not able to trigger I/O errors on the host system mounting an LV from this PV (so the external disk seems to be OK, also checked with smartctl).

(Cannot use F35 inside the VM, because https://bugzilla.redhat.com/show_bug.cgi?id=2032085)
Comment 1 Christoph Karl 2022-01-16 15:38:01 UTC 
Created attachment 1851133 [details]
kickstart file
Comment 2 Christoph Karl 2022-01-16 17:08:37 UTC 
Just reproduced with rawhide (Fedora-Everything-netinst-x86_64-Rawhide-20220114.n.0.iso):
[   45.804981] I/O error, dev vda, sector 3130392 op 0x1:(WRITE) flags 0x0 phys_seg 34 prio class 0
[   45.805004] I/O error, dev vda, sector 3131392 op 0x1:(WRITE) flags 0x4000 phys_seg 98 prio class 0
[   45.805008] I/O error, dev vda, sector 3133952 op 0x1:(WRITE) flags 0x4000 phys_seg 102 prio class 0
[   45.805011] I/O error, dev vda, sector 3136512 op 0x1:(WRITE) flags 0x0 phys_seg 57 prio class 0
[   45.805013] EXT4-fs warning (device vda2): ext4_end_bio:342: I/O error 10 writing to inode 132855 starting block 392064)
and so on
Comment 3 Christoph Karl 2022-01-17 14:16:46 UTC 
Just reproduced with an external (USB) HDD disc:
[  106.963337] blk_update_request: I/O error, dev vda, sector 5786624 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[  106.963390] blk_update_request: I/O error, dev vda, sector 5787648 op 0x1:(WRITE) flags 0x4000 phys_seg 1 prio class 0
[  106.963444] blk_update_request: I/O error, dev vda, sector 5790208 op 0x1:(WRITE) flags 0x4000 phys_seg 1 prio class 0
[  106.963493] blk_update_request: I/O error, dev vda, sector 5792768 op 0x1:(WRITE) flags 0x4000 phys_seg 1 prio class 0
[  106.963544] blk_update_request: I/O error, dev vda, sector 5795328 op 0x1:(WRITE) flags 0x4000 phys_seg 1 prio class 0
[  106.963617] blk_update_request: I/O error, dev vda, sector 5797888 op 0x1:(WRITE) flags 0x4000 phys_seg 1 prio class 0
[  106.963664] blk_update_request: I/O error, dev vda, sector 5800448 op 0x1:(WRITE) flags 0x4000 phys_seg 1 prio class 0
[  106.963711] blk_update_request: I/O error, dev vda, sector 5803008 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[  106.963722] EXT4-fs warning (device dm-0): ext4_end_bio:342: I/O error 10 writing to inode 1867 starting block 210944)
Comment 4 Christoph Karl 2022-01-30 08:34:08 UTC 
An intensive test with 100 VMs and all filesystem qcow2 was successful.
Comment 5 Christoph Karl 2022-01-30 08:35:51 UTC 
The error is reproducible on another host (also a laptop).
Comment 6 Christoph Karl 2022-01-30 08:38:37 UTC 
The error is not reproducible on the internal disk (PCI M2 SSD).
Comment 7 Christoph Karl 2022-02-08 17:43:33 UTC 
Played arround with
virt-install
  --disk vol=extssd/fedora-00,io=io_uring
and other setting but none worked.
Comment 8 Christoph Karl 2022-02-08 18:24:13 UTC 

*** This bug has been marked as a duplicate of bug 2026747 ***