Bug 2174676
| Summary: | Guest hit EXT4-fs error on host 4K disk when repeatedly hot-plug/unplug running IO disk [RHEL9] | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | qing.wang <qinwang> |
| Component: | qemu-kvm | Assignee: | Hanna Czenczek <hreitz> |
| qemu-kvm sub component: | virtio-blk,scsi | QA Contact: | qing.wang <qinwang> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | medium | CC: | coli, esandeen, hreitz, jinzhao, juzhang, kwolf, lijin, mrezanin, qzhang, vgoyal, virt-maint, xuwei, yfu, zhenyzha |
| Version: | 9.2 | Keywords: | Triaged |
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | qemu-kvm-8.0.0-10.el9 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 2141964 | Environment: | |
| Last Closed: | 2023-11-07 08:27:12 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 2141964 | ||
| Bug Blocks: | 2140910 | ||
|
Comment 1
qing.wang
2023-03-02 06:50:59 UTC
Can you please test whether http://brew-task-repos.usersys.redhat.com/repos/scratch/hreitz/qemu-kvm/7.2.0/10.el9_2.hreitz202303031240/ passes the test? Thanks! (In reply to Hanna Czenczek from comment #3) > Can you please test whether > http://brew-task-repos.usersys.redhat.com/repos/scratch/hreitz/qemu-kvm/7.2. > 0/10.el9_2.hreitz202303031240/ passes the test? Thanks! It passes on the following version. (running over 40 hours) Red Hat Enterprise Linux release 9.2 Beta (Plow) 5.14.0-279.el9.x86_64 qemu-kvm-7.2.0-10.el9_2.hreitz202303031240.x86_64 seabios-bin-1.16.1-1.el9.noarch edk2-ovmf-20221207gitfff6d81270b5-6.el9.noarch virtio-win-prewhql-0.1-234.iso Summarizing a discussion Qing Wang and me had on Slack about whether this has customer impact or not: It’s hard to say whether customers have faced or will face this bug. It can only appear if, on a host 4k disk, the guest issues 512-byte-aligned O_DIRECT requests with an extremely large number of buffers in the vector (precisely 1023 or 1024), which to me sounds like it will only happen in benchmarks. If guests do heavy I/O, they’re well-advised to have the guest block size match the host block size, or qemu will, to the detriment of performance, have to pad the requests, which is where the bug is. Furthermore, if the guest does any caching at all, which I think most non-benchmarking applications will do one way or another (be it through the guest page cache or some custom cache), the cache entries will probably be larger than 512 bytes, e.g. the page cache has 4k-sized entries. They’ll probably be aligned to their size, which would then make the bug not appear if the alignment is at least 4k. But there’s a “but”: If you indeed have for one reason or another (e.g. migration) configured your guest to show a sector size of 512, while on a 4k-sectored host disk, and you run heavy O_DIRECT I/O that uses extremely long I/O vectors and that is not aligned to 4k, then it is possible that very rarely this bug manifests. Comment 0 shows one such application (which is a benchmark, i.e. not a real-world application), and it takes 100 tries to see the bug with >50 %. I can’t imagine this is something that customers do, but my imagination is of course very limited. Now, we haven’t had any bug reports but this (and BZ 2141964 for 8.x), but at least upstream qemu has had the bug since December 2020. However, it is reasonable to assume that nobody reported this bug because it appears so rarely and manifests just as an I/O error, which might be anything else, especially if it’s basically impossible to reproduce. All in all, I think this bug has not impacted customers so far, but it is impossible to rule out. In any case, we should fix it rather sooner than later, not least because 4k-sectored disks are becoming more and more common. --- Upstream, fixes are here: https://lists.nongnu.org/archive/html/qemu-block/2023-04/msg00186.html – they are fully reviewed, but not merged yet. Hit this issue on Red Hat Enterprise Linux release 9.3 Beta (Plow) 5.14.0-312.el9.x86_64 qemu-kvm-8.0.0-2.el9.x86_64 seabios-bin-1.16.1-1.el9.noarch edk2-ovmf-20230301gitf80f052277c8-3.el9.noarch libvirt-9.0.0-10.1.el9_2.x86_64 http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section2/images_backup/qbugs/2174676/2023-05-17/ auto script: python ConfigTest.py --testcase=hotplug_unplug_during_io_repeat.default.q35 --iothread_scheme=roundrobin --nr_iothreads=2 --platform=x86_64 --guestname=RHEL.9.3.0 --driveformat=virtio_blk --nicmodel=virtio_net --imageformat=qcow2 --machines=q35 --customsparams="vm_mem_limit = 12G\nimage_aio=threads" --firmware=ovmf --netdst=virbr0 QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass. Passed on Red Hat Enterprise Linux release 9.3 Beta (Plow) 5.14.0-348.el9.x86_64 qemu-kvm-8.0.0-10.el9.x86_64 seabios-bin-1.16.1-1.el9.noarch edk2-ovmf-20230524-2.el9.noarch libvirt-9.3.0-2.el9.x86_64 virtio-win-prewhql-0.1-240.iso python ConfigTest.py --testcase=hotplug_unplug_during_io_repeat.default,block_io_with_unaligned_offset --iothread_scheme=roundrobin --nr_iothreads=2 --platform=x86_64 --guestname=RHEL.9.3.0 --driveformat=virtio_blk,virtio_scsi --imageformat=qcow2 --machines=q35 --customsparams="vm_mem_limit = 12G\nimage_aio=native" --firmware=default_bios --netdst=virbr0 --nrepeat=20 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:6368 |