Bug 2026747
| Summary: | I/O errors on f34 s390x kvm guests under f35 hypervisor | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Kevin Fenzi <kevin> | ||||||
| Component: | qemu | Assignee: | Fedora Virtualization Maintainers <virt-maint> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||
| Severity: | unspecified | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 35 | CC: | berrange, cfergeau, crobinso, ondrejj, pampelmuse, pbonzini, philmd, rjones, smitterl, virt-maint | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | qemu-6.1.0-14.fc35 | Doc Type: | If docs needed, set a value | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2022-02-15 01:37:44 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
|
Description
Kevin Fenzi
2021-11-25 17:51:28 UTC
If you're using libvirt, please provide the libvirt XML config and the corresponding QEMU log file /var/log/libvirt/qemu/$GUEST.log. If using QEMU direct, just the full QEMU command line you use. Created attachment 1843618 [details]
libvirt xml for one of the guests
Here's the libvirt xml for a guest.
I did also try changing between the s390-ccw-virtio-6.1 (what f35 virt-install puts there) and s390-ccw-virtio-4.1 (what the f34 virt-install has there) and that didn't seem to matter.
Created attachment 1843619 [details]
log for guest
Here's the /var/log/libvirt/qemu/buildvm-s390x-23.s390.fedoraproject.org.log
Hi Kevin! We've seen a similar problem in downstream RHEL ... The fix is likely https://gitlab.com/qemu-project/qemu/-/commit/cc071629539dc1f303175a7 ... do you happen to know how to rebuild the package to have a try whether that fixes the issue for you? (In reply to Thomas Huth from comment #6) > Hi Kevin! We've seen a similar problem in downstream RHEL ... The fix is > likely https://gitlab.com/qemu-project/qemu/-/commit/cc071629539dc1f303175a7 > ... do you happen to know how to rebuild the package to have a try whether > that fixes the issue for you? Agreed. A) Could you try with qemu 6.2-rc to see if it still reproduces? B) Can you confirm a) the device attached to target 'vda' is backed by virtio-blk and a dasd block device (/dev/dasdX)? b) if so, with qemu 6.1 version you mention this should reproduce only easily reproduced without nested: 1. attach disk /dev/dasdX as virtio-blk to L1 guest 2. inside L1 guest format and create filesystem ext4 3. read write, check logs for I/O errors Here is a scrath build with the mentioned upstream fix: https://koji.fedoraproject.org/koji/taskinfo?taskID=79265813 if you confirm whether that solves the I/O errors, I'll do a formal build I know everyone's talking about s390x and dasd's and such, but is it possible this same bug could affect armv7 guest on aarch64 host (Fedora Rawhide in both cases)? For about a month I've been seeing errors like this inside the guest, and they look very similar to this BZ. Also Dan if there's a scratch building including aarch64 I could test it. [16784.102801] blk_update_request: I/O error, dev vda, sector 9689856 op 0x1:(WRITE) flags 0x800 phys_seg 16 prio class 0 [16784.130038] blk_update_request: I/O error, dev vda, sector 9690368 op 0x1:(WRITE) flags 0x4800 phys_seg 80 prio class 0 [16784.134209] blk_update_request: I/O error, dev vda, sector 9692928 op 0x1:(WRITE) flags 0x4800 phys_seg 80 prio class 0 [16784.138263] blk_update_request: I/O error, dev vda, sector 9695488 op 0x1:(WRITE) flags 0x800 phys_seg 96 prio class 0 [16784.142771] blk_update_request: I/O error, dev vda, sector 9696752 op 0x1:(WRITE) flags 0x4800 phys_seg 205 prio class 0 [16784.146855] blk_update_request: I/O error, dev vda, sector 9699312 op 0x1:(WRITE) flags 0x800 phys_seg 52 prio class 0 [16784.150877] blk_update_request: I/O error, dev vda, sector 9700136 op 0x1:(WRITE) flags 0x4800 phys_seg 157 prio class 0 [16784.155033] blk_update_request: I/O error, dev vda, sector 9702696 op 0x1:(WRITE) flags 0x4800 phys_seg 80 prio class 0 [16784.159096] blk_update_request: I/O error, dev vda, sector 9705256 op 0x1:(WRITE) flags 0x800 phys_seg 19 prio class 0 [16784.163163] blk_update_request: I/O error, dev vda, sector 9705864 op 0x1:(WRITE) flags 0x4800 phys_seg 80 prio class 0 [16784.188541] vda3: writeback error on inode 4750392, offset 43192320, sector 9682176 (In reply to Richard W.M. Jones from comment #9) > I know everyone's talking about s390x and dasd's and such, but is it > possible this same bug could affect armv7 guest on aarch64 host (Fedora > Rawhide in both cases)? For about a month I've been seeing errors > like this inside the guest, and they look very similar to this BZ. Which was filed as: https://bugzilla.redhat.com/show_bug.cgi?id=2004120 Scratch build with the patch, on top of Rawhide, all arches: https://koji.fedoraproject.org/koji/taskinfo?taskID=79266649 Can confirm it fixes the armv7/aarch64 problem here. *** Bug 2004120 has been marked as a duplicate of this bug. *** Build for Rawhide: https://koji.fedoraproject.org/koji/taskinfo?taskID=79268037 Build for F35: https://koji.fedoraproject.org/koji/taskinfo?taskID=79268098 (In reply to Richard W.M. Jones from comment #14) > Build for F35: > https://koji.fedoraproject.org/koji/taskinfo?taskID=79268098 The GCC / systemtap / armv7 bug of doom is now affecting F35 too. New F35 build including my workaround for that bug: https://koji.fedoraproject.org/koji/taskinfo?taskID=79268858 FEDORA-2021-12f6c46ad8 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-12f6c46ad8 FEDORA-2021-12f6c46ad8 has been pushed to the Fedora 35 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-12f6c46ad8` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-12f6c46ad8 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates. I've upgraded to that version of qemu and restarted a guest. Will see how it looks in a while. :) Thanks for the quick fixes. So far that guest is fine. :) So, looks good here... Seems to be a duplicate of this bug: https://bugzilla.redhat.com/show_bug.cgi?id=2041227 *** Bug 2041227 has been marked as a duplicate of this bug. *** FEDORA-2022-3a60c34473 has been pushed to the Fedora 35 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-3a60c34473` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-3a60c34473 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates. FEDORA-2022-3a60c34473 has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report. |