Bug 1693965
| Summary: | Lockups/unresponsivenes with new default io scheduler in kernel 5.0.3 | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Peter Levart <peter.levart> | ||||
| Component: | kernel | Assignee: | Kernel Maintainer List <kernel-maint> | ||||
| Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 29 | CC: | airlied, bskeggs, hdegoede, ichavero, itamar, jarodwilson, jeremy, jglisse, john.j5live, jonathan, josef, kernel-maint, labbott, linville, mchehab, mjg59, steved, vvs009 | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-09-17 20:01:52 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Peter Levart
2019-03-29 07:51:18 UTC
I updated Fedora 29 to kernel 5.0.3 on another desktop with similar configuration (SATA SSD + SATA HDD) and I observed no such problems (no lockups) with mq-deadline io scheduler there. The differences between those two desktops that I can think of are:
The PC with lockups:
- older CPU (i7-2600K)
- HDD/SSD are:
SSD: [ 2.063941] sd 2:0:0:0: [sda] 351651888 512-byte logical blocks: (180 GB/168 GiB)
HDD: [ 8.474454] sd 2:0:1:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
The PC with not lockups:
- newer CPU (i7-4771)
- HDD/SSD are:
SSD: [ 1.554146] sd 1:0:0:0: [sdb] 488397168 512-byte logical blocks: (250 GB/233 GiB)
HDD: [ 1.245587] sd 0:0:0:0: [sda] 976773168 512-byte logical blocks: (500 GB/466 GiB)
Moving the HDD from SATA 3 to SATA 6 port on the Motherboard fixes the lockups. Now even with mq-deadline io scheduler the system stays responsive when big files are copied from/to HDD... So previously, the SSD/HDD were (SSD on SATA 6 port, HDD on SATA 3 port):
SSD: [ 2.063941] sd 2:0:0:0: [sda] 351651888 512-byte logical blocks: (180 GB/168 GiB)
HDD: [ 8.474454] sd 2:0:1:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
...and this caused lockups, now they are (SSD on SATA 6 port, HDD on SATA 6 port):
SSD: [ 2.763960] sd 2:0:0:0: [sda] 351651888 512-byte logical blocks: (180 GB/168 GiB)
HDD: [ 2.764260] sd 3:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
I'm just stunned to see how recklessly this update was thrown on the users. This new block multiqueue scheduler is completely useless for anything except perhaps watching videos on notebook. Checking 2 TB Btrfs file system with RAID-1 profile took just 20 minutes on kernel 4.20 before but it takes over an hour just to check a small fraction of its extents on new kernel 5.0.4. I've tried every available new scheduler: mq-deadline, bfq, kyber and none - but to no avail. And there is no old cfq anymore to back-up. All threads with I/O requests are just sitting there idle doing nothing, neither using CPU nor I/O activity. Even editing a tiny text file in Vim results in Vim hanging, not able to save the file. Now I have random lock-ups from screen saver under I/O load. Using LVM with thin provisioning for BTRS RAID volumes isn't possible with blk_mq, period. I've restored kernel-4.20.16 and blocked any new updates to it until it's resolved. For the original reporter, is your workaround acceptable? There were major io changes as you noted so I think further pinpointing the issue is going to require discussion upstream. There were also supposed to be more improvements with 5.1 so it would be worth testing a 5.1-rc rawhide kernel to see if that improves anything. This is now fixed for me in kernel-5.0.6. Looks like Jens Axboe's commit did the trick. And thanks to Laura Abbott for pushing it to Koji so soon, so the problem could be mitigated now for many users, I hope. *********** MASS BUG UPDATE ************** We apologize for the inconvenience. There are a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 29 kernel bugs. Fedora 29 has now been rebased to 5.2.9-100.fc29. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 30, and are still experiencing this issue, please change the version to Fedora 30. If you experience different issues, please open a new bug report for those. *********** MASS BUG UPDATE ************** This bug is being closed with INSUFFICIENT_DATA as there has not been a response in 3 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously. I'm using Fedora 30 now with latest updated kernel and am not experiencing this problem any more. Thanks. |