Fedora Account System
Red Hat Associate
Red Hat Customer
Created attachment 1549328 [details] journalctl --no-hostname -k > dmesg.txt 1. Please describe the problem: Since upgrade of kernel to 5.0.3 in Fedora 29, my desktop has become unresponsive when copying large files from a HDD to a NFS mounted filesystem. Investigating I found that the set of available io schedulers has been changed. Previously (on 4.x kernels) the set of available schedulers (compiled into kernel) were: noop, deadline, cfq, mq-deadline. For my desktop having classic SATA SSD (sda) and HDD (sdb), the "cfq" was choosen by default. Now (on 5.0.3 kernel) the set of available schedulers (compiled into kernel) are: none, mq-deadline, with default choosen at boot being "mq-deadline" on my machine. This one causes almost total system lockups (Window system and GUI unresponsive) for the entire time of copying large files from HDD. I tried other io schedulers that ship as modules (bfq, kyber) and they were not any better in keeping the machine responsive during IO. The only one that keeps my machine responsive now is "none" (is this the same as noop in 4.x kernels?). I found that appending "elevator=none" to the kernel startup line in grub does not affect the chosen io scheduler (mq-deadline is still choosen). Currently I work around by placing a file with the following content into the /etc/udev/rules.d: ACTION=="add|change", SUBSYSTEM=="block" , KERNEL=="sdb*", RUN+="/bin/sh -c 'echo none > /sys/block/sdb/queue/scheduler'" ACTION=="add|change", SUBSYSTEM=="block" , KERNEL=="sda*", RUN+="/bin/sh -c 'echo none > /sys/block/sda/queue/scheduler'" If this behavior is consistent for SATA HDD drives then perhaps the kernel should choose "none" by default when it detects such drive or many people will not want to install Fedora any more. 2. What is the Version-Release number of the kernel: 5.0.3-200.fc29.x86_64 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at https://koji.fedoraproject.org/koji/packageinfo?packageID=8 : 5.0.3-200.fc29.x86_64 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: Boot with 5.0.3-200.fc29.x86_64 on a machine having classic SATA HDD then copy a large file from the HDD to the network (or do dd if=large_file of=/dev/null) and then check the responsiveness of the desktop while IO is occurring. 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: not tried yet - will try. 6. Are you running any modules that not shipped with directly Fedora's kernel?: no 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag.
I updated Fedora 29 to kernel 5.0.3 on another desktop with similar configuration (SATA SSD + SATA HDD) and I observed no such problems (no lockups) with mq-deadline io scheduler there. The differences between those two desktops that I can think of are: The PC with lockups: - older CPU (i7-2600K) - HDD/SSD are: SSD: [ 2.063941] sd 2:0:0:0: [sda] 351651888 512-byte logical blocks: (180 GB/168 GiB) HDD: [ 8.474454] sd 2:0:1:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB) The PC with not lockups: - newer CPU (i7-4771) - HDD/SSD are: SSD: [ 1.554146] sd 1:0:0:0: [sdb] 488397168 512-byte logical blocks: (250 GB/233 GiB) HDD: [ 1.245587] sd 0:0:0:0: [sda] 976773168 512-byte logical blocks: (500 GB/466 GiB)
Moving the HDD from SATA 3 to SATA 6 port on the Motherboard fixes the lockups. Now even with mq-deadline io scheduler the system stays responsive when big files are copied from/to HDD...
So previously, the SSD/HDD were (SSD on SATA 6 port, HDD on SATA 3 port): SSD: [ 2.063941] sd 2:0:0:0: [sda] 351651888 512-byte logical blocks: (180 GB/168 GiB) HDD: [ 8.474454] sd 2:0:1:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB) ...and this caused lockups, now they are (SSD on SATA 6 port, HDD on SATA 6 port): SSD: [ 2.763960] sd 2:0:0:0: [sda] 351651888 512-byte logical blocks: (180 GB/168 GiB) HDD: [ 2.764260] sd 3:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
I'm just stunned to see how recklessly this update was thrown on the users. This new block multiqueue scheduler is completely useless for anything except perhaps watching videos on notebook. Checking 2 TB Btrfs file system with RAID-1 profile took just 20 minutes on kernel 4.20 before but it takes over an hour just to check a small fraction of its extents on new kernel 5.0.4. I've tried every available new scheduler: mq-deadline, bfq, kyber and none - but to no avail. And there is no old cfq anymore to back-up. All threads with I/O requests are just sitting there idle doing nothing, neither using CPU nor I/O activity. Even editing a tiny text file in Vim results in Vim hanging, not able to save the file. Now I have random lock-ups from screen saver under I/O load. Using LVM with thin provisioning for BTRS RAID volumes isn't possible with blk_mq, period. I've restored kernel-4.20.16 and blocked any new updates to it until it's resolved.
For the original reporter, is your workaround acceptable? There were major io changes as you noted so I think further pinpointing the issue is going to require discussion upstream. There were also supposed to be more improvements with 5.1 so it would be worth testing a 5.1-rc rawhide kernel to see if that improves anything.
This is now fixed for me in kernel-5.0.6. Looks like Jens Axboe's commit did the trick. And thanks to Laura Abbott for pushing it to Koji so soon, so the problem could be mitigated now for many users, I hope.
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There are a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 29 kernel bugs. Fedora 29 has now been rebased to 5.2.9-100.fc29. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 30, and are still experiencing this issue, please change the version to Fedora 30. If you experience different issues, please open a new bug report for those.
*********** MASS BUG UPDATE ************** This bug is being closed with INSUFFICIENT_DATA as there has not been a response in 3 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.
I'm using Fedora 30 now with latest updated kernel and am not experiencing this problem any more. Thanks.