Bug 2017148 - BFQ unstable; revert back to mq-deadline
Summary: BFQ unstable; revert back to mq-deadline
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 34
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-10-25 18:56 UTC by George Petasis
Modified: 2023-01-24 00:37 UTC (History)
24 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2022-06-07 22:49:40 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description George Petasis 2021-10-25 18:56:52 UTC
1. Please describe the problem:

PC constantly freezes.

2. What is the Version-Release number of the kernel:

All versions for the last 2.5 years.


3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :

I do not know, I suspect the problem started with Fedora 31, where "bfq" was made default disk scheduler.

4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:

Just install Fedora, and wait a couple of days.

5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:

Yes, it affects all kernels.

6. Are you running any modules that not shipped with directly Fedora's kernel?:

No.

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.

Several logs available here:
https://www.dropbox.com/sh/wbhadgrtlsqmv83/AADDrAntGjCK5NelrRCXBFWHa?dl=0

Comment 1 George Petasis 2021-10-25 19:15:28 UTC
I just want to add that this bug has been causing me a severe instability problem, and I have spent money unnecessarily in trying to fix it, as I thought I had a hardware problem instead of a software one.

When my initial system started being unstable (exhibiting kernel freezes), I thought it was due to system age, thus I replace everything except psu & disks.
But kernel freezes continue with the new system for more than a year. I finally decided to RMA the processor, and complained about the bad RMA process in a forum, where a user noticed that all my freezes are due to functions starting with "bfq_".
Then I learned what bfq is, and that Fedora made it default in 31.
Which explains all my freezes, for about two years now.

I have started receiving random freezes to two more fedora 34 systems, without anything written in the logs, and I suspect the cause is the same: bfq!

Comment 2 Justin M. Forbes 2021-11-02 12:28:32 UTC
https://www.kernel.org/doc/html/latest/block/switching-sched.html offers a great way to test that theory.

Comment 3 George Petasis 2021-11-02 14:00:59 UTC
Not a theory, a fact. I have switched to mq-deadline, and not a single freeze for 19 days.
In fact I have switched ALL my fedora installs.

BFQ is buggy.

Fedora should either get rid of bfq from default, or provide an easy way to switch to another scheduler.
Filling crontab with echo, one for each disk, is silly.

Comment 4 George Petasis 2021-11-22 11:13:38 UTC
I just found another similar bug: https://bugzilla.redhat.com/show_bug.cgi?id=1851783

Comment 5 Yaroslav Sidlovsky 2021-11-22 13:15:44 UTC
Thanks for the information!
I've seen very similar freezes on my ASUS ROG CROSSHAIR VII mother board with Ryzen 3600 CPU on Fedora 34.
I'll try to disable bfq and watch if system stability will increase.

Comment 6 Chris Murphy 2021-11-22 14:19:28 UTC
Maybe there's some sysfs tracing that can be enabled and capture the problem as it happens?

Comment 7 George Petasis 2021-11-24 16:39:04 UTC
Perhaps, but I do not know how to do it. Is there a guide?
Just a reminder, in my initial posts I have included several logs that managed to write something before the freeze.

Comment 8 Chris Murphy 2021-11-26 17:24:44 UTC
From one of the logs:
>Οκτ 03 02:09:14.473516 socialwebobservatory.iit.demokritos.gr kernel: CPU: 5 PID: 1139508 Comm: kworker/u64:2 Tainted: P           OE     5.13.19-200.fc34.x86_64 #1

I suggest updating the kernel to 5.14 or 5.15 series, because 5.13 is no longer maintained. I'm not sure if what you've hit is related to this bfq bug: https://bugzilla.kernel.org/show_bug.cgi?id=214503 but it's just easier to try a newer kernel and see if the problem is already fixed.

Comment 9 Chris Murphy 2021-11-26 17:33:44 UTC
Of course, still another alternative is to just change the io scheduler.

The file affecting this is:

$ sudo cat /usr/lib/udev/rules.d/60-block-scheduler.rules
# do not edit this file, it will be overwritten on update

ACTION=="add", SUBSYSTEM=="block", \
  KERNEL=="mmcblk*[0-9]|msblk*[0-9]|mspblk*[0-9]|sd*[!0-9]|sr*", \
  ENV{DEVTYPE}=="disk", \
  ATTR{queue/scheduler}="bfq"


Which you can change by creating

$cat /etc/udev/rules.d/60-block-scheduler.rules
ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="sd[a-z]|mmcblk[0-9]*|loop[0-9]*", ENV{DEVTYPE}=="disk", ATTR{queue/scheduler}="mq-deadline"


The downside is possibly there's a legit bfq bug that needs to get figured out and fixed, and the above just papers over that problem by avoiding it.

Comment 10 Ben Cotton 2022-05-12 15:55:52 UTC
This message is a reminder that Fedora Linux 34 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 34 on 2022-06-07.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '34'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 34 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 11 Ben Cotton 2022-06-07 22:49:40 UTC
Fedora Linux 34 entered end-of-life (EOL) status on 2022-06-07.

Fedora Linux 34 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 12 Brothad45 2023-01-24 00:37:12 UTC
Many people do the Christmas Savings Challenge with each other. If you do not know about it, then read this post carefully and you will know what it actually is. These people start saving from their income for next Christmas. While I was reading this post, I found a link to the website of https://socanadiancasino.com/casino-reviews/goodman/ best goodman casino review Canada  and I shared that link with my friend. I asked him that this is some website. He told me that this is a review page of one of the best casino gambling websites that you sent me the link to. I have also heard a lot about this website.


Note You need to log in before you can comment on or make changes to this bug.