This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2180423 - kworker/1:0+cifsiod is using 100% of a CPU core and freezes my server
Summary: kworker/1:0+cifsiod is using 100% of a CPU core and freezes my server
Keywords:
Status: CLOSED MIGRATED
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: cifs-utils
Version: CentOS Stream
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Nobody
QA Contact: xiaoli feng
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-03-21 12:58 UTC by daniel_ulmer
Modified: 2023-09-23 11:35 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-09-23 11:35:38 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Its a screenshot of the process which is running wild (162.53 KB, image/png)
2023-03-21 12:58 UTC, daniel_ulmer
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker   RHEL-7939 0 None Migrated None 2023-09-23 11:35:35 UTC
Red Hat Issue Tracker RHELPLAN-152568 0 None None None 2023-03-21 13:01:01 UTC

Description daniel_ulmer 2023-03-21 12:58:19 UTC
Created attachment 1952401 [details]
Its a screenshot of the process which is running wild

Description of problem:
My server was running on CentOS9 and everything worked perfectly. Until one day the system told me there is an operating system update. I updated and restartet the PC and after running on the new update the pc randomly freezes and i can't use it anymore and have to plug the cable because it wouldn't turn off by itself. And if i look which process is causing trouble its kworker/1:0cifsiod

My theory is its a cifs problem because in var/log/messages it shows:

Mar 20 11:26:00 ersatzwinserver kernel: watchdog: BUG: soft lockup - CPU#1 stuck for 805s! [kworker/1:1:149]
Workqueue: cifsiod smb2_reconnect_server [cifs]
Mar 20 11:26:00 ersatzwinserver kernel: RIP: 0010:native_queued_spin_lock_slowpath.part.0+0x42/0x1a0
Mar 20 11:26:00 ersatzwinserver kernel: Code: 0f ba 2a 08 0f 92 c1 8b 02 0f b6 c9 c1 e1 08 30 e4 09 c8 a9 00 01 ff ff 0f 85 f5 00 00 00 85 c0 74 0e 8b 02 84 c0 74 08 f3 90 <8b> 02 84 c0 75 f8 b8 01 00 00 00 66 89 02 c3 cc cc cc cc 8b 37 b8
Mar 20 11:26:00 ersatzwinserver kernel: RSP: 0018:ffffa91e40667d78 EFLAGS: 00000202
Mar 20 11:26:00 ersatzwinserver kernel: RAX: 0000000000000101 RBX: ffff9907437c2800 RCX: 0000000000000000
Mar 20 11:26:00 ersatzwinserver kernel: RDX: ffff9907905aa028 RSI: 0000000000000000 RDI: ffff9907905aa028
Mar 20 11:26:00 ersatzwinserver kernel: RBP: ffff9907905aa028 R08: ffffa91e40667e38 R09: 0000000000000000
Mar 20 11:26:00 ersatzwinserver kernel: R10: ffffa91e40667ce8 R11: 0000000000000018 R12: 0000000000000000
Mar 20 11:26:00 ersatzwinserver kernel: R13: ffff9907905aa000 R14: 000000000000ffff R15: 000000000000ffff
Mar 20 11:26:00 ersatzwinserver kernel: FS:  0000000000000000(0000) GS:ffff9916be340000(0000) knlGS:0000000000000000
Mar 20 11:26:00 ersatzwinserver kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 20 11:26:00 ersatzwinserver kernel: CR2: 000055a5b88c2000 CR3: 0000000fe4410004 CR4: 00000000003706e0
Mar 20 11:26:00 ersatzwinserver kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 20 11:26:00 ersatzwinserver kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

Comment 2 fabien.magagnosc@tecnotree.com 2023-03-22 02:30:37 UTC
Hi there,


we got the same issue (CentOS Stream 9), the CPUX is stucks, we can see massive CPU usage on a single kworker+cifsiod
And we find that the culpid (after testing multiple CIFS version, even SMB) was the upgrade of the cifs-utils, from 6.14-1.el9 to 7.0-1.el9 which happened before the week end (we didn't saw it then, no load)
Here some details, maybe to help :


We are mounting a Windows 2012 server, using CIFS 3.0 : 
//server/directory /mnt/directory cifs nobrl,rw,noperm,user=XXXXX,pass=XXXXX,mfsymlinks,uid=2000,gid=2000,dir_mode=0755,file_mode=0755,vers=3.0 0 0


Here, the latest kernel as per the 22th of March 2023 installed :
Linux WWWWW 5.14.0-285.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Mar 7 17:32:48 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux


Here, the situation DNF wise (after we locked to the previous cifs-utils)

Installed Packages
cifs-utils.x86_64                                                                                                      6.14-1.el9                                                                                                      @baseos
Available Packages
cifs-utils.x86_64                                                                                                      6.11-5.el9                                                                                                      baseos 
cifs-utils.x86_64                                                                                                      6.14-1.el9                                                                                                      baseos 
cifs-utils.x86_64                                                                                                      7.0-1.el9                                                                                                       baseos

Comment 3 xiaoli feng 2023-05-15 07:13:50 UTC
Hello,

There is a similar bug bz2177562 and it is fixed from kernel-5.14.0-301.el9. Could you try to test on kernel-5.14.0-301.el9 or later version?

Thanks.

Comment 4 fabien.magagnosc@tecnotree.com 2023-05-15 07:27:04 UTC
@xiaoli feng => To be clear, you means update :
   _ udapte the CIFS utils pacakges
   _ update the kernel to at least 5.14.0-301.el9

isn't it ?


Right now this is what I run perfectly stable :


[tecnotree@teamcity-repository9 ~]$ sudo dnf --showduplicates list cifs-utils
Last metadata expiration check: 2:10:27 ago on Mon 15 May 2023 08:12:29 AM EEST.
Installed Packages
cifs-utils.x86_64                                                                            6.14-1.el9                                                                             @baseos
Available Packages
cifs-utils.x86_64                                                                            6.11-5.el9                                                                             baseos 
cifs-utils.x86_64                                                                            6.14-1.el9                                                                             baseos 
cifs-utils.x86_64                                                                            7.0-1.el9                                                                              baseos 
[tecnotree@teamcity-repository9 ~]$ uname -a
Linux teamcity-repository9.tecnotree.com 5.14.0-307.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed May 3 06:16:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux


And it's perfectly stable since we rollbacked the cifs-utils and any kernels

I cannot access the bug 2177562 so I'm not even sure this is related.
But I'll try to update this week end, and see if the server can sustain the week end (it uses to break after few hours)

Comment 5 ryan.brothers 2023-05-16 19:44:09 UTC
I am running into this issue too after upgrading to 5.14.0-284.11.1.el9_2.  Will a fix for it be released soon?

Comment 6 bugzilla 2023-05-21 18:31:21 UTC
Same here, most of the time when waking up from suspend.

Comment 7 daniel_ulmer 2023-05-22 05:40:07 UTC
Sorry for the late response but after the latest update of CentOS it worked again for me. That's why i didn't response earlier. I left everything the same and didn't change anything and it startet working again. The update was somewhere around early Mai.

Comment 8 bugzilla 2023-05-22 08:24:46 UTC
If that was a kernel update for CentOS 9 stream then I suspect it hasn't been released yet for RHEL (and therefore Alma and Rocky Linux).

Comment 9 ryan.brothers 2023-06-01 17:08:16 UTC
Is there any timeline of when the fix will be released for RHEL 9?

Comment 11 Ronnie Sahlberg 2023-06-06 01:26:36 UTC
This should be fixed in kernel-5.14.0-284.17.1.el9_2

Comment 12 ryan.brothers 2023-06-07 19:30:16 UTC
Thanks, do you know when kernel-5.14.0-284.17.1.el9_2 will be released?

Comment 14 RHEL Program Management 2023-09-23 11:34:25 UTC
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

Comment 15 RHEL Program Management 2023-09-23 11:35:38 UTC
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.


Note You need to log in before you can comment on or make changes to this bug.