1966259 – stalld service crash openshift 4.8 nodes

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1966259 - stalld service crash openshift 4.8 nodes

Summary: stalld service crash openshift 4.8 nodes

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 8
Classification:	Red Hat
Component:	stalld
Sub Component:
Version:	8.4
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	urgent
Target Milestone:	beta
Target Release:	---
Assignee:	Fernando Pacheco
QA Contact:	Mark Simmons
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1976589
TreeView+	depends on / blocked

Reported:	2021-05-31 18:19 UTC by Sebastian Scheinkman
Modified:	2021-11-10 00:43 UTC (History)
CC List:	12 users (show)
Fixed In Version:	stalld-1.14.1-1.el8
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-11-09 18:41:43 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2021:4252	0	None	None	None	2021-11-09 18:42:00 UTC

Description Sebastian Scheinkman 2021-05-31 18:19:27 UTC

Description of problem:
enabling stalld on openshift using tuned or performance addon operator makes the node to enter into boot loop

ocp version:
Client Version: 4.8.0-0.nightly-2021-05-26-071911
Server Version: 4.8.0-0.nightly-2021-05-26-071911
Kubernetes Version: v1.21.0-rc.0+936c9e2

kernel version:
4.18.0-305.el8.x86_64
also
4.18.0-305.rt7.72.el8.x86_64

stalld version:
1.9

dmesg:

[  320.383641] stalld[167745]: segfault at 0 ip 00007f374b27200c sp 00007ffd26a40ce8 error 4 in libc-2.28.so[7f374b115000+1bc000]
[  320.396577] Code: 82 29 c8 c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 89 f9 c5 f9 6e c6 c4 41 31 ef c9 c4 e2 7d 78 c0 83 e1 3f 83 f9 20 77 34 <c5> 7e 6f 07 c4 c1 7d 74 c8 c4 c1 35 74 d0 c5 ed eb c9 c5 fd d7 c1
[  320.464230] coreos-boot-mount-generator: using dev-disk-by\x2dlabel-boot for boot mount to /boot
[  321.811442] coreos-boot-mount-generator: using dev-disk-by\x2dlabel-boot for boot mount to /boot


journalctl messages:

May 31 17:47:39 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Stopping irqbalance daemon...
May 31 17:47:39 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: irqbalance.service: Succeeded.
May 31 17:47:39 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Stopped irqbalance daemon.
May 31 17:47:39 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: irqbalance.service: Consumed 221ms CPU time
May 31 17:47:39 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Started irqbalance daemon.
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Starting Stall Monitor...
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com stalld[167743]: Disabled RT throttling
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Started Stall Monitor.
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com stalld[167745]: boosted pid 0 using SCHED_DEADLINE
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com stalld[167745]: using SCHED_DEADLINE for boosting
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Reloading.
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com stalld[167745]: initial config_buffer_size set to 1228800
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com stalld[167745]: detected new task format
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com kernel: stalld[167745]: segfault at 0 ip 00007f374b27200c sp 00007ffd26a40ce8 error 4 in libc-2.28.so[7f374b115000+1bc000]
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com kernel: Code: 82 29 c8 c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 89 f9 c5 f9 6e c6 c4 41 31 ef c9 c4 e2 7d 78 c0 83 e1 3f 83 f9 20 77 34 <c5> 7e 6f 07 c4 c1 7d 74 c8 c4 c1 35 74 d0 c5 ed eb c9 c5 fd d7 c1
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com coreos-boot-mount-generator: using dev-disk-by\x2dlabel-boot for boot mount to /boot
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: /usr/lib/systemd/system/bootupd.service:22: Unknown lvalue 'ProtectHostname' in section 'Service'
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: /usr/lib/systemd/system/rhcos-usrlocal-selinux-fixup.service:16: Ignoring unknown escape sequences: "if ! test -f /var/lib/.coreos-usrlocal-fixup.stamp; then find /var/usrlocal/sbin -executable -mount -exec chcon -v --reference=/usr/sbin {} \; && touch /var/lib/.coreos-usrlocal-fixup.stamp; fi"
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Started Process Core Dump (PID 167762/UID 0).
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[167796]: selinux: avc:  netlink recvfrom: error 9
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[167796]: selinux: avc:  netlink recvfrom: error 9
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[167796]: selinux: avc:  netlink recvfrom: error 9
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[167796]: selinux: avc:  netlink recvfrom: error 9
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[167796]: selinux: avc:  netlink recvfrom: error 9
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[167796]: selinux: avc:  netlink recvfrom: error 9
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[167796]: selinux: avc:  netlink recvfrom: error 9
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[167796]: selinux: avc:  netlink recvfrom: error 9
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[167796]: selinux: avc:  netlink recvfrom: error 9
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[167796]: selinux: avc:  netlink recvfrom: error 9
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[167796]: selinux: avc:  netlink recvfrom: error 9
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[167796]: selinux: avc:  netlink recvfrom: error 9
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[167796]: selinux: avc:  netlink recvfrom: error 9
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[167796]: selinux: avc:  netlink recvfrom: error 9
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[167796]: selinux: avc:  netlink recvfrom: error 9
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[167796]: selinux: avc:  netlink recvfrom: error 9
May 31 17:47:40 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[167796]: selinux: avc:  netlink recvfrom: error 9
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd-coredump[167796]: Process 167745 (stalld) of user 0 dumped core.
                                                                                        
                                                                                        Stack trace of thread 167745:
                                                                                        #0  0x00007f374b27200c __strchr_avx2 (libc.so.6)
                                                                                        #1  0x0000557ed8253736 get_cpu_idle_time (stalld)
                                                                                        #2  0x0000557ed82539c6 get_cpu_busy_list (stalld)
                                                                                        #3  0x0000557ed8255003 conservative_main (stalld)
                                                                                        #4  0x0000557ed825301e main (stalld)
                                                                                        #5  0x00007f374b138493 __libc_start_main (libc.so.6)
                                                                                        #6  0x0000557ed825318e _start (stalld)
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: stalld.service: Main process exited, code=dumped, status=11/SEGV
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: systemd-coredump: Succeeded.
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: systemd-coredump: Consumed 398ms CPU time
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Reloading.
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com stalld[167831]: Restored RT throttling
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com coreos-boot-mount-generator: using dev-disk-by\x2dlabel-boot for boot mount to /boot
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: /usr/lib/systemd/system/bootupd.service:22: Unknown lvalue 'ProtectHostname' in section 'Service'
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: /usr/lib/systemd/system/rhcos-usrlocal-selinux-fixup.service:16: Ignoring unknown escape sequences: "if ! test -f /var/lib/.coreos-usrlocal-fixup.stamp; then find /var/usrlocal/sbin -executable -mount -exec chcon -v --reference=/usr/sbin {} \; && touch /var/lib/.coreos-usrlocal-fixup.stamp; fi"
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:41 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: Couldn't stat device /dev/char/10:200: No such file or directory
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: stalld.service: Failed with result 'core-dump'.
May 31 17:47:42 cnfdc8-bm-master-0.t5g.lab.eng.bos.redhat.com systemd[1]: stalld.service: Consumed 68ms CPU time

Comment 1 Sebastian Scheinkman 2021-05-31 18:20:56 UTC

not sure if this is a diplucate of https://bugzilla.redhat.com/show_bug.cgi?id=1962690

In the other BZ the node never crash.

From BMC I see this message:

2021-05-31 18:07:01 	OSE0001 	A runtime critical stop occurred.	
	
Log Sequence Number:
583583
Detailed Description:
The system halted due to an exception while the operating system was running. This is a kernel panic or bug check event.
Recommended Action:
Review operating system logs and system video for additional information.

Comment 2 Fernando Pacheco 2021-06-01 16:24:40 UTC

(In reply to Sebastian Scheinkman from comment #1)
> not sure if this is a diplucate of
> https://bugzilla.redhat.com/show_bug.cgi?id=1962690
> 
> In the other BZ the node never crash.
> 
> From BMC I see this message:
> 
> 2021-05-31 18:07:01 	OSE0001 	A runtime critical stop occurred.	
> 	
> Log Sequence Number:
> 583583
> Detailed Description:
> The system halted due to an exception while the operating system was
> running. This is a kernel panic or bug check event.
> Recommended Action:
> Review operating system logs and system video for additional information.

I don't think the two bz are related. Looks like we hit a NULL pointer dereference
in get_cpu_idle_time. This code is not properly checking for NULL return values on
calls to strstr. Still need to do a little digging to find out why the input is not
in a form that we expect. But adding the NULL checks should clean this up.

Best,
Fernando

Comment 3 Fernando Pacheco 2021-06-02 18:36:50 UTC

Possible fix (prevent the NULL dereference):

https://gitlab.com/pachecof/stalld/-/commit/8ebca325487e5edd7fbaa7dc55cc5495cedc1b3e

Comment 6 Juri Lelli 2021-06-03 15:46:31 UTC

Hi Fernando,

(In reply to Fernando Pacheco from comment #3)
> Possible fix (prevent the NULL dereference):
> 
> https://gitlab.com/pachecof/stalld/-/commit/
> 8ebca325487e5edd7fbaa7dc55cc5495cedc1b3e

Would you be able to create a build for this change that
Sebastian can test on his setup?

Comment 7 Fernando Pacheco 2021-06-03 20:36:13 UTC

(In reply to Juri Lelli from comment #6)
> Hi Fernando,
> 
> (In reply to Fernando Pacheco from comment #3)
> > Possible fix (prevent the NULL dereference):
> > 
> > https://gitlab.com/pachecof/stalld/-/commit/
> > 8ebca325487e5edd7fbaa7dc55cc5495cedc1b3e
> 
> Would you be able to create a build for this change that
> Sebastian can test on his setup?

I can create a scratch build, but I thought OCP pulled stalld
in to their tuned operator. Will anything I build work for them?

Comment 8 Juri Lelli 2021-06-04 08:17:45 UTC

(In reply to Fernando Pacheco from comment #7)
> (In reply to Juri Lelli from comment #6)
> > Hi Fernando,
> > 
> > (In reply to Fernando Pacheco from comment #3)
> > > Possible fix (prevent the NULL dereference):
> > > 
> > > https://gitlab.com/pachecof/stalld/-/commit/
> > > 8ebca325487e5edd7fbaa7dc55cc5495cedc1b3e
> > 
> > Would you be able to create a build for this change that
> > Sebastian can test on his setup?
> 
> I can create a scratch build, but I thought OCP pulled stalld
> in to their tuned operator. Will anything I build work for them?

Sebastian, what would you need to test Fernando's fix?

Comment 10 Sebastian Scheinkman 2021-06-07 09:34:30 UTC

Hi Juri,

If I am able to get the stalld binary file I think I will be able just to switch the one that is in my environment for a small test.

In general we will need to request tuned operator folks to bump the stalld version they provide.

Thanks!
Sebastian

Comment 12 Sebastian Scheinkman 2021-06-09 15:25:16 UTC

Hi folks,

I validate it.

● stalld.service - Stall Monitor
   Loaded: loaded (/usr/lib/systemd/system/stalld.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-06-09 15:22:41 UTC; 6s ago
  Process: 3177746 ExecStartPre=/usr/bin/throttlectl off (code=exited, status=0/SUCCESS)
 Main PID: 3177752 (stalld)
    Tasks: 1 (limit: 543549)
   Memory: 768.0K
      CPU: 71ms
   CGroup: /system.slice/stalld.service
           └─3177752 /tmp/stalld --systemd -p 1000000000 -r 20000 -d 3 -t 30 --foreground --pidfile /run/stalld.pid

Jun 09 15:22:46 cnfdc3-bm-master-0.t5g.lab.eng.bos.redhat.com stalld[3177752]:   stat info not present for cpu94
Jun 09 15:22:46 cnfdc3-bm-master-0.t5g.lab.eng.bos.redhat.com stalld[3177752]:   stat info not present for cpu95
Jun 09 15:22:46 cnfdc3-bm-master-0.t5g.lab.eng.bos.redhat.com stalld[3177752]:   stat info not present for cpu96
Jun 09 15:22:46 cnfdc3-bm-master-0.t5g.lab.eng.bos.redhat.com stalld[3177752]:   stat info not present for cpu97
Jun 09 15:22:46 cnfdc3-bm-master-0.t5g.lab.eng.bos.redhat.com stalld[3177752]:   stat info not present for cpu98
Jun 09 15:22:46 cnfdc3-bm-master-0.t5g.lab.eng.bos.redhat.com stalld[3177752]:   stat info not present for cpu99
Jun 09 15:22:46 cnfdc3-bm-master-0.t5g.lab.eng.bos.redhat.com stalld[3177752]:   stat info not present for cpu100
Jun 09 15:22:46 cnfdc3-bm-master-0.t5g.lab.eng.bos.redhat.com stalld[3177752]:   stat info not present for cpu101
Jun 09 15:22:46 cnfdc3-bm-master-0.t5g.lab.eng.bos.redhat.com stalld[3177752]:   stat info not present for cpu102
Jun 09 15:22:46 cnfdc3-bm-master-0.t5g.lab.eng.bos.redhat.com stalld[3177752]:   stat info not present for cpu103


Is the code already merged?

Should I be able to contact the cluster-node-tuning-operator and request them to bump the stalld version for ocp?

Thanks!
Sebastian

Comment 13 Fernando Pacheco 2021-06-09 15:55:09 UTC

(In reply to Sebastian Scheinkman from comment #12)
> Hi folks,
> 
> I validate it.
> 
> ● stalld.service - Stall Monitor
>    Loaded: loaded (/usr/lib/systemd/system/stalld.service; enabled; vendor
> preset: disabled)
>    Active: active (running) since Wed 2021-06-09 15:22:41 UTC; 6s ago
>   Process: 3177746 ExecStartPre=/usr/bin/throttlectl off (code=exited,
> status=0/SUCCESS)
>  Main PID: 3177752 (stalld)
>     Tasks: 1 (limit: 543549)
>    Memory: 768.0K
>       CPU: 71ms
>    CGroup: /system.slice/stalld.service
>            └─3177752 /tmp/stalld --systemd -p 1000000000 -r 20000 -d 3 -t 30
> --foreground --pidfile /run/stalld.pid
> 
> Jun 09 15:22:46 cnfdc3-bm-master-0.t5g.lab.eng.bos.redhat.com
> stalld[3177752]:   stat info not present for cpu94
> Jun 09 15:22:46 cnfdc3-bm-master-0.t5g.lab.eng.bos.redhat.com
> stalld[3177752]:   stat info not present for cpu95
> Jun 09 15:22:46 cnfdc3-bm-master-0.t5g.lab.eng.bos.redhat.com
> stalld[3177752]:   stat info not present for cpu96
> Jun 09 15:22:46 cnfdc3-bm-master-0.t5g.lab.eng.bos.redhat.com
> stalld[3177752]:   stat info not present for cpu97
> Jun 09 15:22:46 cnfdc3-bm-master-0.t5g.lab.eng.bos.redhat.com
> stalld[3177752]:   stat info not present for cpu98
> Jun 09 15:22:46 cnfdc3-bm-master-0.t5g.lab.eng.bos.redhat.com
> stalld[3177752]:   stat info not present for cpu99
> Jun 09 15:22:46 cnfdc3-bm-master-0.t5g.lab.eng.bos.redhat.com
> stalld[3177752]:   stat info not present for cpu100
> Jun 09 15:22:46 cnfdc3-bm-master-0.t5g.lab.eng.bos.redhat.com
> stalld[3177752]:   stat info not present for cpu101
> Jun 09 15:22:46 cnfdc3-bm-master-0.t5g.lab.eng.bos.redhat.com
> stalld[3177752]:   stat info not present for cpu102
> Jun 09 15:22:46 cnfdc3-bm-master-0.t5g.lab.eng.bos.redhat.com
> stalld[3177752]:   stat info not present for cpu103
> 
> 
> Is the code already merged?

Not yet - the merge request still needs proper review.

Thanks for validating the fix!

> 
> Should I be able to contact the cluster-node-tuning-operator and request
> them to bump the stalld version for ocp?
> 
> Thanks!
> Sebastian

Comment 19 Fernando Pacheco 2021-07-08 16:42:05 UTC

This has been addressed by MR34 and is included in stalld v1.13.

https://gitlab.com/rt-linux-tools/stalld/-/merge_requests/34

Comment 20 Sebastian Scheinkman 2021-07-08 16:48:21 UTC

Hi Jiri will you be able to bump stalld in NTO?

I see it was bumped to 1.12.0 at Jun28

we may need also to backport this one to 4.8

Comment 24 Jiří Mencák 2021-07-09 07:39:26 UTC

(In reply to Sebastian Scheinkman from comment #20)
> Hi Jiri will you be able to bump stalld in NTO?
> 
> I see it was bumped to 1.12.0 at Jun28
> 
> we may need also to backport this one to 4.8

Hi Sebastian, thanks for bringing this to my attention!
Upstream PR for NTO in OCP 4.9: https://github.com/openshift/cluster-node-tuning-operator/pull/246
And yes, looking at the fix we'll probably need to backport to 4.8.

Comment 25 Clark Williams 2021-07-09 14:38:19 UTC

We just updated to 1.13 with more bugfixes.

Comment 28 Mark Simmons 2021-07-16 16:03:23 UTC

Can someone from OpenShift either take QA contact or test?

Comment 34 Sebastian Scheinkman 2021-08-01 11:10:52 UTC

Hi Fernando and Jiri,

I validate this on the same environment as before just with a newer ocp version.
stalld is running I only see a lot of messages like this "get_cpu_busy_list: unable to parse idle time for cpu94" is this expected?

stalld.service - Stall Monitor
   Loaded: loaded (/etc/systemd/system/stalld.service; static; vendor preset: disabled)
   Active: active (running) since Mon 2021-07-26 11:27:40 UTC; 5 days ago
 Main PID: 78845 (stalld)
    Tasks: 1 (limit: 609017)
   Memory: 1.8M
      CPU: 2min 35.431s
   CGroup: /system.slice/stalld.service
           └─78845 /usr/local/bin/stalld --systemd -p 1000000000 -r 10000 -d 3 -t 20 --log_syslog --log_kmsg --foreground --pidfile /run/stalld.pid

Aug 01 10:28:52 cnfdc8.clus2.t5g.lab.eng.bos.redhat.com stalld[78845]:   get_cpu_busy_list: unable to parse idle time for cpu94
Aug 01 10:28:52 cnfdc8.clus2.t5g.lab.eng.bos.redhat.com stalld[78845]:   get_cpu_busy_list: unable to parse idle time for cpu95
Aug 01 10:28:52 cnfdc8.clus2.t5g.lab.eng.bos.redhat.com stalld[78845]:   get_cpu_busy_list: unable to parse idle time for cpu96
Aug 01 10:28:52 cnfdc8.clus2.t5g.lab.eng.bos.redhat.com stalld[78845]:   get_cpu_busy_list: unable to parse idle time for cpu97
Aug 01 10:28:52 cnfdc8.clus2.t5g.lab.eng.bos.redhat.com stalld[78845]:   get_cpu_busy_list: unable to parse idle time for cpu98
Aug 01 10:28:52 cnfdc8.clus2.t5g.lab.eng.bos.redhat.com stalld[78845]:   get_cpu_busy_list: unable to parse idle time for cpu99
Aug 01 10:28:52 cnfdc8.clus2.t5g.lab.eng.bos.redhat.com stalld[78845]:   get_cpu_busy_list: unable to parse idle time for cpu100
Aug 01 10:28:52 cnfdc8.clus2.t5g.lab.eng.bos.redhat.com stalld[78845]:   get_cpu_busy_list: unable to parse idle time for cpu101
Aug 01 10:28:52 cnfdc8.clus2.t5g.lab.eng.bos.redhat.com stalld[78845]:   get_cpu_busy_list: unable to parse idle time for cpu102
Aug 01 10:28:52 cnfdc8.clus2.t5g.lab.eng.bos.redhat.com stalld[78845]:   get_cpu_busy_list: unable to parse idle time for cpu103
[root@cnfdc8 core]# oc vers^C
[root@cnfdc8 core]# exit
[core@cnfdc8 ~]$ logout

Connection to cnfdc8.clus2.t5g.lab.eng.bos.redhat.com closed.
[root@cnfdc8-installer ~]# oc version
Client Version: 4.9.0-0.nightly-2021-07-25-025749
Server Version: 4.9.0-0.nightly-2021-07-25-025749
Kubernetes Version: v1.21.1+8268f88

Comment 35 Jiří Mencák 2021-08-02 05:21:52 UTC

I do not see these messages.

sh-4.4# systemctl status stalld
â stalld.service - Stall Monitor
   Loaded: loaded (/etc/systemd/system/stalld.service; static; vendor preset: disabled)
   Active: active (running) since Mon 2021-08-02 05:11:10 UTC; 4min 7s ago
 Main PID: 3836 (stalld)
    Tasks: 1 (limit: 95498)
   Memory: 352.0K
      CPU: 72ms
   CGroup: /system.slice/stalld.service
           ââ3836 /usr/local/bin/stalld --systemd -p 1000000000 -r 10000 -d 3 -t 20 --log_syslog --log_kmsg --foreground --pidfile /run/stalld.pid

Aug 02 05:11:10 jmencak-hh6hp-worker-a-mntdx.c.openshift-gce-devel.internal systemd[1]: Starting Stall Monitor...
Aug 02 05:11:10 jmencak-hh6hp-worker-a-mntdx.c.openshift-gce-devel.internal systemd[1]: Started Stall Monitor.
Aug 02 05:11:10 jmencak-hh6hp-worker-a-mntdx.c.openshift-gce-devel.internal stalld[3836]: dl_runtime is shorter than 1ms, setting HRTICK_DL
Aug 02 05:11:10 jmencak-hh6hp-worker-a-mntdx.c.openshift-gce-devel.internal stalld[3836]: trying to open file /sys/kernel/debug/sched/debug returned -1
Aug 02 05:11:10 jmencak-hh6hp-worker-a-mntdx.c.openshift-gce-devel.internal stalld[3836]: trying to open file /proc/sched_debug returned 3
Aug 02 05:11:10 jmencak-hh6hp-worker-a-mntdx.c.openshift-gce-devel.internal stalld[3836]: boosted pid 0 using SCHED_DEADLINE
Aug 02 05:11:10 jmencak-hh6hp-worker-a-mntdx.c.openshift-gce-devel.internal stalld[3836]: using SCHED_DEADLINE for boosting
Aug 02 05:11:10 jmencak-hh6hp-worker-a-mntdx.c.openshift-gce-devel.internal stalld[3836]: initial config_buffer_size set to 450560
Aug 02 05:11:10 jmencak-hh6hp-worker-a-mntdx.c.openshift-gce-devel.internal stalld[3836]: detected new task format
Aug 02 05:11:10 jmencak-hh6hp-worker-a-mntdx.c.openshift-gce-devel.internal stalld[3836]: single threaded mode

I believe we need stalld authors to take a look at this.  Clark, Daniel (sorry to pull you in), can you take a look?

Comment 36 Jiří Mencák 2021-08-02 10:53:12 UTC

Gathered some logs from the affected box.  It looks like stalld doesn't play well with "nosmt" kernel option and multithreaded CPUs.  The warnings were on this system:

[root@cnfdc8 core]# lscpu -p
# The following is the parsable format, which can be fed to other
# programs. Each different item in every column has an unique ID
# starting from zero.
# CPU,Core,Socket,Node,,L1d,L1i,L2,L3
0,0,0,0,,0,0,0,0
1,1,1,1,,1,1,1,1
2,2,0,0,,2,2,2,0
3,3,1,1,,3,3,3,1
... output cut for brevity ...
49,49,1,1,,49,49,49,1
50,50,0,0,,50,50,50,0
51,51,1,1,,51,51,51,1

and the warnings from stalld were starting for CPUs >= 52.  Statistics for those CPUs do not exist in /proc/stat.

I *believe* 

get_cpu_busy_list: unable to parse idle time for cpu52+

warnings are safe to ignore, but I'll check with `stalld` developers.

Comment 37 Jiří Mencák 2021-08-02 11:30:54 UTC

Created upstream issue:
https://gitlab.com/rt-linux-tools/stalld/-/issues/15

Comment 38 Fernando Pacheco 2021-08-03 01:04:44 UTC

(In reply to Jiří Mencák from comment #36)
> Gathered some logs from the affected box.  It looks like stalld doesn't play
> well with "nosmt" kernel option and multithreaded CPUs.  The warnings were
> on this system:
> 
> [root@cnfdc8 core]# lscpu -p
> # The following is the parsable format, which can be fed to other
> # programs. Each different item in every column has an unique ID
> # starting from zero.
> # CPU,Core,Socket,Node,,L1d,L1i,L2,L3
> 0,0,0,0,,0,0,0,0
> 1,1,1,1,,1,1,1,1
> 2,2,0,0,,2,2,2,0
> 3,3,1,1,,3,3,3,1
> ... output cut for brevity ...
> 49,49,1,1,,49,49,49,1
> 50,50,0,0,,50,50,50,0
> 51,51,1,1,,51,51,51,1
> 
> and the warnings from stalld were starting for CPUs >= 52.  Statistics for
> those CPUs do not exist in /proc/stat.
> 
> I *believe* 
> 
> get_cpu_busy_list: unable to parse idle time for cpu52+
> 
> warnings are safe to ignore, but I'll check with `stalld` developers.

Warnings should be okay to ignore. Aside from the noise we do the right thing here and treat the offline cpus as idle.

I have an (untested) patch that will issue the warning only when we are unable to parse /proc/stat for *online* cpus.
I can submit upstream if there is any interest in this - after testing of course.

Comment 40 Daniel Bristot de Oliveira 2021-08-03 12:04:24 UTC

(In reply to Fernando Pacheco from comment #38)
> (In reply to Jiří Mencák from comment #36)
> > Gathered some logs from the affected box.  It looks like stalld doesn't play
> > well with "nosmt" kernel option and multithreaded CPUs.  The warnings were
> > on this system:
> > 
> > [root@cnfdc8 core]# lscpu -p
> > # The following is the parsable format, which can be fed to other
> > # programs. Each different item in every column has an unique ID
> > # starting from zero.
> > # CPU,Core,Socket,Node,,L1d,L1i,L2,L3
> > 0,0,0,0,,0,0,0,0
> > 1,1,1,1,,1,1,1,1
> > 2,2,0,0,,2,2,2,0
> > 3,3,1,1,,3,3,3,1
> > ... output cut for brevity ...
> > 49,49,1,1,,49,49,49,1
> > 50,50,0,0,,50,50,50,0
> > 51,51,1,1,,51,51,51,1
> > 
> > and the warnings from stalld were starting for CPUs >= 52.  Statistics for
> > those CPUs do not exist in /proc/stat.
> > 
> > I *believe* 
> > 
> > get_cpu_busy_list: unable to parse idle time for cpu52+
> > 
> > warnings are safe to ignore, but I'll check with `stalld` developers.
> 
> Warnings should be okay to ignore. Aside from the noise we do the right
> thing here and treat the offline cpus as idle.
> 
> I have an (untested) patch that will issue the warning only when we are
> unable to parse /proc/stat for *online* cpus.
> I can submit upstream if there is any interest in this - after testing of
> course.

Please, sent it!

-- Daniel

Comment 47 errata-xmlrpc 2021-11-09 18:41:43 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (stalld bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4252

Note You need to log in before you can comment on or make changes to this bug.