RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1666562 - tuned is failing when applying the realtime-virtual-host profile : Failed to start rt-entsk.service: Unit rt-entsk.service not found.
Summary: tuned is failing when applying the realtime-virtual-host profile : Failed to ...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: tuned
Version: 8.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 8.0
Assignee: Jaroslav Škarvada
QA Contact: qe-baseos-daemons
URL:
Whiteboard:
Depends On:
Blocks: 1640832 1665995
TreeView+ depends on / blocked
 
Reported: 2019-01-16 03:30 UTC by Pei Zhang
Modified: 2019-10-22 13:33 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-01-24 14:14:08 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Pei Zhang 2019-01-16 03:30:41 UTC 
Description of problem:
In real time host, using tuned to isolated cores, there are errors in tuned.log.


Version-Release number of selected component (if applicable):
4.18.0-60.rt9.112.el8.x86_64
tuned-profiles-nfv-host-2.10.0-14.el8.noarch
tuned-2.10.0-14.el8.noarch
tuned-profiles-realtime-2.10.0-14.el8.noarch
tuned-profiles-cpu-partitioning-2.10.0-14.el8.noarch
tuned-profiles-nfv-host-bin-0-0.1.20180302git1edfa966.el8.x86_64


How reproducible:
4/4

Steps to Reproduce:
1. Install a real time rhel8 host

2. Isolate cores using tuned
# cat /etc/tuned/realtime-virtual-host-variables.conf 
isolated_cores=2,4,6,8,10,12,14,16,18

# tuned-adm profile realtime-virtual-host

4. Check /var/log/tuned/tuned.log, there are ERROR info

# cat /var/log/tuned/tuned.log | grep ERROR
...
2019-01-16 03:15:43,013 ERROR    tuned.utils.commands: Executing sysctl error: sysctl: cannot stat /proc/sys/net/core/busy_read: No such file or directory
2019-01-16 03:15:43,028 ERROR    tuned.utils.commands: Executing sysctl error: sysctl: cannot stat /proc/sys/net/core/busy_poll: No such file or directory
2019-01-16 03:15:43,060 ERROR    tuned.utils.commands: Executing sysctl error: sysctl: cannot stat /proc/sys/kernel/numa_balancing: No such file or directory
2019-01-16 03:15:43,755 ERROR    tuned.plugins.plugin_script: script '/usr/lib/tuned/realtime-virtual-host/script.sh' error output: 'Failed to start rt-entsk.service: Unit rt-entsk.service not found.'

5. Reboot host

6. Check /var/log/tuned/tuned.log, there are more ERROR info and "BUG: Unhandled exception in start_tuning: dictionary changed size during iteration".

# cat /var/log/tuned/tuned.log | grep ERROR
...
2019-01-16 03:17:35,427 ERROR    tuned.units.manager: BUG: Unhandled exception in stop_tuning: dictionary changed size during iteration
2019-01-16 03:17:35,650 ERROR    tuned.plugins.plugin_script: script '/usr/lib/tuned/realtime-virtual-host/script.sh' error output: 'Failed to stop rt-entsk.service: Unit rt-entsk.service not loaded.'
2019-01-16 03:17:35,650 ERROR    tuned.plugins.plugin_script: script '/usr/lib/tuned/realtime-virtual-host/script.sh' returned error code: 5
2019-01-16 03:22:34,255 ERROR    tuned.utils.commands: Executing sysctl error: sysctl: cannot stat /proc/sys/net/core/busy_read: No such file or directory
2019-01-16 03:22:34,269 ERROR    tuned.utils.commands: Executing sysctl error: sysctl: cannot stat /proc/sys/net/core/busy_poll: No such file or directory
2019-01-16 03:22:34,299 ERROR    tuned.utils.commands: Executing sysctl error: sysctl: cannot stat /proc/sys/kernel/numa_balancing: No such file or directory
2019-01-16 03:22:34,803 ERROR    tuned.plugins.plugin_script: script '/usr/lib/tuned/realtime-virtual-host/script.sh' error output: 'Failed to start rt-entsk.service: Unit rt-entsk.service not found.'
2019-01-16 03:22:34,859 ERROR    tuned.units.manager: BUG: Unhandled exception in start_tuning: dictionary changed size during iteration


Actual results:
There are ERROR infos in /var/log/tuned/tuned.log. These errors may cause latency spike in kvm-rt testing.

Expected results:
There should be no ERROR infos.

Additional info:
Comment 1 Ondřej Lysoněk 2019-01-16 11:10:38 UTC 
So there's several issues here, here's what I found out so far.

(In reply to Pei Zhang from comment #0)
> 2019-01-16 03:15:43,013 ERROR    tuned.utils.commands: Executing sysctl
> error: sysctl: cannot stat /proc/sys/net/core/busy_read: No such file or
> directory
> 2019-01-16 03:15:43,028 ERROR    tuned.utils.commands: Executing sysctl
> error: sysctl: cannot stat /proc/sys/net/core/busy_poll: No such file or
> directory

The net.core.busy_read and net.core.busy_poll sysctl settings seem not to be present in the RHEL-8 realtime kernel. I'll file a bug for these.

> 2019-01-16 03:15:43,060 ERROR    tuned.utils.commands: Executing sysctl
> error: sysctl: cannot stat /proc/sys/kernel/numa_balancing: No such file or
> directory

I don't see the kernel.numa_balancing sysctl setting even in RHEL-7. Perhaps it shows up only on certain hardware?

Either way, perhaps we should print a warning instead of an error to the logs when a sysctl setting doesn't exist. We can fix that along with bug#1666678, which deals with reworking the sysctl plugin.

> 2019-01-16 03:15:43,755 ERROR    tuned.plugins.plugin_script: script
> '/usr/lib/tuned/realtime-virtual-host/script.sh' error output: 'Failed to
> start rt-entsk.service: Unit rt-entsk.service not found.'

The rt-entsk.service needs to be installed. On RHEL-7, it's provided by the rt-setup package, but I don't see it in RHEL-8.

> 6. Check /var/log/tuned/tuned.log, there are more ERROR info and "BUG:
> Unhandled exception in start_tuning: dictionary changed size during
> iteration".

This is probably the same issue as bug#1666680.
Comment 2 Ondřej Lysoněk 2019-01-16 13:19:46 UTC 
(In reply to Ondřej Lysoněk from comment #1)
> (In reply to Pei Zhang from comment #0)
> > 2019-01-16 03:15:43,013 ERROR    tuned.utils.commands: Executing sysctl
> > error: sysctl: cannot stat /proc/sys/net/core/busy_read: No such file or
> > directory
> > 2019-01-16 03:15:43,028 ERROR    tuned.utils.commands: Executing sysctl
> > error: sysctl: cannot stat /proc/sys/net/core/busy_poll: No such file or
> > directory
> 
> The net.core.busy_read and net.core.busy_poll sysctl settings seem not to be
> present in the RHEL-8 realtime kernel. I'll file a bug for these.

See bug#1666724.

> > 2019-01-16 03:15:43,060 ERROR    tuned.utils.commands: Executing sysctl
> > error: sysctl: cannot stat /proc/sys/kernel/numa_balancing: No such file or
> > directory
> 
> I don't see the kernel.numa_balancing sysctl setting even in RHEL-7. Perhaps
> it shows up only on certain hardware?

The kernel.numa_balancing sysctl setting is not present, because CONFIG_NUMA_BALANCING is disabled in the realtime kernel. It's also disabled in the RHEL-7 realtime kernel, so I suspect the error has always been printed. I think disabling kernel.numa_balancing in the tuned profile is still useful on non-rt kernels, so we should keep it. And we should consider printing the log message as a warning, not error, as already suggested.
Comment 3 Ondřej Lysoněk 2019-01-16 13:50:41 UTC 
(In reply to Ondřej Lysoněk from comment #1)
> > 2019-01-16 03:15:43,755 ERROR    tuned.plugins.plugin_script: script
> > '/usr/lib/tuned/realtime-virtual-host/script.sh' error output: 'Failed to
> > start rt-entsk.service: Unit rt-entsk.service not found.'
> 
> The rt-entsk.service needs to be installed. On RHEL-7, it's provided by the
> rt-setup package, but I don't see it in RHEL-8.

Unfortunatelly, the rt-setup package is not available in a public RHEL-8 repo, it's available only in buildroot.

Marcelo, you added the rt-entsk.service handling to the realtime Tuned profiles. Can you please confirm whether starting the service is still needed on RHEL-8?
Comment 4 Luiz Capitulino 2019-01-16 16:43:24 UTC 
(In reply to Ondřej Lysoněk from comment #1)
> So there's several issues here, here's what I found out so far.
> 
> (In reply to Pei Zhang from comment #0)
> > 2019-01-16 03:15:43,013 ERROR    tuned.utils.commands: Executing sysctl
> > error: sysctl: cannot stat /proc/sys/net/core/busy_read: No such file or
> > directory
> > 2019-01-16 03:15:43,028 ERROR    tuned.utils.commands: Executing sysctl
> > error: sysctl: cannot stat /proc/sys/net/core/busy_poll: No such file or
> > directory
> 
> The net.core.busy_read and net.core.busy_poll sysctl settings seem not to be
> present in the RHEL-8 realtime kernel. I'll file a bug for these.

That's correct, thanks for opening the BZ. I knew this for sometime but
forgot opening the BZ.
 
> > 2019-01-16 03:15:43,060 ERROR    tuned.utils.commands: Executing sysctl
> > error: sysctl: cannot stat /proc/sys/kernel/numa_balancing: No such file or
> > directory
> 
> I don't see the kernel.numa_balancing sysctl setting even in RHEL-7. Perhaps
> it shows up only on certain hardware?

I guess it's (intentionally) disabled for the RT kernel.
 
> Either way, perhaps we should print a warning instead of an error to the
> logs when a sysctl setting doesn't exist. We can fix that along with
> bug#1666678, which deals with reworking the sysctl plugin.

I think we need a way to tell in tuned.conf which sysctl and sysfs settings
are required and which are optional. For optional ones, it's fine to
just print a warning if it fails to be set. For required ones, we have
to fail.
 
> > 2019-01-16 03:15:43,755 ERROR    tuned.plugins.plugin_script: script
> > '/usr/lib/tuned/realtime-virtual-host/script.sh' error output: 'Failed to
> > start rt-entsk.service: Unit rt-entsk.service not found.'
> 
> The rt-entsk.service needs to be installed. On RHEL-7, it's provided by the
> rt-setup package, but I don't see it in RHEL-8.

OK.
 
> > 6. Check /var/log/tuned/tuned.log, there are more ERROR info and "BUG:
> > Unhandled exception in start_tuning: dictionary changed size during
> > iteration".
> 
> This is probably the same issue as bug#1666680.

OK, so this BZ is about rt-entsk.service at this point. But note that bug
1666680 is very high priority for us.
Comment 5 Ondřej Lysoněk 2019-01-16 18:08:42 UTC 
(In reply to Luiz Capitulino from comment #4)
> > Either way, perhaps we should print a warning instead of an error to the
> > logs when a sysctl setting doesn't exist. We can fix that along with
> > bug#1666678, which deals with reworking the sysctl plugin.
> 
> I think we need a way to tell in tuned.conf which sysctl and sysfs settings
> are required and which are optional. For optional ones, it's fine to
> just print a warning if it fails to be set. For required ones, we have
> to fail.

Can you specify what percisely you mean by "have to fail"? Would an error message in the log be enough?

> OK, so this BZ is about rt-entsk.service at this point.

So can you confirm that starting rt-entsk.service is still relevant for RHEL-8? (I want to make sure that's the case, because it's possible adding the rt-setup package to RHEL-8 now won't be easy.)

Thanks.
Comment 6 Luiz Capitulino 2019-01-16 18:24:29 UTC 
(In reply to Ondřej Lysoněk from comment #5)
> (In reply to Luiz Capitulino from comment #4)
> > > Either way, perhaps we should print a warning instead of an error to the
> > > logs when a sysctl setting doesn't exist. We can fix that along with
> > > bug#1666678, which deals with reworking the sysctl plugin.
> > 
> > I think we need a way to tell in tuned.conf which sysctl and sysfs settings
> > are required and which are optional. For optional ones, it's fine to
> > just print a warning if it fails to be set. For required ones, we have
> > to fail.
> 
> Can you specify what percisely you mean by "have to fail"? Would an error
> message in the log be enough?

Yes, having the error message is enough (like it is today). However, it has
been my dream to see tuned fail with more verbosity and rolling back all
the profile. By more verbosity I mean failing in systemd and maybe also
writing it to /var/log/messages.
 
> > OK, so this BZ is about rt-entsk.service at this point.
> 
> So can you confirm that starting rt-entsk.service is still relevant for
> RHEL-8? (I want to make sure that's the case, because it's possible adding
> the rt-setup package to RHEL-8 now won't be easy.)

I can't, it has to be Marcelo. I didn't even know/remember that we had
rt-entsk.service...
Comment 7 Marcelo Tosatti 2019-01-18 15:41:32 UTC 
(In reply to Luiz Capitulino from comment #6)
> (In reply to Ondřej Lysoněk from comment #5)
> > (In reply to Luiz Capitulino from comment #4)
> > > > Either way, perhaps we should print a warning instead of an error to the
> > > > logs when a sysctl setting doesn't exist. We can fix that along with
> > > > bug#1666678, which deals with reworking the sysctl plugin.
> > > 
> > > I think we need a way to tell in tuned.conf which sysctl and sysfs settings
> > > are required and which are optional. For optional ones, it's fine to
> > > just print a warning if it fails to be set. For required ones, we have
> > > to fail.
> > 
> > Can you specify what percisely you mean by "have to fail"? Would an error
> > message in the log be enough?
> 
> Yes, having the error message is enough (like it is today). However, it has
> been my dream to see tuned fail with more verbosity and rolling back all
> the profile. By more verbosity I mean failing in systemd and maybe also
> writing it to /var/log/messages.
>  
> > > OK, so this BZ is about rt-entsk.service at this point.
> > 
> > So can you confirm that starting rt-entsk.service is still relevant for
> > RHEL-8? (I want to make sure that's the case, because it's possible adding
> > the rt-setup package to RHEL-8 now won't be easy.)
> 
> I can't, it has to be Marcelo. I didn't even know/remember that we had
> rt-entsk.service...

Yes it is necessary to workaround the chronyd bug.
Comment 8 Marcelo Tosatti 2019-01-18 15:53:58 UTC 
This is the bug:

[Bug 1619407] rt-setup: include enable-netsocket-tstamp-static-key.c [RT-8  ]

https://bugzilla.redhat.com/show_bug.cgi?id=1619407

It should be fixed by pulling the daemon rt-entsk.service from rt-setup upstream.
Comment 9 Marcelo Tosatti 2019-01-18 15:57:04 UTC 
Clark, how do we install rt-setup on RHEL-8 ? Is there a separate repo?(In reply to Ondřej Lysoněk from comment #3)
> (In reply to Ondřej Lysoněk from comment #1)
> > > 2019-01-16 03:15:43,755 ERROR    tuned.plugins.plugin_script: script
> > > '/usr/lib/tuned/realtime-virtual-host/script.sh' error output: 'Failed to
> > > start rt-entsk.service: Unit rt-entsk.service not found.'
> > 
> > The rt-entsk.service needs to be installed. On RHEL-7, it's provided by the
> > rt-setup package, but I don't see it in RHEL-8.
> 
> Unfortunatelly, the rt-setup package is not available in a public RHEL-8
> repo, it's available only in buildroot.

Well, installing it should fix it. 

> 
> Marcelo, you added the rt-entsk.service handling to the realtime Tuned
> profiles. Can you please confirm whether starting the service is still
> needed on RHEL-8?
Comment 18 Ondřej Lysoněk 2019-01-24 14:10:39 UTC 
(In reply to Luiz Capitulino from comment #4)
> > > 2019-01-16 03:15:43,060 ERROR    tuned.utils.commands: Executing sysctl
> > > error: sysctl: cannot stat /proc/sys/kernel/numa_balancing: No such file or
> > > directory
> > 
> > I don't see the kernel.numa_balancing sysctl setting even in RHEL-7. Perhaps
> > it shows up only on certain hardware?
> 
> I guess it's (intentionally) disabled for the RT kernel.
>  
> > Either way, perhaps we should print a warning instead of an error to the
> > logs when a sysctl setting doesn't exist. We can fix that along with
> > bug#1666678, which deals with reworking the sysctl plugin.
> 
> I think we need a way to tell in tuned.conf which sysctl and sysfs settings
> are required and which are optional. For optional ones, it's fine to
> just print a warning if it fails to be set. For required ones, we have
> to fail.

This will be addressed in rhbz#1669171.
Note You need to log in before you can comment on or make changes to this bug.