Bug 1666562
Summary: | tuned is failing when applying the realtime-virtual-host profile : Failed to start rt-entsk.service: Unit rt-entsk.service not found. | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 8 | Reporter: | Pei Zhang <pezhang> |
Component: | tuned | Assignee: | Jaroslav Škarvada <jskarvad> |
Status: | CLOSED NOTABUG | QA Contact: | qe-baseos-daemons |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | 8.1 | CC: | bhu, chayang, hhuang, jeder, jskarvad, juzhang, jwboyer, lcapitulino, michen, mtosatti, olysonek, williams |
Target Milestone: | rc | ||
Target Release: | 8.0 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2019-01-24 14:14:08 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 1640832, 1665995 |
Description
Pei Zhang
2019-01-16 03:30:41 UTC
So there's several issues here, here's what I found out so far. (In reply to Pei Zhang from comment #0) > 2019-01-16 03:15:43,013 ERROR tuned.utils.commands: Executing sysctl > error: sysctl: cannot stat /proc/sys/net/core/busy_read: No such file or > directory > 2019-01-16 03:15:43,028 ERROR tuned.utils.commands: Executing sysctl > error: sysctl: cannot stat /proc/sys/net/core/busy_poll: No such file or > directory The net.core.busy_read and net.core.busy_poll sysctl settings seem not to be present in the RHEL-8 realtime kernel. I'll file a bug for these. > 2019-01-16 03:15:43,060 ERROR tuned.utils.commands: Executing sysctl > error: sysctl: cannot stat /proc/sys/kernel/numa_balancing: No such file or > directory I don't see the kernel.numa_balancing sysctl setting even in RHEL-7. Perhaps it shows up only on certain hardware? Either way, perhaps we should print a warning instead of an error to the logs when a sysctl setting doesn't exist. We can fix that along with bug#1666678, which deals with reworking the sysctl plugin. > 2019-01-16 03:15:43,755 ERROR tuned.plugins.plugin_script: script > '/usr/lib/tuned/realtime-virtual-host/script.sh' error output: 'Failed to > start rt-entsk.service: Unit rt-entsk.service not found.' The rt-entsk.service needs to be installed. On RHEL-7, it's provided by the rt-setup package, but I don't see it in RHEL-8. > 6. Check /var/log/tuned/tuned.log, there are more ERROR info and "BUG: > Unhandled exception in start_tuning: dictionary changed size during > iteration". This is probably the same issue as bug#1666680. (In reply to Ondřej Lysoněk from comment #1) > (In reply to Pei Zhang from comment #0) > > 2019-01-16 03:15:43,013 ERROR tuned.utils.commands: Executing sysctl > > error: sysctl: cannot stat /proc/sys/net/core/busy_read: No such file or > > directory > > 2019-01-16 03:15:43,028 ERROR tuned.utils.commands: Executing sysctl > > error: sysctl: cannot stat /proc/sys/net/core/busy_poll: No such file or > > directory > > The net.core.busy_read and net.core.busy_poll sysctl settings seem not to be > present in the RHEL-8 realtime kernel. I'll file a bug for these. See bug#1666724. > > 2019-01-16 03:15:43,060 ERROR tuned.utils.commands: Executing sysctl > > error: sysctl: cannot stat /proc/sys/kernel/numa_balancing: No such file or > > directory > > I don't see the kernel.numa_balancing sysctl setting even in RHEL-7. Perhaps > it shows up only on certain hardware? The kernel.numa_balancing sysctl setting is not present, because CONFIG_NUMA_BALANCING is disabled in the realtime kernel. It's also disabled in the RHEL-7 realtime kernel, so I suspect the error has always been printed. I think disabling kernel.numa_balancing in the tuned profile is still useful on non-rt kernels, so we should keep it. And we should consider printing the log message as a warning, not error, as already suggested. (In reply to Ondřej Lysoněk from comment #1) > > 2019-01-16 03:15:43,755 ERROR tuned.plugins.plugin_script: script > > '/usr/lib/tuned/realtime-virtual-host/script.sh' error output: 'Failed to > > start rt-entsk.service: Unit rt-entsk.service not found.' > > The rt-entsk.service needs to be installed. On RHEL-7, it's provided by the > rt-setup package, but I don't see it in RHEL-8. Unfortunatelly, the rt-setup package is not available in a public RHEL-8 repo, it's available only in buildroot. Marcelo, you added the rt-entsk.service handling to the realtime Tuned profiles. Can you please confirm whether starting the service is still needed on RHEL-8? (In reply to Ondřej Lysoněk from comment #1) > So there's several issues here, here's what I found out so far. > > (In reply to Pei Zhang from comment #0) > > 2019-01-16 03:15:43,013 ERROR tuned.utils.commands: Executing sysctl > > error: sysctl: cannot stat /proc/sys/net/core/busy_read: No such file or > > directory > > 2019-01-16 03:15:43,028 ERROR tuned.utils.commands: Executing sysctl > > error: sysctl: cannot stat /proc/sys/net/core/busy_poll: No such file or > > directory > > The net.core.busy_read and net.core.busy_poll sysctl settings seem not to be > present in the RHEL-8 realtime kernel. I'll file a bug for these. That's correct, thanks for opening the BZ. I knew this for sometime but forgot opening the BZ. > > 2019-01-16 03:15:43,060 ERROR tuned.utils.commands: Executing sysctl > > error: sysctl: cannot stat /proc/sys/kernel/numa_balancing: No such file or > > directory > > I don't see the kernel.numa_balancing sysctl setting even in RHEL-7. Perhaps > it shows up only on certain hardware? I guess it's (intentionally) disabled for the RT kernel. > Either way, perhaps we should print a warning instead of an error to the > logs when a sysctl setting doesn't exist. We can fix that along with > bug#1666678, which deals with reworking the sysctl plugin. I think we need a way to tell in tuned.conf which sysctl and sysfs settings are required and which are optional. For optional ones, it's fine to just print a warning if it fails to be set. For required ones, we have to fail. > > 2019-01-16 03:15:43,755 ERROR tuned.plugins.plugin_script: script > > '/usr/lib/tuned/realtime-virtual-host/script.sh' error output: 'Failed to > > start rt-entsk.service: Unit rt-entsk.service not found.' > > The rt-entsk.service needs to be installed. On RHEL-7, it's provided by the > rt-setup package, but I don't see it in RHEL-8. OK. > > 6. Check /var/log/tuned/tuned.log, there are more ERROR info and "BUG: > > Unhandled exception in start_tuning: dictionary changed size during > > iteration". > > This is probably the same issue as bug#1666680. OK, so this BZ is about rt-entsk.service at this point. But note that bug 1666680 is very high priority for us. (In reply to Luiz Capitulino from comment #4) > > Either way, perhaps we should print a warning instead of an error to the > > logs when a sysctl setting doesn't exist. We can fix that along with > > bug#1666678, which deals with reworking the sysctl plugin. > > I think we need a way to tell in tuned.conf which sysctl and sysfs settings > are required and which are optional. For optional ones, it's fine to > just print a warning if it fails to be set. For required ones, we have > to fail. Can you specify what percisely you mean by "have to fail"? Would an error message in the log be enough? > OK, so this BZ is about rt-entsk.service at this point. So can you confirm that starting rt-entsk.service is still relevant for RHEL-8? (I want to make sure that's the case, because it's possible adding the rt-setup package to RHEL-8 now won't be easy.) Thanks. (In reply to Ondřej Lysoněk from comment #5) > (In reply to Luiz Capitulino from comment #4) > > > Either way, perhaps we should print a warning instead of an error to the > > > logs when a sysctl setting doesn't exist. We can fix that along with > > > bug#1666678, which deals with reworking the sysctl plugin. > > > > I think we need a way to tell in tuned.conf which sysctl and sysfs settings > > are required and which are optional. For optional ones, it's fine to > > just print a warning if it fails to be set. For required ones, we have > > to fail. > > Can you specify what percisely you mean by "have to fail"? Would an error > message in the log be enough? Yes, having the error message is enough (like it is today). However, it has been my dream to see tuned fail with more verbosity and rolling back all the profile. By more verbosity I mean failing in systemd and maybe also writing it to /var/log/messages. > > OK, so this BZ is about rt-entsk.service at this point. > > So can you confirm that starting rt-entsk.service is still relevant for > RHEL-8? (I want to make sure that's the case, because it's possible adding > the rt-setup package to RHEL-8 now won't be easy.) I can't, it has to be Marcelo. I didn't even know/remember that we had rt-entsk.service... (In reply to Luiz Capitulino from comment #6) > (In reply to Ondřej Lysoněk from comment #5) > > (In reply to Luiz Capitulino from comment #4) > > > > Either way, perhaps we should print a warning instead of an error to the > > > > logs when a sysctl setting doesn't exist. We can fix that along with > > > > bug#1666678, which deals with reworking the sysctl plugin. > > > > > > I think we need a way to tell in tuned.conf which sysctl and sysfs settings > > > are required and which are optional. For optional ones, it's fine to > > > just print a warning if it fails to be set. For required ones, we have > > > to fail. > > > > Can you specify what percisely you mean by "have to fail"? Would an error > > message in the log be enough? > > Yes, having the error message is enough (like it is today). However, it has > been my dream to see tuned fail with more verbosity and rolling back all > the profile. By more verbosity I mean failing in systemd and maybe also > writing it to /var/log/messages. > > > > OK, so this BZ is about rt-entsk.service at this point. > > > > So can you confirm that starting rt-entsk.service is still relevant for > > RHEL-8? (I want to make sure that's the case, because it's possible adding > > the rt-setup package to RHEL-8 now won't be easy.) > > I can't, it has to be Marcelo. I didn't even know/remember that we had > rt-entsk.service... Yes it is necessary to workaround the chronyd bug. This is the bug: [Bug 1619407] rt-setup: include enable-netsocket-tstamp-static-key.c [RT-8 ] https://bugzilla.redhat.com/show_bug.cgi?id=1619407 It should be fixed by pulling the daemon rt-entsk.service from rt-setup upstream. Clark, how do we install rt-setup on RHEL-8 ? Is there a separate repo?(In reply to Ondřej Lysoněk from comment #3) > (In reply to Ondřej Lysoněk from comment #1) > > > 2019-01-16 03:15:43,755 ERROR tuned.plugins.plugin_script: script > > > '/usr/lib/tuned/realtime-virtual-host/script.sh' error output: 'Failed to > > > start rt-entsk.service: Unit rt-entsk.service not found.' > > > > The rt-entsk.service needs to be installed. On RHEL-7, it's provided by the > > rt-setup package, but I don't see it in RHEL-8. > > Unfortunatelly, the rt-setup package is not available in a public RHEL-8 > repo, it's available only in buildroot. Well, installing it should fix it. > > Marcelo, you added the rt-entsk.service handling to the realtime Tuned > profiles. Can you please confirm whether starting the service is still > needed on RHEL-8? (In reply to Luiz Capitulino from comment #4) > > > 2019-01-16 03:15:43,060 ERROR tuned.utils.commands: Executing sysctl > > > error: sysctl: cannot stat /proc/sys/kernel/numa_balancing: No such file or > > > directory > > > > I don't see the kernel.numa_balancing sysctl setting even in RHEL-7. Perhaps > > it shows up only on certain hardware? > > I guess it's (intentionally) disabled for the RT kernel. > > > Either way, perhaps we should print a warning instead of an error to the > > logs when a sysctl setting doesn't exist. We can fix that along with > > bug#1666678, which deals with reworking the sysctl plugin. > > I think we need a way to tell in tuned.conf which sysctl and sysfs settings > are required and which are optional. For optional ones, it's fine to > just print a warning if it fails to be set. For required ones, we have > to fail. This will be addressed in rhbz#1669171. |