RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1784645 - Need a new tuned plug-in, irqbalance
Summary: Need a new tuned plug-in, irqbalance
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: tuned
Version: 8.3
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 8.0
Assignee: Jaroslav Škarvada
QA Contact: Robin Hack
URL:
Whiteboard:
Depends On:
Blocks: 1817044 1825061
TreeView+ depends on / blocked
 
Reported: 2019-12-17 22:01 UTC by Andrew Theurer
Modified: 2020-11-04 02:07 UTC (History)
9 users (show)

Fixed In Version: tuned-2.14.0-0.1.rc1.el8
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-11-04 02:03:07 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github redhat-performance tuned pull 274 0 None closed Add irqbalance plugin 2021-02-17 22:49:09 UTC
Red Hat Product Errata RHBA-2020:4559 0 None None None 2020-11-04 02:03:27 UTC

Description Andrew Theurer 2019-12-17 22:01:06 UTC
Description of problem:

The current method of configuring irqbalance requires the use of the script plug-in.  Supporting the script plug-in for RHEL-based CoreOS is problematic.  To fix this, we would like to have an official "irqbalance" plug-in for tuned.  In this plug-in, the user would be able to specify the CPUs which should not have interrupts delivered.  This cpu list should be able to use the $isolated_cores variable defined in a tuned profile's configuration file (such as /etc/tuned-cpu-partitioning-variables.conf).

Comment 1 Andrew Theurer 2019-12-17 22:04:44 UTC
When applying or re-applying this profile, tuned would update the /etc/sysconfig/irqbalance config file and restart the irqbalance service.

Comment 2 Marcelo Tosatti 2020-01-10 01:53:01 UTC
Andrew, why exactly a new irqbalance plugin for tuned is necessary?

What is necessary is that the Tuned operator performs the configurations
necessary, in /sys/, to add or remove certain CPUs from the interrupt mask 
of irqs.

Can you please explain?

Comment 3 Andrew Theurer 2020-01-17 13:08:50 UTC
Marcelo, I am not sure I understand your question, but the request here is based on avoiding the use of 'script' plug-in, and changing the irqbalance configuration currently happens with 'script' plug-in.  There is no configuration that happens in /sys for irqs.  And as long as irqbalance daemon is active, it is necessary to tell it to not use certain CPUs.  There is no plan for tuned to alter /proc/irq/N/smp_affinity masks directly (while having irqbalance disabled).

Comment 4 Marcelo Tosatti 2020-01-20 22:15:33 UTC
(In reply to Andrew Theurer from comment #3)
> Marcelo, I am not sure I understand your question, but the request here is
> based on avoiding the use of 'script' plug-in, and changing the irqbalance
> configuration currently happens with 'script' plug-in.  There is no
> configuration that happens in /sys for irqs.  And as long as irqbalance
> daemon is active, it is necessary to tell it to not use certain CPUs.  There
> is no plan for tuned to alter /proc/irq/N/smp_affinity masks directly (while
> having irqbalance disabled).

OK!

Comment 5 Jiří Mencák 2020-01-31 14:18:07 UTC
Would https://github.com/redhat-performance/tuned/pull/243 work until this is implemented properly as a tuned plugin?

Comment 6 Andrew Theurer 2020-02-03 13:49:33 UTC
(In reply to jmencak from comment #5)
> Would https://github.com/redhat-performance/tuned/pull/243 work until this
> is implemented properly as a tuned plugin?

It depends, what calls irqbalance_banned_cpus_setup() and is this something you can do in NTO?

Comment 7 Jiří Mencák 2020-02-03 14:24:09 UTC
(In reply to Andrew Theurer from comment #6)
> (In reply to jmencak from comment #5)
> > Would https://github.com/redhat-performance/tuned/pull/243 work until this
> > is implemented properly as a tuned plugin?
> 
> It depends, what calls irqbalance_banned_cpus_setup() and is this something
> you can do in NTO?

Both the cpu-partitioning and realtime profiles call this via the script plugin.
Yes, I can use/do this in NTO.

Comment 8 Ondřej Lysoněk 2020-02-05 10:05:24 UTC
(In reply to Andrew Theurer from comment #1)
> When applying or re-applying this profile, tuned would update the
> /etc/sysconfig/irqbalance config file and restart the irqbalance service.

I think 'systemctl try-restart' is more appropriate instead of a hard restart, isn't it?

(In reply to Andrew Theurer from comment #3)
> There
> is no plan for tuned to alter /proc/irq/N/smp_affinity masks directly (while
> having irqbalance disabled).

I'm not sure what you're trying to say here, but note that the scheduler plugin already sets these affinities. To fixed values, that is. It doesn't do any balancing.

[1] https://github.com/redhat-performance/tuned/blob/041333f9c5daf37f96340f418f4168347551f52a/tuned/plugins/plugin_scheduler.py#L572

Comment 11 Ondřej Lysoněk 2020-04-23 12:18:39 UTC
I've been looking into this, and it's unclear to me whether we need to set IRQBALANCE_BANNED_CPUS at all. At least on RHEL-8, irqbalance parses /sys/devices/system/cpu/isolated and /sys/devices/system/cpu/nohz_full and uses that as the default value for the list of banned CPUs. So the list of banned CPUs seems to be automatically populated just like we want it. I've tested it (with modified cpu-partitioning and realtime* profiles; on regular, non-RT kernel) and it seems to work.

The only case this would not work would be if someone explicitly set IRQBALANCE_BANNED_CPUS in /etc/sysconfig/irqbalance, as that would override the autodetection. Question is, do we care? If the admin explicitly set the variable, they likely have a good reason for it and Tuned shouldn't touch it.

Am I missing something? Can someone provide more context? Andrew, Luiz?

Thanks.

I'll go check how it behaves on RHEL-7.

Comment 12 Ondřej Lysoněk 2020-04-23 16:23:16 UTC
So I've done some testing with different RHEL releases and all related Tuned profiles (cpu-partitioning, realtime*). I've modified the profiles so that they don't touch the irqbalance sysconfig file. irqbalance (almost) always automatically populates the list of banned CPUs with correct values. Here's a summary:
- RHEL-8.3, RHEL-8.3 with RT kernel, RHEL-7.9, RHEL-7.4, RHEL-7.2: list of banned CPUs is correctly set
- RHEL-7.3: there seems to be a bug - the list of banned CPUs contains one more CPU than is necessary

For the record, the way I got the effective list of banned CPUs on RHEL-8 is this:
python3 -c "import socket; s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM); s.connect(b'\\0irqbalance$(pidof irqbalance).sock'); s.send(b'setup'); print(s.recv(1024)); s.close()"

On RHEL-7, the following was necessary:
gdb -p $(pidof irqbalance) -ex 'print banned_cpus' -ex detach -ex quit

Comment 13 Ondřej Lysoněk 2020-04-23 16:27:45 UTC
It looks to me like we don't really need an irqbalance plugin, because irqbalance sets the list of banned CPUs automatically. I think we should simply drop the irqbalance config handling from Tuned profiles entirely. (Please see my previous comments). What do you guys think?

Comment 14 Jiří Mencák 2020-05-14 07:36:59 UTC
(In reply to Ondřej Lysoněk from comment #11)
> I've been looking into this, and it's unclear to me whether we need to set
> IRQBALANCE_BANNED_CPUS at all. At least on RHEL-8, irqbalance parses
> /sys/devices/system/cpu/isolated and /sys/devices/system/cpu/nohz_full and
> uses that as the default value for the list of banned CPUs. So the list of
> banned CPUs seems to be automatically populated just like we want it. I've
> tested it (with modified cpu-partitioning and realtime* profiles; on
> regular, non-RT kernel) and it seems to work.
> 
> The only case this would not work would be if someone explicitly set
> IRQBALANCE_BANNED_CPUS in /etc/sysconfig/irqbalance, as that would override
> the autodetection. Question is, do we care? If the admin explicitly set the
> variable, they likely have a good reason for it and Tuned shouldn't touch it.
> 
> Am I missing something? Can someone provide more context? Andrew, Luiz?

Looking at the code of irqbalance 1.4.0/cputree.c/setup_banned_cpus(),
I think the problem is the assumption the profiles (cpu-partitioning/realtime)
will always use isolcpus and/or nohz_full.  This may not the case (especially
for other profiles and possibly for profiles inheriting these and perhaps 
removing isolcpus/nohz_full), therefore the plugin is still needed AFAICS.

> 
> Thanks.
> 
> I'll go check how it behaves on RHEL-7.

Comment 15 Ondřej Lysoněk 2020-05-18 09:45:07 UTC
(In reply to jmencak from comment #14)
> Looking at the code of irqbalance 1.4.0/cputree.c/setup_banned_cpus(),
> I think the problem is the assumption the profiles
> (cpu-partitioning/realtime)
> will always use isolcpus and/or nohz_full.  This may not the case (especially
> for other profiles and possibly for profiles inheriting these and perhaps 
> removing isolcpus/nohz_full)

You say "especially". Can you elaborate on when it's not the case with our profiles?

Is there a use case for removing isolcpus/nonhz_full while still tuning irqbalance? Let's talk real use cases, not hypothetical ones.

Comment 16 Ondřej Lysoněk 2020-05-18 11:30:34 UTC
(In reply to Ondřej Lysoněk from comment #15)
> (In reply to jmencak from comment #14)
> > Looking at the code of irqbalance 1.4.0/cputree.c/setup_banned_cpus(),
> > I think the problem is the assumption the profiles
> > (cpu-partitioning/realtime)
> > will always use isolcpus and/or nohz_full.  This may not the case (especially
> > for other profiles and possibly for profiles inheriting these and perhaps 
> > removing isolcpus/nohz_full)
> 
> You say "especially". Can you elaborate on when it's not the case with our
> profiles?
> 
> Is there a use case for removing isolcpus/nonhz_full while still tuning
> irqbalance? Let's talk real use cases, not hypothetical ones.

Perhaps I misunderstood what you were getting at. Are you saying that someone might already be doing what you described? And that we shouldn't remove irqbalance handling from our profiles because it might break someone's setup?

Comment 17 Andrew Theurer 2020-05-18 12:12:40 UTC
(In reply to Ondřej Lysoněk from comment #15)
> (In reply to jmencak from comment #14)
> > Looking at the code of irqbalance 1.4.0/cputree.c/setup_banned_cpus(),
> > I think the problem is the assumption the profiles
> > (cpu-partitioning/realtime)
> > will always use isolcpus and/or nohz_full.  This may not the case (especially
> > for other profiles and possibly for profiles inheriting these and perhaps 
> > removing isolcpus/nohz_full)
> 
> You say "especially". Can you elaborate on when it's not the case with our
> profiles?
> 
> Is there a use case for removing isolcpus/nonhz_full while still tuning
> irqbalance? Let's talk real use cases, not hypothetical ones.

We already have the situation where we can't use nohz_full because of problems with CFS-quota, but we still need to control where interrupts go.  This is for Openshift.

Comment 18 Ondřej Lysoněk 2020-05-18 15:42:33 UTC
(In reply to Andrew Theurer from comment #17)
> (In reply to Ondřej Lysoněk from comment #15)
> > (In reply to jmencak from comment #14)
> > > Looking at the code of irqbalance 1.4.0/cputree.c/setup_banned_cpus(),
> > > I think the problem is the assumption the profiles
> > > (cpu-partitioning/realtime)
> > > will always use isolcpus and/or nohz_full.  This may not the case (especially
> > > for other profiles and possibly for profiles inheriting these and perhaps 
> > > removing isolcpus/nohz_full)
> > 
> > You say "especially". Can you elaborate on when it's not the case with our
> > profiles?
> > 
> > Is there a use case for removing isolcpus/nonhz_full while still tuning
> > irqbalance? Let's talk real use cases, not hypothetical ones.
> 
> We already have the situation where we can't use nohz_full because of
> problems with CFS-quota, but we still need to control where interrupts go. 
> This is for Openshift.

Hi Andrew,

thank you very much for the information!

So to expand on that, I suppose an example of a use case would be using the cpu-partitioning profile with the nohz_full setting removed. Given that cpu-partitioning doesn't use isolcpus, neither /sys/devices/system/cpu/isolated nor /sys/devices/system/cpu/nohz_full will be set in such a case, so irqbalance has no way of knowing what the banned CPUs should be.

Comment 19 Andrew Theurer 2020-05-18 19:59:49 UTC
Yes, that is what we are doing on OCP, but we are not quite to the point where we actually use cpu-partitioning today (tuned is not used yet, but NTO/tuned will be for this work very soon), but that is the plan for OCP 4.6: using a cpu-part profile, but with nohz_full removed, but still need the irqs moved away.

Comment 20 Ondřej Lysoněk 2020-06-03 08:14:30 UTC
Upstream PR:
https://github.com/redhat-performance/tuned/pull/274

Comment 31 errata-xmlrpc 2020-11-04 02:03:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (tuned bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4559


Note You need to log in before you can comment on or make changes to this bug.