Bug 2221016

Summary:	Add 'numad' and 'numactl'
Product:	Red Hat Enterprise Linux 8	Reporter:	Rupesh Patel <rupatel>
Component:	toolbox-container	Assignee:	Debarshi Ray <debarshir>
Status:	CLOSED WONTFIX	QA Contact:	Petr Schindler <pschindl>
Severity:	medium	Docs Contact:	Gabriela Nečasová <gnecasov>
Priority:	medium
Version:	8.9	CC:	desktop-qa-list, jwboyer, tpopela, travier
Target Milestone:	rc
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-07-13 08:24:52 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Rupesh Patel 2023-07-07 03:27:08 UTC

With NUMA aware workload feature in openshift [0], with toolbox (sosreport) we have to collect numa related information [1] from NUMA based systems. I think sosreport checks for these rpms [2] with numa plugin enabled    

[0] https://docs.openshift.com/container-platform/4.13/scalability_and_performance/cnf-numa-aware-scheduling.html

[1]

sos/report/plugins/numa.py:            "numastat",
sos/report/plugins/numa.py:            "numastat -m",
sos/report/plugins/numa.py:            "numastat -n",
sos/report/plugins/numa.py:            "numactl --show",
sos/report/plugins/numa.py:            "numactl --hardware",

[2]

sos/report/plugins/numa.py:    plugin_name = 'numa'
sos/report/plugins/numa.py:    packages = ('numad', 'numactl')

Comment 1 Rupesh Patel 2023-07-07 03:32:46 UTC

Just an example where we do check those commands output 

https://github.com/kubernetes/kubernetes/issues/84597

Comment 2 Debarshi Ray 2023-07-07 13:28:03 UTC

Is there any reason for comment 1 to be private?  If not, could you please open it up?  It will help people follow along.

Comment 3 Debarshi Ray 2023-07-07 13:44:09 UTC

Red Hat OpenShift Container Platform uses a different 'toolbox' implementation than Red Hat Enterprise Linux.  The former uses https://github.com/coreos/toolbox while that latter uses https://github.com/containers/toolbox

The toolbox-container component corresponds to the OCI images [1] used by the latter implementation.

Given that, I don't know if addressing this bug will help you in the context of OpenShift.  Will it?

Secondly, these OCI images [1] are built from UBI content, because they are also used by non-RHEL customers on non-RHEL hosts.  Specifically, Arch, Fedora and Ubuntu users can use them on their hosts.  As far as I can make out, only numactl-libs is part of UBI.  The actual numactl and numad RPMs are not.  That means we can't add them to these images without adding them to UBI first.

Third, are the numactl and numad RPMs installed by default on RHEL hosts?  The goal for these OCI images is to replicate the default command line user experience of a RHEL host.

I see that the sos RPM doesn't have any NUMA dependency:
https://gitlab.com/redhat/centos-stream/rpms/sos/-/blob/c9s/sos.spec

What do customers using sosreport on RHEL hosts do?  Do they install the NUMA RPMs separately?

[1] registry.access.redhat.com/ubi9/toolbox
    registry.access.redhat.com/ubi8/toolbox

Comment 4 Rupesh Patel 2023-07-11 04:42:29 UTC

Thank you for the clarity here. 

>> Given that, I don't know if addressing this bug will help you in the context of OpenShift.  Will it?

I think it will. 
Where do we host RHEL toolbox images? 
The toolbox uses images hosted at 'registry.redhat.io/rhelX/support-tools' and we can change this and use the customized image as well [0]. 

As you said, the OCI images are built from UBI which doesn't have these rpms, therefore, we have to first add these rpms to UBI images (which may be not accepted). As you said it is run by other non-RHEL hosts as well and maybe the size and all might also matter here).

>> Third, are the numactl and numad RPMs installed by default on RHEL hosts?  The goal for these OCI images is to replicate the default command line user experience of a RHEL host.
>> What do customers using sosreport on RHEL hosts do?  Do they install the NUMA RPMs separately?

Afaik those rpms are not installed by default but with RHEL it is easy to add them. With CoreOS it is not an option if UBI repos don't have it. 

tx

[0] https://docs.openshift.com/container-platform/4.13/support/gathering-cluster-data.html#starting-an-alternative-image-with-toolbox_gathering-cluster-data

Comment 5 Rupesh Patel 2023-07-11 04:49:53 UTC

Adding more details here, this page have some more details - https://access.redhat.com/solutions/4929021

Comment 6 Debarshi Ray 2023-07-11 12:46:21 UTC

(In reply to Rupesh Patel from comment #4)
> Thank you for the clarity here. 
> 
> >> Given that, I don't know if addressing this bug will help you in the context of OpenShift.  Will it?
> 
> I think it will. 
> Where do we host RHEL toolbox images?

registry.access.redhat.com/ubi9/toolbox
https://catalog.redhat.com/software/containers/ubi9/toolbox/61532d7dd2c7f84a4d2ed86b

registry.access.redhat.com/ubi8/toolbox
https://catalog.redhat.com/software/containers/ubi8/toolbox/611bd665bd674341b5c5ed46

> The toolbox uses images hosted at 'registry.redhat.io/rhelX/support-tools'
> and we can change this and use the customized image as well [0]. 

Not, 'toolbox', but github.com/coreos/toolbox uses registry.redhat.io/rhelX/support-tools.  The terminology is important because github.com/containers/toolbox exists, and is what most people know as 'toolbox'.  :)

In theory anybody can use the UBI 'toolbox' images above, but they are meant to be only used with github.com/containers/toolbox.  So bugs arising out of other unsupported use-cases might get WONTFIX:ed.

The right thing to do would be for Red Hat OpenShift Container Platform to also use github.com/containers/toolbox.  This has been discussed many times before already, and I think we cover all the features.  I suppose it's a matter of everybody having lots to do.

> As you said, the OCI images are built from UBI which doesn't have these
> rpms, therefore, we have to first add these rpms to UBI images (which may be
> not accepted). As you said it is run by other non-RHEL hosts as well and
> maybe the size and all might also matter here).

Yes, the size of the images is a concern.  Note that github.com/containers/toolbox is very widely used rootless as a development environment, and it's not just for troubleshooting with root privileges.  So, it's a matter of striking a balance across a variety of use-cases.

That's why we draw the line at what's installed by default on RHEL hosts.

I think it's worth exploring if numactl and numad can be added to UBI.  If they do get added, it doesn't mean that we have to include them in the UBI 'toolbox' images.  However, it's still going to be easier for anybody to install them if needed, which is still an improvement.
 
> >> Third, are the numactl and numad RPMs installed by default on RHEL hosts?
> >> The goal for these OCI images is to replicate the default command line user
> >> experience of a RHEL host.
> >> What do customers using sosreport on RHEL hosts do?  Do they install the NUMA RPMs separately?
> 
> Afaik those rpms are not installed by default but with RHEL it is easy to
> add them. With CoreOS it is not an option if UBI repos don't have it. 

Out of curiosity, are the numactl and numad RPMs installed by default on Red Hat CoreOS?

Comment 8 Debarshi Ray 2023-07-11 13:02:52 UTC

(In reply to Rupesh Patel from comment #5)
> Adding more details here, this page have some more details -
> https://access.redhat.com/solutions/4929021

Thanks you for this link.  It does make some things a lot clearer.

I didn't know that Red Hat CoreOS nodes aren't subscribed through subscription-manager in the typical RHEL way.

This part is interesting:

> Note: a message like
> Error: registry.fedoraproject.org/f33/fedora-toolbox:latest does
> not have a label of RUN is not an actual error, but just emitted
> while checking if the image has a RUN label. If it hasn't, it
> defaults to running a privileged container with host filesystem
> mounted (pretty similar to what RHEL8 support tools run label
> does). So, in most cases, this can be disregarded.

The registry.fedoraproject.org/f33/fedora-toolbox:latest image is explicitly for github.com/containers/toolbox.  That message is being caused by using the image with something else.  This is an example of 'bugs arising out of other unsupported use-cases might get WONTFIX:ed' in comment 6

These days there's a whole ecosystem of images targeting github.com/containers/toolbox beyond those that are part of the project.  eg., Timothée Ravier (from CoreOS) is a steward for https://github.com/toolbx-images/images

This is why it would be good if RHOCP could (some day) move over to github.com/containers/toolbox.

(CCed Timothée to keep him in the loop.)

Comment 9 Josh Boyer 2023-07-11 13:37:46 UTC

(In reply to Rupesh Patel from comment #4)
> Thank you for the clarity here. 
> 
> >> Given that, I don't know if addressing this bug will help you in the context of OpenShift.  Will it?
> 
> I think it will. 
> Where do we host RHEL toolbox images? 
> The toolbox uses images hosted at 'registry.redhat.io/rhelX/support-tools'
> and we can change this and use the customized image as well [0]. 

We can't add these packages to UBI, but the support-tools container can include them directly.

Comment 10 Debarshi Ray 2023-07-11 16:09:23 UTC

(In reply to Josh Boyer from comment #9)
> 
> We can't add these packages to UBI

Okay, understood!

Rupesh, I am tempted to close this bug (as WONTFIX?).

Comment 11 Rupesh Patel 2023-07-13 04:46:43 UTC

(In reply to Debarshi Ray from comment #10)
> (In reply to Josh Boyer from comment #9)
> > 
> > We can't add these packages to UBI
> 
> Okay, understood!
> 
> Rupesh, I am tempted to close this bug (as WONTFIX?).

Ack, I'll check with the team's own support-tools.

Comment 12 Rupesh Patel 2023-07-13 04:55:56 UTC

(In reply to Debarshi Ray from comment #10)
> (In reply to Josh Boyer from comment #9)
> > 
> > We can't add these packages to UBI
> 
> Okay, understood!
> 
> Rupesh, I am tempted to close this bug (as WONTFIX?).

@Debarshi, is it fine if I move the Jira to a different project? Will closing this bz also close Jira?

Comment 13 Debarshi Ray 2023-07-13 08:15:36 UTC

(In reply to Rupesh Patel from comment #12)
> (In reply to Debarshi Ray from comment #10)
> > 
> > Rupesh, I am tempted to close this bug (as WONTFIX?).
> 
> @Debarshi, is it fine if I move the Jira to a different project?

Sure, feel free to move it.

> Will
> closing this bz also close Jira?

I don't know.

Comment 14 Tomas Popela 2023-07-13 08:17:16 UTC

(In reply to Debarshi Ray from comment #13)
> (In reply to Rupesh Patel from comment #12)
> > (In reply to Debarshi Ray from comment #10)
> > > 
> > > Rupesh, I am tempted to close this bug (as WONTFIX?).
> > 
> > @Debarshi, is it fine if I move the Jira to a different project?
> 
> Sure, feel free to move it.
> 
> > Will
> > closing this bz also close Jira?
> 
> I don't know.

That JIRA is a mirror of this bug, please don't do anything with it as it will be closed automatically once this bug is closed.

Comment 15 RHEL Program Management 2023-07-13 08:24:52 UTC

Development Management has reviewed and declined this request. You may appeal this decision by reopening this request.