With NUMA aware workload feature in openshift [0], with toolbox (sosreport) we have to collect numa related information [1] from NUMA based systems. I think sosreport checks for these rpms [2] with numa plugin enabled [0] https://docs.openshift.com/container-platform/4.13/scalability_and_performance/cnf-numa-aware-scheduling.html [1] sos/report/plugins/numa.py: "numastat", sos/report/plugins/numa.py: "numastat -m", sos/report/plugins/numa.py: "numastat -n", sos/report/plugins/numa.py: "numactl --show", sos/report/plugins/numa.py: "numactl --hardware", [2] sos/report/plugins/numa.py: plugin_name = 'numa' sos/report/plugins/numa.py: packages = ('numad', 'numactl')
Just an example where we do check those commands output https://github.com/kubernetes/kubernetes/issues/84597
Is there any reason for comment 1 to be private? If not, could you please open it up? It will help people follow along.
Red Hat OpenShift Container Platform uses a different 'toolbox' implementation than Red Hat Enterprise Linux. The former uses https://github.com/coreos/toolbox while that latter uses https://github.com/containers/toolbox The toolbox-container component corresponds to the OCI images [1] used by the latter implementation. Given that, I don't know if addressing this bug will help you in the context of OpenShift. Will it? Secondly, these OCI images [1] are built from UBI content, because they are also used by non-RHEL customers on non-RHEL hosts. Specifically, Arch, Fedora and Ubuntu users can use them on their hosts. As far as I can make out, only numactl-libs is part of UBI. The actual numactl and numad RPMs are not. That means we can't add them to these images without adding them to UBI first. Third, are the numactl and numad RPMs installed by default on RHEL hosts? The goal for these OCI images is to replicate the default command line user experience of a RHEL host. I see that the sos RPM doesn't have any NUMA dependency: https://gitlab.com/redhat/centos-stream/rpms/sos/-/blob/c9s/sos.spec What do customers using sosreport on RHEL hosts do? Do they install the NUMA RPMs separately? [1] registry.access.redhat.com/ubi9/toolbox registry.access.redhat.com/ubi8/toolbox
Thank you for the clarity here. >> Given that, I don't know if addressing this bug will help you in the context of OpenShift. Will it? I think it will. Where do we host RHEL toolbox images? The toolbox uses images hosted at 'registry.redhat.io/rhelX/support-tools' and we can change this and use the customized image as well [0]. As you said, the OCI images are built from UBI which doesn't have these rpms, therefore, we have to first add these rpms to UBI images (which may be not accepted). As you said it is run by other non-RHEL hosts as well and maybe the size and all might also matter here). >> Third, are the numactl and numad RPMs installed by default on RHEL hosts? The goal for these OCI images is to replicate the default command line user experience of a RHEL host. >> What do customers using sosreport on RHEL hosts do? Do they install the NUMA RPMs separately? Afaik those rpms are not installed by default but with RHEL it is easy to add them. With CoreOS it is not an option if UBI repos don't have it. tx [0] https://docs.openshift.com/container-platform/4.13/support/gathering-cluster-data.html#starting-an-alternative-image-with-toolbox_gathering-cluster-data
Adding more details here, this page have some more details - https://access.redhat.com/solutions/4929021
(In reply to Rupesh Patel from comment #4) > Thank you for the clarity here. > > >> Given that, I don't know if addressing this bug will help you in the context of OpenShift. Will it? > > I think it will. > Where do we host RHEL toolbox images? registry.access.redhat.com/ubi9/toolbox https://catalog.redhat.com/software/containers/ubi9/toolbox/61532d7dd2c7f84a4d2ed86b registry.access.redhat.com/ubi8/toolbox https://catalog.redhat.com/software/containers/ubi8/toolbox/611bd665bd674341b5c5ed46 > The toolbox uses images hosted at 'registry.redhat.io/rhelX/support-tools' > and we can change this and use the customized image as well [0]. Not, 'toolbox', but github.com/coreos/toolbox uses registry.redhat.io/rhelX/support-tools. The terminology is important because github.com/containers/toolbox exists, and is what most people know as 'toolbox'. :) In theory anybody can use the UBI 'toolbox' images above, but they are meant to be only used with github.com/containers/toolbox. So bugs arising out of other unsupported use-cases might get WONTFIX:ed. The right thing to do would be for Red Hat OpenShift Container Platform to also use github.com/containers/toolbox. This has been discussed many times before already, and I think we cover all the features. I suppose it's a matter of everybody having lots to do. > As you said, the OCI images are built from UBI which doesn't have these > rpms, therefore, we have to first add these rpms to UBI images (which may be > not accepted). As you said it is run by other non-RHEL hosts as well and > maybe the size and all might also matter here). Yes, the size of the images is a concern. Note that github.com/containers/toolbox is very widely used rootless as a development environment, and it's not just for troubleshooting with root privileges. So, it's a matter of striking a balance across a variety of use-cases. That's why we draw the line at what's installed by default on RHEL hosts. I think it's worth exploring if numactl and numad can be added to UBI. If they do get added, it doesn't mean that we have to include them in the UBI 'toolbox' images. However, it's still going to be easier for anybody to install them if needed, which is still an improvement. > >> Third, are the numactl and numad RPMs installed by default on RHEL hosts? > >> The goal for these OCI images is to replicate the default command line user > >> experience of a RHEL host. > >> What do customers using sosreport on RHEL hosts do? Do they install the NUMA RPMs separately? > > Afaik those rpms are not installed by default but with RHEL it is easy to > add them. With CoreOS it is not an option if UBI repos don't have it. Out of curiosity, are the numactl and numad RPMs installed by default on Red Hat CoreOS?
(In reply to Rupesh Patel from comment #5) > Adding more details here, this page have some more details - > https://access.redhat.com/solutions/4929021 Thanks you for this link. It does make some things a lot clearer. I didn't know that Red Hat CoreOS nodes aren't subscribed through subscription-manager in the typical RHEL way. This part is interesting: > Note: a message like > Error: registry.fedoraproject.org/f33/fedora-toolbox:latest does > not have a label of RUN is not an actual error, but just emitted > while checking if the image has a RUN label. If it hasn't, it > defaults to running a privileged container with host filesystem > mounted (pretty similar to what RHEL8 support tools run label > does). So, in most cases, this can be disregarded. The registry.fedoraproject.org/f33/fedora-toolbox:latest image is explicitly for github.com/containers/toolbox. That message is being caused by using the image with something else. This is an example of 'bugs arising out of other unsupported use-cases might get WONTFIX:ed' in comment 6 These days there's a whole ecosystem of images targeting github.com/containers/toolbox beyond those that are part of the project. eg., Timothée Ravier (from CoreOS) is a steward for https://github.com/toolbx-images/images This is why it would be good if RHOCP could (some day) move over to github.com/containers/toolbox. (CCed Timothée to keep him in the loop.)
(In reply to Rupesh Patel from comment #4) > Thank you for the clarity here. > > >> Given that, I don't know if addressing this bug will help you in the context of OpenShift. Will it? > > I think it will. > Where do we host RHEL toolbox images? > The toolbox uses images hosted at 'registry.redhat.io/rhelX/support-tools' > and we can change this and use the customized image as well [0]. We can't add these packages to UBI, but the support-tools container can include them directly.
(In reply to Josh Boyer from comment #9) > > We can't add these packages to UBI Okay, understood! Rupesh, I am tempted to close this bug (as WONTFIX?).
(In reply to Debarshi Ray from comment #10) > (In reply to Josh Boyer from comment #9) > > > > We can't add these packages to UBI > > Okay, understood! > > Rupesh, I am tempted to close this bug (as WONTFIX?). Ack, I'll check with the team's own support-tools.
(In reply to Debarshi Ray from comment #10) > (In reply to Josh Boyer from comment #9) > > > > We can't add these packages to UBI > > Okay, understood! > > Rupesh, I am tempted to close this bug (as WONTFIX?). @Debarshi, is it fine if I move the Jira to a different project? Will closing this bz also close Jira?
(In reply to Rupesh Patel from comment #12) > (In reply to Debarshi Ray from comment #10) > > > > Rupesh, I am tempted to close this bug (as WONTFIX?). > > @Debarshi, is it fine if I move the Jira to a different project? Sure, feel free to move it. > Will > closing this bz also close Jira? I don't know.
(In reply to Debarshi Ray from comment #13) > (In reply to Rupesh Patel from comment #12) > > (In reply to Debarshi Ray from comment #10) > > > > > > Rupesh, I am tempted to close this bug (as WONTFIX?). > > > > @Debarshi, is it fine if I move the Jira to a different project? > > Sure, feel free to move it. > > > Will > > closing this bz also close Jira? > > I don't know. That JIRA is a mirror of this bug, please don't do anything with it as it will be closed automatically once this bug is closed.
Development Management has reviewed and declined this request. You may appeal this decision by reopening this request.