Bug 2030580

Summary: "tuned-adm recommend" reports non-existing "atomic-guest"
Product: Red Hat Enterprise Linux 9 Reporter: Martin Pitt <mpitt>
Component: tunedAssignee: Jaroslav Škarvada <jskarvad>
Status: CLOSED ERRATA QA Contact: Robin Hack <rhack>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.0CC: bstinson, jeder, jskarvad, jwboyer, jzerdik, mmarusak, ykarel
Target Milestone: rcKeywords: Patch, Regression, Reopened, TestCaseNeeded, Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: CockpitTest
Fixed In Version: tuned-2.19.0-0.1.rc1.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-15 11:17:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Martin Pitt 2021-12-09 08:09:45 UTC
Description of problem: About two days ago, cockpit's tuned tests spontaneously broke in CentOS 9 Stream. I am investigating this in [1], but the root cause of the problem is this:

  # tuned-adm recommend
  atomic-guest

  # tuned-adm auto_profile
  Requested profile 'atomic-guest' doesn't exist.

(see [2] for the full log)

It is certainly plausible to clean up the "atomic-guest" name, as atomic has not been a thing since RHEL 7 - but then it should not recommend this profile.

This is a bit difficult to investigate locally. So far it only happens on Testing Farm machines (packit, Fedora/RHEL gating), but in a local VM it says "virtual-guest". So these Testing Farm machines are AWS EC2 instances I think, but under the hood they should also just be qemu -- but apparently with some magic options.


Version-Release number of selected component (if applicable):

kernel 5.14.0-22.el9.x86_64
tuned 2.16.0-4.el9 

All other packae versions in [2]

Additional info:

[1] https://github.com/cockpit-project/cockpit/pull/16709
[2] http://artifacts.dev.testing-farm.io/8fbea3bd-2161-4234-94e7-ad632de41539/
[3] http://artifacts.dev.testing-farm.io/8fbea3bd-2161-4234-94e7-ad632de41539/work-allq6efFs/log.txt

Comment 1 Martin Pitt 2021-12-09 08:13:00 UTC
Adding regression as this was fine a few days ago. Three days ago there was a new version, but it only contained a dependency change [1], so apparently something more subtle going on due to the rebuild?

One more data point: `tuned-adm recommend` says "atomic-guest" both when tuned.service is not running (then it additionally warns about "Cannot talk to TuneD daemon via DBus. Is TuneD daemon running?"). In https://github.com/cockpit-project/cockpit/pull/16709 I flip this around to avoid the warning, and was hoping that running tuned would have a better answer -- but it doesn't.

[1] https://gitlab.com/redhat/centos-stream/rpms/tuned/-/commit/23f34274263f7a1364cec0ab1f0b908142ecd8de

Comment 2 Martin Pitt 2022-01-26 05:51:36 UTC
This is apparently fixed in 2.17, which is in current RHEL 9.0 nightlies. Thanks!

Comment 3 Martin Pitt 2022-01-26 08:11:57 UTC
Sorry, it still happens in current CentOS 9 Stream, and thus also in RHEL 9: http://artifacts.dev.testing-farm.io/c64adb27-6c06-41df-b790-b06c2e497c62/

Comment 6 Jaroslav Škarvada 2022-03-17 13:04:41 UTC
It seems it detects it as atomic. The system probably has atomic role in the syspurpose. The TuneD auto-rule is the following:

[atomic-guest]
virt=.+
syspurpose_role=.*atomic.*

I.e. it has to be virtual machine (not bare metal) and it has to have atomic substring in the syspurpose role.

Could you provide output of the following command:
# subscription-manager syspurpose --show

Comment 7 Martin Pitt 2022-03-18 07:18:12 UTC
It's just as much effort for me as for you to investigate it on the Testing Farm.. (Good opportunity to fix your c9s tests? [1])

I made a test PR with these commands [2]. In the first run [3] it shows the bug:

+ systemctl status tuned.service
○ tuned.service - Dynamic System Tuning Daemon
     Loaded: loaded (/usr/lib/systemd/system/tuned.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: man:tuned(8)
             man:tuned.conf(5)
             man:tuned-adm(8)
+ true
+ systemctl stop tuned.service
+ tuned-adm recommend
Cannot talk to TuneD daemon via DBus. Is TuneD daemon running?
2022-03-18 07:08:22,225 ERROR    tuned.utils.profile_recommender: Failed to process 'syspurpose_role' in '/usr/lib/tuned/recommend.d/50-tuned.conf', the syspurpose module is not available
2022-03-18 07:08:22,260 ERROR    tuned.utils.profile_recommender: Failed to process 'syspurpose_role' in '/usr/lib/tuned/recommend.d/50-tuned.conf', the syspurpose module is not available
atomic-guest
+ subscription-manager syspurpose --show
./run-verify-host.sh: line 9: subscription-manager: command not found
+ true
+ systemctl start tuned.service
+ tuned-adm recommend
atomic-guest
+ subscription-manager syspurpose --show
./run-verify-host.sh: line 13: subscription-manager: command not found
+ true
+ tuned-adm auto_profile
Requested profile 'atomic-guest' doesn't exist.


So it insists on atomic-guest, and subscription-manager is not installed.


[1] https://artifacts.dev.testing-farm.io/6da3a3af-4655-4fc3-86d3-ec1d4561d3de/
[2] https://github.com/cockpit-project/cockpit/pull/17148
[3] https://artifacts.dev.testing-farm.io/ef6aa8fb-55cc-496c-bbaa-d181b6494ffe/

Comment 8 Martin Pitt 2022-03-18 07:40:21 UTC
With version 2 [1] I installed subscription-manager, and then the test [2] actually succeeds:


+ systemctl status tuned.service
○ tuned.service - Dynamic System Tuning Daemon
     Loaded: loaded (/usr/lib/systemd/system/tuned.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: man:tuned(8)
             man:tuned.conf(5)
             man:tuned-adm(8)
+ true
+ systemctl stop tuned.service
+ tuned-adm recommend
Cannot talk to TuneD daemon via DBus. Is TuneD daemon running?
virtual-guest
+ subscription-manager syspurpose --show
{}
+ systemctl start tuned.service
+ tuned-adm recommend
virtual-guest
+ subscription-manager syspurpose --show
{}
+ tuned-adm auto_profile


So the existence or absence of subscription-manager influences the default profile. sub-man isn't installed on the c9s machines by default, nor on our test (and that makes sense -- it's not useful on centos).

[1] https://github.com/cockpit-project/cockpit/blob/3c620c9314d43ab20a3bee8afccd181f50f4cfc6/test/run-verify-host.sh
[2] https://artifacts.dev.testing-farm.io/35d317e8-e147-410a-8e8b-5193338f9da2/

Comment 9 Martin Pitt 2022-03-18 08:06:07 UTC
In version 3 [1] I removed the subscription-manager dependency again, and added `grep -r atomic /etc/ /var/`. The output is in the log [2], but there is nothing *at all* related to subscription-manager or the Atomic product.

This is trivial to reproduce and at that point this requires digging into tuned itself -- I'll leave that to you :-)

Thanks!

[1] https://github.com/cockpit-project/cockpit/blob/fcb4eebac8cf37201459dd3ad08fb4cbb1015b6e/test/run-verify-host.sh
[2] https://artifacts.dev.testing-farm.io/5855d107-fc13-4a73-a4a0-9e3b54f59c58/

Comment 10 Jaroslav Škarvada 2022-03-24 22:34:14 UTC
This is bug, thanks for reporting, upstream PR:
https://github.com/redhat-performance/tuned/pull/428

Comment 11 Jaroslav Škarvada 2022-03-24 22:42:24 UTC
I am moving it to the 9.0. It's easier for us to track. The fix should also get to the Cent OS stream.

Comment 12 Jaroslav Škarvada 2022-03-24 23:03:52 UTC
(In reply to Martin Pitt from comment #1)
> Adding regression as this was fine a few days ago. Three days ago there was
> a new version, but it only contained a dependency change [1], so apparently
> something more subtle going on due to the rebuild?
>
There was no related change on the TuneD side - the last change in the profile recommender was from the 2021-09-22 and in the recommend rules from the 2021-02-09. Before the commit you pointed out it always required subscription-manager on RHEL/CentOS, thus it probably worked because subscription manager was installed and there was no room for the TuneD bug (which was present all the time) to manifest.

> One more data point: `tuned-adm recommend` says "atomic-guest" both when
> tuned.service is not running (then it additionally warns about "Cannot talk
> to TuneD daemon via DBus. Is TuneD daemon running?"). In
> https://github.com/cockpit-project/cockpit/pull/16709 I flip this around to
> avoid the warning, and was hoping that running tuned would have a better
> answer -- but it doesn't.
> 
It should behave the same way no matter whether the TuneD daemon is running or not.

Comment 13 Jaroslav Škarvada 2022-03-24 23:13:21 UTC
(In reply to Martin Pitt from comment #7)
> It's just as much effort for me as for you to investigate it on the Testing
> Farm.. (Good opportunity to fix your c9s tests? [1])
> 
Our c9s tests should be OK. The test in question failed due to some error on the testing farm which is out of our reach. I.e. in this case it seems the testing farm RPM packages were signed with the wrong GPG key:

Problem opening package beakerlib-1.28-1.20220211112859719800.master.4.g484a443.el9.noarch.rpm. Failing package is: beakerlib-1.28-1.20220211112859719800.master.4.g484a443.el9.noarch
 GPG Keys are configured as: https://download.copr.fedorainfracloud.org/results/packit/beakerlib-beakerlib-master/pubkey.gpg
Error: GPG check FAILED

I.e. the failure had nothing to do with the TuneD or TuneD upstream test suite.

Comment 14 Martin Pitt 2022-03-25 05:33:21 UTC
Hello Jaroslav,

> Before the commit you pointed out it always required subscription-manager on RHEL/CentOS

That makes sense -- this would indeed have hidden the bug.

Thanks for fixing this!

Comment 15 Martin Pitt 2022-06-26 04:53:56 UTC
For the record: This still breaks in C9S: https://artifacts.dev.testing-farm.io/83437dab-10c1-41af-bd77-73ae1c4d9265/

Comment 17 RHEL Program Management 2022-07-26 07:27:49 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Comment 18 Martin Pitt 2022-07-26 07:44:49 UTC
Reopening -- this is rather unacceptable. We created a regression in C9S (which is a product!), this has to be fixed. It's bad enough that this has been broken for 8 months already.

Comment 20 Yatin Karel 2022-07-26 10:10:53 UTC
Facing similar issue on CentOS 9-Stream baremetal host with or without subscription-manager installed(it detects as atomic-host):-

# rpm -q tuned
tuned-2.18.0-2.el9.noarch

# tuned-adm recommend
atomic-host

# tuned-adm auto_profile
Requested profile 'atomic-host' doesn't exist.

Seems like [1] fixing it as per my local tests, can this be included in tuned rpm?

[1] https://github.com/redhat-performance/tuned/pull/428/commits/1e7dd713346ef1e0b6f6bc602e4a23541681f34a

Comment 21 Jaroslav Škarvada 2022-07-26 10:30:51 UTC
(In reply to Martin Pitt from comment #18)
> Reopening -- this is rather unacceptable. We created a regression in C9S
> (which is a product!), this has to be fixed. It's bad enough that this has
> been broken for 8 months already.

Sorry, it was closed by automaton.

The fix has been merged upstream some time ago:
https://github.com/redhat-performance/tuned/pull/428

It should be part of the downstream rebase (bug 2057609).

Comment 32 errata-xmlrpc 2022-11-15 11:17:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (tuned bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8321