Bug 1931429

+++ This bug was initially created as a clone of Bug #1930037 +++

Description of problem: Until at least 8 days ago [1], with rhel-guest-image-8.4-594.x86_64.qcow2 [2], subscription-manager was enabled by default in dnf:

# grep enabled= /etc/yum/pluginconf.d/subscription-manager.conf
enabled=1

That was subscription-manager-1.28.11-1.el8.x86_64.

With the most recent RHEL 8.4 nightlies this now broke [3]

# grep enabled= /etc/yum/pluginconf.d/subscription-manager.conf
enabled=0

This breaks at least cockpit and apparently also subscription-manager-cockpit [4]


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Download current cloud image and some cloud-init.iso that will allow you to log in; using cockpit CI's here, but that's nothing special
curl -O http://download.eng.brq.redhat.com/rhel-8/nightly/RHEL-8/latest-RHEL-8.4/compose/BaseOS/x86_64/images/rhel-guest-image-8.4-649.x86_64.qcow2
curl -L -O https://github.com/cockpit-project/bots/raw/master/machine/cloud-init.iso
2. Boot the image
qemu-system-x86_64 -enable-kvm  -nographic -m 2048 -drive file="rhel-guest-image-8.4-649.x86_64.qcow2",if=virtio -snapshot -cdrom cloud-init.iso
3. Log in as root:foobar
4. grep enabled= /etc/yum/pluginconf.d/subscription
5. Open cockpit, check Health box
6. Check the status on the CLI:
subscription-manager status`
7. grep enabled= /etc/yum/pluginconf.d/subscription

Actual results:
4. sub-man is disabled in dnf by default
5. No complaint about "This system is not subscribed"
7. `subscription-manager status` *changed* /etc/yum/pluginconf.d/subscription and set enabled=1

Expected results:
4. sub-man should be enabled in dnf by default on RHEL (but not on Fedora/CentOS)
5. Shows "This system is not subscribed"
7. Calling a status command has no business changing config files in /etc. This confuses Ansible and other tools. such state belongs into /var.

Especially 7 is really upsetting, please don't do this.

If you insist on this for whatever reason (I don't claim I have insights in all the details here), please advise how cockpit and sub-man-cockpit are supposed to check if the OS requires subscriptions to be able to install/update packages.

Thank you!

Additional info:

[1] https://github.com/cockpit-project/bots/pull/1645
[2] http://download.eng.blr.redhat.com/rhel-8/nightly/RHEL-8/latest-RHEL-8.4/compose/BaseOS/x86_64/images//rhel-guest-image-8.4-594.x86_64.qcow2
[3] https://github.com/cockpit-project/bots/pull/1684
[4] see https://github.com/candlepin/subscription-manager/pull/2447 and https://github.com/candlepin/subscription-manager/pull/2448 for some related discussion

--- Additional comment from Martin Pitt on 2021-02-18 09:45:51 UTC ---

This is apparently related to some changes in osbuild: https://issues.redhat.com/browse/COMPOSER-717 So please reassign to osbuild if that is the case. Thank you!

--- Additional comment from Martin Pitt on 2021-02-18 09:53:03 UTC ---

> Version-Release number of selected component (if applicable):

Whoops, forgot to paste that: subscription-manager-1.28.12-1.el8.x86_64

--- Additional comment from Martin Pitt on 2021-02-18 14:33:25 UTC ---

Tomas just kindly pointed out that enabled= in /etc/yum/pluginconf.d/product-id.conf has the exact same behaviour. That may also explain the weird errors that the subscription-manager tests get now:

   [Errno 20] Not a directory: '/etc/pki/product'

See test log: https://logs.cockpit-project.org/logs/pull-1684-20210218-111709-5c46a7d0-rhel-8-4-candlepin-subscription-manager/log.html#1-2
and screenshot: https://logs.cockpit-project.org/logs/pull-1684-20210218-111709-5c46a7d0-rhel-8-4-candlepin-subscription-manager/TestSubscriptions-testInsights-rhel-8-4-127.0.0.2-2301-FAIL.png

and indeed, when I set enabled=1 in product-id.conf as well (as I just did in https://github.com/cockpit-project/bots/pull/1684), then subscription-manager tests are happy again.

--- Additional comment from Tomáš Hozza on 2021-02-18 15:15:41 UTC ---

Just a note, that the fact that RHSM DNF plugins are disabled on RHEL Guest images is not a Regression. This has been the situation since RHEL-6.

8.4 images produced before osbuild-composer has been used have these plugins also disabled (e.g. http://download.eng.brq.redhat.com/rhel-8/nightly/RHEL-8/RHEL-8.4.0-20201119.n.0/compose/BaseOS/x86_64/images/rhel-guest-image-8.4-294.x86_64.qcow2).

While 8.4 images produced since ~Dec 2020, until we fixed the discrepancy in ~Feb 2021 had RHSM DNF plugins enabled (e.g. the first still available image built by osbuild-composer where DNF plugins got enabled - http://download.eng.brq.redhat.com/rhel-8/nightly/RHEL-8/RHEL-8.4.0-20201209.n.0/compose/BaseOS/x86_64/images/rhel-guest-image-8.4-338.x86_64.qcow2).

--- Additional comment from Martin Pitt on 2021-02-18 15:38:41 UTC ---

Thanks Tomas! So is this what happened then:

 - The qcow images had enabled=0 for a long time, until in December osbuild took over building them
 - That had an unexpected change that it set enabled=1 from December to ~ 1 week ago
 - A few days ago, osbuild switched it back to off

cockpit CI switching from virt-install to qcow cloud images happened right during these few weeks (we switched in January).

That would explain both your and my point of view, and clear the confusion. Assuming that is what happened, I still have some new questions, though:

 * I take it that we still expect users to install some packages on a cloud image, which will require a subscription. What is the *reason* why sub-man is disabled by default on cloud images? I.e. what does this dynamic enabled=0 → 1 achieve over a static enabled=1 config?

 * What is the official documented thing that a user has to invoke to enable sub-man on these images?

 * Cockpit is one such supported place that tells you that your system is not subscribed, and Cockpit is installed by default. So obviously, the current situation of not saying that is a bug. What *should* cockpit do to test that a system is not subscribed, but needs to be? We have tested for enabled=1 in /etc/yum/pluginconf.d/subscription for years, but obviously that is not the right thing. But what is?

Thanks!

--- Additional comment from Tomáš Hozza on 2021-02-18 16:46:44 UTC ---

(In reply to Martin Pitt from comment #5)
> Thanks Tomas! So is this what happened then:
> 
>  - The qcow images had enabled=0 for a long time, until in December osbuild
> took over building them
>  - That had an unexpected change that it set enabled=1 from December to ~ 1
> week ago
>  - A few days ago, osbuild switched it back to off
> 
> cockpit CI switching from virt-install to qcow cloud images happened right
> during these few weeks (we switched in January).
> 
> That would explain both your and my point of view, and clear the confusion.
> Assuming that is what happened, I still have some new questions, though:

Yes, that is exactly what happened ;)


>  * I take it that we still expect users to install some packages on a cloud
> image, which will require a subscription. What is the *reason* why sub-man
> is disabled by default on cloud images? I.e. what does this dynamic
> enabled=0 → 1 achieve over a static enabled=1 config?

"cloud images" is an overloaded term, so I'll make more distinction.

The images that you used are (KVM) Guest images, which are intended for use only on KVM-based hypervisors. I was not able to determine the reason why RHSM DNF plugins have been disabled on KVM Guest images. My assumption would be that in the past, they were intended also for use with cloud providers, which is not the case any more (but I don't know). So I really don't know...

Disabling these plugins on images used with cloud providers (e.g. EC2 images) makes sense, because the content is delivered via other means, specifically via RHUI (Red Hat Update Infrastructure). While for any special content, one has to use sub-man, RHEL content with couple of addons is available via RHUI. If these DNF plugins are not disabled, then DNF produces misleading log messages on each command, that the system is not subscribed, while subscribing it is not needed to access content.

The reasoning is that if you really need to subscribe the system, then doing it via GUI/CLI will auto-enable these plugins and everything works.

>  * What is the official documented thing that a user has to invoke to enable
> sub-man on these images?

Nothing AFAIK. DNF plugins are auto-enabled on the very first subscription-manager command. Actually any command.

>  * Cockpit is one such supported place that tells you that your system is
> not subscribed, and Cockpit is installed by default. So obviously, the
> current situation of not saying that is a bug. What *should* cockpit do to
> test that a system is not subscribed, but needs to be? We have tested for
> enabled=1 in /etc/yum/pluginconf.d/subscription for years, but obviously
> that is not the right thing. But what is?

I'm not sub-man devel, but based on my information, the fact that the RHSM DNF plugins are enabled tells you nothing about the subscription status of the OS. I would say that probably checking the output of a specific sub-man command, but I will deffer the answer to actual members of the subscription-manager team... Also the situation is a bit complicated if the system uses RHUI or the latest RHSM feature - SCA (Simple Content Access)

> 
> Thanks!

--- Additional comment from Martin Pitt on 2021-02-19 04:28:55 UTC ---

(In reply to Tomáš Hozza from comment #6)
> > What *should* cockpit do to test that a system is not subscribed, but needs to be?
> > We have tested for enabled=1 in /etc/yum/pluginconf.d/subscription for years, but obviously that is not the right thing. But what is?
>
> the fact that the RHSM DNF plugins are enabled tells you nothing about the subscription status of the OS.

Right, and that's also not what I meant. Cockpit obviously calls rhsm D-Bus to see if the system *is* subscribed or not -- but that is not sufficient, as sub-man says "wah wah not subscribed" on Fedora or CentOS as well. So we additionally need to check if the system *needs* to be subscribed, and that's a question to dnf, not to rhsm. We thought querying the dnf sub-man plugin enablement status was appropriate for that.

Hence my question to the sub-man developers: What is a robust replacement for that, if it's not the config files?

Thanks!

--- Additional comment from Tomáš Hozza on 2021-02-19 08:58:06 UTC ---

(In reply to Martin Pitt from comment #7)
> Right, and that's also not what I meant. Cockpit obviously calls rhsm D-Bus
> to see if the system *is* subscribed or not -- but that is not sufficient,
> as sub-man says "wah wah not subscribed" on Fedora or CentOS as well. So we
> additionally need to check if the system *needs* to be subscribed, and
> that's a question to dnf, not to rhsm. We thought querying the dnf sub-man
> plugin enablement status was appropriate for that.

Thanks for the wider context, I now understand better the problem that you are trying to solve. What I meant mostly is that the RHSM DNF plugins are by default enabled when the subcsription-manager RPM is installed. So unless the image is built in a specific way (like the RHEL qcow2), the subscription-manager and product-id DNF plugins will be most probably enabled even if the system still needs subscribing.

--- Additional comment from Martin Pitt on 2021-02-22 12:04:34 UTC ---

We just had a meeting to resolve this. The sub-man team's recommendation was to stop looking at the dnf config file, and instead use the RHSM ListInstalledProducts() call.

On current RHEL 8.4 cloud image:

# busctl call com.redhat.RHSM1 /com/redhat/RHSM1/Products com.redhat.RHSM1.Products ListInstalledProducts 'sa{sv}s' '' 0 ''
s "[[\"Red Hat Enterprise Linux for x86_64 Beta\", \"486\", \"8.4 Beta\", \"x86_64\", \"unknown\", [], \"\", \"\"]]"

where as on CentOS 8 stream or Fedora it returns

s "[]"

so that's as expected and good.

This needs to be done in subscription-manager-cockpit, so this bug report can stay on the subscription-manager component. I'll clone it to track making that change to Cockpit's Overview and Software Updates pages.

[1] https://www.candlepinproject.org/docs/subscription-manager/dbus_objects.html