Bug 2186015

Summary: Every invocation of the microdnf command overwrite /etc/yum.repos.d/redhat.repo
Product: Red Hat Enterprise Linux 8 Reporter: Jihoon Kim <jaykim>
Component: microdnfAssignee: Packaging Maintenance Team <packaging-team-maint>
Status: CLOSED WONTFIX QA Contact: swm-qe
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.7CC: amatej, dcantrell, dornelas, jcastran, jrohel
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-07-31 09:17:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jihoon Kim 2023-04-11 20:36:42 UTC
Description of problem:
- When using ubi8-minimal and ubi9-minimal, every invocation of the microdnf command completely overwrites the /etc/yum.repos.d/redhat.repo file.

- due to the above, it is not possible to permanently enable/disabld repositories in the ubi minimal container.

Version-Release number of selected component (if applicable):


How reproducible:

1. launching container:
[root@localhost ~]# podman run -it --rm --name 9minimal registry.access.redhat.com/ubi9-minimal /bin/bash
[root@6f7d79b725d5 /]#

2. /etc/yum.repos.d/redhat.repo is generated after invoking microdnf:

[root@6f7d79b725d5 /]# ls -l /etc/yum.repos.d/
total 4
rw-r--r--. 1 root root 2736 Feb 22 09:26 ubi.repo
 
[root@6f7d79b725d5 /]# microdnf repolist
repo id                                                                                                          repo name
rhel-9-for-x86_64-appstream-rpms                                                                                 Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)
rhel-9-for-x86_64-baseos-rpms                                                                                    Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)
ubi-9-appstream-rpms                                                                                             Red Hat Universal Base Image 9 (RPMs) - AppStream
ubi-9-baseos-rpms                                                                                                Red Hat Universal Base Image 9 (RPMs) - BaseOS
ubi-9-codeready-builder                                                                                          Red Hat Universal Base Image 9 (RPMs) - CodeReady Builder
 
[root@6f7d79b725d5 /]# ls -l /etc/yum.repos.d/
total 152
rw-r--r--. 1 root root 147900 Apr  9 20:58 redhat.repo
rw-r--r--. 1 root root   2736 Feb 22 09:26 ubi.repo

3. editing redhat.repo:

[root@6f7d79b725d5 /]# grep -A 10 rhel-9-for-x86_64-baseos-rpms /etc/yum.repos.d/redhat.repo
[rhel-9-for-x86_64-baseos-rpms]
name=Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)
baseurl=https://cdn.redhat.com/content/dist/rhel9/$releasever/x86_64/baseos/os
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
gpgcheck=true
metadata_expire=86400
sslclientcert=/etc/pki/entitlement-host/4341998516435641305.pem
sslclientkey=/etc/pki/entitlement-host/4341998516435641305-key.pem
sslcacert=/etc/rhsm-host/ca/redhat-uep.pem
sslverify=true
 
[root@6f7d79b725d5 /]# sed -i '/\['"rhel-9-for-x86_64-baseos-rpms"'\]/,/^ *\[/ s/enabled=1/enabled=0/' /etc/yum.repos.d/redhat.repo
 
[root@6f7d79b725d5 /]# grep -A 10 rhel-9-for-x86_64-baseos-rpms /etc/yum.repos.d/redhat.repo
[rhel-9-for-x86_64-baseos-rpms]
name=Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)
baseurl=https://cdn.redhat.com/content/dist/rhel9/$releasever/x86_64/baseos/os
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
gpgcheck=true
metadata_expire=86400
sslclientcert=/etc/pki/entitlement-host/4341998516435641305.pem
sslclientkey=/etc/pki/entitlement-host/4341998516435641305-key.pem
sslcacert=/etc/rhsm-host/ca/redhat-uep.pem
sslverify=true

4. microdnf overwrites redhat.repo:

[root@6f7d79b725d5 /]# microdnf repolist
repo id                                                                                                          repo name
rhel-9-for-x86_64-appstream-rpms                                                                                 Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)
rhel-9-for-x86_64-baseos-rpms                                                                                    Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)
ubi-9-appstream-rpms                                                                                             Red Hat Universal Base Image 9 (RPMs) - AppStream
ubi-9-baseos-rpms                                                                                                Red Hat Universal Base Image 9 (RPMs) - BaseOS
ubi-9-codeready-builder                                                                                          Red Hat Universal Base Image 9 (RPMs) - CodeReady Builder
 
[root@6f7d79b725d5 /]# grep -A 10 rhel-9-for-x86_64-baseos-rpms /etc/yum.repos.d/redhat.repo
[rhel-9-for-x86_64-baseos-rpms]
name=Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)
baseurl=https://cdn.redhat.com/content/dist/rhel9/$releasever/x86_64/baseos/os
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
gpgcheck=true
metadata_expire=86400
sslclientcert=/etc/pki/entitlement-host/4341998516435641305.pem
sslclientkey=/etc/pki/entitlement-host/4341998516435641305-key.pem
sslcacert=/etc/rhsm-host/ca/redhat-uep.pem
sslverify=true

5. editing rhsm.conf (manage_repos and full_refresh_on_yum to 0) doesn't resolve issue:

[root@6f7d79b725d5 /]# grep -iE "manage_repos|full_refresh" /etc/rhsm-host/rhsm.conf
manage_repos = 0
full_refresh_on_yum = 0
 
[root@6f7d79b725d5 /]# sed -i '/\['"rhel-9-for-x86_64-baseos-rpms"'\]/,/^ *\[/ s/enabled=1/enabled=0/' /etc/yum.repos.d/redhat.repo
 
[root@6f7d79b725d5 /]# grep -A 10 rhel-9-for-x86_64-baseos-rpms /etc/yum.repos.d/redhat.repo
[rhel-9-for-x86_64-baseos-rpms]
name=Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)
baseurl=https://cdn.redhat.com/content/dist/rhel9/$releasever/x86_64/baseos/os
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
gpgcheck=true
metadata_expire=86400
sslclientcert=/etc/pki/entitlement-host/4341998516435641305.pem
sslclientkey=/etc/pki/entitlement-host/4341998516435641305-key.pem
sslcacert=/etc/rhsm-host/ca/redhat-uep.pem
sslverify=true
 
[root@6f7d79b725d5 /]# microdnf repolist
repo id                                                                                                          repo name
rhel-9-for-x86_64-appstream-rpms                                                                                 Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)
rhel-9-for-x86_64-baseos-rpms                                                                                    Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)
ubi-9-appstream-rpms                                                                                             Red Hat Universal Base Image 9 (RPMs) - AppStream
ubi-9-baseos-rpms                                                                                                Red Hat Universal Base Image 9 (RPMs) - BaseOS
ubi-9-codeready-builder                                                                                          Red Hat Universal Base Image 9 (RPMs) - CodeReady Builder
 
[root@6f7d79b725d5 /]# grep -A 10 rhel-9-for-x86_64-baseos-rpms /etc/yum.repos.d/redhat.repo
 
[rhel-9-for-x86_64-baseos-rpms]
name=Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)
baseurl=https://cdn.redhat.com/content/dist/rhel9/$releasever/x86_64/baseos/os
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
gpgcheck=true
metadata_expire=86400
sslclientcert=/etc/pki/entitlement-host/4341998516435641305.pem
sslclientkey=/etc/pki/entitlement-host/4341998516435641305-key.pem
sslcacert=/etc/rhsm-host/ca/redhat-uep.pem
sslverify=true


Actual results:
- /etc/yum.repos.d/redhat.repo gets overwritten every time microdnf is invoked.

Expected results:
- microdnf command does not overwrite /etc/yum.repos.d/redhat.repo

Comment 1 Derrick Ornelas 2023-04-12 15:41:50 UTC
I was under the impression that overwriting redhat.repo was expected behavior for both dnf and microdnf, but I did some testing with yum/dnf that confirmed 'enabled' changes do not get overwritten. Maybe this is to support the --set-enabled/--set-disabled options in yum-config-manager? If we intentionally want to support being able to manually enable/disable RH repos, then this is a true bug in microdnf, yes?

Comment 2 Jaroslav Rohel 2023-04-18 09:05:09 UTC
I'm not an expert on "subscription-manager", but as far as I know, the correct way to enable/disable the repository provided by the content service is to use "subscription-manager".

It probably works somehow with DNF thanks to the "dnf-plugin-subscription-manager" plugin. But even in the DNF documentation there is "Warning: DNF config-manager can misbehave when enabling/disabling repositories generated by tools like subscription-manager on RHEL. In this case you should use subscription-manager to perform such actions." https://dnf-plugins-core.readthedocs.io/en/latest/config_manager.html

Microdnf doesn't know anything about "subscription-manager", but support is in the context part of the "libdnf" library. More precisely, the "libdnf" library uses functions from the "librhsm" library for this.

RHEL version of "libdnf" library, tests if "subscription manager" is installed - if "/var/lib/rhsm" exists. If not, the code generating "redhat.repo" is automatically activated. If the new content differs from that on disk, the file on disk is overwritten. It is a minimal fallback for systems without "subscription manager" installed.

Support for this fallback is "hardcoded" in libdnf. But some time ago I implemented an option to turn off this support just as if it were a plugin. Can be disabled just like the "subscritpion-manager" plugin in DNF.
"microdnf --disableplugin subscription-manager COMMAND".
Then the fallback code will stop being used and the "redhat.repo" file will not be generated. With all the consequences.

Comment 3 Derrick Ornelas 2023-04-18 19:00:02 UTC
(In reply to Jaroslav Rohel from comment #2)
> I'm not an expert on "subscription-manager", but as far as I know, the
> correct way to enable/disable the repository provided by the content service
> is to use "subscription-manager".
> 

Subscription-manager is explicitly disabled (refuses to run) in a container

# podman run --rm -ti ubi8 /bin/bash
[root@539407bc9444 /]# subscription-manager list --enabled
subscription-manager is disabled when running inside a container. Please refer to your host system for subscription management.

Generally, we've been instructing users to use yum/dnf --enablerepo & --disablerepo options. In the early days they also used yum-config-manager. Since this was the normal workflow in containers, microdnf also had to eventually add --enablerepo/--disablerepo options. 
And, because subscription-manager is not expected to run in a container, it is not included in Red Hat's Minimal images where microdnf is used. The Minimal image provides no python, not even platform-python, so installing it is not an option (unless you want to blow the image size back up). 

# podman run --rm -ti ubi8-minimal rpm -q subscription-manager
package subscription-manager is not installed


> It probably works somehow with DNF thanks to the
> "dnf-plugin-subscription-manager" plugin. But even in the DNF documentation
> there is "Warning: DNF config-manager can misbehave when enabling/disabling
> repositories generated by tools like subscription-manager on RHEL. In this
> case you should use subscription-manager to perform such actions."
> https://dnf-plugins-core.readthedocs.io/en/latest/config_manager.html
> 
> Microdnf doesn't know anything about "subscription-manager", but support is
> in the context part of the "libdnf" library. More precisely, the "libdnf"
> library uses functions from the "librhsm" library for this.
> 
> RHEL version of "libdnf" library, tests if "subscription manager" is
> installed - if "/var/lib/rhsm" exists. If not, the code generating
> "redhat.repo" is automatically activated. If the new content differs from
> that on disk, the file on disk is overwritten. It is a minimal fallback for
> systems without "subscription manager" installed.
> 
> Support for this fallback is "hardcoded" in libdnf. But some time ago I
> implemented an option to turn off this support just as if it were a plugin.
> Can be disabled just like the "subscritpion-manager" plugin in DNF.
> "microdnf --disableplugin subscription-manager COMMAND".
> Then the fallback code will stop being used and the "redhat.repo" file will
> not be generated. With all the consequences.

So it seems the root cause is that /var/lib/rhsm does not exist (because subscription-manager is purposefully not included), so microdnf/libdnf/librhsm always overrides redhat.repo. 

I wonder if it would be worth shipping an empty /var/lib/rhsm as part of the base image.

Comment 4 jcastran 2023-04-19 17:28:16 UTC
If we ship an empty /var/lib/rhsm, then redhat.repo will not be created as it doesn't exist until the first run.

I would suggest to make a kcs to suggest this workaround which would require
 - start the ubi
 - microdnf repolist
 - mkdir /var/lib/rhsm
 - modify redhat.repo as required

Once /var/lib/rhsm has been created in a ubi-minimal image, the redhat.repo file is no longer maintained. If there is a need to regenerate or rewrite the redhat.repo file, /var/lib/rhsm must be removed, and microdnf re-run.

> Note
This is only for ubi-minimal. 

This is the commit which the workaround of using /var/lib/rhsm/ was introduced

    https://github.com/rpm-software-management/libdnf/commit/e7cde66428ca0589eaf9fdf32256ba3037874ddc

This is the full page. 

    https://github.com/rpm-software-management/libdnf/blob/d559dbb52d73e825453a3b2b516ea880508b4717/libdnf/dnf-context.cpp#L2105

Comment 5 Derrick Ornelas 2023-04-20 18:54:39 UTC
Ah, so the two of you are saying that when using DNF the dnf-plugin-subscription-manager plugin controls redhat.repo, but for microdnf it falls back directly to librhsm, correct? 

(In reply to jcastran from comment #4)
> 
> I would suggest to make a kcs to suggest this workaround which would require
>  - start the ubi
>  - microdnf repolist
>  - mkdir /var/lib/rhsm
>  - modify redhat.repo as required
> 
> Once /var/lib/rhsm has been created in a ubi-minimal image, the redhat.repo
> file is no longer maintained. If there is a need to regenerate or rewrite
> the redhat.repo file, /var/lib/rhsm must be removed, and microdnf re-run.
> 

Wouldn't it make more sense to stick with utilizing the --enablerepo/--disable repo functionality rather than creating a hacky workflow? Ideally, we would have the same behavior/workflows between dnf and microdnf. I do understand that what you're suggesting would allow _this_ user to get their desired behavior immediately.

Comment 6 Jaroslav Mracek 2023-06-08 08:39:03 UTC
(In reply to Derrick Ornelas from comment #5)
> Ah, so the two of you are saying that when using DNF the
> dnf-plugin-subscription-manager plugin controls redhat.repo, but for
> microdnf it falls back directly to librhsm, correct? 
> 
> (In reply to jcastran from comment #4)
> > 
> > I would suggest to make a kcs to suggest this workaround which would require
> >  - start the ubi
> >  - microdnf repolist
> >  - mkdir /var/lib/rhsm
> >  - modify redhat.repo as required
> > 
> > Once /var/lib/rhsm has been created in a ubi-minimal image, the redhat.repo
> > file is no longer maintained. If there is a need to regenerate or rewrite
> > the redhat.repo file, /var/lib/rhsm must be removed, and microdnf re-run.
> > 
> 
> Wouldn't it make more sense to stick with utilizing the
> --enablerepo/--disable repo functionality rather than creating a hacky
> workflow? Ideally, we would have the same behavior/workflows between dnf and
> microdnf. I do understand that what you're suggesting would allow _this_
> user to get their desired behavior immediately.

Yes I agree, it make sense to integrate it and use --enablerepo/--disable repo functionality or other integrated approach. But we cannot modify behavior during lifecycle of RHEL. So many things depend on the current behavior therefore we cannot change it.

There is already a discussion in upstream - https://github.com/rpm-software-management/dnf5/discussions/422, https://github.com/rpm-software-management/dnf5/discussions/423.

I am suggesting to close the bug because there is a workaround and improve the integration in DNF5 (RHEL10).

Comment 7 Jaroslav Mracek 2023-07-31 09:17:48 UTC
Because there is no activity, In comment 4, there is a workaround, I am closing the issue.