Bug 1847946

Summary: libdnf behavior has changed unexpectedly in 8.3
Product: Red Hat Enterprise Linux 8 Reporter: Jeff Needle <jneedle>
Component: libdnfAssignee: Pavla Kratochvilova <pkratoch>
Status: CLOSED ERRATA QA Contact: Radek Bíba <rbiba>
Severity: unspecified Docs Contact: Katerina Nemcova <knemcova>
Priority: high    
Version: 8.3CC: dlane, lmanasko, lmiksik, nsella, pkratoch, smaitra, vipatel, yguenane
Target Milestone: rcKeywords: Triaged
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libdnf-0.48.0-3.el8 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-04 01:53:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jeff Needle 2020-06-17 12:13:50 UTC
Description of problem:
Ansible Tower installation is failing on RHEL 8.3 alpha due to changes in RHEL 8.3's version of libdnf.  The change seems unexpected and is likely a bug.

Version-Release number of selected component (if applicable):

python3-libdnf-0.48.0-2.el8.x86_64
libdnf-0.48.0-2.el8.x86_64

How reproducible:
100% reproducible

Steps to Reproduce:

Here's a simple reproducer - run on RHEL8.2 and RHEL8.3.

#!/usr/bin/env python

import dnf
import dnf.sack
import dnf.subject

pkg_spec = 'ansible-tower == 3.6.4'

base = dnf.Base()

base.read_all_repos()
repos = base.repos
enablerepo = ['ansible-tower', 'ansible-tower-dependencies']

for repo_pattern in enablerepo:
    for repo in repos.get_matching(repo_pattern):
        repo.enable()

base.fill_sack(load_system_repo='auto')
subj = dnf.subject.Subject(pkg_spec)
solution = subj.get_best_solution(base.sack, forms=None, with_src=False)

print(solution['query']._name_dict())

Actual results:
On RHEL 8.3: 

[root@rhel-8 cloud-user]# rpm -qa | grep libdnf
python3-libdnf-0.48.0-2.el8.x86_64
libdnf-0.48.0-2.el8.x86_64

[root@rhel-8 cloud-user]# /usr/libexec/platform-python repro.py 
{}

[root@rhel-8 cloud-user]# uname -a
Linux rhel-8.3.0-20200616.0-3.5.6 4.18.0-214.el8.x86_64 #1 SMP Fri Jun 12 08:55:08 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Expected results:
On RHEL 8.2:

[root@rhel-8 cloud-user]# rpm -qa | grep libdnf
libdnf-0.39.1-5.el8.x86_64
python3-libdnf-0.39.1-5.el8.x86_64

[root@rhel-8 cloud-user]# /usr/libexec/platform-python repro.py 
{'ansible-tower': [<hawkey.Package object id 11, ansible-tower-3.6.4-1.el8at.x86_64, @System>, <hawkey.Package object id 7651, ansible-tower-3.6.4-1.el8at.x86_64, ansible-tower>]}

[root@rhel-8 cloud-user]# uname -a
Linux rhel-8.2.0-20200404.0-3.6.4 4.18.0-193.el8.x86_64 #1 SMP Fri Mar 27 14:35:58 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
 

Additional info:
These are the actual failures we are seeing with our interop testing:
http://jenkins.ansible.eng.rdu2.redhat.com/job/Pipelines/job/lptesting-pipeline/283/
http://jenkins.ansible.eng.rdu2.redhat.com/job/Pipelines/job/lptesting-pipeline/281/
http://jenkins.ansible.eng.rdu2.redhat.com/job/Pipelines/job/lptesting-pipeline/282/

Comment 1 Yanis Guenane 2020-06-17 12:23:38 UTC
Yum repo file to use in order to reproduce


[ansible-tower]
name=Ansible Tower Repository - $releasever $basearch
baseurl=https://releases.ansible.com/ansible-tower/rpm/epel-8-$basearch
enabled=1
gpgcheck=1

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
[ansible-tower-dependencies]
name=Ansible Tower Dependencies Repository - $releasever $basearch
baseurl=https://releases.ansible.com/ansible-tower/rpm/dependencies/3.6/epel-8-$basearch
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

Comment 2 Pavla Kratochvilova 2020-06-17 16:24:47 UTC
As a workaround, you can replace 'ansible-tower == 3.6.4' with 'ansible-tower = 3.6.4'.

Btw, weird thing is, I can reproduce the bug (with different packages) on Fedora even with much older libdnf (libdnf-0.28.1-1.fc30.x86_64).

Comment 3 Jeff Needle 2020-06-17 17:01:48 UTC
I can confirm that changing == to = does work around this.  I have nothing on why this is reproducible on older libdnf, but it's definitely a regression between 8.2 and 8.3.

Comment 4 Pavla Kratochvilova 2020-06-18 10:02:30 UTC
I just found out that the '==' operator is not valid when defining dependencies in spec file (see the rpm documentation at https://rpm.org/user_doc/dependencies.html or in /usr/share/doc/rpm/dependencies) and the fact that libdnf accepted it was actually a temporary mistake.

Is it possible to replace the '==' with '=' in all the cases at your end?

Comment 5 Yanis Guenane 2020-06-18 10:28:47 UTC
> Is it possible to replace the '==' with '=' in all the cases at your end?

For future Ansible Tower releases, yes. The problem is for the releases that are currently out there it will be impossible to install them on 8.3 onward until a maintenance release is released.
From a user perspective, it would simply be broken/regression.

As much as I understand this was a temporary mistake, Would it be possible to re-enable it for 8.y time frame to no provide non-backward compatible change ? Then, once we start testing RHEL 9.y we would be find removing it.

Comment 6 Pavla Kratochvilova 2020-06-22 12:08:57 UTC
(In reply to Yanis Guenane from comment #5)
> > Is it possible to replace the '==' with '=' in all the cases at your end?
> 
> For future Ansible Tower releases, yes. The problem is for the releases that
> are currently out there it will be impossible to install them on 8.3 onward
> until a maintenance release is released.
> From a user perspective, it would simply be broken/regression.

Would it be possible to fix this in the Ansible Tower maintenance release (by changing '==' to '=')?
If it's not possible, could you provide more details on Ansible Tower schedule - when a new release is planned?

> As much as I understand this was a temporary mistake, Would it be possible
> to re-enable it for 8.y time frame to no provide non-backward compatible
> change ? Then, once we start testing RHEL 9.y we would be find removing it.

The problem is, bringing back the '==' support in dnf would mean inconsistency with other tools, such as rpm, and what is more important, the behavior is undefined and we cannot guarantee it's correct. For that reason, we are a bit reluctant to bring it back to RHEL 8.
If there's no other way, we can bring it back, but we'll emit a warning about using an unsupported behavior that will be dropped in RHEL 9 and Fedora.

Comment 7 Yanis Guenane 2020-06-22 13:03:54 UTC
This behaviour introduces a regression in our installation procedure in all currently supported versions (3.4.z, 3.5.z, 3.6.z and 3.7.z).
Because of this regression we would need to re-release all currently supported versions of Tower to work around this regression.

While we can update our installation procedure for this not to happen in the future, we would still want the old behaviour to still be available for RHEL8 for the time being.

> If there's no other way, we can bring it back, but we'll emit a warning about using an unsupported behavior that will be dropped in RHEL 9 and Fedora.

This is def. fine with us.

Comment 8 Pavla Kratochvilova 2020-06-24 12:32:35 UTC
I made a patch that brings back the '==' support with deprecation warning: https://github.com/rpm-software-management/libdnf/pull/985

Comment 21 errata-xmlrpc 2020-11-04 01:53:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (yum bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:4510