Description of problem: Some packages/source packages for OSP 16.2 GA Errata 66969 can't be installed/downloaded against Prod Candlepin/CDN. The list of unavailable packages: (1)repo: repo openstack-16.2-for-rhel-8-x86_64-debug-rpms Cython-debugsource-0.29.2-10.el8ost.1.x86_64 numpy-debugsource-1.17.0-7.el8ost.2.x86_64 numpy-debugsource-1.17.0-7.el8ost.2.x86_64 python-psutil-debugsource-5.6.3-3.el8ost.x86_64 (2)repo: openstack-16.2-for-rhel-8-x86_64-source-rpms Cython-0.29.2-10.el8ost.1.src numpy-1.17.0-7.el8ost.2.src pytest-3.9.1-7.el8ost.1.src python-funcsigs-1.0.2-8.el8ost.1.src python-markupsafe-1.1.0-7.el8ost.1.src python-mock-3.0.5-12.el8ost.1.src python-pluggy-0.8.1-7.el8ost.1.src python-psutil-5.6.3-3.el8ost.src python-six-1.12.0-2.el8ost.src (3)repo: openstack-16.2-deployment-tools-for-rhel-8-x86_64-source-rpms numpy-1.17.0-7.el8ost.2.src python-funcsigs-1.0.2-8.el8ost.1.src python-markupsafe-1.1.0-7.el8ost.1.src python-psutil-5.6.3-3.el8ost.src python-psutil-5.6.3-3.el8ost.src python-six-1.12.0-2.el8ost.src (4)repo: openstack-16.2-deployment-tools-for-rhel-8-x86_64-debug-rpms numpy-debugsource-1.17.0-7.el8ost.2.x86_64 python-markupsafe-debugsource-1.1.0-7.el8ost.1.x86_64 python-psutil-debugsource-5.6.3-3.el8ost.x86_64 (5)repo: openstack-16-tools-for-rhel-8-x86_64-debug-rpms numpy-debugsource-1.17.0-7.el8ost.2.x86_64 python-markupsafe-debugsource-1.1.0-7.el8ost.1.x86_64 (6)repo: openstack-16-tools-for-rhel-8-x86_64-source-rpms numpy-1.17.0-7.el8ost.2.src python-funcsigs-1.0.2-8.el8ost.1.src python-markupsafe-1.1.0-7.el8ost.1.src python-six-1.12.0-2.el8ost.src (7)repo: openstack-16.2-cinderlib-for-rhel-8-x86_64-source-rpms numpy-1.17.0-7.el8ost.2.src python-funcsigs-1.0.2-8.el8ost.1.src python-markupsafe-1.1.0-7.el8ost.1.src python-mock-3.0.5-12.el8ost.1.src python-psutil-5.6.3-3.el8ost.src python-six-1.12.0-2.el8ost.src (8)repo: openstack-16.2-cinderlib-for-rhel-8-x86_64-debug-rpms numpy-debugsource-1.17.0-7.el8ost.2.x86_64 python-markupsafe-debugsource-1.1.0-7.el8ost.1.x86_64 python-psutil-debugsource-5.6.3-3.el8ost.x86_64 (9)repo: openstack-16.2-for-rhel-8-ppc64le-debug-rpms Cython-debugsource-0.29.2-10.el8ost.1.ppc64le numpy-debugsource-1.17.0-7.el8ost.2.ppc64le python-markupsafe-debugsource-1.1.0-7.el8ost.1.ppc64le python-psutil-debugsource-5.6.3-3.el8ost.ppc64le (10)repo: openstack-16.2-for-rhel-8-ppc64le-source-rpms Cython-0.29.2-10.el8ost.1.src numpy-1.17.0-7.el8ost.2.src pytest-3.9.1-7.el8ost.1.src python-funcsigs-1.0.2-8.el8ost.1.src python-markupsafe-1.1.0-7.el8ost.1.src python-mock-3.0.5-12.el8ost.1.src python-pluggy-0.8.1-7.el8ost.1.src python-psutil-5.6.3-3.el8ost.src python-six-1.12.0-2.el8ost.src (11)repo: openstack-16.2-deployment-tools-for-rhel-8-ppc64le-source-rpms numpy-1.17.0-7.el8ost.2.src python-funcsigs-1.0.2-8.el8ost.1.src python-markupsafe-1.1.0-7.el8ost.1.src python-psutil-5.6.3-3.el8ost.src python-six-1.12.0-2.el8ost.src (12)repo: openstack-16.2-deployment-tools-for-rhel-8-ppc64le-debug-rpms numpy-debugsource-1.17.0-7.el8ost.2.ppc64le python-markupsafe-debugsource-1.1.0-7.el8ost.1.ppc64le python-psutil-debugsource-5.6.3-3.el8ost.ppc64le (13)repo: openstack-16.2-cinderlib-for-rhel-8-ppc64le-source-rpms numpy-1.17.0-7.el8ost.2.src python-funcsigs-1.0.2-8.el8ost.1.src python-markupsafe-1.1.0-7.el8ost.1.src python-mock-3.0.5-12.el8ost.1.src python-psutil-5.6.3-3.el8ost.src python-six-1.12.0-2.el8ost.src (14)repo: openstack-16.2-cinderlib-for-rhel-8-ppc64le-debug-rpms numpy-debugsource-1.17.0-7.el8ost.2.ppc64le python-markupsafe-debugsource-1.1.0-7.el8ost.1.ppc64le python-psutil-debugsource-5.6.3-3.el8ost.ppc64le Version-Release number of selected component (if applicable): OSP 16.2 GA How reproducible: 100% Testing details: Testing Errata: https://errata.engineering.redhat.com/errata/details/66969 Errata manifest: https://manifest-api.upshift.redhat.com/errata-manifest/66969 Testing job: https://main-jenkins-csb-mpentqe.apps.ocp4.prod.psi.redhat.com/view/MP/job/mp_server_ppc64le/70/ https://main-jenkins-csb-mpentqe.apps.ocp4.prod.psi.redhat.com/view/MP/job/mp_server_x86_64/230/ Steps to Reproduce: 1.Register and entitle [root@cloud-qe-16-vm-09 ~]# cat /etc/redhat-release Red Hat Enterprise Linux release 8.4 (Ootpa) [root@cloud-qe-16-vm-09 ~]# ls /etc/pki/product-default 479.pem [root@cloud-qe-16-vm-09 ~]# cat /etc/rhsm/rhsm.conf | egrep "^hostname|^baseurl" hostname = subscription.rhsm.redhat.com baseurl = https://cdn.redhat.com [root@cloud-qe-16-vm-09 ~]# subscription-manager register --username=multientqe --password='*******' Registering to: subscription.rhsm.redhat.com:443/subscription The system has been registered with ID: 25ccfcbe-d96e-40b5-bf53-57aba6a3d5e1 The registered system name is: cloud-qe-16-vm-09.idmqe.lab.eng.bos.redhat.com [root@cloud-qe-16-vm-09 ~]# subscription-manager attach --pool=8a85f99974b666790174be81b0314baf Successfully attached a subscription for: Red Hat Enterprise Linux Server with Smart Virtualization, Premium (Embedded, Billing) [root@cloud-qe-16-vm-09 ~]# subscription-manager attach --pool=8a85f99b76fc3c2a017747d6bfd03185 Successfully attached a subscription for: Red Hat OpenStack Platform, Standard Support (4 Sockets, NFR, Partner Only) 2.Enable testing repos [root@cloud-qe-16-vm-09 ~]# subscription-manager repos --enable=openstack-16.2-for-rhel-8-x86_64-debug-rpms Repository 'openstack-16.2-for-rhel-8-x86_64-debug-rpms' is enabled for this system. 3.Try to install the package [root@cloud-qe-16-vm-09 ~]# yum -y install Cython-debugsource-0.29.2-10.el8ost.1.x86_64 Updating Subscription Management repositories. Last metadata expiration check: 0:01:50 ago on Thu 16 Sep 2021 02:51:04 AM EDT. All matches were filtered out by modular filtering for argument: Cython-debugsource-0.29.2-10.el8ost.1.x86_64 Error: Unable to find a match: Cython-debugsource-0.29.2-10.el8ost.1.x86_64 [root@cloud-qe-16-vm-09 ~]# yum -y install numpy-debugsource-1.17.0-7.el8ost.2.x86_64 Updating Subscription Management repositories. Last metadata expiration check: 0:02:52 ago on Thu 16 Sep 2021 02:51:04 AM EDT. All matches were filtered out by modular filtering for argument: numpy-debugsource-1.17.0-7.el8ost.2.x86_64 Error: Unable to find a match: numpy-debugsource-1.17.0-7.el8ost.2.x86_64 Actual results: can't find and install the package Expected results: The package should be able to install successfully. Additional info:
Some oddities noted, where are you getting the list of rpms to try to install? > (1)repo: repo openstack-16.2-for-rhel-8-x86_64-debug-rpms > Cython-debugsource-0.29.2-10.el8ost.1.x86_64 > numpy-debugsource-1.17.0-7.el8ost.2.x86_64 > numpy-debugsource-1.17.0-7.el8ost.2.x86_64 > python-psutil-debugsource-5.6.3-3.el8ost.x86_64 I found the following are available via the internal cdn mirrors python3-Cython-debuginfo-0.29.2-10.el8ost.1.x86_64 python3-numpy-debuginfo-1.17.0-7.el8ost.2.x86_64.rpm python3-psutil-debuginfo-5.6.3-3.el8ost.x86_64
Hi, Jon The packages to be tested of installation are listed in the manifest file of Errate 66969 :https://manifest-api.upshift.redhat.com/errata-manifest/66969. I have checked the packages could be found in CDN, but can't be installed via yum command or downloaded via yumdownloader command.
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/developing_c_and_cpp_applications_in_rhel_8/enabling-debugging-with-debugging-information_debugging-applications#getting-debuginfo-packages-for-an-application-or-library-manually_enabling-debugging-with-debugging-information might be helpful. It looks like debugsource needs a special tool to be installed, I wonder if that is part of the issue.
Hi, Jason Thanks for your information. I've read the document and I don't think it's relevant to this issue. The tool "debuginfo-install" is used for users that don't know the debug package name. This tool could find the corresponding debug package for a package. For this issue, the debug package names are known and should be installed directly with "yum install" command.
I'm not sure what next steps are to get this resolved. The issue it appears is either with the repo or issue with dnf as some of these rpms overlap with modules in appstream.
Explanation from @mburns : The root cause for this one is 2-fold, but the basic issue is "CANTFIX". The packages shipped correctly -- you can see them if you manually browse the repositories. The problem is that RHEL also shipped versions of these packages in a module in RHEL 8.4 appstream. Because of this, standard dnf commands can't find the package versions you're trying to install since it *always* uses modules if there is a module available. The secondary issue is that we should not have shipped our own versions of these packages and instead used the RHEL versions. I believe (and Jason can confirm/deny) that functionally, we're using the RHEL versions of the packages in our containers. Going forward, we won't be shipping new versions of these packages and instead using the RHEL versions, but we can't change the fact that we already shipped these packages. We're reviewing our release checklists to ensure consistency in the future so this situation doesn't happen again.
Tried to disable repo "rhel-8-for-x86_64-appstream-rpms", and then install/download the unavailable packages. All the unavailable packages could be installed/downloaded via yum/yumdownloader command with repo "rhel-8-for-x86_64-appstream-rpms" disabled. That confirmed the explanation from Mike, all packages shipped correctly, but influenced by the packages from appstream.
Yum command with option "--setopt=<repo_id>.module_hotfixes=true" could also install the unavailable packages. But this is not a recommended method.