Description of problem: We have a ansible task in the playbook /usr/share/ansible/roles/tripleo-podman/tasks/tripleo_podman_install.yml which is supposed to install podman packages on the overcloud nodes: +++ - name: Install block become: true block: - name: ensure podman and deps are installed package: name: "{{ tripleo_podman_packages }}" state: latest +++ notice that the state is set to the value 'latest' and the package module uses the built in package manager which in this case is dnf (RHEL 8) which inturn updates the below podman packages to the latest version: +++ podman podman-docker +++ pushing these packages to versions: +++ 2020-11-02T09:12:29Z SUBDEBUG Upgrade: podman-1.9.3-2.module+el8.2.1+6867+366c07d6.x86_64 2020-11-02T09:12:31Z SUBDEBUG Upgrade: podman-docker-1.9.3-2.module+el8.2.1+6867+366c07d6.noarch +++ when you then try to run minor updates on the ceph nodes with: +++ openstack overcloud update run --limit CephStorage --playbook all +++ the minor update process fails with: +++ FAILED! => {"changed": false, "failures": [], "msg": "Depsolve Error occured: \n Problem 1: package python3-paunch-5.3.3-1.20200826193407.ed2c015.el8ost.noarch requires podman = 1.6.4, but none of the prov iders can be installed\n - cannot install both podman-1.6.4-10.module+el8.2.0+6064+4ca4827e.x86_64 and podman-1.9.3-2.module+el8.2.1+6867+366c07d6.x86_64\n - cannot install both podman-1.6.4-12.module+el8.2.0+6670+014d0ff8.x86_6 4 and podman-1.9.3-2.module+el8.2.1+6867+366c07d6.x86_64\n - cannot install both podman-1.6.4-11.module+el8.2.0+6369+1f4293b4.x86_64 and podman-1.9.3-2.module+el8.2.1+6867+366c07d6.x86_64\n - cannot install both podman-1.6.4-15. module+el8.2.0+7290+954fb593.x86_64 and podman-1.9.3-2.module+el8.2.1+6867+366c07d6.x86_64\n - cannot install the best update candidate for package python3-paunch-5.3.3-0.20200527083422.16ae5e4.el8ost.noarch\n - problem with ins talled package podman-1.9.3-2.module+el8.2.1+6867+366c07d6.x86_64\n - package podman-1.6.4-11.module+el8.2.0+6368+cf16aa14.x86_64 is filtered out by modular filtering\n - package podman-1.6.4-10.module+el8.2.0+6063+e761893a.x86_ 64 is filtered out by modular filtering\n - package podman-1.6.4-4.module+el8.1.1+5885+44006e55.x86_64 is filtered out by modular filtering\n - package podman-1.6.4-12.module+el8.2.0+6669+dde598ec.x86_64 is filtered out by modul ar filtering\n - package podman-1.6.4-2.module+el8.1.1+5363+bf8ff1af.x86_64 is filtered out by modular filtering\n Problem 2: package rhosp-release-16.1.2-2.el8ost.noarch requires podman <= 1.9, but none of the providers can be i nstalled\n - cannot install both podman-1.6.4-10.module+el8.2.0+6064+4ca4827e.x86_64 and podman-1.9.3-2.module+el8.2.1+6867+366c07d6.x86_64\n - cannot install both podman-1.6.4-12.module+el8.2.0+6670+014d0ff8.x86_64 and podman-1 .9.3-2.module+el8.2.1+6867+366c07d6.x86_64\n - cannot install both podman-1.6.4-11.module+el8.2.0+6369+1f4293b4.x86_64 and podman-1.9.3-2.module+el8.2.1+6867+366c07d6.x86_64\n - cannot install both podman-1.6.4-15.module+el8.2.0 +7290+954fb593.x86_64 and podman-1.9.3-2.module+el8.2.1+6867+366c07d6.x86_64\n - cannot install the best update candidate for package rhosp-release-16.1.1-1.el8ost.noarch\n - cannot install the best update candidate for package podman-1.9.3-2.module+el8.2.1+6867+366c07d6.x86_64\n - package podman-1.0.3-1.git9d78c0c.module+el8.0.0.z+3717+fdd07b7c.x86_64 is filtered out by modular filtering\n - package podman-1.0.0-4.git921f98f.module+el8.2.0+6370+6fb6c8 ca.x86_64 is filtered out by modular filtering\n - package podman-1.4.2-5.module+el8.1.0+4240+893c1ab8.x86_64 is filtered out by modular filtering\n - package podman-1.0.5-1.gitf604175.module+el8.0.0+4017+bbba319f.x86_64 is filt ered out by modular filtering\n - package podman-1.0.0-2.git921f98f.module+el8.0.0+2956+30df4692.x86_64 is filtered out by modular filtering\n - package podman-1.6.4-11.module+el8.2.0+6368+cf16aa14.x86_64 is filtered out by modu lar filtering\n - package podman-1.0.0-3.git921f98f.module+el8.1.0+4241+a7060183.x86_64 is filtered out by modular filtering\n - package podman-1.6.4-10.module+el8.2.0+6063+e761893a.x86_64 is filtered out by modular filtering\n - package podman-1.6.4-4.module+el8.1.1+5885+44006e55.x86_64 is filtered out by modular filtering\n - package podman-1.0.0-2.git921f98f.module+el8+2784+9a0c1dfe.x86_64 is filtered out by modular filtering\n - package podman-1.4 .2-6.module+el8.1.0+4830+f49150d7.x86_64 is filtered out by modular filtering\n - package podman-1.0.0-2.git921f98f.module+el8.0.0+4014+8662b6b2.x86_64 is filtered out by modular filtering\n - package podman-1.0.0-4.git921f98f.m odule+el8.1.0+4908+72a45cef.x86_64 is filtered out by modular filtering\n - package podman-1.0.0-2.git921f98f.module+el8+2785+ff8a053f.x86_64 is filtered out by modular filtering\n - package podman-1.6.4-12.module+el8.2.0+6669+d de598ec.x86_64 is filtered out by modular filtering\n - package podman-1.6.4-2.module+el8.1.1+5363+bf8ff1af.x86_64 is filtered out by modular filtering\n - package podman-1.0.0-2.git921f98f.module+el8.0.0+2958+4e823551.x86_64 is filtered out by modular filtering", "rc": 1, "results": []} +++ the reason why this happens is because while minor update we also try to update python3-paunch which tries to pull podman = 1.6.4 as a depencendy but since we already have podman installed on a higher version during the stack update process; this creates a package version conflict and causes the minor update to fail This happens on the ceph nodes because we are not supposed to use eus repos on ceph nodes and we can not lock them on a specific rhel version The eus repos contain the required package versions for podman: +++ REPO NAME: rhel-8-for-x86_64-appstream-eus-rpms Provide: podman-1.6.4* +++ non eus repos contain higher version the workaround here was to manually downgrade the podman package and then proceed with minor update Version-Release number of selected component (if applicable): tripleo-ansible-0.5.1-1.20200914163925.el8ost.noarch How reproducible: Always Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Sounds like a subscription issue. Please check https://access.redhat.com/solutions/5528711. Closing it. Please reopen if it does not resolve your issue.
As already mentioned in the initial description of the issue is it not possible to use EUS subscriptions for Ceph node. The Ceph subscription does not include RHEL EUS. Control nodes (full image) -> RHEL 8.2 EUS (release set to 8.2) Compute nodes (full image) -> RHEL 8.2 EUS (release set to 8.2) Ceph nodes (minimal image) -> RHEL 8.2 non-EUS (release not specified) The solution link is not suitable for Ceph nodes.
This is actually unrelated to EUS and non-EUS. Ceph nodes can be non-EUS. The issue here is the version of the container-tools module. The default is "rhel8" which will give podman 1.9.3 (or newer). OpenStack is only tested with container-tools:2.0 which includes podman-1.6.4. dnf module disable container-tools:rhel8 dnf module enable container-tools:2.0
(In reply to Mike Burns from comment #4) > This is actually unrelated to EUS and non-EUS. Ceph nodes can be non-EUS. > > The issue here is the version of the container-tools module. The default is > "rhel8" which will give podman 1.9.3 (or newer). OpenStack is only tested > with container-tools:2.0 which includes podman-1.6.4. > > dnf module disable container-tools:rhel8 > dnf module enable container-tools:2.0 This is in the update documentation: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/keeping_red_hat_openstack_platform_updated/index#setting-the-container-tools-module-version Could you check with the customer if they applied it properly?
Hello, Looking at one of the ceph nodes; I can see that the container tools module with the expected version is enabled: +++ dnf module list --enabled Updating Subscription Management repositories. /usr/lib/python3.6/site-packages/dateutil/parser/_parser.py:70: UnicodeWarning: decode() called on unicode string, see https://bugzilla.redhat.com/show_bug.cgi?id=1693751 instream = instream.decode() Red Hat OpenStack Platform 16.1 Director Deployment Tools for RHEL 8 x86_64 (RPMs) 40 kB/s | 2.3 kB 00:00 Red Hat Ceph Storage OSD 4 for RHEL 8 x86_64 (RPMs) 42 kB/s | 2.3 kB 00:00 Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs) 50 kB/s | 2.8 kB 00:00 Red Hat Satellite Tools 6.5 for RHEL 8 x86_64 (RPMs) 36 kB/s | 2.1 kB 00:00 Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs) 42 kB/s | 2.4 kB 00:00 Red Hat Ansible Engine 2.9 for RHEL 8 x86_64 (RPMs) 42 kB/s | 2.3 kB 00:00 Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs) Name Stream Profiles Summary container-tools 2.0 [e] common [d] Common tools and dependencies for container runtimes idm client [d][e] common [d] RHEL IdM long term support client module python36 3.6 [d][e] build, common [d] [i] Python programming language, version 3.6 ruby 2.5 [d][e] common [d] An interpreter of object-oriented scripting language satellite-5-client 1.0 [d][e] common [d], gui Red Hat Satellite 5 client packages virt rhel [d][e] common [d] Virtualization module +++ Additionally the customer mentioned that they performed this change as per the update documentation i.e before running the update step on the overcloud nodes
Yeah - this looks like in 16.1, we had hardcoded container-tools to 2.0 on the ceph storage nodes. For 16.2, we can also set it to container-tools:2.0.
So, here's what happens: - podman 1.6.4 is installed in the image - when deployed, the container-tools stream is not selected So, whatever is performing upgrades must set up container-tools:2.0 module stream after deployment and/or prior to upgrades, since paunch has a hard requirement on 1.6.4. If it did not have a hard requirement, we could simply fix the 16.1 & 16.2 images to use latest container-tools:rhel8 podman packages,
(In reply to Lon Hohberger from comment #15) > So, here's what happens: > > - podman 1.6.4 is installed in the image > - when deployed, the container-tools stream is not selected > > So, whatever is performing upgrades must set up container-tools:2.0 module > stream after deployment and/or prior to upgrades, since paunch has a hard > requirement on 1.6.4. If it did not have a hard requirement, we could simply > fix the 16.1 & 16.2 images to use latest container-tools:rhel8 podman > packages, thanks for helping Lon, the docs suggest to use container-tools:2.0 indeed [1] can we fix that and try again? 1. https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/keeping_red_hat_openstack_platform_updated/preparing-for-a-minor-update#setting-the-container-tools-module-version_keeping-updated