Bug 2154764 - common component periodic jobs for rhos17.1 are failing consistently with 29 missing modules
Summary: common component periodic jobs for rhos17.1 are failing consistently with 29 ...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-selinux
Version: 17.1 (Wallaby)
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: ---
: ---
Assignee: Julie Pichon
QA Contact: nlevinki
URL:
Whiteboard:
: 2155466 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-12-19 07:48 UTC by pojadhav
Modified: 2023-08-04 10:42 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-12-21 14:51:11 UTC
Target Upstream Version:
Embargoed:
jpichon: needinfo-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker OSP-20941 0 None None None 2022-12-19 07:51:28 UTC

Description pojadhav 2022-12-19 07:48:36 UTC
Description of problem:

common component periodic jobs for rhos17.1 are failing consistently with 29 missing modules with below traceback :

2022-12-19 02:30:22.274793 | fa163ee8-8947-adbb-c367-000000000276 |      FATAL | Ensure packages are actually well installed | standalone | error={"changed": true, "cmd": ["rpm", "-V", "driverctl", "lvm2", "jq", "openvswitch", "openstack-heat-agents", "openstack-selinux", "os-net-config", "python3-libselinux", "python3-pyyaml", "puppet-tripleo", "rsync", "tmpwatch", "sysstat"], "delta": "0:00:00.966137", "end": "2022-12-19 02:30:22.240777", "failed_when_result": true, "msg": "non-zero return code", "rc": 2, "start": "2022-12-19 02:30:21.274640", "stderr": "error: %verify(openstack-selinux-0.8.37-1.20221215081212.34c2ecc.el9osttrunk.noarch) scriptlet failed, exit status 1", "stderr_lines": ["error: %verify(openstack-selinux-0.8.37-1.20221215081212.34c2ecc.el9osttrunk.noarch) scriptlet failed, exit status 1"], "stdout": "package openvswitch is not installed\nMissing os-ovs!\nMissing os-swift!\nMissing os-nova!\nMissing os-neutron!\nMissing os-mysql!\nMissing os-glance!\nMissing os-rsync!\nMissing os-rabbitmq!\nMissing os-keepalived!\nMissing os-keystone!\nMissing os-haproxy!\nMissing os-mongodb!\nMissing os-ipxe!\nMissing os-redis!\nMissing os-cinder!\nMissing os-httpd!\nMissing os-gnocchi!\nMissing os-collectd!\nMissing os-virt!\nMissing os-dnsmasq!\nMissing os-octavia!\nMissing os-podman!\nMissing os-rsyslog!\nMissing os-pbis!\nMissing os-barbican!\nMissing os-logrotate!\nMissing os-certmonger!\nMissing os-timemaster!\nMissing os-ovs-el9!\nFound 29 missing module(s).", "stdout_lines": ["package openvswitch is not installed", "Missing os-ovs!", "Missing os-swift!", "Missing os-nova!", "Missing os-neutron!", "Missing os-mysql!", "Missing os-glance!", "Missing os-rsync!", "Missing os-rabbitmq!", "Missing os-keepalived!", "Missing os-keystone!", "Missing os-haproxy!", "Missing os-mongodb!", "Missing os-ipxe!", "Missing os-redis!", "Missing os-cinder!", "Missing os-httpd!", "Missing os-gnocchi!", "Missing os-collectd!", "Missing os-virt!", "Missing os-dnsmasq!", "Missing os-octavia!", "Missing os-podman!", "Missing os-rsyslog!", "Missing os-pbis!", "Missing os-barbican!", "Missing os-logrotate!", "Missing os-certmonger!", "Missing os-timemaster!", "Missing os-ovs-el9!", "Found 29 missing module(s)."]}
2022-12-19 02:30:22.276019 | fa163ee8-8947-adbb-c367-000000000276 |     TIMING | tripleo_bootstrap : Ensure packages are actually well installed | standalone | 0:00:14.356477 | 1.19s


Version-Release number of selected component (if applicable):

rhos17.1 on rhel9

List of job failing with above failure :

1. periodic-tripleo-ci-rhel-9-containers-multinode-common-rhos-17.1
2. periodic-tripleo-ci-rhel-9-scenario001-standalone-common-rhos-17.1            3. periodic-tripleo-ci-rhel-9-scenario002-standalone-common-rhos-17.1 
4. periodic-tripleo-ci-rhel-9-standalone-common-rhos-17.1

Comment 2 Julie Pichon 2022-12-19 09:09:18 UTC
It looks like the openstack-selinux package could not be installed properly. I recommend following the steps at [1] to determine if this is a problem with openstack-selinux or container-selinux, and include the full trace from the rpm reinstall after that.

[1] https://github.com/redhat-openstack/openstack-selinux/blob/master/doc/TROUBLESHOOTING.md#how-to-resolve-missing-os-ovs--found-xx-missing-modules-errors

Comment 4 Julie Pichon 2022-12-19 11:52:25 UTC
Thank you for the reply. The container-selinux package seems fine, and the issue is coming from openstack-selinux.

os-barbican: libsepol.policydb_read: policydb module version 21 does not match my version range 4-20

Interestingly, we saw that recently in https://bugzilla.redhat.com/show_bug.cgi?id=2143562#c8 for container-selinux and the issue was with the mixed environment. The deps repository was leaking content too recent (comment 11). Could this be the issue with osptrunk-candidate-deps as well? "Candidate" in the name is making me wonder. What are the versions for the SELinux libraries installed ($ rpm -qa | grep selinux | sort)?

Comment 6 Julie Pichon 2022-12-19 12:07:12 UTC
Thanks. By the way, as a workaround for now, can you downgrade openstack-selinux and see if that helps? I think it should install without errors then.

Comment 8 Julie Pichon 2022-12-19 13:50:55 UTC
I poked around the system:

- I noticed the delorean-component-common repo is disabled when that's where openstack-selinux would come from. However, that version also doesn't work when trying to install it (same error)
- I temporarily enabled osp-trunk-deps and downgraded openstack-selinux from there. The rpm openstack-selinux-0.8.34-0.20220711150342.a82a63a.el9ost installed correctly, however I'm concerned this may bring other issues down the line as it's missing a couple of patches (0.8.34 vs 0.8.37)

Could we compare with the latest openstack-selinux package version from a working job?

Additionally, I'm looking at brew for openstack-selinux-0.8.37-1.20221215081212.34c2ecc.el9osttrunk.noarch and I think we may be hitting a mixed content issue again. If I understand correctly, it was built against selinux-policy-34.1.43-1.el9 and libselinux-3.4-3.el9.x86_64 (vs running with selinux-policy-34.1.29-1.el9_0.2, libselinux-3.3-2.el9.x86_64 in comment 5).

So:
1. Can we confirm the previously working version of openstack-selinux in the jobs and perhaps force the jobs to use that for now as a workaround, if it works?
2. Lon, I'm looking at the "minimum selinux-policy version" at [1]. Do you think that's something we could make use of here (i.e. requiring selinux-policy-34.1.43 maybe) so the openstack-selinux package cannot be installed with an older selinux policy on the system? I'm not sure if I'm understanding it correctly, or if it might be irrelevant here because libselinux is what actually matters at build time.

Thank you!

[1] https://github.com/rdo-packages/openstack-selinux-distgit/blob/rpm-master/openstack-selinux.spec#L19

Comment 17 pojadhav 2022-12-21 11:52:57 UTC
*** Bug 2155466 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.