Bug 2154764
| Summary: | common component periodic jobs for rhos17.1 are failing consistently with 29 missing modules | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | pojadhav |
| Component: | openstack-selinux | Assignee: | Julie Pichon <jpichon> |
| Status: | CLOSED WORKSFORME | QA Contact: | nlevinki <nlevinki> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 17.1 (Wallaby) | CC: | amoralej, apevec, jjoyce, jpichon, jschluet, lhh, lvrabec, mburns, mgarciac, rlandy, shrjoshi |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | --- | Flags: | jpichon:
needinfo-
|
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-12-21 14:51:11 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
It looks like the openstack-selinux package could not be installed properly. I recommend following the steps at [1] to determine if this is a problem with openstack-selinux or container-selinux, and include the full trace from the rpm reinstall after that. [1] https://github.com/redhat-openstack/openstack-selinux/blob/master/doc/TROUBLESHOOTING.md#how-to-resolve-missing-os-ovs--found-xx-missing-modules-errors Thank you for the reply. The container-selinux package seems fine, and the issue is coming from openstack-selinux. os-barbican: libsepol.policydb_read: policydb module version 21 does not match my version range 4-20 Interestingly, we saw that recently in https://bugzilla.redhat.com/show_bug.cgi?id=2143562#c8 for container-selinux and the issue was with the mixed environment. The deps repository was leaking content too recent (comment 11). Could this be the issue with osptrunk-candidate-deps as well? "Candidate" in the name is making me wonder. What are the versions for the SELinux libraries installed ($ rpm -qa | grep selinux | sort)? Thanks. By the way, as a workaround for now, can you downgrade openstack-selinux and see if that helps? I think it should install without errors then. I poked around the system: - I noticed the delorean-component-common repo is disabled when that's where openstack-selinux would come from. However, that version also doesn't work when trying to install it (same error) - I temporarily enabled osp-trunk-deps and downgraded openstack-selinux from there. The rpm openstack-selinux-0.8.34-0.20220711150342.a82a63a.el9ost installed correctly, however I'm concerned this may bring other issues down the line as it's missing a couple of patches (0.8.34 vs 0.8.37) Could we compare with the latest openstack-selinux package version from a working job? Additionally, I'm looking at brew for openstack-selinux-0.8.37-1.20221215081212.34c2ecc.el9osttrunk.noarch and I think we may be hitting a mixed content issue again. If I understand correctly, it was built against selinux-policy-34.1.43-1.el9 and libselinux-3.4-3.el9.x86_64 (vs running with selinux-policy-34.1.29-1.el9_0.2, libselinux-3.3-2.el9.x86_64 in comment 5). So: 1. Can we confirm the previously working version of openstack-selinux in the jobs and perhaps force the jobs to use that for now as a workaround, if it works? 2. Lon, I'm looking at the "minimum selinux-policy version" at [1]. Do you think that's something we could make use of here (i.e. requiring selinux-policy-34.1.43 maybe) so the openstack-selinux package cannot be installed with an older selinux policy on the system? I'm not sure if I'm understanding it correctly, or if it might be irrelevant here because libselinux is what actually matters at build time. Thank you! [1] https://github.com/rdo-packages/openstack-selinux-distgit/blob/rpm-master/openstack-selinux.spec#L19 *** Bug 2155466 has been marked as a duplicate of this bug. *** |
Description of problem: common component periodic jobs for rhos17.1 are failing consistently with 29 missing modules with below traceback : 2022-12-19 02:30:22.274793 | fa163ee8-8947-adbb-c367-000000000276 | FATAL | Ensure packages are actually well installed | standalone | error={"changed": true, "cmd": ["rpm", "-V", "driverctl", "lvm2", "jq", "openvswitch", "openstack-heat-agents", "openstack-selinux", "os-net-config", "python3-libselinux", "python3-pyyaml", "puppet-tripleo", "rsync", "tmpwatch", "sysstat"], "delta": "0:00:00.966137", "end": "2022-12-19 02:30:22.240777", "failed_when_result": true, "msg": "non-zero return code", "rc": 2, "start": "2022-12-19 02:30:21.274640", "stderr": "error: %verify(openstack-selinux-0.8.37-1.20221215081212.34c2ecc.el9osttrunk.noarch) scriptlet failed, exit status 1", "stderr_lines": ["error: %verify(openstack-selinux-0.8.37-1.20221215081212.34c2ecc.el9osttrunk.noarch) scriptlet failed, exit status 1"], "stdout": "package openvswitch is not installed\nMissing os-ovs!\nMissing os-swift!\nMissing os-nova!\nMissing os-neutron!\nMissing os-mysql!\nMissing os-glance!\nMissing os-rsync!\nMissing os-rabbitmq!\nMissing os-keepalived!\nMissing os-keystone!\nMissing os-haproxy!\nMissing os-mongodb!\nMissing os-ipxe!\nMissing os-redis!\nMissing os-cinder!\nMissing os-httpd!\nMissing os-gnocchi!\nMissing os-collectd!\nMissing os-virt!\nMissing os-dnsmasq!\nMissing os-octavia!\nMissing os-podman!\nMissing os-rsyslog!\nMissing os-pbis!\nMissing os-barbican!\nMissing os-logrotate!\nMissing os-certmonger!\nMissing os-timemaster!\nMissing os-ovs-el9!\nFound 29 missing module(s).", "stdout_lines": ["package openvswitch is not installed", "Missing os-ovs!", "Missing os-swift!", "Missing os-nova!", "Missing os-neutron!", "Missing os-mysql!", "Missing os-glance!", "Missing os-rsync!", "Missing os-rabbitmq!", "Missing os-keepalived!", "Missing os-keystone!", "Missing os-haproxy!", "Missing os-mongodb!", "Missing os-ipxe!", "Missing os-redis!", "Missing os-cinder!", "Missing os-httpd!", "Missing os-gnocchi!", "Missing os-collectd!", "Missing os-virt!", "Missing os-dnsmasq!", "Missing os-octavia!", "Missing os-podman!", "Missing os-rsyslog!", "Missing os-pbis!", "Missing os-barbican!", "Missing os-logrotate!", "Missing os-certmonger!", "Missing os-timemaster!", "Missing os-ovs-el9!", "Found 29 missing module(s)."]} 2022-12-19 02:30:22.276019 | fa163ee8-8947-adbb-c367-000000000276 | TIMING | tripleo_bootstrap : Ensure packages are actually well installed | standalone | 0:00:14.356477 | 1.19s Version-Release number of selected component (if applicable): rhos17.1 on rhel9 List of job failing with above failure : 1. periodic-tripleo-ci-rhel-9-containers-multinode-common-rhos-17.1 2. periodic-tripleo-ci-rhel-9-scenario001-standalone-common-rhos-17.1 3. periodic-tripleo-ci-rhel-9-scenario002-standalone-common-rhos-17.1 4. periodic-tripleo-ci-rhel-9-standalone-common-rhos-17.1