Bug 1829609

Hi, thank you for the report. Could you clarify why you think this is related to the container-selinux package? Are there SELinux denials? Could you include the audit.log file, ideally in permissive mode?

If updating the container-selinux package resolved the issue, could you confirm from what version to what version? And same for the openstack-selinux rpms. Thank you!

Hi Julie,

OSP13 RH7.8  -> we remove openstack-selinux -> we setup OSP16 el8.2 repos => we run LEAPP but leapp it self ignores modules so container module is not enabled and container-selinux package is installed from clean EL8.2 => OSP16.1 EL8 we run UpgradeInit command which enables modules but does not update packages -> we install openstack-selinux which silently fails -> we update packages where we

It's kind of related to the way we test the OSP13 to OSP16.1 upgrade (Lukas Bezdicka could add more details if needed, as he is the one who debugged the issue deeply) but if I understand correctly:
1. We have the Undercloud in OSP13 with RHEL7.8
2. We remove openstack-selinux before performing the RHEL upgrade from RHEL7 to RHEL8.
3. We set up OSP16.1 el8.2 repos.
4. Run Leapp which self ignores modules, so it won't enable the container-tools module for RHEL8.2 and container-selinux package is installed from clean EL8.2.
5. We set up again OSP16.1 EL8.2 repos after Leapp upgrade, but no package update is run.
6. Installed openstack-selinux, which silently fails.
7. We update packages during upgrade tasks, point in which we retrieve the right containers-selinux package from containers-tools module.


Lukas can add more details on his findings to identify it was a selinux issue, but basically running in the Undercloud:

   28  sudo rhos-release -x
   29  sudo rhos-release 16.1-trunk -r 8.2
   30  sudo dnf reinstall openstack-selinux
   31  sudo dnf reinstall containers-selinux

And relaunchiong the FFWD upgrade fixed the issue and now all containers are up and running. So it certainly is related to openstack-selinux and the container-selinux package.

The proper order of things is sudo rhos-release 16.1-trunk -r 8.2; sudo dnf update containers-selinux ; sudo dnf install openstack-selinux.

The reason is that nova module fails to get inserted because containers-selinux that is in clean 8.2 does not know it. The package that comes from enabled module provides what openstack-selinux needs.

Thank you both for all the addition information! I will try to reproduce although if you could include the dnf logs that show the versions for everything and also the nova module error, that would be really helpful.

Also was this working upgrading to 16.0? I'm not sure if we were testing with that version too.

Yes, in OSP16 we didn't see such an issue when we were testing from OSP13 to OSP16.1, it's happening now that we moved to OSP16.1.

Here you can find some additional logs from a CI job in which I manage to reproduce the issue yesterday:

http://cougar11.scl.lab.tlv.redhat.com/DFG-upgrades-ffu-ffu-upgrade-13-16.1_director-rhel-virthost-3cont_2comp_3ceph-ipv6-vxlan-HA/1/undercloud-0.tar.gz?undercloud-0/var/log/audit/audit.log
http://cougar11.scl.lab.tlv.redhat.com/DFG-upgrades-ffu-ffu-upgrade-13-16.1_director-rhel-virthost-3cont_2comp_3ceph-ipv6-vxlan-HA/1/undercloud-0.tar.gz?undercloud-0/var/log/dnf.log
http://cougar11.scl.lab.tlv.redhat.com/DFG-upgrades-ffu-ffu-upgrade-13-16.1_director-rhel-virthost-3cont_2comp_3ceph-ipv6-vxlan-HA/1/undercloud-0.tar.gz?undercloud-0/var/log/messages
Job Full Logs: http://cougar11.scl.lab.tlv.redhat.com/DFG-upgrades-ffu-ffu-upgrade-13-16.1_director-rhel-virthost-3cont_2comp_3ceph-ipv6-vxlan-HA/1/

Thank you for all the additional logs. I'm struggling to connect them to the error mentioned in the description at the moment. Would you be able to point me to a log that shows the "Can't find config file /etc/puppet/hiera.yaml" issue from the description? The only references to hiera.yaml I can find (in the messages logs) seem to show that it can be read/found fine. There doesn't seem to be AVC denials related to the file in the audit log either.

I'm still trying to create a minimum viable reproducer locally, though not successfully yet. You said "4. Run Leapp which self ignores modules, so it won't enable the container-tools module for RHEL8.2 and container-selinux package is installed from clean EL8.2." but if I disable the container-tools module, there is no container-selinux package found. Where does it come from for the "clean EL8.2" environment? Can you share what version that is?

Lukas, would you have the full error about the "nova module failing to insert"? I'm guessing it couldn't find some of the symbols. Though the os-nova policy hasn't changed in a while, and the symbols it uses from container-selinux have been in there since versions v1.121 from what I can tell.

More information on the error, and the package versions that failed and worked would be super helpful. 

Thank you!

Forgot needinfo flags for the questions about logs / versions above

Note also that the upgrade must enable container-tools:2.0 - and disable container-tools:rhel8

i.e.:

(enable 8.2, AV, & 16.1 repos)
dnf -y module disable virt:rhel
dnf -y module enable virt:8.2
dnf -y module disable container-tools:rhel8
dnf -y module enable container-tools:2.0
(rest of upgrades)

The container-selinux (and podman, and all the rest of the container bits) from that module are the versions we need for 16.1; the default module stream isn't the one we want.

I'm not sure how to tie this in to the rest of the upgrade process; do we need to somehow inject callouts within leapp?

I believe we should never be using virt:rhel content nor container-tools:rhel8 content with 16.1. Unfortunately, the container-tools:rhel8 content is default and is "fast-moving", so versions of RPMs within it may supersede what is in container-tools:2.0 - so it's important that these are disabled before installing any podman/container-selinux/etc. content, or "upgrading" to the container-tools:2.0 builds will eventually not work.

Within openstack-selinux, we could require exact versions of RPMs - but I think DNF will simply fail if container-tools:2.0 is not enabled. I am happy to test this.

It sounds like the problem is with step 4 - leapp is not enabling specific modules.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3148

*** Bug 1871241 has been marked as a duplicate of this bug. ***

Created KCS to solve the issue once encountered: https://access.redhat.com/solutions/5568401