Description of problem: During the FFWD2 (OSP13 to OSP16) the Undercloud and Overcloud upgrade steps fail when bringing up the OSP16.1 containers, as all puppet related containers fail with: 2020-04-29T10:42:11.995417386-04:00 stderr F Cannot find config file: /etc/puppet/hiera.yaml [root@undercloud-0 stdouts]# podman ps --all [76/1982] CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES daef301f70eb undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-conductor:16.1_20200428.1 /usr/bin/bootstra... 6 hours ago Exited (1) 6 hours ag o nova_db_sync 5374da1bd3fc undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-api:16.1_20200428.1 /usr/bin/bootstra... 6 hours ago Exited (1) 6 hours ag o nova_api_ensure_default_cell a13c05e326f0 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-placement-api:16.1_20200428.1 /usr/bin/bootstra... 6 hours ago Exited (1) 6 hours ag o placement_api_db_sync 49d24235f731 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-api:16.1_20200428.1 /usr/bin/bootstra... 6 hours ago Exited (1) 6 hours ag o nova_api_map_cell0 8b378374f33c undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-placement-api:16.1_20200428.1 /usr/bin/bootstra... 6 hours ago Exited (1) 6 hours ag o placement_api_db_extract_data_from_nova_api 8a740f842bca undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-api:16.1_20200428.1 /usr/bin/bootstra... 6 hours ago Exited (1) 6 hours ag o nova_api_db_sync 240a00abd8b1 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-ironic-inspector:16.1_20200428.1 true 6 hours ago Exited (0) 6 hours ag o ironic_inspector_get_ipa a731b33c15d1 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-ironic-inspector:16.1_20200428.1 /usr/bin/bootstra... 6 hours ago Exited (1) 6 hours ag o ironic_inspector_db_sync fc916fa48d04 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp15-openstack-nova-api:20200115.1 /usr/bin/bootstra... 6 hours ago Exited (1) 6 hours ag o nova_api_online_data_migrations_stein 5847bc213029 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-keystone:16.1_20200428.1 kolla_start 6 hours ago Up 6 hours ago keystone 7c620a465131 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-iscsid:16.1_20200428.1 kolla_start 6 hours ago Up 6 hours ago iscsid 1e774c12b251 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp15-openstack-ironic-api:20200115.1 /usr/bin/bootstra... 6 hours ago Exited (1) 6 hours ag o ironic_online_data_migrations_stein c96cec68abd1 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp15-openstack-ironic-inspector:20200115.1 /usr/bin/bootstra... 6 hours ago Exited (1) 6 hours ag o ironic_inspector_db_sync_stein 3e3d5948dcbc undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp15-openstack-ironic-api:20200115.1 /usr/bin/bootstra... 6 hours ago Exited (1) 6 hours ag o ironic_db_sync_stein 3d13a9ea28a0 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-ironic-api:16.1_20200428.1 /usr/bin/bootstra... 6 hours ago Exited (1) 6 hours ag o ironic_db_sync 4ac3a3602ecd undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-swift-account:16.1_20200428.1 chown -R swift: /... 6 hours ago Exited (0) 6 hours ag o swift_setup_srv 660ac674bfe1 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-swift-object:16.1_20200428.1 /bin/bash -c sed ... 6 hours ago Exited (0) 6 hours ag o swift_rsync_fix bb5b2754bafb undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp15-openstack-nova-conductor:20200115.1 /bin/bash -c if /... 6 hours ago Exited (0) 6 hours ag o nova_db_sync_stein The issue seems to be related to containers-selinux package not being the right version for the openstack-selinux package. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Hi, thank you for the report. Could you clarify why you think this is related to the container-selinux package? Are there SELinux denials? Could you include the audit.log file, ideally in permissive mode? If updating the container-selinux package resolved the issue, could you confirm from what version to what version? And same for the openstack-selinux rpms. Thank you!
Hi Julie, OSP13 RH7.8 -> we remove openstack-selinux -> we setup OSP16 el8.2 repos => we run LEAPP but leapp it self ignores modules so container module is not enabled and container-selinux package is installed from clean EL8.2 => OSP16.1 EL8 we run UpgradeInit command which enables modules but does not update packages -> we install openstack-selinux which silently fails -> we update packages where we It's kind of related to the way we test the OSP13 to OSP16.1 upgrade (Lukas Bezdicka could add more details if needed, as he is the one who debugged the issue deeply) but if I understand correctly: 1. We have the Undercloud in OSP13 with RHEL7.8 2. We remove openstack-selinux before performing the RHEL upgrade from RHEL7 to RHEL8. 3. We set up OSP16.1 el8.2 repos. 4. Run Leapp which self ignores modules, so it won't enable the container-tools module for RHEL8.2 and container-selinux package is installed from clean EL8.2. 5. We set up again OSP16.1 EL8.2 repos after Leapp upgrade, but no package update is run. 6. Installed openstack-selinux, which silently fails. 7. We update packages during upgrade tasks, point in which we retrieve the right containers-selinux package from containers-tools module. Lukas can add more details on his findings to identify it was a selinux issue, but basically running in the Undercloud: 28 sudo rhos-release -x 29 sudo rhos-release 16.1-trunk -r 8.2 30 sudo dnf reinstall openstack-selinux 31 sudo dnf reinstall containers-selinux And relaunchiong the FFWD upgrade fixed the issue and now all containers are up and running. So it certainly is related to openstack-selinux and the container-selinux package.
The proper order of things is sudo rhos-release 16.1-trunk -r 8.2; sudo dnf update containers-selinux ; sudo dnf install openstack-selinux. The reason is that nova module fails to get inserted because containers-selinux that is in clean 8.2 does not know it. The package that comes from enabled module provides what openstack-selinux needs.
Thank you both for all the addition information! I will try to reproduce although if you could include the dnf logs that show the versions for everything and also the nova module error, that would be really helpful.
Also was this working upgrading to 16.0? I'm not sure if we were testing with that version too.
Yes, in OSP16 we didn't see such an issue when we were testing from OSP13 to OSP16.1, it's happening now that we moved to OSP16.1. Here you can find some additional logs from a CI job in which I manage to reproduce the issue yesterday: http://cougar11.scl.lab.tlv.redhat.com/DFG-upgrades-ffu-ffu-upgrade-13-16.1_director-rhel-virthost-3cont_2comp_3ceph-ipv6-vxlan-HA/1/undercloud-0.tar.gz?undercloud-0/var/log/audit/audit.log http://cougar11.scl.lab.tlv.redhat.com/DFG-upgrades-ffu-ffu-upgrade-13-16.1_director-rhel-virthost-3cont_2comp_3ceph-ipv6-vxlan-HA/1/undercloud-0.tar.gz?undercloud-0/var/log/dnf.log http://cougar11.scl.lab.tlv.redhat.com/DFG-upgrades-ffu-ffu-upgrade-13-16.1_director-rhel-virthost-3cont_2comp_3ceph-ipv6-vxlan-HA/1/undercloud-0.tar.gz?undercloud-0/var/log/messages Job Full Logs: http://cougar11.scl.lab.tlv.redhat.com/DFG-upgrades-ffu-ffu-upgrade-13-16.1_director-rhel-virthost-3cont_2comp_3ceph-ipv6-vxlan-HA/1/
Thank you for all the additional logs. I'm struggling to connect them to the error mentioned in the description at the moment. Would you be able to point me to a log that shows the "Can't find config file /etc/puppet/hiera.yaml" issue from the description? The only references to hiera.yaml I can find (in the messages logs) seem to show that it can be read/found fine. There doesn't seem to be AVC denials related to the file in the audit log either. I'm still trying to create a minimum viable reproducer locally, though not successfully yet. You said "4. Run Leapp which self ignores modules, so it won't enable the container-tools module for RHEL8.2 and container-selinux package is installed from clean EL8.2." but if I disable the container-tools module, there is no container-selinux package found. Where does it come from for the "clean EL8.2" environment? Can you share what version that is? Lukas, would you have the full error about the "nova module failing to insert"? I'm guessing it couldn't find some of the symbols. Though the os-nova policy hasn't changed in a while, and the symbols it uses from container-selinux have been in there since versions v1.121 from what I can tell. More information on the error, and the package versions that failed and worked would be super helpful. Thank you!
Forgot needinfo flags for the questions about logs / versions above
Note also that the upgrade must enable container-tools:2.0 - and disable container-tools:rhel8 i.e.: (enable 8.2, AV, & 16.1 repos) dnf -y module disable virt:rhel dnf -y module enable virt:8.2 dnf -y module disable container-tools:rhel8 dnf -y module enable container-tools:2.0 (rest of upgrades) The container-selinux (and podman, and all the rest of the container bits) from that module are the versions we need for 16.1; the default module stream isn't the one we want.
I'm not sure how to tie this in to the rest of the upgrade process; do we need to somehow inject callouts within leapp? I believe we should never be using virt:rhel content nor container-tools:rhel8 content with 16.1. Unfortunately, the container-tools:rhel8 content is default and is "fast-moving", so versions of RPMs within it may supersede what is in container-tools:2.0 - so it's important that these are disabled before installing any podman/container-selinux/etc. content, or "upgrading" to the container-tools:2.0 builds will eventually not work. Within openstack-selinux, we could require exact versions of RPMs - but I think DNF will simply fail if container-tools:2.0 is not enabled. I am happy to test this.
It sounds like the problem is with step 4 - leapp is not enabling specific modules.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3148
*** Bug 1871241 has been marked as a duplicate of this bug. ***
Created KCS to solve the issue once encountered: https://access.redhat.com/solutions/5568401