Description of problem: While proceeding with FFU steps, this command is timing out on the bootstrap controller Haproxy container was not able to start /var/lib/haproxy/stats is having wrong SELinux contexts Version-Release number of selected component (if applicable): From Image: RHEL-7.7 + OSP-13.0.10 and contrail-image: - 1912.L1-46 To image: RHEL-8.2 + OSP16.1.6 and contrail-image: - 2011.L2.372 How reproducible: openstack overcloud upgrade run --yes --stack overcloud --limit $host --playbook all Steps to Reproduce: 1.hosts=vmwebrhoctrl1,vmwebjuctrl1,vmwebjudatabase1,vmwebjuengine1 2.openstack overcloud upgrade run --yes --stack overcloud --limit $host --playbook all 3. Actual results: # ls -lZ /var/lib/haproxy/stats srw-------. 1 root root system_u:object_r:haproxy_var_lib_t:s0 0 8 nov. 10:05 /var/lib/haproxy/stats Expected results: It should be system_u:object_r:container_file_t:s0 Additional info:
Some thoughts, ideas, solutions and others: - a patch is available as train-only here: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/817079 - it's a "follow-up" of the other one I already provided earlier, related to the OVN service: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/808964 As previously said, we can't just backport [1] down to train/osp-16.x because it would create a complete service outage on day-2 operations such as scale-out - this would be a bad experience for the operator, and we can't allow that. Now, why do we see this situation: while FFU-ing, we get the mighty cleanup of the OSP related packages and services (with containers), the LEAPP to upgrade the OS, the addition of /.autorelabel, and the reboot; then, we get the installation of the new OSP packages, the "upgrades" steps as described in tripleo-heat-templates, and a deploy, in order to start the services and get things back up. The issue happens with the /.autorelabel, and the following restorecon happening system-wide: since openstack-selinux is removed at this point, and since new system SELinux packages are installed, the restorecon will override anything we set with ":z" flag. More over, since we cannot override system/core fcontext from within openstack-selinux, we may end with non-applicable rules (such as /var/log/containers for osp-16.2 conflicting with a newer version of container-selinux). So we may end with wrongly labelled directories and, more importantly, directory content. Now, by default, we don't do anything related to selinux labels/context in the upgrade - so let's move to the deploy itself. Here, nested in tripleo-heat-templates and tripleo-ansible, we do manage some specific "setype" associated to some directory creations. But we're not doing this recursively - it's usually useless, and time consuming (ansible.builtin.file is slow in recursion, really). So we just ensure the directories are present, with the wanted context. And we usually count on the ":z" option set to most of the container volumes in order to get the proper context for anything.... But that flag is missing from pacemaker managed bundles (at least until wallaby/osp-17). This may lead to weird situation, where a container isn't started from the deploy actions, but only by pacemaker - so it will end with the directory having the correct context (thanks to the deploy "host_prep_tasks" steps), but not its content - such as this "stat" file in /var/lib/haproxy. And crash. [1] https://review.opendev.org/c/openstack/puppet-tripleo/+/809132
Failing QA on customer environment :(
Turned out the patch is missing tasks for pacemaker controlled services which is what majority of installations is. I'm working on patch resolving this and will test it directly on customer environment. For next time verification tasks: Ensure correct label on mysql data directory and content Ensure correct label on rabbitmq data directory and content Ensure correct label on haproxy data directory and content have to be present in upgrade run logs.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1.8 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:0986