Bug 2021525

Summary: [16.1] openstack overcloud upgrade run times out / HAProxy container fails to start
Product: Red Hat OpenStack Reporter: gdamien <gdamien>
Component: openstack-tripleo-heat-templatesAssignee: Lukas Bezdicka <lbezdick>
Status: CLOSED ERRATA QA Contact: Jason Grosso <jgrosso>
Severity: medium Docs Contact:
Priority: medium    
Version: 16.1 (Train)CC: amcleod, dciabrin, jpretori, kthakre, lbezdick, lmiccini, m.andre, mburns, spower
Target Milestone: z8Keywords: Triaged
Target Release: 16.1 (Train on RHEL 8.2)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-11.3.2-1.20220114223344.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2021575 (view as bug list) Environment:
Last Closed: 2022-03-24 11:02:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2021575    

Description gdamien 2021-11-09 14:29:25 UTC 
Description of problem:
While proceeding with FFU steps, this command is timing out on the bootstrap controller

Haproxy container was not able to start
 /var/lib/haproxy/stats is having wrong SELinux contexts 

Version-Release number of selected component (if applicable):
From Image: 	RHEL-7.7 + OSP-13.0.10 and contrail-image: - 1912.L1-46 
To image: 	RHEL-8.2 + OSP16.1.6 and contrail-image: - 2011.L2.372 


How reproducible:
openstack overcloud upgrade run --yes   --stack overcloud --limit $host --playbook all


Steps to Reproduce:
1.hosts=vmwebrhoctrl1,vmwebjuctrl1,vmwebjudatabase1,vmwebjuengine1
2.openstack overcloud upgrade run --yes   --stack overcloud --limit $host --playbook all
3.

Actual results:
# ls -lZ /var/lib/haproxy/stats 
srw-------. 1 root root system_u:object_r:haproxy_var_lib_t:s0 0  8 nov.  10:05 /var/lib/haproxy/stats

Expected results:
It should be system_u:object_r:container_file_t:s0


Additional info:
Comment 1 Cédric Jeanneret 2021-11-09 15:41:12 UTC 
Some thoughts, ideas, solutions and others:

- a patch is available as train-only here: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/817079
- it's a "follow-up" of the other one I already provided earlier, related to the OVN service: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/808964

As previously said, we can't just backport [1] down to train/osp-16.x because it would create a complete service outage on day-2 operations such as scale-out - this would be a bad experience for the operator, and we can't allow that. 

Now, why do we see this situation:
while FFU-ing, we get the mighty cleanup of the OSP related packages and services (with containers), the LEAPP to upgrade the OS, the addition of /.autorelabel, and the reboot; then, we get the installation of the new OSP packages, the "upgrades" steps as described in tripleo-heat-templates, and a deploy, in order to start the services and get things back up.
The issue happens with the /.autorelabel, and the following restorecon happening system-wide: since openstack-selinux is removed at this point, and since new system SELinux packages are installed, the restorecon will override anything we set with ":z" flag. More over, since we cannot override system/core fcontext from within openstack-selinux, we may end with non-applicable rules (such as /var/log/containers for osp-16.2 conflicting with a newer version of container-selinux).

So we may end with wrongly labelled directories and, more importantly, directory content.
Now, by default, we don't do anything related to selinux labels/context in the upgrade - so let's move to the deploy itself. Here, nested in tripleo-heat-templates and tripleo-ansible, we do manage some specific "setype" associated to some directory creations. But we're not doing this recursively - it's usually useless, and time consuming (ansible.builtin.file is slow in recursion, really). So we just ensure the directories are present, with the wanted context. And we usually count on the ":z" option set to most of the container volumes in order to get the proper context for anything.... But that flag is missing from pacemaker managed bundles (at least until wallaby/osp-17).

This may lead to weird situation, where a container isn't started from the deploy actions, but only by pacemaker - so it will end with the directory having the correct context (thanks to the deploy "host_prep_tasks" steps), but not its content - such as this "stat" file in /var/lib/haproxy. And crash.


[1] https://review.opendev.org/c/openstack/puppet-tripleo/+/809132
Comment 12 Lukas Bezdicka 2022-02-10 20:54:14 UTC 
Failing QA on customer environment :(
Comment 13 Lukas Bezdicka 2022-02-10 20:56:17 UTC 
Turned out the patch is missing tasks for pacemaker controlled services which is what majority of installations is. I'm working on patch resolving this and will test it directly on customer environment.

For next time verification tasks:
Ensure correct label on mysql data directory and content
Ensure correct label on rabbitmq data directory and content
Ensure correct label on haproxy data directory and content

have to be present in upgrade run logs.
Comment 20 errata-xmlrpc 2022-03-24 11:02:17 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.8 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0986