Bug 1744259 - During update rabbitmq container cannot restart.
Summary: During update rabbitmq container cannot restart.
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-selinux
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
Target Milestone: rc
: 15.0 (Stein)
Assignee: Michele Baldessari
QA Contact: Raviv Bar-Tal
Depends On:
Blocks: 1727808
TreeView+ depends on / blocked
Reported: 2019-08-21 16:22 UTC by Sofer Athlan-Guyot
Modified: 2019-09-27 10:44 UTC (History)
9 users (show)

Fixed In Version: openstack-selinux-0.8.20-0.20190823110429.50e6b42.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
Last Closed: 2019-09-21 11:24:25 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github redhat-openstack openstack-selinux pull 37 0 None closed Make sure that /var/log/pacemaker/* is of cluster_var_log_t type 2021-01-26 13:55:37 UTC
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:25:00 UTC

Description Sofer Athlan-Guyot 2019-08-21 16:22:42 UTC
Description of problem: Doing a update of osp15 from beta-1.0 to passed_phase2 (currently 20190819), the rabbitmq container didn't restart.

Aug 21 14:29:27 controller-0 pacemaker-controld  [99252] (process_lrm_event)    notice: Result of start operation for rabbitmq-bundle-podman-0 on controller-0: 1 (unknown error) | call=117 key=rabbitmq-bundle-podman-0_start_0 confirmed=true cib-update=161   
Aug 21 14:29:27 controller-0 pacemaker-controld  [99252] (process_lrm_event)    notice: controller-0-rabbitmq-bundle-podman-0_start_0:117 [ error getting image "rabbitmq-bundle-podman-0": unable to find a name and tag match for rabbitmq-bundle-podman-0 in repotags: no such image\nerror getting image "rabbitmq-bundle-podman-0": unable to find a name and tag match for rabbitmq-bundle-podman-0 in repotags: no such image\nocf-exit-reason:Newly created podman container exited  after start\n ]
ERROR:__main__:Unexpected error:                                               
Traceback (most recent call last):                                             
  File "/usr/local/bin/kolla_set_configs", line 417, in main                   
  File "/usr/local/bin/kolla_set_configs", line 383, in execute_config_strategy
  File "/usr/local/bin/kolla_set_configs", line 306, in copy_config            
  File "/usr/local/bin/kolla_set_configs", line 150, in copy                   
    self._merge_directories(source, dest)                                      
  File "/usr/local/bin/kolla_set_configs", line 99, in _merge_directories      
    self._copy_file(source, dest)                                              
  File "/usr/local/bin/kolla_set_configs", line 82, in _copy_file              
    shutil.copy(source, dest)                                                  
  File "/usr/lib64/python3.6/shutil.py", line 245, in copy                     
    copyfile(src, dst, follow_symlinks=follow_symlinks)                        
  File "/usr/lib64/python3.6/shutil.py", line 121, in copyfile                 
    with open(dst, 'wb') as fdst:                                              
PermissionError: [Errno 13] Permission denied: '/var/log/btmp' 

Setting setenforce to 0, we were able to restart the rabbitmq container.

During the update the openstack-selinux package was upgraded from 

 - openstack-selinux-0.8.19-0.20190606150404.06faac7.el8ost.noarch


 - openstack-selinux-0.8.19-0.20190813150447.72046d3.el8ost.noarch

Started initial debug with Damien and Michele, which led to that https://github.com/redhat-openstack/openstack-selinux/pull/31 . Assigning to pidone as requested.

Comment 10 Damien Ciabrini 2019-09-04 16:18:03 UTC
Verified by restarting pacemaker-managed rabbitmq container.
(we can't exercise the full minor update with SELinux engage because it's broken in other way currently)


1. stop rabbitmq on all nodes

pcs resource disable rabbitmq-bundle

1. On all controller nodes, force reinstall openstack-selinux to ensure that a SELinux relabelling happens with the SELinux rules from that package.

yum reinstall -y openstack-selinux
Running transaction check
Transaction check succeeded.


2. fix another SELinux error that this package doesn't fix yet. Those specific errors are handled in https://bugzilla.redhat.com/show_bug.cgi?id=1747948

chcon -R -t container_file_t /var/log/containers

3. Verify that pacemaker log files are labelled properly

[root@controller-0 rabbitmq-bundle-0]# ls -laZ /var/log/pacemaker/bundles/rabbitmq-bundle-0
total 0
drwxr-x--x. 4 root      root     system_u:object_r:cluster_var_log_t:s0  47 Sep  4 15:57 .
drwxrwx---. 6 hacluster haclient system_u:object_r:cluster_var_log_t:s0 100 Aug 28 15:53 ..
-rw-------. 1 root      utmp     system_u:object_r:cluster_var_log_t:s0   0 Sep  4 15:57 btmp
drwxr-xr-x. 3 root      root     system_u:object_r:cluster_var_log_t:s0  22 Aug 28 15:48 kolla
drwxr-xr-x. 2 root      root     system_u:object_r:cluster_var_log_t:s0   6 Aug 28 15:48 rabbitmq

4. Restart rabbitmq on all nodes

pcs resource enable rabbitmq-bundle

pcs status | grep rabbitmq
GuestOnline: [ galera-bundle-0@controller-0 galera-bundle-1@controller-1 galera-bundle-2@controller-2 ovn-dbs-bundle-0@controller-2 ovn-dbs-bundle-1@controller-1 ovn-dbs-bundle-2@controller-2 rabbitmq-bundle-0@controller-0 rabbitmq-bundle-1@controller-1 rabbitmq-bundle-2@controller-2 redis-bundle-0@controller-1 redis-bundle-1@controller-1 redis-bundle-2@controller-2 ]
 podman container set: rabbitmq-bundle []
   rabbitmq-bundle-0    (ocf::heartbeat:rabbitmq-cluster):      Started controller-0
   rabbitmq-bundle-1    (ocf::heartbeat:rabbitmq-cluster):      Started controller-1
   rabbitmq-bundle-2    (ocf::heartbeat:rabbitmq-cluster):      Started controller-2

Rabbitmq is restarted as expected on all the nodes because it can access  /var/log/pacemaker/bundles/rabbitmq-bundle-0/btmp at startup

Comment 14 errata-xmlrpc 2019-09-21 11:24:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.