Bug 1744259
| Summary: | During update rabbitmq container cannot restart. | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Sofer Athlan-Guyot <sathlang> |
| Component: | openstack-selinux | Assignee: | Michele Baldessari <michele> |
| Status: | CLOSED ERRATA | QA Contact: | Raviv Bar-Tal <rbartal> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 15.0 (Stein) | CC: | dciabrin, jpichon, lhh, lmiccini, lvrabec, michele, rbartal, sclewis, zcaplovi |
| Target Milestone: | rc | Keywords: | Triaged |
| Target Release: | 15.0 (Stein) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-selinux-0.8.20-0.20190823110429.50e6b42.el8ost | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-09-21 11:24:25 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1727808 | ||
Verified by restarting pacemaker-managed rabbitmq container. (we can't exercise the full minor update with SELinux engage because it's broken in other way currently) Steps: 1. stop rabbitmq on all nodes pcs resource disable rabbitmq-bundle 1. On all controller nodes, force reinstall openstack-selinux to ensure that a SELinux relabelling happens with the SELinux rules from that package. yum reinstall -y openstack-selinux Running transaction check Transaction check succeeded. [...] Upgraded: openstack-selinux-0.8.20-0.20190823110429.50e6b42.el8ost.noarch 2. fix another SELinux error that this package doesn't fix yet. Those specific errors are handled in https://bugzilla.redhat.com/show_bug.cgi?id=1747948 chcon -R -t container_file_t /var/log/containers 3. Verify that pacemaker log files are labelled properly [root@controller-0 rabbitmq-bundle-0]# ls -laZ /var/log/pacemaker/bundles/rabbitmq-bundle-0 total 0 drwxr-x--x. 4 root root system_u:object_r:cluster_var_log_t:s0 47 Sep 4 15:57 . drwxrwx---. 6 hacluster haclient system_u:object_r:cluster_var_log_t:s0 100 Aug 28 15:53 .. -rw-------. 1 root utmp system_u:object_r:cluster_var_log_t:s0 0 Sep 4 15:57 btmp drwxr-xr-x. 3 root root system_u:object_r:cluster_var_log_t:s0 22 Aug 28 15:48 kolla drwxr-xr-x. 2 root root system_u:object_r:cluster_var_log_t:s0 6 Aug 28 15:48 rabbitmq 4. Restart rabbitmq on all nodes pcs resource enable rabbitmq-bundle pcs status | grep rabbitmq GuestOnline: [ galera-bundle-0@controller-0 galera-bundle-1@controller-1 galera-bundle-2@controller-2 ovn-dbs-bundle-0@controller-2 ovn-dbs-bundle-1@controller-1 ovn-dbs-bundle-2@controller-2 rabbitmq-bundle-0@controller-0 rabbitmq-bundle-1@controller-1 rabbitmq-bundle-2@controller-2 redis-bundle-0@controller-1 redis-bundle-1@controller-1 redis-bundle-2@controller-2 ] podman container set: rabbitmq-bundle [192.168.24.1:8787/rhosp15/openstack-rabbitmq:pcmklatest] rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started controller-0 rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Started controller-1 rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Started controller-2 Rabbitmq is restarted as expected on all the nodes because it can access /var/log/pacemaker/bundles/rabbitmq-bundle-0/btmp at startup Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:2811 |
Description of problem: Doing a update of osp15 from beta-1.0 to passed_phase2 (currently 20190819), the rabbitmq container didn't restart. Aug 21 14:29:27 controller-0 pacemaker-controld [99252] (process_lrm_event) notice: Result of start operation for rabbitmq-bundle-podman-0 on controller-0: 1 (unknown error) | call=117 key=rabbitmq-bundle-podman-0_start_0 confirmed=true cib-update=161 Aug 21 14:29:27 controller-0 pacemaker-controld [99252] (process_lrm_event) notice: controller-0-rabbitmq-bundle-podman-0_start_0:117 [ error getting image "rabbitmq-bundle-podman-0": unable to find a name and tag match for rabbitmq-bundle-podman-0 in repotags: no such image\nerror getting image "rabbitmq-bundle-podman-0": unable to find a name and tag match for rabbitmq-bundle-podman-0 in repotags: no such image\nocf-exit-reason:Newly created podman container exited after start\n ] ERROR:__main__:Unexpected error: Traceback (most recent call last): File "/usr/local/bin/kolla_set_configs", line 417, in main execute_config_strategy(config) File "/usr/local/bin/kolla_set_configs", line 383, in execute_config_strategy copy_config(config) File "/usr/local/bin/kolla_set_configs", line 306, in copy_config config_file.copy() File "/usr/local/bin/kolla_set_configs", line 150, in copy self._merge_directories(source, dest) File "/usr/local/bin/kolla_set_configs", line 99, in _merge_directories self._copy_file(source, dest) File "/usr/local/bin/kolla_set_configs", line 82, in _copy_file shutil.copy(source, dest) File "/usr/lib64/python3.6/shutil.py", line 245, in copy copyfile(src, dst, follow_symlinks=follow_symlinks) File "/usr/lib64/python3.6/shutil.py", line 121, in copyfile with open(dst, 'wb') as fdst: PermissionError: [Errno 13] Permission denied: '/var/log/btmp' Setting setenforce to 0, we were able to restart the rabbitmq container. During the update the openstack-selinux package was upgraded from - openstack-selinux-0.8.19-0.20190606150404.06faac7.el8ost.noarch to - openstack-selinux-0.8.19-0.20190813150447.72046d3.el8ost.noarch Started initial debug with Damien and Michele, which led to that https://github.com/redhat-openstack/openstack-selinux/pull/31 . Assigning to pidone as requested.