Bug 1738303
| Summary: | podman: containers may get stopped by systemd instead of pacemaker on shutdown [rhel-8.0.0.z] | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Oneata Mircea Teodor <toneata> |
| Component: | resource-agents | Assignee: | Oyvind Albrigtsen <oalbrigt> |
| Status: | CLOSED ERRATA | QA Contact: | pkomarov |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 8.0 | CC: | agk, aherr, cfeist, cluster-maint, cluster-qe, dciabrin, fdinitto, oalbrigt, pkomarov |
| Target Milestone: | rc | Keywords: | ZStream |
| Target Release: | 8.1 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | resource-agents-4.1.1-17.el8_0.5 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | 1736746 | Environment: | |
| Last Closed: | 2019-09-10 13:13:15 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1736746 | ||
| Bug Blocks: | |||
|
Comment 3
Damien Ciabrini
2019-08-07 14:15:31 UTC
Verified , [stack@undercloud-0 ~]$ ansible overcloud_nodes -b -mshell -a'rpm -qa|grep resource-agents' [WARNING]: Found both group and host with same name: undercloud [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. messaging-0 | CHANGED | rc=0 >> resource-agents-4.1.1-17.el8_0.5.x86_64 compute-0 | CHANGED | rc=0 >> resource-agents-4.1.1-17.el8_0.5.x86_64 controller-2 | CHANGED | rc=0 >> resource-agents-4.1.1-17.el8_0.5.x86_64 controller-0 | CHANGED | rc=0 >> resource-agents-4.1.1-17.el8_0.5.x86_64 controller-1 | CHANGED | rc=0 >> resource-agents-4.1.1-17.el8_0.5.x86_64 messaging-1 | CHANGED | rc=0 >> resource-agents-4.1.1-17.el8_0.5.x86_64 messaging-2 | CHANGED | rc=0 >> resource-agents-4.1.1-17.el8_0.5.x86_64 database-0 | CHANGED | rc=0 >> resource-agents-4.1.1-17.el8_0.5.x86_64 database-1 | CHANGED | rc=0 >> resource-agents-4.1.1-17.el8_0.5.x86_64 database-2 | CHANGED | rc=0 >> resource-agents-4.1.1-17.el8_0.5.x86_64 [stack@undercloud-0 ~]$ ansible database -mshell -b -a'podman ps --filter name=galera-bundle' [WARNING]: Found both group and host with same name: undercloud database-1 | CHANGED | rc=0 >> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b30795498639 192.168.24.1:8787/rhosp15/openstack-mariadb:20190725.1 dumb-init -- /bin... 5 hours ago Up 5 hours ago galera-bundle-podman-0 database-0 | CHANGED | rc=0 >> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e1305bed957b 192.168.24.1:8787/rhosp15/openstack-mariadb:20190725.1 dumb-init -- /bin... 6 hours ago Up 6 hours ago galera-bundle-podman-2 database-2 | CHANGED | rc=0 >> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3f6c7f4471da 192.168.24.1:8787/rhosp15/openstack-mariadb:20190725.1 dumb-init -- /bin... 6 hours ago Up 6 hours ago galera-bundle-podman-1 [stack@undercloud-0 ~]$ ansible database -mshell -b -a'ls /run/systemd/transient/libpod*$(podman ps -q --filter name=galera-bundle)*' [WARNING]: Found both group and host with same name: undercloud database-0 | CHANGED | rc=0 >> /run/systemd/transient/libpod-conmon-e1305bed957bc84a9199e17ba63d030bf9017dac13ae5d5c87d36126ab4eb0b2.scope /run/systemd/transient/libpod-e1305bed957bc84a9199e17ba63d030bf9017dac13ae5d5c87d36126ab4eb0b2.scope database-1 | CHANGED | rc=0 >> /run/systemd/transient/libpod-b3079549863931a8f750b818e12137bf3c00312cf7e6238162118b53ae6dba9f.scope /run/systemd/transient/libpod-conmon-b3079549863931a8f750b818e12137bf3c00312cf7e6238162118b53ae6dba9f.scope database-2 | CHANGED | rc=0 >> /run/systemd/transient/libpod-3f6c7f4471da95fca398855526ceb7cb9441184ebe5e0c3567b88f5f7a56169c.scope /run/systemd/transient/libpod-conmon-3f6c7f4471da95fca398855526ceb7cb9441184ebe5e0c3567b88f5f7a56169c.scope [stack@undercloud-0 ~]$ ansible controller-0 -mshell -b -a'pcs resource disable galera-bundle' [WARNING]: Found both group and host with same name: undercloud controller-0 | CHANGED | rc=0 >> [stack@undercloud-0 ~]$ ansible controller-0 -mshell -b -a'pcs status|grep galera-bundle' [WARNING]: Found both group and host with same name: undercloud controller-0 | CHANGED | rc=0 >> podman container set: galera-bundle [192.168.24.1:8787/rhosp15/openstack-mariadb:pcmklatest] galera-bundle-0 (ocf::heartbeat:galera): Stopped (disabled) galera-bundle-1 (ocf::heartbeat:galera): Stopped (disabled) galera-bundle-2 (ocf::heartbeat:galera): Stopped (disabled) [stack@undercloud-0 ~]$ ansible overcloud_nodes -mshell -b -a'touch /etc/sysconfig/podman_drop_in;ls /etc/sysconfig/podman_drop_in' [WARNING]: Found both group and host with same name: undercloud [WARNING]: Consider using the file module with state=touch rather than running 'touch'. If you need to use command because file is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. messaging-0 | CHANGED | rc=0 >> /etc/sysconfig/podman_drop_in compute-0 | CHANGED | rc=0 >> /etc/sysconfig/podman_drop_in controller-1 | CHANGED | rc=0 >> /etc/sysconfig/podman_drop_in controller-2 | CHANGED | rc=0 >> /etc/sysconfig/podman_drop_in controller-0 | CHANGED | rc=0 >> /etc/sysconfig/podman_drop_in messaging-2 | CHANGED | rc=0 >> /etc/sysconfig/podman_drop_in messaging-1 | CHANGED | rc=0 >> /etc/sysconfig/podman_drop_in database-0 | CHANGED | rc=0 >> /etc/sysconfig/podman_drop_in database-1 | CHANGED | rc=0 >> /etc/sysconfig/podman_drop_in database-2 | CHANGED | rc=0 >> /etc/sysconfig/podman_drop_in [stack@undercloud-0 ~]$ ansible controller-0 -mshell -b -a'pcs resource enable galera-bundle;sleep 5s;pcs status|grep galera-bundle' [WARNING]: Found both group and host with same name: undercloud controller-0 | CHANGED | rc=0 >> GuestOnline: [ galera-bundle-0@overcloud-controller-1 galera-bundle-1@overcloud-controller-1 galera-bundle-2@overcloud-controller-1 ovn-dbs-bundle-0@overcloud-controller-0 ovn-dbs-bundle-1@overcloud-controller-2 ovn-dbs-bundle-2@overcloud-controller-1 rabbitmq-bundle-0@overcloud-controller-1 rabbitmq-bundle-1@overcloud-controller-1 rabbitmq-bundle-2@overcloud-controller-1 redis-bundle-0@overcloud-controller-1 redis-bundle-1@overcloud-controller-0 redis-bundle-2@overcloud-controller-2 ] podman container set: galera-bundle [192.168.24.1:8787/rhosp15/openstack-mariadb:pcmklatest] galera-bundle-0 (ocf::heartbeat:galera): Stopped overcloud-database-0 galera-bundle-1 (ocf::heartbeat:galera): Stopped overcloud-database-1 galera-bundle-2 (ocf::heartbeat:galera): Stopped overcloud-database-2 [stack@undercloud-0 ~]$ ansible controller-0 -mshell -b -a'pcs status|grep galera-bundle' [WARNING]: Found both group and host with same name: undercloud controller-0 | CHANGED | rc=0 >> GuestOnline: [ galera-bundle-0@overcloud-controller-1 galera-bundle-1@overcloud-controller-1 galera-bundle-2@overcloud-controller-1 ovn-dbs-bundle-0@overcloud-controller-0 ovn-dbs-bundle-1@overcloud-controller-2 ovn-dbs-bundle-2@overcloud-controller-1 rabbitmq-bundle-0@overcloud-controller-1 rabbitmq-bundle-1@overcloud-controller-1 rabbitmq-bundle-2@overcloud-controller-1 redis-bundle-0@overcloud-controller-1 redis-bundle-1@overcloud-controller-0 redis-bundle-2@overcloud-controller-2 ] podman container set: galera-bundle [192.168.24.1:8787/rhosp15/openstack-mariadb:pcmklatest] galera-bundle-0 (ocf::heartbeat:galera): Promoting overcloud-database-0 galera-bundle-1 (ocf::heartbeat:galera): Master overcloud-database-1 galera-bundle-2 (ocf::heartbeat:galera): Master overcloud-database-2 [stack@undercloud-0 ~]$ ansible controller-0 -mshell -b -a'pcs status|grep galera-bundle' [WARNING]: Found both group and host with same name: undercloud controller-0 | CHANGED | rc=0 >> GuestOnline: [ galera-bundle-0@overcloud-controller-1 galera-bundle-1@overcloud-controller-1 galera-bundle-2@overcloud-controller-1 ovn-dbs-bundle-0@overcloud-controller-0 ovn-dbs-bundle-1@overcloud-controller-2 ovn-dbs-bundle-2@overcloud-controller-1 rabbitmq-bundle-0@overcloud-controller-1 rabbitmq-bundle-1@overcloud-controller-1 rabbitmq-bundle-2@overcloud-controller-1 redis-bundle-0@overcloud-controller-1 redis-bundle-1@overcloud-controller-0 redis-bundle-2@overcloud-controller-2 ] podman container set: galera-bundle [192.168.24.1:8787/rhosp15/openstack-mariadb:pcmklatest] galera-bundle-0 (ocf::heartbeat:galera): Master overcloud-database-0 galera-bundle-1 (ocf::heartbeat:galera): Master overcloud-database-1 galera-bundle-2 (ocf::heartbeat:galera): Master overcloud-database-2 [stack@undercloud-0 ~]$ ansible database -mshell -b -a'podman ps --filter name=galera-bundle' [WARNING]: Found both group and host with same name: undercloud database-2 | CHANGED | rc=0 >> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 65aab860bd16 192.168.24.1:8787/rhosp15/openstack-mariadb:20190725.1 dumb-init -- /bin... 2 minutes ago Up 2 minutes ago galera-bundle-podman-2 database-0 | CHANGED | rc=0 >> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c7d560ce484a 192.168.24.1:8787/rhosp15/openstack-mariadb:20190725.1 dumb-init -- /bin... 2 minutes ago Up 2 minutes ago galera-bundle-podman-0 database-1 | CHANGED | rc=0 >> CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 324ee8f480f8 192.168.24.1:8787/rhosp15/openstack-mariadb:20190725.1 dumb-init -- /bin... 2 minutes ago Up 2 minutes ago galera-bundle-podman-1 [stack@undercloud-0 ~]$ ansible database -mshell -b -a'ls /run/systemd/transient/libpod*$(podman ps -q --filter name=galera-bundle)*' [WARNING]: Found both group and host with same name: undercloud database-0 | CHANGED | rc=0 >> /run/systemd/transient/libpod-c7d560ce484a019d286b23c8c0d608d10e9c9cd92d042e5a8467ca11f82d3b05.scope /run/systemd/transient/libpod-conmon-c7d560ce484a019d286b23c8c0d608d10e9c9cd92d042e5a8467ca11f82d3b05.scope /run/systemd/transient/libpod-c7d560ce484a019d286b23c8c0d608d10e9c9cd92d042e5a8467ca11f82d3b05.scope.d: dep.conf /run/systemd/transient/libpod-conmon-c7d560ce484a019d286b23c8c0d608d10e9c9cd92d042e5a8467ca11f82d3b05.scope.d: dep.conf database-1 | CHANGED | rc=0 >> /run/systemd/transient/libpod-324ee8f480f81b69761c899115be0866bc325a3f8305d9d01f8e611217073c1e.scope /run/systemd/transient/libpod-conmon-324ee8f480f81b69761c899115be0866bc325a3f8305d9d01f8e611217073c1e.scope /run/systemd/transient/libpod-324ee8f480f81b69761c899115be0866bc325a3f8305d9d01f8e611217073c1e.scope.d: dep.conf /run/systemd/transient/libpod-conmon-324ee8f480f81b69761c899115be0866bc325a3f8305d9d01f8e611217073c1e.scope.d: dep.conf database-2 | CHANGED | rc=0 >> /run/systemd/transient/libpod-65aab860bd16c16c1a949a62a5da5f62ca26a164d89bd5d343523068546ff49f.scope /run/systemd/transient/libpod-conmon-65aab860bd16c16c1a949a62a5da5f62ca26a164d89bd5d343523068546ff49f.scope /run/systemd/transient/libpod-65aab860bd16c16c1a949a62a5da5f62ca26a164d89bd5d343523068546ff49f.scope.d: dep.conf /run/systemd/transient/libpod-conmon-65aab860bd16c16c1a949a62a5da5f62ca26a164d89bd5d343523068546ff49f.scope.d: dep.conf [stack@undercloud-0 ~]$ ansible database -mshell -b -a'systemctl cat libpod*$(podman ps -q --filter name=galera-bundle)*' [WARNING]: Found both group and host with same name: undercloud database-0 | CHANGED | rc=0 >> # /run/systemd/transient/libpod-conmon-c7d560ce484a019d286b23c8c0d608d10e9c9cd92d042e5a8467ca11f82d3b05.scope # This is a transient unit file, created programmatically via the systemd API. Do not edit. [Scope] Slice=machine.slice Delegate=yes [Unit] DefaultDependencies=no # /run/systemd/transient/libpod-conmon-c7d560ce484a019d286b23c8c0d608d10e9c9cd92d042e5a8467ca11f82d3b05.scope.d/dep.conf [Unit] Before=pacemaker.service # /run/systemd/transient/libpod-c7d560ce484a019d286b23c8c0d608d10e9c9cd92d042e5a8467ca11f82d3b05.scope # This is a transient unit file, created programmatically via the systemd API. Do not edit. [Unit] Description=libcontainer container c7d560ce484a019d286b23c8c0d608d10e9c9cd92d042e5a8467ca11f82d3b05 [Scope] Slice=machine.slice Delegate=yes MemoryAccounting=yes CPUAccounting=yes BlockIOAccounting=yes [Unit] DefaultDependencies=yes # /run/systemd/transient/libpod-c7d560ce484a019d286b23c8c0d608d10e9c9cd92d042e5a8467ca11f82d3b05.scope.d/dep.conf [Unit] Before=pacemaker.service database-1 | CHANGED | rc=0 >> # /run/systemd/transient/libpod-conmon-324ee8f480f81b69761c899115be0866bc325a3f8305d9d01f8e611217073c1e.scope # This is a transient unit file, created programmatically via the systemd API. Do not edit. [Scope] Slice=machine.slice Delegate=yes [Unit] DefaultDependencies=no # /run/systemd/transient/libpod-conmon-324ee8f480f81b69761c899115be0866bc325a3f8305d9d01f8e611217073c1e.scope.d/dep.conf [Unit] Before=pacemaker.service # /run/systemd/transient/libpod-324ee8f480f81b69761c899115be0866bc325a3f8305d9d01f8e611217073c1e.scope # This is a transient unit file, created programmatically via the systemd API. Do not edit. [Unit] Description=libcontainer container 324ee8f480f81b69761c899115be0866bc325a3f8305d9d01f8e611217073c1e [Scope] Slice=machine.slice Delegate=yes MemoryAccounting=yes CPUAccounting=yes BlockIOAccounting=yes [Unit] DefaultDependencies=yes # /run/systemd/transient/libpod-324ee8f480f81b69761c899115be0866bc325a3f8305d9d01f8e611217073c1e.scope.d/dep.conf [Unit] Before=pacemaker.service database-2 | CHANGED | rc=0 >> # /run/systemd/transient/libpod-conmon-65aab860bd16c16c1a949a62a5da5f62ca26a164d89bd5d343523068546ff49f.scope # This is a transient unit file, created programmatically via the systemd API. Do not edit. [Scope] Slice=machine.slice Delegate=yes [Unit] DefaultDependencies=no # /run/systemd/transient/libpod-conmon-65aab860bd16c16c1a949a62a5da5f62ca26a164d89bd5d343523068546ff49f.scope.d/dep.conf [Unit] Before=pacemaker.service # /run/systemd/transient/libpod-65aab860bd16c16c1a949a62a5da5f62ca26a164d89bd5d343523068546ff49f.scope # This is a transient unit file, created programmatically via the systemd API. Do not edit. [Unit] Description=libcontainer container 65aab860bd16c16c1a949a62a5da5f62ca26a164d89bd5d343523068546ff49f [Scope] Slice=machine.slice Delegate=yes MemoryAccounting=yes CPUAccounting=yes BlockIOAccounting=yes [Unit] DefaultDependencies=yes # /run/systemd/transient/libpod-65aab860bd16c16c1a949a62a5da5f62ca26a164d89bd5d343523068546ff49f.scope.d/dep.conf [Unit] Before=pacemaker.service [root@overcloud-database-0 ~]# journalctl -b -1 |grep -A 99999 'Aug 25 18:30:29'|grep 'pacemaker\|galera' Aug 25 18:30:29 overcloud-database-0 pacemaker-remoted[852298]: notice: Caught 'Terminated' signal Aug 25 18:30:29 overcloud-database-0 pacemaker-remoted[852298]: notice: TLS server session ended Aug 25 18:30:29 overcloud-database-0 pacemaker-remoted[403303]: notice: Caught 'Terminated' signal Aug 25 18:30:29 overcloud-database-0 pacemaker-remoted[403303]: notice: TLS server session ended Aug 25 18:30:29 overcloud-database-0 pacemaker-remoted[852298]: notice: Caught 'Terminated' signal Aug 25 18:30:29 overcloud-database-0 pacemaker-remoted[852298]: notice: Waiting for cluster to stop resources before exiting Aug 25 18:30:31 overcloud-database-0 galera(galera)[875725]: DEBUG: MySQL still hasn't stopped yet. Waiting... Aug 25 18:30:32 overcloud-database-0 galera(galera)[875731]: DEBUG: MySQL still hasn't stopped yet. Waiting... Aug 25 18:30:33 overcloud-database-0 galera(galera)[875741]: DEBUG: MySQL still hasn't stopped yet. Waiting... Aug 25 18:30:33 overcloud-database-0 galera(galera)[875745]: INFO: MySQL stopped Aug 25 18:30:34 overcloud-database-0 galera(galera)[875759]: INFO: attempting to read safe_to_bootstrap flag from /var/lib/mysql/grastate.dat Aug 25 18:30:34 overcloud-database-0 galera(galera)[875766]: INFO: attempting to detect last commit version by reading /var/lib/mysql/grastate.dat Aug 25 18:30:34 overcloud-database-0 galera(galera)[875773]: INFO: Last commit version found: 1805208 Aug 25 18:30:35 overcloud-database-0 galera(galera)[875829]: INFO: MySQL is not running Aug 25 18:30:36 overcloud-database-0 pacemaker-remoted[852298]: notice: Cleaning up after remote client pacemaker-remote-overcloud-database-0:3123 disconnected Aug 25 18:30:36 overcloud-database-0 podman(galera-bundle-podman-0)[875891]: NOTICE: Cleaning up inactive container, galera-bundle-podman-0. Aug 25 18:30:36 overcloud-database-0 podman(galera-bundle-podman-0)[875911]: INFO: c7d560ce484a019d286b23c8c0d608d10e9c9cd92d042e5a8467ca11f82d3b05 Aug 25 18:30:36 overcloud-database-0 podman(galera-bundle-podman-0)[875915]: DEBUG: galera-bundle-podman-0 stop : 0 Aug 25 18:30:36 overcloud-database-0 pacemaker-remoted[403303]: notice: Cleaning up after remote client pacemaker-remote-172.17.1.132:3121 disconnected [root@overcloud-database-0 ~]# cat /var/log/pacemaker/pacemaker.log|grep -i 'pacemaker\|galera'|grep '18:3'|head Aug 25 18:30:29 overcloud-database-0 pacemaker-remoted [403303] (crm_signal_dispatch) notice: Caught 'Terminated' signal | 15 (invoking handler) Aug 25 18:30:29 overcloud-database-0 pacemaker-remoted [403303] (lrmd_shutdown) info: Sending shutdown request to cluster Aug 25 18:30:29 overcloud-database-0 pacemaker-remoted [403303] (lrmd_remote_connection_destroy) notice: TLS server session ended Aug 25 18:30:29 overcloud-database-0 pacemaker-remoted [403303] (handle_shutdown_ack) info: Received shutdown ack Aug 25 18:30:36 overcloud-database-0 pacemaker-remoted [403303] (cancel_recurring_action) info: Cancelling ocf operation galera-bundle-podman-0_monitor_60000 Aug 25 18:30:36 overcloud-database-0 pacemaker-remoted [403303] (log_execute) info: executing - rsc:galera-bundle-podman-0 action:stop call_id:1113 Aug 25 18:30:36 overcloud-database-0 pacemaker-remoted [403303] (log_finished) info: finished - rsc:galera-bundle-podman-0 action:stop call_id:1113 pid:875851 exit-code:0 exec-time:188ms queue-time:1ms Aug 25 18:30:36 overcloud-database-0 pacemaker-remoted [403303] (lrmd_remote_client_msg) info: Remote client disconnected while reading from it Aug 25 18:30:36 overcloud-database-0 pacemaker-remoted [403303] (lrmd_remote_client_destroy) notice: Cleaning up after remote client pacemaker-remote-172.17.1.132:3121 disconnected | id=35ccc46b-c3ab-4adf-a849-82fad5b9cce6 Aug 25 18:30:36 overcloud-database-0 pacemaker-remoted [403303] (lrmd_exit) info: Terminating with 0 clients Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2700 |