Bug 1823178
| Summary: | [OSP16.1] Openvswitch segfaults multiple times causing API failures | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Roman Safronov <rsafrono> |
| Component: | openvswitch | Assignee: | RHOS Maint <rhos-maint> |
| Status: | CLOSED DUPLICATE | QA Contact: | nlevinki <nlevinki> |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | 16.1 (Train) | CC: | apevec, chrisw, jjoyce, jschluet, lmiccini, michele, rhos-maint, slinaber, tvignaud |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-04-16 12:16:22 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
[root@controller-0 mysql]# podman exec -it galera-bundle-podman-0 rpm -qa | grep maria mariadb-common-10.3.17-1.module+el8.1.0+3974+90eded84.x86_64 mariadb-connector-c-3.0.7-1.el8.x86_64 mariadb-errmsg-10.3.17-1.module+el8.1.0+3974+90eded84.x86_64 mariadb-server-utils-10.3.17-1.module+el8.1.0+3974+90eded84.x86_64 mariadb-server-galera-10.3.17-1.module+el8.1.0+3974+90eded84.x86_64 mariadb-connector-c-config-3.0.7-1.el8.noarch mariadb-10.3.17-1.module+el8.1.0+3974+90eded84.x86_64 mariadb-server-10.3.17-1.module+el8.1.0+3974+90eded84.x86_64 mariadb-backup-10.3.17-1.module+el8.1.0+3974+90eded84.x86_64 Hi Roman, fyi the logs in the sosreports do not cover the timestamps in the description so I am not 100% sure I am looking at the right stuff, here what happens on controller-0: Apr 9 18:45:21 controller-0 corosync[36483]: [KNET ] link: host: 3 link: 0 is down Apr 9 18:45:21 controller-0 corosync[36483]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) Apr 9 18:45:21 controller-0 corosync[36483]: [KNET ] host: host: 3 has no active links Apr 9 18:45:22 controller-0 corosync[36483]: [TOTEM ] Token has not been received in 844 ms Apr 9 18:45:23 controller-0 corosync[36483]: [KNET ] link: host: 2 link: 0 is down Apr 9 18:45:23 controller-0 corosync[36483]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) Apr 9 18:45:23 controller-0 corosync[36483]: [KNET ] host: host: 2 has no active links Apr 9 18:45:23 controller-0 corosync[36483]: [TOTEM ] A processor failed, forming new configuration. Apr 9 18:45:25 controller-0 corosync[36483]: [TOTEM ] A new membership (1.d) was formed. Members left: 2 3 Apr 9 18:45:25 controller-0 corosync[36483]: [TOTEM ] Failed to receive the leave message. failed: 2 3 Apr 9 18:45:26 controller-0 corosync[36483]: [KNET ] rx: host: 3 link: 0 is up Apr 9 18:45:26 controller-0 corosync[36483]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) Apr 9 18:45:26 controller-0 corosync[36483]: [KNET ] rx: host: 2 link: 0 is up Apr 9 18:45:26 controller-0 corosync[36483]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) Apr 9 18:45:28 controller-0 corosync[36483]: [KNET ] link: host: 3 link: 0 is down Apr 9 18:45:28 controller-0 corosync[36483]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) Apr 9 18:45:28 controller-0 corosync[36483]: [KNET ] host: host: 3 has no active links Apr 9 18:45:30 controller-0 corosync[36483]: [TOTEM ] Token has not been received in 3274 ms Apr 9 18:45:31 controller-0 corosync[36483]: [KNET ] link: host: 2 link: 0 is down Apr 9 18:45:31 controller-0 corosync[36483]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) Apr 9 18:45:31 controller-0 corosync[36483]: [KNET ] host: host: 2 has no active links Apr 9 18:45:32 controller-0 corosync[36483]: [TOTEM ] A new membership (1.19) was formed. Members Apr 9 18:45:32 controller-0 corosync[36483]: [KNET ] rx: host: 3 link: 0 is up Apr 9 18:45:32 controller-0 corosync[36483]: [KNET ] rx: host: 2 link: 0 is up Apr 9 18:45:32 controller-0 corosync[36483]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) Apr 9 18:45:32 controller-0 corosync[36483]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) Apr 9 18:45:33 controller-0 corosync[36483]: [TOTEM ] A new membership (1.1d) was formed. Members joined: 2 Apr 9 18:45:33 controller-0 corosync[36483]: [TOTEM ] A new membership (1.21) was formed. Members joined: 3 Apr 9 18:52:45 controller-0 corosync[36483]: [KNET ] link: host: 2 link: 0 is down Apr 9 18:52:45 controller-0 corosync[36483]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) Apr 9 18:52:45 controller-0 corosync[36483]: [KNET ] host: host: 2 has no active links Apr 9 18:52:46 controller-0 corosync[36483]: [KNET ] link: host: 3 link: 0 is down Apr 9 18:52:46 controller-0 corosync[36483]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) Apr 9 18:52:46 controller-0 corosync[36483]: [KNET ] host: host: 3 has no active links Apr 9 18:52:46 controller-0 corosync[36483]: [TOTEM ] Token has not been received in 1237 ms Apr 9 18:52:47 controller-0 corosync[36483]: [TOTEM ] A processor failed, forming new configuration. Apr 9 18:52:49 controller-0 corosync[36483]: [TOTEM ] A new membership (1.25) was formed. Members left: 2 3 Apr 9 18:52:49 controller-0 corosync[36483]: [TOTEM ] Failed to receive the leave message. failed: 2 3 Apr 9 18:52:49 controller-0 corosync[36483]: [KNET ] rx: host: 3 link: 0 is up Apr 9 18:52:49 controller-0 corosync[36483]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) Apr 9 18:52:52 controller-0 corosync[36483]: [KNET ] rx: host: 2 link: 0 is up Apr 9 18:52:52 controller-0 corosync[36483]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) Apr 9 18:52:52 controller-0 corosync[36483]: [TOTEM ] Token has not been received in 2186 ms Apr 9 18:52:52 controller-0 corosync[36483]: [TOTEM ] A new membership (1.29) was formed. Members joined: 2 3 Apr 9 18:55:29 controller-0 corosync[36483]: [KNET ] link: host: 3 link: 0 is down Apr 9 18:55:29 controller-0 corosync[36483]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) Apr 9 18:55:29 controller-0 corosync[36483]: [KNET ] host: host: 3 has no active links Apr 9 18:55:30 controller-0 corosync[36483]: [TOTEM ] Token has not been received in 451 ms Apr 9 18:55:30 controller-0 corosync[36483]: [KNET ] link: host: 2 link: 0 is down Apr 9 18:55:30 controller-0 corosync[36483]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) Apr 9 18:55:30 controller-0 corosync[36483]: [KNET ] host: host: 2 has no active links Apr 9 18:55:30 controller-0 corosync[36483]: [TOTEM ] A processor failed, forming new configuration. Apr 9 18:55:33 controller-0 corosync[36483]: [KNET ] rx: host: 3 link: 0 is up Apr 9 18:55:33 controller-0 corosync[36483]: [KNET ] rx: host: 2 link: 0 is up Apr 9 18:55:33 controller-0 corosync[36483]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) Apr 9 18:55:33 controller-0 corosync[36483]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) Apr 9 18:55:34 controller-0 corosync[36483]: [TOTEM ] A new membership (1.35) was formed. Members left: 2 3 Apr 9 18:55:34 controller-0 corosync[36483]: [TOTEM ] Failed to receive the leave message. failed: 2 3 Apr 9 18:55:34 controller-0 corosync[36483]: [TOTEM ] A new membership (1.39) was formed. Members joined: 2 3 Apr 9 18:56:47 controller-0 corosync[36483]: [KNET ] link: host: 3 link: 0 is down Apr 9 18:56:47 controller-0 corosync[36483]: [KNET ] link: host: 2 link: 0 is down Apr 9 18:56:47 controller-0 corosync[36483]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) Apr 9 18:56:47 controller-0 corosync[36483]: [KNET ] host: host: 3 has no active links Apr 9 18:56:47 controller-0 corosync[36483]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) Apr 9 18:56:47 controller-0 corosync[36483]: [KNET ] host: host: 2 has no active links Apr 9 18:56:47 controller-0 corosync[36483]: [TOTEM ] Token has not been received in 829 ms Apr 9 18:56:47 controller-0 corosync[36483]: [TOTEM ] A processor failed, forming new configuration. Apr 9 18:56:49 controller-0 corosync[36483]: [TOTEM ] A new membership (1.3d) was formed. Members left: 2 3 Apr 9 18:56:49 controller-0 corosync[36483]: [TOTEM ] Failed to receive the leave message. failed: 2 3 Apr 9 18:56:51 controller-0 corosync[36483]: [KNET ] rx: host: 3 link: 0 is up Apr 9 18:56:51 controller-0 corosync[36483]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) Apr 9 18:56:52 controller-0 corosync[36483]: [KNET ] rx: host: 2 link: 0 is up Apr 9 18:56:52 controller-0 corosync[36483]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) Apr 9 18:56:52 controller-0 corosync[36483]: [TOTEM ] A new membership (1.41) was formed. Members joined: 2 Apr 9 18:56:53 controller-0 corosync[36483]: [TOTEM ] A new membership (1.45) was formed. Members joined: 3 same stuff on the other nodes. Controllers are randomly getting isolated and since you don't have fencing configured/enabled pacemaker can't recover promptly (if at all). It seems to me that the root cause could be the load on the environment but since we don't have enough data in the sosreports I can't be 100% sure. If you happen to reproduce maybe give us a ping and we'll look at the live env. Openvswitch version is openvswitch2.13-2.13.0-0.20200117git8ae6a5f.el8fdp.1.x86_64 [heat-admin@controller-0 ~]$ sudo podman exec -it ovn_controller rpm -qa | grep openvswitch rhosp-openvswitch-2.13-7.el8ost.noarch rhosp-openvswitch-ovn-host-2.13-7.el8ost.noarch openvswitch-selinux-extra-policy-1.0-22.el8fdp.noarch python3-openvswitch2.13-2.13.0-0.20200117git8ae6a5f.el8fdp.1.x86_64 python3-rhosp-openvswitch-2.13-7.el8ost.noarch network-scripts-openvswitch2.13-2.13.0-0.20200117git8ae6a5f.el8fdp.1.x86_64 openvswitch2.13-2.13.0-0.20200117git8ae6a5f.el8fdp.1.x86_64 *** This bug has been marked as a duplicate of bug 1821185 *** |
Description of problem: While working with OSP16.1 (usual scenario, create security group, keypair, networks, run instance) I noticed that sometimes API requests are failing. When I rerun same request next time it usually works. In /var/log/messages on controller I found multiple errors that mysql server is often not running: Apr 12 11:11:52 controller-0 galera(galera)[31084]: ERROR: MySQL is not running Apr 12 11:12:00 controller-0 galera(galera)[31211]: ERROR: MySQL is not running Apr 12 11:12:04 controller-0 pacemaker-schedulerd[2775]: warning: Unexpected result (error: local node <controller-0> is started, but not in primary mode. Unknown state.) was recorded for monitor of galera:0 on galera-bundle-0 at Apr 12 11:03:13 2020 Apr 12 11:12:05 controller-0 galera(galera)[31338]: ERROR: MySQL is not running Apr 12 11:12:18 controller-0 pacemaker-schedulerd[2775]: warning: Unexpected result (error: local node <controller-0> is started, but not in primary mode. Unknown state.) was recorded for monitor of galera:0 on galera-bundle-0 at Apr 12 11:03:13 2020 Apr 12 11:12:31 controller-0 pacemaker-schedulerd[2775]: warning: Unexpected result (error: local node <controller-0> is started, but not in primary mode. Unknown state.) was recorded for monitor of galera:0 on galera-bundle-0 at Apr 12 11:03:13 2020 Apr 12 11:15:45 controller-0 pacemaker-schedulerd[2775]: warning: Unexpected result (error: local node <controller-0> is started, but not in primary mode. Unknown state.) was recorded for monitor of galera:0 on galera-bundle-0 at Apr 12 11:03:13 2020 Apr 12 11:15:51 controller-0 pacemaker-schedulerd[2775]: warning: Unexpected result (error: local node <controller-0> is started, but not in primary mode. Unknown state.) was recorded for monitor of galera:0 on galera-bundle-0 at Apr 12 11:03:13 2020 Apr 12 11:15:52 controller-0 galera(galera)[33428]: ERROR: MySQL is not running Apr 12 11:15:52 controller-0 pacemaker-controld[2778]: error: Failed to retrieve meta-data for ocf:ovn:ovndb-servers Apr 12 11:15:53 controller-0 pacemaker-controld[2778]: error: Failed to retrieve meta-data for ocf:ovn:ovndb-servers Apr 12 11:15:53 controller-0 galera(galera)[33492]: ERROR: MySQL is not running Apr 12 11:16:15 controller-0 galera(galera)[33556]: ERROR: MySQL is not running Apr 12 11:16:23 controller-0 galera(galera)[33620]: ERROR: MySQL is not running Apr 12 11:16:26 controller-0 pacemaker-schedulerd[2775]: warning: Unexpected result (error: local node <controller-0> is started, but not in primary mode. Unknown state.) was recorded for monitor of galera:0 on galera-bundle-0 at Apr 12 11:03:13 2020 Apr 12 11:16:26 controller-0 pacemaker-controld[2778]: error: Failed to retrieve meta-data for ocf:ovn:ovndb-servers Apr 12 11:16:27 controller-0 galera(galera)[33684]: ERROR: MySQL is not running Apr 12 11:16:27 controller-0 pacemaker-controld[2778]: error: Failed to retrieve meta-data for ocf:ovn:ovndb-servers Apr 12 11:16:40 controller-0 pacemaker-schedulerd[2775]: warning: Unexpected result (error: local node <controller-0> is started, but not in primary mode. Unknown state.) was recorded for monitor of galera:0 on galera-bundle-0 at Apr 12 11:03:13 2020 Apr 12 11:16:40 controller-0 pacemaker-controld[2778]: error: Failed to retrieve meta-data for ocf:ovn:ovndb-servers Apr 12 11:16:40 controller-0 pacemaker-controld[2778]: error: Failed to retrieve meta-data for ocf:ovn:ovndb-servers Apr 12 11:20:28 controller-0 pacemaker-schedulerd[2775]: warning: Unexpected result (error: local node <controller-0> is started, but not in primary mode. Unknown state.) was recorded for monitor of galera:0 on galera-bundle-0 at Apr 12 11:03:13 2020 Apr 12 11:20:29 controller-0 pacemaker-controld[2778]: error: Failed to retrieve meta-data for ocf:ovn:ovndb-servers Apr 12 11:20:30 controller-0 pacemaker-controld[2778]: error: Failed to retrieve meta-data for ocf:ovn:ovndb-servers Apr 12 11:20:30 controller-0 pacemaker-controld[2778]: error: Failed to retrieve meta-data for ocf:ovn:ovndb-servers Apr 12 11:20:32 controller-0 pacemaker-remoted[7]: notice: rabbitmq_stop_0:168013:stderr [ Error: unable to perform an operation on node 'rabbit@controller-0'. Please see diagnostics information and suggestions below. ] Apr 12 11:20:35 controller-0 pacemaker-schedulerd[2775]: warning: Unexpected result (error: local node <controller-0> is started, but not in primary mode. Unknown state.) was recorded for monitor of galera:0 on galera-bundle-0 at Apr 12 11:03:13 2020 Apr 12 11:20:35 controller-0 galera(galera)[35862]: ERROR: MySQL is not running Apr 12 11:20:37 controller-0 galera(galera)[35926]: ERROR: MySQL is not running Apr 12 11:20:50 controller-0 pacemaker-remoted[7]: notice: rabbitmq_start_0:168570:stderr [ Error: unable to perform an operation on node 'rabbit@controller-0'. Please see diagnostics information and suggestions below. ] Apr 12 11:20:50 controller-0 pacemaker-remoted[7]: notice: rabbitmq_start_0:168570:stderr [ Error: unable to perform an operation on node 'rabbit@controller-0'. Please see diagnostics information and suggestions below. ] Apr 12 11:20:58 controller-0 galera(galera)[35996]: ERROR: MySQL is not running Apr 12 11:20:59 controller-0 pacemaker-schedulerd[2775]: warning: Unexpected result (error: local node <controller-0> is started, but not in primary mode. Unknown state.) was recorded for monitor of galera:0 on galera-bundle-0 at Apr 12 11:03:13 2020 Apr 12 11:20:59 controller-0 pacemaker-controld[2778]: error: Failed to retrieve meta-data for ocf:ovn:ovndb-servers Apr 12 11:21:00 controller-0 pacemaker-controld[2778]: error: Failed to retrieve meta-data for ocf:ovn:ovndb-servers Apr 12 11:21:08 controller-0 pacemaker-schedulerd[2775]: warning: Unexpected result (error: local node <controller-0> is started, but not in primary mode. Unknown state.) was recorded for monitor of galera:0 on galera-bundle-0 at Apr 12 11:03:13 2020 Apr 12 11:21:08 controller-0 pacemaker-controld[2778]: error: Failed to retrieve meta-data for ocf:ovn:ovndb-servers Apr 12 11:21:08 controller-0 pacemaker-controld[2778]: error: Failed to retrieve meta-data for ocf:ovn:ovndb-servers Apr 12 11:21:21 controller-0 pacemaker-schedulerd[2775]: warning: Unexpected result (error: local node <controller-0> is started, but not in primary mode. Unknown state.) was recorded for monitor of galera:0 on galera-bundle-0 at Apr 12 11:03:13 2020 Apr 12 11:21:22 controller-0 galera(galera)[36056]: ERROR: MySQL is not running Apr 12 11:21:39 controller-0 pacemaker-schedulerd[2775]: warning: Unexpected result (error: local node <controller-0> is started, but not in primary mode. Unknown state.) was recorded for monitor of galera:0 on galera-bundle-0 at Apr 12 11:03:13 2020 from mysqld.log 2020-04-12 11:03:04 0 [ERROR] WSREP: exception from gcomm, backend must be restarted: evs::proto(9dec0f6b, GATHER, view_id(REG,5cb51706,19)) failed to form singleton view after exceeding max _install_timeouts 3, giving up (FATAL) at gcomm/src/evs_proto.cpp:handle_install_timer():742 2020-04-12 11:03:04 0 [Note] WSREP: gcomm: terminating thread 2020-04-12 11:03:04 0 [Note] WSREP: gcomm: joining thread 2020-04-12 11:03:04 0 [Note] WSREP: gcomm: closing backend 2020-04-12 11:03:04 0 [Note] WSREP: Forced PC close 2020-04-12 11:03:04 0 [Warning] WSREP: discarding 18 messages from message index 2020-04-12 11:03:04 0 [Note] WSREP: gcomm: closed 2020-04-12 11:03:04 0 [Note] WSREP: Received self-leave message. 2020-04-12 11:03:04 0 [Note] WSREP: comp msg error in core 103 2020-04-12 11:03:04 0 [Note] WSREP: Closing send monitor... 2020-04-12 11:03:04 2 [Note] WSREP: New cluster view: global state: 00000000-0000-0000-0000-000000000000:0, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version -1 2020-04-12 11:03:04 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2020-04-12 11:03:04 2 [Note] WSREP: applier thread exiting (code:6) 2020-04-12 11:03:04 664 [Warning] WSREP: Send action {(nil), 1784, TORDERED} returned -103 (Software caused connection abort) 2020-04-12 11:03:04 0 [Note] WSREP: Closed send monitor. 2020-04-12 11:03:04 0 [Note] WSREP: Closing replication queue. 2020-04-12 11:03:04 0 [Note] WSREP: Closing slave action queue. 2020-04-12 11:03:04 0 [Note] WSREP: Shifting SYNCED -> CLOSED (TO: 370206) 2020-04-12 11:03:04 0 [Note] WSREP: RECV thread exiting -103: Software caused connection abort 2020-04-12 11:03:06 685 [Warning] Aborted connection 685 to db: 'cinder' user: 'cinder' host: '172.17.1.118' (Got an error reading communication packets) 2020-04-12 11:03:07 693 [Warning] Aborted connection 693 to db: 'nova' user: 'nova' host: '172.17.1.118' (Got an error reading communication packets) 2020-04-12 11:03:07 692 [Warning] Aborted connection 692 to db: 'nova' user: 'nova' host: '172.17.1.118' (Got an error reading communication packets) 2020-04-12 11:03:07 688 [Warning] Aborted connection 688 to db: 'nova' user: 'nova' host: '172.17.1.118' (Got an error reading communication packets) 2020-04-12 11:03:07 687 [Warning] Aborted connection 687 to db: 'nova' user: 'nova' host: '172.17.1.118' (Got an error reading communication packets) 2020-04-12 11:03:07 686 [Warning] Aborted connection 686 to db: 'nova' user: 'nova' host: '172.17.1.118' (Got an error reading communication packets) 2020-04-12 11:03:07 678 [Warning] Aborted connection 678 to db: 'heat' user: 'heat' host: '172.17.1.118' (Got an error reading communication packets) 2020-04-12 11:03:07 677 [Warning] Aborted connection 677 to db: 'heat' user: 'heat' host: '172.17.1.118' (Got an error reading communication packets) 2020-04-12 11:03:07 675 [Warning] Aborted connection 675 to db: 'nova' user: 'nova' host: '172.17.1.118' (Got an error reading communication packets) 2020-04-12 11:03:07 679 [Warning] Aborted connection 679 to db: 'cinder' user: 'cinder' host: '172.17.1.118' (Got an error reading communication packets) 2020-04-12 11:03:07 682 [Warning] Aborted connection 682 to db: 'cinder' user: 'cinder' host: '172.17.1.118' (Got an error reading communication packets) 2020-04-12 11:03:07 684 [Warning] Aborted connection 684 to db: 'nova' user: 'nova' host: '172.17.1.118' (Got an error reading communication packets) 2020-04-12 11:03:07 667 [Warning] Aborted connection 667 to db: 'nova' user: 'nova' host: '172.17.1.118' (Got an error reading communication packets) 2020-04-12 11:03:07 666 [Warning] Aborted connection 666 to db: 'nova' user: 'nova' host: '172.17.1.118' (Got an error reading communication packets) 2020-04-12 11:03:07 665 [Warning] Aborted connection 665 to db: 'nova' user: 'nova' host: '172.17.1.118' (Got an error reading communication packets) 2020-04-12 11:03:07 664 [Warning] Aborted connection 664 to db: 'ovs_neutron' user: 'neutron' host: '172.17.1.118' (Got an error reading communication packets) 2020-04-12 11:03:07 668 [Warning] Aborted connection 668 to db: 'nova' user: 'nova' host: '172.17.1.118' (Got an error reading communication packets) 2020-04-12 11:03:15 0 [Note] /usr/libexec/mysqld (initiated by: unknown): Normal shutdown Version-Release number of selected component (if applicable): 16.1-RHEL-8/RHOS-16.1-RHEL-8-20200407.n.0 puppet-mysql-10.4.0-0.20200328015238.95f9b98.el8ost.noarch How reproducible: 100% Steps to Reproduce: Try to create network, router, security group with icmp and ssh rules, keypair, launch instance using openstack CLI client Actual results: After 2-3 API calls error returned. See [1], [2] below note: In case I rerun command once again it works. Expected results: No unexpected errors when running CLI commands Additional info: [1] (overcloud) [stack@undercloud-0 roman]$ openstack security group create overcloud_sg Unable to establish connection to http://10.0.0.111:5000/v3/auth/tokens: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',)) [2] (overcloud) [stack@undercloud-0 roman]$ openstack floating ip create nova Failed to discover available identity versions when contacting http://10.0.0.111:5000. Attempting to parse version from URL. Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://10.0.0.111:5000: HTTPConnectionPool(host='10.0.0.111', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6190957940>: Failed to establish a new connection: [Errno 111] Connection refused',)) Logs available here: http://rhos-release.virt.bos.redhat.com/log/bzosp161_neutron_no_response