Attachment 1491282 Details for Bug 1636714 – logs

logs

restart_ovn_north_d.log (text/plain), 16.44 KB, created by Eran Kuris on 2018-10-07 07:39:07 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Eran Kuris

Created: 2018-10-07 07:39:07 UTC

Size: 16.44 KB

patch

obsolete

>step  to reproduce
>
>[heat-admin@controller-0 ~]$ sudo -i 
>[root@controller-0 ~]# pcs status 
>Cluster name: tripleo_cluster
>Stack: corosync
>Current DC: controller-1 (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
>Last updated: Sun Oct  7 05:51:47 2018
>Last change: Wed Oct  3 16:50:47 2018 by root via cibadmin on controller-0
>
>15 nodes configured
>46 resources configured
>
>Online: [ controller-0 controller-1 controller-2 ]
>GuestOnline: [ galera-bundle-0@controller-0 galera-bundle-1@controller-1 galera-bundle-2@controller-2 ovn-dbs-bundle-0@controller-0 ovn-dbs-bundle-1@controller-1 ovn-dbs-bundle-2@controller-2 rabbitmq-bundle-0@controller-0 rabbitmq-bundle-1@controller-1 rabbitmq-bundle-2@controller-2 redis-bundle-0@controller-0 redis-bundle-1@controller-1 redis-bundle-2@controller-2 ]
>
>Full list of resources:
>
> Docker container set: rabbitmq-bundle [192.168.24.1:8787/rhosp14/openstack-rabbitmq:pcmklatest]
>   rabbitmq-bundle-0	(ocf::heartbeat:rabbitmq-cluster):	Started controller-0
>   rabbitmq-bundle-1	(ocf::heartbeat:rabbitmq-cluster):	Started controller-1
>   rabbitmq-bundle-2	(ocf::heartbeat:rabbitmq-cluster):	Started controller-2
> Docker container set: galera-bundle [192.168.24.1:8787/rhosp14/openstack-mariadb:pcmklatest]
>   galera-bundle-0	(ocf::heartbeat:galera):	Master controller-0
>   galera-bundle-1	(ocf::heartbeat:galera):	Master controller-1
>   galera-bundle-2	(ocf::heartbeat:galera):	Master controller-2
> Docker container set: redis-bundle [192.168.24.1:8787/rhosp14/openstack-redis:pcmklatest]
>   redis-bundle-0	(ocf::heartbeat:redis):	Master controller-0
>   redis-bundle-1	(ocf::heartbeat:redis):	Slave controller-1
>   redis-bundle-2	(ocf::heartbeat:redis):	Slave controller-2
> ip-192.168.24.6	(ocf::heartbeat:IPaddr2):	Started controller-0
> ip-10.0.0.104	(ocf::heartbeat:IPaddr2):	Started controller-1
> ip-172.17.1.18	(ocf::heartbeat:IPaddr2):	Started controller-2
> ip-172.17.1.23	(ocf::heartbeat:IPaddr2):	Started controller-0
> ip-172.17.3.10	(ocf::heartbeat:IPaddr2):	Started controller-1
> ip-172.17.4.33	(ocf::heartbeat:IPaddr2):	Started controller-2
> Docker container set: haproxy-bundle [192.168.24.1:8787/rhosp14/openstack-haproxy:pcmklatest]
>   haproxy-bundle-docker-0	(ocf::heartbeat:docker):	Started controller-0
>   haproxy-bundle-docker-1	(ocf::heartbeat:docker):	Started controller-1
>   haproxy-bundle-docker-2	(ocf::heartbeat:docker):	Started controller-2
> Docker container set: ovn-dbs-bundle [192.168.24.1:8787/rhosp14/openstack-ovn-northd:pcmklatest]
>   ovn-dbs-bundle-0	(ocf::ovn:ovndb-servers):	Master controller-0
>   ovn-dbs-bundle-1	(ocf::ovn:ovndb-servers):	Slave controller-1
>   ovn-dbs-bundle-2	(ocf::ovn:ovndb-servers):	Slave controller-2
> Docker container: openstack-cinder-volume [192.168.24.1:8787/rhosp14/openstack-cinder-volume:pcmklatest]
>   openstack-cinder-volume-docker-0	(ocf::heartbeat:docker):	Started controller-0
>
>Daemon Status:
>  corosync: active/enabled
>  pacemaker: active/enabled
>  pcsd: active/enabled
>[root@controller-0 ~]# docker ps | grep ovn 
>d2d80f7c2730        192.168.24.1:8787/rhosp14/openstack-ovn-controller:2018-10-01.1            "kolla_start"            3 days ago          Up 3 days                                   ovn_controller
>2f6351ac2b48        192.168.24.1:8787/rhosp14/openstack-neutron-server-ovn:2018-10-01.1        "kolla_start"            3 days ago          Up 3 days (healthy)                         neutron_api
>6ffc3f666301        192.168.24.1:8787/rhosp14/openstack-nova-novncproxy:2018-10-01.1           "kolla_start"            3 days ago          Up 3 days (healthy)                         nova_vnc_proxy
>772c87448739        192.168.24.1:8787/rhosp14/openstack-ovn-northd:pcmklatest                  "/bin/bash /usr/lo..."   3 days ago          Up 3 days                                   ovn-dbs-bundle-docker-0
>[root@controller-0 ~]# docker restart ovn-dbs-bundle-docker-0
>
>
>[root@controller-0 ~]# pcs status 
>Cluster name: tripleo_cluster
>Stack: corosync
>Current DC: controller-1 (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
>Last updated: Sun Oct  7 07:18:30 2018
>Last change: Sun Oct  7 07:12:28 2018 by ovn-dbs-bundle-1 via crm_attribute on controller-1
>
>15 nodes configured
>46 resources configured
>
>Online: [ controller-0 controller-1 controller-2 ]
>GuestOnline: [ galera-bundle-0@controller-0 galera-bundle-1@controller-1 galera-bundle-2@controller-2 ovn-dbs-bundle-0@controller-0 ovn-dbs-bundle-1@controller-1 ovn-dbs-bundle-2@controller-2 rabbitmq-bundle-0@controller-0 rabbitmq-bundle-1@controller-1 rabbitmq-bundle-2@controller-2 redis-bundle-0@controller-0 redis-bundle-1@controller-1 redis-bundle-2@controller-2 ]
>
>Full list of resources:
>
> Docker container set: rabbitmq-bundle [192.168.24.1:8787/rhosp14/openstack-rabbitmq:pcmklatest]
>   rabbitmq-bundle-0	(ocf::heartbeat:rabbitmq-cluster):	Started controller-0
>   rabbitmq-bundle-1	(ocf::heartbeat:rabbitmq-cluster):	Started controller-1
>   rabbitmq-bundle-2	(ocf::heartbeat:rabbitmq-cluster):	Started controller-2
> Docker container set: galera-bundle [192.168.24.1:8787/rhosp14/openstack-mariadb:pcmklatest]
>   galera-bundle-0	(ocf::heartbeat:galera):	Master controller-0
>   galera-bundle-1	(ocf::heartbeat:galera):	Master controller-1
>   galera-bundle-2	(ocf::heartbeat:galera):	Master controller-2
> Docker container set: redis-bundle [192.168.24.1:8787/rhosp14/openstack-redis:pcmklatest]
>   redis-bundle-0	(ocf::heartbeat:redis):	Master controller-0
>   redis-bundle-1	(ocf::heartbeat:redis):	Slave controller-1
>   redis-bundle-2	(ocf::heartbeat:redis):	Slave controller-2
> ip-192.168.24.6	(ocf::heartbeat:IPaddr2):	Started controller-0
> ip-10.0.0.104	(ocf::heartbeat:IPaddr2):	Started controller-1
> ip-172.17.1.18	(ocf::heartbeat:IPaddr2):	Started controller-2
> ip-172.17.1.23	(ocf::heartbeat:IPaddr2):	Started controller-1
> ip-172.17.3.10	(ocf::heartbeat:IPaddr2):	Started controller-1
> ip-172.17.4.33	(ocf::heartbeat:IPaddr2):	Started controller-2
> Docker container set: haproxy-bundle [192.168.24.1:8787/rhosp14/openstack-haproxy:pcmklatest]
>   haproxy-bundle-docker-0	(ocf::heartbeat:docker):	Started controller-0
>   haproxy-bundle-docker-1	(ocf::heartbeat:docker):	Started controller-1
>   haproxy-bundle-docker-2	(ocf::heartbeat:docker):	Started controller-2
> Docker container set: ovn-dbs-bundle [192.168.24.1:8787/rhosp14/openstack-ovn-northd:pcmklatest]
>   ovn-dbs-bundle-0	(ocf::ovn:ovndb-servers):	Stopped controller-0
>   ovn-dbs-bundle-1	(ocf::ovn:ovndb-servers):	Master controller-1
>   ovn-dbs-bundle-2	(ocf::ovn:ovndb-servers):	Slave controller-2
> Docker container: openstack-cinder-volume [192.168.24.1:8787/rhosp14/openstack-cinder-volume:pcmklatest]
>   openstack-cinder-volume-docker-0	(ocf::heartbeat:docker):	Started controller-0
>
>Failed Actions:
>* ovn-dbs-bundle-0_monitor_60000 on controller-0 'unknown error' (1): call=20, status=Error, exitreason='',
>    last-rc-change='Sun Oct  7 07:08:38 2018', queued=0ms, exec=0ms
>* ovndb_servers_start_0 on ovn-dbs-bundle-0 'unknown error' (1): call=8, status=Timed Out, exitreason='',
>    last-rc-change='Sun Oct  7 07:08:53 2018', queued=0ms, exec=200002ms
>
>
>Daemon Status:
>  corosync: active/enabled
>  pacemaker: active/enabled
>  pcsd: active/enabled
>
>2018-10-07T03:14:01.968Z|00067|vlog|INFO|opened log file /var/log/openvswitch/ovn-controller.log
>2018-10-07T07:08:38.149Z|00068|reconnect|INFO|tcp:172.17.1.23:6642: connection closed by peer
>2018-10-07T07:08:39.150Z|00069|reconnect|INFO|tcp:172.17.1.23:6642: connecting...
>2018-10-07T07:08:39.157Z|00070|reconnect|INFO|tcp:172.17.1.23:6642: connection attempt failed (Connection refused)
>2018-10-07T07:08:39.157Z|00071|reconnect|INFO|tcp:172.17.1.23:6642: waiting 2 seconds before reconnect
>2018-10-07T07:08:41.159Z|00072|reconnect|INFO|tcp:172.17.1.23:6642: connecting...
>2018-10-07T07:08:41.166Z|00073|reconnect|INFO|tcp:172.17.1.23:6642: connection attempt failed (Connection refused)
>2018-10-07T07:08:41.166Z|00074|reconnect|INFO|tcp:172.17.1.23:6642: waiting 4 seconds before reconnect
>2018-10-07T07:08:45.169Z|00075|reconnect|INFO|tcp:172.17.1.23:6642: connecting...
>2018-10-07T07:08:45.181Z|00076|reconnect|INFO|tcp:172.17.1.23:6642: connection attempt failed (Connection refused)
>2018-10-07T07:08:45.181Z|00077|reconnect|INFO|tcp:172.17.1.23:6642: continuing to reconnect in the background but suppressing further logging
>2018-10-07T07:09:01.215Z|00078|reconnect|INFO|tcp:172.17.1.23:6642: connected
>2018-10-07T07:12:23.078Z|00079|reconnect|ERR|tcp:172.17.1.23:6642: no response to inactivity probe after 5 seconds, disconnecting
>2018-10-07T07:12:23.078Z|00080|reconnect|INFO|tcp:172.17.1.23:6642: connection dropped
>2018-10-07T07:12:24.079Z|00081|reconnect|INFO|tcp:172.17.1.23:6642: connecting...
>2018-10-07T07:12:24.087Z|00082|reconnect|INFO|tcp:172.17.1.23:6642: connected
>
>2018-10-07T07:08:55.021Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server-nb.log
>2018-10-07T07:08:55.030Z|00002|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.10.0
>2018-10-07T07:08:55.030Z|00003|reconnect|INFO|tcp:192.0.2.254:6641: connecting...
>2018-10-07T07:08:56.034Z|00004|reconnect|INFO|tcp:192.0.2.254:6641: connection attempt timed out
>2018-10-07T07:08:57.035Z|00005|reconnect|INFO|tcp:192.0.2.254:6641: connecting...
>2018-10-07T07:08:58.035Z|00006|reconnect|INFO|tcp:192.0.2.254:6641: connection attempt timed out
>2018-10-07T07:08:58.035Z|00007|reconnect|INFO|tcp:192.0.2.254:6641: waiting 2 seconds before reconnect
>2018-10-07T07:09:00.036Z|00008|reconnect|INFO|tcp:192.0.2.254:6641: connecting...
>2018-10-07T07:09:02.037Z|00009|reconnect|INFO|tcp:192.0.2.254:6641: connection attempt timed out
>2018-10-07T07:09:02.037Z|00010|reconnect|INFO|tcp:192.0.2.254:6641: waiting 4 seconds before reconnect
>2018-10-07T07:09:05.037Z|00011|memory|INFO|4412 kB peak resident set size after 10.0 seconds
>2018-10-07T07:09:05.037Z|00012|memory|INFO|cells:490 monitors:2 sessions:8
>2018-10-07T07:09:06.037Z|00013|reconnect|INFO|tcp:192.0.2.254:6641: connecting...
>2018-10-07T07:09:10.040Z|00014|reconnect|INFO|tcp:192.0.2.254:6641: connection attempt timed out
>2018-10-07T07:09:10.040Z|00015|reconnect|INFO|tcp:192.0.2.254:6641: continuing to reconnect in the background but suppressing further logging
>2018-10-07T07:18:02.545Z|00016|reconnect|ERR|tcp:172.17.1.23:44120: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:02.545Z|00017|reconnect|ERR|tcp:172.17.1.23:44122: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:02.547Z|00018|reconnect|ERR|tcp:172.17.1.21:49104: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:02.547Z|00019|reconnect|ERR|tcp:172.17.1.21:49110: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:02.551Z|00020|reconnect|ERR|tcp:172.17.1.11:53124: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:02.551Z|00021|reconnect|ERR|tcp:172.17.1.11:53126: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:12.664Z|00022|reconnect|ERR|tcp:172.17.1.21:49102: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:12.688Z|00023|reconnect|ERR|tcp:172.17.1.11:53118: no response to inactivity probe after 180 seconds, disconnecting
>
>2018-10-07T07:08:55.090Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server-sb.log
>2018-10-07T07:08:55.101Z|00002|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.10.0
>2018-10-07T07:08:55.101Z|00003|reconnect|INFO|tcp:192.0.2.254:6642: connecting...
>2018-10-07T07:08:56.102Z|00004|reconnect|INFO|tcp:192.0.2.254:6642: connection attempt timed out
>2018-10-07T07:08:57.102Z|00005|reconnect|INFO|tcp:192.0.2.254:6642: connecting...
>2018-10-07T07:08:58.105Z|00006|reconnect|INFO|tcp:192.0.2.254:6642: connection attempt timed out
>2018-10-07T07:08:58.105Z|00007|reconnect|INFO|tcp:192.0.2.254:6642: waiting 2 seconds before reconnect
>2018-10-07T07:09:00.106Z|00008|reconnect|INFO|tcp:192.0.2.254:6642: connecting...
>2018-10-07T07:09:02.117Z|00009|reconnect|INFO|tcp:192.0.2.254:6642: connection attempt timed out
>2018-10-07T07:09:02.117Z|00010|reconnect|INFO|tcp:192.0.2.254:6642: waiting 4 seconds before reconnect
>2018-10-07T07:09:05.109Z|00011|memory|INFO|5444 kB peak resident set size after 10.0 seconds
>2018-10-07T07:09:05.109Z|00012|memory|INFO|cells:2483 monitors:4 sessions:35
>2018-10-07T07:09:06.118Z|00013|reconnect|INFO|tcp:192.0.2.254:6642: connecting...
>2018-10-07T07:09:10.122Z|00014|reconnect|INFO|tcp:192.0.2.254:6642: connection attempt timed out
>2018-10-07T07:09:10.123Z|00015|reconnect|INFO|tcp:192.0.2.254:6642: continuing to reconnect in the background but suppressing further logging
>2018-10-07T07:18:01.159Z|00016|reconnect|ERR|tcp:172.17.1.10:47164: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.168Z|00017|reconnect|ERR|tcp:172.17.1.20:54240: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.234Z|00018|reconnect|ERR|tcp:172.17.1.11:59898: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.238Z|00019|reconnect|ERR|tcp:172.17.1.21:47442: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.244Z|00020|reconnect|ERR|tcp:172.17.1.11:59896: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.250Z|00021|reconnect|ERR|tcp:172.17.1.23:44498: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.253Z|00022|reconnect|ERR|tcp:172.17.1.10:47182: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.253Z|00023|reconnect|ERR|tcp:172.17.1.10:47184: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.253Z|00024|reconnect|ERR|tcp:172.17.1.10:47190: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.255Z|00025|reconnect|ERR|tcp:172.17.1.10:47168: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.255Z|00026|reconnect|ERR|tcp:172.17.1.10:47170: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.260Z|00027|reconnect|ERR|tcp:172.17.1.21:47444: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.260Z|00028|reconnect|ERR|tcp:172.17.1.23:44500: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.265Z|00029|reconnect|ERR|tcp:172.17.1.10:47172: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.265Z|00030|reconnect|ERR|tcp:172.17.1.10:47174: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.265Z|00031|reconnect|ERR|tcp:172.17.1.10:47176: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.289Z|00032|reconnect|ERR|tcp:172.17.1.20:54242: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.289Z|00033|reconnect|ERR|tcp:172.17.1.20:54244: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.289Z|00034|reconnect|ERR|tcp:172.17.1.20:54246: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.289Z|00035|reconnect|ERR|tcp:172.17.1.20:54248: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.331Z|00036|reconnect|ERR|tcp:172.17.1.20:54250: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.331Z|00037|reconnect|ERR|tcp:172.17.1.20:54252: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.387Z|00038|reconnect|ERR|tcp:172.17.1.10:47178: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:01.388Z|00039|reconnect|ERR|tcp:172.17.1.10:47186: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:02.189Z|00040|reconnect|ERR|tcp:172.17.1.10:47180: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:02.189Z|00041|reconnect|ERR|tcp:172.17.1.10:47188: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:02.256Z|00042|reconnect|ERR|tcp:172.17.1.20:54256: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:02.257Z|00043|reconnect|ERR|tcp:172.17.1.20:54254: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:12.676Z|00044|reconnect|ERR|tcp:172.17.1.21:47436: no response to inactivity probe after 180 seconds, disconnecting
>2018-10-07T07:18:12.692Z|00045|reconnect|ERR|tcp:172.17.1.11:59888: no response to inactivity probe after 180 seconds, disconnecting
>
>~

Actions: View

Attachments on bug 1636714: 1491282