Bug 1677319 (Gluster_5_Affecting_oVirt_4.3) - [Tracker] Gluster 5/6 issues affecting oVirt 4.3
Summary: [Tracker] Gluster 5/6 issues affecting oVirt 4.3
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: Gluster_5_Affecting_oVirt_4.3
Product: ovirt-distribution
Classification: oVirt
Component: General
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
high
high with 2 votes
Target Milestone: ---
: ---
Assignee: Sahina Bose
QA Contact: bipin
URL:
Whiteboard:
Depends On: 1649054 1651246 1671556 1672711 1673058 1674406 1677160 1678570 1679406 1679965 1679968 1682925 1683008 1683900 1684385
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-14 14:23 UTC by Sandro Bonazzola
Modified: 2019-11-12 07:38 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-12 07:38:38 UTC
oVirt Team: Gluster
Embargoed:


Attachments (Terms of Use)

Description Sandro Bonazzola 2019-02-14 14:23:34 UTC
Tracking Gluster 5 bugs reported affecting oVirt 4.3

Comment 1 James 2019-02-14 14:48:48 UTC
I'm also having problems with Gluster bricks going offline since upgrading to oVirt 4.3 yesterday (previously I've never had a single issue with gluster nor have had a brick ever go down).  I suspect this will continue to happen daily as some other users on this group have suggested.  I was able to pull some logs from engine and gluster from around the time the brick dropped.  My setup is 3 node HCI and I was previously running the latest 4.2 updates (before upgrading to 4.3).  My hardware is has a lot of overhead and I'm on 10Gbe gluster backend (the servers were certainly not under any significant amount of load when the brick went offline).  To recover I had to place the host in maintenance mode and reboot (although I suspect I could have simply unmounted and remounted gluster mounts). 

grep "2019-02-14" engine.log-20190214 | grep "GLUSTER_BRICK_STATUS_CHANGED"
2019-02-14 02:41:48,018-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ff5b093] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of volume non_prod_b of cluster Default from UP to DOWN via cli.
2019-02-14 03:20:11,189-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [760f7851] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/engine/engine of volume engine of cluster Default from DOWN to UP via cli.
2019-02-14 03:20:14,819-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [760f7851] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/prod_b/prod_b of volume prod_b of cluster Default from DOWN to UP via cli.
2019-02-14 03:20:19,692-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [760f7851] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/isos/isos of volume isos of cluster Default from DOWN to UP via cli.
2019-02-14 03:20:25,022-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [760f7851] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/prod_a/prod_a of volume prod_a of cluster Default from DOWN to UP via cli.
2019-02-14 03:20:29,088-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [760f7851] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of volume non_prod_b of cluster Default from DOWN to UP via cli.
2019-02-14 03:20:34,099-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [760f7851] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/non_prod_a/non_prod_a of volume non_prod_a of cluster Default from DOWN to UP via cli

glusterd.log

# grep -B20 -A20 "2019-02-14 02:41" glusterd.log
[2019-02-14 02:36:49.585034] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_b
[2019-02-14 02:36:49.597788] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 2 times between [2019-02-14 02:36:49.597788] and [2019-02-14 02:36:49.900505]
[2019-02-14 02:36:53.437539] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_a
[2019-02-14 02:36:53.452816] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
[2019-02-14 02:36:53.864153] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_a
[2019-02-14 02:36:53.875835] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
[2019-02-14 02:36:30.958649] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume engine
[2019-02-14 02:36:35.322129] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_b
[2019-02-14 02:36:39.639645] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume isos
[2019-02-14 02:36:45.301275] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_a
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 2 times between [2019-02-14 02:36:53.875835] and [2019-02-14 02:36:54.180780]
[2019-02-14 02:37:59.193409] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
[2019-02-14 02:38:44.065560] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume engine
[2019-02-14 02:38:44.072680] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume isos
[2019-02-14 02:38:44.077841] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_a
[2019-02-14 02:38:44.082798] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_b
[2019-02-14 02:38:44.088237] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_a
[2019-02-14 02:38:44.093518] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_b
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 2 times between [2019-02-14 02:37:59.193409] and [2019-02-14 02:38:44.100494]
[2019-02-14 02:41:58.649683] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 6 times between [2019-02-14 02:41:58.649683] and [2019-02-14 02:43:00.286999]
[2019-02-14 02:43:46.366743] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume engine
[2019-02-14 02:43:46.373587] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume isos
[2019-02-14 02:43:46.378997] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_a
[2019-02-14 02:43:46.384324] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_b
[2019-02-14 02:43:46.390310] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_a
[2019-02-14 02:43:46.397031] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_b
[2019-02-14 02:43:46.404083] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
[2019-02-14 02:45:47.302884] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume engine
[2019-02-14 02:45:47.309697] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume isos
[2019-02-14 02:45:47.315149] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_a
[2019-02-14 02:45:47.320806] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_b
[2019-02-14 02:45:47.326865] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_a
[2019-02-14 02:45:47.332192] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_b
[2019-02-14 02:45:47.338991] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
[2019-02-14 02:46:47.789575] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_b
[2019-02-14 02:46:47.795276] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_a
[2019-02-14 02:46:47.800584] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_b
[2019-02-14 02:46:47.770601] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume engine
[2019-02-14 02:46:47.778161] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume isos
[2019-02-14 02:46:47.784020] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_a

engine.log

# grep -B20 -A20 "2019-02-14 02:41:48" engine.log-20190214
2019-02-14 02:41:43,495-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), log id: 172c9ee8
2019-02-14 02:41:43,609-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@479fcb69, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6443e68f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2b4cf035, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5864f06a, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6119ac8c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1a9549be, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5614cf81, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@290c9289, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5dd26e8, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@35355754, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@452deeb4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8f8b442, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@647e29d3, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7bee4dff, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@511c4478, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c0bb0bd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@92e325e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@260731, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@33aaacc9, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@72657c59, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@aa10c89], log id: 172c9ee8
2019-02-14 02:41:43,610-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), log id: 3a0e9d63
2019-02-14 02:41:43,703-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@5ca4a20f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@57a8a76, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@7bd1b14], log id: 3a0e9d63
2019-02-14 02:41:43,704-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVDOVolumeListVDSCommand(HostName = Host1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), log id: 49966b05
2019-02-14 02:41:44,213-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 49966b05
2019-02-14 02:41:44,214-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 30db0ce2
2019-02-14 02:41:44,311-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@61a309b5, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@ea9cb2e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@749d57bd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c49f9d0, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@655eb54d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@256ee273, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3bd079dc, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6804900f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@78e0a49f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2acfbc8a, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12e92e96, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5ea1502c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2398c33b, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7464102e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2f221daa, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7b561852, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1eb29d18, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4a030b80, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@75739027, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3eac8253, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@34fc82c3], log id: 30db0ce2
2019-02-14 02:41:44,312-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 6671d0d7
2019-02-14 02:41:44,329-04 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}'
2019-02-14 02:41:44,345-04 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}'
2019-02-14 02:41:44,374-04 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}'
2019-02-14 02:41:44,405-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@f6a9696, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@558e3332, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@5b449da], log id: 6671d0d7
2019-02-14 02:41:44,406-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVDOVolumeListVDSCommand(HostName = Host2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 6d2bc6d3
2019-02-14 02:41:44,908-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 6d2bc6d3
2019-02-14 02:41:44,909-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Host0, GlusterVolumeAdvancedDetailsVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5', volumeName='non_prod_b'}), log id: 36ae23c6
2019-02-14 02:41:47,336-04 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}'
2019-02-14 02:41:47,351-04 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}'
2019-02-14 02:41:47,379-04 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}'
2019-02-14 02:41:47,979-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterVolumeAdvancedDetailsVDSCommand, return: org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails@7a4a787b, log id: 36ae23c6
2019-02-14 02:41:48,018-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ff5b093] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of volume non_prod_b of cluster Default from UP to DOWN via cli.
2019-02-14 02:41:48,046-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ff5b093] EVENT_ID: GLUSTER_BRICK_STATUS_DOWN(4,151), Status of brick host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of volume non_prod_b on cluster Default is down.
2019-02-14 02:41:48,139-04 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler1) [5ff5b093] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}'
2019-02-14 02:41:48,140-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler3) [7b9bd2d] START, GlusterServersListVDSCommand(HostName = Host0, VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: e1fb23
2019-02-14 02:41:48,911-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler3) [7b9bd2d] FINISH, GlusterServersListVDSCommand, return: [10.12.0.220/24:CONNECTED, host1.replaced.domain.com:CONNECTED, host2.replaced.domain.com:CONNECTED], log id: e1fb23
2019-02-14 02:41:48,930-04 INFO  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler1) [5ff5b093] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}'
2019-02-14 02:41:48,931-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler3) [7b9bd2d] START, GlusterVolumesListVDSCommand(HostName = Host0, GlusterVolumesListVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 68f1aecc
2019-02-14 02:41:49,366-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler3) [7b9bd2d] FINISH, GlusterVolumesListVDSCommand, return: {6c05dfc6-4dc0-41e3-a12f-55b4767f1d35=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@1952a85, 3f8f6a0f-aed4-48e3-9129-18a2a3f64eef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@2f6688ae, 71ff56d9-79b8-445d-b637-72ffc974f109=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@730210fb, 752a9438-cd11-426c-b384-bc3c5f86ed07=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@c3be510c, c3e7447e-8514-4e4a-9ff5-a648fe6aa537=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@450befac, 79e8e93c-57c8-4541-a360-726cec3790cf=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@1926e392}, log id: 68f1aecc
2019-02-14 02:41:49,489-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host0, VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 38debe74
2019-02-14 02:41:49,581-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5e5a7925, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2cdf5c9e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@443cb62, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@49a3e880, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@443d23c0, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1250bc75, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8d27d86, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5e6363f4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@73ed78db, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@64c9d1c7, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7fecbe95, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3a551e5f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2266926e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@88b380c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1209279e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3c6466, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@16df63ed, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@47456262, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c2b88c3, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7f57c074, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12fa0478], log id: 38debe74
2019-02-14 02:41:49,582-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host0, VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 7ec02237
2019-02-14 02:41:49,660-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@3eedd0bc, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@7f78e375, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@3d63e126], log id: 7ec02237
2019-02-14 02:41:49,661-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVDOVolumeListVDSCommand(HostName = Host0, VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 42cdad27
2019-02-14 02:41:50,142-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 42cdad27
2019-02-14 02:41:50,143-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), log id: 12f5fdf2
2019-02-14 02:41:50,248-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2aaed792, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8e66930, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@276d599e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1aca2aec, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@46846c60, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7d103269, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@30fc25fc, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7baae445, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1ea8603c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@62578afa, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@33d58089, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1f71d27a, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4205e828, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c5bbac8, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@395a002, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12664008, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7f4faec4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3e03d61f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1038e46d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@307e8062, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@32453127], log id: 12f5fdf2
2019-02-14 02:41:50,249-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), log id: 1256aa5e
2019-02-14 02:41:50,338-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@459a2ff5, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@123cab4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@1af41fbe], log id: 1256aa5e
2019-02-14 02:41:50,339-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVDOVolumeListVDSCommand(HostName = Host1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), log id: 3dd752e4
2019-02-14 02:41:50,847-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 3dd752e4
2019-02-14 02:41:50,848-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 29a6272c
2019-02-14 02:41:50,954-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@364f3ec6, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@c7cce5e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@b3bed47, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@13bc244b, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5cca81f4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@36aeba0d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@62ab384a, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1047d628, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@188a30f5, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5bb79f3b, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@60e5956f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4e3df9cd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7796567, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@60d06cf4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2cd2d36c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@d80a4aa, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@411eaa20, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@22cac93b, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@18b927bd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@101465f4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@246f927c], log id: 29a6272c
2019-02-14 02:41:50,955-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 501814db
2019-02-14 02:41:51,044-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@1cd55aa, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@32c5aba2, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@6ae123f4], log id: 501814db
2019-02-14 02:41:51,045-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVDOVolumeListVDSCommand(HostName = Host2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 7acf4cbf
2019-02-14 02:41:51,546-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 7acf4cbf
2019-02-14 02:41:51,547-04 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Host0, GlusterVolumeAdvancedDetailsVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5', volumeName='non_prod_a'}), log id: 11c42649

Comment 2 James 2019-02-14 16:17:15 UTC
Brick log from around the time the brick went down:


[2019-02-14 02:20:29.611099] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:20:29.611131] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:ee716e24-e187-4b57-a371-cab544f41162-GRAPH_ID:0-PID:30671-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3)
[2019-02-14 02:20:29.619521] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:ee716e24-e187-4b57-a371-cab544f41162-GRAPH_ID:0-PID:30671-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:20:29.619867] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:ee716e24-e187-4b57-a371-cab544f41162-GRAPH_ID:0-PID:30671-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 19988 times between [2019-02-14 02:19:31.377315] and [2019-02-14 02:21:14.033991]
[2019-02-14 02:21:30.303440] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 10 times between [2019-02-14 02:21:30.303440] and [2019-02-14 02:23:20.421140]
[2019-02-14 02:23:33.142281] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 34 times between [2019-02-14 02:23:33.142281] and [2019-02-14 02:25:29.115156]
[2019-02-14 02:25:30.326469] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
[2019-02-14 02:25:53.973830] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.222"
[2019-02-14 02:25:53.973896] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:25:53.973928] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:4a6b8860-8274-4b3b-b400-d66cbfb97349-GRAPH_ID:0-PID:33522-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3)
[2019-02-14 02:25:54.627728] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:4a6b8860-8274-4b3b-b400-d66cbfb97349-GRAPH_ID:0-PID:33522-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:25:54.628149] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:4a6b8860-8274-4b3b-b400-d66cbfb97349-GRAPH_ID:0-PID:33522-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:25:56.396855] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.220"
[2019-02-14 02:25:56.396926] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:25:56.396957] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:963c2196-108c-485d-aca6-a236906d2acf-GRAPH_ID:0-PID:33635-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3)
[2019-02-14 02:25:56.404566] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:963c2196-108c-485d-aca6-a236906d2acf-GRAPH_ID:0-PID:33635-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:25:56.404866] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:963c2196-108c-485d-aca6-a236906d2acf-GRAPH_ID:0-PID:33635-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 25 times between [2019-02-14 02:25:30.326469] and [2019-02-14 02:27:25.965601]
[2019-02-14 02:28:10.538374] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 38 times between [2019-02-14 02:28:10.538374] and [2019-02-14 02:29:22.622679]
[2019-02-14 02:29:48.891040] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
[2019-02-14 02:29:56.026002] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
[2019-02-14 02:31:22.494824] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.221"
[2019-02-14 02:31:22.494914] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:31:22.494946] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:210b837a-1c98-4d88-aab8-5ce895398d81-GRAPH_ID:0-PID:18765-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3)
[2019-02-14 02:31:22.503920] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:210b837a-1c98-4d88-aab8-5ce895398d81-GRAPH_ID:0-PID:18765-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:31:22.504205] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:210b837a-1c98-4d88-aab8-5ce895398d81-GRAPH_ID:0-PID:18765-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 15301 times between [2019-02-14 02:29:48.891040] and [2019-02-14 02:31:28.963054]
[2019-02-14 02:31:31.016063] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 628 times between [2019-02-14 02:31:31.016063] and [2019-02-14 02:33:27.936726]
[2019-02-14 02:33:35.878655] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 1776 times between [2019-02-14 02:33:35.878655] and [2019-02-14 02:35:28.143659]
[2019-02-14 02:35:29.607848] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
[2019-02-14 02:36:00.483996] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.220"
[2019-02-14 02:36:00.484066] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:36:00.484098] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:9f2cd861-23c9-412b-be2c-8c45658bb155-GRAPH_ID:0-PID:39820-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3)
[2019-02-14 02:36:00.779203] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:9f2cd861-23c9-412b-be2c-8c45658bb155-GRAPH_ID:0-PID:39820-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:36:00.779662] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:9f2cd861-23c9-412b-be2c-8c45658bb155-GRAPH_ID:0-PID:39820-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:36:49.892372] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.222"
[2019-02-14 02:36:49.892451] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:36:49.892481] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:91cc19de-30ae-4639-a407-c6fce4682fa4-GRAPH_ID:0-PID:39353-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3)
[2019-02-14 02:36:49.900530] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:91cc19de-30ae-4639-a407-c6fce4682fa4-GRAPH_ID:0-PID:39353-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:36:49.900863] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:91cc19de-30ae-4639-a407-c6fce4682fa4-GRAPH_ID:0-PID:39353-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 15772 times between [2019-02-14 02:35:29.607848] and [2019-02-14 02:37:24.862502]
[2019-02-14 02:38:11.704945] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 23 times between [2019-02-14 02:38:11.704945] and [2019-02-14 02:39:24.469485]
[2019-02-14 02:39:47.550186] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 34 times between [2019-02-14 02:39:47.550186] and [2019-02-14 02:41:28.642587]
[2019-02-14 02:41:29.746302] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
[2019-02-14 02:42:15.811205] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.220"
[2019-02-14 02:42:15.811284] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:42:15.811317] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:6de46e97-ef84-43c8-a472-4e942471a8f6-GRAPH_ID:0-PID:43752-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3)
[2019-02-14 02:42:15.819841] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:6de46e97-ef84-43c8-a472-4e942471a8f6-GRAPH_ID:0-PID:43752-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:42:15.820156] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:6de46e97-ef84-43c8-a472-4e942471a8f6-GRAPH_ID:0-PID:43752-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 5147 times between [2019-02-14 02:41:29.746302] and [2019-02-14 02:43:28.777825]
[2019-02-14 02:43:35.252750] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 17909 times between [2019-02-14 02:43:35.252750] and [2019-02-14 02:45:29.331082]
[2019-02-14 02:45:29.497786] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
[2019-02-14 02:46:08.243167] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.222"
[2019-02-14 02:46:08.243230] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:46:08.243259] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:56fb6f47-3e05-4f07-abdf-cb49213cf5c0-GRAPH_ID:0-PID:44348-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3)
[2019-02-14 02:46:09.512844] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:56fb6f47-3e05-4f07-abdf-cb49213cf5c0-GRAPH_ID:0-PID:44348-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:46:09.513302] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:56fb6f47-3e05-4f07-abdf-cb49213cf5c0-GRAPH_ID:0-PID:44348-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 2892 times between [2019-02-14 02:45:29.497786] and [2019-02-14 02:47:23.957871]
[2019-02-14 02:47:30.087716] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
[2019-02-14 02:47:42.722682] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.221"
[2019-02-14 02:47:42.722779] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:47:42.722810] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:34bb8b68-3483-4744-bd33-7a47cd4214b2-GRAPH_ID:0-PID:27248-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3)
[2019-02-14 02:47:42.731133] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:34bb8b68-3483-4744-bd33-7a47cd4214b2-GRAPH_ID:0-PID:27248-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:47:42.731447] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:34bb8b68-3483-4744-bd33-7a47cd4214b2-GRAPH_ID:0-PID:27248-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 564 times between [2019-02-14 02:47:30.087716] and [2019-02-14 02:49:28.480491]
[2019-02-14 02:49:32.129061] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 15058 times between [2019-02-14 02:49:32.129061] and [2019-02-14 02:51:26.667738]
[2019-02-14 02:52:35.362594] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
[2019-02-14 02:53:08.127443] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.222"
[2019-02-14 02:53:08.127528] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:53:08.127562] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:3fbc7931-7ac7-4dfd-9169-6271a18fc0f9-GRAPH_ID:0-PID:48068-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3)
[2019-02-14 02:53:08.135384] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:3fbc7931-7ac7-4dfd-9169-6271a18fc0f9-GRAPH_ID:0-PID:48068-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:53:08.135682] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:3fbc7931-7ac7-4dfd-9169-6271a18fc0f9-GRAPH_ID:0-PID:48068-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 11 times between [2019-02-14 02:52:35.362594] and [2019-02-14 02:53:24.927967]
[2019-02-14 02:53:56.793284] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 38 times between [2019-02-14 02:53:56.793284] and [2019-02-14 02:55:29.101365]
[2019-02-14 02:55:30.445077] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
[2019-02-14 02:56:14.259883] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.222"
[2019-02-14 02:56:14.259948] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:56:14.259980] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:d7f3996c-4c68-4e15-b0e2-0197bc358ddd-GRAPH_ID:0-PID:885-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3)
[2019-02-14 02:56:14.336000] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:d7f3996c-4c68-4e15-b0e2-0197bc358ddd-GRAPH_ID:0-PID:885-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:56:14.336327] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:d7f3996c-4c68-4e15-b0e2-0197bc358ddd-GRAPH_ID:0-PID:885-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 19 times between [2019-02-14 02:55:30.445077] and [2019-02-14 02:57:28.589205]
[2019-02-14 02:57:35.071899] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
[2019-02-14 02:58:33.855724] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.220"
[2019-02-14 02:58:33.855805] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:58:33.855838] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:868382a8-7e38-481c-b5a3-80c1bdd21f28-GRAPH_ID:0-PID:4631-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3)
[2019-02-14 02:58:33.864814] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:868382a8-7e38-481c-b5a3-80c1bdd21f28-GRAPH_ID:0-PID:4631-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:58:33.865124] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:868382a8-7e38-481c-b5a3-80c1bdd21f28-GRAPH_ID:0-PID:4631-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 35 times between [2019-02-14 02:57:35.071899] and [2019-02-14 02:59:29.398638]
[2019-02-14 02:59:33.501166] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
[2019-02-14 03:00:01.622537] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 21104 times between [2019-02-14 02:59:33.501166] and [2019-02-14 03:01:29.325701]
[2019-02-14 03:01:30.635130] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 640 times between [2019-02-14 03:01:30.635130] and [2019-02-14 03:03:29.020755]
[2019-02-14 03:03:31.238543] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler
[2019-02-14 03:04:00.722382] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.221"
[2019-02-14 03:04:00.722466] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 03:04:00.722499] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:ca4e5703-7129-4242-9fbf-9b119de9e0eb-GRAPH_ID:0-PID:35692-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3)

Comment 3 Netbulae 2019-02-18 11:43:20 UTC
Can we add https://bugzilla.redhat.com/show_bug.cgi?id=1306562 to this list? 

We would like to start using libgfapi but without live migration this doesn't work for us.

Comment 4 Netbulae 2019-02-18 12:07:00 UTC
(In reply to Netbulae from comment #3)
> Can we add https://bugzilla.redhat.com/show_bug.cgi?id=1306562 to this list? 
> 
> We would like to start using libgfapi but without live migration this
> doesn't work for us.

Ah sorry, this is a libvirt issue not GlusterFS. Please ignore

Comment 5 Sandro Bonazzola 2019-07-23 06:53:29 UTC
Moving to 4.3.6, not completed in 4.3.5.

Comment 6 Timo Eissler 2019-09-26 12:17:58 UTC
When will this issue be fixed?
The target milestone was 4.3.2 initially and now it was moved to 4.3.7!

We can't upgrade our oVirt infrastructure to 4.3 since this problem is a show stopper for us.
We just can't manually bring up the bricks when they go down as we don't have that many resources.

Is there a specific configuration which causes this or are there any other workarounds than restart the corresponding gluster service?

Kind regards,
Timo

Comment 7 Sahina Bose 2019-09-26 12:32:36 UTC
(In reply to Timo Eissler from comment #6)
> When will this issue be fixed?
> The target milestone was 4.3.2 initially and now it was moved to 4.3.7!
> 
> We can't upgrade our oVirt infrastructure to 4.3 since this problem is a
> show stopper for us.
> We just can't manually bring up the bricks when they go down as we don't
> have that many resources.
> 
> Is there a specific configuration which causes this or are there any other
> workarounds than restart the corresponding gluster service?
> 
> Kind regards,
> Timo

This should be working with existing release 4.3.6
All tracking issues in this bug have been closed except for bug 1672318, which seems to be hitting 1 particular user. We will check on that and update the state of this bug.
Setting a NI on me to follow up

Comment 8 Sahina Bose 2019-10-11 10:23:36 UTC
Moving this to ON_QA as bug 1672318 seems like a setup issue specific to user

Comment 9 SATHEESARAN 2019-11-12 07:36:26 UTC
(In reply to Sahina Bose from comment #8)
> Moving this to ON_QA as bug 1672318 seems like a setup issue specific to user

Based on this comment, removing this bug 1672318 from the tracker

Comment 10 SATHEESARAN 2019-11-12 07:38:38 UTC
This bug is the TRACKER, that tracks the individual bugs that are targetted specifically for the release.
It doesn't makes sense to move the TRACKER to ON_QA. CLOSING this tracker bug as all the bugs tracked are already CLOSED.


Note You need to log in before you can comment on or make changes to this bug.