Tracking Gluster 5 bugs reported affecting oVirt 4.3
I'm also having problems with Gluster bricks going offline since upgrading to oVirt 4.3 yesterday (previously I've never had a single issue with gluster nor have had a brick ever go down). I suspect this will continue to happen daily as some other users on this group have suggested. I was able to pull some logs from engine and gluster from around the time the brick dropped. My setup is 3 node HCI and I was previously running the latest 4.2 updates (before upgrading to 4.3). My hardware is has a lot of overhead and I'm on 10Gbe gluster backend (the servers were certainly not under any significant amount of load when the brick went offline). To recover I had to place the host in maintenance mode and reboot (although I suspect I could have simply unmounted and remounted gluster mounts). grep "2019-02-14" engine.log-20190214 | grep "GLUSTER_BRICK_STATUS_CHANGED" 2019-02-14 02:41:48,018-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ff5b093] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of volume non_prod_b of cluster Default from UP to DOWN via cli. 2019-02-14 03:20:11,189-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [760f7851] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/engine/engine of volume engine of cluster Default from DOWN to UP via cli. 2019-02-14 03:20:14,819-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [760f7851] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/prod_b/prod_b of volume prod_b of cluster Default from DOWN to UP via cli. 2019-02-14 03:20:19,692-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [760f7851] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/isos/isos of volume isos of cluster Default from DOWN to UP via cli. 2019-02-14 03:20:25,022-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [760f7851] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/prod_a/prod_a of volume prod_a of cluster Default from DOWN to UP via cli. 2019-02-14 03:20:29,088-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [760f7851] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of volume non_prod_b of cluster Default from DOWN to UP via cli. 2019-02-14 03:20:34,099-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [760f7851] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/non_prod_a/non_prod_a of volume non_prod_a of cluster Default from DOWN to UP via cli glusterd.log # grep -B20 -A20 "2019-02-14 02:41" glusterd.log [2019-02-14 02:36:49.585034] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_b [2019-02-14 02:36:49.597788] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 2 times between [2019-02-14 02:36:49.597788] and [2019-02-14 02:36:49.900505] [2019-02-14 02:36:53.437539] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_a [2019-02-14 02:36:53.452816] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 02:36:53.864153] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_a [2019-02-14 02:36:53.875835] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 02:36:30.958649] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume engine [2019-02-14 02:36:35.322129] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_b [2019-02-14 02:36:39.639645] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume isos [2019-02-14 02:36:45.301275] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_a The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 2 times between [2019-02-14 02:36:53.875835] and [2019-02-14 02:36:54.180780] [2019-02-14 02:37:59.193409] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 02:38:44.065560] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume engine [2019-02-14 02:38:44.072680] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume isos [2019-02-14 02:38:44.077841] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_a [2019-02-14 02:38:44.082798] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_b [2019-02-14 02:38:44.088237] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_a [2019-02-14 02:38:44.093518] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_b The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 2 times between [2019-02-14 02:37:59.193409] and [2019-02-14 02:38:44.100494] [2019-02-14 02:41:58.649683] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 6 times between [2019-02-14 02:41:58.649683] and [2019-02-14 02:43:00.286999] [2019-02-14 02:43:46.366743] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume engine [2019-02-14 02:43:46.373587] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume isos [2019-02-14 02:43:46.378997] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_a [2019-02-14 02:43:46.384324] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_b [2019-02-14 02:43:46.390310] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_a [2019-02-14 02:43:46.397031] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_b [2019-02-14 02:43:46.404083] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 02:45:47.302884] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume engine [2019-02-14 02:45:47.309697] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume isos [2019-02-14 02:45:47.315149] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_a [2019-02-14 02:45:47.320806] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_b [2019-02-14 02:45:47.326865] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_a [2019-02-14 02:45:47.332192] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_b [2019-02-14 02:45:47.338991] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 02:46:47.789575] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_b [2019-02-14 02:46:47.795276] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_a [2019-02-14 02:46:47.800584] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume prod_b [2019-02-14 02:46:47.770601] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume engine [2019-02-14 02:46:47.778161] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume isos [2019-02-14 02:46:47.784020] I [MSGID: 106499] [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management: Received status volume req for volume non_prod_a engine.log # grep -B20 -A20 "2019-02-14 02:41:48" engine.log-20190214 2019-02-14 02:41:43,495-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), log id: 172c9ee8 2019-02-14 02:41:43,609-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@479fcb69, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6443e68f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2b4cf035, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5864f06a, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6119ac8c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1a9549be, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5614cf81, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@290c9289, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5dd26e8, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@35355754, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@452deeb4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8f8b442, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@647e29d3, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7bee4dff, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@511c4478, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c0bb0bd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@92e325e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@260731, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@33aaacc9, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@72657c59, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@aa10c89], log id: 172c9ee8 2019-02-14 02:41:43,610-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), log id: 3a0e9d63 2019-02-14 02:41:43,703-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@5ca4a20f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@57a8a76, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@7bd1b14], log id: 3a0e9d63 2019-02-14 02:41:43,704-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVDOVolumeListVDSCommand(HostName = Host1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), log id: 49966b05 2019-02-14 02:41:44,213-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 49966b05 2019-02-14 02:41:44,214-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 30db0ce2 2019-02-14 02:41:44,311-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@61a309b5, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@ea9cb2e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@749d57bd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c49f9d0, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@655eb54d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@256ee273, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3bd079dc, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6804900f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@78e0a49f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2acfbc8a, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12e92e96, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5ea1502c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2398c33b, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7464102e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2f221daa, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7b561852, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1eb29d18, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4a030b80, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@75739027, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3eac8253, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@34fc82c3], log id: 30db0ce2 2019-02-14 02:41:44,312-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 6671d0d7 2019-02-14 02:41:44,329-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}' 2019-02-14 02:41:44,345-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}' 2019-02-14 02:41:44,374-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}' 2019-02-14 02:41:44,405-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@f6a9696, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@558e3332, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@5b449da], log id: 6671d0d7 2019-02-14 02:41:44,406-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVDOVolumeListVDSCommand(HostName = Host2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 6d2bc6d3 2019-02-14 02:41:44,908-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 6d2bc6d3 2019-02-14 02:41:44,909-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Host0, GlusterVolumeAdvancedDetailsVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5', volumeName='non_prod_b'}), log id: 36ae23c6 2019-02-14 02:41:47,336-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}' 2019-02-14 02:41:47,351-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}' 2019-02-14 02:41:47,379-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}' 2019-02-14 02:41:47,979-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterVolumeAdvancedDetailsVDSCommand, return: org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails@7a4a787b, log id: 36ae23c6 2019-02-14 02:41:48,018-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ff5b093] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of volume non_prod_b of cluster Default from UP to DOWN via cli. 2019-02-14 02:41:48,046-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ff5b093] EVENT_ID: GLUSTER_BRICK_STATUS_DOWN(4,151), Status of brick host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of volume non_prod_b on cluster Default is down. 2019-02-14 02:41:48,139-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler1) [5ff5b093] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}' 2019-02-14 02:41:48,140-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler3) [7b9bd2d] START, GlusterServersListVDSCommand(HostName = Host0, VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: e1fb23 2019-02-14 02:41:48,911-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler3) [7b9bd2d] FINISH, GlusterServersListVDSCommand, return: [10.12.0.220/24:CONNECTED, host1.replaced.domain.com:CONNECTED, host2.replaced.domain.com:CONNECTED], log id: e1fb23 2019-02-14 02:41:48,930-04 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler1) [5ff5b093] Failed to acquire lock and wait lock 'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]', sharedLocks=''}' 2019-02-14 02:41:48,931-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler3) [7b9bd2d] START, GlusterVolumesListVDSCommand(HostName = Host0, GlusterVolumesListVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 68f1aecc 2019-02-14 02:41:49,366-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler3) [7b9bd2d] FINISH, GlusterVolumesListVDSCommand, return: {6c05dfc6-4dc0-41e3-a12f-55b4767f1d35=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@1952a85, 3f8f6a0f-aed4-48e3-9129-18a2a3f64eef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@2f6688ae, 71ff56d9-79b8-445d-b637-72ffc974f109=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@730210fb, 752a9438-cd11-426c-b384-bc3c5f86ed07=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@c3be510c, c3e7447e-8514-4e4a-9ff5-a648fe6aa537=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@450befac, 79e8e93c-57c8-4541-a360-726cec3790cf=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@1926e392}, log id: 68f1aecc 2019-02-14 02:41:49,489-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host0, VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 38debe74 2019-02-14 02:41:49,581-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5e5a7925, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2cdf5c9e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@443cb62, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@49a3e880, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@443d23c0, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1250bc75, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8d27d86, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5e6363f4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@73ed78db, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@64c9d1c7, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7fecbe95, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3a551e5f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2266926e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@88b380c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1209279e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3c6466, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@16df63ed, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@47456262, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c2b88c3, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7f57c074, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12fa0478], log id: 38debe74 2019-02-14 02:41:49,582-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host0, VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 7ec02237 2019-02-14 02:41:49,660-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@3eedd0bc, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@7f78e375, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@3d63e126], log id: 7ec02237 2019-02-14 02:41:49,661-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVDOVolumeListVDSCommand(HostName = Host0, VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 42cdad27 2019-02-14 02:41:50,142-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 42cdad27 2019-02-14 02:41:50,143-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), log id: 12f5fdf2 2019-02-14 02:41:50,248-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2aaed792, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8e66930, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@276d599e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1aca2aec, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@46846c60, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7d103269, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@30fc25fc, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7baae445, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1ea8603c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@62578afa, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@33d58089, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1f71d27a, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4205e828, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c5bbac8, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@395a002, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12664008, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7f4faec4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3e03d61f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1038e46d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@307e8062, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@32453127], log id: 12f5fdf2 2019-02-14 02:41:50,249-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), log id: 1256aa5e 2019-02-14 02:41:50,338-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@459a2ff5, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@123cab4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@1af41fbe], log id: 1256aa5e 2019-02-14 02:41:50,339-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVDOVolumeListVDSCommand(HostName = Host1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}), log id: 3dd752e4 2019-02-14 02:41:50,847-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 3dd752e4 2019-02-14 02:41:50,848-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 29a6272c 2019-02-14 02:41:50,954-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalLogicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@364f3ec6, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@c7cce5e, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@b3bed47, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@13bc244b, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5cca81f4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@36aeba0d, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@62ab384a, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1047d628, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@188a30f5, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5bb79f3b, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@60e5956f, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4e3df9cd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7796567, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@60d06cf4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2cd2d36c, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@d80a4aa, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@411eaa20, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@22cac93b, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@18b927bd, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@101465f4, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@246f927c], log id: 29a6272c 2019-02-14 02:41:50,955-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 501814db 2019-02-14 02:41:51,044-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterLocalPhysicalVolumeListVDSCommand, return: [org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@1cd55aa, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@32c5aba2, org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@6ae123f4], log id: 501814db 2019-02-14 02:41:51,045-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVDOVolumeListVDSCommand(HostName = Host2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 7acf4cbf 2019-02-14 02:41:51,546-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] FINISH, GetGlusterVDOVolumeListVDSCommand, return: [], log id: 7acf4cbf 2019-02-14 02:41:51,547-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand] (DefaultQuartzScheduler1) [5ff5b093] START, GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Host0, GlusterVolumeAdvancedDetailsVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5', volumeName='non_prod_a'}), log id: 11c42649
Brick log from around the time the brick went down: [2019-02-14 02:20:29.611099] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 [2019-02-14 02:20:29.611131] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:ee716e24-e187-4b57-a371-cab544f41162-GRAPH_ID:0-PID:30671-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3) [2019-02-14 02:20:29.619521] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:ee716e24-e187-4b57-a371-cab544f41162-GRAPH_ID:0-PID:30671-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 [2019-02-14 02:20:29.619867] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:ee716e24-e187-4b57-a371-cab544f41162-GRAPH_ID:0-PID:30671-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 19988 times between [2019-02-14 02:19:31.377315] and [2019-02-14 02:21:14.033991] [2019-02-14 02:21:30.303440] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 10 times between [2019-02-14 02:21:30.303440] and [2019-02-14 02:23:20.421140] [2019-02-14 02:23:33.142281] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 34 times between [2019-02-14 02:23:33.142281] and [2019-02-14 02:25:29.115156] [2019-02-14 02:25:30.326469] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 02:25:53.973830] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.222" [2019-02-14 02:25:53.973896] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 [2019-02-14 02:25:53.973928] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:4a6b8860-8274-4b3b-b400-d66cbfb97349-GRAPH_ID:0-PID:33522-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3) [2019-02-14 02:25:54.627728] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:4a6b8860-8274-4b3b-b400-d66cbfb97349-GRAPH_ID:0-PID:33522-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 [2019-02-14 02:25:54.628149] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:4a6b8860-8274-4b3b-b400-d66cbfb97349-GRAPH_ID:0-PID:33522-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 [2019-02-14 02:25:56.396855] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.220" [2019-02-14 02:25:56.396926] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 [2019-02-14 02:25:56.396957] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:963c2196-108c-485d-aca6-a236906d2acf-GRAPH_ID:0-PID:33635-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3) [2019-02-14 02:25:56.404566] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:963c2196-108c-485d-aca6-a236906d2acf-GRAPH_ID:0-PID:33635-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 [2019-02-14 02:25:56.404866] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:963c2196-108c-485d-aca6-a236906d2acf-GRAPH_ID:0-PID:33635-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 25 times between [2019-02-14 02:25:30.326469] and [2019-02-14 02:27:25.965601] [2019-02-14 02:28:10.538374] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 38 times between [2019-02-14 02:28:10.538374] and [2019-02-14 02:29:22.622679] [2019-02-14 02:29:48.891040] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 02:29:56.026002] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 02:31:22.494824] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.221" [2019-02-14 02:31:22.494914] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 [2019-02-14 02:31:22.494946] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:210b837a-1c98-4d88-aab8-5ce895398d81-GRAPH_ID:0-PID:18765-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3) [2019-02-14 02:31:22.503920] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:210b837a-1c98-4d88-aab8-5ce895398d81-GRAPH_ID:0-PID:18765-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0 [2019-02-14 02:31:22.504205] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:210b837a-1c98-4d88-aab8-5ce895398d81-GRAPH_ID:0-PID:18765-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0 The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 15301 times between [2019-02-14 02:29:48.891040] and [2019-02-14 02:31:28.963054] [2019-02-14 02:31:31.016063] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 628 times between [2019-02-14 02:31:31.016063] and [2019-02-14 02:33:27.936726] [2019-02-14 02:33:35.878655] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 1776 times between [2019-02-14 02:33:35.878655] and [2019-02-14 02:35:28.143659] [2019-02-14 02:35:29.607848] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 02:36:00.483996] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.220" [2019-02-14 02:36:00.484066] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 [2019-02-14 02:36:00.484098] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:9f2cd861-23c9-412b-be2c-8c45658bb155-GRAPH_ID:0-PID:39820-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3) [2019-02-14 02:36:00.779203] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:9f2cd861-23c9-412b-be2c-8c45658bb155-GRAPH_ID:0-PID:39820-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 [2019-02-14 02:36:00.779662] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:9f2cd861-23c9-412b-be2c-8c45658bb155-GRAPH_ID:0-PID:39820-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 [2019-02-14 02:36:49.892372] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.222" [2019-02-14 02:36:49.892451] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 [2019-02-14 02:36:49.892481] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:91cc19de-30ae-4639-a407-c6fce4682fa4-GRAPH_ID:0-PID:39353-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3) [2019-02-14 02:36:49.900530] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:91cc19de-30ae-4639-a407-c6fce4682fa4-GRAPH_ID:0-PID:39353-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 [2019-02-14 02:36:49.900863] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:91cc19de-30ae-4639-a407-c6fce4682fa4-GRAPH_ID:0-PID:39353-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 15772 times between [2019-02-14 02:35:29.607848] and [2019-02-14 02:37:24.862502] [2019-02-14 02:38:11.704945] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 23 times between [2019-02-14 02:38:11.704945] and [2019-02-14 02:39:24.469485] [2019-02-14 02:39:47.550186] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 34 times between [2019-02-14 02:39:47.550186] and [2019-02-14 02:41:28.642587] [2019-02-14 02:41:29.746302] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 02:42:15.811205] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.220" [2019-02-14 02:42:15.811284] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 [2019-02-14 02:42:15.811317] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:6de46e97-ef84-43c8-a472-4e942471a8f6-GRAPH_ID:0-PID:43752-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3) [2019-02-14 02:42:15.819841] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:6de46e97-ef84-43c8-a472-4e942471a8f6-GRAPH_ID:0-PID:43752-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 [2019-02-14 02:42:15.820156] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:6de46e97-ef84-43c8-a472-4e942471a8f6-GRAPH_ID:0-PID:43752-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 5147 times between [2019-02-14 02:41:29.746302] and [2019-02-14 02:43:28.777825] [2019-02-14 02:43:35.252750] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 17909 times between [2019-02-14 02:43:35.252750] and [2019-02-14 02:45:29.331082] [2019-02-14 02:45:29.497786] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 02:46:08.243167] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.222" [2019-02-14 02:46:08.243230] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 [2019-02-14 02:46:08.243259] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:56fb6f47-3e05-4f07-abdf-cb49213cf5c0-GRAPH_ID:0-PID:44348-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3) [2019-02-14 02:46:09.512844] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:56fb6f47-3e05-4f07-abdf-cb49213cf5c0-GRAPH_ID:0-PID:44348-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 [2019-02-14 02:46:09.513302] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:56fb6f47-3e05-4f07-abdf-cb49213cf5c0-GRAPH_ID:0-PID:44348-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 2892 times between [2019-02-14 02:45:29.497786] and [2019-02-14 02:47:23.957871] [2019-02-14 02:47:30.087716] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 02:47:42.722682] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.221" [2019-02-14 02:47:42.722779] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 [2019-02-14 02:47:42.722810] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:34bb8b68-3483-4744-bd33-7a47cd4214b2-GRAPH_ID:0-PID:27248-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3) [2019-02-14 02:47:42.731133] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:34bb8b68-3483-4744-bd33-7a47cd4214b2-GRAPH_ID:0-PID:27248-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0 [2019-02-14 02:47:42.731447] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:34bb8b68-3483-4744-bd33-7a47cd4214b2-GRAPH_ID:0-PID:27248-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0 The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 564 times between [2019-02-14 02:47:30.087716] and [2019-02-14 02:49:28.480491] [2019-02-14 02:49:32.129061] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 15058 times between [2019-02-14 02:49:32.129061] and [2019-02-14 02:51:26.667738] [2019-02-14 02:52:35.362594] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 02:53:08.127443] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.222" [2019-02-14 02:53:08.127528] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 [2019-02-14 02:53:08.127562] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:3fbc7931-7ac7-4dfd-9169-6271a18fc0f9-GRAPH_ID:0-PID:48068-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3) [2019-02-14 02:53:08.135384] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:3fbc7931-7ac7-4dfd-9169-6271a18fc0f9-GRAPH_ID:0-PID:48068-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 [2019-02-14 02:53:08.135682] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:3fbc7931-7ac7-4dfd-9169-6271a18fc0f9-GRAPH_ID:0-PID:48068-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 11 times between [2019-02-14 02:52:35.362594] and [2019-02-14 02:53:24.927967] [2019-02-14 02:53:56.793284] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 38 times between [2019-02-14 02:53:56.793284] and [2019-02-14 02:55:29.101365] [2019-02-14 02:55:30.445077] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 02:56:14.259883] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.222" [2019-02-14 02:56:14.259948] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 [2019-02-14 02:56:14.259980] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:d7f3996c-4c68-4e15-b0e2-0197bc358ddd-GRAPH_ID:0-PID:885-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3) [2019-02-14 02:56:14.336000] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:d7f3996c-4c68-4e15-b0e2-0197bc358ddd-GRAPH_ID:0-PID:885-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 [2019-02-14 02:56:14.336327] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:d7f3996c-4c68-4e15-b0e2-0197bc358ddd-GRAPH_ID:0-PID:885-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 19 times between [2019-02-14 02:55:30.445077] and [2019-02-14 02:57:28.589205] [2019-02-14 02:57:35.071899] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 02:58:33.855724] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.220" [2019-02-14 02:58:33.855805] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 [2019-02-14 02:58:33.855838] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:868382a8-7e38-481c-b5a3-80c1bdd21f28-GRAPH_ID:0-PID:4631-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3) [2019-02-14 02:58:33.864814] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting connection from CTX_ID:868382a8-7e38-481c-b5a3-80c1bdd21f28-GRAPH_ID:0-PID:4631-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 [2019-02-14 02:58:33.865124] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down connection CTX_ID:868382a8-7e38-481c-b5a3-80c1bdd21f28-GRAPH_ID:0-PID:4631-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0 The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 35 times between [2019-02-14 02:57:35.071899] and [2019-02-14 02:59:29.398638] [2019-02-14 02:59:33.501166] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 03:00:01.622537] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 21104 times between [2019-02-14 02:59:33.501166] and [2019-02-14 03:01:29.325701] [2019-02-14 03:01:30.635130] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 640 times between [2019-02-14 03:01:30.635130] and [2019-02-14 03:03:29.020755] [2019-02-14 03:03:31.238543] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler [2019-02-14 03:04:00.722382] I [addr.c:54:compare_addr_and_update] 0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr = "10.11.0.221" [2019-02-14 03:04:00.722466] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3 [2019-02-14 03:04:00.722499] I [MSGID: 115029] [server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted client from CTX_ID:ca4e5703-7129-4242-9fbf-9b119de9e0eb-GRAPH_ID:0-PID:35692-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0 (version: 5.3)
Can we add https://bugzilla.redhat.com/show_bug.cgi?id=1306562 to this list? We would like to start using libgfapi but without live migration this doesn't work for us.
(In reply to Netbulae from comment #3) > Can we add https://bugzilla.redhat.com/show_bug.cgi?id=1306562 to this list? > > We would like to start using libgfapi but without live migration this > doesn't work for us. Ah sorry, this is a libvirt issue not GlusterFS. Please ignore
Moving to 4.3.6, not completed in 4.3.5.
When will this issue be fixed? The target milestone was 4.3.2 initially and now it was moved to 4.3.7! We can't upgrade our oVirt infrastructure to 4.3 since this problem is a show stopper for us. We just can't manually bring up the bricks when they go down as we don't have that many resources. Is there a specific configuration which causes this or are there any other workarounds than restart the corresponding gluster service? Kind regards, Timo
(In reply to Timo Eissler from comment #6) > When will this issue be fixed? > The target milestone was 4.3.2 initially and now it was moved to 4.3.7! > > We can't upgrade our oVirt infrastructure to 4.3 since this problem is a > show stopper for us. > We just can't manually bring up the bricks when they go down as we don't > have that many resources. > > Is there a specific configuration which causes this or are there any other > workarounds than restart the corresponding gluster service? > > Kind regards, > Timo This should be working with existing release 4.3.6 All tracking issues in this bug have been closed except for bug 1672318, which seems to be hitting 1 particular user. We will check on that and update the state of this bug. Setting a NI on me to follow up
Moving this to ON_QA as bug 1672318 seems like a setup issue specific to user
(In reply to Sahina Bose from comment #8) > Moving this to ON_QA as bug 1672318 seems like a setup issue specific to user Based on this comment, removing this bug 1672318 from the tracker
This bug is the TRACKER, that tracks the individual bugs that are targetted specifically for the release. It doesn't makes sense to move the TRACKER to ON_QA. CLOSING this tracker bug as all the bugs tracked are already CLOSED.