Bug 1565577

Summary: [geo-rep]: Lot of changelogs retries and "dict is null" errors in geo-rep logs
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Rochelle <rallan>
Component: distributeAssignee: Mohit Agrawal <moagrawa>
Status: CLOSED ERRATA QA Contact: Rochelle <rallan>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rhgs-3.4CC: csaba, khiremat, moagrawa, rhinduja, rhs-bugs, sheggodu, storage-qa-internal
Target Milestone: ---   
Target Release: RHGS 3.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.12.2-10 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1571069 1576767 1580215 (view as bug list) Environment:
Last Closed: 2018-09-04 06:46:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1499520, 1600671    
Bug Blocks: 1503137, 1571069, 1576767, 1580215    

Description Rochelle 2018-04-10 10:54:27 UTC
Description of problem:
=======================
Observed excessive 'dict is null' errors on the master and the slave:

Master:
-------
[2018-04-10 06:57:16.611887] E [MSGID: 101046] [dht-common.c:749:dht_discover_complete] 0-master-dht: dict is null
The message "E [MSGID: 101046] [dht-common.c:749:dht_discover_complete] 0-master-dht: dict is null" repeated 6693 times between [2018-04-10 06:57:16.611887] and [2018-04-10 06:58:16.426846]
[2018-04-10 06:58:34.040023] E [MSGID: 101046] [dht-common.c:749:dht_discover_complete] 0-master-dht: dict is null
The message "E [MSGID: 101046] [dht-common.c:749:dht_discover_complete] 0-master-dht: dict is null" repeated 11449 times between [2018-04-10 06:58:34.040023] and [2018-04-10 07:00:12.429952]
[2018-04-10 07:00:26.919063] E [MSGID: 101046] [dht-common.c:749:dht_discover_complete] 0-master-dht: dict is null
The message "E [MSGID: 101046] [dht-common.c:749:dht_discover_complete] 0-master-dht: dict is null" repeated 9760 times between [2018-04-10 07:00:26.919063] and [2018-04-10 07:01:52.179336]


Slave:
------
The message "E [MSGID: 101046] [dht-common.c:749:dht_discover_complete] 0-slave-dht: dict is null" repeated 51 times between [2018-04-10 06:24:36.408769] and [2018-04-10 06:24:37.168309]
[2018-04-10 06:24:37.179356] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-slave-dht: Found anomalies in (null) (gfid = aee3b531-d3ea-4a1b-a030-91f8e98b566c). Holes=1 overlaps=0
[2018-04-10 06:24:37.213912] E [MSGID: 101046] [dht-common.c:749:dht_discover_complete] 0-slave-dht: dict is null
The message "E [MSGID: 101046] [dht-common.c:749:dht_discover_complete] 0-slave-dht: dict is null" repeated 2 times between [2018-04-10 06:24:37.213912] and [2018-04-10 06:24:37.233486]
[2018-04-10 06:24:37.244772] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 0-slave-dht: Found anomalies in (null) (gfid = 152cd127-929e-432c-af50-9e2f5de008bd). Holes=1 overlaps=0
[2018-04-10 06:24:37.256863] E [MSGID: 101046] [dht-common.c:749:dht_discover_complete] 0-slave-dht: dict is null
The message "E [MSGID: 101046] [dht-common.c:749:dht_discover_complete] 0-slave-dht: dict is null" repeated 2 times between [2018-04-10 06:24:37.256863] and [2018-04-10 06:24:37.275916]




Version-Release number of selected component (if applicable):
==============================================================
[root@dhcp42-58 geo-replication-slaves]# rpm -qa | grep gluster
glusterfs-3.12.2-7.el7rhgs.x86_64
vdsm-gluster-4.19.43-2.3.el7rhgs.noarch
libvirt-daemon-driver-storage-gluster-3.9.0-14.el7.x86_64
glusterfs-api-3.12.2-7.el7rhgs.x86_64
glusterfs-geo-replication-3.12.2-7.el7rhgs.x86_64
glusterfs-fuse-3.12.2-7.el7rhgs.x86_64
python2-gluster-3.12.2-7.el7rhgs.x86_64
gluster-nagios-common-0.2.4-1.el7rhgs.noarch
glusterfs-client-xlators-3.12.2-7.el7rhgs.x86_64
glusterfs-server-3.12.2-7.el7rhgs.x86_64
glusterfs-rdma-3.12.2-7.el7rhgs.x86_64
gluster-nagios-addons-0.2.10-2.el7rhgs.x86_64
glusterfs-cli-3.12.2-7.el7rhgs.x86_64
glusterfs-libs-3.12.2-7.el7rhgs.x86_6

How reproducible:
=================
2/2

Steps to Reproduce:
===================
1.Create and start master and slave volumes
2.Create data on the master mount
3.Create and start a geo-replication session
4.Calculate the checksum of master and slave (matches)
5.rm -rf * on master
6.Checksum matches.

Comment 17 errata-xmlrpc 2018-09-04 06:46:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607