Bug 1569490 - [geo-rep]: in-service upgrade fails, session in FAULTY state
Summary: [geo-rep]: in-service upgrade fails, session in FAULTY state
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: geo-replication
Version: rhgs-3.4
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: RHGS 3.4.0
Assignee: Kotresh HR
QA Contact: Rochelle
URL:
Whiteboard:
Depends On:
Blocks: 1474012 1503137 1575490 1577862 1611104
TreeView+ depends on / blocked
 
Reported: 2018-04-19 11:49 UTC by Rochelle
Modified: 2018-09-14 05:26 UTC (History)
6 users (show)

Fixed In Version: glusterfs-3.12.2-9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1575490 (view as bug list)
Environment:
Last Closed: 2018-09-04 06:46:03 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:2607 0 None None None 2018-09-04 06:47:38 UTC

Description Rochelle 2018-04-19 11:49:19 UTC
Description of problem:
=======================
While upgrading from glusterfs-3.8.4-54.el7rhgs.x86_64.rpm  (RHEL 7.4) to glusterfs-3.12.2-7.el7rhgs.x86_64 (RHEL 7.5) encountered a FAULTY session where there was only one worker ACTIVE.

[root@dhcp42-53 master]# gluster volume geo-replication master 10.70.42.164::slave status
 
MASTER NODE     MASTER VOL    MASTER BRICK      SLAVE USER    SLAVE                  SLAVE NODE      STATUS    CRAWL STATUS     LAST_SYNCED          
------------------------------------------------------------------------------------------------------------------------------------------
10.70.42.53     master        /rhs/brick1/b1    root          10.70.42.164::slave    N/A             Faulty    N/A              N/A                  
10.70.42.53     master        /rhs/brick2/b4    root          10.70.42.164::slave    N/A             Faulty    N/A              N/A                  
10.70.42.138    master        /rhs/brick1/b3    root          10.70.42.164::slave    10.70.42.164    Active    History Crawl    N/A                  
10.70.42.138    master        /rhs/brick2/b6    root          10.70.42.164::slave    N/A             Faulty    N/A              N/A                  
10.70.42.160    master        /rhs/brick1/b2    root          10.70.42.164::slave    N/A             Faulty    N/A              N/A                  
10.70.42.160    master        /rhs/brick2/b5    root          10.70.42.164::slave    N/A             Faulty    N/A              N/A  



Traceback in geo-rep logs:
--------------------------------
Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 210, in main
    main_i()
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 802, in main_i
    local.service_loop(*[r for r in [remote] if r])
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1676, in service_loop
    g3.crawlwrap(oneshot=True)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 597, in crawlwrap
    self.crawl()
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1470, in crawl
    self.changelogs_batch_process(changes)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1370, in changelogs_batch_process
    self.process(batch)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1204, in process
    self.process_change(change, done, retry)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1123, in process_change
    entry_stime_to_update[0])
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncdstatus.py", line 200, in set_field
    return self._update(merger)
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncdstatus.py", line 161, in _update
    data = mergerfunc(data)
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncdstatus.py", line 194, in merger
    if data[key] == value:
KeyError: 'last_synced_entry'


Version-Release number of selected component (if applicable):
=============================================================
[root@dhcp42-53 glusterfs]# rpm -qa | grep gluster
libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.2.x86_64
glusterfs-3.12.2-7.el7rhgs.x86_64
glusterfs-fuse-3.12.2-7.el7rhgs.x86_64
glusterfs-geo-replication-3.12.2-7.el7rhgs.x86_64
gluster-nagios-common-0.2.4-1.el7rhgs.noarch
glusterfs-libs-3.12.2-7.el7rhgs.x86_64
glusterfs-client-xlators-3.12.2-7.el7rhgs.x86_64
glusterfs-server-3.12.2-7.el7rhgs.x86_64
vdsm-gluster-4.19.43-2.3.el7rhgs.noarch
glusterfs-api-3.12.2-7.el7rhgs.x86_64
glusterfs-rdma-3.12.2-7.el7rhgs.x86_64
glusterfs-events-3.12.2-7.el7rhgs.x86_64
gluster-nagios-addons-0.2.10-2.el7rhgs.x86_64
glusterfs-cli-3.12.2-7.el7rhgs.x86_64
python2-gluster-3.12.2-7.el7rhgs.x86_64


How reproducible:
=================
1/1


Actual results:
===============
Session is FAULTY.

Expected results:
=================
Session should not be FAULTY.

Comment 12 errata-xmlrpc 2018-09-04 06:46:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607


Note You need to log in before you can comment on or make changes to this bug.