Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1232216 - [geo-rep]: UnboundLocalError: local variable 'fd' referenced before assignment
[geo-rep]: UnboundLocalError: local variable 'fd' referenced before assignment
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: geo-replication (Show other bugs)
3.1
x86_64 Linux
unspecified Severity medium
: ---
: RHGS 3.1.1
Assigned To: Kotresh HR
Rahul Hinduja
: ZStream
Depends On:
Blocks: 1216951 1233411 1240607 1251815
  Show dependency treegraph
 
Reported: 2015-06-16 06:05 EDT by Rahul Hinduja
Modified: 2015-10-05 03:11 EDT (History)
10 users (show)

See Also:
Fixed In Version: glusterfs-3.7.1-12
Doc Type: Bug Fix
Doc Text:
Previously, if a meta-volume is configured, there was a small race window, where geo-replication worker access the unreferenced fd of the lock file maintained in shared storage volume. As a consequence, the geo-replication worker died and restarted. A fix is made in worker to always get the right fd. Now, the geo-replication worker does not die and restart.
Story Points: ---
Clone Of:
: 1233411 (view as bug list)
Environment:
Last Closed: 2015-10-05 03:11:39 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2015:1845 normal SHIPPED_LIVE Moderate: Red Hat Gluster Storage 3.1 update 2015-10-05 07:06:22 EDT

  None (edit)
Description Rahul Hinduja 2015-06-16 06:05:06 EDT
Description of problem:
=======================

Happened to see the below traceback in the geo-rep log with the use of meta_volume.

[2015-06-16 20:11:43.911356] I [master(/bricks/brick1/master_brick2):528:crawlwrap] _GMaster: crawl interval: 1 seconds
[2015-06-16 20:11:43.912364] I [master(/bricks/brick0/master_brick0):519:crawlwrap] _GMaster: primary master with volume id a6de699b-83c5-4747-95a7-81c36f5d79c6 ...
[2015-06-16 20:11:43.917547] I [master(/bricks/brick0/master_brick0):528:crawlwrap] _GMaster: crawl interval: 1 seconds
[2015-06-16 20:11:43.918723] I [master(/bricks/brick1/master_brick2):455:mgmt_lock] _GMaster: Creating geo-rep directory in meta volume...
[2015-06-16 20:11:43.923168] I [master(/bricks/brick0/master_brick0):455:mgmt_lock] _GMaster: Creating geo-rep directory in meta volume...
[2015-06-16 20:11:43.938770] E [syncdutils(/bricks/brick0/master_brick0):276:log_raise_exception] <top>: FAIL: 
Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 165, in main
    main_i()
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 659, in main_i
    local.service_loop(*[r for r in [remote] if r])
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1438, in service_loop
    g3.crawlwrap(oneshot=True)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 531, in crawlwrap
    crawl = self.should_crawl()
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 485, in should_crawl
    return self.mgmt_lock()
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 469, in mgmt_lock
    os.close(fd)
UnboundLocalError: local variable 'fd' referenced before assignment
[2015-06-16 20:11:43.941852] I [syncdutils(/bricks/brick0/master_brick0):220:finalize] <top>: exiting.
[2015-06-16 20:11:43.945887] I [repce(agent):92:service_loop] RepceServer: terminating on reaching EOF.
[2015-06-16 20:11:43.946471] I [syncdutils(agent):220:finalize] <top>: exiting.


Version-Release number of selected component (if applicable):
=============================================================

glusterfs-3.7.1-3.el7rhgs.x86_64
Comment 3 Rahul Hinduja 2015-07-04 06:40:26 EDT
Hit this multiple times during Rename and Remove operations. 

[2015-07-04 15:04:45.75320] E [syncdutils(/rhs/brick1/b1):276:log_raise_exception] <top>: FAIL: 
Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 165, in main
    main_i()
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 659, in main_i
    local.service_loop(*[r for r in [remote] if r])
  File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1438, in service_loop
    g3.crawlwrap(oneshot=True)
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 531, in crawlwrap
    crawl = self.should_crawl()
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 485, in should_crawl
    return self.mgmt_lock()
  File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 469, in mgmt_lock
    os.close(fd)
UnboundLocalError: local variable 'fd' referenced before assignment
[2015-07-04 15:04:45.78900] I [syncdutils(/rhs/brick1/b1):220:finalize] <top>: exiting.
Comment 4 monti lawrence 2015-07-22 16:43:26 EDT
Doc text is edited. Please sign off to be included in Known Issues.
Comment 5 Kotresh HR 2015-07-27 02:27:14 EDT
Edited DocText. Please check.
Comment 6 Anjana Suparna Sriram 2015-07-27 22:47:23 EDT
Included the edited text.
Comment 7 Kotresh HR 2015-08-06 02:07:54 EDT
Upstream Patch (Master):
http://review.gluster.org/#/c/11318/

Upstream Patch (3.7):
http://review.gluster.org/#/c/11563/
Comment 10 Kotresh HR 2015-08-11 05:07:26 EDT
Merged in upstream (master) and upstream (3.7). Hence moving it to POST
Comment 11 Kotresh HR 2015-08-13 03:20:42 EDT
Downstream Patch:

https://code.engineering.redhat.com/gerrit/#/c/55050/
Comment 12 Rahul Hinduja 2015-09-01 07:13:47 EDT
Verified with build: glusterfs-3.7.1-14.el7rhgs.x86_64

Didn't see the Traceback. Moving the bug to verified state. 

[root@georep1 ~]# 
[root@georep1 ~]# grep -i "UnboundLocalError" /var/log/glusterfs/geo-replication/master/ssh%3A%2F%2Froot%4010.70.46.167%3Agluster%3A%2F%2F127.0.0.1%3Aslave.*
[root@georep1 ~]# 
[root@georep1 ~]# 

[root@georep2 ~]# 
[root@georep2 ~]# grep -i "UnboundLocalError" /var/log/glusterfs/geo-replication/master/ssh%3A%2F%2Froot%4010.70.46.167%3Agluster%3A%2F%2F127.0.0.1%3Aslave.*
[root@georep2 ~]#

[root@georep3 ~]# 
[root@georep3 ~]# grep -i "UnboundLocalError" /var/log/glusterfs/geo-replication/master/ssh%3A%2F%2Froot%4010.70.46.167%3Agluster%3A%2F%2F127.0.0.1%3Aslave.*
[root@georep3 ~]# 


[root@georep4 ~]# 
[root@georep4 ~]# grep -i "UnboundLocalError" /var/log/glusterfs/geo-replication/master/ssh%3A%2F%2Froot%4010.70.46.167%3Agluster%3A%2F%2F127.0.0.1%3Aslave.*
[root@georep4 ~]# 


[root@georep1 ~]# gluster volume list
gluster_shared_storage
master
[root@georep1 ~]#
Comment 15 errata-xmlrpc 2015-10-05 03:11:39 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1845.html

Note You need to log in before you can comment on or make changes to this bug.