Bug 1339683

Summary:	[geo-rep]: Converting existing geo-rep session to non-root session fails with permission denied
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Rahul Hinduja <rhinduja>
Component:	geo-replication	Assignee:	Bug Updates Notification Mailing List <rhs-bugs>
Status:	CLOSED NOTABUG	QA Contact:	storage-qa-internal <storage-qa-internal>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.1	CC:	avishwan, csaba
Target Milestone:	---	Keywords:	ZStream
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-05-26 11:22:00 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Rahul Hinduja 2016-05-25 15:26:25 UTC

Description of problem:
=======================

From 3.1.3 onwards, we have made changes to handle create validation better as mentioned in bz 1261838 and bz 1286791 by using slave uuid. This enables a use case to convert the existing geo-rep session to secured session(non-root), which is failing with the following traceback:


[2016-05-25 15:22:55.995728] E [resource(/rhs/brick1/b1):230:logerr] Popen: ssh> Traceback (most recent call last):
[2016-05-25 15:22:55.995903] E [resource(/rhs/brick2/b7):230:logerr] Popen: ssh> Traceback (most recent call last):
[2016-05-25 15:22:55.995934] E [resource(/rhs/brick1/b1):230:logerr] Popen: ssh>   File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 201, in main
[2016-05-25 15:22:55.996125] E [resource(/rhs/brick2/b7):230:logerr] Popen: ssh>   File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 201, in main
[2016-05-25 15:22:55.996160] E [resource(/rhs/brick1/b1):230:logerr] Popen: ssh>     main_i()
[2016-05-25 15:22:55.996381] E [resource(/rhs/brick2/b7):230:logerr] Popen: ssh>     main_i()
[2016-05-25 15:22:55.996407] E [resource(/rhs/brick1/b1):230:logerr] Popen: ssh>   File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 525, in main_i
[2016-05-25 15:22:55.996578] E [resource(/rhs/brick2/b7):230:logerr] Popen: ssh>   File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 525, in main_i
[2016-05-25 15:22:55.996623] E [resource(/rhs/brick1/b1):230:logerr] Popen: ssh>     upgrade_config_file(rconf['config_file'])
[2016-05-25 15:22:55.996773] E [resource(/rhs/brick2/b7):230:logerr] Popen: ssh>     upgrade_config_file(rconf['config_file'])
[2016-05-25 15:22:55.996834] E [resource(/rhs/brick1/b1):230:logerr] Popen: ssh>   File "/usr/libexec/glusterfs/python/syncdaemon/configinterface.py", line 121, in upgrade_config_file
[2016-05-25 15:22:55.996994] E [resource(/rhs/brick2/b7):230:logerr] Popen: ssh>   File "/usr/libexec/glusterfs/python/syncdaemon/configinterface.py", line 121, in upgrade_config_file
[2016-05-25 15:22:55.997055] E [resource(/rhs/brick1/b1):230:logerr] Popen: ssh>     shutil.move(tempConfigFile.name, path)
[2016-05-25 15:22:55.997234] E [resource(/rhs/brick2/b7):230:logerr] Popen: ssh>     shutil.move(tempConfigFile.name, path)
[2016-05-25 15:22:55.997307] E [resource(/rhs/brick1/b1):230:logerr] Popen: ssh>   File "/usr/lib64/python2.7/shutil.py", line 301, in move
[2016-05-25 15:22:55.997441] E [resource(/rhs/brick2/b7):230:logerr] Popen: ssh>   File "/usr/lib64/python2.7/shutil.py", line 301, in move
[2016-05-25 15:22:55.997509] E [resource(/rhs/brick1/b1):230:logerr] Popen: ssh>     copy2(src, real_dst)
[2016-05-25 15:22:55.997636] E [resource(/rhs/brick2/b7):230:logerr] Popen: ssh>     copy2(src, real_dst)
[2016-05-25 15:22:55.997714] E [resource(/rhs/brick1/b1):230:logerr] Popen: ssh>   File "/usr/lib64/python2.7/shutil.py", line 130, in copy2
[2016-05-25 15:22:55.997827] E [resource(/rhs/brick2/b7):230:logerr] Popen: ssh>   File "/usr/lib64/python2.7/shutil.py", line 130, in copy2
[2016-05-25 15:22:55.997960] E [resource(/rhs/brick1/b1):230:logerr] Popen: ssh>     copyfile(src, dst)
[2016-05-25 15:22:55.998039] E [resource(/rhs/brick2/b7):230:logerr] Popen: ssh>     copyfile(src, dst)
[2016-05-25 15:22:55.998229] E [resource(/rhs/brick1/b1):230:logerr] Popen: ssh>   File "/usr/lib64/python2.7/shutil.py", line 83, in copyfile
[2016-05-25 15:22:55.998296] E [resource(/rhs/brick2/b7):230:logerr] Popen: ssh>   File "/usr/lib64/python2.7/shutil.py", line 83, in copyfile
[2016-05-25 15:22:55.998436] E [resource(/rhs/brick1/b1):230:logerr] Popen: ssh>     with open(dst, 'wb') as fdst:
[2016-05-25 15:22:55.998500] E [resource(/rhs/brick2/b7):230:logerr] Popen: ssh>     with open(dst, 'wb') as fdst:
[2016-05-25 15:22:55.998629] E [resource(/rhs/brick1/b1):230:logerr] Popen: ssh> IOError: [Errno 13] Permission denied: '/var/lib/glusterd/geo-replication/gsyncd_template.conf'
[2016-05-25 15:22:55.998692] E [resource(/rhs/brick2/b7):230:logerr] Popen: ssh> IOError: [Errno 13] Permission denied: '/var/lib/glusterd/geo-replication/gsyncd_template.conf'
[2016-05-25 15:22:55.998819] E [resource(/rhs/brick1/b1):230:logerr] Popen: ssh> failed with IOError.
[2016-05-25 15:22:55.998887] E [resource(/rhs/brick2/b7):230:logerr] Popen: ssh> failed with IOError.



Version-Release number of selected component (if applicable):
=============================================================

glusterfs-geo-replication-3.7.9-6.el7rhgs.x86_64
glusterfs-3.7.9-6.el7rhgs.x86_64


How reproducible:
=================

Always


Steps to Reproduce:
===================
1. Create geo-rep session between master and slave
2. Convert the geo-rep session from non secured to secured (non-root)

Comment 3 Rahul Hinduja 2016-05-25 16:00:26 UTC

RCA by Aravinda: 
In slave we need to enable r+w permission for non root group for the directory /var/lib/glusterd/geo-replication .
This was existing issue but we are seeing now due to the latest changes in geo-rep config. monitor.status and pid files are renamed, so Geo-rep is trying to upgrade the old config file which fails with permission denied.

Comment 4 Aravinda VK 2016-05-26 11:22:00 UTC

Opened a doc bug to update these steps in documentation. BZ 1340048

Closing this bug. Please reopen if the issue is not resolved with the documented steps. Thanks