Bug 1442760

Summary:	snapshot: snapshots appear to be failing with respect to secure geo-rep slave
Product:	[Community] GlusterFS	Reporter:	Kotresh HR <khiremat>
Component:	snapshot	Assignee:	Kotresh HR <khiremat>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	high	Docs Contact:
Priority:	high
Version:	mainline	CC:	ashah, atumball, avishwan, khiremat, olim, rhs-bugs, storage-qa-internal
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.11.0	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1438972
Clones:	1443010 1443012 (view as bug list)		Environment:
Last Closed:	2017-05-30 18:50:19 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1438972, 1443010, 1443012, 1445593

Comment 1 Kotresh HR 2017-04-17 12:38:36 UTC

Description of problem:

Attempting to create snapshots fails with the following error messages:

# gluster snapshot create master-snap master
snapshot create: failed: Commit failed on localhost. Please check log file for details.
Snapshot command failed

glusterd.log:
[2017-03-27 18:43:11.316696] E [MSGID: 106029] [glusterd-snapshot.c:544:glusterd_copy_geo_rep_session_files] 0-management: Session files not present in /var/lib/glusterd/geo-replication/master_geoaccount@fedora1_slave [No such file or directory]
[2017-03-27 18:43:11.316730] E [MSGID: 106029] [glusterd-snapshot.c:752:glusterd_copy_geo_rep_files] 0-management: Failed to copy files related to session master_geoaccount@fedora1_slave
[2017-03-27 18:43:11.316737] E [MSGID: 106031] [glusterd-snapshot.c:5434:glusterd_do_snap_vol] 0-management: Failed to copy geo-rep config and status files for volume master
[2017-03-27 18:43:11.316754] E [MSGID: 106033] [glusterd-store.c:1746:glusterd_store_delete_volume] 0-management: Failed to rename volume directory for volume 6c5ff90e838a486897e43ffd801b6c91 [No such file or directory]
[2017-03-27 18:43:11.316762] W [MSGID: 106071] [glusterd-snapshot.c:3164:glusterd_snap_volume_remove] 0-management: Failed to remove volume 6c5ff90e838a486897e43ffd801b6c91 from store
[2017-03-27 18:43:11.316780] W [MSGID: 106030] [glusterd-snapshot.c:7002:glusterd_snapshot_create_commit] 0-management: taking the snapshot of the volume lss failed
[2017-03-27 18:43:11.316882] E [MSGID: 106030] [glusterd-snapshot.c:8449:glusterd_snapshot] 0-management: Failed to create snapshot
[2017-03-27 18:43:11.316899] W [MSGID: 106123] [glusterd-mgmt.c:283:gd_mgmt_v3_commit_fn] 0-management: Snapshot Commit Failed
[2017-03-27 18:43:11.316912] E [MSGID: 106123] [glusterd-mgmt.c:1456:glusterd_mgmt_v3_commit] 0-management: Commit failed for operation Snapshot on local node
[2017-03-27 18:43:11.316919] E [MSGID: 106123] [glusterd-mgmt.c:2333:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Commit Op Failed
[2017-03-27 18:43:12.456142] E [MSGID: 106062] [glusterd-snapshot.c:6424:glusterd_do_snap_cleanup] 0-glusterd: Unable to get volume name
[2017-03-27 18:43:12.456166] W [MSGID: 106040] [glusterd-snapshot.c:8269:glusterd_snapshot_create_postvalidate] 0-management: cleanup operation failed
[2017-03-27 18:43:12.456173] W [MSGID: 106030] [glusterd-snapshot.c:9235:glusterd_snapshot_postvalidate] 0-management: Snapshot create post-validation failed
[2017-03-27 18:43:12.456179] W [MSGID: 106121] [glusterd-mgmt.c:373:gd_mgmt_v3_post_validate_fn] 0-management: postvalidate operation failed
[2017-03-27 18:43:12.456185] E [MSGID: 106121] [glusterd-mgmt.c:1689:glusterd_mgmt_v3_post_validate] 0-management: Post Validation failed for operation Snapshot on local node
[2017-03-27 18:43:12.456192] E [MSGID: 106122] [glusterd-mgmt.c:2392:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Post Validation Failed

Comment 2 Worker Ant 2017-04-17 12:42:58 UTC

REVIEW: https://review.gluster.org/17067 (glusterd: Fix snapshot failure in non-root geo-rep setup) posted (#1) for review on master by Kotresh HR (khiremat)

Comment 3 Worker Ant 2017-04-18 09:39:49 UTC

COMMIT: https://review.gluster.org/17067 committed in master by Atin Mukherjee (amukherj) 
------
commit cc839523364e47dea715cd7241772cd68f05f76c
Author: Kotresh HR <khiremat>
Date:   Mon Apr 17 08:39:30 2017 -0400

    glusterd: Fix snapshot failure in non-root geo-rep setup
    
    Geo-replication session directory name has the form
    '<mastervol>_<slavehost>_<slavevol>'. But in non-root
    geo-replication setup, while preparing geo-replication
    session directory name, glusterd is including 'user@'
    resulting in "<mastervol>_<user@slavehost>_<slavevol>".
    Hence snapshot is failing to copy geo-rep specific
    session files. Fixing the same.
    
    Change-Id: Id214d3186e40997d2827a0bb60d3676ca2552df7
    BUG: 1442760
    Signed-off-by: Kotresh HR <khiremat>
    Reviewed-on: https://review.gluster.org/17067
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Reviewed-by: Amar Tumballi <amarts>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Aravinda VK <avishwan>

Comment 4 Shyamsundar 2017-05-30 18:50:19 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.11.0, please open a new bug report.

glusterfs-3.11.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2017-May/000073.html
[2] https://www.gluster.org/pipermail/gluster-users/