Bug 1443010

Summary:	snapshot: snapshots appear to be failing with respect to secure geo-rep slave
Product:	[Community] GlusterFS	Reporter:	Kotresh HR <khiremat>
Component:	snapshot	Assignee:	bugs <bugs>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	high	Docs Contact:
Priority:	high
Version:	3.10	CC:	ashah, atumball, avishwan, bugs, khiremat, olim, rhs-bugs, rtalur, storage-qa-internal
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.10.2	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1442760	Environment:
Last Closed:	2017-05-31 20:46:29 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1442760
Bug Blocks:	1438972, 1442603

Comment 1 Worker Ant 2017-04-18 09:52:57 UTC

REVIEW: https://review.gluster.org/17070 (glusterd: Fix snapshot failure in non-root geo-rep setup) posted (#1) for review on release-3.10 by Kotresh HR (khiremat)

Comment 2 Kotresh HR 2017-04-18 09:54:44 UTC

Description of problem:

Attempting to create snapshots fails with the following error messages:

# gluster snapshot create master-snap master
snapshot create: failed: Commit failed on localhost. Please check log file for details.
Snapshot command failed

glusterd.log:
[2017-03-27 18:43:11.316696] E [MSGID: 106029] [glusterd-snapshot.c:544:glusterd_copy_geo_rep_session_files] 0-management: Session files not present in /var/lib/glusterd/geo-replication/master_geoaccount@fedora1_slave [No such file or directory]
[2017-03-27 18:43:11.316730] E [MSGID: 106029] [glusterd-snapshot.c:752:glusterd_copy_geo_rep_files] 0-management: Failed to copy files related to session master_geoaccount@fedora1_slave
[2017-03-27 18:43:11.316737] E [MSGID: 106031] [glusterd-snapshot.c:5434:glusterd_do_snap_vol] 0-management: Failed to copy geo-rep config and status files for volume master
[2017-03-27 18:43:11.316754] E [MSGID: 106033] [glusterd-store.c:1746:glusterd_store_delete_volume] 0-management: Failed to rename volume directory for volume 6c5ff90e838a486897e43ffd801b6c91 [No such file or directory]
[2017-03-27 18:43:11.316762] W [MSGID: 106071] [glusterd-snapshot.c:3164:glusterd_snap_volume_remove] 0-management: Failed to remove volume 6c5ff90e838a486897e43ffd801b6c91 from store
[2017-03-27 18:43:11.316780] W [MSGID: 106030] [glusterd-snapshot.c:7002:glusterd_snapshot_create_commit] 0-management: taking the snapshot of the volume lss failed
[2017-03-27 18:43:11.316882] E [MSGID: 106030] [glusterd-snapshot.c:8449:glusterd_snapshot] 0-management: Failed to create snapshot
[2017-03-27 18:43:11.316899] W [MSGID: 106123] [glusterd-mgmt.c:283:gd_mgmt_v3_commit_fn] 0-management: Snapshot Commit Failed
[2017-03-27 18:43:11.316912] E [MSGID: 106123] [glusterd-mgmt.c:1456:glusterd_mgmt_v3_commit] 0-management: Commit failed for operation Snapshot on local node
[2017-03-27 18:43:11.316919] E [MSGID: 106123] [glusterd-mgmt.c:2333:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Commit Op Failed
[2017-03-27 18:43:12.456142] E [MSGID: 106062] [glusterd-snapshot.c:6424:glusterd_do_snap_cleanup] 0-glusterd: Unable to get volume name
[2017-03-27 18:43:12.456166] W [MSGID: 106040] [glusterd-snapshot.c:8269:glusterd_snapshot_create_postvalidate] 0-management: cleanup operation failed
[2017-03-27 18:43:12.456173] W [MSGID: 106030] [glusterd-snapshot.c:9235:glusterd_snapshot_postvalidate] 0-management: Snapshot create post-validation failed
[2017-03-27 18:43:12.456179] W [MSGID: 106121] [glusterd-mgmt.c:373:gd_mgmt_v3_post_validate_fn] 0-management: postvalidate operation failed
[2017-03-27 18:43:12.456185] E [MSGID: 106121] [glusterd-mgmt.c:1689:glusterd_mgmt_v3_post_validate] 0-management: Post Validation failed for operation Snapshot on local node
[2017-03-27 18:43:12.456192] E [MSGID: 106122] [glusterd-mgmt.c:2392:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Post Validation Failed

Comment 3 Worker Ant 2017-04-27 10:45:04 UTC

COMMIT: https://review.gluster.org/17070 committed in release-3.10 by Raghavendra Talur (rtalur) 
------
commit 6cd1f12ae58f1cace89104664a6d41b682b7ee88
Author: Kotresh HR <khiremat>
Date:   Mon Apr 17 08:39:30 2017 -0400

    glusterd: Fix snapshot failure in non-root geo-rep setup
    
    Geo-replication session directory name has the form
    '<mastervol>_<slavehost>_<slavevol>'. But in non-root
    geo-replication setup, while preparing geo-replication
    session directory name, glusterd is including 'user@'
    resulting in "<mastervol>_<user@slavehost>_<slavevol>".
    Hence snapshot is failing to copy geo-rep specific
    session files. Fixing the same.
    
    > BUG: 1442760
    > Signed-off-by: Kotresh HR <khiremat>
    > Reviewed-on: https://review.gluster.org/17067
    > Smoke: Gluster Build System <jenkins.org>
    > NetBSD-regression: NetBSD Build System <jenkins.org>
    > Reviewed-by: Amar Tumballi <amarts>
    > CentOS-regression: Gluster Build System <jenkins.org>
    > Reviewed-by: Aravinda VK <avishwan>
      (cherry picked from commit cc839523364e47dea715cd7241772cd68f05f76c)
    
    Change-Id: Id214d3186e40997d2827a0bb60d3676ca2552df7
    BUG: 1443010
    Signed-off-by: Kotresh HR <khiremat>
    Reviewed-on: https://review.gluster.org/17070
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Reviewed-by: Atin Mukherjee <amukherj>
    CentOS-regression: Gluster Build System <jenkins.org>

Comment 4 Raghavendra Talur 2017-05-31 20:46:29 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.2, please open a new bug report.