Bug 1278254

Summary: [Snapshot]: Clone creation fails on tiered volume with pre-validation failed message
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Shashank Raj <sraj>
Component: snapshotAssignee: Avra Sengupta <asengupt>
Status: CLOSED ERRATA QA Contact: storage-qa-internal <storage-qa-internal>
Severity: urgent Docs Contact:
Priority: urgent    
Version: rhgs-3.1CC: annair, asengupt, asrivast, nchilaka, rhs-bugs, rjoseph, sankarshan, sashinde, storage-qa-internal
Target Milestone: ---Keywords: ZStream
Target Release: RHGS 3.1.2   
Hardware: x86_64   
OS: Linux   
Whiteboard: SNAPSHOT
Fixed In Version: glusterfs-3.7.5-9 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1279327 (view as bug list) Environment:
Last Closed: 2016-03-01 05:51:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1260783, 1279327, 1287538    

Description Shashank Raj 2015-11-05 05:51:48 UTC 
Description of problem:

Clone creation fails on tiered volume with pre-validation failed message

Version-Release number of selected component (if applicable):

glusterfs-3.7.5-5

How reproducible:

Always

Steps to Reproduce:

1.Create a 2x(4+2) EC volume and start it.
2.Enable quota on the volume
3.Attach a tier with 2x2 dist-rep on the volume.
4.Create a snapshot of the tiered volume make sure it is successful.
5.Create a clone of this snapshot and observe that it fails with error message "Pre-validation failed"

below logs are observed in /var/log/glusterfs/etc-glusterfs-glusterd.vol.log

2015-11-05 05:38:24.339519] E [MSGID: 106027] [glusterd-op-sm.c:6550:glusterd_bricks_select_rebalance_volume] 0-glusterd: Volume 7d697e1e9f904a5f85777904a5b154bd does not exist
[2015-11-05 05:38:24.339598] E [MSGID: 106125] [glusterd-syncop.c:1646:gd_brick_op_phase] 0-management: Volume 7d697e1e9f904a5f85777904a5b154bd does not exist
[2015-11-05 05:38:24.339616] E [MSGID: 106572] [glusterd-snapshot.c:1998:glusterd_snapshot_pause_tier] 0-management: Failed to pause tier. Errstr=Volume 7d697e1e9f904a5f85777904a5b154bd does not exist
[2015-11-05 05:38:24.339633] E [MSGID: 106572] [glusterd-snapshot.c:2397:glusterd_snapshot_clone_prevalidate] 0-management: Failed to pause tier in clone prevalidate.
[2015-11-05 05:38:24.339653] W [MSGID: 106443] [glusterd-snapshot.c:8392:glusterd_snapshot_prevalidate] 0-management: Snapshot clone pre-validation failed
[2015-11-05 05:38:24.339670] W [MSGID: 106122] [glusterd-mgmt.c:166:gd_mgmt_v3_pre_validate_fn] 0-management: Snapshot Prevalidate Failed
[2015-11-05 05:38:24.339687] E [MSGID: 106122] [glusterd-mgmt.c:820:glusterd_mgmt_v3_pre_validate] 0-management: Pre Validation failed for operation Snapshot on local node
[2015-11-05 05:38:24.339706] E [MSGID: 106122] [glusterd-mgmt.c:2164:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Pre Validation Failed
[2015-11-05 05:38:24.339750] E [MSGID: 106027] [glusterd-snapshot.c:7940:glusterd_snapshot_clone_postvalidate] 0-management: unable to find clone testclone volinfo
[2015-11-05 05:38:24.339781] W [MSGID: 106444] [glusterd-snapshot.c:8831:glusterd_snapshot_postvalidate] 0-management: Snapshot create post-validation failed
[2015-11-05 05:38:24.339796] W [MSGID: 106121] [glusterd-mgmt.c:323:gd_mgmt_v3_post_validate_fn] 0-management: postvalidate operation failed
[2015-11-05 05:38:24.339812] E [MSGID: 106121] [glusterd-mgmt.c:1585:glusterd_mgmt_v3_post_validate] 0-management: Post Validation failed for operation Snapshot on local node
[2015-11-05 05:38:24.339826] E [MSGID: 106122] [glusterd-mgmt.c:2284:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Post Validation Failed
[2015-11-05 05:38:25.723507] W [socket.c:588:__socket_rwv] 0-nfs: readv on /var/run/gluster/11f5a41d4df7a19d42d4e641eb784bfa.socket failed (Invalid argument)

Actual results:

Clone creation is not successful.


Expected results:

Clone creation should be successful without any errors.

Additional info:
Comment 2 Avra Sengupta 2015-11-09 08:45:57 UTC 
FIx sent for master (upstream) http://review.gluster.org/#/c/12548/2
Comment 4 Avra Sengupta 2015-12-03 06:38:26 UTC 
Master URL : http://review.gluster.org/#/c/12548
Release 3.7 URL : http://review.gluster.org/#/c/12846/
RHGS 3.1.2 URL : https://code.engineering.redhat.com/gerrit/#/c/62894/
Comment 5 Shashank Raj 2015-12-04 09:54:16 UTC 
Verified this bug with latest build glusterfs-3.7.5-9 and it is working as expected.

Output as below:

[root@dhcp35-228 ~]# gluster --version
glusterfs 3.7.5 built on Dec  3 2015 13:01:46

[root@dhcp35-228 ~]# rpm -qa|grep glusterfs
glusterfs-libs-3.7.5-9.el7rhgs.x86_64
glusterfs-fuse-3.7.5-9.el7rhgs.x86_64
glusterfs-rdma-3.7.5-9.el7rhgs.x86_64
glusterfs-3.7.5-9.el7rhgs.x86_64
glusterfs-api-3.7.5-9.el7rhgs.x86_64
glusterfs-cli-3.7.5-9.el7rhgs.x86_64
glusterfs-geo-replication-3.7.5-9.el7rhgs.x86_64
glusterfs-debuginfo-3.7.5-9.el7rhgs.x86_64
glusterfs-client-xlators-3.7.5-9.el7rhgs.x86_64
glusterfs-server-3.7.5-9.el7rhgs.x86_64

[root@dhcp35-228 ~]# gluster snapshot create test_snap tiervolume no-timestamp
snapshot create: success: Snap test_snap created successfully

[root@dhcp35-228 ~]# gluster snapshot activate test_snap
Snapshot activate: test_snap: Snap activated successfully

[root@dhcp35-228 ~]# gluster snapshot clone test_clone test_snap
snapshot clone: success: Clone test_clone created successfully

[root@dhcp35-228 ~]# gluster snapshot clone test_clone2 test_snap
snapshot clone: success: Clone test_clone2 created successfully

[root@dhcp35-228 ~]# gluster snapshot clone test_clone3 test_snap
snapshot clone: success: Clone test_clone3 created successfully

[root@dhcp35-228 ~]# gluster snapshot clone test_clone4 test_snap
snapshot clone: success: Clone test_clone4 created successfully

[root@dhcp35-228 ~]# gluster snapshot clone test_clone5 test_snap
snapshot clone: success: Clone test_clone5 created successfully

Note: Clone creation has been performed while promotion/demotion of files are going on.

Since we are not facing this bug now, Hence marking this bug as Verified.
Comment 7 errata-xmlrpc 2016-03-01 05:51:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html