Description of problem: Clone creation fails on tiered volume with pre-validation failed message Version-Release number of selected component (if applicable): glusterfs-3.7.5-5 How reproducible: Always Steps to Reproduce: 1.Create a 2x(4+2) EC volume and start it. 2.Enable quota on the volume 3.Attach a tier with 2x2 dist-rep on the volume. 4.Create a snapshot of the tiered volume make sure it is successful. 5.Create a clone of this snapshot and observe that it fails with error message "Pre-validation failed" below logs are observed in /var/log/glusterfs/etc-glusterfs-glusterd.vol.log 2015-11-05 05:38:24.339519] E [MSGID: 106027] [glusterd-op-sm.c:6550:glusterd_bricks_select_rebalance_volume] 0-glusterd: Volume 7d697e1e9f904a5f85777904a5b154bd does not exist [2015-11-05 05:38:24.339598] E [MSGID: 106125] [glusterd-syncop.c:1646:gd_brick_op_phase] 0-management: Volume 7d697e1e9f904a5f85777904a5b154bd does not exist [2015-11-05 05:38:24.339616] E [MSGID: 106572] [glusterd-snapshot.c:1998:glusterd_snapshot_pause_tier] 0-management: Failed to pause tier. Errstr=Volume 7d697e1e9f904a5f85777904a5b154bd does not exist [2015-11-05 05:38:24.339633] E [MSGID: 106572] [glusterd-snapshot.c:2397:glusterd_snapshot_clone_prevalidate] 0-management: Failed to pause tier in clone prevalidate. [2015-11-05 05:38:24.339653] W [MSGID: 106443] [glusterd-snapshot.c:8392:glusterd_snapshot_prevalidate] 0-management: Snapshot clone pre-validation failed [2015-11-05 05:38:24.339670] W [MSGID: 106122] [glusterd-mgmt.c:166:gd_mgmt_v3_pre_validate_fn] 0-management: Snapshot Prevalidate Failed [2015-11-05 05:38:24.339687] E [MSGID: 106122] [glusterd-mgmt.c:820:glusterd_mgmt_v3_pre_validate] 0-management: Pre Validation failed for operation Snapshot on local node [2015-11-05 05:38:24.339706] E [MSGID: 106122] [glusterd-mgmt.c:2164:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Pre Validation Failed [2015-11-05 05:38:24.339750] E [MSGID: 106027] [glusterd-snapshot.c:7940:glusterd_snapshot_clone_postvalidate] 0-management: unable to find clone testclone volinfo [2015-11-05 05:38:24.339781] W [MSGID: 106444] [glusterd-snapshot.c:8831:glusterd_snapshot_postvalidate] 0-management: Snapshot create post-validation failed [2015-11-05 05:38:24.339796] W [MSGID: 106121] [glusterd-mgmt.c:323:gd_mgmt_v3_post_validate_fn] 0-management: postvalidate operation failed [2015-11-05 05:38:24.339812] E [MSGID: 106121] [glusterd-mgmt.c:1585:glusterd_mgmt_v3_post_validate] 0-management: Post Validation failed for operation Snapshot on local node [2015-11-05 05:38:24.339826] E [MSGID: 106122] [glusterd-mgmt.c:2284:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Post Validation Failed [2015-11-05 05:38:25.723507] W [socket.c:588:__socket_rwv] 0-nfs: readv on /var/run/gluster/11f5a41d4df7a19d42d4e641eb784bfa.socket failed (Invalid argument) Actual results: Clone creation is not successful. Expected results: Clone creation should be successful without any errors. Additional info:
FIx sent for master (upstream) http://review.gluster.org/#/c/12548/2
Master URL : http://review.gluster.org/#/c/12548 Release 3.7 URL : http://review.gluster.org/#/c/12846/ RHGS 3.1.2 URL : https://code.engineering.redhat.com/gerrit/#/c/62894/
Verified this bug with latest build glusterfs-3.7.5-9 and it is working as expected. Output as below: [root@dhcp35-228 ~]# gluster --version glusterfs 3.7.5 built on Dec 3 2015 13:01:46 [root@dhcp35-228 ~]# rpm -qa|grep glusterfs glusterfs-libs-3.7.5-9.el7rhgs.x86_64 glusterfs-fuse-3.7.5-9.el7rhgs.x86_64 glusterfs-rdma-3.7.5-9.el7rhgs.x86_64 glusterfs-3.7.5-9.el7rhgs.x86_64 glusterfs-api-3.7.5-9.el7rhgs.x86_64 glusterfs-cli-3.7.5-9.el7rhgs.x86_64 glusterfs-geo-replication-3.7.5-9.el7rhgs.x86_64 glusterfs-debuginfo-3.7.5-9.el7rhgs.x86_64 glusterfs-client-xlators-3.7.5-9.el7rhgs.x86_64 glusterfs-server-3.7.5-9.el7rhgs.x86_64 [root@dhcp35-228 ~]# gluster snapshot create test_snap tiervolume no-timestamp snapshot create: success: Snap test_snap created successfully [root@dhcp35-228 ~]# gluster snapshot activate test_snap Snapshot activate: test_snap: Snap activated successfully [root@dhcp35-228 ~]# gluster snapshot clone test_clone test_snap snapshot clone: success: Clone test_clone created successfully [root@dhcp35-228 ~]# gluster snapshot clone test_clone2 test_snap snapshot clone: success: Clone test_clone2 created successfully [root@dhcp35-228 ~]# gluster snapshot clone test_clone3 test_snap snapshot clone: success: Clone test_clone3 created successfully [root@dhcp35-228 ~]# gluster snapshot clone test_clone4 test_snap snapshot clone: success: Clone test_clone4 created successfully [root@dhcp35-228 ~]# gluster snapshot clone test_clone5 test_snap snapshot clone: success: Clone test_clone5 created successfully Note: Clone creation has been performed while promotion/demotion of files are going on. Since we are not facing this bug now, Hence marking this bug as Verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0193.html