Bug 1218589 - Snapshots failing on tiered volumes (with EC)
Summary: Snapshots failing on tiered volumes (with EC)
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: tiering
Version: mainline
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Mohammed Rafi KC
QA Contact: bugs@gluster.org
URL:
Whiteboard:
Depends On:
Blocks: 1229267 1260923
TreeView+ depends on / blocked
 
Reported: 2015-05-05 10:37 UTC by Anoop
Modified: 2015-12-01 16:45 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1229267 (view as bug list)
Environment:
Last Closed: 2015-08-06 07:30:41 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)
sosreports (16.16 MB, application/x-gzip)
2015-05-05 10:37 UTC, Anoop
no flags Details

Description Anoop 2015-05-05 10:37:38 UTC
Created attachment 1022152 [details]
sosreports

Description of problem:
Seeing failures while trying to snap a tiered volume 

Version-Release number of selected component (if applicable):
glusterfs-server-3.7dev-0.994.git0d36d4f.el6.x86_64


How reproducible:


Steps to Reproduce:
1. Created a tiered volume [Distribute (Hot) + EC (Cold)]
2. Mount the volume on the clinet and start linux untar on the mount
3. While the untar is happening, take snapshots

Actual results:
Snapshot are failing.
[2015-05-05 14:59:36.250915] I [glusterd-handler.c:1317:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-05-05 14:59:36.251808] I [glusterd-handler.c:1317:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-05-05 14:59:36.252740] I [glusterd-handler.c:1317:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-05-05 15:03:21.723471] I [socket.c:3432:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1)
[2015-05-05 15:03:21.723488] E [rpcsvc.c:1299:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1, Program: GlusterD svc cli, ProgVers: 2, Proc: 39) to rpc-transport (socket.management)
[2015-05-05 15:03:21.723499] E [glusterd-utils.c:409:glusterd_submit_reply] 0-: Reply submission failed
[2015-05-05 15:07:13.759341] W [glusterd-mgmt.c:190:gd_mgmt_v3_brick_op_fn] 0-management: snapshot brickop failed
[2015-05-05 15:07:13.759356] E [glusterd-mgmt.c:943:glusterd_mgmt_v3_brick_op] 0-management: Brick ops failed for operation Snapshot on local node
[2015-05-05 15:07:13.759362] E [glusterd-mgmt.c:2028:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Brick Ops Failed
[2015-05-05 15:08:19.961699] I [socket.c:3432:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1)
[2015-05-05 15:08:19.961717] E [rpcsvc.c:1299:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1, Program: GlusterD svc cli, ProgVers: 2, Proc: 39) to rpc-transport (socket.management)
[2015-05-05 15:08:19.961727] E [glusterd-utils.c:409:glusterd_submit_reply] 0-: Reply submission failed
[2015-05-05 15:10:36.465294] I [glusterd-handler.c:1317:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-05-05 15:10:36.466208] I [glusterd-handler.c:1317:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-05-05 15:10:36.467075] I [glusterd-handler.c:1317:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2015-05-05 15:12:33.198395] E [glusterd-op-sm.c:220:glusterd_get_txn_opinfo] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x3246822140] (--> /usr/lib64/glusterfs/3.7dev/xlator/mgmt/glusterd.so(glusterd_get_txn_opinfo+0x197)[0x7fa5e8f4a7b7] (--> /usr/lib64/glusterfs/3.7dev/xlator/mgmt/glusterd.so(__glusterd_handle_stage_op+0x1f0)[0x7fa5e8f2d9e0] (--> /usr/lib64/glusterfs/3.7dev/xlator/mgmt/glusterd.so(glusterd_big_locked_handler+0x3f)[0x7fa5e8f2ad7f] (--> /usr/lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x3246861c72] ))))) 0-management: Unable to get transaction opinfo for transaction ID : 5cc2bef3-3914-45eb-be68-529fbb2cb8d4


Expected results:
Snapshot create should e sucessful.

Additional info:

Attaching logs.

Comment 1 Mohammed Rafi KC 2015-07-31 07:48:58 UTC
I tried to reproduce this issue with latest master code. I'm able to create snapshots during an ongoing I/O on mount.

My test scenario


volume :>>>>

Volume Name: patchy
Type: Tier
Volume ID: 358a6e6a-c0a2-4e3d-8260-2b83ac28c4b5
Status: Started
Number of Bricks: 6
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distribute
Number of Bricks: 2
Brick1: 10.70.43.110:/d/backends/3/patchy_snap_mnt
Brick2: 10.70.43.100:/d/backends/3/patchy_snap_mnt
Cold Tier:
Cold Tier Type : Disperse
Number of Bricks: 1 x (3 + 1) = 4
Brick3: 10.70.43.100:/d/backends/1/patchy_snap_mnt
Brick4: 10.70.43.110:/d/backends/1/patchy_snap_mnt
Brick5: 10.70.43.100:/d/backends/2/patchy_snap_mnt
Brick6: 10.70.43.110:/d/backends/2/patchy_snap_mnt
Options Reconfigured:
cluster.tier-promote-frequency: 10
cluster.tier-demote-frequency: 10
cluster.write-freq-threshold: 0
cluster.read-freq-threshold: 0
performance.io-cache: off
performance.quick-read: off
features.ctr-enabled: on
performance.readdir-ahead: on



on mount point>>
tar -xvf /root/linux-4.1.2.tar.xz

on server (during I/O)>>

for i in {1..100} ; do  gluster snapshot create snap$i patchy no-timestamp;done;

Comment 2 Mohammed Rafi KC 2015-08-06 07:27:02 UTC
repeated the sa,e test on NFS mount also, an bith I/O and snapshot was success.


Note You need to log in before you can comment on or make changes to this bug.