Bug 878874

Summary: mount point broke on client when a lun from a storage backend offline or missing . After there the data are scrap
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Vidya Sakar <vinaraya>
Component: glusterfsAssignee: Pranith Kumar K <pkarampu>
Status: CLOSED NOTABUG QA Contact: Sachidananda Urs <surs>
Severity: urgent Docs Contact:
Priority: medium    
Version: 2.0CC: ahuser, gluster-bugs, rfortier, rhs-bugs, shaines, vbellur
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 874348 Environment:
Last Closed: 2012-12-11 09:52:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 874348    
Bug Blocks:    

Description Vidya Sakar 2012-11-21 12:59:55 UTC 
+++ This bug was initially created as a clone of Bug #874348 +++

Hi, 

i have two zfs Storage Backends with a thin provisoned volume and two GlusterFS Head Server. One unit are one Storage Backend and one GlusterFS Head Server.

This volumes from each Storage Backend are provided to GlusterFS head Server over Infiniband SRP Target.

On the GlusterFS Head the device /dev/sdx1 formated as xfs -i size=512 and mount to /brick

The GlusterFS Volume are configured as replicate without special (only default)

When a copy process startet and i unmount the mount point /brick at one GluserFS Server or i stop the gluster service to test the service all works fine no problems.
But when i go to a Storage backend and set one lun to offline or remove a infiniband cable then crashed the complete gluster volume.

On the client.  Gluster volume mount to /mnt.  When i try a "ls /mnt" i get a input/output error.  After unmount -l and stop gluster volume and reconnect all devices i can start the gluster volume again. But the bricks are different and self heal not work. When i try to remove files from this mount point rm -rf /mnt/*  i get some errors with missing files.

I not sure if this error occur with a iscsi target. But with a Infiniband SRP Target LUN is that not nice.

Many thanks and greetings from germany and sorry for my english :)
Andreas

Error message from Client

[2012-11-08 00:46:22.091743] W [dict.c:439:dict_ref] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x35cd00f875] (-->/usr/lib64/glusterfs/3.3.1/xlator/protocol/client.so(client3_1_lookup_cbk+0x560) [0x7f56572ec040] (-->/usr/lib64/glusterfs/3.3.1/xlator/cluster/replicate.so(afr_lookup_cbk+0x218) [0x7f56570a39c8]))) 0-dict: dict is NULL
[2012-11-08 00:46:22.091786] W [afr-common.c:1411:afr_conflicting_iattrs] 0-vol001-replicate-0: /test2/usr: filetype differs on subvolumes (0, 1)
[2012-11-08 00:46:22.091817] W [afr-common.c:1411:afr_conflicting_iattrs] 0-vol001-replicate-0: /test2/usr: filetype differs on subvolumes (0, 1)
[2012-11-08 00:46:22.091853] W [afr-common.c:1196:afr_detect_self_heal_by_iatt] 0-vol001-replicate-0: /test2/usr: gfid different on subvolume
[2012-11-08 00:46:22.091958] E [afr-common.c:1146:afr_lookup_set_self_heal_params_by_xattr] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xad) [0x35cd01028d] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x35cd00f875] (-->/usr/lib64/glusterfs/3.3.1/xlator/protocol/client.so(client3_1_lookup_cbk+0x560) [0x7f56572ec040]))) 0-: Assertion failed: xattr

--- Additional comment from Pranith Kumar K on 2012-11-12 00:53:24 EST ---

hi Andreas,
   According to the log messages, it seems the entry /test2/usr has changed file type as in, Something like on one machine it could be file and on the other it could be dir. This is the reason you are observing Input/Output Error.

"when i go to a Storage backend and set one lun to offline or remove a infiniband cable then crashed the complete gluster volume."

What exactly do you mean by 'crashed the complete gluster volume'. Does it only mean Input/Output errors you are observing or something more?

--- Additional comment from Andreas Huser on 2012-11-13 09:20:45 EST ---

Hi Pranith, 

test2 it's only a test mount point with a copied folder usr.
I have check  some different environments and scenarios. The errors comes only with a ,,xfs" filesystem. I have no Problems with ext4.

When xfs lost his device /dev/sdX, or whatever else which source (LUN,Disk, Fc-Lun, iSCSI, Raid Controller etc.) xfs make a xfs_do_force_shutdown. The result is a not usable replica mount point for clients. And the mounted volume in a client crash with I/O errors.

To test this build a environment: Infiniband are not needed.
1.) two GlusterFS Server with two Harddrives 
one for system end one for glusterfs volume.
2.) A client to mount the glusterfs volume.

Format each glusterfs Volume with xfs on server 1 and server 2. Create a replica volume and mount this on the client. Now copy a lot of data in this mount point.  During the copy process pull the sata cable from one volume harddrive on the server 1 or 2 and wait a few second. Now you see the glusterfs volume on the client is crashed. You not can remount this volume. You must reset the failed device and restart the glustervolume. 

I hope that helps you 
Many greetings from germany 
Andreas


ov 13 14:43:16 kvm01 sm-notify[2387]: Version 1.2.3 starting
Nov 13 14:44:15 kvm01 kernel: ata4: exception Emask 0x10 SAct 0x0 SErr 0x90000 action 0xe frozen
Nov 13 14:44:15 kvm01 kernel: ata4: irq_stat 0x00400000, PHY RDY changed
Nov 13 14:44:15 kvm01 kernel: ata4: SError: { PHYRdyChg 10B8B }
Nov 13 14:44:15 kvm01 kernel: ata4: hard resetting link
Nov 13 14:44:15 kvm01 kernel: ata4: SATA link down (SStatus 0 SControl 300)
Nov 13 14:44:20 kvm01 kernel: ata4: hard resetting link
Nov 13 14:44:21 kvm01 kernel: ata4: SATA link down (SStatus 0 SControl 300)
Nov 13 14:44:21 kvm01 kernel: ata4: limiting SATA link speed to 1.5 Gbps
Nov 13 14:44:26 kvm01 kernel: ata4: hard resetting link
Nov 13 14:44:26 kvm01 kernel: ata4: SATA link down (SStatus 0 SControl 310)
Nov 13 14:44:26 kvm01 kernel: ata4.00: disabled
Nov 13 14:44:26 kvm01 kernel: ata4: EH complete
Nov 13 14:44:26 kvm01 kernel: ata4.00: detaching (SCSI 3:0:0:0)
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] killing request
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] Unhandled error code
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] CDB: Write(10): 2a 00 12 8e 1e 1f 00 00 08 00
Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108397
Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1
Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108439
Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1
Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108465
Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1
Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108571
Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1
Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108620
Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1
Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108621
Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1
Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108622
Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1
Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108623
Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1
Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108624
Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1
Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108625
Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1
Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): Device sdb1: metadata write error block 0xe8e24b0
Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): I/O error occurred: meta-data dev sdb1 block 0x1d1c4e19       ("xlog_iodone") error 5 buf count 32768
Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): xfs_do_force_shutdown(0x2) called from line 891 of file fs/xfs/xfs_log.c.  Return address = 0xffffffffa087c8dc
Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): Log I/O Error Detected.  Shutting down filesystem
Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): Please umount the filesystem and rectify the problem(s)
Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): xfs_log_force: error 5 returned.
Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): xfs_log_force: error 5 returned.
Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): xfs_do_force_shutdown(0x1) called from line 1056 of file fs/xfs/linux-2.6/xfs_buf.c.  Return address = 0xffffffffa0898283
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] Synchronizing SCSI cache
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] Stopping disk
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] START_STOP FAILED
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Nov 13 14:44:38 kvm01 kernel: XFS (sdb1): xfs_log_force: error 5 returned.
Comment 2 Vijay Bellur 2012-12-11 09:52:56 UTC 
Not a valid deployment scenario for RHS. Closing this bug.
Comment 3 Andreas Huser 2012-12-11 10:24:45 UTC 
I'm not understand why it is not a bug?
I have two server in replicated mode. One server lost a harddrive or cable or controller or an other part. The complete high availability of this replicated volume are lost. RedHat recommend format the volume with xfs. But with xfs i have a faulty volume when one server have a problem.
Is GlusterFS in replicated mode not for high availability?
Or what make i wrong?

Many thanks
Andreas
Comment 4 Vijay Bellur 2012-12-11 11:23:35 UTC 
(In reply to comment #3)
> I'm not understand why it is not a bug?
> I have two server in replicated mode. One server lost a harddrive or cable
> or controller or an other part. The complete high availability of this
> replicated volume are lost. RedHat recommend format the volume with xfs. But
> with xfs i have a faulty volume when one server have a problem.
> Is GlusterFS in replicated mode not for high availability?
> Or what make i wrong?
> 

Please note that this bug is not relevant to product Red Hat Storage. For GlusterFS, this is still relevant and the issue will continue to be tracked through the original bug https://bugzilla.redhat.com/show_bug.cgi?id=874348

Thanks,
Vijay
Comment 5 Andreas Huser 2012-12-11 11:34:15 UTC 
okay i understand sorry. it gives two reports one for glusterfs and one for Red Hat Storage. And the bug report for glusterfs is still open.

thanks
Andreas