Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 874348

Summary:	mount point broke on client when a lun from a storage backend offline or missing . After there the data are scrap
Product:	[Community] GlusterFS	Reporter:	Andreas Huser <ahuser>
Component:	replicate	Assignee:	Pranith Kumar K <pkarampu>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	ahuser, bfoster, gluster-bugs, jdarcy
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.4.0	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	878874 (view as bug list)		Environment:
Last Closed:	2013-07-24 17:55:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	878874

Description Andreas Huser 2012-11-08 01:11:47 UTC

Hi, 

i have two zfs Storage Backends with a thin provisoned volume and two GlusterFS Head Server. One unit are one Storage Backend and one GlusterFS Head Server.

This volumes from each Storage Backend are provided to GlusterFS head Server over Infiniband SRP Target.

On the GlusterFS Head the device /dev/sdx1 formated as xfs -i size=512 and mount to /brick

The GlusterFS Volume are configured as replicate without special (only default)

When a copy process startet and i unmount the mount point /brick at one GluserFS Server or i stop the gluster service to test the service all works fine no problems.
But when i go to a Storage backend and set one lun to offline or remove a infiniband cable then crashed the complete gluster volume.

On the client.  Gluster volume mount to /mnt.  When i try a "ls /mnt" i get a input/output error.  After unmount -l and stop gluster volume and reconnect all devices i can start the gluster volume again. But the bricks are different and self heal not work. When i try to remove files from this mount point rm -rf /mnt/*  i get some errors with missing files.

I not sure if this error occur with a iscsi target. But with a Infiniband SRP Target LUN is that not nice.

Many thanks and greetings from germany and sorry for my english :)
Andreas

Error message from Client

[2012-11-08 00:46:22.091743] W [dict.c:439:dict_ref] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x35cd00f875] (-->/usr/lib64/glusterfs/3.3.1/xlator/protocol/client.so(client3_1_lookup_cbk+0x560) [0x7f56572ec040] (-->/usr/lib64/glusterfs/3.3.1/xlator/cluster/replicate.so(afr_lookup_cbk+0x218) [0x7f56570a39c8]))) 0-dict: dict is NULL
[2012-11-08 00:46:22.091786] W [afr-common.c:1411:afr_conflicting_iattrs] 0-vol001-replicate-0: /test2/usr: filetype differs on subvolumes (0, 1)
[2012-11-08 00:46:22.091817] W [afr-common.c:1411:afr_conflicting_iattrs] 0-vol001-replicate-0: /test2/usr: filetype differs on subvolumes (0, 1)
[2012-11-08 00:46:22.091853] W [afr-common.c:1196:afr_detect_self_heal_by_iatt] 0-vol001-replicate-0: /test2/usr: gfid different on subvolume
[2012-11-08 00:46:22.091958] E [afr-common.c:1146:afr_lookup_set_self_heal_params_by_xattr] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xad) [0x35cd01028d] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x35cd00f875] (-->/usr/lib64/glusterfs/3.3.1/xlator/protocol/client.so(client3_1_lookup_cbk+0x560) [0x7f56572ec040]))) 0-: Assertion failed: xattr

Comment 1 Pranith Kumar K 2012-11-12 05:53:24 UTC

hi Andreas,
   According to the log messages, it seems the entry /test2/usr has changed file type as in, Something like on one machine it could be file and on the other it could be dir. This is the reason you are observing Input/Output Error.

"when i go to a Storage backend and set one lun to offline or remove a infiniband cable then crashed the complete gluster volume."

What exactly do you mean by 'crashed the complete gluster volume'. Does it only mean Input/Output errors you are observing or something more?

Comment 2 Andreas Huser 2012-11-13 14:20:45 UTC

Hi Pranith, 

test2 it's only a test mount point with a copied folder usr.
I have check  some different environments and scenarios. The errors comes only with a ,,xfs" filesystem. I have no Problems with ext4.

When xfs lost his device /dev/sdX, or whatever else which source (LUN,Disk, Fc-Lun, iSCSI, Raid Controller etc.) xfs make a xfs_do_force_shutdown. The result is a not usable replica mount point for clients. And the mounted volume in a client crash with I/O errors.

To test this build a environment: Infiniband are not needed.
1.) two GlusterFS Server with two Harddrives 
one for system end one for glusterfs volume.
2.) A client to mount the glusterfs volume.

Format each glusterfs Volume with xfs on server 1 and server 2. Create a replica volume and mount this on the client. Now copy a lot of data in this mount point.  During the copy process pull the sata cable from one volume harddrive on the server 1 or 2 and wait a few second. Now you see the glusterfs volume on the client is crashed. You not can remount this volume. You must reset the failed device and restart the glustervolume. 

I hope that helps you 
Many greetings from germany 
Andreas


ov 13 14:43:16 kvm01 sm-notify[2387]: Version 1.2.3 starting
Nov 13 14:44:15 kvm01 kernel: ata4: exception Emask 0x10 SAct 0x0 SErr 0x90000 action 0xe frozen
Nov 13 14:44:15 kvm01 kernel: ata4: irq_stat 0x00400000, PHY RDY changed
Nov 13 14:44:15 kvm01 kernel: ata4: SError: { PHYRdyChg 10B8B }
Nov 13 14:44:15 kvm01 kernel: ata4: hard resetting link
Nov 13 14:44:15 kvm01 kernel: ata4: SATA link down (SStatus 0 SControl 300)
Nov 13 14:44:20 kvm01 kernel: ata4: hard resetting link
Nov 13 14:44:21 kvm01 kernel: ata4: SATA link down (SStatus 0 SControl 300)
Nov 13 14:44:21 kvm01 kernel: ata4: limiting SATA link speed to 1.5 Gbps
Nov 13 14:44:26 kvm01 kernel: ata4: hard resetting link
Nov 13 14:44:26 kvm01 kernel: ata4: SATA link down (SStatus 0 SControl 310)
Nov 13 14:44:26 kvm01 kernel: ata4.00: disabled
Nov 13 14:44:26 kvm01 kernel: ata4: EH complete
Nov 13 14:44:26 kvm01 kernel: ata4.00: detaching (SCSI 3:0:0:0)
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] killing request
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] Unhandled error code
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] CDB: Write(10): 2a 00 12 8e 1e 1f 00 00 08 00
Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108397
Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1
Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108439
Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1
Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108465
Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1
Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108571
Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1
Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108620
Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1
Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108621
Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1
Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108622
Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1
Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108623
Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1
Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108624
Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1
Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108625
Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1
Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): Device sdb1: metadata write error block 0xe8e24b0
Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): I/O error occurred: meta-data dev sdb1 block 0x1d1c4e19       ("xlog_iodone") error 5 buf count 32768
Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): xfs_do_force_shutdown(0x2) called from line 891 of file fs/xfs/xfs_log.c.  Return address = 0xffffffffa087c8dc
Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): Log I/O Error Detected.  Shutting down filesystem
Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): Please umount the filesystem and rectify the problem(s)
Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): xfs_log_force: error 5 returned.
Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): xfs_log_force: error 5 returned.
Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): xfs_do_force_shutdown(0x1) called from line 1056 of file fs/xfs/linux-2.6/xfs_buf.c.  Return address = 0xffffffffa0898283
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] Synchronizing SCSI cache
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] Stopping disk
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] START_STOP FAILED
Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Nov 13 14:44:38 kvm01 kernel: XFS (sdb1): xfs_log_force: error 5 returned.

Comment 3 Pranith Kumar K 2013-03-21 12:52:49 UTC

Brian,
    This seems a lot like the bug 892730. Could you take a look at these xfs logs and confirm that this is the same.

Here is the commit log of the bug to refresh your memory.

Pranith.

commit 679cb2399fc1f8e97f2b29654ec422f267b03783
Author: Brian Foster <bfoster>
Date:   Thu Jan 10 10:49:17 2013 -0500

    afr: conditionally prioritize EIO errors over ENOENT
    
    The most important errno logic historically only prioritized ESTALE
    over ENOENT. Commit c8c0942d added EIO prioritization over ENOENT
    to ensure that split-brain was reported when it occurs in
    conjunction with bricks missing the file entry. The unintended side
    effect of this change is that (non split-brain) EIO errors reported
    from the bricks themselves are now reported to the client when the
    expectation is that afr should squash said errors in favor of
    marking the file inconsistent.
    
    The high-level problem is that EIO is overloaded with different
    meanings from different contexts. This commit adds an eio parameter
    to the errno priority logic to conditionally flag when EIO is of
    higher priority and should be propagated to the client.
    
    BUG: 892730

Comment 4 Brian Foster 2013-03-24 23:42:37 UTC

(In reply to comment #3)
> Brian,
>     This seems a lot like the bug 892730. Could you take a look at these xfs
> logs and confirm that this is the same.
> 

It's certainly similar and could be a factor. I think the caveat to note is that 82730 addressed an error prioritization issue as opposed to an error suppression issue. If I recall correctly, the invalid exposure of EIO was tied to a lookup failure which was expected to fail, but with ENOENT instead of EIO.

I suppose that could be source of errors here if a fileset copy is in progress. I don't recall seeing such a blatant problem as ls failure on the mountpoint however. This might be worth attempting to simulate in a VM.

> Here is the commit log of the bug to refresh your memory.
> 
> Pranith.
> 
> commit 679cb2399fc1f8e97f2b29654ec422f267b03783
> Author: Brian Foster <bfoster>
> Date:   Thu Jan 10 10:49:17 2013 -0500
> 
>     afr: conditionally prioritize EIO errors over ENOENT
>     
>     The most important errno logic historically only prioritized ESTALE
>     over ENOENT. Commit c8c0942d added EIO prioritization over ENOENT
>     to ensure that split-brain was reported when it occurs in
>     conjunction with bricks missing the file entry. The unintended side
>     effect of this change is that (non split-brain) EIO errors reported
>     from the bricks themselves are now reported to the client when the
>     expectation is that afr should squash said errors in favor of
>     marking the file inconsistent.
>     
>     The high-level problem is that EIO is overloaded with different
>     meanings from different contexts. This commit adds an eio parameter
>     to the errno priority logic to conditionally flag when EIO is of
>     higher priority and should be propagated to the client.
>     
>     BUG: 892730

Comment 5 Pranith Kumar K 2013-03-25 09:08:25 UTC

Andreas Huser,
      The bug is most probably fixed by Brian's patch. Could you let us know the results with 3.4 alpha. You can install them through the packages at http://download.gluster.org/pub/gluster/glusterfs/qa-releases/3.4.0alpha/

Pranith.

Comment 6 Andreas Huser 2013-03-25 09:28:26 UTC

Hi Pranith, 

gladly i do this, but i must build a new test environment. this can take a few days.
Many thanks for your great work!  

Regards Andreas

Comment 7 Pranith Kumar K 2013-03-25 09:32:07 UTC

Andreas Huser,
     Please add a comment with your findings after you get to test it.

Pranith

Comment 8 Pranith Kumar K 2013-04-04 08:45:19 UTC

Andrea,
     Please feel free to re-open the bug if you think that previous patch does not address the problem. I am closing it for now.

Pranith

Comment 9 Andreas Huser 2013-04-04 08:51:19 UTC

Hi Pranith,

i'm sorry it's too much work at time. Next week it looks better.
I post the results as soon as posible. 

Many thanks 
Regards Andreas