Fedora Account System
Red Hat Associate
Red Hat Customer
+++ This bug was initially created as a clone of Bug #874348 +++ Hi, i have two zfs Storage Backends with a thin provisoned volume and two GlusterFS Head Server. One unit are one Storage Backend and one GlusterFS Head Server. This volumes from each Storage Backend are provided to GlusterFS head Server over Infiniband SRP Target. On the GlusterFS Head the device /dev/sdx1 formated as xfs -i size=512 and mount to /brick The GlusterFS Volume are configured as replicate without special (only default) When a copy process startet and i unmount the mount point /brick at one GluserFS Server or i stop the gluster service to test the service all works fine no problems. But when i go to a Storage backend and set one lun to offline or remove a infiniband cable then crashed the complete gluster volume. On the client. Gluster volume mount to /mnt. When i try a "ls /mnt" i get a input/output error. After unmount -l and stop gluster volume and reconnect all devices i can start the gluster volume again. But the bricks are different and self heal not work. When i try to remove files from this mount point rm -rf /mnt/* i get some errors with missing files. I not sure if this error occur with a iscsi target. But with a Infiniband SRP Target LUN is that not nice. Many thanks and greetings from germany and sorry for my english :) Andreas Error message from Client [2012-11-08 00:46:22.091743] W [dict.c:439:dict_ref] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x35cd00f875] (-->/usr/lib64/glusterfs/3.3.1/xlator/protocol/client.so(client3_1_lookup_cbk+0x560) [0x7f56572ec040] (-->/usr/lib64/glusterfs/3.3.1/xlator/cluster/replicate.so(afr_lookup_cbk+0x218) [0x7f56570a39c8]))) 0-dict: dict is NULL [2012-11-08 00:46:22.091786] W [afr-common.c:1411:afr_conflicting_iattrs] 0-vol001-replicate-0: /test2/usr: filetype differs on subvolumes (0, 1) [2012-11-08 00:46:22.091817] W [afr-common.c:1411:afr_conflicting_iattrs] 0-vol001-replicate-0: /test2/usr: filetype differs on subvolumes (0, 1) [2012-11-08 00:46:22.091853] W [afr-common.c:1196:afr_detect_self_heal_by_iatt] 0-vol001-replicate-0: /test2/usr: gfid different on subvolume [2012-11-08 00:46:22.091958] E [afr-common.c:1146:afr_lookup_set_self_heal_params_by_xattr] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xad) [0x35cd01028d] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x35cd00f875] (-->/usr/lib64/glusterfs/3.3.1/xlator/protocol/client.so(client3_1_lookup_cbk+0x560) [0x7f56572ec040]))) 0-: Assertion failed: xattr --- Additional comment from Pranith Kumar K on 2012-11-12 00:53:24 EST --- hi Andreas, According to the log messages, it seems the entry /test2/usr has changed file type as in, Something like on one machine it could be file and on the other it could be dir. This is the reason you are observing Input/Output Error. "when i go to a Storage backend and set one lun to offline or remove a infiniband cable then crashed the complete gluster volume." What exactly do you mean by 'crashed the complete gluster volume'. Does it only mean Input/Output errors you are observing or something more? --- Additional comment from Andreas Huser on 2012-11-13 09:20:45 EST --- Hi Pranith, test2 it's only a test mount point with a copied folder usr. I have check some different environments and scenarios. The errors comes only with a ,,xfs" filesystem. I have no Problems with ext4. When xfs lost his device /dev/sdX, or whatever else which source (LUN,Disk, Fc-Lun, iSCSI, Raid Controller etc.) xfs make a xfs_do_force_shutdown. The result is a not usable replica mount point for clients. And the mounted volume in a client crash with I/O errors. To test this build a environment: Infiniband are not needed. 1.) two GlusterFS Server with two Harddrives one for system end one for glusterfs volume. 2.) A client to mount the glusterfs volume. Format each glusterfs Volume with xfs on server 1 and server 2. Create a replica volume and mount this on the client. Now copy a lot of data in this mount point. During the copy process pull the sata cable from one volume harddrive on the server 1 or 2 and wait a few second. Now you see the glusterfs volume on the client is crashed. You not can remount this volume. You must reset the failed device and restart the glustervolume. I hope that helps you Many greetings from germany Andreas ov 13 14:43:16 kvm01 sm-notify[2387]: Version 1.2.3 starting Nov 13 14:44:15 kvm01 kernel: ata4: exception Emask 0x10 SAct 0x0 SErr 0x90000 action 0xe frozen Nov 13 14:44:15 kvm01 kernel: ata4: irq_stat 0x00400000, PHY RDY changed Nov 13 14:44:15 kvm01 kernel: ata4: SError: { PHYRdyChg 10B8B } Nov 13 14:44:15 kvm01 kernel: ata4: hard resetting link Nov 13 14:44:15 kvm01 kernel: ata4: SATA link down (SStatus 0 SControl 300) Nov 13 14:44:20 kvm01 kernel: ata4: hard resetting link Nov 13 14:44:21 kvm01 kernel: ata4: SATA link down (SStatus 0 SControl 300) Nov 13 14:44:21 kvm01 kernel: ata4: limiting SATA link speed to 1.5 Gbps Nov 13 14:44:26 kvm01 kernel: ata4: hard resetting link Nov 13 14:44:26 kvm01 kernel: ata4: SATA link down (SStatus 0 SControl 310) Nov 13 14:44:26 kvm01 kernel: ata4.00: disabled Nov 13 14:44:26 kvm01 kernel: ata4: EH complete Nov 13 14:44:26 kvm01 kernel: ata4.00: detaching (SCSI 3:0:0:0) Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] killing request Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: rejecting I/O to offline device Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] Unhandled error code Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] CDB: Write(10): 2a 00 12 8e 1e 1f 00 00 08 00 Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108397 Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1 Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108439 Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1 Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108465 Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1 Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108571 Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1 Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108620 Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1 Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108621 Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1 Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108622 Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1 Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108623 Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1 Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108624 Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1 Nov 13 14:44:26 kvm01 kernel: Buffer I/O error on device sdb1, logical block 61108625 Nov 13 14:44:26 kvm01 kernel: lost page write due to I/O error on sdb1 Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): Device sdb1: metadata write error block 0xe8e24b0 Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): I/O error occurred: meta-data dev sdb1 block 0x1d1c4e19 ("xlog_iodone") error 5 buf count 32768 Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): xfs_do_force_shutdown(0x2) called from line 891 of file fs/xfs/xfs_log.c. Return address = 0xffffffffa087c8dc Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): Log I/O Error Detected. Shutting down filesystem Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): Please umount the filesystem and rectify the problem(s) Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): xfs_log_force: error 5 returned. Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): xfs_log_force: error 5 returned. Nov 13 14:44:26 kvm01 kernel: XFS (sdb1): xfs_do_force_shutdown(0x1) called from line 1056 of file fs/xfs/linux-2.6/xfs_buf.c. Return address = 0xffffffffa0898283 Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] Synchronizing SCSI cache Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] Stopping disk Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] START_STOP FAILED Nov 13 14:44:26 kvm01 kernel: sd 3:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK Nov 13 14:44:38 kvm01 kernel: XFS (sdb1): xfs_log_force: error 5 returned.
Not a valid deployment scenario for RHS. Closing this bug.
I'm not understand why it is not a bug? I have two server in replicated mode. One server lost a harddrive or cable or controller or an other part. The complete high availability of this replicated volume are lost. RedHat recommend format the volume with xfs. But with xfs i have a faulty volume when one server have a problem. Is GlusterFS in replicated mode not for high availability? Or what make i wrong? Many thanks Andreas
(In reply to comment #3) > I'm not understand why it is not a bug? > I have two server in replicated mode. One server lost a harddrive or cable > or controller or an other part. The complete high availability of this > replicated volume are lost. RedHat recommend format the volume with xfs. But > with xfs i have a faulty volume when one server have a problem. > Is GlusterFS in replicated mode not for high availability? > Or what make i wrong? > Please note that this bug is not relevant to product Red Hat Storage. For GlusterFS, this is still relevant and the issue will continue to be tracked through the original bug https://bugzilla.redhat.com/show_bug.cgi?id=874348 Thanks, Vijay
okay i understand sorry. it gives two reports one for glusterfs and one for Red Hat Storage. And the bug report for glusterfs is still open. thanks Andreas