Created attachment 788316 [details] ext4 corruption snapshot Description of problem: After adding a pair of bricks to the distributed-replicate volume and following rebalance lead to ext4 file system corruption on the app vm. Version-Release number of selected component (if applicable): [root@rhs1-beta vss3]# rpm -qa | grep gluster glusterfs-geo-replication-3.4.0.19rhs-2.el6rhs.x86_64 glusterfs-rdma-3.4.0.19rhs-2.el6rhs.x86_64 gluster-swift-container-1.8.0-6.3.el6rhs.noarch glusterfs-libs-3.4.0.19rhs-2.el6rhs.x86_64 glusterfs-fuse-3.4.0.19rhs-2.el6rhs.x86_64 glusterfs-api-3.4.0.19rhs-2.el6rhs.x86_64 glusterfs-server-3.4.0.19rhs-2.el6rhs.x86_64 glusterfs-devel-3.4.0.19rhs-2.el6rhs.x86_64 gluster-swift-1.8.0-6.3.el6rhs.noarch gluster-swift-proxy-1.8.0-6.3.el6rhs.noarch gluster-swift-account-1.8.0-6.3.el6rhs.noarch vdsm-gluster-4.10.2-23.0.1.el6rhs.noarch glusterfs-3.4.0.19rhs-2.el6rhs.x86_64 glusterfs-api-devel-3.4.0.19rhs-2.el6rhs.x86_64 glusterfs-debuginfo-3.4.0.19rhs-2.el6rhs.x86_64 gluster-swift-object-1.8.0-6.3.el6rhs.noarch gluster-swift-plugin-1.8.0-2.el6rhs.noarch How reproducible: frequently "NOT" reproducible Steps to Reproduce: 1.I had a 7x2 distributed-replicate volume 2.created a storage domain out of the above created volume and created 5 vms on it 3. added one more pair of brick and started rebalance, while rebalance is running i/o on the vms we happening using for i in {1..100000} do dd if=/dev/urandom of=$i bs=512k count=1 done 4. rebalance ran to completion 5. at some point in time one of the vm had ext4 corruption Actual results: one of the app vm had ext4 corruption Expected results: No vm corruption should happen Additional info: Volume Name: vmstore Type: Distributed-Replicate Volume ID: 10b93f79-2a1d-4737-8632-05f57c97db93 Status: Started Number of Bricks: 8 x 2 = 16 Transport-type: tcp Bricks: Brick1: 10.70.37.113:/brick1/vss1 Brick2: 10.70.37.133:/brick1/vss1 Brick3: 10.70.37.113:/brick2/vss2 Brick4: 10.70.37.133:/brick2/vss2 Brick5: 10.70.37.113:/brick3/vss3 Brick6: 10.70.37.133:/brick3/vss3 Brick7: 10.70.37.113:/brick4/vss4 Brick8: 10.70.37.133:/brick4/vss4 Brick9: 10.70.37.113:/brick4/vss5 Brick10: 10.70.37.133:/brick5/vss5 Brick11: 10.70.37.113:/brick6/vss6 Brick12: 10.70.37.133:/brick6/vss6 Brick13: 10.70.37.113:/brick1/vss7 Brick14: 10.70.37.133:/brick1/vss7 Brick15: 10.70.37.113:/brick1/vss8 Brick16: 10.70.37.133:/brick1/vss8 Options Reconfigured: storage.owner-gid: 36 storage.owner-uid: 36 network.remote-dio: enable cluster.eager-lock: enable performance.stat-prefetch: off performance.io-cache: off performance.read-ahead: off performance.quick-read: off cluster info ============ RHS nodes --------- 10.70.37.113 10.70.37.133 Hypervisor ========== rhs-client36.lab.eng.blr.redhat.com Mount point =========== /rhev/data-center/mnt/10.70.37.113:vmstore Mount log messages =================== 2013-08-14 12:08:13.805531] I [client.c:2103:client_rpc_notify] 0-vmstore-client-11: disconnected from 10.70.37.133:49170. Client pro cess will keep trying to connect to glusterd until brick's port is available. [2013-08-14 12:08:13.805540] E [afr-common.c:3832:afr_notify] 0-vmstore-replicate-5: All subvolumes are down. Going offline until atle ast one of them comes back up. [2013-08-14 12:08:13.805556] I [client.c:2103:client_rpc_notify] 0-vmstore-client-12: disconnected from 10.70.37.113:49164. Client pro cess will keep trying to connect to glusterd until brick's port is available. [2013-08-14 12:08:13.805574] I [client.c:2103:client_rpc_notify] 0-vmstore-client-13: disconnected from 10.70.37.133:49171. Client pro cess will keep trying to connect to glusterd until brick's port is available. [2013-08-14 12:08:13.805583] E [afr-common.c:3832:afr_notify] 0-vmstore-replicate-6: All subvolumes are down. Going offline until atle ast one of them comes back up. [2013-08-14 12:08:13.806891] W [client-rpc-fops.c:2604:client3_3_lookup_cbk] 1-vmstore-client-14: remote operation failed: Permission denied. Path: /05ba73ee-552a-4eb4-9368-6db52bac31ef (00000000-0000-0000-0000-000000000000) [2013-08-14 12:08:13.807476] W [client-rpc-fops.c:2604:client3_3_lookup_cbk] 1-vmstore-client-14: remote operation failed: Permission denied. Path: /05ba73ee-552a-4eb4-9368-6db52bac31ef (00000000-0000-0000-0000-000000000000) [2013-08-14 12:08:13.813842] I [dht-layout.c:633:dht_layout_normalize] 1-vmstore-dht: found anomalies in /05ba73ee-552a-4eb4-9368-6db5 2bac31ef. holes=1 overlaps=0 missing=0 down=0 misc=1 [2013-08-14 12:08:13.813876] W [dht-selfheal.c:916:dht_selfheal_directory] 1-vmstore-dht: 1 subvolumes have unrecoverable errors [2013-08-14 12:08:13.814402] I [dht-layout.c:633:dht_layout_normalize] 1-vmstore-dht: found anomalies in /05ba73ee-552a-4eb4-9368-6db52bac31ef. holes=1 overlaps=0 missing=0 down=0 misc=1 [2013-08-14 12:08:13.814421] W [dht-selfheal.c:916:dht_selfheal_directory] 1-vmstore-dht: 1 subvolumes have unrecoverable errors [2013-08-14 12:08:13.815177] W [client-rpc-fops.c:2604:client3_3_lookup_cbk] 1-vmstore-client-14: remote operation failed: Permission denied. Path: /05ba73ee-552a-4eb4-9368-6db52bac31ef (140eaaf5-c667-4a71-aef1-a69a50c249b0) [2013-08-14 12:08:13.815222] I [dht-common.c:567:dht_revalidate_cbk] 1-vmstore-dht: subvolume vmstore-replicate-7 for /05ba73ee-552a-4eb4-9368-6db52bac31ef returned -1 (Permission denied) [2013-08-14 12:08:13.815493] W [client-rpc-fops.c:2604:client3_3_lookup_cbk] 1-vmstore-client-14: remote operation failed: Permission denied. Path: /05ba73ee-552a-4eb4-9368-6db52bac31ef (140eaaf5-c667-4a71-aef1-a69a50c249b0) [2013-08-14 12:08:13.815510] I [dht-common.c:567:dht_revalidate_cbk] 1-vmstore-dht: subvolume vmstore-replicate-7 for /05ba73ee-552a-4eb4-9368-6db52bac31ef returned -1 (Permission denied) [2013-08-14 12:08:13.834774] I [dht-layout.c:633:dht_layout_normalize] 1-vmstore-dht: found anomalies in <gfid:140eaaf5-c667-4a71-aef1-a69a50c249b0>. holes=1 overlaps=0 missing=1 down=0 misc=0 [2013-08-14 12:08:13.835080] I [dht-layout.c:633:dht_layout_normalize] 1-vmstore-dht: found anomalies in <gfid:140eaaf5-c667-4a71-aef1-a69a50c249b0>. holes=1 overlaps=0 missing=1 down=0 misc=0 [2013-08-14 12:08:13.835545] W [client-rpc-fops.c:519:client3_3_stat_cbk] 1-vmstore-client-14: remote operation failed: No such file or directory [2013-08-14 12:08:13.836324] W [client-rpc-fops.c:807:client3_3_statfs_cbk] 1-vmstore-client-14: remote operation failed: No such file there are some permission denied errors on one of the brick. Brick15 and Brick16 are the newly added bricks and then rebalance was invoked No vm pausing is seen, but ext4 curruption was seen on the vm (attached the ext4 corrutpion message snapshot)
Please provide the ls -l output from all the bricks. Also the same o/p from all bricks for this path <brick-path>/.glusterfs/05/ba/05ba73ee-552a-4eb4-9368-6db52bac31ef
I don't have the setup now. I will try to reproduce the bug and will let you know.
Targeting for 2.1.z (Big Bend) U1.
https://code.engineering.redhat.com/gerrit/#/c/16039/ should fix this. All rebalance issues open should be fixed with above patch.
Tested with glusterfs-3.4.0.55rhs-1 and RHEVM IS31 ( 3.3.0-0.44.el6ev) Performed the following step using RHEVM UI, 1. Created a 4 Node cluster 2. Created a distribute replicate volume with 2X2 3. Optimized the volume for virt-store (i.e) gluster volume set <vol-name> group virt Changed the ownership to 36:36 (i.e) gluster voume set <vol-name> storage.owner-uid 36 gluster voume set <vol-name> storage.owner-gid 36 4. Started the volume 5. Fuse mounted the volume 6. Created 3 raw images each of size 30GB 7. Created 3 App VMs with the above created images and then installed them with RHEL 6.5 8. Ran the cotinuous load on App VMs using dd command as follows !/bin/bash vols=(/home/vol1 /home/vol2 /home/vol3 /home/vol4 /home/vol5) while true do for vol in ${vols[@]};do #touch some files touch ${vol}/file{1,2,3,4,5,6,7,8,9} # dd some files of varying size for k in 10 50 100 200 500 700 1000 10000 20000 do dd if=/dev/urandom of=${vol}/ddfile${k} bs=128k count=$k done # remove files rm -rf ${vol}/* done done 9. Added more bricks to the volume 10. Started rebalance App Vms are healthy after rebalance operations
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2014-0208.html