Bug 998796 - [RHEV-RHS] : ext4 corruption on app vm after add-brick and rebalance
[RHEV-RHS] : ext4 corruption on app vm after add-brick and rebalance
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterfs (Show other bugs)
2.1
x86_64 Linux
medium Severity medium
: ---
: RHGS 2.1.2
Assigned To: Bug Updates Notification Mailing List
SATHEESARAN
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-20 01:52 EDT by shylesh
Modified: 2015-05-15 14:20 EDT (History)
8 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0.44.1u2rhs-1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
virt rhev rhs integration
Last Closed: 2014-02-25 02:35:02 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
ext4 corruption snapshot (192.58 KB, image/png)
2013-08-20 01:52 EDT, shylesh
no flags Details

  None (edit)
Description shylesh 2013-08-20 01:52:43 EDT
Created attachment 788316 [details]
ext4 corruption snapshot

Description of problem:
After adding a pair of bricks to the distributed-replicate volume and following rebalance lead to ext4 file system corruption on the app vm.

Version-Release number of selected component (if applicable):
[root@rhs1-beta vss3]# rpm -qa | grep gluster
glusterfs-geo-replication-3.4.0.19rhs-2.el6rhs.x86_64
glusterfs-rdma-3.4.0.19rhs-2.el6rhs.x86_64
gluster-swift-container-1.8.0-6.3.el6rhs.noarch
glusterfs-libs-3.4.0.19rhs-2.el6rhs.x86_64
glusterfs-fuse-3.4.0.19rhs-2.el6rhs.x86_64
glusterfs-api-3.4.0.19rhs-2.el6rhs.x86_64
glusterfs-server-3.4.0.19rhs-2.el6rhs.x86_64
glusterfs-devel-3.4.0.19rhs-2.el6rhs.x86_64
gluster-swift-1.8.0-6.3.el6rhs.noarch
gluster-swift-proxy-1.8.0-6.3.el6rhs.noarch
gluster-swift-account-1.8.0-6.3.el6rhs.noarch
vdsm-gluster-4.10.2-23.0.1.el6rhs.noarch
glusterfs-3.4.0.19rhs-2.el6rhs.x86_64
glusterfs-api-devel-3.4.0.19rhs-2.el6rhs.x86_64
glusterfs-debuginfo-3.4.0.19rhs-2.el6rhs.x86_64
gluster-swift-object-1.8.0-6.3.el6rhs.noarch
gluster-swift-plugin-1.8.0-2.el6rhs.noarch

How reproducible:

frequently "NOT" reproducible

Steps to Reproduce:
1.I had a  7x2 distributed-replicate volume
2.created a storage domain out of the above created volume and created 5 vms on it
3. added one more pair of brick and started rebalance, while rebalance is running i/o on the vms we happening using
for i in {1..100000}
do
dd if=/dev/urandom of=$i bs=512k count=1
done
4. rebalance ran to completion
5. at some point in time one of the vm had ext4 corruption

Actual results:
one of the app vm had ext4 corruption

Expected results:
No vm corruption should happen

Additional info:

Volume Name: vmstore
Type: Distributed-Replicate
Volume ID: 10b93f79-2a1d-4737-8632-05f57c97db93
Status: Started
Number of Bricks: 8 x 2 = 16
Transport-type: tcp
Bricks:
Brick1: 10.70.37.113:/brick1/vss1
Brick2: 10.70.37.133:/brick1/vss1
Brick3: 10.70.37.113:/brick2/vss2
Brick4: 10.70.37.133:/brick2/vss2
Brick5: 10.70.37.113:/brick3/vss3
Brick6: 10.70.37.133:/brick3/vss3
Brick7: 10.70.37.113:/brick4/vss4
Brick8: 10.70.37.133:/brick4/vss4
Brick9: 10.70.37.113:/brick4/vss5
Brick10: 10.70.37.133:/brick5/vss5
Brick11: 10.70.37.113:/brick6/vss6
Brick12: 10.70.37.133:/brick6/vss6
Brick13: 10.70.37.113:/brick1/vss7
Brick14: 10.70.37.133:/brick1/vss7
Brick15: 10.70.37.113:/brick1/vss8
Brick16: 10.70.37.133:/brick1/vss8
Options Reconfigured:
storage.owner-gid: 36
storage.owner-uid: 36
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off



cluster info
============
RHS nodes
---------
10.70.37.113
10.70.37.133


Hypervisor
==========
rhs-client36.lab.eng.blr.redhat.com

Mount point
===========
/rhev/data-center/mnt/10.70.37.113:vmstore



Mount log messages
===================
2013-08-14 12:08:13.805531] I [client.c:2103:client_rpc_notify] 0-vmstore-client-11: disconnected from 10.70.37.133:49170. Client pro
cess will keep trying to connect to glusterd until brick's port is available.
[2013-08-14 12:08:13.805540] E [afr-common.c:3832:afr_notify] 0-vmstore-replicate-5: All subvolumes are down. Going offline until atle
ast one of them comes back up.
[2013-08-14 12:08:13.805556] I [client.c:2103:client_rpc_notify] 0-vmstore-client-12: disconnected from 10.70.37.113:49164. Client pro
cess will keep trying to connect to glusterd until brick's port is available.
[2013-08-14 12:08:13.805574] I [client.c:2103:client_rpc_notify] 0-vmstore-client-13: disconnected from 10.70.37.133:49171. Client pro
cess will keep trying to connect to glusterd until brick's port is available.
[2013-08-14 12:08:13.805583] E [afr-common.c:3832:afr_notify] 0-vmstore-replicate-6: All subvolumes are down. Going offline until atle
ast one of them comes back up.
[2013-08-14 12:08:13.806891] W [client-rpc-fops.c:2604:client3_3_lookup_cbk] 1-vmstore-client-14: remote operation failed: Permission
denied. Path: /05ba73ee-552a-4eb4-9368-6db52bac31ef (00000000-0000-0000-0000-000000000000)
[2013-08-14 12:08:13.807476] W [client-rpc-fops.c:2604:client3_3_lookup_cbk] 1-vmstore-client-14: remote operation failed: Permission
denied. Path: /05ba73ee-552a-4eb4-9368-6db52bac31ef (00000000-0000-0000-0000-000000000000)
[2013-08-14 12:08:13.813842] I [dht-layout.c:633:dht_layout_normalize] 1-vmstore-dht: found anomalies in /05ba73ee-552a-4eb4-9368-6db5
2bac31ef. holes=1 overlaps=0 missing=0 down=0 misc=1
[2013-08-14 12:08:13.813876] W [dht-selfheal.c:916:dht_selfheal_directory] 1-vmstore-dht: 1 subvolumes have unrecoverable errors
[2013-08-14 12:08:13.814402] I [dht-layout.c:633:dht_layout_normalize] 1-vmstore-dht: found anomalies in /05ba73ee-552a-4eb4-9368-6db52bac31ef. holes=1 overlaps=0 missing=0 down=0 misc=1
[2013-08-14 12:08:13.814421] W [dht-selfheal.c:916:dht_selfheal_directory] 1-vmstore-dht: 1 subvolumes have unrecoverable errors
[2013-08-14 12:08:13.815177] W [client-rpc-fops.c:2604:client3_3_lookup_cbk] 1-vmstore-client-14: remote operation failed: Permission denied. Path: /05ba73ee-552a-4eb4-9368-6db52bac31ef (140eaaf5-c667-4a71-aef1-a69a50c249b0)
[2013-08-14 12:08:13.815222] I [dht-common.c:567:dht_revalidate_cbk] 1-vmstore-dht: subvolume vmstore-replicate-7 for /05ba73ee-552a-4eb4-9368-6db52bac31ef returned -1 (Permission denied)
[2013-08-14 12:08:13.815493] W [client-rpc-fops.c:2604:client3_3_lookup_cbk] 1-vmstore-client-14: remote operation failed: Permission denied. Path: /05ba73ee-552a-4eb4-9368-6db52bac31ef (140eaaf5-c667-4a71-aef1-a69a50c249b0)
[2013-08-14 12:08:13.815510] I [dht-common.c:567:dht_revalidate_cbk] 1-vmstore-dht: subvolume vmstore-replicate-7 for /05ba73ee-552a-4eb4-9368-6db52bac31ef returned -1 (Permission denied)
[2013-08-14 12:08:13.834774] I [dht-layout.c:633:dht_layout_normalize] 1-vmstore-dht: found anomalies in <gfid:140eaaf5-c667-4a71-aef1-a69a50c249b0>. holes=1 overlaps=0 missing=1 down=0 misc=0
[2013-08-14 12:08:13.835080] I [dht-layout.c:633:dht_layout_normalize] 1-vmstore-dht: found anomalies in <gfid:140eaaf5-c667-4a71-aef1-a69a50c249b0>. holes=1 overlaps=0 missing=1 down=0 misc=0
[2013-08-14 12:08:13.835545] W [client-rpc-fops.c:519:client3_3_stat_cbk] 1-vmstore-client-14: remote operation failed: No such file or directory
[2013-08-14 12:08:13.836324] W [client-rpc-fops.c:807:client3_3_statfs_cbk] 1-vmstore-client-14: remote operation failed: No such file



there are some permission denied errors on one of the brick.

Brick15 and Brick16 are the newly added bricks and then rebalance was invoked

No vm pausing is seen, but ext4 curruption was seen on the vm

(attached the ext4 corrutpion message snapshot)
Comment 3 shishir gowda 2013-08-21 00:21:07 EDT
Please provide the ls -l output from all the bricks.

Also the same o/p from all bricks for this path <brick-path>/.glusterfs/05/ba/05ba73ee-552a-4eb4-9368-6db52bac31ef
Comment 4 shylesh 2013-08-22 01:46:38 EDT
I don't have the setup now. I will try to reproduce the bug and will let you know.
Comment 5 Scott Haines 2013-09-23 15:47:09 EDT
Targeting for 2.1.z (Big Bend) U1.
Comment 6 Amar Tumballi 2013-11-26 02:14:41 EST
https://code.engineering.redhat.com/gerrit/#/c/16039/ should fix this.

All rebalance issues open should be fixed with above patch.
Comment 7 SATHEESARAN 2014-01-12 21:58:27 EST
Tested with glusterfs-3.4.0.55rhs-1 and RHEVM IS31 ( 3.3.0-0.44.el6ev)

Performed the following step using RHEVM UI,

1. Created a 4 Node cluster
2. Created a distribute replicate volume with 2X2
3. Optimized the volume for virt-store
(i.e) gluster volume set <vol-name> group virt
Changed the ownership to 36:36
(i.e) gluster voume set <vol-name> storage.owner-uid 36
      gluster voume set <vol-name> storage.owner-gid 36
4. Started the volume
5. Fuse mounted the volume
6. Created 3 raw images each of size 30GB
7. Created 3 App VMs with the above created images and then installed them with RHEL 6.5
8. Ran the cotinuous load on App VMs using dd command as follows
!/bin/bash
vols=(/home/vol1 /home/vol2 /home/vol3 /home/vol4 /home/vol5)
while true
do
    for vol in ${vols[@]};do
        #touch some files
        touch ${vol}/file{1,2,3,4,5,6,7,8,9}
    
       # dd some files of varying size
        for k in 10 50 100 200 500 700 1000 10000 20000
        do
            dd if=/dev/urandom of=${vol}/ddfile${k} bs=128k count=$k
        done

        # remove files
        rm -rf ${vol}/*
    done
done
9. Added more bricks to the volume
10. Started rebalance

App Vms are healthy after rebalance operations
Comment 9 errata-xmlrpc 2014-02-25 02:35:02 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-0208.html

Note You need to log in before you can comment on or make changes to this bug.