Bug 983737

Summary: [RHS-RHOS] EXT4 FS I/O error on a attached cinder volume when performing rebalance followed by fix-layout
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: SATHEESARAN <sasundar>
Component: glusterfsAssignee: shishir gowda <sgowda>
Status: CLOSED ERRATA QA Contact: Sudhir D <sdharane>
Severity: high Docs Contact:
Priority: high    
Version: 2.0CC: amarts, ashetty, eharney, grajaiya, nsathyan, rhs-bugs, sasundar, sgowda, shaines, tkatarki, vagarwal, vbellur
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.4.0.20rhs-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
virt rhos cinder integration
Last Closed: 2013-09-23 22:35:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
screenshot showing the affected volumes are not mounted post reboot, during rebalance operation
none
paused VM instances post rebalance operation
none
errors in rebalance log files
none
EXT4 FS IO errors while doing ls on mount point in VM
none
error messages in /var/log/messages none

Description SATHEESARAN 2013-07-11 20:14:26 UTC 
Description of problem:
In this setup, cinder is configured to use glusterfs volumes.
cinder volumes are created and attached to VM Instance.

These volumes are EXT4 formatted and mounted, and IO was happening continuously.

Adding a pair of bricks to the glusterfs volume, and performing rebalance operation, following fix-layout, causes EXT4 FS IO error, on the mounted volumes attached to the VM Instance 

Version-Release number of selected component (if applicable):
RHS  - glusterfs-3.3.0.11rhs-1.x86_64
RHOS - http://download.lab.bos.redhat.com/rel-eng/OpenStack/Grizzly/2013-07-08.1/
cinder - 1.0.4


How reproducible:
1 time / 1 try

Steps to Reproduce:
Following is the steps executed to hit this bug,
1. Created gluster volume of 6X2 distributed replicate
   (i.e) gluster volume create cinder-vol replica 2 <brick1> ... <brick12>

2. Tag the volume with group virt
   (i.e) gluster volume set cinder-vol group virt

3. Set owner uid and gid to the volume
   (i.e) gluster volume set cinder-vol storage.owner-uid 165
         gluster volume set cinder-vol storage.owner-gid 165

4. gluster volume info and gluster volume status
=================================================

[Thu Jul 11 19:00:52 UTC 2013 root.37.73:~ ] # gluster volume info cinder-vol
 
Volume Name: cinder-vol
Type: Distributed-Replicate
Volume ID: c7d79599-c54e-47e2-babe-a7bcc5d2fed2
Status: Started
Number of Bricks: 7 x 2 = 14
Transport-type: tcp
Bricks:
Brick1: 10.70.37.73:/rhs/brick1/cinder1
Brick2: 10.70.37.166:/rhs/brick1/cinder1
Brick3: 10.70.37.73:/rhs/brick2/cinder2
Brick4: 10.70.37.166:/rhs/brick2/cinder2
Brick5: 10.70.37.73:/rhs/brick3/cinder3
Brick6: 10.70.37.166:/rhs/brick3/cinder3
Brick7: 10.70.37.124:/rhs/brick1/cinder1
Brick8: 10.70.37.217:/rhs/brick1/cinder1
Brick9: 10.70.37.124:/rhs/brick2/cinder2
Brick10: 10.70.37.217:/rhs/brick2/cinder2
Brick11: 10.70.37.124:/rhs/brick3/cinder3
Brick12: 10.70.37.217:/rhs/brick3/cinder3
Brick13: 10.70.37.166:/rhs/brick4/cinder-add1
Brick14: 10.70.37.217:/rhs/brick4/cinder-add1
Options Reconfigured:
storage.owner-uid: 165
storage.owner-gid: 165
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: on

[Thu Jul 11 19:00:58 UTC 2013 root.37.73:~ ] # gluster volume status cinder-vol
Status of volume: cinder-vol
Gluster process                                         Port    Online  Pid
------------------------------------------------------------------------------
Brick 10.70.37.73:/rhs/brick1/cinder1                   24015   Y       1705
Brick 10.70.37.166:/rhs/brick1/cinder1                  24014   Y       1653
Brick 10.70.37.73:/rhs/brick2/cinder2                   24016   Y       1710
Brick 10.70.37.166:/rhs/brick2/cinder2                  24015   Y       1659
Brick 10.70.37.73:/rhs/brick3/cinder3                   24017   Y       1716
Brick 10.70.37.166:/rhs/brick3/cinder3                  24016   Y       1665
Brick 10.70.37.124:/rhs/brick1/cinder1                  24015   Y       2990
Brick 10.70.37.217:/rhs/brick1/cinder1                  24014   Y       19186
Brick 10.70.37.124:/rhs/brick2/cinder2                  24016   Y       1697
Brick 10.70.37.217:/rhs/brick2/cinder2                  24015   Y       19191
Brick 10.70.37.124:/rhs/brick3/cinder3                  24017   Y       1703
Brick 10.70.37.217:/rhs/brick3/cinder3                  24016   Y       19197
Brick 10.70.37.166:/rhs/brick4/cinder-add1              24021   Y       7488
Brick 10.70.37.217:/rhs/brick4/cinder-add1              24020   Y       22468
NFS Server on localhost                                 38467   Y       23069
Self-heal Daemon on localhost                           N/A     Y       23075
NFS Server on 10.70.37.166                              38467   Y       7495
Self-heal Daemon on 10.70.37.166                        N/A     Y       7501
NFS Server on 10.70.37.124                              38467   Y       13747
Self-heal Daemon on 10.70.37.124                        N/A     Y       13753
NFS Server on 10.70.37.217                              38467   Y       22474
Self-heal Daemon on 10.70.37.217                        N/A     Y       22480

5. Configure cinder to use glusterfs volume, created in step 1
==============================================================
This setup makes use of multi-backend, so there are 2 glusterfs volumes and a nfs volume.modification in /etc/cinder/cinder.conf file is shown below,
  <snip>
    [glusterfsdriver1]
    volume_driver = cinder.volume.drivers.glusterfs.GlusterfsDriver
    glusterfs_shares_config = /etc/cinder/shares.conf
    glusterfs_mount_point_base = /var/lib/cinder/volumes
    volume_backend_name=GLUSTERFS_unique1

    [glusterfsdriver2]
    volume_driver = cinder.volume.drivers.glusterfs.GlusterfsDriver
    glusterfs_shares_config = /etc/cinder/shares1.conf
    glusterfs_mount_point_base = /var/lib/cinder/volumes
    volume_backend_name=GLUSTERFS_unique1

    [nfsdriver]
    volume_driver = cinder.volume.drivers.nfs.NfsDriver
    nfs_shares_config = /etc/cinder/nfsshares.conf
    nfs_mount_point_base = /var/lib/cinder/nfsvol
    volume_backend_name = NFS_unique1
  </snip> 

6. Restart cinder services
  (i.e) for i in api scheduler volume; do service openstack-cinder-$i restart; done

7. Create a 'type' for different backend
   (i.e) cinder type-create glusterfs
         cinder type-create nfs

8. Associate 'volume_backend_name' to corresponding type
   (i.e) cinder type-key glusterfs set volume_backend_name=GLUSTERFS_unique1
         cinder type-key nfs set volume_backend_name=NFS_unique1

9. Create 50 cinder volumes, each of size 10G
   (i.e) for i in {1..50}; do cinder create 10 --display-name volume${i} --volume-type glusterfs;done

10. Created 3 new images[this images are used to boot VM instances], using oz-install and those images are uploaded to glance,which is also using glusterfs volume, 'glance-vol',in this case. 

This images are of centos6.4,rhel6.3,rhel6.4
Images are available @  https://url.corp.redhat.com/imgs

NOTE: All this images are created with minimum RAM 1024MB and minimum hard disk 5GB

11.Create 9 VM Instances, using glance images [3 VMs using centos, 3 VMs using rhel6.3, 3 VMs using rhel6.4],with custom flavor as below

[Thu Jul 11 17:13:31 UTC 2013 root@rhs-client9:~ ] # nova flavor-show 02d50a8b-333e-4de4-bbb5-4ba554931aa4
+----------------------------+--------------------------------------+
| Property                   | Value                                |
+----------------------------+--------------------------------------+
| name                       | m1.custom1                           |
| ram                        | 1024                                 |
| OS-FLV-DISABLED:disabled   | False                                |
| vcpus                      | 2                                    |
| extra_specs                | {}                                   |
| swap                       |                                      |
| os-flavor-access:is_public | True                                 |
| rxtx_factor                | 1.0                                  |
| OS-FLV-EXT-DATA:ephemeral  | 0                                    |
| disk                       | 12                                   |
| id                         | 02d50a8b-333e-4de4-bbb5-4ba554931aa4 |
+----------------------------+--------------------------------------+ 

12. After all 9 VMs are in to running state, (Out of 50 available volumes) attach 5 volumes per VM instance, with target as /dev/vdb,/dev/vdc,/dev/vdd,/dev/vde,/dev/vdf respectively. After repeating this on all 9 VMs,
rest of 5 volumes are in 'available' state

13. On each VM instance, on each attached cinder volume, /dev/vd{b,c,d,e,f}, create a new partition on it, say /dev/vdb1,/dev/vdc1,/dev/vdd1,/dev/vde1./dev/vdf1, format these partitions with EXT4 and mounted it on /home/vol{1,2,3,4,5}. 
  (i.e) for i in b c d e f; do
            echo "n
            p
            1
            
            
            w
            "|fdisk -cu /dev/vd${i};mkfs.ext4 /dev/vd${i}1;done

14. Create directories to mount the EXT4 partitions created in step 13
    (i.e) mkdir /home/vol{1,2,3,4,5}

15. Added the volume entry to /etc/fstab
    (i.e) count=0;for i in b c d e f; do echo "/dev/vd${i}1 /home/vol${count} ext4 defaults 0 0">> /etc/fstab;((count++));done;mount -a
NOTE: step 14, step 15 are done on all 9 VMs 

16. To simulate apps running on these VMS, I just used the shell script, that continuously touches file, uses dd command to create many files, then deletes them all. This scripts does IO on all 5 mounted ext4 volumes (/home/vol{1,2,3,4,5}) consecutively. <script is attached>

17. On gluster end, added 2 new bricks, (i.e) 10.70.37.166:/rhs/brick4/cinder-add1 and Brick14: 10.70.37.217:/rhs/brick4/cinder-add1 
  (i.e) gluster volume add-brick cinder-vol 10.70.37.166:/rhs/brick4/cinder-add1 10.70.37.217:/rhs/brick4/cinder-add1

[Thu Jul 11 19:00:52 UTC 2013 root.37.73:~ ] # gluster volume info cinder-vol
 
Volume Name: cinder-vol
Type: Distributed-Replicate
Volume ID: c7d79599-c54e-47e2-babe-a7bcc5d2fed2
Status: Started
Number of Bricks: 7 x 2 = 14
Transport-type: tcp
Bricks:
Brick1: 10.70.37.73:/rhs/brick1/cinder1
Brick2: 10.70.37.166:/rhs/brick1/cinder1
Brick3: 10.70.37.73:/rhs/brick2/cinder2
Brick4: 10.70.37.166:/rhs/brick2/cinder2
Brick5: 10.70.37.73:/rhs/brick3/cinder3
Brick6: 10.70.37.166:/rhs/brick3/cinder3
Brick7: 10.70.37.124:/rhs/brick1/cinder1
Brick8: 10.70.37.217:/rhs/brick1/cinder1
Brick9: 10.70.37.124:/rhs/brick2/cinder2
Brick10: 10.70.37.217:/rhs/brick2/cinder2
Brick11: 10.70.37.124:/rhs/brick3/cinder3
Brick12: 10.70.37.217:/rhs/brick3/cinder3
Brick13: 10.70.37.166:/rhs/brick4/cinder-add1 --> new brick
Brick14: 10.70.37.217:/rhs/brick4/cinder-add1 --> new brick
Options Reconfigured:
storage.owner-uid: 165
storage.owner-gid: 165
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: on

18. Executed, rebalance fix-layout
    (i.e) gluster volume rebalance cinder-vol fix-layout start

19. Start rebalance on the volume
    (i.e) gluster volume rebalance cinder-vol start 

20. Check for rebalance status
    (i.e)[Thu Jul 11 19:02:14 UTC 2013 root.37.73:~ ] # gluster volume rebalance cinder-vol status
                                    Node Rebalanced-files          size       scanned      failures         status
                               ---------      -----------   -----------   -----------   -----------   ------------
                               localhost                5  53687091200           54           16      completed
                            10.70.37.166                0            0           50            0      completed
                            10.70.37.217                0            0           50            0      completed
                            10.70.37.124               11 118111600640           62           18      completed

21. Check for mounted cinder volumes status in VM Instance (i.e) /dev/sd{b,c,d,e,f}1 mounted on /home/vol{1,2,3,4,5}, through console logs of VM instance using Horizon [dashboard], during/after rebalance process

Actual results
==============
On the console log of VM instance,noticed EXT4 FS Error, on the mounted volumes(provided by cinder) [viewed through horizon(dashboard)]

Expected results
================
All mounted volumes should be healthy and there should not be any errors during/post glusterfs rebalance operation 

Additional info:

1. gluster nodes ip information
================================
RHS Servers  : 10.70.37.73,  10.70.37.124,  10.70.37.166,  10.70.37.217
All commands executed from       : 10.70.37.73
cinder volumes are mounted using : 10.70.37.73

2. RHOS all-in-one node information
===================================
RHOS installed in : 10.70.36.33 [client9.lab.eng.blr.redhat.com]

NOTE: sosreports have their ip suffixed to their names

3. contents of cinder shares.conf, shares1.conf, nfsshares.conf
================================================================
Here are 3 shares.conf as this cinder has multi-backend. Check point 5 in steps

[Thu Jul 11 19:57:18 UTC 2013 root@rhs-client9:~ ] # cat /etc/cinder/shares.conf
10.70.37.73:cinder-vol
[Thu Jul 11 19:57:36 UTC 2013 root@rhs-client9:~ ] # cat /etc/cinder/shares1.conf
10.70.37.73:another-vol
[Thu Jul 11 19:57:41 UTC 2013 root@rhs-client9:~ ] # cat /etc/cinder/nfsshares.conf
10.70.35.5:/home/nfsvol

4.output of mount command and df -Th command
=============================================
[Thu Jul 11 19:57:50 UTC 2013 root@rhs-client9:~ ] # mount
/dev/mapper/vg_rhsclient9-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/vg_rhsclient9-lv_home on /home type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
10.70.37.73:glance-vol on /var/lib/glance/images type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)
10.70.37.73:cinder-vol on /var/lib/cinder/volumes/61fc21e43cc39e53f00341f56a6e13c7 type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)
10.70.37.73:another-vol on /var/lib/cinder/volumes/1272ed240174f05b9b0f7a6c6207edfe type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)            
10.70.35.5:/home/nfsvol on /var/lib/cinder/nfsvol/002834169f5e66a124d096cb0892f32b type nfs (rw,vers=4,addr=10.70.35.5,clientaddr=10.70.36.33)                          
10.70.37.73:cinder-vol on /var/lib/nova/mnt/61fc21e43cc39e53f00341f56a6e13c7 type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)                   
10.70.35.5:/home/nfsvol on /var/lib/nova/mnt/002834169f5e66a124d096cb0892f32b type nfs (rw,vers=4,addr=10.70.35.5,clientaddr=10.70.36.33)                               
[Thu Jul 11 19:58:42 UTC 2013 root@rhs-client9:~ ] # df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_rhsclient9-lv_root
                       50G  5.5G   42G  12% /
tmpfs                 7.8G     0  7.8G   0% /dev/shm
/dev/sda1             485M   91M  369M  20% /boot
/dev/mapper/vg_rhsclient9-lv_home
                      1.8T   25G  1.7T   2% /home
10.70.37.73:glance-vol
                      600G  367G  234G  62% /var/lib/glance/images
10.70.37.73:cinder-vol
                      700G  396G  305G  57% /var/lib/cinder/volumes/61fc21e43cc39e53f00341f56a6e13c7
10.70.37.73:another-vol
                      100G   30G   71G  30% /var/lib/cinder/volumes/1272ed240174f05b9b0f7a6c6207edfe
10.70.35.5:/home/nfsvol
                      4.3G   24M  4.3G   1% /var/lib/cinder/nfsvol/002834169f5e66a124d096cb0892f32b
10.70.37.73:cinder-vol
                      700G  396G  305G  57% /var/lib/nova/mnt/61fc21e43cc39e53f00341f56a6e13c7
10.70.35.5:/home/nfsvol
                      4.3G   24M  4.3G   1% /var/lib/nova/mnt/002834169f5e66a124d096cb0892f32b
Comment 3 SATHEESARAN 2013-07-11 20:33:15 UTC 
script used to run IO inside VM Instance,

#!/bin/bash
vols=(/home/vol1 /home/vol2 /home/vol3 /home/vol4 /home/vol5)
while true
do
    for vol in ${vols[@]};do
        #touch some files
        touch ${vol}/file{1,2,3,4,5,6,7,8,9}
        
        # dd some files of varying size
        for k in 10 50 100 200 500 700 1000 10000 20000 100000 1000000
        do
            dd if=/dev/urandom of=${vol}/ddfile${k} bs=128k count=$k
        done

        # remove files
        rm -rf ${vol}/*
    done
done
Comment 4 SATHEESARAN 2013-07-11 20:43:20 UTC 
OBSERVATION
===========

1. while rebalance is in progress, I could see some EXT4 errors in the VM Instance 'vmac1', as follows, I captured the console-log of VM since its booted, it shows the occurrence of this error

Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32279.31.1.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.64) (GCC) ) #1 SMP Sun May 26 06:54:41 EDT 2013
Command line: ro root=/dev/mapper/VolGroup00LogVol00 rd_NO_LUKS serial=tty0 console=ttyS0,115200n8 console=ttyS0,115200 LANG=en_US.UTF8 rd_NO_MD SYSFONT=latarcyrhebsun16 crashkernel=auto rd_LVM_LV=VolGroup00/LogVol01 rd_LVM_LV=VolGroup00/LogVol00  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
Disabled fast string operations
BIOSprovided physical RAM map:
 BIOSe820: 0000000000000000  0000000000093c00 (usable)
 BIOSe820: 0000000000093c00  00000000000a0000 (reserved)
 BIOSe820: 00000000000f0000  0000000000100000 (reserved)
 BIOSe820: 0000000000100000  000000003fffd000 (usable)
 BIOSe820: 000000003fffd000  0000000040000000 (reserved)
 BIOSe820: 00000000fffbc000  0000000100000000 (reserved)
DMI 2.4 present.
SMBIOS version 2.4 @ 0xFD9F0
last_pfn = 0x3fffd max_arch_pfn = 0x400000000
x86 PAT enabled: cpu 0, old 0x70106, new 0x7010600070106
init_memory_mapping: 0000000000000000000000003fffd000
RAMDISK: 370db000  37fef04d
ACPI: RSDP 00000000000fd9c0 00014 (v00 BOCHS )
ACPI: RSDT 000000003fffd600 00034 (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)
ACPI: FACP 000000003ffffdd0 00074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)
ACPI: DSDT 000000003fffd880 024F1 (v01   BXPC   BXDSDT 00000001 INTL 20090123)
ACPI: FACS 000000003ffffd80 00040
ACPI: SSDT 000000003fffd780 000FF (v01 BOCHS  BXPCSSDT 00000001 BXPC 00000001)
ACPI: APIC 000000003fffd690 00080 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
ACPI: SSDT 000000003fffd640 0004E (v01   BXPC BXSSDTPC 00000001 INTL 20090123)
No NUMA configuration found
Faking a node at 0000000000000000000000003fffd000
Bootmem setup node 0 0000000000000000000000003fffd000
  NODE_DATA [0000000000009000  000000000003cfff]
  bootmap [000000000003d000   0000000000044fff] pages 8
(7 early reservations) ==> bootmem [0000000000  003fffd000]
  #0 [0000000000  0000001000]   BIOS data page ==> [0000000000  0000001000]
  #1 [0000006000  0000008000]       TRAMPOLINE ==> [0000006000  0000008000]
  #2 [0001000000  00020117e4]    TEXT DATA BSS ==> [0001000000  00020117e4]
  #3 [00370db000  0037fef04d]          RAMDISK ==> [00370db000  0037fef04d]
  #4 [0000093c00  0000100000]    BIOS reserved ==> [0000093c00  0000100000]
  #5 [0002012000  00020120c5]              BRK ==> [0002012000  00020120c5]
  #6 [0000008000  0000009000]          PGTABLE ==> [0000008000  0000009000]
found SMP MPtable at [ffff8800000fda10] fda10
kvmclock: Using msrs 4b564d01 and 4b564d00
kvmclock: cpu 0, msr 0:1c1f601, boot clock
Zone PFN ranges:
  DMA      0x00000001 > 0x00001000
  DMA32    0x00001000 > 0x00100000
  Normal   0x00100000 > 0x00100000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000001 > 0x00000093
    0: 0x00000100 > 0x0003fffd
ACPI: PMTimer IO Port: 0xb008
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 023
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
Using ACPI (MADT) for SMP configuration information
SMP: Allowing 2 CPUs, 0 hotplug CPUs
PM: Registered nosave memory: 0000000000093000  0000000000094000
PM: Registered nosave memory: 0000000000094000  00000000000a0000
PM: Registered nosave memory: 00000000000a0000  00000000000f0000
PM: Registered nosave memory: 00000000000f0000  0000000000100000
Allocating PCI resources starting at 40000000 (gap: 40000000:bffbc000)
Booting paravirtualized kernel on KVM
NR_CPUS:4096 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1
PERCPU: Embedded 31 pages/cpu @ffff880002200000 s94424 r8192 d24360 u1048576
pcpualloc: s94424 r8192 d24360 u1048576 alloc=1*2097152
pcpualloc: [0] 0 1 
kvmclock: cpu 0, msr 0:2216601, primary cpu clock
kvmstealtime: cpu 0, msr 220e840
Built 1 zonelists in Node order, mobility grouping on.  Total pages: 258335
Policy zone: DMA32
Kernel command line: ro root=/dev/mapper/VolGroup00LogVol00 rd_NO_LUKS serial=tty0 console=ttyS0,115200n8 console=ttyS0,115200 LANG=en_US.UTF8 rd_NO_MD SYSFONT=latarcyrhebsun16  rd_LVM_LV=VolGroup00/LogVol01 rd_LVM_LV=VolGroup00/LogVol00  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM
PID hash table entries: 4096 (order: 3, 32768 bytes)
xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340
Checking aperture...
No AGP bridge found
Memory: 1001136k/1048564k available (5074k kernel code, 440k absent, 46988k reserved, 7245k data, 1256k init)
Hierarchical RCU implementation.
NR_IRQS:33024 nr_irqs:424
Console: colour VGA+ 80x25
console [ttyS0] enabled
allocated 8388608 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
Detected 1999.999 MHz processor.
Calibrating delay loop (skipped) preset value.. 3999.99 BogoMIPS (lpj=1999999)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux:  Initializing.
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
Inodecache hash table entries: 65536 (order: 7, 524288 bytes)
Mountcache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
Initializing cgroup subsys perf_event
Initializing cgroup subsys net_prio
Disabled fast string operations
mce: CPU supports 10 MCE banks
alternatives: switching to unfair spinlock
ACPI: Core revision 20090903
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 20999 entries in 83 pages
Enabling x2apic
Enabled x2apic
Setting APIC routing to physical x2apic
..TIMER: vector=0x30 apic1=0 pin1=2 apic2=1 pin2=1
CPU0: Intel Xeon E312xx (Sandy Bridge) stepping 01
Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only.
NMI watchdog disabled (cpu0): hardware events not enabled
Booting Node   0, Processors  #1 Ok.
kvmclock: cpu 1, msr 0:2316601, secondary cpu clock
kvmstealtime: cpu 1, msr 230e840
Disabled fast string operations
Brought up 2 CPUs
Total of 2 processors activated (7999.99 BogoMIPS).
devtmpfs: initialized
regulator: core version 0.5
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using configuration type 1 for base access
bio: create slab <bio0> at 0
ACPI: Interpreter enabled
ACPI: (supports S0 S5)
ACPI: Using IOAPIC for interrupt routing
ACPI: No dock devices found.
HEST: Table not found.
PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug
ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00ff])
pci 0000:00:01.3: quirk: [io  0xb0000xb03f] claimed by PIIX4 ACPI
pci 0000:00:01.3: quirk: [io  0xb1000xb10f] claimed by PIIX4 SMB
ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
ACPI: PCI Interrupt Link [LNKS] (IRQs 9) *0
vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
vgaarb: loaded
vgaarb: bridge control possible 0000:00:02.0
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: Using ACPI for IRQ routing
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
Switching to clocksource kvmclock
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp: PnP ACPI: found 7 devices
ACPI: ACPI bus type pnp unregistered
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 6, 262144 bytes)
TCP established hash table entries: 131072 (order: 9, 2097152 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
NET: Registered protocol family 1
pci 0000:00:00.0: Limiting direct PCI/PCI transfers
pci 0000:00:01.0: PIIX3: Enabling Passive Release
pci 0000:00:01.0: Activating ISA DMA hang workarounds
ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
pci 0000:00:01.2: PCI INT D > Link[LNKD] > GSI 11 (level, high) > IRQ 11
pci 0000:00:01.2: PCI INT D disabled
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 15440k freed
audit: initializing netlink socket (disabled)
type=2000 audit(1373553830.408:1): initialized
HugeTLB registered 2 MB page size, preallocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquotcache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 1985
alg: No test for stdrng (krng)
ksign: Installing public key data
Loading keyring
 Added public key 519D953B3D9E4FF1
 User ID: Red Hat, Inc. (Kernel Module GPG key)
 Added public key D4A26C9CCD09BEDA
 User ID: Red Hat Enterprise Linux Driver Update Program <secalert>
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
acpiphp: Slot [1] registered
acpiphp: Slot [2] registered
acpiphp: Slot [3] registered
acpiphp: Slot [4] registered
acpiphp: Slot [5] registered
acpiphp: Slot [6] registered
acpiphp: Slot [7] registered
acpiphp: Slot [8] registered
acpiphp: Slot [9] registered
acpiphp: Slot [10] registered
acpiphp: Slot [11] registered
acpiphp: Slot [12] registered
acpiphp: Slot [13] registered
acpiphp: Slot [14] registered
acpiphp: Slot [15] registered
acpiphp: Slot [16] registered
acpiphp: Slot [17] registered
acpiphp: Slot [18] registered
acpiphp: Slot [19] registered
acpiphp: Slot [20] registered
acpiphp: Slot [21] registered
acpiphp: Slot [22] registered
acpiphp: Slot [23] registered
acpiphp: Slot [24] registered
acpiphp: Slot [25] registered
acpiphp: Slot [26] registered
acpiphp: Slot [27] registered
acpiphp: Slot [28] registered
acpiphp: Slot [29] registered
acpiphp: Slot [30] registered
acpiphp: Slot [31] registered
input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
ACPI: Power Button [PWRF]
ERST: Table is not found!
GHES: HEST is not enabled!
Nonvolatile memory driver v1.3
Linux agpgart interface v0.103
crash memory driver: version 1.1
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
?serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:06: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input1
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
uhci_hcd 0000:00:01.2: PCI INT D > Link[LNKD] > GSI 11 (level, high) > IRQ 11
uhci_hcd 0000:00:01.2: UHCI Host Controller
uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1
uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020
usb usb1: New USB device found, idVendor=1d6b, idProduct=0001
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: UHCI Host Controller
usb usb1: Manufacturer: Linux 2.6.32279.31.1.el6.x86_64 uhci_hcd
usb usb1: SerialNumber: 0000:00:01.2
usb usb1: configuration #1 chosen from 1 choice
hub 10:1.0: USB hub found
hub 10:1.0: 2 ports detected
PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
rtc_cmos 00:01: RTC can wake from S4
input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2
rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0
rtc0: alarms up to one day, 114 bytes nvram
cpuidle: using governor ladder
cpuidle: using governor menu
EFI Variables Facility v0.08 2004May17
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
rtc_cmos 00:01: setting system clock to 20130711 14:43:49 UTC (1373553829)
Initalizing network drop monitor service
Freeing unused kernel memory: 1256k freed
Write protecting the kernel readonly data: 10240k
Freeing unused kernel memory: 1052k freed
Freeing unused kernel memory: 1732k freed
dracut: dracut004284.el6_3.1
dracut: rd_NO_LUKS: removing cryptoluks activation
devicemapper: uevent: version 1.0.3
devicemapper: ioctl: 4.22.6ioctl (20111019) initialised: dmdevel
udev: starting version 147
dracut: Starting plymouth daemon
dracut: rd_NO_DM: removing DM RAID activation
dracut: rd_NO_MD: removing MD RAID activation
scsi0 : ata_piix
scsi1 : ata_piix
ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14
ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15
%G%GACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10
virtiopci 0000:00:03.0: PCI INT A > Link[LNKC] > GSI 10 (level, high) > IRQ 10
virtiopci 0000:00:04.0: PCI INT A > Link[LNKD] > GSI 11 (level, high) > IRQ 11
ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
virtiopci 0000:00:05.0: PCI INT A > Link[LNKA] > GSI 10 (level, high) > IRQ 10
ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11
virtiopci 0000:00:06.0: PCI INT A > Link[LNKB] > GSI 11 (level, high) > IRQ 11
virtiopci 0000:00:07.0: PCI INT A > Link[LNKC] > GSI 10 (level, high) > IRQ 10
virtiopci 0000:00:08.0: PCI INT A > Link[LNKD] > GSI 11 (level, high) > IRQ 11
virtiopci 0000:00:09.0: PCI INT A > Link[LNKA] > GSI 10 (level, high) > IRQ 10
virtiopci 0000:00:0a.0: PCI INT A > Link[LNKB] > GSI 11 (level, high) > IRQ 11
usb 11: new full speed USB device number 2 using uhci_hcd
Refined TSC clocksource calibration: 1999.999 MHz.
input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
 vda: vda1 vda2
 vdb: vdb1
 vdc: vdc1
 vdd: vdd1
 vde: vde1
 vdf: vdf1
usb 11: New USB device found, idVendor=0627, idProduct=0001
usb 11: New USB device strings: Mfr=1, Product=3, SerialNumber=5
usb 11: Product: QEMU USB Tablet
usb 11: Manufacturer: QEMU 0.12.1
usb 11: SerialNumber: 42
usb 11: configuration #1 chosen from 1 choice
input: QEMU 0.12.1 QEMU USB Tablet as /devices/pci0000:00/0000:00:01.2/usb1/11/11:1.0/input/input4
genericusb 0003:0627:0001.0001: input,hidraw0: USB HID v0.01 Pointer [QEMU 0.12.1 QEMU USB Tablet] on usb0000:00:01.21/input0
dracut: Scanning devices vda2  for LVM logical volumes VolGroup00/LogVol01 VolGroup00/LogVol00 
dracut: inactive '/dev/VolGroup00/LogVol01' [1.50 GiB] inherit
dracut: inactive '/dev/VolGroup00/LogVol00' [8.28 GiB] inherit
EXT4fs (dm1): INFO: recovery required on readonly filesystem
EXT4fs (dm1): write access will be enabled during recovery
EXT4fs (dm1): orphan cleanup on readonly fs
EXT4fs (dm1): 67 orphan inodes deleted
EXT4fs (dm1): recovery complete
EXT4fs (dm1): mounted filesystem with ordered data mode. Opts: 
dracut: Mounted root filesystem /dev/mapper/VolGroup00LogVol00
dracut: Loading SELinux policy
type=1404 audit(1373553832.256:2): enforcing=1 old_enforcing=0 auid=4294967295 ses=4294967295
type=1403 audit(1373553832.803:3): policy loaded auid=4294967295 ses=4294967295
dracut: 
dracut: Switching root
		Welcome to 		Welcome to Red HatRed Hat Enterprise Linux Server
 Enterprise Linux Server
Starting udev: Starting udev: udev: starting version 147
%G%Gpiix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0
udev: renamed network interface eth0 to eth1
microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1
platform microcode: firmware: requesting intelucode/062a01
microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1
platform microcode: firmware: requesting intelucode/062a01
Microcode Update Driver: v2.00 <tigran.co.uk>, Peter Oruba
[  OK  ]
[  OK  ]
Setting hostname localhost.localdomain:  Setting hostname localhost.localdomain:  [  OK  ]
[  OK  ]
Setting up Logical Volume Management: Setting up Logical Volume Management:   2 logical volume(s) in volume group "VolGroup00" now active
  2 logical volume(s) in volume group "VolGroup00" now active
[  OK  ]
[  OK  ]
Checking filesystems
Checking filesystems
Checking all file systems.
[/sbin/fsck.ext4 (1)  /] fsck.ext4 a /dev/mapper/VolGroup00LogVol00 
Checking all file systems.
[/sbin/fsck.ext4 (1)  /] fsck.ext4 a /dev/mapper/VolGroup00LogVol00 
/dev/mapper/VolGroup00LogVol00: clean, 34698/543504 files, 325491/2170880 blocks
/dev/mapper/VolGroup00LogVol00: clean, 34698/543504 files, 325491/2170880 blocks
[/sbin/fsck.ext4 (1)  /boot] fsck.ext4 a /dev/vda1 
[/sbin/fsck.ext4 (1)  /boot] fsck.ext4 a /dev/vda1 
/dev/vda1: recovering journal
/dev/vda1: recovering journal
/dev/vda1: clean, 44/51200 files, 56434/204800 blocks
/dev/vda1: clean, 44/51200 files, 56434/204800 blocks
[  OK  ]
[  OK  ]
Remounting root filesystem in readwrite mode:  Remounting root filesystem in readwrite mode:  [  OK  ]
[  OK  ]
Mounting local filesystems:  Mounting local filesystems:  EXT4fs (vda1): mounted filesystem with ordered data mode. Opts: 
EXT4fs (vdb1): mounted filesystem with ordered data mode. Opts: 
EXT4fs (vdc1): recovery complete
EXT4fs (vdc1): mounted filesystem with ordered data mode. Opts: 
EXT4fs (vdd1): recovery complete
EXT4fs (vdd1): mounted filesystem with ordered data mode. Opts: 
EXT4fs (vde1): recovery complete
EXT4fs (vde1): mounted filesystem with ordered data mode. Opts: 
EXT4fs (vdf1): mounted filesystem with ordered data mode. Opts: 
[  OK  ]
Enabling local filesystem quotas:  [  OK  ]
Enabling local filesystem quotas:  [  OK  ]
[  OK  ]
Enabling /etc/fstab swaps:  Enabling /etc/fstab swaps:  Adding 1572856k swap on /dev/mapper/VolGroup00LogVol01.  Priority:1 extents:1 across:1572856k 
[  OK  ]
[  OK  ]
Entering noninteractive startup
Entering noninteractive startup
Calling the system activity data collector (sadc): Calling the system activity data collector (sadc): 

Starting monitoring for VG VolGroup00: Starting monitoring for VG VolGroup00:   2 logical volume(s) in volume group "VolGroup00" monitored
  2 logical volume(s) in volume group "VolGroup00" monitored
[  OK  ]
[  OK  ]
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
Bringing up loopback interface:  Bringing up loopback interface:  [  OK  ]
[  OK  ]
Bringing up interface eth0:  Bringing up interface eth0:  Device eth0 does not seem to be present, delaying initialization.
Device eth0 does not seem to be present, delaying initialization.
[FAILED][FAILED]

Starting auditd: Starting auditd: [  OK  ]
[  OK  ]
Starting system logger: Starting system logger: [  OK  ]
[  OK  ]
Starting irqbalance: Starting irqbalance: [  OK  ]
[  OK  ]
Starting kdump:[FAILED]Starting kdump:[FAILED]

Starting system message bus: Starting system message bus: [  OK  ]
[  OK  ]
Mounting other filesystems:  Mounting other filesystems:  [  OK  ]
[  OK  ]
Starting acpi daemon: Starting acpi daemon: [  OK  ][  OK  ]

Starting HAL daemon: Starting HAL daemon: [  OK  ][  OK  ]

Retrigger failed udev eventsRetrigger failed udev events[  OK  ]
[  OK  ]
Starting sshd: Starting sshd: [  OK  ][  OK  ]

Starting postfix: Starting postfix: [  OK  ][  OK  ]

Starting abrt daemon: Starting abrt daemon: [  OK  ][  OK  ]

Starting crond: Starting crond: [  OK  ]
[  OK  ]
Starting atd: Starting atd: [  OK  ]
[  OK  ]
Starting Red Hat Network Daemon: Starting Red Hat Network Daemon: [  OK  ]
[  OK  ]
Starting rhsmcertd 240 1440Starting rhsmcertd 240 1440[  OK  ][  OK  ]


Red Hat Enterprise Linux Server release 6.3 (Santiago)
Kernel 2.6.32279.31.1.el6.x86_64 on an x86_64

localhost.localdomain login: EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
Buffer I/O error on device vde1, logical block 0
lost page write due to I/O error on vde1
EXT4fs error (device vde1) in ext4_new_inode: IO failure
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
Buffer I/O error on device vde1, logical block 0
lost page write due to I/O error on vde1
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1) in ext4_new_inode: IO failure
Buffer I/O error on device vde1, logical block 0
lost page write due to I/O error on vde1
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
Buffer I/O error on device vde1, logical block 0
lost page write due to I/O error on vde1
EXT4fs error (device vde1) in ext4_new_inode: IO failure
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
end_request: I/O error, dev vde, sector 2048
Buffer I/O error on device vde1, logical block 0
lost page write due to I/O error on vde1
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1) in ext4_new_inode: IO failure
end_request: I/O error, dev vde, sector 2048
Buffer I/O error on device vde1, logical block 0
lost page write due to I/O error on vde1
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
end_request: I/O error, dev vde, sector 2048
Buffer I/O error on device vde1, logical block 0
lost page write due to I/O error on vde1
EXT4fs error (device vde1) in ext4_new_inode: IO failure
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
end_request: I/O error, dev vde, sector 2048
Buffer I/O error on device vde1, logical block 0
lost page write due to I/O error on vde1
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1) in ext4_new_inode: IO failure
end_request: I/O error, dev vde, sector 2048
Buffer I/O error on device vde1, logical block 0
lost page write due to I/O error on vde1
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
end_request: I/O error, dev vde, sector 2048
Buffer I/O error on device vde1, logical block 0
lost page write due to I/O error on vde1
EXT4fs error (device vde1) in ext4_new_inode: IO failure
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1) in ext4_new_inode: IO failure
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
end_request: I/O error, dev vde, sector 2048
EXT4fs error (device vde1) in ext4_new_inode: IO failure
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1) in ext4_new_inode: IO failure
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
end_request: I/O error, dev vde, sector 2048
EXT4fs error (device vde1) in ext4_new_inode: IO failure
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1) in ext4_new_inode: IO failure
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
end_request: I/O error, dev vde, sector 2048
EXT4fs error (device vde1) in ext4_new_inode: IO failure
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1) in ext4_new_inode: IO failure
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
end_request: I/O error, dev vde, sector 2048
EXT4fs error (device vde1) in ext4_new_inode: IO failure
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1) in ext4_new_inode: IO failure
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
end_request: I/O error, dev vde, sector 2048
EXT4fs error (device vde1) in ext4_new_inode: IO failure
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1) in ext4_new_inode: IO failure
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
end_request: I/O error, dev vde, sector 2048
EXT4fs error (device vde1) in ext4_new_inode: IO failure
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1) in ext4_new_inode: IO failure
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 72968
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vde1): previous I/O error to superblock detected
__ratelimit: 62 callbacks suppressed
Aborting journal on device vdf18.
EXT4fs error (device vdf1): ext4_journal_start_sb: Detected aborted journal
EXT4fs (vdf1): Remounting filesystem readonly
__ratelimit: 20 callbacks suppressed
Buffer I/O error on device vde1, logical block 1081344
lost page write due to I/O error on vde1
JBD2: I/O error detected when updating journal superblock for vde18.
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
Buffer I/O error on device vde1, logical block 0
lost page write due to I/O error on vde1
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1) in ext4_new_inode: IO failure
Buffer I/O error on device vde1, logical block 0
lost page write due to I/O error on vde1
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
Buffer I/O error on device vde1, logical block 0
lost page write due to I/O error on vde1
EXT4fs error (device vde1) in ext4_new_inode: IO failure
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
Buffer I/O error on device vde1, logical block 0
lost page write due to I/O error on vde1
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1) in ext4_new_inode: IO failure
end_request: I/O error, dev vde, sector 2048
Buffer I/O error on device vde1, logical block 0
lost page write due to I/O error on vde1
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
end_request: I/O error, dev vde, sector 2048
Buffer I/O error on device vde1, logical block 0
lost page write due to I/O error on vde1
EXT4fs error (device vde1) in ext4_new_inode: IO failure
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
end_request: I/O error, dev vde, sector 2048
Buffer I/O error on device vde1, logical block 0
lost page write due to I/O error on vde1
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1) in ext4_new_inode: IO failure
end_request: I/O error, dev vde, sector 2048
Buffer I/O error on device vde1, logical block 0
lost page write due to I/O error on vde1
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
end_request: I/O error, dev vde, sector 2048
Buffer I/O error on device vde1, logical block 0
lost page write due to I/O error on vde1
EXT4fs error (device vde1) in ext4_new_inode: IO failure
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1) in ext4_new_inode: IO failure
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
end_request: I/O error, dev vde, sector 2048
EXT4fs error (device vde1) in ext4_new_inode: IO failure
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1) in ext4_new_inode: IO failure
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
end_request: I/O error, dev vde, sector 2048
EXT4fs error (device vde1) in ext4_new_inode: IO failure
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1) in ext4_new_inode: IO failure
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
end_request: I/O error, dev vde, sector 2048
EXT4fs error (device vde1) in ext4_new_inode: IO failure
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1) in ext4_new_inode: IO failure
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
end_request: I/O error, dev vde, sector 2048
EXT4fs error (device vde1) in ext4_new_inode: IO failure
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1) in ext4_new_inode: IO failure
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
end_request: I/O error, dev vde, sector 2048
EXT4fs error (device vde1) in ext4_new_inode: IO failure
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1) in ext4_new_inode: IO failure
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
end_request: I/O error, dev vde, sector 2048
EXT4fs error (device vde1) in ext4_new_inode: IO failure
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vde1): previous I/O error to superblock detected
EXT4fs error (device vde1) in ext4_new_inode: IO failure
end_request: I/O error, dev vde, sector 2048
end_request: I/O error, dev vde, sector 72968
EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 7304
EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
end_request: I/O error, dev vde, sector 2048
EXT4fs error (device vde1) in ext4_new_inode: IO failure
EXT4fs (vde1): previous I/O error to superblock detected
end_request: I/O error, dev vde, sector 72968
__ratelimit: 62 callbacks suppressed
Aborting journal on device vde18.

2. 'vmac1' was throwing IO error on /dev/vde1 and /dev/vdf1

3. Rebooting the VM prevented those partitions from mounting. This is captured in attached screenshot. I have also captured logs, of 'vmac1' after rebooting post IO errors,

    end_request: I/O error, dev vdf, sector 0
    end_request: I/O error, dev vde, sector 128
    end_request: I/O error, dev vdf, sector 0
    end_request: I/O error, dev vde, sector 0
    end_request: I/O error, dev vdf, sector 0
    end_request: I/O error, dev vde, sector 0
    end_request: I/O error, dev vdf, sector 8
    end_request: I/O error, dev vde, sector 4096
    end_request: I/O error, dev vdf, sector 128
    end_request: I/O error, dev vdf, sector 0
    end_request: I/O error, dev vdf, sector 0
    end_request: I/O error, dev vdf, sector 4096
    [  OK  ]
    [  OK  ]
    Retrigger failed udev eventsRetrigger failed udev events[  OK  ]
    [  OK  ]
    Starting sshd: Starting sshd: [  OK  ][  OK  ]
     
    Starting postfix: Starting postfix: [  OK  ][  OK  ]
     
    Starting abrt daemon: Starting abrt daemon: [  OK  ][  OK  ]
     
    Starting crond: Starting crond: [  OK  ]
    [  OK  ]
    Starting atd: Starting atd: [  OK  ]
    [  OK  ]
    Starting Red Hat Network Daemon: Starting Red Hat Network Daemon: [  OK  ]
    [  OK  ]
    Starting rhsmcertd 240 1440Starting rhsmcertd 240 1440[  OK  ][  OK  ]
     
     
    Red Hat Enterprise Linux Server release 6.3 (Santiago)
    Kernel 2.6.32279.31.1.el6.x86_64 on an x86_64
     
    localhost.localdomain login:

4. After rebalance process is completed, 5 VMs out of 9 VMs went to PAUSED state. I manually rebooted one machine 'vmac1' to know about its behavior post reboot. Screenshot is attached
Comment 5 SATHEESARAN 2013-07-11 20:47:51 UTC 
Created attachment 772422 [details]
screenshot showing the affected volumes are not mounted post reboot, during rebalance operation
Comment 6 SATHEESARAN 2013-07-11 20:48:37 UTC 
Created attachment 772423 [details]
paused VM instances post rebalance operation
Comment 9 SATHEESARAN 2013-07-16 10:48:00 UTC 
I did a test without involving cinder, and here I am giving my observations,

1. created a 6x2 distributed replicate volume

[root@localhost ~]# gluster volume info
 
Volume Name: drvol6x2
Type: Distributed-Replicate
Volume ID: 5bab11e9-0d24-4036-bcb1-2b81c134012b
Status: Started
Number of Bricks: 6 x 2 = 12
Transport-type: tcp
Bricks:
Brick1: 10.70.37.115:/rhs/brick1/dir1
Brick2: 10.70.37.127:/rhs/brick1/dir1
Brick3: 10.70.37.115:/rhs/brick2/dir2
Brick4: 10.70.37.127:/rhs/brick2/dir2
Brick5: 10.70.37.115:/rhs/brick3/dir3
Brick6: 10.70.37.127:/rhs/brick3/dir3
Brick7: 10.70.37.211:/rhs/brick1/dir1
Brick8: 10.70.37.55:/rhs/brick1/dir1
Brick9: 10.70.37.211:/rhs/brick2/dir2
Brick10: 10.70.37.55:/rhs/brick2/dir2
Brick11: 10.70.37.211:/rhs/brick3/dir3
Brick12: 10.70.37.55:/rhs/brick3/dir3
Options Reconfigured:
storage.owner-gid: 107
storage.owner-uid: 107
network.remote-dio: on                                                                                                                                                  
cluster.eager-lock: enable                                                                                                                                              
performance.stat-prefetch: off                                                                                                                                          
performance.io-cache: off                                                                                                                                               
performance.read-ahead: off                                                                                                    
performance.quick-read: off

2. Mounted it in hypervisor
[root@rhs-client39 ~]# mount
/dev/mapper/vg_rhsclient39-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)                                                                                                                                                         
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)                                                                                                                                                         
10.70.37.115:drvol6x2 on /mnt/drvol6x2 type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) 

[root@rhs-client39 ~]# df -Th                                                                                                                                                                                  
Filesystem    Type    Size  Used Avail Use% Mounted on                                                                                                                                                         
/dev/mapper/vg_rhsclient39-lv_root                                                                                                                                                                             
              ext4    1.8T   23G  1.7T   2% /                                                                                                                                                                  
tmpfs        tmpfs    7.8G     0  7.8G   0% /dev/shm                                                                                                                                                           
/dev/sda1     ext4    485M   64M  396M  14% /boot                                                                                                                                                              
10.70.37.115:drvol6x2                                                                                                                                                                                          
    fuse.glusterfs    595G  142G  454G  24% /mnt/drvol6x2 

3. Created 50 raw image files in this mounted gluster volume, each of size 10G.
(i.e) [root@rhs-client39 drvol6x2(keystone_admin)]# for i in {1..50}; do qemu-img create -f raw volume${i} 10G;done

4. Create 4 Nova Instances using horizon. These VM instances are used, for attaching the raw images created in step 3 as additional disks ( Any VMs can be used for this purpose )

(i.e) [root@rhs-client39 ~]# virsh list                                                                                                                                                                              
 Id    Name                           State                                                                                                                                                                    
----------------------------------------------------                                                                                                                                                           
 1     instance-0000000d              running                                                                                                                                                                  
 2     instance-0000000e              running                                                                                                                                                                  
 3     instance-0000000f              running                                                                                                                                                                  
 5     instance-00000011              running


4. Attached 7 such files to VMs.
(i.e) count=1;for i in b c d e f g h;do virsh detach-disk instance-0000000d  --target vd${i};((count++));done

above attaching is done for rest of instances also
NOTE: 7 disks are attached with target as /dev/vd{b,c,d,e,f,g,h}

5. Check for the additional disk images inside VM
6. Create a single partition extending the entire disk
  (i.e) for i in b c d e f g h; do
            echo "n
            p
            1
            
            
            w
            "|fdisk -cu /dev/vd${i};mkfs.ext4 /dev/vd${i}1;done

7. Create directories to mount the EXT4 partitions created in step 6
    (i.e) mkdir /home/vol{1,2,3,4,5,6,7}

8. Added the volume entry to /etc/fstab
    (i.e) count=0;for i in b c d e f g h; do echo "/dev/vd${i}1 /home/vol${count} ext4 defaults 0 0">> /etc/fstab;((count++));done;mount -a
NOTE: step 6, step 7, & step 8 are done on all 9 VMs 

9. To simulate apps running on these VMS, I just used the shell script, that continuously touches file, uses dd command to create many files, then deletes them all. This scripts does IO on all 5 mounted ext4 volumes (/home/vol{1,2,3,4,5,6,7}) consecutively. <script is shown in comment3>

10. On gluster end, added 2 new bricks, (i.e) 10.70.37.115:/rhs/brick4/extra-dir1 and 10.70.37.127:/rhs/brick4/extra-dir1 
  (i.e) [root@localhost ~]# gluster volume add-brick drvol6x2 10.70.37.115:/rhs/brick4/extra-dir1 10.70.37.127:/rhs/brick4/extra-dir1
Add Brick successful

[root@localhost ~]# gluster v info
 
Volume Name: drvol6x2
Type: Distributed-Replicate
Volume ID: 5bab11e9-0d24-4036-bcb1-2b81c134012b
Status: Started
Number of Bricks: 7 x 2 = 14
Transport-type: tcp
Bricks:
Brick1: 10.70.37.115:/rhs/brick1/dir1
Brick2: 10.70.37.127:/rhs/brick1/dir1
Brick3: 10.70.37.115:/rhs/brick2/dir2
Brick4: 10.70.37.127:/rhs/brick2/dir2
Brick5: 10.70.37.115:/rhs/brick3/dir3
Brick6: 10.70.37.127:/rhs/brick3/dir3
Brick7: 10.70.37.211:/rhs/brick1/dir1
Brick8: 10.70.37.55:/rhs/brick1/dir1
Brick9: 10.70.37.211:/rhs/brick2/dir2
Brick10: 10.70.37.55:/rhs/brick2/dir2
Brick11: 10.70.37.211:/rhs/brick3/dir3
Brick12: 10.70.37.55:/rhs/brick3/dir3
Brick13: 10.70.37.115:/rhs/brick4/extra-dir1
Brick14: 10.70.37.127:/rhs/brick4/extra-dir1
Options Reconfigured:
storage.owner-gid: 107
storage.owner-uid: 107
network.remote-dio: on
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off

11. Started rebalance on those volumes
[root@localhost ~]# gluster volume rebalance drvol6x2 start
Starting rebalance on volume drvol6x2 has been successful
[root@localhost ~]# gluster volume rebalance drvol6x2 status
                                    Node Rebalanced-files          size       scanned      failures         status
                               ---------      -----------   -----------   -----------   -----------   ------------
                               localhost                0            0            0            0    in progress
                             10.70.37.55                0            0            0            0    in progress
                            10.70.37.211                0            0            0            0    in progress
                            10.70.37.127                0            0            0            0    in progress

[root@localhost ~]# gluster volume rebalance drvol6x2 status
                                    Node Rebalanced-files          size       scanned      failures         status
                               ---------      -----------   -----------   -----------   -----------   ------------
                               localhost                0            0            1            0    in progress
                             10.70.37.55                0            0           50            0      completed
                            10.70.37.127                0            0           50            0      completed
                            10.70.37.211                0            0           34            4    in progress

12. checked rebalance status
[root@localhost ~]# gluster volume rebalance drvol6x2 status
                                    Node Rebalanced-files          size       scanned      failures         status
                               ---------      -----------   -----------   -----------   -----------   ------------
                               localhost                3  32212254720           22           17    in progress
                            10.70.37.127                0            0           50            0      completed
                             10.70.37.55                0            0           50            0      completed
                            10.70.37.211                9  96636764160           43            4    in progress

13. while rebalance is going on, I could see IO errors on the mounted EXT4 volumes on VMs

in vm3 - instance-0000000f I could able to see these errors

<snip>
EXT4fs error (device vdd1) in ext4_new_inode: IO failure
EXT4fs (vdd1): previous I/O error to superblock detected
end_request: I/O error, dev vdd, sector 2048
end_request: I/O error, dev vdd, sector 72968
EXT4fs error (device vdd1): ext4_find_entry: reading directory #2 offset 0
EXT4fs (vdd1): previous I/O error to superblock detected
end_request: I/O error, dev vdd, sector 2048
end_request: I/O error, dev vdd, sector 7304
EXT4fs error (device vdd1): ext4_read_inode_bitmap: Cannot read inode bitmap  block_group = 0, inode_bitmap = 657
EXT4fs (vdd1): previous I/O error to superblock detected
end_request: I/O error, dev vdd, sector 2048
EXT4fs error (device vdd1) in ext4_new_inode: IO failure
EXT4fs (vdd1): previous I/O error to superblock detected
</snip>

14. ls on the mounted volume throwed following error
[root@192-168-122-103 home]# ls /home/vol6
ls: reading directory /home/vol6" Input/output error

Rebalance is still in progress, so more info - sosreports will be added once rebalance operation is complete
Comment 11 SATHEESARAN 2013-07-17 06:42:28 UTC 
I could able to see error messages in log as follows in rebalance logs,


[2013-07-16 08:36:51.167289] E [dht-common.c:1955:dht_getxattr] 0-drvol6x2-dht: layout is NULL
[2013-07-16 08:36:51.167396] E [dht-rebalance.c:1163:gf_defrag_migrate_data] 0-drvol6x2-dht: Failed to get node-uuid for /volume34
[2013-07-16 08:36:51.194048] I [afr-self-heal-common.c:1849:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-drvol6x2-replicate-1: Non blocking entrylks failed.
[2013-07-16 08:36:51.194115] E [afr-self-heal-common.c:2069:afr_self_heal_completion_cbk] 0-drvol6x2-replicate-1: background  meta-data data entry missing-entry gfid self-heal failed on /volume37
[2013-07-16 08:36:51.194145] I [dht-common.c:997:dht_lookup_everywhere_cbk] 0-drvol6x2-dht: deleting stale linkfile /volume37 on drvol6x2-replicate-1
[2013-07-16 08:36:51.262147] I [dht-common.c:997:dht_lookup_everywhere_cbk] 0-drvol6x2-dht: deleting stale linkfile /volume14 on drvol6x2-replicate-3
[2013-07-16 08:36:51.275755] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-drvol6x2-client-6: remote operation failed: File exists. Path: /volume14 (00000000-0000-0000-0000-000000000000)
[2013-07-16 08:36:51.276003] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-drvol6x2-client-7: remote operation failed: File exists. Path: /volume14 (00000000-0000-0000-0000-000000000000)
[2013-07-16 08:36:51.276139] E [dht-common.c:1955:dht_getxattr] 0-drvol6x2-dht: layout is NULL
[2013-07-16 08:36:51.276180] E [dht-rebalance.c:1163:gf_defrag_migrate_data] 0-drvol6x2-dht: Failed to get node-uuid for /volume14
Comment 12 SATHEESARAN 2013-07-17 06:44:32 UTC 
Created attachment 774641 [details]
errors in rebalance log files
Comment 13 SATHEESARAN 2013-07-17 06:46:17 UTC 
Created attachment 774643 [details]
EXT4 FS IO errors while doing ls on mount point in VM
Comment 14 SATHEESARAN 2013-07-17 06:46:55 UTC 
Created attachment 774645 [details]
error messages in /var/log/messages
Comment 15 Amar Tumballi 2013-07-22 10:34:16 UTC 
Noticed that this issue poped up in RHS2.0 U5 bits. Should we treat this blocker for Big Bend?
Comment 16 Amar Tumballi 2013-07-30 11:42:40 UTC 
There is another blocker bug for similar issue (bug 988262) for Big Bend builds. And hence taking the blocker flag off the bug, and also marking it for 2.0.z (if we do it anytime)
Comment 17 Gowrishankar Rajaiyan 2013-08-05 12:23:53 UTC 
Oh! I didn't realize the blocker flag was intentionally removed. Removing the blocker request since this appears to be triaged.
Comment 18 shishir gowda 2013-08-06 08:01:46 UTC 
This looks like a duplicate of bug 922183 (see comment 16 of bug 922183)

errors from the sos-reports
[2013-07-16 10:44:52.922272] W [fuse-bridge.c:1960:fuse_readv_cbk] 0-glusterfs-fuse: 881650: READ => -1 (Invalid argument)
[2013-07-16 10:44:52.923894] I [afr-lk-common.c:1443:afr_nonblocking_inodelk] 1-drvol6x2-replicate-6: unable to get fd ctx for fd=0x25d7c58
[2013-07-16 10:44:52.923922] W [fuse-bridge.c:2037:fuse_writev_cbk] 0-glusterfs-fuse: 881652: WRITE => -1 (Invalid argument)
[2013-07-16 10:44:52.925521] W [fuse-bridge.c:1960:fuse_readv_cbk] 0-glusterfs-fuse: 881653: READ => -1 (Invalid argument)
[2013-07-16 10:44:52.925600] W [fuse-bridge.c:1960:fuse_readv_cbk] 0-glusterfs-fuse: 881654: READ => -1 (Invalid argument)
[2013-07-16 10:44:52.928255] I [afr-lk-common.c:1443:afr_nonblocking_inodelk] 1-drvol6x2-replicate-6: unable to get fd ctx for fd=0x25d7c58
[2013-07-16 10:44:52.928287] W [fuse-bridge.c:2037:fuse_writev_cbk] 0-glusterfs-fuse: 881658: WRITE => -1 (Invalid argument)
[2013-07-16 10:44:52.931025] I [afr-lk-common.c:1443:afr_nonblocking_inodelk] 1-drvol6x2-replicate-6: unable to get fd ctx for fd=0x25d7c58
[2013-07-16 10:44:52.931058] W [fuse-bridge.c:2037:fuse_writev_cbk] 0-glusterfs-fuse: 881660: WRITE => -1 (Invalid argument)
[2013-07-16 10:44:52.932717] W [fuse-bridge.c:1960:fuse_readv_cbk] 0-glusterfs-fuse: 881661: READ => -1 (Invalid argument)

The fix for the above bug is available from glusterfs-3.4.0.2rhs-1 and above.

Could we verify if the issue still exists in rhs-2.1 releases?
Comment 19 Anush Shetty 2013-08-19 12:26:11 UTC 
Tried this case again with glusterfs-3.4.0.20rhs-2.el6rhs.x86_64. 

Created cinder-volumes and was formatting them (mkfs.ext4) from the instances while the rebalance was performed. Didn't see this issue. Please feel free to move this bug to ON_QA
Comment 20 Amar Tumballi 2013-08-19 12:30:30 UTC 
as per comment #19
Comment 21 Anush Shetty 2013-08-19 12:36:37 UTC 
Verified with glusterfs-3.4.0.20rhs-2.el6rhs.x86_64.
Comment 22 Scott Haines 2013-09-23 22:35:49 UTC 
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html