Description of problem: In this setup, cinder is configured to use glusterfs volumes. cinder volumes are created and attached to VM Instance. These volumes are EXT4 formatted and mounted, and IO was happening continuously. Adding a pair of bricks to the glusterfs volume, and performing rebalance operation, following fix-layout, causes EXT4 FS IO error, on the mounted volumes attached to the VM Instance Version-Release number of selected component (if applicable): RHS - glusterfs-3.3.0.11rhs-1.x86_64 RHOS - http://download.lab.bos.redhat.com/rel-eng/OpenStack/Grizzly/2013-07-08.1/ cinder - 1.0.4 How reproducible: 1 time / 1 try Steps to Reproduce: Following is the steps executed to hit this bug, 1. Created gluster volume of 6X2 distributed replicate (i.e) gluster volume create cinder-vol replica 2 <brick1> ... <brick12> 2. Tag the volume with group virt (i.e) gluster volume set cinder-vol group virt 3. Set owner uid and gid to the volume (i.e) gluster volume set cinder-vol storage.owner-uid 165 gluster volume set cinder-vol storage.owner-gid 165 4. gluster volume info and gluster volume status ================================================= [Thu Jul 11 19:00:52 UTC 2013 root.37.73:~ ] # gluster volume info cinder-vol Volume Name: cinder-vol Type: Distributed-Replicate Volume ID: c7d79599-c54e-47e2-babe-a7bcc5d2fed2 Status: Started Number of Bricks: 7 x 2 = 14 Transport-type: tcp Bricks: Brick1: 10.70.37.73:/rhs/brick1/cinder1 Brick2: 10.70.37.166:/rhs/brick1/cinder1 Brick3: 10.70.37.73:/rhs/brick2/cinder2 Brick4: 10.70.37.166:/rhs/brick2/cinder2 Brick5: 10.70.37.73:/rhs/brick3/cinder3 Brick6: 10.70.37.166:/rhs/brick3/cinder3 Brick7: 10.70.37.124:/rhs/brick1/cinder1 Brick8: 10.70.37.217:/rhs/brick1/cinder1 Brick9: 10.70.37.124:/rhs/brick2/cinder2 Brick10: 10.70.37.217:/rhs/brick2/cinder2 Brick11: 10.70.37.124:/rhs/brick3/cinder3 Brick12: 10.70.37.217:/rhs/brick3/cinder3 Brick13: 10.70.37.166:/rhs/brick4/cinder-add1 Brick14: 10.70.37.217:/rhs/brick4/cinder-add1 Options Reconfigured: storage.owner-uid: 165 storage.owner-gid: 165 performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off cluster.eager-lock: enable network.remote-dio: on [Thu Jul 11 19:00:58 UTC 2013 root.37.73:~ ] # gluster volume status cinder-vol Status of volume: cinder-vol Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.37.73:/rhs/brick1/cinder1 24015 Y 1705 Brick 10.70.37.166:/rhs/brick1/cinder1 24014 Y 1653 Brick 10.70.37.73:/rhs/brick2/cinder2 24016 Y 1710 Brick 10.70.37.166:/rhs/brick2/cinder2 24015 Y 1659 Brick 10.70.37.73:/rhs/brick3/cinder3 24017 Y 1716 Brick 10.70.37.166:/rhs/brick3/cinder3 24016 Y 1665 Brick 10.70.37.124:/rhs/brick1/cinder1 24015 Y 2990 Brick 10.70.37.217:/rhs/brick1/cinder1 24014 Y 19186 Brick 10.70.37.124:/rhs/brick2/cinder2 24016 Y 1697 Brick 10.70.37.217:/rhs/brick2/cinder2 24015 Y 19191 Brick 10.70.37.124:/rhs/brick3/cinder3 24017 Y 1703 Brick 10.70.37.217:/rhs/brick3/cinder3 24016 Y 19197 Brick 10.70.37.166:/rhs/brick4/cinder-add1 24021 Y 7488 Brick 10.70.37.217:/rhs/brick4/cinder-add1 24020 Y 22468 NFS Server on localhost 38467 Y 23069 Self-heal Daemon on localhost N/A Y 23075 NFS Server on 10.70.37.166 38467 Y 7495 Self-heal Daemon on 10.70.37.166 N/A Y 7501 NFS Server on 10.70.37.124 38467 Y 13747 Self-heal Daemon on 10.70.37.124 N/A Y 13753 NFS Server on 10.70.37.217 38467 Y 22474 Self-heal Daemon on 10.70.37.217 N/A Y 22480 5. Configure cinder to use glusterfs volume, created in step 1 ============================================================== This setup makes use of multi-backend, so there are 2 glusterfs volumes and a nfs volume.modification in /etc/cinder/cinder.conf file is shown below, <snip> [glusterfsdriver1] volume_driver = cinder.volume.drivers.glusterfs.GlusterfsDriver glusterfs_shares_config = /etc/cinder/shares.conf glusterfs_mount_point_base = /var/lib/cinder/volumes volume_backend_name=GLUSTERFS_unique1 [glusterfsdriver2] volume_driver = cinder.volume.drivers.glusterfs.GlusterfsDriver glusterfs_shares_config = /etc/cinder/shares1.conf glusterfs_mount_point_base = /var/lib/cinder/volumes volume_backend_name=GLUSTERFS_unique1 [nfsdriver] volume_driver = cinder.volume.drivers.nfs.NfsDriver nfs_shares_config = /etc/cinder/nfsshares.conf nfs_mount_point_base = /var/lib/cinder/nfsvol volume_backend_name = NFS_unique1 </snip> 6. Restart cinder services (i.e) for i in api scheduler volume; do service openstack-cinder-$i restart; done 7. Create a 'type' for different backend (i.e) cinder type-create glusterfs cinder type-create nfs 8. Associate 'volume_backend_name' to corresponding type (i.e) cinder type-key glusterfs set volume_backend_name=GLUSTERFS_unique1 cinder type-key nfs set volume_backend_name=NFS_unique1 9. Create 50 cinder volumes, each of size 10G (i.e) for i in {1..50}; do cinder create 10 --display-name volume${i} --volume-type glusterfs;done 10. Created 3 new images[this images are used to boot VM instances], using oz-install and those images are uploaded to glance,which is also using glusterfs volume, 'glance-vol',in this case. This images are of centos6.4,rhel6.3,rhel6.4 Images are available @ https://url.corp.redhat.com/imgs NOTE: All this images are created with minimum RAM 1024MB and minimum hard disk 5GB 11.Create 9 VM Instances, using glance images [3 VMs using centos, 3 VMs using rhel6.3, 3 VMs using rhel6.4],with custom flavor as below [Thu Jul 11 17:13:31 UTC 2013 root@rhs-client9:~ ] # nova flavor-show 02d50a8b-333e-4de4-bbb5-4ba554931aa4 +----------------------------+--------------------------------------+ | Property | Value | +----------------------------+--------------------------------------+ | name | m1.custom1 | | ram | 1024 | | OS-FLV-DISABLED:disabled | False | | vcpus | 2 | | extra_specs | {} | | swap | | | os-flavor-access:is_public | True | | rxtx_factor | 1.0 | | OS-FLV-EXT-DATA:ephemeral | 0 | | disk | 12 | | id | 02d50a8b-333e-4de4-bbb5-4ba554931aa4 | +----------------------------+--------------------------------------+ 12. After all 9 VMs are in to running state, (Out of 50 available volumes) attach 5 volumes per VM instance, with target as /dev/vdb,/dev/vdc,/dev/vdd,/dev/vde,/dev/vdf respectively. After repeating this on all 9 VMs, rest of 5 volumes are in 'available' state 13. On each VM instance, on each attached cinder volume, /dev/vd{b,c,d,e,f}, create a new partition on it, say /dev/vdb1,/dev/vdc1,/dev/vdd1,/dev/vde1./dev/vdf1, format these partitions with EXT4 and mounted it on /home/vol{1,2,3,4,5}. (i.e) for i in b c d e f; do echo "n p 1 w "|fdisk -cu /dev/vd${i};mkfs.ext4 /dev/vd${i}1;done 14. Create directories to mount the EXT4 partitions created in step 13 (i.e) mkdir /home/vol{1,2,3,4,5} 15. Added the volume entry to /etc/fstab (i.e) count=0;for i in b c d e f; do echo "/dev/vd${i}1 /home/vol${count} ext4 defaults 0 0">> /etc/fstab;((count++));done;mount -a NOTE: step 14, step 15 are done on all 9 VMs 16. To simulate apps running on these VMS, I just used the shell script, that continuously touches file, uses dd command to create many files, then deletes them all. This scripts does IO on all 5 mounted ext4 volumes (/home/vol{1,2,3,4,5}) consecutively. <script is attached> 17. On gluster end, added 2 new bricks, (i.e) 10.70.37.166:/rhs/brick4/cinder-add1 and Brick14: 10.70.37.217:/rhs/brick4/cinder-add1 (i.e) gluster volume add-brick cinder-vol 10.70.37.166:/rhs/brick4/cinder-add1 10.70.37.217:/rhs/brick4/cinder-add1 [Thu Jul 11 19:00:52 UTC 2013 root.37.73:~ ] # gluster volume info cinder-vol Volume Name: cinder-vol Type: Distributed-Replicate Volume ID: c7d79599-c54e-47e2-babe-a7bcc5d2fed2 Status: Started Number of Bricks: 7 x 2 = 14 Transport-type: tcp Bricks: Brick1: 10.70.37.73:/rhs/brick1/cinder1 Brick2: 10.70.37.166:/rhs/brick1/cinder1 Brick3: 10.70.37.73:/rhs/brick2/cinder2 Brick4: 10.70.37.166:/rhs/brick2/cinder2 Brick5: 10.70.37.73:/rhs/brick3/cinder3 Brick6: 10.70.37.166:/rhs/brick3/cinder3 Brick7: 10.70.37.124:/rhs/brick1/cinder1 Brick8: 10.70.37.217:/rhs/brick1/cinder1 Brick9: 10.70.37.124:/rhs/brick2/cinder2 Brick10: 10.70.37.217:/rhs/brick2/cinder2 Brick11: 10.70.37.124:/rhs/brick3/cinder3 Brick12: 10.70.37.217:/rhs/brick3/cinder3 Brick13: 10.70.37.166:/rhs/brick4/cinder-add1 --> new brick Brick14: 10.70.37.217:/rhs/brick4/cinder-add1 --> new brick Options Reconfigured: storage.owner-uid: 165 storage.owner-gid: 165 performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off cluster.eager-lock: enable network.remote-dio: on 18. Executed, rebalance fix-layout (i.e) gluster volume rebalance cinder-vol fix-layout start 19. Start rebalance on the volume (i.e) gluster volume rebalance cinder-vol start 20. Check for rebalance status (i.e)[Thu Jul 11 19:02:14 UTC 2013 root.37.73:~ ] # gluster volume rebalance cinder-vol status Node Rebalanced-files size scanned failures status --------- ----------- ----------- ----------- ----------- ------------ localhost 5 53687091200 54 16 completed 10.70.37.166 0 0 50 0 completed 10.70.37.217 0 0 50 0 completed 10.70.37.124 11 118111600640 62 18 completed 21. Check for mounted cinder volumes status in VM Instance (i.e) /dev/sd{b,c,d,e,f}1 mounted on /home/vol{1,2,3,4,5}, through console logs of VM instance using Horizon [dashboard], during/after rebalance process Actual results ============== On the console log of VM instance,noticed EXT4 FS Error, on the mounted volumes(provided by cinder) [viewed through horizon(dashboard)] Expected results ================ All mounted volumes should be healthy and there should not be any errors during/post glusterfs rebalance operation Additional info: 1. gluster nodes ip information ================================ RHS Servers : 10.70.37.73, 10.70.37.124, 10.70.37.166, 10.70.37.217 All commands executed from : 10.70.37.73 cinder volumes are mounted using : 10.70.37.73 2. RHOS all-in-one node information =================================== RHOS installed in : 10.70.36.33 [client9.lab.eng.blr.redhat.com] NOTE: sosreports have their ip suffixed to their names 3. contents of cinder shares.conf, shares1.conf, nfsshares.conf ================================================================ Here are 3 shares.conf as this cinder has multi-backend. Check point 5 in steps [Thu Jul 11 19:57:18 UTC 2013 root@rhs-client9:~ ] # cat /etc/cinder/shares.conf 10.70.37.73:cinder-vol [Thu Jul 11 19:57:36 UTC 2013 root@rhs-client9:~ ] # cat /etc/cinder/shares1.conf 10.70.37.73:another-vol [Thu Jul 11 19:57:41 UTC 2013 root@rhs-client9:~ ] # cat /etc/cinder/nfsshares.conf 10.70.35.5:/home/nfsvol 4.output of mount command and df -Th command ============================================= [Thu Jul 11 19:57:50 UTC 2013 root@rhs-client9:~ ] # mount /dev/mapper/vg_rhsclient9-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0") /dev/sda1 on /boot type ext4 (rw) /dev/mapper/vg_rhsclient9-lv_home on /home type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) 10.70.37.73:glance-vol on /var/lib/glance/images type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) 10.70.37.73:cinder-vol on /var/lib/cinder/volumes/61fc21e43cc39e53f00341f56a6e13c7 type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) 10.70.37.73:another-vol on /var/lib/cinder/volumes/1272ed240174f05b9b0f7a6c6207edfe type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) 10.70.35.5:/home/nfsvol on /var/lib/cinder/nfsvol/002834169f5e66a124d096cb0892f32b type nfs (rw,vers=4,addr=10.70.35.5,clientaddr=10.70.36.33) 10.70.37.73:cinder-vol on /var/lib/nova/mnt/61fc21e43cc39e53f00341f56a6e13c7 type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) 10.70.35.5:/home/nfsvol on /var/lib/nova/mnt/002834169f5e66a124d096cb0892f32b type nfs (rw,vers=4,addr=10.70.35.5,clientaddr=10.70.36.33) [Thu Jul 11 19:58:42 UTC 2013 root@rhs-client9:~ ] # df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_rhsclient9-lv_root 50G 5.5G 42G 12% / tmpfs 7.8G 0 7.8G 0% /dev/shm /dev/sda1 485M 91M 369M 20% /boot /dev/mapper/vg_rhsclient9-lv_home 1.8T 25G 1.7T 2% /home 10.70.37.73:glance-vol 600G 367G 234G 62% /var/lib/glance/images 10.70.37.73:cinder-vol 700G 396G 305G 57% /var/lib/cinder/volumes/61fc21e43cc39e53f00341f56a6e13c7 10.70.37.73:another-vol 100G 30G 71G 30% /var/lib/cinder/volumes/1272ed240174f05b9b0f7a6c6207edfe 10.70.35.5:/home/nfsvol 4.3G 24M 4.3G 1% /var/lib/cinder/nfsvol/002834169f5e66a124d096cb0892f32b 10.70.37.73:cinder-vol 700G 396G 305G 57% /var/lib/nova/mnt/61fc21e43cc39e53f00341f56a6e13c7 10.70.35.5:/home/nfsvol 4.3G 24M 4.3G 1% /var/lib/nova/mnt/002834169f5e66a124d096cb0892f32b
script used to run IO inside VM Instance, #!/bin/bash vols=(/home/vol1 /home/vol2 /home/vol3 /home/vol4 /home/vol5) while true do for vol in ${vols[@]};do #touch some files touch ${vol}/file{1,2,3,4,5,6,7,8,9} # dd some files of varying size for k in 10 50 100 200 500 700 1000 10000 20000 100000 1000000 do dd if=/dev/urandom of=${vol}/ddfile${k} bs=128k count=$k done # remove files rm -rf ${vol}/* done done
OBSERVATION =========== 1. while rebalance is in progress, I could see some EXT4 errors in the VM Instance 'vmac1', as follows, I captured the console-log of VM since its booted, it shows the occurrence of this error Initializing cgroup subsys cpuset Initializing cgroup subsys cpu Linux version 2.6.32279.31.1.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.64) (GCC) ) #1 SMP Sun May 26 06:54:41 EDT 2013 Command line: ro root=/dev/mapper/VolGroup00LogVol00 rd_NO_LUKS serial=tty0 console=ttyS0,115200n8 console=ttyS0,115200 LANG=en_US.UTF8 rd_NO_MD SYSFONT=latarcyrhebsun16 crashkernel=auto rd_LVM_LV=VolGroup00/LogVol01 rd_LVM_LV=VolGroup00/LogVol00 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM KERNEL supported cpus: Intel GenuineIntel AMD AuthenticAMD Centaur CentaurHauls Disabled fast string operations BIOSprovided physical RAM map: BIOSe820: 0000000000000000 0000000000093c00 (usable) BIOSe820: 0000000000093c00 00000000000a0000 (reserved) BIOSe820: 00000000000f0000 0000000000100000 (reserved) BIOSe820: 0000000000100000 000000003fffd000 (usable) BIOSe820: 000000003fffd000 0000000040000000 (reserved) BIOSe820: 00000000fffbc000 0000000100000000 (reserved) DMI 2.4 present. SMBIOS version 2.4 @ 0xFD9F0 last_pfn = 0x3fffd max_arch_pfn = 0x400000000 x86 PAT enabled: cpu 0, old 0x70106, new 0x7010600070106 init_memory_mapping: 0000000000000000000000003fffd000 RAMDISK: 370db000 37fef04d ACPI: RSDP 00000000000fd9c0 00014 (v00 BOCHS ) ACPI: RSDT 000000003fffd600 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) ACPI: FACP 000000003ffffdd0 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) ACPI: DSDT 000000003fffd880 024F1 (v01 BXPC BXDSDT 00000001 INTL 20090123) ACPI: FACS 000000003ffffd80 00040 ACPI: SSDT 000000003fffd780 000FF (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) ACPI: APIC 000000003fffd690 00080 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) ACPI: SSDT 000000003fffd640 0004E (v01 BXPC BXSSDTPC 00000001 INTL 20090123) No NUMA configuration found Faking a node at 0000000000000000000000003fffd000 Bootmem setup node 0 0000000000000000000000003fffd000 NODE_DATA [0000000000009000 000000000003cfff] bootmap [000000000003d000 0000000000044fff] pages 8 (7 early reservations) ==> bootmem [0000000000 003fffd000] #0 [0000000000 0000001000] BIOS data page ==> [0000000000 0000001000] #1 [0000006000 0000008000] TRAMPOLINE ==> [0000006000 0000008000] #2 [0001000000 00020117e4] TEXT DATA BSS ==> [0001000000 00020117e4] #3 [00370db000 0037fef04d] RAMDISK ==> [00370db000 0037fef04d] #4 [0000093c00 0000100000] BIOS reserved ==> [0000093c00 0000100000] #5 [0002012000 00020120c5] BRK ==> [0002012000 00020120c5] #6 [0000008000 0000009000] PGTABLE ==> [0000008000 0000009000] found SMP MPtable at [ffff8800000fda10] fda10 kvmclock: Using msrs 4b564d01 and 4b564d00 kvmclock: cpu 0, msr 0:1c1f601, boot clock Zone PFN ranges: DMA 0x00000001 > 0x00001000 DMA32 0x00001000 > 0x00100000 Normal 0x00100000 > 0x00100000 Movable zone start PFN for each node early_node_map[2] active PFN ranges 0: 0x00000001 > 0x00000093 0: 0x00000100 > 0x0003fffd ACPI: PMTimer IO Port: 0xb008 ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 023 ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) Using ACPI (MADT) for SMP configuration information SMP: Allowing 2 CPUs, 0 hotplug CPUs PM: Registered nosave memory: 0000000000093000 0000000000094000 PM: Registered nosave memory: 0000000000094000 00000000000a0000 PM: Registered nosave memory: 00000000000a0000 00000000000f0000 PM: Registered nosave memory: 00000000000f0000 0000000000100000 Allocating PCI resources starting at 40000000 (gap: 40000000:bffbc000) Booting paravirtualized kernel on KVM NR_CPUS:4096 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 PERCPU: Embedded 31 pages/cpu @ffff880002200000 s94424 r8192 d24360 u1048576 pcpualloc: s94424 r8192 d24360 u1048576 alloc=1*2097152 pcpualloc: [0] 0 1 kvmclock: cpu 0, msr 0:2216601, primary cpu clock kvmstealtime: cpu 0, msr 220e840 Built 1 zonelists in Node order, mobility grouping on. Total pages: 258335 Policy zone: DMA32 Kernel command line: ro root=/dev/mapper/VolGroup00LogVol00 rd_NO_LUKS serial=tty0 console=ttyS0,115200n8 console=ttyS0,115200 LANG=en_US.UTF8 rd_NO_MD SYSFONT=latarcyrhebsun16 rd_LVM_LV=VolGroup00/LogVol01 rd_LVM_LV=VolGroup00/LogVol00 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM PID hash table entries: 4096 (order: 3, 32768 bytes) xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340 Checking aperture... No AGP bridge found Memory: 1001136k/1048564k available (5074k kernel code, 440k absent, 46988k reserved, 7245k data, 1256k init) Hierarchical RCU implementation. NR_IRQS:33024 nr_irqs:424 Console: colour VGA+ 80x25 console [ttyS0] enabled allocated 8388608 bytes of page_cgroup please try 'cgroup_disable=memory' option if you don't want memory cgroups Detected 1999.999 MHz processor. Calibrating delay loop (skipped) preset value.. 3999.99 BogoMIPS (lpj=1999999) pid_max: default: 32768 minimum: 301 Security Framework initialized SELinux: Initializing. Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes) Inodecache hash table entries: 65536 (order: 7, 524288 bytes) Mountcache hash table entries: 256 Initializing cgroup subsys ns Initializing cgroup subsys cpuacct Initializing cgroup subsys memory Initializing cgroup subsys devices Initializing cgroup subsys freezer Initializing cgroup subsys net_cls Initializing cgroup subsys blkio Initializing cgroup subsys perf_event Initializing cgroup subsys net_prio Disabled fast string operations mce: CPU supports 10 MCE banks alternatives: switching to unfair spinlock ACPI: Core revision 20090903 ftrace: converting mcount calls to 0f 1f 44 00 00 ftrace: allocating 20999 entries in 83 pages Enabling x2apic Enabled x2apic Setting APIC routing to physical x2apic ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=1 pin2=1 CPU0: Intel Xeon E312xx (Sandy Bridge) stepping 01 Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only. NMI watchdog disabled (cpu0): hardware events not enabled Booting Node 0, Processors #1 Ok. kvmclock: cpu 1, msr 0:2316601, secondary cpu clock kvmstealtime: cpu 1, msr 230e840 Disabled fast string operations Brought up 2 CPUs Total of 2 processors activated (7999.99 BogoMIPS). devtmpfs: initialized regulator: core version 0.5 NET: Registered protocol family 16 ACPI: bus type pci registered PCI: Using configuration type 1 for base access bio: create slab <bio0> at 0 ACPI: Interpreter enabled ACPI: (supports S0 S5) ACPI: Using IOAPIC for interrupt routing ACPI: No dock devices found. HEST: Table not found. PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00ff]) pci 0000:00:01.3: quirk: [io 0xb0000xb03f] claimed by PIIX4 ACPI pci 0000:00:01.3: quirk: [io 0xb1000xb10f] claimed by PIIX4 SMB ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) ACPI: PCI Interrupt Link [LNKS] (IRQs 9) *0 vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none vgaarb: loaded vgaarb: bridge control possible 0000:00:02.0 SCSI subsystem initialized usbcore: registered new interface driver usbfs usbcore: registered new interface driver hub usbcore: registered new device driver usb PCI: Using ACPI for IRQ routing NetLabel: Initializing NetLabel: domain hash size = 128 NetLabel: protocols = UNLABELED CIPSOv4 NetLabel: unlabeled traffic allowed by default Switching to clocksource kvmclock pnp: PnP ACPI init ACPI: bus type pnp registered pnp: PnP ACPI: found 7 devices ACPI: ACPI bus type pnp unregistered NET: Registered protocol family 2 IP route cache hash table entries: 32768 (order: 6, 262144 bytes) TCP established hash table entries: 131072 (order: 9, 2097152 bytes) TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) TCP: Hash tables configured (established 131072 bind 65536) TCP reno registered NET: Registered protocol family 1 pci 0000:00:00.0: Limiting direct PCI/PCI transfers pci 0000:00:01.0: PIIX3: Enabling Passive Release pci 0000:00:01.0: Activating ISA DMA hang workarounds ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 pci 0000:00:01.2: PCI INT D > Link[LNKD] > GSI 11 (level, high) > IRQ 11 pci 0000:00:01.2: PCI INT D disabled Trying to unpack rootfs image as initramfs... Freeing initrd memory: 15440k freed audit: initializing netlink socket (disabled) type=2000 audit(1373553830.408:1): initialized HugeTLB registered 2 MB page size, preallocated 0 pages VFS: Disk quotas dquot_6.5.2 Dquotcache hash table entries: 512 (order 0, 4096 bytes) msgmni has been set to 1985 alg: No test for stdrng (krng) ksign: Installing public key data Loading keyring Added public key 519D953B3D9E4FF1 User ID: Red Hat, Inc. (Kernel Module GPG key) Added public key D4A26C9CCD09BEDA User ID: Red Hat Enterprise Linux Driver Update Program <secalert> Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) io scheduler noop registered io scheduler anticipatory registered io scheduler deadline registered io scheduler cfq registered (default) pci_hotplug: PCI Hot Plug PCI Core version: 0.5 pciehp: PCI Express Hot Plug Controller Driver version: 0.4 acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 acpiphp: Slot [1] registered acpiphp: Slot [2] registered acpiphp: Slot [3] registered acpiphp: Slot [4] registered acpiphp: Slot [5] registered acpiphp: Slot [6] registered acpiphp: Slot [7] registered acpiphp: Slot [8] registered acpiphp: Slot [9] registered acpiphp: Slot [10] registered acpiphp: Slot [11] registered acpiphp: Slot [12] registered acpiphp: Slot [13] registered acpiphp: Slot [14] registered acpiphp: Slot [15] registered acpiphp: Slot [16] registered acpiphp: Slot [17] registered acpiphp: Slot [18] registered acpiphp: Slot [19] registered acpiphp: Slot [20] registered acpiphp: Slot [21] registered acpiphp: Slot [22] registered acpiphp: Slot [23] registered acpiphp: Slot [24] registered acpiphp: Slot [25] registered acpiphp: Slot [26] registered acpiphp: Slot [27] registered acpiphp: Slot [28] registered acpiphp: Slot [29] registered acpiphp: Slot [30] registered acpiphp: Slot [31] registered input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 ACPI: Power Button [PWRF] ERST: Table is not found! GHES: HEST is not enabled! Nonvolatile memory driver v1.3 Linux agpgart interface v0.103 crash memory driver: version 1.1 Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled ?serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A 00:06: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A brd: module loaded loop: module loaded input: Macintosh mouse button emulation as /devices/virtual/input/input1 Fixed MDIO Bus: probed ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver uhci_hcd: USB Universal Host Controller Interface driver uhci_hcd 0000:00:01.2: PCI INT D > Link[LNKD] > GSI 11 (level, high) > IRQ 11 uhci_hcd 0000:00:01.2: UHCI Host Controller uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb1: Product: UHCI Host Controller usb usb1: Manufacturer: Linux 2.6.32279.31.1.el6.x86_64 uhci_hcd usb usb1: SerialNumber: 0000:00:01.2 usb usb1: configuration #1 chosen from 1 choice hub 10:1.0: USB hub found hub 10:1.0: 2 ports detected PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 serio: i8042 KBD port at 0x60,0x64 irq 1 serio: i8042 AUX port at 0x60,0x64 irq 12 mice: PS/2 mouse device common for all mice rtc_cmos 00:01: RTC can wake from S4 input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2 rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 rtc0: alarms up to one day, 114 bytes nvram cpuidle: using governor ladder cpuidle: using governor menu EFI Variables Facility v0.08 2004May17 usbcore: registered new interface driver hiddev usbcore: registered new interface driver usbhid usbhid: v2.6:USB HID core driver TCP cubic registered Initializing XFRM netlink socket NET: Registered protocol family 17 registered taskstats version 1 rtc_cmos 00:01: setting system clock to 20130711 14:43:49 UTC (1373553829) Initalizing network drop monitor service Freeing unused kernel memory: 1256k freed Write protecting the kernel readonly data: 10240k Freeing unused kernel memory: 1052k freed Freeing unused kernel memory: 1732k freed dracut: dracut004284.el6_3.1 dracut: rd_NO_LUKS: removing cryptoluks activation devicemapper: uevent: version 1.0.3 devicemapper: ioctl: 4.22.6ioctl (20111019) initialised: dmdevel udev: starting version 147 dracut: Starting plymouth daemon dracut: rd_NO_DM: removing DM RAID activation dracut: rd_NO_MD: removing MD RAID activation scsi0 : ata_piix scsi1 : ata_piix ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 %G%GACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 virtiopci 0000:00:03.0: PCI INT A > Link[LNKC] > GSI 10 (level, high) > IRQ 10 virtiopci 0000:00:04.0: PCI INT A > Link[LNKD] > GSI 11 (level, high) > IRQ 11 ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 virtiopci 0000:00:05.0: PCI INT A > Link[LNKA] > GSI 10 (level, high) > IRQ 10 ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11 virtiopci 0000:00:06.0: PCI INT A > Link[LNKB] > GSI 11 (level, high) > IRQ 11 virtiopci 0000:00:07.0: PCI INT A > Link[LNKC] > GSI 10 (level, high) > IRQ 10 virtiopci 0000:00:08.0: PCI INT A > Link[LNKD] > GSI 11 (level, high) > IRQ 11 virtiopci 0000:00:09.0: PCI INT A > Link[LNKA] > GSI 10 (level, high) > IRQ 10 virtiopci 0000:00:0a.0: PCI INT A > Link[LNKB] > GSI 11 (level, high) > IRQ 11 usb 11: new full speed USB device number 2 using uhci_hcd Refined TSC clocksource calibration: 1999.999 MHz. input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3 vda: vda1 vda2 vdb: vdb1 vdc: vdc1 vdd: vdd1 vde: vde1 vdf: vdf1 usb 11: New USB device found, idVendor=0627, idProduct=0001 usb 11: New USB device strings: Mfr=1, Product=3, SerialNumber=5 usb 11: Product: QEMU USB Tablet usb 11: Manufacturer: QEMU 0.12.1 usb 11: SerialNumber: 42 usb 11: configuration #1 chosen from 1 choice input: QEMU 0.12.1 QEMU USB Tablet as /devices/pci0000:00/0000:00:01.2/usb1/11/11:1.0/input/input4 genericusb 0003:0627:0001.0001: input,hidraw0: USB HID v0.01 Pointer [QEMU 0.12.1 QEMU USB Tablet] on usb0000:00:01.21/input0 dracut: Scanning devices vda2 for LVM logical volumes VolGroup00/LogVol01 VolGroup00/LogVol00 dracut: inactive '/dev/VolGroup00/LogVol01' [1.50 GiB] inherit dracut: inactive '/dev/VolGroup00/LogVol00' [8.28 GiB] inherit EXT4fs (dm1): INFO: recovery required on readonly filesystem EXT4fs (dm1): write access will be enabled during recovery EXT4fs (dm1): orphan cleanup on readonly fs EXT4fs (dm1): 67 orphan inodes deleted EXT4fs (dm1): recovery complete EXT4fs (dm1): mounted filesystem with ordered data mode. Opts: dracut: Mounted root filesystem /dev/mapper/VolGroup00LogVol00 dracut: Loading SELinux policy type=1404 audit(1373553832.256:2): enforcing=1 old_enforcing=0 auid=4294967295 ses=4294967295 type=1403 audit(1373553832.803:3): policy loaded auid=4294967295 ses=4294967295 dracut: dracut: Switching root Welcome to Welcome to Red HatRed Hat Enterprise Linux Server Enterprise Linux Server Starting udev: Starting udev: udev: starting version 147 %G%Gpiix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 udev: renamed network interface eth0 to eth1 microcode: CPU0 sig=0x206a1, pf=0x1, revision=0x1 platform microcode: firmware: requesting intelucode/062a01 microcode: CPU1 sig=0x206a1, pf=0x1, revision=0x1 platform microcode: firmware: requesting intelucode/062a01 Microcode Update Driver: v2.00 <tigran.co.uk>, Peter Oruba [ OK ] [ OK ] Setting hostname localhost.localdomain: Setting hostname localhost.localdomain: [ OK ] [ OK ] Setting up Logical Volume Management: Setting up Logical Volume Management: 2 logical volume(s) in volume group "VolGroup00" now active 2 logical volume(s) in volume group "VolGroup00" now active [ OK ] [ OK ] Checking filesystems Checking filesystems Checking all file systems. [/sbin/fsck.ext4 (1) /] fsck.ext4 a /dev/mapper/VolGroup00LogVol00 Checking all file systems. [/sbin/fsck.ext4 (1) /] fsck.ext4 a /dev/mapper/VolGroup00LogVol00 /dev/mapper/VolGroup00LogVol00: clean, 34698/543504 files, 325491/2170880 blocks /dev/mapper/VolGroup00LogVol00: clean, 34698/543504 files, 325491/2170880 blocks [/sbin/fsck.ext4 (1) /boot] fsck.ext4 a /dev/vda1 [/sbin/fsck.ext4 (1) /boot] fsck.ext4 a /dev/vda1 /dev/vda1: recovering journal /dev/vda1: recovering journal /dev/vda1: clean, 44/51200 files, 56434/204800 blocks /dev/vda1: clean, 44/51200 files, 56434/204800 blocks [ OK ] [ OK ] Remounting root filesystem in readwrite mode: Remounting root filesystem in readwrite mode: [ OK ] [ OK ] Mounting local filesystems: Mounting local filesystems: EXT4fs (vda1): mounted filesystem with ordered data mode. Opts: EXT4fs (vdb1): mounted filesystem with ordered data mode. Opts: EXT4fs (vdc1): recovery complete EXT4fs (vdc1): mounted filesystem with ordered data mode. Opts: EXT4fs (vdd1): recovery complete EXT4fs (vdd1): mounted filesystem with ordered data mode. Opts: EXT4fs (vde1): recovery complete EXT4fs (vde1): mounted filesystem with ordered data mode. Opts: EXT4fs (vdf1): mounted filesystem with ordered data mode. Opts: [ OK ] Enabling local filesystem quotas: [ OK ] Enabling local filesystem quotas: [ OK ] [ OK ] Enabling /etc/fstab swaps: Enabling /etc/fstab swaps: Adding 1572856k swap on /dev/mapper/VolGroup00LogVol01. Priority:1 extents:1 across:1572856k [ OK ] [ OK ] Entering noninteractive startup Entering noninteractive startup Calling the system activity data collector (sadc): Calling the system activity data collector (sadc): Starting monitoring for VG VolGroup00: Starting monitoring for VG VolGroup00: 2 logical volume(s) in volume group "VolGroup00" monitored 2 logical volume(s) in volume group "VolGroup00" monitored [ OK ] [ OK ] NET: Registered protocol family 10 lo: Disabled Privacy Extensions Bringing up loopback interface: Bringing up loopback interface: [ OK ] [ OK ] Bringing up interface eth0: Bringing up interface eth0: Device eth0 does not seem to be present, delaying initialization. Device eth0 does not seem to be present, delaying initialization. [FAILED][FAILED] Starting auditd: Starting auditd: [ OK ] [ OK ] Starting system logger: Starting system logger: [ OK ] [ OK ] Starting irqbalance: Starting irqbalance: [ OK ] [ OK ] Starting kdump:[FAILED]Starting kdump:[FAILED] Starting system message bus: Starting system message bus: [ OK ] [ OK ] Mounting other filesystems: Mounting other filesystems: [ OK ] [ OK ] Starting acpi daemon: Starting acpi daemon: [ OK ][ OK ] Starting HAL daemon: Starting HAL daemon: [ OK ][ OK ] Retrigger failed udev eventsRetrigger failed udev events[ OK ] [ OK ] Starting sshd: Starting sshd: [ OK ][ OK ] Starting postfix: Starting postfix: [ OK ][ OK ] Starting abrt daemon: Starting abrt daemon: [ OK ][ OK ] Starting crond: Starting crond: [ OK ] [ OK ] Starting atd: Starting atd: [ OK ] [ OK ] Starting Red Hat Network Daemon: Starting Red Hat Network Daemon: [ OK ] [ OK ] Starting rhsmcertd 240 1440Starting rhsmcertd 240 1440[ OK ][ OK ] Red Hat Enterprise Linux Server release 6.3 (Santiago) Kernel 2.6.32279.31.1.el6.x86_64 on an x86_64 localhost.localdomain login: EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 Buffer I/O error on device vde1, logical block 0 lost page write due to I/O error on vde1 EXT4fs error (device vde1) in ext4_new_inode: IO failure EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 Buffer I/O error on device vde1, logical block 0 lost page write due to I/O error on vde1 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1) in ext4_new_inode: IO failure Buffer I/O error on device vde1, logical block 0 lost page write due to I/O error on vde1 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 Buffer I/O error on device vde1, logical block 0 lost page write due to I/O error on vde1 EXT4fs error (device vde1) in ext4_new_inode: IO failure EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 end_request: I/O error, dev vde, sector 2048 Buffer I/O error on device vde1, logical block 0 lost page write due to I/O error on vde1 end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1) in ext4_new_inode: IO failure end_request: I/O error, dev vde, sector 2048 Buffer I/O error on device vde1, logical block 0 lost page write due to I/O error on vde1 end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 end_request: I/O error, dev vde, sector 2048 Buffer I/O error on device vde1, logical block 0 lost page write due to I/O error on vde1 EXT4fs error (device vde1) in ext4_new_inode: IO failure EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 end_request: I/O error, dev vde, sector 2048 Buffer I/O error on device vde1, logical block 0 lost page write due to I/O error on vde1 end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1) in ext4_new_inode: IO failure end_request: I/O error, dev vde, sector 2048 Buffer I/O error on device vde1, logical block 0 lost page write due to I/O error on vde1 end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 end_request: I/O error, dev vde, sector 2048 Buffer I/O error on device vde1, logical block 0 lost page write due to I/O error on vde1 EXT4fs error (device vde1) in ext4_new_inode: IO failure EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1) in ext4_new_inode: IO failure end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 end_request: I/O error, dev vde, sector 2048 EXT4fs error (device vde1) in ext4_new_inode: IO failure EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1) in ext4_new_inode: IO failure end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 end_request: I/O error, dev vde, sector 2048 EXT4fs error (device vde1) in ext4_new_inode: IO failure EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1) in ext4_new_inode: IO failure end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 end_request: I/O error, dev vde, sector 2048 EXT4fs error (device vde1) in ext4_new_inode: IO failure EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1) in ext4_new_inode: IO failure end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 end_request: I/O error, dev vde, sector 2048 EXT4fs error (device vde1) in ext4_new_inode: IO failure EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1) in ext4_new_inode: IO failure end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 end_request: I/O error, dev vde, sector 2048 EXT4fs error (device vde1) in ext4_new_inode: IO failure EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1) in ext4_new_inode: IO failure end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 end_request: I/O error, dev vde, sector 2048 EXT4fs error (device vde1) in ext4_new_inode: IO failure EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1) in ext4_new_inode: IO failure end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 72968 end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vde1): previous I/O error to superblock detected __ratelimit: 62 callbacks suppressed Aborting journal on device vdf18. EXT4fs error (device vdf1): ext4_journal_start_sb: Detected aborted journal EXT4fs (vdf1): Remounting filesystem readonly __ratelimit: 20 callbacks suppressed Buffer I/O error on device vde1, logical block 1081344 lost page write due to I/O error on vde1 JBD2: I/O error detected when updating journal superblock for vde18. EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 Buffer I/O error on device vde1, logical block 0 lost page write due to I/O error on vde1 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1) in ext4_new_inode: IO failure Buffer I/O error on device vde1, logical block 0 lost page write due to I/O error on vde1 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 Buffer I/O error on device vde1, logical block 0 lost page write due to I/O error on vde1 EXT4fs error (device vde1) in ext4_new_inode: IO failure EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 Buffer I/O error on device vde1, logical block 0 lost page write due to I/O error on vde1 end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1) in ext4_new_inode: IO failure end_request: I/O error, dev vde, sector 2048 Buffer I/O error on device vde1, logical block 0 lost page write due to I/O error on vde1 end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 end_request: I/O error, dev vde, sector 2048 Buffer I/O error on device vde1, logical block 0 lost page write due to I/O error on vde1 EXT4fs error (device vde1) in ext4_new_inode: IO failure EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 end_request: I/O error, dev vde, sector 2048 Buffer I/O error on device vde1, logical block 0 lost page write due to I/O error on vde1 end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1) in ext4_new_inode: IO failure end_request: I/O error, dev vde, sector 2048 Buffer I/O error on device vde1, logical block 0 lost page write due to I/O error on vde1 end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 end_request: I/O error, dev vde, sector 2048 Buffer I/O error on device vde1, logical block 0 lost page write due to I/O error on vde1 EXT4fs error (device vde1) in ext4_new_inode: IO failure EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1) in ext4_new_inode: IO failure end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 end_request: I/O error, dev vde, sector 2048 EXT4fs error (device vde1) in ext4_new_inode: IO failure EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1) in ext4_new_inode: IO failure end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 end_request: I/O error, dev vde, sector 2048 EXT4fs error (device vde1) in ext4_new_inode: IO failure EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1) in ext4_new_inode: IO failure end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 end_request: I/O error, dev vde, sector 2048 EXT4fs error (device vde1) in ext4_new_inode: IO failure EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1) in ext4_new_inode: IO failure end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 end_request: I/O error, dev vde, sector 2048 EXT4fs error (device vde1) in ext4_new_inode: IO failure EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1) in ext4_new_inode: IO failure end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 end_request: I/O error, dev vde, sector 2048 EXT4fs error (device vde1) in ext4_new_inode: IO failure EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1) in ext4_new_inode: IO failure end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 end_request: I/O error, dev vde, sector 2048 EXT4fs error (device vde1) in ext4_new_inode: IO failure EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vde1): previous I/O error to superblock detected EXT4fs error (device vde1) in ext4_new_inode: IO failure end_request: I/O error, dev vde, sector 2048 end_request: I/O error, dev vde, sector 72968 EXT4fs error (device vde1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 7304 EXT4fs error (device vde1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 end_request: I/O error, dev vde, sector 2048 EXT4fs error (device vde1) in ext4_new_inode: IO failure EXT4fs (vde1): previous I/O error to superblock detected end_request: I/O error, dev vde, sector 72968 __ratelimit: 62 callbacks suppressed Aborting journal on device vde18. 2. 'vmac1' was throwing IO error on /dev/vde1 and /dev/vdf1 3. Rebooting the VM prevented those partitions from mounting. This is captured in attached screenshot. I have also captured logs, of 'vmac1' after rebooting post IO errors, end_request: I/O error, dev vdf, sector 0 end_request: I/O error, dev vde, sector 128 end_request: I/O error, dev vdf, sector 0 end_request: I/O error, dev vde, sector 0 end_request: I/O error, dev vdf, sector 0 end_request: I/O error, dev vde, sector 0 end_request: I/O error, dev vdf, sector 8 end_request: I/O error, dev vde, sector 4096 end_request: I/O error, dev vdf, sector 128 end_request: I/O error, dev vdf, sector 0 end_request: I/O error, dev vdf, sector 0 end_request: I/O error, dev vdf, sector 4096 [ OK ] [ OK ] Retrigger failed udev eventsRetrigger failed udev events[ OK ] [ OK ] Starting sshd: Starting sshd: [ OK ][ OK ] Starting postfix: Starting postfix: [ OK ][ OK ] Starting abrt daemon: Starting abrt daemon: [ OK ][ OK ] Starting crond: Starting crond: [ OK ] [ OK ] Starting atd: Starting atd: [ OK ] [ OK ] Starting Red Hat Network Daemon: Starting Red Hat Network Daemon: [ OK ] [ OK ] Starting rhsmcertd 240 1440Starting rhsmcertd 240 1440[ OK ][ OK ] Red Hat Enterprise Linux Server release 6.3 (Santiago) Kernel 2.6.32279.31.1.el6.x86_64 on an x86_64 localhost.localdomain login: 4. After rebalance process is completed, 5 VMs out of 9 VMs went to PAUSED state. I manually rebooted one machine 'vmac1' to know about its behavior post reboot. Screenshot is attached
Created attachment 772422 [details] screenshot showing the affected volumes are not mounted post reboot, during rebalance operation
Created attachment 772423 [details] paused VM instances post rebalance operation
I did a test without involving cinder, and here I am giving my observations, 1. created a 6x2 distributed replicate volume [root@localhost ~]# gluster volume info Volume Name: drvol6x2 Type: Distributed-Replicate Volume ID: 5bab11e9-0d24-4036-bcb1-2b81c134012b Status: Started Number of Bricks: 6 x 2 = 12 Transport-type: tcp Bricks: Brick1: 10.70.37.115:/rhs/brick1/dir1 Brick2: 10.70.37.127:/rhs/brick1/dir1 Brick3: 10.70.37.115:/rhs/brick2/dir2 Brick4: 10.70.37.127:/rhs/brick2/dir2 Brick5: 10.70.37.115:/rhs/brick3/dir3 Brick6: 10.70.37.127:/rhs/brick3/dir3 Brick7: 10.70.37.211:/rhs/brick1/dir1 Brick8: 10.70.37.55:/rhs/brick1/dir1 Brick9: 10.70.37.211:/rhs/brick2/dir2 Brick10: 10.70.37.55:/rhs/brick2/dir2 Brick11: 10.70.37.211:/rhs/brick3/dir3 Brick12: 10.70.37.55:/rhs/brick3/dir3 Options Reconfigured: storage.owner-gid: 107 storage.owner-uid: 107 network.remote-dio: on cluster.eager-lock: enable performance.stat-prefetch: off performance.io-cache: off performance.read-ahead: off performance.quick-read: off 2. Mounted it in hypervisor [root@rhs-client39 ~]# mount /dev/mapper/vg_rhsclient39-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0") /dev/sda1 on /boot type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) 10.70.37.115:drvol6x2 on /mnt/drvol6x2 type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) [root@rhs-client39 ~]# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg_rhsclient39-lv_root ext4 1.8T 23G 1.7T 2% / tmpfs tmpfs 7.8G 0 7.8G 0% /dev/shm /dev/sda1 ext4 485M 64M 396M 14% /boot 10.70.37.115:drvol6x2 fuse.glusterfs 595G 142G 454G 24% /mnt/drvol6x2 3. Created 50 raw image files in this mounted gluster volume, each of size 10G. (i.e) [root@rhs-client39 drvol6x2(keystone_admin)]# for i in {1..50}; do qemu-img create -f raw volume${i} 10G;done 4. Create 4 Nova Instances using horizon. These VM instances are used, for attaching the raw images created in step 3 as additional disks ( Any VMs can be used for this purpose ) (i.e) [root@rhs-client39 ~]# virsh list Id Name State ---------------------------------------------------- 1 instance-0000000d running 2 instance-0000000e running 3 instance-0000000f running 5 instance-00000011 running 4. Attached 7 such files to VMs. (i.e) count=1;for i in b c d e f g h;do virsh detach-disk instance-0000000d --target vd${i};((count++));done above attaching is done for rest of instances also NOTE: 7 disks are attached with target as /dev/vd{b,c,d,e,f,g,h} 5. Check for the additional disk images inside VM 6. Create a single partition extending the entire disk (i.e) for i in b c d e f g h; do echo "n p 1 w "|fdisk -cu /dev/vd${i};mkfs.ext4 /dev/vd${i}1;done 7. Create directories to mount the EXT4 partitions created in step 6 (i.e) mkdir /home/vol{1,2,3,4,5,6,7} 8. Added the volume entry to /etc/fstab (i.e) count=0;for i in b c d e f g h; do echo "/dev/vd${i}1 /home/vol${count} ext4 defaults 0 0">> /etc/fstab;((count++));done;mount -a NOTE: step 6, step 7, & step 8 are done on all 9 VMs 9. To simulate apps running on these VMS, I just used the shell script, that continuously touches file, uses dd command to create many files, then deletes them all. This scripts does IO on all 5 mounted ext4 volumes (/home/vol{1,2,3,4,5,6,7}) consecutively. <script is shown in comment3> 10. On gluster end, added 2 new bricks, (i.e) 10.70.37.115:/rhs/brick4/extra-dir1 and 10.70.37.127:/rhs/brick4/extra-dir1 (i.e) [root@localhost ~]# gluster volume add-brick drvol6x2 10.70.37.115:/rhs/brick4/extra-dir1 10.70.37.127:/rhs/brick4/extra-dir1 Add Brick successful [root@localhost ~]# gluster v info Volume Name: drvol6x2 Type: Distributed-Replicate Volume ID: 5bab11e9-0d24-4036-bcb1-2b81c134012b Status: Started Number of Bricks: 7 x 2 = 14 Transport-type: tcp Bricks: Brick1: 10.70.37.115:/rhs/brick1/dir1 Brick2: 10.70.37.127:/rhs/brick1/dir1 Brick3: 10.70.37.115:/rhs/brick2/dir2 Brick4: 10.70.37.127:/rhs/brick2/dir2 Brick5: 10.70.37.115:/rhs/brick3/dir3 Brick6: 10.70.37.127:/rhs/brick3/dir3 Brick7: 10.70.37.211:/rhs/brick1/dir1 Brick8: 10.70.37.55:/rhs/brick1/dir1 Brick9: 10.70.37.211:/rhs/brick2/dir2 Brick10: 10.70.37.55:/rhs/brick2/dir2 Brick11: 10.70.37.211:/rhs/brick3/dir3 Brick12: 10.70.37.55:/rhs/brick3/dir3 Brick13: 10.70.37.115:/rhs/brick4/extra-dir1 Brick14: 10.70.37.127:/rhs/brick4/extra-dir1 Options Reconfigured: storage.owner-gid: 107 storage.owner-uid: 107 network.remote-dio: on cluster.eager-lock: enable performance.stat-prefetch: off performance.io-cache: off performance.read-ahead: off performance.quick-read: off 11. Started rebalance on those volumes [root@localhost ~]# gluster volume rebalance drvol6x2 start Starting rebalance on volume drvol6x2 has been successful [root@localhost ~]# gluster volume rebalance drvol6x2 status Node Rebalanced-files size scanned failures status --------- ----------- ----------- ----------- ----------- ------------ localhost 0 0 0 0 in progress 10.70.37.55 0 0 0 0 in progress 10.70.37.211 0 0 0 0 in progress 10.70.37.127 0 0 0 0 in progress [root@localhost ~]# gluster volume rebalance drvol6x2 status Node Rebalanced-files size scanned failures status --------- ----------- ----------- ----------- ----------- ------------ localhost 0 0 1 0 in progress 10.70.37.55 0 0 50 0 completed 10.70.37.127 0 0 50 0 completed 10.70.37.211 0 0 34 4 in progress 12. checked rebalance status [root@localhost ~]# gluster volume rebalance drvol6x2 status Node Rebalanced-files size scanned failures status --------- ----------- ----------- ----------- ----------- ------------ localhost 3 32212254720 22 17 in progress 10.70.37.127 0 0 50 0 completed 10.70.37.55 0 0 50 0 completed 10.70.37.211 9 96636764160 43 4 in progress 13. while rebalance is going on, I could see IO errors on the mounted EXT4 volumes on VMs in vm3 - instance-0000000f I could able to see these errors <snip> EXT4fs error (device vdd1) in ext4_new_inode: IO failure EXT4fs (vdd1): previous I/O error to superblock detected end_request: I/O error, dev vdd, sector 2048 end_request: I/O error, dev vdd, sector 72968 EXT4fs error (device vdd1): ext4_find_entry: reading directory #2 offset 0 EXT4fs (vdd1): previous I/O error to superblock detected end_request: I/O error, dev vdd, sector 2048 end_request: I/O error, dev vdd, sector 7304 EXT4fs error (device vdd1): ext4_read_inode_bitmap: Cannot read inode bitmap block_group = 0, inode_bitmap = 657 EXT4fs (vdd1): previous I/O error to superblock detected end_request: I/O error, dev vdd, sector 2048 EXT4fs error (device vdd1) in ext4_new_inode: IO failure EXT4fs (vdd1): previous I/O error to superblock detected </snip> 14. ls on the mounted volume throwed following error [root@192-168-122-103 home]# ls /home/vol6 ls: reading directory /home/vol6" Input/output error Rebalance is still in progress, so more info - sosreports will be added once rebalance operation is complete
I could able to see error messages in log as follows in rebalance logs, [2013-07-16 08:36:51.167289] E [dht-common.c:1955:dht_getxattr] 0-drvol6x2-dht: layout is NULL [2013-07-16 08:36:51.167396] E [dht-rebalance.c:1163:gf_defrag_migrate_data] 0-drvol6x2-dht: Failed to get node-uuid for /volume34 [2013-07-16 08:36:51.194048] I [afr-self-heal-common.c:1849:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-drvol6x2-replicate-1: Non blocking entrylks failed. [2013-07-16 08:36:51.194115] E [afr-self-heal-common.c:2069:afr_self_heal_completion_cbk] 0-drvol6x2-replicate-1: background meta-data data entry missing-entry gfid self-heal failed on /volume37 [2013-07-16 08:36:51.194145] I [dht-common.c:997:dht_lookup_everywhere_cbk] 0-drvol6x2-dht: deleting stale linkfile /volume37 on drvol6x2-replicate-1 [2013-07-16 08:36:51.262147] I [dht-common.c:997:dht_lookup_everywhere_cbk] 0-drvol6x2-dht: deleting stale linkfile /volume14 on drvol6x2-replicate-3 [2013-07-16 08:36:51.275755] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-drvol6x2-client-6: remote operation failed: File exists. Path: /volume14 (00000000-0000-0000-0000-000000000000) [2013-07-16 08:36:51.276003] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-drvol6x2-client-7: remote operation failed: File exists. Path: /volume14 (00000000-0000-0000-0000-000000000000) [2013-07-16 08:36:51.276139] E [dht-common.c:1955:dht_getxattr] 0-drvol6x2-dht: layout is NULL [2013-07-16 08:36:51.276180] E [dht-rebalance.c:1163:gf_defrag_migrate_data] 0-drvol6x2-dht: Failed to get node-uuid for /volume14
Created attachment 774641 [details] errors in rebalance log files
Created attachment 774643 [details] EXT4 FS IO errors while doing ls on mount point in VM
Created attachment 774645 [details] error messages in /var/log/messages
Noticed that this issue poped up in RHS2.0 U5 bits. Should we treat this blocker for Big Bend?
There is another blocker bug for similar issue (bug 988262) for Big Bend builds. And hence taking the blocker flag off the bug, and also marking it for 2.0.z (if we do it anytime)
Oh! I didn't realize the blocker flag was intentionally removed. Removing the blocker request since this appears to be triaged.
This looks like a duplicate of bug 922183 (see comment 16 of bug 922183) errors from the sos-reports [2013-07-16 10:44:52.922272] W [fuse-bridge.c:1960:fuse_readv_cbk] 0-glusterfs-fuse: 881650: READ => -1 (Invalid argument) [2013-07-16 10:44:52.923894] I [afr-lk-common.c:1443:afr_nonblocking_inodelk] 1-drvol6x2-replicate-6: unable to get fd ctx for fd=0x25d7c58 [2013-07-16 10:44:52.923922] W [fuse-bridge.c:2037:fuse_writev_cbk] 0-glusterfs-fuse: 881652: WRITE => -1 (Invalid argument) [2013-07-16 10:44:52.925521] W [fuse-bridge.c:1960:fuse_readv_cbk] 0-glusterfs-fuse: 881653: READ => -1 (Invalid argument) [2013-07-16 10:44:52.925600] W [fuse-bridge.c:1960:fuse_readv_cbk] 0-glusterfs-fuse: 881654: READ => -1 (Invalid argument) [2013-07-16 10:44:52.928255] I [afr-lk-common.c:1443:afr_nonblocking_inodelk] 1-drvol6x2-replicate-6: unable to get fd ctx for fd=0x25d7c58 [2013-07-16 10:44:52.928287] W [fuse-bridge.c:2037:fuse_writev_cbk] 0-glusterfs-fuse: 881658: WRITE => -1 (Invalid argument) [2013-07-16 10:44:52.931025] I [afr-lk-common.c:1443:afr_nonblocking_inodelk] 1-drvol6x2-replicate-6: unable to get fd ctx for fd=0x25d7c58 [2013-07-16 10:44:52.931058] W [fuse-bridge.c:2037:fuse_writev_cbk] 0-glusterfs-fuse: 881660: WRITE => -1 (Invalid argument) [2013-07-16 10:44:52.932717] W [fuse-bridge.c:1960:fuse_readv_cbk] 0-glusterfs-fuse: 881661: READ => -1 (Invalid argument) The fix for the above bug is available from glusterfs-3.4.0.2rhs-1 and above. Could we verify if the issue still exists in rhs-2.1 releases?
Tried this case again with glusterfs-3.4.0.20rhs-2.el6rhs.x86_64. Created cinder-volumes and was formatting them (mkfs.ext4) from the instances while the rebalance was performed. Didn't see this issue. Please feel free to move this bug to ON_QA
as per comment #19
Verified with glusterfs-3.4.0.20rhs-2.el6rhs.x86_64.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1262.html