Bug 1014062

Summary: [RHEL5] Error : EXT4-fs error (device sda8): ext4_ext_search_left: inode #8126473: (comm db2sysc) ix (60672) != EXT_FIRST_INDEX (0) (depth 1)! leads to DB2 instance to crash
Product: Red Hat Enterprise Linux 5 Reporter: Stuart Auchterlonie <sauchter>
Component: e4fsprogsAssignee: Eric Sandeen <esandeen>
Status: CLOSED CURRENTRELEASE QA Contact: Eryu Guan <eguan>
Severity: high Docs Contact:
Priority: urgent    
Version: 5.11CC: atomlin, djeffery, eguan, ekuric, esandeen, fs-maint, jcastillo, jharriga, jsvarova, kdudka, lmcilroy, lmiksik, mgandhi, plambri, rwheeler, sauchter, sct, spanjikk, vincent
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: e4fsprogs-1.41.12-4.el5 Doc Type: Bug Fix
Doc Text:
Previously, the resize4fs utility mishandled the resizing of an ext4 file system to a smaller size. As a consequence, files containing many extents could become corrupted if they were moved during the resize process. With this update, resize4fs now maintains a consistent extent tree when moving files containing many extents, and such files no longer become corrupted in this scenario.
 Story Points: ---
Clone Of: 873201
: 1033201 (view as bug list) Environment:
Last Closed: 2014-01-09 15:12:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 873201, 1022175    
Bug Blocks: 961026, 1033201, 1033548    

Description Stuart Auchterlonie 2013-10-01 10:48:29 UTC 
+++ This bug was initially created as a clone of Bug #873201 +++

Description of problem:

When starting db2 instance on RHEL 6.3 OS it will crash and will not work as expected 


Version-Release number of selected component (if applicable):

RHEL 6.3 
Kernel : 2.6.32-131.21.1.el6.x86_64 #1 SMP Fri Nov 11 11:50:54 EST 2011 x86_64 x86_64 x86_64 GNU/Linux


How reproducible:

Install DB2 on RHEL 6.3 and try to start it.


  
Actual results:

DB2 instance fails to start with error message showed below in /var/log/messages



Additional info:

In this case issue is visible on one file system mounted as below ( I have full /etc/fstab, here is only part ) 


/dev/sda8 on /tsm/db type ext4 (rw) [tsmdb]

/dev/sda8: LABEL="tsmdb" UUID="eb82e64d-7f22-4627-a630-316b8be85e90" TYPE="ext4" 

/dev/sda8            216277748  90920132 114371288  45% /tsm/db

It was run e2fsck on this file system and file system mount works fine.e2fsck does not report file system errors 

# e2fsck -f /dev/sda8
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
tsmdb: 48/13737984 files (2.1% non-contiguous), 23592236/54931640 blocks
[root@slx900 mte]# mount -a


Error message in logs is : 

Nov  1 15:19:44 server kernel: EXT4-fs error (device sda8): ext4_ext_search_left: inode #8126473: (comm db2sysc) ix (60672) != EXT_FIRST_INDEX (0) (depth 1)!
Nov  1 15:19:44 server kernel: EXT4-fs error (device sda8): ext4_ext_search_left: inode #8126473: (comm db2sysc) ix (60672) != EXT_FIRST_INDEX (0) (depth 1)!


Error message in logs after  error appear and reboot 

Nov  1 12:00:31 server MR_MONITOR[2678]: <MRMON044> Controller ID: 0  Time
established since power on: Time 2012-11-01,12:00:31 3860 Seconds
Nov  1 13:00:31 server MR_MONITOR[2678]: <MRMON044> Controller ID: 0  Time
established since power on: Time 2012-11-01,13:00:31 7460 Seconds
Nov  1 13:01:02 server kernel: process `grep' is using deprecated sysctl
(syscall) net.ipv6.neigh.default.retrans_time; Use
net.ipv6.neigh.default.retrans_t
ime_ms instead.
Nov  1 13:59:45 server kernel: EXT4-fs error (device sda8):
ext4_ext_search_left: inode #8126473: (comm db2sysc) ix (60672) !=
EXT_FIRST_INDEX (0) (depth 1)
!
Nov  1 13:59:45 server kernel: EXT4-fs error (device sda8):
ext4_ext_search_left: inode #8126473: (comm db2sysc) ix (60672) !=
EXT_FIRST_INDEX (0) (depth 1)
!
Nov  1 14:00:31 server MR_MONITOR[2678]: <MRMON044> Controller ID: 0  Time
established since power on: Time 2012-11-01,14:00:31 11060 Seconds
Nov  1 14:18:24 server kernel: EXT4-fs error (device sda8):
ext4_ext_search_left: inode #8126473: (comm db2sysc) ix (60672) !=
EXT_FIRST_INDEX (0) (depth 1)!
Nov  1 14:18:25 server kernel: EXT4-fs error (device sda8):
ext4_ext_search_left: inode #8126473: (comm db2sysc) ix (60672) !=
EXT_FIRST_INDEX (0) (depth 1)!
Nov  1 14:28:43 server kernel: EXT4-fs error (device sda8):
ext4_ext_search_left: inode #8126473: (comm db2sysc) ix (60672) !=
EXT_FIRST_INDEX (0) (depth 1)!
Nov  1 14:28:44 server kernel: EXT4-fs error (device sda8):
ext4_ext_search_left: inode #8126473: (comm db2sysc) ix (60672) !=
EXT_FIRST_INDEX (0) (depth 1)!
Nov  1 15:00:31 server MR_MONITOR[2678]: <MRMON044> Controller ID: 0  Time
established since power on: Time 2012-11-01,15:00:31 14660 Seconds
Nov  1 15:04:43 server kernel: EXT4-fs (sda8): mounted filesystem with ordered
data mode
Nov  1 15:19:44 server kernel: EXT4-fs error (device sda8):
ext4_ext_search_left: inode #8126473: (comm db2sysc) ix (60672) !=
EXT_FIRST_INDEX (0) (depth 1)!
Nov  1 15:19:44 server kernel: EXT4-fs error (device sda8):
ext4_ext_search_left: inode #8126473: (comm db2sysc) ix (60672) !=
EXT_FIRST_INDEX (0) (depth 1)!
Nov  1 15:25:00 server init: tty (/dev/tty2) main process (2702) killed by
TERM signal
Nov  1 15:25:00 server init: tty (/dev/tty3) main process (2704) killed by
TERM signal
Nov  1 15:25:00 server init: tty (/dev/tty4) main process (2706) killed by
TERM signal
Nov  1 15:25:00 server init: tty (/dev/tty5) main process (2708) killed by
TERM signal
Nov  1 15:25:00 server init: tty (/dev/tty6) main process (2711) killed by
TERM signal
Nov  1 15:25:00 server avahi-daemon[2029]: Got SIGTERM, quitting.
Nov  1 15:25:00 server avahi-daemon[2029]: Leaving mDNS multicast group on
interface eth0.IPv4 with address 10.8.20.217.
Nov  1 15:25:00 server avahi-daemon[2029]: Leaving mDNS multicast group on
interface usb0.IPv4 with address 169.254.95.120.
Nov  1 15:25:03 server abrtd: UnRegistered Analyzer plugin CCpp
Nov  1 15:25:03 server abrtd: UnRegistered Analyzer plugin Kerneloops
Nov  1 15:25:03 server abrtd: UnRegistered Action plugin KerneloopsScanner
Nov  1 15:25:03 server abrtd: UnRegistered Reporter plugin Logger
Nov  1 15:25:03 server abrtd: UnRegistered Reporter plugin Mailx
Nov  1 15:25:03 server abrtd: UnRegistered Analyzer plugin Python
Nov  1 15:25:03 server abrtd: UnRegistered Reporter plugin RHTSupport
Nov  1 15:25:03 server abrtd: UnRegistered Action plugin SOSreport
Nov  1 15:25:03 server abrtd: UnRegistered Database plugin SQLite3
Nov  1 15:25:03 server abrtd: Got signal 15, exiting
Nov  1 15:25:06 server acpid: exiting
Nov  1 15:25:07 server ntpd[2196]: ntpd exiting on signal 15
Nov  1 15:25:07 server lin_taped[2008]: lin_taped terminated.
Nov  1 15:25:09 server rpcbind: rpcbind terminating on signal. Restart with
"rpcbind -w"
Nov  1 15:25:09 server init: Disconnected from system bus
Nov  1 15:25:09 server auditd[3061]: The audit daemon is exiting.


Nov  1 15:25:09 server kernel: type=1305 audit(1351779909.381:54268):
audit_pid=0 old=3061 auid=4294967295 ses=4294967295
subj=system_u:system_r:auditd_t:s0 res=1
Comment 7 Eric Sandeen 2013-11-21 19:44:35 UTC 
Built in e4fsprogs-1.41.12-4.el5