Bug 734705

Summary: Read-only file system
Product: Red Hat Enterprise Linux 5 Reporter: Sachin <sachin.vatsaraj>
Component: lvm2Assignee: LVM and device-mapper development team <lvm-team>
Status: CLOSED WORKSFORME QA Contact: Corey Marthaler <cmarthal>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 5.5CC: agk, bmr, dwysocha, esandeen, heinzm, jbrassow, mbroz, mchristi, prajnoha, prockai, rwheeler, sachin.vatsaraj, thenzl, thornber, zkabelac
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-10-21 16:05:27 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
zip of /var/log/messages
none
dmseg O/P
none
sosreport none

Description Sachin 2011-08-31 08:25:03 UTC
Description of problem:File systems becomes Read Only after application is started.
Observed following errors in /var/log/messages -:

Aug 25 17:43:37 P618XISDLP kernel:  emcpowerp: emcpowerp1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowero: emcpowero1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowerl: emcpowerl1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowerk: emcpowerk1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowerj: emcpowerj1
Aug 25 17:43:37 P618XISDLP kernel:  emcpoweri: emcpoweri1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowerb: unknown partition table
Aug 25 17:43:37 P618XISDLP kernel:  emcpowerh: emcpowerh1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowere: emcpowere1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowerd: emcpowerd1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowern: emcpowern1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowerm: emcpowerm1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowerg: emcpowerg1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowerf: emcpowerf1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowerc: emcpowerc1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowera: emcpowera1

Aug 27 17:49:19 P618XISDLP kernel: SCSI device sdaf: 142606336 512-byte hdwr sectors (73014 MB)
Aug 27 17:49:19 P618XISDLP kernel: sdaf: Write Protect is off
Aug 27 17:49:19 P618XISDLP kernel: SCSI device sdaf: drive cache: write through
Aug 27 17:49:19 P618XISDLP kernel:  sdaf: sdaf1
Aug 27 17:49:19 P618XISDLP kernel:  emcpowerb: emcpowerb1
Aug 27 17:49:21 P618XISDLP kernel: SCSI device sdaf: 142606336 512-byte hdwr sectors (73014 MB)
Aug 27 17:49:21 P618XISDLP kernel: sdaf: Write Protect is off
Aug 27 17:49:21 P618XISDLP kernel: SCSI device sdaf: drive cache: write through
Aug 27 17:49:21 P618XISDLP kernel:  sdaf: sdaf1
Aug 27 17:49:21 P618XISDLP kernel:  emcpowerb: emcpowerb1



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:Connected to EMC Clariion storage array.HBA QLE2462 , Driver 8.03.01.06.05.06-k, BIOS 3.00

Comment 1 Zdenek Kabelac 2011-08-31 08:39:55 UTC
This bug-report does not contain enough information to analyze your problem.


Please attach complete  dmesg output  (Add an attachment)
Fill in version of your installed packages.
What kind of application have you started?
(Steps to Reproduce)
Is it repeatable ?
Is the hw working properly?

Comment 2 Sachin 2011-08-31 08:45:00 UTC
Created attachment 520767 [details]
zip of /var/log/messages

Comment 3 Zdenek Kabelac 2011-08-31 09:05:05 UTC
From the message log - your system has been in heavily broken state already at the beginning of this log - i.e. Aug 21.
 
This most probably lead to very serious filesystem data corruption.

Configuration of  dm devices is probably wrong:

kernel: dm-0: rw=0, want=2249180728, limit=2147483648

Wasn't there someone doing some device resize operation without resize of filesystem living on this device?

Also still info about package version of  lvm/dm tools is missing.

Also you need to find out on which day the error started to happen. It's before Aug 21. One you find it in the log   cut&paste surrounding info (no need to upload another multi megabyte file)

Comment 4 Sachin 2011-08-31 09:48:11 UTC
Created attachment 520774 [details]
dmseg O/P

Comment 5 Zdenek Kabelac 2011-08-31 09:54:57 UTC
Bug looks like duplicate of  bug 696807.

Unfortunately there is also not a lot of info.

Comment 6 Sachin 2011-08-31 10:02:53 UTC
I found out following errors in old msg files,

Aug 17 17:40:59 P618XISDLP kernel: EXT3-fs warning (device cciss/c0d0p9): dx_probe: Unrecognised inode hash code 36
Aug 17 17:40:59 P618XISDLP kernel: EXT3-fs warning (device cciss/c0d0p9): dx_probe: Corrupt dir inode 5897988, running e2fsck is recommended.
Aug 17 17:40:59 P618XISDLP kernel: EXT3-fs warning (device cciss/c0d0p9): dx_probe: Unrecognised inode hash code 36
Aug 17 17:40:59 P618XISDLP kernel: EXT3-fs warning (device cciss/c0d0p9): dx_probe: Corrupt dir inode 5897988, running e2fsck is recommended.

Below is the first time when file system got in to read only mode,

Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998953 in dir #5963921
Aug 19 10:40:24 P618XISDLP kernel: Aborting journal on device cciss/c0d0p9.
Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998959 in dir #5963921
Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998962 in dir #5963921
Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998967 in dir #5963921
Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998968 in dir #5963921
Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998969 in dir #5963921
Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998970 in dir #5963921
Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998973 in dir #5963921
Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998974 in dir #5963921
Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998975 in dir #5963921
Aug 19 10:40:24 P618XISDLP kernel: ext3_abort called.
Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_journal_start_sb: Detected aborted journal
Aug 19 10:40:24 P618XISDLP kernel: Remounting filesystem read-only

Comment 7 Zdenek Kabelac 2011-08-31 11:06:56 UTC
Ok - could you attach output of 'sosreport'  command tool?

I think it would be also helpful to have the full look from Aug 17 (if it's the first day when this error has been noticed) ?


Could you also better describe system configuration?


Looks quite similar to this problem:

http://old.nabble.com/journal-has-aborted-td13556995.html

Comment 8 Sachin 2011-08-31 11:50:29 UTC
Created attachment 520802 [details]
sosreport

Comment 9 Sachin 2011-08-31 11:52:26 UTC
sosreport has been uploaded.

Thank you guys for all the wonderful support I am getting.

Comment 10 Bryn M. Reeves 2011-09-01 17:46:25 UTC
Please note that Red Hat Bugzilla is not an avenue for technical assistance or support, but simply a bug tracking system. As such there are no service level agreements or other guarantees associated with defects reported in Bugzilla.

If you have active support entitlements for the systems mentioned in this report please file a technical support case with Red Hat Global Support Services either via your normal support representative or via the customer portal located at the following URL:

  https://access.redhat.com/support/

This will enable a Red Hat technical support engineer to follow up on the problems reported here directly.

Comment 11 Milan Broz 2011-10-21 16:05:27 UTC
I hope the problem was solved through support channel as mentioned in Comment #10. If not, please use the report procedure mentioned above, thanks.