Bug 734705 - Read-only file system
Summary: Read-only file system
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: lvm2
Version: 5.5
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: rc
: ---
Assignee: LVM and device-mapper development team
QA Contact: Corey Marthaler
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-08-31 08:25 UTC by Sachin
Modified: 2011-10-21 16:05 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-10-21 16:05:27 UTC


Attachments (Terms of Use)
zip of /var/log/messages (7.92 MB, application/x-zip-compressed)
2011-08-31 08:45 UTC, Sachin
no flags Details
dmseg O/P (120.36 KB, text/plain)
2011-08-31 09:48 UTC, Sachin
no flags Details
sosreport (6.39 MB, application/octet-stream)
2011-08-31 11:50 UTC, Sachin
no flags Details

Description Sachin 2011-08-31 08:25:03 UTC
Description of problem:File systems becomes Read Only after application is started.
Observed following errors in /var/log/messages -:

Aug 25 17:43:37 P618XISDLP kernel:  emcpowerp: emcpowerp1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowero: emcpowero1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowerl: emcpowerl1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowerk: emcpowerk1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowerj: emcpowerj1
Aug 25 17:43:37 P618XISDLP kernel:  emcpoweri: emcpoweri1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowerb: unknown partition table
Aug 25 17:43:37 P618XISDLP kernel:  emcpowerh: emcpowerh1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowere: emcpowere1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowerd: emcpowerd1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowern: emcpowern1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowerm: emcpowerm1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowerg: emcpowerg1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowerf: emcpowerf1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowerc: emcpowerc1
Aug 25 17:43:37 P618XISDLP kernel:  emcpowera: emcpowera1

Aug 27 17:49:19 P618XISDLP kernel: SCSI device sdaf: 142606336 512-byte hdwr sectors (73014 MB)
Aug 27 17:49:19 P618XISDLP kernel: sdaf: Write Protect is off
Aug 27 17:49:19 P618XISDLP kernel: SCSI device sdaf: drive cache: write through
Aug 27 17:49:19 P618XISDLP kernel:  sdaf: sdaf1
Aug 27 17:49:19 P618XISDLP kernel:  emcpowerb: emcpowerb1
Aug 27 17:49:21 P618XISDLP kernel: SCSI device sdaf: 142606336 512-byte hdwr sectors (73014 MB)
Aug 27 17:49:21 P618XISDLP kernel: sdaf: Write Protect is off
Aug 27 17:49:21 P618XISDLP kernel: SCSI device sdaf: drive cache: write through
Aug 27 17:49:21 P618XISDLP kernel:  sdaf: sdaf1
Aug 27 17:49:21 P618XISDLP kernel:  emcpowerb: emcpowerb1



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:Connected to EMC Clariion storage array.HBA QLE2462 , Driver 8.03.01.06.05.06-k, BIOS 3.00

Comment 1 Zdenek Kabelac 2011-08-31 08:39:55 UTC
This bug-report does not contain enough information to analyze your problem.


Please attach complete  dmesg output  (Add an attachment)
Fill in version of your installed packages.
What kind of application have you started?
(Steps to Reproduce)
Is it repeatable ?
Is the hw working properly?

Comment 2 Sachin 2011-08-31 08:45:00 UTC
Created attachment 520767 [details]
zip of /var/log/messages

Comment 3 Zdenek Kabelac 2011-08-31 09:05:05 UTC
From the message log - your system has been in heavily broken state already at the beginning of this log - i.e. Aug 21.
 
This most probably lead to very serious filesystem data corruption.

Configuration of  dm devices is probably wrong:

kernel: dm-0: rw=0, want=2249180728, limit=2147483648

Wasn't there someone doing some device resize operation without resize of filesystem living on this device?

Also still info about package version of  lvm/dm tools is missing.

Also you need to find out on which day the error started to happen. It's before Aug 21. One you find it in the log   cut&paste surrounding info (no need to upload another multi megabyte file)

Comment 4 Sachin 2011-08-31 09:48:11 UTC
Created attachment 520774 [details]
dmseg O/P

Comment 5 Zdenek Kabelac 2011-08-31 09:54:57 UTC
Bug looks like duplicate of  bug 696807.

Unfortunately there is also not a lot of info.

Comment 6 Sachin 2011-08-31 10:02:53 UTC
I found out following errors in old msg files,

Aug 17 17:40:59 P618XISDLP kernel: EXT3-fs warning (device cciss/c0d0p9): dx_probe: Unrecognised inode hash code 36
Aug 17 17:40:59 P618XISDLP kernel: EXT3-fs warning (device cciss/c0d0p9): dx_probe: Corrupt dir inode 5897988, running e2fsck is recommended.
Aug 17 17:40:59 P618XISDLP kernel: EXT3-fs warning (device cciss/c0d0p9): dx_probe: Unrecognised inode hash code 36
Aug 17 17:40:59 P618XISDLP kernel: EXT3-fs warning (device cciss/c0d0p9): dx_probe: Corrupt dir inode 5897988, running e2fsck is recommended.

Below is the first time when file system got in to read only mode,

Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998953 in dir #5963921
Aug 19 10:40:24 P618XISDLP kernel: Aborting journal on device cciss/c0d0p9.
Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998959 in dir #5963921
Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998962 in dir #5963921
Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998967 in dir #5963921
Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998968 in dir #5963921
Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998969 in dir #5963921
Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998970 in dir #5963921
Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998973 in dir #5963921
Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998974 in dir #5963921
Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998975 in dir #5963921
Aug 19 10:40:24 P618XISDLP kernel: ext3_abort called.
Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_journal_start_sb: Detected aborted journal
Aug 19 10:40:24 P618XISDLP kernel: Remounting filesystem read-only

Comment 7 Zdenek Kabelac 2011-08-31 11:06:56 UTC
Ok - could you attach output of 'sosreport'  command tool?

I think it would be also helpful to have the full look from Aug 17 (if it's the first day when this error has been noticed) ?


Could you also better describe system configuration?


Looks quite similar to this problem:

http://old.nabble.com/journal-has-aborted-td13556995.html

Comment 8 Sachin 2011-08-31 11:50:29 UTC
Created attachment 520802 [details]
sosreport

Comment 9 Sachin 2011-08-31 11:52:26 UTC
sosreport has been uploaded.

Thank you guys for all the wonderful support I am getting.

Comment 10 Bryn M. Reeves 2011-09-01 17:46:25 UTC
Please note that Red Hat Bugzilla is not an avenue for technical assistance or support, but simply a bug tracking system. As such there are no service level agreements or other guarantees associated with defects reported in Bugzilla.

If you have active support entitlements for the systems mentioned in this report please file a technical support case with Red Hat Global Support Services either via your normal support representative or via the customer portal located at the following URL:

  https://access.redhat.com/support/

This will enable a Red Hat technical support engineer to follow up on the problems reported here directly.

Comment 11 Milan Broz 2011-10-21 16:05:27 UTC
I hope the problem was solved through support channel as mentioned in Comment #10. If not, please use the report procedure mentioned above, thanks.


Note You need to log in before you can comment on or make changes to this bug.