Description of problem:File systems becomes Read Only after application is started. Observed following errors in /var/log/messages -: Aug 25 17:43:37 P618XISDLP kernel: emcpowerp: emcpowerp1 Aug 25 17:43:37 P618XISDLP kernel: emcpowero: emcpowero1 Aug 25 17:43:37 P618XISDLP kernel: emcpowerl: emcpowerl1 Aug 25 17:43:37 P618XISDLP kernel: emcpowerk: emcpowerk1 Aug 25 17:43:37 P618XISDLP kernel: emcpowerj: emcpowerj1 Aug 25 17:43:37 P618XISDLP kernel: emcpoweri: emcpoweri1 Aug 25 17:43:37 P618XISDLP kernel: emcpowerb: unknown partition table Aug 25 17:43:37 P618XISDLP kernel: emcpowerh: emcpowerh1 Aug 25 17:43:37 P618XISDLP kernel: emcpowere: emcpowere1 Aug 25 17:43:37 P618XISDLP kernel: emcpowerd: emcpowerd1 Aug 25 17:43:37 P618XISDLP kernel: emcpowern: emcpowern1 Aug 25 17:43:37 P618XISDLP kernel: emcpowerm: emcpowerm1 Aug 25 17:43:37 P618XISDLP kernel: emcpowerg: emcpowerg1 Aug 25 17:43:37 P618XISDLP kernel: emcpowerf: emcpowerf1 Aug 25 17:43:37 P618XISDLP kernel: emcpowerc: emcpowerc1 Aug 25 17:43:37 P618XISDLP kernel: emcpowera: emcpowera1 Aug 27 17:49:19 P618XISDLP kernel: SCSI device sdaf: 142606336 512-byte hdwr sectors (73014 MB) Aug 27 17:49:19 P618XISDLP kernel: sdaf: Write Protect is off Aug 27 17:49:19 P618XISDLP kernel: SCSI device sdaf: drive cache: write through Aug 27 17:49:19 P618XISDLP kernel: sdaf: sdaf1 Aug 27 17:49:19 P618XISDLP kernel: emcpowerb: emcpowerb1 Aug 27 17:49:21 P618XISDLP kernel: SCSI device sdaf: 142606336 512-byte hdwr sectors (73014 MB) Aug 27 17:49:21 P618XISDLP kernel: sdaf: Write Protect is off Aug 27 17:49:21 P618XISDLP kernel: SCSI device sdaf: drive cache: write through Aug 27 17:49:21 P618XISDLP kernel: sdaf: sdaf1 Aug 27 17:49:21 P618XISDLP kernel: emcpowerb: emcpowerb1 Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:Connected to EMC Clariion storage array.HBA QLE2462 , Driver 8.03.01.06.05.06-k, BIOS 3.00
This bug-report does not contain enough information to analyze your problem. Please attach complete dmesg output (Add an attachment) Fill in version of your installed packages. What kind of application have you started? (Steps to Reproduce) Is it repeatable ? Is the hw working properly?
Created attachment 520767 [details] zip of /var/log/messages
From the message log - your system has been in heavily broken state already at the beginning of this log - i.e. Aug 21. This most probably lead to very serious filesystem data corruption. Configuration of dm devices is probably wrong: kernel: dm-0: rw=0, want=2249180728, limit=2147483648 Wasn't there someone doing some device resize operation without resize of filesystem living on this device? Also still info about package version of lvm/dm tools is missing. Also you need to find out on which day the error started to happen. It's before Aug 21. One you find it in the log cut&paste surrounding info (no need to upload another multi megabyte file)
Created attachment 520774 [details] dmseg O/P
Bug looks like duplicate of bug 696807. Unfortunately there is also not a lot of info.
I found out following errors in old msg files, Aug 17 17:40:59 P618XISDLP kernel: EXT3-fs warning (device cciss/c0d0p9): dx_probe: Unrecognised inode hash code 36 Aug 17 17:40:59 P618XISDLP kernel: EXT3-fs warning (device cciss/c0d0p9): dx_probe: Corrupt dir inode 5897988, running e2fsck is recommended. Aug 17 17:40:59 P618XISDLP kernel: EXT3-fs warning (device cciss/c0d0p9): dx_probe: Unrecognised inode hash code 36 Aug 17 17:40:59 P618XISDLP kernel: EXT3-fs warning (device cciss/c0d0p9): dx_probe: Corrupt dir inode 5897988, running e2fsck is recommended. Below is the first time when file system got in to read only mode, Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998953 in dir #5963921 Aug 19 10:40:24 P618XISDLP kernel: Aborting journal on device cciss/c0d0p9. Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998959 in dir #5963921 Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998962 in dir #5963921 Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998967 in dir #5963921 Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998968 in dir #5963921 Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998969 in dir #5963921 Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998970 in dir #5963921 Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998973 in dir #5963921 Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998974 in dir #5963921 Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_lookup: unlinked inode 5998975 in dir #5963921 Aug 19 10:40:24 P618XISDLP kernel: ext3_abort called. Aug 19 10:40:24 P618XISDLP kernel: EXT3-fs error (device cciss/c0d0p9): ext3_journal_start_sb: Detected aborted journal Aug 19 10:40:24 P618XISDLP kernel: Remounting filesystem read-only
Ok - could you attach output of 'sosreport' command tool? I think it would be also helpful to have the full look from Aug 17 (if it's the first day when this error has been noticed) ? Could you also better describe system configuration? Looks quite similar to this problem: http://old.nabble.com/journal-has-aborted-td13556995.html
Created attachment 520802 [details] sosreport
sosreport has been uploaded. Thank you guys for all the wonderful support I am getting.
Please note that Red Hat Bugzilla is not an avenue for technical assistance or support, but simply a bug tracking system. As such there are no service level agreements or other guarantees associated with defects reported in Bugzilla. If you have active support entitlements for the systems mentioned in this report please file a technical support case with Red Hat Global Support Services either via your normal support representative or via the customer portal located at the following URL: https://access.redhat.com/support/ This will enable a Red Hat technical support engineer to follow up on the problems reported here directly.
I hope the problem was solved through support channel as mentioned in Comment #10. If not, please use the report procedure mentioned above, thanks.