Description of problem: The kernel in RHEL4u6 (2.6.9-67) will not enable barriers if it detects WCE (write cache enabled) as being disabled, which is expected. However the PERC 6/i controller on Dell 2950 hardware improperly reports WCE as being disabled while still maintaining a write cache. This causes file system corruption after a power failure, because data was being maintained in controller cache while the kernel was told data was being written through. Version-Release number of selected component (if applicable): kernel-2.6.9-67.EL.i686 Steps to Reproduce: 1. Add barrier=1 to fstab for ext3 fs on disk presented by RAID controller 2. Kernel will report barriers are disabled because the PERC reports WCE is disabled (disks are presented as write-through) 3. With WCE being disabled the block queue stays unordered 4. Start writing data to drive and attempt to flush cache 5. Yank power Actual results: ext3 filesystem gets eaten on a power loss Expected results: ext3 filesystem may have corrupt data on power loss, but it's in a recoverable state Additional info: Barriers need to be enforced on this hardware because it is improperly reporting it is write-through Issue has been triaged by rwheeler, tcoughlan, jbacik.
Note that this is a generic issue - not just a PERC controller one. We do not enable barriers by default and recommend that users run their local storage with write cache disabled when they do not have battery backup. However, if you do run with write cache enabled and mount with barriers, this fix is required for any drive exported via libata or normal SCSI.
Created attachment 369486 [details] proposed patch proposed fix.
posted 11/13
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: The solution to this issue doesn't actually give us cache flushes, it just provides proper software ordering. So this will give us proper data integrity with devices that have no write caching or have write caching disabled, but if the device has write cache enabled it will not get cache flushes and leave the possibility for data loss.
Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1 +1 @@ -The solution to this issue doesn't actually give us cache flushes, it just provides proper software ordering. So this will give us proper data integrity with devices that have no write caching or have write caching disabled, but if the device has write cache enabled it will not get cache flushes and leave the possibility for data loss.+The solution to this issue doesn't actually give us cache flushes, it just provides proper software ordering. So this will give us proper data integrity with devices that have no write caching (or write caching is disabled) and no command queuing. If you have command queuing or write cache enabled there is no guarantee of data integrity.
Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1 +1 @@ -The solution to this issue doesn't actually give us cache flushes, it just provides proper software ordering. So this will give us proper data integrity with devices that have no write caching (or write caching is disabled) and no command queuing. If you have command queuing or write cache enabled there is no guarantee of data integrity.+The solution to this issue doesn't actually give us cache flushes, it just provides proper software ordering. So this will give us proper data integrity with devices that have no write caching (or write caching is disabled) and no command queuing. If you have command queuing or write cache enabled there is no guarantee of data integrity after a crash.
Committed in 89.21.EL . RPMS are available at http://people.redhat.com/vgoyal/rhel4/
Please help to verify this bug with the rhel4 kernel -89.21.EL. Thanks.
Followed the simple test from bug 560563 comment 7, the file/sec rates showed no big difference between barrier=1 and barrier=0 on 2.6.9-99 kernel. But I did notice that there was a message shown up on 2.6.9-89 kernel when mount ext3 with barrier=1 mount option. "JBD: barrier-based sync failed on sda1 - disabling barriers" On 2.6.9-99 kernel, there was no such message. I also ran fsstress, no issue found. Set it to VERIFIED.
Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1 +1,3 @@ -The solution to this issue doesn't actually give us cache flushes, it just provides proper software ordering. So this will give us proper data integrity with devices that have no write caching (or write caching is disabled) and no command queuing. If you have command queuing or write cache enabled there is no guarantee of data integrity after a crash.+Red Hat Enterprise Linux 4 kernel 2.6.9-67 does not enable barriers if it detects WCE (Write Cache Enabled) as being disabled. However, the PERC 6/i controller on Dell 2950 hardware improperly reports WCE as being disabled while still maintaining a write cache. This could have caused file system corruption in the event of a power outage due to data being maintained in the controller cache even though the kernel was told that data was being written through to disk. + +The solution to this issue provides proper software ordering, with the result that proper data integrity is achieved with devices that have no write caching (or for which write caching is disabled), and no command queuing. If command queuing or write caching is enabled, there is no guarantee of data integrity following a crash.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0263.html