Bug 573185
| Summary: | large storage data corruption on 32 bit | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 5 | Reporter: | Kapetanakis Giannis <bilias> | ||||||
| Component: | kernel | Assignee: | Stanislaw Gruszka <sgruszka> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Storage QE <storage-qe> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | low | ||||||||
| Version: | 5.4 | CC: | bdonahue, coughlan, esandeen, fge, rwheeler, sgruszka | ||||||
| Target Milestone: | rc | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | i686 | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2011-01-13 21:17:24 UTC | Type: | --- | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
|
Description
Kapetanakis Giannis
2010-03-13 09:30:19 UTC
Thanks! We can see if we can reproduce this... The gpt label is also corrupted after the fs is crashed.
The gpt label is corrupted on /dev/md0 /dev/sdb and /dev/sdc
This happens with or without LVM.
So it's either the software RAID0, the adaptec driver
or maybe the firmware of the raid controllers.
/dev/sdb: Adaptec 5805z with latest firmware 5.2-0 (17544)
hardware raid 5 with 6 ST31500341AS + 1 spare
/dev/sdc same as above
/dev/md0: soft raid0 of /dev/sdb /dev/sdc
prior crash:
#parted /dev/md0
(parted) p
Model: Unknown (unknown)
Disk /dev/md0: 15.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size File system Name Flags
#fdisk -l /dev/md0
Disk /dev/md0: 14978.6 GB, 14978676948992 bytes
255 heads, 63 sectors/track, 1821053 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/md0p1 1 267350 2147483647+ ee EFI GPT
after crash:
#parted /dev/md0
(parted) p
Error: Unable to open /dev/md0 - unrecognised disk label.
#fdisk -l /dev/md0
WARNING: GPT (GUID Partition Table) detected on '/dev/md0'! The util fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/md0: 14978.6 GB, 14978676948992 bytes
2 heads, 4 sectors/track, -638063744 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Created attachment 399941 [details]
lspci
I've installed x64 Centos 5.4 and all seem better so far.
LVM quick test passed.
I'm doing a full write on disk with fs_mark and I will report.
The only problem (maybe it's normal) is that LVM erases the gpt label
on /dev/md0
So probably the gpt label was erased by LVM and not by fs corruption.
#fdisk -l /dev/md0
WARNING: GPT (GUID Partition Table) detected on '/dev/md0'! The util fdisk doesn't support GPT. Use GNU Parted.
WARNING: The size of this disk is 15.0 TB (14978676948992 bytes).
DOS partition table format can not be used on drives for volumes
larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID
partition table format (GPT).
Disk /dev/md0: 14978.6 GB, 14978676948992 bytes
255 heads, 63 sectors/track, 1821053 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/md0p1 1 267350 2147483647+ ee EFI GPT
# pvcreate /dev/md0
Physical volume "/dev/md0" successfully created
#fdisk -l /dev/md0
WARNING: GPT (GUID Partition Table) detected on '/dev/md0'! The util fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/md0: 14978.6 GB, 14978676948992 bytes
2 heads, 4 sectors/track, -638063744 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md0 doesn't contain a valid partition table
# parted /dev/md0
GNU Parted 1.8.1
Using /dev/md0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Error: Unable to open /dev/md0 - unrecognised disk label.
Is this normal?
ok full write on 14+TB completed without any errors. Seems I have problems only on 32bit OS so I'm ticking with 64bit atm. Only problem as I said earlier is that LVM is destroying the gpt label on soft raid0. Don't know if that is a problem. regards, Giannis We should definitely figure out why this works on 64 and not 32 bit. If we have a limitation, we should be checking and preventing users from creating very large file systems. Tom, we also seem to have a bad interaction between GPT labels and LVM that might be new?
According to pvcreate(8)
-Z, --zero y|n
Whether or not the first 4 sectors (2048 bytes) of the device
should be wiped. If this option is not given, the default is to
wipe these sectors unless either or both of the --restorefile or
--uuid options were specified.
This might be related to the erase of the gpt label (start of it and not the end)
Kapetanakis,
Are you using raid0, right? We have variable overflow bug in raid0 code, that can cause silent data corruption on 32 bit. This is fixed by this upstream commit:
commit 787f17feb204ed1c6331892fb8124b80dc9fe288
Author: NeilBrown <neilb>
Date: Wed May 23 13:58:09 2007 -0700
md: avoid overflow in raid0 calculation with large components
Thanx for the update. We're using 64 bit version the last 7 months without problems Giannis Hi Giannis Are you willing to test 32-bit kernel with fix from comment 8 if I prepare it? I'm not sure if you still have such possibility. If not, I will assume 32 bit overflow variable in raid0 is what happens, and post patch for that bug to be applied in our kernel. No I'm sorry but I can't test it in my live system. It has 11TB of data. I can't afford loosing them nor have I any system of that size to copy them for the test :/ regards, Giannis I have no 12TB handy as well to do testing :-( . But I'm pretty sure 32 bit variable overflow is the problem here. I had similar storage configuration and meat the same problem on my previous company, so I knew that issue and the solution before. Besides I believe our QE will do proper testing and assure we have no data corruption large storages on 32 bit. Created attachment 452053 [details]
corrupt_raid0.sh
Scripts for reproduce raid0 corruption on sparse 8TB files, need to be run ext4 or xfs.
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release. in kernel-2.6.18-227.el5 You can download this test kernel (or newer) from http://people.redhat.com/jwilson/el5 Detailed testing feedback is always welcomed. Reproduced this by the script attached on RHEL5.5 GA. Problem has been fixed in kernel-2.6.18-233.el5 Also confirmed RHEL 5.5 GA x86_64 kernel doesn't have this issue. Thank everyone's great work. An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0017.html |