Description of problem: Version-Release number of selected component (if applicable): mdadm-2.6.9-2.el5.x86_64 How reproducible: always Steps to Reproduce: 1. Create a software RAID like this: # cat /proc/mdstat Personalities : [raid1] md0 : active raid1 sdb1[1] sda1[0] 264960 blocks [2/2] [UU] md1 : active raid1 sdb2[1] sda2[0] 155983040 blocks [2/2] [UU] 2. Let /etc/cron.weekly/99-raid-check execute Actual results: md0 is being checked for consistency, md1 is not being checked for consistency. Expected results: md0 and md1 are being checked for consistency. Additional info: As soon as "echo check > /sys/block/md0/md/sync_action" is being issued /sys/block/md1/md/array_state changes from clean to active. Therefor when the array_state of md1 is being checked in line 25 of /etc/cron.weekly/99-raid-check the check for "active" == "clean" fails and thus md1 will never be checked.
The reporter is correct. Basically if you have two md devices living on different partitions of the same physical disks it fails to check the second one. This is not an uncommon occurrence e.g. /boot a RAID 1 (md0) on /dev/sda1, /dev/sdb1 LVM or / a RAID 5 (md1) with components /dev/sda2, /dev/sdb2 and others. You have to have the RAID 1 /boot so grub/BIOS can boot it (they can't do a RAID 5 boot). (or another reason /boot md0 being ext3 to make it bootable by grub) As soon at the "check" action is echo'd to /sys/block/md0/md/sync_action the other device (sharing the same physical disks) goes to active. There by it never gets set to check the second md device living on these physical devices. i.e # echo "check" >/sys/block/md0/md/sync_action ; cat /sys/block/md1/md/array_state active It works fine if the md device is on different underlying physical devices. How to fix? One way is to instead of having two loops an initiator and a result checker, change to just a single loop that initiates a check on that single md device. Then have an inner loop waiting for it to finish on that single device, then repeat on the next md device. Am I being over CPU sensitive, but does 3 seconds seem a bit low for a wait period? Every minute would seem a bit lighter, a 1 TB mirror takes a while to check. I do like the script reporting mismatch_cnt, that's really good.
Created attachment 367522 [details] Fixed script This updated script solves the problem. It gets the state from all the arrays on all the drives before it ever initiates the first check operation.
Script seems to work, once I commented out declare -A check Which doesn't seem present as a function when I run from the shell (maybe declared somewhere else in the cron scripts?) Maybe I'm being really complaining, but would it not be better to serialize the checks, rather than run them all at once?
That declare -A doesn't work means I'll have to modify the script slightly for the particular version of bash on rhel5 (I wrote the script with bash-4.0 on Fedora, and declare -A is a bash built-in there). As far as serializing the checks, the script allows for either check or repair operations, and we wait on one operation and not the other. Serializing would have no point of serialization in the event of repair operations. In addition, if we have multiple arrays on different drives/controllers then it makes no sense to serialize those check operations, especially considering that large arrays may take very significant amounts of time to check. We most definitely want those operations happening in parallel. In fact, it's only the specific case of multiple arrays on the same physical drives (common with /boot and / arrays, but almost never done for large data arrays) where we don't want things to run in parallel, and since the md raid layer will already make sure that doesn't happen for us, adding logic to detect that situation to what is arguably a very simple script would be redundant and simply another possible source of failure. So, no, I don't think serializing this script is the right way to go.
Shouldn't that be "declare -a check".
Created attachment 373146 [details] RHEL5 compatible version of script No, for the way it was being used, declare -A is correct. However, it's not supported by the version of bash in rhel5. So, I modified the script to do things differently. Updated script attached.
Both attached scripts are identical and use declare -A.
Created attachment 376776 [details] Updated RHEL5 compatible raid-check script Sorry, I failed to get the updated script the first time around. Re-attaching.
Verified, updated script works as expected.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2010-0006.html