Bug 543044

I've read the threads you listed from the centos mailing lists.  There is a misunderstanding on your part and on Jancio Wodnik's part.  The script is *not* rebuilding your array, despite the messages in your kernel logs (whether you initiate a rebuild, or a simple check, the kernel logs look the same).  The script is running a 'check' operation, which doesn't rebuild anything, it only reads every sector on each device so that we have a correct mismatch_cnt (if we didn't scan the drives, we wouldn't know for sure what the mismatch_cnt is).  This is not a bug, and it is not scanning your arrays *because* mismatch_cnt is non-0, it's scanning them to see *if* your mismatch_cnt is non-0.  If you don't want the script to scan the drives weekly to see what the mismatch_cnt is, then add these decives to the SKIP_DEVS list in /etc/sysconfig/raid-check.

it seems you do NOT read the whole thread! it IS rebuild all of my raid1 array in all weekend. it's turn out the in case of raid1 there can be none 0 mismatch_cnt even without real error. so the script /etc/cron.weekly/99-raid-check start the repair in case of none 0 mismatch_cnt in case of raid1 too.

and yes i understand that i can trun it off byt setting all md device in /etc/sysconfig/raid-check to SKIP_DEVS. but it's still a bug since by default all raid1 device are scanned and all none 0 mismatch_cnt are start to repair ie. rebuild.

(In reply to comment #2)
> it seems you do NOT read the whole thread! it IS rebuild all of my raid1 array
> in all weekend.

It runs a check of your arrays every weekend.  That's different than a rebuild, and it is intentional.  Even for raid1 arrays.

> it's turn out the in case of raid1 there can be none 0
> mismatch_cnt even without real error.

It is possible on swap partitions that this occurs, but it should be rare that it would occur on a non-swap raid1 device.  So, we don't treat a non-0 mismatch count on a raid1 device any differently than we do on another device.

> so the script
> /etc/cron.weekly/99-raid-check start the repair in case of none 0 mismatch_cnt
> in case of raid1 too.

Unless you go in and change the configuration to be a repair, it's a check.  They are different.

> and yes i understand that i can trun it off byt setting all md device in
> /etc/sysconfig/raid-check to SKIP_DEVS. but it's still a bug since by default
> all raid1 device are scanned and all none 0 mismatch_cnt are start to repair
> ie. rebuild.

No, *none* of them do anything on a non-0 mismatch count.  Have you read the 99-raid-check script?  If we find a non-0 mismatch count, we print a warning, and then we exit the script.  It does *NOTHING* besides print a warning about the non-0 mismatch count.  So how exactly do you expect the script to start a repair/rebuild when all it does once it finds a non-0 mismatch_cnt is to exit?

So, I'll reiterate how the script works:

It runs a 'check' operation on all devices - which we want, including raid1 devices
It waits for the 'check' to complete
When the 'check' is complete it tests the mismatch_cnt of all devices it ran a check on - again which we want, including raid1 devices as mismatch_cnt should be rare on raid1 devices even though it is possible for it to exist
We print out the names of any devices that had a non-0 mismatch_cnt, then we exit

So, where in that process is the script rebuilding your raid1 array?  Nowhere.  And where is it repairing your raid1 array?  Nowhere.  And should we skip raid1 arrays because it's technically possible to have a non-0 mismatch_cnt?  No.  Amongst other issues, the fact that a non-0 mismatch_cnt is possible on anything other than a swap partition is considered an issue that needs resolved and upstream is currently considering just *how* to solve the problem, the fact that this script makes people aware of the issue helps put pressure on upstream to resolve the problem.  Hence, it should stay as it is.

yes you're right that the script "looks like" this, BUT it's still rebuild all of my raid1 array on all weekend. and i double check:

in /etc/sysconfig/raid-check there is "check" and nit "repair":
==========================================
ENABLED=yes
CHECK=check
# To check devs /dev/md0 and /dev/md3, use "md0 md3"
CHECK_DEVS=""
REPAIR_DEVS=""
SKIP_DEVS="md1"
==========================================

in my /etc/mdadm.conf
==========================================
PROGRAM /usr/local/sbin/md-notify
==========================================

in my /usr/local/sbin/md-notify:
==========================================
#!/bin/bash
# usage: md-notify event device <component>

if [ "NewArray" == $1 -o "SparesMissing" == $1 ]; then
  exit;
fi


DATE=`date`;
mail -s "RAID event on $HOSTNAME" root <<EOF
---------------- E R R O R -----------------
mdadm report the following error:
Host:		$HOSTNAME
Date:		$DATE
Event:		$1
Device:		$2
Component:	$3
---------------- E R R O R -----------------
EOF
==========================================

and i've got 6 email from all of my raid1 arrays as start to rebuild and 20/40/60/80/finished rebuild.
are you sure there is no some hidden "feature" which cause the rebuilding?
we can wait till the next weekend and i can reopen this bug then...
i'm just check and i can't find any non zero mismatch_cnt:-(

It's still just a check.  In older kernels, they hadn't yet included different printk messages for different operations, so whether it was a check, a repair, or a rebuild caused for some other reason, the kernel messages all say the same thing.  You can verify this yourself by doing the following:

echo "check" > /sys/block/md<number>/md/sync_action
cat /sys/block/md<number>/md/sync_actino
cat /proc/mdstat

Do this and you will see for yourself that it is not rebuilding your arrays, but instead mdmonitor and your notify program are reading the check as a rebuild even when it isn't.

-------------------------------------
[root@atom ~]# echo "check" > /sys/block/md1/md/sync_action
[root@atom ~]# cat /sys/block/md1/md/sync_action 
check
[root@atom ~]# mdadm --detail /dev/md1
/dev/md1:
        Version : 0.90
  Creation Time : Mon Apr 20 19:38:47 2009
     Raid Level : raid1
     Array Size : 8385792 (8.00 GiB 8.59 GB)
  Used Dev Size : 8385792 (8.00 GiB 8.59 GB)
   Raid Devices : 3
  Total Devices : 3
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Wed Dec  2 23:33:53 2009
          State : clean, resyncing
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

 Rebuild Status : 0% complete

           UUID : 97590506:59e51a72:33f0a4a7:8c0ab238
         Events : 0.14

    Number   Major   Minor   RaidDevice State
       0       8       49        0      active sync   /dev/sdd1
       1       8       65        1      active sync   /dev/sde1
       2       8       81        2      active sync   /dev/sdf1
[root@atom ~]# cat /proc/mdstat 
Personalities : [raid1] [raid6] [raid5] [raid4] 
md1 : active raid1 sdf1[2] sde1[1] sdd1[0]
      8385792 blocks [3/3] [UUU]
      [=>...................]  resync =  5.0% (426688/8385792) finish=1.2min speed=106672K/sec
      
md2 : active raid6 sdf2[5] sde2[4] sdd2[3] sdc2[0] sdb2[1] sda2[2]
      3873496064 blocks level 6, 256k chunk, algorithm 2 [6/6] [UUUUUU]
      
md0 : active raid1 sdc1[0] sdb1[1] sda1[2]
      8385792 blocks [3/3] [UUU]
      
unused devices: <none>
-------------------------------------
are you sure this is not rebuilding????
and from my weekend log does it takes 6 hours to check?:
---------------- E R R O R -----------------
mdadm report the following error:
Host:           atom.int.bppiac.hu
Date:           Sun Nov 29 04:22:01 CET 2009
Event:          RebuildStarted
Device:         /dev/md2
Component:
---------------- E R R O R -----------------
---------------- E R R O R -----------------
mdadm report the following error:
Host:           atom.int.bppiac.hu
Date:           Sun Nov 29 10:12:37 CET 2009
Event:          RebuildFinished
Device:         /dev/md2
Component:
---------------- E R R O R -----------------

so in conclusion i don't know it's a kernel raid bug or mdadm bug or something else, but i'm sure that this new /etc/cron.weekly/99-raid-check cause that all of my raid1 array always rebuild.

(In reply to comment #6)
> -------------------------------------
> [root@atom ~]# echo "check" > /sys/block/md1/md/sync_action
> [root@atom ~]# cat /sys/block/md1/md/sync_action 
> check

See, it's check, not rebuild, not repair.  It's check.  Just like it's supposed to be.

> [root@atom ~]# mdadm --detail /dev/md1
> /dev/md1:
>         Version : 0.90
>   Creation Time : Mon Apr 20 19:38:47 2009
>      Raid Level : raid1
>      Array Size : 8385792 (8.00 GiB 8.59 GB)
>   Used Dev Size : 8385792 (8.00 GiB 8.59 GB)
>    Raid Devices : 3
>   Total Devices : 3
> Preferred Minor : 1
>     Persistence : Superblock is persistent
> 
>     Update Time : Wed Dec  2 23:33:53 2009
>           State : clean, resyncing

And here too, the array state is clean and resyncing.  Which is correct.  If it were rebuilding or repairing, it wouldn't be clean.  The resyncing is just signalling the fact that we are in the middle of a check, the clean tells it is a check and not a rebuild.

> [root@atom ~]# cat /proc/mdstat 
> Personalities : [raid1] [raid6] [raid5] [raid4] 
> md1 : active raid1 sdf1[2] sde1[1] sdd1[0]
>       8385792 blocks [3/3] [UUU]
>       [=>...................]  resync =  5.0% (426688/8385792) finish=1.2min
> speed=106672K/sec

mdstat overloads resync and uses it for multiple conditions, one of which is the check condition.

> -------------------------------------
> are you sure this is not rebuilding????

Yes, 100% positive.  In fact, if it were rebuilding, your mismatch_cnt would go to 0 because a rebuild would forcefully correct the mismatch_cnt not being 0.

> and from my weekend log does it takes 6 hours to check?:

It takes exactly the same amount of time to check as it would to rebuild.  The only difference between the two is that a check reads every sector on both devices looking for differences, while a rebuild reads every sector on both devices and when it finds differences it writes from one device to the other to eliminate the differences.  That's the only difference between a check and a repair/rebuild, and that's why they take the same amount of time.

> ---------------- E R R O R -----------------
> mdadm report the following error:
> Host:           atom.int.bppiac.hu
> Date:           Sun Nov 29 04:22:01 CET 2009
> Event:          RebuildStarted
> Device:         /dev/md2
> Component:
> ---------------- E R R O R -----------------
> ---------------- E R R O R -----------------
> mdadm report the following error:
> Host:           atom.int.bppiac.hu
> Date:           Sun Nov 29 10:12:37 CET 2009
> Event:          RebuildFinished
> Device:         /dev/md2
> Component:
> ---------------- E R R O R -----------------

The check operation is fairly new, and there was no CheckStarted or CheckFinished event when the current md event API was written.  So, the Rebuild event was overloaded to also be used for the Check events.  I'm sorry that this has caused your little notify program to confuse you into believing that a rebuild is happening when a check is happening instead, but that is precisely what has happened here.

> so in conclusion i don't know it's a kernel raid bug or mdadm bug or something
> else, but i'm sure that this new /etc/cron.weekly/99-raid-check cause that all
> of my raid1 array always rebuild

This is the last time I will say it: your arrays are not being rebuilt, they are being checked.  If you don't want them checked, then disable that in the config file /etc/sysconfig/raid-check.  There is no bug as the script is *supposed* to check your raid arrays weekly.

(In reply to comment #7)
> Yes, 100% positive.  In fact, if it were rebuilding, your mismatch_cnt would go
> to 0 because a rebuild would forcefully correct the mismatch_cnt not being 0.

after the weekend mismatch_cnt always 0 and i assume before the /etc/cron.weekly/99-raid-check is should have to be none 0 otherwise the check don't run. so it's check or rebuild?
 
> This is the last time I will say it: your arrays are not being rebuilt, they
> are being checked.  If you don't want them checked, then disable that in the
> config file /etc/sysconfig/raid-check.  There is no bug as the script is
> *supposed* to check your raid arrays weekly.  

ok let we see from other point of view:
why it's bothers me if my arrays rebuild when it's not needed? because it cause a huge io load and my server becomes less usable. so from my (the server user) point of view it's totally unimportant whether it's a check or a repair when it's not needed both are bug. it's something like my daily mlocate scripts running all day and cause high io load. of course it's not a bug as rebuild wouldn't be a bug if it's needed, but it's useless.

(In reply to comment #8)
> (In reply to comment #7)
> > Yes, 100% positive.  In fact, if it were rebuilding, your mismatch_cnt would go
> > to 0 because a rebuild would forcefully correct the mismatch_cnt not being 0.
> 
> after the weekend mismatch_cnt always 0 and i assume before the
> /etc/cron.weekly/99-raid-check is should have to be none 0

Then you assume incorrectly.

> otherwise the check
> don't run.

Incorrect.  The check runs regardless of what mismatch_cnt is prior to the check.

> so it's check or rebuild?

It's a check.  It's purpose is to scan the drives looking for errors.  If it only ran when it knew errors existed, then it wouldn't be a check.

> > This is the last time I will say it: your arrays are not being rebuilt, they
> > are being checked.  If you don't want them checked, then disable that in the
> > config file /etc/sysconfig/raid-check.  There is no bug as the script is
> > *supposed* to check your raid arrays weekly.  
> 
> ok let we see from other point of view:
> why it's bothers me if my arrays rebuild when it's not needed?

It is a preventative measure designed to catch sectors that have gone bad but have not been read recently.  In general, most servers don't read large parts of their disks for months on end.  In that time, sectors that haven't been read can silently go bad.  This weekly check scans the entire disk looking for bad sectors.  If any bad sectors are found, they are corrected from data off of other disks.  In the process of scanning for bad sectors, it also verifies that the data on each disk is consistent.  If it finds any sectors that are not consistent, it reports that fact via mismatch_cnt.  Whether or not it is needed is a matter of preference.  If you want the system to performance proactive scans of your disks to find sectors that have gone bad since the last time they were accessed, then it's needed.  If you don't, then it's not.  That's why there is a config file, so you can turn it on or off.

> because it cause
> a huge io load and my server becomes less usable. so from my (the server user)
> point of view it's totally unimportant whether it's a check or a repair when
> it's not needed both are bug.

And you are free to turn it off.  Other users disagree about the importance of proactive measures to find bad sectors before they cause a problem later on, and it's one of those users that filed the bug requesting this feature and for those users in general that this feature was enabled.

> it's something like my daily mlocate scripts
> running all day and cause high io load. of course it's not a bug as rebuild
> wouldn't be a bug if it's needed, but it's useless.

This is an issue for me too, that most of our RAID 1 systems email us every week about mismatch_cnt. Not so useful to get 20 odd emails every week when there is no real issue. I'm surprised it's closed, it certainly is a bug in one sense it's reporting a problem when none really exits (okay a potential problem but not helpful to a busy sysadmin).

On systems without swap on the devices I see failing I'm betting on memory mapped files (or some systems that have VM filesystems) being the problem for RAID 1 devices.

It's a difficult one to decide on the best approach to fix. 

Either:

1/ Have some setting allowing the user to skip mismatch checking on certain devices (but still cause it to do the "check", thereby bad blocks will still be detected). Basic workaround solution.

2/ Don't do mismatch_cnt checking on RAID 1 devices (as this functionality in some level is broken in the kernel, it's telling you something but being no zero doesn't indicate an error). Maybe tunable in /etc/sysconfig ?

3/ Is there a kernel tuning parameter that allows the writes to be buffered in RAID 1 (as the RAID 5 md driver does)? Is so, turn it on at the start of the script and off at then end. Then the performance degradation associated with doing this will only be seen during this checking and not all the time. At some level there is a kernel bug here, the kernel probably should buffer RAID 1 writes during check/repair operations would maybe be a sensible thing to do (with a buffer flush).

Hi Colin,

This bug is closed because the original reporter was claiming that even though the script purported to be performing a check, he was convinced it was performing a repair instead.  This isn't true, it's just that /proc/mdstat uses resync as the state whether it's doing a check or a repair.  Once I cleared that up, I closed this bug out.  You're situation, frequent non-0 mismatch_cnt results, is also a known issue and something I intend to address.  I have a slightly modified raid-check script that simply doesn't perform a mismatch_cnt check on raid1 devices as it's valid for them to have a non-0 mismatch_cnt under a number of different circumstances.  If you would like to be included on a rhel5 bug for the mismatch_cnt issue, feel free to add yourself to bug 566828.