Bug 2227026 - [RAID 1] mdraid configured with hot spare disk does not start rebuild automatically when one of the leg got removed
Summary: [RAID 1] mdraid configured with hot spare disk does not start rebuild automat...
Keywords:
Status: ASSIGNED
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: mdadm
Version: 9.2
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Nigel Croxon
QA Contact: Storage QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-07-27 13:15 UTC by pratapsingh
Modified: 2023-08-18 01:10 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-163692 0 None None None 2023-07-27 13:18:25 UTC
Red Hat Issue Tracker RHELPLAN-163693 0 None None None 2023-07-27 13:18:30 UTC

Description pratapsingh 2023-07-27 13:15:15 UTC
Description of problem: [RAID 1] mdraid configured with hot spare disk does not start rebuilding automatically when one of the legs got removed if md raid is not mounted


Version-Release number of selected component (if applicable):
5.14.0-284.11.1.el9_2.x86_64
mdadm-4.2-8.el9.x86_64
Red Hat Enterprise Linux release 9.2 (Plow)

How reproducible:
Always

Steps to Reproduce:

1. Create a RAID1 with a spare leg

#mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sda /dev/sdb --spare-device=1 /dev/sdc

2. Once it's created remove one of the active legs

# echo 1 > /sys/block/sdb/device/delete 

3. Auto rebuilding has not started


Actual results:

1. As mentioned in step 2 above I removed the disk after that disk got removed successfully.

2. The disk was marked as "removed" however auto rebuilding doesn't happen automatically with the spare leg even after I waited for a few mins.

[root@~]# mdadm -D /dev/md1 
/dev/md1:
           Version : 1.2
     Creation Time : Thu Jul 27 07:36:07 2023
        Raid Level : raid1
        Array Size : 5237760 (5.00 GiB 5.36 GB)
     Used Dev Size : 5237760 (5.00 GiB 5.36 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Thu Jul 27 07:53:40 2023
             State : clean, degraded 
    Active Devices : 1
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 1

Consistency Policy : resync

              Name : 1
              UUID : 1094d9ed:3d47bc39:49fd5afc:2f352617
            Events : 37

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       -       0        0        1      removed

       2       8       32        -      spare   /dev/sdc

3. If adding any new disk rebuilding started automatically.

#  mdadm --manage /dev/md1 --add /dev/sdd
mdadm: added /dev/sdd

Consistency Policy : resync

    Rebuild Status : 13% complete

              Name : 1
              UUID : 1094d9ed:3d47bc39:49fd5afc:2f352617
            Events : 42

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       3       8       48        1      spare rebuilding   /dev/sdd

       2       8       32        -      spare   /dev/sdc



Expected results:

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       0       0        0        1      removed     
       2       8       32        -     spare rebuilding /dev/sdc

Additional info:

1. If /dev/md1 is mounted then auto rebuilding happens automatically if any leg got removed.
2. The status of the disk showing as removed in case /dev/md1 is not mounted. 
3. The status of the disk showing as missing if /dev/md1 is mounted.
4. I am not sure if it's expected behavior or a kind of bug because the issue is the same on all kernels RHEL7,8 and 9 and customer expectation is to rebuild or resync automatically if one leg got removed.

Comment 1 Nigel Croxon 2023-07-27 18:46:55 UTC
Expected behavior.  Not a bug.

Human intervention is required, as you have done to add the disk back into the array.

You also do not have a bitmap, which would make the resync faster when the disk is re-added.

Comment 2 pratapsingh 2023-07-28 05:51:12 UTC
(In reply to Nigel Croxon from comment #1)
> Expected behavior.  Not a bug.
> 
> Human intervention is required, as you have done to add the disk back into
> the array.
> 
> You also do not have a bitmap, which would make the resync faster when the
> disk is re-added.

Hello Nigel,

If /dev/md1 is mounted then auto rebuilding happens automatically if any leg got removed. And the status of the disk also shows as missing instead of removed.


[root@~]# mount /dev/md1 /test-raid
[root@~]# df -h /test-raid/
Filesystem      Size  Used Avail Use% Mounted on
/dev/md1        5.0G   68M  5.0G   2% /test-raid
[root@~]# mdadm -D /dev/md1 
/dev/md1:
           Version : 1.2
     Creation Time : Thu Jul 27 07:36:07 2023
        Raid Level : raid1
        Array Size : 5237760 (5.00 GiB 5.36 GB)
     Used Dev Size : 5237760 (5.00 GiB 5.36 GB)
      Raid Devices : 2
     Total Devices : 3
       Persistence : Superblock is persistent

       Update Time : Thu Jul 27 08:00:07 2023
             State : clean 
    Active Devices : 2
   Working Devices : 3
    Failed Devices : 0
     Spare Devices : 1

Consistency Policy : resync

              Name : 1
              UUID : 1094d9ed:3d47bc39:49fd5afc:2f352617
            Events : 57

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       3       8       48        1      active sync   /dev/sdd

       2       8       32        -      spare   /dev/sdc

--> I removed the active device

[root@~]# echo 1 > /sys/block/sdd/device/delete 

--> It will show as missing and after a few seconds md1 start auto-resync with the spare leg. 

[root@~]# mdadm -D /dev/md1 
/dev/md1:
           Version : 1.2
     Creation Time : Thu Jul 27 07:36:07 2023
        Raid Level : raid1
        Array Size : 5237760 (5.00 GiB 5.36 GB)
     Used Dev Size : 5237760 (5.00 GiB 5.36 GB)
      Raid Devices : 2
     Total Devices : 3
       Persistence : Superblock is persistent

       Update Time : Thu Jul 27 08:00:07 2023
             State : clean, degraded 
    Active Devices : 2
   Working Devices : 3
    Failed Devices : 0
     Spare Devices : 1

Consistency Policy : resync

              Name : 1
              UUID : 1094d9ed:3d47bc39:49fd5afc:2f352617
            Events : 57

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       3       8       48        1      active sync   missing

       2       8       32        -      spare   /dev/sdc
[root@~]# mdadm -D /dev/md1 
/dev/md1:
           Version : 1.2
     Creation Time : Thu Jul 27 07:36:07 2023
        Raid Level : raid1
        Array Size : 5237760 (5.00 GiB 5.36 GB)
     Used Dev Size : 5237760 (5.00 GiB 5.36 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Thu Jul 27 08:00:41 2023
             State : clean, degraded, recovering 
    Active Devices : 1
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 1

Consistency Policy : resync

    Rebuild Status : 3% complete

              Name : 1
              UUID : 1094d9ed:3d47bc39:49fd5afc:2f352617
            Events : 60

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       2       8       32        1      spare rebuilding   /dev/sdc

Why md1 does not start auto-rebuilding with the spare leg as with the mounted disk?

Regards,
Pratapsingh


Note You need to log in before you can comment on or make changes to this bug.