Bug 736521 - mdadm crash/oops when stopping array in installer environment
mdadm crash/oops when stopping array in installer environment
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
16
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Kernel Maintainer List
Fedora Extras Quality Assurance
AcceptedBlocker
:
: 737278 (view as bug list)
Depends On:
Blocks: F16Beta/F16BetaBlocker
  Show dependency treegraph
 
Reported: 2011-09-07 18:15 EDT by David Lehman
Modified: 2011-09-16 15:18 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-09-16 13:57:33 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
syslog (70.40 KB, text/plain)
2011-09-07 18:15 EDT, David Lehman
no flags Details
log of commands run by anaconda (30.54 KB, text/plain)
2011-09-07 18:17 EDT, David Lehman
no flags Details
20110913_preRC_boot.x86_64.iso rescue/VT1 (65.57 KB, image/png)
2011-09-14 05:06 EDT, Tom H
no flags Details
20110913_preRC_boot.x86_64.iso rescue/VT3 (60.23 KB, image/png)
2011-09-14 05:06 EDT, Tom H
no flags Details
20110913_preRC_boot.x86_64.iso rescue/VT4 (78.63 KB, image/png)
2011-09-14 05:07 EDT, Tom H
no flags Details
20110913_preRC_boot.x86_64.iso rescue/VT5 (23.89 KB, image/png)
2011-09-14 05:08 EDT, Tom H
no flags Details
contents of /tmp (93.28 KB, application/x-gzip)
2011-09-14 09:29 EDT, Clyde E. Kunkel
no flags Details

  None (edit)
Description David Lehman 2011-09-07 18:15:34 EDT
Created attachment 522014 [details]
syslog

Description of problem:
When trying to stop an array in the anaconda runtime environment in f16-beta.tc1 mdadm crashes, which seems to lead to an unkillable udev-spawned blkid process. The udev/blkid part is less clear to me than the mdadm oops.

To be clear mdadm says the array is stopped, and it appears to be stopped. However, any subsequent mdadm commands on that array seem to hang. Also, within a short time there is an 'add' event on md0 that triggers the unkillable blkid trying to probe the non-existent md0.

Version-Release number of selected component (if applicable):
mdadm-3.2.2-6

How reproducible:
Always, from what I can tell

Steps to Reproduce:
1. Boot some f16-beta.tc1 media
2. switch to shell on tty2
3. deactivate any active md array
  
Actual results:
mdadm seems to succeed except for the crash dump in the syslog and the fact that the stopped array is now unusable

Expected results:
array stopped successfully

Additional info:
In trying to track this down I made a custom image without 65-md-incremental.rules. I also manually started the rsyslog service from the shell on tty2 before running any other commands.
Comment 1 David Lehman 2011-09-07 18:17:27 EDT
Created attachment 522015 [details]
log of commands run by anaconda

This was not done by manually starting/stopping the array on tty2 -- I let anaconda do it. The result appears to be the same in either case.
Comment 2 Doug Ledford 2011-09-08 20:16:44 EDT
Dave, you said you made a custom image without 65-md-incremental.rules.  Did that custom image also show this problem (I suspect it did)?

Regardless, this is a kernel oops, not an mdadm bug.  It is, however, very likely that it's a kernel MD raid stack bug.
Comment 3 David Lehman 2011-09-09 11:09:03 EDT
(In reply to comment #2)
> Dave, you said you made a custom image without 65-md-incremental.rules.  Did
> that custom image also show this problem (I suspect it did)?

Yes, the problem still occurred without the incremental rules.

> 
> Regardless, this is a kernel oops, not an mdadm bug.  It is, however, very
> likely that it's a kernel MD raid stack bug.

Ok. Reassigning.
Comment 4 Doug Ledford 2011-09-09 11:55:55 EDT
I've pinged Neil Brown about this issue.  I suspect it's something that needs fixed prior to the final 3.1 kernel release and therefore he needed to be informed.
Comment 5 Josh Boyer 2011-09-09 13:37:23 EDT
(In reply to comment #4)
> I've pinged Neil Brown about this issue.  I suspect it's something that needs
> fixed prior to the final 3.1 kernel release and therefore he needed to be
> informed.

Bruno filed bug 737076 for some other raid related issues and has been working with Neil on them already.  Unfortunately, there isn't much info in that bug and bugzilla.kernel.org is down.  I thought I would mention it just in case it winds up being related.
Comment 6 Clyde E. Kunkel 2011-09-10 12:55:11 EDT
I filed bz 737278 which I expect is a dupe of this one.

I used syslog=192.168.0.11:6666 on the kernel cmd line.  It looks like it started.  However, on 192.168.0.11 machine, when I enter eval 'scripts/analog ....' I get:  bash: scripts/analog:  no such file or directory.  

What do I need to do to get the logs?

Thanks

(PS:  what about going back to the kernel used with the alpha tests?)
Comment 7 Adam Williamson 2011-09-12 11:36:07 EDT
*** Bug 737278 has been marked as a duplicate of this bug. ***
Comment 8 Tim Flink 2011-09-12 11:45:38 EDT
Discussed at the 2011-09-12 Fedora QA meeting. Accepted as a blocker for Fedora 16 Beta due to violation of the following criterion [1]:

The installer must be able to create and install to software, hardware or BIOS RAID-0, RAID-1 or RAID-5 partitions for anything except /boot

This bug breaks installation on systems that already have an existing mdraid and is suspected to cause problems with newly created mdraid arrays
Comment 9 Tom H 2011-09-12 15:48:06 EDT
Re: https://bugzilla.redhat.com/show_bug.cgi?id=737278#c6

This may or may not have been fixed in BZ 737076 mentioned above in comment #5 but I re-created an F16 mdraid install, attached the disks to another F16 install, and wasn't able to assemble the array.

[root@f16test ~]# lsblk
NAME      MAJ:MIN RM   SIZE RO MOUNTPOINT
sda         8:0    0     8G  0 
├─sda1      8:1    0     1M  0 
├─sda2      8:2    0     7G  0 /
└─sda3      8:3    0   700M  0 [SWAP]
sdb         8:16   0     8G  0 
├─sdb1      8:17   0     1M  0 
├─sdb2      8:18   0     7G  0 
│ └─md126   9:126  0         0 
└─sdb3      8:19   0   700M  0 
  └─md127   9:127  0         0 
sdc         8:32   0     8G  0 
├─sdc1      8:33   0     1M  0 
├─sdc2      8:34   0     7G  0 
│ └─md126   9:126  0         0 
└─sdc3      8:35   0   700M  0 
  └─md127   9:127  0         0 

[root@f16test ~]# cat /proc/mdstat 
Personalities : 
md126 : inactive sdb2[0] sdc2[1]
      14745576 blocks super 1.0
       
md127 : inactive sdb3[0] sdc3[1]
      1433576 blocks super 1.2
       
unused devices: <none>
[root@f16test ~]# mdadm --examine /dev/sdb2
/dev/sdb2:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x1
     Array UUID : 86e84327:d3cda684:aac981df:84084e50
           Name : localhost.localdomain:0
  Creation Time : Mon Sep 12 13:38:13 2011
     Raid Level : raid1
   Raid Devices : 2
 Avail Dev Size : 14745576 (7.03 GiB 7.55 GB)
     Array Size : 14745576 (7.03 GiB 7.55 GB)
   Super Offset : 14745584 sectors
          State : clean
    Device UUID : d699d597:a48cbf76:5903b724:e2672b04
Internal Bitmap : -8 sectors from superblock
    Update Time : Mon Sep 12 13:44:11 2011
       Checksum : 6d5c01d4 - correct
         Events : 27
   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing)

[root@f16test ~]# mdadm --examine /dev/sdc2
/dev/sdc2:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x1
     Array UUID : 86e84327:d3cda684:aac981df:84084e50
           Name : localhost.localdomain:0
  Creation Time : Mon Sep 12 13:38:13 2011
     Raid Level : raid1
   Raid Devices : 2
 Avail Dev Size : 14745576 (7.03 GiB 7.55 GB)
     Array Size : 14745576 (7.03 GiB 7.55 GB)
   Super Offset : 14745584 sectors
          State : clean
    Device UUID : 33d9f694:8468aa53:ce731827:f997aa13
Internal Bitmap : -8 sectors from superblock
    Update Time : Mon Sep 12 13:44:11 2011
       Checksum : 5948bd9e - correct
         Events : 27
   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing)

[root@f16test ~]# mdadm --detail /dev/md126
/dev/md126:
        Version : 1.0
  Creation Time : Mon Sep 12 13:38:13 2011
     Raid Level : raid1
  Used Dev Size : 7372788 (7.03 GiB 7.55 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent
    Update Time : Mon Sep 12 13:44:11 2011
          State : active, Not Started
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
           Name : localhost.localdomain:0
           UUID : 86e84327:d3cda684:aac981df:84084e50
         Events : 27
    Number   Major   Minor   RaidDevice State
       0       8       18        0      active sync   /dev/sdb2
       1       8       34        1      active sync   /dev/sdc2

[root@f16test ~]# mdadm --assemble /dev/md126 /dev/sdb2 /dev/sdc2
mdadm: cannot open device /dev/sdb2: Device or resource busy
mdadm: /dev/sdb2 has no superblock - assembly aborted

[root@f16test ~]# mdadm --assemble /dev/md127 /dev/sdb3 /dev/sdc3
mdadm: cannot open device /dev/sdb3: Device or resource busy
mdadm: /dev/sdb3 has no superblock - assembly aborted
Comment 10 Josh Boyer 2011-09-12 18:35:50 EDT
I've started a 3.1-rc6 build that contains some md fixes.  Please test this kernel when the build completes:

http://koji.fedoraproject.org/koji/taskinfo?taskID=3346292
Comment 11 Tim Flink 2011-09-13 23:15:55 EDT
I made a test boot iso with the 3.1-rc6 kernel for testing:

http://tflink.fedorapeople.org/iso/20110913_preRC_boot.x86_64.iso
http://tflink.fedorapeople.org/iso/20110913_preRC_boot.x86_64.iso.sha256

I don't have any raid arrays that I can test this with, if someone could try the new kernel out, that would be great.
Comment 12 Adam Williamson 2011-09-14 00:22:09 EDT
I wanted to reproduce this to test the fix, but I can't - TC2 can stop my Intel BIOS RAID-0 array just fine. Dunno if that helps narrow it down any, but it means we need David, Clyde and/or Tom to test the fix. Please grab the ISO from comment #11 and let us know if it helps. thanks!
Comment 13 Tom H 2011-09-14 05:06:18 EDT
Created attachment 523093 [details]
20110913_preRC_boot.x86_64.iso rescue/VT1

I tried Anaconda's rescue mode with my previous mdraid install. It failed but this time my laptop wasn't killed. I couldn't access the logs on VT2 but was able to screen dump the VTs.
Comment 14 Tom H 2011-09-14 05:06:58 EDT
Created attachment 523094 [details]
20110913_preRC_boot.x86_64.iso rescue/VT3
Comment 15 Tom H 2011-09-14 05:07:33 EDT
Created attachment 523095 [details]
20110913_preRC_boot.x86_64.iso rescue/VT4
Comment 16 Tom H 2011-09-14 05:08:02 EDT
Created attachment 523096 [details]
20110913_preRC_boot.x86_64.iso rescue/VT5
Comment 17 Clyde E. Kunkel 2011-09-14 09:22:44 EDT
(In reply to comment #11)
> I made a test boot iso with the 3.1-rc6 kernel for testing:
>

uname -r says this is a rc5 kernel.  Anyway, it looks like it fails in the same manner for me.  Will be attaching /tmp obtained via remote syslog shortly.
Comment 18 Clyde E. Kunkel 2011-09-14 09:29:18 EDT
Created attachment 523148 [details]
contents of /tmp

As mentioned in comment 17
Comment 19 Clyde E. Kunkel 2011-09-14 09:37:52 EDT
yeah, syslog shows:


[  183.790520] Pid: 2953, comm: mdadm Not tainted 3.1.0-0.rc5.git0.0.fc16.x86_64 #1 System manufacturer P5K-E/P5K-E                        ^^^
                                                           |
                                                           |
Note--------------------------------------------------------

So, wrong kernel used? Make a new boot.iso and I'll test.
Comment 20 Tim Flink 2011-09-14 10:05:36 EDT
(In reply to comment #17)
> (In reply to comment #11)
> > I made a test boot iso with the 3.1-rc6 kernel for testing:
> >
> 
> uname -r says this is a rc5 kernel.  Anyway, it looks like it fails in the same
> manner for me.  Will be attaching /tmp obtained via remote syslog shortly.

Good catch, thanks for letting me know. There was a typo in the command I used to build the iso and it pulled in rc5 instead of rc6.

New boot.iso (I double checked the kernel version this time) at:

http://tflink.fedorapeople.org/iso/20110914_preRC_boot2.x86_64.iso
http://tflink.fedorapeople.org/iso/20110914_preRC_boot2.x86_64.iso.sha256
Comment 21 Tom H 2011-09-14 10:29:57 EDT
(In reply to comment #20)
> 
> Good catch, thanks for letting me know. There was a typo in the command I used
> to build the iso and it pulled in rc5 instead of rc6.
> 
> New boot.iso (I double checked the kernel version this time) at:

I'm downloading it and will try it asap.

FYI:

(1) After trying rescue mode (comment #13), I tried a new install and it froze my computer after installing 10 packages. No way to get any data but since it was the previous kernel, I won't try again to see if I can preserve something.

(2) I installed 3.1-rc6 on another F16 install and the arrays from my failed install were active and mountable unlike in comment #9, so I'm hopeful that your new iso'll work...
Comment 22 Clyde E. Kunkel 2011-09-14 11:11:12 EDT
Success!!!  Install is humming along nicely with rc6 kernel, LVs over raid10 and a mirror raid present.

Good work all.  Thanks!!

--
Regards
OldFart
Comment 23 Tom H 2011-09-14 11:39:04 EDT
I was able to install F16, stop the array within the anaconda environment, and then boot into anaconda's rescue mode and mount and chroot into the array.

But I'm back to booting to an unusable "grub rescue" prompt.
Comment 24 Chuck Ebbert 2011-09-15 14:16:11 EDT
Does everyone agree that 3.1-rc6 fixes this bug?
Comment 25 Clyde E. Kunkel 2011-09-15 14:56:30 EDT
(In reply to comment #24)
> Does everyone agree that 3.1-rc6 fixes this bug?

Comment 22 for me is a yes.
Comment 26 Adam Williamson 2011-09-15 18:04:27 EDT
it would be best if people can test with Beta RC1:

http://dl.fedoraproject.org/pub/alt/stage/16-Beta.RC1/

to confirm. if this issue is fixed but you run into another later, please file that separately (of course, first check if someone else has already filed it).

thanks!
Comment 27 Doug Ledford 2011-09-16 13:57:33 EDT
(In reply to comment #23)
> I was able to install F16, stop the array within the anaconda environment, and
> then boot into anaconda's rescue mode and mount and chroot into the array.
> 
> But I'm back to booting to an unusable "grub rescue" prompt.

Tom, this is certain to be something wrong with the grub installation (the "grub rescue prompt thing) and not a problem with your raid arrays.  The specific problem you ran into with your raid arrays was that mdadm was not able to request the kernel module to support your raid level (which is why the arrays showed up in /proc/mdstat, but weren't running, as the raid1 personality wasn't loaded).  So, the issue in comment #9 was fixed by the selinux update, starting and stopping arrays was fixed by this bug, and now we need to get to the bottom of the boot issue on your machine (which may just boil down to you need to select a different install point for grub).  In any case, your remaining issue isn't related to this bug, and I hereby declare this issue fixed based upon all the feedback present.
Comment 28 Tom H 2011-09-16 15:18:21 EDT
I agree that my "grub rescue" problem isn't related to this bug; I'm sorry that I wasn't clearer in #23. I've been too busy to report the grub2 problem but I'll do so this weekend. (FYI, "grub2-install..." in Anaconda's rescue mode fails but "grub2-mkimage ...; grub2-setup ..." succeeds.) Thanks for the mdstat explanation; I hadn't thought of checking that!

Note You need to log in before you can comment on or make changes to this bug.