Bug 1310264 - udev/mdadm timeouts at boot with very large mdadm devices
udev/mdadm timeouts at boot with very large mdadm devices
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: kernel (Show other bugs)
7.2
x86_64 Linux
unspecified Severity high
: rc
: 7.3
Assigned To: Jes Sorensen
guazhang@redhat.com
:
Depends On:
Blocks: 1298243
  Show dependency treegraph
 
Reported: 2016-02-19 17:20 EST by John Pittman
Modified: 2016-11-23 15:28 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-11-23 15:28:36 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description John Pittman 2016-02-19 17:20:34 EST
Created attachment 1128673 [details]
journalctl part 1

Description of problem:
With mdadm devices that are very large, at boot, the device will not create fast enough causing device timeout and udev settle to fail.

Version-Release number of selected component (if applicable):

3.10.0-327.4.5.el7.x86_64

How reproducible:
Create a very large mdadm device.  At boot, from what I can tell, assembly takes too long and we timeout.

Steps to Reproduce:
1.Create mdadm device
   - My example ~400TB w/ 74 legs
2.Format w/ XFS and add line to fstab
3.Allow to assemble, reboot

Actual results:

Boot fails with:

Feb 11 06:50:41 host systemd[1]: dev-md1.device changed dead -> tentative
Feb 11 06:50:41 host systemd[1]: Installed new job dev-md1.device/start as 2456
Feb 11 06:53:42 host systemd[1]: Job dev-md1.device/start timed out.
Feb 11 06:53:42 host systemd[1]: Job dev-md1.device/start finished, result=timeout
Feb 11 06:53:42 host systemd[1]: Timed out waiting for device /dev/md1.
Feb 11 06:53:42 host systemd[1]: Job dev-md1.device/start failed with result 'timeout'.

(1 of 2) A start job is running for dev-md1.device (51s / 1min 30s)
(1 of 2) A start job is running for dev-md1.device (51s / 1min 30s)
(2 of 2) A start job is running for udev Wait for Complete Device Initialization (49s / 2min 57s)
(2 of 2) A start job is running for udev Wait for Complete Device Initialization (49s / 2min 57s)
(2 of 2) A start job is running for udev Wait for Complete Device Initialization (50s / 2min 57s)
A start job is running for udev Wait for Complete Device Initialization (1min 57s / 2min 57s)
Failed to start udev Wait for Complete Device Initialization.
See 'systemctl status systemd-udev-settle.service' for details.
         Starting Activation of DM RAID sets...
 Started Activation of DM RAID sets.
Reached target Local File Systems.

UNIT                 ACTIVE    SUB        DESCRIPTION
dev-md1.device       loaded    activating tentative /dev/md1
systemd-udev-settle.service loaded    failed     failed    udev Wait for Complete Device Initialization

Expected results:

We should wait for the md device to complete initialization.  I found https://bugzilla.redhat.com/show_bug.cgi?id=1208680 ; I was going to ask there if this issue could be resolved by the possible fix mentioned there, but did not want to crowd the bug in case I was incorrect.

Additional info:

Workaround for issue is to add nofail to the fstab entry for the md device, then mount manually post boot.  The disks are "JBOD"; seagate ST6000NM0034 behind Fusion MPT SAS Host

I've had a bit of a hard time fully isolating things from the logs, so I will attach the full logs to the bug.  Also, still waiting on /proc/mdstat output, and array_state at the moment of failure; will update with that when I get it.
Comment 4 Jes Sorensen 2016-02-19 17:37:48 EST
74 legs?

Second, please provide some _reasonable_ log files. The files > 300MB and go
back 5 days - please reduce it to something that can actually be analyzed.

Last, you talk about failure - did the array synchronize prior to reboot, or
is it still in the process of needing to be synchronized?

It sounds like systemd kills the script during startup because it takes too
long. How long does it take if you assemble the array on the command line?

Jes
Comment 5 John Pittman 2016-02-19 17:45:09 EST
Thanks for looking into this Jes; I will get the information you asked for asap.

Wanted to mention as well, the customer has a second environment with 44 disk raid6 array showing the same symptoms.  I tried a recreate with 45 disks but was not successful; I am assuming due to disk size.  

For clarity /proc/mdstat from 74 disk array after boot.

md1 : active raid6 sdbl1[28] sdbn1[30] sdbr1[34] sdbu1[37] sdbd1[20] sdce1[48] sdau1[10] sdax1[13] sdaz1[15] sdbj1[26] sdbq1[33] sdbg1[23] sdbs1[35] sdca1[44] sdby1[41] sdv1[69] sds1[66] sdbf1[22] sdaw1[12] sdcc1[46] sdcf1[49] sdbx1[40] sdbi1[25] sdaq1[6] sdbk1[27] sdar1[7] sdap1[5] sdcb1[45] sdbt1[36] sdav1[11] sday1[14] sdbv1[38] sdbw1[39] sdao1[4] sdy1[72] sdbe1[21] sdba1[17] sdcg1[50] sdat1[9] sdal1[1] sdcd1[47] sdas1[8] sdbz1[42] sdam1[2] sdw1[70] sdbm1[29] sdbp1[32] sdbh1[24] sdbc1[19] sdbo1[31] sdx1[71] sdu1[68] sdbb1[18] sdt1[67] sdan1[3] sdak1[0] sdr1[65] sdz1[73] sdq1[64] sdp1[63] sdk1[58] sdj1[57] sdi1[56] sdh1[55] sdg1[54] sde1[52] sdf1[53] sdd1[51] sdo1[62] sdm1[60] sdl1[59] sdn1[61] sdc1[43] sdb1[16]
      421948071936 blocks super 1.2 level 6, 512k chunk, algorithm 2 [74/74] [UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU]
      bitmap: 20/44 pages [80KB], 65536KB chunk

John
Comment 7 John Pittman 2016-02-23 15:18:29 EST
Hi Jes,

I attached a smaller log file; only the messages from Feb 13th.  The customer is working within their next maintenance window to grab us the additional data you asked for.  Will update as soon as I get the next set of data.

John

Note You need to log in before you can comment on or make changes to this bug.