Bug 1700451
| Summary: | Booting with a large number of multipath devices drops into emergency shell | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Ben Marzinski <bmarzins> | |
| Component: | device-mapper-multipath | Assignee: | Ben Marzinski <bmarzins> | |
| Status: | CLOSED ERRATA | QA Contact: | Lin Li <lilin> | |
| Severity: | unspecified | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | --- | CC: | agk, bmarson, bmarzins, heinzm, jbrassow, lilin, msnitzer, prajnoha, rhandlin, rpeterso, toneata, ttracy, zkabelac | |
| Target Milestone: | rc | Keywords: | ZStream | |
| Target Release: | 8.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | device-mapper-multipath-0.8.0-2.el8 | Doc Type: | Bug Fix | |
| Doc Text: |
Cause:
When multipath is determining whether it should claim a block device as a path device in udev, it checks if multipathd is running by openning a socket connection to it. If multipathd hasn't started up yet adn there are a large number of block devics, this can hang, causing udev to hang as well.
Consequence:
udev processing for block devices can be delayed on bootup, possibly causing the bootup to fail to the emergency shell.
Fix:
multipath now tries to connect to the multipathd socket in a nonblocking manner. If that fails, it looks at the error to determine if multipathd will be starting up.
Result:
multipath no longer causes udev processing of block devices to hang in setups with a large number of block devices.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1723746 (view as bug list) | Environment: | ||
| Last Closed: | 2019-11-05 22:18:16 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1723746 | |||
Think there is a more fundamental issue for this particular testbed: the root volume is _not_ using multipathing; so the initramfs shouldn't even be doing anything related to multipath. The initramfs isn't doing anything with multipathing. But the /home directory isn't being mounted in the initramfs, and after the pivot root, the multipath stall waiting for multipathd on all the other devices is causing the local device used for /home to not get re-initialized in time. Now, it does seem that since the local device is currently there with active LVs already using it for the root directory and /boot, that udev should know about it and not need to recheck it before it can mount /home. But regardless, multipath shouldn't have to wait for multipathd to get started before it can even begin to check if a device should be claimed as a multipath path. The intention of the code was not to wait, it's just that the socket autoactivation makes this happen. I have a patch that checks if multipathd is running without accessing the socket, so that it doesn't trigger this. The other solution is to drop the autoactivation, since multipathd should always be running. A different solution was agreed upon upstream, so the test packages do not reflect the actual solution. Instead of not accessing the multipathd socket, the multipath -u command now tries to open it non-blocking, and on failure checks the error code to see if multipathd will be starting up later. Hello Barry, Could you provide test result following up your steps with fixed version ? I will reproduce this issue with a large number of scsi_debug devices. Thanks in advance! Hello Barry, Could you provide me steps to reproduce? I want to reproduce it using your steps. Thanks in advance! I simply have a large number of multipath devices. In my case, there are 48x8(multipath) LUNS. Upon booting, a local volume group does not properly initialize and we drop into maintenance mode Barry Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:3578 |
Description of problem: If there are a large number of devices on a system, and device-mapper-multipath is configured, even if it is configured to blacklist all of the devices. It can cause the system to timeout trying to initialize devices needed for filesystem mounts on bootup, and drop into the emergency shell. Version-Release number of selected component (if applicable): device-mapper-multipath-0.7.8-7.el8 How reproducible: Always on effected setups Steps to Reproduce: 1. Install RHEL8 on a node with a large number of devices 2. Set up multipathing 3. Update the node. I'm not sure why this set appears necessary to reproduce the issue Actual results: On boot, the node drops into the emergency shell. The devices will be initialized automatically, so no action needs to be taken except exitting the emergency shell when the filesystems have mounted. Expected results: The node will not drop into the emergecny shell on boot. Additional info: This issue appears to be caused by multipath attempting to determine if it should claim devices (set ENV{DM_MULTIPATH_DEVICE_PATH}="1") in 62-multipath.rules. As part of doing this, it checks if multipathd is running. In rhel-8, this triggers a socket activation of multipathd through the /lib/systemd/system/multipathd.socket file. It appears that sometimes, this can take some time, and while that is happening, the multipath command is stalled. This bug can be worked around simply by removing /lib/systemd/system/multipathd.socket Removing that file shouldn't cause other issues, since multipathd should be running all the time anyway. Also, if multipathd isn't currently running, multipath checks if systemd has multipathd enabled, so that it will be started, and acts as the same as if multipath is running. However, it should be possible to fix this so that socket autoactivation remains, but the multipath claiming code doesn't trigger the autoactivation.