Bug 1260728

Summary: multipathd is inactive (dead) after boot
Product: Red Hat Enterprise Linux 7 Reporter: Martin Hoyer <mhoyer>
Component: device-mapper-multipathAssignee: Ben Marzinski <bmarzins>
Status: CLOSED ERRATA QA Contact: Martin Hoyer <mhoyer>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.2CC: agk, bgoncalv, bmarzins, hannsj_uhl, heinzm, jbrassow, mhoyer, msnitzer, prajnoha, tlavigne
Target Milestone: rcKeywords: TestBlocker
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: device-mapper-multipath-0.4.9-85.el7 Doc Type: Bug Fix
Doc Text:
No documentation necessary. This bug was caused by a rhel-7.2 fix. Code with this issue never went to customers.
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-19 12:58:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Martin Hoyer 2015-09-07 14:18:21 UTC
Description of problem:
multipathd service is dead after booting into OS.

So far this issue is only seen on iSCSI boot server with qla8200 HBA

#systemctl status multipathd -l
● multipathd.service - Device-Mapper Multipath Device Controller
   Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Mon 2015-09-07 13:14:11 CEST; 2h 2min ago
 Main PID: 390 (code=exited, status=0/SUCCESS)

multipathd[390]: path checkers start up
systemd[1]: Started Device-Mapper Multipath Device Controller.
multipathd[390]: sda: add path (uevent)
multipathd[390]: mpatha: load table [0 104878080 multipath 0 0 1 1 service-time 0 1 1 8:0 1]
multipathd[390]: mpatha: event checker started
multipathd[390]: sda [8:0]: path added to devmap mpatha
multipathd[390]: sdb: add path (uevent)
multipathd[390]: mpatha: load table [0 104878080 multipath 0 0 2 1 service-time 0 1 1 8:0 1 service-time 0 1 1 8:16 1]
multipathd[390]: sdb [8:16]: path added to devmap mpatha
multipathd[390]: sdc: add path (uevent)


# multipathd show paths
ux_socket_connect: Connection refused

# multipath -l
mpatha (360fff19abdd9552f8a36e5355226ba27) dm-0 EQLOGIC ,100E-00         
size=50G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| `- 0:0:0:0 sda 8:0  active undef running
`-+- policy='service-time 0' prio=0 status=enabled
  `- 1:0:0:0 sdb 8:16 active undef running

Version-Release number of selected component (if applicable):
kernel-3.10.0-313.el7
device-mapper-multipath-0.4.9-82.el7

How reproducible:
100%

Steps to Reproduce:
1.Boot to OS
2.Check multipathd status

Actual results:
multipathd service is inactive(dead)

Expected results:
multipathd service is active

Comment 5 Martin Hoyer 2015-09-15 13:43:30 UTC
Hi Ben,
abrt is not reporting any problems. Do You have any suggestion what should I do to provide more info about this?

Only thing I found out is this from dmesg:
systemd[1]: Cannot add dependency job for unit multipathd.service, ignoring: Unit blk-availability.service failed to load: No such file or directory.

# systemctl is-enabled multipathd.service
enabled

Do you want to loan the system for debug?

Martin

Comment 6 Ben Marzinski 2015-09-15 18:42:46 UTC
(In reply to Martin Hoyer from comment #5)
> Hi Ben,
> abrt is not reporting any problems. Do You have any suggestion what should I
> do to provide more info about this?
> 
> Only thing I found out is this from dmesg:
> systemd[1]: Cannot add dependency job for unit multipathd.service, ignoring:
> Unit blk-availability.service failed to load: No such file or directory.

The multipath.service now depends on the blk-availability.service, which should be installed with device-mapper >= 1.02.96, which is the rpm was updated to require. I'm sure that what's causing this. I'll double-check that I'm requiring the correct version.


> Do you want to loan the system for debug?

No. With that dmseg output, I can probably track this down on my machines. Thanks.

> Martin

Comment 7 Ben Marzinski 2015-09-17 00:46:28 UTC
So the issue is that I didn't include the epoch in the device-mapper version I was Requiring, which caused every RHEL7 version of device-mapper to be new enough.  I'll update the Requires line.

If the blocker flag gets set, I'm gladly push this into 7.2. However, the rhel-7.2 version of device-mapper is new enough, so if a user updates all their packages, this won't be an issue.  Also, the 7.1 version of lvm2 included the necessary file.  So to hit this, a user would need to update lvm2 and device-mapper-multipath, but not device-mapper, to the 7.2 versions.

Comment 18 Martin Hoyer 2015-09-30 13:19:37 UTC
The issue is not present with RHEL-7.2-20150924.0 
However there is still this message:
dmesg | grep multipath
[    9.796642] systemd[1]: Cannot add dependency job for unit multipathd.service, ignoring: Unit blk-availability.service failed to load: No such file or directory.

blk-availability is active and enabled by default.

Is this expected? Can I verify it?

kernel-3.10.0-319.el7.x86_64
device-mapper-multipath-0.4.9-85.el7.x86_64
lvm2-2.02.130-2.el7.x86_64

Comment 19 Ben Marzinski 2015-09-30 18:28:12 UTC
This is because blk-availability.service isn't in the initramfs. However since it's job is to deactivate block devices correctly during shutdown, I don't think it needs to be.  I'm not sure if there is a way to make multipathd.service not require it during startup. At any rate, as long as it is active once the system has booted, everything should be fine.

Comment 20 Ben Marzinski 2015-09-30 22:51:03 UTC
With the latest RHEL-7.2 build, multipathd is working fine after boot for me as well. You can have the machine back.

Comment 21 Ben Marzinski 2015-09-30 22:52:32 UTC
So, I'd say that we can safely verify this, and consider opening a new bug for the

[    9.796642] systemd[1]: Cannot add dependency job for unit multipathd.service, ignoring: Unit blk-availability.service failed to load: No such file or directory.

message.

Comment 22 Martin Hoyer 2015-10-01 07:04:03 UTC
This bug is not present anymore in RHEL-7.2-20150924.0 build with device-mapper-multipath-0.4.9-85.el7

Comment 23 errata-xmlrpc 2015-11-19 12:58:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2132.html