Red Hat Bugzilla – Bug 431689
After update, multipathd does not start any more
Last modified: 2008-05-21 12:06:59 EDT
Description of problem:
After doing a "yum update", the multipathd does not run any more.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
# service multipathd start
Starting multipathd daemon: [ OK ]
# service multipathd status
multipathd dead but pid file exists
The machine is multipath-booted from a Netapp Filer:
/dev/mapper/smtp3p5 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/mapper/smtp3p1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
/dev/mapper/smtp3p2 on /var type ext3 (rw)
/dev/mapper/smtp3p3 on /var/spool type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
I think the problem is related to selinux because after doing a "setenforce 0",
"ps -ef" shows at least a running daemon. But /var/cache/multipathd is empty,
mount does not show a ramfs and there are strange error messages in the syslog.
I already unsuccessfully did a "touch /.autorelabel" and a reboot.
"strace -F -f /sbin/multipathd" sometimes shows this:
[pid 6398] open("/sbin/dasd_id", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 6398] open("/sbin/gnbd_import", O_RDONLY) = -1 ENOENT (No such file or
[pid 6398] mount(NULL, "/var/cache/multipathd", "ramfs"..., MS_SYNCHRONOUS,
"maxsize=530258"...) = -1 EACCES (Permission denied)
[pid 6398] exit_group(1) = ?
Process 6398 detached
/var/log/messages: (only after doing "setenforce 0")
Feb 6 15:19:15 smtp3 multipathd: cannot open mpath_prio_hds_modular : No such
file or directory
Feb 6 15:19:15 smtp3 multipathd: cannot open /sbin/dasd_id : No such file or
Feb 6 15:19:15 smtp3 multipathd: cannot open /sbin/gnbd_import : No such file
Feb 6 15:19:15 smtp3 multipathd: [copy.c] cannot open mpath_prio_hds_modular
Feb 6 15:19:15 smtp3 multipathd: cannot copy mpath_prio_hds_modular in ramfs :
No such file or directory
Feb 6 15:19:15 smtp3 multipathd: [copy.c] cannot open /sbin/dasd_id
Feb 6 15:19:15 smtp3 multipathd: cannot copy /sbin/dasd_id in ramfs : No such
file or directory
Feb 6 15:19:15 smtp3 multipathd: [copy.c] cannot open /sbin/gnbd_import
Feb 6 15:19:15 smtp3 multipathd: cannot copy /sbin/gnbd_import in ramfs : No
such file or directory
Feb 6 15:19:15 smtp3 multipathd: smtp3: event checker started
Created attachment 294108 [details]
Created attachment 294109 [details]
grep multipath /var/log/audit/audit.log
Created attachment 294240 [details]
Selinux policy fix.
This patch applies cleanly to selinux-policy-2.4.6-106.el5_1.3 and
This is dumb oversight on my part. I accidentally tested the fix for bz #428338
on a machine with selinux disabled. The fix for this bug requires a selinux
policy change. Without the policy change, multipathd won't work at all.
Unfortunately, bz #428338 has a RHEL 5.1.z zstream fix associated with it, bz
#355961, so the selinux policy needs to be fixed there as well. The patch in
Comment #3 fixes the problem in both RHEL-5.2 and RHEL-5.1.z. Reassigning this
bug to selinux-policy.
I don't believe most of this patch is necessary with selinux-policy-2.4.6-120.el5
All of the ramfs avc's seem to be handled. Now why does it need all of the
mounton privs. Does multipathd have a builtin mount command?
I developed the patch on selinux-policy-2.4.6-106.el5_1.3, and then simply
applied the patch and retested with selinux-policy-2.4.6-120.el5. It applied
cleanly, and worked correctly, but it's definitely possible that it does more
than necessary. I can check.
The mountons were necessary on selinux-policy-2.4.6-106.el5_1.3 because of what
multipathd does to avoid the no-paths failure. It creates its own namespace,
and mounts the ramfs over /bin, /sbin, etc. It does this so that users don't
need to modify their configurations to point to the ramfs version of the binaries.
Trying this without the patch on selinux-policy-2.4.6-121.el5, everything seems
to work fine.
Ok fixed in selinux-policy-2.4.6-121.el5
I've downgraded to device-mapper-multipath-0.4.7-12.el5 to make the multipathd
run again. But a "yum update" would still install
device-mapper-multipath i386 0.4.7-12.el5_1.2 rhel-i386-server-5 2.0 M
kpartx i386 0.4.7-12.el5_1.2 rhel-i386-server-5 403 k
As far as I know I run the current official supported production Version of RHEL 5.1
Did I make a mistake at setting up my systems or will the multipathd stop to
work on all RHEL5.1/selinux enabled systems in the world if the administrator
makes a "yum update"? If yes, when will this be changed? Maybe you could exclude
this version from the automatic upgrade list to save the systems that have not
selinux-policy-2.4.6-121.el5 will be released with RHEL 5.2 You need a RHEL 5.1
zstream update for selinux-policy.
I hit this today with a fresh install of 5.1 and all patches applied via yum.
Multipathd is broken out of the box. Why is there no fix for RHEL 5.1? The
only workaround I can see that doesn't involve writing policy is to disable SELinux.
Which policy package did you install?
Has a preview of the U2 selinux-policy
For me, the following has worked:
rpm --nodeps -e device-mapper-multipath
rpm --nodeps -e kpartx
mv /etc/multipath.conf.rpmsave /etc/multipath.conf
yum install device-mapper-multipath-0.4.7-12.el5
And I have to take care not to run a "yum update" because the broken version is
still distributed, more than 3 weeks after my initial report.
I've learned that "zstreams" are package releases between the main releases.
There is another bug #433289 related to this bug and zstream, but it has no
attachment yet... maybe the fix for RHEL5.1 will be released under that bugid?
A new version of device-mapper-multipath is distributed since yesterday:
0.4.7-12.el5_1.3. But no new selinux-policy. Is this the reason why the new
version fails on selinux enabled machines as the version before?
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.