Red Hat Bugzilla – Bug 431689
After update, multipathd does not start any more
Last modified: 2008-05-21 12:06:59 EDT
Description of problem: After doing a "yum update", the multipathd does not run any more. Version-Release number of selected component (if applicable): device-mapper-multipath-0.4.7-12.el5_1.2 How reproducible: always Steps to Reproduce: # service multipathd start Starting multipathd daemon: [ OK ] # service multipathd status multipathd dead but pid file exists Actual results: Expected results: Additional info: The machine is multipath-booted from a Netapp Filer: # mount /dev/mapper/smtp3p5 on / type ext3 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/mapper/smtp3p1 on /boot type ext3 (rw) tmpfs on /dev/shm type tmpfs (rw) /dev/mapper/smtp3p2 on /var type ext3 (rw) /dev/mapper/smtp3p3 on /var/spool type ext3 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) I think the problem is related to selinux because after doing a "setenforce 0", "ps -ef" shows at least a running daemon. But /var/cache/multipathd is empty, mount does not show a ramfs and there are strange error messages in the syslog. I already unsuccessfully did a "touch /.autorelabel" and a reboot. "strace -F -f /sbin/multipathd" sometimes shows this: ... [pid 6398] open("/sbin/dasd_id", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 6398] open("/sbin/gnbd_import", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 6398] mount(NULL, "/var/cache/multipathd", "ramfs"..., MS_SYNCHRONOUS, "maxsize=530258"...) = -1 EACCES (Permission denied) [pid 6398] exit_group(1) = ? Process 6398 detached ... /var/log/messages: (only after doing "setenforce 0") Feb 6 15:19:15 smtp3 multipathd: cannot open mpath_prio_hds_modular : No such file or directory Feb 6 15:19:15 smtp3 multipathd: cannot open /sbin/dasd_id : No such file or directory Feb 6 15:19:15 smtp3 multipathd: cannot open /sbin/gnbd_import : No such file or directory Feb 6 15:19:15 smtp3 multipathd: [copy.c] cannot open mpath_prio_hds_modular Feb 6 15:19:15 smtp3 multipathd: cannot copy mpath_prio_hds_modular in ramfs : No such file or directory Feb 6 15:19:15 smtp3 multipathd: [copy.c] cannot open /sbin/dasd_id Feb 6 15:19:15 smtp3 multipathd: cannot copy /sbin/dasd_id in ramfs : No such file or directory Feb 6 15:19:15 smtp3 multipathd: [copy.c] cannot open /sbin/gnbd_import Feb 6 15:19:15 smtp3 multipathd: cannot copy /sbin/gnbd_import in ramfs : No such file or directory Feb 6 15:19:15 smtp3 multipathd: smtp3: event checker started
Created attachment 294108 [details] My multipath.conf
Created attachment 294109 [details] grep multipath /var/log/audit/audit.log
Created attachment 294240 [details] Selinux policy fix. This patch applies cleanly to selinux-policy-2.4.6-106.el5_1.3 and selinux-policy-2.4.6-120.el5.
This is dumb oversight on my part. I accidentally tested the fix for bz #428338 on a machine with selinux disabled. The fix for this bug requires a selinux policy change. Without the policy change, multipathd won't work at all. Unfortunately, bz #428338 has a RHEL 5.1.z zstream fix associated with it, bz #355961, so the selinux policy needs to be fixed there as well. The patch in Comment #3 fixes the problem in both RHEL-5.2 and RHEL-5.1.z. Reassigning this bug to selinux-policy.
I don't believe most of this patch is necessary with selinux-policy-2.4.6-120.el5 All of the ramfs avc's seem to be handled. Now why does it need all of the mounton privs. Does multipathd have a builtin mount command?
I developed the patch on selinux-policy-2.4.6-106.el5_1.3, and then simply applied the patch and retested with selinux-policy-2.4.6-120.el5. It applied cleanly, and worked correctly, but it's definitely possible that it does more than necessary. I can check. The mountons were necessary on selinux-policy-2.4.6-106.el5_1.3 because of what multipathd does to avoid the no-paths failure. It creates its own namespace, and mounts the ramfs over /bin, /sbin, etc. It does this so that users don't need to modify their configurations to point to the ramfs version of the binaries.
Trying this without the patch on selinux-policy-2.4.6-121.el5, everything seems to work fine.
Ok fixed in selinux-policy-2.4.6-121.el5
I've downgraded to device-mapper-multipath-0.4.7-12.el5 to make the multipathd run again. But a "yum update" would still install Updating: device-mapper-multipath i386 0.4.7-12.el5_1.2 rhel-i386-server-5 2.0 M kpartx i386 0.4.7-12.el5_1.2 rhel-i386-server-5 403 k without selinux-policy-2.4.6-121.el5. As far as I know I run the current official supported production Version of RHEL 5.1 Did I make a mistake at setting up my systems or will the multipathd stop to work on all RHEL5.1/selinux enabled systems in the world if the administrator makes a "yum update"? If yes, when will this be changed? Maybe you could exclude this version from the automatic upgrade list to save the systems that have not yet upgraded?
selinux-policy-2.4.6-121.el5 will be released with RHEL 5.2 You need a RHEL 5.1 zstream update for selinux-policy.
I hit this today with a fresh install of 5.1 and all patches applied via yum. Multipathd is broken out of the box. Why is there no fix for RHEL 5.1? The only workaround I can see that doesn't involve writing policy is to disable SELinux.
Which policy package did you install? http://people.redhat.com/dwalsh/SELinux/RHEL5 Has a preview of the U2 selinux-policy
For me, the following has worked: rpm --nodeps -e device-mapper-multipath rpm --nodeps -e kpartx mv /etc/multipath.conf.rpmsave /etc/multipath.conf yum install device-mapper-multipath-0.4.7-12.el5 And I have to take care not to run a "yum update" because the broken version is still distributed, more than 3 weeks after my initial report. I've learned that "zstreams" are package releases between the main releases. There is another bug #433289 related to this bug and zstream, but it has no attachment yet... maybe the fix for RHEL5.1 will be released under that bugid?
A new version of device-mapper-multipath is distributed since yesterday: 0.4.7-12.el5_1.3. But no new selinux-policy. Is this the reason why the new version fails on selinux enabled machines as the version before?
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2008-0465.html