Bug 431689 - After update, multipathd does not start any more
After update, multipathd does not start any more
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: selinux-policy (Show other bugs)
5.1
i386 Linux
urgent Severity high
: rc
: ---
Assigned To: Daniel Walsh
GSSApproved
: ZStream
Depends On:
Blocks: 432541 433289
  Show dependency treegraph
 
Reported: 2008-02-06 09:55 EST by Thorsten Schlichting
Modified: 2008-05-21 12:06 EDT (History)
16 users (show)

See Also:
Fixed In Version: RHBA-2008-0465
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-05-21 12:06:59 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
My multipath.conf (3.37 KB, text/plain)
2008-02-06 09:55 EST, Thorsten Schlichting
no flags Details
grep multipath /var/log/audit/audit.log (14.75 KB, text/plain)
2008-02-06 10:01 EST, Thorsten Schlichting
no flags Details
Selinux policy fix. (988 bytes, patch)
2008-02-07 13:05 EST, Ben Marzinski
no flags Details | Diff

  None (edit)
Description Thorsten Schlichting 2008-02-06 09:55:12 EST
Description of problem:
After doing a "yum update", the multipathd does not run any more.

Version-Release number of selected component (if applicable):
device-mapper-multipath-0.4.7-12.el5_1.2

How reproducible:
always

Steps to Reproduce:
# service multipathd start
Starting multipathd daemon:                                [  OK  ]
# service multipathd status
multipathd dead but pid file exists

  
Actual results:


Expected results:


Additional info:

The machine is multipath-booted from a Netapp Filer:
# mount
/dev/mapper/smtp3p5 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/mapper/smtp3p1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
/dev/mapper/smtp3p2 on /var type ext3 (rw)
/dev/mapper/smtp3p3 on /var/spool type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

I think the problem is related to selinux because after doing a "setenforce 0",
"ps -ef" shows at least a running daemon. But /var/cache/multipathd is empty,
mount does not show a ramfs and there are strange error messages in the syslog.

I already unsuccessfully did a "touch /.autorelabel" and a reboot.

"strace -F -f /sbin/multipathd" sometimes shows this:

...
[pid  6398] open("/sbin/dasd_id", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid  6398] open("/sbin/gnbd_import", O_RDONLY) = -1 ENOENT (No such file or
directory)
[pid  6398] mount(NULL, "/var/cache/multipathd", "ramfs"..., MS_SYNCHRONOUS,
"maxsize=530258"...) = -1 EACCES (Permission denied)
[pid  6398] exit_group(1)               = ?
Process 6398 detached
...

/var/log/messages: (only after doing "setenforce 0")
Feb  6 15:19:15 smtp3 multipathd: cannot open mpath_prio_hds_modular : No such
file or directory
Feb  6 15:19:15 smtp3 multipathd: cannot open /sbin/dasd_id : No such file or
directory
Feb  6 15:19:15 smtp3 multipathd: cannot open /sbin/gnbd_import : No such file
or directory
Feb  6 15:19:15 smtp3 multipathd: [copy.c] cannot open mpath_prio_hds_modular
Feb  6 15:19:15 smtp3 multipathd: cannot copy mpath_prio_hds_modular in ramfs :
No such file or directory
Feb  6 15:19:15 smtp3 multipathd: [copy.c] cannot open /sbin/dasd_id
Feb  6 15:19:15 smtp3 multipathd: cannot copy /sbin/dasd_id in ramfs : No such
file or directory
Feb  6 15:19:15 smtp3 multipathd: [copy.c] cannot open /sbin/gnbd_import
Feb  6 15:19:15 smtp3 multipathd: cannot copy /sbin/gnbd_import in ramfs : No
such file or directory
Feb  6 15:19:15 smtp3 multipathd: smtp3: event checker started
Comment 1 Thorsten Schlichting 2008-02-06 09:55:12 EST
Created attachment 294108 [details]
My multipath.conf
Comment 2 Thorsten Schlichting 2008-02-06 10:01:10 EST
Created attachment 294109 [details]
grep multipath /var/log/audit/audit.log
Comment 3 Ben Marzinski 2008-02-07 13:05:41 EST
Created attachment 294240 [details]
Selinux policy fix.

This patch applies cleanly to selinux-policy-2.4.6-106.el5_1.3 and
selinux-policy-2.4.6-120.el5.
Comment 4 Ben Marzinski 2008-02-07 13:15:17 EST
This is dumb oversight on my part. I accidentally tested the fix for bz #428338
on a machine with selinux disabled. The fix for this bug requires a selinux
policy change. Without the policy change, multipathd won't work at all.
Unfortunately, bz #428338 has a RHEL 5.1.z zstream fix associated with it, bz
#355961, so the selinux policy needs to be fixed there as well. The patch in
Comment #3 fixes the problem in both RHEL-5.2 and RHEL-5.1.z. Reassigning this
bug to selinux-policy.
Comment 6 Daniel Walsh 2008-02-07 15:07:24 EST
I don't believe most of this patch is necessary with selinux-policy-2.4.6-120.el5

All of the ramfs avc's seem to be handled.  Now why does it need all of the
mounton privs.  Does multipathd have a builtin mount command?
Comment 7 Ben Marzinski 2008-02-08 15:38:25 EST
I developed the patch on selinux-policy-2.4.6-106.el5_1.3, and then simply
applied the patch and retested with selinux-policy-2.4.6-120.el5. It applied
cleanly, and worked correctly, but it's definitely possible that it does more
than necessary. I can check.

The mountons were necessary on selinux-policy-2.4.6-106.el5_1.3 because of what
multipathd does to avoid the no-paths failure.  It creates its own namespace,
and mounts the ramfs over /bin, /sbin, etc.  It does this so that users don't
need to modify their configurations to point to the ramfs version of the binaries.
Comment 8 Ben Marzinski 2008-02-08 16:15:29 EST
Trying this without the patch on selinux-policy-2.4.6-121.el5, everything seems
to work fine.
Comment 9 Daniel Walsh 2008-02-08 16:43:28 EST
Ok fixed in selinux-policy-2.4.6-121.el5
Comment 11 Thorsten Schlichting 2008-02-12 08:24:11 EST
I've downgraded to device-mapper-multipath-0.4.7-12.el5 to make the multipathd
run again. But a "yum update" would still install 

Updating:
 device-mapper-multipath  i386       0.4.7-12.el5_1.2  rhel-i386-server-5  2.0 M
 kpartx                  i386       0.4.7-12.el5_1.2  rhel-i386-server-5  403 k

without selinux-policy-2.4.6-121.el5.

As far as I know I run the current official supported production Version of RHEL 5.1

Did I make a mistake at setting up my systems or will the multipathd stop to
work on all RHEL5.1/selinux enabled systems in the world if the administrator
makes a "yum update"? If yes, when will this be changed? Maybe you could exclude
this version from the automatic upgrade list to save the systems that have not
yet upgraded?
Comment 13 Ben Marzinski 2008-02-12 12:47:42 EST
selinux-policy-2.4.6-121.el5 will be released with RHEL 5.2  You need a RHEL 5.1
zstream update for selinux-policy.
Comment 17 Bruce A. Locke 2008-02-28 17:28:36 EST
I hit this today with a fresh install of 5.1 and all patches applied via yum.

Multipathd is broken out of the box.  Why is there no fix for RHEL 5.1?  The
only workaround I can see that doesn't involve writing policy is to disable SELinux.

Comment 18 Daniel Walsh 2008-02-29 11:24:26 EST
Which policy package did you install?

http://people.redhat.com/dwalsh/SELinux/RHEL5

Has a preview of the U2 selinux-policy 
Comment 19 Thorsten Schlichting 2008-02-29 11:44:02 EST
For me, the following has worked:

rpm --nodeps -e device-mapper-multipath
rpm --nodeps -e kpartx
mv /etc/multipath.conf.rpmsave /etc/multipath.conf
yum install device-mapper-multipath-0.4.7-12.el5

And I have to take care not to run a "yum update" because the broken version is
still distributed, more than 3 weeks after my initial report.

I've learned that "zstreams" are package releases between the main releases.
There is another bug #433289 related to this bug and zstream, but it has no
attachment yet... maybe the fix for RHEL5.1 will be released under that bugid?
Comment 20 Thorsten Schlichting 2008-03-12 05:32:13 EDT
A new version of device-mapper-multipath is distributed since yesterday:
0.4.7-12.el5_1.3. But no new selinux-policy. Is this the reason why the new
version fails on selinux enabled machines as the version before?
Comment 22 errata-xmlrpc 2008-05-21 12:06:59 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2008-0465.html

Note You need to log in before you can comment on or make changes to this bug.