Bug 447666 - multipathd child namespace fails to open/provide /dev/console (no such device) under Xen kernels
multipathd child namespace fails to open/provide /dev/console (no such device...
Status: CLOSED INSUFFICIENT_DATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: device-mapper-multipath (Show other bugs)
5.1
x86_64 Linux
low Severity urgent
: rc
: ---
Assigned To: Ben Marzinski
Corey Marthaler
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-05-20 22:19 EDT by Eli Stair
Modified: 2014-02-06 14:18 EST (History)
14 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-02-06 14:18:11 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Eli Stair 2008-05-20 22:19:00 EDT
centos 5.1 + x86_64,
xen-3.0.3-41.el5_1.5,
device-mapper-multipath-0.4.7-12.el5_1.4, device-mapper-multipath-0.4.7-12.el5_1.3,
kernels = 2.6.18-53.1.6xen 2.6.18-53.1.19xen, 2.6.18-53.1.6
selinux DISABLED

'multipathd' under Xen kernels is failing to start, apparently due to an issue
with the cloned namespace. Confirmed that on the same system booting the non-xen
kernel, multipathd starts fine and children have no problem accessing
/dev/console. Incidentally, I'm not clear why the newer multipathd is passing
newer flags (CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD) to clone() in the
second "intermediary" child process. It's immediately after this (in the third
PID) that child namespace does not appear to have a /dev/console (on the Xen
kernel):

  open("/dev/console", O_WRONLY) = -1 ENODEV (No such device)

Launching the process in a non-Xen kernel shows it functioning:

  open("/dev/console", O_WRONLY) = 4

In all cases /dev/console exists with the same perms:

  crw------- 1 root root 5, 1 May 20 18:25 /dev/console


This failure is occurring on the two most recent (el5_1.4, el5_1.3)
device-mapper-multipath 'multipathd' binaries at least. Comparing strace output
of runs of the two, it's notable that the second child process which clone()'s
the third PID is passing additional flags to clone() on the newer versions,
haven't gotten far enough to determine if this is related:


Here's the failing newer environment, on Xen kernel: {

    ######multipathd-0.4.7-12.el5_1.4-11420
< snipped for brevity
    getuid() = 0
    chdir("/") = 0
    umask(077) = 022
    umask(022) = 077
    clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x2aaaaaac51d0) = 11421
    exit_group(0) =

    ######multipathd-0.4.7-12.el5_1.4-11421
    setsid() = 11421
    clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x2aaaaaac51d0) = 11422
    exit_group(0) = ?

    ######multipathd-0.4.7-12.el5_1.4-11422
    unshare(CLONE_NEWNS) = 0
    open("/dev/null", O_RDONLY) = 3
    open("/dev/console", O_WRONLY) = -1 ENODEV (No such device)
    write(2, "cannot open /dev/console for out"..., 53) = 53
    exit_group(0) = ?
< snipped for brevity
} /NEW


VS. same environment but NON-XEN kernel that works {

  ######multipathd-0.4.7-12.el5_1.4-no-xen.4943
< snipped for brevity
    getuid() = 0
    chdir("/") = 0
    umask(077) = 022
    umask(022) = 077
    clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x2aaaaaac51d0) = 4494
    exit_group(0) = ?

  ######multipathd-0.4.7-12.el5_1.4-4494
    setsid() = 4494
    clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x2aaaaaac51d0) = 4495
    exit_group(0) = ?

  ######multipathd-0.4.7-12.el5_1.4-4495
    unshare(CLONE_NEWNS) = 0
    open("/dev/null", O_RDONLY) = 3
    open("/dev/console", O_WRONLY) = 4
    close(0) = 0
    dup(3) = 0
<snip... etc, this works>


VS. older (device-mapper-multipath-0.4.7-12.el5_1.2) non-Xen environment: {

   ######multipathd-.0.4.7-12.el5_1.2-2019
< snipped for brevity
    getuid() = 0
    chdir("/") = 0
    umask(077) = 022
    umask(022) = 077
    clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x2aaaaaae01b0) = 2020
    exit_group(0) = ?

  ######multipathd-.0.4.7-12.el5_1.2-2020
    setsid() = 2020
    clone(child_stack=0x1c04e670, flags=CLONE_NEWNS) = 2021
    exit_group(0) = ?

  ######multipathd-.0.4.7-12.el5_1.2-2021
    getpid() = 2021
    open("/dev/null", O_RDONLY) = 3
    open("/dev/console", O_WRONLY) = 4
    close(0) = 0
    dup(3) = 0
<snip... etc, this works>
Comment 1 Ben Marzinski 2008-05-21 14:55:38 EDT
The difference between device-mapper-multipath-0.4.7-12.el5_1.2 uses the clone()
library call, while device-mapper-multipath-0.4.7-12.el5_1.4 uses the fork()
library call (which calls the clone system call).  This was done because clone()
wasn't playing well with SELinux. However they all create their own namespace.
Did you test this with device-mapper-multipath-0.4.7-12.el5_1.2 in a Xen setup?

If you could test this with device-mapper-multipath-0.4.7-12.el5_1.1, which
doesn't create it's own namespace, it would be a big help to learn if that is
the cause.
Comment 2 Ben Marzinski 2008-05-22 16:22:54 EDT
I have been unable to recreate this on a DomU with
device-mapper-multipath-0.4.7-12.el5_1.4. I was using more recent Xen and kernel
packages. I will retry with the identical patches.
Comment 3 Eli Stair 2008-05-22 19:11:16 EDT
I will make an attempt (ASAP for me) building a system with the requested older
device-mapper-multipath-0.4.7-12.el5_1.1, on a couple of recent RHEL kernels.

Which kernel were you attempting this on, a disted RHEL kernel, or patched
kernel.org 2.6.18?

/eli
Comment 4 Ben Marzinski 2008-05-23 11:47:11 EDT
I ran the test with device-mapper-multipath-0.4.7-12.el5_1.4, xen-3.0.3-64.el5,
and kernel-xen-2.6.18-92.el5. The later two are the latest RHEL5.2 versions of
the packages. They should have been just released.  I'm not sure when centos
will pick up the new versions.
Comment 5 Ben Marzinski 2008-07-03 16:43:01 EDT
I have not been able to recreate this at all. Have you tried with the newer
versions?
Comment 6 Jakub Wojcik 2009-09-01 06:15:58 EDT
We're having the same problem with the most recent versions 5.3 x86_64:
- kernel-xen-2.6.18-128.7.1.el5
- xen-3.0.3-80.el5_3.3
- device-mapper-multipath-0.4.7-23.el5_3.4

----------------------
child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b2cf8aa61d0) = 8242
[pid  8241] exit_group(0)               = ?
Process 8241 detached
unshare(CLONE_NEWNS)                    = 0
open("/dev/null", O_RDONLY)             = 3
open("/dev/console", O_WRONLY)          = -1 ENODEV (No such device)
----------------------

As a workaround to get multipathd running we did:

rm /dev/console
ln -s /dev/tty6 /dev/console

(xencons=tty6 in xen module options) - but not sure if "xm console" will work properly
Comment 8 Ben Marzinski 2011-02-23 18:45:57 EST
Is this still a problem. I have not been able to recreate this issue.
Comment 9 RHEL Product and Program Management 2014-01-29 05:40:40 EST
This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux release.  Product Management has
requested further review of this request by Red Hat Engineering, for
potential inclusion in a Red Hat Enterprise Linux release for currently
deployed products.  This request is not yet committed for inclusion in
a release.

Note You need to log in before you can comment on or make changes to this bug.