Bug 1241938

systemd by default saves core dump to / directory. Please have a look and upload the core file if you find any?

Created attachment 1050704 [details]
Core dump

The stack trace is:

(gdb) t a a bt

Thread 1 (LWP 15221):
#0  0x00007fcc893f58a7 in kill () from /lib64/libc.so.6
#1  0x00007fcc8ad0e6a3 in crash.2985 (sig=11) at src/core/main.c:168
#2  <signal handler called>
#3  0x00007fcc8ad49260 in bus_socket_make_message (
    bus=bus@entry=0x7fcc8c37c780, size=176)
    at src/libsystemd/sd-bus/bus-socket.c:900
#4  0x00007fcc8ad494d8 in bus_socket_read_message (
    bus=bus@entry=0x7fcc8c37c780) at src/libsystemd/sd-bus/bus-socket.c:1014
#5  0x00007fcc8ad4e423 in bus_read_message (bus=bus@entry=0x7fcc8c37c780, 
    hint_priority=hint_priority@entry=false, priority=0)
    at src/libsystemd/sd-bus/sd-bus.c:1624
#6  0x00007fcc8acf0bc2 in dispatch_rqueue (priority=0, m=<synthetic pointer>, 
    hint_priority=false, bus=0x7fcc8c37c780)
    at src/libsystemd/sd-bus/sd-bus.c:1661
#7  process_running (priority=0, ret=0x0, hint_priority=false, 
    bus=0x7fcc8c37c780) at src/libsystemd/sd-bus/sd-bus.c:2519
#8  bus_process_internal (bus=0x7fcc8c37c780, ret=0x0, priority=0, 
    hint_priority=false) at src/libsystemd/sd-bus/sd-bus.c:2714
#9  0x00007fcc8ad36e61 in sd_bus_process (ret=0x0, bus=<optimized out>)
    at src/libsystemd/sd-bus/sd-bus.c:2733
#10 io_callback.50558 (s=<optimized out>, fd=<optimized out>, 
    revents=<optimized out>, userdata=<optimized out>)
    at src/libsystemd/sd-bus/sd-bus.c:2992
#11 0x00007fcc8ad3da40 in source_dispatch (s=s@entry=0x7fcc8c2ffd30)
    at src/libsystemd/sd-event/sd-event.c:2115
#12 0x00007fcc8ad4099a in sd_event_dispatch (e=e@entry=0x7fcc8c2d5ad0)
    at src/libsystemd/sd-event/sd-event.c:2472
#13 0x00007fcc8ad56e1f in sd_event_run (timeout=18446744073709551615, 
    e=0x7fcc8c2d5ad0) at src/libsystemd/sd-event/sd-event.c:2501
#14 manager_loop (m=m@entry=0x7fcc8c2d54c0) at src/core/manager.c:2056
#15 0x00007fcc8acb8499 in main (argc=<optimized out>, argv=<optimized out>)
    at src/core/main.c:1756

There is no crashing dereference at bus-socket.c:900 (frame 3) that I
can see so far ...

Darius, if there are newer core dumps in / could you upload the newest
of those.

I think we already pin-pointed the problem. We didn't see it in test runs because we test on machines where SELinux is enabled. AFAICT, this bug occurs only when SELinux is disabled. Darius can you verify that on the machine where you see bug there is SELinux disabled?

OK I see it.  The crash happens in this function call:

900	        r = bus_message_from_malloc(bus,
901	                                    bus->rbuffer, size,
902	                                    bus->fds, bus->n_fds,
903	                                    !bus->bus_client && bus->ucred_valid ? &bus->ucred : NULL,
904	                                    !bus->bus_client && bus->label[0] ? bus->label : NULL,
905	                                    &t);

on line 904 when bus->label[0] is evaluated.  In gdb:

(gdb) print bus->label
$34 = 0x0
(gdb) print bus->label[0]
Cannot access memory at address 0x0

In upstream systemd, the bus->label field was changed from a
char[NAME_MAX] to a char * in this commit:

commit c4e6556c46cea1b7195cfb81c8cfab8342ebd852
Author: Zbigniew Jędrzejewski-Szmek <zbyszek.pl>
Date:   Sat Jun 6 21:24:45 2015 -0400

    sd-bus: store selinux context at connection time
    
    This appears to be the right time to do it for SOCK_STREAM
    unix sockets.
    
    Also: condition bus_get_owner_creds_dbus1 was reversed. Split
    it out to a separate variable for clarity and fix.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1224211

I was chatting about this bug to Darius in IRC, and we believe that
SELinux was disabled when the crash happened.

We backported following bugfix,

https://github.com/lnykryn/systemd-rhel/commit/61a6ce79defd59fee00cd2bc28d58f7c3e637ae2

*** Bug 1242053 has been marked as a duplicate of this bug. ***

This was fixed with BZ#1230190, so qa_acking..

Verified on RHEL-7.2:

# grep SELINUX /etc/selinux/config 
# SELINUX= can take one of these three values:
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
SELINUXTYPE=targeted

# Upgrade to systemd-219-18.el7
# yum upgrade ...
<Completed without errors/segfaults>


Old version for comparison:

# grep SELINUX /etc/selinux/config 
# SELINUX= can take one of these three values:
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
SELINUXTYPE=targeted

# Upgrade to systemd-219-5.el7
# yum upgrade ...
<truncated>
Message from syslogd@xxx at Oct  8 08:24:21 ...
 kernel:systemd[1]: segfault at 0 ip 00007f4477fec260 sp 00007fff8c98c6d0 error 4 in systemd[7f4477f37000
+146000]

Broadcast message from systemd-journald.com (Thu 2015-10-08 08:24:21 EDT):

systemd[1]: Caught <SEGV>, dumped core as pid 1173.


Broadcast message from systemd-journald.com (Thu 2015-10-08 08:24:21 EDT):

systemd[1]: Freezing execution.


Message from syslogd@xxx at Oct  8 08:24:21 ...
 systemd:Caught <SEGV>, dumped core as pid 1173.

Message from syslogd@xxx at Oct  8 08:24:21 ...
 systemd:Freezing execution.
/var/tmp/rpm-tmp.kdW7pk: line 3:  1152 Segmentation fault      systemctl daemon-reexec > /dev/null 2>&1
<truncated>

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2092.html