Bug 1167044

Created attachment 960296 [details]
File: backtrace

Created attachment 960297 [details]
File: cgroup

Created attachment 960298 [details]
File: core_backtrace

Created attachment 960299 [details]
File: dso_list

Created attachment 960300 [details]
File: limits

Created attachment 960301 [details]
File: maps

Created attachment 960302 [details]
File: open_fds

Created attachment 960303 [details]
File: proc_pid_status

Created attachment 960304 [details]
File: var_log_messages

Also hit a crash on F20 -> F21 upgrade:


Nov 30 00:13:49 nostromo systemd: Unknown serialization item 'subscribed=:1.195'
Nov 30 00:13:49 nostromo systemd: Configuration file /usr/lib/systemd/system/wpa
_supplicant.service is marked executable. Please remove executable permission bi
ts. Proceeding anyway.
Nov 30 00:13:49 nostromo systemd: Unknown serialization item 'subscribed=:1.195'
Nov 30 00:13:49 nostromo systemd: Configuration file /usr/lib/systemd/system/wpa
_supplicant.service is marked executable. Please remove executable permission bi
ts. Proceeding anyway.
Nov 30 00:13:50 nostromo systemd: Failed to reset devices.list on /system.slice:
 Invalid argument
Nov 30 00:13:50 nostromo kernel: [28411.492780] systemd[1]: segfault at 7fb67178
8400 ip 00007fb60dd11d23 sp 00007fffda6c06b0 error 4 in systemd[7fb60dc34000+137
000]
Nov 30 00:13:50 nostromo kernel: systemd[1]: segfault at 7fb671788400 ip 00007fb
60dd11d23 sp 00007fffda6c06b0 error 4 in systemd[7fb60dc34000+137000]
Nov 30 00:13:50 nostromo systemd: Caught <SEGV>, dumped core as pid 9704.

#0  0x00007fb60c757f99 in raise () from /lib64/libpthread.so.0
#1  0x00007fb60dcd7ce8 in crash.lto_priv ()
#2  <signal handler called>
#3  0x00007fb60dd11d23 in signal_agent_released.lto_priv ()
#4  0x00007fb60dc851a5 in bus_match_run ()
#5  0x00007fb60dc84dfe in bus_match_run ()
#6  0x00007fb60dc84dd8 in bus_match_run ()
#7  0x00007fb60dc84dfe in bus_match_run ()
#8  0x00007fb60dc84dd8 in bus_match_run ()
#9  0x00007fb60dc84dfe in bus_match_run ()
#10 0x00007fb60dc84dd8 in bus_match_run ()
#11 0x00007fb60dc84dfe in bus_match_run ()
#12 0x00007fb60dc84dd8 in bus_match_run ()
#13 0x00007fb60dc84dfe in bus_match_run ()
#14 0x00007fb60dc84cee in bus_match_run ()
#15 0x00007fb60dce59d8 in process_match.lto_priv ()
#16 0x00007fb60dd03ae0 in bus_process_internal.constprop ()
#17 0x00007fb60dc66b71 in io_callback ()
#18 0x00007fb60dc6a700 in source_dispatch ()
#19 0x00007fb60dc6b4ae in sd_event_dispatch ()
#20 0x00007fb60dceee80 in manager_loop ()
#21 0x00007fb60dc57239 in main ()
(gdb)

(I was on 208-22.fc20)

I have a 64-bit Fedora 20 VM image where this happens completely
consistently and reproducibly during a yum upgrade from Fedora 20 to
Fedora 21. It is using the latest updates, so the starting Fedora 20
version is systemd-208-28.fc20.x86_64 (and I have an intact systemd
journal from the latest upgrade). The yum upgrade is done in the
standard way, with 'yum --releasever 21 distro-sync'.

With either patience or stubbing /usr/bin/systemctl out to /bin/true
so that all of the postinstall 'systemctl daemon-reload' and so on
commands finish any time soon, the upgrade will complete. However for
me the resulting Fedora 21 kernel initramfs stalls during boot; I have
to force a boot off the Fedora 20 kernel and rebuild the initramfs.
This may be unrelated.

I took a look at my systemd core dump. My stack backtrace is the same
as the original reporter, with what is clearly a bogus 'path' pointer.
This pointer comes from a structure element:

#9  0x00007f0523292d13 in manager_notify_cgroup_empty (cgroup=<optimized out>, 
    m=0x7f0524c54350) at ../src/core/cgroup.c:981
981                     r = cg_is_empty_recursive(SYSTEMD_CGROUP_CONTROLLER, u->cgroup_path, true);

I dumped the structure's nominal contents and the raw memory of it in
gdb, which I will put in an attachment because it's relatively large,
but to me it looks like the memory the structure is theoretically at
has been smashed and partially overwritten by, eg, string data.

I would be happy to make the core dump itself available to interested
parties who can determine more from it than I can.

Created attachment 967745 [details]
gdb dump of the bad 'Unit' structure contents

*** Bug 1130633 has been marked as a duplicate of this bug. ***

(In reply to Chris Siebenmann from comment #13)
> I would be happy to make the core dump itself available to interested
> parties who can determine more from it than I can.

if that can help, my /var/tmp/abrt still holds the data so devs can ping me on irc/in person for a copy/access too

For the record: since Fedora 21 just released an updated systemd, I
just retried a virtual machine upgrade with an up to date Fedora 20
system image and this repeatable systemd crash still happens with
systemd-216-14.fc21.x86_64.

(In reply to Karel Volný from comment #0)
> and I was unable to even reboot
'systemctl reboot -f' and 'reboot -f' should work even when systemd has segfaulted.
The advantage is that disks are synced.

(In reply to Chris Siebenmann from comment #13)
> I took a look at my systemd core dump. My stack backtrace is the same
> as the original reporter, with what is clearly a bogus 'path' pointer.
> This pointer comes from a structure element:
> 
> #9  0x00007f0523292d13 in manager_notify_cgroup_empty (cgroup=<optimized
> out>, 
>     m=0x7f0524c54350) at ../src/core/cgroup.c:981
> 981                     r = cg_is_empty_recursive(SYSTEMD_CGROUP_CONTROLLER,
> u->cgroup_path, true);
> 
> I dumped the structure's nominal contents and the raw memory of it in
> gdb, which I will put in an attachment because it's relatively large,
> but to me it looks like the memory the structure is theoretically at
> has been smashed and partially overwritten by, eg, string data.
Most likely that unit has been freed, but the pointer to it was retained in
another structure (most probably one of the hashmaps with units). This is
more likely than the memory being overwritten by overflow of another data
structure.

(In reply to Chris Siebenmann from comment #12)
> I have a 64-bit Fedora 20 VM image where this happens completely
> consistently and reproducibly during a yum upgrade from Fedora 20 to
> Fedora 21. It is using the latest updates, so the starting Fedora 20
> version is systemd-208-28.fc20.x86_64 (and I have an intact systemd
> journal from the latest upgrade). The yum upgrade is done in the
> standard way, with 'yum --releasever 21 distro-sync'.
OK, I'll try to reproduce this that way.

(In reply to Zbigniew Jędrzejewski-Szmek from comment #18)
> (In reply to Karel Volný from comment #0)
> > and I was unable to even reboot
> 'systemctl reboot -f' and 'reboot -f' should work even when systemd has
> segfaulted.
> The advantage is that disks are synced.

nice, I haven't noticed it now accepts this option ... and does it have any advantage over the SysRq method?

(In reply to Karel Volný from comment #19)
> nice, I haven't noticed it now accepts this option ... and does it have any
> advantage over the SysRq method?
It might be easier to use, since with sysrq you have to first sync and remount ro, wait for the kernel to print the message that this is done, and then reboot. It also might be easier over ssh and similar. Finally, sysrq reboot is disabled by default.

This message is a reminder that Fedora 21 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 21. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '21'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 21 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 21 changed to end-of-life (EOL) status on 2015-12-01. Fedora 21 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.