Bug 873898 - systemd crashes with "Caught <BUS>" leaving the whole machine crippled
Summary: systemd crashes with "Caught <BUS>" leaving the whole machine crippled
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 16
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-11-06 22:45 UTC by Michal Jaegermann
Modified: 2013-02-13 22:33 UTC (History)
10 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2013-02-13 22:33:09 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
core file dropped by systemd (compressed with xz) (195.27 KB, application/octet-stream)
2012-11-07 20:54 UTC, Michal Jaegermann
no flags Details

Description Michal Jaegermann 2012-11-06 22:45:52 UTC
Description of problem:

After recovery I found the following in logs:

systemd[1]: Caught <BUS>, dumped core as pid 16169.
systemd[1]: Freezing execution.

Soon after this entries like that started to show up in logs:

systemd-cgroups-agent[16355]: Failed to get D-Bus connection: Failed to connect to socket /org/freedesktop/systemd1/private: Connection refused

and a machine was effectively crippled.  As this was a remote machine then for all practical purposes it was gone, as sshd was rejecting attempts to connect, and with no path to a recovery.  Even a local reboot was difficult and required few retries before this was acomplished.

abrtd did not catch anything bug systemd left a core as /core.16169.  With
systemd-debuginfo loaded gdb reports the following:


Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/bin/systemd --log-level info --log-target syslog-or-kmsg --system --dump-core'.
Program terminated with signal 7, Bus error.
#0  0x00000033b1c0f3cb in raise () from /lib64/libpthread.so.0

and a backtrace is:

#0  0x00000033b1c0f3cb in raise () from /lib64/libpthread.so.0
#1  0x000000000040b42c in crash (sig=7) at src/main.c:123
#2  <signal handler called>
#3  hashmap_iterate (h=0x7f495a4510c0, i=0x7fff5b69cc20, key=0x0)
    at src/hashmap.c:474
#4  0x0000000000412eb8 in transaction_add_job_and_dependencies (tr=0x15ac8f0, 
    type=<optimized out>, unit=<optimized out>, by=<optimized out>, 
    matters=false, override=false, conflicts=false, ignore_requirements=false, 
    ignore_order=false, e=0x7fff5b69ce30) at src/transaction.c:904
#5  0x0000000000412a2a in transaction_add_job_and_dependencies (tr=0x15ac8f0, 
    type=JOB_START, unit=<optimized out>, by=<optimized out>, matters=false, 
    override=false, conflicts=false, ignore_requirements=false, 
    ignore_order=false, e=0x7fff5b69ce30) at src/transaction.c:872
#6  0x0000000000412a2a in transaction_add_job_and_dependencies (tr=0x15ac8f0, 
    type=JOB_RESTART, unit=<optimized out>, by=<optimized out>, matters=true, 
    override=false, conflicts=false, ignore_requirements=false, 
    ignore_order=false, e=0x7fff5b69ce30) at src/transaction.c:872
#7  0x000000000040cb26 in manager_add_job (m=0x159d5b0, type=JOB_RESTART, 
    unit=0x160bca0, mode=JOB_FAIL, override=false, e=0x7fff5b69ce30, _ret=0x0)
    at src/manager.c:654
#8  0x0000000000420dd2 in service_enter_restart (s=0x160bca0)
    at src/service.c:2134
#9  service_timer_event (u=0x160bca0, elapsed=<optimized out>, 
    w=<optimized out>) at src/service.c:2947
#10 0x0000000000410d72 in process_event (ev=0x7fff5b69cea0, m=0x159d5b0)
    at src/manager.c:1330
#11 manager_loop (m=0x159d5b0) at src/manager.c:1427
#12 0x0000000000409207 in main (argc=<optimized out>, argv=<optimized out>)
    at src/main.c:1466

This is a loop which bombed out:

1465	        for (;;) {
1466	                if ((r = manager_loop(m)) < 0) {
1467	                        log_error("Failed to run mainloop: %s", strerror(-r));
1468	                        goto finish;
1469	                }

A value for 'm' is '(Manager *) 0x159d5b0' and this is a rather big structure.

In 'manager_loop' (frame #11) this is:

1427	                if ((r = process_event(m, &event)) < 0)
1428	                        return r;

and 'event' has a value {events = 1, data = {ptr = 0x160c110, fd = 23118096, u32 = 23118096, u64 = 23118096}}

Version-Release number of selected component (if applicable):
systemd-37-25.fc16.x86_64

How reproducible:
No idea.  I did not see that before.
  
Actual results:
A mess.

Expected results:
A machine does not fall apart because systemd has hiccups.

Comment 1 Michal Schmidt 2012-11-07 08:25:25 UTC
Could you attach the core file?

Comment 2 Michal Jaegermann 2012-11-07 20:54:12 UTC
Created attachment 640363 [details]
core file dropped by systemd (compressed with xz)

> Could you attach the core file?

Sure.  Here it is; compressed with xz as this significantly reduces size of that file

Comment 3 Fedora End Of Life 2013-01-16 17:43:37 UTC
This message is a reminder that Fedora 16 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 16. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '16'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 16's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 16 is end of life. If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora, you are encouraged to click on 
"Clone This Bug" and open it against that version of Fedora.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 4 Jóhann B. Guðmundsson 2013-02-12 02:03:04 UTC
Michal should we move this one onwards to rawhide?

Comment 5 Michal Jaegermann 2013-02-12 14:32:15 UTC
(In reply to comment #4)
> Michal should we move this one onwards to rawhide?

It is really up to you.  OTOH I do not recall seeing anything such dramatic ever later.  If you think that this was a demonstration of some subtle race then this code path does not seem to be exercised very often so it may be hard to repeat.  Or maybe indeed these were cosmic rays?

Comment 6 Fedora End Of Life 2013-02-13 22:33:13 UTC
Fedora 16 changed to end-of-life (EOL) status on 2013-02-12. Fedora 16 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.