Bug 996547 - Systemd assertion failure in cg_is_empty_recursive
Systemd assertion failure in cg_is_empty_recursive
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: systemd (Show other bugs)
7.0
x86_64 Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: systemd-maint
Jan Ščotka
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-13 08:12 EDT by Nikolai Kondrashov
Modified: 2014-01-31 08:48 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-31 08:48:52 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
core.28556.gz (356.14 KB, application/x-gzip)
2013-08-13 08:12 EDT, Nikolai Kondrashov
no flags Details
core.15923.gz (353.42 KB, application/x-gzip)
2013-08-13 14:16 EDT, Nikolai Kondrashov
no flags Details
core.3272.gz (341.11 KB, application/x-gzip)
2013-08-28 11:45 EDT, Nikolai Kondrashov
no flags Details

  None (edit)
Description Nikolai Kondrashov 2013-08-13 08:12:45 EDT
Created attachment 786130 [details]
core.28556.gz

Description of problem:
Systemd hits assertion failure (compressed coredump attached) after repeatedly restarting sssd, with the following in /var/log/messages:

Aug 13 14:52:38 client-rhel7 systemd: Stopping System Security Services Daemon...
Aug 13 14:52:38 client-rhel7 sssd[pam]: Shutting down
Aug 13 14:52:38 client-rhel7 sssd[be[LDAP]]: Shutting down
Aug 13 14:52:38 client-rhel7 sssd[nss]: Shutting down
Aug 13 14:52:38 client-rhel7 systemd: Stopped System Security Services Daemon.
Aug 13 14:52:38 client-rhel7 systemd: Assertion 'path' failed at src/shared/cgroup-util.c:866, function cg_is_empty_recursive(). Aborting.
Aug 13 14:52:38 client-rhel7 systemd: Caught <ABRT>, dumped core as pid 28556.
Aug 13 14:52:38 client-rhel7 systemd: Freezing execution.

Shortly before that systemd refused to restart sssd with the following in
"journalctl -xn" output:

-- Subject: Unit sssd.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit sssd.service has begun starting up.
Aug 13 14:44:55 client-rhel7.sss-test.test systemd[1]: sssd.service start request repeated too quickly, refusing to start.
Aug 13 14:44:55 client-rhel7.sss-test.test systemd[1]: Failed to start System Security Services Daemon.
-- Subject: Unit sssd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/catalog/be02cf6855d2428ba40df7e9d022f03d
--
-- Unit sssd.service has failed.
--
-- The result is failed.

Version-Release number of selected component (if applicable):
systemd-libs-206-3.el7.x86_64
systemd-206-3.el7.x86_64
systemd-journal-gateway-206-3.el7.x86_64
systemd-sysv-206-3.el7.x86_64
systemd-debuginfo-206-3.el7.x86_64

How reproducible:
Encountered once so far

Steps to Reproduce:
Restart sssd repeatedly with about one second between restarts.

Backtrace:
#0  0x00007fb093d59ffb in raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/pt-raise.c:37
#1  0x00007fb0958a195e in crash (sig=6) at src/core/main.c:144
#2  <signal handler called>
#3  0x00007fb0939bf999 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#4  0x00007fb0939c10a8 in __GI_abort () at abort.c:90
#5  0x00007fb0958fa533 in log_assert (text=<optimized out>, file=0x7fb095953463 "src/shared/cgroup-util.c", line=866, func=0x7fb095953620 <__PRETTY_FUNCTION__.7898> "cg_is_empty_recursive", format=format@entry=0x7fb095954a10 "Assertion '%s' failed at %s:%u, function %s(). Aborting.") at src/shared/log.c:695
#6  0x00007fb0958fade0 in log_assert_failed (text=<optimized out>, file=<optimized out>, line=<optimized out>, func=<optimized out>) at src/shared/log.c:700
#7  0x00007fb0958f13f3 in cg_is_empty_recursive (controller=controller@entry=0x7fb09594bd07 "name=systemd", path=0x0, ignore_self=ignore_self@entry=true) at src/shared/cgroup-util.c:866
#8  0x00007fb0958e2890 in manager_notify_cgroup_empty (m=m@entry=0x7fb09766a360, cgroup=<optimized out>) at src/core/cgroup.c:736
#9  0x00007fb0958d338d in private_bus_message_filter (connection=0x7fb097770e40, message=0x7fb097978de0, data=0x7fb09766a360) at src/core/dbus.c:491
#10 0x00007fb0943949e6 in dbus_connection_dispatch (connection=connection@entry=0x7fb097770e40) at dbus-connection.c:4631
#11 0x00007fb0958d3dda in bus_dispatch (m=m@entry=0x7fb09766a360) at src/core/dbus.c:525
#12 0x00007fb0958a769f in manager_loop (m=0x7fb09766a360) at src/core/manager.c:1816
#13 0x00007fb09589efb6 in main (argc=5, argv=0x7fff6416bc88) at src/core/main.c:1705
Comment 2 Nikolai Kondrashov 2013-08-13 14:16:56 EDT
Created attachment 786233 [details]
core.15923.gz

Another crash, this time SIGSEGV, but, it seems, during the same assertion failure.

Backtrace:
#0  0x00007f4c3d4deffb in raise (sig=sig@entry=11) at ../nptl/sysdeps/unix/sysv/linux/pt-raise.c:37
#1  0x00007f4c3f02695e in crash (sig=11) at src/core/main.c:144
#2  <signal handler called>
#3  0x00007f4c3f07572b in isempty (p=0x3a30303a30303030 <Address 0x3a30303a30303030 out of bounds>) at src/shared/util.h:106
#4  join_path (controller=0x7f4c3f0a2e22 "systemd", path=0x3a30303a30303030 <Address 0x3a30303a30303030 out of bounds>, suffix=0x7f4c3f0d84ef "cgroup.procs", fs=0x7fffecb49eb8) at src/shared/cgroup-util.c:454
#5  0x00007f4c3f075ea8 in cg_enumerate_processes (controller=controller@entry=0x7f4c3f0d0d07 "name=systemd", path=path@entry=0x3a30303a30303030 <Address 0x3a30303a30303030 out of bounds>, _f=_f@entry=0x7fffecb49ee8) at src/shared/cgroup-util.c:50
#6  0x00007f4c3f07619a in cg_is_empty (controller=controller@entry=0x7f4c3f0d0d07 "name=systemd", path=path@entry=0x3a30303a30303030 <Address 0x3a30303a30303030 out of bounds>, ignore_self=ignore_self@entry=true) at src/shared/cgroup-util.c:827
#7  0x00007f4c3f07630f in cg_is_empty_recursive (controller=controller@entry=0x7f4c3f0d0d07 "name=systemd", path=0x3a30303a30303030 <Address 0x3a30303a30303030 out of bounds>, ignore_self=ignore_self@entry=true) at src/shared/cgroup-util.c:868
#8  0x00007f4c3f067890 in manager_notify_cgroup_empty (m=m@entry=0x7f4c3f4f7360, cgroup=<optimized out>) at src/core/cgroup.c:736
#9  0x00007f4c3f05838d in private_bus_message_filter (connection=0x7f4c3f5ba450, message=0x7f4c3f5ba560, data=0x7f4c3f4f7360) at src/core/dbus.c:491
#10 0x00007f4c3db199e6 in dbus_connection_dispatch (connection=connection@entry=0x7f4c3f5ba450) at dbus-connection.c:4631
#11 0x00007f4c3f058dda in bus_dispatch (m=m@entry=0x7f4c3f4f7360) at src/core/dbus.c:525
#12 0x00007f4c3f02c69f in manager_loop (m=0x7f4c3f4f7360) at src/core/manager.c:1816
#13 0x00007f4c3f023fb6 in main (argc=5, argv=0x7fffecb4a9d8) at src/core/main.c:1705
Comment 3 Kay Sievers 2013-08-14 06:46:44 EDT
Hmm, just to check, was there an earlier version of systemd on the
machine before?

If yes, was the machine rebooted after the update to the systemd.rpm
version 206?
Comment 4 Nikolai Kondrashov 2013-08-14 06:53:04 EDT
I'm not sure about the first crash (will try to check), but the second was after a reboot. Note though, that I haven't seen this again so far.
Comment 5 Nikolai Kondrashov 2013-08-28 11:45:50 EDT
Created attachment 791441 [details]
core.3272.gz

Another crash after a fresh reboot. This time with the following package versions:

systemd-libs-206-6.el7.x86_64
systemd-journal-gateway-206-6.el7.x86_64
systemd-debuginfo-206-6.el7.x86_64
systemd-206-6.el7.x86_64  
systemd-sysv-206-6.el7.x86_64

Repeated sssd restarts were involved as well.

Backtrace:
#0  0x00007ff25d0e4ffb in raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/pt-raise.c:37
#1  0x00007ff25ec2c95e in crash (sig=6) at src/core/main.c:144
#2  <signal handler called>
#3  0x00007ff25cd4a999 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#4  0x00007ff25cd4c0a8 in __GI_abort () at abort.c:90
#5  0x00007ff25ec85563 in log_assert (text=<optimized out>, file=0x7ff25ecde503 "src/shared/cgroup-util.c", line=866, func=0x7ff25ecde6c0 <__PRETTY_FUNCTION__.7913> "cg_is_empty_recursive", format=format@entry=0x7ff25ecdfab0 "Assertion '%s' failed at %s:%u, function %s(). Aborting.") at src/shared/log.c:699
#6  0x00007ff25ec85e10 in log_assert_failed (text=<optimized out>, file=<optimized out>, line=<optimized out>, func=<optimized out>) at src/shared/log.c:704
#7  0x00007ff25ec7c3f3 in cg_is_empty_recursive (controller=controller@entry=0x7ff25ecd6da7 "name=systemd", path=0x0, ignore_self=ignore_self@entry=true) at src/shared/cgroup-util.c:866
#8  0x00007ff25ec6d890 in manager_notify_cgroup_empty (m=m@entry=0x7ff25f6e1ad0, cgroup=<optimized out>) at src/core/cgroup.c:741
#9  0x00007ff25ec5e37d in private_bus_message_filter (connection=0x7ff25f7a15d0, message=0x7ff25f7a16e0, data=0x7ff25f6e1ad0) at src/core/dbus.c:491
#10 0x00007ff25d71f9e6 in dbus_connection_dispatch (connection=connection@entry=0x7ff25f7a15d0) at dbus-connection.c:4631
#11 0x00007ff25ec5edca in bus_dispatch (m=m@entry=0x7ff25f6e1ad0) at src/core/dbus.c:525
#12 0x00007ff25ec3269f in manager_loop (m=0x7ff25f6e1ad0) at src/core/manager.c:1816
#13 0x00007ff25ec29fb6 in main (argc=5, argv=0x7fff83ebf748) at src/core/main.c:1705
Comment 6 Harald Hoyer 2013-08-28 11:58:05 EDT
should be fixed with systemd-206-7 , which was built today

Note You need to log in before you can comment on or make changes to this bug.