Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be unavailable on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1202598 - systemd throttling, taking up 25%+ CPU: "Looping too fast. Throttling execution a little."
Summary: systemd throttling, taking up 25%+ CPU: "Looping too fast. Throttling executi...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 22
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-03-17 01:40 UTC by Trevor Cordes
Modified: 2015-10-03 17:40 UTC (History)
12 users (show)

Fixed In Version: systemd-222-6.fc23
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-10-03 17:40:03 UTC
Type: Bug


Attachments (Terms of Use)
strace of systemd ps 1 while it was showing the bug, big file, compressed (755.95 KB, application/x-bzip)
2015-03-17 01:40 UTC, Trevor Cordes
no flags Details

Description Trevor Cordes 2015-03-17 01:40:03 UTC
Created attachment 1002585 [details]
strace of systemd ps 1 while it was showing the bug, big file, compressed

Description of problem:
System is acting slowly, top shows systemd using 25% CPU constantly, forever.  Can't do systemd things like restart a service.  Yum updates hang if they have to do any systemd stuff (like an httpd update).


Version-Release number of selected component (if applicable):
systemd-216-20.fc21.i686
kernel-PAE-core-3.18.7-200.fc21.i686


How reproducible:
It just happened once, I don't know if it will happen again.  Hardware is very new ECC/Xeon Intel Server-class, and unlikely to be faulty.


Steps to Reproduce:
1. Leave the system to run for a long time
2. Notice it's slow


Actual results:
systemd CPU is at 25%, systemd is "hung" as far as commands to it go.
journalctl says every <1s forever:
Mar 16 20:24:46 foo.com systemd[1]: Looping too fast. Throttling execution a little.


Expected results:
systemd at normal 0-1% CPU, systemctl commands work fine.


Additional info:
I got an strace of 1 while it was hung, see attached.

I looked at bugs 1187695 and 1186018 and while they give the same error, I don't think they are necessarily dupes because I wasn't yum/dnf updating anything when this started happening.

Comment 1 Trevor Cordes 2015-03-17 17:19:13 UTC
Further: I checked the logs more carefully.  The "Looping too fast" started at Mar 15 04:01:02.  That is precisely the time I have cron do a systemctl restart of clamd@scan.service and then clamav-milter.

Unlike bug #1187695, I had not updated anything for a while.  The last time dbus and systemd were updated together was Mar 7 and this bug did not hit until Mar 15.

So now just a simple systemctl restart is causing the "Looping too fast" symptoms.

P.S. Is there a way to get systemd to regain sanity in these cases without than a reboot?  I'm afraid that when systemd is this messed up it won't even do a clean shutdown.

Comment 2 Michal Sekletar 2015-03-18 12:54:56 UTC
Not sure if that will help but you can try sending SIGTERM to systemd when this happens. systemd should reexecute itself.

Comment 3 Lukáš Nykrýn 2015-03-20 15:20:33 UTC
If this is related to https://bugzilla.redhat.com/show_bug.cgi?id=1201964 maybe it would be sufficient to send SIGUSR1.

Comment 4 Lukáš Nykrýn 2015-03-20 15:21:06 UTC
Ugh wrong release please ignore.

Comment 5 Alexey I. Froloff 2015-05-22 14:06:17 UTC
This happened several time on my desktop during dnf upgrade process.  systemd loops this way when rpm %post scripts executing "systemctl try-restart whatever.service".  Killing the systemctl process break the loop for a while.

Comment 6 Alexey I. Froloff 2015-06-02 12:31:13 UTC
Tried to debug this issue as described in http://lists.freedesktop.org/archives/systemd-devel/2015-February/028541.html , but systemd-219-15.fc22.x86_64 doesn't seem to have the source_dispatch function.

Comment 7 flashl 2015-06-10 13:06:41 UTC
Systemd throttling : "Looping too fast. Throttling execution a little" appeared after upgrade from fedora 20 to 21 using fedup. Unable to reach multi-user target. Throttling appears to be related to cascading failures after systemctl fedora-import-state.service failure.

I can confirm #Comment 6, although I'm in single user mode using kernel 4.0.4.-202.fc21.x86_64+debug, troubleshooting fails because source_dispatch function is not found.

For me, this error is #urgent. Need means to resolve error.

Comment 8 Henrique Martins 2015-07-18 14:00:56 UTC
Same as in comment #5 with several systemctl try-restart that I had to kill.

Comment 9 Oleg Fayans 2015-09-02 07:41:02 UTC
Confirm the same with kernel 4.0.8-300.fc22.x86_64 and systemd-219-18.fc22.x86_64
In my case it caused the system to reboot on it's own.

Comment 10 Fedora Update System 2015-09-25 13:35:30 UTC
systemd-222-6.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-1e06faabb7

Comment 11 Fedora Update System 2015-09-27 00:55:03 UTC
systemd-222-6.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update systemd'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-1e06faabb7

Comment 12 Fedora Update System 2015-10-03 17:38:59 UTC
systemd-222-6.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.