Created attachment 1104182 [details]
strace from systemd while in broken event loop state

Description of problem:
The currently available version of systemd-219 for Fedora-22 is subject to a malfunction in the sd-event pending_prioq_compare function which can swap a disabled event source with an enabled one, causing a broken event loop followed by systemctl hangs eventually requiring a reboot.

The issue has been identified and fixed upstream:
http://lists.freedesktop.org/archives/systemd-devel/2015-September/034356.html
https://github.com/systemd/systemd/pull/1366


Version-Release number of selected component (if applicable):
systemd-219-25.fc22.x86_64


How reproducible:
6/10

With syscall tracing we were able to observe (under production load) the epoll_wait POLLOUT looping after our monitoring system noticed 'systemctl' processes piling up. The systems in question run between 5000~7000 units. I've attached the strace.

We were _not_ able to reproduce per the systemd mailing list post (which described a future version - 227, ahead of 219)

Steps to Reproduce:
1. gdb, attach to #1, b pending_prioq_compare
2. break the sd-event queue
3. inspect x, y locals, look for disabled event source, if not, continue (gdb script can help)
4. strace #1 to observe POLLOUT infinite loop / broken sd-event loop

Actual results:
- non-deterministic piling up systemctl processes
- heavy epoll_wait activity by #1 with an infinitely-increasing POLLOUT list

Expected results:
- systemctl processes not piling up
- normal (paired) epoll_wait2 behavior from #1, no disabled event sources swapped with enabled ones

Additional info:
We have deployed a custom build with this patch and have so far not been able to observe the infinite loop/sd-event malfunction under load.