Description of problem: I have many journal files, due to many users using the system and each creating their own journal: ls /var/log/journal/e3f1ec33551971df031878511bf6c160/ | wc -l 1126 If ulimit -n is not set sufficiently above 1024, journalctl silently fails to read many journal files and hangs without tailing logs when I use a "journalctl -f". If ulimit is set high enough, I get informed of what journal files are skipped: ulimit -n 200000 journalctl -n ...[clip 100 lines]... Too many open journal files, not adding /var/log/journal/e3f1ec33551971df031878511bf6c160/user-41277871.journal. Too many open journal files, not adding /var/log/journal/e3f1ec33551971df031878511bf6c160/user-55024. Error was encountered while opening journal files: Too many references: cannot splice Version-Release number of selected component (if applicable): systemd-216-13.fc21.x86_64 How reproducible: Problem has been in journalctl for some time, only just got to report it.
I now changed git upstream to bump RLIMIT_NOFILE to 16K if possible (effectively this is limited to 4K though for normal journalctl invocations since the default hard limit for user code defaults to 4K on fedora). http://cgit.freedesktop.org/systemd/systemd/commit/?id=de45d726034f33afdb0a185e62fc61bf10a0acd2 This is not a full fix though. We need to come up with a strategy what to do when we hit the limit. In the worst case we need to fail gracefully. This could mean warning the user that files are not shown and ensuring that the oldest files are the ones missing, not the newest ones...
Thanks for the quick attention! Not that I know the code, but do you also need to change the hard-coded constant in sd-journal.c ./sd-journal.c:#define JOURNAL_FILES_MAX 1024
I removed the check now in commit b914ea8d379b446c4c9fac4ba181771676ef38cd.
systemd-216-16.fc21 has been submitted as an update for Fedora 21. https://admin.fedoraproject.org/updates/systemd-216-16.fc21
Package systemd-216-16.fc21: * should fix your issue, * was pushed to the Fedora 21 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing systemd-216-16.fc21' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2015-0916/systemd-216-16.fc21 then log in and leave karma (feedback).
systemd-216-16.fc21 has been pushed to the Fedora 21 stable repository. If problems still persist, please make note of it in this bug report.