Description of problem: I wanted to list all the boots in the journal, so I ran `journalctl --list-boots`. The application does not finish and is permanently at 100 % cpu usage. I have limited the disk usage to 2GB and `journalctl --disk-usage` reports Archived and active journals take up 2.1G on disk. I ran `journalctl --verify`, and came up with a few corrupt log files. I deleted those, but the problem still persists. Does anyone have an idea what is going on? Version-Release number of selected component (if applicable): systemd-219-19.fc22.x86_64 How reproducible: Steps to Reproduce: 1. Run journalctl --list-boots Actual results: journalctl hangs with 100 % cpu usage Expected results: journalctl displays all boots in the log Additional info:
This is happening to me on systemd-219-20.fc22.x86_64. I've got a 36 MiB core dump, and a 62 MiB tar file of my /var/log/journal directory, both LZMA compressed if any Fedora developers would like to look at them. I'd rather provide them privately, because I can't be sure there's nothing sensitive, considering these are full system logs. -------------------------------- lsof(8) output: journalct 4094 jreinhart 0u CHR 136,0 0t0 3 /dev/pts/0 journalct 4094 jreinhart 1u CHR 136,0 0t0 3 /dev/pts/0 journalct 4094 jreinhart 2u CHR 136,0 0t0 3 /dev/pts/0 journalct 4094 jreinhart 5r REG 253,0 8388608 2509681 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/user-1001 journalct 4094 jreinhart 6r REG 253,0 8388608 2504326 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/user-1001 journalct 4094 jreinhart 7r REG 253,0 92274688 2504207 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/system~ journalct 4094 jreinhart 8r REG 253,0 16777216 2502050 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/user-1000 journalct 4094 jreinhart 9r REG 253,0 75497472 2490940 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/system journalct 4094 jreinhart 10r REG 253,0 8388608 2510372 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/user-1000 journalct 4094 jreinhart 11r REG 253,0 8388608 2505234 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/user-1001 journalct 4094 jreinhart 12r REG 253,0 8388608 2504322 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/user-1000 journalct 4094 jreinhart 13r REG 253,0 8388608 2510374 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/user-1001 journalct 4094 jreinhart 14r REG 253,0 16777216 2504325 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/user-1000~ journalct 4094 jreinhart 15r REG 253,0 8388608 2491048 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/system~ journalct 4094 jreinhart 16r REG 253,0 100663296 2510396 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/system journalct 4094 jreinhart 17r REG 253,0 8388608 2510384 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/user-1001.journal journalct 4094 jreinhart 18r REG 253,0 16777216 2493267 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/user-1000 journalct 4094 jreinhart 19r REG 253,0 8388608 2506305 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/user-1000~ journalct 4094 jreinhart 20r REG 253,0 8388608 2510397 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/user-1000 journalct 4094 jreinhart 21r REG 253,0 100663296 2494199 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/system journalct 4094 jreinhart 22r REG 253,0 8388608 2505147 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/user-1001 journalct 4094 jreinhart 23r REG 253,0 100663296 2509587 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/system journalct 4094 jreinhart 24r REG 253,0 8388608 2505143 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/user-1000 journalct 4094 jreinhart 25r REG 253,0 8388608 2502006 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/system.journal journalct 4094 jreinhart 26r REG 253,0 75497472 2506152 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/system journalct 4094 jreinhart 28r REG 253,0 100663296 2490449 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/system journalct 4094 jreinhart 29r REG 253,0 83886080 2504283 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/system journalct 4094 jreinhart 30r REG 253,0 16777216 2509677 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/user-1000 journalct 4094 jreinhart 31r REG 253,0 8388608 2510422 /var/log/journal/3a5255a2b5164201a3521ea3bf0939cc/user-1000.journal -------------------------------- A couple backtraces: #0 0x00007f9ee5fa9067 in munmap () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f9ee741fabf in window_unlink () #2 0x00007f9ee7420b73 in mmap_cache_get () #3 0x00007f9ee741ad04 in journal_file_move_to.lto_priv () #4 0x00007f9ee741ae07 in journal_file_move_to_object () #5 0x00007f9ee7432342 in test_object_realtime.lto_priv () #6 0x00007f9ee7432983 in generic_array_bisect.lto_priv () #7 0x00007f9ee7428fed in real_journal_next.lto_priv () #8 0x00007f9ee73fb9fd in get_boots.lto_priv () #9 0x00007f9ee73f85ea in main () #0 mmap64 () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f9ee74208ec in mmap_cache_get () #2 0x00007f9ee741ad04 in journal_file_move_to.lto_priv () #3 0x00007f9ee741ae07 in journal_file_move_to_object () #4 0x00007f9ee7432342 in test_object_realtime.lto_priv () #5 0x00007f9ee7432983 in generic_array_bisect.lto_priv () #6 0x00007f9ee7428fed in real_journal_next.lto_priv () #7 0x00007f9ee73fb9fd in get_boots.lto_priv () #8 0x00007f9ee73f85ea in main () -------------------------------- strace output shows: mmap(NULL, 7122944, PROT_READ, MAP_SHARED, 7, 0x5135000) = 0x7f4b23580000 munmap(0x7f4b2dc4b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 8, 0x4a6000) = 0x7f4b2dc4b000 munmap(0x7f4b3644b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 9, 0x1300000) = 0x7f4b3644b000 munmap(0x7f4b2144b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 9, 0x3a42000) = 0x7f4b2144b000 munmap(0x7f4b2a44b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 9, 0x3e59000) = 0x7f4b2a44b000 munmap(0x7f4b22c4b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 14, 0x605000) = 0x7f4b22d80000 munmap(0x7f4b287ae000, 4837376) = 0 mmap(NULL, 4239360, PROT_READ, MAP_SHARED, 14, 0xbf5000) = 0x7f4b31040000 munmap(0x7f4b2d44b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 16, 0x3f9000) = 0x7f4b2d44b000 munmap(0x7f4b26fae000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 16, 0xc78000) = 0x7f4b26fae000 munmap(0x7f4b2244b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 16, 0x2629000) = 0x7f4b22580000 munmap(0x7f4b21c4b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 16, 0x56db000) = 0x7f4b21d80000 munmap(0x7f4b267ae000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 18, 0x6c0000) = 0x7f4b267ae000 munmap(0x7f4b27fae000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 21, 0x3f3000) = 0x7f4b2844b000 munmap(0x7f4b277ae000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 21, 0x2630000) = 0x7f4b27c4b000 munmap(0x7f4b2cc4b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 21, 0x56f5000) = 0x7f4b2cc4b000 munmap(0x7f4b25fae000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 23, 0x6a2000) = 0x7f4b25fae000 munmap(0x7f4b30840000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 23, 0x12cf000) = 0x7f4b30840000 munmap(0x7f4b3344b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 23, 0x2dee000) = 0x7f4b3344b000 munmap(0x7f4b25fae000, 8388608) = 0 mmap(NULL, 4837376, PROT_READ, MAP_SHARED, 23, 0x5b63000) = 0x7f4b277ae000 munmap(0x7f4b30840000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 26, 0x14ca000) = 0x7f4b30840000 munmap(0x7f4b3344b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 26, 0x3d67000) = 0x7f4b3344b000 munmap(0x7f4b277ae000, 4837376) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 28, 0x3f1000) = 0x7f4b25fae000 munmap(0x7f4b20c4b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 28, 0x2617000) = 0x7f4b20c4b000 munmap(0x7f4b30840000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 28, 0x5717000) = 0x7f4b30840000 munmap(0x7f4b3344b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 29, 0x1598000) = 0x7f4b3344b000 munmap(0x7f4b25fae000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 29, 0x3f9a000) = 0x7f4b25fae000 munmap(0x7f4b2944b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 29, 0x47cc000) = 0x7f4b2944b000 munmap(0x7f4b30840000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 30, 0x5c8000) = 0x7f4b30840000 munmap(0x7f4b20c4b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 23, 0x1bfa000) = 0x7f4b20c4b000 munmap(0x7f4b3344b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 23, 0x244e000) = 0x7f4b3344b000 munmap(0x7f4b25fae000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 23, 0x29e4000) = 0x7f4b25fae000 munmap(0x7f4b2944b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 28, 0x101e000) = 0x7f4b2944b000 munmap(0x7f4b30840000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 23, 0x6a2000) = 0x7f4b30840000 munmap(0x7f4b28c4b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 23, 0x12cf000) = 0x7f4b28c4b000 munmap(0x7f4b2e44b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 23, 0x2dee000) = 0x7f4b2e44b000 munmap(0x7f4b23580000, 7122944) = 0 mmap(NULL, 4837376, PROT_READ, MAP_SHARED, 23, 0x5b63000) = 0x7f4b277ae000 munmap(0x7f4b2dc4b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 26, 0x14ca000) = 0x7f4b2dc4b000 munmap(0x7f4b3644b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 26, 0x3d67000) = 0x7f4b3644b000 munmap(0x7f4b2144b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 28, 0x3f1000) = 0x7f4b21580000 munmap(0x7f4b2a44b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 28, 0x2617000) = 0x7f4b2a44b000 munmap(0x7f4b31040000, 4239360) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 28, 0x5717000) = 0x7f4b1fc4b000 munmap(0x7f4b22d80000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 29, 0x1cf6000) = 0x7f4b2344b000 munmap(0x7f4b3144b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 29, 0x1598000) = 0x7f4b3144b000 munmap(0x7f4b26fae000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 29, 0x3f9a000) = 0x7f4b26fae000 munmap(0x7f4b22580000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 29, 0x47cc000) = 0x7f4b22c4b000 munmap(0x7f4b21d80000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 30, 0x5c8000) = 0x7f4b2244b000 munmap(0x7f4b267ae000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 7, 0x14a3000) = 0x7f4b267ae000 munmap(0x7f4b2d44b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 7, 0x262f000) = 0x7f4b2d44b000 munmap(0x7f4b2844b000, 8388608) = 0 mmap(NULL, 7122944, PROT_READ, MAP_SHARED, 7, 0x5135000) = 0x7f4b28580000 munmap(0x7f4b27c4b000, 8388608) = 0 mmap(NULL, 8388608, PROT_READ, MAP_SHARED, 8, 0x4a6000) = 0x7f4b27d80000 ...and you can see it repeats from here.
Looks like a bug in journal libs. If you send me (zbyszek.pl) the files (a link to somewhere where I can download them from would be best), I'll take a look.
The files have been sent.
*** This bug has been marked as a duplicate of bug 1324663 ***