Description of problem: In Fedora 19, the pstore filesystem is automatically mounted on EFI systems at /sys/fs/pstore. If the kernel crashes, it will store dump files under this location which can often be used to get failure data for hard to diagnose problems. This is often the only source of crash information for "doesn't boot" or "hung machine" type bugs. It would be beneficial for ABRT to look in this directory upon startup, assemble a backtrace from the files if they exist, and report a bug. Additionally, ABRT should delete these files from the /sys/fs/pstore/ directory. They are saved in UEFI NVRAM, which is a very limited resource. Leaving these around will eventually prevent new UEFI variables from being created.
Just FYI, this will trigger behaviour like 928982 even once we fix the problem that's currently causing that bug, so it is pretty important.
Given "They are saved in UEFI NVRAM, which is a very limited resource", should we perhaps consider disabling this until abrt is actually ready to handle the dumps?
> It would be beneficial for ABRT to look in this directory upon startup, assemble a backtrace from the files if they exist, and report a bug. Please give me a few examples of /sys/fs/pstore contents. Re deletion: what should be deleted there? Everyting?
(In reply to comment #3) > > It would be beneficial for ABRT to look in this directory upon startup, assemble a backtrace from the files if they exist, and report a bug. > > Please give me a few examples of /sys/fs/pstore contents. > Re deletion: what should be deleted there? Everyting? Anaconda deletes everything under /sys/fs/pstore on installation, but that probably isn't the proper thing to do for ABRT. The files ABRT will be interested in are of the type: dmesg-efi-N where N is a number. E.g. [jwboyer@sb ~]$ ls -l /sys/fs/pstore/ total 0 -r--r--r--. 1 root root 998 Apr 16 14:14 dmesg-efi-1 -r--r--r--. 1 root root 978 Apr 16 14:14 dmesg-efi-10 -r--r--r--. 1 root root 976 Apr 16 14:14 dmesg-efi-11 -r--r--r--. 1 root root 977 Apr 16 14:14 dmesg-efi-2 -r--r--r--. 1 root root 1022 Apr 16 14:14 dmesg-efi-3 -r--r--r--. 1 root root 1005 Apr 16 14:14 dmesg-efi-4 -r--r--r--. 1 root root 980 Apr 16 14:14 dmesg-efi-5 -r--r--r--. 1 root root 981 Apr 16 14:14 dmesg-efi-6 -r--r--r--. 1 root root 1016 Apr 16 14:14 dmesg-efi-7 -r--r--r--. 1 root root 897 Apr 16 14:14 dmesg-efi-8 -r--r--r--. 1 root root 946 Apr 16 14:14 dmesg-efi-9 [jwboyer@sb ~]$ The contents contained within are records for backtraces that look like this: [jwboyer@sb ~]$ cat /sys/fs/pstore/dmesg-efi-1 Panic#2 Part1 <4>[ 793.321074] Call Trace: <4>[ 793.323855] [<ffffffff8144be47>] __handle_sysrq+0x127/0x190 <4>[ 793.326659] [<ffffffff8144beb0>] ? __handle_sysrq+0x190/0x190 <4>[ 793.329474] [<ffffffff8144befa>] write_sysrq_trigger+0x4a/0x50 <4>[ 793.332285] [<ffffffff812529ac>] proc_reg_write+0x7c/0xc0 <4>[ 793.335102] [<ffffffff811e108f>] vfs_write+0xaf/0x190 <4>[ 793.337915] [<ffffffff811e1545>] sys_write+0x55/0xa0 <4>[ 793.340720] [<ffffffff81740519>] system_call_fastpath+0x16/0x1b <4>[ 793.343536] Code: e1 f7 ff ff eb d8 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 66 66 66 66 90 55 c7 05 3c d9 ce 00 01 00 00 00 48 89 e5 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 66 66 66 66 90 55 48 89 e5 53 48 <1>[ 793.349823] RIP [<ffffffff8144b616>] sysrq_handle_crash+0x16/0x20 <4>[ 793.352888] RSP <ffff880104a01e28> <4>[ 793.355937] CR2: 0000000000000000 <4>[ 793.372892] ---[ end trace 1675b8f5d63a931e ]--- <0>[ 793.534721] Kernel panic - not syncing: Fatal exception
Don't assume that they'll contain -efi- in the filename - different backends will have different strings there, and they should probably all be handled.
That's it? No kernel version? No explanation of the reason why kernel panics? Here is an example of a "regular" oops: BUG: unable to handle kernel NULL pointer dereference at 00000000 IP: [<f88dec25>] :radeon:radeon_cp_init_ring_buffer+0x90/0x302 *pde = 6f5c6067 Oops: 0000 [#1] SMP. Modules linked in: r8169 mii fuse nfsd lockd nfs_acl auth_rpcgss exportfs bridge stp bnep sco l2cap bl Pid: 8003, comm: Xorg Not tainted (2.6.27.9-159.fc10.i686 #1) ^^^^^^^^^^^^^^^^^^^^^^ important! EIP: 0060:[<f88dec25>] EFLAGS: 00213246 CPU: 1 EIP is at radeon_cp_init_ring_buffer+0x90/0x302 [radeon] EAX: 00000000 EBX: f78b4000 ECX: f78b4000 EDX: 00000000 ESI: f5dbe800 EDI: 00006458 EBP: f0a0cf18 ESP: f0a0cf08 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 Process Xorg (pid: 8003, ti=f0a0c000 task=f2380000 task.ti=f0a0c000) Stack: f0a0cf18 f78b4000 f5dbe800 00006458 f0a0cf28 f88e11c7 f8911a24 00000000. f0a0cf4c f88745f8 f30c3ba0 f5dbe800 f88e114a f5dbe828 f890fd78 f097ac00. 00000000 f0a0cf68 c049b1c0 00000000 00006458 f097ac00 f097ac00 00000000. Call Trace: [<f88e11c7>] radeon_cp_resume+0x7d/0xbc [radeon] [<f88745f8>] drm_ioctl+0x1b0/0x225 [drm] [<f88e114a>] radeon_cp_resume+0x0/0xbc [radeon] [<c049b1c0>] vfs_ioctl+0x50/0x69 [<c049b414>] do_vfs_ioctl+0x23b/0x247 [<c0460a56>] audit_syscall_entry+0xf9/0x123 [<c049b460>] sys_ioctl+0x40/0x5c [<c0403c76>] syscall_call+0x7/0xb ======================= Code: 66 31 d2 09 c2 89 d8 e8 fc e7 ff ff 8b 83 cc 00 00 00 8b 53 34 03 10 8b 86 70 02 00 00 2b 50 44 EIP: [<f88dec25>] radeon_cp_init_ring_buffer+0x90/0x302 [radeon] SS:ESP 0068:f0a0cf08 The example you provided looks like merely a *tail* of such oops. It even has no kernel version. Imagine that we (abrt reporter tool) are trying to file a BZ. Under which version we should file it? Is it really that bad?
No. Josh provided the end of the oops. You need to reassemble the files in order to obtain the full backtrace.
(In reply to comment #7) > No. Josh provided the end of the oops. You need to reassemble the files in > order to obtain the full backtrace. Please give me a complete example (as in: set of /sys/fs/pstore files which constitutes one complete oops).
Committed to upstream git: commit 79d189a1e0a35108954f1476dd4ddbd46760b0fe Author: Denys Vlasenko <dvlasenk> Date: Fri Apr 26 17:33:19 2013 +0200 tests/runtests/uefioops: new test Signed-off-by: Denys Vlasenko <dvlasenk> Signed-off-by: Martin Milata <mmilata> commit c8f39aacf8356cd70763fc2802b544da38a24fbb Author: Denys Vlasenko <dvlasenk> Date: Fri Apr 26 17:33:18 2013 +0200 specfile: hook up abrt-uefioops service Signed-off-by: Denys Vlasenko <dvlasenk> Signed-off-by: Martin Milata <mmilata> commit 865f7b88b391f2fbd71baa1aa3d6320455d4f519 Author: Denys Vlasenko <dvlasenk> Date: Fri Apr 26 17:33:17 2013 +0200 abrt-uefioops: new service This service, once per boot, scans /sys/fs/pstore/* and crates oops problem dirs from this data. Signed-off-by: Denys Vlasenko <dvlasenk> Signed-off-by: Martin Milata <mmilata>
Pstore isn't UEFI-specific - calling it uefioops probably isn't the best idea.
Fixed in abrt git.
uefi-oops renamed to pstore-oops in commit 205a7d29cdb91bec9a7174dad90e5c39d466434e
gnome-abrt-0.3.1-1.fc19,abrt-2.1.7-1.fc19,libreport-2.1.7-1.fc19,satyr-0.9-1.fc19 has been submitted as an update for Fedora 19. https://admin.fedoraproject.org/updates/gnome-abrt-0.3.1-1.fc19,abrt-2.1.7-1.fc19,libreport-2.1.7-1.fc19,satyr-0.9-1.fc19
Package gnome-abrt-0.3.1-1.fc19, abrt-2.1.7-1.fc19, libreport-2.1.7-1.fc19, satyr-0.9-1.fc19: * should fix your issue, * was pushed to the Fedora 19 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing gnome-abrt-0.3.1-1.fc19 abrt-2.1.7-1.fc19 libreport-2.1.7-1.fc19 satyr-0.9-1.fc19' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2013-16707/gnome-abrt-0.3.1-1.fc19,abrt-2.1.7-1.fc19,libreport-2.1.7-1.fc19,satyr-0.9-1.fc19 then log in and leave karma (feedback).
gnome-abrt-0.3.1-1.fc19, abrt-2.1.7-1.fc19, libreport-2.1.7-1.fc19, satyr-0.9-1.fc19 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report.