Description of problem: This is an RFE to add kernel source support into retrace-server in some way. This should be an optional feature. The design needs worked out and of course this will increase storage requirements. We could support it directly in retrace-server or via a custom hook 'extension' to retrace-server, provided we implement the custom hooks (see https://bugzilla.redhat.com/show_bug.cgi?id=1082376) and the kernel version of the vmcore is easily available (it is today in /cores/retrace/tasks/<taskid>/kernelver). Version-Release number of selected component (if applicable): retrace-server-1.11-4.el6.noarch Actual results: Unable to list source code via 'gdb list' on a retrace-server machine. People have to clone git trees or rpmbuild -bp the kernel source to look at the source tree for every vmcore. Expected results: Ability to use 'gdb list' to show source code of vmcores. crash> bt -l PID: 6674 TASK: ffff88007b8f0ae0 CPU: 0 COMMAND: "test_locks5" ... [exception RIP: locks_remove_flock+253] RIP: ffffffff811d92bd RSP: ffff88007b8f3de8 RFLAGS: 00010246 RAX: 0000000000000081 RBX: ffff88007abe73c0 RCX: 0000000000000000 RDX: ffff88007b8f0ae0 RSI: ffff880037cd17c0 RDI: 0000000000000282 RBP: ffff88007b8f3eb8 R8: ffff880079ccfc80 R9: 0000000000000002 R10: 0000000000000001 R11: 0000000000000000 R12: ffff88007b28d730 R13: ffff88007b8f3de8 R14: ffff88007b329780 R15: ffff8800376996c0 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000 #7 [ffff88007b8f3de0] locks_remove_flock at ffffffff811d9283 /usr/src/debug/kernel-2.6.32-431.17.1.el6/linux-2.6.32-431.17.1.el6.x86_64/fs/locks.c: 2026 #8 [ffff88007b8f3ec0] __fput at ffffffff8118a3c0 /usr/src/debug/kernel-2.6.32-431.17.1.el6/linux-2.6.32-431.17.1.el6.x86_64/fs/file_table.c: 248 #9 [ffff88007b8f3f10] fput at ffffffff8118a525 /usr/src/debug/kernel-2.6.32-431.17.1.el6/linux-2.6.32-431.17.1.el6.x86_64/fs/file_table.c: 200 #10 [ffff88007b8f3f20] filp_close at ffffffff8118584d /usr/src/debug/kernel-2.6.32-431.17.1.el6/linux-2.6.32-431.17.1.el6.x86_64/fs/open.c: 977 #11 [ffff88007b8f3f50] sys_close at ffffffff81185925 /usr/src/debug/kernel-2.6.32-431.17.1.el6/linux-2.6.32-431.17.1.el6.x86_64/fs/open.c: 1007 #12 [ffff88007b8f3f80] tracesys at ffffffff8100b288 (via system_call) /usr/src/debug////////kernel-2.6.32-431.17.1.el6/linux-2.6.32-431.17.1.el6.x86_64/arch/x86/kernel/entry_64.S: 606 RIP: 00007f7fddfd65ad RSP: 00007f7fde3f5e70 RFLAGS: 00000293 RAX: ffffffffffffffda RBX: ffffffff8100b288 RCX: ffffffffffffffff RDX: 0000000000000004 RSI: 0000000000400000 RDI: 0000000000000003 RBP: 00007f7fde3f5eb0 R8: 00000000ffffffff R9: 00007f7fde3f6700 R10: 00007f7fde3f5c00 R11: 0000000000000293 R12: 00007f7fde3f69c0 R13: 00007fffb3475d70 R14: 00007f7fd80008c0 R15: 0000000000000003 ORIG_RAX: 0000000000000003 CS: 0033 SS: 002b crash> gdb list fs/locks.c:2026 2021 } 2022 2023 lock_kernel(); 2024 before = &inode->i_flock; 2025 2026 while ((fl = *before) != NULL) { 2027 if (fl->fl_file == filp) { 2028 if (IS_FLOCK(fl)) { 2029 locks_delete_lock(before); 2030 continue; Additional info: For practical / implementation purposes, we'll need to account for increased storage space as well. Storage space is outside the scope of this RFE but we may want to mention it in documentation and/or comment above any config option in /etc/retrace-server.conf The question is, how do we do the trees? If git trees are available, a lot of space can be saved with 'git clone --reference'. If we do "rpmbuild -bp" then it may be more reliable.
This is definitely a useful feature - there have many times when I've opened a vmcore and wished this feature was available. The ability to quickly look at the faulting source code could result in an equally quick confirmation of a known problem without needing to find a repository and checkout the correct branch. But with most vmcore analyses I do it's only a matter of time before I need the full kernel source indexed with cscope/tags so I can search it and the recent change history too (and not to mention other trees to compare with too). So while this is a very convenient feature it does have it's limits.
This is indeed an interesting feature, that can increase our efficiency when doing the "first touch" on a vmcore and, as Lachlan points out, quickly confirm known issues, or checking the vmcore sources against the most current ones. Since many parts of the kernel remain unchanged between minor releases and z-streams, an engineer could be very efficient with a checkout of the most recent sources and comparing it against what crash shows for the vmcore. If the sources are the same for the problem under consideration, the engineer can quickly confirm or discard a hypothesis without waiting for the local checkout for that particular release, or decide to check the commit log for the particular file under consideration to quickly determine why something changed. My only concern would be how efficient we can make the sources storage space. I wonder how much we can use hardlinking and deduplication to keep that manageable.
Yes I was thinking it would be useful to have the source even if all it was used for was listing out the source code of the faulting task. Having the source right there automatically for every vmcore also opens more possibilities for automated analysis. The other thing is it just feels incomplete to me to have a core (userspace or kernel) analysis system without source code integration. Point taken that a lot of people that work on vmcores have their own trees and/or may need more than just the one tree. I would like to hear from some kernel TSE and non-SEGs.
This message is a reminder that EPEL 6 is nearing its end of life. Fedora will stop maintaining and issuing updates for EPEL 6 on 2020-11-30. It is our policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of 'el6'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later EPEL version. Thank you for reporting this issue and we are sorry that we were not able to fix it before EPEL 6 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above.
EPEL el6 changed to end-of-life (EOL) status on 2020-11-30. EPEL el6 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of EPEL please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.