Bug 241362

Summary: kdump: read on /proc/kcore returns EPERM always
Product: [Fedora] Fedora Reporter: Zing <zing>
Component: kexec-toolsAssignee: Neil Horman <nhorman>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: 8CC: davej, fche, lkundrak
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: kernel-2_6_24_7-92_fc8 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-05-07 20:02:10 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 238768    
Bug Blocks:    
Attachments:
Description Flags
patch to re-enable kcore access in the kernel
none
new version of patch to enable kcore access
none
patch to enable kcore on x86_64 none

Description Zing 2007-05-25 13:45:35 UTC
Description of problem:
This bug is a split off from bug #238768

The machine here is identical to the one in #238768:
intel xeon dual processor blade (8843-E9U) in an IBM Bladecenter H
(8852-4XU) but in this case with Fedora Core 6 x86_64 installed on it & updated.

The version of kexec-tools is the one located in the following link (which has
been patched to use /dev/crash):
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=238768#c17

Version-Release number of selected component (if applicable):
kexec-tools-1.101-173.test1.x86_64
kernel-2.6.20-1.2948.fc6.x86_64
kernel-kdump-2.6.20-1.2948.fc6.x86_64

/etc/kdump.conf:
ext3 /dev/mainvg/varlv

/etc/sysconfig/kdump:
KDUMP_COMMANDLINE=""
KDUMP_COMMANDLINE_APPEND="irqpoll maxcpus=1"
KEXEC_ARGS=" --args-linux"
KDUMP_BOOTDIR="/boot"
KDUMP_IMG_EXT=""
KDUMP_IMG="vmlinux"

With the above kdump config this error is received:
# service kdump start
No kdump kernel image found.                               [WARNING]
Tried to locate /boot/vmlinux-2.6.20-1.2948.fc6
Starting kdump:                                            [FAILED]

It has a little problem "auto finding" the kdump kernel.

Changing the following in /etc/sysconfig/kdump:
KDUMP_KERNELVER="2.6.20-1.2948.fc6kdump"

# service kdump start
Detected /etc/kdump.conf or /boot/vmlinux-2.6.20-1.2948.fc6kdump change
Rebuilding /boot/initrd-2.6.20-1.2948.fc6kdumpkdump.img
Starting kdump:                                            [FAILED]

/var/log/messages:
kernel: crash memory driver: version 1.0
kernel: crash memory driver: !page_is_ram(pfn: 0)
kdump: read on /dev/crash of 4096 bytes failed: Bad address Cannot read
/dev/crash: Bad address Cannot load /boot/vmlinux-2.6.20-1.2948.fc6kdump
kdump: kexec: failed to load kdump kernel
kdump: failed to start up

kexec command:
/sbin/kexec  --args-linux -p --command-line="ro root=LABEL=/ irqpoll maxcpus=1"
          --initrd=/boot/initrd-2.6.20-1.2948.fc6kdumpkdump.img
/boot/vmlinux-2.6.20-1.2948.fc6kdump

Comment 1 Neil Horman 2007-05-25 15:22:55 UTC
This is going to have to wait until such time as 238768 is fixed, since the
patch that causes this problem isn't checked into the bulid yet

Comment 2 Zing 2007-05-31 19:15:39 UTC
I just ran kexec-tools-1.101-69.fc7.x86_64 and the error is:

#/sbin/kexec  --args-linux -p          --command-line="ro root=LABEL=/  irqpoll
maxcpus=1"             --initrd=/boot/initrd-2.6.20-1.2948.fc6kdumpkdump.img
/boot/vmlinux-2.6.20-1.2948.fc6kdump
Cannot open /proc/kcore: Operation not permitted
Cannot read /proc/kcore: Operation not permitted
Cannot load /boot/vmlinux-2.6.20-1.2948.fc6kdump

Actually, I can open a new bug starting with this version of kexec-tools if you
want to scrap this one.  thanks.

Comment 3 Neil Horman 2007-05-31 20:01:13 UTC
updating bug summary to reflect current behavior, rather than behavior of my
last patch

Comment 6 Neil Horman 2007-06-04 18:23:40 UTC
Created attachment 156105 [details]
patch to re-enable kcore access in the kernel

Heres the patch to re-enable access to /proc/kcore.  I've spoken with the
selinux guys and we're are all in agreement that the best way forward here is
to re-enable kernel access and protect the file with selinux policy.

Comment 7 Neil Horman 2007-06-04 20:00:11 UTC
Created attachment 156119 [details]
new version of patch to enable kcore access

New version of the patch.  After discussing with davej, consensus was that
since its clear no one else needs kcore access anyway (evident by the fact it
is inaccessible), we decided to only make it accessible if the crashkernel
parameter is on the kernel command line.  This patch does that, on only does it
for x86_64 systems since kexec only needs access on that arch.

Comment 8 Neil Horman 2007-06-22 19:03:57 UTC
setting to waiting on davej until this goes into the fedora kernel


Comment 9 Dave Jones 2007-07-07 00:44:35 UTC
This is in rawhide now, and will trickle back to F7/F6 when we rebase to 2.6.22
soon.

Comment 10 Neil Horman 2007-07-07 16:01:17 UTC
Thanks dave!

Comment 11 Lubomir Kundrak 2008-04-09 11:43:42 UTC
Seems like this reappeared, so I was advised to reopen this bug report.

# /etc/init.d/kdump restart
Stopping kdump:                                            [  OK  ]
Starting kdump:                                            [FAILED]
# sh -x /etc/init.d/kdump restart 2>&1 |grep /sbin/kexec
+ KEXEC=/sbin/kexec
+ /sbin/kexec -p -u
+ /sbin/kexec --args-linux -p '--command-line=ro root=/dev/gophervg/rootlv 
irqpoll maxcpus=1' --initrd=/boot/initrd-2.6.24.4-69.fc8kdump.img 
                  /boot/vmlinuz-2.6.24.4-69.fc8
# /sbin/kexec --args-linux -p '--command-line=ro root=/dev/gophervg/rootlv 
irqpoll maxcpus=1' --initrd=/boot/initrd-2.6.24.4-69.fc8kdump.img 
                  /boot/vmlinuz-2.6.24.4-69.fc8
Cannot open /proc/kcore: Operation not permitted
Cannot read /proc/kcore: Operation not permitted
Cannot load /boot/vmlinuz-2.6.24.4-69.fc8
# 

# cat /proc/kcore
cat: /proc/kcore: Operation not permitted
# cat /proc/cmdline 
ro root=/dev/gophervg/rootlv crashkernel=128M@16M
# 

This was configured by system-config-kdump

Comment 12 Neil Horman 2008-04-09 12:48:48 UTC
dave, did the patch for this bug get dropped?

Comment 13 Dave Jones 2008-04-24 19:10:01 UTC
Arjan rewrote the diff to be a lot less invasive, and hopefully more palatable
for upstream.  I switched over to using his patch a while back. 

The odd thing is that the new patch doesn't do as much as the old one. It only
disables /dev/kmem, so why /proc/kcore isn't working is odd.

hmm, it seems to work for me..
$ sudo hexdump /proc/kcore   | head
0000000 457f 464c 0102 0001 0000 0000 0000 0000
0000010 0004 003e 0001 0000 0000 0000 0000 0000
0000020 0040 0000 0000 0000 0000 0000 0000 0000
0000030 0000 0000 0040 0038 0006 0000 0000 0000
0000040 0004 0000 0000 0000 0190 0000 0000 0000
....


Comment 14 Lubomir Kundrak 2008-04-24 19:31:33 UTC
Dave did you try on F8?

bash-3.2# hexdump /proc/kcore |head
hexdump: /proc/kcore: Operation not permitted
hexdump: /proc/kcore: Bad file descriptor
bash-3.2# uname -r
2.6.24.4-84.fc8
bash-3.2# 

Comment 15 Dave Jones 2008-04-24 19:42:50 UTC
ah, sorry, mistook this for a rawhide bug.
Hmm.  The code in F8 shouldn't have changed in ages.  The patch above is still
applied.   Strange.


Comment 16 Neil Horman 2008-04-24 20:39:28 UTC
could this be an selinux issue?  Can you check your logs for avc deinals?  It
may be that its denying your right to access that file, despite permissions?

Comment 17 Frank Ch. Eigler 2008-05-06 17:13:02 UTC
selinux/audit.log is all quiet on the matter.  (setenforce 0 doesn't make a
difference.)

[root@very ~]# uname -r
2.6.24.3-34.fc8
[root@very ~]# cat /proc/kcore
cat: /proc/kcore: Operation not permitted
[root@very ~]# id
uid=0(root) gid=0(root)
groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel)
context=unconfined_u:system_r:unconfined_t:s0-s0:c0.c1023
[root@very ~]# ls -al /proc/kcore
-r-------- 1 root root 5100277760 2008-05-06 13:12 /proc/kcore


Comment 18 Neil Horman 2008-05-06 17:35:17 UTC
Thanks frank.  One more question.  Its probably silly of me to ask, but you do
have a creashkernel= command line parameter specified on the production kernel
command line right?  The patch that grants  access to /proc/kcore only enables
read-permissions in the event that you specify that parameter.  Without it the
file is unreadable.

can you check /proc/cmdline and verify your crashkernel parameter please?  Thanks!

Comment 19 Frank Ch. Eigler 2008-05-06 18:02:38 UTC
Certainly:
% cat /proc/cmdline 
ro root=/dev/VolGroup00/LogVol00 crashkernel=64m@64m printk.time=1

... and confirmed by /proc/iomem.
The kernel code could log the reason for its EPERM return, in case there is more
than one possibility, to help debugging.


Comment 20 Neil Horman 2008-05-06 20:14:20 UTC
Created attachment 304681 [details]
patch to enable kcore on x86_64

hmm, ok.

wait a sec....I see the problem.

There was a screw up with the 32/64 bit x86 merge.  The bits that enabled kcore
access when passing the crashkernel parameter were applied to the 32 bit setup
code, but not the 64 bit setup code.  This patch should fix it.  Could you test
& confirm please?  Thanks frank!

apologies for my blidness :)

Comment 21 Frank Ch. Eigler 2008-05-06 20:52:07 UTC
sorry, don't have a suitable build/machine available just now; but I agree, the
patch looks obvious.

Comment 22 Neil Horman 2008-05-07 12:49:21 UTC
Dave, I'm looking at the rawhide tree, and we don't include any of these
allow_kcore_access bits at all (which means everything should just work).  Is
the answer here to just strip it all out of F-8 as well, or shall we forward
port the lot to rawhide?

Comment 23 Dave Jones 2008-05-07 15:26:26 UTC
I imagine it's going to be needed for F9 & rawhide too, or kdump isn't going to
do the right thing is it ?

Comment 24 Neil Horman 2008-05-07 16:51:48 UTC
well, thats rather the ironic thing about all this.  Upstream kernels work fine,
since to open /proc/kcore all you need is privs on CAP_RAW_IO.  Somewhere along
the line Fedora/RHEL (at least in RHEL5 & F-8) we decided to carry a security
patch to disable all access to /proc/kcore (it always returns 0).  That
necessitated the addition of the allow_kcore_access bits, so that the always
deny policy could bee overridden if you were using kexec.  Its all rather
hackish.  Ijust tested  and my patch works fine, so if you'd like to add it in ,
that would be great. That being said, I've always felt its rather a hack, so if
instead, you'd just like to revert the allow_kcore_access bits and mirror the
open_kcore function from upstream, that would be ideal in my mind.

Comment 25 Neil Horman 2008-05-07 20:02:10 UTC
after talking with davej, I've updated F-8 to behave like upstream, in which
only CAP_RAW_IO is needed to open /proc/kcore.  Tested and fixed.  Thanks!

Comment 26 Fedora Update System 2008-05-14 21:32:03 UTC
kernel-2.6.24.7-92.fc8 has been pushed to the Fedora 8 stable repository.  If problems still persist, please make note of it in this bug report.