Bugzilla will be upgraded to version 5.0 on December 2, 2018. The outage period for the upgrade will start at 0:00 UTC and have a duration of 12 hours
Bug 613578 - Free the reserved memory for crashkernel inside xen-guest
Free the reserved memory for crashkernel inside xen-guest
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: initscripts (Show other bugs)
6.0
All Linux
high Severity medium
: rc
: ---
Assigned To: initscripts Maintenance Team
Yulia Kopkova
: Reopened
Depends On: 608320 628817
Blocks:
  Show dependency treegraph
 
Reported: 2010-07-12 06:19 EDT by Cong Wang
Modified: 2013-09-29 22:18 EDT (History)
13 users (show)

See Also:
Fixed In Version: initscripts-9.03.14-1.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-11-10 15:40:33 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Cong Wang 2010-07-12 06:19:43 EDT
Description of problem:
Xen guest on both RHEL5 and RHEL6 should be using the hyper calls
in xen, so kexec-tools need to put a check inside of xen-guest
os that it won't load the kexec kernel; since kexec kernel won't work
in xen guest, and can panic the cpu.

Also on RHEL6, since the memory (on >4G i686+x86_64 or >8G on ppc64 system);
the kdump memory are reserved by default. kexec-tools will not enable kdump service inside Xen-guest, so we needs to free those memory in xen guest since xen guest doens't use the kexec kdump mechanism to capture core.

See also Bug 608320.
Comment 2 Bill Nottingham 2010-07-12 10:43:59 EDT
Moving to kexec-tools... I don't see what this has to do with initscripts.
Comment 3 Cong Wang 2010-07-12 22:07:26 EDT
Bill, please check: https://bugzilla.redhat.com/show_bug.cgi?id=608320#c10
/etc/rc.d/rc.local does belong to initscripts.

The reserved memory needs to be freed no matter if kdump service is started, so kdump.init is not a good place to do this.
Comment 4 Harald Hoyer 2010-07-13 04:40:52 EDT
huh? rc.local is for the administrator to put things in. So, you might want that in rc.sysinit.

How is the memory freed in shell code?
Comment 5 Cong Wang 2010-07-13 22:50:28 EDT
(In reply to comment #4)
> huh? rc.local is for the administrator to put things in. So, you might want
> that in rc.sysinit.


Yeah, you know initscripts better than me. :)

> 
> How is the memory freed in shell code?    

echo 0 > /sys/kernel/kexec_crash_size

The important part is not this, it is checking if we are inside a xen-guest.
Please refer Bug 608320.
Comment 6 Bill Nottingham 2010-07-14 10:47:11 EDT
The kernel knows if it's a xen kernel, the kernel knows the kexec_crash_size that's reserved. The initscripts neither knows nor cares about either. This should be fixed in the kernel, there's no reason the kernel should set a nonsensical config for itself and rely on userspace to clean it up.
Comment 7 RHEL Product and Program Management 2010-07-14 10:54:47 EDT
Development Management has reviewed and declined this request.  You may appeal
this decision by reopening this request.
Comment 8 Cong Wang 2010-07-16 05:04:21 EDT
(In reply to comment #6)
> The kernel knows if it's a xen kernel, the kernel knows the kexec_crash_size
> that's reserved. The initscripts neither knows nor cares about either. This
> should be fixed in the kernel, there's no reason the kernel should set a
> nonsensical config for itself and rely on userspace to clean it up.    

I think the reason why do this in user-space instead of kernel-space is that it's hard to make xen-guest work well with kdump, so we decided not to allow user to use kdump at all. Based on this, moving this policy into kernel is not a good idea.

I think Chris can give more information.
Comment 9 Chris Lalancette 2010-07-16 09:02:10 EDT
What's going on here is that we can't reliably make kdump work for RHEL-6 guests running on a Xen hypervisor (to be clear, on a RHEL-5 Xen hypervisor).  That's OK; Xen has a hypercall mechanism to force dom0 to take a core, so we at least have a mechanism to get cores when a RHEL-6 Xen guest fails.

However, because we are passing "crashkernel=auto" (or the new, equivalent syntax that we will use in RHEL-6.1), this means that in a Xen guest, we are wasting the reserved crash memory that can never be successfully used.  I see a number of ways of solving this:

1)  Have the kernel free up the memory if it knows it is running on Xen.  This could be done; it will probably require separate paths for PV and HVM Xen guests.  The downside here is that we are putting a policy into the kernel.  If you know what you are doing, there are specific instances where you could use kdump or kexec work in a Xen environment, so we would be disallowing those use cases.

2)  Have the kdump startup script free up the memory if it knows it is running on Xen.  This has the advantage that we don't put the policy in the kernel.  The downside is that if the kdump service is chkconfig --off, then the memory will never be freed.

3)  Have the system initscripts free up the memory if they know they are running on Xen.  This has the advantage that we don't put policy in the kernel and also has the advantage that it will always run.  It pretty much has the same disadvantage as 1), in that we are encoding a policy into the initscripts.  That being said, it's much easier to get around it (by modifying the initscripts) than a kernel hack would be.

I'm actually leaning towards 2), though I'm not entirely certain that is the right thing to do.  Amerigo, a couple of questions about the kdump service:

1)  Is it installed in all of the default package sets? (Server, AP, etc?)
2)  Is it chkconfig on by default?

Chris Lalancette
Comment 10 Bill Nottingham 2010-07-16 11:27:04 EDT
What happens in the case of a non-Xen-guest machine that's booted with 'crashkernel=auto', that would never want to use kdump? How is its memory freed?
Comment 11 Bill Nottingham 2010-07-16 11:27:52 EDT
To be specific, the problem appears to me that the kernel is automatically reserving memory for a feature that it doesn't know if the user actually wants to use.
Comment 12 Chris Lalancette 2010-07-16 12:28:18 EDT
(In reply to comment #11)
> To be specific, the problem appears to me that the kernel is automatically
> reserving memory for a feature that it doesn't know if the user actually wants
> to use.    

Yes, this is true.  However, the word from support is that *not* reserving memory by default is a much worse situation then possibly wasting the memory.  The experience from RHEL-5 shows that customers either forget to reserve the kdump memory when booting, or don't want to do the extra reboot cycle required after firstboot.  What this led to is that many users thought they would get coredumps, but when it came down to it, they had a misconfiguration.

The proposal for RHEL-6 is to automatically reserve the memory, getting rid of those two problems.  Since RHEL-6 *also* has the ability to free up the memory if it's not being used, then it seems best to always reserve the memory, and just free it up in the case the user doesn't want to use it.  What remains to be decided is where we do this freeing.

Chris Lalancette
Comment 13 Cong Wang 2010-07-18 22:27:54 EDT
(In reply to comment #9)
> I'm actually leaning towards 2), though I'm not entirely certain that is the
> right thing to do.

I prefer 3) so opened this bug. :)

> Amerigo, a couple of questions about the kdump service:
> 
> 1)  Is it installed in all of the default package sets? (Server, AP, etc?)

I don't think so, on some RHTS machines, I have to install kexec-tools manually.


> 2)  Is it chkconfig on by default?
> 

Yes.
Comment 14 Phil Knirsch 2010-07-21 09:50:36 EDT
Alright, i know that either Bill or Harald will kill me now for suggesting this (or both actually), but how about if kexec-tools or kdump would have a %post and a %postun section where the line to free the reserved memory would be added to /etc/rc.local in case it isn't there yet during installation (%post) and removed from /etc/rc.local if is is there during deinstallation(%postun)?

I know this is a hack just like the other proposals are, but at least the component being responsible for all this would at least do it and we wouldn't hardwire it for eternity in initscripts itself. This would have the distinct advantage that it could be easily changed/removed once the kernel can take care of everything itself.

Just my $0.02.

Thanks & regards, Phil

PS: Bug #608320 is already a blocker, so it could still be added there in case you want to go down that road.
Comment 15 Harald Hoyer 2010-07-21 10:28:10 EDT
(In reply to comment #14)
> Alright, i know that either Bill or Harald will kill me now for suggesting this
> (or both actually), but how about if kexec-tools or kdump would have a %post
> and a %postun section where the line to free the reserved memory would be added
> to /etc/rc.local in case it isn't there yet during installation (%post) and
> removed from /etc/rc.local if is is there during deinstallation(%postun)?

the problem is, that it should be freed, especially if kexec-tools or kdump are _not_ installed
Comment 16 Phil Knirsch 2010-07-21 10:29:13 EDT
Then add it to %post in kernel?

/me runs, fast...


Regards, Phil
Comment 17 Harald Hoyer 2010-07-21 10:45:43 EDT
as far as I understand, this could go to /etc/rc.sysinit.

if ! { [[ -f /proc/xen/capabilities ]] \
  && grep -q "control_d" /proc/xen/capabilities 2>/dev/null; }; then  
    echo OK
    if { [[ -f /sys/hypervisor/type ]] \
         && grep -q "xen" /sys/hypervisor/type 2>/dev/null; } \
       || { [[ -f /proc/acpi/dsdt ]] \
         && grep -qi "xen" /proc/acpi/dsdt 2>/dev/null;} ; then
        echo 0 > /sys/kernel/kexec_crash_size
    fi
fi

or it could go in an initscript which is always installed on a xen-guest.
Comment 18 Harald Hoyer 2010-07-21 10:46:26 EDT
doh.. with "echo OK" :)

if ! { [[ -f /proc/xen/capabilities ]] \
  && grep -q "control_d" /proc/xen/capabilities 2>/dev/null; }; then  
    if { [[ -f /sys/hypervisor/type ]] \
         && grep -q "xen" /sys/hypervisor/type 2>/dev/null; } \
       || { [[ -f /proc/acpi/dsdt ]] \
         && grep -qi "xen" /proc/acpi/dsdt 2>/dev/null;} ; then
        echo 0 > /sys/kernel/kexec_crash_size
    fi
fi
Comment 19 Bill Nottingham 2010-07-26 13:51:10 EDT
Given that that 'echo' command crashes a stock Fedora install, I'm loath to include this in any upstream repo.
Comment 20 Bill Nottingham 2010-07-26 13:51:26 EDT
(2.6.34-45.fc14.x86_64 was the kernel, FWIW.)
Comment 21 Neil Horman 2010-07-30 09:08:13 EDT
That echo command to kexec_crash_size should be fixed in RHEL6 at this point (I assume upstream as well).

RHEL6 commit: 
af96b2bf327929de3aa3419ba52858396852ec1f
upstrean commit from akpms git tree: 5f3bdd935fec7725420d76c7b1bbb0e89e144195
Comment 22 Vivek Goyal 2010-07-30 09:56:06 EDT
On what product variants to enable/disable kdump is a matter of policy and I think it should be implemented in user space. Xen case one can still think of
solving in kernel but similar question came up for minimal-install configuration,
how would we free up memory there? Kernel does not know that it is a minimal-install.

To me following might be one of the solutions.

- Make kexec-tools package install on every product variant as default and make
  kdump service ON by default.

- Let kexec-tools internally create a rule/policy file where it knows on what
  product varinats (minimal-install, xen kernel,....) etc to disable the kdump
  and free up the memory.

- This tool should also modify grub entry to get rid of any crashkernel= command
  lines. So that if user later removes kexec-tools package, it is not an issue
  and we don't want to get into the exercise of reserving and freeing up memory
  on every boot.
Comment 23 Neil Horman 2010-07-30 10:57:36 EDT
That seems rather heavy handed, I think it would be simpler, and perhaps  more user comprehensible to do the following:

1) add a check in rc.sysinit:
if [ ! -x /sbin/kexec -a `chkconfig --list kdump` == off ]
then
     echo 0 > /sys/kernel/kexec_crash_size
fi

2) INstall kexec-tools by default on only the plaforms we need it installed on

3) add logic in kdump.init to deny service startup if we're running as a xen guest

Thats seems a bit less invasive, and allows users to install kexec-tools later on without needing to mess with grub.conf at the same time
Comment 24 Neil Horman 2010-07-30 14:04:43 EDT
amerigo, given the discussion on kexec-list about this bug, since Bill has expressed some satisfaction with the idea in comment 23 above, do you want to go ahead and implement it, or would you like me to do the kexec side of this?
Comment 25 Cong Wang 2010-08-01 22:42:44 EDT
(In reply to comment #24)
> amerigo, given the discussion on kexec-list about this bug, since Bill has
> expressed some satisfaction with the idea in comment 23 above, do you want to
> go ahead and implement it, or would you like me to do the kexec side of this?    

1) belongs to init-scripts, 3) is done, you mean 2)?
Comment 27 Neil Horman 2010-08-03 07:21:49 EDT
amerigo, sorry, yes, you've done item (3).  and (1) belongs to the initscripts team.  So all thats left (if its left at all is for you to check and see if theres any install profile that we need to include/exclude kexec-tools from.
Comment 28 Bill Nottingham 2010-08-03 09:49:53 EDT
I don't really *like* the idea of (1) - my preferred solution would be for kexec to be able to reserve the memory when the service starts. But since that can't be done now, acking something along the lines of (1).
Comment 29 Bill Nottingham 2010-08-03 17:09:32 EDT
http://git.fedorahosted.org/git/?p=initscripts.git;a=commitdiff;h=b92eda2753c564d96a5c52b5fbfc5eca1ab3f8b3

Implemented as an upstart job for assorted technical and non-technical reasons.
Comment 32 releng-rhel@redhat.com 2010-11-10 15:40:33 EST
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.

Note You need to log in before you can comment on or make changes to this bug.