Bug 613578
Summary: | Free the reserved memory for crashkernel inside xen-guest | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 6 | Reporter: | Cong Wang <amwang> |
Component: | initscripts | Assignee: | initscripts Maintenance Team <initscripts-maint-list> |
Status: | CLOSED CURRENTRELEASE | QA Contact: | Yulia Kopkova <ykopkova> |
Severity: | medium | Docs Contact: | |
Priority: | high | ||
Version: | 6.0 | CC: | borgan, clalance, ddumas, harald, lwang, nhorman, notting, pknirsch, qcai, rkhan, snagar, vgoyal, ykopkova |
Target Milestone: | rc | Keywords: | Reopened |
Target Release: | --- | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | initscripts-9.03.14-1.el6 | Doc Type: | Bug Fix |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2010-11-10 20:40:33 UTC | Type: | --- |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 608320, 628817 | ||
Bug Blocks: |
Description
Cong Wang
2010-07-12 10:19:43 UTC
Moving to kexec-tools... I don't see what this has to do with initscripts. Bill, please check: https://bugzilla.redhat.com/show_bug.cgi?id=608320#c10 /etc/rc.d/rc.local does belong to initscripts. The reserved memory needs to be freed no matter if kdump service is started, so kdump.init is not a good place to do this. huh? rc.local is for the administrator to put things in. So, you might want that in rc.sysinit. How is the memory freed in shell code? (In reply to comment #4) > huh? rc.local is for the administrator to put things in. So, you might want > that in rc.sysinit. Yeah, you know initscripts better than me. :) > > How is the memory freed in shell code? echo 0 > /sys/kernel/kexec_crash_size The important part is not this, it is checking if we are inside a xen-guest. Please refer Bug 608320. The kernel knows if it's a xen kernel, the kernel knows the kexec_crash_size that's reserved. The initscripts neither knows nor cares about either. This should be fixed in the kernel, there's no reason the kernel should set a nonsensical config for itself and rely on userspace to clean it up. Development Management has reviewed and declined this request. You may appeal this decision by reopening this request. (In reply to comment #6) > The kernel knows if it's a xen kernel, the kernel knows the kexec_crash_size > that's reserved. The initscripts neither knows nor cares about either. This > should be fixed in the kernel, there's no reason the kernel should set a > nonsensical config for itself and rely on userspace to clean it up. I think the reason why do this in user-space instead of kernel-space is that it's hard to make xen-guest work well with kdump, so we decided not to allow user to use kdump at all. Based on this, moving this policy into kernel is not a good idea. I think Chris can give more information. What's going on here is that we can't reliably make kdump work for RHEL-6 guests running on a Xen hypervisor (to be clear, on a RHEL-5 Xen hypervisor). That's OK; Xen has a hypercall mechanism to force dom0 to take a core, so we at least have a mechanism to get cores when a RHEL-6 Xen guest fails. However, because we are passing "crashkernel=auto" (or the new, equivalent syntax that we will use in RHEL-6.1), this means that in a Xen guest, we are wasting the reserved crash memory that can never be successfully used. I see a number of ways of solving this: 1) Have the kernel free up the memory if it knows it is running on Xen. This could be done; it will probably require separate paths for PV and HVM Xen guests. The downside here is that we are putting a policy into the kernel. If you know what you are doing, there are specific instances where you could use kdump or kexec work in a Xen environment, so we would be disallowing those use cases. 2) Have the kdump startup script free up the memory if it knows it is running on Xen. This has the advantage that we don't put the policy in the kernel. The downside is that if the kdump service is chkconfig --off, then the memory will never be freed. 3) Have the system initscripts free up the memory if they know they are running on Xen. This has the advantage that we don't put policy in the kernel and also has the advantage that it will always run. It pretty much has the same disadvantage as 1), in that we are encoding a policy into the initscripts. That being said, it's much easier to get around it (by modifying the initscripts) than a kernel hack would be. I'm actually leaning towards 2), though I'm not entirely certain that is the right thing to do. Amerigo, a couple of questions about the kdump service: 1) Is it installed in all of the default package sets? (Server, AP, etc?) 2) Is it chkconfig on by default? Chris Lalancette What happens in the case of a non-Xen-guest machine that's booted with 'crashkernel=auto', that would never want to use kdump? How is its memory freed? To be specific, the problem appears to me that the kernel is automatically reserving memory for a feature that it doesn't know if the user actually wants to use. (In reply to comment #11) > To be specific, the problem appears to me that the kernel is automatically > reserving memory for a feature that it doesn't know if the user actually wants > to use. Yes, this is true. However, the word from support is that *not* reserving memory by default is a much worse situation then possibly wasting the memory. The experience from RHEL-5 shows that customers either forget to reserve the kdump memory when booting, or don't want to do the extra reboot cycle required after firstboot. What this led to is that many users thought they would get coredumps, but when it came down to it, they had a misconfiguration. The proposal for RHEL-6 is to automatically reserve the memory, getting rid of those two problems. Since RHEL-6 *also* has the ability to free up the memory if it's not being used, then it seems best to always reserve the memory, and just free it up in the case the user doesn't want to use it. What remains to be decided is where we do this freeing. Chris Lalancette (In reply to comment #9) > I'm actually leaning towards 2), though I'm not entirely certain that is the > right thing to do. I prefer 3) so opened this bug. :) > Amerigo, a couple of questions about the kdump service: > > 1) Is it installed in all of the default package sets? (Server, AP, etc?) I don't think so, on some RHTS machines, I have to install kexec-tools manually. > 2) Is it chkconfig on by default? > Yes. Alright, i know that either Bill or Harald will kill me now for suggesting this (or both actually), but how about if kexec-tools or kdump would have a %post and a %postun section where the line to free the reserved memory would be added to /etc/rc.local in case it isn't there yet during installation (%post) and removed from /etc/rc.local if is is there during deinstallation(%postun)? I know this is a hack just like the other proposals are, but at least the component being responsible for all this would at least do it and we wouldn't hardwire it for eternity in initscripts itself. This would have the distinct advantage that it could be easily changed/removed once the kernel can take care of everything itself. Just my $0.02. Thanks & regards, Phil PS: Bug #608320 is already a blocker, so it could still be added there in case you want to go down that road. (In reply to comment #14) > Alright, i know that either Bill or Harald will kill me now for suggesting this > (or both actually), but how about if kexec-tools or kdump would have a %post > and a %postun section where the line to free the reserved memory would be added > to /etc/rc.local in case it isn't there yet during installation (%post) and > removed from /etc/rc.local if is is there during deinstallation(%postun)? the problem is, that it should be freed, especially if kexec-tools or kdump are _not_ installed Then add it to %post in kernel? /me runs, fast... Regards, Phil as far as I understand, this could go to /etc/rc.sysinit. if ! { [[ -f /proc/xen/capabilities ]] \ && grep -q "control_d" /proc/xen/capabilities 2>/dev/null; }; then echo OK if { [[ -f /sys/hypervisor/type ]] \ && grep -q "xen" /sys/hypervisor/type 2>/dev/null; } \ || { [[ -f /proc/acpi/dsdt ]] \ && grep -qi "xen" /proc/acpi/dsdt 2>/dev/null;} ; then echo 0 > /sys/kernel/kexec_crash_size fi fi or it could go in an initscript which is always installed on a xen-guest. doh.. with "echo OK" :) if ! { [[ -f /proc/xen/capabilities ]] \ && grep -q "control_d" /proc/xen/capabilities 2>/dev/null; }; then if { [[ -f /sys/hypervisor/type ]] \ && grep -q "xen" /sys/hypervisor/type 2>/dev/null; } \ || { [[ -f /proc/acpi/dsdt ]] \ && grep -qi "xen" /proc/acpi/dsdt 2>/dev/null;} ; then echo 0 > /sys/kernel/kexec_crash_size fi fi Given that that 'echo' command crashes a stock Fedora install, I'm loath to include this in any upstream repo. (2.6.34-45.fc14.x86_64 was the kernel, FWIW.) That echo command to kexec_crash_size should be fixed in RHEL6 at this point (I assume upstream as well). RHEL6 commit: af96b2bf327929de3aa3419ba52858396852ec1f upstrean commit from akpms git tree: 5f3bdd935fec7725420d76c7b1bbb0e89e144195 On what product variants to enable/disable kdump is a matter of policy and I think it should be implemented in user space. Xen case one can still think of solving in kernel but similar question came up for minimal-install configuration, how would we free up memory there? Kernel does not know that it is a minimal-install. To me following might be one of the solutions. - Make kexec-tools package install on every product variant as default and make kdump service ON by default. - Let kexec-tools internally create a rule/policy file where it knows on what product varinats (minimal-install, xen kernel,....) etc to disable the kdump and free up the memory. - This tool should also modify grub entry to get rid of any crashkernel= command lines. So that if user later removes kexec-tools package, it is not an issue and we don't want to get into the exercise of reserving and freeing up memory on every boot. That seems rather heavy handed, I think it would be simpler, and perhaps more user comprehensible to do the following: 1) add a check in rc.sysinit: if [ ! -x /sbin/kexec -a `chkconfig --list kdump` == off ] then echo 0 > /sys/kernel/kexec_crash_size fi 2) INstall kexec-tools by default on only the plaforms we need it installed on 3) add logic in kdump.init to deny service startup if we're running as a xen guest Thats seems a bit less invasive, and allows users to install kexec-tools later on without needing to mess with grub.conf at the same time amerigo, given the discussion on kexec-list about this bug, since Bill has expressed some satisfaction with the idea in comment 23 above, do you want to go ahead and implement it, or would you like me to do the kexec side of this? (In reply to comment #24) > amerigo, given the discussion on kexec-list about this bug, since Bill has > expressed some satisfaction with the idea in comment 23 above, do you want to > go ahead and implement it, or would you like me to do the kexec side of this? 1) belongs to init-scripts, 3) is done, you mean 2)? amerigo, sorry, yes, you've done item (3). and (1) belongs to the initscripts team. So all thats left (if its left at all is for you to check and see if theres any install profile that we need to include/exclude kexec-tools from. I don't really *like* the idea of (1) - my preferred solution would be for kexec to be able to reserve the memory when the service starts. But since that can't be done now, acking something along the lines of (1). http://git.fedorahosted.org/git/?p=initscripts.git;a=commitdiff;h=b92eda2753c564d96a5c52b5fbfc5eca1ab3f8b3 Implemented as an upstart job for assorted technical and non-technical reasons. Red Hat Enterprise Linux 6.0 is now available and should resolve the problem described in this bug report. This report is therefore being closed with a resolution of CURRENTRELEASE. You may reopen this bug report if the solution does not work for you. |