Bug 510816 - kdump fails to send vmcore via ssh/scp when configured using short hostname
kdump fails to send vmcore via ssh/scp when configured using short hostname
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kexec-tools (Show other bugs)
5.4
All Linux
high Severity medium
: rc
: 5.5
Assigned To: Neil Horman
Red Hat Kernel QE team
: Regression
Depends On:
Blocks: 499522
  Show dependency treegraph
 
Reported: 2009-07-10 18:36 EDT by Dave Maley
Modified: 2012-05-28 07:29 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
A previous enhancement to kdump ensured that kdump would perform reliably even when the DNS on a network changed. However, this required that host names specified in the kdump.conf script needed to be entered as fully- qualified domain names. Therefore, if a hostname was entered in a short form, kdump would fail to dump to a ssh/scp target. The documentation for kdump has been revised to make it clear that hostnames must always be specified as FQDNs. When users specify hostnames in this format, dumps to ssh/scp targets perform as expected.
Story Points: ---
Clone Of:
: 600579 (view as bug list)
Environment:
Last Closed: 2010-03-30 03:47:13 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
kdump config (1.12 KB, application/octet-stream)
2009-07-13 13:13 EDT, Dave Maley
no flags Details
log of failed attempt (49.64 KB, application/octet-stream)
2009-07-13 13:15 EDT, Dave Maley
no flags Details
log of successful attempt (43.21 KB, application/octet-stream)
2009-07-13 13:17 EDT, Dave Maley
no flags Details
resolv.conf from lab (114 bytes, application/octet-stream)
2009-07-13 16:17 EDT, Dave Maley
no flags Details
patch to ensure that we always use an fqdn (795 bytes, patch)
2009-07-14 09:35 EDT, Neil Horman
no flags Details | Diff
[updated] patch to ensure that we always use an fqdn (795 bytes, patch)
2009-07-14 15:41 EDT, Dave Maley
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2010:0179 normal SHIPPED_LIVE kexec-tools bug fix update 2010-03-29 08:18:38 EDT

  None (edit)
Description Dave Maley 2009-07-10 18:36:39 EDT
Description of problem:
when kdump is configured to dump to a ssh/scp target using short hostname the vmcore is written to the local disk and nothing makes it to the remote system.  rolling back to kexec-tools-1.102pre-56.el5 resolves things.

*regression from 5.3 thus escalating to BZ immediately


Version-Release number of selected component (if applicable):
kexec-tools-1.102pre-74.el5


How reproducible:
100%


Steps to Reproduce:
1. configure kdump for scp/ssh to server using short hostname
2. propagate and restart service
3. crash system

  
Actual results:
vmcore captured on localhost


Expected results:
vmcore captured on server defined in kdump.conf


Additional info:
-reproduced in RDU GSS lab
  -> client: hp-dl385g5-1.gsslab.rdu.redhat.com
  -> server: dl385g2.gsslab.rdu.redhat.com
- scp using short hostname does work
Comment 1 Neil Horman 2009-07-10 21:14:19 EDT
I'm out all next week, so this is way too late to get into 5.4.  Proposing for 5.5.

when you say 'short' hostname, I assume you're referring to just the hostname, not the fqdn?  Its not recommended that you use a non-fqdn in your configuration for kdump (all the examples use an fqdn).  If you want to get me your kdump.conf and the serial console log of a kdump session, I might be able to make this a bit more robust, but there are a number of factors in resolving a non-fqdn for kdump, and so this might be a cantfix/wontfix.
Comment 3 Dave Maley 2009-07-13 13:12:10 EDT
> when you say 'short' hostname, I assume you're referring to just the hostname,
> not the fqdn?

correct

> Its not recommended that you use a non-fqdn in your configuration for kdump 
> (all the examples use an fqdn).

Do note that using short hostnames worked in 5.3.  On the same system, w/ same config, I can simply rollback to kexec-tools-1.102pre-56.el5 and it works.  Maybe that was by chance (ie. not expected) however this has been reported to us as being a regression.
Comment 4 Dave Maley 2009-07-13 13:13:58 EDT
Created attachment 351496 [details]
kdump config

config used for lab testing
Comment 5 Dave Maley 2009-07-13 13:15:50 EDT
Created attachment 351497 [details]
log of failed attempt

serial console log of failed attempt (kexec-tools-1.102pre-74.el5)
Comment 6 Dave Maley 2009-07-13 13:17:11 EDT
Created attachment 351498 [details]
log of successful attempt

serial console log of successful attempt (kexec-tools-1.102pre-56.el5)
Comment 7 Dave Maley 2009-07-13 13:19:10 EDT
There's nothing obvious in the log from the failed attempt, it just says:

Saving to remote location kdump-test@dl385g2
lost connection

I'm going to start reviewing the changes between pre-56 and pre-74 to see if I can find any clues ....
Comment 8 Dave Maley 2009-07-13 14:37:38 EDT
I've determined this regression was introduced w/ the mkdumprd change from bug 493690.  Reverting the changes that were introduced in that bug resolves this regression.

Interesting to note that according to:
https://bugzilla.redhat.com/show_bug.cgi?id=493690#c12

..this scenario (short hostname) was tested and reported as working (see case #5).  Tho I suppose it's possible the engineer meant username@hostname.domain rather than username@hostname.
Comment 11 Neil Horman 2009-07-13 16:10:27 EDT
Can you please attach the /etc/resolve.conf file for this system thats failing?  Thanks!
Comment 13 Dave Maley 2009-07-13 16:17:45 EDT
Created attachment 351514 [details]
resolv.conf from lab

resolv.conf from lab system used during my reproduction efforts
Comment 15 Neil Horman 2009-07-13 16:23:58 EDT
does it work if you add this line to /etc/resolv.conf:
domain gsslab.rdu.redhat.com

Note that you'll have to touch /etc/kdump.conf and restart the kdump services.
Comment 17 Dave Maley 2009-07-13 17:19:52 EDT
adding the suggested line from comment 15 made no difference.
Comment 18 Neil Horman 2009-07-14 09:35:27 EDT
Created attachment 351584 [details]
patch to ensure that we always use an fqdn

think I found the problem.  In 5.3 we had a request to enhance kdump so that dns changes didn't require kdump restarts (we had previously resolved dns entries statically and just put the ip address in the initramfs.  We fixed this by simply putting the host name in, and the resolution setup in the kdump kernel is preventing simple hostname resolution in our environment. I could enhance the resolve setup so that it would work in our environment, but that doesn't solve the general problem if someones dns requires fqdn's to resolve an address. I think the best thing to do is twofold:
1) Transform non-fqdn's into fqdn's in the mkdumprd script
2) Add documentation to tell people to always use fqdns.

This patch does (1).  If you could test it out please and confirm dave, I'd appreciate it.  Thanks!
Comment 19 Dave Maley 2009-07-14 15:41:54 EDT
Created attachment 351661 [details]
[updated] patch to ensure that we always use an fqdn

Yep this works in my test setup.  I'll get a test pkg to the TAM and request that we get confirmation from the partner as well.

I had to fix a couple typos in the patch and so I've uploaded the corrected version.
Comment 22 Dave Maley 2009-07-20 09:49:21 EDT
partner has verified the proposed fix resolves this regression for them
Comment 25 Dave Maley 2009-08-07 12:51:33 EDT
The document "I cannot get a vmcore sent to a remote host via scp when kdump is configured using a short hostname" has been published in Knowledgebase English space. The article is now visible to the public at the following URL:

http://kbase.redhat.com/faq/docs/DOC-17880
Comment 26 Issue Tracker 2009-09-04 05:59:51 EDT
Event posted on 09-04-2009 06:59pm JST by mfuruta@redhat.com

Hi Samukawa-san,

----
Since RHEL5.4 is released, customers have to change their configuration
to specify the server by IP address or FQDN.  Even if we change this
behavior in RHEL5.5, it will leave a different limitation in how the
hostname could be specified.  The knowledgebase must be changed again
in RHEL5.5 and it will confuse the customers.

Hence, NEC would like to withdraw the original request to enable using
short hostnames, and will settle with the current knowledgebase article.
----

Thanks for comments, I got your situation. 
Let me confirm that you'd NOT like to change this behavior and want
KnowledgeBase Article only, right?

Thanks in advance.

Regards,

Masaki Furuta



Internal Status set to 'Waiting on Customer'
Status set to: Waiting on Client
Ticket type set to: 'Problem'

This event sent from IssueTracker by mfuruta@redhat.com 
 issue 315411
Comment 28 Dave Maley 2009-09-09 14:25:07 EDT
Neil - Since NEC has stated that they prefer to move forward w/ the kbase only, and to my knowledge we haven't received any additional reports, I'd suggest we:

1. update the kdump howto to state that fqdn is required
2. update the documentation in kdump.conf to state fqdn is required
3. update the kbase to remove the statement about the possibility of a fix

If you agree I'll go ahead and propose the changes to the howto and config files, and will get the kbase updated.  I'll also clear the 5.4.z request as there's no reason these changes can't wait for 5.5.
Comment 29 Dave Maley 2009-09-29 17:39:03 EDT
* removing 5.4.z request based on comment 27

Neil - lmk how you like to proceed wrt comment 28
Comment 30 Neil Horman 2009-10-21 15:56:33 EDT
yeah, that sounds like a fine plan.  Thanks Dave!
Comment 31 Neil Horman 2009-10-23 13:24:21 EDT
updated docs in the pkg in -84.el5.  Thanks!
Comment 32 Dave Maley 2009-10-26 15:45:39 EDT
Neil - I've updated the kbase so we should be all set here.  Thanks!
Comment 38 Ruediger Landmann 2010-03-19 01:00:41 EDT
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.

New Contents:
A previous enhancement to kdump ensured that kdump would perform reliably
even when the DNS on a network changed. However, this required that host
names specified in the kdump.conf script needed to be entered as fully-
qualified domain names. Therefore, if a hostname was entered in a short form, 
kdump would fail to dump to a ssh/scp target. The documentation for kdump
has been revised to make it clear that hostnames must always be specified as
FQDNs. When users specify hostnames in this format, dumps to ssh/scp targets
perform as expected.
Comment 39 errata-xmlrpc 2010-03-30 03:47:13 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2010-0179.html

Note You need to log in before you can comment on or make changes to this bug.