Bug 1478707

Summary: kdump over ssh to remote target server with system role
Product: Red Hat Enterprise Linux 7 Reporter: Terry Bowling <tbowling>
Component: rhel-system-rolesAssignee: Baoquan He <bhe>
Status: CLOSED ERRATA QA Contact: Leos Pol <lpol>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 7.4CC: bhe, dyoung, ovasik, pcahyna, salmy, tbowling
Target Milestone: rcKeywords: Extras, FutureFeature
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rhel-system-roles-0.5-3.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-01-25 12:55:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1489746    
Bug Blocks: 1477926    
Attachments:
Description Flags
Fix the wonr conditional for ssh checking. none

Description Terry Bowling 2017-08-06 11:28:16 UTC
Description of problem:

Verify and test that configuration of kdump over ssh works correctly according to customer provided requirements.

1. Supports RHEL 6, & 7  (5 as a stretch goal but not required)
2. Set crashkernel=128M in grub.conf  (need to verify this with best practices, possibly rely on current defaults or auto)
3. Set target to SSH
4. “core_collector makedumpfile -F -c -d 31”
5. Need to populate target servers ssh key into  ~/.ssh/known_hosts
6. Need to populate kdump_id_rsa key into remote target server authorized_keys file (currently using ansible copy module)  Could probably use the Ansible SSH authorized_key module http://docs.ansible.com/ansible/latest/authorized_key_module.html
7. Restart kdump

Comment 1 Baoquan He 2017-08-07 13:14:09 UTC
Hi Pavel,

Please don't assign this bug to me. As I discussed with Terry, I mentioned that I don't know how to copy ssh authorized file on a ansible server from system with kdump enabled to a dump target machine. Means, I Know there's ssh methods to copy ssh authorized file on A from A to B, but I don't know how to copy it on A from B to C. I ever asked other people, no one knows it. And I also raised this issue to Terry, he said he will ask people to hanlde it. I can tell what need to be done for this on kdump, better find a person who knows this.

Thanks
Baoquan

Comment 3 Baoquan He 2017-08-08 04:08:12 UTC
Think about it a while, if customer can copy the ssh authorized public key file to dump target server, and keep the private key file on dumping machine, ansible code will be simple to add to kdump module. While this may need all dumping machine to own the same private ssh key file. It may not be good. The ssh key file copying is a problem I ever tried at the first version of kdump module.

Since kdump provides "kdumpctl propagate" to copy ssh key file to dumping target server, while it will prompt with a ssh password input while it's not seen under ansible task running, it's not available any more.

Comment 4 Baoquan He 2017-09-08 09:28:21 UTC
Created a bz 1489746 to ask ansible component to implement a way to provide password between two remote ansible clients for ssh key file copying.

If they can fix 1489746, or suggest a way to satisfy the requirement, this bug can be fixed easily.

Comment 5 Terry Bowling 2017-11-01 12:02:14 UTC
from email thread:

On 10/15/17 at 10:04pm, James Tanner wrote:
> Vault does not affect the transport or remote system security. It's job is
> to encrypt/decrypt secrets on the ansible controller only. Transport
> security is ssh.
>
> #1 The expect module can drive generic commands that require stdin
> http://docs.ansible.com/ansible/latest/expect_module.html. For those with a
> QA background, tcl/expect is rarely perfect and is quite often fragile.
>
> #2 There's a hundred different ways to create and propagate ssh keys, so I
> don't feel as if trying to adhere to kdumpctl's obtuse interface should be
> a requirement here. Copying the key directly from the kdump-client to the
> kdump-dump-target is also not a requirement....
>
> 1. Use shell or authorized keys module to generate the keys on the
> kdump-client
> 2. Use the fetch module to copy the key back to the ansible controller
> 3. Use the copy or authorized keys module to send the controller's copy to
> the dump-target

I personnally think this is the closest way to the original 'kdumpctl propagate'.
Thanks for your great idea, James.

Other people, any comment?

Thanks
Baoquan

Comment 6 Terry Bowling 2017-11-01 12:04:12 UTC
from email thread:

> Thanks everyone.
> I emailed two of our customers who manage large environments and they both
> confirmed that they simply manage a single key-pair for their kdump user
> account from their deployment server.  They have the kdump user locked down
> as no login, no home dir.  They use this same kdump UID/GID and key-pair on
> every client node to dump to the target server.
>
> If we generate a new key-pair on the remote node, I suggest that we ensure
> to created as "USER-kdump.rsa", if the username does not already contain
> the kdump term.  That way if they are reusing another user account, or
> root, we are not distributing a potentially more important key. We I'll ask
> our security team to weigh in on the BZ for any guidance they might have in
> how we are handling keys.

Thanks for telling this, Terry.

We provide an option in /etc/kdump.conf:

#       sshkey /root/.ssh/kdump_id_rsa

Default to use /root/.ssh/kdump_id_rsa, user can specify their favorite
name, like 'USER-kdump.rsa'.

Comment 7 Terry Bowling 2017-11-01 12:10:24 UTC
Baoquan,

Do you have what you need to move forward?  Can we close out https://bugzilla.redhat.com/show_bug.cgi?id=1489746 as no longer needed?

Comment 8 Baoquan He 2017-11-01 13:21:48 UTC
(In reply to Terry Bowling from comment #7)
> Baoquan,
> 
> Do you have what you need to move forward?  Can we close out
> https://bugzilla.redhat.com/show_bug.cgi?id=1489746 as no longer needed?

Yes, we can close it since it's not needed. 

Have closed it as NOTABUG.

By the way, I have made a pull request for the fix of this bug:
https://github.com/linux-system-roles/kdump/pull/9

Hope people can help review.

Comment 10 Leos Pol 2017-11-21 12:43:10 UTC
By default, dump_target.kind is not defined, but following conditional in tasks/main.yml is evaluated as true:

when: dump_target.kind == "ssh"

... which causes including tasks/ssh.yml and exception due to undefined other variable - ssh_dump_server.

Proposed fix:
- include_tasks: ssh.yml
  when: 
    - dump_target
    - dump_target.kind == "ssh"


Similar issue found in templates/kdump.conf.j2. Proposed fix:

{% if dump_target and dump_target.kind == "ssh" and sshkey != '/root/.ssh/kdump_id_rsa' %}
sshkey {{ sshkey }}
{% endif %}


Other possible solution is to change dump_target in defaults/main.yml.

Comment 11 Baoquan He 2017-11-29 05:58:10 UTC
(In reply to Leos Pol from comment #10)
> By default, dump_target.kind is not defined, but following conditional in
> tasks/main.yml is evaluated as true:
> 
> when: dump_target.kind == "ssh"
> 
> ... which causes including tasks/ssh.yml and exception due to undefined
> other variable - ssh_dump_server.
> 
> Proposed fix:
> - include_tasks: ssh.yml
>   when: 
>     - dump_target
>     - dump_target.kind == "ssh"
> 
> 
> Similar issue found in templates/kdump.conf.j2. Proposed fix:
> 
> {% if dump_target and dump_target.kind == "ssh" and sshkey !=
> '/root/.ssh/kdump_id_rsa' %}
> sshkey {{ sshkey }}
> {% endif %}
> 
> 
> Other possible solution is to change dump_target in defaults/main.yml.

Thanks, Leos. I changed as you suggested and test passed.

Hi Pavel,

I will attach the patch in the BZ since the github code is not synced with latest code.

Thanks
Baoquan

Comment 12 Baoquan He 2017-11-29 05:59:23 UTC
Created attachment 1360165 [details]
Fix the wonr conditional for ssh checking.

Comment 13 Pavel Cahyna 2017-12-19 20:05:05 UTC
Thank you, I committed your fix to my repo and sent you a PR, together with some test improvements (removal of hardcoded values). I made a PR in the upstream project as well (#10), CI now works with the test.

Comment 14 Leos Pol 2018-01-15 13:22:39 UTC
rhel-system-roles-0.5-3.el7 works for me

Comment 17 errata-xmlrpc 2018-01-25 12:55:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0189