Bug 1448062 - Unable to log in via SSH to compute nodes with the heat-admin user
Summary: Unable to log in via SSH to compute nodes with the heat-admin user
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: async
: 10.0 (Newton)
Assignee: Angus Thomas
QA Contact: Amit Ugol
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-05-04 13:04 UTC by Marius Cornea
Modified: 2017-05-17 12:26 UTC (History)
10 users (show)

Fixed In Version: openstack-tripleo-heat-templates-5.2.0-15.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-05-17 12:26:17 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:1242 0 normal SHIPPED_LIVE Important: Red Hat OpenStack Platform director security update 2017-05-17 16:19:05 UTC

Description Marius Cornea 2017-05-04 13:04:13 UTC
Description of problem:
Unable to log in  via SSH to compute nodes with the heat-admin user

Version-Release number of selected component (if applicable):
OSP10 latest
openstack-tripleo-heat-templates-5.2.0-13.el7ost.noarch
puppet-tripleo-5.5.0-11.el7ost.noarch
puppet-nova-9.5.0-4.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP10 latest
2. SSH to one of the compute nodes
[stack@undercloud-0 ~]$ ssh heat-admin@192.168.0.18
Connection closed by 192.168.0.18


Actual results:
[stack@undercloud-0 ~]$ ssh heat-admin@192.168.0.18 -v
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug1: Connecting to 192.168.0.18 [192.168.0.18] port 22.
debug1: Connection established.
debug1: identity file /home/stack/.ssh/id_rsa type 1
debug1: identity file /home/stack/.ssh/id_rsa-cert type -1
debug1: identity file /home/stack/.ssh/id_dsa type -1
debug1: identity file /home/stack/.ssh/id_dsa-cert type -1
debug1: identity file /home/stack/.ssh/id_ecdsa type -1
debug1: identity file /home/stack/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/stack/.ssh/id_ed25519 type -1
debug1: identity file /home/stack/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA ca:7f:c4:cb:b9:ea:de:2b:ea:e6:11:9a:d3:9b:d1:10
debug1: Host '192.168.0.18' is known and matches the ECDSA host key.
debug1: Found key in /home/stack/.ssh/known_hosts:2
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
Connection closed by 192.168.0.18
[stack@undercloud-0 ~]$ 

Expected results:
SSH connection is successfuly established.

Additional info:

Comment 1 Marian Krcmarik 2017-05-04 13:52:07 UTC
It seems that a typo was introduced in sshd_config on overcloud compute nodes while trying to configure ssh for live migration. It's not possible to ssh into compute nodes and sshd on computes complains about the bad line 21:
"Match LocalAddress 192.168.24.16, User nova_migration"
I believe that the comma is not supposed to be there and once I remove the comma, ssh starts to work as expected.
There is possibly one more time a problem with comma at line:
Match LocalAddress !192.168.24.16,!

For sure sshd complains about the line 21 of the sshd_config where the comma should not be.
If we have there more local migration addresses then they are supposed to be separated by comma but It's not the case of our setups.

Example of created sshd_config by puppet:
# File is managed by Puppet

AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
AuthorizedKeysFile .ssh/authorized_keys
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials no
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
PasswordAuthentication no
PrintMotd no
Subsystem sftp /usr/libexec/openssh/sftp-server
SyslogFacility AUTHPRIV
UsePAM yes
UsePrivilegeSeparation sandbox
X11Forwarding yes
Match LocalAddress 192.168.24.16, User nova_migration
    AllowTcpForwarding no
    AuthorizedKeysFile /etc/nova/migration/authorized_keys
    ForceCommand /bin/nova-migration-wrapper
    PasswordAuthentication no
    X11Forwarding no
Match LocalAddress !192.168.24.16,!
    DenyUsers nova_migration

Comment 2 Marian Krcmarik 2017-05-04 14:18:10 UTC
The related reviews seem to be:
https://review.openstack.org/#/c/458082/
https://review.openstack.org/#/c/458077/

It seems that the problem is that live_migration_ssh_inbound_addr is empty and thus tripleo::profile::base::nova::migration_ssh_localaddrs is something like:
['192.168.24.16', '']

From my /etc/puppet/hieradata/service_configs.yaml:
.
.
.
cold_migration_ssh_inbound_addr: 192.168.24.16
kernel_modules: {
  "nf_conntrack": {}
}
live_migration_ssh_inbound_addr:
.
.
.
tripleo::profile::base::nova::migration_ssh_localaddrs: [
  "%{hiera('cold_migration_ssh_inbound_addr')}",
  "%{hiera('live_migration_ssh_inbound_addr')}"
]
.
.

Comment 5 errata-xmlrpc 2017-05-17 12:26:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:1242


Note You need to log in before you can comment on or make changes to this bug.