Bug 1085081

Summary:	Docker sshd closing connection RHEL 6.5
Product:	[Fedora] Fedora EPEL	Reporter:	aschulz <arnoschulz>
Component:	docker-io	Assignee:	Lokesh Mandvekar <lsm5>
Status:	CLOSED NEXTRELEASE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	el6	CC:	acathrow, admiller, arnoschulz, dwalsh, golang-updates, jkeck, lsm5, mattdm, mgoldman, skottler, vbatts
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-06-03 20:31:34 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description aschulz 2014-04-07 18:42:59 UTC

Description of problem:

When running a docker container on RHEL 6.5, with an open-ssh server I'm unable to login into the container.

To be precise, the authentication via password or public key works, the session is closed after successful authentication.

This seems to be a problem affecting RHEL 6.5 and related distribution (tried CentOS 6.4 and 6.5)

External Bug -- Docker https://github.com/dotcloud/docker/issues/5032

The very same container has no issues running on an Ubuntu host (tested 12.04, 13.04 & 13.10)

Version-Release number of selected component (if applicable):

RHEL 6.5
CentOS 6.4/6.5

docker 0.9.0-3
Also tested 0.9.1-1 with RHEL 6.5

How reproducible:

100% in RHEL 6.5, CentOS 6.4 + 6.5

Steps to Reproduce:
1. install docker 0.9.0-3 or 0.9.1-1 (from epel 6)
2. build new sshd docker container (http://docs.docker.io/en/latest/examples/running_ssh_service/) - it will fail if using a fedora or arch linux container as well.
3. run sshd container in detached mode or interactive mode (docker run -d -p 2222:22 sshd)

Actual results:

ssh into container ssh -vvvv root@host01 -p 2222
---------------------------------------------------
OpenSSH_4.6p1, OpenSSL 0.9.8e 23 Feb 2007
debug1: Reading configuration data /c/Users/aschulz//.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to host01[10.242.182.84] port 2222.
debug1: Connection established.
debug1: identity file /c/Users/aschulz/.ssh/identity type -1
debug3: Not a RSA1 key file /c/Users/aschulz/.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /c/Users/aschulz/.ssh/id_rsa type 1
debug1: identity file /c/Users/aschulz/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.6
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib,zlib
debug2: kex_parse_kexinit: none,zlib,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_init: found hmac-md5
debug1: kex: server->client aes128-cbc hmac-md5 none
debug2: mac_init: found hmac-md5
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 140/256
debug2: bits set: 511/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: put_host_port: [10.242.182.84]:2222
debug3: put_host_port: [ohiswappdev01]:2222
debug3: check_host_in_hostfile: filename /c/Users/aschulz/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug3: check_host_in_hostfile: filename /c/Users/aschulz/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug1: Host '[ohiswappdev01]:2222' is known and matches the RSA host key.
debug1: Found key in /c/Users/aschulz/.ssh/known_hosts:1
debug2: bits set: 524/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /c/Users/aschulz/.ssh/id_rsa (0xa01cc88)
debug2: key: /c/Users/aschulz/.ssh/identity (0x0)
debug2: key: /c/Users/aschulz/.ssh/id_dsa (0x0)
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /c/Users/aschulz/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Trying private key: /c/Users/aschulz/.ssh/identity
debug3: no such identity: /c/Users/aschulz/.ssh/identity
debug1: Trying private key: /c/Users/aschulz/.ssh/id_dsa
debug3: no such identity: /c/Users/aschulz/.ssh/id_dsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
root@ohiswappdev01's password:
debug3: packet_send2: adding 64 (len 55 padlen 9 extra_pad 64)
debug2: we sent a password packet, wait for reply
debug1: Authentication succeeded (password).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Entering interactive session.
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 client-session (t3 r-1 i0/0 o0/0 fd 4/5 cfd -1)

debug3: channel 0: close_fds r 4 w 5 e 6 c -1
Connection to host01closed by remote host.
Connection to host01closed.
debug1: Transferred: stdin 0, stdout 0, stderr 89 bytes in 0.0 seconds
debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 12714.4
debug1: Exit status -1
-------------------------------------

Expected results:

A bash shell in the container.

Additional info:

+ I've checked the permission of the host keys
+ The issue affects all base images (failed test with CentOS, fedora, ubuntu, Arch as base container for sshd) on a RHEL host.
+ I've disabled PAM (as a similar issue was reported with a CentOS container http://stackoverflow.com/questions/18173889/cannot-access-centos-sshd-on-docker)
+ I am unable to find any logs while running the container in interactive mode
  - nothing in /var/log/messages
  - nothing in /var/log/auth.log 
  - nothing in /var/log/secure
  - nothing in /var/log/faillog
+ I see no server side notification of successful authentication I see on the client side while running the container in interactive mode

Comment 2 aschulz 2014-04-08 17:45:25 UTC

Running sshd -dddd

I've got 

debug1: SELinux support enabled
debug3: ssh_selinux_setup_exec_context: setting execution context
ssh_selinux_getctxbyname: Failed to get default SELinux security context for root
ssh_selinux_setup_exec_context: security_getenforce() failed
debug1: do_cleanup

which pointed me to https://groups.google.com/forum/#!msg/docker-user/7EyZthXHcww/B3YAV0XsxNAJ

Disabling SELinux worked, setting it to permissive didn't.

There's a pull requested that pull request regarding SELinux support in docker that was merged 12 days ago, perhaps will be included in the next rpm release https://github.com/dotcloud/docker/pull/4211.

Comment 3 aschulz 2014-04-16 13:09:23 UTC

I've checked and SELinux pull was included in Docker 0.10.0 and I got the RPM (ftp://195.220.108.108/linux/epel/testing/6/x86_64/docker-io-0.10.0-2.el6.x86_64.rpm) for RHEL installed but I still get the same error as Guy

debug1: SELinux support enabled
debug3: ssh_selinux_setup_pty: setting TTY context on /dev/pts/0
ssh_selinux_getctxbyname: ssh_selinux_getctxbyname: security_getenforce() failed
debug1: do_cleanup
debug1: PAM: cleanup
debug1: PAM: closing session

Similar by disabling SELinux everything works like a charm, but setting it to either Enforcing or Permissive and it fails.

No logs on either the host or container under /var/log/messages for SELinux errors

Comment 4 Lokesh Mandvekar 2014-04-16 14:29:27 UTC

(In reply to aschulz from comment #3)

> Similar by disabling SELinux everything works like a charm, but setting it
> to either Enforcing or Permissive and it fails.

Copying dwalsh.

Comment 5 aschulz 2014-05-01 15:41:35 UTC

We've updated the images (yum update) and now SSH works with SELinux enabled (tested with both Ubuntu and Fedora containers) tested on RHEL 6.5 host.

No idea as to what caused the initial issue.

Comment 6 aschulz 2014-05-01 17:20:30 UTC

One final note,

The issue seems to be still present for Ubuntu 12.04 containers but 12.10, 13.04, 13.10 and 14.04 containers work without a hitch.

Comment 7 Daniel Walsh 2014-05-28 16:48:20 UTC

http://people.redhat.com/dwalsh/SELinux/RHEL6/ contains an update libselinux which should be added to rhel6.5 image, which should allow stuff like useradd/groupadd to work.

Comment 8 Daniel Walsh 2014-06-03 20:31:34 UTC

When we ship an image it will have the fixed libselinux. Until then use the library I provided.