Description of problem: When running a docker container on RHEL 6.5, with an open-ssh server I'm unable to login into the container. To be precise, the authentication via password or public key works, the session is closed after successful authentication. This seems to be a problem affecting RHEL 6.5 and related distribution (tried CentOS 6.4 and 6.5) External Bug -- Docker https://github.com/dotcloud/docker/issues/5032 The very same container has no issues running on an Ubuntu host (tested 12.04, 13.04 & 13.10) Version-Release number of selected component (if applicable): RHEL 6.5 CentOS 6.4/6.5 docker 0.9.0-3 Also tested 0.9.1-1 with RHEL 6.5 How reproducible: 100% in RHEL 6.5, CentOS 6.4 + 6.5 Steps to Reproduce: 1. install docker 0.9.0-3 or 0.9.1-1 (from epel 6) 2. build new sshd docker container (http://docs.docker.io/en/latest/examples/running_ssh_service/) - it will fail if using a fedora or arch linux container as well. 3. run sshd container in detached mode or interactive mode (docker run -d -p 2222:22 sshd) Actual results: ssh into container ssh -vvvv root@host01 -p 2222 --------------------------------------------------- OpenSSH_4.6p1, OpenSSL 0.9.8e 23 Feb 2007 debug1: Reading configuration data /c/Users/aschulz//.ssh/config debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug2: ssh_connect: needpriv 0 debug1: Connecting to host01[10.242.182.84] port 2222. debug1: Connection established. debug1: identity file /c/Users/aschulz/.ssh/identity type -1 debug3: Not a RSA1 key file /c/Users/aschulz/.ssh/id_rsa. debug2: key_type_from_name: unknown key type '-----BEGIN' debug3: key_read: missing keytype debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug3: key_read: missing whitespace debug2: key_type_from_name: unknown key type '-----END' debug3: key_read: missing keytype debug1: identity file /c/Users/aschulz/.ssh/id_rsa type 1 debug1: identity file /c/Users/aschulz/.ssh/id_dsa type -1 debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3 debug1: match: OpenSSH_5.3 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_4.6 debug2: fd 3 setting O_NONBLOCK debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa,ssh-dss debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc.se,aes128-ctr,aes192-ctr,aes256-ctr debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc.se,aes128-ctr,aes192-ctr,aes256-ctr debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: none,zlib,zlib debug2: kex_parse_kexinit: none,zlib,zlib debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc.se debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc.se debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: none,zlib debug2: kex_parse_kexinit: none,zlib debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: mac_init: found hmac-md5 debug1: kex: server->client aes128-cbc hmac-md5 none debug2: mac_init: found hmac-md5 debug1: kex: client->server aes128-cbc hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug2: dh_gen_key: priv key bits set: 140/256 debug2: bits set: 511/1024 debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug3: put_host_port: [10.242.182.84]:2222 debug3: put_host_port: [ohiswappdev01]:2222 debug3: check_host_in_hostfile: filename /c/Users/aschulz/.ssh/known_hosts debug3: check_host_in_hostfile: match line 1 debug3: check_host_in_hostfile: filename /c/Users/aschulz/.ssh/known_hosts debug3: check_host_in_hostfile: match line 1 debug1: Host '[ohiswappdev01]:2222' is known and matches the RSA host key. debug1: Found key in /c/Users/aschulz/.ssh/known_hosts:1 debug2: bits set: 524/1024 debug1: ssh_rsa_verify: signature correct debug2: kex_derive_keys debug2: set_newkeys: mode 1 debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug2: set_newkeys: mode 0 debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_REQUEST sent debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug2: key: /c/Users/aschulz/.ssh/id_rsa (0xa01cc88) debug2: key: /c/Users/aschulz/.ssh/identity (0x0) debug2: key: /c/Users/aschulz/.ssh/id_dsa (0x0) debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password debug3: preferred publickey,keyboard-interactive,password debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Offering public key: /c/Users/aschulz/.ssh/id_rsa debug3: send_pubkey_test debug2: we sent a publickey packet, wait for reply debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password debug1: Trying private key: /c/Users/aschulz/.ssh/identity debug3: no such identity: /c/Users/aschulz/.ssh/identity debug1: Trying private key: /c/Users/aschulz/.ssh/id_dsa debug3: no such identity: /c/Users/aschulz/.ssh/id_dsa debug2: we did not send a packet, disable method debug3: authmethod_lookup password debug3: remaining preferred: ,password debug3: authmethod_is_enabled password debug1: Next authentication method: password root@ohiswappdev01's password: debug3: packet_send2: adding 64 (len 55 padlen 9 extra_pad 64) debug2: we sent a password packet, wait for reply debug1: Authentication succeeded (password). debug1: channel 0: new [client-session] debug3: ssh_session2_open: channel_new: 0 debug2: channel 0: send open debug1: Entering interactive session. debug1: channel 0: free: client-session, nchannels 1 debug3: channel 0: status: The following connections are open: #0 client-session (t3 r-1 i0/0 o0/0 fd 4/5 cfd -1) debug3: channel 0: close_fds r 4 w 5 e 6 c -1 Connection to host01closed by remote host. Connection to host01closed. debug1: Transferred: stdin 0, stdout 0, stderr 89 bytes in 0.0 seconds debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 12714.4 debug1: Exit status -1 ------------------------------------- Expected results: A bash shell in the container. Additional info: + I've checked the permission of the host keys + The issue affects all base images (failed test with CentOS, fedora, ubuntu, Arch as base container for sshd) on a RHEL host. + I've disabled PAM (as a similar issue was reported with a CentOS container http://stackoverflow.com/questions/18173889/cannot-access-centos-sshd-on-docker) + I am unable to find any logs while running the container in interactive mode - nothing in /var/log/messages - nothing in /var/log/auth.log - nothing in /var/log/secure - nothing in /var/log/faillog + I see no server side notification of successful authentication I see on the client side while running the container in interactive mode
Running sshd -dddd I've got debug1: SELinux support enabled debug3: ssh_selinux_setup_exec_context: setting execution context ssh_selinux_getctxbyname: Failed to get default SELinux security context for root ssh_selinux_setup_exec_context: security_getenforce() failed debug1: do_cleanup which pointed me to https://groups.google.com/forum/#!msg/docker-user/7EyZthXHcww/B3YAV0XsxNAJ Disabling SELinux worked, setting it to permissive didn't. There's a pull requested that pull request regarding SELinux support in docker that was merged 12 days ago, perhaps will be included in the next rpm release https://github.com/dotcloud/docker/pull/4211.
I've checked and SELinux pull was included in Docker 0.10.0 and I got the RPM (ftp://195.220.108.108/linux/epel/testing/6/x86_64/docker-io-0.10.0-2.el6.x86_64.rpm) for RHEL installed but I still get the same error as Guy debug1: SELinux support enabled debug3: ssh_selinux_setup_pty: setting TTY context on /dev/pts/0 ssh_selinux_getctxbyname: ssh_selinux_getctxbyname: security_getenforce() failed debug1: do_cleanup debug1: PAM: cleanup debug1: PAM: closing session Similar by disabling SELinux everything works like a charm, but setting it to either Enforcing or Permissive and it fails. No logs on either the host or container under /var/log/messages for SELinux errors
(In reply to aschulz from comment #3) > Similar by disabling SELinux everything works like a charm, but setting it > to either Enforcing or Permissive and it fails. Copying dwalsh.
We've updated the images (yum update) and now SSH works with SELinux enabled (tested with both Ubuntu and Fedora containers) tested on RHEL 6.5 host. No idea as to what caused the initial issue.
One final note, The issue seems to be still present for Ubuntu 12.04 containers but 12.10, 13.04, 13.10 and 14.04 containers work without a hitch.
http://people.redhat.com/dwalsh/SELinux/RHEL6/ contains an update libselinux which should be added to rhel6.5 image, which should allow stuff like useradd/groupadd to work.
When we ship an image it will have the fixed libselinux. Until then use the library I provided.