Bug 844025

Assigned.

I have not been able to reproduce this bug, and the problem is described by the two reports as intermittent. I have reached out to them for more info. I believe the severity can be reduced due to the small number of people affected, the intermittent nature of the problem, and the available workaround (manually setting up ssh port forwarding)

Lowering severity because of inability to reproduce.

Hi there.
Just wanted to add my input on this as I am seeing this issue consistently.

I've tested with the same (scalable) app from both windows (installed client tools yesterday via ruby - not cygwin) and FC17 (I ran 'gem update rhc' today).

I can ssh into my app no problem. Here is the output with -v:

OpenSSH_5.9p1, OpenSSL 1.0.0j-fips 10 May 2012
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 50: Applying options for *
debug1: Connecting to lab-iksystems.rhcloud.com [23.22.23.80] port 22.
debug1: Connection established.
debug1: identity file /home/jon/.ssh/id_rsa type 1
debug1: identity file /home/jon/.ssh/id_rsa-cert type -1
debug1: identity file /home/jon/.ssh/id_dsa type -1
debug1: identity file /home/jon/.ssh/id_dsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.9
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA cf:ee:77:cb:0e:fc:02:d7:72:7e:ae:80:c0:90:88:a7
debug1: Host 'lab-iksystems.rhcloud.com' is known and matches the RSA host key.
debug1: Found key in /home/jon/.ssh/known_hosts:11
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_1000' not found

debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_1000' not found

debug1: Unspecified GSS failure.  Minor code may provide more information


debug1: Unspecified GSS failure.  Minor code may provide more information
Credentials cache file '/tmp/krb5cc_1000' not found

debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/jon/.ssh/id_rsa
debug1: Remote: Forced command: /usr/bin/trap-user
debug1: Remote: X11 forwarding disabled.
debug1: Server accepts key: pkalg ssh-rsa blen 151
debug1: Remote: Forced command: /usr/bin/trap-user
debug1: Remote: X11 forwarding disabled.
debug1: Authentication succeeded (publickey).
Authenticated to lab-iksystems.rhcloud.com ([23.22.23.80]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env XMODIFIERS = @im=none
debug1: Sending env LANG = en_US.utf8

    *********************************************************************

    You are accessing a service that is for use only by authorized users.  
    If you do not have authorization, discontinue use at once. 
    Any use of the services is subject to the applicable terms of the 
    agreement which can be found at: 
    https://openshift.redhat.com/app/legal

    *********************************************************************

    Welcome to OpenShift shell

    This shell will assist you in managing OpenShift applications.

    !!! IMPORTANT !!! IMPORTANT !!! IMPORTANT !!!
    Shell access is quite powerful and it is possible for you to
    accidentally damage your application.  Proceed with care!
    If worse comes to worst, destroy your application with 'rhc app destroy'
    and recreate it
    !!! IMPORTANT !!! IMPORTANT !!! IMPORTANT !!!

    Type "help" for more info.

But if I try port forwarding:

rhc-port-forward -a lab -l server.admin
Password: **********

Checking available ports...
/usr/local/share/gems/gems/net-ssh-2.5.2/lib/net/ssh.rb:200:in `start': 87fe67d3eb964a089b05db723de7d339 (Net::SSH::AuthenticationFailed)
	from /usr/local/share/gems/gems/rhc-0.96.9/bin/rhc-port-forward:155:in `block (2 levels) in <top (required)>'
	from /usr/local/share/gems/gems/rhc-0.96.9/bin/rhc-port-forward:151:in `each'
	from /usr/local/share/gems/gems/rhc-0.96.9/bin/rhc-port-forward:151:in `block in <top (required)>'
	from /usr/local/share/gems/gems/net-ssh-2.5.2/lib/net/ssh.rb:193:in `start'
	from /usr/local/share/gems/gems/rhc-0.96.9/bin/rhc-port-forward:110:in `<top (required)>'
	from /usr/local/bin/rhc-port-forward:23:in `load'
	from /usr/local/bin/rhc-port-forward:23:in `<main>'

Output of rhc-chk:
Password: **********

Loaded suite /usr/local/bin/rhc-chk
Started
....F
===============================================================================
Failure:
test_02_ssh_agent(Test3_SSH)
/usr/local/share/gems/gems/rhc-0.96.9/bin/rhc-chk:393:in `test_02_ssh_agent'
     390:   def test_02_ssh_agent
     391:     require_agent_keys
     392: 
  => 393:     assert agent_key_names.include?(File.expand_path(@libra_kfile)) ,error_for(:pubkey_not_loaded, ": #{@libra_kpfile}")
     394:   end
     395: 
     396:   def test_03_remote_ssh_keys
<> expected but was
<>

diff:
  nil
===============================================================================
..

Finished in 9.04025881 seconds.

7 tests, 12 assertions, 1 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
85.7143% passed

0.77 tests/s, 1.33 assertions/s



Hope this helps with the debugging! Shout if you need more info - I'll watch this thread...

Jon

This bug is produced on INT but not reproduced with devenv_2010.

Please refer to the details as below:

Build:
int.openshift.redhat.com 
rhc.0.97.11.gem

Steps:
1. Create an scalable jbossas app on INT.
#rhc app create -a jbosap -t jossas-7 -p xx -s
2. Run rhc port-forward
#rhc port-forward -a jbosap

Actual results:
Error displayed as below:
# rhc port-forward  -a jbosap -p redhat
Checking available ports...
/usr/local/share/gems/gems/net-ssh-2.5.2/lib/net/ssh.rb:200:in `start': 109a5c7c2904464b8ecd19b8fac185ef (Net::SSH::AuthenticationFailed)
	from /usr/local/share/gems/gems/rhc-0.97.11/bin/rhc-port-forward:155:in `block (2 levels) in <top (required)>'
	from /usr/local/share/gems/gems/rhc-0.97.11/bin/rhc-port-forward:151:in `each'
	from /usr/local/share/gems/gems/rhc-0.97.11/bin/rhc-port-forward:151:in `block in <top (required)>'
	from /usr/local/share/gems/gems/net-ssh-2.5.2/lib/net/ssh.rb:193:in `start'
	from /usr/local/share/gems/gems/rhc-0.97.11/bin/rhc-port-forward:110:in `<top (required)>'
	from /usr/local/bin/rhc-port-forward:23:in `load'
	from /usr/local/bin/rhc-port-forward:23:in `<main>'


Additional info:
1. It works well with non-scalable app.
2. It works well with both scalable and non-scalable app on devenv_2010.

Thanks for the additional info; I've been able to reproduce this. There is definitely a defect in the way that rhc-port-forward works with scaled apps. I have opened a discussion on the OpenShift developer's list because there are a few different ways that this can be resolved. If you are not subscribed to the list, you can sign up here:
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

My discussion thread on the list can be read here:
http://lists.openshift.redhat.com/openshift-archives/dev/2012-August/msg00073.html

*** Bug 835501 has been marked as a duplicate of this bug. ***

This needs to be moved into a user story in the backlog

I have created a user story for this issue. I have also modified the utility. For the time being, when a user attempts to run rhc-port-forward with a scaled app, the utility will exit with a message to the user:


This utility does not currently support scaled applications.
You will need to set up port forwarding manually.

Harrison: Thanks for looking into this and posting the links to the list - I've read through with interest. Modifying the returned message on scaled apps is a good idea - should prevent duplicate bugs and threads.

Are you able to post a link to any documentation on how to configure this manually? I've been looking around, but struggled to find any.

Thanks again, Jon

Jon - here is the workaround for you. I'm keeping this out of the dev list discussion because this is only temporary until this issue can be addressed.

1. SSH into the primary app instance (the one available from the "Want to log into your application?" link in the web UI).

2. Look in ~/haproxy-1.4/conf/gear-registry.db for your gears. They will be of the format:

<UUID>@<IP ADDRESS>:<INTERNAL NAME>;<INTERNAL NAME>-<USER NAMESPACE>.rhcloud.com

Everything after the ';' is collectively the <INSTANCE HOST>

3. To create a new port-forwarding setup with this, run:

ssh -N -L <local port>:<INSTANCE HOST>:<remote port> <UUID>@<INSTANCE HOST>

This bug has been verified and fixed.
Please refer to the details as below:

Verified build:
int.openshift.redhat.com
rhc-0.97.15.gem

Verified steps:
1. Create an scalable jbossas app on INT.
#rhc app create -a jbosap -t jossas-7 -p xx -s
2. Run rhc port-forward
#rhc port-forward -a jbosap

Actual results:
The reasonable information is displayed as below:

# rhc port-forward -a scaljbos -p redhat

This utility does not currently support scaled applications.
You will need to set up port forwarding manually.


Then try to setup port forward manually as below:
ssh -N -L 127.12.185.1:8080:127.12.185.1:8080 c0cea2d14b924531907e400039e691a0.rhcloud.com

The forwarded url can be accessed. 

So mark this bug as Verified. Thanks.