Bug 2173155

Summary:	REX jobs through jump hosts are failing as ProxyCommand=none is hard coded
Product:	Red Hat Satellite	Reporter:	Nagoor Shaik <nshaik>
Component:	Remote Execution	Assignee:	satellite6-bugs <satellite6-bugs>
Status:	NEW ---	QA Contact:	Satellite QE Team <sat-qe-bz-list>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	6.12.1	CC:	aruzicka, ashipati, ben.argyle
Target Milestone:	Unspecified	Keywords:	Triaged
Target Release:	Unused	Flags:	nshaik: needinfo-
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Nagoor Shaik 2023-02-24 10:38:13 UTC

Description of problem:
Remote Execution jobs are failing when triggered actions goes through the jump host/bastion server.

Satellite ----> Jump Host/Bastion Server ---> Client

Even after configuring correct ssh parameters under ~foreman-proxy/.ssh/config REX jobs fails because the ProxyCommand=none is hard coded as per https://github.com/theforeman/smart_proxy_remote_execution_ssh/pull/89

Version-Release number of selected component (if applicable):
Satellite 6.12+

How reproducible:
100%

Steps to Reproduce:
1. Install Satellite 6.12
2. Register a client behind a NATed network (Ensure the only way to reach the host from Satellite is through a Jump Host)
3. Configure Jump Host configuration as below 

 # vi ~foreman-proxy/.ssh/config
Host bastion.example.com
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  IdentityFile ~foreman-proxy/.ssh/id_rsa_foreman_proxy

Host * !bastion.example.com
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  IdentityFile ~foreman-proxy/.ssh/id_rsa_foreman_proxy
  ProxyJump root.com

4. Ensure foreman-proxy SSH pub keys are copied to both bastion and client
5. Execute a simple REX job which fails with below error

~~~
1: Error initializing command: RuntimeError - Failed to establish connection to remote host, exit code: 255
2: Exit status: EXCEPTION
~~~

Actual results:
REX job fails

Expected results:
REX job should succeed and ProxyCommand argument shouldn't be hard-coded 

Additional info:
IMO, we need to revert this change https://github.com/theforeman/smart_proxy_remote_execution_ssh/pull/89

and this bug https://bugzilla.redhat.com/show_bug.cgi?id=2108637 can easily be addressed by just opening a documentation to include a switch `--no-ssh` argument with the ipa-client-install command.

Comment 1 Adam Ruzicka 2023-02-28 10:27:03 UTC

If I enroll a machine with ipa-client-install and later rerun it as ipa-client-install --no-ssh, will it fix things or does the argument need to be passed the first time the machine is enrolled?

Comment 2 Nagoor Shaik 2023-03-01 08:47:46 UTC

Unfortunately no, its not possible. It has to be passed in for the first time. In case a user has not configured it with --no-ssh option, then they have to run the ipa-client-install --uninstall first before re-running the "ipa-client-install --no-ssh" again.

See https://bugzilla.redhat.com/show_bug.cgi?id=1544379#c7

Comment 3 Adam Ruzicka 2023-03-01 12:23:08 UTC

That doesn't sound like a good user experience. Would having ssh options configurable the same way as we do for ansible be acceptable?

Comment 4 Nagoor Shaik 2023-03-02 04:27:11 UTC

Yes, that looks good plan. But, I strongly feel that --no-ssh is better option for Satellite 6, because it has to be executed just on the Satellite/Capsules which doesn't involve too much effort for the end-user.

Even if the customer has executed ipa-client-install without this option, they can just uninstall and then use the --no-ssh option which wouldn't cause any major outage and also it is common practice in IDM world as its recommended in https://bugzilla.redhat.com/show_bug.cgi?id=1544379#c7

Comment 6 Ben 2023-06-06 07:52:20 UTC

Just for reference, I've been fixing this for my own <= 6.12.2 (and now 6.13.1) install by commenting out the line

ssh_options << "-o ProxyCommand=none"

in file 

/usr/share/gems/gems/smart_proxy_remote_execution_ssh-0.10.1/lib/smart_proxy_remote_execution_ssh/multiplexed_ssh_connection.rb 

(previously under 6.12.2 it was in 

/usr/share/gems/gems/smart_proxy_remote_execution_ssh-0.7.3/lib/smart_proxy_remote_execution_ssh/runners/script_runner.rb)

I'd love not to have to do so as it messes up "satellite-maintain upgrade check" runs (-: