Bug 1605147

Created redmine issue https://projects.theforeman.org/issues/24367 from this bug

note that ansible core does not provide this, we could only workaround it through  ssh-agent, it would be possible though if we used ansible-runner as an engine

Thank you for your interest in Satellite 6. We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in the product in the foreseeable future. This is due to other priorities for the product, and not a reflection on the request itself. We are therefore closing this out as WONTFIX. If you have any concerns about this, please do not reopen. Instead, feel free to contact Red Hat Technical Support. Thank you.

Reopening for consideration now that we have ansible-runner in Satellite 6.8 and a customer request for this feature.

The runner has support for keys and passwords, it could be done in combinations with emitted password. This part is being improved right now, so we'd need to wait a little, but this is now possible to achieve.

https://ansible-runner.readthedocs.io/en/stable/intro.html#env-ssh-key
https://ansible-runner.readthedocs.io/en/stable/intro.html#env-passwords

Upon review of our valid but aging backlog the Satellite Team has concluded that this Bugzilla does not meet the criteria for a resolution in the near term, and are planning to close in one month's time. If you have any concerns about this, please contact your Red Hat Account team.  Thank you.

*** This bug has been marked as a duplicate of bug 1942782 ***

Thank you for your interest in Satellite 6. We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in the product in the foreseeable future. This is due to other priorities for the product, and not a reflection on the request itself. We are therefore closing this out as WONTFIX. If you have any concerns about this, please do not reopen. Instead, feel free to contact your Red Hat Account Team. Thank you.

Upstream bug assigned to mhulan

Upstream bug assigned to mhulan

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/24367 has been resolved.

*** Bug 1986422 has been marked as a duplicate of this bug. ***

*** Satellite 6.8.6 Hotfix Available ***

Note that this includes the fix for 1605147 as well as an additional critical hotfix in this area of code, BZ 1971395.

1) Download tfm-rubygem-foreman_ansible_core-3.0.4.1-3.HFRHBZ1971395.1605147.el7sat.noarch.rpm from this bugzilla to your Satellite

2) stop services:

satellite-maintain service stop

3) Install:

rpm -Uvh tfm-rubygem-foreman_ansible_core-3.0.4.1-3.HFRHBZ1971395.1605147.el7sat.noarch.rpm

4) restart:

satellite-maintain service start

5) resume operations

Created attachment 1811561 [details]
tfm-rubygem-foreman_ansible_core-3.0.4.1-3.HFRHBZ1971395.1605147.el7sat.noarch.rpm

To test:
0) Check that running an ansible job against a target host works before changing anything

1) Add a passphrase to the already-existing key
sudo -u foreman-proxy ssh-keygen -p -f ~foreman-proxy/.ssh/id_rsa_foreman_proxy -N $password

2) Restart capsule
systemctl restart foreman-proxy

3) Set the passphrase in Satellite
hammer setting set --name remote_execution_ssh_key_passphrase --value $password

4) Run an ansible job

Actual results:
The job fails since it cannot open the key

Expected results:
The job passes

Failed QA on Satellite 6.10, snap 20 (tfm-rubygem-foreman_ansible-6.3.4-1.el7sat.noarch).

Steps to Test:
1. Register a RHEL 7 host and a RHEL 8 host to Satellite 6.10 and configure the hosts for remote execution with the Global Registration Template.
2. Verify that a job of category 'Ansible Commands' and template 'Run Command - Ansible Default' is able to execute successfully on both hosts. I ran `touch test_ansible`, which succeeded on both hosts.
3. Add a passphrase to the foreman-proxy private key:

# sudo -u foreman-proxy ssh-keygen -p -f ~foreman-proxy/.ssh/id_rsa_foreman_proxy -N password

4. Restart the foreman-proxy service:

# systemctl restart foreman-proxy

5. Set the passphrase for the key in Satellite:

# hammer setting set --name remote_execution_ssh_key_passphrase --value password

6. Attempt to run another remote job on the hosts. The settings for this job were the same as in step 2, except that the command used was `touch test_ansible_with_passphrase`.

Actual Results:
The `Actions::RemoteExecution::RunHostsJob` task and its two `Actions::RemoteExecution::RunHostJob` subtasks stay in result 'Running' and state 'Pending' for fifteen minutes.

Expected Results:
The remote jobs finish successfully.

Notes:
- The results were the same when attempting the same job and entering the passphrase in the 'Private key passphrase' field on the job invocations page.
- The private key was able to authenticate successfully when attempting to log in from the Satellite to the hosts using `ssh -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy root.com` and entering the passphrase interactively.
- I did not find any error messages in /var/log/foreman-proxy/proxy.log, /var/log/foreman/production.log, or /var/log/messages on the Satellite around the time the unsuccessful jobs were run.

I think the fix actually landed in smart_proxy_ansible, it requires this specific commit https://github.com/theforeman/smart_proxy_ansible/commit/a169a9518b67b42088fdd2c45da6515a9c13367f, which doesn't seem to be present in any released version. I guess we'd need a cherrypick for 6.10.

Adam can you please double-check your Fixed in version information is correct?

> I think the fix actually landed in smart_proxy_ansible

While that is true, 6.10 still has foreman_ansible before the core extraction happened, so the fix for 6.10 will be in foreman_ansible and foreman_ansible_core. Apparently the fix was never backported to foreman_ansible_core, if we don't count hotfixes.

foreman_ansible-6.3.3 has fix for the behavior described in #24, the rest just went out in foreman_ansible_core-4.2.0.

Failed QA on Satellite 6.10, snap 21 (tfm-rubygem-foreman_ansible-6.3.4-1.el7sat.noarch, tfm-rubygem-foreman_ansible_core-4.2.0-1.el7sat.noarch).

I used the same steps to test as in comment #28. Now, though, `Ansible Command` jobs stay in result 'Running' and state 'Pending' indefinitely both when a passphrase is defined and when no passphrase is defined. 

For comparison, equivalent remote jobs (`touch ssh_test` and `touch ssh_test_with_passphrase`) run against two hosts using the `Run Command - SSH Default` job template finish in ten seconds both when a passphrase is defined and when no passphrase is defined.

See also bug 2010863

I'd say that if ansible-runner hangs on no passphrase forever, it's primarilly a bug in ansible-runner. There should be some timeout. From the Satellite process we can't tell whether the ansible-runner process still does something (e.g. configuring a system through ansible) after 10 minutes or it only waits for the input. We can't really even tell if the SSH key is passphrased prior the actual run.

However based on the comment 31, it seems that even if the passphrase is specified, it hangs, correct? That would of course be an issue. How does it behave when the key is not passphrased at all? Does that still work?

Yesterday, with no passphrase specified on the key, I left an `Ansible Commands` job running on snap 21 when I signed off for the day. This morning, it had been running for ~18 hours with no change. The behavior currently appears to be the same with Ansible Commands jobs regardless of whether a passphrase is set.

On Satellite 6.10, snap 20, though, Ansible Commands were working properly when no passphrase was set, so there seems to have been a regression between snap 20 and snap 21

Non-Ansible remote jobs over SSH are working properly on snap 21 both with no passphrase set and with a passphrase set.

Verified on Satellite 6.10, snap 23 (tfm-rubygem-foreman_ansible-6.3.4-1.el7sat.noarch).

I re-ran the steps from comment 28, and Ansible Commands jobs are now completing successfully both with and without a passphrase assigned to the foreman-proxy private key.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4702