Marking as blocker because oVirt/RHV 4.2 can't add a host to cluster when engine is running on the host attached to IPA. The only workaround for this issue is to remove/comment below line in /etc/ssh/ssh_config:

 ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h

Have you tried installing the client with --no-ssh?

It may fix(In reply to Rob Crittenden from comment #3)
> Have you tried installing the client with --no-ssh?

I didn't. It may fix the issue, but I think I can't say user to do that, because he may want to install the ipa-client with ssh enabled.

It doesn't disable ssh. It disables the configuration option that is causing you grief.

From ipa-client-install(1):

       --no-ssh
              Do not configure OpenSSH client.

(In reply to Rob Crittenden from comment #5)
> It doesn't disable ssh. It disables the configuration option that is causing
> you grief.
> 
> From ipa-client-install(1):
> 
>        --no-ssh
>               Do not configure OpenSSH client.

Is there any way how to reconfigure IPA installation to remove offending SSH configuration (something like "ipa-client-config --no-ssh")? If so, it could be valid workaround for users, but I don't think so that manual edit of ssh_config is acceptable.

IPA provides no mechanism to reconfigure post-installation. The only option other than manually changing the file would be to run ipa-client-install --uninstall and then re-run ipa-client-install --no-ssh.

Can you describe the scenario further? Who/what is doing these IPA client installations?

Sorry, I missed the need_info request.

The issue is that we are executing 'ansible-playbook' command from the Java application as external proccess. So ansible-playbook command will load ssh configuration. The issue is that the Java apllication is running as user who don't have any shell - it's using /sbin/nologin, when running the ansible-playbook command as such user it fails because it can't execute '/usr/bin/sss_ssh_knownhostsproxy'.

Ok, is there a reason you can't pass the --no-ssh option to the client installer?

Well it's customer who do that. So we must instruct them to do the re-installation with --no-ssh if during oVirt installation we detect that it is configured with --ssh. Ok, I think you can close this bug. Thanks.

Ok, closing as NOTABUG.

We hit this issue again in RHV in slightly different scenario. Unfortunately reinstalling ipa-client described in comment #7 does not sound acceptable.

Is this behavior of sss_ssh_knownhostsproxy by design or a bug? If it is by design, would it be at least possible to include configuration option that would disable it?

I'll change the component to openssh to get a confirmation from the openssh maintainers that it is expected that the ProxyCommand option only works for users with a proper shell and that there is no work-around if the shell is /bin/noloing.

Manual page for ssh_config clearly says this is expected behavior:

> ProxyCommand
>       Specifies the command to use to connect to the server.  The command
>       string extends to the end of the line, and is executed using the user's
>       shell ‘exec’ directive to avoid a lingering shell process.

There is no other way how to do this function, if you want to provide a way to specify somehow rich commands with IO redirection and others.

If you need a way to work around this, I propose to use -F switch of ssh for loading your own configuration file or just use /dev/null if you do not depend on any modified configuration.



On the side note for IPA, I wanted to ask you if you could stop modifying ssh_config and use the drop-in directory in /etc/ssh/ssh_config.d/ for your modification for future releases, if you don't do that yet.

Is there anything else to resolve in this bug?

Jakub, thank you for the details. I switch the ticket back to IPA to evaluate the /etc/ssh/ssh_config.d/ suggestion. Is there /etc/ssh/sshd_config.d/ as well? Btw, it looks like /etc/ssh/ssh_config.d/ is not mentioned in the man pages.

Tomáš, would it work for you to call 'ssh -F /dev/null ....' as a workaround?

(In reply to Sumit Bose from comment #17)

> Tomáš, would it work for you to call 'ssh -F /dev/null ....' as a workaround?

Definitely not. This prevents the application that calls ssh (or scp in our case) from using any system wide configuration which is far from ideal (think e.g. of Ciphers).

We're better of installing IPA with --no-ssh. At least the inherent risks and side effects of that are more obvious.

The client side drop in directory was not announced a lot and is implemented by simple include in the main ssh_config as you can notice. This drop-in directory is unfortunately not provided by upstream out of the box and it might not work on other distributions, but we already use it for distribution modification of default configuration.

The server side sshd_config include was not yet merged upstream so it is not part of our package.

Thinking about the original problem, you (ipa) could try to use Match exec block to check the user has useful shell before setting the ProxyCommand, but this would add an overhead of one more shell invocation for every ssh invocation, which might or might not be acceptable for you. Something like this should do the job:

# assumes that if a user does not have shell (/sbin/nologin),
# this will return nonzero exit code and proxy command will be ignored
Match exec true   
        ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h

Thanks again Jakub, now I really switch the ticket back to IPA.

Upstream ticket:
https://pagure.io/freeipa/issue/7676

Can you expand on the reproduction steps?

As far as I can gather, you have a user in IPA with /sbin/nologin as the shell with a password so you authenticate as the user?

You're sshing from what to what? I assume both IPA-enrolled hosts?

How are you becoming this shell-less user, su? What ssh command-line are you using? (e.g. what are you doing with ssh)

How does it fail to ssh?

I don't know the details of the original issue, but the second time we hit that in oVirt was in this configuration:

- have VDSM host in IPA
- on host there is 'vdsm' user that has nologin as shell
- on host there is a service running as 'vdsm' user
- the service spawns other commands as subprocesses that in turn run ssh/scp to machine that is not in IPA (the command is virt-v2v)

I was able to dig up the failure from my old emails regarding the issue. Running virt-v2v fails with:
    ...
    [   0.2] Opening the source -i vmx ssh://root.x.x/vmfs/volumes/datastore12/fdupont-test_migration/fdupont-test_migration.vmx
    scp 'root'@'x.x.x.x':''\''/vmfs/volumes/datastore12/fdupont-test_migration/fdupont-test_migration.vmx'\''' '/var/tmp/vmx.FeUhiy/source.vmx'
    ssh_exchange_identification: Connection closed by remote host
    virt-v2v: error: could not copy the VMX file from the remote server, see
    earlier error messages

Manual reproducer with SSH is:

    # sudo -u vdsm ssh -v root.x.x
    ...
    debug1: Enabling compatibility mode for protocol 2.0
    debug1: Local version string SSH-2.0-OpenSSH_7.4
    debug1: ssh_exchange_identification: This account is currently not available.

Fixed upstream
master:
https://pagure.io/freeipa/c/15b30e15a42b8adbbd4cf718e43406941e475db8
https://pagure.io/freeipa/c/16616e576d6a2e52a9967a3a55a676dba56c6a77

Fixed upstream
ipa-4-9:
https://pagure.io/freeipa/c/a525b2ebf01ffff83d0a5925035f4be0fc5c700c
https://pagure.io/freeipa/c/d89e3abf2714092baae1607afd83da1c944d6c9f

The ssh_config(5) man page documents that a ProxyComamnd executes <shell> -c <proxy command>

Support older nologin versions that do not support -c.

Upstream PR https://github.com/freeipa/freeipa/pull/5401

Fixed upstream
master:
https://pagure.io/freeipa/c/dfa084217ed96c60bd9317de98e74d197f793420

Fixed upstream
ipa-4-9:
https://pagure.io/freeipa/c/ca9f8d1c9feda6fd58220f1424970dcca5b730e0

Verified using nightly run and version:

2021-02-02T14:14:09+0000     name: ipa-server
2021-02-02T14:14:09+0000     release: 1.module+el8.4.0+9665+c9815399
2021-02-02T14:14:09+0000     source: rpm
2021-02-02T14:14:09+0000     version: 4.9.1

Test log:
============================= test session starts ==============================
platform linux -- Python 3.6.8, pytest-3.10.1, py-1.10.0, pluggy-0.13.1 -- /usr/libexec/platform-python
cachedir: /home/cloud-user/.pytest_cache
metadata: {'Python': '3.6.8', 'Platform': 'Linux-4.18.0-280.el8.x86_64-x86_64-with-redhat-8.4-Ootpa', 'Packages': {'pytest': '3.10.1', 'py': '1.10.0', 'pluggy': '0.13.1'}, 'Plugins': {'metadata': '1.11.0', 'html': '1.22.1', 'multihost': '3.0', 'sourceorder': '0.5'}}
rootdir: /usr/lib/python3.6/site-packages/ipatests, inifile:
plugins: metadata-1.11.0, html-1.22.1, multihost-3.0, sourceorder-0.5
collecting ... collected 35 items

test_integration/test_commands.py::TestIPACommand::test_aes_sha_kerberos_enctypes PASSED [  2%]
test_integration/test_commands.py::TestIPACommand::test_certmap_match_issue7520 PASSED [  5%]
....
test_integration/test_commands.py::TestIPACommand::test_ipa_nis_manage_enable_incorrect_password PASSED [ 88%]
test_integration/test_commands.py::TestIPACommand::test_pkispawn_log_is_present PASSED [ 91%]
test_integration/test_commands.py::TestIPACommand::test_reset_password_unlock PASSED [ 94%]
test_integration/test_commands.py::TestIPACommand::test_certupdate_no_schema PASSED [ 97%]
test_integration/test_commands.py::TestIPACommand::test_proxycommand_invalid_shell PASSED [100%]

=================== 35 passed, 2 warnings in 1678.06 seconds ===================

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: idm:DL1 and idm:client security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1846