Bug 2151383

Summary: sss_ssh_knownhostsproxy causes high latency with X11 forwarding
Product: Red Hat Enterprise Linux 8 Reporter: Chance Callahan <ccallaha>
Component: sssdAssignee: Alexey Tikhonov <atikhono>
Status: CLOSED WONTFIX QA Contact: Anuj Borah <aborah>
Severity: medium Docs Contact:
Priority: low    
Version: 8.6CC: aboscatt, atikhono, dchen, pbrezina, sgadekar
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard: sync-to-jira
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-06-13 16:36:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Chance Callahan 2022-12-06 22:25:30 UTC
Description of problem:

When the sss_ssh_knownhostsproxy is enabled on OpenSSH, it causes high latency with X11 forwarding.

Version-Release number of selected component (if applicable):


libssh2-1.9.0-5.el8.x86_64                                  Tue Oct  4 11:43:45 2022
libssh2-devel-1.9.0-5.el8.x86_64                            Tue Oct  4 11:43:46 2022
libssh-0.9.6-3.el8.x86_64                                   Tue Oct  4 11:39:11 2022
libssh-host4ig-0.9.6-3.el8.noarch                            Tue Oct  4 11:39:06 2022
libssh-devel-0.9.6-3.el8.x86_64                             Tue Oct  4 11:40:24 2022
libxkbcommon-x11-0.9.1-1.el8.x86_64                         Tue Oct  4 11:38:45 2022
openssh-8.0p1-13.el8.x86_64                                 Tue Oct  4 11:39:25 2022
openssh-askpass-8.0p1-13.el8.x86_64                         Tue Oct  4 11:43:24 2022
openssh-clients-8.0p1-13.el8.x86_64                         Tue Oct  4 11:40:19 2022
openssh-server-8.0p1-13.el8.x86_64                          Tue Oct  4 11:40:19 2022
python3-sssdhost4ig-2.6.2-4.el86.1.noarch                   Tue Oct  4 11:44:50 2022
sssd-client-2.6.2-4.el8_6.1.x86_64                          Tue Oct  4 11:39:38 2022
sssd-common-2.6.2-4.el8_6.1.x86_64                          Tue Oct  4 11:39:44 2022
sssd-common-pac-2.6.2-4.el8_6.1.x86_64                      Tue Oct  4 11:44:51 2022
sssd-dbus-2.6.2-4.el8_6.1.x86_64                            Tue Oct  4 11:44:49 2022
sssd-ipa-2.6.2-4.el8_6.1.x86_64                             Tue Oct  4 11:44:51 2022
sssd-kcm-2.6.2-4.el8_6.1.x86_64                             Tue Oct  4 11:40:00 2022
sssd-krb5-common-2.6.2-4.el8_6.1.x86_64                     Tue Oct  4 11:44:50 2022
sssd-nfs-host56ap-2.6.2-4.el8_6.1.x86_64                       Tue Oct  4 11:39:44 2022
sssd-tools-2.6.2-4.el8_6.1.x86_64                           Tue Oct  4 11:44:51 2022
x11vnc-0.9.16-3.el8.x86_64                                  Tue Oct  4 11:43:46 2022
xorg-x11-apps-7.7-21.el8.x86_64                             Tue Oct  4 11:39:31 2022
xorg-x11-drv-fbdev-0.5.0-2.el8.x86_64                       Tue Oct  4 11:43:20 2022
xorg-x11-drv-libinput-0.29.0-1.el8.x86_64                   Tue Oct  4 11:43:20 2022
xorg-x11-drv-vesa-2.4.0-3.el8.x86_64                        Tue Oct  4 11:43:20 2022
xorg-x11-fonts-75dpi-7.5-19.el8.noarch                      Tue Oct  4 11:40:26 2022
xorg-x11-fonts-100dpi-7.5-19.el8.noarch                     Tue Oct  4 11:40:25 2022
xorg-x11-fonts-ISO8859-1-75dpi-7.5-19.el8.noarch            Tue Oct  4 11:38:52 2022
xorg-x11-fonts-ISO8859-1-100dpi-7.5-19.el8.noarch           Tue Oct  4 11:38:52 2022
xorg-x11-fonts-Type1-7.5-19.el8.noarch                      Tue Oct  4 11:43:08 2022
xorg-x11-fonts-misc-7.5-19.el8.noarch                       Tue Oct  4 11:39:23 2022
xorg-x11-font-utils-7.5-41.el8.x86_64                       Tue Oct  4 11:38:51 2022
xorg-x11-proto-devel-2020.1-3.el8.noarch                    Tue Oct  4 11:38:51 2022
xorg-x11-server-Xorg-1.20.11-5.el8_6.2.x86_64               Tue Oct  4 11:43:21 2022
xorg-x11-server-Xvfb-1.20.11-5.el8_6.2.x86_64               Tue Oct  4 11:43:45 2022
xorg-x11-server-common-1.20.11-5.el8_6.2.x86_64             Tue Oct  4 11:43:20 2022
xorg-x11-server-utils-7.7-27.el8.x86_64                     Tue Oct  4 11:39:07 2022
xorg-x11-utils-7.5-28.el8.x86_64                            Tue Oct  4 11:39:07 2022
xorg-x11-xauth-1.0.9-12.el8.x86_64                          Tue Oct  4 11:39:07 2022
xorg-x11-xbitmaps-1.1.1-13.el8.noarch                       Tue Oct  4 11:38:51 2022
xorg-x11-xinit-1.3.4-18.el8.x86_64                          Tue Oct  4 11:39:44 2022
xorg-x11-xkb-utils-7.7-28.el8.x86_64                        Tue Oct  4 11:43:19 2022

How reproducible:

I've had trouble reproducing, but the customer has been able to successfully on multiple machines.

Steps to Reproduce:
1. Make sure glxgears is installed
2. $ ssh -X -o 'ProxyCommand=/usr/bin/sss_ssh_knownhostsproxy -p %p %h' user@machine glxgears

Actual results:

45 frames in 5.0 seconds =  8.978 FPS
71 frames in 5.0 seconds = 14.158 FPS
64 frames in 5.0 seconds = 12.783 FPS
75 frames in 5.1 seconds = 14.679 FPS

Expected results:

705 frames in 5.0 seconds = 140.936 FPS
649 frames in 5.0 seconds = 129.643 FPS
701 frames in 5.0 seconds = 140.121 FPS
717 frames in 5.0 seconds = 143.343 FPS

Additional info:

Case c#13 has a detailed example from customer on how to reproduce it.

Comment 1 Alexey Tikhonov 2022-12-12 21:28:24 UTC
Probably performance of `sss_ssh_knownhostsproxy :: proxy_data()` could be improved using something like `sendfile()` (or other zero-copy technique (MSG_ZEROCOPY?))

But not sure if it's worth the effort taking into account `sss_ssh_knownhostsproxy` is deprecated and will be removed / needs to be replaced by `KnownHostsCommand` - see https://github.com/SSSD/sssd/issues/5518

Comment 2 Alexey Tikhonov 2022-12-13 09:10:12 UTC
(In reply to Alexey Tikhonov from comment #1)
> 
> But not sure if it's worth the effort taking into account
> `sss_ssh_knownhostsproxy` is deprecated

A note: I didn't mean "oficially deprecated here". I merely meant "have to be replaced by `KnownHostsCommand`".

Comment 10 Alexey Tikhonov 2023-06-13 16:36:32 UTC
Well, an attempt to improve performance of SSSD proxy helper - https://github.com/SSSD/sssd/pull/6757 - didn't help a real user.

No further attempt will be made to improve it.

Instead team will focus on implementing support of new ssh configuration option - `KnownHostsCommand`, that should be much more promising area to put effort into (see https://github.com/SSSD/sssd/issues/5518 )

Comment 11 Ding-Yi Chen 2023-07-14 01:44:40 UTC
The behaviour of sss_ssh_knownhostsproxy is:

1. Query IdM server and get the host public key
2. Store the publickey to /var/lib/sss/pubconf/known_hosts

So if 1 is slow, patching one SSSD side probably will not help much.

Let time how long does it require to get the public key


~~~
time ssh_ssh_knownhostproxy -k -p $PORT $HOST_NAME
~~~

Comment 12 Alexey Tikhonov 2023-07-17 12:44:33 UTC
(In reply to Ding-Yi Chen from comment #11)
> The behaviour of sss_ssh_knownhostsproxy is:
> 
> 1. Query IdM server and get the host public key
> 2. Store the publickey to /var/lib/sss/pubconf/known_hosts
> 
> So if 1 is slow, patching one SSSD side probably will not help much.

What is slow is:
3. proxy data: https://github.com/SSSD/sssd/blob/34ef9c5f3e90d5c50c7ac5161c39daa2840c92f2/src/sss_client/ssh/sss_ssh_knownhostsproxy.c#L77