Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2151383

Summary: sss_ssh_knownhostsproxy causes high latency with X11 forwarding
Product: Red Hat Enterprise Linux 8 Reporter: Chance Callahan <ccallaha>
Component: sssdAssignee: Alexey Tikhonov <atikhono>
Status: CLOSED WONTFIX QA Contact: Anuj Borah <aborah>
Severity: medium Docs Contact:
Priority: low    
Version: 8.6CC: aboscatt, atikhono, dchen, pbrezina, sgadekar
Target Milestone: rcKeywords: Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: All   
OS: Linux   
Whiteboard: sync-to-jira
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-06-13 16:36:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Chance Callahan 2022-12-06 22:25:30 UTC 
Description of problem:

When the sss_ssh_knownhostsproxy is enabled on OpenSSH, it causes high latency with X11 forwarding.

Version-Release number of selected component (if applicable):


libssh2-1.9.0-5.el8.x86_64                                  Tue Oct  4 11:43:45 2022
libssh2-devel-1.9.0-5.el8.x86_64                            Tue Oct  4 11:43:46 2022
libssh-0.9.6-3.el8.x86_64                                   Tue Oct  4 11:39:11 2022
libssh-host4ig-0.9.6-3.el8.noarch                            Tue Oct  4 11:39:06 2022
libssh-devel-0.9.6-3.el8.x86_64                             Tue Oct  4 11:40:24 2022
libxkbcommon-x11-0.9.1-1.el8.x86_64                         Tue Oct  4 11:38:45 2022
openssh-8.0p1-13.el8.x86_64                                 Tue Oct  4 11:39:25 2022
openssh-askpass-8.0p1-13.el8.x86_64                         Tue Oct  4 11:43:24 2022
openssh-clients-8.0p1-13.el8.x86_64                         Tue Oct  4 11:40:19 2022
openssh-server-8.0p1-13.el8.x86_64                          Tue Oct  4 11:40:19 2022
python3-sssdhost4ig-2.6.2-4.el86.1.noarch                   Tue Oct  4 11:44:50 2022
sssd-client-2.6.2-4.el8_6.1.x86_64                          Tue Oct  4 11:39:38 2022
sssd-common-2.6.2-4.el8_6.1.x86_64                          Tue Oct  4 11:39:44 2022
sssd-common-pac-2.6.2-4.el8_6.1.x86_64                      Tue Oct  4 11:44:51 2022
sssd-dbus-2.6.2-4.el8_6.1.x86_64                            Tue Oct  4 11:44:49 2022
sssd-ipa-2.6.2-4.el8_6.1.x86_64                             Tue Oct  4 11:44:51 2022
sssd-kcm-2.6.2-4.el8_6.1.x86_64                             Tue Oct  4 11:40:00 2022
sssd-krb5-common-2.6.2-4.el8_6.1.x86_64                     Tue Oct  4 11:44:50 2022
sssd-nfs-host56ap-2.6.2-4.el8_6.1.x86_64                       Tue Oct  4 11:39:44 2022
sssd-tools-2.6.2-4.el8_6.1.x86_64                           Tue Oct  4 11:44:51 2022
x11vnc-0.9.16-3.el8.x86_64                                  Tue Oct  4 11:43:46 2022
xorg-x11-apps-7.7-21.el8.x86_64                             Tue Oct  4 11:39:31 2022
xorg-x11-drv-fbdev-0.5.0-2.el8.x86_64                       Tue Oct  4 11:43:20 2022
xorg-x11-drv-libinput-0.29.0-1.el8.x86_64                   Tue Oct  4 11:43:20 2022
xorg-x11-drv-vesa-2.4.0-3.el8.x86_64                        Tue Oct  4 11:43:20 2022
xorg-x11-fonts-75dpi-7.5-19.el8.noarch                      Tue Oct  4 11:40:26 2022
xorg-x11-fonts-100dpi-7.5-19.el8.noarch                     Tue Oct  4 11:40:25 2022
xorg-x11-fonts-ISO8859-1-75dpi-7.5-19.el8.noarch            Tue Oct  4 11:38:52 2022
xorg-x11-fonts-ISO8859-1-100dpi-7.5-19.el8.noarch           Tue Oct  4 11:38:52 2022
xorg-x11-fonts-Type1-7.5-19.el8.noarch                      Tue Oct  4 11:43:08 2022
xorg-x11-fonts-misc-7.5-19.el8.noarch                       Tue Oct  4 11:39:23 2022
xorg-x11-font-utils-7.5-41.el8.x86_64                       Tue Oct  4 11:38:51 2022
xorg-x11-proto-devel-2020.1-3.el8.noarch                    Tue Oct  4 11:38:51 2022
xorg-x11-server-Xorg-1.20.11-5.el8_6.2.x86_64               Tue Oct  4 11:43:21 2022
xorg-x11-server-Xvfb-1.20.11-5.el8_6.2.x86_64               Tue Oct  4 11:43:45 2022
xorg-x11-server-common-1.20.11-5.el8_6.2.x86_64             Tue Oct  4 11:43:20 2022
xorg-x11-server-utils-7.7-27.el8.x86_64                     Tue Oct  4 11:39:07 2022
xorg-x11-utils-7.5-28.el8.x86_64                            Tue Oct  4 11:39:07 2022
xorg-x11-xauth-1.0.9-12.el8.x86_64                          Tue Oct  4 11:39:07 2022
xorg-x11-xbitmaps-1.1.1-13.el8.noarch                       Tue Oct  4 11:38:51 2022
xorg-x11-xinit-1.3.4-18.el8.x86_64                          Tue Oct  4 11:39:44 2022
xorg-x11-xkb-utils-7.7-28.el8.x86_64                        Tue Oct  4 11:43:19 2022

How reproducible:

I've had trouble reproducing, but the customer has been able to successfully on multiple machines.

Steps to Reproduce:
1. Make sure glxgears is installed
2. $ ssh -X -o 'ProxyCommand=/usr/bin/sss_ssh_knownhostsproxy -p %p %h' user@machine glxgears

Actual results:

45 frames in 5.0 seconds =  8.978 FPS
71 frames in 5.0 seconds = 14.158 FPS
64 frames in 5.0 seconds = 12.783 FPS
75 frames in 5.1 seconds = 14.679 FPS

Expected results:

705 frames in 5.0 seconds = 140.936 FPS
649 frames in 5.0 seconds = 129.643 FPS
701 frames in 5.0 seconds = 140.121 FPS
717 frames in 5.0 seconds = 143.343 FPS

Additional info:

Case c#13 has a detailed example from customer on how to reproduce it.
Comment 1 Alexey Tikhonov 2022-12-12 21:28:24 UTC 
Probably performance of `sss_ssh_knownhostsproxy :: proxy_data()` could be improved using something like `sendfile()` (or other zero-copy technique (MSG_ZEROCOPY?))

But not sure if it's worth the effort taking into account `sss_ssh_knownhostsproxy` is deprecated and will be removed / needs to be replaced by `KnownHostsCommand` - see https://github.com/SSSD/sssd/issues/5518
Comment 2 Alexey Tikhonov 2022-12-13 09:10:12 UTC 
(In reply to Alexey Tikhonov from comment #1)
> 
> But not sure if it's worth the effort taking into account
> `sss_ssh_knownhostsproxy` is deprecated

A note: I didn't mean "oficially deprecated here". I merely meant "have to be replaced by `KnownHostsCommand`".
Comment 10 Alexey Tikhonov 2023-06-13 16:36:32 UTC 
Well, an attempt to improve performance of SSSD proxy helper - https://github.com/SSSD/sssd/pull/6757 - didn't help a real user.

No further attempt will be made to improve it.

Instead team will focus on implementing support of new ssh configuration option - `KnownHostsCommand`, that should be much more promising area to put effort into (see https://github.com/SSSD/sssd/issues/5518 )
Comment 11 Ding-Yi Chen 2023-07-14 01:44:40 UTC 
The behaviour of sss_ssh_knownhostsproxy is:

1. Query IdM server and get the host public key
2. Store the publickey to /var/lib/sss/pubconf/known_hosts

So if 1 is slow, patching one SSSD side probably will not help much.

Let time how long does it require to get the public key


~~~
time ssh_ssh_knownhostproxy -k -p $PORT $HOST_NAME
~~~
Comment 12 Alexey Tikhonov 2023-07-17 12:44:33 UTC 
(In reply to Ding-Yi Chen from comment #11)
> The behaviour of sss_ssh_knownhostsproxy is:
> 
> 1. Query IdM server and get the host public key
> 2. Store the publickey to /var/lib/sss/pubconf/known_hosts
> 
> So if 1 is slow, patching one SSSD side probably will not help much.

What is slow is:
3. proxy data: https://github.com/SSSD/sssd/blob/34ef9c5f3e90d5c50c7ac5161c39daa2840c92f2/src/sss_client/ssh/sss_ssh_knownhostsproxy.c#L77