Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2222812

Summary: ipa cli doesn't work over ssh session
Product: Red Hat Enterprise Linux 9 Reporter: Tomasz Kepczynski <tomek>
Component: ipaAssignee: Florence Blanc-Renaud <frenaud>
Status: CLOSED WORKSFORME QA Contact: ipa-qe
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.2CC: bnater, bstinson, cheimes, rcritten, tscherf
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-09-12 07:07:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Tomasz Kepczynski 2023-07-13 19:55:01 UTC
Description of problem:
I cannot use ipa cli when logged in remotely (using ssh with gssapi authentication and exchange).

Version-Release number of selected component (if applicable):
ipa-client-4.10.1-7.el9_2.x86_64
ipa-client-common-4.10.1-7.el9_2.noarch
ipa-common-4.10.1-7.el9_2.noarch
ipa-selinux-4.10.1-7.el9_2.noarch
openssh-8.7p1-29.el9_2.x86_64
openssh-clients-8.7p1-29.el9_2.x86_64
openssh-server-8.7p1-29.el9_2.x86_64

How reproducible:
Nearly always. IPA clients on AlmaLinux 9.2 fail always or nearly always. IPA server on AlmaLinux 8.8 doesn't seem to exhibit this issue.

Steps to Reproduce:

vesemir:~> # list kerberos tickets
vesemir:~> klist 
Ticket cache: KEYRING:persistent:20000003:krb_ccache_t2Cdat7
Default principal: tomek.NET

Valid starting       Expires              Service principal
13.07.2023 21:41:58  14.07.2023 20:46:02  host/paulie.xxxxx.org@
        renew until 14.07.2023 20:50:02
        Ticket server: host/paulie.xxxxx.org.NET
13.07.2023 21:41:51  14.07.2023 20:46:02  krbtgt/IPA.XXXXX.NET.NET
        renew until 14.07.2023 20:50:02
vesemir:~> # login to remote client
vesemir:~> ssh paulie.xxxxx.org 
Last login: Thu Jul 13 21:39:20 2023 from 2a0X:XXXXX:XXXXX:3000:94cb:2ef5:6321:e82f
paulie:~> # list kerberos tickets
paulie:~> klist 
Ticket cache: KEYRING:persistent:20000003:krb_ccache_l81ErTf
Default principal: tomek.NET

Valid starting       Expires              Service principal
13.07.2023 21:42:27  14.07.2023 20:46:02  krbtgt/IPA.XXXXX.NET.NET
        renew until 14.07.2023 20:50:02
paulie:~> # try to use ipa cli
paulie:~> ipa user-find
ipa: ERROR: cannot connect to 'any of the configured servers': https://zoltan.xxxxx.org/ipa/json, https://triss.xxxxx.net/ipa/json, https://tissaia.xxxxx.net/ipa/json
paulie:~> # now try to get another ticket
paulie:~> kinit 
Password for tomek.NET: 
paulie:~> klist 
Ticket cache: KEYRING:persistent:20000003:krb_ccache_l81ErTf
Default principal: tomek.NET

Valid starting       Expires              Service principal
13.07.2023 21:43:30  14.07.2023 21:42:53  krbtgt/IPA.XXXXX.NET.NET
paulie:~> # give it another try
paulie:~> ipa user-find
---------------------
Pasuje 6 użytkowników
---------------------
  Login użytkownika: admin
  Nazwisko: Administrator
  Katalog domowy: /home/admin
  Powłoka logowania: /bin/bash
  Nazwa naczelnika: admin.NET
  Principal alias: admin.NET, root.NET
  UID: 20000000
  GID: 20000000
  Account disabled: False

:
[=== cut ===]
:
----------------------------
Number of entries returned 6
----------------------------
paulie:~> # now it worked....

Actual results:
ipa user-find command fails right after a login, apparently the kerberos ticket passed by ssh might be at fault here. After a kinit on remote system new ticket is obtained and then the command works.

Expected results:
ipa cli should work with the kerberos ticket passed by ssh.

Additional info:
I am puzzled by this issue. At first I thought it might be caused by updated freeipa on AlmaLinux 9.2 (contrary to 8.8). I had a thought it might have been caused by overrides.

Now what I see most consistently is:
- the command works on local systems (the systems where I login on console);
- the command doesn't work on remotes logged in with ssh (I do use SSO with both gssapi authentication and key exchange enabled);
- but apparently it is NOT an issue on AlmaLinux 8.8 IPA replica, while it can still be reproduced on AlmaLinux 9.2 replicas (2 of them).

This MIGHT be the configuration issue but I have NO idea how to pinpoint it...

Comment 1 Christian Heimes 2023-07-13 20:39:26 UTC
I'm unable to reproduce the issue on my RHEL 9.2 test cluster. Ticket forwarding and delegation works as expected.

$ kinit admin
$ klist
Ticket cache: KCM:1000
Default principal: admin

Valid starting     Expires            Service principal
07/13/23 16:26:50  07/14/23 15:54:31  krbtgt/IPA.TEST

$ ssh -oGSSAPIDelegateCredentials=yes admin.test

$ klist
Ticket cache: KCM:271600000:79189
Default principal: admin

Valid starting     Expires            Service principal
07/13/23 16:27:35  07/14/23 15:54:31  krbtgt/IPA.TEST
$ ipa user-show admin
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin, root
  UID: 271600000
  GID: 271600000
  Account disabled: False
  Password: True
  Member of groups: trust admins, admins
  Kerberos keys available: True

$ rpm -qa ipa-client krb5-workstation
krb5-workstation-1.20.1-9.el9_2.x86_64
ipa-client-4.10.1-7.el9_2.x86_64


Since you are an AlmaLinux user, please report your issue to AlmaLinux and ask their engineers to look into the problem.

If you are a Red Hat customer, please open a support case in the Customer Portal so we can have our support team help you appropriately.

You can also use the freeipa-users mailing list and ask the community for assistance. The list is hosted at https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org/ .

Comment 3 Tomasz Kepczynski 2023-09-02 17:45:14 UTC
This issue no longer reproduces, after upgrading the remaining IPA masters to AlmaLinux / RHEL 9.2. It probably can be closed as not reproducible.

I would suggest testing using ipa commands over remote sessions where tickets are passed from local to remote systems in a deployment where some IPA masters are from current and some from preceding RHEL major release as this would be a typical scenario during an upgrade to new major release.

Comment 4 Florence Blanc-Renaud 2023-09-12 07:07:39 UTC
Thanks for your update. According to your comment, I am closing this issue as not reproducible.