RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1438016 - gssapi errors after IPA server upgrade
Summary: gssapi errors after IPA server upgrade
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa
Version: 7.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: IPA Maintainers
QA Contact: Nikhil Dehadrai
URL:
Whiteboard:
Depends On: 1438390
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-31 15:59 UTC by Scott Poore
Modified: 2017-08-01 09:47 UTC (History)
7 users (show)

Fixed In Version: ipa-4.5.0-16.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1438390 (view as bug list)
Environment:
Last Closed: 2017-08-01 09:47:49 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:2304 0 normal SHIPPED_LIVE ipa bug fix and enhancement update 2017-08-01 12:41:35 UTC

Description Scott Poore 2017-03-31 15:59:34 UTC
Description of problem:

After upgrading the IPA server from IPA version 4.4 (rhel7.3) to 4.5 (rhel7.4), I'm seeing gssapi errors and internal server error's when running commands.

After upgrade:

[root@auto-hv-02-guest10 /]# ipa user-find
ipa: ERROR: cannot connect to 'https://auto-hv-02-guest10.isc03081.test/ipa/json': Internal Server Error

In /var/log/httpd/error_log I see a backtrace and a lot of these:


   File "misc.pyx", line 174, in gssapi.raw.misc._display_status (gssapi/raw/misc.c:1797)
   File "misc.pyx", line 216, in gssapi.raw.misc.GSSErrorRegistry.__call__ (gssapi/raw/misc.c:2719)
   File "misc.pyx", line 272, in gssapi.raw.misc.GSSError.__init__ (gssapi/raw/misc.c:3143)
   File "misc.pyx", line 324, in gssapi.raw.misc.GSSError.gen_message (gssapi/raw/misc.c:3862)
   File "misc.pyx", line 291, in gssapi.raw.misc.GSSError.get_all_statuses (gssapi/raw/misc.c:3298)
 RuntimeError: maximum recursion depth exceeded while calling a Python object


Version-Release number of selected component (if applicable):
ipa-server-4.5.0-4.el7.x86_64
gssproxy-0.7.0-3.el7.x86_64
mod_wsgi-3.4-12.el7_0.x86_64


How reproducible:
Unknown.


Steps to Reproduce:
1.  on rhel7.3 install 4.4 ipa server
2.  upgrade to 4.5 version
3.  ipa user-find

Actual results:
internal server error

Expected results:
find users and not errors.

Additional info:

/var/log/httpd/error_log entries:

[Fri Mar 31 11:41:18.567914 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 274, in route
[Fri Mar 31 11:41:18.567933 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     return app(environ, start_response)
[Fri Mar 31 11:41:18.567941 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 468, in __call__
[Fri Mar 31 11:41:18.567965 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     response = super(jsonserver, self).__call__(environ, start_response)
[Fri Mar 31 11:41:18.567974 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 648, in __call__
[Fri Mar 31 11:41:18.567988 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     self.create_context(ccache=user_ccache)
[Fri Mar 31 11:41:18.568007 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 125, in create_context
[Fri Mar 31 11:41:18.568076 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     time_limit=None)
[Fri Mar 31 11:41:18.568088 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 66, in connect
[Fri Mar 31 11:41:18.568104 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     conn = self.create_connection(*args, **kw)
[Fri Mar 31 11:41:18.568113 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipaserver/plugins/ldap2.py", line 205, in create_connection
[Fri Mar 31 11:41:18.568248 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     principal = krb_utils.get_principal(ccache_name=ccache)
[Fri Mar 31 11:41:18.568266 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipalib/krb_utils.py", line 168, in get_principal
[Fri Mar 31 11:41:18.568334 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     creds = get_credentials(ccache_name=ccache_name)
[Fri Mar 31 11:41:18.568361 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipalib/krb_utils.py", line 147, in get_credentials
[Fri Mar 31 11:41:18.568379 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     return gssapi.Credentials(usage='initiate', name=name, store=store)
[Fri Mar 31 11:41:18.568395 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__
[Fri Mar 31 11:41:18.568519 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     store=store)
[Fri Mar 31 11:41:18.568530 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire
[Fri Mar 31 11:41:18.568547 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     usage)
[Fri Mar 31 11:41:18.568554 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c:1726)
[Fri Mar 31 11:41:18.568657 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "misc.pyx", line 216, in gssapi.raw.misc.GSSErrorRegistry.__call__ (gssapi/raw/misc.c:2719)
[Fri Mar 31 11:41:18.568737 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "misc.pyx", line 272, in gssapi.raw.misc.GSSError.__init__ (gssapi/raw/misc.c:3143)
[Fri Mar 31 11:41:18.568814 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "misc.pyx", line 325, in gssapi.raw.misc.GSSError.gen_message (gssapi/raw/misc.c:3890)
[Fri Mar 31 11:41:18.568891 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "misc.pyx", line 295, in gssapi.raw.misc.GSSError.get_all_statuses (gssapi/raw/misc.c:3360)
[Fri Mar 31 11:41:18.568979 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "misc.pyx", line 174, in gssapi.raw.misc._display_status (gssapi/raw/misc.c:1797)
[Fri Mar 31 11:41:18.603558 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84] RuntimeError: maximum recursion depth exceeded while calling a Python object

Comment 2 Martin Babinsky 2017-04-03 07:45:34 UTC
Looks like https://pagure.io/freeipa/issue/6796 to me.

Comment 3 Petr Vobornik 2017-04-03 10:34:15 UTC
This bz was cloned to python-gssapi as triage of upstream IPA 6796 suggests.

The main fix is in python-gssapi but also a sanity fix should be done on IPA side.

Comment 4 Petr Vobornik 2017-04-13 16:31:34 UTC
Upstream ticket:
https://pagure.io/freeipa/issue/6796

Comment 5 Petr Vobornik 2017-05-05 15:03:36 UTC
Blocking bug 1438390 was fixed so we can also raise requires to python-gssapi-1.2.0-3.el7

Comment 9 Nikhil Dehadrai 2017-06-07 10:43:45 UTC
IPA-server-version: ipa-server-4.5.0-15.el7.x86_64

Verified the bug on the basis of below observations:
1. Verified that upgrade of IPA-MASTER is successful.
2. Verified that after upgrade commands "ipa user-find, ipa user-show, ipa host-find" are run successfully without any errors.
3. Also no error are observed inside "/var/log/httpd/error_log".
4. Verified the same for following upgrade paths:
- Rhel 7.3.z > 7.4
- Rhel 7.3GA > 7.4
- Rhel 7.2.z > 7.4
- Rhel 7.1.z > 7.4- Upgrade fails for which a separate bug is updated BZ#1438731 (comment#6)
5. Refer console output from one of the upgrade paths:(RHel 7.3.z > 7.4)

[root@inferno ~]# tail -1 /var/log/ipaupgrade.log 
2017-06-07T10:03:21Z INFO The ipa-server-upgrade command was successful

[root@inferno ~]# rpm -q ipa-server
ipa-server-4.5.0-15.el7.x86_64

[root@inferno ~]# kinit admin
Password for admin: 

[root@inferno ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful

[root@inferno ~]# ipactl restart
Stopping pki-tomcatd Service
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting ntpd Service
Restarting pki-tomcatd Service
Restarting ipa-otpd Service
Restarting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful

[root@inferno ~]# ipa user-find
--------------
1 user matched
--------------
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin
  UID: 1075400000
  GID: 1075400000
  Account disabled: False
----------------------------
Number of entries returned 1
----------------------------
[root@inferno ~]# ipa host-find
--------------
1 host matched
--------------
  Host name: inferno.testrelm.test
  Principal name: host/inferno.testrelm.test
  Principal alias: host/inferno.testrelm.test
  SSH public key fingerprint: SHA256:LF8wIaQeKN6ww4llCkbPs6IuinEPL1O9At2QpyE23Qw (ssh-rsa),
                              SHA256:8jo0PBAD920N1MPQ/Kns9cspcu97gixeAvatoNbc4o0 (ssh-ed25519),
                              SHA256:8Yi1pl7+Nm8jaBwDDI3mjGnxVFqehziZ1CedR8sLjI0 (ecdsa-
                              sha2-nistp256)
----------------------------
Number of entries returned 1
----------------------------

[root@inferno ~]# ipa user-show
User login: admin
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin
  UID: 1075400000
  GID: 1075400000
  Account disabled: False
  Password: True
  Member of groups: admins, trust admins
  Kerberos keys available: True

[root@inferno ~]# cat /var/log/httpd/error_log | grep -rn "maximum recursion depth"
[root@inferno ~]# cat /var/log/httpd/error_log | grep -rn "recursion"
[root@inferno ~]# cat /var/log/httpd/error_log | grep -rn "maximum"
[root@inferno ~]# cat /var/log/httpd/error_log | grep -rn "gssapi"
[root@inferno ~]# cat /var/log/httpd/error_log | grep -rn "GSSError"

Thus on the basis of above observations, marking status of bug to "VERIFIED".

Comment 11 Martin Bašti 2017-06-07 13:26:21 UTC
ipa-4-5:

* 15d5ddd417d801a2356dcb043feef1aed8f76a25 Bump version of python-gssapi

Comment 13 Nikhil Dehadrai 2017-06-12 05:04:30 UTC
IPA-server-version: ipa-server-4.5.0-16.el7.x86_64

Verified the bug on the basis of below observations:
1. Verified that upgrade of IPA-MASTER is successful.
2. Verified that after upgrade commands "ipa user-find, ipa user-show, ipa host-find" are run successfully without any errors.
3. Also no errors are observed inside "/var/log/httpd/error_log".
4. Verified the same for following upgrade paths:
- Rhel 7.3.z > 7.4
- Rhel 7.3GA > 7.4
- Rhel 7.2.z > 7.4
- Rhel 7.1.z > 7.4- Upgrade fails for which a separate bug is updated BZ#1438731 (comment#6)
5. Refer console output from one of the upgrade paths:(Rhel 7.3.z > 7.4)

[root@auto-hv-01-guest03 ~]# rpm -q ipa-server
ipa-server-4.5.0-16.el7.x86_64

[root@auto-hv-01-guest03 ~]# rpm -q python-gssapi
python-gssapi-1.2.0-3.el7.x86_64

[root@auto-hv-01-guest03 ~]# tail -1 /var/log/ipaupgrade.log 
2017-06-12T04:47:06Z INFO The ipa-server-upgrade command was successful

[root@auto-hv-01-guest03 ~]# kinit admin
Password for admin: 

[root@auto-hv-01-guest03 ~]# ipa user-find
--------------
1 user matched
--------------
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin
  UID: 579000000
  GID: 579000000
  Account disabled: False
----------------------------
Number of entries returned 1
----------------------------

[root@auto-hv-01-guest03 ~]# ipa user-find tuser
--------------
1 user matched
--------------
  User login: tuser
  First name: test
  Last name: user
  Home directory: /home/tuser
  Login shell: /bin/sh
  Principal name: tuser
  Principal alias: tuser
  Email address: tuser
  UID: 579000001
  GID: 579000001
  Account disabled: False
----------------------------
Number of entries returned 1
----------------------------
[root@auto-hv-01-guest03 ~]# ipa user-find tuser --all
--------------
1 user matched
--------------
  dn: uid=tuser,cn=users,cn=accounts,dc=testrelm,dc=test
  User login: tuser
  First name: test
  Last name: user
  Full name: test user
  Display name: test user
  Initials: tu
  Home directory: /home/tuser
  GECOS: test user
  Login shell: /bin/sh
  Principal name: tuser
  Principal alias: tuser
  Email address: tuser
  UID: 579000001
  GID: 579000001
  Account disabled: False
  Preserved user: False
  Member of groups: ipausers
  ipauniqueid: 63edee6a-4f2c-11e7-80ae-525400cc38fd
  mepmanagedentry: cn=tuser,cn=groups,cn=accounts,dc=testrelm,dc=test
  objectclass: top, person, organizationalperson, inetorgperson, inetuser, posixaccount,
               krbprincipalaux, krbticketpolicyaux, ipaobject, ipasshuser, ipaSshGroupOfPubKeys,
               mepOriginEntry
----------------------------
Number of entries returned 1
----------------------------
[root@auto-hv-01-guest03 ~]# ipa user-find tuser --raw
--------------
1 user matched
--------------
  uid: tuser
  givenname: test
  sn: user
  homedirectory: /home/tuser
  loginshell: /bin/sh
  krbcanonicalname: tuser
  krbprincipalname: tuser
  mail: tuser
  uidnumber: 579000001
  gidnumber: 579000001
  nsaccountlock: FALSE
----------------------------
Number of entries returned 1
----------------------------

[root@auto-hv-01-guest03 ~]# ipa user-show
User login: admin
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin
  UID: 579000000
  GID: 579000000
  Account disabled: False
  Password: True
  Member of groups: admins, trust admins
  Kerberos keys available: True
[root@auto-hv-01-guest03 ~]# ipa host-find
--------------
1 host matched
--------------
  Host name: auto-hv-01-guest03.testrelm.test
  Principal name: host/auto-hv-01-guest03.testrelm.test
  Principal alias: host/auto-hv-01-guest03.testrelm.test
  SSH public key fingerprint: SHA256:81w5bMII4U0OBeCkwFrUSMvqCXuPGaTwj0v0DP51EWc (ssh-rsa),
                              SHA256:hMRDycHsxmY+M3JDMzwuV6RwrJzLKr6f5HOvqKOEX+Q (ecdsa-
                              sha2-nistp256), SHA256:9t9sRoJT5n4svMoMW2f2ok9ubc/UIgxoA+4NTqrmRB0
                              (ssh-ed25519)
----------------------------
Number of entries returned 1
----------------------------
[root@auto-hv-01-guest03 ~]# cat /var/log/httpd/error_log | grep -rn "maximum recursion depth"
[root@auto-hv-01-guest03 ~]# cat /var/log/httpd/error_log | grep -rn "recursion"
[root@auto-hv-01-guest03 ~]# cat /var/log/httpd/error_log | grep -rn "maximum"
[root@auto-hv-01-guest03 ~]# cat /var/log/httpd/error_log | grep -rn "gssapi"
[root@auto-hv-01-guest03 ~]# cat /var/log/httpd/error_log | grep -rn "GSSError"
[root@auto-hv-01-guest03 ~]# 

Thus on the basis of above observations, marking status of bug to "VERIFIED".

Comment 14 errata-xmlrpc 2017-08-01 09:47:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2304


Note You need to log in before you can comment on or make changes to this bug.