Bug 1438016

Summary: gssapi errors after IPA server upgrade
Product: Red Hat Enterprise Linux 7 Reporter: Scott Poore <spoore>
Component: ipaAssignee: IPA Maintainers <ipa-maint>
Status: CLOSED ERRATA QA Contact: Nikhil Dehadrai <ndehadra>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.4CC: ksiddiqu, mbabinsk, mbasti, ovasik, pvoborni, rcritten, tscherf
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ipa-4.5.0-16.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1438390 (view as bug list) Environment:
Last Closed: 2017-08-01 09:47:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1438390    
Bug Blocks:    

Description Scott Poore 2017-03-31 15:59:34 UTC
Description of problem:

After upgrading the IPA server from IPA version 4.4 (rhel7.3) to 4.5 (rhel7.4), I'm seeing gssapi errors and internal server error's when running commands.

After upgrade:

[root@auto-hv-02-guest10 /]# ipa user-find
ipa: ERROR: cannot connect to 'https://auto-hv-02-guest10.isc03081.test/ipa/json': Internal Server Error

In /var/log/httpd/error_log I see a backtrace and a lot of these:


   File "misc.pyx", line 174, in gssapi.raw.misc._display_status (gssapi/raw/misc.c:1797)
   File "misc.pyx", line 216, in gssapi.raw.misc.GSSErrorRegistry.__call__ (gssapi/raw/misc.c:2719)
   File "misc.pyx", line 272, in gssapi.raw.misc.GSSError.__init__ (gssapi/raw/misc.c:3143)
   File "misc.pyx", line 324, in gssapi.raw.misc.GSSError.gen_message (gssapi/raw/misc.c:3862)
   File "misc.pyx", line 291, in gssapi.raw.misc.GSSError.get_all_statuses (gssapi/raw/misc.c:3298)
 RuntimeError: maximum recursion depth exceeded while calling a Python object


Version-Release number of selected component (if applicable):
ipa-server-4.5.0-4.el7.x86_64
gssproxy-0.7.0-3.el7.x86_64
mod_wsgi-3.4-12.el7_0.x86_64


How reproducible:
Unknown.


Steps to Reproduce:
1.  on rhel7.3 install 4.4 ipa server
2.  upgrade to 4.5 version
3.  ipa user-find

Actual results:
internal server error

Expected results:
find users and not errors.

Additional info:

/var/log/httpd/error_log entries:

[Fri Mar 31 11:41:18.567914 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 274, in route
[Fri Mar 31 11:41:18.567933 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     return app(environ, start_response)
[Fri Mar 31 11:41:18.567941 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 468, in __call__
[Fri Mar 31 11:41:18.567965 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     response = super(jsonserver, self).__call__(environ, start_response)
[Fri Mar 31 11:41:18.567974 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 648, in __call__
[Fri Mar 31 11:41:18.567988 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     self.create_context(ccache=user_ccache)
[Fri Mar 31 11:41:18.568007 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 125, in create_context
[Fri Mar 31 11:41:18.568076 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     time_limit=None)
[Fri Mar 31 11:41:18.568088 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 66, in connect
[Fri Mar 31 11:41:18.568104 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     conn = self.create_connection(*args, **kw)
[Fri Mar 31 11:41:18.568113 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipaserver/plugins/ldap2.py", line 205, in create_connection
[Fri Mar 31 11:41:18.568248 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     principal = krb_utils.get_principal(ccache_name=ccache)
[Fri Mar 31 11:41:18.568266 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipalib/krb_utils.py", line 168, in get_principal
[Fri Mar 31 11:41:18.568334 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     creds = get_credentials(ccache_name=ccache_name)
[Fri Mar 31 11:41:18.568361 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipalib/krb_utils.py", line 147, in get_credentials
[Fri Mar 31 11:41:18.568379 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     return gssapi.Credentials(usage='initiate', name=name, store=store)
[Fri Mar 31 11:41:18.568395 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__
[Fri Mar 31 11:41:18.568519 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     store=store)
[Fri Mar 31 11:41:18.568530 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire
[Fri Mar 31 11:41:18.568547 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     usage)
[Fri Mar 31 11:41:18.568554 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c:1726)
[Fri Mar 31 11:41:18.568657 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "misc.pyx", line 216, in gssapi.raw.misc.GSSErrorRegistry.__call__ (gssapi/raw/misc.c:2719)
[Fri Mar 31 11:41:18.568737 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "misc.pyx", line 272, in gssapi.raw.misc.GSSError.__init__ (gssapi/raw/misc.c:3143)
[Fri Mar 31 11:41:18.568814 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "misc.pyx", line 325, in gssapi.raw.misc.GSSError.gen_message (gssapi/raw/misc.c:3890)
[Fri Mar 31 11:41:18.568891 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "misc.pyx", line 295, in gssapi.raw.misc.GSSError.get_all_statuses (gssapi/raw/misc.c:3360)
[Fri Mar 31 11:41:18.568979 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "misc.pyx", line 174, in gssapi.raw.misc._display_status (gssapi/raw/misc.c:1797)
[Fri Mar 31 11:41:18.603558 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84] RuntimeError: maximum recursion depth exceeded while calling a Python object

Comment 2 Martin Babinsky 2017-04-03 07:45:34 UTC
Looks like https://pagure.io/freeipa/issue/6796 to me.

Comment 3 Petr Vobornik 2017-04-03 10:34:15 UTC
This bz was cloned to python-gssapi as triage of upstream IPA 6796 suggests.

The main fix is in python-gssapi but also a sanity fix should be done on IPA side.

Comment 4 Petr Vobornik 2017-04-13 16:31:34 UTC
Upstream ticket:
https://pagure.io/freeipa/issue/6796

Comment 5 Petr Vobornik 2017-05-05 15:03:36 UTC
Blocking bug 1438390 was fixed so we can also raise requires to python-gssapi-1.2.0-3.el7

Comment 9 Nikhil Dehadrai 2017-06-07 10:43:45 UTC
IPA-server-version: ipa-server-4.5.0-15.el7.x86_64

Verified the bug on the basis of below observations:
1. Verified that upgrade of IPA-MASTER is successful.
2. Verified that after upgrade commands "ipa user-find, ipa user-show, ipa host-find" are run successfully without any errors.
3. Also no error are observed inside "/var/log/httpd/error_log".
4. Verified the same for following upgrade paths:
- Rhel 7.3.z > 7.4
- Rhel 7.3GA > 7.4
- Rhel 7.2.z > 7.4
- Rhel 7.1.z > 7.4- Upgrade fails for which a separate bug is updated BZ#1438731 (comment#6)
5. Refer console output from one of the upgrade paths:(RHel 7.3.z > 7.4)

[root@inferno ~]# tail -1 /var/log/ipaupgrade.log 
2017-06-07T10:03:21Z INFO The ipa-server-upgrade command was successful

[root@inferno ~]# rpm -q ipa-server
ipa-server-4.5.0-15.el7.x86_64

[root@inferno ~]# kinit admin
Password for admin: 

[root@inferno ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful

[root@inferno ~]# ipactl restart
Stopping pki-tomcatd Service
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting ntpd Service
Restarting pki-tomcatd Service
Restarting ipa-otpd Service
Restarting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful

[root@inferno ~]# ipa user-find
--------------
1 user matched
--------------
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin
  UID: 1075400000
  GID: 1075400000
  Account disabled: False
----------------------------
Number of entries returned 1
----------------------------
[root@inferno ~]# ipa host-find
--------------
1 host matched
--------------
  Host name: inferno.testrelm.test
  Principal name: host/inferno.testrelm.test
  Principal alias: host/inferno.testrelm.test
  SSH public key fingerprint: SHA256:LF8wIaQeKN6ww4llCkbPs6IuinEPL1O9At2QpyE23Qw (ssh-rsa),
                              SHA256:8jo0PBAD920N1MPQ/Kns9cspcu97gixeAvatoNbc4o0 (ssh-ed25519),
                              SHA256:8Yi1pl7+Nm8jaBwDDI3mjGnxVFqehziZ1CedR8sLjI0 (ecdsa-
                              sha2-nistp256)
----------------------------
Number of entries returned 1
----------------------------

[root@inferno ~]# ipa user-show
User login: admin
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin
  UID: 1075400000
  GID: 1075400000
  Account disabled: False
  Password: True
  Member of groups: admins, trust admins
  Kerberos keys available: True

[root@inferno ~]# cat /var/log/httpd/error_log | grep -rn "maximum recursion depth"
[root@inferno ~]# cat /var/log/httpd/error_log | grep -rn "recursion"
[root@inferno ~]# cat /var/log/httpd/error_log | grep -rn "maximum"
[root@inferno ~]# cat /var/log/httpd/error_log | grep -rn "gssapi"
[root@inferno ~]# cat /var/log/httpd/error_log | grep -rn "GSSError"

Thus on the basis of above observations, marking status of bug to "VERIFIED".

Comment 11 Martin Bašti 2017-06-07 13:26:21 UTC
ipa-4-5:

* 15d5ddd417d801a2356dcb043feef1aed8f76a25 Bump version of python-gssapi

Comment 13 Nikhil Dehadrai 2017-06-12 05:04:30 UTC
IPA-server-version: ipa-server-4.5.0-16.el7.x86_64

Verified the bug on the basis of below observations:
1. Verified that upgrade of IPA-MASTER is successful.
2. Verified that after upgrade commands "ipa user-find, ipa user-show, ipa host-find" are run successfully without any errors.
3. Also no errors are observed inside "/var/log/httpd/error_log".
4. Verified the same for following upgrade paths:
- Rhel 7.3.z > 7.4
- Rhel 7.3GA > 7.4
- Rhel 7.2.z > 7.4
- Rhel 7.1.z > 7.4- Upgrade fails for which a separate bug is updated BZ#1438731 (comment#6)
5. Refer console output from one of the upgrade paths:(Rhel 7.3.z > 7.4)

[root@auto-hv-01-guest03 ~]# rpm -q ipa-server
ipa-server-4.5.0-16.el7.x86_64

[root@auto-hv-01-guest03 ~]# rpm -q python-gssapi
python-gssapi-1.2.0-3.el7.x86_64

[root@auto-hv-01-guest03 ~]# tail -1 /var/log/ipaupgrade.log 
2017-06-12T04:47:06Z INFO The ipa-server-upgrade command was successful

[root@auto-hv-01-guest03 ~]# kinit admin
Password for admin: 

[root@auto-hv-01-guest03 ~]# ipa user-find
--------------
1 user matched
--------------
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin
  UID: 579000000
  GID: 579000000
  Account disabled: False
----------------------------
Number of entries returned 1
----------------------------

[root@auto-hv-01-guest03 ~]# ipa user-find tuser
--------------
1 user matched
--------------
  User login: tuser
  First name: test
  Last name: user
  Home directory: /home/tuser
  Login shell: /bin/sh
  Principal name: tuser
  Principal alias: tuser
  Email address: tuser
  UID: 579000001
  GID: 579000001
  Account disabled: False
----------------------------
Number of entries returned 1
----------------------------
[root@auto-hv-01-guest03 ~]# ipa user-find tuser --all
--------------
1 user matched
--------------
  dn: uid=tuser,cn=users,cn=accounts,dc=testrelm,dc=test
  User login: tuser
  First name: test
  Last name: user
  Full name: test user
  Display name: test user
  Initials: tu
  Home directory: /home/tuser
  GECOS: test user
  Login shell: /bin/sh
  Principal name: tuser
  Principal alias: tuser
  Email address: tuser
  UID: 579000001
  GID: 579000001
  Account disabled: False
  Preserved user: False
  Member of groups: ipausers
  ipauniqueid: 63edee6a-4f2c-11e7-80ae-525400cc38fd
  mepmanagedentry: cn=tuser,cn=groups,cn=accounts,dc=testrelm,dc=test
  objectclass: top, person, organizationalperson, inetorgperson, inetuser, posixaccount,
               krbprincipalaux, krbticketpolicyaux, ipaobject, ipasshuser, ipaSshGroupOfPubKeys,
               mepOriginEntry
----------------------------
Number of entries returned 1
----------------------------
[root@auto-hv-01-guest03 ~]# ipa user-find tuser --raw
--------------
1 user matched
--------------
  uid: tuser
  givenname: test
  sn: user
  homedirectory: /home/tuser
  loginshell: /bin/sh
  krbcanonicalname: tuser
  krbprincipalname: tuser
  mail: tuser
  uidnumber: 579000001
  gidnumber: 579000001
  nsaccountlock: FALSE
----------------------------
Number of entries returned 1
----------------------------

[root@auto-hv-01-guest03 ~]# ipa user-show
User login: admin
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin
  UID: 579000000
  GID: 579000000
  Account disabled: False
  Password: True
  Member of groups: admins, trust admins
  Kerberos keys available: True
[root@auto-hv-01-guest03 ~]# ipa host-find
--------------
1 host matched
--------------
  Host name: auto-hv-01-guest03.testrelm.test
  Principal name: host/auto-hv-01-guest03.testrelm.test
  Principal alias: host/auto-hv-01-guest03.testrelm.test
  SSH public key fingerprint: SHA256:81w5bMII4U0OBeCkwFrUSMvqCXuPGaTwj0v0DP51EWc (ssh-rsa),
                              SHA256:hMRDycHsxmY+M3JDMzwuV6RwrJzLKr6f5HOvqKOEX+Q (ecdsa-
                              sha2-nistp256), SHA256:9t9sRoJT5n4svMoMW2f2ok9ubc/UIgxoA+4NTqrmRB0
                              (ssh-ed25519)
----------------------------
Number of entries returned 1
----------------------------
[root@auto-hv-01-guest03 ~]# cat /var/log/httpd/error_log | grep -rn "maximum recursion depth"
[root@auto-hv-01-guest03 ~]# cat /var/log/httpd/error_log | grep -rn "recursion"
[root@auto-hv-01-guest03 ~]# cat /var/log/httpd/error_log | grep -rn "maximum"
[root@auto-hv-01-guest03 ~]# cat /var/log/httpd/error_log | grep -rn "gssapi"
[root@auto-hv-01-guest03 ~]# cat /var/log/httpd/error_log | grep -rn "GSSError"
[root@auto-hv-01-guest03 ~]# 

Thus on the basis of above observations, marking status of bug to "VERIFIED".

Comment 14 errata-xmlrpc 2017-08-01 09:47:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2304