Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
First Last Prev Next    This bug is not in your last search results.
Bug 1438016 - gssapi errors after IPA server upgrade
gssapi errors after IPA server upgrade
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa (Show other bugs)
7.4
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: IPA Maintainers
Nikhil Dehadrai
:
Depends On: 1438390
Blocks:
  Show dependency tree / graph
 
Reported: 2017-03-31 11:59 EDT by Scott Poore
Modified: 2017-08-01 05:47 EDT (History)
7 users (show)

See Also:
Fixed In Version: ipa-4.5.0-16.el7
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1438390 (view as bug list)
Environment:
Last Closed: 2017-08-01 05:47:49 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:2304 normal SHIPPED_LIVE ipa bug fix and enhancement update 2017-08-01 08:41:35 EDT

  None (edit)
Description Scott Poore 2017-03-31 11:59:34 EDT 
Description of problem:

After upgrading the IPA server from IPA version 4.4 (rhel7.3) to 4.5 (rhel7.4), I'm seeing gssapi errors and internal server error's when running commands.

After upgrade:

[root@auto-hv-02-guest10 /]# ipa user-find
ipa: ERROR: cannot connect to 'https://auto-hv-02-guest10.isc03081.test/ipa/json': Internal Server Error

In /var/log/httpd/error_log I see a backtrace and a lot of these:


   File "misc.pyx", line 174, in gssapi.raw.misc._display_status (gssapi/raw/misc.c:1797)
   File "misc.pyx", line 216, in gssapi.raw.misc.GSSErrorRegistry.__call__ (gssapi/raw/misc.c:2719)
   File "misc.pyx", line 272, in gssapi.raw.misc.GSSError.__init__ (gssapi/raw/misc.c:3143)
   File "misc.pyx", line 324, in gssapi.raw.misc.GSSError.gen_message (gssapi/raw/misc.c:3862)
   File "misc.pyx", line 291, in gssapi.raw.misc.GSSError.get_all_statuses (gssapi/raw/misc.c:3298)
 RuntimeError: maximum recursion depth exceeded while calling a Python object


Version-Release number of selected component (if applicable):
ipa-server-4.5.0-4.el7.x86_64
gssproxy-0.7.0-3.el7.x86_64
mod_wsgi-3.4-12.el7_0.x86_64


How reproducible:
Unknown.


Steps to Reproduce:
1.  on rhel7.3 install 4.4 ipa server
2.  upgrade to 4.5 version
3.  ipa user-find

Actual results:
internal server error

Expected results:
find users and not errors.

Additional info:

/var/log/httpd/error_log entries:

[Fri Mar 31 11:41:18.567914 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 274, in route
[Fri Mar 31 11:41:18.567933 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     return app(environ, start_response)
[Fri Mar 31 11:41:18.567941 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 468, in __call__
[Fri Mar 31 11:41:18.567965 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     response = super(jsonserver, self).__call__(environ, start_response)
[Fri Mar 31 11:41:18.567974 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 648, in __call__
[Fri Mar 31 11:41:18.567988 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     self.create_context(ccache=user_ccache)
[Fri Mar 31 11:41:18.568007 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 125, in create_context
[Fri Mar 31 11:41:18.568076 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     time_limit=None)
[Fri Mar 31 11:41:18.568088 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 66, in connect
[Fri Mar 31 11:41:18.568104 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     conn = self.create_connection(*args, **kw)
[Fri Mar 31 11:41:18.568113 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipaserver/plugins/ldap2.py", line 205, in create_connection
[Fri Mar 31 11:41:18.568248 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     principal = krb_utils.get_principal(ccache_name=ccache)
[Fri Mar 31 11:41:18.568266 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipalib/krb_utils.py", line 168, in get_principal
[Fri Mar 31 11:41:18.568334 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     creds = get_credentials(ccache_name=ccache_name)
[Fri Mar 31 11:41:18.568361 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib/python2.7/site-packages/ipalib/krb_utils.py", line 147, in get_credentials
[Fri Mar 31 11:41:18.568379 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     return gssapi.Credentials(usage='initiate', name=name, store=store)
[Fri Mar 31 11:41:18.568395 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__
[Fri Mar 31 11:41:18.568519 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     store=store)
[Fri Mar 31 11:41:18.568530 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acquire
[Fri Mar 31 11:41:18.568547 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]     usage)
[Fri Mar 31 11:41:18.568554 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cred_from (gssapi/raw/ext_cred_store.c:1726)
[Fri Mar 31 11:41:18.568657 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "misc.pyx", line 216, in gssapi.raw.misc.GSSErrorRegistry.__call__ (gssapi/raw/misc.c:2719)
[Fri Mar 31 11:41:18.568737 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "misc.pyx", line 272, in gssapi.raw.misc.GSSError.__init__ (gssapi/raw/misc.c:3143)
[Fri Mar 31 11:41:18.568814 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "misc.pyx", line 325, in gssapi.raw.misc.GSSError.gen_message (gssapi/raw/misc.c:3890)
[Fri Mar 31 11:41:18.568891 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "misc.pyx", line 295, in gssapi.raw.misc.GSSError.get_all_statuses (gssapi/raw/misc.c:3360)
[Fri Mar 31 11:41:18.568979 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84]   File "misc.pyx", line 174, in gssapi.raw.misc._display_status (gssapi/raw/misc.c:1797)
[Fri Mar 31 11:41:18.603558 2017] [:error] [pid 25246] [remote ipa_master_ip_address:84] RuntimeError: maximum recursion depth exceeded while calling a Python object
Comment 2 Martin Babinsky 2017-04-03 03:45:34 EDT 
Looks like https://pagure.io/freeipa/issue/6796 to me.
Comment 3 Petr Vobornik 2017-04-03 06:34:15 EDT 
This bz was cloned to python-gssapi as triage of upstream IPA 6796 suggests.

The main fix is in python-gssapi but also a sanity fix should be done on IPA side.
Comment 4 Petr Vobornik 2017-04-13 12:31:34 EDT 
Upstream ticket:
https://pagure.io/freeipa/issue/6796
Comment 5 Petr Vobornik 2017-05-05 11:03:36 EDT 
Blocking bug 1438390 was fixed so we can also raise requires to python-gssapi-1.2.0-3.el7
Comment 9 Nikhil Dehadrai 2017-06-07 06:43:45 EDT 
IPA-server-version: ipa-server-4.5.0-15.el7.x86_64

Verified the bug on the basis of below observations:
1. Verified that upgrade of IPA-MASTER is successful.
2. Verified that after upgrade commands "ipa user-find, ipa user-show, ipa host-find" are run successfully without any errors.
3. Also no error are observed inside "/var/log/httpd/error_log".
4. Verified the same for following upgrade paths:
- Rhel 7.3.z > 7.4
- Rhel 7.3GA > 7.4
- Rhel 7.2.z > 7.4
- Rhel 7.1.z > 7.4- Upgrade fails for which a separate bug is updated BZ#1438731 (comment#6)
5. Refer console output from one of the upgrade paths:(RHel 7.3.z > 7.4)

[root@inferno ~]# tail -1 /var/log/ipaupgrade.log 
2017-06-07T10:03:21Z INFO The ipa-server-upgrade command was successful

[root@inferno ~]# rpm -q ipa-server
ipa-server-4.5.0-15.el7.x86_64

[root@inferno ~]# kinit admin
Password for admin@TESTRELM.TEST: 

[root@inferno ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful

[root@inferno ~]# ipactl restart
Stopping pki-tomcatd Service
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting ntpd Service
Restarting pki-tomcatd Service
Restarting ipa-otpd Service
Restarting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful

[root@inferno ~]# ipa user-find
--------------
1 user matched
--------------
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin@TESTRELM.TEST
  UID: 1075400000
  GID: 1075400000
  Account disabled: False
----------------------------
Number of entries returned 1
----------------------------
[root@inferno ~]# ipa host-find
--------------
1 host matched
--------------
  Host name: inferno.testrelm.test
  Principal name: host/inferno.testrelm.test@TESTRELM.TEST
  Principal alias: host/inferno.testrelm.test@TESTRELM.TEST
  SSH public key fingerprint: SHA256:LF8wIaQeKN6ww4llCkbPs6IuinEPL1O9At2QpyE23Qw (ssh-rsa),
                              SHA256:8jo0PBAD920N1MPQ/Kns9cspcu97gixeAvatoNbc4o0 (ssh-ed25519),
                              SHA256:8Yi1pl7+Nm8jaBwDDI3mjGnxVFqehziZ1CedR8sLjI0 (ecdsa-
                              sha2-nistp256)
----------------------------
Number of entries returned 1
----------------------------

[root@inferno ~]# ipa user-show
User login: admin
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin@TESTRELM.TEST
  UID: 1075400000
  GID: 1075400000
  Account disabled: False
  Password: True
  Member of groups: admins, trust admins
  Kerberos keys available: True

[root@inferno ~]# cat /var/log/httpd/error_log | grep -rn "maximum recursion depth"
[root@inferno ~]# cat /var/log/httpd/error_log | grep -rn "recursion"
[root@inferno ~]# cat /var/log/httpd/error_log | grep -rn "maximum"
[root@inferno ~]# cat /var/log/httpd/error_log | grep -rn "gssapi"
[root@inferno ~]# cat /var/log/httpd/error_log | grep -rn "GSSError"

Thus on the basis of above observations, marking status of bug to "VERIFIED".
Comment 11 Martin Bašti 2017-06-07 09:26:21 EDT 
ipa-4-5:

* 15d5ddd417d801a2356dcb043feef1aed8f76a25 Bump version of python-gssapi
Comment 13 Nikhil Dehadrai 2017-06-12 01:04:30 EDT 
IPA-server-version: ipa-server-4.5.0-16.el7.x86_64

Verified the bug on the basis of below observations:
1. Verified that upgrade of IPA-MASTER is successful.
2. Verified that after upgrade commands "ipa user-find, ipa user-show, ipa host-find" are run successfully without any errors.
3. Also no errors are observed inside "/var/log/httpd/error_log".
4. Verified the same for following upgrade paths:
- Rhel 7.3.z > 7.4
- Rhel 7.3GA > 7.4
- Rhel 7.2.z > 7.4
- Rhel 7.1.z > 7.4- Upgrade fails for which a separate bug is updated BZ#1438731 (comment#6)
5. Refer console output from one of the upgrade paths:(Rhel 7.3.z > 7.4)

[root@auto-hv-01-guest03 ~]# rpm -q ipa-server
ipa-server-4.5.0-16.el7.x86_64

[root@auto-hv-01-guest03 ~]# rpm -q python-gssapi
python-gssapi-1.2.0-3.el7.x86_64

[root@auto-hv-01-guest03 ~]# tail -1 /var/log/ipaupgrade.log 
2017-06-12T04:47:06Z INFO The ipa-server-upgrade command was successful

[root@auto-hv-01-guest03 ~]# kinit admin
Password for admin@TESTRELM.TEST: 

[root@auto-hv-01-guest03 ~]# ipa user-find
--------------
1 user matched
--------------
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin@TESTRELM.TEST
  UID: 579000000
  GID: 579000000
  Account disabled: False
----------------------------
Number of entries returned 1
----------------------------

[root@auto-hv-01-guest03 ~]# ipa user-find tuser
--------------
1 user matched
--------------
  User login: tuser
  First name: test
  Last name: user
  Home directory: /home/tuser
  Login shell: /bin/sh
  Principal name: tuser@TESTRELM.TEST
  Principal alias: tuser@TESTRELM.TEST
  Email address: tuser@testrelm.test
  UID: 579000001
  GID: 579000001
  Account disabled: False
----------------------------
Number of entries returned 1
----------------------------
[root@auto-hv-01-guest03 ~]# ipa user-find tuser --all
--------------
1 user matched
--------------
  dn: uid=tuser,cn=users,cn=accounts,dc=testrelm,dc=test
  User login: tuser
  First name: test
  Last name: user
  Full name: test user
  Display name: test user
  Initials: tu
  Home directory: /home/tuser
  GECOS: test user
  Login shell: /bin/sh
  Principal name: tuser@TESTRELM.TEST
  Principal alias: tuser@TESTRELM.TEST
  Email address: tuser@testrelm.test
  UID: 579000001
  GID: 579000001
  Account disabled: False
  Preserved user: False
  Member of groups: ipausers
  ipauniqueid: 63edee6a-4f2c-11e7-80ae-525400cc38fd
  mepmanagedentry: cn=tuser,cn=groups,cn=accounts,dc=testrelm,dc=test
  objectclass: top, person, organizationalperson, inetorgperson, inetuser, posixaccount,
               krbprincipalaux, krbticketpolicyaux, ipaobject, ipasshuser, ipaSshGroupOfPubKeys,
               mepOriginEntry
----------------------------
Number of entries returned 1
----------------------------
[root@auto-hv-01-guest03 ~]# ipa user-find tuser --raw
--------------
1 user matched
--------------
  uid: tuser
  givenname: test
  sn: user
  homedirectory: /home/tuser
  loginshell: /bin/sh
  krbcanonicalname: tuser@TESTRELM.TEST
  krbprincipalname: tuser@TESTRELM.TEST
  mail: tuser@testrelm.test
  uidnumber: 579000001
  gidnumber: 579000001
  nsaccountlock: FALSE
----------------------------
Number of entries returned 1
----------------------------

[root@auto-hv-01-guest03 ~]# ipa user-show
User login: admin
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin@TESTRELM.TEST
  UID: 579000000
  GID: 579000000
  Account disabled: False
  Password: True
  Member of groups: admins, trust admins
  Kerberos keys available: True
[root@auto-hv-01-guest03 ~]# ipa host-find
--------------
1 host matched
--------------
  Host name: auto-hv-01-guest03.testrelm.test
  Principal name: host/auto-hv-01-guest03.testrelm.test@TESTRELM.TEST
  Principal alias: host/auto-hv-01-guest03.testrelm.test@TESTRELM.TEST
  SSH public key fingerprint: SHA256:81w5bMII4U0OBeCkwFrUSMvqCXuPGaTwj0v0DP51EWc (ssh-rsa),
                              SHA256:hMRDycHsxmY+M3JDMzwuV6RwrJzLKr6f5HOvqKOEX+Q (ecdsa-
                              sha2-nistp256), SHA256:9t9sRoJT5n4svMoMW2f2ok9ubc/UIgxoA+4NTqrmRB0
                              (ssh-ed25519)
----------------------------
Number of entries returned 1
----------------------------
[root@auto-hv-01-guest03 ~]# cat /var/log/httpd/error_log | grep -rn "maximum recursion depth"
[root@auto-hv-01-guest03 ~]# cat /var/log/httpd/error_log | grep -rn "recursion"
[root@auto-hv-01-guest03 ~]# cat /var/log/httpd/error_log | grep -rn "maximum"
[root@auto-hv-01-guest03 ~]# cat /var/log/httpd/error_log | grep -rn "gssapi"
[root@auto-hv-01-guest03 ~]# cat /var/log/httpd/error_log | grep -rn "GSSError"
[root@auto-hv-01-guest03 ~]# 

Thus on the basis of above observations, marking status of bug to "VERIFIED".
Comment 14 errata-xmlrpc 2017-08-01 05:47:49 EDT 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2304
Note You need to log in before you can comment on or make changes to this bug.
First Last Prev Next    This bug is not in your last search results.