Bug 1328522

Summary: [abrt] [faf] pki-core: get_server_tracker(): /usr/lib/python2.7/site-packages/pki/server/upgrade.py killed by TypeError
Product: Red Hat Enterprise Linux 7 Reporter: Nikhil Dehadrai <ndehadra>
Component: pki-coreAssignee: Matthew Harmsen <mharmsen>
Status: CLOSED DUPLICATE QA Contact: Asha Akkiangady <aakkiang>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.2CC: abokovoy, arubin, cheimes, edewata, ekeck, lmiksik, ndehadra, nkinder, pasik, pvoborni, rcritten, yamakasi.014
Target Milestone: rcKeywords: Regression, Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
URL: http://faf-report.itos.redhat.com/reports/bthash/da94500e4ee43a6ea03c8319a8395dfc735ed1af/
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-26 21:31:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nikhil Dehadrai 2016-04-19 14:57:06 UTC 
This bug has been created based on an anonymous crash report requested by the package maintainer.

Report URL: http://faf-report.itos.redhat.com/reports/bthash/da94500e4ee43a6ea03c8319a8395dfc735ed1af/
Comment 1 Nikhil Dehadrai 2016-04-19 15:05:20 UTC 
Noticed this crash behavior on performing following upgrade tests:
=====================================================================
1. Setup IPA server for ipa-upgrade test suite execution on beaker (In this case its 7.1.z > 7.2 up4). 
2. Make sure latest repo links are set correctly for test execution.
3. Initiate automation for ipa-upgrade test suite on beaker.


Observation:
==============
cmdline:        /usr/bin/python /sbin/pki-server-upgrade --silent
uid:            0 (root)
abrt_version:   2.1.11
dso_list:       pki-base-10.1.2-7.el7.noarch
event_log:      
executable:     /sbin/pki-server-upgrade
hostname:       auto-hv-01-guest06.testrelm.test
kernel:         3.10.0-229.20.1.el7.x86_64
last_occurrence: 1461076764
pid:            11161
pkg_arch:       noarch
pkg_epoch:      0
pkg_name:       pki-server
pkg_release:    7.el7
pkg_version:    10.1.2
runlevel:       N 3
username:       root


backtrace:
:upgrade.py:259:get_server_tracker:TypeError: unhashable type: 'PKIInstance'
:
:Traceback (most recent call last):
:  File "/sbin/pki-server-upgrade", line 194, in <module>
:    main(sys.argv)
:  File "/sbin/pki-server-upgrade", line 187, in main
:    upgrader.upgrade()
:  File "/usr/lib/python2.7/site-packages/pki/upgrade.py", line 650, in upgrade
:    versions = self.versions()
:  File "/usr/lib/python2.7/site-packages/pki/upgrade.py", line 483, in versions
:    current_version = self.get_current_version()
:  File "/usr/lib/python2.7/site-packages/pki/server/upgrade.py", line 299, in get_current_version
:    tracker = self.get_server_tracker(instance, subsystem)
:  File "/usr/lib/python2.7/site-packages/pki/server/upgrade.py", line 259, in get_server_tracker
:    tracker = self.subsystem_trackers[instance]
:TypeError: unhashable type: 'PKIInstance'


Additional information:
=======================
This issue was also observed for upgrade path 7.1(GA) to 7.2 up4.
Comment 4 Petr Vobornik 2016-04-20 10:14:32 UTC 
*** Bug 1328602 has been marked as a duplicate of this bug. ***
Comment 5 Petr Vobornik 2016-04-20 10:17:50 UTC 
This is a PKI bug which breaks also IPA upgrade with a result that CA won't start, see bug 1328602. Christian, thanks for investigation. 

This needs to be fixed in u4.
Comment 6 Petr Vobornik 2016-04-20 11:20:56 UTC 
bug 1328807 is probably a dup as well
Comment 7 Endi Sukma Dewata 2016-04-20 22:57:34 UTC 
According to cheimes it's already fixed upstream in this ticket:
https://fedorahosted.org/pki/ticket/2216
Comment 8 Matthew Harmsen 2016-04-20 23:00:32 UTC 
Upstream ticket:
https://fedorahosted.org/pki/ticket/2216
Comment 9 Matthew Harmsen 2016-04-20 23:05:53 UTC 
Pushed to master d5360c915761773e93333b6894df4bd8ac8a2f13
Comment 10 Matthew Harmsen 2016-04-20 23:06:36 UTC 
Checked into DOGTAG_10_2_6_BRANCH:

    9c83d154939f00011c9682ff1ccaabb78626c99f 

Checked into DOGTAG_10_2_BRANCH:

    7a9612411834649cac2089afc6c19942d8062ab7 

Resolved in:

    ​pki-core-10.2.6-17.fc23
Comment 11 Nathan Kinder 2016-04-20 23:31:23 UTC 
Business Justification:
This is a regression that was introduced in a 7.2.z batch update build for pki-core (still trying to determine exactly which build).  This bug breaks IdM upgrades, resulting in IdM services that won't start after updating packages.  We should fix this as quickly as possible (ideally in the batch4 update).
Comment 14 Matthew Harmsen 2016-04-21 18:38:52 UTC 

*** This bug has been marked as a duplicate of bug 1301546 ***
Comment 15 Matthew Harmsen 2016-04-21 18:42:40 UTC 

*** This bug has been marked as a duplicate of bug 1318302 ***
Comment 16 Ade Lee 2016-05-05 19:43:08 UTC 
*** Bug 1328807 has been marked as a duplicate of this bug. ***
Comment 17 Yamakasi 2016-07-05 18:12:50 UTC 
I see this happening again on VERSION: 4.2.0, API_VERSION: 2.156

Is this known and how to fix ?

I'm kinda puzzled.
Comment 19 Yamakasi 2016-07-08 10:35:06 UTC 
I uploaded my ipaupgrade.log
Comment 20 Yamakasi 2016-07-08 10:36:26 UTC 
Created attachment 1177608 [details]
ipaupgrade.log from CentOS Linux release 7.2.1511 (Core)
Comment 22 Christian Heimes 2016-07-11 19:34:47 UTC 
The problem should be fixed by Patch17: pki-core-Make-PKIInstance-and-PKISubsystem-hashable.patch. The patch was added in pki-core-10.2.5-10.

I don't see a TypeError in the upgrade log. The upgrade fails with some Java exception and followed by a final Python exception in IPA:


javax.servlet.ServletException: Servlet.init() for servlet Resteasy threw exception
        org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:610)
        org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
        org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
        org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
        org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
        org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        java.lang.Thread.run(Thread.java:745)

java.lang.RuntimeException: java.lang.ClassNotFoundException: com.netscape.ca.CertificateAuthorityApplication
        org.jboss.resteasy.spi.ResteasyDeployment.createApplication(ResteasyDeployment.java:292)
        org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:95)
        org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:36)
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        java.lang.reflect.Method.invoke(Method.java:498)
        org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:277)
        org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:274)
        java.security.AccessController.doPrivileged(Native Method)
        javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
        org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:309)
        org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:169)
        org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:123)
        org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:610)
        org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
        org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
        org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
        org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
        org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        java.lang.Thread.run(Thread.java:745)

java.lang.ClassNotFoundException: com.netscape.ca.CertificateAuthorityApplication
        org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720)
        org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
        org.jboss.resteasy.spi.ResteasyDeployment.createApplication(ResteasyDeployment.java:288)
        org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:95)
        org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:36)
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        java.lang.reflect.Method.invoke(Method.java:498)
        org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:277)
        org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:274)
        java.security.AccessController.doPrivileged(Native Method)
        javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
        org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:309)
        org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:169)
        org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:123)
        org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:610)
        org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
        org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
        org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
        org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
        org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        java.lang.Thread.run(Thread.java:745)


2016-07-08T10:25:13Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 48, in run
    server.upgrade()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1618, in upgrade
    upgrade_configuration()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1548, in upgrade_configuration
    ca_enable_ldap_profile_subsystem(ca)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 341, in ca_enable_ldap_profile_subsystem
    cainstance.migrate_profiles_to_ldap(caconfig)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1868, in migrate_profiles_to_ldap
    _create_dogtag_profile(profile_id, profile_data, overwrite=False)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 1874, in _create_dogtag_profile
    with api.Backend.ra_certprofile as profile_api:
  File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 2038, in __enter__
    raise errors.RemoteRetrieveError(reason=_('Failed to authenticate to CA REST API'))
Comment 24 Endi Sukma Dewata 2016-07-25 21:48:26 UTC 
Hi Yamakasi,

Do you mind opening a separate bug about the "ClassNotFoundException"? I think that issue is different from the "TypeError: unhashable type: 'PKIInstance'" that was originally reported in this bug (and has been fixed in another bug). Could you also provide the steps to reproduce the problem? Thanks.
Comment 27 Endi Sukma Dewata 2016-07-26 21:31:03 UTC 
The TypeError problem was already fixed as part of bug #1318302 and the problem is no longer reproducible. According to cheimes the ClassNotFoundException is a symptom of the same problem:

https://bugzilla.redhat.com/show_bug.cgi?id=1328602#c8

I'm closing this bug as a duplicate.

*** This bug has been marked as a duplicate of bug 1318302 ***