Bug 1478366 - Crash noticed during IPA upgrade process due to ipa package.
Summary: Crash noticed during IPA upgrade process due to ipa package.
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa
Version: 7.4
Hardware: Unspecified
OS: Unspecified
Target Milestone: rc
: ---
Assignee: IPA Maintainers
QA Contact: ipa-qe
Aneta Šteflová Petrová
Depends On:
TreeView+ depends on / blocked
Reported: 2017-08-04 12:46 UTC by Nikhil Dehadrai
Modified: 2018-10-30 10:56 UTC (History)
10 users (show)

Fixed In Version: ipa-4.6.4-1.el7
Doc Type: Known Issue
Doc Text:
A crash is reported after an unsuccessful lightweight CA key retrieval When using Identity Management (IdM), if retrieving the lightweight certificate authority (CA) key fails for some reason, the operation terminates unexpectedly with an uncaught exception. The exception results in a crash report.
Clone Of:
Last Closed: 2018-10-30 10:56:00 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:3187 0 None None None 2018-10-30 10:56:48 UTC

Description Nikhil Dehadrai 2017-08-04 12:46:58 UTC
Description of problem:
While upgrading IPA server from RHEL 7.4( to RHEL 7.4.1(4.5.0-21.el7_4.1), found crash report related to package ipa-server-4.5.0-21.el7.

Full Backtrace:
models.py:834:raise_for_status:HTTPError: 401 Client Error: Unauthorized

Traceback (most recent call last):
  File "/usr/libexec/ipa/ipa-pki-retrieve-key", line 32, in <module>
    print(client.fetch_key(keyname, store=False))
  File "/usr/lib/python2.7/site-packages/ipaserver/secrets/client.py", line 101, in fetch_key
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 834, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 401 Client Error: Unauthorized

Local variables in innermost frame:
self: <Response [401]>
http_error_msg: '401 Client Error: Unauthorized'

Additional information:
:JAVA_OPTS=-DRESTEASY_LIB=/usr/share/java/resteasy-base -Djava.library.path=/usr/lib64/nuxwdog-jni

Comment 3 Fraser Tweedale 2017-08-08 12:54:42 UTC
Probably related to Custodia.  Could you please provide
/var/log/pki/pki-tomcat/ca/debug and also describe what is the

Comment 4 Nikhil Dehadrai 2017-08-14 06:59:36 UTC
Hi Fraseer / Petr,

As far as impact is considered, not entirely sure at this point of time. Also another thing to note here is the crash is noticed before upgrade is run.

Please find attached debug log as requested. I also noticed avc's during this  run for which I have already logged bz1478371, if that helps.

Let me know, if you need anymore information

Comment 6 Petr Vobornik 2017-08-14 07:44:08 UTC
Probably the related AVC mentioned (but the time is not the same) in bug 1478371 is:

time->Fri Aug  4 09:14:59 2017
type=PROCTITLE msg=audit(1501852499.897:1007): proctitle=2F7573722F62696E2F707974686F6E32002F7573722F6C6962657865632F6970612F6970612D706B692D72657472696576652D6B65790063615369676E696E674365727420636572742D706B692D63612037353339353237612D306366372D346564372D626536322D323962373064613931323963006175746F2D68762D3031
type=SYSCALL msg=audit(1501852499.897:1007): arch=c000003e syscall=4 success=no exit=-13 a0=18267a0 a1=7ffee9b7f720 a2=7ffee9b7f720 a3=4 items=0 ppid=26998 pid=1626 auid=4294967295 uid=17 gid=17 euid=17 suid=17 fsuid=17 egid=17 sgid=17 fsgid=17 tty=(none) ses=4294967295 comm="ipa-pki-retriev" exe="/usr/bin/python2.7" subj=system_u:system_r:tomcat_t:s0 key=(null)
type=AVC msg=audit(1501852499.897:1007): avc:  denied  { getattr } for  pid=1626 comm="ipa-pki-retriev" path="/etc/openldap/certs" dev="dm-0" ino=100951062 scontext=system_u:system_r:tomcat_t:s0 tcontext=system_u:object_r:slapd_cert_t:s0 tclass=dir

Comment 10 Fraser Tweedale 2017-08-15 08:45:10 UTC
It looks like the ipa-pki-retrieve-key program is crashing due to the
ipa-custodia server returning non-200 response.

This isn't really a problem, there are a variety of conditions that can
cause this e.g. LDAP server being temporarily down (as in the above

IMO it is not a serious issue; the "fix" is to catch all exceptions and
exit cleanly with nonzero exit status instead of allowing the
interpreter to crash with uncaught exception.

Comment 11 Petr Vobornik 2017-08-17 14:51:00 UTC
Fraser, do we know when ipa-pki-retrieve-key was called and if it was OK/expected for it to fail? I.e. are we sure that it doesn't break any functionality.

Comment 12 Fraser Tweedale 2017-08-17 23:02:17 UTC
It was called to replicate lightweight CA signing keys.  Without knowing
more about the history of the topology in question it's impossible to say
whether the invocation of the ipa-pki-retrieve-key is expected or not,
or the root cause as to why key retrieval is not succeeding.

Comment 13 Petr Vobornik 2017-08-18 16:24:34 UTC
Could you point us to time when this crash happen in relation to rpm upgrade process/ipa-server-upgrade? Or part of the beaker job where it happened?

Comment 16 Fraser Tweedale 2017-08-23 01:22:09 UTC
Upstream ticket for dealing specifically with the crashes:

Comment 17 Fraser Tweedale 2017-08-23 10:57:01 UTC
Related Dogtag issue (regression in LWCA key replication):

Comment 18 Standa Laznicka 2017-09-14 06:01:17 UTC
Fixed upstream

ipa-pki-retrieve-key should not be crashing anymore, although from the discussion I see that might have not been the root cause of the problem, so I am not setting this BZ to POST.

If you think otherwise, you can change the status.

Comment 19 Nikhil Dehadrai 2017-09-25 09:53:09 UTC
Noticed similar crash during ipa-server upgrade process from :
1) RHEL 7.4-0day > RHEL 7.4 update2
2) RHEL 7.4 update1 > RHEL 7.4 update2

Comment 27 Sudhir Menon 2018-09-06 06:39:44 UTC
Crash is not seen while performing below upgrade path.

1. 7.4.z to 7.6

Sep 05 05:02:35 Installed: ipa-server-4.5.0-22.el7_4.x86_64
Sep 05 06:39:03 Updated: ipa-server-4.6.4-8.el7.x86_64

[root@master abrt]# pwd
[root@master abrt]# ls -l
total 0

2. 7.5.z to 7.6
Sep 06 01:26:42 Installed: ipa-server-4.5.4-10.el7_5.4.3.x86_64
Sep 06 01:57:07 Updated: ipa-server-4.6.4-8.el7.x86_64

[root@master abrt]# pwd
[root@master abrt]# ls -l
total 0

Hence marking the bug as VERIFIED.

Comment 29 errata-xmlrpc 2018-10-30 10:56:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.