Bug 1478366 - Crash noticed during IPA upgrade process due to ipa package.
Crash noticed during IPA upgrade process due to ipa package.
Status: POST
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa (Show other bugs)
7.4
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: IPA Maintainers
ipa-qe
Aneta Šteflová Petrová
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-04 08:46 EDT by Nikhil Dehadrai
Modified: 2017-12-12 12:07 EST (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Known Issue
Doc Text:
A crash is reported after an unsuccessful lightweight CA key retrieval When using Identity Management (IdM), if retrieving the lightweight certificate authority (CA) key fails for some reason, the operation terminates unexpectedly with an uncaught exception. The exception results in a crash report.
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Nikhil Dehadrai 2017-08-04 08:46:58 EDT
Description of problem:
While upgrading IPA server from RHEL 7.4(4.5.0.21) to RHEL 7.4.1(4.5.0-21.el7_4.1), found crash report related to package ipa-server-4.5.0-21.el7.


Full Backtrace:
models.py:834:raise_for_status:HTTPError: 401 Client Error: Unauthorized

Traceback (most recent call last):
  File "/usr/libexec/ipa/ipa-pki-retrieve-key", line 32, in <module>
    print(client.fetch_key(keyname, store=False))
  File "/usr/lib/python2.7/site-packages/ipaserver/secrets/client.py", line 101, in fetch_key
    r.raise_for_status()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 834, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 401 Client Error: Unauthorized

Local variables in innermost frame:
self: <Response [401]>
http_error_msg: '401 Client Error: Unauthorized'

Additional information:
environ:
:USE_NUXWDOG=false
:JAVA_OPTS=-DRESTEASY_LIB=/usr/share/java/resteasy-base -Djava.library.path=/usr/lib64/nuxwdog-jni
:LOGNAME=pkiuser
:USER=pkiuser
:TOMCAT7_USER=pkiuser
:TOMCAT_LOG=/var/log/pki/pki-tomcat/tomcat-initd.log
:CATALINA_HOME=/usr/share/tomcat
:LANG=en_US.UTF-8
:SHELL=/sbin/nologin
:CATALINA_TMPDIR=/var/lib/pki/pki-tomcat/temp
:SHLVL=0
:NSS_ENABLE_PKIX_VERIFY=1
:TOMCAT7_SECURITY=true
:JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk
:HOME=/usr/share/pki
:TOMCATS_BASE=/var/lib/tomcats/
:PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
:TOMCAT_USER=pkiuser
:NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat
:SECURITY_MANAGER=true
:NAME=pki-tomcat
:TOMCAT_CFG_LOADED=1
:OLDPWD=/
:CATALINA_PID=/var/run/pki/tomcat/pki-tomcat.pid
:CATALINA_BASE=/var/lib/pki/pki-tomcat
:PWD=/usr/share/tomcat
:PKI_VERSION=10.4.1
:XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt
Comment 3 Fraser Tweedale 2017-08-08 08:54:42 EDT
Probably related to Custodia.  Could you please provide
/var/log/pki/pki-tomcat/ca/debug and also describe what is the
impact?
Comment 4 Nikhil Dehadrai 2017-08-14 02:59:36 EDT
Hi Fraseer / Petr,

As far as impact is considered, not entirely sure at this point of time. Also another thing to note here is the crash is noticed before upgrade is run.

Please find attached debug log as requested. I also noticed avc's during this  run for which I have already logged bz1478371, if that helps.

Let me know, if you need anymore information
Comment 6 Petr Vobornik 2017-08-14 03:44:08 EDT
Probably the related AVC mentioned (but the time is not the same) in bug 1478371 is:

time->Fri Aug  4 09:14:59 2017
type=PROCTITLE msg=audit(1501852499.897:1007): proctitle=2F7573722F62696E2F707974686F6E32002F7573722F6C6962657865632F6970612F6970612D706B692D72657472696576652D6B65790063615369676E696E674365727420636572742D706B692D63612037353339353237612D306366372D346564372D626536322D323962373064613931323963006175746F2D68762D3031
type=SYSCALL msg=audit(1501852499.897:1007): arch=c000003e syscall=4 success=no exit=-13 a0=18267a0 a1=7ffee9b7f720 a2=7ffee9b7f720 a3=4 items=0 ppid=26998 pid=1626 auid=4294967295 uid=17 gid=17 euid=17 suid=17 fsuid=17 egid=17 sgid=17 fsgid=17 tty=(none) ses=4294967295 comm="ipa-pki-retriev" exe="/usr/bin/python2.7" subj=system_u:system_r:tomcat_t:s0 key=(null)
type=AVC msg=audit(1501852499.897:1007): avc:  denied  { getattr } for  pid=1626 comm="ipa-pki-retriev" path="/etc/openldap/certs" dev="dm-0" ino=100951062 scontext=system_u:system_r:tomcat_t:s0 tcontext=system_u:object_r:slapd_cert_t:s0 tclass=dir
Comment 10 Fraser Tweedale 2017-08-15 04:45:10 EDT
It looks like the ipa-pki-retrieve-key program is crashing due to the
ipa-custodia server returning non-200 response.

This isn't really a problem, there are a variety of conditions that can
cause this e.g. LDAP server being temporarily down (as in the above
case).

IMO it is not a serious issue; the "fix" is to catch all exceptions and
exit cleanly with nonzero exit status instead of allowing the
interpreter to crash with uncaught exception.
Comment 11 Petr Vobornik 2017-08-17 10:51:00 EDT
Fraser, do we know when ipa-pki-retrieve-key was called and if it was OK/expected for it to fail? I.e. are we sure that it doesn't break any functionality.
Comment 12 Fraser Tweedale 2017-08-17 19:02:17 EDT
It was called to replicate lightweight CA signing keys.  Without knowing
more about the history of the topology in question it's impossible to say
whether the invocation of the ipa-pki-retrieve-key is expected or not,
or the root cause as to why key retrieval is not succeeding.
Comment 13 Petr Vobornik 2017-08-18 12:24:34 EDT
Could you point us to time when this crash happen in relation to rpm upgrade process/ipa-server-upgrade? Or part of the beaker job where it happened?
Comment 16 Fraser Tweedale 2017-08-22 21:22:09 EDT
Upstream ticket for dealing specifically with the crashes:
https://pagure.io/freeipa/issue/7115
Comment 17 Fraser Tweedale 2017-08-23 06:57:01 EDT
Related Dogtag issue (regression in LWCA key replication):
https://bugzilla.redhat.com/show_bug.cgi?id=1484359
Comment 18 Stanislav Laznicka 2017-09-14 02:01:17 EDT
Fixed upstream
master:
https://pagure.io/freeipa/c/09f746f56823ec6120437ba625f0db9b5d704e3e
ipa-4-6:
https://pagure.io/freeipa/c/f9074dcc9c025594d6961dcbb03805b9a20bb220

ipa-pki-retrieve-key should not be crashing anymore, although from the discussion I see that might have not been the root cause of the problem, so I am not setting this BZ to POST.

If you think otherwise, you can change the status.
Comment 19 Nikhil Dehadrai 2017-09-25 05:53:09 EDT
Noticed similar crash during ipa-server upgrade process from :
1) RHEL 7.4-0day > RHEL 7.4 update2
2) RHEL 7.4 update1 > RHEL 7.4 update2

Note You need to log in before you can comment on or make changes to this bug.