Bug 1455541 - after upgrade login from web ui breaks [NEEDINFO]
Summary: after upgrade login from web ui breaks
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa
Version: 7.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: IPA Maintainers
QA Contact: Nikhil Dehadrai
URL:
Whiteboard:
Depends On:
Blocks: 1452216
TreeView+ depends on / blocked
 
Reported: 2017-05-25 12:23 UTC by Petr Vobornik
Modified: 2017-08-01 09:51 UTC (History)
10 users (show)

Fixed In Version: ipa-4.5.0-15.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-01 09:51:24 UTC
Target Upstream Version:
pvoborni: needinfo? (ndehadra)


Attachments (Terms of Use)
Login-success-after-upgrade (44.72 KB, image/png)
2017-06-13 05:46 UTC, Nikhil Dehadrai
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:2304 normal SHIPPED_LIVE ipa bug fix and enhancement update 2017-08-01 12:41:35 UTC

Description Petr Vobornik 2017-05-25 12:23:08 UTC
Cloned from upstream: https://pagure.io/freeipa/issue/6973

Apparently kdc.crt is readable only by root now,. However the from based auth code uses it for anchors when users autenthicate locally.
So after updating my test VMs to latest master I see this when I try to auth:
[Mon May 22 14:36:37.274858 2017] [wsgi:error] [pid 26269] [remote 192.168.123.1:45548] ipa: DEBUG: args=/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_26269 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
[Mon May 22 14:36:37.287311 2017] [wsgi:error] [pid 26269] [remote 192.168.123.1:45548] ipa: DEBUG: Process finished, return code=1
[Mon May 22 14:36:37.287379 2017] [wsgi:error] [pid 26269] [remote 192.168.123.1:45548] ipa: DEBUG: stdout=Password for WELLKNOWN/ANONYMOUS@IPA.TEST:
[Mon May 22 14:36:37.287383 2017] [wsgi:error] [pid 26269] [remote 192.168.123.1:45548]
[Mon May 22 14:36:37.287420 2017] [wsgi:error] [pid 26269] [remote 192.168.123.1:45548] ipa: DEBUG: stderr=kinit: Pre-authentication failed: Cannot open file '/var/kerberos/krb5kdc/kdc.crt': Permission denied while getting initial credentials
[Mon May 22 14:36:37.287423 2017] [wsgi:error] [pid 26269] [remote 192.168.123.1:45548]

Comment 2 Petr Vobornik 2017-05-25 12:23:24 UTC
Upstream ticket:
https://pagure.io/freeipa/issue/6973

Comment 5 Nikhil Dehadrai 2017-05-26 10:15:49 UTC
IPA Version: ipa-server-4.5.0-13.el7.x86_64

Tested that after upgrading IPA server to latest version (In my case RHEL 7.3.z to Rhel 7.4):
1) IPA web UI is accessible.
2) We are able to log in to the server.

Comment 13 Nikhil Dehadrai 2017-06-07 09:37:45 UTC
IPA-server: ipa-server-4.5.0-15.el7.x86_64


Tested the bug for WEBUI login to ipa server after upgrade( in my case RHEL 7.3.z > 7.4), but the login still fails with same error message "Login failed due to an unknown reason. "

Refer following logs:

[root@vm-idm-029 ~]# rpm -q ipa-server
ipa-server-4.5.0-15.el7.x86_64

[root@vm-idm-029 ~]# tail -f /var/log/httpd/error_log
[Wed Jun 07 15:03:51.010984 2017] [:error] [pid 29940] [remote 10.67.116.122:212] mod_wsgi (pid=29940): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'.
[Wed Jun 07 15:03:51.011104 2017] [:error] [pid 29940] [remote 10.67.116.122:212] Traceback (most recent call last):
[Wed Jun 07 15:03:51.011155 2017] [:error] [pid 29940] [remote 10.67.116.122:212]   File "/usr/share/ipa/wsgi.py", line 51, in application
[Wed Jun 07 15:03:51.011241 2017] [:error] [pid 29940] [remote 10.67.116.122:212]     return api.Backend.wsgi_dispatch(environ, start_response)
[Wed Jun 07 15:03:51.011271 2017] [:error] [pid 29940] [remote 10.67.116.122:212]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 262, in __call__
[Wed Jun 07 15:03:51.011315 2017] [:error] [pid 29940] [remote 10.67.116.122:212]     return self.route(environ, start_response)
[Wed Jun 07 15:03:51.011335 2017] [:error] [pid 29940] [remote 10.67.116.122:212]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 274, in route
[Wed Jun 07 15:03:51.011372 2017] [:error] [pid 29940] [remote 10.67.116.122:212]     return app(environ, start_response)
[Wed Jun 07 15:03:51.011393 2017] [:error] [pid 29940] [remote 10.67.116.122:212]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 923, in __call__
[Wed Jun 07 15:03:51.011429 2017] [:error] [pid 29940] [remote 10.67.116.122:212]     self.kinit(user_principal, password, ipa_ccache_name)
[Wed Jun 07 15:03:51.011449 2017] [:error] [pid 29940] [remote 10.67.116.122:212]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 959, in kinit
[Wed Jun 07 15:03:51.011483 2017] [:error] [pid 29940] [remote 10.67.116.122:212]     pkinit_anchors=[paths.KDC_CERT, paths.KDC_CA_BUNDLE_PEM],
[Wed Jun 07 15:03:51.011506 2017] [:error] [pid 29940] [remote 10.67.116.122:212]   File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 122, in kinit_armor
[Wed Jun 07 15:03:51.011566 2017] [:error] [pid 29940] [remote 10.67.116.122:212]     run(args, env=env, raiseonerr=True, capture_error=True)
[Wed Jun 07 15:03:51.011594 2017] [:error] [pid 29940] [remote 10.67.116.122:212]   File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 498, in run
[Wed Jun 07 15:03:51.011637 2017] [:error] [pid 29940] [remote 10.67.116.122:212]     raise CalledProcessError(p.returncode, arg_string, str(output))
[Wed Jun 07 15:03:51.011732 2017] [:error] [pid 29940] [remote 10.67.116.122:212] CalledProcessError: Command '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_29940 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned non-zero exit status 1

Thus on the basis of bove observations, changing status to "ASSIGNED".

Comment 14 Martin Babinsky 2017-06-07 09:49:50 UTC
Can you please switch on framework debugging in /etc/ipa/default.conf, re-run the test and then post the httpd error_log? also please check audit log for AVCs although these should be fixed in selinux-policy.

Comment 15 Martin Babinsky 2017-06-07 10:23:49 UTC
Which brings me to the question whether you use selinux-policy-3.13.1-159.el7.noarch which was built as part of https://bugzilla.redhat.com/show_bug.cgi?id=1436689, can you check that?

Comment 16 Varun Mylaraiah 2017-06-07 12:34:55 UTC
(In reply to Martin Babinsky from comment #14)
> Can you please switch on framework debugging in /etc/ipa/default.conf,
> re-run the test and then post the httpd error_log? also please check audit
> log for AVCs although these should be fixed in selinux-policy.

[root@vm-idm-001 ~]# rpm -qa ipa-server
ipa-server-4.5.0-15.el7.x86_64

/var/log/httpd/error_log

[Wed Jun 07 17:51:47.350374 2017] [:error] [pid 16997] [remote 10.67.116.109:4] mod_wsgi (pid=16997): Exception occurred processing WSGI script '/usr/share/ipa/wsgi.py'.
[Wed Jun 07 17:51:47.350481 2017] [:error] [pid 16997] [remote 10.67.116.109:4] Traceback (most recent call last):
[Wed Jun 07 17:51:47.350513 2017] [:error] [pid 16997] [remote 10.67.116.109:4]   File "/usr/share/ipa/wsgi.py", line 51, in application
[Wed Jun 07 17:51:47.350570 2017] [:error] [pid 16997] [remote 10.67.116.109:4]     return api.Backend.wsgi_dispatch(environ, start_response)
[Wed Jun 07 17:51:47.350601 2017] [:error] [pid 16997] [remote 10.67.116.109:4]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 262, in __call__
[Wed Jun 07 17:51:47.350640 2017] [:error] [pid 16997] [remote 10.67.116.109:4]     return self.route(environ, start_response)
[Wed Jun 07 17:51:47.350653 2017] [:error] [pid 16997] [remote 10.67.116.109:4]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 274, in route
[Wed Jun 07 17:51:47.350671 2017] [:error] [pid 16997] [remote 10.67.116.109:4]     return app(environ, start_response)
[Wed Jun 07 17:51:47.350682 2017] [:error] [pid 16997] [remote 10.67.116.109:4]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 923, in __call__
[Wed Jun 07 17:51:47.350701 2017] [:error] [pid 16997] [remote 10.67.116.109:4]     self.kinit(user_principal, password, ipa_ccache_name)
[Wed Jun 07 17:51:47.350712 2017] [:error] [pid 16997] [remote 10.67.116.109:4]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 959, in kinit
[Wed Jun 07 17:51:47.350729 2017] [:error] [pid 16997] [remote 10.67.116.109:4]     pkinit_anchors=[paths.KDC_CERT, paths.KDC_CA_BUNDLE_PEM],
[Wed Jun 07 17:51:47.350743 2017] [:error] [pid 16997] [remote 10.67.116.109:4]   File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 122, in kinit_armor
[Wed Jun 07 17:51:47.350782 2017] [:error] [pid 16997] [remote 10.67.116.109:4]     run(args, env=env, raiseonerr=True, capture_error=True)
[Wed Jun 07 17:51:47.350793 2017] [:error] [pid 16997] [remote 10.67.116.109:4]   File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 498, in run
[Wed Jun 07 17:51:47.350815 2017] [:error] [pid 16997] [remote 10.67.116.109:4]     raise CalledProcessError(p.returncode, arg_string, str(output))
[Wed Jun 07 17:51:47.350854 2017] [:error] [pid 16997] [remote 10.67.116.109:4] CalledProcessError: Command '/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_16997 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem' returned non-zero exit status 1

selinux-policy-3.13.1-160.el7.noarch
[root@vm-idm-001 ~]# ausearch -m AVC -ts today
<no matches>

[root@vm-idm-001 ~]# cat /var/log/audit/audit.log |audit2allow
Nothing to do
[root@vm-idm-001 ~]#

Comment 19 Petr Vobornik 2017-06-07 17:10:18 UTC
I've managed to reproduce with clean install. The issue was that kinit which obtains armor required pre-auth. 

After restarting gssproxy and then krb5kdc it no longer required it and Web UI login started to work again.

I'm not sure if both services needs to be restarted or only on of them. I did not manage to reproduce after un-installation and re-instalaltion on the same host.

Note the initial installation was with older selinux policy so maybe my reproducer did not match this scenario, but the work-around is worth trying.

Comment 20 Scott Poore 2017-06-07 19:56:00 UTC
FYI, I tried restarting gssproxy then krb5kdc with no luck.  I thought I'd try again with SELinux in permissive mode and that worked:

Afterwards I did see these AVC denials in the log:

time->Wed Jun  7 14:53:44 2017
type=SYSCALL msg=audit(1496865224.775:130): arch=c000003e syscall=2 success=yes exit=3 a0=563698bc2ed5 a1=0 a2=1b6 a3=24 items=0 ppid=2515 pid=2584 auid=4294967295 uid=387 gid=387 euid=387 suid=387 fsuid=387 egid=387 sgid=387 fsgid=387 tty=(none) ses=4294967295 comm="kinit" exe="/usr/bin/kinit" subj=system_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(1496865224.775:130): avc:  denied  { open } for  pid=2584 comm="kinit" path="/var/lib/ipa-client/pki/kdc-ca-bundle.pem" dev="dm-0" ino=4520247 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:realmd_var_lib_t:s0 tclass=file
type=AVC msg=audit(1496865224.775:130): avc:  denied  { read } for  pid=2584 comm="kinit" name="kdc-ca-bundle.pem" dev="dm-0" ino=4520247 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:realmd_var_lib_t:s0 tclass=file
----
time->Wed Jun  7 14:53:44 2017
type=SYSCALL msg=audit(1496865224.775:131): arch=c000003e syscall=5 success=yes exit=0 a0=3 a1=7fff570e49d0 a2=7fff570e49d0 a3=0 items=0 ppid=2515 pid=2584 auid=4294967295 uid=387 gid=387 euid=387 suid=387 fsuid=387 egid=387 sgid=387 fsgid=387 tty=(none) ses=4294967295 comm="kinit" exe="/usr/bin/kinit" subj=system_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(1496865224.775:131): avc:  denied  { getattr } for  pid=2584 comm="kinit" path="/var/lib/ipa-client/pki/kdc-ca-bundle.pem" dev="dm-0" ino=4520247 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:realmd_var_lib_t:s0 tclass=file

Comment 21 Standa Laznicka 2017-06-08 11:37:47 UTC
Seems like these policies slipped between our fingers, I already reported these denials in the massive IPA SELinux bugzilla:

https://bugzilla.redhat.com/show_bug.cgi?id=1436689#c109

Comment 22 Nikhil Dehadrai 2017-06-13 05:45:38 UTC
IPA-server: ipa-server-4.5.0-16.el7.x86_64
SELINUX: selinux-policy-3.13.1-161.el7.noarch

Verified the bug for WEBUI login to ipa server after upgrade (in my case RHEL 7.3.z > 7.4), the login is now successful.

Refer attached screenshot.

Comment 23 Nikhil Dehadrai 2017-06-13 05:46:19 UTC
Created attachment 1287162 [details]
Login-success-after-upgrade

Login-success-after-upgrade

Comment 24 errata-xmlrpc 2017-08-01 09:51:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2304


Note You need to log in before you can comment on or make changes to this bug.