Jul 24 17:43:04 replica.testrelm.test setroubleshoot[32378]: SELinux is preventing /usr/libexec/platform-python3.6 from search access on the directory krb5. 
For complete SELinux messages run: sealert -l e3514db6-a4fb-4acc-ac69-03e17e028844
Jul 24 17:43:04 replica.testrelm.test platform-python[32378]: SELinux is preventing /usr/libexec/platform-python3.6 from search access on the directory krb5.
 *****  Plugin catchall (100. confidence) suggests   **************************
                                                              
If you believe that platform-python3.6 should be allowed search access on the krb5 directory by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'ipa-pki-retriev' --raw | audit2allow -M my-ipapkiretriev
# semodule -X 300 -i my-ipapkiretriev.pp

(In reply to Sudhir Menon from comment #3)
> Jul 24 17:43:04 replica.testrelm.test setroubleshoot[32378]: SELinux is
> preventing /usr/libexec/platform-python3.6 from search access on the
> directory krb5. 
> For complete SELinux messages run: sealert -l
> e3514db6-a4fb-4acc-ac69-03e17e028844
> Jul 24 17:43:04 replica.testrelm.test platform-python[32378]: SELinux is
> preventing /usr/libexec/platform-python3.6 from search access on the
> directory krb5.
>  *****  Plugin catchall (100. confidence) suggests  
> **************************
>                                                               
> If you believe that platform-python3.6 should be allowed search access on
> the krb5 directory by default.
> Then you should report this as a bug.
> You can generate a local policy module to allow this access.
> Do
> allow this access for now by executing:
> # ausearch -c 'ipa-pki-retriev' --raw | audit2allow -M my-ipapkiretriev
> # semodule -X 300 -i my-ipapkiretriev.pp

Sudhir,

Can you try this manually with steps? If reproducible with manual steps, please share the steps.

Reproduce Steps:

Master:
[root@master alias]# ipa ca-add test_subca_master --subject 'cn=test_subca_master' --desc='subca'
[root@master alias]# certutil -d . -L 

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI
caSigningCert cert-pki-ca d156a636-121b-4ed5-a2e0-14dc3d1ebcd0 u,u,u

Replica:
[root@replica alias]#ipa ca-find
  Name: test_subca_master
  Description: subca
  Authority ID: d156a636-121b-4ed5-a2e0-14dc3d1ebcd0
  Subject DN: CN=test_subca_master
  Issuer DN: CN=Certificate Authority,O=RHEL83.TEST

[root@replica alias]#ipa ca-add test_subca_replica --subject 'cn=test_subca_replica' --desc='subca'
[root@replica alias]#ipa ca-find
  Name: test_subca_replica
  Description: subca
  Authority ID: 5f3fd4d8-5d22-4f97-92c4-a49e31edef5c
  Subject DN: CN=test_subca_replica
  Issuer DN: CN=Certificate Authority,O=RHEL83.TEST

[root@replica alias]# certutil -d . -L 
Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI
caSigningCert cert-pki-ca 5f3fd4d8-5d22-4f97-92c4-a49e31edef5c u,u,u

[root@replica alias]# certutil -d . -L -n 'caSigningCert cert-pki-ca d156a636-121b-4ed5-a2e0-14dc3d1ebcd0'
certutil: Could not find cert: caSigningCert cert-pki-ca d156a636-121b-4ed5-a2e0-14dc3d1ebcd0
: PR_FILE_NOT_FOUND_ERROR: File not found

[root@replica alias]# ausearch -m AVC
time->Thu Aug 13 16:36:52 2020
type=PROCTITLE msg=audit(1597316812.733:836): proctitle=2F7573722F6C6962657865632F706C6174666F726D2D707974686F6E002D49002F7573722F6C6962657865632F6970612F6970612D706B692D72657472696576652D6B65790063615369676E696E674365727420636572742D706B692D63612064313536613633362D313231622D346564352D613265302D3134646333643165
type=SYSCALL msg=audit(1597316812.733:836): arch=c000003e syscall=257 success=no exit=-13 a0=ffffff9c a1=55fc20814360 a2=0 a3=0 items=0 ppid=31417 pid=31469 auid=4294967295 uid=17 gid=17 euid=17 suid=17 fsuid=17 egid=17 sgid=17 fsgid=17 tty=(none) ses=4294967295 comm="ipa-pki-retriev" exe="/usr/libexec/platform-python3.6" subj=system_u:system_r:tomcat_t:s0 key=(null)
type=AVC msg=audit(1597316812.733:836): avc:  denied  { search } for  pid=31469 comm="ipa-pki-retriev" name="krb5" dev="dm-0" ino=33907336 scontext=system_u:system_r:tomcat_t:s0 tcontext=system_u:object_r:krb5_keytab_t:s0 tclass=dir permissive=0

[root@replica alias]# cat /var/log/messages
Aug 13 16:37:00 vm-idm-026 server[31417]: Traceback (most recent call last):
Aug 13 16:37:00 vm-idm-026 server[31417]:  File "/usr/libexec/ipa/ipa-pki-retrieve-key", line 54, in main
Aug 13 16:37:00 vm-idm-026 server[31417]:    resp = client.fetch_key(path, store=False)
Aug 13 16:37:00 vm-idm-026 server[31417]:  File "/usr/lib/python3.6/site-packages/ipaserver/secrets/client.py", line 120, in fetch_key
Aug 13 16:37:00 vm-idm-026 server[31417]:    r.raise_for_status()
Aug 13 16:37:00 vm-idm-026 server[31417]:  File "/usr/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
Aug 13 16:37:00 vm-idm-026 server[31417]:    raise HTTPError(http_error_msg, response=self)
Aug 13 16:37:00 vm-idm-026 server[31417]: requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://master.rhel83.test/ipa/keys/ca_wrapped/caSigningCert%20cert-pki-ca%20d156a636-121b-4ed5-a2e0-14dc3d1ebcd0/2.16.840.1.101.3.4.1.2?type=kem&value=eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkEyNTZDQkMtSFM1MTIiLCJraWQiOm51bGx9.EeBh-exmW-pDOWiwkVBt8xE9RS9g-Ffb0QO_XhpzmebAalrN0R53z0TOG0QnQ3tNtayXRHMG_et1eDqFV0PYpuvw1BBqhBnWxaTZxcAeHDBzFVwW6K6AwJ8w-A2pvuFFM4iehwEoqebqkHeI41bqfrRkEPVSOYC_aOljgCezfmMCbl7or2qctKDVUnURhUQOAXDU4atVBHOg8pkkMqtR6_7DZ5S960Y9HlTiUln_IMpzZ0yqMrPd8CKYij0xrAlr8mmg4Kdbn4syu6F00UHO5ENUVlqeVsSvf4xUWcSP7abpZ1P76ZDAM9IF-igo6nHiOP3mJFs5auPH7SUswU16eg.Fg-gC97LXlYEglZKxIgPJQ.T9iVxzHGeL8DNnoGWUuIaWkWepbmY0oP5LUzB7y7zDy-JGeJDz-FaBJVTx1Z9gKwxQyOyzZP9u8NP3-smLGTfAe-qK9O54DHdj1wMnSrv4_kgqeZu7_82-myVJYmyaJL7WrE_o688mD8TTcN15o5KCEJSs3Eo3-a-dfRAtkD8zYydgrcGetkcNf1V31eUgWq8o8aOSK4en4BiSxa0B3W5dxgVpv4PAkuc5nbqcGEX1JuxqykByvMlRKyOkQhGDaO2v29v9R9qpKOd_2YhTwjo7fDy7ZWhSDNgVbOgGO8yMvz6LqrMazhFS5NHzNQr4wX_TP4uTzJIh2dOzPkNOOKnehDXBI-q6ZGArHy07e1J7rfG2GodyZNxoEVCOW6Z3Q-7WOll9tAJuS4Q9H67P3mv1bsgMgkFGBGX568lt7dDNJPtwkycAyZW-J4p7l2aMGtpxgnyys8vB9avEmkGZTJ8MU5RoDPEeSPw-T82-Cg04u3aoZYXNXNzo3-eHJttq9l_Af5jUxYHZSdTKGvSRaUnywfomtHmLLEvkaW7xDyq8EK2SseQtrts2wvM3hN2RL9dAVYn2pjWHIZzVIBhHJHBRTGSfxom6_is0B9ULHNgnC9IK31kQTTadcQHhfsbpGONH7SK4fEouOCe0R8_4qdI5g1_s8wScoHL7AVH71L6R2ifpSUxTvoe12KKYTnqhX1faxF2u-dWopqVTwicBxWRkfigSieMocq1GAZI0YdjoxkXArCIGRBB6KQC_ysDfW83wVxmpxlGtnXohr9q3rVqA.SDnFby8462I1AwxuAGjYvAVJav1l4xRKfbVWJwkVuL8

Sudhir,

Could you reproduce the issue with SELinux set to Permissive on both VMs and post the results e.g. "ausearch -m AVC"?
There might be more than one AVC there but in Enforcing mode the first AVC will hide the others.

Thank you,
François

Minimal reproducer:

* on master:
$ kinit admin
$ ipa ca-add test_subca_master --subject cn=test_subca_master --desc subca
$ ipa ca-show test_subca_master

* on replica:
$ kinit admin
$ ipa ca-show test_subca_master
$ /usr/bin/certutil -d /etc/pki/pki-tomcat/alias -L -n "caSigningCert cert-pki-ca <ID>"
$ echo $?

The following module on the master side fixes the issue:

#################
module local 1.0;

require {
	type pki_tomcat_cert_t;
	type ipa_custodia_t;
	type node_t;
	class tcp_socket { bind create node_bind };
	class process execmem;
	class file { create unlink };
	class dir remove_name;
}

#============= ipa_custodia_t ==============

allow ipa_custodia_t self:tcp_socket { bind create };
allow ipa_custodia_t node_t:tcp_socket node_bind;
allow ipa_custodia_t pki_tomcat_cert_t:dir remove_name;
allow ipa_custodia_t pki_tomcat_cert_t:file create;
allow ipa_custodia_t pki_tomcat_cert_t:file unlink;
allow ipa_custodia_t self:process execmem;
#################

There are remaining AVCs on the master:
type=AVC msg=audit(1599232734.116:171): avc:  denied  { read } for  pid=2020 comm="java" name="cpu" dev="sysfs" ino=33 scontext=system_u:system_r:ipa_custodia_t:s0 tcontext=system_u:object_r:sysfs_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1599232734.117:172): avc:  denied  { map } for  pid=2020 comm="java" path="/tmp/hsperfdata_pkiuser/2020" dev="dm-0" ino=9534196 scontext=system_u:system_r:ipa_custodia_t:s0 tcontext=system_u:object_r:ipa_custodia_tmp_t:s0 tclass=file permissive=0
type=AVC msg=audit(1599232734.801:173): avc:  denied  { read } for  pid=2020 comm="java" name="if_inet6" dev="proc" ino=4026532230 scontext=system_u:system_r:ipa_custodia_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0
type=AVC msg=audit(1599232734.801:174): avc:  denied  { read } for  pid=2020 comm="java" name="ipv6_route" dev="proc" ino=4026532231 scontext=system_u:system_r:ipa_custodia_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0
type=AVC msg=audit(1599232734.801:175): avc:  denied  { read } for  pid=2020 comm="java" name="if_inet6" dev="proc" ino=4026532230 scontext=system_u:system_r:ipa_custodia_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0
type=AVC msg=audit(1599232735.032:176): avc:  denied  { read } for  pid=2020 comm="java" name="random" dev="devtmpfs" ino=9223 scontext=system_u:system_r:ipa_custodia_t:s0 tcontext=system_u:object_r:random_device_t:s0 tclass=chr_file permissive=0
type=AVC msg=audit(1599232735.032:177): avc:  denied  { read } for  pid=2020 comm="java" name="random" dev="devtmpfs" ino=9223 scontext=system_u:system_r:ipa_custodia_t:s0 tcontext=system_u:object_r:random_device_t:s0 tclass=chr_file permissive=0
type=AVC msg=audit(1599232735.032:178): avc:  denied  { read } for  pid=2020 comm="java" name="random" dev="devtmpfs" ino=9223 scontext=system_u:system_r:ipa_custodia_t:s0 tcontext=system_u:object_r:random_device_t:s0 tclass=chr_file permissive=0
type=AVC msg=audit(1599232736.514:179): avc:  denied  { read } for  pid=2045 comm="java" name="cpu" dev="sysfs" ino=33 scontext=system_u:system_r:ipa_custodia_t:s0 tcontext=system_u:object_r:sysfs_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1599232736.515:180): avc:  denied  { map } for  pid=2045 comm="java" path="/tmp/hsperfdata_pkiuser/2045" dev="dm-0" ino=9534196 scontext=system_u:system_r:ipa_custodia_t:s0 tcontext=system_u:object_r:ipa_custodia_tmp_t:s0 tclass=file permissive=0
type=AVC msg=audit(1599232737.665:181): avc:  denied  { read } for  pid=2045 comm="java" name="if_inet6" dev="proc" ino=4026532230 scontext=system_u:system_r:ipa_custodia_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0
type=AVC msg=audit(1599232737.665:182): avc:  denied  { read } for  pid=2045 comm="java" name="ipv6_route" dev="proc" ino=4026532231 scontext=system_u:system_r:ipa_custodia_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0
type=AVC msg=audit(1599232737.665:183): avc:  denied  { read } for  pid=2045 comm="java" name="if_inet6" dev="proc" ino=4026532230 scontext=system_u:system_r:ipa_custodia_t:s0 tcontext=system_u:object_r:proc_net_t:s0 tclass=file permissive=0
type=AVC msg=audit(1599232737.880:184): avc:  denied  { read } for  pid=2045 comm="java" name="random" dev="devtmpfs" ino=9223 scontext=system_u:system_r:ipa_custodia_t:s0 tcontext=system_u:object_r:random_device_t:s0 tclass=chr_file permissive=0
type=AVC msg=audit(1599232737.880:185): avc:  denied  { read } for  pid=2045 comm="java" name="random" dev="devtmpfs" ino=9223 scontext=system_u:system_r:ipa_custodia_t:s0 tcontext=system_u:object_r:random_device_t:s0 tclass=chr_file permissive=0
type=AVC msg=audit(1599232737.880:186): avc:  denied  { read } for  pid=2045 comm="java" name="random" dev="devtmpfs" ino=9223 scontext=system_u:system_r:ipa_custodia_t:s0 tcontext=system_u:object_r:random_device_t:s0 tclass=chr_file permissive=0

And on the replica:
type=AVC msg=audit(1599232733.330:92): avc:  denied  { search } for  pid=1837 comm="ipa-pki-retriev" name="krb5" dev="dm-0" ino=8620822 scontext=system_u:system_r:tomcat_t:s0 tcontext=system_u:object_r:krb5_keytab_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1599232733.344:93): avc:  denied  { search } for  pid=1837 comm="ipa-pki-retriev" name="krb5" dev="dm-0" ino=8620822 scontext=system_u:system_r:tomcat_t:s0 tcontext=system_u:object_r:krb5_keytab_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1599232733.411:94): avc:  denied  { search } for  pid=1837 comm="ipa-pki-retriev" name="krb5" dev="dm-0" ino=8620822 scontext=system_u:system_r:tomcat_t:s0 tcontext=system_u:object_r:krb5_keytab_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1599232733.422:95): avc:  denied  { search } for  pid=1837 comm="ipa-pki-retriev" name="krb5" dev="dm-0" ino=8620822 scontext=system_u:system_r:tomcat_t:s0 tcontext=system_u:object_r:krb5_keytab_t:s0 tclass=dir permissive=0

I'll work on adapting FreeIPA's SELinux policy so that this works OOTB.

Fixed upstream
master:
https://pagure.io/freeipa/c/68328299c881df497dab32b145157d6e0e42d6c6

Fixed upstream
ipa-4-8:
https://pagure.io/freeipa/c/438285470610dee4aa6a56523df22307840ede87

Upstream ticket: https://pagure.io/freeipa/issue/8488

Data points:
- upstream test results, SELinux in Enforcing mode, without the fix: http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/ea5f1404-f1dd-11ea-b70d-fa163e53827b/report.html
- upstream test results, SELinux in Enforcing mode, with the fix: http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/a4c68fec-f1e9-11ea-ab26-fa163eef7a64/report.html

Complete diff:
https://github.com/freeipa/freeipa/commit/68328299c881df497dab32b145157d6e0e42d6c6

No code change, only SELinux policy change.

Fixed upstream
master:
https://pagure.io/freeipa/c/dfeea1644a35307d95a62fd47a71e9c97f20e066
https://pagure.io/freeipa/c/7823da0630379d642f4a19b3ae0c063963041a82
https://pagure.io/freeipa/c/ea9db4a9032b768486b0a93e4222a087053f87bb
https://pagure.io/freeipa/c/820beca4ac2aaa00a713e0bf5e50ecf79fe89dfc
https://pagure.io/freeipa/c/09816f4dbcb00e6c58754c2d74a6d8072e2871c3
https://pagure.io/freeipa/c/4b3c4b84d4ca80019d27a7643fcb1a03bb208072
https://pagure.io/freeipa/c/f774642b6384c5e2da644c1c812d8dd48946dacc
https://pagure.io/freeipa/c/2f2bce43108db5f1fa651a63eea38b33242495cf

Fixed upstream
ipa-4-8:
https://pagure.io/freeipa/c/52929cbadf0252fcac1019b74663a2808061ea1b
https://pagure.io/freeipa/c/5a5962426d8174212f0b7efef1a9e53aaecb5901
https://pagure.io/freeipa/c/c126610ea6605a1ff36cecf2e2f5b2cb97130831
https://pagure.io/freeipa/c/310dbd6eec337f0747d73fa87363083a742fc5dc
https://pagure.io/freeipa/c/0518c63768b50973f3d3129547f5b4b95335f4a8
https://pagure.io/freeipa/c/25cf7af0d41bbd34621f37c95802675b42baeae9
https://pagure.io/freeipa/c/7ad04841245668e3126cb1718ef7ec1b744526e8
https://pagure.io/freeipa/c/6a31605c1d249416ed7627755bca23a1cc45a581

Fixed upstream
master:
https://pagure.io/freeipa/c/7651d335b3fe7c644ae9b8de2590b315303375cf
https://pagure.io/freeipa/c/36c6a2e7493f8135b0adaaf1c056486a9d576c84

This bug has two root causes:
* IPA SELinux policy not allowing subca key replication to happen properly. This is fixed with the above commits.
* A race condition in nss that randomly causes rapid-fire subca creation/deletion to fail. This is tracked in https://bugzilla.redhat.com/show_bug.cgi?id=1881999#c2

Fixed upstream
ipa-4-8:
https://pagure.io/freeipa/c/58c3343a67a3922dcc84d3d4b1deca515c48a6f8

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: idm:DL1 and idm:client security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4670