Description of problem: When the undercloud cert monitored by certmonger is renewed the post save command does not have permission to interact with the container. [stack@undercloud-0 ~]$ sudo grep -i denied /var/log/audit/audit.log type=AVC msg=audit(1555482722.248:3216): avc: denied { read } for pid=29948 comm="logrotate" name="heat" dev="vda1" ino=205528870 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555482902.089:3359): avc: denied { search } for pid=14253 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555482902.136:3360): avc: denied { search } for pid=14256 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555482902.182:3361): avc: denied { search } for pid=14258 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555482902.231:3362): avc: denied { search } for pid=14260 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555482902.282:3363): avc: denied { search } for pid=14262 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555483089.292:3837): avc: denied { search } for pid=31328 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555483089.347:3838): avc: denied { search } for pid=31330 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555483089.401:3839): avc: denied { search } for pid=31332 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555483089.457:3840): avc: denied { search } for pid=31334 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555483089.512:3841): avc: denied { search } for pid=31336 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555483158.261:3998): avc: denied { search } for pid=37158 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555483158.307:3999): avc: denied { search } for pid=37160 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555483158.353:4000): avc: denied { search } for pid=37162 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555483158.400:4001): avc: denied { search } for pid=37164 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555483158.447:4002): avc: denied { search } for pid=37166 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555483388.614:4278): avc: denied { search } for pid=46806 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555483388.662:4279): avc: denied { search } for pid=46808 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555483388.711:4280): avc: denied { search } for pid=46810 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555483388.760:4281): avc: denied { search } for pid=46812 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555483388.811:4282): avc: denied { search } for pid=46814 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555483454.786:4762): avc: denied { search } for pid=54745 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555483454.836:4763): avc: denied { search } for pid=54747 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555483454.883:4764): avc: denied { search } for pid=54749 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555483454.932:4765): avc: denied { search } for pid=54751 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1555483454.979:4766): avc: denied { search } for pid=54753 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1587104711.294:522): avc: denied { search } for pid=89142 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1587104711.342:523): avc: denied { search } for pid=89144 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1587104711.389:524): avc: denied { search } for pid=89146 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1587104711.436:525): avc: denied { search } for pid=89148 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1587104711.484:526): avc: denied { search } for pid=89150 comm="ruby" name="puppet" dev="vda1" ino=167784818 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:puppet_etc_t:s0 tclass=dir permissive=0 Running in permissive mode for a renewal shows the following items. (undercloud) [root@undercloud-0 ~]# grep -i denied /var/log/audit/audit.log |audit2allow #============= certmonger_t ============== allow certmonger_t NetworkManager_t:dir { getattr search }; allow certmonger_t NetworkManager_t:file { open read }; allow certmonger_t auditd_t:dir { getattr search }; allow certmonger_t auditd_t:file { open read }; allow certmonger_t container_runtime_exec_t:file { execute execute_no_trans getattr ioctl open read }; allow certmonger_t container_runtime_t:dir { getattr search }; allow certmonger_t container_runtime_t:file { open read }; allow certmonger_t crond_t:dir { getattr search }; allow certmonger_t crond_t:file { open read }; allow certmonger_t dhcpc_t:dir { getattr search }; allow certmonger_t dhcpc_t:file { open read }; allow certmonger_t getty_t:dir { getattr search }; allow certmonger_t getty_t:file { open read }; allow certmonger_t gssproxy_t:dir { getattr search }; allow certmonger_t gssproxy_t:file { open read }; allow certmonger_t irqbalance_t:dir { getattr search }; allow certmonger_t irqbalance_t:file { open read }; allow certmonger_t kernel_t:dir { getattr search }; allow certmonger_t kernel_t:file { open read }; allow certmonger_t lvm_t:dir { getattr search }; allow certmonger_t lvm_t:file { open read }; allow certmonger_t ntpd_t:dir { getattr search }; allow certmonger_t ntpd_t:file { open read }; allow certmonger_t openvswitch_t:dir { getattr search }; allow certmonger_t openvswitch_t:file { open read }; allow certmonger_t policykit_t:dir { getattr search }; allow certmonger_t policykit_t:file { open read }; allow certmonger_t postfix_master_t:dir { getattr search }; allow certmonger_t postfix_master_t:file { open read }; allow certmonger_t postfix_pickup_t:dir { getattr search }; allow certmonger_t postfix_pickup_t:file { open read }; allow certmonger_t postfix_qmgr_t:dir { getattr search }; allow certmonger_t postfix_qmgr_t:file { open read }; allow certmonger_t puppet_etc_t:dir search; allow certmonger_t puppet_etc_t:file { getattr ioctl open read }; allow certmonger_t rhnsd_t:dir { getattr search }; allow certmonger_t rhnsd_t:file { open read }; allow certmonger_t rpcbind_t:dir { getattr search }; allow certmonger_t rpcbind_t:file { open read }; allow certmonger_t spc_t:dir { getattr search }; allow certmonger_t spc_t:file { open read }; allow certmonger_t spc_t:process signal; allow certmonger_t sshd_t:dir { getattr search }; allow certmonger_t sshd_t:file { open read }; allow certmonger_t sysctl_net_t:dir search; allow certmonger_t sysctl_net_t:file { open read }; allow certmonger_t syslogd_t:dir { getattr search }; allow certmonger_t syslogd_t:file { open read }; allow certmonger_t system_dbusd_t:dir { getattr search }; allow certmonger_t system_dbusd_t:file { open read }; allow certmonger_t systemd_logind_t:dir { getattr search }; allow certmonger_t systemd_logind_t:file { open read }; allow certmonger_t tuned_t:dir { getattr search }; allow certmonger_t tuned_t:file { open read }; allow certmonger_t udev_t:dir { getattr search }; allow certmonger_t udev_t:file { open read }; allow certmonger_t unconfined_service_t:dir { getattr search }; allow certmonger_t unconfined_service_t:file { open read }; allow certmonger_t unconfined_t:dir { getattr search }; allow certmonger_t unconfined_t:file { open read }; allow certmonger_t virt_qemu_ga_t:dir { getattr search }; allow certmonger_t virt_qemu_ga_t:file { open read }; #============= logrotate_t ============== #!!!! This avc can be allowed using the boolean 'logrotate_read_inside_containers' allow logrotate_t container_file_t:dir read; Version-Release number of selected component (if applicable): osp14 How reproducible: always Steps to Reproduce: 1. disable and stop ntpd 2. find the date for the cert to expire and check the cert [stack@undercloud-0 ~]$ sudo getcert list Number of certificates and requests being tracked: 1. Request ID 'haproxy-external-cert': status: MONITORING stuck: no key pair storage: type=FILE,location='/etc/pki/tls/private/haproxy/overcloud-haproxy-external.key' certificate: type=FILE,location='/etc/pki/tls/certs/haproxy/overcloud-haproxy-external.crt' CA: local issuer: CN=e76b6864-6aa64a3d-b5bf9a7f-a6a70d8d,CN=Local Signing Authority subject: CN=192.168.24.2 expires: 2020-04-17 06:35:00 UTC eku: id-kp-clientAuth,id-kp-serverAuth pre-save command: post-save command: /usr/bin/certmonger-haproxy-refresh.sh reload external track: yes auto-renew: yes [stack@undercloud-0 ~]$ sudo cat /etc/pki/tls/certs/haproxy/overcloud-haproxy-external.crt -----BEGIN CERTIFICATE----- MIIDfTCCAmWgAwIBAgIRAOdraGRqpko9tb+af6anDZIwDQYJKoZIhvcNAQELBQAw UDEgMB4GA1UEAwwXTG9jYWwgU2lnbmluZyBBdXRob3JpdHkxLDAqBgNVBAMMI2U3 NmI2ODY0LTZhYTY0YTNkLWI1YmY5YTdmLWE2YTcwZDhkMB4XDTE5MDQxNzA2NDQx NFoXDTIwMDQxNzA2MzUwMFowFzEVMBMGA1UEAxMMMTkyLjE2OC4yNC4yMIIBIjAN BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxlptQ+cRrvXcKYC+MjDgDCGOW+11 0KTjxVMqAzvua2ct3XgxVgOUWIAJJFCWOq3/cl+roHxVABbXZaWyIJLtC9q7Tn5o 2R/b5zKc8++GCeCNEdsc7zrnmxcM4n1HMO6ZCVbEDPBZvt9SQUgfCsCcytMNIOd0 8x3I+88vAxnX+bcqAC6s9sJ3DjUJ04RvPZEd9aoB3xlPdmgFeL0cCASGC8WgcMnJ 3PYIGNT6CEoQuN5PiHh8EvVBuY6Tip0nv9iJd+qr/pxOyrvs0vbdMS04habTnkRR vr1K91NSefLNAOMTlARNB/eZpW2qOw91BzvP8QmbwUVxM0rAFUn1mRD/fwIDAQAB o4GKMIGHMBIGA1UdEQEBAAQIMAaHBMCoGAIwIAYDVR0lAQEABBYwFAYIKwYBBQUH AwIGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwIAYDVR0OAQEABBYEFJllj+nI6+uB g+/i7/RM2vQCBI4NMB8GA1UdIwQYMBaAFIbgbeIDDqH+JxyL7onK2fe6U8isMA0G CSqGSIb3DQEBCwUAA4IBAQCtdTumx7WcrPN75o97IYgJhc3F0eMbTplw4+kl/6QA c87WDWwoMLTXklA1BdVPGjbgknp+ysVPEhCr/2uJwf4S5YnpMyISqLxlx48Fjivl d+5vss7XbKHbp1BljQv6kmt0dK2CcYQmKETGIvDtVLzQGgBVr7/E3FFxtiFzSpJW cO1fPGVoRC+zHAkvq7bRKL1D3C66lzAM8D4jRckSBoX0la/4uLXdYPOra+7r/jKr 8HaFgxibKQjTyXVCmR8ek1M+AYajj0NCuf+wFqWNinKAGKFC85n8+MVecJQv6uXF 3fjT/meR4ld8rMs68El8oXtxZJm3W8bRjdi4H4sMCa2b -----END CERTIFICATE----- 3. Confirm haproxy is using the cert. [stack@undercloud-0 ~]$ openssl s_client -connect 192.168.24.2:13000 CONNECTED(00000003) depth=1 CN = Local Signing Authority, CN = e76b6864-6aa64a3d-b5bf9a7f-a6a70d8d verify return:1 depth=0 CN = 192.168.24.2 verify return:1 --- Certificate chain 0 s:/CN=192.168.24.2 i:/CN=Local Signing Authority/CN=e76b6864-6aa64a3d-b5bf9a7f-a6a70d8d 1 s:/CN=Local Signing Authority/CN=e76b6864-6aa64a3d-b5bf9a7f-a6a70d8d i:/CN=Local Signing Authority/CN=e76b6864-6aa64a3d-b5bf9a7f-a6a70d8d --- Server certificate -----BEGIN CERTIFICATE----- MIIDfTCCAmWgAwIBAgIRAOdraGRqpko9tb+af6anDZIwDQYJKoZIhvcNAQELBQAw UDEgMB4GA1UEAwwXTG9jYWwgU2lnbmluZyBBdXRob3JpdHkxLDAqBgNVBAMMI2U3 NmI2ODY0LTZhYTY0YTNkLWI1YmY5YTdmLWE2YTcwZDhkMB4XDTE5MDQxNzA2NDQx NFoXDTIwMDQxNzA2MzUwMFowFzEVMBMGA1UEAxMMMTkyLjE2OC4yNC4yMIIBIjAN BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxlptQ+cRrvXcKYC+MjDgDCGOW+11 0KTjxVMqAzvua2ct3XgxVgOUWIAJJFCWOq3/cl+roHxVABbXZaWyIJLtC9q7Tn5o 2R/b5zKc8++GCeCNEdsc7zrnmxcM4n1HMO6ZCVbEDPBZvt9SQUgfCsCcytMNIOd0 8x3I+88vAxnX+bcqAC6s9sJ3DjUJ04RvPZEd9aoB3xlPdmgFeL0cCASGC8WgcMnJ 3PYIGNT6CEoQuN5PiHh8EvVBuY6Tip0nv9iJd+qr/pxOyrvs0vbdMS04habTnkRR vr1K91NSefLNAOMTlARNB/eZpW2qOw91BzvP8QmbwUVxM0rAFUn1mRD/fwIDAQAB o4GKMIGHMBIGA1UdEQEBAAQIMAaHBMCoGAIwIAYDVR0lAQEABBYwFAYIKwYBBQUH AwIGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwIAYDVR0OAQEABBYEFJllj+nI6+uB g+/i7/RM2vQCBI4NMB8GA1UdIwQYMBaAFIbgbeIDDqH+JxyL7onK2fe6U8isMA0G CSqGSIb3DQEBCwUAA4IBAQCtdTumx7WcrPN75o97IYgJhc3F0eMbTplw4+kl/6QA c87WDWwoMLTXklA1BdVPGjbgknp+ysVPEhCr/2uJwf4S5YnpMyISqLxlx48Fjivl d+5vss7XbKHbp1BljQv6kmt0dK2CcYQmKETGIvDtVLzQGgBVr7/E3FFxtiFzSpJW cO1fPGVoRC+zHAkvq7bRKL1D3C66lzAM8D4jRckSBoX0la/4uLXdYPOra+7r/jKr 8HaFgxibKQjTyXVCmR8ek1M+AYajj0NCuf+wFqWNinKAGKFC85n8+MVecJQv6uXF 3fjT/meR4ld8rMs68El8oXtxZJm3W8bRjdi4H4sMCa2b -----END CERTIFICATE----- subject=/CN=192.168.24.2 issuer=/CN=Local Signing Authority/CN=e76b6864-6aa64a3d-b5bf9a7f-a6a70d8d --- No client certificate CA names sent Peer signing digest: SHA512 Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 2473 bytes and written 415 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: 7DC50A720D5C00349758AD5EDD453A986F1FD3EB957EC4F79E3F28128CEFF23B Session-ID-ctx: Master-Key: 362A10614AB853CD48ABF54C3A68EAB862268296B03EAC418FA130801D21778033A22D13214D2D0585E0CE1E19796ED6 Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None TLS session ticket lifetime hint: 300 (seconds) TLS session ticket: 0000 - de 8a 7d 1f 69 79 82 bc-97 48 a4 76 3a f2 af 9e ..}.iy...H.v:... 0010 - 1d 64 cd 01 38 52 61 d7-1f 09 e6 b6 26 9e 98 11 .d..8Ra.....&... 0020 - 88 b0 71 f6 5e 18 ae bf-1f c6 b9 9c 89 4a bb c7 ..q.^........J.. 0030 - 57 20 08 be 54 37 d5 d6-16 d9 22 1e bd a0 bd 1f W ..T7...."..... 0040 - db ad 5c 5c ca 3c c0 e7-75 01 77 dc 84 9f f8 8c ..\\.<..u.w..... 0050 - 3f 87 3e 59 1a d4 2a 88-4c b8 cb 5b 58 6b 67 20 ?.>Y..*.L..[Xkg 0060 - 8c 32 70 6b 52 a2 34 68-96 e0 69 df 83 3d dc 26 .2pkR.4h..i..=.& 0070 - c7 b6 42 d7 83 89 ea e8-e7 50 f1 ce ce da 1e 56 ..B......P.....V 0080 - 42 33 b0 18 c1 1b f8 6b-f7 f0 8e 0d 9d 0a 6b 1b B3.....k......k. 0090 - 59 cd 61 36 73 16 80 f8-ff f0 6f f0 de 41 bc d4 Y.a6s.....o..A.. Start Time: 1555601505 Timeout : 300 (sec) Verify return code: 0 (ok) --- 4. Set the system time so the cert will expire. [stack@undercloud-0 ~]$ sudo date --set "2020-04-17 06:25:00" --utc Fri Apr 17 06:25:00 UTC 2020 [stack@undercloud-0 ~]$ date --utc Fri Apr 17 06:25:25 UTC 2020 5. Check and confirm certmonger got a new cert and check its contents. [stack@undercloud-0 ~]$ sudo getcert list Number of certificates and requests being tracked: 1. Request ID 'haproxy-external-cert': status: MONITORING stuck: no key pair storage: type=FILE,location='/etc/pki/tls/private/haproxy/overcloud-haproxy-external.key' certificate: type=FILE,location='/etc/pki/tls/certs/haproxy/overcloud-haproxy-external.crt' CA: local issuer: CN=e76b6864-6aa64a3d-b5bf9a7f-a6a70d8d,CN=Local Signing Authority subject: CN=192.168.24.2 expires: 2021-04-17 06:25:11 UTC eku: id-kp-clientAuth,id-kp-serverAuth pre-save command: post-save command: /usr/bin/certmonger-haproxy-refresh.sh reload external track: yes auto-renew: yes [stack@undercloud-0 ~]$ sudo cat /etc/pki/tls/certs/haproxy/overcloud-haproxy-external.crt -----BEGIN CERTIFICATE----- MIIDfTCCAmWgAwIBAgIRAOdraGRqpko9tb+af6anDZQwDQYJKoZIhvcNAQELBQAw UDEgMB4GA1UEAwwXTG9jYWwgU2lnbmluZyBBdXRob3JpdHkxLDAqBgNVBAMMI2U3 NmI2ODY0LTZhYTY0YTNkLWI1YmY5YTdmLWE2YTcwZDhkMB4XDTIwMDQxNzA2MjUx MVoXDTIxMDQxNzA2MjUxMVowFzEVMBMGA1UEAxMMMTkyLjE2OC4yNC4yMIIBIjAN BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxlptQ+cRrvXcKYC+MjDgDCGOW+11 0KTjxVMqAzvua2ct3XgxVgOUWIAJJFCWOq3/cl+roHxVABbXZaWyIJLtC9q7Tn5o 2R/b5zKc8++GCeCNEdsc7zrnmxcM4n1HMO6ZCVbEDPBZvt9SQUgfCsCcytMNIOd0 8x3I+88vAxnX+bcqAC6s9sJ3DjUJ04RvPZEd9aoB3xlPdmgFeL0cCASGC8WgcMnJ 3PYIGNT6CEoQuN5PiHh8EvVBuY6Tip0nv9iJd+qr/pxOyrvs0vbdMS04habTnkRR vr1K91NSefLNAOMTlARNB/eZpW2qOw91BzvP8QmbwUVxM0rAFUn1mRD/fwIDAQAB o4GKMIGHMBIGA1UdEQEBAAQIMAaHBMCoGAIwIAYDVR0lAQEABBYwFAYIKwYBBQUH AwIGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwIAYDVR0OAQEABBYEFJllj+nI6+uB g+/i7/RM2vQCBI4NMB8GA1UdIwQYMBaAFLpYHe8mM1TXwix/2D8+6oGSUWFbMA0G CSqGSIb3DQEBCwUAA4IBAQB6/G9Ri7Pk11Jab+CNeN5c6gx6KGlBXhfWPPqBVbM9 HWinS5a46SXvS3nWI6EPQrJgQh2koPRso0yc4w7V0TJpbYwCQ4AEIPutc/7Rdro3 Rx7S1ERSXumzK+OXDifcTMWouq4/GzHJLaP7W8pVUiLnui9ck3mrNgZIG7ROOuzo l97uWuRRKMAfCDbM+2lp83WS8k2ZOoFGXxM4lkjF/Pmx/wguMOE/vEe4285j/ZIX o4DAofI0LNiHHABvVk4tfJOPJzQ0JdXUhrcjEE/u6UMzJig2NYi2DS1sqv/mfL5k ZqLfoq/qGwem1WYvEU3jBBTyGhEDYzy4RN+KSGfc2NzR -----END CERTIFICATE----- 6. Check haproxy again. the old cert is still in place and does not match the new one. [stack@undercloud-0 ~]$ openssl s_client -connect 192.168.24.2:13000 CONNECTED(00000003) depth=1 CN = Local Signing Authority, CN = e76b6864-6aa64a3d-b5bf9a7f-a6a70d8d verify return:1 depth=0 CN = 192.168.24.2 verify return:1 --- Certificate chain 0 s:/CN=192.168.24.2 i:/CN=Local Signing Authority/CN=e76b6864-6aa64a3d-b5bf9a7f-a6a70d8d 1 s:/CN=Local Signing Authority/CN=e76b6864-6aa64a3d-b5bf9a7f-a6a70d8d i:/CN=Local Signing Authority/CN=e76b6864-6aa64a3d-b5bf9a7f-a6a70d8d --- Server certificate -----BEGIN CERTIFICATE----- MIIDfTCCAmWgAwIBAgIRAOdraGRqpko9tb+af6anDZIwDQYJKoZIhvcNAQELBQAw UDEgMB4GA1UEAwwXTG9jYWwgU2lnbmluZyBBdXRob3JpdHkxLDAqBgNVBAMMI2U3 NmI2ODY0LTZhYTY0YTNkLWI1YmY5YTdmLWE2YTcwZDhkMB4XDTE5MDQxNzA2NDQx NFoXDTIwMDQxNzA2MzUwMFowFzEVMBMGA1UEAxMMMTkyLjE2OC4yNC4yMIIBIjAN BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxlptQ+cRrvXcKYC+MjDgDCGOW+11 0KTjxVMqAzvua2ct3XgxVgOUWIAJJFCWOq3/cl+roHxVABbXZaWyIJLtC9q7Tn5o 2R/b5zKc8++GCeCNEdsc7zrnmxcM4n1HMO6ZCVbEDPBZvt9SQUgfCsCcytMNIOd0 8x3I+88vAxnX+bcqAC6s9sJ3DjUJ04RvPZEd9aoB3xlPdmgFeL0cCASGC8WgcMnJ 3PYIGNT6CEoQuN5PiHh8EvVBuY6Tip0nv9iJd+qr/pxOyrvs0vbdMS04habTnkRR vr1K91NSefLNAOMTlARNB/eZpW2qOw91BzvP8QmbwUVxM0rAFUn1mRD/fwIDAQAB o4GKMIGHMBIGA1UdEQEBAAQIMAaHBMCoGAIwIAYDVR0lAQEABBYwFAYIKwYBBQUH AwIGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwIAYDVR0OAQEABBYEFJllj+nI6+uB g+/i7/RM2vQCBI4NMB8GA1UdIwQYMBaAFIbgbeIDDqH+JxyL7onK2fe6U8isMA0G CSqGSIb3DQEBCwUAA4IBAQCtdTumx7WcrPN75o97IYgJhc3F0eMbTplw4+kl/6QA c87WDWwoMLTXklA1BdVPGjbgknp+ysVPEhCr/2uJwf4S5YnpMyISqLxlx48Fjivl d+5vss7XbKHbp1BljQv6kmt0dK2CcYQmKETGIvDtVLzQGgBVr7/E3FFxtiFzSpJW cO1fPGVoRC+zHAkvq7bRKL1D3C66lzAM8D4jRckSBoX0la/4uLXdYPOra+7r/jKr 8HaFgxibKQjTyXVCmR8ek1M+AYajj0NCuf+wFqWNinKAGKFC85n8+MVecJQv6uXF 3fjT/meR4ld8rMs68El8oXtxZJm3W8bRjdi4H4sMCa2b -----END CERTIFICATE----- subject=/CN=192.168.24.2 issuer=/CN=Local Signing Authority/CN=e76b6864-6aa64a3d-b5bf9a7f-a6a70d8d --- No client certificate CA names sent Peer signing digest: SHA512 Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 2473 bytes and written 415 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: AF9CBC8D59A7409F8293B45F2ABC1A4B75E632EC28ED86685A69DF3EEE6C955A Session-ID-ctx: Master-Key: 465AC697858DB11B7A103CD8556008CF37234B4200E4A8A447712216E90DFAF584A55EE73EB4255F7C1F30702FE00F24 Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None TLS session ticket lifetime hint: 300 (seconds) TLS session ticket: 0000 - de 8a 7d 1f 69 79 82 bc-97 48 a4 76 3a f2 af 9e ..}.iy...H.v:... 0010 - d0 f4 5e 10 49 c2 b2 c8-c1 b8 5f 36 60 e9 9d da ..^.I....._6`... 0020 - be cf 0b 20 1c 86 5f 23-4a f4 20 dd 28 9a 5c 11 ... .._#J. .(.\. 0030 - 13 d5 71 93 48 39 6c d1-af df 7d ec 34 8b 52 49 ..q.H9l...}.4.RI 0040 - b7 04 01 01 9b 45 c7 c8-d3 7a aa 9e 07 85 02 7f .....E...z...... 0050 - 16 bc a2 bd 4d 1c 74 95-3b 7f 76 6b 1c a9 32 fb ....M.t.;.vk..2. 0060 - 8b bf fb f8 91 7a ff 8c-3f 20 22 7e 2d cb 6b ad .....z..? "~-.k. 0070 - b5 fa 81 be 80 71 f9 3e-25 30 3c 4e 65 bd 24 44 .....q.>%0<Ne.$D 0080 - 79 d9 90 0a 54 cc a5 e5-9f 7b a0 6d 58 5d f3 69 y...T....{.mX].i 0090 - fd 7d 0c e1 6b 17 c4 0c-0a 40 3c 51 dc 81 33 02 .}..k....@<Q..3. Start Time: 1587104752 Timeout : 300 (sec) Verify return code: 0 (ok) --- Actual results: The cert will not be updated in the haproxy container and connecting to haproxy will show the old cert. Expected results: The cert will be updated in the haproxy container and connecting to haproxy will show the new cert. Additional info:
Thank you for the report. Could you attach a full copy of the audit.log from the permissive run, so that we can look in details at the AVCs? Some of these don't look related to OpenStack. Thank you.
openstack-selinux and selinux-policy versions would be helpful as well.
Created attachment 1559999 [details] audit log from after the post run command in permissive mode
Attached the audit log. here is the version info. [stack@undercloud-0 ~]$ rpm -q openstack-selinux openstack-selinux-0.8.18-1.el7ost.noarch [stack@undercloud-0 ~]$ rpm -q selinux-policy selinux-policy-3.13.1-229.el7_6.12.noarch This is what the script thats running with the "reload external" passed options. [stack@undercloud-0 ~]$ sudo cat /usr/bin/certmonger-haproxy-refresh.sh #!/bin/bash # This script is meant to reload HAProxy when certmonger triggers a certificate # renewal. It'll concatenate the needed certificates for the PEM file that # HAProxy reads. die() { echo "$*" 1>&2 ; exit 1; } [[ $# -eq 2 ]] || die "Invalid number of arguments" [[ $1 == @(reload|restart) ]] || die "First argument must be one of 'reload' or 'restart'." ACTION=$1 NETWORK=$2 certmonger_ca=$(hiera -c /etc/puppet/hiera.yaml certmonger_ca) container_cli=$(hiera -c /etc/puppet/hiera.yaml container_cli docker) service_certificate="$(hiera -c /etc/puppet/hiera.yaml tripleo::certmonger::haproxy_dirs::certificate_dir)/overcloud-haproxy-$NETWORK.crt" service_key="$(hiera -c /etc/puppet/hiera.yaml tripleo::certmonger::haproxy_dirs::key_dir)/overcloud-haproxy-$NETWORK.key" ca_path="" if [ "$certmonger_ca" == "local" ]; then ca_path="/etc/pki/ca-trust/source/anchors/cm-local-ca.pem" elif [ "$certmonger_ca" == "IPA" ]; then ca_path="/etc/ipa/ca.crt" fi if [ "$NETWORK" != "external" ]; then service_pem="$(hiera -c /etc/puppet/hiera.yaml tripleo::certmonger::haproxy_dirs::certificate_dir)/overcloud-haproxy-$NETWORK.pem" else service_pem="$(hiera -c /etc/puppet/hiera.yaml tripleo::haproxy::service_certificate)" fi cat "$service_certificate" "$ca_path" "$service_key" > "$service_pem" haproxy_container_name=$($container_cli ps --format="{{.Names}}" | grep haproxy) if [ "$ACTION" == "reload" ]; then # Copy the new cert from the mount-point to the real path $container_cli exec "$haproxy_container_name" cp "/var/lib/kolla/config_files/src-tls$service_pem" "$service_pem" # Set appropriate permissions $container_cli exec "$haproxy_container_name" chown haproxy:haproxy "$service_pem" # Trigger a reload for HAProxy to read the new certificates pkill -f -HUP haproxy-systemd-wrapper elif [ "$ACTION" == "restart" ]; then # Copying the certificate and permissions will be handled by kolla's start # script. $container_cli restart "$haproxy_container_name" fi
Thank you for the audit.log and including the script details! This is very helpful. Where does the script come from, by the way? Is it something we ship in OpenStack? There are 140 unique AVC denials in the logs. I thought it strange that 121 of them are related to "pkill" and certmonger trying to kill everything, which led me to this upstream spec for Train/OSP16, https://specs.openstack.org/openstack/tripleo-specs/specs/train/certificate-management.html : "The main issue now is the use of "pkill", especially for httpd services. Since Certmonger has no knowledge of what container has an httpd service running, it uses a wide fly swatter in the hope all related services will effectively be reloaded with the new certificate." If we only look at the non-pkill AVC denials, the following rules (as well as enabling logrotate_read_inside_containers, which seems generally useful) are enough to resolve all denials but 3: allow certmonger_t puppet_etc_t:dir search; allow certmonger_t puppet_etc_t:file { getattr ioctl open read }; allow certmonger_t container_runtime_exec_t:file { execute execute_no_trans getattr ioctl open read }; allow certmonger_t container_runtime_t:dir { getattr search }; allow certmonger_t container_runtime_t:file { open read }; This generally seems in line with what the script does. The 3 other denials seem somewhat unrelated... type=AVC msg=audit(1588173023.389:4931): avc: denied { search } for pid=145752 comm="docker-current" name="net" dev="proc" ino=8799 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:sysctl_net_t:s0 tclass=dir permissive=1 bz1703242:11:type=AVC msg=audit(1588173023.389:4931): avc: denied { read } for pid=145752 comm="docker-current" name="somaxconn" dev="proc" ino=63146 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:sysctl_net_t:s0 tclass=file permissive=1 bz1703242:12:type=AVC msg=audit(1588173023.389:4931): avc: denied { open } for pid=145752 comm="docker-current" path="/proc/sys/net/core/somaxconn" dev="proc" ino=63146 scontext=system_u:system_r:certmonger_t:s0 tcontext=system_u:object_r:sysctl_net_t:s0 tclass=file permissive=1 I do see there is an explicit pkill call in the script, "pkill -f -HUP haproxy-systemd-wrapper" that I can't match explicitly to one of the pkill AVC denials in the logs. I'm not sure if the rules above are enough to handle that case as well. Jeremy, would you be able to run "ps -efZ | grep haproxy" so that we can see the full label for the haproxy-systemd-wrapper process? Thank you. I'd like to understand better how certmonger is meant to be handled in 14 until the Train upstream spec above is implemented... Cédric, I see you wrote up that spec, would you have any pointers?
Hello Julie, I indeed wrote that spec, but the whole certmonger and related TLS configurations are more from DFG:Security - I therefore add a needinfo() for Grzegorz (xek). pkill is used because we have shared certificates, and certmonger doesn't know what services uses said certificate when renewing them - therefore, the pkill was added in order to force-reload upon renewal. I'm not really sure we will be able to backport the spec content to 14 nor 15 though, so I'm pretty positive we will need something in-between, being some SELinux rules or equivalent. I don't have a "nice" solution for that, unfortunately :(. The rules you listed should indeed be OK, and shouldn't create big issues regarding system security. But if I understand you correctly, they aren't meant for the pkill calls directly, probably more for some process listing and the like, right? Cheers, C.
Thanks Cédric! I am hoping once we know the label for the haproxy-systemd-wrapper process this will be enough to narrow down the pkill rule we need, but it's also possible I misunderstand how this is all supposed to work. At the moment I doubt the rules I mentioned will be enough. Thanks for bringing in expertise from DFG:Security as well!
I believe most of the AVCs denials preventing the cert renewal should be resolved thanks to bug 1777368 and bug 1777263. We will want to rebase the OSP14 openstack-selinux package to include these patches. The read-only AVCs denials related to pkill will likely remain but that is sort of expected behaviour (running "pkill" involves reading through every process to get their info in order to get at the right one, so if the process is unrelated the caller (certmonger here) won't have permission to do anything with them.)
the "reload external" passed options are used on the controller: [heat-admin@overcloud-controller-0 ~]$ ps -efZ | grep haproxy-systemd-wrapper unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 heat-ad+ 125682 124084 0 09:37 pts/1 00:00:00 grep --color=auto haproxy-systemd-wrapper system_u:system_r:spc_t:s0 root 255232 255111 0 2019 ? 00:00:00 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg
I think this is resolved based on the bugs linked to in comment 8. The blocked bug is also closed, and links to another few bugs from that one show a number of related certmonger changes happened in other components (puppet-tripleo, THT, etc). Based on all this I am closing this bz, but feel free to reopen against a newer version if there are still related issues. Thank you.