Bug 2001082

Summary: Certmonger certificates stuck in NEED_GUIDANCE
Product: Red Hat Enterprise Linux 9 Reporter: Rob Crittenden <rcritten>
Component: certmongerAssignee: Rob Crittenden <rcritten>
Status: CLOSED ERRATA QA Contact: ipa-qe <ipa-qe>
Severity: medium Docs Contact:
Priority: medium    
Version: 9.0CC: dsedgmen, ipa-qe, juqiao, ksiddiqu, myusuf, rcritten, sumenon
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: certmonger-0.79.14-5.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1992439 Environment:
Last Closed: 2022-05-17 13:13:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1992439    
Bug Blocks: 2001079    

Description Rob Crittenden 2021-09-03 18:05:05 UTC 
+++ This bug was initially created as a clone of Bug #1992439 +++

Description of problem:
When resubmit certificates through certmonger gets stuck in NEED_GUIDANCE.
This seems to be related to how long the certmonger process has been running. 

After restarting the service there is no issue resubmitting certificates

Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux Server release 7.9 (Maipo)
certmonger-0.78.4-14.el7.x86_64

How reproducible:
Hard to tell, it seems to be based on how long the service has been running. 
In the test environment where the certmonger has been up for 8 months. 
Of the 3 of 8 servers we tried to replicate on, we could replicate it each time.

Steps to Reproduce:
1. Submit certificates to IPA through certmonger
2. Wait an undetermined about of time
3. resubmit the certificates 

Actual results:

Request ID 'haproxy-storage_mgmt-cert':
	status: NEED_GUIDANCE
	stuck: yes
	key pair storage: type=FILE,location='/etc/pki/tls/private/haproxy/overcloud-haproxy-storage_mgmt.key'
	certificate: type=FILE,location='/etc/pki/tls/certs/haproxy/overcloud-haproxy-storage_mgmt.crt'
	CA: IPA
	issuer: CN=Certificate Authority,O=REDHAT.LOCAL
	subject: CN=controller-0.storagemgmt.redhat.local,O=REDHAT.LOCAL
	expires: 2023-04-10 05:18:02 UTC
	dns: overcloud.storagemgmt.redhat.local,controller-0.storagemgmt.redhat.local
	principal name: haproxy/controller-0.storagemgmt.redhat.local
	key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
	eku: id-kp-serverAuth,id-kp-clientAuth
	pre-save command: 
	post-save command: /usr/bin/certmonger-haproxy-refresh.sh reload storage_mgmt
	track: yes
	auto-renew: yes

Expected results:

Request ID 'haproxy-storage_mgmt-cert':
	status: MONITORING
	stuck: no
	key pair storage: type=FILE,location='/etc/pki/tls/private/haproxy/overcloud-haproxy-storage_mgmt.key'
	certificate: type=FILE,location='/etc/pki/tls/certs/haproxy/overcloud-haproxy-storage_mgmt.crt'
	CA: IPA
	issuer: CN=Certificate Authority,O=REDHAT.LOCAL
	subject: CN=controller-0.storagemgmt.redhat.local,O=REDHAT.LOCAL
	expires: 2023-04-10 05:18:02 UTC
	dns: overcloud.storagemgmt.redhat.local,controller-0.storagemgmt.redhat.local
	principal name: haproxy/controller-0.storagemgmt.redhat.local
	key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
	eku: id-kp-serverAuth,id-kp-clientAuth
	pre-save command: 
	post-save command: /usr/bin/certmonger-haproxy-refresh.sh reload storage_mgmt
	track: yes
	auto-renew: yes


Additional info:


From what I can see from the strace is there Bad file descriptor when try to read -1, 0x55f57c3507d0, 8192 right after sending the submitting status to dbus. 

After this is goes into NEED_GUIDANCE
~~~
15104 1628051842.134790 close(1022</var/lib/certmonger/requests/20201112060423.tmp>) = 0 <0.000019>
15104 1628051842.134844 munmap(0x7fde2bcec000, 4096) = 0 <0.000022>
15104 1628051842.134895 rename("/var/lib/certmonger/requests/20201112060423.tmp", "/var/lib/certmonger/requests/20201112060423") = 0 <0.000051>
15104 1628051842.135021 sendmsg(7<UNIX:[63674->62964]>, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\4\1\1O\0\0\0\230\34\0\0\214\0\0\0\1\1o\0.\0\0\0/org/fedorahosted/certmonger/requests/Request5\0\0\2\1s\0\37\0\0\0org.freedesktop.
DBus.Properties\0\3\1s\0\21\0\0\0PropertiesChanged\0\0\0\0\0\0\0\10\1g\0\6sa{sv}\0\0\0\0\0", iov_len=160}, {iov_base="#\0\0\0org.fedorahosted.certmonger.request\0\37\0\0\0\0\0\0\0\6\0\0\0status\0\1s\0\0\0\n\0\0\0SUBMITTING\0", iov_len=79}
], msg_iovlen=2, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 239 <0.000025>
15104 1628051842.135103 epoll_wait(3<anon_inode:[eventpoll]>, [], 1, 4999) = 0 <5.000301>
15104 1628051847.135591 read(-1, 0x55f57c3507d0, 8192) = - <0.000030>
15104 1628051847.135691 close(-1)       = -1 EBADF (Bad file descriptor) <0.000022>
15104 1628051847.135752 wait4(0, 0x55f57c32e0bc, 0, NULL) = -1 ECHILD (No child processes) <0.000018>
15104 1628051847.135837 open("/var/lib/certmonger/requests/20201112060423.tmp", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 1022</var/lib/certmonger/requests/20201112060423.tmp> <0.000130>
15104 1628051847.136067 fstat(1022</var/lib/certmonger/requests/20201112060423.tmp>, {st_dev=makedev(252, 2), st_ino=14349509, st_mode=S_IFREG|0600, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=0, st_atime=1628051
847 /* 2021-08-04T04:37:27.135056204+0000 */, st_atime_nsec=135056204, st_mtime=1628051847 /* 2021-08-04T04:37:27.135056204+0000 */, st_mtime_nsec=135056204, st_ctime=1628051847 /* 2021-08-04T04:37:27.135056204+0000 */, st_ctime_nsec=1350
56204}) = 0 <0.000023>
15104 1628051847.136171 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fde2bcec000 <0.000028>
~~~

--- Additional comment from Rob Crittenden on 2021-09-03 17:37:08 UTC ---

Fixed upstream
master: b4c090d2e12956a2df6157592839936adf4024f4
Comment 1 Rob Crittenden 2021-09-23 15:37:08 UTC 
Fixed upstream
master: b4c090d2e12956a2df6157592839936adf4024f4
Comment 8 Mohammad Rizwan 2021-12-06 07:36:30 UTC 
version:
certmonger-0.79.14-5.el9.x86_64

[root@master ~]#  lsof -p `pidof certmonger` 
COMMAND     PID USER   FD      TYPE             DEVICE SIZE/OFF     NODE NAME
certmonge 23413 root  cwd       DIR              252,3      235      128 /
certmonge 23413 root  rtd       DIR              252,3      235      128 /
certmonge 23413 root  txt       REG              252,3   478144 25986524 /usr/sbin/certmonger
certmonge 23413 root  mem       REG              252,3   617376      403 /usr/lib64/libpcre2-8.so.0.10.2
certmonge 23413 root  mem       REG              252,3   153568      373 /usr/lib64/libgpg-error.so.0.32.0
certmonge 23413 root  mem       REG              252,3   359688      622 /usr/lib64/libgssapi_krb5.so.2.2
certmonge 23413 root  mem       REG              252,3   201808      338 /usr/lib64/libcrypt.so.2.0.0
certmonge 23413 root  mem       REG              252,3   172632      406 /usr/lib64/libselinux.so.1
certmonge 23413 root  mem       REG              252,3   108128      159 /usr/lib64/libgcc_s-11-20211019.so.1
certmonge 23413 root  mem       REG              252,3  1301680      424 /usr/lib64/libgcrypt.so.20.3.3
certmonge 23413 root  mem       REG              252,3    36912      330 /usr/lib64/libcap.so.2.48
certmonge 23413 root  mem       REG              252,3   144120      401 /usr/lib64/liblz4.so.1.9.3
certmonge 23413 root  mem       REG              252,3  1004496      336 /usr/lib64/libzstd.so.1.5.0
certmonge 23413 root  mem       REG              252,3   178704      203 /usr/lib64/liblzma.so.5.2.5
certmonge 23413 root  mem       REG              252,3   685216      613 /usr/lib64/libssl.so.3.0.0
certmonge 23413 root  mem       REG              252,3   129496      839 /usr/lib64/libsasl2.so.3.0.0
certmonge 23413 root  mem       REG              252,3    70864      841 /usr/lib64/liblber-2.4.so.2.11.7
certmonge 23413 root  mem       REG              252,3  1591920      399 /usr/lib64/libunistring.so.2.1.0
certmonge 23413 root  mem       REG              252,3    71424      174 /usr/lib64/libresolv.so.2
certmonge 23413 root  mem       REG              252,3    24616      371 /usr/lib64/libkeyutils.so.1.9
certmonge 23413 root  mem       REG              252,3    66880      634 /usr/lib64/libkrb5support.so.0.1
certmonge 23413 root  mem       REG              252,3    24384      348 /usr/lib64/libcom_err.so.2.1
certmonge 23413 root  mem       REG              252,3    99872      626 /usr/lib64/libk5crypto.so.3.1
certmonge 23413 root  mem       REG              252,3   871512      618 /usr/lib64/libsystemd.so.0.32.0
certmonge 23413 root  mem       REG              252,3   103184      201 /usr/lib64/libz.so.1.2.11
certmonge 23413 root  mem       REG              252,3  2389584      165 /usr/lib64/libc.so.6
certmonge 23413 root  mem       REG              252,3   365976      843 /usr/lib64/libldap_r-2.4.so.2.11.7
certmonge 23413 root  mem       REG              252,3    58368      334 /usr/lib64/libpopt.so.0.0.1
certmonge 23413 root  mem       REG              252,3    32816      344 /usr/lib64/libuuid.so.1.3.0
certmonge 23413 root  mem       REG              252,3   130952      409 /usr/lib64/libidn2.so.0.3.7
certmonge 23413 root  mem       REG              252,3   920296      632 /usr/lib64/libkrb5.so.3.3
certmonge 23413 root  mem       REG              252,3   271040   690910 /usr/lib64/libnspr4.so
certmonge 23413 root  mem       REG              252,3    25128   690911 /usr/lib64/libplc4.so
certmonge 23413 root  mem       REG              252,3    16784   690912 /usr/lib64/libplds4.so
certmonge 23413 root  mem       REG              252,3   212392   690913 /usr/lib64/libnssutil3.so
certmonge 23413 root  mem       REG              252,3  1322048   706239 /usr/lib64/libnss3.so
certmonge 23413 root  mem       REG              252,3   180856   706240 /usr/lib64/libsmime3.so
certmonge 23413 root  mem       REG              252,3   424368   706241 /usr/lib64/libssl3.so
certmonge 23413 root  mem       REG              252,3    58192      375 /usr/lib64/libtalloc.so.2.3.2
certmonge 23413 root  mem       REG              252,3    87976      421 /usr/lib64/libtevent.so.0.11.0
certmonge 23413 root  mem       REG              252,3   340824      638 /usr/lib64/libdbus-1.so.3.19.13
certmonge 23413 root  mem       REG              252,3  4446224      611 /usr/lib64/libcrypto.so.3.0.0
certmonge 23413 root  mem       REG              252,3   884984      161 /usr/lib64/ld-linux-x86-64.so.2
certmonge 23413 root    0r      CHR                1,3      0t0        4 /dev/null
certmonge 23413 root    1u     unix 0xffff8bc58400f740      0t0    68929 type=STREAM (CONNECTED)
certmonge 23413 root    2u     unix 0xffff8bc58400f740      0t0    68929 type=STREAM (CONNECTED)
certmonge 23413 root    3u  a_inode               0,14        0     9754 [eventpoll:5,6,7]
certmonge 23413 root    4uW     REG              252,3        0 42044516 /var/lib/certmonger/lock
certmonge 23413 root    5u  a_inode               0,14        0     9754 [eventfd:14]
certmonge 23413 root    6u  netlink                         0t0    68932 ROUTE
certmonge 23413 root    7u     unix 0xffff8bc58400f300      0t0    68933 type=STREAM (CONNECTED)
certmonge 23413 root    8uW     REG               0,25        6     1445 /run/certmonger.pid


As we can see expected 8 open file descriptor. Marking the bug as verified.
Comment 10 errata-xmlrpc 2022-05-17 13:13:50 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: certmonger), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2478