RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2001079 - Certmonger certificates stuck in NEED_GUIDANCE
Summary: Certmonger certificates stuck in NEED_GUIDANCE
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: certmonger
Version: 8.5
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Rob Crittenden
QA Contact: ipa-qe
URL:
Whiteboard:
Depends On: 1992439 2001082
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-09-03 18:00 UTC by Rob Crittenden
Modified: 2022-05-10 14:01 UTC (History)
8 users (show)

Fixed In Version: certmonger-0.79.13-4.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1992439
Environment:
Last Closed: 2022-05-10 13:38:10 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker FREEIPA-6870 0 None None None 2021-09-21 13:59:24 UTC
Red Hat Issue Tracker RHELPLAN-96181 0 None None None 2021-09-03 18:02:56 UTC
Red Hat Knowledge Base (Solution) 6759001 0 None None None 2022-02-25 04:26:49 UTC
Red Hat Product Errata RHBA-2022:1789 0 None None None 2022-05-10 13:38:20 UTC

Description Rob Crittenden 2021-09-03 18:00:52 UTC 
+++ This bug was initially created as a clone of Bug #1992439 +++

Description of problem:
When resubmit certificates through certmonger gets stuck in NEED_GUIDANCE.
This seems to be related to how long the certmonger process has been running. 

After restarting the service there is no issue resubmitting certificates

Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux Server release 7.9 (Maipo)
certmonger-0.78.4-14.el7.x86_64

How reproducible:
Hard to tell, it seems to be based on how long the service has been running. 
In the test environment where the certmonger has been up for 8 months. 
Of the 3 of 8 servers we tried to replicate on, we could replicate it each time.

Steps to Reproduce:
1. Submit certificates to IPA through certmonger
2. Wait an undetermined about of time
3. resubmit the certificates 

Actual results:

Request ID 'haproxy-storage_mgmt-cert':
	status: NEED_GUIDANCE
	stuck: yes
	key pair storage: type=FILE,location='/etc/pki/tls/private/haproxy/overcloud-haproxy-storage_mgmt.key'
	certificate: type=FILE,location='/etc/pki/tls/certs/haproxy/overcloud-haproxy-storage_mgmt.crt'
	CA: IPA
	issuer: CN=Certificate Authority,O=REDHAT.LOCAL
	subject: CN=controller-0.storagemgmt.redhat.local,O=REDHAT.LOCAL
	expires: 2023-04-10 05:18:02 UTC
	dns: overcloud.storagemgmt.redhat.local,controller-0.storagemgmt.redhat.local
	principal name: haproxy/controller-0.storagemgmt.redhat.local
	key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
	eku: id-kp-serverAuth,id-kp-clientAuth
	pre-save command: 
	post-save command: /usr/bin/certmonger-haproxy-refresh.sh reload storage_mgmt
	track: yes
	auto-renew: yes

Expected results:

Request ID 'haproxy-storage_mgmt-cert':
	status: MONITORING
	stuck: no
	key pair storage: type=FILE,location='/etc/pki/tls/private/haproxy/overcloud-haproxy-storage_mgmt.key'
	certificate: type=FILE,location='/etc/pki/tls/certs/haproxy/overcloud-haproxy-storage_mgmt.crt'
	CA: IPA
	issuer: CN=Certificate Authority,O=REDHAT.LOCAL
	subject: CN=controller-0.storagemgmt.redhat.local,O=REDHAT.LOCAL
	expires: 2023-04-10 05:18:02 UTC
	dns: overcloud.storagemgmt.redhat.local,controller-0.storagemgmt.redhat.local
	principal name: haproxy/controller-0.storagemgmt.redhat.local
	key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
	eku: id-kp-serverAuth,id-kp-clientAuth
	pre-save command: 
	post-save command: /usr/bin/certmonger-haproxy-refresh.sh reload storage_mgmt
	track: yes
	auto-renew: yes


Additional info:


From what I can see from the strace is there Bad file descriptor when try to read -1, 0x55f57c3507d0, 8192 right after sending the submitting status to dbus. 

After this is goes into NEED_GUIDANCE
~~~
15104 1628051842.134790 close(1022</var/lib/certmonger/requests/20201112060423.tmp>) = 0 <0.000019>
15104 1628051842.134844 munmap(0x7fde2bcec000, 4096) = 0 <0.000022>
15104 1628051842.134895 rename("/var/lib/certmonger/requests/20201112060423.tmp", "/var/lib/certmonger/requests/20201112060423") = 0 <0.000051>
15104 1628051842.135021 sendmsg(7<UNIX:[63674->62964]>, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\4\1\1O\0\0\0\230\34\0\0\214\0\0\0\1\1o\0.\0\0\0/org/fedorahosted/certmonger/requests/Request5\0\0\2\1s\0\37\0\0\0org.freedesktop.
DBus.Properties\0\3\1s\0\21\0\0\0PropertiesChanged\0\0\0\0\0\0\0\10\1g\0\6sa{sv}\0\0\0\0\0", iov_len=160}, {iov_base="#\0\0\0org.fedorahosted.certmonger.request\0\37\0\0\0\0\0\0\0\6\0\0\0status\0\1s\0\0\0\n\0\0\0SUBMITTING\0", iov_len=79}
], msg_iovlen=2, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 239 <0.000025>
15104 1628051842.135103 epoll_wait(3<anon_inode:[eventpoll]>, [], 1, 4999) = 0 <5.000301>
15104 1628051847.135591 read(-1, 0x55f57c3507d0, 8192) = - <0.000030>
15104 1628051847.135691 close(-1)       = -1 EBADF (Bad file descriptor) <0.000022>
15104 1628051847.135752 wait4(0, 0x55f57c32e0bc, 0, NULL) = -1 ECHILD (No child processes) <0.000018>
15104 1628051847.135837 open("/var/lib/certmonger/requests/20201112060423.tmp", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 1022</var/lib/certmonger/requests/20201112060423.tmp> <0.000130>
15104 1628051847.136067 fstat(1022</var/lib/certmonger/requests/20201112060423.tmp>, {st_dev=makedev(252, 2), st_ino=14349509, st_mode=S_IFREG|0600, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=0, st_atime=1628051
847 /* 2021-08-04T04:37:27.135056204+0000 */, st_atime_nsec=135056204, st_mtime=1628051847 /* 2021-08-04T04:37:27.135056204+0000 */, st_mtime_nsec=135056204, st_ctime=1628051847 /* 2021-08-04T04:37:27.135056204+0000 */, st_ctime_nsec=1350
56204}) = 0 <0.000023>
15104 1628051847.136171 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fde2bcec000 <0.000028>
~~~

--- Additional comment from Rob Crittenden on 2021-09-03 17:37:08 UTC ---

Fixed upstream
master: b4c090d2e12956a2df6157592839936adf4024f4
Comment 1 Rob Crittenden 2021-09-23 15:36:59 UTC 
Fixed upstream
master: b4c090d2e12956a2df6157592839936adf4024f4
Comment 7 Mohammad Rizwan 2021-11-25 06:01:52 UTC 
Steps:
1. install ipa-server
2. install lsfo packge
3. Do $lsof -p `pidof certmonger`, the open numbered file descriptor should be 8.


[root@master ~]# lsof -p `pidof certmonger`
COMMAND     PID USER   FD      TYPE             DEVICE SIZE/OFF     NODE NAME
certmonge 24235 root  cwd       DIR              252,3      224      128 /
certmonge 24235 root  rtd       DIR              252,3      224      128 /
certmonge 24235 root  txt       REG              252,3   518280 26259739 /usr/sbin/certmonger
certmonge 24235 root  mem       REG              252,3   543160      385 /usr/lib64/libpcre2-8.so.0.7.1
certmonge 24235 root  mem       REG              252,3   144392      460 /usr/lib64/libgpg-error.so.0.24.2
certmonge 24235 root  mem       REG              252,3   343600     1797 /usr/lib64/libblkid.so.1.1.0
certmonge 24235 root  mem       REG              252,3   355920     1750 /usr/lib64/libgssapi_krb5.so.2.2
certmonge 24235 root  mem       REG              252,3   136040      468 /usr/lib64/libcrypt.so.1.1.0
certmonge 24235 root  mem       REG              252,3   168536      388 /usr/lib64/libselinux.so.1
certmonge 24235 root  mem       REG              252,3    99672      380 /usr/lib64/libgcc_s-8-20210514.so.1
certmonge 24235 root  mem       REG              252,3  1187360      509 /usr/lib64/libgcrypt.so.20.2.5
certmonge 24235 root  mem       REG              252,3   371384     1786 /usr/lib64/libmount.so.1.1.0
certmonge 24235 root  mem       REG              252,3    25112      454 /usr/lib64/libcap.so.2.26
certmonge 24235 root  mem       REG              252,3   119760      544 /usr/lib64/liblz4.so.1.8.3
certmonge 24235 root  mem       REG              252,3   192016      446 /usr/lib64/liblzma.so.5.2.4
certmonge 24235 root  mem       REG              252,3   615704     1806 /usr/lib64/libssl.so.1.1.1k
certmonge 24235 root  mem       REG              252,3   125328      747 /usr/lib64/libsasl2.so.3.0.0
certmonge 24235 root  mem       REG              252,3    67232     1729 /usr/lib64/liblber-2.4.so.2.10.9
certmonge 24235 root  mem       REG              252,3  1760264      511 /usr/lib64/libunistring.so.2.1.0
certmonge 24235 root  mem       REG              252,3   123264      432 /usr/lib64/libresolv-2.28.so
certmonge 24235 root  mem       REG              252,3    16192      486 /usr/lib64/libkeyutils.so.1.6
certmonge 24235 root  mem       REG              252,3    71480     1762 /usr/lib64/libkrb5support.so.0.1
certmonge 24235 root  mem       REG              252,3    69120      434 /usr/lib64/librt-2.28.so
certmonge 24235 root  mem       REG              252,3  1375560     1793 /usr/lib64/libsystemd.so.0.23.0
certmonge 24235 root  mem       REG              252,3    95416      444 /usr/lib64/libz.so.1.2.11
certmonge 24235 root  mem       REG              252,3  3167176      416 /usr/lib64/libc-2.28.so
certmonge 24235 root  mem       REG              252,3   329184     1731 /usr/lib64/libldap-2.4.so.2.10.9
certmonge 24235 root  mem       REG              252,3    54688      448 /usr/lib64/libpopt.so.0.0.1
certmonge 24235 root  mem       REG              252,3    33480      464 /usr/lib64/libuuid.so.1.3.0
certmonge 24235 root  mem       REG              252,3   162224      513 /usr/lib64/libidn2.so.0.3.6
certmonge 24235 root  mem       REG              252,3    16776      450 /usr/lib64/libcom_err.so.2.1
certmonge 24235 root  mem       REG              252,3    96216     1754 /usr/lib64/libk5crypto.so.3.1
certmonge 24235 root  mem       REG              252,3   971456     1760 /usr/lib64/libkrb5.so.3.3
certmonge 24235 root  mem       REG              252,3    28840      418 /usr/lib64/libdl-2.28.so
certmonge 24235 root  mem       REG              252,3   321536      430 /usr/lib64/libpthread-2.28.so
certmonge 24235 root  mem       REG              252,3   262992    13044 /usr/lib64/libnspr4.so
certmonge 24235 root  mem       REG              252,3    20992    13045 /usr/lib64/libplc4.so
certmonge 24235 root  mem       REG              252,3    16712    13046 /usr/lib64/libplds4.so
certmonge 24235 root  mem       REG              252,3   206728    13047 /usr/lib64/libnssutil3.so
certmonge 24235 root  mem       REG              252,3  1308184  1081440 /usr/lib64/libnss3.so
certmonge 24235 root  mem       REG              252,3   175136  1081441 /usr/lib64/libsmime3.so
certmonge 24235 root  mem       REG              252,3   419760  1081442 /usr/lib64/libssl3.so
certmonge 24235 root  mem       REG              252,3    87536      505 /usr/lib64/libtalloc.so.2.3.2
certmonge 24235 root  mem       REG              252,3    87792      546 /usr/lib64/libtevent.so.0.11.0
certmonge 24235 root  mem       REG              252,3   351576     1788 /usr/lib64/libdbus-1.so.3.19.7
certmonge 24235 root  mem       REG              252,3  3079592     1804 /usr/lib64/libcrypto.so.1.1.1k
certmonge 24235 root  mem       REG              252,3   273696      409 /usr/lib64/ld-2.28.so
certmonge 24235 root    0r      CHR                1,3      0t0     9371 /dev/null
certmonge 24235 root    1u     unix 0xffffa07674e7b600      0t0    94365 type=STREAM
certmonge 24235 root    2u     unix 0xffffa07674e7b600      0t0    94365 type=STREAM
certmonge 24235 root    3u  a_inode               0,14        0     9365 [eventpoll]
certmonge 24235 root    4uW     REG              252,3        0 33654365 /var/lib/certmonger/lock
certmonge 24235 root    5u  a_inode               0,14        0     9365 [eventfd]
certmonge 24235 root    6u  netlink                         0t0    93624 ROUTE
certmonge 24235 root    7u     unix 0xffffa0774dea0d80      0t0    93625 type=STREAM
certmonge 24235 root    8uW     REG               0,24        6    93627 /run/certmonger.pid
[root@master ~]# 
[root@master ~]# 
[root@master ~]# rpm -qa | grep certmo
certmonger-0.79.13-5.el8.x86_64
[root@master ~]# 
[root@master ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.6 Beta (Ootpa)

As we can see expected open file descriptor. Marking the bug as verified.
Comment 11 errata-xmlrpc 2022-05-10 13:38:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (certmonger bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1789
Note You need to log in before you can comment on or make changes to this bug.