RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2001082 - Certmonger certificates stuck in NEED_GUIDANCE
Summary: Certmonger certificates stuck in NEED_GUIDANCE
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: certmonger
Version: 9.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Rob Crittenden
QA Contact: ipa-qe
URL:
Whiteboard:
Depends On: 1992439
Blocks: 2001079
TreeView+ depends on / blocked
 
Reported: 2021-09-03 18:05 UTC by Rob Crittenden
Modified: 2022-05-17 13:29 UTC (History)
7 users (show)

Fixed In Version: certmonger-0.79.14-5.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1992439
Environment:
Last Closed: 2022-05-17 13:13:50 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker FREEIPA-6911 0 None None None 2021-09-23 15:37:56 UTC
Red Hat Issue Tracker RHELPLAN-96182 0 None None None 2021-09-03 18:07:00 UTC
Red Hat Product Errata RHBA-2022:2478 0 None None None 2022-05-17 13:13:56 UTC

Description Rob Crittenden 2021-09-03 18:05:05 UTC 
+++ This bug was initially created as a clone of Bug #1992439 +++

Description of problem:
When resubmit certificates through certmonger gets stuck in NEED_GUIDANCE.
This seems to be related to how long the certmonger process has been running. 

After restarting the service there is no issue resubmitting certificates

Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux Server release 7.9 (Maipo)
certmonger-0.78.4-14.el7.x86_64

How reproducible:
Hard to tell, it seems to be based on how long the service has been running. 
In the test environment where the certmonger has been up for 8 months. 
Of the 3 of 8 servers we tried to replicate on, we could replicate it each time.

Steps to Reproduce:
1. Submit certificates to IPA through certmonger
2. Wait an undetermined about of time
3. resubmit the certificates 

Actual results:

Request ID 'haproxy-storage_mgmt-cert':
	status: NEED_GUIDANCE
	stuck: yes
	key pair storage: type=FILE,location='/etc/pki/tls/private/haproxy/overcloud-haproxy-storage_mgmt.key'
	certificate: type=FILE,location='/etc/pki/tls/certs/haproxy/overcloud-haproxy-storage_mgmt.crt'
	CA: IPA
	issuer: CN=Certificate Authority,O=REDHAT.LOCAL
	subject: CN=controller-0.storagemgmt.redhat.local,O=REDHAT.LOCAL
	expires: 2023-04-10 05:18:02 UTC
	dns: overcloud.storagemgmt.redhat.local,controller-0.storagemgmt.redhat.local
	principal name: haproxy/controller-0.storagemgmt.redhat.local
	key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
	eku: id-kp-serverAuth,id-kp-clientAuth
	pre-save command: 
	post-save command: /usr/bin/certmonger-haproxy-refresh.sh reload storage_mgmt
	track: yes
	auto-renew: yes

Expected results:

Request ID 'haproxy-storage_mgmt-cert':
	status: MONITORING
	stuck: no
	key pair storage: type=FILE,location='/etc/pki/tls/private/haproxy/overcloud-haproxy-storage_mgmt.key'
	certificate: type=FILE,location='/etc/pki/tls/certs/haproxy/overcloud-haproxy-storage_mgmt.crt'
	CA: IPA
	issuer: CN=Certificate Authority,O=REDHAT.LOCAL
	subject: CN=controller-0.storagemgmt.redhat.local,O=REDHAT.LOCAL
	expires: 2023-04-10 05:18:02 UTC
	dns: overcloud.storagemgmt.redhat.local,controller-0.storagemgmt.redhat.local
	principal name: haproxy/controller-0.storagemgmt.redhat.local
	key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
	eku: id-kp-serverAuth,id-kp-clientAuth
	pre-save command: 
	post-save command: /usr/bin/certmonger-haproxy-refresh.sh reload storage_mgmt
	track: yes
	auto-renew: yes


Additional info:


From what I can see from the strace is there Bad file descriptor when try to read -1, 0x55f57c3507d0, 8192 right after sending the submitting status to dbus. 

After this is goes into NEED_GUIDANCE
~~~
15104 1628051842.134790 close(1022</var/lib/certmonger/requests/20201112060423.tmp>) = 0 <0.000019>
15104 1628051842.134844 munmap(0x7fde2bcec000, 4096) = 0 <0.000022>
15104 1628051842.134895 rename("/var/lib/certmonger/requests/20201112060423.tmp", "/var/lib/certmonger/requests/20201112060423") = 0 <0.000051>
15104 1628051842.135021 sendmsg(7<UNIX:[63674->62964]>, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\4\1\1O\0\0\0\230\34\0\0\214\0\0\0\1\1o\0.\0\0\0/org/fedorahosted/certmonger/requests/Request5\0\0\2\1s\0\37\0\0\0org.freedesktop.
DBus.Properties\0\3\1s\0\21\0\0\0PropertiesChanged\0\0\0\0\0\0\0\10\1g\0\6sa{sv}\0\0\0\0\0", iov_len=160}, {iov_base="#\0\0\0org.fedorahosted.certmonger.request\0\37\0\0\0\0\0\0\0\6\0\0\0status\0\1s\0\0\0\n\0\0\0SUBMITTING\0", iov_len=79}
], msg_iovlen=2, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 239 <0.000025>
15104 1628051842.135103 epoll_wait(3<anon_inode:[eventpoll]>, [], 1, 4999) = 0 <5.000301>
15104 1628051847.135591 read(-1, 0x55f57c3507d0, 8192) = - <0.000030>
15104 1628051847.135691 close(-1)       = -1 EBADF (Bad file descriptor) <0.000022>
15104 1628051847.135752 wait4(0, 0x55f57c32e0bc, 0, NULL) = -1 ECHILD (No child processes) <0.000018>
15104 1628051847.135837 open("/var/lib/certmonger/requests/20201112060423.tmp", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 1022</var/lib/certmonger/requests/20201112060423.tmp> <0.000130>
15104 1628051847.136067 fstat(1022</var/lib/certmonger/requests/20201112060423.tmp>, {st_dev=makedev(252, 2), st_ino=14349509, st_mode=S_IFREG|0600, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=0, st_atime=1628051
847 /* 2021-08-04T04:37:27.135056204+0000 */, st_atime_nsec=135056204, st_mtime=1628051847 /* 2021-08-04T04:37:27.135056204+0000 */, st_mtime_nsec=135056204, st_ctime=1628051847 /* 2021-08-04T04:37:27.135056204+0000 */, st_ctime_nsec=1350
56204}) = 0 <0.000023>
15104 1628051847.136171 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fde2bcec000 <0.000028>
~~~

--- Additional comment from Rob Crittenden on 2021-09-03 17:37:08 UTC ---

Fixed upstream
master: b4c090d2e12956a2df6157592839936adf4024f4
Comment 1 Rob Crittenden 2021-09-23 15:37:08 UTC 
Fixed upstream
master: b4c090d2e12956a2df6157592839936adf4024f4
Comment 8 Mohammad Rizwan 2021-12-06 07:36:30 UTC 
version:
certmonger-0.79.14-5.el9.x86_64

[root@master ~]#  lsof -p `pidof certmonger` 
COMMAND     PID USER   FD      TYPE             DEVICE SIZE/OFF     NODE NAME
certmonge 23413 root  cwd       DIR              252,3      235      128 /
certmonge 23413 root  rtd       DIR              252,3      235      128 /
certmonge 23413 root  txt       REG              252,3   478144 25986524 /usr/sbin/certmonger
certmonge 23413 root  mem       REG              252,3   617376      403 /usr/lib64/libpcre2-8.so.0.10.2
certmonge 23413 root  mem       REG              252,3   153568      373 /usr/lib64/libgpg-error.so.0.32.0
certmonge 23413 root  mem       REG              252,3   359688      622 /usr/lib64/libgssapi_krb5.so.2.2
certmonge 23413 root  mem       REG              252,3   201808      338 /usr/lib64/libcrypt.so.2.0.0
certmonge 23413 root  mem       REG              252,3   172632      406 /usr/lib64/libselinux.so.1
certmonge 23413 root  mem       REG              252,3   108128      159 /usr/lib64/libgcc_s-11-20211019.so.1
certmonge 23413 root  mem       REG              252,3  1301680      424 /usr/lib64/libgcrypt.so.20.3.3
certmonge 23413 root  mem       REG              252,3    36912      330 /usr/lib64/libcap.so.2.48
certmonge 23413 root  mem       REG              252,3   144120      401 /usr/lib64/liblz4.so.1.9.3
certmonge 23413 root  mem       REG              252,3  1004496      336 /usr/lib64/libzstd.so.1.5.0
certmonge 23413 root  mem       REG              252,3   178704      203 /usr/lib64/liblzma.so.5.2.5
certmonge 23413 root  mem       REG              252,3   685216      613 /usr/lib64/libssl.so.3.0.0
certmonge 23413 root  mem       REG              252,3   129496      839 /usr/lib64/libsasl2.so.3.0.0
certmonge 23413 root  mem       REG              252,3    70864      841 /usr/lib64/liblber-2.4.so.2.11.7
certmonge 23413 root  mem       REG              252,3  1591920      399 /usr/lib64/libunistring.so.2.1.0
certmonge 23413 root  mem       REG              252,3    71424      174 /usr/lib64/libresolv.so.2
certmonge 23413 root  mem       REG              252,3    24616      371 /usr/lib64/libkeyutils.so.1.9
certmonge 23413 root  mem       REG              252,3    66880      634 /usr/lib64/libkrb5support.so.0.1
certmonge 23413 root  mem       REG              252,3    24384      348 /usr/lib64/libcom_err.so.2.1
certmonge 23413 root  mem       REG              252,3    99872      626 /usr/lib64/libk5crypto.so.3.1
certmonge 23413 root  mem       REG              252,3   871512      618 /usr/lib64/libsystemd.so.0.32.0
certmonge 23413 root  mem       REG              252,3   103184      201 /usr/lib64/libz.so.1.2.11
certmonge 23413 root  mem       REG              252,3  2389584      165 /usr/lib64/libc.so.6
certmonge 23413 root  mem       REG              252,3   365976      843 /usr/lib64/libldap_r-2.4.so.2.11.7
certmonge 23413 root  mem       REG              252,3    58368      334 /usr/lib64/libpopt.so.0.0.1
certmonge 23413 root  mem       REG              252,3    32816      344 /usr/lib64/libuuid.so.1.3.0
certmonge 23413 root  mem       REG              252,3   130952      409 /usr/lib64/libidn2.so.0.3.7
certmonge 23413 root  mem       REG              252,3   920296      632 /usr/lib64/libkrb5.so.3.3
certmonge 23413 root  mem       REG              252,3   271040   690910 /usr/lib64/libnspr4.so
certmonge 23413 root  mem       REG              252,3    25128   690911 /usr/lib64/libplc4.so
certmonge 23413 root  mem       REG              252,3    16784   690912 /usr/lib64/libplds4.so
certmonge 23413 root  mem       REG              252,3   212392   690913 /usr/lib64/libnssutil3.so
certmonge 23413 root  mem       REG              252,3  1322048   706239 /usr/lib64/libnss3.so
certmonge 23413 root  mem       REG              252,3   180856   706240 /usr/lib64/libsmime3.so
certmonge 23413 root  mem       REG              252,3   424368   706241 /usr/lib64/libssl3.so
certmonge 23413 root  mem       REG              252,3    58192      375 /usr/lib64/libtalloc.so.2.3.2
certmonge 23413 root  mem       REG              252,3    87976      421 /usr/lib64/libtevent.so.0.11.0
certmonge 23413 root  mem       REG              252,3   340824      638 /usr/lib64/libdbus-1.so.3.19.13
certmonge 23413 root  mem       REG              252,3  4446224      611 /usr/lib64/libcrypto.so.3.0.0
certmonge 23413 root  mem       REG              252,3   884984      161 /usr/lib64/ld-linux-x86-64.so.2
certmonge 23413 root    0r      CHR                1,3      0t0        4 /dev/null
certmonge 23413 root    1u     unix 0xffff8bc58400f740      0t0    68929 type=STREAM (CONNECTED)
certmonge 23413 root    2u     unix 0xffff8bc58400f740      0t0    68929 type=STREAM (CONNECTED)
certmonge 23413 root    3u  a_inode               0,14        0     9754 [eventpoll:5,6,7]
certmonge 23413 root    4uW     REG              252,3        0 42044516 /var/lib/certmonger/lock
certmonge 23413 root    5u  a_inode               0,14        0     9754 [eventfd:14]
certmonge 23413 root    6u  netlink                         0t0    68932 ROUTE
certmonge 23413 root    7u     unix 0xffff8bc58400f300      0t0    68933 type=STREAM (CONNECTED)
certmonge 23413 root    8uW     REG               0,25        6     1445 /run/certmonger.pid


As we can see expected 8 open file descriptor. Marking the bug as verified.
Comment 10 errata-xmlrpc 2022-05-17 13:13:50 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: certmonger), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2478
Note You need to log in before you can comment on or make changes to this bug.