Bug 1992439
| Summary: | Certmonger certificates stuck in NEED_GUIDANCE | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | David Sedgmen <dsedgmen> | |
| Component: | certmonger | Assignee: | Rob Crittenden <rcritten> | |
| Status: | CLOSED ERRATA | QA Contact: | ipa-qe <ipa-qe> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 7.9 | CC: | amore, jreznik, juqiao, rcritten, shtiwari, ssidhaye, tapazogl | |
| Target Milestone: | rc | Keywords: | Triaged, ZStream | |
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
|
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | certmonger-0.78.4-16.el7_9 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2001079 2001082 (view as bug list) | Environment: | ||
| Last Closed: | 2021-10-12 15:31:05 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 2001079, 2001082 | |||
If your reproducer is still running can you see if /etc/sysconfig/certmonger exists? Hopefully it does and bumps up the debug level a bit. Otherwise the journal should still include some minor output which could be useful. An lsof of the pid may be useful as well. It can open and rename files ok but a fd of 1022 seems a bit on the edge. Hi Rob There is no "/etc/sysconfig/certmonger" also from what I have seen there is not journal output either when we hit the issue. Also the issue is once we enable debug we won't be able to replicate again for undetermined amount of time. I can't get the lsof of the process from that strace, but I get it from one another one we replicated on. I can see this leak in a Fedora installation I have which, when I first started poking, had 190 open descriptors, most of which are FIFO. By using the refresh-ca command I can provoke a leak of 8 file descriptors! The CA being refreshed doesn't seem to matter: # lsof -p `pidof certmonger` | tail -1 certmonge 2841750 root 229r FIFO 0,13 0t0 8840919 pipe # getcert refresh-ca -c local Data for CA 'local' being refreshed. # lsof -p `pidof certmonger` | tail -1 certmonge 2841750 root 237r FIFO 0,13 0t0 8843863 pipe The leak I was seeing in certmonger-0.79.13-1.fc33 is fixed in certmonger-0.79.14-1.fc34. Will try to reproduce similar in el7. Upstream PR https://pagure.io/certmonger/pull-request/218 I should note that this PR eliminates *a* fd leak, I don't have confidence that it eliminates the reported leak which I've yet to reproduce with current upstream. Fixed upstream master: b4c090d2e12956a2df6157592839936adf4024f4 This is easiest to verify in an IPA installation. - yum -y install ipa-server-dns lsof - ipa-server-install <options> - lsof -p `pidof certmonger` In prior releases this would be a number in the 120's or higher. With the fixed version this should be 8. Verified using : certmonger-0.78.4-16.el7_9.x86_64 [root@master ~]# lsof -p `pidof certmonger` COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME .. .. certmonge 12175 root 0r CHR 1,3 0t0 1028 /dev/null certmonge 12175 root 1u unix 0xffff925446911dc0 0t0 50844 socket certmonge 12175 root 2u unix 0xffff925446911dc0 0t0 50844 socket certmonge 12175 root 3u a_inode 0,10 0 6397 [eventpoll] certmonge 12175 root 4uW REG 253,1 0 50356946 /var/lib/certmonger/lock certmonge 12175 root 5u a_inode 0,10 0 6397 [eventfd] certmonge 12175 root 6u netlink 0t0 50845 ROUTE certmonge 12175 root 7u unix 0xffff925446913300 0t0 50846 socket certmonge 12175 root 8uW REG 0,20 6 51813 /run/certmonger.pid [root@master ~]# rpm -qa ipa-server certmonger certmonger-0.78.4-16.el7_9.x86_64 ipa-server-4.6.8-5.el7_9.9.x86_64 [root@master ~]# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Reproduced using: certmonger-0.78.4-15.el7_9.x86_64 [root@master ~]# lsof -p `pidof certmonger` COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME .. .. certmonge 12262 root mem REG 253,1 163312 7895 /usr/lib64/ld-2.17.so certmonge 12262 root 0r CHR 1,3 0t0 1028 /dev/null certmonge 12262 root 1u unix 0xffff8fb0f44f9980 0t0 51775 socket certmonge 12262 root 2u unix 0xffff8fb0f44f9980 0t0 51775 socket certmonge 12262 root 3u a_inode 0,10 0 6397 [eventpoll] certmonge 12262 root 4uW REG 253,1 0 75524016 /var/lib/certmonger/lock certmonge 12262 root 5u a_inode 0,10 0 6397 [eventfd] certmonge 12262 root 6u netlink 0t0 52539 ROUTE certmonge 12262 root 7u unix 0xffff8fb0f7135dc0 0t0 52540 socket certmonge 12262 root 8uW REG 0,20 6 52541 /run/certmonger.pid certmonge 12262 root 9r FIFO 0,9 0t0 52542 pipe certmonge 12262 root 10r FIFO 0,9 0t0 52547 pipe certmonge 12262 root 11r FIFO 0,9 0t0 52613 pipe certmonge 12262 root 12r FIFO 0,9 0t0 51785 pipe certmonge 12262 root 13r FIFO 0,9 0t0 51821 pipe certmonge 12262 root 14r FIFO 0,9 0t0 51790 pipe certmonge 12262 root 15r FIFO 0,9 0t0 52697 pipe certmonge 12262 root 16r FIFO 0,9 0t0 51792 pipe certmonge 12262 root 17r FIFO 0,9 0t0 51823 pipe certmonge 12262 root 18r FIFO 0,9 0t0 51794 pipe certmonge 12262 root 19r FIFO 0,9 0t0 51941 pipe certmonge 12262 root 20r FIFO 0,9 0t0 52576 pipe certmonge 12262 root 21r FIFO 0,9 0t0 51825 pipe certmonge 12262 root 22r FIFO 0,9 0t0 51796 pipe certmonge 12262 root 23r FIFO 0,9 0t0 51943 pipe certmonge 12262 root 24r FIFO 0,9 0t0 51798 pipe certmonge 12262 root 25r FIFO 0,9 0t0 51827 pipe certmonge 12262 root 26r FIFO 0,9 0t0 52578 pipe certmonge 12262 root 27r FIFO 0,9 0t0 51945 pipe certmonge 12262 root 28r FIFO 0,9 0t0 51800 pipe certmonge 12262 root 29r FIFO 0,9 0t0 52615 pipe certmonge 12262 root 30r FIFO 0,9 0t0 52580 pipe certmonge 12262 root 31r FIFO 0,9 0t0 51947 pipe certmonge 12262 root 32r FIFO 0,9 0t0 51802 pipe certmonge 12262 root 33r FIFO 0,9 0t0 52617 pipe certmonge 12262 root 34r FIFO 0,9 0t0 52582 pipe certmonge 12262 root 35r FIFO 0,9 0t0 52700 pipe certmonge 12262 root 36r FIFO 0,9 0t0 51804 pipe certmonge 12262 root 37r FIFO 0,9 0t0 52619 pipe certmonge 12262 root 38r FIFO 0,9 0t0 52584 pipe certmonge 12262 root 39r FIFO 0,9 0t0 52704 pipe certmonge 12262 root 40r FIFO 0,9 0t0 51806 pipe certmonge 12262 root 41r FIFO 0,9 0t0 51831 pipe certmonge 12262 root 42r FIFO 0,9 0t0 52586 pipe certmonge 12262 root 43r FIFO 0,9 0t0 52706 pipe certmonge 12262 root 44r FIFO 0,9 0t0 51808 pipe certmonge 12262 root 45r FIFO 0,9 0t0 51833 pipe certmonge 12262 root 46r FIFO 0,9 0t0 52588 pipe certmonge 12262 root 47r FIFO 0,9 0t0 52760 pipe certmonge 12262 root 48r FIFO 0,9 0t0 52590 pipe certmonge 12262 root 49r FIFO 0,9 0t0 51835 pipe certmonge 12262 root 50r FIFO 0,9 0t0 52592 pipe certmonge 12262 root 51r FIFO 0,9 0t0 52004 pipe certmonge 12262 root 52r FIFO 0,9 0t0 52594 pipe certmonge 12262 root 53r FIFO 0,9 0t0 51838 pipe certmonge 12262 root 54r FIFO 0,9 0t0 52596 pipe certmonge 12262 root 55r FIFO 0,9 0t0 58388 pipe certmonge 12262 root 56r FIFO 0,9 0t0 52598 pipe certmonge 12262 root 57r FIFO 0,9 0t0 51842 pipe certmonge 12262 root 58r FIFO 0,9 0t0 51813 pipe certmonge 12262 root 59r FIFO 0,9 0t0 52006 pipe certmonge 12262 root 60r FIFO 0,9 0t0 51815 pipe certmonge 12262 root 61r FIFO 0,9 0t0 51844 pipe certmonge 12262 root 62r FIFO 0,9 0t0 51817 pipe certmonge 12262 root 63r FIFO 0,9 0t0 58105 pipe certmonge 12262 root 64r FIFO 0,9 0t0 52602 pipe certmonge 12262 root 65r FIFO 0,9 0t0 52626 pipe certmonge 12262 root 66r FIFO 0,9 0t0 52604 pipe certmonge 12262 root 67r FIFO 0,9 0t0 52008 pipe certmonge 12262 root 68r FIFO 0,9 0t0 52606 pipe certmonge 12262 root 69r FIFO 0,9 0t0 52630 pipe certmonge 12262 root 70r FIFO 0,9 0t0 52608 pipe certmonge 12262 root 71r FIFO 0,9 0t0 58364 pipe certmonge 12262 root 72r FIFO 0,9 0t0 51848 pipe certmonge 12262 root 73r FIFO 0,9 0t0 52010 pipe certmonge 12262 root 74r FIFO 0,9 0t0 51850 pipe certmonge 12262 root 75r FIFO 0,9 0t0 58390 pipe certmonge 12262 root 76r FIFO 0,9 0t0 51855 pipe certmonge 12262 root 77r FIFO 0,9 0t0 52012 pipe certmonge 12262 root 78r FIFO 0,9 0t0 51861 pipe certmonge 12262 root 79r FIFO 0,9 0t0 58500 pipe certmonge 12262 root 80r FIFO 0,9 0t0 51863 pipe certmonge 12262 root 81r FIFO 0,9 0t0 52014 pipe certmonge 12262 root 82r FIFO 0,9 0t0 51866 pipe certmonge 12262 root 83r FIFO 0,9 0t0 58107 pipe certmonge 12262 root 84r FIFO 0,9 0t0 51868 pipe certmonge 12262 root 85r FIFO 0,9 0t0 52016 pipe certmonge 12262 root 86r FIFO 0,9 0t0 51874 pipe certmonge 12262 root 87r FIFO 0,9 0t0 62188 pipe certmonge 12262 root 88r FIFO 0,9 0t0 58109 pipe certmonge 12262 root 89r FIFO 0,9 0t0 58366 pipe certmonge 12262 root 90r FIFO 0,9 0t0 58111 pipe certmonge 12262 root 91r FIFO 0,9 0t0 60849 pipe certmonge 12262 root 92r FIFO 0,9 0t0 58113 pipe certmonge 12262 root 93r FIFO 0,9 0t0 58368 pipe certmonge 12262 root 94r FIFO 0,9 0t0 58115 pipe certmonge 12262 root 95r FIFO 0,9 0t0 62238 pipe certmonge 12262 root 96r FIFO 0,9 0t0 59394 pipe certmonge 12262 root 97r FIFO 0,9 0t0 62190 pipe certmonge 12262 root 98r FIFO 0,9 0t0 59404 pipe certmonge 12262 root 99r FIFO 0,9 0t0 60883 pipe certmonge 12262 root 100r FIFO 0,9 0t0 58527 pipe certmonge 12262 root 101r FIFO 0,9 0t0 62192 pipe certmonge 12262 root 102r FIFO 0,9 0t0 59406 pipe certmonge 12262 root 103r FIFO 0,9 0t0 60914 pipe certmonge 12262 root 104r FIFO 0,9 0t0 62194 pipe certmonge 12262 root 105r FIFO 0,9 0t0 62240 pipe certmonge 12262 root 106r FIFO 0,9 0t0 62196 pipe certmonge 12262 root 107r FIFO 0,9 0t0 62273 pipe certmonge 12262 root 108r FIFO 0,9 0t0 62198 pipe certmonge 12262 root 109r FIFO 0,9 0t0 62242 pipe certmonge 12262 root 110r FIFO 0,9 0t0 62200 pipe certmonge 12262 root 111r FIFO 0,9 0t0 62479 pipe certmonge 12262 root 112r FIFO 0,9 0t0 62244 pipe certmonge 12262 root 113r FIFO 0,9 0t0 62275 pipe certmonge 12262 root 114r FIFO 0,9 0t0 62246 pipe certmonge 12262 root 115r FIFO 0,9 0t0 62481 pipe certmonge 12262 root 116r FIFO 0,9 0t0 62248 pipe certmonge 12262 root 117r FIFO 0,9 0t0 62277 pipe certmonge 12262 root 118r FIFO 0,9 0t0 62258 pipe certmonge 12262 root 119r FIFO 0,9 0t0 62483 pipe certmonge 12262 root 120r FIFO 0,9 0t0 62279 pipe certmonge 12262 root 121r FIFO 0,9 0t0 62485 pipe certmonge 12262 root 122r FIFO 0,9 0t0 62281 pipe certmonge 12262 root 123r FIFO 0,9 0t0 62487 pipe certmonge 12262 root 124r FIFO 0,9 0t0 62283 pipe certmonge 12262 root 125r FIFO 0,9 0t0 61432 pipe certmonge 12262 root 126r FIFO 0,9 0t0 62285 pipe certmonge 12262 root 127r FIFO 0,9 0t0 61436 pipe certmonge 12262 root 128r FIFO 0,9 0t0 63640 pipe certmonge 12262 root 129r FIFO 0,9 0t0 61438 pipe certmonge 12262 root 130r FIFO 0,9 0t0 62547 pipe certmonge 12262 root 132r FIFO 0,9 0t0 62549 pipe certmonge 12262 root 134r FIFO 0,9 0t0 62551 pipe certmonge 12262 root 136r FIFO 0,9 0t0 62553 pipe certmonge 12262 root 138r FIFO 0,9 0t0 62555 pipe certmonge 12262 root 140r FIFO 0,9 0t0 63642 pipe certmonge 12262 root 142r FIFO 0,9 0t0 63644 pipe [root@master ~]# rpm -qa ipa-server certmonger ipa-server-4.6.8-5.el7_9.7.x86_64 certmonger-0.78.4-15.el7_9.x86_64 [root@master ~]# Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (certmonger bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:3808 |
Description of problem: When resubmit certificates through certmonger gets stuck in NEED_GUIDANCE. This seems to be related to how long the certmonger process has been running. After restarting the service there is no issue resubmitting certificates Version-Release number of selected component (if applicable): Red Hat Enterprise Linux Server release 7.9 (Maipo) certmonger-0.78.4-14.el7.x86_64 How reproducible: Hard to tell, it seems to be based on how long the service has been running. In the test environment where the certmonger has been up for 8 months. Of the 3 of 8 servers we tried to replicate on, we could replicate it each time. Steps to Reproduce: 1. Submit certificates to IPA through certmonger 2. Wait an undetermined about of time 3. resubmit the certificates Actual results: Request ID 'haproxy-storage_mgmt-cert': status: NEED_GUIDANCE stuck: yes key pair storage: type=FILE,location='/etc/pki/tls/private/haproxy/overcloud-haproxy-storage_mgmt.key' certificate: type=FILE,location='/etc/pki/tls/certs/haproxy/overcloud-haproxy-storage_mgmt.crt' CA: IPA issuer: CN=Certificate Authority,O=REDHAT.LOCAL subject: CN=controller-0.storagemgmt.redhat.local,O=REDHAT.LOCAL expires: 2023-04-10 05:18:02 UTC dns: overcloud.storagemgmt.redhat.local,controller-0.storagemgmt.redhat.local principal name: haproxy/controller-0.storagemgmt.redhat.local key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/bin/certmonger-haproxy-refresh.sh reload storage_mgmt track: yes auto-renew: yes Expected results: Request ID 'haproxy-storage_mgmt-cert': status: MONITORING stuck: no key pair storage: type=FILE,location='/etc/pki/tls/private/haproxy/overcloud-haproxy-storage_mgmt.key' certificate: type=FILE,location='/etc/pki/tls/certs/haproxy/overcloud-haproxy-storage_mgmt.crt' CA: IPA issuer: CN=Certificate Authority,O=REDHAT.LOCAL subject: CN=controller-0.storagemgmt.redhat.local,O=REDHAT.LOCAL expires: 2023-04-10 05:18:02 UTC dns: overcloud.storagemgmt.redhat.local,controller-0.storagemgmt.redhat.local principal name: haproxy/controller-0.storagemgmt.redhat.local key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/bin/certmonger-haproxy-refresh.sh reload storage_mgmt track: yes auto-renew: yes Additional info: From what I can see from the strace is there Bad file descriptor when try to read -1, 0x55f57c3507d0, 8192 right after sending the submitting status to dbus. After this is goes into NEED_GUIDANCE ~~~ 15104 1628051842.134790 close(1022</var/lib/certmonger/requests/20201112060423.tmp>) = 0 <0.000019> 15104 1628051842.134844 munmap(0x7fde2bcec000, 4096) = 0 <0.000022> 15104 1628051842.134895 rename("/var/lib/certmonger/requests/20201112060423.tmp", "/var/lib/certmonger/requests/20201112060423") = 0 <0.000051> 15104 1628051842.135021 sendmsg(7<UNIX:[63674->62964]>, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\4\1\1O\0\0\0\230\34\0\0\214\0\0\0\1\1o\0.\0\0\0/org/fedorahosted/certmonger/requests/Request5\0\0\2\1s\0\37\0\0\0org.freedesktop. DBus.Properties\0\3\1s\0\21\0\0\0PropertiesChanged\0\0\0\0\0\0\0\10\1g\0\6sa{sv}\0\0\0\0\0", iov_len=160}, {iov_base="#\0\0\0org.fedorahosted.certmonger.request\0\37\0\0\0\0\0\0\0\6\0\0\0status\0\1s\0\0\0\n\0\0\0SUBMITTING\0", iov_len=79} ], msg_iovlen=2, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 239 <0.000025> 15104 1628051842.135103 epoll_wait(3<anon_inode:[eventpoll]>, [], 1, 4999) = 0 <5.000301> 15104 1628051847.135591 read(-1, 0x55f57c3507d0, 8192) = - <0.000030> 15104 1628051847.135691 close(-1) = -1 EBADF (Bad file descriptor) <0.000022> 15104 1628051847.135752 wait4(0, 0x55f57c32e0bc, 0, NULL) = -1 ECHILD (No child processes) <0.000018> 15104 1628051847.135837 open("/var/lib/certmonger/requests/20201112060423.tmp", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 1022</var/lib/certmonger/requests/20201112060423.tmp> <0.000130> 15104 1628051847.136067 fstat(1022</var/lib/certmonger/requests/20201112060423.tmp>, {st_dev=makedev(252, 2), st_ino=14349509, st_mode=S_IFREG|0600, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=0, st_atime=1628051 847 /* 2021-08-04T04:37:27.135056204+0000 */, st_atime_nsec=135056204, st_mtime=1628051847 /* 2021-08-04T04:37:27.135056204+0000 */, st_mtime_nsec=135056204, st_ctime=1628051847 /* 2021-08-04T04:37:27.135056204+0000 */, st_ctime_nsec=1350 56204}) = 0 <0.000023> 15104 1628051847.136171 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fde2bcec000 <0.000028> ~~~