1763745 – certmonger.service starts but certmonger does not reply on D-Bus in container

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1763745 - certmonger.service starts but certmonger does not reply on D-Bus in container

Summary: certmonger.service starts but certmonger does not reply on D-Bus in container

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 8
Classification:	Red Hat
Component:	certmonger
Sub Component:
Version:	8.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	8.0
Assignee:	Rob Crittenden
QA Contact:	ipa-qe
Docs Contact:
URL:
Whiteboard:
Depends On:	1656519
Blocks:
TreeView+	depends on / blocked

Reported:	2019-10-21 13:56 UTC by Rob Crittenden
Modified:	2023-02-12 22:51 UTC (History)
CC List:	8 users (show)
Fixed In Version:	certmonger-0.79.7-4
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:	1656519
Environment:
Last Closed:	2020-04-28 16:01:49 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	FREEIPA-9439	None	None	None	2023-02-12 22:51:03 UTC
Red Hat Issue Tracker	RHELPLAN-30192	None	None	None	2023-02-12 22:48:50 UTC
Red Hat Product Errata	RHBA-2020:1704	None	None	None	2020-04-28 16:01:54 UTC

Description Rob Crittenden 2019-10-21 13:56:23 UTC

+++ This bug was initially created as a clone of Bug #1656519 +++

Description of problem:

Starting with Fedora 27

Version-Release number of selected component (if applicable):

In the container:

certmonger-0.79.6-3.fc29.x86_64
dbus-1.12.10-1.fc29.x86_64
systemd-239-3.fc29.x86_64

On the host:

docker-1.13.1-62.git9cb56fd.fc29.x86_64

How reproducible:

Not deterministic.

Steps to Reproduce:
1. Allow running systemd in containers:
   setsebool -P container_manage_cgroup 1
2. Run systemd container:
   docker run -e container=oci --name=systemd -d --rm registry.fedoraproject.org/fedora:29 /usr/sbin/init
3. Install certmonger in the container:
   docker exec systemd dnf install -y certmonger
4. In the container, start certmonger, check that systemd sees it running, and try to introspect it:
   docker exec systemd bash -c 'systemctl restart certmonger ; systemctl status certmonger ; dbus-send --system --type=method_call --print-reply --dest=org.fedorahosted.certmonger /org/fedorahosted/certmonger org.freedesktop.DBus.Introspectable.Introspect'

Actual results:

● certmonger.service - Certificate monitoring and PKI enrollment
   Loaded: loaded (/usr/lib/systemd/system/certmonger.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-12-05 15:28:13 UTC; 48ms ago
 Main PID: 241 (certmonger)
   CGroup: /system.slice/docker-d0a9b82bcc42b9759b76b701a6e75e23cccff1e96db61c43a089501e1014da2f.scope/system.slice/certmonger.service
           ├─241 /usr/sbin/certmonger -S -p /var/run/certmonger.pid -n
           └─243 /usr/sbin/certmonger -S -p /var/run/certmonger.pid -n

Dec 05 15:28:12 d0a9b82bcc42 systemd[1]: Starting Certificate monitoring and PKI enrollment...
Dec 05 15:28:13 d0a9b82bcc42 systemd[1]: Started Certificate monitoring and PKI enrollment.

Error org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

Expected results:

● certmonger.service - Certificate monitoring and PKI enrollment
   Loaded: loaded (/usr/lib/systemd/system/certmonger.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-12-05 15:28:13 UTC; 48ms ago
 Main PID: 241 (certmonger)
   CGroup: /system.slice/docker-d0a9b82bcc42b9759b76b701a6e75e23cccff1e96db61c43a089501e1014da2f.scope/system.slice/certmonger.service
           ├─241 /usr/sbin/certmonger -S -p /var/run/certmonger.pid -n
           └─243 /usr/sbin/certmonger -S -p /var/run/certmonger.pid -n

Dec 05 15:28:12 d0a9b82bcc42 systemd[1]: Starting Certificate monitoring and PKI enrollment...
Dec 05 15:28:13 d0a9b82bcc42 systemd[1]: Started Certificate monitoring and PKI enrollment.

method return time=1544023753.120209 sender=:1.3 -> destination=:1.5 serial=22 reply_serial=2
   string "<!DOCTYPE node PUBLIC "-//freedesktop//DTD D-BUS Object Introspection 1.0//EN"
"http://www.freedesktop.org/standards/dbus/1.0/introspect.dtd">

<node name="/org/fedorahosted/certmonger">
 <interface name="org.freedesktop.DBus.Introspectable">
  <method name="Introspect">
   <arg name="xml_data" type="s" direction="out"/>
  </method>
 </interface>
 <interface name="org.freedesktop.DBus.Properties">
  <method name="Get">
   <arg name="interface_name" type="s" direction="in"/>
   <arg name="property_name" type="s" direction="in"/>
   <arg name="value" type="v" direction="out"/>
  </method>
  <method name="Set">
   <arg name="interface_name" type="s" direction="in"/>
   <arg name="property_name" type="s" direction="in"/>
   <arg name="value" type="v" direction="in"/>
  </method>
[...]

Additional info:

https://github.com/freeipa/freeipa-container/issues/187#issuecomment-391687818

--- Additional comment from Jan Pazdziora on 2018-12-05 16:57:17 UTC ---

Beaker reproducer job:
https://beaker.engineering.redhat.com/jobs/3207096

--- Additional comment from Jan Pazdziora on 2018-12-05 17:10:24 UTC ---

The

/usr/lib/systemd/system/certmonger.service

is defined as

[Unit]
Description=Certificate monitoring and PKI enrollment
After=syslog.target network.target dbus.service

[Service]
Type=dbus
PIDFile=/var/run/certmonger.pid
EnvironmentFile=-/etc/sysconfig/certmonger
ExecStart=/usr/sbin/certmonger -S -p /var/run/certmonger.pid -n $OPTS
BusName=org.fedorahosted.certmonger

[Install]
WantedBy=multi-user.target

And systemd.service(5) says

           Behavior of dbus is similar to simple; however, it is expected that
           the daemon acquires a name on the D-Bus bus, as configured by
           BusName=. systemd will proceed with starting follow-up units after
           the D-Bus bus name has been acquired. Service units with this
           option configured implicitly gain dependencies on the dbus.socket
           unit. This type is the default if BusName= is specified.

It does not specifically say that the service will be made active/running after it has acquired the D-Bus bus name but it used to behave this way up until Fedora 26 -- ipa-server-install (where the issues was first observed and from which it was minimized to this bug report) tests were not failing because of certmonger.service startup.

--- Additional comment from Jan Pazdziora on 2018-12-05 17:24:05 UTC ---

An example of beaker job with fedora:27 image (and dbus and certmonger which is in that Fedora version) on Fedora 28 host:
https://beaker.engineering.redhat.com/jobs/3207431

--- Additional comment from Jan Pazdziora on 2018-12-05 20:12:51 UTC ---

Beaker job reproducer, using podman instead of docker:
https://beaker.engineering.redhat.com/jobs/3207481

--- Additional comment from Jan Pazdziora on 2019-07-16 14:42:03 UTC ---

After certmonger was started, list of defunct child processes gradually grows up to

root       786  0.2  0.1  17424  7460 ?        Ss   14:38   0:00 /usr/sbin/certmonger -S -p /var/run/certmonger.pid -n
root       788  4.9  0.0      0     0 ?        Z    14:38   0:01  \_ [ipa-submit] <defunct>
root       793  3.2  0.0      0     0 ?        Z    14:38   0:00  \_ [ipa-submit] <defunct>
root       794  3.2  0.0      0     0 ?        Z    14:38   0:00  \_ [ipa-submit] <defunct>
root       796  3.4  0.0      0     0 ?        Z    14:38   0:00  \_ [ipa-submit] <defunct>
root       798  3.5  0.0      0     0 ?        Z    14:38   0:00  \_ [ipa-submit] <defunct>
root       800  3.7  0.0      0     0 ?        Z    14:38   0:00  \_ [ipa-submit] <defunct>
root       801  3.9  0.0      0     0 ?        Z    14:38   0:00  \_ [ipa-submit] <defunct>
root       803  4.0  0.0      0     0 ?        Z    14:38   0:00  \_ [ipa-submit] <defunct>
root       805  4.1  0.0      0     0 ?        Z    14:38   0:00  \_ [certmaster-subm] <defunct>
root       807  4.4  0.0      0     0 ?        Z    14:39   0:00  \_ [certmaster-subm] <defunct>
root       808  4.7  0.0      0     0 ?        Z    14:39   0:00  \_ [certmaster-subm] <defunct>
root       810  4.6  0.0      0     0 ?        Z    14:39   0:00  \_ [certmaster-subm] <defunct>
root       812  5.0  0.0      0     0 ?        Z    14:39   0:00  \_ [certmaster-subm] <defunct>
root       814  5.4  0.0      0     0 ?        Z    14:39   0:00  \_ [certmaster-subm] <defunct>
root       816  5.8  0.0      0     0 ?        Z    14:39   0:00  \_ [certmaster-subm] <defunct>
root       817  5.6  0.0      0     0 ?        Z    14:39   0:00  \_ [certmaster-subm] <defunct>
root       819  6.2  0.0      0     0 ?        Z    14:39   0:00  \_ [dogtag-ipa-rene] <defunct>
root       821  6.8  0.0      0     0 ?        Z    14:39   0:00  \_ [dogtag-ipa-rene] <defunct>
root       823  7.8  0.0      0     0 ?        Z    14:39   0:00  \_ [dogtag-ipa-rene] <defunct>
root       825  8.2  0.0      0     0 ?        Z    14:39   0:00  \_ [dogtag-ipa-rene] <defunct>
root       826  8.4  0.0      0     0 ?        Z    14:39   0:00  \_ [dogtag-ipa-rene] <defunct>
root       828  9.3  0.0      0     0 ?        Z    14:39   0:00  \_ [dogtag-ipa-rene] <defunct>
root       830 10.7  0.0      0     0 ?        Z    14:39   0:00  \_ [dogtag-ipa-rene] <defunct>
root       832 12.5  0.0      0     0 ?        Z    14:39   0:00  \_ [dogtag-ipa-rene] <defunct>
root       833 12.5  0.0      0     0 ?        Z    14:39   0:00  \_ [local-submit] <defunct>
root       835 15.2  0.0      0     0 ?        Z    14:39   0:00  \_ [local-submit] <defunct>
root       837 32.7  0.0      0     0 ?        Z    14:39   0:01  \_ [local-submit] <defunct>
root       839 25.3  0.0      0     0 ?        Z    14:39   0:00  \_ [local-submit] <defunct>
root       840 37.5  0.0      0     0 ?        Z    14:39   0:00  \_ [local-submit] <defunct>
root       842 76.0  0.0      0     0 ?        Z    14:39   0:00  \_ [local-submit] <defunct>
root       844  0.0  0.0  17424  1016 ?        R    14:39   0:00  \_ /usr/sbin/certmonger -S -p /var/run/certmonger.pid -n

During that time, certmonger does not seem to respond over D-Bus.

The container does not have anything about dogtag or ipa installed, so I'm not sure what certmonger is trying to do here.

--- Additional comment from Jan Pazdziora on 2019-07-16 14:56:26 UTC ---

It seems like on the host (outside of container), these children are executed as well during systemctl restart certmonger (bash -c 'echo $$' before and after the systemctl start shows 38 pids go by) but it all happens much faster, at least looking at the journalctl -f output (for certmonger configured with OPTS="-d 10" in /etc/sysconfig/certmonger) so I never manage to see those children in the ps output.

--- Additional comment from Rob Crittenden on 2019-07-16 15:20:14 UTC ---

certmonger validates the CA configuration. For example, for the local CA if the self-signed CA is not present it will generate one.

Does certmonger ever eventually reply once these tasks clean up?

--- Additional comment from Jan Pazdziora on 2019-07-16 15:31:33 UTC ---

If I rerun that dbus-send after a while, it will reply. But it can take many seconds for those forked processes to clear out in the container case. On the host, on the same machine, they go away (and certmonger get responsive) much faster.

The problem is that ipa-server-install runs systemctl restart certmonger and when that systemctl command finishes, it starts the next operation which tries to use certmonger. But at that point certmonger is still running those startup validation steps, making FreeIPA setup fail in nondeterministic fashion; it also kills FreeIPA upgrades.

--- Additional comment from Rob Crittenden on 2019-07-16 15:38:37 UTC ---

Ok, fair point. I've seen others complain about this behavior of tons of spawned processes hanging around a while, even in the non-container case, but haven't been able to replicate it. I'll have to see if there is a way I can tell what is going on.

--- Additional comment from Rob Crittenden on 2019-08-29 16:28:44 UTC ---

I have a candidate IPA patch which should mitigate this somewhat, https://github.com/freeipa/freeipa/pull/3596

The IPA certmonger CA uses a locker, ipa-server-guard, to ensure serial renewals. This was executing on operations that it didn't need to. Return earlier for operations not supported by ipa-submit. This avoids a bunch of locked calls that are guaranteed to fail.

The other piece is in the renewal script for the CA subsystem certificates. The IPA API was always initialized whether it need to be or not, which is costly. I also moved some imports around so that some are deferred and only used if needed. This should help in startup time.

There is still a flurry of activity during startup but in my testing it ends 50% sooner with less overall load.

--- Additional comment from Rob Crittenden on 2019-09-03 16:43:57 UTC ---

Duplicating some info from the PR that is more relevant for this more vanilla case.

During startup certmonger forks each CA a number of times performing operations on it to obtain their capabilities, six in total. So the number of forks is 6 x # of CAs.

A strategy for reducing these startup time would be to remove CA's that will not be used in the container using getcert remove-ca <name>

I wasn't able to reproduce the failure on a F30 VM running the F29 container in the steps, it connects the DBus after a second or two pause.

I realize this BZ isn't specifically about IPA/dogtag but is a more vanilla case but IPA exacerbates some of the issues because it ends up serializing some of the requests using locks. The above PR is an attempt to mitigate that specific case.

--- Additional comment from Rob Crittenden on 2019-09-03 16:45:55 UTC ---

Typo, it's 8 forks per CA not 6.

--- Additional comment from Jan Pazdziora on 2019-09-05 14:00:57 UTC ---

The real reason for the different and much slower behaviour in containers is the code 

        for (i = getdtablesize() - 1; i >= 3; i--) {
                if ((i == fd) ||
                    (i == fd2) ||
                    (i == fd3)) {
                        continue;
                }
                l = fcntl(i, F_GETFD);
                if (l != -1) {
                        if (fcntl(i, F_SETFD, l | FD_CLOEXEC) != 0) {
                                cm_log(0, "Potentially leaking FD %d.\n", i);
                        }
                }
        }

in src/subproc.c.

In containers, the ulimit seems to be set to

# ulimit -n
1048576

so the processes loop over huge number of invalid file descriptors.

--- Additional comment from Jan Pazdziora on 2019-09-05 14:10:32 UTC ---

Setting the limit to the same value as observed on the host seems to help:

( echo "[Service]" ; echo "LimitNOFILE=1024" ) > /usr/lib/systemd/system/certmonger.service.d/ulimit.conf

--- Additional comment from Jan Pazdziora on 2019-09-05 14:25:48 UTC ---

Alternatively:

echo "DefaultLimitNOFILE=1024" >> /etc/systemd/system.conf

--- Additional comment from Jan Pazdziora on 2019-09-05 15:05:46 UTC ---

I guess this bugzilla can be closed because ensuring FD_CLOEXEC is set is a good thing, and there isn't a better way in Linux than cycling through all potential file descriptors, which getdtablesize() is a good indication. So there's nothing to be done in certmonger, really.

Any patch that certmonger can get to minimize unneeded work is of course great so feel free to track it here if you prefer to. But the real cause needs to be handled in the container, setting the limits appropriately. We have a pull request https://github.com/freeipa/freeipa-container/pull/283 under testing now.

--- Additional comment from Jan Pazdziora on 2019-09-05 15:16:06 UTC ---

Actually, listing fds with /proc/self/fd instead of using the getdtablesize() approach might be a valid approach, should certmonger team prefer to do that.

--- Additional comment from Rob Crittenden on 2019-09-05 17:21:05 UTC ---

Suggestion implemented in https://pagure.io/certmonger/pull-request/130

--- Additional comment from Rob Crittenden on 2019-09-06 17:49:20 UTC ---

master: 9bbb628620d4e586941344e1bdbbc166a885c0a9

--- Additional comment from Rob Crittenden on 2019-10-10 20:46:01 UTC ---

Additional fix to not close STDERR when fetching CA data.

master: b7bcb1b3b953c2052e2d89cb2b3e9d9ccd1b3864

Comment 2 Jan Pazdziora 2019-11-27 09:08:57 UTC

It should be possible to test this on host (outside of containers) by increasing the limit of open files (or setting it to unlimited). Something like (untested):

echo "DefaultLimitNOFILE=infinity" >> /etc/systemd/system.conf
systemctl daemon-reload
ipa-server-instal ...

Comment 3 Rob Crittenden 2019-12-02 16:28:39 UTC

Or use strace to ensure that all 1024 (or whatever) fds are not being closed on fork.

Comment 5 Sumedh Sidhaye 2020-03-06 07:47:48 UTC

    F 30 container
         
        [root@user1 ]# docker exec systemd dnf install -y certmonger
        Last metadata expiration check: 0:11:46 ago on Fri Mar  6 04:34:32 2020.
        Package certmonger-0.79.9-1.fc30.x86_64 is already installed.
        Dependencies resolved.
        Nothing to do.
        Complete!
        [root@user1 ]# docker exec systemd bash -c 'systemctl restart certmonger ; systemctl status certmonger ; dbus-send --system --type=method_call --print-reply --dest=org.fedorahosted.certmonger /org/fedorahosted/certmonger org.freedesktop.DBus.Introspectable.Introspect'
        ● certmonger.service - Certificate monitoring and PKI enrollment
           Loaded: loaded (/usr/lib/systemd/system/certmonger.service; disabled; vendor preset: disabled)
           Active: active (running) since Fri 2020-03-06 04:46:41 UTC; 14ms ago
         Main PID: 248 (certmonger)
           CGroup: /system.slice/docker-da73fb1b228ab838b1e5ca98c0256522a35b46a5fc92db3d53f78baad4a6331a.scope/system.slice/certmonger.service
                   ├─248 /usr/sbin/certmonger -S -p /run/certmonger.pid -n
                   ├─259 /usr/libexec/certmonger/certmaster-submit
                   ├─260 /usr/libexec/certmonger/certmaster-submit
                   ├─261 /usr/libexec/certmonger/certmaster-submit
                   ├─262 /usr/libexec/certmonger/certmaster-submit
                   └─263 /usr/sbin/certmonger -S -p /run/certmonger.pid -n
         
        Mar 06 04:46:41 da73fb1b228a systemd[1]: Starting Certificate monitoring and PKI enrollment...
        Mar 06 04:46:41 da73fb1b228a systemd[1]: Started Certificate monitoring and PKI enrollment.
        method return time=1583470001.370465 sender=:1.4 -> destination=:1.5 serial=13 reply_serial=2
           string "<!DOCTYPE node PUBLIC "-//freedesktop//DTD D-BUS Object Introspection 1.0//EN"
        "http://www.freedesktop.org/standards/dbus/1.0/introspect.dtd">
         
        <node name="/org/fedorahosted/certmonger">
         <interface name="org.freedesktop.DBus.Introspectable">
          <method name="Introspect">
           <arg name="xml_data" type="s" direction="out"/>
          </method>
         </interface>
         <interface name="org.freedesktop.DBus.Properties">
          <method name="Get">
           <arg name="interface_name" type="s" direction="in"/>
           <arg name="property_name" type="s" direction="in"/>
           <arg name="value" type="v" direction="out"/>
          </method>
          <method name="Set">
           <arg name="interface_name" type="s" direction="in"/>
           <arg name="property_name" type="s" direction="in"/>
           <arg name="value" type="v" direction="in"/>
          </method>
          <method name="GetAll">
           <arg name="interface_name" type="s" direction="in"/>
           <arg name="props" type="a{sv}" direction="out"/>
          </method>
          <signal name="PropertiesChanged">
           <arg name="interface_name" type="s"/>
           <arg name="changed_properties" type="a{sv}"/>
           <arg name="invalidated_properties" type="as"/>
          </signal>
         </interface>
         <interface name="org.fedorahosted.certmonger">
          <method name="add_known_ca">
           <arg name="nickname" type="s" direction="in"/>
           <arg name="command" type="s" direction="in"/>
           <arg name="known_names" type="as" direction="in"/>
           <arg name="status" type="b" direction="out"/>
           <arg name="name" type="o" direction="out"/>
          </method>
          <method name="add_request">
           <arg name="template" type="a{sv}" direction="in"/>
           <arg name="status" type="b" direction="out"/>
           <arg name="name" type="o" direction="out"/>
          </method>
          <method name="find_ca_by_nickname">
           <arg name="nickname" type="s" direction="in"/>
           <arg name="ca" type="o" direction="out"/>
          </method>
          <method name="find_request_by_nickname">
           <arg name="nickname" type="s" direction="in"/>
           <arg name="request" type="o" direction="out"/>
          </method>
          <method name="get_known_cas">
           <arg name="ca_list" type="ao" direction="out"/>
          </method>
          <method name="get_requests">
           <arg name="requests" type="ao" direction="out"/>
          </method>
          <method name="get_supported_key_types">
           <arg name="key_type_list" type="as" direction="out"/>
          </method>
          <method name="get_supported_key_storage">
           <arg name="key_storage_type_list" type="as" direction="out"/>
          </method>
          <method name="get_supported_cert_storage">
           <arg name="cert_storage_type_list" type="as" direction="out"/>
          </method>
          <method name="remove_known_ca">
           <arg name="ca" type="o" direction="in"/>
           <arg name="status" type="b" direction="out"/>
          </method>
          <method name="remove_request">
           <arg name="request" type="o" direction="in"/>
           <arg name="status" type="b" direction="out"/>
          </method>
         </interface>
         <node name="requests"/>
         <node name="cas"/>
        </node>"
        [root@user1 ]# docker ps -a
        CONTAINER ID        IMAGE                                  COMMAND             CREATED             STATUS              PORTS               NAMES
        da73fb1b228a        registry.fedoraproject.org/fedora:30   "/usr/sbin/init"    23 minutes ago      Up 23 minutes                           systemd
        [root@user1 ]# docker run -e container=oci --name=systemd2 -d --rm registry.fedoraproject.org/fedora:31 /usr/sbin/init
        d09ab0211d00bb35c84ed6ad3b77df25de6df4fc03d29b508f040e9414110507
        [root@user1 ]# docker ps -a
        CONTAINER ID        IMAGE                                  COMMAND             CREATED             STATUS              PORTS               NAMES
        d09ab0211d00        registry.fedoraproject.org/fedora:31   "/usr/sbin/init"    4 seconds ago       Up 3 seconds                            systemd2
        da73fb1b228a        registry.fedoraproject.org/fedora:30   "/usr/sbin/init"    23 minutes ago      Up 23 minutes                           systemd
         
         
        F31 container
         
         
        [root@user1 ]# docker exec systemd2 bash -c 'systemctl restart certmonger ; systemctl status certmonger ; dbus-send --system --type=method_call --print-reply --dest=org.fedorahosted.certmonger /org/fedorahosted/certmonger org.freedesktop.DBus.Introspectable.Introspect'
        ● certmonger.service - Certificate monitoring and PKI enrollment
           Loaded: loaded (/usr/lib/systemd/system/certmonger.service; disabled; vendor preset: disabled)
           Active: active (running) since Fri 2020-03-06 04:57:23 UTC; 35ms ago
         Main PID: 129 (certmonger)
           CGroup: /system.slice/docker-d09ab0211d00bb35c84ed6ad3b77df25de6df4fc03d29b508f040e9414110507.scope/system.slice/certmonger.service
                   ├─129 /usr/sbin/certmonger -S -p /run/certmonger.pid -n
                   ├─138 /usr/libexec/certmonger/ipa-submit
                   ├─139 /usr/libexec/certmonger/certmaster-submit
                   ├─140 /usr/libexec/certmonger/certmaster-submit
                   ├─141 /usr/libexec/certmonger/certmaster-submit
                   ├─142 /usr/libexec/certmonger/certmaster-submit
                   └─143 [certmaster-subm]
         
        Mar 06 04:57:23 d09ab0211d00 systemd[1]: Starting Certificate monitoring and PKI enrollment...
        Mar 06 04:57:23 d09ab0211d00 systemd[1]: Started Certificate monitoring and PKI enrollment.
        method return time=1583470644.015827 sender=:1.3 -> destination=:1.4 serial=13 reply_serial=2
           string "<!DOCTYPE node PUBLIC "-//freedesktop//DTD D-BUS Object Introspection 1.0//EN"
        "http://www.freedesktop.org/standards/dbus/1.0/introspect.dtd">
         
        <node name="/org/fedorahosted/certmonger">
         <interface name="org.freedesktop.DBus.Introspectable">
          <method name="Introspect">
           <arg name="xml_data" type="s" direction="out"/>
          </method>
         </interface>
         <interface name="org.freedesktop.DBus.Properties">
          <method name="Get">
           <arg name="interface_name" type="s" direction="in"/>
           <arg name="property_name" type="s" direction="in"/>
           <arg name="value" type="v" direction="out"/>
          </method>
          <method name="Set">
           <arg name="interface_name" type="s" direction="in"/>
           <arg name="property_name" type="s" direction="in"/>
           <arg name="value" type="v" direction="in"/>
          </method>
          <method name="GetAll">
           <arg name="interface_name" type="s" direction="in"/>
           <arg name="props" type="a{sv}" direction="out"/>
          </method>
          <signal name="PropertiesChanged">
           <arg name="interface_name" type="s"/>
           <arg name="changed_properties" type="a{sv}"/>
           <arg name="invalidated_properties" type="as"/>
          </signal>
         </interface>
         <interface name="org.fedorahosted.certmonger">
          <method name="add_known_ca">
           <arg name="nickname" type="s" direction="in"/>
           <arg name="command" type="s" direction="in"/>
           <arg name="known_names" type="as" direction="in"/>
           <arg name="status" type="b" direction="out"/>
           <arg name="name" type="o" direction="out"/>
          </method>
          <method name="add_request">
           <arg name="template" type="a{sv}" direction="in"/>
           <arg name="status" type="b" direction="out"/>
           <arg name="name" type="o" direction="out"/>
          </method>
          <method name="find_ca_by_nickname">
           <arg name="nickname" type="s" direction="in"/>
           <arg name="ca" type="o" direction="out"/>
          </method>
          <method name="find_request_by_nickname">
           <arg name="nickname" type="s" direction="in"/>
           <arg name="request" type="o" direction="out"/>
          </method>
          <method name="get_known_cas">
           <arg name="ca_list" type="ao" direction="out"/>
          </method>
          <method name="get_requests">
           <arg name="requests" type="ao" direction="out"/>
          </method>
          <method name="get_supported_key_types">
           <arg name="key_type_list" type="as" direction="out"/>
          </method>
          <method name="get_supported_key_storage">
           <arg name="key_storage_type_list" type="as" direction="out"/>
          </method>
          <method name="get_supported_cert_storage">
           <arg name="cert_storage_type_list" type="as" direction="out"/>
          </method>
          <method name="remove_known_ca">
           <arg name="ca" type="o" direction="in"/>
           <arg name="status" type="b" direction="out"/>
          </method>
          <method name="remove_request">
           <arg name="request" type="o" direction="in"/>
           <arg name="status" type="b" direction="out"/>
          </method>
         </interface>
         <node name="requests"/>
         <node name="cas"/>
        </node>"
        [root@user1 ]# docker exec systemd2 rpm -q certmonger
        certmonger-0.79.9-1.fc31.x86_64
        [root@user1 ]#
     
     
     
     
     
     
     
     
     
    non-containerized environment
     
    dnf module enable idm:DL1
    dnf module install idm:DL1/dns
    echo "DefaultLimitNOFILE=infinity" >> /etc/systemd/system.conf
    systemctl daemon-reload
    systemctl restart certmonger ; systemctl status certmonger ; dbus-send --system --type=method_call --print-reply --dest=org.fedorahosted.certmonger /org/fedorahosted/certmonger org.freedesktop.DBus.Introspectable.Introspect
    ipa-server-install --domain ipa.test --realm IPA.TEST -a Secret123 -p Secret123 -U
    systemctl restart certmonger ; systemctl status certmonger ; dbus-send --system --type=method_call --print-reply --dest=org.fedorahosted.certmonger /org/fedorahosted/certmonger org.freedesktop.DBus.Introspectable.Introspect
     
     
    [root@ci-vm-10-0-136-225 ~]# echo "DefaultLimitNOFILE=infinity" >> /etc/systemd/system.conf
    [root@ci-vm-10-0-136-225 ~]# systemctl daemon-reload
    [root@ci-vm-10-0-136-225 ~]# cat /etc/redhat-release
    Red Hat Enterprise Linux release 8.2 Beta (Ootpa)
    [root@ci-vm-10-0-136-225 ~]# rpm -q certmonger ipa-server
    certmonger-0.79.7-6.el8.x86_64
    ipa-server-4.8.4-6.module+el8.2.0+5773+68ace8c5.x86_64
     
     
    [root@ci-vm-10-0-136-225 ~]# ipa-server-install --domain ipa.test --realm IPA.TEST -a Secret123 -p Secret123 -U
     
    The log file for this installation can be found in /var/log/ipaserver-install.log
    ==============================================================================
    This program will set up the IPA Server.
    Version 4.8.4
     
    This includes:
      * Configure a stand-alone CA (dogtag) for certificate management
      * Configure the NTP client (chronyd)
      * Create and configure an instance of Directory Server
      * Create and configure a Kerberos Key Distribution Center (KDC)
      * Configure Apache (httpd)
      * Configure the KDC to enable PKINIT
     
     
    The IPA Master Server will be configured with:
    Hostname:       ci-vm-10-0-136-225.hosted.upshift.rdu2.redhat.com
    IP address(es): 10.0.136.225
    Domain name:    ipa.test
    Realm name:     IPA.TEST
     
    The CA will be configured with:
    Subject DN:   CN=Certificate Authority,O=IPA.TEST
    Subject base: O=IPA.TEST
    Chaining:     self-signed
     
    Disabled p11-kit-proxy
    Synchronizing time
    No SRV records of NTP servers found and no NTP server or pool address was provided.
    Using default chrony configuration.
    Attempting to sync time with chronyc.
    Process chronyc waitsync failed to sync time!
    Unable to sync time with chrony server, assuming the time is in sync. Please check that 123 UDP port is opened, and any time server is on network.
    Warning: IPA was unable to sync time with chrony!
             Time synchronization is required for IPA to work correctly
    Configuring directory server (dirsrv). Estimated time: 30 seconds
      [1/44]: creating directory server instance
      [2/44]: configure autobind for root
      [3/44]: stopping directory server
      [4/44]: updating configuration in dse.ldif
      [5/44]: starting directory server
      [6/44]: adding default schema
      [7/44]: enabling memberof plugin
      [8/44]: enabling winsync plugin
      [9/44]: configure password logging
      [10/44]: configuring replication version plugin
      [11/44]: enabling IPA enrollment plugin
      [12/44]: configuring uniqueness plugin
      [13/44]: configuring uuid plugin
      [14/44]: configuring modrdn plugin
      [15/44]: configuring DNS plugin
      [16/44]: enabling entryUSN plugin
      [17/44]: configuring lockout plugin
      [18/44]: configuring topology plugin
      [19/44]: creating indices
      [20/44]: enabling referential integrity plugin
      [21/44]: configuring certmap.conf
      [22/44]: configure new location for managed entries
      [23/44]: configure dirsrv ccache and keytab
      [24/44]: enabling SASL mapping fallback
      [25/44]: restarting directory server
      [26/44]: adding sasl mappings to the directory
      [27/44]: adding default layout
      [28/44]: adding delegation layout
      [29/44]: creating container for managed entries
      [30/44]: configuring user private groups
      [31/44]: configuring netgroups from hostgroups
      [32/44]: creating default Sudo bind user
      [33/44]: creating default Auto Member layout
      [34/44]: adding range check plugin
      [35/44]: creating default HBAC rule allow_all
      [36/44]: adding entries for topology management
      [37/44]: initializing group membership
      [38/44]: adding master entry
      [39/44]: initializing domain level
      [40/44]: configuring Posix uid/gid generation
      [41/44]: adding replication acis
      [42/44]: activating sidgen plugin
      [43/44]: activating extdom plugin
      [44/44]: configuring directory to start on boot
    Done configuring directory server (dirsrv).
    Configuring Kerberos KDC (krb5kdc)
      [1/10]: adding kerberos container to the directory
      [2/10]: configuring KDC
      [3/10]: initialize kerberos container
      [4/10]: adding default ACIs
      [5/10]: creating a keytab for the directory
      [6/10]: creating a keytab for the machine
      [7/10]: adding the password extension to the directory
      [8/10]: creating anonymous principal
      [9/10]: starting the KDC
      [10/10]: configuring KDC to start on boot
    Done configuring Kerberos KDC (krb5kdc).
    Configuring kadmin
      [1/2]: starting kadmin
      [2/2]: configuring kadmin to start on boot
    Done configuring kadmin.
    Configuring ipa-custodia
      [1/5]: Making sure custodia container exists
      [2/5]: Generating ipa-custodia config file
      [3/5]: Generating ipa-custodia keys
      [4/5]: starting ipa-custodia
      [5/5]: configuring ipa-custodia to start on boot
    Done configuring ipa-custodia.
    Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
      [1/29]: configuring certificate server instance
      [2/29]: Add ipa-pki-wait-running
      [3/29]: reindex attributes
      [4/29]: exporting Dogtag certificate store pin
      [5/29]: stopping certificate server instance to update CS.cfg
      [6/29]: backing up CS.cfg
      [7/29]: disabling nonces
      [8/29]: set up CRL publishing
      [9/29]: enable PKIX certificate path discovery and validation
      [10/29]: starting certificate server instance
      [11/29]: configure certmonger for renewals
      [12/29]: requesting RA certificate from CA
      [13/29]: setting audit signing renewal to 2 years
      [14/29]: restarting certificate server
      [15/29]: publishing the CA certificate
      [16/29]: adding RA agent as a trusted user
      [17/29]: authorizing RA to modify profiles
      [18/29]: authorizing RA to manage lightweight CAs
      [19/29]: Ensure lightweight CAs container exists
      [20/29]: configure certificate renewals
      [21/29]: Configure HTTP to proxy connections
      [22/29]: restarting certificate server
      [23/29]: updating IPA configuration
      [24/29]: enabling CA instance
      [25/29]: migrating certificate profiles to LDAP
      [26/29]: importing IPA certificate profiles
      [27/29]: adding default CA ACL
      [28/29]: adding 'ipa' CA entry
      [29/29]: configuring certmonger renewal for lightweight CAs
    Done configuring certificate server (pki-tomcatd).
    Configuring directory server (dirsrv)
      [1/3]: configuring TLS for DS instance
      [2/3]: adding CA certificate entry
      [3/3]: restarting directory server
    Done configuring directory server (dirsrv).
    Configuring ipa-otpd
      [1/2]: starting ipa-otpd
      [2/2]: configuring ipa-otpd to start on boot
    Done configuring ipa-otpd.
    Configuring the web interface (httpd)
      [1/21]: stopping httpd
      [2/21]: backing up ssl.conf
      [3/21]: disabling nss.conf
      [4/21]: configuring mod_ssl certificate paths
      [5/21]: setting mod_ssl protocol list
      [6/21]: configuring mod_ssl log directory
      [7/21]: disabling mod_ssl OCSP
      [8/21]: adding URL rewriting rules
      [9/21]: configuring httpd
    Nothing to do for configure_httpd_wsgi_conf
      [10/21]: setting up httpd keytab
      [11/21]: configuring Gssproxy
      [12/21]: setting up ssl
      [13/21]: configure certmonger for renewals
      [14/21]: publish CA cert
      [15/21]: clean up any existing httpd ccaches
      [16/21]: configuring SELinux for httpd
      [17/21]: create KDC proxy config
      [18/21]: enable KDC proxy
      [19/21]: starting httpd
      [20/21]: configuring httpd to start on boot
      [21/21]: enabling oddjobd
    Done configuring the web interface (httpd).
    Configuring Kerberos KDC (krb5kdc)
      [1/1]: installing X509 Certificate for PKINIT
    Done configuring Kerberos KDC (krb5kdc).
    Applying LDAP updates
    Upgrading IPA:. Estimated time: 1 minute 30 seconds
      [1/10]: stopping directory server
      [2/10]: saving configuration
      [3/10]: disabling listeners
      [4/10]: enabling DS global lock
      [5/10]: disabling Schema Compat
      [6/10]: starting directory server
      [7/10]: upgrading server
      [8/10]: stopping directory server
      [9/10]: restoring configuration
      [10/10]: starting directory server
    Done.
    Restarting the KDC
    Configuring client side components
    This program will set up IPA client.
    Version 4.8.4
     
    Using existing certificate '/etc/ipa/ca.crt'.
    Client hostname: ci-vm-10-0-136-225.hosted.upshift.rdu2.redhat.com
    Realm: IPA.TEST
    DNS Domain: ipa.test
    IPA Server: ci-vm-10-0-136-225.hosted.upshift.rdu2.redhat.com
    BaseDN: dc=ipa,dc=test
     
    Configured sudoers in /etc/authselect/user-nsswitch.conf
    Configured /etc/sssd/sssd.conf
    Systemwide CA database updated.
    Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub
    Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub
    Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
    Could not update DNS SSHFP records.
    SSSD enabled
    Configured /etc/openldap/ldap.conf
    Configured /etc/ssh/ssh_config
    Configured /etc/ssh/sshd_config
    Configuring ipa.test as NIS domain.
    Client configuration complete.
    The ipa-client-install command was successful
     
    Please add records in this file to your DNS system: /tmp/ipa.system.records.6ih2oifn.db
    ==============================================================================
    Setup complete
     
    Next steps:
            1. You must make sure these network ports are open:
                    TCP Ports:
                      * 80, 443: HTTP/HTTPS
                      * 389, 636: LDAP/LDAPS
                      * 88, 464: kerberos
                    UDP Ports:
                      * 88, 464: kerberos
                      * 123: ntp
     
            2. You can now obtain a kerberos ticket using the command: 'kinit admin'
               This ticket will allow you to use the IPA tools (e.g., ipa user-add)
               and the web user interface.
     
    Be sure to back up the CA certificates stored in /root/cacert.p12
    These files are required to create replicas. The password for these
    files is the Directory Manager password
    The ipa-server-install command was successful
    [root@ci-vm-10-0-136-225 ~]# systemctl restart certmonger ; systemctl status certmonger ; dbus-send --system --type=method_call --print-reply --dest=org.fedorahosted.certmonger /org/fedorahosted/certmonger org.freedesktop.DBus.Introspectable.Introspect
    ● certmonger.service - Certificate monitoring and PKI enrollment
       Loaded: loaded (/usr/lib/systemd/system/certmonger.service; enabled; vendor preset: disabled)
       Active: active (running) since Fri 2020-03-06 00:30:59 EST; 82ms ago
     Main PID: 11488 (certmonger)
        Tasks: 3
       Memory: 5.1M
       CGroup: /system.slice/certmonger.service
               ├─11488 /usr/sbin/certmonger -S -p /var/run/certmonger.pid -n -d2
               ├─11508 /usr/libexec/platform-python -I /usr/libexec/certmonger/ipa-server-guard /usr/libexec/certmonger/ipa-submit
               ├─11509 /usr/libexec/platform-python -I /usr/libexec/certmonger/ipa-server-guard /usr/libexec/certmonger/ipa-submit
               ├─11510 /usr/libexec/platform-python -I /usr/libexec/certmonger/ipa-server-guard /usr/libexec/certmonger/ipa-submit
               └─11511 /usr/sbin/certmonger -S -p /var/run/certmonger.pid -n -d2
     
    Mar 06 00:30:58 ci-vm-10-0-136-225.hosted.upshift.rdu2.redhat.com certmonger[11488]: 2020-03-06 00:30:58 [11494] Token is named "NSS Generi…pping.
    Mar 06 00:30:58 ci-vm-10-0-136-225.hosted.upshift.rdu2.redhat.com certmonger[11488]: 2020-03-06 00:30:58 [11496] Token is named "NSS Generi…pping.
    Mar 06 00:30:59 ci-vm-10-0-136-225.hosted.upshift.rdu2.redhat.com certmonger[11488]: 2020-03-06 00:30:59 [11498] Token is named "NSS Generi…pping.
    Mar 06 00:30:59 ci-vm-10-0-136-225.hosted.upshift.rdu2.redhat.com certmonger[11488]: 2020-03-06 00:30:59 [11500] Token is named "NSS Generi…pping.
    Mar 06 00:30:59 ci-vm-10-0-136-225.hosted.upshift.rdu2.redhat.com certmonger[11488]: 2020-03-06 00:30:59 [11502] Token is named "NSS Generi…pping.
    Mar 06 00:30:59 ci-vm-10-0-136-225.hosted.upshift.rdu2.redhat.com systemd[1]: Started Certificate monitoring and PKI enrollment.
    Mar 06 00:30:59 ci-vm-10-0-136-225.hosted.upshift.rdu2.redhat.com certmonger[11488]: 2020-03-06 00:30:59 [11508] Running enrollment/cadata …uard".
    Mar 06 00:30:59 ci-vm-10-0-136-225.hosted.upshift.rdu2.redhat.com certmonger[11488]: 2020-03-06 00:30:59 [11509] Running enrollment/cadata …uard".
    Mar 06 00:30:59 ci-vm-10-0-136-225.hosted.upshift.rdu2.redhat.com certmonger[11488]: 2020-03-06 00:30:59 [11510] Running enrollment/cadata …uard".
    Mar 06 00:30:59 ci-vm-10-0-136-225.hosted.upshift.rdu2.redhat.com certmonger[11488]: 2020-03-06 00:30:59 [11511] Running enrollment/cadata …uard".
    Hint: Some lines were ellipsized, use -l to show in full.
    method return time=1583472661.974697 sender=:1.135 -> destination=:1.136 serial=5 reply_serial=2
       string "<!DOCTYPE node PUBLIC "-//freedesktop//DTD D-BUS Object Introspection 1.0//EN"
    "http://www.freedesktop.org/standards/dbus/1.0/introspect.dtd">
     
    <node name="/org/fedorahosted/certmonger">
     <interface name="org.freedesktop.DBus.Introspectable">
      <method name="Introspect">
       <arg name="xml_data" type="s" direction="out"/>
      </method>
     </interface>
     <interface name="org.freedesktop.DBus.Properties">
      <method name="Get">
       <arg name="interface_name" type="s" direction="in"/>
       <arg name="property_name" type="s" direction="in"/>
       <arg name="value" type="v" direction="out"/>
      </method>
      <method name="Set">
       <arg name="interface_name" type="s" direction="in"/>
       <arg name="property_name" type="s" direction="in"/>
       <arg name="value" type="v" direction="in"/>
      </method>
      <method name="GetAll">
       <arg name="interface_name" type="s" direction="in"/>
       <arg name="props" type="a{sv}" direction="out"/>
      </method>
      <signal name="PropertiesChanged">
       <arg name="interface_name" type="s"/>
       <arg name="changed_properties" type="a{sv}"/>
       <arg name="invalidated_properties" type="as"/>
      </signal>
     </interface>
     <interface name="org.fedorahosted.certmonger">
      <method name="add_known_ca">
       <arg name="nickname" type="s" direction="in"/>
       <arg name="command" type="s" direction="in"/>
       <arg name="known_names" type="as" direction="in"/>
       <arg name="status" type="b" direction="out"/>
       <arg name="name" type="o" direction="out"/>
      </method>
      <method name="add_request">
       <arg name="template" type="a{sv}" direction="in"/>
       <arg name="status" type="b" direction="out"/>
       <arg name="name" type="o" direction="out"/>
      </method>
      <method name="find_ca_by_nickname">
       <arg name="nickname" type="s" direction="in"/>
       <arg name="ca" type="o" direction="out"/>
      </method>
      <method name="find_request_by_nickname">
       <arg name="nickname" type="s" direction="in"/>
       <arg name="request" type="o" direction="out"/>
      </method>
      <method name="get_known_cas">
       <arg name="ca_list" type="ao" direction="out"/>
      </method>
      <method name="get_requests">
       <arg name="requests" type="ao" direction="out"/>
      </method>
      <method name="get_supported_key_types">
       <arg name="key_type_list" type="as" direction="out"/>
      </method>
      <method name="get_supported_key_storage">
       <arg name="key_storage_type_list" type="as" direction="out"/>
      </method>
      <method name="get_supported_cert_storage">
       <arg name="cert_storage_type_list" type="as" direction="out"/>
      </method>
      <method name="remove_known_ca">
       <arg name="ca" type="o" direction="in"/>
       <arg name="status" type="b" direction="out"/>
      </method>
      <method name="remove_request">
       <arg name="request" type="o" direction="in"/>
       <arg name="status" type="b" direction="out"/>
      </method>
     </interface>
     <node name="requests"/>
     <node name="cas"/>
    </node>"
    [root@ci-vm-10-0-136-225 ~]#
    [root@ci-vm-10-0-136-225 ~]# strace -e trace=close -p 11488 -o /tmp/strace_certmonger.out -s 256 -f &
    [1] 12090
    [root@ci-vm-10-0-136-225 ~]# strace: Process 11488 attached
    [root@ci-vm-10-0-136-225 ~]# cat /tmp/strace_certmonger.out
    11488 --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=1, si_uid=0} ---
    11488 --- SIGCONT {si_signo=SIGCONT, si_code=SI_USER, si_pid=1, si_uid=0} ---
    11488 close(11)                         = 0
    11488 close(11)                         = 0
    11488 close(11)                         = 0
    11488 close(11)                         = 0
    11488 close(11)                         = 0
    11488 close(11)                         = 0
    11488 close(11)                         = 0
    11488 close(11)                         = 0
    11488 close(11)                         = 0
    11488 close(11)                         = 0
    11488 close(11)                         = 0
    11488 close(11)                         = 0
    11488 close(11)                         = 0
    11488 close(11)                         = 0
    11488 close(11)                         = 0
    11488 close(11)                         = 0
    11488 close(5)                          = 0
    11488 close(3)                          = 0
    11488 close(8)                          = 0
    11488 +++ exited with 0 +++


Based on above observations marking Bugzilla verified.

Comment 7 errata-xmlrpc 2020-04-28 16:01:49 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1704

Note You need to log in before you can comment on or make changes to this bug.