Bug 1504688

Summary: certmonger is not able to set well-known bus name "org.fedorahosted.certmonger"
Product: Red Hat Enterprise Linux 7 Reporter: Pavel Vomacka <pvomacka>
Component: dbusAssignee: David King <dking>
Status: CLOSED NOTABUG QA Contact: Desktop QE <desktop-qa-list>
Severity: urgent Docs Contact:
Priority: high    
Version: 7.5CC: plarsen, rcritten, spoore
Target Milestone: rcKeywords: Reopened, TestBlocker
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-27 16:03:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pavel Vomacka 2017-10-20 12:03:42 UTC
Description of problem:
When trying to install IPA server on RHEL. It fails on configuring certmonger.

Version-Release number of selected component (if applicable):
ipa-4.5.4-1.el7
dbus-1.10.24-3.el7.x86_64
certmonger-0.78.4-3.el7.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Run ipa-server-install: $ sudo ipa-server-install -p Passwd123 -a Passwd123 -n EXAMPLE.COM -r EXAMPLE.COM 
2. Installation fails with following error:

CalledProcessError: Command '/bin/systemctl start certmonger.service' returned non-zero exit status 1
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    Command '/bin/systemctl start certmonger.service' returned non-zero exit status 1
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information


Actual results:
Installation fails

Expected results:
Installation passes

Additional info:
Traceback seen in /var/log/ipaserver-install.log:

2017-10-20T08:25:04Z DEBUG stderr=Job for certmonger.service failed because the control process exited with error code. See "systemctl status certmonger.service" and "journalctl -xe" for details.

2017-10-20T08:25:04Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 504, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 494, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dogtaginstance.py", line 250, in configure_certmonger_renewal
    cmonger.start()
  File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 294, in start
    skip_output=not capture_output)
  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 523, in run
    raise CalledProcessError(p.returncode, arg_string, str(output))
CalledProcessError: Command '/bin/systemctl start certmonger.service' returned non-zero exit status 1

2017-10-20T08:25:04Z DEBUG   [error] CalledProcessError: Command '/bin/systemctl start certmonger.service' returned non-zero exit status 1
2017-10-20T08:25:04Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 333, in run
    cfgr.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 368, in run
    self.execute()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 392, in execute
    for _nothing in self._executor():
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 434, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 463, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 453, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 424, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 658, in _configure
    next(executor)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 434, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 463, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 521, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 453, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 518, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 453, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 424, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 63, in _install
    for _nothing in self._installer(self.parent):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/__init__.py", line 578, in main
    master_install(self)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/install.py", line 248, in decorated
    func(installer)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/install.py", line 798, in install
    ca.install_step_0(False, None, options)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ca.py", line 284, in install_step_0
    use_ldaps=standalone)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 447, in configure_instance
    self.start_creation(runtime=runtime)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 504, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 494, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dogtaginstance.py", line 250, in configure_certmonger_renewal
    cmonger.start()
  File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 294, in start
    skip_output=not capture_output)
  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 523, in run
    raise CalledProcessError(p.returncode, arg_string, str(output))

2017-10-20T08:25:04Z DEBUG The ipa-server-install command failed, exception: CalledProcessError: Command '/bin/systemctl start certmonger.service' returned non-zero exit status 1




output of command # journalctl -u certmonger.service: 

Oct 20 10:45:04 vm.example.com systemd[1]: Starting Certificate monitoring and PKI enrollment...
Oct 20 10:45:04 vm.example.com certmonger[40723]: 2017-10-20 10:45:04 [40723] Unable to set well-known bus name "org.fedorahosted.certmonger": Connection ":1.59" is not allowed to own the service "org.fedorahosted.certmonger" due to security policies in the configuration file(-1).
Oct 20 10:45:04 vm.example.com certmonger[40723]: Error connecting to D-Bus.
Oct 20 10:45:04 vm.example.com systemd[1]: certmonger.service: main process exited, code=exited, status=1/FAILURE
Oct 20 10:45:04 vm.example.com systemd[1]: Failed to start Certificate monitoring and PKI enrollment.
Oct 20 10:45:04 vm.example.com systemd[1]: Unit certmonger.service entered failed state.
Oct 20 10:45:04 vm.example.com systemd[1]: certmonger.service failed.

Comment 2 Pavel Vomacka 2017-10-20 12:12:38 UTC
I forgot to add exact step of IPA installation when it failed:

Configuring NTP daemon (ntpd)
  [1/4]: stopping ntpd
  [2/4]: writing configuration
  [3/4]: configuring ntpd to start on boot
  [4/4]: starting ntpd
Done configuring NTP daemon (ntpd).
Configuring directory server (dirsrv). Estimated time: 30 seconds
  [1/45]: creating directory server instance
  [2/45]: enabling ldapi
  [3/45]: configure autobind for root
  [4/45]: stopping directory server
  [5/45]: updating configuration in dse.ldif
  [6/45]: starting directory server
  [7/45]: adding default schema
  [8/45]: enabling memberof plugin
  [9/45]: enabling winsync plugin
  [10/45]: configuring replication version plugin
  [11/45]: enabling IPA enrollment plugin
  [12/45]: configuring uniqueness plugin
  [13/45]: configuring uuid plugin
  [14/45]: configuring modrdn plugin
  [15/45]: configuring DNS plugin
  [16/45]: enabling entryUSN plugin
  [17/45]: configuring lockout plugin
  [18/45]: configuring topology plugin
  [19/45]: creating indices
  [20/45]: enabling referential integrity plugin
  [21/45]: configuring certmap.conf
  [22/45]: configure new location for managed entries
  [23/45]: configure dirsrv ccache
  [24/45]: enabling SASL mapping fallback
  [25/45]: restarting directory server
  [26/45]: adding sasl mappings to the directory
  [27/45]: adding default layout
  [28/45]: adding delegation layout
  [29/45]: creating container for managed entries
  [30/45]: configuring user private groups
  [31/45]: configuring netgroups from hostgroups
  [32/45]: creating default Sudo bind user
  [33/45]: creating default Auto Member layout
  [34/45]: adding range check plugin
  [35/45]: creating default HBAC rule allow_all
  [36/45]: adding entries for topology management
  [37/45]: initializing group membership
  [38/45]: adding master entry
  [39/45]: initializing domain level
  [40/45]: configuring Posix uid/gid generation
  [41/45]: adding replication acis
  [42/45]: activating sidgen plugin
  [43/45]: activating extdom plugin
  [44/45]: tuning directory server
  [45/45]: configuring directory to start on boot
Done configuring directory server (dirsrv).
Configuring Kerberos KDC (krb5kdc)
  [1/10]: adding kerberos container to the directory
  [2/10]: configuring KDC
  [3/10]: initialize kerberos container
WARNING: Your system is running out of entropy, you may experience long delays
  [4/10]: adding default ACIs
  [5/10]: creating a keytab for the directory
  [6/10]: creating a keytab for the machine
  [7/10]: adding the password extension to the directory
  [8/10]: creating anonymous principal
  [9/10]: starting the KDC
  [10/10]: configuring KDC to start on boot
Done configuring Kerberos KDC (krb5kdc).
Configuring kadmin
  [1/2]: starting kadmin 
  [2/2]: configuring kadmin to start on boot
Done configuring kadmin.
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/29]: configuring certificate server instance
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure CA instance: Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpEeIPqY' returned non-zero exit status 1
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the installation logs and the following files/directories for more information:
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL   /var/log/pki/pki-tomcat
  [error] RuntimeError: CA configuration failed.
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    CA configuration failed.
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

Comment 5 Rob Crittenden 2017-10-23 18:03:54 UTC
I've been unable to reproduce this. My IPA install worked ok using the Oct 20 nightly.

Comment 6 Rob Crittenden 2017-11-08 18:18:38 UTC
I've heard no other reports of this and haven't run into it myself. Closing. We can re-open if it comes up again.

Comment 7 Peter Larsen 2018-01-02 23:20:53 UTC
Reproduced - 4.5.

Comment 8 Peter Larsen 2018-01-02 23:21:31 UTC
running "certmonger -S -d 10" produces this:

2018-01-02 18:19:04 [31960] Adding disabled DBus watch on FD 7 (for Write) for 0x55c5484cae90.
2018-01-02 18:19:04 [31960] Adding a watch group for FD 7 for 0x55c5484cae90.
2018-01-02 18:19:04 [31960] Dequeuing FD 7 for 0x55c5484cae90:(nil).
2018-01-02 18:19:04 [31960] Not queuing FD 7 for 0x55c5484cae90.
2018-01-02 18:19:04 [31960] Adding enabled DBus watch on FD 7 (for Read) for 0x55c5484cae90.
2018-01-02 18:19:04 [31960] Dequeuing FD 7 for 0x55c5484cae90:(nil).
2018-01-02 18:19:04 [31960] Queuing FD 7 for Read for 0x55c5484cae90:0x55c5484ce890.
2018-01-02 18:19:04 [31960] Unable to set well-known bus name "org.fedorahosted.certmonger": (2).
Error connecting to D-Bus.

Comment 9 Scott Poore 2018-02-26 16:09:06 UTC
I just saw this bug occur:

Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/29]: configuring certificate server instance
  [2/29]: exporting Dogtag certificate store pin
  [3/29]: stopping certificate server instance to update CS.cfg
  [4/29]: backing up CS.cfg
  [5/29]: disabling nonces
  [6/29]: set up CRL publishing
  [7/29]: enable PKIX certificate path discovery and validation
  [8/29]: starting certificate server instance
  [9/29]: configure certmonger for renewals
  [error] CalledProcessError: Command '/bin/systemctl start certmonger.service' returned non-zero exit status 1
 
 
STDERR:
 
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    Command '/bin/systemctl start certmonger.service' returned non-zero exit status 1
ipa.ipapython.install.cli.install_tool(CompatServerMasterInstall): ERROR    The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information
 

From journalctl:

Feb 26 10:31:35 intel-sugarbay-dh-02 systemd[1]: Starting Certificate monitoring and PKI enrollment...
Feb 26 10:31:35 intel-sugarbay-dh-02 certmonger[18924]: 2018-02-26 10:31:35 [18924] Unable to set well-known bus name "org.fedorahosted.certmonger": Connection ":1.49" is not allowed to own the service "org.fedorahosted.certmonger" due to security policies in the configuration file(-1).
Feb 26 10:31:35 intel-sugarbay-dh-02 certmonger[18924]: Error connecting to D-Bus.
Feb 26 10:31:35 intel-sugarbay-dh-02 systemd[1]: certmonger.service: main process exited, code=exited, status=1/FAILURE
Feb 26 10:31:35 intel-sugarbay-dh-02 systemd[1]: Failed to start Certificate monitoring and PKI enrollment.
Feb 26 10:31:35 intel-sugarbay-dh-02 systemd[1]: Unit certmonger.service entered failed state.
Feb 26 10:31:35 intel-sugarbay-dh-02 systemd[1]: certmonger.service failed.


[root@intel-sugarbay-dh-02 log]# cat /etc/dbus-1/system.d/certmonger.conf
<!DOCTYPE busconfig PUBLIC
 "-//freedesktop//DTD D-BUS Bus Configuration 1.0//EN"
 "http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd">
<busconfig>

  <!-- Only root can own and communicate with the service -->

  <policy context="default">
    <deny own="org.fedorahosted.certmonger"/>
    <deny send_destination="org.fedorahosted.certmonger"/>
  </policy>

  <policy user="root">
    <allow own="org.fedorahosted.certmonger"/>

    <allow send_destination="org.fedorahosted.certmonger"
           send_interface="org.fedorahosted.certmonger"/>
    <allow send_destination="org.fedorahosted.certmonger"
           send_interface="org.fedorahosted.certmonger.ca"/>
    <allow send_destination="org.fedorahosted.certmonger"
           send_interface="org.fedorahosted.certmonger.request"/>
    <allow send_destination="org.fedorahosted.certmonger"
           send_interface="org.freedesktop.DBus.Introspectable"/>
    <allow send_destination="org.fedorahosted.certmonger"
           send_interface="org.freedesktop.DBus.Properties"/>
  </policy>

</busconfig>


[root@intel-sugarbay-dh-02 log]# ps -ef|grep dbus
dbus       814     1  0 10:17 ?        00:00:01 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation


[root@intel-sugarbay-dh-02 log]# certmonger -S -d 10
2018-02-26 11:07:05 [19194] Starting up.
2018-02-26 11:07:05 [19194] Changing to root directory.
2018-02-26 11:07:05 [19194] Obtaining system lock.
2018-02-26 11:07:05 [19194] CA1('SelfSign').identify starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA1('SelfSign').certs starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA1('SelfSign').profiles starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA1('SelfSign').default_profile starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA1('SelfSign').enrollment_reqs starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA1('SelfSign').renewal_reqs starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA1('SelfSign').capabilities starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA1('SelfSign').encryption_certs starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA2('IPA').identify starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA2('IPA').certs starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA2('IPA').profiles starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA2('IPA').default_profile starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA2('IPA').enrollment_reqs starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA2('IPA').renewal_reqs starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA2('IPA').capabilities starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA2('IPA').encryption_certs starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA3('certmaster').identify starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA3('certmaster').certs starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA3('certmaster').profiles starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA3('certmaster').default_profile starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA3('certmaster').enrollment_reqs starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA3('certmaster').renewal_reqs starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA3('certmaster').capabilities starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA3('certmaster').encryption_certs starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA4('dogtag-ipa-renew-agent').identify starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA4('dogtag-ipa-renew-agent').certs starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA4('dogtag-ipa-renew-agent').profiles starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA4('dogtag-ipa-renew-agent').default_profile starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA4('dogtag-ipa-renew-agent').enrollment_reqs starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA4('dogtag-ipa-renew-agent').renewal_reqs starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA4('dogtag-ipa-renew-agent').capabilities starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA4('dogtag-ipa-renew-agent').encryption_certs starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA5('local').identify starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA5('local').certs starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA5('local').profiles starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA5('local').default_profile starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA5('local').enrollment_reqs starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA5('local').renewal_reqs starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA5('local').capabilities starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] CA5('local').encryption_certs starts (NEED_TO_REFRESH)
2018-02-26 11:07:05 [19194] Adding disabled DBus watch on FD 7 (for Write) for 0x557534321f50.
2018-02-26 11:07:05 [19194] Adding a watch group for FD 7 for 0x557534321f50.
2018-02-26 11:07:05 [19194] Dequeuing FD 7 for 0x557534321f50:(nil).
2018-02-26 11:07:05 [19194] Not queuing FD 7 for 0x557534321f50.
2018-02-26 11:07:05 [19194] Adding enabled DBus watch on FD 7 (for Read) for 0x557534321f50.
2018-02-26 11:07:05 [19194] Dequeuing FD 7 for 0x557534321f50:(nil).
2018-02-26 11:07:05 [19194] Queuing FD 7 for Read for 0x557534321f50:0x557534324d50.
2018-02-26 11:07:05 [19194] Unable to set well-known bus name "org.fedorahosted.certmonger": Connection ":1.69" is not allowed to own the service "org.fedorahosted.certmonger" due to security policies in the configuration file(-1).
Error connecting to D-Bus.


If I restart dbus, it then seems to work.

Comment 11 David King 2018-02-27 16:03:53 UTC
(In reply to Scott Poore from comment #9)
>…
> If I restart dbus, it then seems to work.

Several bugs of this sort have cropped up recently, so please see bug 1519206 and bug 1531506 for some explanation. In summary, dbus is not able to be restarted after upgrading it on a running system, and the system as a whole needs to be restarted. This is because dbus is low-level enough to be used by several systemd daemons (among others), and restarts of the service are poorly tolerated (or simply not handled at all).