When I tested this on Scott's machine what I saw was that list-cas caused it to fail.

I could list requests and start/stop tracking of certs but not list the CAs.

I stopped my investigation at that point.

There are two things going on here.

The first is a bug that happens when certmonger reconnects to the system message bus, which it's only able to do so because it marks its original connection with the "don't exit on disconnect" flag.  It fails to mark the new connection with the same flag, so if it's disconnected a second time, it exits.  This will be fixed upstream shortly.

The second is that before 0.51, the identifiers for CAs and certificate requests were incorporated into the names which were assigned to them when they were exposed via D-Bus as part of the API which certmonger provides to its clients (getcert is such a client).  However, the name "dogtag-ipa-renew-agent" is not part of a valid object name as far as D-Bus is concerned, so when a client asked it for a list of known CAs, and certmonger attempted to respond with a list of their object names, the message bus daemon would notice that the reply message included invalid data and would drop certmonger.  This was fixed upstream in version 0.51, so versions of the package in EL 6.3 and later didn't suffer from it.

When the second bug was triggered, the first bug was triggered, and if the second bug was triggered again, the service would be stopped, and that's what's happening here.  The patch for it is pretty trivial, so I'll leave this one open to track the first bug, as the second one is already fixed.

So it's that one dogtag-ipa-renew-agent name causing all to be dropped and then certmonger stopped?  

Should those files have been left behind after uninstall/downgrade I did?  Or should they have been cleaned up by the rpm removal?

Can I test for this bug with something like this:

After uninstall:

grep id=dogtag-ipa-renew-agent /var/lib/certmonger/cas/*

If it's found, and version is < 0.51, then, we've hit this bug?


Thanks,
Scott

IPA should probably remove this file on uninstall.

Resetting priority/severity back to medium as I just realized I accidentally unset that.

(In reply to Scott Poore from comment #3)
> So it's that one dogtag-ipa-renew-agent name causing all to be dropped and
> then certmonger stopped?  
> 
> Should those files have been left behind after uninstall/downgrade I did? 
> Or should they have been cleaned up by the rpm removal?
> 
> Can I test for this bug with something like this:
> 
> After uninstall:
> 
> grep id=dogtag-ipa-renew-agent /var/lib/certmonger/cas/*
> 
> If it's found, and version is < 0.51, then, we've hit this bug?

Yes.  6.4 went out with 0.61, so what exactly are the expectations here?

I was looking for a way to confirm if the bug is seen.  My upgrade testing can start on older versions of RHEL.  So the re-install can occur on an older version of RHEL 6.  I'm just looking to see if there's an easy way to flag this when it is seen.

So, the newer version of certmonger does cleanup of the file with id=dogtag-ipa-renew-agent now or this is something that ipa should handle?  Or does it handle this now?

Thanks

The packaging starting with 0.75.5 should remove the dogtag-ipa-renew-cert CA when it detects a version older than 0.58 being installed.

Nalin,

So to do the test for this bug, will this process work:

upgrade from 6.3 to 6.6
uninstall
reinstall 6.3

And this should no longer be triggered right?

The uninstall step should now clean up or will it still leave things in a state that is expected to show this bug on the reinstall of IPA for the  RHEL6.3 version?

Thanks,
Scott

If you're doing a from-scratch reinstall of the OS, you wouldn't have been hitting this problem.  The newer package notices when it's being downgraded to an older version, and removes its record of the CA that the older version won't be able to handle after the downgrade completes.  (If the packages is being removed entirely, then the older version installed rather than downgraded, then the newer package won't be able to do anything to help.)

Not from scratch reinstall.  Sorry, wasn't clear there.  The upgrade/uninstall/reinstall refers to ipa and related components' rpms.

And, as of right now, certmonger is completely removed by the test scripts during the uninstall step.  It's not downgraded.  So removal still leaves things behind that the older version will have a problem with in this case?

However, it sounds like I should now be able to move certmonger from full remove to a downgrade and no longer see a problem?  well for downgrades from this fixed version to the older?

(In reply to Scott Poore from comment #12)
> Not from scratch reinstall.  Sorry, wasn't clear there.  The
> upgrade/uninstall/reinstall refers to ipa and related components' rpms.
> 
> And, as of right now, certmonger is completely removed by the test scripts
> during the uninstall step.  It's not downgraded.  So removal still leaves
> things behind that the older version will have a problem with in this case?

Yes.  I guess we could do this on post-uninstalls too, but how would we know we should even bother?

> However, it sounds like I should now be able to move certmonger from full
> remove to a downgrade and no longer see a problem?  well for downgrades from
> this fixed version to the older?

Yes.

I have confirmed the same failure with downgrade from 6.5 version of IPA and certmonger back to 6.2 and then re-installed:

root        : ERROR    certmonger failed starting to track certificate: Command '/usr/bin/ipa-getcert start-tracking -d /etc/httpd/alias -n
Server-Cert -p /etc/httpd/alias/pwdfile.txt' returned non-zero exit status 1

Now cleanup reinstall 6.2 version and incrementally upgraded to 6.6 including certmonger:

[root@nocp12 ~]# rpm -q certmonger
certmonger-0.75.6-1.el6.x86_64

But, When I uninstall/reinstall there, I now see a client install error during the ipa-server-install:

+ ipa-server-install -r EXAMPLE.COM -p Secret123 -a Secret123 -U

The log file for this installation can be found in /var/log/ipaserver-install.log
==============================================================================
This program will set up the IPA Server.

This includes:
  * Configure a stand-alone CA (dogtag) for certificate management
  * Configure the Network Time Daemon (ntpd)
  * Create and configure an instance of Directory Server
  * Create and configure a Kerberos Key Distribution Center (KDC)
  * Configure Apache (httpd)

To accept the default shown in brackets, press the Enter key.

The domain name has been calculated based on the host name.

The IPA Master Server will be configured with
Hostname:    zippyvm1.$DOMAIN
IP address:  10.8.60.60
Domain name: $DOMAIN

Configuring ntpd
  [1/4]: stopping ntpd
  [2/4]: writing configuration
  [3/4]: configuring ntpd to start on boot
  [4/4]: starting ntpd
done configuring ntpd.
Configuring directory server for the CA: Estimated time 30 seconds
  [1/3]: creating directory server user
  [2/3]: creating directory server instance
  [3/3]: restarting directory server
done configuring pkids.
Configuring certificate server: Estimated time 3 minutes 30 seconds
  [1/17]: creating certificate server user
  [2/17]: creating pki-ca instance
  [3/17]: configuring certificate server instance
  [4/17]: disabling nonces
  [5/17]: creating CA agent PKCS#12 file in /root
  [6/17]: creating RA agent certificate database
  [7/17]: importing CA chain to RA certificate database
  [8/17]: fixing RA database permissions
  [9/17]: setting up signing cert profile
  [10/17]: set up CRL publishing
  [11/17]: set certificate subject base
  [12/17]: configuring certificate server to start on boot
  [13/17]: restarting certificate server
  [14/17]: requesting RA certificate from CA
  [15/17]: issuing RA agent certificate
  [16/17]: adding RA agent as a trusted user
  [17/17]: Configure HTTP to proxy connections
done configuring pki-cad.
Configuring directory server: Estimated time 1 minute
  [1/35]: creating directory server user
  [2/35]: creating directory server instance
  [3/35]: adding default schema
  [4/35]: enabling memberof plugin
  [5/35]: enabling referential integrity plugin
  [6/35]: enabling winsync plugin
  [7/35]: configuring replication version plugin
  [8/35]: enabling IPA enrollment plugin
  [9/35]: enabling ldapi
  [10/35]: configuring uniqueness plugin
  [11/35]: configuring uuid plugin
  [12/35]: configuring modrdn plugin
  [13/35]: enabling entryUSN plugin
  [14/35]: configuring lockout plugin
  [15/35]: creating indices
  [16/35]: configuring ssl for ds instance
  [17/35]: configuring certmap.conf
  [18/35]: configure autobind for root
  [19/35]: configure new location for managed entries
  [20/35]: restarting directory server
  [21/35]: adding default layout
  [22/35]: adding delegation layout
  [23/35]: adding replication acis
  [24/35]: creating container for managed entries
  [25/35]: configuring user private groups
  [26/35]: configuring netgroups from hostgroups
  [27/35]: creating default Sudo bind user
  [28/35]: creating default Auto Member layout
  [29/35]: creating default HBAC rule allow_all
  [30/35]: initializing group membership
  [31/35]: adding master entry
  [32/35]: configuring Posix uid/gid generation
  [33/35]: enabling compatibility plugin
Restarting IPA to initialize updates before performing deletes:
  [1/2]: stopping directory server
  [2/2]: starting directory server
done configuring dirsrv.
  [34/35]: tuning directory server
  [35/35]: configuring directory to start on boot
done configuring dirsrv.
Configuring Kerberos KDC: Estimated time 30 seconds
  [1/14]: setting KDC account password
  [2/14]: adding sasl mappings to the directory
  [3/14]: adding kerberos entries to the DS
  [4/14]: adding default ACIs
  [5/14]: configuring KDC
  [6/14]: adding default keytypes
  [7/14]: adding default password policy
  [8/14]: creating a keytab for the directory
  [9/14]: creating a keytab for the machine
  [10/14]: exporting the kadmin keytab
  [11/14]: adding the password extension to the directory
  [12/14]: adding the kerberos master key to the directory
  [13/14]: starting the KDC
  [14/14]: configuring KDC to start on boot
done configuring krb5kdc.
Configuring ipa_kpasswd
  [1/2]: starting ipa_kpasswd 
  [2/2]: configuring ipa_kpasswd to start on boot
done configuring ipa_kpasswd.
Configuring the web interface: Estimated time 1 minute
  [1/13]: disabling mod_ssl in httpd
  [2/13]: setting mod_nss port to 443
  [3/13]: setting mod_nss password file
  [4/13]: enabling mod_nss renegotiate
  [5/13]: adding URL rewriting rules
  [6/13]: configuring httpd
  [7/13]: setting up ssl
  [8/13]: setting up browser autoconfig
  [9/13]: publish CA cert
  [10/13]: creating a keytab for httpd
  [11/13]: configuring SELinux for httpd
  [12/13]: restarting httpd
  [13/13]: configuring httpd to start on boot
done configuring httpd.
Applying LDAP updates
Restarting IPA to initialize updates before performing deletes:
  [1/2]: stopping directory server
  [2/2]: starting directory server
done configuring dirsrv.
Restarting the directory server
Restarting the KDC
Restarting the web server
Sample zone file for bind has been created in /tmp/sample.zone.FS8AV6.db
Configuration of client side components failed!
ipa-client-install returned: Command '/usr/sbin/ipa-client-install --on-master --unattended --domain $DOMAIN --server zippyvm1.$DOMAIN --realm
EXAMPLE.COM --hostname zippyvm1.$DOMAIN' returned non-zero exit status 1



From ipaserver-install.log:

2014-07-06 21:40:18,296 DEBUG args=/usr/sbin/ipa-client-install --on-master --unattended --domain $DOMAIN --server
zippyvm1.$DOMAIN --realm EXAMPLE.COM --hostname zippyvm1.$DOMAIN
2014-07-06 21:40:18,296 DEBUG stdout=ESC[?1034hDiscovery was successful!
Hostname: zippyvm1.$DOMAIN
Realm: EXAMPLE.COM
DNS Domain: $DOMAIN
IPA Server: zippyvm1.$DOMAIN
BaseDN: dc=example,dc=com


New SSSD config will be created.

2014-07-06 21:40:18,296 DEBUG stderr=DNS domain 'example.com' is not configured for automatic KDC address lookup.
KDC address will be set to fixed value.

Traceback (most recent call last):
  File "/usr/sbin/ipa-client-install", line 1292, in <module>
    sys.exit(main())
  File "/usr/sbin/ipa-client-install", line 1279, in main
    rval = install(options, env, fstore, statestore)
  File "/usr/sbin/ipa-client-install", line 1119, in install
    if configure_sssd_conf(fstore, cli_realm, cli_domain, cli_server, options):
  File "/usr/sbin/ipa-client-install", line 692, in configure_sssd_conf
    del sssdconfig
UnboundLocalError: local variable 'sssdconfig' referenced before assignment

It should be noted in this example test is run from:

ipa-server-install -r EXAMPLE.COM -p Secret123 -a Secret123 -U

So, no DNS and the realm differs from the DNS domain.

(In reply to Scott Poore from comment #14)
> Traceback (most recent call last):
>   File "/usr/sbin/ipa-client-install", line 1292, in <module>
>     sys.exit(main())
>   File "/usr/sbin/ipa-client-install", line 1279, in main
>     rval = install(options, env, fstore, statestore)
>   File "/usr/sbin/ipa-client-install", line 1119, in install
>     if configure_sssd_conf(fstore, cli_realm, cli_domain, cli_server,
> options):
>   File "/usr/sbin/ipa-client-install", line 692, in configure_sssd_conf
>     del sssdconfig
> UnboundLocalError: local variable 'sssdconfig' referenced before assignment

It doesn't seem likely that changes to certmonger would cause this.  Does it still happen if the version of the certmonger package is left unchanged during the test?

Yeah, I didn't think so but, thought I'd report it here since this is where I saw it and I wasn't sure if I was hitting something related here or something else.

I did test again (I hope) as suggested.  I installed on 6.2.  Updated ipa/certmonger components to rhel6.6 versions.  I uninstalled/downgraded as described earlier but, this time I left the newer versions of certmonger/nss/nspr:

certmonger-0.75.6-1.el6.x86_64
nss-3.16.1-8.el6.x86_64
nspr-4.10.6-1.el6.x86_64

I did see the same results where ipa-client-install of ipa-server-install (reinstall) failed.

So at this point, I guess I take that over to a different bug.

Do I have enough yet though to verify this bug as no longer occurring since I did not see the initially reported error when I tested in comment #14?

Thanks

Verified.

Version ::

certmonger.x86_64 0:0.75.6-1.el6 

Results ::

* Installed RHEL6.2 version of IPA and Certmonger.

* Upgraded to RHEL6.6 version of IPA and Certmonger.

* Uninstalled IPA server

[root@zippyvm4 bz948993]# ipa-server-install --uninstall -U
Shutting down all IPA services
Removing IPA client configuration
Unconfiguring ntpd
Unconfiguring CA directory server
Unconfiguring CA
Unconfiguring web server
Unconfiguring krb5kdc
Unconfiguring directory server
Unconfiguring ipa_memcached

[root@zippyvm4 bz948993]# yum -y remove 'ipa*' '389-ds-base*' bind krb5-workstation bind-dyndb-ldap krb5-pkinit-openssl httpd httpd-tools
...
Complete!

[root@zippyvm4 bz948993]# 

[root@zippyvm4 bz948993]# yum -y remove sssd libipa_hbac krb5-server slapi-nis sssd-client 'pki*' 'tomcat6*' mod_nss
...      

Complete!


[root@zippyvm4 bz948993]# service certmonger status
certmonger is stopped
[root@zippyvm4 bz948993]# grep id=dogtag-ipa-renew-agent /var/lib/certmonger/cas/*
/var/lib/certmonger/cas/20140707173619:id=dogtag-ipa-renew-agent
[root@zippyvm4 bz948993]# 
[root@zippyvm4 bz948993]# 
[root@zippyvm4 bz948993]# 
[root@zippyvm4 bz948993]# 
[root@zippyvm4 bz948993]# 
[root@zippyvm4 bz948993]# 
[root@zippyvm4 bz948993]# 
[root@zippyvm4 bz948993]# yum -y downgrade certmonger 
Loaded plugins: product-id, security, subscription-manager
Updating certificate-based repositories.
Setting up Downgrade Process
Resolving Dependencies
--> Running transaction check
---> Package certmonger.x86_64 0:0.50-3.el6 will be a downgrade
---> Package certmonger.x86_64 0:0.75.6-1.el6 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================================
 Package                 Arch                Version                  Repository                  Size
=======================================================================================================
Downgrading:
 certmonger              x86_64              0.50-3.el6               beaker-Server              203 k

Transaction Summary
=======================================================================================================
Downgrade     1 Package(s)

Total download size: 203 k
Downloading Packages:
certmonger-0.50-3.el6.x86_64.rpm                                                | 203 kB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
  Installing : certmonger-0.50-3.el6.x86_64                                                        1/2 
  Cleanup    : certmonger-0.75.6-1.el6.x86_64                                                      2/2 
Installed products updated.

Removed:
  certmonger.x86_64 0:0.75.6-1.el6                                                                     

Installed:
  certmonger.x86_64 0:0.50-3.el6                                                                       

Complete!

[root@zippyvm4 bz948993]# grep id=dogtag-ipa-renew-agent /var/lib/certmonger/cas/*
[root@zippyvm4 bz948993]# 

So, I can see that the patch removes the problem entry.


### Also with the addition of some new cleanup steps in uninstall phase, I was able to get a good re-install:

[root@zippyvm4 bz948993]# ipa-server-install -r EXAMPLE.COM -p Secret123 -a Secret123 -U

The log file for this installation can be found in /var/log/ipaserver-install.log
==============================================================================
This program will set up the IPA Server.

This includes:
  * Configure a stand-alone CA (dogtag) for certificate management
  * Configure the Network Time Daemon (ntpd)
  * Create and configure an instance of Directory Server
  * Create and configure a Kerberos Key Distribution Center (KDC)
  * Configure Apache (httpd)

To accept the default shown in brackets, press the Enter key.

The domain name has been calculated based on the host name.

The IPA Master Server will be configured with
Hostname:    zippyvm4.idm.lab.eng.rdu2.redhat.com
IP address:  10.8.60.63
Domain name: idm.lab.eng.rdu2.redhat.com

Configuring ntpd
  [1/4]: stopping ntpd
  [2/4]: writing configuration
  [3/4]: configuring ntpd to start on boot
  [4/4]: starting ntpd
done configuring ntpd.
Configuring directory server for the CA: Estimated time 30 seconds
  [1/3]: creating directory server user
  [2/3]: creating directory server instance
  [3/3]: restarting directory server
done configuring pkids.
Configuring certificate server: Estimated time 3 minutes 30 seconds
  [1/17]: creating certificate server user
  [2/17]: creating pki-ca instance
  [3/17]: configuring certificate server instance
  [4/17]: disabling nonces
  [5/17]: creating CA agent PKCS#12 file in /root
  [6/17]: creating RA agent certificate database
  [7/17]: importing CA chain to RA certificate database
  [8/17]: fixing RA database permissions
  [9/17]: setting up signing cert profile
  [10/17]: set up CRL publishing
  [11/17]: set certificate subject base
  [12/17]: configuring certificate server to start on boot
  [13/17]: restarting certificate server
  [14/17]: requesting RA certificate from CA
  [15/17]: issuing RA agent certificate
  [16/17]: adding RA agent as a trusted user
  [17/17]: Configure HTTP to proxy connections
done configuring pki-cad.
Configuring directory server: Estimated time 1 minute
  [1/35]: creating directory server user
  [2/35]: creating directory server instance
  [3/35]: adding default schema
  [4/35]: enabling memberof plugin
  [5/35]: enabling referential integrity plugin
  [6/35]: enabling winsync plugin
  [7/35]: configuring replication version plugin
  [8/35]: enabling IPA enrollment plugin
  [9/35]: enabling ldapi
  [10/35]: configuring uniqueness plugin
  [11/35]: configuring uuid plugin
  [12/35]: configuring modrdn plugin
  [13/35]: enabling entryUSN plugin
  [14/35]: configuring lockout plugin
  [15/35]: creating indices
  [16/35]: configuring ssl for ds instance
  [17/35]: configuring certmap.conf
  [18/35]: configure autobind for root
  [19/35]: configure new location for managed entries
  [20/35]: restarting directory server
  [21/35]: adding default layout
  [22/35]: adding delegation layout
  [23/35]: adding replication acis
  [24/35]: creating container for managed entries
  [25/35]: configuring user private groups
  [26/35]: configuring netgroups from hostgroups
  [27/35]: creating default Sudo bind user
  [28/35]: creating default Auto Member layout
  [29/35]: creating default HBAC rule allow_all
  [30/35]: initializing group membership
  [31/35]: adding master entry
  [32/35]: configuring Posix uid/gid generation
  [33/35]: enabling compatibility plugin
Restarting IPA to initialize updates before performing deletes:
  [1/2]: stopping directory server
  [2/2]: starting directory server
done configuring dirsrv.
  [34/35]: tuning directory server
  [35/35]: configuring directory to start on boot
done configuring dirsrv.
Configuring Kerberos KDC: Estimated time 30 seconds
  [1/14]: setting KDC account password
  [2/14]: adding sasl mappings to the directory
  [3/14]: adding kerberos entries to the DS
  [4/14]: adding default ACIs
  [5/14]: configuring KDC
  [6/14]: adding default keytypes
  [7/14]: adding default password policy
  [8/14]: creating a keytab for the directory
  [9/14]: creating a keytab for the machine
  [10/14]: exporting the kadmin keytab
  [11/14]: adding the password extension to the directory
  [12/14]: adding the kerberos master key to the directory
  [13/14]: starting the KDC
  [14/14]: configuring KDC to start on boot
done configuring krb5kdc.
Configuring ipa_kpasswd
  [1/2]: starting ipa_kpasswd 
  [2/2]: configuring ipa_kpasswd to start on boot
done configuring ipa_kpasswd.
Configuring the web interface: Estimated time 1 minute
  [1/13]: disabling mod_ssl in httpd
  [2/13]: setting mod_nss port to 443
  [3/13]: setting mod_nss password file
  [4/13]: enabling mod_nss renegotiate
  [5/13]: adding URL rewriting rules
  [6/13]: configuring httpd
  [7/13]: setting up ssl
  [8/13]: setting up browser autoconfig
  [9/13]: publish CA cert
  [10/13]: creating a keytab for httpd
  [11/13]: configuring SELinux for httpd
  [12/13]: restarting httpd
  [13/13]: configuring httpd to start on boot
done configuring httpd.
Applying LDAP updates
Restarting IPA to initialize updates before performing deletes:
  [1/2]: stopping directory server
  [2/2]: starting directory server
done configuring dirsrv.
Restarting the directory server
Restarting the KDC
Restarting the web server
Sample zone file for bind has been created in /tmp/sample.zone.cDoA2x.db
==============================================================================
Setup complete

Next steps:
	1. You must make sure these network ports are open:
		TCP Ports:
		  * 80, 443: HTTP/HTTPS
		  * 389, 636: LDAP/LDAPS
		  * 88, 464: kerberos
		UDP Ports:
		  * 88, 464: kerberos
		  * 123: ntp

	2. You can now obtain a kerberos ticket using the command: 'kinit admin'
	   This ticket will allow you to use the IPA tools (e.g., ipa user-add)
	   and the web user interface.

Be sure to back up the CA certificate stored in /root/cacert.p12
This file is required to create replicas. The password for this
file is the Directory Manager password
[root@zippyvm4 bz948993]# 

[root@zippyvm4 bz948993]# rpm -q ipa-server certmonger
ipa-server-2.1.3-9.el6.x86_64
certmonger-0.50-3.el6.x86_64
[root@zippyvm4 bz948993]# kinit admin
Password for admin: 
[root@zippyvm4 bz948993]# ipa user-find
--------------
1 user matched
--------------
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  UID: 1464800000
  GID: 1464800000
  Account disabled: False
  Keytab: True
  Password: True
----------------------------
Number of entries returned 1
----------------------------
[root@zippyvm4 bz948993]#

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1512.html