Created attachment 779088 [details]
/var/log/ipaupgrade.log

The log is a concatenation of the output produced during fedup and when running the work around.

FreeIPA upgrade should work even in the case when all IPA services are shut down (like in the fedup environment). So I think the key question here is why the 389-ds-base socket is not available in the fedup process.

Can we rule out the SELinux? I think the last time we investigated it together, you said that there were no related AVCs in /var/log/audit/audit.log after the upgrade.

In case you have chance to try running the upgrade again, would it may any change to update /etc/selinux/config and change SELINUX to permissive to confirm or rule out that SELinux is the cause?


(In reply to Dean Hunter from comment #0)
> But there remains a problem starting pki-tomcat.

What problem? I saw no example outputs in this bug or the previous one. Maybe the pki-ca is just starting too slow. In that case it may help to increase the time that ipactl waits for processes to start, e.g.:

echo "startup_timeout=240" >> /etc/ipa/default.conf

Please, I have no in depth knowledge of FreeIPA or Linux other than what I have learned stumbling around the last few months trying to make it work.  If you need me to do something to gather more information it would be helpful to me if you told me both what you need and how to get it.

1. Where in the "Steps to Reproduce" would you like me to to change to SELinux permissive mode?  Are you requesting:

   3. yum install fedup
      setenforce 0
   4. fedup -network 19 --instrepo http://host.hunter.org/repos/fedora19/iso
      setenforce 1
   5. reboot

2. Does this mean that SELinux alerts are not logged during fedup where they could be reported by ausearch?

3. Where would you expect to find more information about why pki-tomcat failed to start?

(In reply to Dean Hunter from comment #3)
> Please, I have no in depth knowledge of FreeIPA or Linux other than what I
> have learned stumbling around the last few months trying to make it work. 
> If you need me to do something to gather more information it would be
> helpful to me if you told me both what you need and how to get it.
> 
> 1. Where in the "Steps to Reproduce" would you like me to to change to
> SELinux permissive mode?  Are you requesting:
> 
>    3. yum install fedup
>       setenforce 0
>    4. fedup -network 19 --instrepo http://host.hunter.org/repos/fedora19/iso
>       setenforce 1
>    5. reboot

I think Martin is suggesting modifying /etc/selinux/config directly so it remains in permissive mode through reboots.
 
> 2. Does this mean that SELinux alerts are not logged during fedup where they
> could be reported by ausearch?

No. Permissive mode still logs failures, but allows the access that would have been denied in Enforcing mode.
 
> 3. Where would you expect to find more information about why pki-tomcat
> failed to start?

/var/log/messages, /var/log/pki/pki-tomcat/catalina.out, /var/log/pki/pki-tomcat/ca/debug are where I'd start

Thank you for the clarification.

1. In "Steps to Reproduce" I have added a step:

   3. yum install fedup
   +  sed -i s/SELINUX=enforcing/SELINUX=permissive/ /set/selinux/config
   4. fedup -network 19 --instrepo http://host.hunter.org/repos/fedora19/iso
   5. reboot

2. Are we changing the SELinux mode to permissive because the SELinux are not logged during the upgrade where they could be reported by ausearch?

We want to be sure it is in permissive the entire cycle of the upgrade.

It should have no impact on what is printed, other than it may print more since it won't be blocked by a single denial.

The results look the same to me.  I will attach all of the mentioned log files.

Created attachment 780119 [details]
upgrade logs

/var/log/ipaupgrade.log
/var/log/messages
/var/log/pki/pki-tomcat/catalina.out
/var/log/pki/pki-tomcat/ca/debug

Created attachment 780121 [details]
all avc messages

Thanks Dean(In reply to Dean Hunter from comment #3)
> Please, I have no in depth knowledge of FreeIPA or Linux other than what I
> have learned stumbling around the last few months trying to make it work. 
> If you need me to do something to gather more information it would be
> helpful to me if you told me both what you need and how to get it.

No problem at all, just ask as you just did in case you do not understand my instructions. So far, your feedback and testing was very valuable.

I finally found  the root cause of why the upgrade process does not work:

2013-07-29T19:26:22Z DEBUG   [4/8]: starting directory server
2013-07-29T19:26:22Z DEBUG Starting external process
2013-07-29T19:26:22Z DEBUG args=/bin/systemctl start dirsrv.target
2013-07-29T19:26:22Z DEBUG Process finished, return code=0
2013-07-29T19:26:22Z DEBUG stdout=
<<<
2013-07-29T19:26:22Z DEBUG stderr=Running in chroot, ignoring request.
>>>
2013-07-29T19:26:22Z DEBUG   duration: 0 seconds
2013-07-29T19:26:22Z DEBUG   [5/8]: upgrading server
2013-07-29T19:27:02Z ERROR Upgrade failed with [Errno 2] No such file or directory
2013-07-29T19:27:02Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py", line 115, in __upgrade
    ld = ldapupdate.LDAPUpdate(dm_password='', ldapi=True, live_run=self.live_run, plugins=True)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 174, in __init__
    conn.do_external_bind(self.pw_name)
  File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1734, in do_external_bind
    self.conn.sasl_interactive_bind_s, timeout, None, auth_tokens)
  File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1720, in __bind_with_wait
    self.__wait_for_connection(timeout)
  File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1706, in __wait_for_connection
    wait_for_open_socket(lurl.hostport, timeout)
  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 1115, in wait_for_open_socket
    raise e
error: [Errno 2] No such file or directory

It is indeed not SELinux related. The problem is that fedup runs in chroot-ed environment which does not allow dirsrv service to be started. Thus, the only workable workaround so far is the one you already acknowledged, i.e. running the upgrade processes after the Fedora 19 started.

Rob or Alexander, any idea how to mitigate this? I am thinking we may want to improve reporting of the upgrade process failure so that user is well notified about it after the fedup finished. We will also need to update the upgrade instructions in the release notes to run the update commands in case system is upgraded via fedup.

I have no work around.  Running the upgrade manually results in errors with pki-tomcat, remember?  Log files for pki-tomcat are attached

To get my network onto Fedora 19 I rebuilt IPA.  I am running these tests on a pair of isolated VMs to try to save the next person some grief.

Yeah, there are actually 2 issues:

1) Upgrade process does not work in fedup (explained in Comment 10)

2) PKI does not start. I found this error in PKI-CA debug log, looks related:

[29/Jul/2013:14:14:16][Thread-2]: Can't create master connection in LdapBoundConnFactory::getConn! Could not connect to LDAP server host ipa2.hunter.org port 636 Error netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)

When your dirsrv starts, is the port on? You can find out this way for example:

# netstat -pnl | grep  636

If it isn't, can you pleas check a value of "nsslapd-security" in /etc/dirsrv/slapd-HUNTER-ORG/dse.ldif?

I am sorry, but I do not understand "When your dirsrv starts, ....".  

Do you mean after performing the upgrade manually as in:

# /usr/sbin/ipa-ldap-updater --upgrade
# /usr/sbin/ipa-upgradeconfig
# netstat -pnl | grep  636

(In reply to Dean Hunter from comment #13)
> I am sorry, but I do not understand "When your dirsrv starts, ....".  
> 
> Do you mean after performing the upgrade manually as in:

Yes.

He wants to see if we are restoring some values in dse.ldif that affect what ports 389-ds listens to. During an upgrade we disable all network ports to ensure that things are quiet. It should always restore those values, even on catastrophic failure. Since tomcat is failing to start the thinking is that the secure port isn't being re-enabled.

I gather that dirsrv is actually starting up ok since the KDC and bind are starting.

[root@ipa2 ~]#   /usr/sbin/ipa-ldap-updater --upgrade
Upgrading IPA:
  [1/8]: stopping directory server
  [2/8]: saving configuration
  [3/8]: disabling listeners
  [4/8]: starting directory server
  [5/8]: upgrading server
PRE_UPDATE
Parsing update file '/usr/share/ipa/updates/10-60basev2.update'
...
Updating existing entry: cn=CAcert,cn=ipa,cn=etc,dc=hunter,dc=org
Done
  [6/8]: stopping directory server
  [7/8]: restoring configuration
  [8/8]: starting directory server
Done.
The ipa-ldap-updater command was successful

[root@ipa2 ~]#   /usr/sbin/ipa-upgradeconfig
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that CA proxy configuration is correct]
[Verifying that KDC configuration is using ipa-kdb backend]
[Fix DS schema file syntax]
[Removing self-signed CA]
Configuring ipa-otpd
  [1/2]: starting ipa-otpd 
  [2/2]: configuring ipa-otpd to start on boot
Done configuring ipa-otpd.
[Checking for deprecated KDC configuration files]
[Setting up Firefox extension]
/usr/share/ipa/html/krb.js exists, skipping install of Firefox extension
[Add missing CA DNS records]
[Enabling persistent search in DNS]
[Enabling serial autoincrement in DNS]
[Updating GSSAPI configuration in DNS]
[Updating pid-file configuration in DNS]
Changes to named.conf have been made, restart named
[Enable certificate renewal]
[Verifying that CA service certificate profile is updated]
[Certificate renewal should stop the CA]
Already configured to stop CA
The ipa-upgradeconfig command was successful

[root@ipa2 ~]#   ss -lnp | grep  636
tcp    LISTEN     0      128                   :::636                  :::*      users:(("ns-slapd",12396,7))

[root@ipa2 ~]#

[root@ipa2 ~]# ipactl status
Directory Service: RUNNING
ipa: INFO: The ipactl command was successful

[root@ipa2 ~]# ipactl stop
Stopping Directory Service
ipa: INFO: The ipactl command was successful

[root@ipa2 ~]# ipactl start
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting ipa_memcached Service
Starting httpd Service
Starting pki-tomcatd Service
Failed to start pki-tomcatd Service
Shutting down
Aborting ipactl

[root@ipa2 ~]# ipactl status
Directory Service: STOPPED
Directory Service must be running in order to obtain status of other services
ipa: INFO: The ipactl command was successful

[root@ipa2 ~]#

Created attachment 780792 [details]
excerpt from /var/log/messages during ipactl start

[root@ipa2 ~]# cat /var/log/messages
...
Jul 30 13:14:22 ipa2 systemd[1]: Starting 389 Directory Server.
Jul 30 13:14:22 ipa2 systemd[1]: Reached target 389 Directory Server.
Jul 30 13:14:22 ipa2 systemd[1]: Starting 389 Directory Server HUNTER-ORG....
Jul 30 13:14:22 ipa2 systemd[1]: Ignoring invalid environment 'export KRB5_KTNAME=/etc/dirsrv/ds.keytab': /etc/sysconfig/dirsrv
Jul 30 13:14:22 ipa2 systemd[1]: Starting 389 Directory Server PKI-IPA....
Jul 30 13:14:22 ipa2 systemd[1]: Ignoring invalid environment 'export KRB5_KTNAME=/etc/dirsrv/ds.keytab': /etc/sysconfig/dirsrv
Jul 30 13:14:22 ipa2 systemd[1]: Failed to load environment files: No such file or directory
Jul 30 13:14:22 ipa2 systemd[1]: dirsrv failed to run 'start' task: No such file or directory
Jul 30 13:14:22 ipa2 systemd[1]: Failed to start 389 Directory Server PKI-IPA..
....

[root@ipa2 ~]# cat /etc/sysconfig/dirsrv
# This file is sourced by dirsrv upon startup to set
# the default environment for all directory server instances.
# To set instance specific defaults, use the file in the same
...
# if using systemd, omit the "; export VARNAME" at the end
#PID_TIME=600 ; export PID_TIME
ulimit -n 8192
KRB5_KTNAME=/etc/dirsrv/ds.keytab
export KRB5_KTNAME=/etc/dirsrv/ds.keytab
KRB5CCNAME=/tmp/krb5cc_990

[root@ipa2 ~]# ls -alZ /etc/dirsrv
drwxrwxr-x. root   dirsrv system_u:object_r:dirsrv_config_t:s0 .
drwxr-xr-x. root   root   system_u:object_r:etc_t:s0       ..
drwxr-xr-x. root   root   system_u:object_r:dirsrv_config_t:s0 config
-rw-------. dirsrv dirsrv unconfined_u:object_r:dirsrv_config_t:s0 ds.keytab
drwxr-xr-x. root   root   system_u:object_r:dirsrv_config_t:s0 schema
drwxrwx---. dirsrv dirsrv unconfined_u:object_r:dirsrv_config_t:s0 slapd-HUNTER-ORG

[root@ipa2 ~]#

[root@ipa2 ~]# cat /var/log/messages
...
Jul 30 13:14:26 ipa2 systemd[1]: Starting PKI Tomcat Server.
Jul 30 13:14:26 ipa2 systemd[1]: Reached target PKI Tomcat Server.
Jul 30 13:14:26 ipa2 systemd[1]: Starting PKI Tomcat Server pki-tomcat...
Jul 30 13:14:26 ipa2 pkidaemon[13113]: WARNING:  Symbolic link '/var/lib/pki/pki-tomcat/common/lib/jss4.jar' does NOT exist!
Jul 30 13:14:26 ipa2 pkidaemon[13113]: INFO:  Attempting to create '/var/lib/pki/pki-tomcat/common/lib/jss4.jar' -> '/usr/lib/java/jss4.jar' . . .
Jul 30 13:14:26 ipa2 pkidaemon[13113]: ln: failed to create symbolic link ‘/var/lib/pki/pki-tomcat/common/lib/jss4.jar’: File exists
Jul 30 13:14:26 ipa2 pkidaemon[13113]: ERROR:  Failed to create '/var/lib/pki/pki-tomcat/common/lib/jss4.jar' -> '/usr/lib/java/jss4.jar'!
Jul 30 13:14:26 ipa2 pkidaemon[13113]: WARNING:  Symbolic link '/var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar' does NOT exist!
Jul 30 13:14:26 ipa2 pkidaemon[13113]: INFO:  Attempting to create '/var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar' -> '/usr/share/java/tomcatjss.jar' . . .
Jul 30 13:14:26 ipa2 pkidaemon[13113]: ln: failed to create symbolic link ‘/var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar’: File exists
Jul 30 13:14:26 ipa2 pkidaemon[13113]: ERROR:  Failed to create '/var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar' -> '/usr/share/java/tomcatjss.jar'!
Jul 30 13:14:27 ipa2 systemd[1]: pki-tomcatd: control process exited, code=exited status=1
Jul 30 13:14:27 ipa2 systemd[1]: Failed to start PKI Tomcat Server pki-tomcat.
Jul 30 13:14:27 ipa2 systemd[1]: Unit pki-tomcatd entered failed state.
....

[root@ipa2 ~]# find /usr /var -name jss4.jar -exec ls -lZ {} \;
lrwxrwxrwx. root root system_u:object_r:lib_t:s0       /usr/lib64/jss/jss4.jar -> /usr/lib/java/jss4.jar
-rw-r--r--. root root system_u:object_r:lib_t:s0       /usr/lib/java/jss4.jar
lrwxrwxrwx. pkiuser pkiuser system_u:object_r:pki_tomcat_var_lib_t:s0 /var/lib/pki/pki-tomcat/common/lib/jss4.jar -> /usr/lib64/java/jss4.jar

[root@ipa2 ~]# find /usr /var -name jss.jar -exec ls -lZ {} \;

[root@ipa2 ~]#

(In reply to Dean Hunter from comment #18)

The dirsrv systemd errors are being tracked in https://bugzilla.redhat.com/show_bug.cgi?id=988858

They are benign.

(In reply to Dean Hunter from comment #19)
> [root@ipa2 ~]# cat /var/log/messages
> ...
> Jul 30 13:14:26 ipa2 systemd[1]: Starting PKI Tomcat Server.
> Jul 30 13:14:26 ipa2 systemd[1]: Reached target PKI Tomcat Server.
> Jul 30 13:14:26 ipa2 systemd[1]: Starting PKI Tomcat Server pki-tomcat...
> Jul 30 13:14:26 ipa2 pkidaemon[13113]: WARNING:  Symbolic link
> '/var/lib/pki/pki-tomcat/common/lib/jss4.jar' does NOT exist!
> Jul 30 13:14:26 ipa2 pkidaemon[13113]: INFO:  Attempting to create
> '/var/lib/pki/pki-tomcat/common/lib/jss4.jar' -> '/usr/lib/java/jss4.jar' .
> . .
> Jul 30 13:14:26 ipa2 pkidaemon[13113]: ln: failed to create symbolic link
> ‘/var/lib/pki/pki-tomcat/common/lib/jss4.jar’: File exists

Ade, do you know why the links are failing?

Dean, what version of tomcat, tomcatjss and jss do you have installed?

(In reply to Rob Crittenden from comment #20)
> (In reply to Dean Hunter from comment #18)
> 
> The dirsrv systemd errors are being tracked in
> https://bugzilla.redhat.com/show_bug.cgi?id=988858
> 
> They are benign.

Comparing my new install IPA 3.2.2-1 with my upgrade IPA 3.2.2-1:

/etc/sysconfig/dirsrv
new:
KRB5CCNAME=/tmp/krb5cc_990
ulimit -n 8192
KRB5_KTNAME=/etc/dirsrv/ds.keytab

upgrade:
ulimit -n 8192
KRB5_KTNAME=/etc/dirsrv/ds.keytab
export KRB5_KTNAME=/etc/dirsrv/ds.keytab
KRB5CCNAME=/tmp/krb5cc_990

When I change the upgrade statements to be the same as the new statements I still get:

Jul 30 14:29:42 ipa2 systemd[1]: Failed to load environment files: No such file or directory
Jul 30 14:29:42 ipa2 systemd[1]: dirsrv failed to run 'start' task: No such file or directory
Jul 30 14:29:42 ipa2 systemd[1]: Failed to start 389 Directory Server PKI-IPA..

Are you saying that dirsrv is not required?

(In reply to Rob Crittenden from comment #21)

> Dean, what version of tomcat, tomcatjss and jss do you have installed?

[root@ipa2 ~]# rpm -q tomcat tomcatjss jss
tomcat-7.0.40-2.fc19.noarch
tomcatjss-7.1.0-3.fc19.noarch
jss-4.2.6-30.fc19.x86_64

[root@ipa2 ~]#

I think you need to check this out, http://pki.fedoraproject.org/wiki/Migrating_Dogtag_9_Instances_to_Dogtag_10

1. Is there an error in the first paragraph of the wiki aricle you referenced?  

Overview

Dogtag 9 instances are configured as tomcat 6 instances. Unfortunately, tomcat 6 is no longer supported on Fedora 19. To ensure that users do not inadvertently upgrade to Fedora 18 and render their Dogatg 9 instances inoperable, code has been added to check for the existence of Dogtag 9 instances prior to upgrade and to cause the upgrade to abort if those instances exist. 

Should it be "upgrade from Fedora 18" or "upgrade to Fedora 19" instead of "upgrade to Fedora 18"?

2. Please confirm that the dogtag 9 to 10 upgrade should be performed on the Fedora 18 instance before the Fedora 18 to 19 upgrade?

1. upgrade to Fedora 19 - thanks for catching
2. Yes

(In reply to Rob Crittenden from comment #24)
> I think you need to check this out,
> http://pki.fedoraproject.org/wiki/Migrating_Dogtag_9_Instances_to_Dogtag_10

This is FreeIPA specialized page:
http://www.freeipa.org/page/Howto/Dogtag9ToDogtag10Migration

(In reply to Dean Hunter from comment #25)
> ...
> 2. Please confirm that the dogtag 9 to 10 upgrade should be performed on the
> Fedora 18 instance before the Fedora 18 to 19 upgrade?

This is correct. Just please note, that we cannot do just a simple upgrade (in terms of yum/package update), but a manual migration procedure needs to be followed to migrate your old FreeIPA masters with CA to Fedora 19. The procedure is described on the FreeIPA.org wiki.

If you are using the installed version of tomcat as the indicator then here are the installed package versions before the Fedora 18 to 19 upgrade:

[root@ipa2 ~]# rpm -q freeipa-server tomcat tomcatjss jss
freeipa-server-3.1.5-1.fc18.x86_64
tomcat-7.0.40-1.fc18.noarch
tomcatjss-7.0.0-5.fc18.noarch
jss-4.2.6-28.fc18.x86_64

[root@ipa2 ~]# 

Steps to Reproduce:

1. Build a new Fedora 18 VM
2. Install and configure freeipa-server-3.1.5-1.fc18.x86_64
3. yum install fedup
4. fedup -network 19 --instrepo http://host.hunter.org/repos/fedora19/iso
5. reboot
6. ipactl status

(In reply to Dean Hunter from comment #28)
...
> Steps to Reproduce:
> 
> 1. Build a new Fedora 18 VM
> 2. Install and configure freeipa-server-3.1.5-1.fc18.x86_64
> 3. yum install fedup
> 4. fedup -network 19 --instrepo http://host.hunter.org/repos/fedora19/iso
> 5. reboot
> 6. ipactl status

BTW if you installed FreeIPA CA master in F18, you do not have to undertake the migration procedure described here:
http://www.freeipa.org/page/Howto/Dogtag9ToDogtag10Migration
... as you already use Dogtag10-based CA.

So I think that your pki-ca problem may be a different issue - Ade, what do you think about errors in Comment 19?

I fixed the wiki entry.  Thanks for the catch.

I am totally befuddled by the messages though.  The startup code (in /usr/share/pki/scripts/operations does the following:

if [ -e ${symlink} ]; then
   do stuff ..
else 
   echo "WARNING:  Symbolic link '${symlink}' does NOT exist!"
   # Attempt to create the symbolic link and chown it.
   make_symlink ${symlink} ${target} ${user} ${group}
fi


make_symlink()
{
    symlink="${1}"
    target="${2}"
    user="${3}"
    group="${4}"

    rv=0

    echo "INFO:  Attempting to create '${symlink}' -> '${target}' . . ."
    # Check to make certain that the expected target exists.
    #
    #     NOTE:  The symbolic link does NOT exist at this point.
    #
    if [ -e ${target} ]; then
        # Check that the expected target is fully resolvable!
        if [ ! `readlink -qe ${target}` ]; then
            # Issue an ERROR that the target to which the
            # symbolic link is expected to point is NOT fully resolvable!
            echo "ERROR:  Failed making '${symlink}' -> '${target}'"\
                 "since target '${target}' is NOT fully resolvable!"
            rv=1
        else
            # Attempt to create a symbolic link and 'chown' it.
            ln -s ${target} ${symlink}
            rv=$?
            if [ $rv -eq 0 ]; then
                # NOTE:  Ignore 'chown' errors.
                chown -h ${user}:${group} ${symlink}
                echo "SUCCESS:  Created '${symlink}' -> '${target}'"
            else
                echo "ERROR:  Failed to create '${symlink}' -> '${target}'!"
                rv=1
            fi
        fi
    else
        # Issue an ERROR that the target to which the
        # symbolic link is expected to point does NOT exist.
        echo "ERROR:  Failed making '${symlink}' -> '${target}'"\
             "since target '${target}' does NOT exist!"
        rv=1
    fi

    return $rv
}

So, its only trying to create the symlink if the symlink does not exist - and if the target exists, but then the ln -s is failing to create the link - and is reporting that it does so because the file exists!

Do the links -

/var/lib/pki/pki-tomcat/common/lib/jss4.jar -> /usr/lib/java/jss4.jar 
/var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar -> /usr/share/java/tomcatjss.jar

exist? (Along with their targets?)

I'll have to think more on how this could happen.  As a workaround, you can try to create them manually, and then try restarting the CA.

ln -s /usr/lib/java/jss4.jar /var/lib/pki/pki-tomcat/common/lib/jss4.jar
ln -s /usr/share/java/tomcatjss.jar /var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar
systemctl restart pki-tomcatd

Does this answer your question about links to jss4.jar?

From Comment 19:

[root@ipa2 ~]# find /usr /var -name jss4.jar -exec ls -lZ {} \;
lrwxrwxrwx. root root system_u:object_r:lib_t:s0       /usr/lib64/jss/jss4.jar -> /usr/lib/java/jss4.jar
-rw-r--r--. root root system_u:object_r:lib_t:s0       /usr/lib/java/jss4.jar
lrwxrwxrwx. pkiuser pkiuser system_u:object_r:pki_tomcat_var_lib_t:s0 /var/lib/pki/pki-tomcat/common/lib/jss4.jar -> /usr/lib64/java/jss4.jar

[root@ipa2 ~]#

This is what the links looked like before the upgrade:

[root@ipa2 ~]# find /usr /var -name jss4.jar -exec ls -lZ {} \;
lrwxrwxrwx. root root system_u:object_r:lib_t:s0       /usr/lib64/jss/jss4.jar -> /usr/lib64/java/jss4.jar
-rw-r--r--. root root system_u:object_r:lib_t:s0       /usr/lib64/java/jss4.jar
lrwxrwxrwx. pkiuser pkiuser system_u:object_r:pki_tomcat_var_lib_t:s0 /var/lib/pki/pki-tomcat/common/lib/jss4.jar -> /usr/lib64/java/jss4.jar

[root@ipa2 ~]#

Actually, I think we found a case where the code above will fail.

The case is if the link exists and is pointing to the wrong location.  In that case, the -e test will fail - and then ln -s will fail because the link already exists.

That clearly is the case for the jss jar.  In F19, the JNI directory was moved as indicated.

The same is not true though for tomcatjss.  Can you tell us what the location of the tomcatjss link was?

Here are the links for tomcatjss.jar before the upgrade:

[root@ipa2 ~]# find /usr /var -name tomcatjss.jar -exec ls -lZ {} \;
lrwxrwxrwx. root root system_u:object_r:usr_t:s0       /usr/share/java/tomcatjss.jar -> tomcatjss-6.0.99.jar
lrwxrwxrwx. pkiuser pkiuser system_u:object_r:pki_tomcat_var_lib_t:s0 /var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar -> /usr/share/java/tomcat7jss.jar

[root@ipa2 ~]# 

It will take me a little more time to get these links after the upgrade.

Here are the links for tomcat.jss after the upgrade:

[root@ipa2 ~]# find /usr /var -name tomcatjss.jar -exec ls -lZ {} \;
lrwxrwxrwx. root root system_u:object_r:usr_t:s0       /usr/share/java/tomcatjss.jar -> tomcatjss-7.1.0.jar
lrwxrwxrwx. pkiuser pkiuser system_u:object_r:pki_tomcat_var_lib_t:s0 /var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar -> /usr/share/java/tomcat7jss.jar

[root@ipa2 ~]#

Please verify this summary of what we have found thus far as I find myself getting confused.

There is an error in the IPA upgrade process performed by fedup while upgrading from Fedora 18 to Fedora 19.

1. The directory server can not started or stopped during fedup.  This is an architectural issue between the use of chroot in fedup and the need to control services during the IPA upgrade.  This problem may be seen at the following locations in ipaupgrade.log.

1.1. During ipa-ldap-updater

1.1.1. Line 69: Running in chroot during /bin/systemctl stop dirsrv

1.1.2. Line 83: Running in chroot during /bin/systemctl start dirsrv.target
       Line 88: Errno 2: No such file or directory while upgrading server

1.1.3. Line 109: Running in chroot during /bin/systemctl stop dirsrv

1.1.4. Line 127: Running in chroot during /bin/systemctl is-active dirsrv
       Line 129: wait_for_open_ports: localhost [389] timeout 120

1.2. During ipa-upgradeconfig

1.2.1. Line 279: Running in chroot during /bin/systemctl stop dirsrv

1.2.2. Line 287: Running in chroot during /bin/systemctl start dirsrv

1.2.3. Line 293: Running in chroot during /bin/systemctl is-active dirsrv
       Line 295: wait_for_open_ports: localhost [389] timeout 120

It is possible to run ipa-ldap-updater and ipa-upgradeconfig successfully after booting into Fedora 19.  However there are two remaining problems when trying to start IPA.

2. The systemd environment is not configured correctly for dirsrv.

2.1. In /etc/sysconfig/dirsrv the following line should be removed:

     ulimit -n 8192

     This line is ignored without any messages; see bug 988858

2.2. In /etc/sysconfig/dirsrv the following line should be removed:

     export KRB5_KTNAME=/etc/dirsrv/ds.keytab

     This line is ignored and a message is logged.

2.3. There is an environment file missing that causes the start task to fail.  A message is logged without naming the missing file or directory.

3. Two symbolic links are not configured correctly for pki-tomcat.

3.1. /var/lib/pki/pki-tomcat/common/lib/jss4.jar -> /usr/lib/java/jss4.jar
     It points to /usr/lib64/java/jss4.jar instead and it does not exist.

3.2. /var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar -> /usr/share/java/tomcatjss.jar
     It points to /usr/share/java/tomcat7jss.jar instead and it does not exist.

(In reply to Ade Lee from comment #33)
> Actually, I think we found a case where the code above will fail.
> 
> The case is if the link exists and is pointing to the wrong location.  In
> that case, the -e test will fail - and then ln -s will fail because the link
> already exists.
> 
> That clearly is the case for the jss jar.  In F19, the JNI directory was
> moved as indicated.
> 
> The same is not true though for tomcatjss.  Can you tell us what the
> location of the tomcatjss link was?

Per Dean Hunter's info, the tomcatjss link would have been pointing to '/usr/share/java/tomcat7jss.jar' on Fedora 18, and once the tomcatjss was updated to the version on Fedora 19, it would have left the '/var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar' still pointing to this non-existent file.  Consequently, the -e test would have failed exactly the same as in the jss case.

I believe that I have a potential solution - always attempt to remove a symbolic link prior to creating it (simply ignore the return value because it is a benign error if the symbolic link does not exist).

Per the previous comment, I would suggest adding the following in-line patch to the code from comment #30:

(In reply to Ade Lee from comment #30)
> I fixed the wiki entry.  Thanks for the catch.
> 
> I am totally befuddled by the messages though.  The startup code (in
> /usr/share/pki/scripts/operations does the following:
> 
> if [ -e ${symlink} ]; then
>    do stuff ..
> else 
>    echo "WARNING:  Symbolic link '${symlink}' does NOT exist!"
>    # Attempt to create the symbolic link and chown it.
>    make_symlink ${symlink} ${target} ${user} ${group}
> fi
> 
> 
> make_symlink()
> {
>     symlink="${1}"
>     target="${2}"
>     user="${3}"
>     group="${4}"
> 
>     rv=0
> 
>     echo "INFO:  Attempting to create '${symlink}' -> '${target}' . . ."
>     # Check to make certain that the expected target exists.
>     #
>     #     NOTE:  The symbolic link does NOT exist at this point.
>     #
>     if [ -e ${target} ]; then
>         # Check that the expected target is fully resolvable!
>         if [ ! `readlink -qe ${target}` ]; then
>             # Issue an ERROR that the target to which the
>             # symbolic link is expected to point is NOT fully resolvable!
>             echo "ERROR:  Failed making '${symlink}' -> '${target}'"\
>                  "since target '${target}' is NOT fully resolvable!"
>             rv=1
>         else
              # Always attempt to remove any existing symbolic link that may
              # already exist and be pointing to a non-existent target
              # (ignore any return value as the symlink will usually not exist)
              rm ${symlink}
>             # Attempt to create a symbolic link and 'chown' it.
>             ln -s ${target} ${symlink}
>             rv=$?
>             if [ $rv -eq 0 ]; then
>                 # NOTE:  Ignore 'chown' errors.
>                 chown -h ${user}:${group} ${symlink}
>                 echo "SUCCESS:  Created '${symlink}' -> '${target}'"
>             else
>                 echo "ERROR:  Failed to create '${symlink}' -> '${target}'!"
>                 rv=1
>             fi
>         fi
>     else
>         # Issue an ERROR that the target to which the
>         # symbolic link is expected to point does NOT exist.
>         echo "ERROR:  Failed making '${symlink}' -> '${target}'"\
>              "since target '${target}' does NOT exist!"
>         rv=1
>     fi
> 
>     return $rv
> }
> 
> So, its only trying to create the symlink if the symlink does not exist -
> and if the target exists, but then the ln -s is failing to create the link -
> and is reporting that it does so because the file exists!
> 
> Do the links -
> 
> /var/lib/pki/pki-tomcat/common/lib/jss4.jar -> /usr/lib/java/jss4.jar 
> /var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar ->
> /usr/share/java/tomcatjss.jar
> 
> exist? (Along with their targets?)
> 
> I'll have to think more on how this could happen.  As a workaround, you can
> try to create them manually, and then try restarting the CA.
> 
> ln -s /usr/lib/java/jss4.jar /var/lib/pki/pki-tomcat/common/lib/jss4.jar
> ln -s /usr/share/java/tomcatjss.jar
> /var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar
> systemctl restart pki-tomcatd

(In reply to Matthew Harmsen from comment #38)
> Per the previous comment, I would suggest adding the following in-line patch
> to the code from comment #30:

Ok, I will clone the bug to pki-ca so that you can track the proposed fix there.

(In reply to Dean Hunter from comment #36)
> Please verify this summary of what we have found thus far as I find myself
> getting confused.
> 
> There is an error in the IPA upgrade process performed by fedup while
> upgrading from Fedora 18 to Fedora 19.
> 
> 1. The directory server can not started or stopped during fedup.  This is an
> architectural issue between the use of chroot in fedup and the need to
> control services during the IPA upgrade.  This problem may be seen at the
> following locations in ipaupgrade.log.
> 
> 1.1. During ipa-ldap-updater
> 
> 1.1.1. Line 69: Running in chroot during /bin/systemctl stop
> dirsrv
> 
> 1.1.2. Line 83: Running in chroot during /bin/systemctl start dirsrv.target
>        Line 88: Errno 2: No such file or directory while upgrading server
> 
> 1.1.3. Line 109: Running in chroot during /bin/systemctl stop
> dirsrv
> 
> 1.1.4. Line 127: Running in chroot during /bin/systemctl is-active
> dirsrv
>        Line 129: wait_for_open_ports: localhost [389] timeout 120
> 
> 1.2. During ipa-upgradeconfig
> 
> 1.2.1. Line 279: Running in chroot during /bin/systemctl stop
> dirsrv
> 
> 1.2.2. Line 287: Running in chroot during /bin/systemctl start
> dirsrv
> 
> 1.2.3. Line 293: Running in chroot during /bin/systemctl is-active
> dirsrv
>        Line 295: wait_for_open_ports: localhost [389] timeout 120
> 
> It is possible to run ipa-ldap-updater and ipa-upgradeconfig successfully
> after booting into Fedora 19.  However there are two remaining problems when
> trying to start IPA.

Yes, this is the error that this Bugzilla should fix. Either by improving upgrade instructions in documentation or some other checks in FreeIPA - I will create upstream ticket for that. But until this is fixed, the proposed workaround should be enough to fix the upgrade issue from FreeIPA point of view.

> 
> 2. The systemd environment is not configured correctly for
> dirsrv.

This particular service is quite suspicious actually (and benign). This dirsrv instance seems to be a left-over from previous FreeIPA installation when CA run on a separate dirsrv instance. This should not happen with clean FreeIPA installation in F18 or F19.

> 
> 2.1. In /etc/sysconfig/dirsrv the following line should be removed:
> 
>      ulimit -n 8192
> 
>      This line is ignored without any messages; see bug 988858
> 
> 2.2. In /etc/sysconfig/dirsrv the following line should be removed:
> 
>      export KRB5_KTNAME=/etc/dirsrv/ds.keytab
> 
>      This line is ignored and a message is logged.

Both issues will be fixed in scope of FreeIPA 3.3.1. Upstream ticket:
https://fedorahosted.org/freeipa/ticket/3823

> 
> 2.3. There is an environment file missing that causes the start task to
> fail.  A message is logged without naming the missing file or directory.
> 
> 3. Two symbolic links are not configured correctly for
> pki-tomcat.
> 
> 3.1. /var/lib/pki/pki-tomcat/common/lib/jss4.jar -> /usr/lib/java/jss4.jar
>      It points to /usr/lib64/java/jss4.jar instead and it does not exist.
> 
> 3.2. /var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar ->
> /usr/share/java/tomcatjss.jar
>      It points to /usr/share/java/tomcat7jss.jar instead and it does not
> exist.

Should be fixed by Bug 990892.

BTW, thanks for the nice summary Dean! Your patience and thorough testing helps us avoid these obstacles for other FreeIPA users.

(In reply to Martin Kosek from comment #40)
> (In reply to Dean Hunter from comment #36)
> > 
> > 2. The systemd environment is not configured correctly for
> > dirsrv.
> 
> This particular service is quite suspicious actually (and benign). This
> dirsrv instance seems to be a left-over from previous FreeIPA installation
> when CA run on a separate dirsrv instance. This should not happen with clean
> FreeIPA installation in F18 or F19.
> 
> > 
> > 2.1. In /etc/sysconfig/dirsrv the following line should be removed:
> > 
> >      ulimit -n 8192
> > 
> >      This line is ignored without any messages; see bug 988858
> > 
> > 2.2. In /etc/sysconfig/dirsrv the following line should be removed:
> > 
> >      export KRB5_KTNAME=/etc/dirsrv/ds.keytab
> > 
> >      This line is ignored and a message is logged.
> 
> Both issues will be fixed in scope of FreeIPA 3.3.1. Upstream ticket:
> https://fedorahosted.org/freeipa/ticket/3823
> 
> > 
> > 2.3. There is an environment file missing that causes the start task to
> > fail.  A message is logged without naming the missing file or directory.
> > 

I am sorry but when I examine new builds of Fedora 18 with IPA 3.1.5-1 and Fedora 19 with IPA 3.2.2-1 I find dirsrv.  If the service should not be installed then the environment file errors do not need to be corrected, instead the service should be removed.  So your comments leave me confused how to proceed to clear this error:

Jul 30 14:29:42 ipa2 systemd[1]: Failed to load environment files: No such file or directory
Jul 30 14:29:42 ipa2 systemd[1]: dirsrv failed to run 'start' task: No such file or directory
Jul 30 14:29:42 ipa2 systemd[1]: Failed to start 389 Directory Server PKI-IPA..

Ah, I see now that there are two services incorrectly configured.

2. The systemd environment is not configured correctly for two services.

2.1. For dirsrv.service

2.1.1. In /etc/sysconfig/dirsrv the following line should be removed:

       ulimit -n 8192

       This line is ignored without any messages; see bug 988858

2.1.2. In /etc/sysconfig/dirsrv the following line should be removed:

       export KRB5_KTNAME=/etc/dirsrv/ds.keytab

       This line is ignored and a message is logged.

2.2. For dirsrv

2.2.1. There is an environment file missing that causes the start task to fail.  A message is logged without naming the missing file or directory.

2.2.2. dirsrv should not be starting.  After the upgrade process a wanted-by link to this service exists that was not present before the upgrade.

fedup from Fedora 18 with IPA 3.1.5-1 to Fedora 19 with IPA 3.2.2-1 adds:

  /etc/systemd/system/dirsrv.target.wants/dirsrv
    -> /lib/systemd/system/dirsrv@.service

This link is not present on a new build of Fedora 18 with IPA 3.1.5-1 or Fedora 19 with IPA 3.2.2-1.

Just to confirm, you don't have a /etc/dirsrv/slapd-PKI-IPA right?

I have the feeling this is the fault of freeipa-systemd-upgrade. It unconditionally enables the PKI-IPA dirsrv instance.

[root@ipa2 ~]# find -L /etc -name slapd*
/etc/dirsrv/config/slapd-collations.conf
/etc/dirsrv/slapd-HUNTER-ORG
/etc/dirsrv/slapd-HUNTER-ORG/slapd-collations.conf

[root@ipa2 ~]#

Here is a sequence of commands that will upgrade a new built Fedora 18 with IPA 3.1.5-1 to Fedora 19 with 3.2.2-1 and correct the problems identified to date:

# Upgrade using the local Fedora 19 repositories

  yum --assumeyes install fedup

  fedup --network 19 \
    --instrepo http://host.hunter.org/isos/fedora19/DVD

  reboot

# Work around FreeIPA upgrade problems

  sed -i /^ulimit/d /etc/sysconfig/dirsrv
  sed -i /^export/d /etc/sysconfig/dirsrv
  cat               /etc/sysconfig/dirsrv
    
  rm -f /etc/systemd/system/dirsrv.target.wants/dirsrv
  find -L /etc /usr -name dirsrv* -type f | sort
  
  /usr/sbin/ipa-ldap-updater --upgrade
  /usr/sbin/ipa-upgradeconfig

  rm -f /var/lib/pki/pki-tomcat/common/lib/jss4.jar
  find -L /var /usr -name jss4.jar | sort

  rm -f /var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar
  find -L /var /usr -name tomcatjss.jar | sort

# Verify IPA will stop and start

  ipactl start
  ipactl stop

Trac ticket 699 (https://fedorahosted.org/pki/ticket/699) has been created to track this issue for Dogtag.

pki-core-10.0.4-2 has been created to address this issue.  Please test it out and provide karma to help move the  build to the stable repo.

With this update, the following steps:

 rm -f /var/lib/pki/pki-tomcat/common/lib/jss4.jar
  find -L /var /usr -name jss4.jar | sort

  rm -f /var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar
  find -L /var /usr -name tomcatjss.jar | sort

are no longer required.  The links should fix themselves upon restarting the dogtag instance.

(In reply to Ade Lee from comment #47)
> Trac ticket 699 (https://fedorahosted.org/pki/ticket/699) has been created
> to track this issue for Dogtag.
> 
> pki-core-10.0.4-2 has been created to address this issue.  Please test it
> out and provide karma to help move the  build to the stable repo.

Where is the sequence of commands from comment 46 should I update to 
pki-core-10.0.4-2?

1. Should I use pki-core-10.0.4-2.fc18 before fedup?

2. Should I use pki-core-10.0.4-2.fc19 after the reboot and before /usr/sbin/ipa-ldap-updater?

3. I guess I could even do it once each way?

Ade may specify this more, but IMO doing just 2) is enough. However, you can do both 1) and 2), just to be sure.

Cloning the Bugzilla to FreeIPA upstrem Trac to solve the fedup upgrade issue.

We discussed this issue on our team meeting and we think that the best way would be to:

1. When RPM upgrade succeeds, it writes the upstream version ID to system (/var/lib/ipa/sysupgrade/sysupgrade.state). When it fails, version is not written
2. Next time, when ipa service starts and detects that upgrade was not performed (comparing it's version with stored one), it would execute the upgrade before IPA starts

Upstream ticket:
https://fedorahosted.org/freeipa/ticket/3849

# cd /srv/http/repos/fedora$v/updates-testing
# koji download-build --arch x86_64 pki-core-10.0.4-2.fc$v
# cd -
# createrepo /srv/http/repos/fedora$v/updates-testing

retrieves two packaes:

  pki-symkey-10.0.4-2.fc18.x86_64.rpm
  pki-tools-10.0.4-2.fc18.x86_64.rpm

When I try to update these packages then a number of dependency errors are displayed:

# yum --assumeyes --enablerepo local-testing pki-symkey pki-tools
...
--> Finished Dependency Resolution
Error: Package: pki-tools-10.0.4-2.fc18.x86_64 (local-testing)
           Requires: pki-base = 10.0.4-2.fc18
           Installed: pki-base-10.0.3-2.fc18.noarch (@local-updates)
               pki-base = 10.0.3-2.fc18
           Available: pki-base-10.0.0-1.fc18.noarch (local-fedora)
               pki-base = 10.0.0-1.fc18
           Available: pki-base-10.0.4-1.fc18.noarch (local-testing)
               pki-base = 10.0.4-1.fc18
Error: Package: pki-server-10.0.4-1.fc18.noarch (local-testing)
           Requires: pki-tools = 10.0.4-1.fc18
           Removing: pki-tools-10.0.3-2.fc18.x86_64 (@local-updates)
               pki-tools = 10.0.3-2.fc18
           Updated By: pki-tools-10.0.4-2.fc18.x86_64 (local-testing)
               pki-tools = 10.0.4-2.fc18
           Available: pki-tools-10.0.0-1.fc18.x86_64 (local-fedora)
               pki-tools = 10.0.0-1.fc18
           Available: pki-tools-10.0.4-1.fc18.x86_64 (local-testing)
               pki-tools = 10.0.4-1.fc18
Error: Package: pki-tools-10.0.4-2.fc18.x86_64 (local-testing)
           Requires: pki-base = 10.0.4-2.fc18
           Removing: pki-base-10.0.3-2.fc18.noarch (@local-updates)
               pki-base = 10.0.3-2.fc18
           Updated By: pki-base-10.0.4-1.fc18.noarch (local-testing)
               pki-base = 10.0.4-1.fc18
           Available: pki-base-10.0.0-1.fc18.noarch (local-fedora)
               pki-base = 10.0.0-1.fc18
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

What have I done wrong?

There are required noarch packages too.

I need instructions, please.

(In reply to Dean Hunter from comment #52)
> # koji download-build --arch x86_64 pki-core-10.0.4-2.fc$v

This ^^ downloads just packages with x86_64 architecture. However, some of the pki packages are architecture independent, i.e. "noarch" architecture.

To download both, replace the command above with:
# koji download-build --arch x86_64 --arch noarch pki-core-10.0.4-2.fc$v

Ah!  Thank you.

Which of these packages that I should designate on the "yum update" statement?

[root@host ~]# ls -l /srv/http/repos/fedora18/koji
total 6864
-rw-r--r--. 1 root root  910216 Aug  7 08:01 pki-base-10.0.4-2.fc18.noarch.rpm
-rw-r--r--. 1 root root  429132 Aug  7 08:01 pki-ca-10.0.4-2.fc18.noarch.rpm
-rw-r--r--. 1 root root 1962556 Aug  7 08:01 pki-javadoc-10.0.4-2.fc18.noarch.rpm
-rw-r--r--. 1 root root  213276 Aug  7 08:01 pki-kra-10.0.4-2.fc18.noarch.rpm
-rw-r--r--. 1 root root  132616 Aug  7 08:01 pki-ocsp-10.0.4-2.fc18.noarch.rpm
-rw-r--r--. 1 root root 2786744 Aug  7 08:01 pki-server-10.0.4-2.fc18.noarch.rpm
-rw-r--r--. 1 root root   84184 Aug  7 08:01 pki-symkey-10.0.4-2.fc18.x86_64.rpm
-rw-r--r--. 1 root root  116904 Aug  7 08:01 pki-tks-10.0.4-2.fc18.noarch.rpm
-rw-r--r--. 1 root root  366012 Aug  7 08:01 pki-tools-10.0.4-2.fc18.x86_64.rpm
drwxr-xr-x. 2 root root    4096 Aug  7 08:01 repodata
[root@host ~]#

Doesn't simple "yum update /srv/http/repos/fedora18/koji/pki-*.rpm" work? It should skip pki subpackages you do not have installed on your VM.

I see that pki-core-10.0.4-2 is now available in updates-testing.  I would rather use the repository than what I downloaded from koji.  What packages should I update?

  yum --assumeyes --enablerepo updates-testing update ???

pki-ca should pull in any needed dependencies.

Verified correction for pki 699 and updated karma.

Now my script looks like:

# Upgrade to Fedora 19 from Fedora 18

  yum --assumeyes install fedup
 
  fedup --network 19 \
    --instrepo http://host.hunter.org/isos/fedora19/DVD

  reboot

# Work around FreeIPA upgrade problems

# freeipa 3823: invalid statements in systemd environment

  sed -i /^ulimit/d /etc/sysconfig/dirsrv
  sed -i /^export/d /etc/sysconfig/dirsrv
  cat               /etc/sysconfig/dirsrv

# freeipa ????: orphaned systemd service dependency

  rm -f /etc/systemd/system/dirsrv.target.wants/dirsrv
  find -L /etc /usr -name dirsrv* -type f | sort

# pki 699: can not update broken links

  yum --assumeyes --enablerepo local-testing update pki-ca
  find -L /var /usr -name jss4.jar -exec ls -l {} \;
  find -L /var /usr -name tomcatjss.jar -exec ls -l {} \;

# freeipa 3849: can not upgrade ipa while running in chroot

  /usr/sbin/ipa-ldap-updater --upgrade
  /usr/sbin/ipa-upgradeconfig

# Verify IPA will start

  reboot

Starting with Fedora 18 and FreeIPA 3.1.5-1 and upgrading to Fedora 19 and FreeIPA 3.3.0-1:

# Upgrade to Fedora 19 from Fedora 18

  yum --assumeyes install fedup
 
  fedup --network 19 \
    --instrepo http://host.hunter.org/isos/fedora19/DVD

  reboot

# Work arounds for FreeIPA upgrade problems

  yum --assumeyes --enablerepo local-testing \
    update pki-ca                                # pki-ca-10.0.4-2

  yum --assumeyes --enablerepo local-testing \
    update bind bind-dyndb-ldap freeipa-server   # freeipa-server-3.3.0-1
...
  Updating   : freeipa-server-3.3.0-1.fc19.x86_64                         19/37 
Found IPA server for domain HUNTER.ORG
Converting services setup to systemd
    Upgrade /etc/sysconfig/dirsrv
    Upgrade /etc/sysconfig/krb5kdc
    Re-enable Directory server instances PKI-IPA and HUNTER-ORG 
    Re-enable IPA service
Finished.
  Cleanup    : freeipa-server-3.2.2-1.fc19.x86_64                         20/37 
...

and then yum never completes!

14660_13126,cn=index,cn=tasks,cn=config
[root@ipa2 ~]# tail -20 /var/log/ipaupgrade.log
2013-08-11T21:15:54Z DEBUG only: updated value [u'eq', u'pres', u'sub']
2013-08-11T21:15:54Z DEBUG ---------------------------------------------
2013-08-11T21:15:54Z DEBUG Final value after applying updates
2013-08-11T21:15:54Z DEBUG dn: cn=ipasudorunasgroup,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
2013-08-11T21:15:54Z DEBUG objectClass:
2013-08-11T21:15:54Z DEBUG 	top
2013-08-11T21:15:54Z DEBUG 	nsIndex
2013-08-11T21:15:54Z DEBUG nsIndexType:
2013-08-11T21:15:54Z DEBUG 	eq
2013-08-11T21:15:54Z DEBUG 	pres
2013-08-11T21:15:54Z DEBUG 	sub
2013-08-11T21:15:54Z DEBUG cn:
2013-08-11T21:15:54Z DEBUG 	ipasudorunasgroup
2013-08-11T21:15:54Z DEBUG nsSystemIndex:
2013-08-11T21:15:54Z DEBUG 	false
2013-08-11T21:15:54Z DEBUG [(0, u'nsindextype', ['sub'])]
2013-08-11T21:15:54Z DEBUG Live 1, updated 1
2013-08-11T21:15:54Z INFO Done
2013-08-11T21:15:59Z INFO Creating task to index attribute: ipasudorunasgroup
2013-08-11T21:15:59Z DEBUG Task id: cn=indextask_ipasudorunasgroup_135955485596414660_13126,cn=index,cn=tasks,cn=config
[root@ipa2 ~]#

This sounds serious, thanks for reporting! Sounds like a DS deadlock.

I did a little Bugzilla research and found a related Bug which had a similar behavior - Bug 923407. We should check it did not regress. Adding Noriko and Rich to CC - could you please provide help with debugging the DS side of the issue?

Can we get the errors and access log files from the directory server - /var/log/dirsrv/slapd-DOMAIN/{access,errors}

Is the directory server hung?  If so, please follow these steps to get a stack trace of the hung process:
http://port389.org/wiki/FAQ#Debugging_Crashes
http://port389.org/wiki/FAQ#Debugging_Hangs

This sequence works:

# Upgrade to Fedora 19 from Fedora 18

  yum --assumeyes install fedup
 
  fedup --network 19 \
    --instrepo http://host.hunter.org/isos/fedora19/DVD

  reboot

# Work arounds for FreeIPA server upgrade problems

# pki 699

  yum --assumeyes --enablerepo local-testing \
    update pki-ca                                # pki-ca-10.0.4-2

# freeipa 3823: invalid systemd environment for dirsrv

  sed -i /^ulimit/d /etc/sysconfig/dirsrv
  sed -i /^export/d /etc/sysconfig/dirsrv

# freeipa ????: orphaned systemd service dependency for dirsrv

  rm -f /etc/systemd/system/dirsrv.target.wants/dirsrv

# freeipa 3849: can not upgrade ipa while running in chroot

  /usr/sbin/ipa-ldap-updater --upgrade
  /usr/sbin/ipa-upgradeconfig

# Verify IPA will start and is working correctly

# Upgrade to freeipa 3.3.0-1

  yum --assumeyes --enablerepo local-testing \
    update bind bind-dyndb-ldap freeipa-server   # freeipa-server-3.3.0-1

# freeipa ????: orphaned systemd service dependency for dirsrv

  rm -f /etc/systemd/system/dirsrv.target.wants/dirsrv

(In reply to Rich Megginson from comment #63)
> Can we get the errors and access log files from the directory server -
> /var/log/dirsrv/slapd-DOMAIN/{access,errors}

I will work on this today. I will need to rebuild/restore

> Is the directory server hung?  If so, please follow these steps to get a
> stack trace of the hung process:
> http://port389.org/wiki/FAQ#Debugging_Crashes
> http://port389.org/wiki/FAQ#Debugging_Hangs

How should I test whether the directory server is hung?

(In reply to Dean Hunter from comment #65)
> (In reply to Rich Megginson from comment #63)
> > Can we get the errors and access log files from the directory server -
> > /var/log/dirsrv/slapd-DOMAIN/{access,errors}
> 
> I will work on this today. I will need to rebuild/restore
> 
> > Is the directory server hung?  If so, please follow these steps to get a
> > stack trace of the hung process:
> > http://port389.org/wiki/FAQ#Debugging_Crashes
> > http://port389.org/wiki/FAQ#Debugging_Hangs
> 
> How should I test whether the directory server is hung?

ldapsearch -xLLL -D "cn=directory manager" -W -s base -b "" 'objectclass=*' vendorVersion

If this returns the vendorVersion value, then the server is not completely hung.

However, if yum upgrade is stalled, and it appears to be the fault of the directory server, then it may be that the directory server is deadlocked but not hung, and follow http://port389.org/wiki/FAQ#Debugging_Hangs and http://port389.org/wiki/FAQ#Debugging_Crashes so that we can get a stack trace to examine.

(In reply to Dean Hunter from comment #64)
> This sequence works:
> 
> # Upgrade to Fedora 19 from Fedora 18
> 
>   yum --assumeyes install fedup
>  
>   fedup --network 19 \
>     --instrepo http://host.hunter.org/isos/fedora19/DVD
> 
>   reboot
> 
> # Work arounds for FreeIPA server upgrade problems
> 
> # pki 699
> 
>   yum --assumeyes --enablerepo local-testing \
>     update pki-ca                                # pki-ca-10.0.4-2
> 
> # freeipa 3823: invalid systemd environment for dirsrv
> 
>   sed -i /^ulimit/d /etc/sysconfig/dirsrv
>   sed -i /^export/d /etc/sysconfig/dirsrv
> 
> # freeipa ????: orphaned systemd service dependency for dirsrv
> 
>   rm -f /etc/systemd/system/dirsrv.target.wants/dirsrv
> 
> # freeipa 3849: can not upgrade ipa while running in chroot
> 
>   /usr/sbin/ipa-ldap-updater --upgrade
>   /usr/sbin/ipa-upgradeconfig
> 
> # Verify IPA will start and is working correctly
> 
> # Upgrade to freeipa 3.3.0-1
> 
>   yum --assumeyes --enablerepo local-testing \
>     update bind bind-dyndb-ldap freeipa-server   # freeipa-server-3.3.0-1
> 
> # freeipa ????: orphaned systemd service dependency for dirsrv
> 
>   rm -f /etc/systemd/system/dirsrv.target.wants/dirsrv

Are all of these issues fixed in ipa 3.3?  If so, perhaps you can try to upgrade to ipa 3.3 instead?

Created attachment 786218 [details]
/var/log/dirsrv/slapd-HUNTER-ORG access and errors

(In reply to Dean Hunter from comment #68)
> Created attachment 786218 [details]
> /var/log/dirsrv/slapd-HUNTER-ORG access and errors

Nothing interesting in the errors log.  There is a request without a result in the access log, which could indicate a hang or deadlock.  Need a stack trace.

(In reply to Rich Megginson from comment #67)
> (In reply to Dean Hunter from comment #64)
> 
> Are all of these issues fixed in ipa 3.3?  If so, perhaps you can try to
> upgrade to ipa 3.3 instead?

None of these issues are fixed in IPA 3.3.  The last time I looked they were scheduled for IPA 3.4.  However, as soon as IPA 3.3 moves to the Fedora Updates repository I will test the upgrade from IPA 3.1 to IPA 3.3 because I do not know how to tell fedup to get freeipa-server 3.3 from updates-testing.

The requested stack trace will be available shortly.

(In reply to Rich Megginson from comment #69)
> (In reply to Dean Hunter from comment #68)
> > Created attachment 786218 [details]
> > /var/log/dirsrv/slapd-HUNTER-ORG access and errors
> 
> Nothing interesting in the errors log.  There is a request without a result
> in the access log, which could indicate a hang or deadlock.  Need a stack
> trace.

I am sorry.  I was called out of town for several weeks and now I can not build a viable Fedora 18 VM to test the upgrade.  A new build with all updates applied has problems with:

1. starting gdm
2. the label on /var/tmp/abrt
3. audispd event queue full

I will get the stack trace as soon as I can.

*** Bug 1058222 has been marked as a duplicate of this bug. ***

This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

FreeIPA does not support upgrades from unsupported Fedora releases, see Comment 73. I am thus closing this bug. If you are still experiencing upgrade issues with supported Fedora versions (20, 21), please file a new bug.