This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 989094 - Upgrade to 3.2.2-1 from 3.1.5-1 fails
Upgrade to 3.2.2-1 from 3.1.5-1 fails
Status: CLOSED EOL
Product: Fedora
Classification: Fedora
Component: freeipa (Show other bugs)
19
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: Rob Crittenden
Fedora Extras Quality Assurance
:
: 1058222 (view as bug list)
Depends On:
Blocks: 990892
  Show dependency treegraph
 
Reported: 2013-07-27 09:44 EDT by Dean Hunter
Modified: 2015-01-15 10:29 EST (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 990892 (view as bug list)
Environment:
Last Closed: 2015-01-15 10:29:56 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
/var/log/ipaupgrade.log (5.33 MB, application/octet-stream)
2013-07-27 09:51 EDT, Dean Hunter
no flags Details
upgrade logs (141.04 KB, application/gzip)
2013-07-29 15:47 EDT, Dean Hunter
no flags Details
all avc messages (11.16 KB, text/x-log)
2013-07-29 15:55 EDT, Dean Hunter
no flags Details
excerpt from /var/log/messages during ipactl start (17.42 KB, text/plain)
2013-07-30 14:20 EDT, Dean Hunter
no flags Details
/var/log/dirsrv/slapd-HUNTER-ORG access and errors (28.19 KB, application/gzip)
2013-08-13 13:23 EDT, Dean Hunter
no flags Details

  None (edit)
Description Dean Hunter 2013-07-27 09:44:35 EDT
Description of problem:

Upgrading from Fedora18 and FreeIPA server 3.1.5-1 to Fedora 19 and FreeIPA server 3.2.2-1 fails while upgrading directory server.  The work around of running ipa-ldap-updater and ipa-upgradeconfig following the failure is successful.  But there remains a problem starting pki-tomcat.


Version-Release number of selected component (if applicable):

freeipa-server-3.2.2-1.fc19.x86_64


How reproducible: Consistent


Steps to Reproduce:

1. Build a new Fedora 18 VM
2. Install and configure freeipa-server-3.1.5-1.fc19.x86_64
3. yum install fedup
4. fedup -network 19 --instrepo http://host.hunter.org/repos/fedora19/iso
5. reboot
6. ipactl status


Actual results:

[root@ipa2 ~]# ipactl status
Directory Service: STOPPED
Directory Service must be running in order to obtain status of other services
ipa: INFO: The ipactl command was successful

[root@ipa2 ~]# 

From ipaupgrade.log

2013-07-27T12:49:16Z DEBUG   duration: 0 seconds
2013-07-27T12:49:16Z DEBUG   [5/8]: upgrading server
2013-07-27T12:49:56Z ERROR Upgrade failed with [Errno 2] No such file or directory
2013-07-27T12:49:56Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py", line 115, in __upgrade
    ld = ldapupdate.LDAPUpdate(dm_password='', ldapi=True, live_run=self.live_run, plugins=True)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 174, in __init__
    conn.do_external_bind(self.pw_name)
  File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1734, in do_external_bind
    self.conn.sasl_interactive_bind_s, timeout, None, auth_tokens)
  File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1720, in __bind_with_wait
    self.__wait_for_connection(timeout)
  File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1706, in __wait_for_connection
    wait_for_open_socket(lurl.hostport, timeout)
  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 1115, in wait_for_open_socket
    raise e
error: [Errno 2] No such file or directory


Expected results:

[root@ipa2 ~]# ipactl status
ipa: INFO: The ipactl command was successful
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING

[root@ipa2 ~]# 

And no errors in ipaupgrade.log


Additional info:

[root@ipa2 ~]# ausearch --message avc --start today
<no matches>

[root@ipa2 ~]# ls -laZ /var/run/slapd-*.socket
srw-rw-rw-. root root system_u:object_r:dirsrv_var_run_t:s0 /var/run/slapd-HUNTER-ORG.socket

[root@ipa2 ~]# restorecon -R -v /var/tmp/abrt

[root@ipa2 ~]# /usr/sbin/ipa-ldap-updater --upgrade
Upgrading IPA:
  [1/8]: stopping directory server
  [2/8]: saving configuration
  [3/8]: disabling listeners
  [4/8]: starting directory server
  [5/8]: upgrading server
PRE_UPDATE
Parsing update file '/usr/share/ipa/updates/10-60basev2.update'
Parsing update file '/usr/share/ipa/updates/10-60basev3.update'
Parsing update file '/usr/share/ipa/updates/10-70ipaotp.update'
...
Done
Updating existing entry: cn=CAcert,cn=ipa,cn=etc,dc=hunter,dc=org
Done
  [6/8]: stopping directory server
  [7/8]: restoring configuration
  [8/8]: starting directory server
Done.
The ipa-ldap-updater command was successful

[root@ipa2 ~]# /usr/sbin/ipa-upgradeconfig
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that CA proxy configuration is correct]
[Verifying that KDC configuration is using ipa-kdb backend]
[Fix DS schema file syntax]
[Removing self-signed CA]
Configuring ipa-otpd
  [1/2]: starting ipa-otpd 
  [2/2]: configuring ipa-otpd to start on boot
Done configuring ipa-otpd.
[Checking for deprecated KDC configuration files]
[Setting up Firefox extension]
/usr/share/ipa/html/krb.js exists, skipping install of Firefox extension
[Add missing CA DNS records]
[Enabling persistent search in DNS]
[Enabling serial autoincrement in DNS]
[Updating GSSAPI configuration in DNS]
[Updating pid-file configuration in DNS]
Changes to named.conf have been made, restart named
[Enable certificate renewal]
[Verifying that CA service certificate profile is updated]
[Certificate renewal should stop the CA]
Already configured to stop CA
The ipa-upgradeconfig command was successful

[root@ipa2 ~]# ipactl stop
Stopping Directory Service
ipa: INFO: The ipactl command was successful

[root@ipa2 ~]# ipactl start
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting ipa_memcached Service
Starting httpd Service
Starting pki-tomcatd Service
Failed to start pki-tomcatd Service
Shutting down
Aborting ipactl

[root@ipa2 ~]#
Comment 1 Dean Hunter 2013-07-27 09:51:49 EDT
Created attachment 779088 [details]
/var/log/ipaupgrade.log

The log is a concatenation of the output produced during fedup and when running the work around.
Comment 2 Martin Kosek 2013-07-29 12:00:32 EDT
FreeIPA upgrade should work even in the case when all IPA services are shut down (like in the fedup environment). So I think the key question here is why the 389-ds-base socket is not available in the fedup process.

Can we rule out the SELinux? I think the last time we investigated it together, you said that there were no related AVCs in /var/log/audit/audit.log after the upgrade.

In case you have chance to try running the upgrade again, would it may any change to update /etc/selinux/config and change SELINUX to permissive to confirm or rule out that SELinux is the cause?


(In reply to Dean Hunter from comment #0)
> But there remains a problem starting pki-tomcat.

What problem? I saw no example outputs in this bug or the previous one. Maybe the pki-ca is just starting too slow. In that case it may help to increase the time that ipactl waits for processes to start, e.g.:

echo "startup_timeout=240" >> /etc/ipa/default.conf
Comment 3 Dean Hunter 2013-07-29 12:31:00 EDT
Please, I have no in depth knowledge of FreeIPA or Linux other than what I have learned stumbling around the last few months trying to make it work.  If you need me to do something to gather more information it would be helpful to me if you told me both what you need and how to get it.

1. Where in the "Steps to Reproduce" would you like me to to change to SELinux permissive mode?  Are you requesting:

   3. yum install fedup
      setenforce 0
   4. fedup -network 19 --instrepo http://host.hunter.org/repos/fedora19/iso
      setenforce 1
   5. reboot

2. Does this mean that SELinux alerts are not logged during fedup where they could be reported by ausearch?

3. Where would you expect to find more information about why pki-tomcat failed to start?
Comment 4 Rob Crittenden 2013-07-29 13:18:35 EDT
(In reply to Dean Hunter from comment #3)
> Please, I have no in depth knowledge of FreeIPA or Linux other than what I
> have learned stumbling around the last few months trying to make it work. 
> If you need me to do something to gather more information it would be
> helpful to me if you told me both what you need and how to get it.
> 
> 1. Where in the "Steps to Reproduce" would you like me to to change to
> SELinux permissive mode?  Are you requesting:
> 
>    3. yum install fedup
>       setenforce 0
>    4. fedup -network 19 --instrepo http://host.hunter.org/repos/fedora19/iso
>       setenforce 1
>    5. reboot

I think Martin is suggesting modifying /etc/selinux/config directly so it remains in permissive mode through reboots.
 
> 2. Does this mean that SELinux alerts are not logged during fedup where they
> could be reported by ausearch?

No. Permissive mode still logs failures, but allows the access that would have been denied in Enforcing mode.
 
> 3. Where would you expect to find more information about why pki-tomcat
> failed to start?

/var/log/messages, /var/log/pki/pki-tomcat/catalina.out, /var/log/pki/pki-tomcat/ca/debug are where I'd start
Comment 5 Dean Hunter 2013-07-29 14:57:43 EDT
Thank you for the clarification.

1. In "Steps to Reproduce" I have added a step:

   3. yum install fedup
   +  sed -i s/SELINUX=enforcing/SELINUX=permissive/ /set/selinux/config
   4. fedup -network 19 --instrepo http://host.hunter.org/repos/fedora19/iso
   5. reboot

2. Are we changing the SELinux mode to permissive because the SELinux are not logged during the upgrade where they could be reported by ausearch?
Comment 6 Rob Crittenden 2013-07-29 15:36:06 EDT
We want to be sure it is in permissive the entire cycle of the upgrade.

It should have no impact on what is printed, other than it may print more since it won't be blocked by a single denial.
Comment 7 Dean Hunter 2013-07-29 15:42:06 EDT
The results look the same to me.  I will attach all of the mentioned log files.
Comment 8 Dean Hunter 2013-07-29 15:47:59 EDT
Created attachment 780119 [details]
upgrade logs

/var/log/ipaupgrade.log
/var/log/messages
/var/log/pki/pki-tomcat/catalina.out
/var/log/pki/pki-tomcat/ca/debug
Comment 9 Dean Hunter 2013-07-29 15:55:13 EDT
Created attachment 780121 [details]
all avc messages
Comment 10 Martin Kosek 2013-07-30 03:00:18 EDT
Thanks Dean(In reply to Dean Hunter from comment #3)
> Please, I have no in depth knowledge of FreeIPA or Linux other than what I
> have learned stumbling around the last few months trying to make it work. 
> If you need me to do something to gather more information it would be
> helpful to me if you told me both what you need and how to get it.

No problem at all, just ask as you just did in case you do not understand my instructions. So far, your feedback and testing was very valuable.

I finally found  the root cause of why the upgrade process does not work:

2013-07-29T19:26:22Z DEBUG   [4/8]: starting directory server
2013-07-29T19:26:22Z DEBUG Starting external process
2013-07-29T19:26:22Z DEBUG args=/bin/systemctl start dirsrv.target
2013-07-29T19:26:22Z DEBUG Process finished, return code=0
2013-07-29T19:26:22Z DEBUG stdout=
<<<
2013-07-29T19:26:22Z DEBUG stderr=Running in chroot, ignoring request.
>>>
2013-07-29T19:26:22Z DEBUG   duration: 0 seconds
2013-07-29T19:26:22Z DEBUG   [5/8]: upgrading server
2013-07-29T19:27:02Z ERROR Upgrade failed with [Errno 2] No such file or directory
2013-07-29T19:27:02Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.py", line 115, in __upgrade
    ld = ldapupdate.LDAPUpdate(dm_password='', ldapi=True, live_run=self.live_run, plugins=True)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ldapupdate.py", line 174, in __init__
    conn.do_external_bind(self.pw_name)
  File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1734, in do_external_bind
    self.conn.sasl_interactive_bind_s, timeout, None, auth_tokens)
  File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1720, in __bind_with_wait
    self.__wait_for_connection(timeout)
  File "/usr/lib/python2.7/site-packages/ipapython/ipaldap.py", line 1706, in __wait_for_connection
    wait_for_open_socket(lurl.hostport, timeout)
  File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 1115, in wait_for_open_socket
    raise e
error: [Errno 2] No such file or directory

It is indeed not SELinux related. The problem is that fedup runs in chroot-ed environment which does not allow dirsrv service to be started. Thus, the only workable workaround so far is the one you already acknowledged, i.e. running the upgrade processes after the Fedora 19 started.

Rob or Alexander, any idea how to mitigate this? I am thinking we may want to improve reporting of the upgrade process failure so that user is well notified about it after the fedup finished. We will also need to update the upgrade instructions in the release notes to run the update commands in case system is upgraded via fedup.
Comment 11 Dean Hunter 2013-07-30 08:30:26 EDT
I have no work around.  Running the upgrade manually results in errors with pki-tomcat, remember?  Log files for pki-tomcat are attached

To get my network onto Fedora 19 I rebuilt IPA.  I am running these tests on a pair of isolated VMs to try to save the next person some grief.
Comment 12 Martin Kosek 2013-07-30 11:23:03 EDT
Yeah, there are actually 2 issues:

1) Upgrade process does not work in fedup (explained in Comment 10)

2) PKI does not start. I found this error in PKI-CA debug log, looks related:

[29/Jul/2013:14:14:16][Thread-2]: Can't create master connection in LdapBoundConnFactory::getConn! Could not connect to LDAP server host ipa2.hunter.org port 636 Error netscape.ldap.LDAPException: IO Error creating JSS SSL Socket (-1)

When your dirsrv starts, is the port on? You can find out this way for example:

# netstat -pnl | grep  636

If it isn't, can you pleas check a value of "nsslapd-security" in /etc/dirsrv/slapd-HUNTER-ORG/dse.ldif?
Comment 13 Dean Hunter 2013-07-30 13:46:04 EDT
I am sorry, but I do not understand "When your dirsrv starts, ....".  

Do you mean after performing the upgrade manually as in:

# /usr/sbin/ipa-ldap-updater --upgrade
# /usr/sbin/ipa-upgradeconfig
# netstat -pnl | grep  636
Comment 14 Rob Crittenden 2013-07-30 13:50:16 EDT
(In reply to Dean Hunter from comment #13)
> I am sorry, but I do not understand "When your dirsrv starts, ....".  
> 
> Do you mean after performing the upgrade manually as in:

Yes.

He wants to see if we are restoring some values in dse.ldif that affect what ports 389-ds listens to. During an upgrade we disable all network ports to ensure that things are quiet. It should always restore those values, even on catastrophic failure. Since tomcat is failing to start the thinking is that the secure port isn't being re-enabled.

I gather that dirsrv is actually starting up ok since the KDC and bind are starting.
Comment 15 Dean Hunter 2013-07-30 13:59:51 EDT
[root@ipa2 ~]#   /usr/sbin/ipa-ldap-updater --upgrade
Upgrading IPA:
  [1/8]: stopping directory server
  [2/8]: saving configuration
  [3/8]: disabling listeners
  [4/8]: starting directory server
  [5/8]: upgrading server
PRE_UPDATE
Parsing update file '/usr/share/ipa/updates/10-60basev2.update'
...
Updating existing entry: cn=CAcert,cn=ipa,cn=etc,dc=hunter,dc=org
Done
  [6/8]: stopping directory server
  [7/8]: restoring configuration
  [8/8]: starting directory server
Done.
The ipa-ldap-updater command was successful

[root@ipa2 ~]#   /usr/sbin/ipa-upgradeconfig
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that CA proxy configuration is correct]
[Verifying that KDC configuration is using ipa-kdb backend]
[Fix DS schema file syntax]
[Removing self-signed CA]
Configuring ipa-otpd
  [1/2]: starting ipa-otpd 
  [2/2]: configuring ipa-otpd to start on boot
Done configuring ipa-otpd.
[Checking for deprecated KDC configuration files]
[Setting up Firefox extension]
/usr/share/ipa/html/krb.js exists, skipping install of Firefox extension
[Add missing CA DNS records]
[Enabling persistent search in DNS]
[Enabling serial autoincrement in DNS]
[Updating GSSAPI configuration in DNS]
[Updating pid-file configuration in DNS]
Changes to named.conf have been made, restart named
[Enable certificate renewal]
[Verifying that CA service certificate profile is updated]
[Certificate renewal should stop the CA]
Already configured to stop CA
The ipa-upgradeconfig command was successful

[root@ipa2 ~]#   ss -lnp | grep  636
tcp    LISTEN     0      128                   :::636                  :::*      users:(("ns-slapd",12396,7))

[root@ipa2 ~]#
Comment 16 Dean Hunter 2013-07-30 14:18:56 EDT
[root@ipa2 ~]# ipactl status
Directory Service: RUNNING
ipa: INFO: The ipactl command was successful

[root@ipa2 ~]# ipactl stop
Stopping Directory Service
ipa: INFO: The ipactl command was successful

[root@ipa2 ~]# ipactl start
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting ipa_memcached Service
Starting httpd Service
Starting pki-tomcatd Service
Failed to start pki-tomcatd Service
Shutting down
Aborting ipactl

[root@ipa2 ~]# ipactl status
Directory Service: STOPPED
Directory Service must be running in order to obtain status of other services
ipa: INFO: The ipactl command was successful

[root@ipa2 ~]#
Comment 17 Dean Hunter 2013-07-30 14:20:30 EDT
Created attachment 780792 [details]
excerpt from /var/log/messages during ipactl start
Comment 18 Dean Hunter 2013-07-30 14:33:01 EDT
[root@ipa2 ~]# cat /var/log/messages
...
Jul 30 13:14:22 ipa2 systemd[1]: Starting 389 Directory Server.
Jul 30 13:14:22 ipa2 systemd[1]: Reached target 389 Directory Server.
Jul 30 13:14:22 ipa2 systemd[1]: Starting 389 Directory Server HUNTER-ORG....
Jul 30 13:14:22 ipa2 systemd[1]: Ignoring invalid environment 'export KRB5_KTNAME=/etc/dirsrv/ds.keytab': /etc/sysconfig/dirsrv
Jul 30 13:14:22 ipa2 systemd[1]: Starting 389 Directory Server PKI-IPA....
Jul 30 13:14:22 ipa2 systemd[1]: Ignoring invalid environment 'export KRB5_KTNAME=/etc/dirsrv/ds.keytab': /etc/sysconfig/dirsrv
Jul 30 13:14:22 ipa2 systemd[1]: Failed to load environment files: No such file or directory
Jul 30 13:14:22 ipa2 systemd[1]: dirsrv@PKI-IPA.service failed to run 'start' task: No such file or directory
Jul 30 13:14:22 ipa2 systemd[1]: Failed to start 389 Directory Server PKI-IPA..
....

[root@ipa2 ~]# cat /etc/sysconfig/dirsrv
# This file is sourced by dirsrv upon startup to set
# the default environment for all directory server instances.
# To set instance specific defaults, use the file in the same
...
# if using systemd, omit the "; export VARNAME" at the end
#PID_TIME=600 ; export PID_TIME
ulimit -n 8192
KRB5_KTNAME=/etc/dirsrv/ds.keytab
export KRB5_KTNAME=/etc/dirsrv/ds.keytab
KRB5CCNAME=/tmp/krb5cc_990

[root@ipa2 ~]# ls -alZ /etc/dirsrv
drwxrwxr-x. root   dirsrv system_u:object_r:dirsrv_config_t:s0 .
drwxr-xr-x. root   root   system_u:object_r:etc_t:s0       ..
drwxr-xr-x. root   root   system_u:object_r:dirsrv_config_t:s0 config
-rw-------. dirsrv dirsrv unconfined_u:object_r:dirsrv_config_t:s0 ds.keytab
drwxr-xr-x. root   root   system_u:object_r:dirsrv_config_t:s0 schema
drwxrwx---. dirsrv dirsrv unconfined_u:object_r:dirsrv_config_t:s0 slapd-HUNTER-ORG

[root@ipa2 ~]#
Comment 19 Dean Hunter 2013-07-30 14:49:40 EDT
[root@ipa2 ~]# cat /var/log/messages
...
Jul 30 13:14:26 ipa2 systemd[1]: Starting PKI Tomcat Server.
Jul 30 13:14:26 ipa2 systemd[1]: Reached target PKI Tomcat Server.
Jul 30 13:14:26 ipa2 systemd[1]: Starting PKI Tomcat Server pki-tomcat...
Jul 30 13:14:26 ipa2 pkidaemon[13113]: WARNING:  Symbolic link '/var/lib/pki/pki-tomcat/common/lib/jss4.jar' does NOT exist!
Jul 30 13:14:26 ipa2 pkidaemon[13113]: INFO:  Attempting to create '/var/lib/pki/pki-tomcat/common/lib/jss4.jar' -> '/usr/lib/java/jss4.jar' . . .
Jul 30 13:14:26 ipa2 pkidaemon[13113]: ln: failed to create symbolic link ‘/var/lib/pki/pki-tomcat/common/lib/jss4.jar’: File exists
Jul 30 13:14:26 ipa2 pkidaemon[13113]: ERROR:  Failed to create '/var/lib/pki/pki-tomcat/common/lib/jss4.jar' -> '/usr/lib/java/jss4.jar'!
Jul 30 13:14:26 ipa2 pkidaemon[13113]: WARNING:  Symbolic link '/var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar' does NOT exist!
Jul 30 13:14:26 ipa2 pkidaemon[13113]: INFO:  Attempting to create '/var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar' -> '/usr/share/java/tomcatjss.jar' . . .
Jul 30 13:14:26 ipa2 pkidaemon[13113]: ln: failed to create symbolic link ‘/var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar’: File exists
Jul 30 13:14:26 ipa2 pkidaemon[13113]: ERROR:  Failed to create '/var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar' -> '/usr/share/java/tomcatjss.jar'!
Jul 30 13:14:27 ipa2 systemd[1]: pki-tomcatd@pki-tomcat.service: control process exited, code=exited status=1
Jul 30 13:14:27 ipa2 systemd[1]: Failed to start PKI Tomcat Server pki-tomcat.
Jul 30 13:14:27 ipa2 systemd[1]: Unit pki-tomcatd@pki-tomcat.service entered failed state.
....

[root@ipa2 ~]# find /usr /var -name jss4.jar -exec ls -lZ {} \;
lrwxrwxrwx. root root system_u:object_r:lib_t:s0       /usr/lib64/jss/jss4.jar -> /usr/lib/java/jss4.jar
-rw-r--r--. root root system_u:object_r:lib_t:s0       /usr/lib/java/jss4.jar
lrwxrwxrwx. pkiuser pkiuser system_u:object_r:pki_tomcat_var_lib_t:s0 /var/lib/pki/pki-tomcat/common/lib/jss4.jar -> /usr/lib64/java/jss4.jar

[root@ipa2 ~]# find /usr /var -name jss.jar -exec ls -lZ {} \;

[root@ipa2 ~]#
Comment 20 Rob Crittenden 2013-07-30 15:18:45 EDT
(In reply to Dean Hunter from comment #18)

The dirsrv systemd errors are being tracked in https://bugzilla.redhat.com/show_bug.cgi?id=988858

They are benign.
Comment 21 Rob Crittenden 2013-07-30 15:20:22 EDT
(In reply to Dean Hunter from comment #19)
> [root@ipa2 ~]# cat /var/log/messages
> ...
> Jul 30 13:14:26 ipa2 systemd[1]: Starting PKI Tomcat Server.
> Jul 30 13:14:26 ipa2 systemd[1]: Reached target PKI Tomcat Server.
> Jul 30 13:14:26 ipa2 systemd[1]: Starting PKI Tomcat Server pki-tomcat...
> Jul 30 13:14:26 ipa2 pkidaemon[13113]: WARNING:  Symbolic link
> '/var/lib/pki/pki-tomcat/common/lib/jss4.jar' does NOT exist!
> Jul 30 13:14:26 ipa2 pkidaemon[13113]: INFO:  Attempting to create
> '/var/lib/pki/pki-tomcat/common/lib/jss4.jar' -> '/usr/lib/java/jss4.jar' .
> . .
> Jul 30 13:14:26 ipa2 pkidaemon[13113]: ln: failed to create symbolic link
> ‘/var/lib/pki/pki-tomcat/common/lib/jss4.jar’: File exists

Ade, do you know why the links are failing?

Dean, what version of tomcat, tomcatjss and jss do you have installed?
Comment 22 Dean Hunter 2013-07-30 15:54:04 EDT
(In reply to Rob Crittenden from comment #20)
> (In reply to Dean Hunter from comment #18)
> 
> The dirsrv systemd errors are being tracked in
> https://bugzilla.redhat.com/show_bug.cgi?id=988858
> 
> They are benign.

Comparing my new install IPA 3.2.2-1 with my upgrade IPA 3.2.2-1:

/etc/sysconfig/dirsrv
new:
KRB5CCNAME=/tmp/krb5cc_990
ulimit -n 8192
KRB5_KTNAME=/etc/dirsrv/ds.keytab

upgrade:
ulimit -n 8192
KRB5_KTNAME=/etc/dirsrv/ds.keytab
export KRB5_KTNAME=/etc/dirsrv/ds.keytab
KRB5CCNAME=/tmp/krb5cc_990

When I change the upgrade statements to be the same as the new statements I still get:

Jul 30 14:29:42 ipa2 systemd[1]: Failed to load environment files: No such file or directory
Jul 30 14:29:42 ipa2 systemd[1]: dirsrv@PKI-IPA.service failed to run 'start' task: No such file or directory
Jul 30 14:29:42 ipa2 systemd[1]: Failed to start 389 Directory Server PKI-IPA..

Are you saying that dirsrv@PKI-IPA.service is not required?
Comment 23 Dean Hunter 2013-07-30 15:56:35 EDT
(In reply to Rob Crittenden from comment #21)

> Dean, what version of tomcat, tomcatjss and jss do you have installed?

[root@ipa2 ~]# rpm -q tomcat tomcatjss jss
tomcat-7.0.40-2.fc19.noarch
tomcatjss-7.1.0-3.fc19.noarch
jss-4.2.6-30.fc19.x86_64

[root@ipa2 ~]#
Comment 24 Rob Crittenden 2013-07-30 16:22:33 EDT
I think you need to check this out, http://pki.fedoraproject.org/wiki/Migrating_Dogtag_9_Instances_to_Dogtag_10
Comment 25 Dean Hunter 2013-07-30 17:55:55 EDT
1. Is there an error in the first paragraph of the wiki aricle you referenced?  

Overview

Dogtag 9 instances are configured as tomcat 6 instances. Unfortunately, tomcat 6 is no longer supported on Fedora 19. To ensure that users do not inadvertently upgrade to Fedora 18 and render their Dogatg 9 instances inoperable, code has been added to check for the existence of Dogtag 9 instances prior to upgrade and to cause the upgrade to abort if those instances exist. 

Should it be "upgrade from Fedora 18" or "upgrade to Fedora 19" instead of "upgrade to Fedora 18"?

2. Please confirm that the dogtag 9 to 10 upgrade should be performed on the Fedora 18 instance before the Fedora 18 to 19 upgrade?
Comment 26 Dmitri Pal 2013-07-30 19:03:29 EDT
1. upgrade to Fedora 19 - thanks for catching
2. Yes
Comment 27 Martin Kosek 2013-07-31 02:33:27 EDT
(In reply to Rob Crittenden from comment #24)
> I think you need to check this out,
> http://pki.fedoraproject.org/wiki/Migrating_Dogtag_9_Instances_to_Dogtag_10

This is FreeIPA specialized page:
http://www.freeipa.org/page/Howto/Dogtag9ToDogtag10Migration

(In reply to Dean Hunter from comment #25)
> ...
> 2. Please confirm that the dogtag 9 to 10 upgrade should be performed on the
> Fedora 18 instance before the Fedora 18 to 19 upgrade?

This is correct. Just please note, that we cannot do just a simple upgrade (in terms of yum/package update), but a manual migration procedure needs to be followed to migrate your old FreeIPA masters with CA to Fedora 19. The procedure is described on the FreeIPA.org wiki.
Comment 28 Dean Hunter 2013-07-31 07:53:28 EDT
If you are using the installed version of tomcat as the indicator then here are the installed package versions before the Fedora 18 to 19 upgrade:

[root@ipa2 ~]# rpm -q freeipa-server tomcat tomcatjss jss
freeipa-server-3.1.5-1.fc18.x86_64
tomcat-7.0.40-1.fc18.noarch
tomcatjss-7.0.0-5.fc18.noarch
jss-4.2.6-28.fc18.x86_64

[root@ipa2 ~]# 

Steps to Reproduce:

1. Build a new Fedora 18 VM
2. Install and configure freeipa-server-3.1.5-1.fc18.x86_64
3. yum install fedup
4. fedup -network 19 --instrepo http://host.hunter.org/repos/fedora19/iso
5. reboot
6. ipactl status
Comment 29 Martin Kosek 2013-07-31 08:25:04 EDT
(In reply to Dean Hunter from comment #28)
...
> Steps to Reproduce:
> 
> 1. Build a new Fedora 18 VM
> 2. Install and configure freeipa-server-3.1.5-1.fc18.x86_64
> 3. yum install fedup
> 4. fedup -network 19 --instrepo http://host.hunter.org/repos/fedora19/iso
> 5. reboot
> 6. ipactl status

BTW if you installed FreeIPA CA master in F18, you do not have to undertake the migration procedure described here:
http://www.freeipa.org/page/Howto/Dogtag9ToDogtag10Migration
... as you already use Dogtag10-based CA.

So I think that your pki-ca problem may be a different issue - Ade, what do you think about errors in Comment 19?
Comment 30 Ade Lee 2013-07-31 10:34:24 EDT
I fixed the wiki entry.  Thanks for the catch.

I am totally befuddled by the messages though.  The startup code (in /usr/share/pki/scripts/operations does the following:

if [ -e ${symlink} ]; then
   do stuff ..
else 
   echo "WARNING:  Symbolic link '${symlink}' does NOT exist!"
   # Attempt to create the symbolic link and chown it.
   make_symlink ${symlink} ${target} ${user} ${group}
fi


make_symlink()
{
    symlink="${1}"
    target="${2}"
    user="${3}"
    group="${4}"

    rv=0

    echo "INFO:  Attempting to create '${symlink}' -> '${target}' . . ."
    # Check to make certain that the expected target exists.
    #
    #     NOTE:  The symbolic link does NOT exist at this point.
    #
    if [ -e ${target} ]; then
        # Check that the expected target is fully resolvable!
        if [ ! `readlink -qe ${target}` ]; then
            # Issue an ERROR that the target to which the
            # symbolic link is expected to point is NOT fully resolvable!
            echo "ERROR:  Failed making '${symlink}' -> '${target}'"\
                 "since target '${target}' is NOT fully resolvable!"
            rv=1
        else
            # Attempt to create a symbolic link and 'chown' it.
            ln -s ${target} ${symlink}
            rv=$?
            if [ $rv -eq 0 ]; then
                # NOTE:  Ignore 'chown' errors.
                chown -h ${user}:${group} ${symlink}
                echo "SUCCESS:  Created '${symlink}' -> '${target}'"
            else
                echo "ERROR:  Failed to create '${symlink}' -> '${target}'!"
                rv=1
            fi
        fi
    else
        # Issue an ERROR that the target to which the
        # symbolic link is expected to point does NOT exist.
        echo "ERROR:  Failed making '${symlink}' -> '${target}'"\
             "since target '${target}' does NOT exist!"
        rv=1
    fi

    return $rv
}

So, its only trying to create the symlink if the symlink does not exist - and if the target exists, but then the ln -s is failing to create the link - and is reporting that it does so because the file exists!

Do the links -

/var/lib/pki/pki-tomcat/common/lib/jss4.jar -> /usr/lib/java/jss4.jar 
/var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar -> /usr/share/java/tomcatjss.jar

exist? (Along with their targets?)

I'll have to think more on how this could happen.  As a workaround, you can try to create them manually, and then try restarting the CA.

ln -s /usr/lib/java/jss4.jar /var/lib/pki/pki-tomcat/common/lib/jss4.jar
ln -s /usr/share/java/tomcatjss.jar /var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar
systemctl restart pki-tomcatd@pki-tomcat.service
Comment 31 Dean Hunter 2013-07-31 11:36:38 EDT
Does this answer your question about links to jss4.jar?

From Comment 19:

[root@ipa2 ~]# find /usr /var -name jss4.jar -exec ls -lZ {} \;
lrwxrwxrwx. root root system_u:object_r:lib_t:s0       /usr/lib64/jss/jss4.jar -> /usr/lib/java/jss4.jar
-rw-r--r--. root root system_u:object_r:lib_t:s0       /usr/lib/java/jss4.jar
lrwxrwxrwx. pkiuser pkiuser system_u:object_r:pki_tomcat_var_lib_t:s0 /var/lib/pki/pki-tomcat/common/lib/jss4.jar -> /usr/lib64/java/jss4.jar

[root@ipa2 ~]#
Comment 32 Dean Hunter 2013-07-31 11:46:26 EDT
This is what the links looked like before the upgrade:

[root@ipa2 ~]# find /usr /var -name jss4.jar -exec ls -lZ {} \;
lrwxrwxrwx. root root system_u:object_r:lib_t:s0       /usr/lib64/jss/jss4.jar -> /usr/lib64/java/jss4.jar
-rw-r--r--. root root system_u:object_r:lib_t:s0       /usr/lib64/java/jss4.jar
lrwxrwxrwx. pkiuser pkiuser system_u:object_r:pki_tomcat_var_lib_t:s0 /var/lib/pki/pki-tomcat/common/lib/jss4.jar -> /usr/lib64/java/jss4.jar

[root@ipa2 ~]#
Comment 33 Ade Lee 2013-07-31 12:10:05 EDT
Actually, I think we found a case where the code above will fail.

The case is if the link exists and is pointing to the wrong location.  In that case, the -e test will fail - and then ln -s will fail because the link already exists.

That clearly is the case for the jss jar.  In F19, the JNI directory was moved as indicated.

The same is not true though for tomcatjss.  Can you tell us what the location of the tomcatjss link was?
Comment 34 Dean Hunter 2013-07-31 13:12:42 EDT
Here are the links for tomcatjss.jar before the upgrade:

[root@ipa2 ~]# find /usr /var -name tomcatjss.jar -exec ls -lZ {} \;
lrwxrwxrwx. root root system_u:object_r:usr_t:s0       /usr/share/java/tomcatjss.jar -> tomcatjss-6.0.99.jar
lrwxrwxrwx. pkiuser pkiuser system_u:object_r:pki_tomcat_var_lib_t:s0 /var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar -> /usr/share/java/tomcat7jss.jar

[root@ipa2 ~]# 

It will take me a little more time to get these links after the upgrade.
Comment 35 Dean Hunter 2013-07-31 14:46:18 EDT
Here are the links for tomcat.jss after the upgrade:

[root@ipa2 ~]# find /usr /var -name tomcatjss.jar -exec ls -lZ {} \;
lrwxrwxrwx. root root system_u:object_r:usr_t:s0       /usr/share/java/tomcatjss.jar -> tomcatjss-7.1.0.jar
lrwxrwxrwx. pkiuser pkiuser system_u:object_r:pki_tomcat_var_lib_t:s0 /var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar -> /usr/share/java/tomcat7jss.jar

[root@ipa2 ~]#
Comment 36 Dean Hunter 2013-07-31 15:29:27 EDT
Please verify this summary of what we have found thus far as I find myself getting confused.

There is an error in the IPA upgrade process performed by fedup while upgrading from Fedora 18 to Fedora 19.

1. The directory server can not started or stopped during fedup.  This is an architectural issue between the use of chroot in fedup and the need to control services during the IPA upgrade.  This problem may be seen at the following locations in ipaupgrade.log.

1.1. During ipa-ldap-updater

1.1.1. Line 69: Running in chroot during /bin/systemctl stop dirsrv@HUNTER-ORG.service

1.1.2. Line 83: Running in chroot during /bin/systemctl start dirsrv.target
       Line 88: Errno 2: No such file or directory while upgrading server

1.1.3. Line 109: Running in chroot during /bin/systemctl stop dirsrv@HUNTER-ORG.service

1.1.4. Line 127: Running in chroot during /bin/systemctl is-active dirsrv@HUNTER-ORG.service
       Line 129: wait_for_open_ports: localhost [389] timeout 120

1.2. During ipa-upgradeconfig

1.2.1. Line 279: Running in chroot during /bin/systemctl stop dirsrv@HUNTER-ORG.service

1.2.2. Line 287: Running in chroot during /bin/systemctl start dirsrv@HUNTER-ORG.service

1.2.3. Line 293: Running in chroot during /bin/systemctl is-active dirsrv@HUNTER-ORG.service
       Line 295: wait_for_open_ports: localhost [389] timeout 120

It is possible to run ipa-ldap-updater and ipa-upgradeconfig successfully after booting into Fedora 19.  However there are two remaining problems when trying to start IPA.

2. The systemd environment is not configured correctly for dirsrv@PKI-IPA.service.

2.1. In /etc/sysconfig/dirsrv the following line should be removed:

     ulimit -n 8192

     This line is ignored without any messages; see bug 988858

2.2. In /etc/sysconfig/dirsrv the following line should be removed:

     export KRB5_KTNAME=/etc/dirsrv/ds.keytab

     This line is ignored and a message is logged.

2.3. There is an environment file missing that causes the start task to fail.  A message is logged without naming the missing file or directory.

3. Two symbolic links are not configured correctly for pki-tomcat@pki-tomcat.service.

3.1. /var/lib/pki/pki-tomcat/common/lib/jss4.jar -> /usr/lib/java/jss4.jar
     It points to /usr/lib64/java/jss4.jar instead and it does not exist.

3.2. /var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar -> /usr/share/java/tomcatjss.jar
     It points to /usr/share/java/tomcat7jss.jar instead and it does not exist.
Comment 37 Matthew Harmsen 2013-07-31 21:01:00 EDT
(In reply to Ade Lee from comment #33)
> Actually, I think we found a case where the code above will fail.
> 
> The case is if the link exists and is pointing to the wrong location.  In
> that case, the -e test will fail - and then ln -s will fail because the link
> already exists.
> 
> That clearly is the case for the jss jar.  In F19, the JNI directory was
> moved as indicated.
> 
> The same is not true though for tomcatjss.  Can you tell us what the
> location of the tomcatjss link was?

Per Dean Hunter's info, the tomcatjss link would have been pointing to '/usr/share/java/tomcat7jss.jar' on Fedora 18, and once the tomcatjss was updated to the version on Fedora 19, it would have left the '/var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar' still pointing to this non-existent file.  Consequently, the -e test would have failed exactly the same as in the jss case.

I believe that I have a potential solution - always attempt to remove a symbolic link prior to creating it (simply ignore the return value because it is a benign error if the symbolic link does not exist).
Comment 38 Matthew Harmsen 2013-07-31 21:04:34 EDT
Per the previous comment, I would suggest adding the following in-line patch to the code from comment #30:

(In reply to Ade Lee from comment #30)
> I fixed the wiki entry.  Thanks for the catch.
> 
> I am totally befuddled by the messages though.  The startup code (in
> /usr/share/pki/scripts/operations does the following:
> 
> if [ -e ${symlink} ]; then
>    do stuff ..
> else 
>    echo "WARNING:  Symbolic link '${symlink}' does NOT exist!"
>    # Attempt to create the symbolic link and chown it.
>    make_symlink ${symlink} ${target} ${user} ${group}
> fi
> 
> 
> make_symlink()
> {
>     symlink="${1}"
>     target="${2}"
>     user="${3}"
>     group="${4}"
> 
>     rv=0
> 
>     echo "INFO:  Attempting to create '${symlink}' -> '${target}' . . ."
>     # Check to make certain that the expected target exists.
>     #
>     #     NOTE:  The symbolic link does NOT exist at this point.
>     #
>     if [ -e ${target} ]; then
>         # Check that the expected target is fully resolvable!
>         if [ ! `readlink -qe ${target}` ]; then
>             # Issue an ERROR that the target to which the
>             # symbolic link is expected to point is NOT fully resolvable!
>             echo "ERROR:  Failed making '${symlink}' -> '${target}'"\
>                  "since target '${target}' is NOT fully resolvable!"
>             rv=1
>         else
              # Always attempt to remove any existing symbolic link that may
              # already exist and be pointing to a non-existent target
              # (ignore any return value as the symlink will usually not exist)
              rm ${symlink}
>             # Attempt to create a symbolic link and 'chown' it.
>             ln -s ${target} ${symlink}
>             rv=$?
>             if [ $rv -eq 0 ]; then
>                 # NOTE:  Ignore 'chown' errors.
>                 chown -h ${user}:${group} ${symlink}
>                 echo "SUCCESS:  Created '${symlink}' -> '${target}'"
>             else
>                 echo "ERROR:  Failed to create '${symlink}' -> '${target}'!"
>                 rv=1
>             fi
>         fi
>     else
>         # Issue an ERROR that the target to which the
>         # symbolic link is expected to point does NOT exist.
>         echo "ERROR:  Failed making '${symlink}' -> '${target}'"\
>              "since target '${target}' does NOT exist!"
>         rv=1
>     fi
> 
>     return $rv
> }
> 
> So, its only trying to create the symlink if the symlink does not exist -
> and if the target exists, but then the ln -s is failing to create the link -
> and is reporting that it does so because the file exists!
> 
> Do the links -
> 
> /var/lib/pki/pki-tomcat/common/lib/jss4.jar -> /usr/lib/java/jss4.jar 
> /var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar ->
> /usr/share/java/tomcatjss.jar
> 
> exist? (Along with their targets?)
> 
> I'll have to think more on how this could happen.  As a workaround, you can
> try to create them manually, and then try restarting the CA.
> 
> ln -s /usr/lib/java/jss4.jar /var/lib/pki/pki-tomcat/common/lib/jss4.jar
> ln -s /usr/share/java/tomcatjss.jar
> /var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar
> systemctl restart pki-tomcatd@pki-tomcat.service
Comment 39 Martin Kosek 2013-08-01 03:11:35 EDT
(In reply to Matthew Harmsen from comment #38)
> Per the previous comment, I would suggest adding the following in-line patch
> to the code from comment #30:

Ok, I will clone the bug to pki-ca so that you can track the proposed fix there.
Comment 40 Martin Kosek 2013-08-01 03:37:19 EDT
(In reply to Dean Hunter from comment #36)
> Please verify this summary of what we have found thus far as I find myself
> getting confused.
> 
> There is an error in the IPA upgrade process performed by fedup while
> upgrading from Fedora 18 to Fedora 19.
> 
> 1. The directory server can not started or stopped during fedup.  This is an
> architectural issue between the use of chroot in fedup and the need to
> control services during the IPA upgrade.  This problem may be seen at the
> following locations in ipaupgrade.log.
> 
> 1.1. During ipa-ldap-updater
> 
> 1.1.1. Line 69: Running in chroot during /bin/systemctl stop
> dirsrv@HUNTER-ORG.service
> 
> 1.1.2. Line 83: Running in chroot during /bin/systemctl start dirsrv.target
>        Line 88: Errno 2: No such file or directory while upgrading server
> 
> 1.1.3. Line 109: Running in chroot during /bin/systemctl stop
> dirsrv@HUNTER-ORG.service
> 
> 1.1.4. Line 127: Running in chroot during /bin/systemctl is-active
> dirsrv@HUNTER-ORG.service
>        Line 129: wait_for_open_ports: localhost [389] timeout 120
> 
> 1.2. During ipa-upgradeconfig
> 
> 1.2.1. Line 279: Running in chroot during /bin/systemctl stop
> dirsrv@HUNTER-ORG.service
> 
> 1.2.2. Line 287: Running in chroot during /bin/systemctl start
> dirsrv@HUNTER-ORG.service
> 
> 1.2.3. Line 293: Running in chroot during /bin/systemctl is-active
> dirsrv@HUNTER-ORG.service
>        Line 295: wait_for_open_ports: localhost [389] timeout 120
> 
> It is possible to run ipa-ldap-updater and ipa-upgradeconfig successfully
> after booting into Fedora 19.  However there are two remaining problems when
> trying to start IPA.

Yes, this is the error that this Bugzilla should fix. Either by improving upgrade instructions in documentation or some other checks in FreeIPA - I will create upstream ticket for that. But until this is fixed, the proposed workaround should be enough to fix the upgrade issue from FreeIPA point of view.

> 
> 2. The systemd environment is not configured correctly for
> dirsrv@PKI-IPA.service.

This particular service is quite suspicious actually (and benign). This dirsrv instance seems to be a left-over from previous FreeIPA installation when CA run on a separate dirsrv instance. This should not happen with clean FreeIPA installation in F18 or F19.

> 
> 2.1. In /etc/sysconfig/dirsrv the following line should be removed:
> 
>      ulimit -n 8192
> 
>      This line is ignored without any messages; see bug 988858
> 
> 2.2. In /etc/sysconfig/dirsrv the following line should be removed:
> 
>      export KRB5_KTNAME=/etc/dirsrv/ds.keytab
> 
>      This line is ignored and a message is logged.

Both issues will be fixed in scope of FreeIPA 3.3.1. Upstream ticket:
https://fedorahosted.org/freeipa/ticket/3823

> 
> 2.3. There is an environment file missing that causes the start task to
> fail.  A message is logged without naming the missing file or directory.
> 
> 3. Two symbolic links are not configured correctly for
> pki-tomcat@pki-tomcat.service.
> 
> 3.1. /var/lib/pki/pki-tomcat/common/lib/jss4.jar -> /usr/lib/java/jss4.jar
>      It points to /usr/lib64/java/jss4.jar instead and it does not exist.
> 
> 3.2. /var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar ->
> /usr/share/java/tomcatjss.jar
>      It points to /usr/share/java/tomcat7jss.jar instead and it does not
> exist.

Should be fixed by Bug 990892.

BTW, thanks for the nice summary Dean! Your patience and thorough testing helps us avoid these obstacles for other FreeIPA users.
Comment 41 Dean Hunter 2013-08-01 10:10:00 EDT
(In reply to Martin Kosek from comment #40)
> (In reply to Dean Hunter from comment #36)
> > 
> > 2. The systemd environment is not configured correctly for
> > dirsrv@PKI-IPA.service.
> 
> This particular service is quite suspicious actually (and benign). This
> dirsrv instance seems to be a left-over from previous FreeIPA installation
> when CA run on a separate dirsrv instance. This should not happen with clean
> FreeIPA installation in F18 or F19.
> 
> > 
> > 2.1. In /etc/sysconfig/dirsrv the following line should be removed:
> > 
> >      ulimit -n 8192
> > 
> >      This line is ignored without any messages; see bug 988858
> > 
> > 2.2. In /etc/sysconfig/dirsrv the following line should be removed:
> > 
> >      export KRB5_KTNAME=/etc/dirsrv/ds.keytab
> > 
> >      This line is ignored and a message is logged.
> 
> Both issues will be fixed in scope of FreeIPA 3.3.1. Upstream ticket:
> https://fedorahosted.org/freeipa/ticket/3823
> 
> > 
> > 2.3. There is an environment file missing that causes the start task to
> > fail.  A message is logged without naming the missing file or directory.
> > 

I am sorry but when I examine new builds of Fedora 18 with IPA 3.1.5-1 and Fedora 19 with IPA 3.2.2-1 I find dirsrv@PKI-IPA.service.  If the service should not be installed then the environment file errors do not need to be corrected, instead the service should be removed.  So your comments leave me confused how to proceed to clear this error:

Jul 30 14:29:42 ipa2 systemd[1]: Failed to load environment files: No such file or directory
Jul 30 14:29:42 ipa2 systemd[1]: dirsrv@PKI-IPA.service failed to run 'start' task: No such file or directory
Jul 30 14:29:42 ipa2 systemd[1]: Failed to start 389 Directory Server PKI-IPA..
Comment 42 Dean Hunter 2013-08-01 11:43:06 EDT
Ah, I see now that there are two services incorrectly configured.

2. The systemd environment is not configured correctly for two services.

2.1. For dirsrv.service

2.1.1. In /etc/sysconfig/dirsrv the following line should be removed:

       ulimit -n 8192

       This line is ignored without any messages; see bug 988858

2.1.2. In /etc/sysconfig/dirsrv the following line should be removed:

       export KRB5_KTNAME=/etc/dirsrv/ds.keytab

       This line is ignored and a message is logged.

2.2. For dirsrv@PKI-IPA.service

2.2.1. There is an environment file missing that causes the start task to fail.  A message is logged without naming the missing file or directory.

2.2.2. dirsrv@PKI-IPA.service should not be starting.  After the upgrade process a wanted-by link to this service exists that was not present before the upgrade.
Comment 43 Dean Hunter 2013-08-01 12:47:37 EDT
fedup from Fedora 18 with IPA 3.1.5-1 to Fedora 19 with IPA 3.2.2-1 adds:

  /etc/systemd/system/dirsrv.target.wants/dirsrv@PKI-IPA.service
    -> /lib/systemd/system/dirsrv@.service

This link is not present on a new build of Fedora 18 with IPA 3.1.5-1 or Fedora 19 with IPA 3.2.2-1.
Comment 44 Rob Crittenden 2013-08-01 13:05:29 EDT
Just to confirm, you don't have a /etc/dirsrv/slapd-PKI-IPA right?

I have the feeling this is the fault of freeipa-systemd-upgrade. It unconditionally enables the PKI-IPA dirsrv instance.
Comment 45 Dean Hunter 2013-08-01 13:51:14 EDT
[root@ipa2 ~]# find -L /etc -name slapd*
/etc/dirsrv/config/slapd-collations.conf
/etc/dirsrv/slapd-HUNTER-ORG
/etc/dirsrv/slapd-HUNTER-ORG/slapd-collations.conf

[root@ipa2 ~]#
Comment 46 Dean Hunter 2013-08-01 14:04:00 EDT
Here is a sequence of commands that will upgrade a new built Fedora 18 with IPA 3.1.5-1 to Fedora 19 with 3.2.2-1 and correct the problems identified to date:

# Upgrade using the local Fedora 19 repositories

  yum --assumeyes install fedup

  fedup --network 19 \
    --instrepo http://host.hunter.org/isos/fedora19/DVD

  reboot

# Work around FreeIPA upgrade problems

  sed -i /^ulimit/d /etc/sysconfig/dirsrv
  sed -i /^export/d /etc/sysconfig/dirsrv
  cat               /etc/sysconfig/dirsrv
    
  rm -f /etc/systemd/system/dirsrv.target.wants/dirsrv@PKI-IPA.service
  find -L /etc /usr -name dirsrv* -type f | sort
  
  /usr/sbin/ipa-ldap-updater --upgrade
  /usr/sbin/ipa-upgradeconfig

  rm -f /var/lib/pki/pki-tomcat/common/lib/jss4.jar
  find -L /var /usr -name jss4.jar | sort

  rm -f /var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar
  find -L /var /usr -name tomcatjss.jar | sort

# Verify IPA will stop and start

  ipactl start
  ipactl stop
Comment 47 Ade Lee 2013-08-02 01:11:14 EDT
Trac ticket 699 (https://fedorahosted.org/pki/ticket/699) has been created to track this issue for Dogtag.

pki-core-10.0.4-2 has been created to address this issue.  Please test it out and provide karma to help move the  build to the stable repo.

With this update, the following steps:

 rm -f /var/lib/pki/pki-tomcat/common/lib/jss4.jar
  find -L /var /usr -name jss4.jar | sort

  rm -f /var/lib/pki/pki-tomcat/common/lib/tomcatjss.jar
  find -L /var /usr -name tomcatjss.jar | sort

are no longer required.  The links should fix themselves upon restarting the dogtag instance.
Comment 48 Dean Hunter 2013-08-05 12:10:02 EDT
(In reply to Ade Lee from comment #47)
> Trac ticket 699 (https://fedorahosted.org/pki/ticket/699) has been created
> to track this issue for Dogtag.
> 
> pki-core-10.0.4-2 has been created to address this issue.  Please test it
> out and provide karma to help move the  build to the stable repo.

Where is the sequence of commands from comment 46 should I update to 
pki-core-10.0.4-2?

1. Should I use pki-core-10.0.4-2.fc18 before fedup?

2. Should I use pki-core-10.0.4-2.fc19 after the reboot and before /usr/sbin/ipa-ldap-updater?

3. I guess I could even do it once each way?
Comment 49 Martin Kosek 2013-08-05 12:34:15 EDT
Ade may specify this more, but IMO doing just 2) is enough. However, you can do both 1) and 2), just to be sure.
Comment 50 Martin Kosek 2013-08-06 03:19:01 EDT
Cloning the Bugzilla to FreeIPA upstrem Trac to solve the fedup upgrade issue.

We discussed this issue on our team meeting and we think that the best way would be to:

1. When RPM upgrade succeeds, it writes the upstream version ID to system (/var/lib/ipa/sysupgrade/sysupgrade.state). When it fails, version is not written
2. Next time, when ipa service starts and detects that upgrade was not performed (comparing it's version with stored one), it would execute the upgrade before IPA starts
Comment 51 Martin Kosek 2013-08-06 03:22:26 EDT
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/3849
Comment 52 Dean Hunter 2013-08-06 14:51:38 EDT
# cd /srv/http/repos/fedora$v/updates-testing
# koji download-build --arch x86_64 pki-core-10.0.4-2.fc$v
# cd -
# createrepo /srv/http/repos/fedora$v/updates-testing

retrieves two packaes:

  pki-symkey-10.0.4-2.fc18.x86_64.rpm
  pki-tools-10.0.4-2.fc18.x86_64.rpm

When I try to update these packages then a number of dependency errors are displayed:

# yum --assumeyes --enablerepo local-testing pki-symkey pki-tools
...
--> Finished Dependency Resolution
Error: Package: pki-tools-10.0.4-2.fc18.x86_64 (local-testing)
           Requires: pki-base = 10.0.4-2.fc18
           Installed: pki-base-10.0.3-2.fc18.noarch (@local-updates)
               pki-base = 10.0.3-2.fc18
           Available: pki-base-10.0.0-1.fc18.noarch (local-fedora)
               pki-base = 10.0.0-1.fc18
           Available: pki-base-10.0.4-1.fc18.noarch (local-testing)
               pki-base = 10.0.4-1.fc18
Error: Package: pki-server-10.0.4-1.fc18.noarch (local-testing)
           Requires: pki-tools = 10.0.4-1.fc18
           Removing: pki-tools-10.0.3-2.fc18.x86_64 (@local-updates)
               pki-tools = 10.0.3-2.fc18
           Updated By: pki-tools-10.0.4-2.fc18.x86_64 (local-testing)
               pki-tools = 10.0.4-2.fc18
           Available: pki-tools-10.0.0-1.fc18.x86_64 (local-fedora)
               pki-tools = 10.0.0-1.fc18
           Available: pki-tools-10.0.4-1.fc18.x86_64 (local-testing)
               pki-tools = 10.0.4-1.fc18
Error: Package: pki-tools-10.0.4-2.fc18.x86_64 (local-testing)
           Requires: pki-base = 10.0.4-2.fc18
           Removing: pki-base-10.0.3-2.fc18.noarch (@local-updates)
               pki-base = 10.0.3-2.fc18
           Updated By: pki-base-10.0.4-1.fc18.noarch (local-testing)
               pki-base = 10.0.4-1.fc18
           Available: pki-base-10.0.0-1.fc18.noarch (local-fedora)
               pki-base = 10.0.0-1.fc18
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

What have I done wrong?
Comment 53 Rob Crittenden 2013-08-06 15:17:10 EDT
There are required noarch packages too.
Comment 54 Dean Hunter 2013-08-06 15:35:32 EDT
I need instructions, please.
Comment 55 Martin Kosek 2013-08-07 02:16:06 EDT
(In reply to Dean Hunter from comment #52)
> # koji download-build --arch x86_64 pki-core-10.0.4-2.fc$v

This ^^ downloads just packages with x86_64 architecture. However, some of the pki packages are architecture independent, i.e. "noarch" architecture.

To download both, replace the command above with:
# koji download-build --arch x86_64 --arch noarch pki-core-10.0.4-2.fc$v
Comment 56 Dean Hunter 2013-08-07 09:26:23 EDT
Ah!  Thank you.

Which of these packages that I should designate on the "yum update" statement?

[root@host ~]# ls -l /srv/http/repos/fedora18/koji
total 6864
-rw-r--r--. 1 root root  910216 Aug  7 08:01 pki-base-10.0.4-2.fc18.noarch.rpm
-rw-r--r--. 1 root root  429132 Aug  7 08:01 pki-ca-10.0.4-2.fc18.noarch.rpm
-rw-r--r--. 1 root root 1962556 Aug  7 08:01 pki-javadoc-10.0.4-2.fc18.noarch.rpm
-rw-r--r--. 1 root root  213276 Aug  7 08:01 pki-kra-10.0.4-2.fc18.noarch.rpm
-rw-r--r--. 1 root root  132616 Aug  7 08:01 pki-ocsp-10.0.4-2.fc18.noarch.rpm
-rw-r--r--. 1 root root 2786744 Aug  7 08:01 pki-server-10.0.4-2.fc18.noarch.rpm
-rw-r--r--. 1 root root   84184 Aug  7 08:01 pki-symkey-10.0.4-2.fc18.x86_64.rpm
-rw-r--r--. 1 root root  116904 Aug  7 08:01 pki-tks-10.0.4-2.fc18.noarch.rpm
-rw-r--r--. 1 root root  366012 Aug  7 08:01 pki-tools-10.0.4-2.fc18.x86_64.rpm
drwxr-xr-x. 2 root root    4096 Aug  7 08:01 repodata
[root@host ~]#
Comment 57 Martin Kosek 2013-08-07 09:36:35 EDT
Doesn't simple "yum update /srv/http/repos/fedora18/koji/pki-*.rpm" work? It should skip pki subpackages you do not have installed on your VM.
Comment 58 Dean Hunter 2013-08-07 12:53:12 EDT
I see that pki-core-10.0.4-2 is now available in updates-testing.  I would rather use the repository than what I downloaded from koji.  What packages should I update?

  yum --assumeyes --enablerepo updates-testing update ???
Comment 59 Rob Crittenden 2013-08-07 13:09:23 EDT
pki-ca should pull in any needed dependencies.
Comment 60 Dean Hunter 2013-08-07 14:23:03 EDT
Verified correction for pki 699 and updated karma.

Now my script looks like:

# Upgrade to Fedora 19 from Fedora 18

  yum --assumeyes install fedup
 
  fedup --network 19 \
    --instrepo http://host.hunter.org/isos/fedora19/DVD

  reboot

# Work around FreeIPA upgrade problems

# freeipa 3823: invalid statements in systemd environment

  sed -i /^ulimit/d /etc/sysconfig/dirsrv
  sed -i /^export/d /etc/sysconfig/dirsrv
  cat               /etc/sysconfig/dirsrv

# freeipa ????: orphaned systemd service dependency

  rm -f /etc/systemd/system/dirsrv.target.wants/dirsrv@PKI-IPA.service
  find -L /etc /usr -name dirsrv* -type f | sort

# pki 699: can not update broken links

  yum --assumeyes --enablerepo local-testing update pki-ca
  find -L /var /usr -name jss4.jar -exec ls -l {} \;
  find -L /var /usr -name tomcatjss.jar -exec ls -l {} \;

# freeipa 3849: can not upgrade ipa while running in chroot

  /usr/sbin/ipa-ldap-updater --upgrade
  /usr/sbin/ipa-upgradeconfig

# Verify IPA will start

  reboot
Comment 61 Dean Hunter 2013-08-11 17:32:38 EDT
Starting with Fedora 18 and FreeIPA 3.1.5-1 and upgrading to Fedora 19 and FreeIPA 3.3.0-1:

# Upgrade to Fedora 19 from Fedora 18

  yum --assumeyes install fedup
 
  fedup --network 19 \
    --instrepo http://host.hunter.org/isos/fedora19/DVD

  reboot

# Work arounds for FreeIPA upgrade problems

  yum --assumeyes --enablerepo local-testing \
    update pki-ca                                # pki-ca-10.0.4-2

  yum --assumeyes --enablerepo local-testing \
    update bind bind-dyndb-ldap freeipa-server   # freeipa-server-3.3.0-1
...
  Updating   : freeipa-server-3.3.0-1.fc19.x86_64                         19/37 
Found IPA server for domain HUNTER.ORG
Converting services setup to systemd
    Upgrade /etc/sysconfig/dirsrv
    Upgrade /etc/sysconfig/krb5kdc
    Re-enable Directory server instances PKI-IPA and HUNTER-ORG 
    Re-enable IPA service
Finished.
  Cleanup    : freeipa-server-3.2.2-1.fc19.x86_64                         20/37 
...

and then yum never completes!

14660_13126,cn=index,cn=tasks,cn=config
[root@ipa2 ~]# tail -20 /var/log/ipaupgrade.log
2013-08-11T21:15:54Z DEBUG only: updated value [u'eq', u'pres', u'sub']
2013-08-11T21:15:54Z DEBUG ---------------------------------------------
2013-08-11T21:15:54Z DEBUG Final value after applying updates
2013-08-11T21:15:54Z DEBUG dn: cn=ipasudorunasgroup,cn=index,cn=userRoot,cn=ldbm database,cn=plugins,cn=config
2013-08-11T21:15:54Z DEBUG objectClass:
2013-08-11T21:15:54Z DEBUG 	top
2013-08-11T21:15:54Z DEBUG 	nsIndex
2013-08-11T21:15:54Z DEBUG nsIndexType:
2013-08-11T21:15:54Z DEBUG 	eq
2013-08-11T21:15:54Z DEBUG 	pres
2013-08-11T21:15:54Z DEBUG 	sub
2013-08-11T21:15:54Z DEBUG cn:
2013-08-11T21:15:54Z DEBUG 	ipasudorunasgroup
2013-08-11T21:15:54Z DEBUG nsSystemIndex:
2013-08-11T21:15:54Z DEBUG 	false
2013-08-11T21:15:54Z DEBUG [(0, u'nsindextype', ['sub'])]
2013-08-11T21:15:54Z DEBUG Live 1, updated 1
2013-08-11T21:15:54Z INFO Done
2013-08-11T21:15:59Z INFO Creating task to index attribute: ipasudorunasgroup
2013-08-11T21:15:59Z DEBUG Task id: cn=indextask_ipasudorunasgroup_135955485596414660_13126,cn=index,cn=tasks,cn=config
[root@ipa2 ~]#
Comment 62 Martin Kosek 2013-08-12 03:00:37 EDT
This sounds serious, thanks for reporting! Sounds like a DS deadlock.

I did a little Bugzilla research and found a related Bug which had a similar behavior - Bug 923407. We should check it did not regress. Adding Noriko and Rich to CC - could you please provide help with debugging the DS side of the issue?
Comment 63 Rich Megginson 2013-08-12 11:24:48 EDT
Can we get the errors and access log files from the directory server - /var/log/dirsrv/slapd-DOMAIN/{access,errors}

Is the directory server hung?  If so, please follow these steps to get a stack trace of the hung process:
http://port389.org/wiki/FAQ#Debugging_Crashes
http://port389.org/wiki/FAQ#Debugging_Hangs
Comment 64 Dean Hunter 2013-08-13 12:50:26 EDT
This sequence works:

# Upgrade to Fedora 19 from Fedora 18

  yum --assumeyes install fedup
 
  fedup --network 19 \
    --instrepo http://host.hunter.org/isos/fedora19/DVD

  reboot

# Work arounds for FreeIPA server upgrade problems

# pki 699

  yum --assumeyes --enablerepo local-testing \
    update pki-ca                                # pki-ca-10.0.4-2

# freeipa 3823: invalid systemd environment for dirsrv

  sed -i /^ulimit/d /etc/sysconfig/dirsrv
  sed -i /^export/d /etc/sysconfig/dirsrv

# freeipa ????: orphaned systemd service dependency for dirsrv

  rm -f /etc/systemd/system/dirsrv.target.wants/dirsrv@PKI-IPA.service

# freeipa 3849: can not upgrade ipa while running in chroot

  /usr/sbin/ipa-ldap-updater --upgrade
  /usr/sbin/ipa-upgradeconfig

# Verify IPA will start and is working correctly

# Upgrade to freeipa 3.3.0-1

  yum --assumeyes --enablerepo local-testing \
    update bind bind-dyndb-ldap freeipa-server   # freeipa-server-3.3.0-1

# freeipa ????: orphaned systemd service dependency for dirsrv

  rm -f /etc/systemd/system/dirsrv.target.wants/dirsrv@PKI-IPA.service
Comment 65 Dean Hunter 2013-08-13 12:53:27 EDT
(In reply to Rich Megginson from comment #63)
> Can we get the errors and access log files from the directory server -
> /var/log/dirsrv/slapd-DOMAIN/{access,errors}

I will work on this today. I will need to rebuild/restore

> Is the directory server hung?  If so, please follow these steps to get a
> stack trace of the hung process:
> http://port389.org/wiki/FAQ#Debugging_Crashes
> http://port389.org/wiki/FAQ#Debugging_Hangs

How should I test whether the directory server is hung?
Comment 66 Rich Megginson 2013-08-13 12:59:04 EDT
(In reply to Dean Hunter from comment #65)
> (In reply to Rich Megginson from comment #63)
> > Can we get the errors and access log files from the directory server -
> > /var/log/dirsrv/slapd-DOMAIN/{access,errors}
> 
> I will work on this today. I will need to rebuild/restore
> 
> > Is the directory server hung?  If so, please follow these steps to get a
> > stack trace of the hung process:
> > http://port389.org/wiki/FAQ#Debugging_Crashes
> > http://port389.org/wiki/FAQ#Debugging_Hangs
> 
> How should I test whether the directory server is hung?

ldapsearch -xLLL -D "cn=directory manager" -W -s base -b "" 'objectclass=*' vendorVersion

If this returns the vendorVersion value, then the server is not completely hung.

However, if yum upgrade is stalled, and it appears to be the fault of the directory server, then it may be that the directory server is deadlocked but not hung, and follow http://port389.org/wiki/FAQ#Debugging_Hangs and http://port389.org/wiki/FAQ#Debugging_Crashes so that we can get a stack trace to examine.
Comment 67 Rich Megginson 2013-08-13 13:00:28 EDT
(In reply to Dean Hunter from comment #64)
> This sequence works:
> 
> # Upgrade to Fedora 19 from Fedora 18
> 
>   yum --assumeyes install fedup
>  
>   fedup --network 19 \
>     --instrepo http://host.hunter.org/isos/fedora19/DVD
> 
>   reboot
> 
> # Work arounds for FreeIPA server upgrade problems
> 
> # pki 699
> 
>   yum --assumeyes --enablerepo local-testing \
>     update pki-ca                                # pki-ca-10.0.4-2
> 
> # freeipa 3823: invalid systemd environment for dirsrv
> 
>   sed -i /^ulimit/d /etc/sysconfig/dirsrv
>   sed -i /^export/d /etc/sysconfig/dirsrv
> 
> # freeipa ????: orphaned systemd service dependency for dirsrv
> 
>   rm -f /etc/systemd/system/dirsrv.target.wants/dirsrv@PKI-IPA.service
> 
> # freeipa 3849: can not upgrade ipa while running in chroot
> 
>   /usr/sbin/ipa-ldap-updater --upgrade
>   /usr/sbin/ipa-upgradeconfig
> 
> # Verify IPA will start and is working correctly
> 
> # Upgrade to freeipa 3.3.0-1
> 
>   yum --assumeyes --enablerepo local-testing \
>     update bind bind-dyndb-ldap freeipa-server   # freeipa-server-3.3.0-1
> 
> # freeipa ????: orphaned systemd service dependency for dirsrv
> 
>   rm -f /etc/systemd/system/dirsrv.target.wants/dirsrv@PKI-IPA.service

Are all of these issues fixed in ipa 3.3?  If so, perhaps you can try to upgrade to ipa 3.3 instead?
Comment 68 Dean Hunter 2013-08-13 13:23:21 EDT
Created attachment 786218 [details]
/var/log/dirsrv/slapd-HUNTER-ORG access and errors
Comment 69 Rich Megginson 2013-08-13 13:40:16 EDT
(In reply to Dean Hunter from comment #68)
> Created attachment 786218 [details]
> /var/log/dirsrv/slapd-HUNTER-ORG access and errors

Nothing interesting in the errors log.  There is a request without a result in the access log, which could indicate a hang or deadlock.  Need a stack trace.
Comment 70 Dean Hunter 2013-08-13 13:56:13 EDT
(In reply to Rich Megginson from comment #67)
> (In reply to Dean Hunter from comment #64)
> 
> Are all of these issues fixed in ipa 3.3?  If so, perhaps you can try to
> upgrade to ipa 3.3 instead?

None of these issues are fixed in IPA 3.3.  The last time I looked they were scheduled for IPA 3.4.  However, as soon as IPA 3.3 moves to the Fedora Updates repository I will test the upgrade from IPA 3.1 to IPA 3.3 because I do not know how to tell fedup to get freeipa-server 3.3 from updates-testing.

The requested stack trace will be available shortly.
Comment 71 Dean Hunter 2013-09-04 15:47:13 EDT
(In reply to Rich Megginson from comment #69)
> (In reply to Dean Hunter from comment #68)
> > Created attachment 786218 [details]
> > /var/log/dirsrv/slapd-HUNTER-ORG access and errors
> 
> Nothing interesting in the errors log.  There is a request without a result
> in the access log, which could indicate a hang or deadlock.  Need a stack
> trace.

I am sorry.  I was called out of town for several weeks and now I can not build a viable Fedora 18 VM to test the upgrade.  A new build with all updates applied has problems with:

1. starting gdm
2. the label on /var/tmp/abrt
3. audispd event queue full

I will get the stack trace as soon as I can.
Comment 72 Martin Kosek 2014-01-28 02:53:24 EST
*** Bug 1058222 has been marked as a duplicate of this bug. ***
Comment 73 Fedora End Of Life 2015-01-09 17:13:43 EST
This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 74 Martin Kosek 2015-01-15 10:29:56 EST
FreeIPA does not support upgrades from unsupported Fedora releases, see Comment 73. I am thus closing this bug. If you are still experiencing upgrade issues with supported Fedora versions (20, 21), please file a new bug.

Note You need to log in before you can comment on or make changes to this bug.