Bug 1180325

Summary: RHEL 7.1 ipa-server-4.1.0 upgrade fails
Product: Red Hat Enterprise Linux 7 Reporter: Josh Baird <jbaird>
Component: 389-ds-baseAssignee: Noriko Hosoi <nhosoi>
Status: CLOSED ERRATA QA Contact: Viktor Ashirov <vashirov>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 7.1CC: dpal, drieden, jbaird, lkrispen, mkosek, nkinder, nsoman, parsonsa, pvoborni, rcritten, rmeggins, spoore, tbordaz
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 389-ds-base-1.3.3.1-12.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-03-05 09:40:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
/var/log/ipaupgrade.log
none
/var/log/dirsrv/slapd*/errors
none
journalctl
none
rpm -qa post upgrade
none
rpm -qa pre upgrade
none
dirsrv logs during upgrade to RHEL 7.1
none
/var/log/messages none

Description Josh Baird 2015-01-08 20:48:09 UTC 
Description of problem:

When upgrading from RHEL 7.0 ipa-server-3.3.3 to RHEL 7.1 ipa-server-4.1.0. the 'yum update' fails due to the following error:

 Cleanup    : glibc-common-2.17-55.el7_0.3.x86_64                                                                                       540/543
  Cleanup    : nss-softokn-freebl-3.16.2.3-1.el7_0.x86_64                                                                                541/543
  Cleanup    : glibc-2.17-55.el7_0.3.x86_64                                                                                              542/543
  Cleanup    : libgcc-4.8.2-16.2.el7_0.x86_64                                                                                            543/543
2734 blocks
^C^CPre schema upgrade failed with [Errno 111] Connection refused
[Errno 111] Connection refused
Cancelled.
warning: %posttrans(ipa-server-4.1.0-10.el7.x86_64) scriptlet failed, signal 2
Non-fatal POSTTRANS scriptlet failure in rpm package ipa-server-4.1.0-10.el7.x86_64

IPA was in a running state when the 'yum update' was executed.  This failure leaves the new kernel installed, but no initramfs built which causes the next reboot to fail.

Version-Release number of selected component (if applicable):


How reproducible:

Reproduced on two systems running RHEL 7.0 with an attempted update to RHEL 7.1 beta.

Steps to Reproduce:
1. Locate RHEL 7.0 system with ipa-server installed.
2. Execute 'yum update' against RHEL 7.1 beta repos.

Actual results:

'yum update' will freeze and upon forcing an exit, the error messages that %posttrans failed on ipa-server-4.1.0 appear.  The server will then fail to boot into the new kernel because the initramfs for the new kernel was not built.

Expected results:

Server should be updated to 7.1, ipa-server should be updated, server should boot into the new 7.1 kernel.

Additional info:
Comment 2 Martin Kosek 2015-01-09 12:57:08 UTC 
Thanks for the report. I doubt that the FreeIPA upgrade error would cause kernel update problems though, it sounds as a different issue.

Even RPM seems to admin that the RPM snippet failure was not fatal for it:
Non-fatal POSTTRANS scriptlet failure in rpm package ipa-server-4.1.0-10.el7.x86_64

I wonder where the
[Errno 111] Connection refused
comes from. Are there any SELinux AVCs? Any useful errors in /var/log/ipaupgrade.log?

Martin, does this error sound familiar? You worked on the related update, so I am hoping you can find what's wrong there.
Comment 3 Josh Baird 2015-01-09 13:27:39 UTC 
SELinux is disabled on these servers.  I have attached ipaupgrade.log which seems to indicate that the CA cannot start.

2015-01-09T13:20:31Z DEBUG The CA status is: check interrupted
2015-01-09T13:20:31Z DEBUG Waiting for CA to start...
2015-01-09T13:20:32Z DEBUG request 'https://imqa-d1-dc01.qa-unix.follett.com:443/ca/admin/ca/getStatus'
2015-01-09T13:20:32Z DEBUG request body ''
2015-01-09T13:20:32Z DEBUG request status 500
2015-01-09T13:20:32Z DEBUG request reason_phrase u'Internal Server Error'

I believe the "connection refused" messages are in reference to the LDAP server which appears to be down while the upgrade is attempting to update the schema.  The LDAP server was running successfully before the 'yum update' was executed.

This loops indefinitely which causes the %posttrans script to never complete.  The following is logged to /var/log/messages during this:

Jan  9 07:20:40 imqa-d1-dc01 named[31491]: connection to the LDAP server was lost
Jan  9 07:20:40 imqa-d1-dc01 named[31491]: Failed to get initial credentials (TGT) using principal 'DNS/imqa-d1-dc01.qa-unix.follett.com' and keytab 'FILE:/etc/named.keytab' (Cannot contact any KDC for realm 'QA-UNIX.FOLLETT.COM')

At this point, the new kernel package (kernel-3.10.0-210) is installed, but no initramfs exists in /boot.  This may very well be another bug and/or issue.
Comment 4 Josh Baird 2015-01-09 13:28:17 UTC 
Created attachment 978149 [details]
/var/log/ipaupgrade.log
Comment 5 Martin Kosek 2015-01-09 13:41:18 UTC 
Looking at the log, it looks like the initial problem is indeed the fact that LDAP server cannot be contacted. This reveals the first upgrade issue:

2015-01-09T13:19:10Z DEBUG The ipa-ldap-updater command failed, exception: error: [Errno 111] Connection refused
2015-01-09T13:19:10Z ERROR [Errno 111] Connection refused
2015-01-09T13:19:10Z DEBUG /usr/sbin/ipa-upgradeconfig was invoked with options: {'debug': False, 'quiet': True}


I do not think that second upgrade step (ipa-upgradeconfig) should be run if the first step failed. This is something we can fix pretty easily.

But the main question is why the LDAP server cannot be contacted. Can you please check if there are any errors in

# journalctl --unit dirsrv --since=today

or

/var/log/dirsrv/errors logs?
Comment 6 Josh Baird 2015-01-09 14:46:42 UTC 
I reverted the VM snapshot to 7.0 to ensure that 389 was running correctly before the upgrade.  This is a vanilla ipa-server install.

# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful

However, /var/log/dirsrv/error shows this:

[09/Jan/2015:08:24:30 -0600] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is not connected)
[09/Jan/2015:08:24:30 -0600] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -1 (Can't contact LDAP server)

# journalctl --unit dirsrv --since=today
-- Logs begin at Wed 2015-01-07 12:12:01 CST, end at Fri 2015-01-09 08:23:51 CST. --

It does look like 389 is functional because it's access log shows responses:

09/Jan/2015:08:39:02 -0600] conn=10 op=1349 RESULT err=0 tag=101 nentries=0 etime=0
[09/Jan/2015:08:39:02 -0600] conn=2426 op=1 UNBIND

Next, I will 'yum update' to update the system to 7.1.  At this point, 'yum' freezes once again.  I have attached both /var/log/dirsrv/error and the output of journalctl.

root@imqa-d1-dc01:~# ipactl status
Directory Service: STOPPED
krb5kdc Service: STOPPED
kadmin Service: STOPPED
named Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful
Comment 7 Josh Baird 2015-01-09 14:47:09 UTC 
Created attachment 978185 [details]
/var/log/dirsrv/slapd*/errors
Comment 8 Josh Baird 2015-01-09 14:47:33 UTC 
Created attachment 978186 [details]
journalctl
Comment 9 Martin Kosek 2015-01-09 14:57:41 UTC 
Looking at the journal, it almost looks like as the 389 DS crashed:

Jan 09 08:43:20 imqa-d1-dc01.qa-unix.follett.com systemd[1]: dirsrv: main process exited, code=exited, status=1/FAILURE
Jan 09 08:43:20 imqa-d1-dc01.qa-unix.follett.com systemd[1]: Unit dirsrv entered failed state.


Thierry or Ludwig, can you please advise?
Comment 10 Ludwig 2015-01-09 15:13:48 UTC 
I think the DS does not start because of a schema problem.
[09/Jan/2015:08:44:09 -0600] - Entry "cn=replSchema,cn=config" has unknown object class "nsSchemaPolicy"
[09/Jan/2015:08:44:09 -0600] NSMMReplicationPlugin - Warning: unable to create configuration entry cn=replSchema, cn=config: Object class violation

I need to check in which version this objectclass was introduced and then why it is not in the schema which is deployed
Comment 11 Martin Kosek 2015-01-09 15:23:04 UTC 
BTW, Josh, what is the version of 389-ds-base you are updating to? I assume you are updating to 7.1 Beta packages...
Comment 12 Josh Baird 2015-01-09 15:24:45 UTC 
Yeah, I'm updating using the 7.1 beta repo.

# rpm -qa | grep 389
389-ds-base-1.3.3.1-9.el7.x86_64
389-ds-base-libs-1.3.3.1-9.el7.x86_64
Comment 13 thierry bordaz 2015-01-09 15:26:03 UTC 
Ludwig,

This objectclass has been added in https://fedorahosted.org/389/ticket/47676. This is present since 1.3.2
Comment 14 Ludwig 2015-01-09 15:34:55 UTC 
Josh, can you check your 
/etc/dirsrv/slapd-<instance>/schema/01core389.ldif if it contains nsSchemaPolicy ?
Comment 15 Josh Baird 2015-01-09 15:37:26 UTC 
Doesn't look like it.

root@imqa-d1-dc01:/etc/dirsrv/slapd-QA-UNIX-FOLLETT-COM/schema# grep nsSchemaPolicy 01core389.ldif
root@imqa-d1-dc01:/etc/dirsrv/slapd-QA-UNIX-FOLLETT-COM/schema#
Comment 16 Ludwig 2015-01-09 16:53:13 UTC 
could you also check /etc/dirsrv/schema/01core389.ldif
what timestamps do the files in this directory have ?
Comment 17 Josh Baird 2015-01-09 17:29:53 UTC 
root@imqa-d1-dc01:/etc/dirsrv/schema# ll
total 364
-rw-r--r-- 1 root root 26733 Nov 13 17:57 00core.ldif
-rw-r--r-- 1 root root 59379 Nov 13 17:57 01core389.ldif
-rw-r--r-- 1 root root 33615 Nov 13 17:57 02common.ldif
-rw-r--r-- 1 root root  2620 Nov 13 17:57 05rfc2927.ldif
-rw-r--r-- 1 root root  5854 Nov 13 17:57 05rfc4523.ldif
-rw-r--r-- 1 root root 10481 Nov 13 17:57 05rfc4524.ldif
-rw-r--r-- 1 root root  4750 Nov 13 17:57 06inetorgperson.ldif
-rw-r--r-- 1 root root  5204 Nov 13 17:57 10automember-plugin.ldif
-rw-r--r-- 1 root root  8954 Nov 13 17:57 10dna-plugin.ldif
-rw-r--r-- 1 root root  4331 Nov 13 17:57 10mep-plugin.ldif
-rw-r--r-- 1 root root  9211 Nov 13 17:57 10rfc2307.ldif
-rw-r--r-- 1 root root  6385 Nov 13 17:57 20subscriber.ldif
-rw-r--r-- 1 root root  4624 Nov 13 17:57 25java-object.ldif
-rw-r--r-- 1 root root  3645 Nov 13 17:57 28pilot.ldif
-rw-r--r-- 1 root root 10833 Nov 13 17:57 30ns-common.ldif
-rw-r--r-- 1 root root  8374 Nov 13 17:57 50ns-admin.ldif
-rw-r--r-- 1 root root  2866 Nov 13 17:57 50ns-certificate.ldif
-rw-r--r-- 1 root root 18323 Nov 13 17:57 50ns-directory.ldif
-rw-r--r-- 1 root root 10821 Nov 13 17:57 50ns-mail.ldif
-rw-r--r-- 1 root root  4776 Nov 13 17:57 50ns-value.ldif
-rw-r--r-- 1 root root  2865 Nov 13 17:57 50ns-web.ldif
-rw-r--r-- 1 root root  1948 Nov 13 17:57 60acctpolicy.ldif
-rw-r--r-- 1 root root  1129 Nov 13 17:57 60autofs.ldif
-rw-r--r-- 1 root root  3311 Nov 13 17:57 60eduperson.ldif
-rw-r--r-- 1 root root  6856 Nov 13 17:57 60mozilla.ldif
-rw-r--r-- 1 root root   741 Nov 13 17:57 60nss-ldap.ldif
-rw-r--r-- 1 root root  4269 Nov 13 17:57 60pam-plugin.ldif
-rw-r--r-- 1 root root  2640 Nov 13 17:57 60posix-winsync-plugin.ldif
-rw-r--r-- 1 root root  3552 Nov 13 17:57 60pureftpd.ldif
-rw-r--r-- 1 root root  3497 Nov 13 17:57 60rfc2739.ldif
-rw-r--r-- 1 root root 15312 Nov 13 17:57 60rfc3712.ldif
-rw-r--r-- 1 root root  2045 Nov 13 17:57 60sabayon.ldif
-rw-r--r-- 1 root root  3611 Nov 13 17:57 60sudo.ldif
-rw-r--r-- 1 root root  1281 Nov 13 17:57 60trust.ldif
-rw-r--r-- 1 root root  2210 Nov 13 17:57 99user.ldif

root@imqa-d1-dc01:/etc/dirsrv/schema# grep nsSchemaPolicy *
01core389.ldif:objectClasses: ( 2.16.840.1.113730.3.2.328 NAME 'nsSchemaPolicy' DESC 'Netscape defined objectclass' SUP top  MAY ( cn $ schemaUpdateObjectclassAccept $ schemaUpdateObjectclassReject $ schemaUpdateAttributeAccept $ schemaUpdateAttributeReject) X-ORIGIN 'Netscape Directory Server' )
Comment 18 Ludwig 2015-01-09 17:39:04 UTC 
ok, so the correct file seems to have been installed, but failed to be copied to the schema directory of the instance. 
Hard to tell why this happened. Couls you coy this schema file manually and see if your upgrade woudl maek more progress ?
Comment 19 Josh Baird 2015-01-09 17:59:10 UTC 
How should I resume the upgrade once I copy this file?
Comment 20 Josh Baird 2015-01-09 18:06:54 UTC 
OK - I copied the file, and then re-ran 'ipa-upgradeconfig':

root@imqa-d1-dc01:/etc/dirsrv/schema# ipa-upgradeconfig
[Verifying that root certificate is published]
Failed to backup CS.cfg: Dogtag must be stopped when creating backup of /var/lib/pki/pki-tomcat/conf/ca/CS.cfg
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that CA proxy configuration is correct]
[Verifying that KDC configuration is using ipa-kdb backend]
Restoring /etc/sysconfig/httpd as it is no longer required
[Updating mod_nss protocol versions]
[Fixing trust flags in /etc/httpd/alias]
[Fix DS schema file syntax]
[Removing RA cert from DS NSS database]
[Removing self-signed CA]
[Checking for deprecated KDC configuration files]
[Checking for deprecated backups of Samba configuration files]
[Setting up Firefox extension]
[Add missing CA DNS records]
DNS is not configured
[Removing deprecated DNS configuration options]
[Ensuring minimal number of connections]
[Enabling serial autoincrement in DNS]
[Updating GSSAPI configuration in DNS]
[Updating pid-file configuration in DNS]
[Enabling "dnssec-enable" configuration in DNS]
[Setting "bindkeys-file" option in named.conf]
[Setting "managed-keys-directory" option in named.conf]
[Including named root key in named.conf]
Changes to named.conf have been made, restart named
[Verifying that CA service certificate profile is updated]
[Update certmonger certificate renewal configuration to version 2]
Certmonger certificate renewal configuration updated to version 2
[Enable PKIX certificate path discovery and validation]
pki-ca configuration changed, restart pki-ca
The ipa-upgradeconfig command was successful

root@imqa-d1-dc01:/etc/dirsrv/schema# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful
Comment 21 thierry bordaz 2015-01-09 18:30:14 UTC 
The problem is that during the update procedure, the file /etc/dirsrv/schema/01core389.ldif was updated but not the file the /etc/dirsrv/slapd-QA-UNIX-FOLLETT-COM/schema/01core389.ldif.

This problem was already seen in https://bugzilla.redhat.com/show_bug.cgi?id=1113022#c17 where we failed to identify a reproducible test case.
Comment 22 Josh Baird 2015-01-09 18:32:08 UTC 
I can reproduce this over and over on two separate servers.  Now that the upgrade completed, I still cannot login to IdM:

[Fri Jan 09 12:16:37.495361 2015] [:error] [pid 2103] ipa: ERROR: non-public: KeyError: 'idnsforwardzone'
[Fri Jan 09 12:16:37.495448 2015] [:error] [pid 2103] Traceback (most recent call last):
[Fri Jan 09 12:16:37.495454 2015] [:error] [pid 2103]   File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 348, in wsgi_execute
[Fri Jan 09 12:16:37.495457 2015] [:error] [pid 2103]     result = self.Command[name](*args, **options)
[Fri Jan 09 12:16:37.495460 2015] [:error] [pid 2103]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 439, in __call__
[Fri Jan 09 12:16:37.495463 2015] [:error] [pid 2103]     ret = self.run(*args, **options)
[Fri Jan 09 12:16:37.495465 2015] [:error] [pid 2103]   File "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 754, in run
[Fri Jan 09 12:16:37.495469 2015] [:error] [pid 2103]     return self.execute(*args, **options)
[Fri Jan 09 12:16:37.495471 2015] [:error] [pid 2103]   File "/usr/lib/python2.7/site-packages/ipalib/plugins/internal.py", line 123, in execute
[Fri Jan 09 12:16:37.495474 2015] [:error] [pid 2103]     (o.name, json_serialize(o)) for o in self.api.Object()
[Fri Jan 09 12:16:37.495503 2015] [:error] [pid 2103]   File "/usr/lib/python2.7/site-packages/ipalib/plugins/internal.py", line 123, in <genexpr>
[Fri Jan 09 12:16:37.495547 2015] [:error] [pid 2103]     (o.name, json_serialize(o)) for o in self.api.Object()
[Fri Jan 09 12:16:37.495551 2015] [:error] [pid 2103]   File "/usr/lib/python2.7/site-packages/ipalib/util.py", line 59, in json_serialize
[Fri Jan 09 12:16:37.495554 2015] [:error] [pid 2103]     return json_serialize(obj.__json__())
[Fri Jan 09 12:16:37.495556 2015] [:error] [pid 2103]   File "/usr/lib/python2.7/site-packages/ipalib/plugins/baseldap.py", line 710, in __json__
[Fri Jan 09 12:16:37.495558 2015] [:error] [pid 2103]     attrs = self.api.Backend.ldap2.schema.attribute_types(objectclasses)
[Fri Jan 09 12:16:37.495561 2015] [:error] [pid 2103]   File "/usr/lib64/python2.7/site-packages/ldap/schema/subentry.py", line 377, in attribute_types
[Fri Jan 09 12:16:37.495563 2015] [:error] [pid 2103]     object_class = self.sed[ObjectClass][object_class_oid]
[Fri Jan 09 12:16:37.495565 2015] [:error] [pid 2103] KeyError: 'idnsforwardzone'
Comment 23 Martin Bašti 2015-01-12 08:31:16 UTC 
Hello Josh,

You need to run ipa-ldap-updater --upgrade before ipa-upgradeconfig
Comment 24 Josh Baird 2015-01-12 14:08:16 UTC 
Ok, here is what I have done:

* Reverted back to RHEL 7.0 (VM snapshot)
* Ran 'yum update' against rhel7-server-beta-x86_64
* Waited until yum froze due to the LDAP server not starting
* Broke out of yum
* Re-installed the beta kernel so initramfs was built/installed
* Copied /etc/dirsrv/schema/01core389.ldif to /etc/dirsrv/slapd*/schema
* Ran 'ipa-ldap-updater --upgrade' which failed:

...
Upgrade failed with targetattr "winsyncdirectoryfilter" does not exist in schema. Please add attributeTypes "winsyncdirectoryfilter" to schema if necessary. ACL Syn         tax Error(-5):(targetattr = \22cn || createtimestamp || description || entryusn || modifytimestamp || nsds50ruv || nsds5beginreplicarefresh || nsds5debugreplicatime         out || nsds5flags || nsds5replicaabortcleanruv || nsds5replicaautoreferral || nsds5replicabackoffmax || nsds5replicabackoffmin || nsds5replicabinddn || nsds5replica         bindmethod || nsds5replicabusywaittime: Invalid syntax.
  [8/10]: stopping directory server
  [9/10]: restoring configuration
  [10/10]: starting directory server
Done.

* Next, I found winsyncdirectoryfilter in /etc/dirsrv/schema/02common.ldif and copied it to /etc/dirsrv/slapd*/schema.
* Re-ran 'ipa-ldap-updater --upgrade' which was successful this time.
* Ran 'ipa-upgradeconfig'
* Now, I'm able to login to IdM's web interface.

It would seem that *something* is fundamentally broken with the upgrade process.
Comment 25 Petr Vobornik 2015-01-12 17:25:23 UTC 
Given that `ipa-ldap-updater --upgrade` and `ipa-upgradeconfig` runs fine when the schema is properly copied to instance dir (which should be done in 389 update process), I don't think there is an issue in IPA, but in 389.

Therefore changing component to 389 server.
Comment 26 Ludwig 2015-01-15 15:37:11 UTC 
I got a VM with RHEL 7.0 and
389-ds 1.3.1 and ipa 3.3.3 installed

I created a repo: 
[RHEL-7.0-x86_64-beta]
name=RHEL-7.0-x86_64-updates-testing-debug
baseurl=http://localmirror/cgi-bin/repo.py?http://download.lab.bos.redhat.com/released/RHEL-7/7.1-Beta/Server/x86_64/os/
enabled=0
priority=99
gpgcheck=0

and upgraded 389 DS:
 yum --disablerepo=* --enablerepo=RHEL-7.0-x86_64-beta upgrade 389-ds*

Then checked for nsschemapolicy and winSyncDirectoryFilter in the generic and the instance specific directory:
grep -i nsschemapolicy /etc/dirsrv/schema/*
/etc/dirsrv/schema/01core389.ldif:objectClasses: ( 2.16.840.1.113730.3.2.328 NAME 'nsSchemaPolicy' DESC 'Netscape defined objectclass' SUP top  MAY ( cn $ schemaUpdateObjectclassAccept $ schemaUpdateObjectclassReject $ schemaUpdateAttributeAccept $ schemaUpdateAttributeReject) X-ORIGIN 'Netscape Directory Server' )

# grep -i nsschemapolicy /etc/dirsrv/slapd-IDM-LAB-ENG-BRQ-REDHAT-COM/schema/*
/etc/dirsrv/slapd-IDM-LAB-ENG-BRQ-REDHAT-COM/schema/01core389.ldif:objectClasses: ( 2.16.840.1.113730.3.2.328 NAME 'nsSchemaPolicy' DESC 'Netscape defined objectclass' SUP top  MAY ( cn $ schemaUpdateObjectclassAccept $ schemaUpdateObjectclassReject $ schemaUpdateAttributeAccept $ schemaUpdateAttributeReject) X-ORIGIN 'Netscape Directory Server' )

# grep -i winsyncdirectoryfilter /etc/dirsrv/schema/*
/etc/dirsrv/schema/02common.ldif:attributeTypes: ( 2.16.840.1.113730.3.1.2162 NAME 'winSyncDirectoryFilter' DESC 'Netscape defined attribute type'  SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 SINGLE-VALUE X-ORIGIN 'Netscape Directory Server' )
/etc/dirsrv/schema/02common.ldif:objectClasses: ( 2.16.840.1.113730.3.2.503 NAME 'nsDSWindowsReplicationAgreement' DESC 'Netscape defined objectclass' SUP top MUST ( cn ) MAY ( nsDS5ReplicaHost $ nsDS5ReplicaPort $ nsDS5ReplicaTransportInfo $ nsDS5ReplicaBindDN $ nsDS5ReplicaCredentials $ nsDS5ReplicaBindMethod $ nsDS5ReplicaRoot $ nsDS5ReplicatedAttributeList $ nsDS5ReplicaUpdateSchedule $ nsds5BeginReplicaRefresh $ description $ nsds50ruv $ nsruvReplicaLastModified $ nsds5ReplicaTimeout $ nsds5replicaChangesSentSinceStartup $ nsds5replicaLastUpdateEnd $ nsds5replicaLastUpdateStart $ nsds5replicaLastUpdateStatus $ nsds5replicaUpdateInProgress $ nsds5replicaLastInitEnd $ nsds5replicaLastInitStart $ nsds5replicaLastInitStatus $ nsds5debugreplicatimeout $ nsds5replicaBusyWaitTime $ nsds5replicaSessionPauseTime $ nsds7WindowsReplicaSubtree $ nsds7DirectoryReplicaSubtree $ nsds7NewWinUserSyncEnabled $ nsds7NewWinGroupSyncEnabled $ nsds7WindowsDomain $ nsds7DirsyncCookie $ winSyncInterval $ oneWaySync $ winSyncMoveAction $ nsds5ReplicaEnabled $ winSyncDirectoryFilter $ winSyncWindowsFilter $ winSyncSubtreePair ) X-ORIGIN 'Netscape Directory Server' )

grep -i winsyncdirectoryfilter /etc/dirsrv/slapd-IDM-LAB-ENG-BRQ-REDHAT-COM/schema/*
/etc/dirsrv/slapd-IDM-LAB-ENG-BRQ-REDHAT-COM/schema/02common.ldif:attributeTypes: ( 2.16.840.1.113730.3.1.2162 NAME 'winSyncDirectoryFilter' DESC 'Netscape defined attribute type'  SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 SINGLE-VALUE X-ORIGIN 'Netscape Directory Server' )
/etc/dirsrv/slapd-IDM-LAB-ENG-BRQ-REDHAT-COM/schema/02common.ldif:objectClasses: ( 2.16.840.1.113730.3.2.503 NAME 'nsDSWindowsReplicationAgreement' DESC 'Netscape defined objectclass' SUP top MUST ( cn ) MAY ( nsDS5ReplicaHost $ nsDS5ReplicaPort $ nsDS5ReplicaTransportInfo $ nsDS5ReplicaBindDN $ nsDS5ReplicaCredentials $ nsDS5ReplicaBindMethod $ nsDS5ReplicaRoot $ nsDS5ReplicatedAttributeList $ nsDS5ReplicaUpdateSchedule $ nsds5BeginReplicaRefresh $ description $ nsds50ruv $ nsruvReplicaLastModified $ nsds5ReplicaTimeout $ nsds5replicaChangesSentSinceStartup $ nsds5replicaLastUpdateEnd $ nsds5replicaLastUpdateStart $ nsds5replicaLastUpdateStatus $ nsds5replicaUpdateInProgress $ nsds5replicaLastInitEnd $ nsds5replicaLastInitStart $ nsds5replicaLastInitStatus $ nsds5debugreplicatimeout $ nsds5replicaBusyWaitTime $ nsds5replicaSessionPauseTime $ nsds7WindowsReplicaSubtree $ nsds7DirectoryReplicaSubtree $ nsds7NewWinUserSyncEnabled $ nsds7NewWinGroupSyncEnabled $ nsds7WindowsDomain $ nsds7DirsyncCookie $ winSyncInterval $ oneWaySync $ winSyncMoveAction $ nsds5ReplicaEnabled $ winSyncDirectoryFilter $ winSyncWindowsFilter $ winSyncSubtreePair ) X-ORIGIN 'Netscape Directory Server' 

so 389-ds upgrade seems to work properly by its own
Comment 27 Ludwig 2015-01-16 09:46:33 UTC 
On a new 7.0vm I did install 389 (1.3.1) and ipa (3.3.3) and installed an ipa server.
Then did a full update to 7.1 beta ( repos as specified in last comment) and
the schema in the ds instance was updated correctly.

Can you provide exact info on how to reproduce this ?
Comment 28 Josh Baird 2015-01-16 13:20:26 UTC 
Yes.

1.  Build a VM using rhel7-7.0-kickstart-x86_64.
2.  yum install ipa-server bind bind-dyndb-ldap
3.  ipa-server-install
4.  Verify that ipa-server and all IPA related services are running
5.  Run 'yum --enablerepo=beta update' 

The 'beta' repo is synced from htps://cdn.redhat.com/content/beta/rhel/server/7/x86_64                         4/os

At this point, the 'yum update' gets to the 'cleanup' section and never moves past this:

...

  Cleanup    : glibc-2.17-55.el7_0.1.x86_64                                                                                                                                                                                560/562
  Cleanup    : tzdata-2014i-1.el7.noarch                                                                                                                                                                                   561/562
  Cleanup    : libgcc-4.8.2-16.2.el7_0.x86_64                                                                                                                                                                              562/562
2734 blocks

When I break out of this eventually (20 minutes later):

^CPre schema upgrade failed with [Errno 111] Connection refused
[Errno 111] Connection refused
CA did not start in 300.0s
warning: %posttrans(ipa-server-4.1.0-10.el7.x86_64) scriptlet failed, signal 2
Non-fatal POSTTRANS scriptlet failure in rpm package ipa-server-4.1.0-10.el7.x86_64

At this point, the initramfs for the new kernel (3.10.0-210) is not built for whatever reason.

I have attached:

* rpm -qa of packages before I upgraded to RHEL 7.1 beta
* /var/log/messages during 'yum update'
* /var/log/dirsrv/slapd* logs
* rpm -qa of packages after I upgraded to RHEL 7.1 and the yum locks up

Let me know if you need anything else.
Comment 29 Josh Baird 2015-01-16 13:21:02 UTC 
Created attachment 980838 [details]
rpm -qa post upgrade
Comment 30 Josh Baird 2015-01-16 13:21:20 UTC 
Created attachment 980839 [details]
rpm -qa pre upgrade
Comment 31 Josh Baird 2015-01-16 13:21:47 UTC 
Created attachment 980840 [details]
dirsrv logs during upgrade to RHEL 7.1
Comment 32 Josh Baird 2015-01-16 13:22:17 UTC 
Created attachment 980841 [details]
/var/log/messages
Comment 33 Ludwig 2015-01-16 15:27:52 UTC 
do you have a directory /etc/dirsrv/<instance>/schema.bak ? What is in it ?
Comment 34 Josh Baird 2015-01-16 15:43:11 UTC 
Nope.

root@imqa-d1-dc03:/etc/dirsrv# find . -name schema.bak
root@imqa-d1-dc03:/etc/dirsrv#
Comment 35 Josh Baird 2015-01-16 15:43:11 UTC 
Nope.

root@imqa-d1-dc03:/etc/dirsrv# find . -name schema.bak
root@imqa-d1-dc03:/etc/dirsrv#
Comment 36 Ludwig 2015-01-16 15:56:02 UTC 
ok, the 389 schema upgrade script has a list of potential files to upgrade.
it creates a bak dir and moves these files to the bak dir.
then it calls installSchema which copies all non existing files from /etc/dirsrv/schema to /etc/dirsrv/<instance>/schema.
as far as i understand the schema.bak directory isn't removed, so it should be there, if upgrade fails to create this directory schema files are not copied.
Comment 37 Ludwig 2015-01-19 07:40:54 UTC 
It could also be a failure in teh scripts run before 60upgradeschemafiles.pl.

After upgrade, in the failed state, could you run
setup-ds.pl -d --update 

and capture the output
Comment 38 Josh Baird 2015-01-19 15:26:59 UTC 
[15/01/19:09:26:23] - [Setup] Info This program will update the 389 Directory Server.

It is recommended that you have "root" privilege to perform the update.
Tips for using this  program:
  - Press "Enter" to choose the default and go to the next screen
  - Type "Control-B" or the word "back" then "Enter" to go back to the previous screen
  - Type "Control-C" to cancel the update

[15/01/19:09:26:23] - [Setup] Info Would you like to continue with update?
[15/01/19:09:26:25] - [Setup] Info yes
[15/01/19:09:26:25] - [Setup] Info
The update process can work in one of two modes:

  - Online: The changes are made to the running directory servers using LDAP.
            The operations must be performed as an administrative user.
            You must provide the name and password, for each instance
            if there is more than one instance of directory server.
            Some operations may require a directory server restart to take
            effect.  The update script will notify you if you need to restart
            the server.

  - Offline: The changes are made to the server configuration files.  The
             servers MUST FIRST BE SHUTDOWN BY YOU.  The script will not
             shutdown the servers for you.  You MUST shutdown the
             servers in order to use this mode.  A username and password
             are not required to use Offline mode.  If the servers are not
             shutdown, CHANGES WILL BE LOST.

To summarize:
  Online - servers remain running - you must provide admin name and password
           for each server - servers may need to be restarted
  Offline - servers must be shutdown - no username or password required

[15/01/19:09:26:25] - [Setup] Info Which update mode do you want to use?
[15/01/19:09:26:28] - [Setup] Info offline
Running stage pre update /usr/share/dirsrv/updates/50acctusabilityplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50addchainingsaslpwroles.ldif
Running stage pre update /usr/share/dirsrv/updates/50automemberplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50bitstringsyntaxplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50contentsync.ldif
Running stage pre update /usr/share/dirsrv/updates/50deliverymethodsyntaxplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50derefplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50disableurisyntaxplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50enhancedguidesyntaxplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50entryusnindex.ldif
Running stage pre update /usr/share/dirsrv/updates/50faxnumbersyntaxplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50faxsyntaxplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50guidesyntaxplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50linkedattrsplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50managedentriesplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50memberofindex.ldif
Running stage pre update /usr/share/dirsrv/updates/50memberofplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50nameuidsyntaxplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50nstombstonecsn.ldif
Running stage pre update /usr/share/dirsrv/updates/50numericstringsyntaxplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50posix-winsync-plugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50printablestringsyntaxplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50refintprecedence.ldif
Running stage pre update /usr/share/dirsrv/updates/50replication-plugins.ldif
Running stage pre update /usr/share/dirsrv/updates/50retroclprecedence.ldif
Running stage pre update /usr/share/dirsrv/updates/50rootdnaccesscontrolplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50schemareloadplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50smd5pwdstorageplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50syntaxvalidplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50targetuniqueid.ldif
Running stage pre update /usr/share/dirsrv/updates/50teletexterminalidsyntaxplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50telexnumbersyntaxplugin.ldif
Running stage pre update /usr/share/dirsrv/updates/50updateconfig.ldif
Running stage pre update /usr/share/dirsrv/updates/50usnplugin.ldif
changeOwnerMode: changed mode of /var/run/dirsrv to 770
changeOwnerMode: changed group ownership of /var/run/dirsrv to group 297
changeOwnerMode: changed mode of /var/lib/dirsrv to 775
changeOwnerMode: changed group ownership of /var/lib/dirsrv to group 297
changeOwnerMode: changed mode of /etc/dirsrv to 775
changeOwnerMode: changed group ownership of /etc/dirsrv to group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/bak2db to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/bak2db to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/bak2db.pl to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/bak2db.pl to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/cleanallruv.pl to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/cleanallruv.pl to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/db2bak to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/db2bak to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/db2bak.pl to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/db2bak.pl to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/db2index to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/db2index to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/db2index.pl to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/db2index.pl to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/db2ldif to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/db2ldif to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/db2ldif.pl to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/db2ldif.pl to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/dbverify to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/dbverify to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/dn2rdn to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/dn2rdn to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/fixup-linkedattrs.pl to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/fixup-linkedattrs.pl to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/fixup-memberof.pl to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/fixup-memberof.pl to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/fixup-memberuid.pl to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/fixup-memberuid.pl to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/ldif2db to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/ldif2db to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/ldif2db.pl to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/ldif2db.pl to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/ldif2ldap to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/ldif2ldap to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/monitor to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/monitor to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/ns-accountstatus.pl to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/ns-accountstatus.pl to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/ns-activate.pl to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/ns-activate.pl to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/ns-inactivate.pl to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/ns-inactivate.pl to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/ns-newpwpolicy.pl to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/ns-newpwpolicy.pl to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/restart-slapd to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/restart-slapd to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/restoreconfig to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/restoreconfig to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/saveconfig to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/saveconfig to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/schema-reload.pl to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/schema-reload.pl to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/start-slapd to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/start-slapd to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/stop-slapd to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/stop-slapd to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/suffix2instance to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/suffix2instance to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/syntax-validate.pl to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/syntax-validate.pl to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/upgradednformat to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/upgradednformat to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/usn-tombstone-cleanup.pl to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/usn-tombstone-cleanup.pl to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/verify-db.pl to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/verify-db.pl to user 298 group 297
changeOwnerMode: changed mode of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/vlvindex to 550
changeOwnerMode: changed ownership of /var/lib/dirsrv/scripts-QA-UNIX-FOLLETT-COM/vlvindex to user 298 group 297
Running stage preinst update /usr/share/dirsrv/updates/50acctusabilityplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50addchainingsaslpwroles.ldif
Running stage preinst update /usr/share/dirsrv/updates/50automemberplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50bitstringsyntaxplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50contentsync.ldif
Running stage preinst update /usr/share/dirsrv/updates/50deliverymethodsyntaxplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50derefplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50disableurisyntaxplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50enhancedguidesyntaxplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50entryusnindex.ldif
Running stage preinst update /usr/share/dirsrv/updates/50faxnumbersyntaxplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50faxsyntaxplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50guidesyntaxplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50linkedattrsplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50managedentriesplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50memberofindex.ldif
Running stage preinst update /usr/share/dirsrv/updates/50memberofplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50nameuidsyntaxplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50nstombstonecsn.ldif
Running stage preinst update /usr/share/dirsrv/updates/50numericstringsyntaxplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50posix-winsync-plugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50printablestringsyntaxplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50refintprecedence.ldif
Running stage preinst update /usr/share/dirsrv/updates/50replication-plugins.ldif
Running stage preinst update /usr/share/dirsrv/updates/50retroclprecedence.ldif
Running stage preinst update /usr/share/dirsrv/updates/50rootdnaccesscontrolplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50schemareloadplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50smd5pwdstorageplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50syntaxvalidplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50targetuniqueid.ldif
Running stage preinst update /usr/share/dirsrv/updates/50teletexterminalidsyntaxplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50telexnumbersyntaxplugin.ldif
Running stage preinst update /usr/share/dirsrv/updates/50updateconfig.ldif
Running stage preinst update /usr/share/dirsrv/updates/50usnplugin.ldif
Processing /usr/share/dirsrv/data/template-pampta.ldif ...
Processing /usr/share/dirsrv/data/template-bitwise.ldif ...
Processing /usr/share/dirsrv/data/template-dnaplugin.ldif ...
Processing /usr/share/dirsrv/updates/dnaplugindepends.ldif ...
Processing /usr/share/dirsrv/updates/50updateconfig.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/10cleanupldapi.pl
Running stage runinst update /usr/share/dirsrv/updates/10delautodnsuffix.pl
Running stage runinst update /usr/share/dirsrv/updates/10fixrundir.pl
Running stage runinst update /usr/share/dirsrv/updates/20betxn.pl
Running stage runinst update /usr/share/dirsrv/updates/50acctusabilityplugin.ldif
Processing /usr/share/dirsrv/updates/50acctusabilityplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50addchainingsaslpwroles.ldif
Processing /usr/share/dirsrv/updates/50addchainingsaslpwroles.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50automemberplugin.ldif
Processing /usr/share/dirsrv/updates/50automemberplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50bitstringsyntaxplugin.ldif
Processing /usr/share/dirsrv/updates/50bitstringsyntaxplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50contentsync.ldif
Processing /usr/share/dirsrv/updates/50contentsync.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50deliverymethodsyntaxplugin.ldif
Processing /usr/share/dirsrv/updates/50deliverymethodsyntaxplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50derefplugin.ldif
Processing /usr/share/dirsrv/updates/50derefplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50disableurisyntaxplugin.ldif
Processing /usr/share/dirsrv/updates/50disableurisyntaxplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50enhancedguidesyntaxplugin.ldif
Processing /usr/share/dirsrv/updates/50enhancedguidesyntaxplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50entryusnindex.ldif
Processing /usr/share/dirsrv/updates/50entryusnindex.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50faxnumbersyntaxplugin.ldif
Processing /usr/share/dirsrv/updates/50faxnumbersyntaxplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50faxsyntaxplugin.ldif
Processing /usr/share/dirsrv/updates/50faxsyntaxplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50fixNsState.pl
Running stage runinst update /usr/share/dirsrv/updates/50guidesyntaxplugin.ldif
Processing /usr/share/dirsrv/updates/50guidesyntaxplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50linkedattrsplugin.ldif
Processing /usr/share/dirsrv/updates/50linkedattrsplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50managedentriesplugin.ldif
Processing /usr/share/dirsrv/updates/50managedentriesplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50memberofindex.ldif
Processing /usr/share/dirsrv/updates/50memberofindex.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50memberofplugin.ldif
Processing /usr/share/dirsrv/updates/50memberofplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50nameuidsyntaxplugin.ldif
Processing /usr/share/dirsrv/updates/50nameuidsyntaxplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50nstombstonecsn.ldif
Processing /usr/share/dirsrv/updates/50nstombstonecsn.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50numericstringsyntaxplugin.ldif
Processing /usr/share/dirsrv/updates/50numericstringsyntaxplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50posix-winsync-plugin.ldif
Processing /usr/share/dirsrv/updates/50posix-winsync-plugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50printablestringsyntaxplugin.ldif
Processing /usr/share/dirsrv/updates/50printablestringsyntaxplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50refintprecedence.ldif
Processing /usr/share/dirsrv/updates/50refintprecedence.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50replication-plugins.ldif
Processing /usr/share/dirsrv/updates/50replication-plugins.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50retroclprecedence.ldif
Processing /usr/share/dirsrv/updates/50retroclprecedence.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50rootdnaccesscontrolplugin.ldif
Processing /usr/share/dirsrv/updates/50rootdnaccesscontrolplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50schemareloadplugin.ldif
Processing /usr/share/dirsrv/updates/50schemareloadplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50smd5pwdstorageplugin.ldif
Processing /usr/share/dirsrv/updates/50smd5pwdstorageplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50syntaxvalidplugin.ldif
Processing /usr/share/dirsrv/updates/50syntaxvalidplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50targetuniqueid.ldif
Processing /usr/share/dirsrv/updates/50targetuniqueid.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50teletexterminalidsyntaxplugin.ldif
Processing /usr/share/dirsrv/updates/50teletexterminalidsyntaxplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50telexnumbersyntaxplugin.ldif
Processing /usr/share/dirsrv/updates/50telexnumbersyntaxplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50updateconfig.ldif
Processing /usr/share/dirsrv/updates/50updateconfig.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/50usnplugin.ldif
Processing /usr/share/dirsrv/updates/50usnplugin.ldif ...
Running stage runinst update /usr/share/dirsrv/updates/60upgradeconfigfiles.pl
[15/01/19:09:26:30] - [Setup] Info Could not rename config file '/etc/dirsrv/slapd-QA-UNIX-FOLLETT-COM/slapd-collations.conf' to '/var/lib/dirsrv/slapd-QA-UNIX-FOLLETT-COM/bak.bak/slapd-collations.conf'.  Error: Invalid cross-device link
[15/01/19:09:26:30] - [Setup] Fatal Error: could not update the directory server.
[15/01/19:09:26:30] - [Setup] Fatal Exiting . . .
Log file is '/tmp/setupqMOXdn.log'
Comment 39 Ludwig 2015-01-19 17:32:26 UTC 
so it looks like you have /etc and /var on different file systems.
The upgrade perl scripts use rename to mv a file and this does not work across mount points.

changing this in DS could be a major effort, do you need to have seperate fs ?
Comment 40 Josh Baird 2015-01-19 18:35:00 UTC 
Yes, all of our builds have separate filesystems for /etc/ and /var.  I would think that it's not very uncommon for users to have /var on it's own filesystem.
Comment 41 Aron Parsons 2015-01-19 22:48:06 UTC 
(In reply to Josh Baird from comment #40)
> Yes, all of our builds have separate filesystems for /etc/ and /var.  I
> would think that it's not very uncommon for users to have /var on it's own
> filesystem.

Agreed.  /var is almost always going to be on a separate filesystem except maybe on a box where someone manually walked through the installer and accepted the defaults.

(In reply to Ludwig from comment #39)
> so it looks like you have /etc and /var on different file systems.
> The upgrade perl scripts use rename to mv a file and this does not work
> across mount points.
> 
> changing this in DS could be a major effort, do you need to have seperate fs
> ?

Did the behavior in the update script change recently?  This has never been an issue over the past few years doing IPA upgrades.
Comment 42 Noriko Hosoi 2015-01-19 23:05:32 UTC 
(In reply to Aron Parsons from comment #41)
> (In reply to Josh Baird from comment #40)
> > Yes, all of our builds have separate filesystems for /etc/ and /var.  I
> > would think that it's not very uncommon for users to have /var on it's own
> > filesystem.
> 
> Agreed.  /var is almost always going to be on a separate filesystem except
> maybe on a box where someone manually walked through the installer and
> accepted the defaults.
> 
> (In reply to Ludwig from comment #39)
> > so it looks like you have /etc and /var on different file systems.
> > The upgrade perl scripts use rename to mv a file and this does not work
> > across mount points.
> > 
> > changing this in DS could be a major effort, do you need to have seperate fs
> > ?
> 
> Did the behavior in the update script change recently?  This has never been
> an issue over the past few years doing IPA upgrades.

This 60upgradeconfigfile.pl was introduced to 389-ds-base-1.3.1.6-6.el7 at the rhel-7.0 time frame... (see https://bugzilla.redhat.com/show_bug.cgi?id=1016317)

> Running stage runinst update /usr/share/dirsrv/updates/60upgradeconfigfiles.pl
Comment 43 Ludwig 2015-01-20 08:06:58 UTC 
> Did the behavior in the update script change recently?  This has never been
> an issue over the past few years doing IPA upgrades.

as Noriko said this upgrade script is a relatively new addittion. And the failure is silent, if there wouldn't be the missing schema upgrad in the next step, you probably wouldn't have noticed it this time.

I looked into the scripts, and it looks like only thiy one tries a renam across file systems, just be using the regular backup dir in /var as start.
So I think we can fix it.
Comment 51 Scott Poore 2015-01-26 02:07:01 UTC 
Verified.

Version :;

Before Upgrade:

[root@vm7 lib]# rpm -q ipa-server 389-ds-base
ipa-server-3.3.3-28.el7.x86_64
389-ds-base-1.3.1.6-25.el7.x86_64

After Upgrade:

[root@vm7 lib]# rpm -q ipa-server 389-ds-base
ipa-server-4.1.0-16.el7.x86_64
389-ds-base-1.3.3.1-12.el7.x86_64


Results ::

[root@vm7 lib]# grep -i nsSchemaPolicy /etc/dirsrv/slapd-EXAMPLE-TEST/schema/01core389.ldif
objectClasses: ( 2.16.840.1.113730.3.2.328 NAME 'nsSchemaPolicy' DESC 'Netscape defined objectclass' SUP top  MAY ( cn $ schemaUpdateObjectclassAccept $ schemaUpdateObjectclassReject $ schemaUpdateAttributeAccept $ schemaUpdateAttributeReject) X-ORIGIN 'Netscape Directory Server' )

[root@vm7 lib]# grep -i nsSchemaPolicy /etc/dirsrv/schema/01core389.ldif 
objectClasses: ( 2.16.840.1.113730.3.2.328 NAME 'nsSchemaPolicy' DESC 'Netscape defined objectclass' SUP top  MAY ( cn $ schemaUpdateObjectclassAccept $ schemaUpdateObjectclassReject $ schemaUpdateAttributeAccept $ schemaUpdateAttributeReject) X-ORIGIN 'Netscape Directory Server' )

[root@vm7 lib]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
ipa_memcached Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful
Comment 53 errata-xmlrpc 2015-03-05 09:40:18 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0416.html