Bug 1274905

Summary:	Upgrade of FreeIPA to Fedora 23 failed
Product:	[Fedora] Fedora	Reporter:	Martin Kosek <mkosek>
Component:	freeipa	Assignee:	IPA Maintainers <ipa-maint>
Status:	CLOSED ERRATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	23	CC:	abokovoy, awilliam, devin, fedora, fujisan43, ipa-maint, jhrozek, martin, mkosek, pbrobinson, pviktori, pvoborni, rcritten, rdieter, samuel-rhbugs, ssorce, tomek
Target Milestone:	---	Keywords:	CommonBugs
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	https://fedoraproject.org/wiki/Common_F23_bugs#freeipa-upgrade-fail
Fixed In Version:	freeipa-4.2.3-1.fc23	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-11-03 18:19:54 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1274915
Bug Blocks:

Description Martin Kosek 2015-10-23 19:36:11 UTC

Description of problem:

# ipa-server-upgrade
session memcached servers not running
Missing version: no platform stored
Upgrading IPA:
  [1/8]: saving configuration
  [2/8]: disabling listeners
  [3/8]: enabling DS global lock
  [4/8]: starting directory server
  [error] CalledProcessError: Command ''/bin/systemctl' 'start' 'dirsrv'' returned non-zero exit status 1
  [cleanup]: stopping directory server
  [cleanup]: restoring configuration
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
Unexpected error - see /var/log/ipaupgrade.log for details:
CalledProcessError: Command ''/bin/systemctl' 'start' 'dirsrv'' returned non-zero exit status 1


# systemctl status dirsrv -l
● dirsrv - 389 Directory Server DEMO1-FREEIPA-ORG.
   Loaded: loaded (/usr/lib/systemd/system/dirsrv@.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2015-10-23 19:30:52 UTC; 49s ago
  Process: 1437 ExecStart=/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-%i -i /var/run/dirsrv/slapd-%i.pid -w /var/run/dirsrv/slapd-%i.startpid (code=exited, status=1/FAILURE)

Oct 23 19:30:51 ipa.demo1.freeipa.org systemd[1]: Starting 389 Directory Server DEMO1-FREEIPA-ORG....
Oct 23 19:30:52 ipa.demo1.freeipa.org ns-slapd[1437]: [23/Oct/2015:19:30:52 +0000] - Cannot find parent attribute type "ipaPublicKey"
Oct 23 19:30:52 ipa.demo1.freeipa.org ns-slapd[1437]: [23/Oct/2015:19:30:52 +0000] dse_read_one_file - The entry cn=schema in file /etc/dirsrv/slapd-DEMO1-FREEIPA-ORG/schema/99user.ldif (lineno: 1) is invalid, error code 21 (Invalid syntax) - attribute type ipaVaultPublicKey: Missing parent attribute syntax OID
Oct 23 19:30:52 ipa.demo1.freeipa.org ns-slapd[1437]: [23/Oct/2015:19:30:52 +0000] dse - Please edit the file to correct the reported problems and then restart the server.
Oct 23 19:30:52 ipa.demo1.freeipa.org systemd[1]: dirsrv: Control process exited, code=exited status=1
Oct 23 19:30:52 ipa.demo1.freeipa.org systemd[1]: Failed to start 389 Directory Server DEMO1-FREEIPA-ORG..
Oct 23 19:30:52 ipa.demo1.freeipa.org systemd[1]: dirsrv: Unit entered failed state.
Oct 23 19:30:52 ipa.demo1.freeipa.org systemd[1]: dirsrv: Failed with result 'exit-code'.
Oct 23 19:30:53 ipa.demo1.freeipa.org systemd[1]: Stopped 389 Directory Server DEMO1-FREEIPA-ORG..


Version-Release number of selected component (if applicable):
389-ds-base-1.3.4.4-1.fc23.1.x86_64
pki-ca-10.2.6-7.fc23.noarch
freeipa-server-4.2.2-1.fc23.x86_64

How reproducible:
Happened once, during upgrade

Steps to Reproduce:
1. Install FreeIPA server on F22
2. Upgrade F22 to F23 using dnf
3.

Actual results:
Upgrade fails as 389-DS cannot start

Expected results:
Upgrade passes.

Additional info:

Comment 1 Martin Kosek 2015-10-23 19:37:28 UTC

Related FreeIPA ticket that needs to be backported:
https://fedorahosted.org/freeipa/ticket/5359

Comment 2 Martin Kosek 2015-10-23 19:55:45 UTC

https://fedorahosted.org/freeipa/ticket/5360 has to be also backported - fails the upgrade too.

Comment 3 Adam Williamson 2015-11-02 00:05:54 UTC

This seems to be a major, major problem :/ should've been at least evaluated as a release blocker: you can't upgrade a Fedora Server running a supported role.

Comment 4 Adam Williamson 2015-11-02 00:44:06 UTC

This seems to be a complete nightmare to recover from. After hand-editing dse.ldif to drop the 'SUP ipaPublicKey' - running the upgrade script a second time after patching that problem adds the EQUALITY but doesn't drop the SUP - I got a bit further, but now it seems like nothing from https://fedorahosted.org/pki/ticket/1264#comment:1 has been applied on my system; all those changes are missing from /etc/pki/pki-tomcat/server.xml , so I hit that "java.lang.ClassNotFoundException: org.apache.catalina.core.JasperListener" error.

At this point my FreeIPA server is more or less toast, which is obviously not good at all. I upgraded from 21 to 23 using dnf system-upgrade.

Comment 5 Adam Williamson 2015-11-02 01:11:32 UTC

Upgrades also encounter this:

https://fedorahosted.org/pki/ticket/1310

To do the Tomcat 8 migration manually, do:

pki-server migrate --tomcat 8

After manually fixing up the LDAP upgrade problem and running the Tomcat migration, my server seems to be more or less working again, but something seems to be causing Apache to make 'execmem' calls, which is forbidden by SELinux policy by default, and should not be necessary. I had to do "setsebool -P httpd_execmem 1" for now. Does anyone know what would be causing that?

Comment 6 Fedora Update System 2015-11-02 20:40:29 UTC

freeipa-4.2.3-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-4d94884a7e

Comment 7 Fedora Update System 2015-11-03 18:19:47 UTC

freeipa-4.2.3-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.