Bug 1550555

Summary: freeipa 4.6.1->4.6.3 upgrade breaks in ipa-server-upgrade: No such file or directory: '/var/lib/pki/pki-tomcat/conf/ca/CS.cfg'
Product: [Fedora] Fedora Reporter: James <james>
Component: freeipaAssignee: IPA Maintainers <ipa-maint>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 27CC: abokovoy, frenaud, ipa-maint, jcholast, jhrozek, pvoborni, rcritten, ssorce
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: freeipa-4.6.3-2.fc27 freeipa-4.6.4-2.fc27 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1577805 (view as bug list) Environment:
Last Closed: 2018-08-28 11:55:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1577805    

Description James 2018-03-01 12:51:57 UTC
Description of problem:
Upgrading from FreeIPA 4.6.1 to 4.6.3, on F27. This installation with external CA. ipa-server-upgrade fails at:

ipaserver.install.ipa_server_upgrade: ERROR: IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
ipapython.admintool: DEBUG:   File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 174, in execute
    return_value = self.run()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/ipa_server_upgrade.py", line 50, in run
    server.upgrade()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/upgrade.py", line 1999, in upgrade
    upgrade_configuration()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/upgrade.py", line 1686, in upgrade_configuration
    ca.backup_config()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/dogtaginstance.py", line 475, in backup_config
    shutil.copy(path, path + '.ipabkp')
  File "/usr/lib64/python3.6/shutil.py", line 241, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:

ipapython.admintool: DEBUG: The ipa-server-upgrade command failed, exception: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pki/pki-tomcat/conf/ca/CS.cfg'
ipapython.admintool: ERROR: [Errno 2] No such file or directory: '/var/lib/pki/pki-tomcat/conf/ca/CS.cfg'
ipapython.admintool: ERROR: The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

Comment 1 James 2018-03-01 13:05:36 UTC
Had to roll back to 4.6.1, now back in action. So at least ipa-server-upgrade didn't hose the database...

Comment 2 Florence Blanc-Renaud 2018-03-01 17:17:24 UTC
Upstream ticket:
https://pagure.io/freeipa/issue/7409

Comment 4 James 2018-03-07 19:22:25 UTC
Hold on, hold on. What do I have to do to test this without risking a broken database and having to start all over again?

Comment 5 Rob Crittenden 2018-03-07 21:57:04 UTC
Just update to the fixed packages and that should do it.

Even if the upgrade failed it wouldn't corrupt the database.

Comment 6 James 2018-03-07 22:08:05 UTC
Reopening. This is not fixed. Downgrading again.

# rpm -q freeipa-server
freeipa-server-4.6.3-2.fc27.x86_64

# systemctl status ipa
● ipa.service - Identity, Policy, Audit
   Loaded: loaded (/usr/lib/systemd/system/ipa.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2018-03-07 22:06:01 GMT; 39s ago
  Process: 20965 ExecStart=/usr/sbin/ipactl start (code=exited, status=1/FAILURE)
 Main PID: 20965 (code=exited, status=1/FAILURE)

Mar 07 22:06:00 skipper.cb.ettle ipactl[20965]: [Verifying that root certificate is published]
Mar 07 22:06:00 skipper.cb.ettle ipactl[20965]: IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-serve
Mar 07 22:06:00 skipper.cb.ettle ipactl[20965]: [Errno 2] No such file or directory: '/var/lib/pki/pki-tomcat/conf/ca/CS.cfg'
Mar 07 22:06:00 skipper.cb.ettle ipactl[20965]: The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more informat
Mar 07 22:06:00 skipper.cb.ettle ipactl[20965]: See the upgrade log for more details and/or run /usr/sbin/ipa-server-upgrade again
Mar 07 22:06:00 skipper.cb.ettle ipactl[20965]: Aborting ipactl
Mar 07 22:06:01 skipper.cb.ettle systemd[1]: ipa.service: Main process exited, code=exited, status=1/FAILURE
Mar 07 22:06:01 skipper.cb.ettle systemd[1]: Failed to start Identity, Policy, Audit.
Mar 07 22:06:01 skipper.cb.ettle systemd[1]: ipa.service: Unit entered failed state.
Mar 07 22:06:01 skipper.cb.ettle systemd[1]: ipa.service: Failed with result 'exit-code'.


End of /var/log/ipaupgrade.log:


2018-03-07T22:06:00Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2018-03-07T22:06:00Z DEBUG   File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 174, in execute
    return_value = self.run()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/ipa_server_upgrade.py", line 50, in run
    server.upgrade()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/upgrade.py", line 1999, in upgrade
    upgrade_configuration()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/server/upgrade.py", line 1686, in upgrade_configuration
    ca.backup_config()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/dogtaginstance.py", line 475, in backup_config
    shutil.copy(path, path + '.ipabkp')
  File "/usr/lib64/python3.6/shutil.py", line 241, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:

2018-03-07T22:06:00Z DEBUG The ipa-server-upgrade command failed, exception: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pki/pki-tomcat/conf/ca/CS.cfg'
2018-03-07T22:06:00Z ERROR [Errno 2] No such file or directory: '/var/lib/pki/pki-tomcat/conf/ca/CS.cfg'
2018-03-07T22:06:00Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

Comment 7 James 2018-03-08 23:06:00 UTC
Looking at the SRPM those commits from Comment 3 simply aren't in 4.6.3-2 (ipaserver/install/server/upgrade.py). Patch0001 only deals with KRA-related stuff, but seems to be matching against code from that commit...

Comment 8 Rob Crittenden 2018-03-12 18:52:25 UTC
You're right, I missed a patch. There were two issues, one hiding the other. I'll spin up a new build.

Comment 9 Fedora Update System 2018-03-13 14:15:06 UTC
freeipa-4.6.3-3.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-0a4399f314

Comment 10 James 2018-03-13 19:19:12 UTC
OK - 4.6.3-3.fc27.x86_64 updated cleanly. Server rebooted, confirmed login with OTP, NFS and web interface work. Thanks, Rob!

Comment 11 Fedora Update System 2018-03-14 01:39:18 UTC
freeipa-4.6.3-3.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-0a4399f314

Comment 12 Florence Blanc-Renaud 2018-05-15 15:11:41 UTC
Fixed upstream
ipa-4-5:
https://pagure.io/freeipa/c/035f1cb24a228ba40b3e124d78a507be22aa52bd

Comment 13 Fedora Update System 2018-06-13 15:36:20 UTC
freeipa-4.6.4-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-0492828909

Comment 14 Fedora Update System 2018-06-14 13:48:36 UTC
freeipa-4.6.4-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-0492828909

Comment 15 Fedora Update System 2018-08-20 08:56:20 UTC
freeipa-4.6.4-2.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-39051f69b7

Comment 16 Fedora Update System 2018-08-20 15:50:30 UTC
freeipa-4.6.4-2.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-39051f69b7

Comment 17 Fedora Update System 2018-08-28 11:55:11 UTC
freeipa-4.6.4-2.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.