Bug 1836806 - ipa-server-upgrade fails: /var/lib/pki/pki-tomcat/ca/profiles/ca/caECAdminCert.cfg missing
Summary: ipa-server-upgrade fails: /var/lib/pki/pki-tomcat/ca/profiles/ca/caECAdminCer...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: pki-core
Version: 31
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Matthew Harmsen
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-18 08:28 UTC by Tomasz Torcz
Modified: 2020-10-12 15:50 UTC (History)
16 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2020-10-12 15:50:05 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Tomasz Torcz 2020-05-18 08:28:30 UTC
Description of problem:
After a reboot, I've noticed my ipa.service is failing. The cause is a failure of ipa-server-upgrade. When running manually, I see following error:

ipapython.dogtag: DEBUG: response body (decoded): b''
ipaserver.install.ipa_server_upgrade: ERROR: IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
ipapython.admintool: DEBUG:   File "/usr/lib/python3.7/site-packages/ipapython/admintool.py", line 179, in execute
    return_value = self.run()                                                      
  File "/usr/lib/python3.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run
    server.upgrade()                                                               
  File "/usr/lib/python3.7/site-packages/ipaserver/install/server/upgrade.py", line 2280, in upgrade
    upgrade_configuration()                                                        
  File "/usr/lib/python3.7/site-packages/ipaserver/install/server/upgrade.py", line 2149, in upgrade_configuration
    ca_enable_ldap_profile_subsystem(ca)                                           
  File "/usr/lib/python3.7/site-packages/ipaserver/install/server/upgrade.py", line 414, in ca_enable_ldap_profile_subsystem
    cainstance.migrate_profiles_to_ldap() 
  File "/usr/lib/python3.7/site-packages/ipaserver/install/cainstance.py", line 1932, in migrate_profiles_to_ldap
    with open(filename) as f:

ipapython.admintool: DEBUG: The ipa-server-upgrade command failed, exception: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/pki/pki-tomcat/ca/profiles/ca/caECAdminCert.cfg'              
ipapython.admintool: ERROR: [Errno 2] No such file or directory: '/var/lib/pki/pki-tomcat/ca/profiles/ca/caECAdminCert.cfg'                                           
ipapython.admintool: ERROR: The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information



Version-Release number of selected component (if applicable):
freeipa-server-4.8.6-1.fc31.x86_64

pki-base-10.8.3-2.fc31.noarch
pki-base-java-10.8.3-2.fc31.noarch
pki-ca-10.8.3-2.fc31.noarch
pki-kra-10.8.3-2.fc31.noarch
pki-server-10.8.3-2.fc31.noarch
pki-symkey-10.8.3-2.fc31.x86_64
pki-tools-10.8.3-2.fc31.x86_64
python3-pki-10.8.3-2.fc31.noarch

How reproducible:
Always.

Steps to Reproduce:
1. Try to start ipa.service
2.
3.

Comment 1 Rob Crittenden 2020-05-18 12:49:04 UTC
Can you tell from the dnf logs which version of ipa-server and pki-ca you updated from?

To start ipa without trying to do the update you can run: ipactl --skip-version-check

But this would leave the service in a quasi-updated state. I don't know whether some important change wouldn't be applied since only about half of the update script has been executed.

An alternative would be to copy /usr/share/pki/ca/profiles/ca/caECAdminCert.cfg into /var/lib/pki/pki-tomcat/ca/profiles/ca/ then run ipa-server-upgrade but this just masks the failure and won't lead to the root cause.

Comment 2 Tomasz Torcz 2020-05-18 12:55:37 UTC
From dnf.rpm.log it seems previous version was:

freeipa-server-4.8.4-2.fc31.x86_64 (build in December 2019)

I see there were no other F31 builds between 4.8.4-2 and 4.8.6-1.

Comment 3 Rob Crittenden 2020-05-18 12:59:03 UTC
What about pki-ca? That's what provides the base configuration file.

Comment 4 Tomasz Torcz 2020-05-18 13:17:05 UTC
dnf.rpm.log.1:2020-02-21T13:47:21Z SUBDEBUG Upgrade: pki-ca-10.7.3-3.fc31.noarch
dnf.rpm.log.1:2020-02-21T13:53:03Z SUBDEBUG Upgraded: pki-ca-10.7.3-3.fc30.noarch

dnf.rpm.log.1:2020-03-15T05:33:49Z SUBDEBUG Upgrade: pki-ca-10.8.3-1.fc31.noarch
dnf.rpm.log.1:2020-03-15T05:33:50Z SUBDEBUG Upgraded: pki-ca-10.7.3-3.fc31.noarch

dnf.rpm.log:2020-05-17T14:11:05Z SUBDEBUG Upgrade: pki-ca-10.8.3-2.fc31.noarch
dnf.rpm.log:2020-05-17T14:12:06Z SUBDEBUG Upgraded: pki-ca-10.8.3-1.fc31.noarch

Comment 5 Rob Crittenden 2020-05-18 13:57:15 UTC
cc'ing a dogtag developer to see if this is a known issue.

Comment 6 Dinesh Prasanth 2020-05-18 14:46:58 UTC
(In reply to Tomasz Torcz from comment #0)
> 
> ipapython.admintool: DEBUG: The ipa-server-upgrade command failed,
> exception: FileNotFoundError: [Errno 2] No such file or directory:
> '/var/lib/pki/pki-tomcat/ca/profiles/ca/caECAdminCert.cfg'              
> ipapython.admintool: ERROR: [Errno 2] No such file or directory:
> '/var/lib/pki/pki-tomcat/ca/profiles/ca/caECAdminCert.cfg'                  
> 
> ipapython.admintool: ERROR: The ipa-server-upgrade command failed. See
> /var/log/ipaupgrade.log for more information
> 

Hi Tomasz,

Do you remember which version of IPA/PKI you started with?

My hypothesis based on the error and timeline:

- The `caECAdminCert.cfg` was introduced in PKI 10.6 [1] and has been there since
- The patch to FIX path in CS.cfg was introduced in 10.8.2 [2]. Note that this patch
  DOES NOT copy caECAdminCert.cfg introduced.
- The bug in the above patch was fixed [3] in 10.8.3 and released through 10.8.3-2 on
  Fedora.

So, Tomasz, if you started with IPA before Fedora 27, my above hypothesis
will be true. In that case, I can add a patch to upstream 10.8.3 and make a new release.

The quick solution is copying the file as Rob suggested above and re-running the upgrade
process.

HTH

[1] https://github.com/dogtagpki/pki/commit/27cf99efe1e52249f226db24ef28b0990a654dd5#diff-e97d0d6e9ba2781db5a0ff3a6172b657
[2] https://github.com/dogtagpki/pki/commit/84c039e9d93794df118e926c24aacf1da8fd166e
[3] https://github.com/dogtagpki/pki/pull/355

Comment 7 Tomasz Torcz 2020-05-18 14:58:21 UTC
I do not remember version exactly, but it was in 2015, so around Fedora 22/23.
I will copy the file, but please include it upstream. Thank you!

Comment 8 Adam Williamson 2020-06-23 15:17:15 UTC
I hit this too, and my FreeIPA install is indeed also very old. So Dinesh's theory looks sound.

Comment 9 Dinesh Prasanth 2020-07-10 15:48:27 UTC
The issue has been fixed upstream via PR: https://github.com/dogtagpki/pki/pull/471

commit information:
===================
commit 337cff960cf01c0cfd5ac759c11053a9f0de7e7f
Author: Dinesh Prasanth M K <dmoluguw>
Date:   Thu Jul 2 19:07:29 2020 -0400

    Copy missing profiles between 10.5 and current version (10.9)
    
    This patch copies all missing profiles introduced from 10.6+
    and configures the CS.cfg in existing deployments. This ensures
    that the old deployments (<=10.5) can use the latest profiles
    
    Signed-off-by: Dinesh Prasanth M K <dmoluguw>

Additional Notes:
=================
Please refer the "Additional Notes" section in the PR's description as well as
reviewer's comments on possible things that require manual intervention by the
user.


Setting the status to POST

Comment 11 Adam Williamson 2020-10-07 22:43:55 UTC
So F31 now has pki-core-10.9.4-1.fc31 and dogtag-pki-10.9.4-1.fc31 , which have the commit mentioned above.

Tomas: are you able to test if this is fixed, or shall we just assume it is and close the bug?

Comment 12 Tomasz Torcz 2020-10-12 04:57:22 UTC
I cannot test, I had to fix my important infrastructure.
But there are no more reports, let's assume it is fixed.

Comment 13 Adam Williamson 2020-10-12 15:50:05 UTC
"I cannot test, I had to fix my important infrastructure."

Yeah, same here :/

"But there are no more reports, let's assume it is fixed."

I agree, let's close it.


Note You need to log in before you can comment on or make changes to this bug.