Bug 1619733

Richard, more details are needed about your configuration in place.

For example, we are not using db2 driver for KDC at all and don't have it set up in kdc.conf/krb5.conf. So this is a non-standard configuration where db2 kdb driver overtakes whatever is defined by FreeIPA (ipadb).

I haven't made any non-standard configurations, at least that I'm aware of. I don't even know the procedure for changing KDC driver to db2.

ipadb.so is specified in krb5.conf. See attachment krb5.conf.

There are two IPA servers; server0 and server1. Rough timeline:
2017-10-15 server0 installed with Fedora 26 and IPA server. IPA version 4.4.4-4.fc26
2017-12-30 server0 upgraded to Fedora 27, IPA version 4.6.1-3.fc27
2017-12-31 server1 installed with Fedora 27 and enrolled as IPA client.
2017-12-31 server1 promoted to replica with ipa-replica-install. IPA version 4.6.1-3.fc27
2018-07-14 server0 and server1 upgraded to Fedora 28. IPA version 4.6.90.pre2-3.fc28
2018-08-20 server1 failed upgrade to freeipa-server-4.7.0-1.fc28.x86_64

From server0 /var/log/ipaserver-install.log:
2017-10-15T15:22:55Z DEBUG ipa-server-install was invoked with arguments [] and options: {'no_dns_sshfp': None, 'ignore_topology_disconnect': None, 'verbose': False, 'ip_addresses': [CheckedIPAddress('192.0.2.1'), CheckedIPAddress('2001:DB8::1')], 'domainlevel': None, 'mkhomedir': True, 'no_pkinit': None, 'http_cert_files': None, 'no_ntp': None, 'subject': None, 'no_forwarders': None, 'external_ca_type': None, 'ssh_trust_dns': True, 'domain_name': 'arvanode.net', 'idmax': None, 'http_cert_name': None, 'dirsrv_cert_files': None, 'no_dnssec_validation': None, 'ca_signing_algorithm': None, 'no_reverse': None, 'pkinit_cert_files': None, 'unattended': False, 'auto_reverse': None, 'auto_forwarders': True, 'no_host_dns': None, 'no_sshd': None, 'no_ui_redirect': None, 'ignore_last_of_role': None, 'realm_name': 'ARVANODE.NET', 'forwarders': None, 'idstart': None, 'external_ca': None, 'pkinit_cert_name': None, 'no_ssh': None, 'external_cert_files': None, 'no_hbac_allow': None, 'forward_policy': 'only', 'dirsrv_cert_name': None, 'ca_cert_files': None, 'zonemgr': None, 'quiet': False, 'setup_dns': True, 'host_name': 'server0.example.com', 'dirsrv_config_file': None, 'log_file': None, 'reverse_zones': None, 'allow_zone_overlap': True, 'uninstall': False}
2017-10-15T15:22:55Z DEBUG IPA version 4.4.4-4.fc26

From server1 /var/log/ipareplica-install.log:
2017-12-31T20:19:33Z DEBUG ipa-replica-install was invoked with arguments [] and options: {'unattended': False, 'ip_addresses': None, 'domain_name': None, 'servers': None, 'realm_name': None, 'host_name': None, 'principal': None, 'setup_adtrust': False, 'setup_ca': True, 'setup_kra': False, 'setup_dns': True, 'no_pkinit': False, 'no_ui_redirect': False, 'dirsrv_config_file': None, 'dirsrv_cert_files': None, 'http_cert_files': None, 'pkinit_cert_files': None, 'dirsrv_cert_name': None, 'http_cert_name': None, 'pkinit_cert_name': None, 'keytab': None, 'mkhomedir': True, 'force_join': False, 'no_ntp': False, 'ssh_trust_dns': True, 'no_ssh': False, 'no_sshd': False, 'no_dns_sshfp': False, 'skip_schema_check': False, 'allow_zone_overlap': False, 'reverse_zones': None, 'no_reverse': False, 'auto_reverse': False, 'forwarders': [CheckedIPAddress('192.0.2.254')], 'no_forwarders': False, 'auto_forwarders': False, 'forward_policy': None, 'no_dnssec_validation': False, 'no_host_dns': False, 'add_sids': False, 'add_agents': False, 'enable_compat': False, 'netbios_name': None, 'no_msdcs': False, 'rid_base': None, 'secondary_rid_base': None, 'skip_conncheck': False, 'verbose': False, 'quiet': False, 'log_file': None}
2017-12-31T20:19:33Z DEBUG IPA version 4.6.1-3.fc27

Created attachment 1477660 [details]
krb5.conf server1

Could you please also attach /var/log/ipaupgrade.log?

I assume you also replaced actual realm name with the EXAMPLE.COM before posting, so that is not your actual realm.

What be good also to see output of strace run of kadmin.local as root:
  strace -etrace=file kadmin.local


Robbie, do you have any idea where db2 default comes from? Could it be that kadmin.local defaults to db2 in case a module specified for the realm is not available?

Yes, I believe that's the case.

Based on the configuration file, the IPA KDB module isn't loaded.  It should look like:

[dbmodules]
  EXAMPLE.COM = {
    db_library = ipadb.so
  }

which is present but commented out.  Why is it commented out?

Yes, the realm name has been replaced with EXAMPLE.COM before posting.

The problem was an incorrect krb5.conf. 

Last lines of krb5.conf before running ipa-server-upgrade:
[dbmodules]
  EXAMPLE.COM = {
    db_library = ipadb.so
  }

[plugins]
certauth = {
module = ipakdb:kdb/ipadb.so
enable_only = ipakdb
}


After ipa-server-upgrade:
[dbmodules]
# EXAMPLE.COM = {
#      db_library = ipadb.so
# }

[plugins]
 certauth = {
     module = ipakdb:kdb/ipadb.so
     enable_only = ipakdb
 }
  EXAMPLE.COM = {
    db_library = ipadb.so
  }


ipa-server-upgrade was successful after restoring krb5.conf and placing section [dbmodules] last in krb5.conf like this:
[plugins]
certauth = {
module = ipakdb:kdb/ipadb.so
enable_only = ipakdb
}

[dbmodules]
  EXAMPLE.COM = {
    db_library = ipadb.so
  }

Assigned to me to see if anything needs to be fixed in FreeIPA.

This message is a reminder that Fedora 28 is nearing its end of life.
On 2019-May-28 Fedora will stop maintaining and issuing updates for
Fedora 28. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora 'version' of '28'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 28 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 28 changed to end-of-life (EOL) status on 2019-05-28. Fedora 28 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.