Bug 1757064

Workaround / observation: Run ipactl restart after the ipupgrade process completes

Can you clarify step 4? It seems to me that should be two steps at least and the order is unclear. Are you running ipa-server-upgrade and then yum (!?) update or the reverse (or is the ipa-server-upgrade expected to be implicit)?

If the version was increased then ipactl will re-run the upgrade.

Hi Rob, 
I am running only 'yum -y update' and then the command 'ipa-server-upgrade' post 'yum update'.

# yum -y update
# Upgrade FAILS
# Restart ipactl restart
# ipa-server-upgrade is successful

[root@vm-idm-014 ~]# ipa-server-upgrade
Upgrading IPA:. Estimated time: 1 minute 30 seconds
  [1/11]: stopping directory server
  [2/11]: saving configuration
  [3/11]: disabling listeners
  [4/11]: enabling DS global lock
  [5/11]: disabling Schema Compat
  [6/11]: starting directory server
  [7/11]: updating schema
  [8/11]: upgrading server
  [9/11]: stopping directory server
  [10/11]: restoring configuration
  [11/11]: starting directory server
Done.
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
Disabled p11-kit-proxy
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that KDC configuration is using ipa-kdb backend]
[Fix DS schema file syntax]
Syntax already fixed
[Removing RA cert from DS NSS database]
RA cert already removed
[Enable sidgen and extdom plugins by default]
[Updating HTTPD service IPA configuration]
[Updating HTTPD service IPA WSGI configuration]
Nothing to do for configure_httpd_wsgi_conf
[Migrating from mod_nss to mod_ssl]
Already migrated to mod_ssl
[Moving HTTPD service keytab to gssproxy]
[Removing self-signed CA]
[Removing Dogtag 9 CA]
[Checking for deprecated KDC configuration files]
[Checking for deprecated backups of Samba configuration files]
[Remove FILE: prefix from 'dedicated keytab file' in Samba configuration]
[Update 'max smbd processes' in Samba configuration to prevent unlimited SMBLoris attack amplification]
[Add missing CA DNS records]
IPA CA DNS records already processed
[Removing deprecated DNS configuration options]
[Ensuring minimal number of connections]
[Updating GSSAPI configuration in DNS]
[Updating pid-file configuration in DNS]
[Checking global forwarding policy in named.conf to avoid conflicts with automatic empty zones]
Changes to named.conf have been made, restart named
[Upgrading CA schema]
CA schema update complete (no changes)
[Verifying that CA audit signing cert has 2 year validity]
[Update certmonger certificate renewal configuration]
Certmonger certificate renewal configuration updated
[Enable PKIX certificate path discovery and validation]
PKIX already enabled
[Authorizing RA Agent to modify profiles]
[Authorizing RA Agent to manage lightweight CAs]
[Ensuring Lightweight CAs container exists in Dogtag database]
[Adding default OCSP URI configuration]
pki-tomcat configuration changed, restart pki-tomcat
[Ensuring CA is using LDAPProfileSubsystem]
[Migrating certificate profiles to LDAP]
[Ensuring presence of included profiles]
[Add default CA ACL]
Default CA ACL already added
[Migrating to authselect profile]
Already migrated to authselect profile
[Create systemd-user hbac service and rule]
hbac service systemd-user already exists
[Setup SPAKE]
[Setup PKINIT]
[Enable certauth]
The IPA services were upgraded
The ipa-server-upgrade command was successful

2019-09-30T12:51:52Z DEBUG Default Trust View already present on this server
2019-09-30T12:51:52Z DEBUG Executing upgrade plugin: update_tdo_gidnumber
2019-09-30T12:51:52Z DEBUG raw: update_tdo_gidnumber
2019-09-30T12:51:52Z DEBUG raw: adtrust_is_enabled(version='2.233')
2019-09-30T12:51:52Z DEBUG adtrust_is_enabled(version='2.233')
2019-09-30T12:51:52Z DEBUG Executing upgrade plugin: update_tdo_to_new_layout
2019-09-30T12:51:52Z DEBUG raw: update_tdo_to_new_layout
2019-09-30T12:51:52Z DEBUG raw: adtrust_is_enabled(version='2.233')
2019-09-30T12:51:52Z DEBUG adtrust_is_enabled(version='2.233')
2019-09-30T12:51:52Z DEBUG raw: trustconfig_show(version='2.233')
2019-09-30T12:51:52Z DEBUG trustconfig_show(rights=False, trust_type='ad', all=False, raw=False, version='2.233')
2019-09-30T12:51:52Z DEBUG Processing trust domain object cn=ipaad2k16cin.test,cn=ad,cn=trusts,dc=nd30sep,dc=ndpne
2019-09-30T12:51:52Z DEBUG Destroyed connection context.ldap2_139745159013544
2019-09-30T12:51:52Z ERROR Upgrade failed with name 'drsblobs' is not defined
2019-09-30T12:51:52Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ipaserver/install/upgradeinstance.py", line 276, in __upgrade
    self.modified = (ld.update(self.files) or self.modified)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/ldapupdate.py", line 965, in update
    self._run_updates(all_updates)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/ldapupdate.py", line 926, in _run_updates
    self._run_update_plugin(update['plugin'])
  File "/usr/lib/python3.6/site-packages/ipaserver/install/ldapupdate.py", line 901, in _run_update_plugin
    restart_ds, updates = self.api.Updater[plugin_name]()
  File "/usr/lib/python3.6/site-packages/ipalib/frontend.py", line 1480, in __call__
    return self.execute(**options)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/plugins/adtrust.py", line 667, in execute
    ndr_unpack(drsblobs.trustAuthInOutBlob,
NameError: name 'drsblobs' is not defined

Cloned to https://pagure.io/freeipa/issue/8085

Upstream PR: https://github.com/freeipa/freeipa/pull/3910

Fixed upstream
master:
https://pagure.io/freeipa/c/ba466a802129cbe61964653fdfddacd5d43f6771

ipa-4-8:
https://pagure.io/freeipa/c/18540386230e295087296e58761ced2b781ae4e3

ipa-4-7:
https://pagure.io/freeipa/c/2f8f257d9a9c076bf1a2d28aee06fbac0532aa72

ipa-4-6:
https://pagure.io/freeipa/c/fa23f5a13a326b4cedf6705be7d14da8bc813763

ipa-server version: ipa-server-4.8.4-2.module+el8.2.0+5265+c70de5c4.x86_64


Verified the bug on the basis of following observations:
1) Setup IPA server at RHEL8.1.0z with trust installed.

[ci-vm-10-0-155-4.hos] :: [ 04:35:38 ] :: [  BEGIN   ] :: Running ' /usr/sbin/ipa-server-install --setup-dns  --auto-forwarders --reverse-zone=155.0.10.in-addr.arpa. --allow-zone-overlap --hostname=ci-vm-10-0-155-4.testrelm.test -r TESTRELM.TEST -n testrelm.test -p Secret123 -a Secret123 --ip-address=10.0.155.4 -U'


[ci-vm-10-0-155-4.hos] :: [ 04:40:46 ] :: [   PASS   ] :: Command ' /usr/sbin/ipa-server-install --setup-dns  --auto-forwarders --reverse-zone=155.0.10.in-addr.arpa. --allow-zone-overlap --hostname=ci-vm-10-0-155-4.testrelm.test -r TESTRELM.TEST -n testrelm.test -p Secret123 -a Secret123 --ip-address=10.0.155.4 -U' (Expected 0, got 0)


[ci-vm-10-0-155-4.hos] :: [ 05:21:00 ] :: [  BEGIN   ] :: Running 'rpm -q ipa-server 389-ds-base bind bind-dyndb-ldap pki-ca sssd-ipa'
[ci-vm-10-0-155-4.hos] ipa-server-4.8.0-13.module+el8.1.0+4923+c6efe041.x86_64
[ci-vm-10-0-155-4.hos] 389-ds-base-1.4.1.3-7.module+el8.1.0+4150+5b8c2c1f.x86_64
[ci-vm-10-0-155-4.hos] bind-9.11.4-26.P2.el8.x86_64
[ci-vm-10-0-155-4.hos] bind-dyndb-ldap-11.1-14.module+el8.1.0+4098+f286395e.x86_64
[ci-vm-10-0-155-4.hos] pki-ca-10.7.3-1.module+el8.1.0+3964+500fc130.noarch
[ci-vm-10-0-155-4.hos] sssd-ipa-2.2.0-19.el8.x86_64
[ci-vm-10-0-155-4.hos] :: [ 05:21:00 ] :: [   PASS   ] :: Command 'rpm -q ipa-server 389-ds-base bind bind-dyndb-ldap pki-ca sssd-ipa' (Expected 0, got 0)

[ci-vm-10-0-155-4.hos] :: [ 05:00:52 ] :: [  BEGIN   ] :: Running 'echo Secret123 | ipa trust-add ipaad2k16cin.test --admin Administrator                 --range-type=ipa-ad-trust --password --two-way=True'
[ci-vm-10-0-154-5.hos] *** Current Time: Thu Jan 09 05:00:53 2020  Localwatchdog at: Fri Jan 10 04:03:52 2020
[ci-vm-10-0-155-4.hos] *** Current Time: Thu Jan 09 05:00:54 2020  Localwatchdog at: Fri Jan 10 04:03:54 2020
[ci-vm-10-0-155-4.hos] ----------------------------------------------------------
[ci-vm-10-0-155-4.hos] Added Active Directory trust for realm "ipaad2k16cin.test"
[ci-vm-10-0-155-4.hos] ----------------------------------------------------------
[ci-vm-10-0-155-4.hos]   Realm name: ipaad2k16cin.test
[ci-vm-10-0-155-4.hos]   Domain NetBIOS name: IPAAD2K16CIN
[ci-vm-10-0-155-4.hos]   Domain Security Identifier: S-1-5-21-2842256260-195550463-1751006347
[ci-vm-10-0-155-4.hos]   Trust direction: Two-way trust
[ci-vm-10-0-155-4.hos]   Trust type: Active Directory domain
[ci-vm-10-0-155-4.hos]   Trust status: Established and verified
[ci-vm-10-0-155-4.hos] :: [ 05:00:54 ] :: [   PASS   ] :: Command 'echo Secret123 | ipa trust-add ipaad2k16cin.test --admin Administrator                 --range-type=ipa-ad-trust --password --two-way=True' (Expected 0, got 0)
[ci-vm-10-0-155-4.hos] :: [ 05:00:54 ] :: [  BEGIN   ] :: Running 'systemctl stop sssd'
[ci-vm-10-0-155-4.hos] :: [ 05:00:54 ] :: [   PASS   ] :: Command 'systemctl stop sssd' (Expected 0, got 0)
[ci-vm-10-0-155-4.hos] :: [ 05:00:54 ] :: [  BEGIN   ] :: Running 'rm -frv /var/lib/sss/{db,mc}/*'
[ci-vm-10-0-155-4.hos] removed '/var/lib/sss/db/cache_implicit_files.ldb'
[ci-vm-10-0-155-4.hos] removed '/var/lib/sss/db/cache_testrelm.test.ldb'
[ci-vm-10-0-155-4.hos] removed '/var/lib/sss/db/ccache_TESTRELM.TEST'
[ci-vm-10-0-155-4.hos] removed '/var/lib/sss/db/config.ldb'
[ci-vm-10-0-155-4.hos] removed '/var/lib/sss/db/sssd.ldb'
[ci-vm-10-0-155-4.hos] removed '/var/lib/sss/db/timestamps_implicit_files.ldb'
[ci-vm-10-0-155-4.hos] removed '/var/lib/sss/db/timestamps_testrelm.test.ldb'
[ci-vm-10-0-155-4.hos] removed '/var/lib/sss/mc/group'
[ci-vm-10-0-155-4.hos] removed '/var/lib/sss/mc/initgroups'
[ci-vm-10-0-155-4.hos] removed '/var/lib/sss/mc/passwd'
[ci-vm-10-0-155-4.hos] :: [ 05:00:54 ] :: [   PASS   ] :: Command 'rm -frv /var/lib/sss/{db,mc}/*' (Expected 0, got 0)
[ci-vm-10-0-155-4.hos] :: [ 05:00:54 ] :: [  BEGIN   ] :: Running 'systemctl start sssd'
[ci-vm-10-0-155-4.hos] :: [ 05:00:55 ] :: [   PASS   ] :: Command 'systemctl start sssd' (Expected 0, got 0)
[ci-vm-10-0-155-4.hos] :: [ 05:00:55 ] :: [  BEGIN   ] :: Running 'ipa trust-find ipaad2k16cin.test'
[ci-vm-10-0-155-12.ho] *** Current Time: Thu Jan 09 05:00:54 2020  Localwatchdog at: Fri Jan 10 04:03:54 2020
[ci-vm-10-0-155-4.hos] ---------------
[ci-vm-10-0-155-4.hos] 1 trust matched
[ci-vm-10-0-155-4.hos] ---------------
[ci-vm-10-0-155-4.hos]   Realm name: ipaad2k16cin.test
[ci-vm-10-0-155-4.hos]   Domain NetBIOS name: IPAAD2K16CIN
[ci-vm-10-0-155-4.hos]   Domain Security Identifier: S-1-5-21-2842256260-195550463-1751006347
[ci-vm-10-0-155-4.hos]   Trust type: Active Directory domain
[ci-vm-10-0-155-4.hos]   UPN suffixes: tomupn14.in, upn2016.in, testupnsuffix.test, testupnsuffix
[ci-vm-10-0-155-4.hos] ----------------------------
[ci-vm-10-0-155-4.hos] Number of entries returned 1
[ci-vm-10-0-155-4.hos] ----------------------------
[ci-vm-10-0-155-4.hos] :: [ 05:00:56 ] :: [   PASS   ] :: Command 'ipa trust-find ipaad2k16cin.test' (Expected 0, got 0)
[ci-vm-10-0-155-4.hos] :: [ 05:06:56 ] :: [  BEGIN   ] :: Running 'id administrator'
[ci-vm-10-0-155-4.hos] uid=879000500(administrator) gid=879000500(administrator) groups=879000500(administrator),879000518(schema admins),879000519(enterprise admins),879000512(domain admins),879000513(domain users),879000520(group policy creator owners)
[ci-vm-10-0-155-4.hos] :: [ 05:06:56 ] :: [   PASS   ] :: Command 'id administrator' (Expected 0, got 0)




2) Upgrade the IPA server to RHEL 8.2.0

[ci-vm-10-0-155-4.hos] :: [ 05:21:16 ] :: [  BEGIN   ] :: Initiating upgrade Process :: actually running 'yum -y update'
.
.
.
[ci-vm-10-0-155-4.hos] :: [ 05:30:49 ] :: [   PASS   ] :: Initiating upgrade Process (Expected 0, got 0)
[ci-vm-10-0-155-4.hos] :: [ 05:30:49 ] :: [  BEGIN   ] :: Running 'tail -1 /var/log/ipaupgrade.log | grep 'The ipa-server-upgrade command was successful''
[ci-vm-10-0-155-4.hos] 2020-01-09T10:28:27Z INFO The ipa-server-upgrade command was successful
[ci-vm-10-0-155-4.hos] :: [ 05:30:49 ] :: [   PASS   ] :: Command 'tail -1 /var/log/ipaupgrade.log | grep 'The ipa-server-upgrade command was successful'' (Expected 0, got 0)

[ci-vm-10-0-155-4.hos] :: [ 05:33:31 ] :: [  BEGIN   ] :: Running 'rpm -q ipa-server 389-ds-base bind bind-dyndb-ldap pki-ca sssd-ipa'
[ci-vm-10-0-155-4.hos] ipa-server-4.8.4-2.module+el8.2.0+5265+c70de5c4.x86_64
[ci-vm-10-0-155-4.hos] 389-ds-base-1.4.2.4-4.module+el8.2.0+4930+d4051b3a.x86_64
[ci-vm-10-0-155-4.hos] bind-9.11.13-1.el8.x86_64
[ci-vm-10-0-155-4.hos] bind-dyndb-ldap-11.2-3.module+el8.2.0+4921+923e30d5.x86_64
[ci-vm-10-0-155-4.hos] pki-ca-10.8.0-0.4.module+el8.2.0+5228+a2bc7b32.noarch
[ci-vm-10-0-155-4.hos] sssd-ipa-2.2.3-6.el8.x86_64
[ci-vm-10-0-155-4.hos] :: [ 05:33:31 ] :: [   PASS   ] :: Command 'rpm -q ipa-server 389-ds-base bind bind-dyndb-ldap pki-ca sssd-ipa' (Expected 0, got 0)

3) Validate trust after upgrade using id command
[ci-vm-10-0-155-4.hos] :: [ 05:41:35 ] :: [  BEGIN   ] :: Running 'ipa trust-find ipaad2k16cin.test'
[ci-vm-10-0-155-4.hos] ---------------
[ci-vm-10-0-155-4.hos] 1 trust matched
[ci-vm-10-0-155-4.hos] ---------------
[ci-vm-10-0-155-4.hos]   Realm name: ipaad2k16cin.test
[ci-vm-10-0-155-4.hos]   Domain NetBIOS name: IPAAD2K16CIN
[ci-vm-10-0-155-4.hos]   Domain Security Identifier: S-1-5-21-2842256260-195550463-1751006347
[ci-vm-10-0-155-4.hos]   Trust type: Active Directory domain
[ci-vm-10-0-155-4.hos]   UPN suffixes: tomupn14.in, upn2016.in, testupnsuffix.test, testupnsuffix
[ci-vm-10-0-155-4.hos] ----------------------------
[ci-vm-10-0-155-4.hos] Number of entries returned 1
[ci-vm-10-0-155-4.hos] ----------------------------
[ci-vm-10-0-155-4.hos] :: [ 05:41:36 ] :: [   PASS   ] :: Command 'ipa trust-find ipaad2k16cin.test' (Expected 0, got 0)
[ci-vm-10-0-155-4.hos] :: [ 05:41:36 ] :: [  BEGIN   ] :: Running 'id administrator'
[ci-vm-10-0-155-4.hos] uid=879000500(administrator) gid=879000500(administrator) groups=879000500(administrator),879000518(schema admins),879000519(enterprise admins),879000512(domain admins),879000513(domain users),879000520(group policy creator owners)
[ci-vm-10-0-155-4.hos] :: [ 05:41:36 ] :: [   PASS   ] :: Command 'id administrator' (Expected 0, got 0)


Thus on the basis of above observations marking the status of bug to "VERIFIED"

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:1640