Bug 1376676

1). Checking nsDS5ReplicaCredentials attribute value with older version of 389-ds-base on RHEL6.8
[root@vm-idm-006 dirsrv]# rpm -qa |grep -i 389-ds
389-ds-base-libs-1.2.11.15-74.el6.x86_64
389-ds-base-debuginfo-1.2.11.15-81.el6_8.x86_64
389-ds-base-1.2.11.15-74.el6.x86_64
389-ds-base-devel-1.2.11.15-74.el6.x86_64

[root@vm-idm-006 dirsrv]# ldapsearch -LLL -x -p 1189 -h localhost -D "cn=Directory Manager" -w Secret123 -b "cn=replica,cn=\"dc=passsync,dc=com\",cn=mapping tree,cn=config" nsDS5ReplicaCredentials
dn: cn=replica,cn=dc\3Dpasssync\2Cdc\3Dcom,cn=mapping tree,cn=config

dn: cn=1189_to_1389_on_vm-idm-006.lab.eng.pnq.redhat.com,cn=replica,cn=dc\3Dpa
 sssync\2Cdc\3Dcom,cn=mapping tree,cn=config
nsDS5ReplicaCredentials: {DES}2mO8LgMr/w5ognn2BRK+gQ==

dn: cn=1189_to_1489_on_vm-idm-006.lab.eng.pnq.redhat.com,cn=replica,cn=dc\3Dpa
 sssync\2Cdc\3Dcom,cn=mapping tree,cn=config
nsDS5ReplicaCredentials: {DES}2mO8LgMr/w5ognn2BRK+gQ==

dn: cn=1189_to_1626_on_vm-idm-006.lab.eng.pnq.redhat.com,cn=replica,cn=dc\3Dpa
 sssync\2Cdc\3Dcom,cn=mapping tree,cn=config
nsDS5ReplicaCredentials: {DES}2mO8LgMr/w5ognn2BRK+gQ==

dn: cn=1189_to_2616_on_vm-idm-006.lab.eng.pnq.redhat.com,cn=replica,cn=dc\3Dpa
 sssync\2Cdc\3Dcom,cn=mapping tree,cn=config
nsDS5ReplicaCredentials: {DES}2mO8LgMr/w5ognn2BRK+gQ==

dn: cn=1189_to_2626_on_vm-idm-006.lab.eng.pnq.redhat.com,cn=replica,cn=dc\3Dpa
 sssync\2Cdc\3Dcom,cn=mapping tree,cn=config
nsDS5ReplicaCredentials: {DES}2mO8LgMr/w5ognn2BRK+gQ==

2). Upgrade the server and restart the instances.

[root@vm-idm-006 dirsrv]# rpm -qa |grep -i 389-ds
389-ds-base-1.2.11.15-81.el6_8.x86_64
389-ds-base-debuginfo-1.2.11.15-81.el6_8.x86_64
389-ds-base-libs-1.2.11.15-81.el6_8.x86_64
389-ds-base-devel-1.2.11.15-81.el6_8.x86_64

[root@vm-idm-006 dirsrv]# service dirsrv restart
Shutting down dirsrv: 
    C1...[  OK  ]
    C2...[  OK  ]
    M1...[  OK  ]
    M2...[  OK  ]
    M3...[  OK  ]
    M4...[  OK  ]
Starting dirsrv: 
    C1...[26/Oct/2016:16:26:12 +051800] - Netscape Portable Runtime error -5977: /usr/lib64/dirsrv/plugins/libdes-plugin.so: cannot open shared object file: No such file or directory
[26/Oct/2016:16:26:12 +051800] - Could not open library "/usr/lib64/dirsrv/plugins/libdes-plugin.so" for plugin DES
[26/Oct/2016:16:26:12 +051800] - The plugin entry [cn=DES,cn=Password Storage Schemes,cn=plugins,cn=config] in the configfile /etc/dirsrv/slapd-C1/dse.ldif was invalid
[26/Oct/2016:16:26:12 +051800] startup - The configuration files in directory /etc/dirsrv/slapd-C1 could not be read or were not found.  Please refer to the error log or output for more information.
[FAILED]
    C2...[26/Oct/2016:16:26:12 +051800] - Netscape Portable Runtime error -5977: /usr/lib64/dirsrv/plugins/libdes-plugin.so: cannot open shared object file: No such file or directory
[26/Oct/2016:16:26:12 +051800] - Could not open library "/usr/lib64/dirsrv/plugins/libdes-plugin.so" for plugin DES
[26/Oct/2016:16:26:12 +051800] - The plugin entry [cn=DES,cn=Password Storage Schemes,cn=plugins,cn=config] in the configfile /etc/dirsrv/slapd-C2/dse.ldif was invalid
[26/Oct/2016:16:26:12 +051800] startup - The configuration files in directory /etc/dirsrv/slapd-C2 could not be read or were not found.  Please refer to the error log or output for more information.

It looks like a regression. Please confirm.

Looks like the upgrade scripts were not called.  Perhaps "setup-ds.pl -u" was not called?  Can you try running it manually to see if it helps?

I'll try and reproduce this on a beaker box as well.

(In reply to mreynolds from comment #8)
> Looks like the upgrade scripts were not called.  Perhaps "setup-ds.pl -u"
> was not called?  Can you try running it manually to see if it helps?
> 
> I'll try and reproduce this on a beaker box as well.

This time, I ran setup-ds.pl -u after upgrading the packages. However, the result is observed. Restart of instances failed with the same error as comment #7

Based on comment #8 and comment #9, marking the bug as assigned.

1).
root@vm-idm-006 ~]# rpm -qa |grep -i 389-ds
389-ds-base-1.2.11.15-74.el6.x86_64
389-ds-base-libs-1.2.11.15-74.el6.x86_64
389-ds-base-devel-1.2.11.15-74.el6.x86_64

2). Then, I created 4 way MMR. Encryption with DES
 
# replica, dc\3Dpasssync\2Cdc\3Dcom, mapping tree, config
dn: cn=replica,cn=dc\3Dpasssync\2Cdc\3Dcom,cn=mapping tree,cn=config

# 1189_to_1389_on_vm-idm-006.lab.eng.pnq.redhat.com, replica, dc\3Dpasssync\2Cd
 c\3Dcom, mapping tree, config
dn: cn=1189_to_1389_on_vm-idm-006.lab.eng.pnq.redhat.com,cn=replica,cn=dc\3Dpa
 sssync\2Cdc\3Dcom,cn=mapping tree,cn=config
nsDS5ReplicaCredentials: {DES}2mO8LgMr/w5ognn2BRK+gQ==

3). Upgraded the packages to 389-ds-base-1.2.11.15-82. Ran setup-ds.pl -u to complete the upgrade process.
[root@vm-idm-006 MMR_WINSYNC]# rpm -qa |grep -i 389-ds
389-ds-base-1.2.11.15-82.el6_8.x86_64
389-ds-base-debuginfo-1.2.11.15-82.el6_8.x86_64
389-ds-base-libs-1.2.11.15-82.el6_8.x86_64
389-ds-base-devel-1.2.11.15-82.el6_8.x86_64

4). Restarted directory server instances. 
[root@vm-idm-006 MMR_WINSYNC]# service dirsrv restart
Shutting down dirsrv: 
    C1...[  OK  ]
    C2...[  OK  ]
    M1...[  OK  ]
    M2...[  OK  ]
Starting dirsrv: 
    C1...[28/Oct/2016:17:38:11 +051800] - Netscape Portable Runtime error -5977: /usr/lib64/dirsrv/plugins/libdes-plugin.so: cannot open shared object file: No such file or directory
[28/Oct/2016:17:38:11 +051800] - Could not open library "/usr/lib64/dirsrv/plugins/libdes-plugin.so" for plugin DES
[28/Oct/2016:17:38:11 +051800] - The plugin entry [cn=DES,cn=Password Storage Schemes,cn=plugins,cn=config] in the configfile /etc/dirsrv/slapd-C1/dse.ldif was invalid
[28/Oct/2016:17:38:11 +051800] startup - The configuration files in directory /etc/dirsrv/slapd-C1 could not be read or were not found.  Please refer to the error log or output for more information.
[FAILED]
    C2...[28/Oct/2016:17:38:11 +051800] - Netscape Portable Runtime error -5977: /usr/lib64/dirsrv/plugins/libdes-plugin.so: cannot open shared object file: No such file or directory
[28/Oct/2016:17:38:11 +051800] - Could not open library "/usr/lib64/dirsrv/plugins/libdes-plugin.so" for plugin DES
[28/Oct/2016:17:38:11 +051800] - The plugin entry [cn=DES,cn=Password Storage Schemes,cn=plugins,cn=config] in the configfile /etc/dirsrv/slapd-C2/dse.ldif was invalid
[28/Oct/2016:17:38:11 +051800] startup - The configuration files in directory /etc/dirsrv/slapd-C2 could not be read or were not found.  Please refer to the error log or output for more information.


I will retest with a fresh beaker machine and update my comments here in few hours.

389-ds-base-1.2.11.15-84.el6 still has the same problem (but possibly improved).

I can see that the upgrade script is present on the system, but there is no AES plugin entry in cn=config so the script does not complete:

[28/Oct/2016:10:43:05 -0400] conn=2 op=42 SRCH base="cn=AES,cn=Password Storage Schemes,cn=plugins,cn=config" scope=0 filter="(cn=*)" attrs=ALL
[28/Oct/2016:10:43:05 -0400] conn=2 op=42 RESULT err=32 tag=101 nentries=0 etime=0

/usr/share/dirsrv/updates/52updateAESplugin.pl
...
    my $aes_dn = "cn=AES,cn=Password Storage Schemes,cn=plugins,cn=config";
    my $aes_entry = $conn->search($aes_dn, "base", "(cn=*)");
    if (!$aes_entry) {
        # No AES plugin - nothing to do
        return ();
    }

The AES plugin is present in /usr/share/dirsrv/data/template-dse.ldif, but it's not updating the current dse.ldif when running "setup-ds.pl -u".  Continuing to investigate...

The upgrade script should create the AES plugin entry if it does not exist.

Created upstream ticket:

https://fedorahosted.org/389/ticket/49023

(In reply to mreynolds from comment #13)
> The upgrade script should create the AES plugin entry if it does not exist.
> 
> Created upstream ticket:
> 
> https://fedorahosted.org/389/ticket/49023

This ticket is invalid.  The real problem is that 50AES-pbe-plugin.ldif is missing from the Makefile

Should we also verify if its backward compatible? I mean, downgrade from the latest 389-ds-base(which has the complete fix for AES plugin) to an older version of 389-ds-base(which supports DES by default) and check if server restarts fine and replication works.

(In reply to mreynolds from comment #14)
> (In reply to mreynolds from comment #13)
> > The upgrade script should create the AES plugin entry if it does not exist.
> > 
> > Created upstream ticket:
> > 
> > https://fedorahosted.org/389/ticket/49023
> 
> This ticket is invalid.  The real problem is that 50AES-pbe-plugin.ldif is
> missing from the Makefile

Ah...  Sorry, Mark and Sankar...

We removed autoconf artifacts from RHEL-6.9.  But RHEL-6.8 still has them.  I should have rerun autogen and push them to the tree... :(  

Let me redo it now.

It makes me rethink...  Do we rather want to apply the change -- removing artifacts to RHEL-6.8, as well?  What do you think, Mark?

[root@vm-idm-006 MMR_WINSYNC]# rpm -qa |grep -i 389-ds
389-ds-base-libs-1.2.11.15-74.el6.x86_64
389-ds-base-1.2.11.15-74.el6.x86_64

[root@vm-idm-006 ~]# ldapsearch -x -p 1289 -h localhost -D "cn=Directory Manager" -w Secret123 -b "cn=replica,cn=\"dc=passsync,dc=com\",cn=mapping tree,cn=config" nsDS5ReplicaCredentials |grep -i nsDS5ReplicaCredentials 
# requesting: nsDS5ReplicaCredentials 
nsDS5ReplicaCredentials: {DES}/2IL8UbcCrGTG5/757YC1g==
nsDS5ReplicaCredentials: {DES}/2IL8UbcCrGTG5/757YC1g==
nsDS5ReplicaCredentials: {DES}/2IL8UbcCrGTG5/757YC1g==

[root@vm-idm-006 ~]# yum -y update

[root@vm-idm-006 ~]# setup-ds.pl -u

[root@vm-idm-006 ~]# service dirsrv restart

[root@vm-idm-006 MMR_WINSYNC]# ldapsearch -x -p 1289 -h localhost -D "cn=Directory Manager" -w Secret123 -b "cn=replica,cn=\"dc=passsync,dc=com\",cn=mapping tree,cn=config" nsDS5ReplicaCredentials |grep -i nsDS5ReplicaCredentials 
# requesting: nsDS5ReplicaCredentials 
nsDS5ReplicaCredentials: {AES-TUhNR0NTcUdTSWIzRFFFRkRUQm1NRVVHQ1NxR1NJYjNEUUVG
nsDS5ReplicaCredentials: {AES-TUhNR0NTcUdTSWIzRFFFRkRUQm1NRVVHQ1NxR1NJYjNEUUVG
nsDS5ReplicaCredentials: {AES-TUhNR0NTcUdTSWIzRFFFRkRUQm1NRVVHQ1NxR1NJYjNEUUVG

[root@vm-idm-006 MMR_WINSYNC]# ./AddEntry.sh Users 1189 "ou=people,dc=passsync,dc=com" utestnew 999 localhost

[root@vm-idm-006 ~]# #PORT=1189; SUFF="dc=passsync,dc=com"; for PORT in `echo "1189 1289 1389 1489"`; do /usr/bin/ldapsearch -x -p $PORT -h localhost -D "cn=Directory Manager" -w Secret123 -b $SUFF |grep -i dn: | wc -l ; done
[root@vm-idm-006 ~]# PORT=1189; SUFF="dc=passsync,dc=com"; for PORT in `echo "1189 1289 1389 1489"`; do /usr/bin/ldapsearch -x -p $PORT -h localhost -D "cn=Directory Manager" -w Secret123 -b $SUFF |grep -i dn: | wc -l ; done
1046
1046
1046
1046

[root@vm-idm-006 MMR_WINSYNC]# rpm -qa |grep -i 389-ds
389-ds-base-libs-1.2.11.15-84.el6_8.x86_64
389-ds-base-1.2.11.15-84.el6_8.x86_64

Upgrade is working fine with the latest build of 389-ds-base-1.2.11.15-84. Hence, marking the bug as Verified.

However, the downgrade tests are failing. I heard from Viktor that automated way of downgrading may not be possible at this time. So, we need to document the steps for downgrade tests.

Here is the output...
    M1...[08/Nov/2016:08:41:01 +051800] - Netscape Portable Runtime error -5977: /usr/lib64/dirsrv/plugins/libpbe-plugin.so: cannot open shared object file: No such file or directory
[08/Nov/2016:08:41:01 +051800] - Could not open library "/usr/lib64/dirsrv/plugins/libpbe-plugin.so" for plugin AES
[08/Nov/2016:08:41:01 +051800] - The plugin entry [cn=AES,cn=Password Storage Schemes,cn=plugins,cn=config] in the configfile /etc/dirsrv/slapd-M1/dse.ldif was invalid
[08/Nov/2016:08:41:01 +051800] startup - The configuration files in directory /etc/dirsrv/slapd-M1 could not be read or were not found.  Please refer to the error log or output for more information.
[FAILED]
    M2...[08/Nov/2016:08:41:01 +051800] - Netscape Portable Runtime error -5977: /usr/lib64/dirsrv/plugins/libpbe-plugin.so: cannot open shared object file: No such file or directory
[08/Nov/2016:08:41:01 +051800] - Could not open library "/usr/lib64/dirsrv/plugins/libpbe-plugin.so" for plugin AES
[08/Nov/2016:08:41:01 +051800] - The plugin entry [cn=AES,cn=Password Storage Schemes,cn=plugins,cn=config] in the configfile /etc/dirsrv/slapd-M2/dse.ldif was invalid
[08/Nov/2016:08:41:01 +051800] startup - The configuration files in directory /etc/dirsrv/slapd-M2 could not be read or were not found.  Please refer to the error log or output for more information.
[FAILED]
  *** Error: 4 instance(s) failed to start

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2765.html