Bug 1903350

Description of problem:

( could not select the python3-lib389 component in this bug report)

dsconf replication status fails when replication is set with SSL or TLS, with error "Can't contact LDAP server" / "error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed (self signed certificate in certificate chain)"

also related to
https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org/thread/GVKK5EGFPJQYVJACG5LK7S7ZMFAW3GFZ/
https://github.com/389ds/389-ds-base/issues/4460


the dsconf replication status command seem to use a default value from
/usr/lib/python3.6/site-packages/lib389/__init__.py
and
./src/lib389/lib389/_constants.py:                 SER_SERVERID_PROP: "template",

and it seem the cert_dir may not be initialized, in

/src/lib389/lib389/__init__.py
...
class DirSrv(SimpleLDAPObject, object):

    def get_cert_dir(self):
        return self.ds_paths.cert_dir

it seems like there are a few other non initialized paths in the class DirSrv



dsconf -v m1 replication status --suffix dc=example,dc=test --bind-dn "cn=Directory Manager" --bind-passwd ${DM_BIND_PW}
...snip...
DEBUG: Using external ca certificate /etc/dirsrv/slapd-template
DEBUG: Using certificate policy 1
DEBUG: ldap.OPT_X_TLS_REQUIRE_CERT = 1
DEBUG: Cannot connect to 'ldaps://192.168.122.112:636/'
DEBUG: Connection to consumer (192.168.122.112:389) failed, error: {'desc': "Can't contact LDAP server", 'info': 'error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed (self signed certificate in certificate chain)'}


but an OpenSSL command works just fine:
echo "Q\n" | openssl s_client -connect m1.example.test:636  -no_ign_eof  -state | less
as well as
echo "Q\n" | openssl s_client -connect m2.example.test:636  -no_ign_eof  -state | less
CONNECTED(00000003)
---
Certificate chain
 0 s:CN = m2.example.test
   i:O = example.test, CN = Subordinate CA1 Signing Certificate
 1 s:O = example.test, CN = Subordinate CA1 Signing Certificate
   i:O = Root CA1 Example Test, OU = rootca1, CN = CA Signing Certificate
 2 s:O = Root CA1 Example Test, OU = rootca1, CN = CA Signing Certificate
   i:O = Root CA1 Example Test, OU = rootca1, CN = CA Signing Certificate
---
Server certificate
...snip...
Acceptable client certificate CA names
O = example.test, CN = Subordinate CA1 Signing Certificate
O = Root CA1 Example Test, OU = rootca1, CN = CA Signing Certificate
C = AU, ST = Queensland, L = 389ds, O = testing, CN = ssca.389ds.example.com
Client Certificate Types: RSA sign, ECDSA sign, DSA sign
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:ECDSA+SHA1:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:RSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:ECDSA+SHA1:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:RSA+SHA1
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3604 bytes and written 428 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 7EBE29FCE5887EB794AA9ED39B54A7E1E6B5BCB8D87187E9875100FEC032D203
    Session-ID-ctx: 
    Master-Key: E94291681880FC8E7E04826C9E02F7B7D9A68E5F77972CD4F94700A97406646D492712060F972D0112ACF05FFE4647BF
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1606856229
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---
(END)


as well as a LDAPS search, from "m1" to "m2":

ldapsearch -o ldif-wrap=no -LLLH ldaps://m2.example.test:636 -D "cn=directory manager" -w password -s base -b "" 'objectclass=*' vendorVersion
dn:
vendorVersion: 389-Directory/1.4.2.12 B2020.119.1433



Version-Release number of selected component (if applicable):
RHEL-8.2
389-ds-base-1.4.2.12-2.module+el8dsrv+6428+6e54c518.x86_64
python3-lib389-1.4.2.12-2.module+el8dsrv+6428+6e54c518.noarch


How reproducible:
on demand


Steps to Reproduce:

1. create 2 instances, example for first one:

groupadd -r ldapgroup1
useradd -r -g  ldapgroup1 ldapuser1
dscreate create-template ~/ds11.instance.inf
sed     -e 's/;instance_name = .*/instance_name = m1/g'     -e 's/;user = .*/user = ldapuser1/g'     -e 's/;group = .*/group = ldapgroup1/g'     -e 's/;root_password = .*/root_password = password/g'     -e 's/;suffix = .*/suffix = dc=example,dc=test/g'     ~/ds11.instance.inf > ~/ds11.m1.instance.inf
dscreate from-file ~/ds11.m1.instance.inf

create a second instance
and set replication as 2x mmr


2. prepare for 2 SSL server certificates issued by a subordinate CA, trust chain example for first one:

trust anchor ~/rootca1.crt 
trust anchor ~/subca1.crt 

trust list ca-anchors | grep -i ca1
[root@m1 slapd-m1]# 

trust list | grep -i ca1
    label: Subordinate CA1 Signing Certificate
    label: SecureSign RootCA11
[root@m1 slapd-m1]# 


3. add the new certificates, with the SSL server cert issued by a subordinate CA

note: did not used the "dsctl m1 tls import commands" nor
dsctl m1 tls generate-server-cert-csr --help
dsctl m1 tls import-server-cert --help
( should I ?) , and instead, did a pk12util import:

dsctl m1 stop
pk12util -i /etc/dirsrv/slapd-m1/ssl.server.cert.m1.from.subca1.p12 -d /etc/dirsrv/slapd-m1/ -k /etc/dirsrv/slapd-m1/pwdfile.txt -W password -n sslserverm1fromsubca1
and then changed the nsSSLPersonalitySSL in the web UI

certutil -O -d /etc/dirsrv/slapd-m1 -n sslserverm1fromsubca1
"rootca1" [CN=CA Signing Certificate,OU=rootca1,O=Root CA1 Example Test]
  "subca1" [CN=Subordinate CA1 Signing Certificate,O=example.test]
    "sslserverm1fromsubca1" [CN=m1.example.test]

dsctl m1 start


3. did some replication tests, example
/usr/bin/ldclt -D "cn=directory manager" -w password -h localhost -p 389 -s subtree -b "ou=people,dc=example,dc=test" -e randombinddn,randombinddnlow=100,randombinddnhigh=999 -e random,attreplace=roomNumber:roomXXX -r 1 -R 19  -n 2 -I 32 -q -f "uid=guestXX"


4. verify replication status:
dsconf -v m1 replication status --suffix dc=example,dc=test --bind-dn "cn=Directory Manager" --bind-passwd ${DM_BIND_PW}
...snip...


Actual results:

...snip...
DEBUG: _gen_selector filter = (&(&(objectclass=nsds5Replica))(|(nsDS5ReplicaRoot=dc=example,dc=test)))
DEBUG: cn=replica,cn=dc\3Dexample\2Cdc\3Dtest,cn=mapping tree,cn=config getVal('nsDS5ReplicaRoot')
DEBUG: cn=replica,cn=dc\3Dexample\2Cdc\3Dtest,cn=mapping tree,cn=config getVal('nsDS5ReplicaId')
DEBUG: Allocate <class 'lib389.DirSrv'> with None
DEBUG: Allocate <class 'lib389.DirSrv'> with 192.168.122.112:636
DEBUG: Allocate <class 'lib389.DirSrv'> with 192.168.122.112:636
DEBUG: open(): Connecting to uri ldaps://192.168.122.112:636/
DEBUG: Using dirsrv ca certificate /etc/dirsrv/slapd-template
DEBUG: Using external ca certificate /etc/dirsrv/slapd-template
DEBUG: Using external ca certificate /etc/dirsrv/slapd-template
DEBUG: Using certificate policy 1
DEBUG: ldap.OPT_X_TLS_REQUIRE_CERT = 1
DEBUG: Cannot connect to 'ldaps://192.168.122.112:636/'
DEBUG: Connection to consumer (192.168.122.112:389) failed, error: {'desc': "Can't contact LDAP server", 'info': 'error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed (self signed certificate in certificate chain)'}
INFO: {'agmt-name': ['m1tom2exampleSSL'], 'replica': ['192.168.122.112:636'], 'replica-enabled': ['on'], 'update-in-progress': ['FALSE'], 'last-update-start': ['19700101000000Z'], 'last-update-end': ['19700101000000Z'], 'number-changes-sent': ['0'], 'number-changes-skipped': ['unavailable'], 'last-update-status': ["Error (-1) Problem connecting to replica - LDAP error: Can't contact LDAP server (connection error)"], 'last-init-start': ['19700101000000Z'], 'last-init-end': ['19700101000000Z'], 'last-init-status': ['unavailable'], 'reap-active': ['0'], 'replication-status': ['Not in Synchronization: supplier (Unknown) consumer (Unknown) State (red) Reason (Consumer can not be contacted)'], 'replication-lag-time': ['Unavailable']}
INFO: Command successful.
[root@m1 slapd-m1]# 


Expected results:
yes


Additional info:

related to

https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org/thread/GVKK5EGFPJQYVJACG5LK7S7ZMFAW3GFZ/
https://github.com/389ds/389-ds-base/issues/4460


attempted workaround resulting in a different error:  duplicate the etc instance directory into /etc/dirsrv/slapd-template/ and rehash the certs: "TLS: hostname does not match peer certificate"

# on m1 and m2:
rsync -av /etc/dirsrv/slapd-m1/ /etc/dirsrv/slapd-template/

# on m1:
certutil -L -d /etc/dirsrv/slapd-template/ -n sslserverm1fromsubca1 -a > /etc/dirsrv/slapd-template/sslserverm1fromsubca1.crt

# on m2:
certutil -L -d /etc/dirsrv/slapd-template/ -n sslserverm2fromsubca1 -a > /etc/dirsrv/slapd-template/sslserverm2fromsubca1.crt
certutil -L -d /etc/dirsrv/slapd-template/ -n subca1 -a > /etc/dirsrv/slapd-template/subca1.crt
certutil -L -d /etc/dirsrv/slapd-template/ -n rootca1 -a > /etc/dirsrv/slapd-template/rootca1.crt
openssl rehash  /etc/dirsrv/slapd-template/
ls -l /etc/dirsrv/slapd-template/

# on m1:
dsconf -v m1 replication status --suffix dc=example,dc=test --bind-dn "cn=Directory Manager" --bind-passwd ${DM_BIND_PW}
...snip...
DEBUG: Allocate <class 'lib389.DirSrv'> with 192.168.122.112:636
DEBUG: open(): Connecting to uri ldaps://192.168.122.112:636/
DEBUG: Using dirsrv ca certificate /etc/dirsrv/slapd-template
DEBUG: Using external ca certificate /etc/dirsrv/slapd-template
DEBUG: Using external ca certificate /etc/dirsrv/slapd-template
DEBUG: Using certificate policy 1
DEBUG: ldap.OPT_X_TLS_REQUIRE_CERT = 1
DEBUG: Cannot connect to 'ldaps://192.168.122.112:636/'
DEBUG: Connection to consumer (192.168.122.112:389) failed, error: {'desc': "Can't contact LDAP server", 'info': 'TLS: hostname does not match peer certificate'}
INFO: {'agmt-name': ['m1tom2exampleSSL'], 'replica': ['192.168.122.112:636'], 'replica-enabled': ['on'], 'update-in-progress': ['FALSE'], 'last-update-start': ['19700101000000Z'], 'last-update-end': ['19700101000000Z'], 'number-changes-sent': ['0'], 'number-changes-skipped': ['unavailable'], 'last-update-status': ["Error (-1) Problem connecting to replica - LDAP error: Can't contact LDAP server (connection error)"], 'las
t-init-start': ['19700101000000Z'], 'last-init-end': ['19700101000000Z'], 'last-init-status': ['unavailable'], 'reap-active': ['0'], 'replication-status': ['Not in Synchronization: supplier (Unknown) consumer (Unknown) State (red) Reason (Consumer can not be contacted)'], 'replication-lag-time': ['Unavailable']}
INFO: Command successful.


which is strange as both the "m1" and "m2" SSL server ceritifcates do have a SAN extension populated with a correct corresponding DNS name value:
(hostname correct, but DNS resolution could be the cause in my test, but was not in a customer case with the same exact problem)

certutil -L -d ${nssdir} -n sslserverm1fromsubca1 | less
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 8 (0x8)
        Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
        Issuer: "CN=Subordinate CA1 Signing Certificate,O=example.test"
        Validity:
            Not Before: Mon Nov 23 05:59:20 2020
            Not After : Wed Feb 23 01:31:07 2022
        Subject: "CN=m1.example.test"
...snip...
        Signed Extensions:
            Name: Certificate Authority Key Identifier
            Key ID:
                e2:45:24:43:7c:b1:ee:f1:a3:44:57:2d:54:87:b7:6f:
                19:91:8f:aa

            Name: Certificate Subject Alt Name
            DNS name: "m1.example.test"

            Name: Authority Information Access
            Method: PKIX Online Certificate Status Protocol
            Location: 
                URI: "http://ca1.example.test:8444/ca/ocsp"

            Name: Certificate Key Usage
            Critical: True
            Usages: Digital Signature
                    Key Encipherment
                    Data Encipherment

            Name: Extended Key Usage
                TLS Web Server Authentication Certificate
                TLS Web Client Authentication Certificate


and

certutil -L -d ${nssdir} -n sslserverm2fromsubca1 | less
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 9 (0x9)
        Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
        Issuer: "CN=Subordinate CA1 Signing Certificate,O=example.test"
        Validity:
            Not Before: Mon Nov 23 06:00:17 2020
            Not After : Wed Feb 23 01:31:07 2022
        Subject: "CN=m2.example.test"
...snip...
        Signed Extensions:
            Name: Certificate Authority Key Identifier
            Key ID:
                e2:45:24:43:7c:b1:ee:f1:a3:44:57:2d:54:87:b7:6f:
                19:91:8f:aa

            Name: Certificate Subject Alt Name
            DNS name: "m2.example.test"

            Name: Authority Information Access
            Method: PKIX Online Certificate Status Protocol
            Location: 
                URI: "http://ca1.example.test:8444/ca/ocsp"

            Name: Certificate Key Usage
            Critical: True
            Usages: Digital Signature
                    Key Encipherment
                    Data Encipherment

            Name: Extended Key Usage
                TLS Web Server Authentication Certificate
                TLS Web Client Authentication Certificate
...snip...