Customer ran:

openstack tripleo container image prepare default \
  --output-env-file containers-prepare-parameter.yaml

to generate the containers prepare yaml, and then added the following to the end of it:

- push_destination: true
    includes:
    - openstack-keystone
    set:
      tag: 14.0-102.1557945123

That then fixed their issues.

I can take this. I was active in the review which added this ldap change to keystone and if memory serves me correctly I discovered this exact issue and filed an upstream bug against python-ldap. My assumption is the version of python-ldap paired with keystone does not have the fix. I'll investigate and add more info when I have it.

The upstream bug for initialize() got an unexpected keyword argument 'bytes_mode' is https://github.com/python-ldap/python-ldap/issues/249.
Upstream committed fixes the first week of January 2019.
The upstream fixes first appeared in python-ldap-3.2.0 which was released 2019-03-13.

I need to know the version of python-ldap that was installed. Unfortunately the list of RPM's in the SOS report is empty. (Plus downloading SOS reports is painfully slow). Can you please find out what the version of python-ldap was installed.

That upstream bug is about the new parameter bytes_strictness, not bytes_mode.

I'm not installing any version of python-ldap, I'm using the docker container images provided by RedHat.  You can pull the container and inspect it yourself if you want.  https://access.redhat.com/containers/?tab=security#/registry.access.redhat.com/rhosp14/openstack-keystone/images/14.0-107

As far as I can tell, the rhosp14/openstack-keystone images only have python-ldap 2.4.15-2 which does not support bytes_mode as it was added sometime in the 5 years since 2.4.15 was released.  Keystone project updated their requirement to python-ldap>=3.0.0 about a year ago. More recently, they added use of the bytes_mode parameter. Seems like the building of the rhosp14/openstack-keystone image needs to update what version of python-ldap it pulls in.

*** Bug 1730132 has been marked as a duplicate of this bug. ***

For the record, updating python-ldap to >= 3.0 requires updating pyasn1 to >= 0.3.7 and pyasn1-modules to >= 0.1.5 also coming from RHEL base repos.

I've proposed to update python-ldap and python-pyasn1 from versions in rhel base to required versions in RDO Rocky https://review.rdoproject.org/r/#/c/21643/

@jdennis note that updating python-ldap in OSP means bumping from 2.4.15 included in RHEL to 3.1.0 which potentially may break other packages in rhel using python-ldap. There may be any chance of fixing this by applying any downstream patch only for osp14 (from osp15 on, they run on rhel8 which includes a recent version of python-ldap).

re comment #25: Why would we need to update python-ldap and other packages in RHEL7? Can't the newer versions of the required dependencies be built in another repo, let's say CloudSIG. I'm assuming all the repos are enabled during an OSP install and if the spec file specifies a minimum version that is not satisfied in RHEL but is satisfied in another repo yum will fetch it from a non-RHEL repo.

RHEL is very conservative with rebasing packages so surely the necessity of needing a more current package for OSP than RHEL supplies must have come up before as an issue.

Actually, from reading https://www.rdoproject.org/documentation/requirements/ OSP doesn't install RHEL packages, it installs CentOS packages, isn't that correct?

Bottom line, can't dependencies which are newer than what is in CentOS/RHEL be provided by a non-CentOS/RHEL repo?

(In reply to John Dennis from comment #26)
> re comment #25: Why would we need to update python-ldap and other packages
> in RHEL7? Can't the newer versions of the required dependencies be built in
> another repo, let's say CloudSIG. I'm assuming all the repos are enabled
> during an OSP install and if the spec file specifies a minimum version that
> is not satisfied in RHEL but is satisfied in another repo yum will fetch it
> from a non-RHEL repo.
> 

Yes, I misexplained it. We provide updates in RDO repos as you mentioned and those are pulled and installed as they are newer that in RHEL/CentOS base. The problem is that may be other packages in RHEL that use python-ldap and could be affected by this update.

> RHEL is very conservative with rebasing packages so surely the necessity of
> needing a more current package for OSP than RHEL supplies must have come up
> before as an issue.

Yes, we hit it from time to time and we always try to avoid it if possible :). Just when there is no other option we provide an updated version in our repo for a package provided in CentOS/RHEL.

> 
> Actually, from reading
> https://www.rdoproject.org/documentation/requirements/ OSP doesn't install
> RHEL packages, it installs CentOS packages, isn't that correct?
> 

Yes, but for the context of this discussion, CentOS base packages has the same versions and contents as RHEL.

> Bottom line, can't dependencies which are newer than what is in CentOS/RHEL
> be provided by a non-CentOS/RHEL repo?

Yes, but it has a certain risk so we do if there is not a better alternative (in fact we are already in the way to provide it in https://review.rdoproject.org/r/#/q/topic:update-ldap+(status:open+OR+status:merged) but i want to make sure there is not a better option. Once we provide an update in the official repos, reverting is really a problem.

> We provide updates in RDO repos as you mentioned and those are pulled and
> installed as they are newer that in RHEL/CentOS base. The problem is that
> may be other packages in RHEL that use python-ldap and could be affected
> by this update.

It's a risk but I can't see any other solution. Fortunately even though this is
a major version upgrade according to the release notes the API changes for backward
compatibility are minor, here is what I found:

The functions `ldap.open()`, `ldap.init()`, `ldif.CreateLDIF()`
and `ldif.ParseLDIF()`, which were deprecated for over a decade,
are scheduled for removal in python-ldap 3.1.

The module `ldap.async` is renamed to `ldap.asyncsearch`, due to
`async` becoming a keyword in Python 3.7.
The old module name is deprecated, but will be available as long
as Python 3.6 is supported.

> https://review.rdoproject.org/r/#/q/topic:update-ldap+(status:open+OR+status:merged)

Thank you, I think this is the right solution and the best first step in addressing the
out of sync dependencies. At the moment we have a number of dependencies that need
updating with respect to Keystone and we'll need to revisit this, but for now this
is a good first step (I am going to review what was changed in ldappool just to make
sure that doesn't have to update in lockstep with python-ldap.

OSP14 has python-ldappool-2.3.1 which should be OK from a dependency point of view. However we've fixed bugs in ldappool that first appear in 2.4.0, I see OSP15 has 2.4.0, the OSP14 customers would probably appreciate rebasing to 2.4.0.

For the record, existing pysnmp-4.3.2 does not work fine with pyasn1-0.3.7 (required for python-ldap), we need to update it to pysnmp>4.3.9 (will rebase to 4.4.X which is in u-c), see https://review.rdoproject.org/r/#/c/21681/

(In reply to John Dennis from comment #29)
> OSP14 has python-ldappool-2.3.1 which should be OK from a dependency point
> of view. However we've fixed bugs in ldappool that first appear in 2.4.0, I
> see OSP15 has 2.4.0, the OSP14 customers would probably appreciate rebasing
> to 2.4.0.

We follow the versions that are in upper-constraints.txt for each release. For ldappool in rocky it's 2.3.1:

https://github.com/openstack/requirements/blob/stable/rocky/upper-constraints.txt#L225

since stein it was updated to 2.4.0 so we updated it.

Btw, it was updated to 2.4.1 in master and stein, so we should also update.

I'm updating pysnmp in RDO Rocky in https://review.rdoproject.org/r/#/c/21697/

(In reply to John Dennis from comment #28)
> > We provide updates in RDO repos as you mentioned and those are pulled and
> > installed as they are newer that in RHEL/CentOS base. The problem is that
> > may be other packages in RHEL that use python-ldap and could be affected
> > by this update.
> 
> It's a risk but I can't see any other solution. Fortunately even though this
> is
> a major version upgrade according to the release notes the API changes for
> backward
> compatibility are minor, here is what I found:
> 
> The functions `ldap.open()`, `ldap.init()`, `ldif.CreateLDIF()`
> and `ldif.ParseLDIF()`, which were deprecated for over a decade,
> are scheduled for removal in python-ldap 3.1.
> 
> The module `ldap.async` is renamed to `ldap.asyncsearch`, due to
> `async` becoming a keyword in Python 3.7.
> The old module name is deprecated, but will be available as long
> as Python 3.6 is supported.
> 

yes, tbh, i'm more concerned with pyasn1 update that with python-ldap itself.

> > https://review.rdoproject.org/r/#/q/topic:update-ldap+(status:open+OR+status:merged)
> 
> Thank you, I think this is the right solution and the best first step in
> addressing the
> out of sync dependencies. At the moment we have a number of dependencies
> that need
> updating with respect to Keystone and we'll need to revisit this, but for
> now this
> is a good first step (I am going to review what was changed in ldappool just
> to make
> sure that doesn't have to update in lockstep with python-ldap.

hey David, 

We just applied the workaround described by the customer, removing the bytes_mode argument responsible for the check failure. You can find that change on openstack-keystone-14.1.1-0.20190514013954.00242bd.el7ost

Regards,

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:4340