Bug 1244234 - Use python3 in scripts in 389-ds-base
Summary: Use python3 in scripts in 389-ds-base
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: 389-ds-base
Version: 27
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: mreynolds
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Keywords: Reopened
Depends On:
Blocks: PYTHON3 PYTHON3-PYTHON2
TreeView+ depends on / blocked
 
Reported: 2015-07-17 14:12 UTC by Miro Hrončok
Modified: 2018-07-01 22:36 UTC (History)
11 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2018-07-01 22:36:43 UTC


Attachments (Terms of Use)
Patch for dist-git (13.78 KB, patch)
2015-07-26 23:14 UTC, Miro Hrončok
no flags Details | Diff
Make binaries pointing to python3 (1.51 KB, patch)
2017-05-19 12:09 UTC, Charalampos Stratakis
no flags Details | Diff
Make binaries point to python3 (1.86 KB, patch)
2017-05-19 12:23 UTC, Charalampos Stratakis
no flags Details | Diff
Make binaries pointing to python3 (1.56 KB, application/mbox)
2017-05-25 09:36 UTC, Charalampos Stratakis
no flags Details

Description Miro Hrončok 2015-07-17 14:12:53 UTC
389-ds-base ships with 3 python scripts:

/usr/bin/ds-logpipe.py
/usr/lib64/dirsrv/python/failedbinds.py
/usr/lib64/dirsrv/python/logregex.py

There is an upstream commit that makes those file Python 3 compatible [1].

As for a system wide change for Fedora 23 [2], we need 389-ds-base to use Python 3 instead of Python 2.

Could you please adjust the package, take the patches or a new release if it exists, to make sure all those scripts use Python 3?

If you need help with that, let me know.

[1] https://fedorahosted.org/389/changeset/674eac235a72565075129f125e3aefdecb033a05/
[2] https://fedoraproject.org/wiki/Changes/Python_3_as_Default

Comment 1 Noriko Hosoi 2015-07-20 16:05:56 UTC
Hi Mark,

Since you've already done the research, let me assign this bug to you.

Thanks!
--noriko

Comment 2 Miro Hrončok 2015-07-23 08:43:00 UTC
Mark, the Alpha Freeze is on Tuesday. We need to have this done before then. Is that doable on your side? If not, I could do it.

Comment 3 mreynolds 2015-07-23 11:24:11 UTC
(In reply to Miro Hrončok from comment #2)
> Mark, the Alpha Freeze is on Tuesday. We need to have this done before then.
> Is that doable on your side? If not, I could do it.

We have already pushed the core DS python updates to upstream.  Sorry this bug was not linked to the upstream ticket at that time of the work, so it was not updated.

Comment 4 Miro Hrončok 2015-07-23 11:58:20 UTC
(In reply to mreynolds from comment #3)
> We have already pushed the core DS python updates to upstream.

I know about that, the commit is linked in the bug description.

You haven't actually answered the main question:

(In reply to Miro Hrončok from comment #2)
> Mark, the Alpha Freeze is on Tuesday. We need to have this done before then.
> Is that doable on your side?

Thanks

Comment 5 mreynolds 2015-07-23 12:11:34 UTC
(In reply to Miro Hrončok from comment #4)
> (In reply to mreynolds from comment #3)
> > We have already pushed the core DS python updates to upstream.
> 
> I know about that, the commit is linked in the bug description.
> 
> You haven't actually answered the main question:

I don't understand the question then.  The code is pushed to master branch, not sure what else you need from me.

Mark

> 
> (In reply to Miro Hrončok from comment #2)
> > Mark, the Alpha Freeze is on Tuesday. We need to have this done before then.
> > Is that doable on your side?
> 
> Thanks

Comment 6 Miro Hrončok 2015-07-26 22:27:58 UTC
(In reply to mreynolds from comment #5)
> (In reply to Miro Hrončok from comment #4)
> > (In reply to mreynolds from comment #3)
> > > We have already pushed the core DS python updates to upstream.
> > 
> > I know about that, the commit is linked in the bug description.
> > 
> > You haven't actually answered the main question:
> 
> I don't understand the question then.  The code is pushed to master branch,
> not sure what else you need from me.

The original bug description says: "Could you please adjust the package, take the patches or a new release if it exists, to make sure all those scripts use Python 3?"

We need this to be present in Fedora 23 before Tuesday.

You say it is pushed to (upstream) master branch, which is great, but does not fix the thing we need. Latest Fedora build is from 2015-06-24 and therefor cannot contain the commit from three weeks ago.

I'll attach a patch for dist git in a few minutes.

Comment 7 Miro Hrončok 2015-07-26 23:14:13 UTC
Created attachment 1056405 [details]
Patch for dist-git

Koji scratch build: http://koji.fedoraproject.org/koji/taskinfo?taskID=10489427

Comment 8 Miro Hrončok 2015-07-27 23:39:24 UTC
Since there has been no reaction and the alpha freeze is today, I've built it.

There has been some error with empty line in sources, I believe Noriko fixed faster than me (Thank you).

During the process, I've realized f23 is one version ahead of rawhide/master, which should probably not happen.

Comment 9 Noriko Hosoi 2015-07-28 00:41:28 UTC
Please do not override the necessary changes...

diff --git a/389-ds-base.spec b/389-ds-base.spec
index 68fe3cf..47f38a3 100644
--- a/389-ds-base.spec
+++ b/389-ds-base.spec
@@ -74,8 +74,11 @@ BuildRequires:    tcp_wrappers
 BuildRequires:    pam-devel
 BuildRequires:    systemd-units
 
+# for python3 macros
+BuildRequires:    python3-devel
+
 # this is needed for using semanage from our setup scripts
-Requires:         policycoreutils-python-utils
+Requires:         /usr/sbin/semanage
 
 # the following are needed for some of our scripts
 %if %{use_openldap}
===============================================================
Also, please do not apply the patch (e.g., 389-ds-base-py3.patch) against the 389-ds-base source code.

I have to undo this change you made against 389-ds-base dist-git.
commit 44d1dcbaecd86ad8683f6be8c9eb4c6497dc3288
Author: Miro Hrončok <miro@hroncok.cz>
Date:   Mon Jul 27 01:07:59 2015 +0200

    Resolves: Bug 1244234 - Use python3 in scripts in 389-ds-base
    
    Conflicts:
        389-ds-base.spec

We strictly manage our source code in the upstream git.

Comment 10 Tomas Orsava 2016-06-16 13:06:15 UTC
The 389-ds-base RPM requires both Python 2 and Python 3.

Except in very special circumstances, there is no need for one package
to drag in both Python stacks. Usually, this is a packaging error: for
example, a stray "/usr/bin/python" shebang in a Python 3 package can
introduce a Python 2 dependency.

Please split your package, or remove the stray dependencies.
There is a section on shebangs in the Python RPM Porting Guide [0]
which covers this issue.

It's ok to do this in Rawhide only, however, it would be greatly
appreciated if you could push it to Fedora 24 as well.


If anything is unclear, or if you need any kind of assistance, you can
ask on IRC (#fedora-python on Freenode), or reply here. We'll be happy
to help investigating or fixing this issue!


[0] http://python-rpm-porting.readthedocs.io/en/latest/application-modules.html#are-shebangs-dragging-you-down-to-python-2

Comment 11 Fedora Update System 2016-11-03 20:45:28 UTC
389-ds-base-1.3.5.15-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-8660c7656f

Comment 12 Fedora Update System 2016-11-03 21:17:47 UTC
389-ds-base-1.3.5.15-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-8f9d466bcc

Comment 13 Fedora Update System 2016-11-05 03:34:54 UTC
389-ds-base-1.3.5.15-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-8660c7656f

Comment 14 Fedora Update System 2016-11-05 18:59:53 UTC
389-ds-base-1.3.5.15-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-8f9d466bcc

Comment 15 Fedora Update System 2016-11-06 00:26:33 UTC
389-ds-base-1.3.5.15-1.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

Comment 16 Fedora Update System 2016-11-19 21:11:02 UTC
389-ds-base-1.3.5.15-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 17 Iryna Shcherbina 2017-03-31 10:08:32 UTC
The issue is still valid.

389-ds-base RPM still depends on both Python 2 and Python 3:

$ dnf --enablerepo=rawhide repoquery --requires 389-ds-base | grep python
/usr/bin/python2
/usr/bin/python3
...

Please remove the stray dependency. Let us know if you need any assistance or help.

Comment 18 wibrown@redhat.com 2017-04-02 23:21:30 UTC
I've seen this issue before. It's comes from the fact that one of our python scripts has a python2 in the #! line, and rpm is pulling it in automatically.

This may be difficult to correct, as we must support RHEL in addition to fedora, and python 3 is not in a state where we can depend on it in RHEL.

This leads to the issue that it's very hard to actually develop python tools that can seamlessly change between python 2 and python 3, because of the need to alter the #! line. Without using some kind of autotools substitution, sed, or other hacky solution, this is not easy to achieve.

So we are in a position where we can not without a "hack" support our needs of RHEL and Fedora.

If you could provide some advice or material on how to make this change "better", that would be excellent, but if not, we may need to continue to rely on python 2 during the 1.3.x lifecycle.

Comment 19 Charalampos Stratakis 2017-05-19 12:08:27 UTC
(In reply to wibrown@redhat.com from comment #18)
> I've seen this issue before. It's comes from the fact that one of our python
> scripts has a python2 in the #! line, and rpm is pulling it in automatically.
> 
> This may be difficult to correct, as we must support RHEL in addition to
> fedora, and python 3 is not in a state where we can depend on it in RHEL.
> 
> This leads to the issue that it's very hard to actually develop python tools
> that can seamlessly change between python 2 and python 3, because of the
> need to alter the #! line. Without using some kind of autotools
> substitution, sed, or other hacky solution, this is not easy to achieve.
> 
> So we are in a position where we can not without a "hack" support our needs
> of RHEL and Fedora.
> 
> If you could provide some advice or material on how to make this change
> "better", that would be excellent, but if not, we may need to continue to
> rely on python 2 during the 1.3.x lifecycle.

Hello. Ideally that would be only a downstream modification using sed in the SPEC file to change the shebang to point to python3.

Unfortunately there is not an elegant solution for an upstream project to support both python2 and python3 with a single shebang line for both Fedora and RHEL distros. Some packagers in order to make this happen, provide different subpackages for each python interpreter, with differently named binaries, even though the binaries provide the same functionality. It's something that must be solved at the downstream level. Hopefully at some point in the future, python will mean python3 everywhere, but it's not the case at the moment. 

Nevertheless I did some testing and the dependency on usr/bin/python2 can be removed by changing this configure line [0] to usr/bin/python3

[0] https://pagure.io/389-ds-base/blob/master/f/configure.ac#_423

Will provide a patch.

Comment 20 Charalampos Stratakis 2017-05-19 12:09 UTC
Created attachment 1280377 [details]
Make binaries pointing to python3

Comment 21 Charalampos Stratakis 2017-05-19 12:23 UTC
Created attachment 1280378 [details]
Make binaries point to python3

Amended the previous patch to change the libsemanage-python dependency to libsemanage-python3 as it was dragging python2 as well.

Comment 22 Charalampos Stratakis 2017-05-19 13:03:43 UTC
Output of 'rpm -qp --requires 389-ds-base-1.3.6.4-2.fc27.x86_64.rpm | grep python'

/usr/bin/python3
libsemanage-python3
policycoreutils-python-utils

Comment 23 Charalampos Stratakis 2017-05-25 09:36 UTC
Created attachment 1282192 [details]
Make binaries pointing to python3

Actually scratch the previous patch. Just adding the --with-pythonexec option at the configure will make this work. Providing another patch.

Comment 24 mreynolds 2017-05-25 20:31:49 UTC
I applied your patch and rebuilt rawhide and f26.  Do you need me to do this to other branches?

Comment 25 Charalampos Stratakis 2017-05-25 21:16:24 UTC
(In reply to mreynolds from comment #24)
> I applied your patch and rebuilt rawhide and f26.  Do you need me to do this
> to other branches?

Thanks a lot! Rawhide and F26 are fine, no other branches required.

Comment 26 wibrown@redhat.com 2017-05-25 23:47:10 UTC
This is likely going to break our RHEL7 builds :( 

I have been attempting to integrate python3 for a long time, and it's so far proven impossible it maintain both python2 and python3 that works in RHEL7 and Fedora.

So far I have chosen to support RHEL7 because that is a more valuable target. I think that this may cause us some trouble to try and make our scripts blindly work with python3 only, because they will no longer work on RHEL7.

Comment 27 wibrown@redhat.com 2017-05-25 23:50:07 UTC
Also, just changing the pythonexec flag doesn't change the #! in the scripts, so this may still pull in the python2 files.

Comment 28 Charalampos Stratakis 2017-05-26 08:13:06 UTC
(In reply to wibrown@redhat.com from comment #26)
> This is likely going to break our RHEL7 builds :( 
> 
> I have been attempting to integrate python3 for a long time, and it's so far
> proven impossible it maintain both python2 and python3 that works in RHEL7
> and Fedora.
> 
> So far I have chosen to support RHEL7 because that is a more valuable
> target. I think that this may cause us some trouble to try and make our
> scripts blindly work with python3 only, because they will no longer work on
> RHEL7.

What do you mean with that? This is clearly a downstream change only for Fedora. Maintaining the same SPEC for RHEL and Fedora is just impossible as the distros have diverted way too much.

Comment 29 Charalampos Stratakis 2017-05-26 08:14:40 UTC
(In reply to wibrown@redhat.com from comment #27)
> Also, just changing the pythonexec flag doesn't change the #! in the
> scripts, so this may still pull in the python2 files.

It is true that it doesn't, however due to the scripts not being executable, rpmbuild will not add python2 as a dependency due to these scripts.

I've posted above how you can test the dependencies of the resulting rpm's.

Comment 30 mreynolds 2017-05-26 11:20:14 UTC
(In reply to wibrown@redhat.com from comment #26)
> This is likely going to break our RHEL7 builds :( 
> 
 
This is upstream spec file change only - it has nothing to do with RHEL.

Comment 31 Viktor Ashirov 2017-05-29 08:42:51 UTC
(In reply to wibrown@redhat.com from comment #26)
> This is likely going to break our RHEL7 builds :( 
> 
> I have been attempting to integrate python3 for a long time, and it's so far
> proven impossible it maintain both python2 and python3 that works in RHEL7
> and Fedora.
> 
> So far I have chosen to support RHEL7 because that is a more valuable
> target. I think that this may cause us some trouble to try and make our
> scripts blindly work with python3 only, because they will no longer work on
> RHEL7.

So, actually, we have 3 spec files:
RHEL: http://pkgs.devel.redhat.com/cgit/rpms/389-ds-base
Fedora: http://pkgs.fedoraproject.org/cgit/rpms/389-ds-base.git
Upstream: https://pagure.io/389-ds-base/blob/master/f/rpm/389-ds-base.spec.in

Since these are separate entities, we can maintain python2 for RHEL, python3 for Fedora, and whatever works for upstream. 

I'd like to generate working rpms from upstream bits for both RHEL and Fedora though.

Comment 32 Fedora Update System 2017-05-29 16:56:14 UTC
389-ds-base-1.3.6.6-2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-1a351f960e

Comment 33 Fedora Update System 2017-05-29 23:06:03 UTC
389-ds-base-1.3.6.6-2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-1a351f960e

Comment 35 Fedora Update System 2017-06-09 19:14:56 UTC
389-ds-base-1.3.6.6-2.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 36 Iryna Shcherbina 2017-06-29 14:14:42 UTC
Reopening the issue, as 389-ds-base-1.3.7.1-1.fc27 started to depend on both Python 2 and Python 3 again after the latest update.

The problem is using `#!/usr/bin/python` shebang in `ldap/admin/src/scripts/ds-replcheck` file. This should be fixed if you change shebangs preprocessing [0] to also match this file, e.g:

> sed -r -i '1s|^#!\s*/usr/bin.*python.*|#!%{__python3}|' ldap/admin/src/scripts/{*.py,ds-replcheck}


Please remove the stray dependency. Let us know if you need any assistance or help.

[0] https://src.fedoraproject.org/cgit/rpms/389-ds-base.git/tree/389-ds-base.spec#n239

Comment 37 mreynolds 2017-06-29 15:17:15 UTC
New build done:  389-ds-base-1.3.7.1-2

Comment 38 Jan Kurik 2018-05-31 09:09:29 UTC
This bug is currently reported against a Fedora version which is already unsuported.
I am changing the version to '27', the latest supported release.

Please check whether this bug is still an issue on the '27' release.
If you find this bug not being applicable on this release, please close it.

Comment 39 Iryna Shcherbina 2018-06-11 10:11:54 UTC
Reopening as with update to 389-ds-base-0:1.4.0.10-1, this can be reproduced again:

$ dnf --repo=rawhide repoquery --requires 389-ds-base | grep python
/usr/bin/python2
/usr/bin/python3
...

There are a couple more scripts that need to be checked:
$ grep -r  python2
usr/bin/readnsstate:#!/usr/bin/python2
usr/libexec/dirsrv/ds_selinux_port_query:#!/usr/bin/python2
usr/libexec/dirsrv/ds_selinux_enabled:#!/usr/bin/python2

Comment 40 mreynolds 2018-06-14 15:58:49 UTC
Should now be fixed in 389-ds-base-1.4.0.10-2.fc28

Comment 41 Fedora Update System 2018-06-22 15:34:05 UTC
389-ds-base-1.4.0.11-2.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-a00a3e9b36

Comment 42 Fedora Update System 2018-06-23 21:21:21 UTC
389-ds-base-1.4.0.11-2.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-a00a3e9b36

Comment 43 Fedora Update System 2018-07-01 22:36:43 UTC
389-ds-base-1.4.0.11-2.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.