Bug 1737931

Summary: mercurial depends on Python 2
Product: [Fedora] Fedora Reporter: Lumír Balhar <lbalhar>
Component: mercurialAssignee: Petr Stodulka <pstodulk>
Status: ASSIGNED --- QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 31CC: katzj, mads, mhroncok, ndbecker2, opohorel, pcahyna, pstodulk, pviktori, sebastian.kisela
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1738965, 1738967, 1698500, 1731974, 1738953, 1738954, 1739050    
Attachments:
Description Flags
first attempt for hg with Py3
none
POC: create subpackages for py2 & py3 none

Description Lumír Balhar 2019-08-06 11:41:44 UTC
Python 2.7 will reach end-of-life in January 2020, over 9 years after it was released. This falls within the Fedora 31 lifetime.
Packages that depend on Python 2 are being switched to Python 3 or removed from Fedora: https://fedoraproject.org/wiki/Changes/F31_Mass_Python_2_Package_Removal#Information_on_Remaining_Packages
Python 2 will be retired in Fedora 32: https://fedoraproject.org/wiki/Changes/RetirePython2

To help planning, we'd like to know the plans for mercurial's future. Specifically:


- What is the reason for the Python2 dependency? (Is it software written in Python, or does it just provide Python bindings, or use Python in the build system or test runner?) 

- What are the upstream/community plans/timelines regarding Python 3?

- What is the guidance for porting to Python 3? (Assuming that there is someone who generally knows how to port to Python 3, but doesn't know anything about the particular package, what are the next steps to take?)


This bug is filed semi-automatically, and might not have all the context specific to mercurial.
If you need anything from us, or something is unclear, please mention it here.

Thank you.

Comment 1 Petr Stodulka 2019-08-06 18:51:53 UTC
I already did some investigation. Currently, mercurial is not prepared for Python3. To put here more information, mercurial 5.1 without extensions is possible to use with python3 when building is modified appropriately. Many extensions (not sure how many) are broken - in case that users are using any additional extensions that are not provided in fedora, its utmost sure that it will be broken.

Additionally, mercurial is using their `source_to_code` function they created for transition to Python3, but unfortunately, that breaks some stuff - like hgk extension, which would work otherwise. I can fix at least partially hgk extension (with patch that I would rather not present). I will attach my curernt patch for spec and hgk (no comment) here. I am not sure whether we should already apply it or wait a little - even when we are talking about rawhide.

For now, I am sharing my copr repo with testing rpms:
  $ dnf copr enable pstodulk/mercurial
- provided packages for F30+

Any feedback and help with testing is welcomed (things that are broken and presented by failed tests it is not needed to report [0])

[0] https://copr-be.cloud.fedoraproject.org/results/pstodulk/mercurial/fedora-rawhide-x86_64/00995430-mercurial/builder-live.log.gz
[1] https://copr.fedorainfracloud.org/coprs/pstodulk/mercurial/

Comment 2 Petr Stodulka 2019-08-06 18:55:13 UTC
More info about active port of Mercurial to Python3 can be seen here:
  https://www.mercurial-scm.org/wiki/Python3

Comment 3 Petr Stodulka 2019-08-06 19:10:58 UTC
Created attachment 1601192 [details]
first attempt for hg with Py3

Comment 4 Ben Cotton 2019-08-13 17:10:39 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to 31.

Comment 5 Ben Cotton 2019-08-13 17:11:26 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to '31'.

Comment 6 Lumír Balhar 2019-08-14 09:06:38 UTC
Just sharing a general plan we have:

The current plan is to remove packages with dependency on Python 2 from Fedora 32 in the middle of November 2019. If you want to keep your package in Fedora after that date and you cannot port it to Python 3 yet, you need to request a FESCo exception for the package and all its Python 2 dependencies (even transitive) [1]. If you don't want to maintain it anymore, please orphan the package.

If you're considering filing the exception request, let us know. We can help (for example, we can help find all the dependencies).

[1] https://fedoraproject.org/wiki/Changes/RetirePython2#FESCo_exceptions

Comment 7 Mads Kiilerich 2019-08-14 10:24:55 UTC
Note that a python2 exception not really is an option as the only solution. As many other packages (like python2-urwid) has been removed already, other packages depending on Mercurial *must* have a Python3 Mercurial ASAP so they have the possibility of moving forward.

An instantaneous Mercurial package switch from python2 to python3 will also cause a lot of breakage for all dependencies and is not a good option. Side by side seems like the only good option to me. Perhaps ask for exception to keeep the python2 package around in Fedora 31.

Comment 8 Petr Viktorin 2019-08-14 11:20:07 UTC
Yes, a Python 2 exception is the only solution. Please file for one. Here is an example: https://pagure.io/fesco/issue/2208
but please mention plans for urwid as well, if needed.


We didn't plan to remove python2-urwid while other packages depend on it. What happened is that the maintainer of python-urwid orphaned the package. When it got retired die to lack of maintenance, someone stepped to maintain it -- but took with the python3-urwid variant only.

Do you want to take python2-urwid? We can help to bring it back (if it gets a FESCo exception with e.g. hgview-curses). It would need a miantainer, though.

Comment 9 Petr Stodulka 2019-08-14 12:01:57 UTC
Honestly, I do not plan to take any python2 exception here. Mercurial will be maybe kept for Python2 on F31, but F32 is already planned to be on Python3 - switch to Python3 mercurial in rawhide is planned for tonight. See the thread [0] about that. If we success to prepare everything for Python3 (including other rpms that depends on mercurial) before beta freeze of F31, maybe we could do the update even for F31 - but I guess this will not happen. 

Currently, mercurial is not the only problem we have, there are other rpms that already started to have broken dependencies because of orphaned python2 libraries.

[0] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/7MMYNWCVDDUENRTDQQYMZCLSDFNF6D4P/

Comment 10 Mads Kiilerich 2019-08-14 12:52:38 UTC
@pstodulk: The problem is that an instantaneous switch to Python3 will introduce even more broken dependencies. Please give us a chance to move to a Python 3 Mercurial without having broken dependencies. Broken dependencies would not only make it important, but also urgent.

@petr: Thanks for the explanation of python2-urwid and the idea of getting it back. But no big deal. I will fix my broken dependency by migrating hgview to Python 3 when I can. That's why we need Mercurial packaged for Python 3. (But again: that does *not* mean that we need Python 2 Mercurial to go away while other packages depend on it.)

Comment 11 Petr Stodulka 2019-08-14 23:00:33 UTC
@mads - we are going already to have broken dependencies with python2 - what is making here different level of urgency? As mercurial is not used by so many people (and how many of them are using rawhide..) what is so big problem to switch to F31 branches that are pretty close to rawhide now and switch back later when porting will be done?... They can even stop upgrades of mercurial related rpms until it will be resolved. Or create separate mercurial COPR repo for all related rpms and people can use that.

Anyway, I spent tonight with mercurial to look at way to provide Py2 & Py3 mercurial builds. Honestly, I would like to kickoff Py2-mercurial from the train..

Comment 12 Petr Stodulka 2019-08-27 23:12:39 UTC
Created attachment 1608765 [details]
POC: create subpackages for py2 & py3

The most probably it's broken. I haven't had time to test it yet but maybe there is already something done about that. Extensions will be almost sure broken for such builds but that's all what I could do for now.

One of the main problems here is the executable script. To be able to have script for py2 and py3, I created /usr/bin/hg3 script for mercurial-py3 packages. Not sure whether it is good idea, but it is the only one I had in my mind how to resolve it under one component.

Comment 13 Miro Hrončok 2019-10-07 17:22:25 UTC
What's the current plan? Is it requesting an exception, switching to python3 only, or retiring mercurial?

Comment 14 Petr Stodulka 2019-10-09 11:09:15 UTC
Current plan is to have both, mercurial for Python2 & Python3. But the correct solution is not prepared yet. To have just Python3 version is easy otherwise. Just bunch of other rpms will be broken immediately in such case. So we are trying to have both.

Comment 15 Miro Hrončok 2019-10-09 11:20:22 UTC
In order to keep the Python 2 version a FESCo exception is needed. We can help you draft the request. In order to do that, we need some info:

- Currently, mercurial BuildRequires python2-docutils. Is that dependency mandatory? Can it build docs with python3-docutils instead?

- From the following packages, are you planning to keep some on Python 2 as well? Should they be included in the exception ?

  git-cinnabar
  git-remote-hg
  hg-git
  hgview
  tortoisehg
  trac-mercurial-plugin

Note that they bring a handful of other Python 2 packages.

Comment 16 Mads Kiilerich 2019-10-10 01:16:33 UTC
As we still don't have Mercurial on Python 3 available, all dependent packages are *forced* to stay on Python 2 even if we have plans for moving to Python 3.

I ported hgview to python 3, and it is in latest upstream release.

TortoiseHg is also getting there. The development branch with some patches works with Python 3. It would be valuable to have it packaged next to the stable python2 package and get feedback. But if necessary, it could be the only one.

But yes, because of the timing, I guess everything that depends on Mercurial packaging will need an exception too, and should ideally be included in the exception. But with python2 support falling a part, I doubt it will be feasible for other packages to use it anyway.

Comment 17 Petr Viktorin 2019-10-10 09:13:20 UTC
Keeping mercurial itself and git-cinnabar on Python 2 would be easy; they
don't pull in a lot of other packages:

- mercurial
  - python-docutils (probably only for docs, which can be done on py3 or pregenerated)
- git-cinnabar
  - mercurial
  - python-nose (test-only)
    - python-coverage

trac-mercurial-plugin depends just on
  - mercurial
  - Trac (which is probably also getting an exception)
  - python2-setuptools (which shouldn't be hard to get an exception for; I'll omit it below)



hg-git would be more challenging to coordinate, since it depends on dulwich,
which drags in a lot of network/SSL-related packages.
We'd need to coordinate with the maintainers, and possibly find people to
(co-)maintain the Python 2 bits:

- hg-git
  - mercurial
  - python-dulwich
    - python-certifi
    - python-docutils (doc-only)
    - python-nose (test only)
    - python-urllib3
      - python-idna
      - python-pysocks
        - python-psutil (has an exception already)
        - python2-pytest (test-only)
      - python-six
      - python-backports-ssl_match_hostname (orphaned)
      - python-ipadress (orphaned)
- git-remote-hg
  - mercurial
  - hg-git (above)



And then there are GUI tools, which depend on Qt4, Qt5 *and* GTK:

- hgview
  - mercurial
  - python-docutils (for run time)
  - python-inotify
  - python-pygments
    - python-nose (test-only)
  - PyQt4
    - dbus-python
      - pygobject3 (see below)
      - python-docutils (doc-only?)
    - sip
  - qscintilla
    - PyQt4 (see above)
    - python-qt5
      - dbus-python, sip (see above)
- tortoisehg:
  - mercurial
  - pygobject3
    - pycairo (has an exception already)
  - python-iniparse
    - python-six
  - python-pygments
  - qscintilla, python-qt5 (see above)
  - python-enum34

Comment 18 Miro Hrončok 2019-10-11 16:08:50 UTC
I went ahead and requested an exception request (only) for mercurial: https://pagure.io/fesco/issue/2243

Comment 19 Petr Stodulka 2019-10-14 15:50:04 UTC
Hi guys, sorry for delay. I've got sick again. Yes, we will need exception for otherpakcages as well. I believe I will be able to look at it tomorrow. At least, rebase of mercurial should be done finally for F31. The new version of mercurial requires some small changes in spec as well.

Thanks Miro for opening request for mercurial.

Comment 20 Petr Stodulka 2019-10-20 01:13:07 UTC
I rebased mercurial to version 5.1.2 in rawhide. As well, you can find opened PR to add the mercurial-python3 subpackage (among other things). Hopefully things will be moving faster now.

I looked at the python2-docutils dependency but I it doesn't look that it can be easily dropped. We will need it as well now, unless someone modify build&installation scripts.

Comment 21 Petr Stodulka 2019-10-20 01:24:45 UTC
Here is the PR: https://src.fedoraproject.org/rpms/mercurial/pull-request/5

Comment 22 Petr Stodulka 2019-10-21 11:52:58 UTC
FYI, about git-remote-hg, I dropped the build dependency on hg-git. Checking the code, there is nothing related to hg-git and was there by mistake.

Comment 23 Petr Stodulka 2019-10-21 11:53:40 UTC
- or at least, nowadays it was invalid.

Comment 24 Mads Kiilerich 2019-11-17 18:19:20 UTC
Thanks for the Mercurial py3 subpackage in rawhide. https://koji.fedoraproject.org/koji/taskinfo?taskID=39064842 is now waiting for it.

But also, this py3 packaging of Mercurial 5.1 has a big problem:

Mercurial 5.1 use a custom importer with byte code conversion for Python 3. The .pyc files are thus in a special format. When brp-python-bytecompile overwrite them with standard byte codes, Mercurial will have to re-import them (and try to write out new .pyc files). That can give several seconds of startup time, as seen when running for example a simple `hg debuginstall` (as a regular user or with SELinux enabled). mercurial-py3 is thus not really useful ... oir a significant unnecessary regression.

That custom importer is however gone in Mercurial 5.2, released earlier this month. Python 3 support is also out of preview (except on Windows), and HGPYTHON3 is no longer needed. I have updated the upstream Fedora packaging to Python 3 on Fedora 31.

I thus suggest upgrading rawhide to Mercurial 5.2 ASAP and clean up the packaging.

Considering the time pressure and bitrot of Python2 in Fedora 31, I also suggest backporting the new Python 3 sub package there. Everything considered, I think the least risky option is to upgrade Mercurial to 5.2 there too, so users have a (long term) less risky option of using Python 3.

Comment 25 Miro Hrončok 2019-11-17 21:37:01 UTC
As a short them workaround, brp-python-bytecompile can be turned of for mercurial.

Comment 26 Petr Stodulka 2019-11-19 15:12:09 UTC
Thanks Mads for feedback. I plan to upgrade mercurial to 5.2 in the following week. I wanted to do it the week ago but...

Anyway, I am not sure that rebase in stable fedora is good idea, because I believe it will break additional packages. I would really continue the work just in rawhide. Regarding the time, if anyone else is able to work on that, feel free.

Comment 27 Petr Stodulka 2019-11-26 00:18:31 UTC
I created new PR with Mercurial 5.2: https://src.fedoraproject.org/rpms/mercurial/pull-request/6#

I kept there still the HGPYTHON3 envar which I use to detect that /usr/bin/hg3 should be created (I know it would be used just the PYTHON envar, but this seems to me more obvious solution for now). As well, I found that the previous build was not created (5.1.2-2; sorry guys. that was pebkac combined with time pressure and I have installed local build). So the *-py3 subpackage has not been delivered in the rawhide. It will be finally with the v5.2. 

If you have time, please check the PR. In the worst case, I will merge it this afternoon (Tue: CET) to deliver builds finally.

I will send email during the day to the other maintainers of components depedent on mercurial to see how does it look like with Py3 porting.