Bug 1789613 - python-latexcodec fails to build with Python 3.9
Summary: python-latexcodec fails to build with Python 3.9
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: python-latexcodec
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Jerry James
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: PYTHON39
TreeView+ depends on / blocked
 
Reported: 2020-01-09 22:02 UTC by Charalampos Stratakis
Modified: 2020-01-16 09:19 UTC (History)
4 users (show)

Fixed In Version: 2.0.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-01-15 22:50:52 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Charalampos Stratakis 2020-01-09 22:02:27 UTC
python-latexcodec fails to build with Python 3.9.0a2.

There are many test failures, all similar to:

ERROR: test_double_quotes_unicode (test_latex_codec.TestDecoder)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/latexcodec-1.0.7/test/test_latex_codec.py", line 222, in test_double_quotes_unicode
    self.decode(u"“áâ€", u"``á''".encode("utf8"), "utf8")
  File "/builddir/build/BUILD/latexcodec-1.0.7/test/test_latex_codec.py", line 57, in decode
    decoded, n = codecs.getdecoder(encoding)(text_latex)
  File "/usr/lib64/python3.9/codecs.py", line 970, in getdecoder
    return lookup(encoding).decode
LookupError: unknown encoding: latex+utf8

For the build logs, see:
https://copr-be.cloud.fedoraproject.org/results/@python/python3.9/fedora-rawhide-x86_64/01142170-python-latexcodec/

For all our attempts to build python-latexcodec with Python 3.9, see:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/package/python-latexcodec/

Testing and mass rebuild of packages is happening in copr. You can follow these instructions to test locally in mock if your package builds with Python 3.9:
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/

Let us know here if you have any questions.

Python 3.9 will be included in Fedora 33, but the initial bootstrapping has already started.
A build failure this early in the bootstrap sequence blocks us very much.

Comment 1 Jerry James 2020-01-11 22:16:20 UTC
I cannot do local mock builds to diagnose the problem.  Every attempt ends like this:

Copr repo for python3.9 owned by @python        0.0  B/s |   0  B     02:00    
Errors during downloading metadata for repository 'python39':
  - Curl error (28): Timeout was reached for https://copr-be.cloud.fedoraproject.org/results/@python/python3.9/fedora-rawhide-x86_64/repodata/repomd.xml [Connection timed out after 30000 milliseconds]
  - Curl error (28): Timeout was reached for https://copr-be.cloud.fedoraproject.org/results/@python/python3.9/fedora-rawhide-x86_64/repodata/repomd.xml [Connection timed out after 30001 milliseconds]
Error: Failed to download metadata for repo 'python39': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
ERROR: Exception(python-latexcodec-1.0.7-5.fc32.src.rpm) Config(fedora-rawhide-python39) 2 minutes 5 seconds

I can visit the COPR web site and log in.  I don't know why it is refusing to let me download metadata for the python39 repository.

The issue with python-latexcodec is not obvious to me from the traceback you've shown above, nor from inspecting the code.

I do not know how to proceed.

I would also like to strong suggest that you omit this sentence from future bug reports: "A build failure this early in the bootstrap sequence blocks us very much."  Why?  Because it is a blatant attempt at exerting psychological pressure on package maintainers in order to get them to fix problems caused by a backwards incompatible change, the timing of the introduction of which was not of their choosing.  It seems unprofessional to me.  I think you should just file the bug and exert psychological pressure later if maintainers don't respond in a timely fashion.  I am, in short, rather irritated that you are pressuring *me*, when I am generally very responsive.  Knock it off.

Comment 2 Jerry James 2020-01-11 23:22:52 UTC
I got COPR to talk to me finally and now I can reproduce the problem.

When latexcodec starts up, it registers a function (find_latex) with the codecs module.  With python 3.8 and earlier, code like this:

b"\xfe".decode("latex+latin1")

would result in find_latex being invoked with the argument "latex+latin1".  With python 3.9, it is invoked with "latex_latin1".  Why is the + converted to _?  Is this a documented change in the codecs module?  I don't see anything about it on the python 3.9 changes page.

Comment 3 Charalampos Stratakis 2020-01-13 16:32:41 UTC
(In reply to Jerry James from comment #1)
> 
> I would also like to strong suggest that you omit this sentence from future
> bug reports: "A build failure this early in the bootstrap sequence blocks us
> very much."  Why?  Because it is a blatant attempt at exerting psychological
> pressure on package maintainers in order to get them to fix problems caused
> by a backwards incompatible change, the timing of the introduction of which
> was not of their choosing.  It seems unprofessional to me.  I think you
> should just file the bug and exert psychological pressure later if
> maintainers don't respond in a timely fashion.  I am, in short, rather
> irritated that you are pressuring *me*, when I am generally very responsive.
> Knock it off.

Apologies if the comment seemed like that, the goal is not to exert psychological pressure, we are indeed blocked by the failures. latexcodec is part of the initial bootstrap sequence as it can be seen here: https://github.com/hroncok/rpm-list-builder/blob/python39/python39.yaml

After that sequence has been completed we proceed with the rest of the packages depending on Python which are a bit more than 2000. And with so many packages to deal with, we try to make the reports and the procedure as automated as possible, it's already a big task and we can't be going through each package and packager manually.

Python upstream has already modified their release schedule to better align with Fedora releases, hence the reason we are working as soon as the alpha is out to check out for failures and work with packagers and upstreams for the transition.

I'll try to rework the message to something more appropriate.

In the meantime I am cc'ing on of the core developers in the bug, maybe he could provide some more info on the issue.

Comment 4 Miro Hrončok 2020-01-14 00:39:54 UTC
(In reply to Jerry James from comment #1)
> I would also like to strong suggest that you omit this sentence from future
> bug reports: "A build failure this early in the bootstrap sequence blocks us
> very much."  Why?  Because it is a blatant attempt at exerting psychological
> pressure on package maintainers in order to get them to fix problems caused
> by a backwards incompatible change, the timing of the introduction of which
> was not of their choosing.  It seems unprofessional to me.  I think you
> should just file the bug and exert psychological pressure later if
> maintainers don't respond in a timely fashion.  I am, in short, rather
> irritated that you are pressuring *me*, when I am generally very responsive.
> Knock it off.

As Charalampos said, sorry about tha. The sentence was written by me. It was not our intention to pressure anybody. Our intention was to make sure the maintainers understand that the priority is high for us, despite the fact that Python was not yet updated to 3.9 and Fedora 33 does not exist yet. Do you have a better suggestion? I appreciate your work in Fedora very ,uch and indeed, you are generally very responsive -- thanks for that.

Comment 5 Miro Hrončok 2020-01-14 12:00:16 UTC
Jerry, I have updated the bugzilla template:

https://github.com/hroncok/mini-mass-rebuild/commit/08e3f57e2a908353596e29172789908bc1b5ea33

In the meantime, I am trying to figure out what happen with the codes names in 3.9, so far no luck.

Comment 6 Victor Stinner 2020-01-14 12:31:12 UTC
> b"\xfe".decode("latex+latin1") would result in find_latex being invoked with the argument "latex+latin1".
> With python 3.9, it is invoked with "latex_latin1".  Why is the + converted to _?

That's a deliberate change of Python 3.9 in codecs lookup:

* https://bugs.python.org/issue37751
* https://github.com/python/cpython/commit/20f59fe1f7748ae899aceee4cb560e5e1f528a1f

Can latexcodec be adapted to handle "latex_latin1" instead of "latex+latin1"?

The change is currently only documented in the Changelog:
https://docs.python.org/dev/whatsnew/changelog.html#id9
"bpo-37751: Fix codecs.lookup() to normalize the encoding name the same way than encodings.normalize_encoding(), except that codecs.lookup() also converts the name to lower case."

FYI I merged the change before Marc-Andre Lemburg, who is the big architect of Unicode and codecs, blessed the change: "Jordon is right. Conversion has to be to underscores, not hyphens."

I can reconsider reverting the change if there is no way to update latexcodec for Python 3.9.

Currently, latexcodec find_latex(encoding) uses encoding.partition(u"+"). It seems easy to fix it. The function should be updated to attempt to "_" if encoding contains "_". Something like:

if u"_" in encoding:
    # Python 3.9 now normalizes "latex+latin1" to "latex_latin1"
    # https://bugs.python.org/issue37751
    encoding, _, inputenc_ = encoding.partition(u"_")
else:
    encoding, _, inputenc_ = encoding.partition(u"+")

Comment 7 Victor Stinner 2020-01-14 12:54:54 UTC
I reported the bug upstream and I proposed a PR to fix the issue upstream:

* https://github.com/mcmtroffaes/latexcodec/pull/76
* https://github.com/mcmtroffaes/latexcodec/issues/75

Comment 9 Jerry James 2020-01-15 19:42:40 UTC
I want to apologize for being grumpy.  I was under a lot of stress and let it come through in that last comment.  I'm sorry.

Thank you, Victor, for the upstream fix.  I have built version 2.0.0 in Rawhide.  Let the python 3.9 builds roll on!

Comment 10 Victor Stinner 2020-01-15 22:01:39 UTC
> I want to apologize for being grumpy.  I was under a lot of stress and let it come through in that last comment.  I'm sorry.

It's ok, don't worry. I'm also a maintainer of open source softwares, and I know how stressful it can be.

> Thank you, Victor, for the upstream fix.  I have built version 2.0.0 in Rawhide.  Let the python 3.9 builds roll on!

Wooow, 2.0 is already released! And it's already in Rawhide! That's great :-)

I was feeling guilty since I merged the codecs change in Python usptream which introduced the incompatible change ;-)

Should the "Fixed In Version" field be set to 2.0.0 and status be updated? (I didn't understand if the new package is already built or not, sorry.)

Comment 11 Jerry James 2020-01-15 22:50:52 UTC
(In reply to Victor Stinner from comment #10)
> Should the "Fixed In Version" field be set to 2.0.0 and status be updated?
> (I didn't understand if the new package is already built or not, sorry.)

Oops, I meant to do that.  I should upgrade the memory modules in my head someday....

Version 2.0.0 has been built in Rawhide.  I don't know its status in the python 3.9 copr.

Comment 12 Charalampos Stratakis 2020-01-16 00:39:26 UTC
(In reply to Jerry James from comment #11)
> (In reply to Victor Stinner from comment #10)
> > Should the "Fixed In Version" field be set to 2.0.0 and status be updated?
> > (I didn't understand if the new package is already built or not, sorry.)
> 
> Oops, I meant to do that.  I should upgrade the memory modules in my head
> someday....
> 
> Version 2.0.0 has been built in Rawhide.  I don't know its status in the
> python 3.9 copr.

I've manually applied the upstream fix in copr to allow us to proceed further down the sequence. Next rebuilds will grab the updated version from rawhide, so we are good. Thank you.

Comment 13 Miro Hrončok 2020-01-16 09:19:59 UTC
(In reply to Jerry James from comment #9)
> I want to apologize for being grumpy.  I was under a lot of stress and let
> it come through in that last comment.  I'm sorry.

Don't worry about it, aren't we all stressed sometimes?


Note You need to log in before you can comment on or make changes to this bug.