Bug 1484497 - UnicodeDecodeError in localeconv() makes test_float fail in Koji
Summary: UnicodeDecodeError in localeconv() makes test_float fail in Koji
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: python3
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Charalampos Stratakis
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-08-23 17:58 UTC by Petr Viktorin (pviktori)
Modified: 2018-01-26 18:10 UTC (History)
9 users (show)

Fixed In Version: python3-3.6.4-6.fc27
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-01-26 18:10:04 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Python 31900 0 None None None 2017-10-30 13:41:35 UTC

Description Petr Viktorin (pviktori) 2017-08-23 17:58:16 UTC
I can reproduce this bug only in Koji, and in Mock if the buildroot is cleaned immediately before. I don't know how to reproduce it on a system/chroot I can SSH into.


Under certain circumstances, when LC_NUMERIC is fr_FR.ISO8859-1 but LC_ALL is C.UTF-8, locale.localeconv() fails with
    UnicodeDecodeError: 'locale' codec can't decode byte 0xa0 in position 0: Invalid or incomplete multibyte or wide character

Apparently, the thousands separator (or something else) in the lconv is "\xa0" (unbreakable space in fr_FR.ISO8859-1), and it's being decoded with UTF-8.

This is tripped by Python's test suite, namely test_float.GeneralFloatCases.test_float_with_comma

One failed Koji build is here: https://koji.fedoraproject.org/koji/taskinfo?taskID=21419062

A smaller reproducer is here (self-contained SPEC file): https://gist.github.com/encukou/70b3d3f9ef3e29ac1dbc23a5f7bd6431

Comment 1 Miro Hrončok 2017-08-23 19:15:05 UTC
I was able to reproduce this with the self-contained SPEC file on rawhide, but not on f27 or f26.

rawhide: https://koji.fedoraproject.org/koji/taskinfo?taskID=21425550 (failed)
f27: https://koji.fedoraproject.org/koji/taskinfo?taskID=21425571 (completed)
f26: https://koji.fedoraproject.org/koji/taskinfo?taskID=21425582 (completed)

Comment 2 Miro Hrončok 2017-08-23 19:38:46 UTC
Package diff:

-bash.x86_644.4.12-9
+bash.x86_644.4.12-10

-cracklib.x86_642.9.6-7
+cracklib.x86_642.9.6-9

-crypto-policies.noarch20170816-1.git2618a6c
+crypto-policies.noarch20170823-1.git8d18c27

-curl.x86_647.55.1-2
-libcurl.x86_647.55.1-2
+curl.x86_647.55.1-3
+libcurl.x86_647.55.1-3

-file-libs.x86_645.31-9
+file-libs.x86_645.31-10

-file.x86_645.31-9
+file.x86_645.31-10

-glibc-all-langpacks.x86_642.26-4
-glibc-common.x86_642.26-4
-glibc-devel.x86_642.26-4
-glibc-headers.x86_642.26-4
-glibc.x86_642.26-4
+glibc-all-langpacks.x86_642.26.90-9
+glibc-common.x86_642.26.90-9
+glibc-devel.x86_642.26.90-9
+glibc-headers.x86_642.26.90-9
+glibc.x86_642.26.90-9

-gnutls.x86_643.5.15-1
+gnutls.x86_643.6.0-1

-kernel-headers.x86_644.13.0-0.rc5.git1.1
+kernel-headers.x86_644.13.0-0.rc6.git1.1

-krb5-libs.x86_641.15.1-22
+krb5-libs.x86_641.15.1-24

-libcap-ng.x86_640.7.8-5
+libcap-ng.x86_640.7.8-7

-libcrypt-nss.x86_642.26-4
+libcrypt-nss.x86_642.26.90-9



-libselinux.x86_642.7-1
-libsemanage.x86_642.7-1
+libselinux.x86_642.7-3
+libsemanage.x86_642.7-3

-libxml2.x86_642.9.4-4
+libxml2.x86_642.9.4-5

-setup.noarch2.10.5-3
+setup.noarch2.10.7-1


I bet the change in glibc is relevant.

Comment 3 Charalampos Stratakis 2017-10-30 13:21:06 UTC
I can verify that the issue is with the latest glibc and more specifically with the 2.26.90 version.

My tests compared version 2.26-15.fc27.x86_64 to 2.26.90-3.fc28 [0].

The easiest way to reproduce it, is to initialize a F27 chroot or VM, run the test (it will pass on this step) and then upgrade the five glibc* packages that come with the base install to those from [0] and rerun the test (it will fail here).

dnf will complain during upgrading so it's better to wget the rpm's and then install them through 'rpm -Uvh <rpm's>'.

[0] https://koji.fedoraproject.org/koji/buildinfo?buildID=956358

Comment 4 Charalampos Stratakis 2017-10-30 18:14:43 UTC
A PR has been submitted upstream which fixes the issue [0]. Waiting on it to be merged before backporting it.

[0] https://github.com/python/cpython/pull/4174

Comment 5 Fedora Update System 2018-01-24 08:25:20 UTC
python3-3.6.4-6.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-ff1b193de3

Comment 6 Fedora Update System 2018-01-25 08:37:39 UTC
python3-3.6.4-6.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-ff1b193de3

Comment 7 Fedora Update System 2018-01-26 18:10:04 UTC
python3-3.6.4-6.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.