Bug 1484497

Summary: UnicodeDecodeError in localeconv() makes test_float fail in Koji
Product: [Fedora] Fedora Reporter: Petr Viktorin <pviktori>
Component: python3Assignee: Charalampos Stratakis <cstratak>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: bkabrda, cstratak, ishcherb, mcyprian, mhroncok, pviktori, rkuska, tomspur, torsava
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python3-3.6.4-6.fc27 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-01-26 18:10:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Petr Viktorin 2017-08-23 17:58:16 UTC
I can reproduce this bug only in Koji, and in Mock if the buildroot is cleaned immediately before. I don't know how to reproduce it on a system/chroot I can SSH into.


Under certain circumstances, when LC_NUMERIC is fr_FR.ISO8859-1 but LC_ALL is C.UTF-8, locale.localeconv() fails with
    UnicodeDecodeError: 'locale' codec can't decode byte 0xa0 in position 0: Invalid or incomplete multibyte or wide character

Apparently, the thousands separator (or something else) in the lconv is "\xa0" (unbreakable space in fr_FR.ISO8859-1), and it's being decoded with UTF-8.

This is tripped by Python's test suite, namely test_float.GeneralFloatCases.test_float_with_comma

One failed Koji build is here: https://koji.fedoraproject.org/koji/taskinfo?taskID=21419062

A smaller reproducer is here (self-contained SPEC file): https://gist.github.com/encukou/70b3d3f9ef3e29ac1dbc23a5f7bd6431

Comment 1 Miro Hrončok 2017-08-23 19:15:05 UTC
I was able to reproduce this with the self-contained SPEC file on rawhide, but not on f27 or f26.

rawhide: https://koji.fedoraproject.org/koji/taskinfo?taskID=21425550 (failed)
f27: https://koji.fedoraproject.org/koji/taskinfo?taskID=21425571 (completed)
f26: https://koji.fedoraproject.org/koji/taskinfo?taskID=21425582 (completed)

Comment 2 Miro Hrončok 2017-08-23 19:38:46 UTC
Package diff:

-bash.x86_644.4.12-9
+bash.x86_644.4.12-10

-cracklib.x86_642.9.6-7
+cracklib.x86_642.9.6-9

-crypto-policies.noarch20170816-1.git2618a6c
+crypto-policies.noarch20170823-1.git8d18c27

-curl.x86_647.55.1-2
-libcurl.x86_647.55.1-2
+curl.x86_647.55.1-3
+libcurl.x86_647.55.1-3

-file-libs.x86_645.31-9
+file-libs.x86_645.31-10

-file.x86_645.31-9
+file.x86_645.31-10

-glibc-all-langpacks.x86_642.26-4
-glibc-common.x86_642.26-4
-glibc-devel.x86_642.26-4
-glibc-headers.x86_642.26-4
-glibc.x86_642.26-4
+glibc-all-langpacks.x86_642.26.90-9
+glibc-common.x86_642.26.90-9
+glibc-devel.x86_642.26.90-9
+glibc-headers.x86_642.26.90-9
+glibc.x86_642.26.90-9

-gnutls.x86_643.5.15-1
+gnutls.x86_643.6.0-1

-kernel-headers.x86_644.13.0-0.rc5.git1.1
+kernel-headers.x86_644.13.0-0.rc6.git1.1

-krb5-libs.x86_641.15.1-22
+krb5-libs.x86_641.15.1-24

-libcap-ng.x86_640.7.8-5
+libcap-ng.x86_640.7.8-7

-libcrypt-nss.x86_642.26-4
+libcrypt-nss.x86_642.26.90-9



-libselinux.x86_642.7-1
-libsemanage.x86_642.7-1
+libselinux.x86_642.7-3
+libsemanage.x86_642.7-3

-libxml2.x86_642.9.4-4
+libxml2.x86_642.9.4-5

-setup.noarch2.10.5-3
+setup.noarch2.10.7-1


I bet the change in glibc is relevant.

Comment 3 Charalampos Stratakis 2017-10-30 13:21:06 UTC
I can verify that the issue is with the latest glibc and more specifically with the 2.26.90 version.

My tests compared version 2.26-15.fc27.x86_64 to 2.26.90-3.fc28 [0].

The easiest way to reproduce it, is to initialize a F27 chroot or VM, run the test (it will pass on this step) and then upgrade the five glibc* packages that come with the base install to those from [0] and rerun the test (it will fail here).

dnf will complain during upgrading so it's better to wget the rpm's and then install them through 'rpm -Uvh <rpm's>'.

[0] https://koji.fedoraproject.org/koji/buildinfo?buildID=956358

Comment 4 Charalampos Stratakis 2017-10-30 18:14:43 UTC
A PR has been submitted upstream which fixes the issue [0]. Waiting on it to be merged before backporting it.

[0] https://github.com/python/cpython/pull/4174

Comment 5 Fedora Update System 2018-01-24 08:25:20 UTC
python3-3.6.4-6.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-ff1b193de3

Comment 6 Fedora Update System 2018-01-25 08:37:39 UTC
python3-3.6.4-6.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-ff1b193de3

Comment 7 Fedora Update System 2018-01-26 18:10:04 UTC
python3-3.6.4-6.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.