Bug 1484497

Summary: UnicodeDecodeError in localeconv() makes test_float fail in Koji
Product: [Fedora] Fedora Reporter: Petr Viktorin (pviktori) <pviktori>
Component: python3Assignee: Charalampos Stratakis <cstratak>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: bkabrda, cstratak, ishcherb, mcyprian, mhroncok, pviktori, rkuska, tomspur, torsava
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: python3-3.6.4-6.fc27 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-01-26 18:10:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Petr Viktorin (pviktori) 2017-08-23 17:58:16 UTC
I can reproduce this bug only in Koji, and in Mock if the buildroot is cleaned immediately before. I don't know how to reproduce it on a system/chroot I can SSH into.

Under certain circumstances, when LC_NUMERIC is fr_FR.ISO8859-1 but LC_ALL is C.UTF-8, locale.localeconv() fails with
    UnicodeDecodeError: 'locale' codec can't decode byte 0xa0 in position 0: Invalid or incomplete multibyte or wide character

Apparently, the thousands separator (or something else) in the lconv is "\xa0" (unbreakable space in fr_FR.ISO8859-1), and it's being decoded with UTF-8.

This is tripped by Python's test suite, namely test_float.GeneralFloatCases.test_float_with_comma

One failed Koji build is here: https://koji.fedoraproject.org/koji/taskinfo?taskID=21419062

A smaller reproducer is here (self-contained SPEC file): https://gist.github.com/encukou/70b3d3f9ef3e29ac1dbc23a5f7bd6431

Comment 1 Miro Hrončok 2017-08-23 19:15:05 UTC
I was able to reproduce this with the self-contained SPEC file on rawhide, but not on f27 or f26.

rawhide: https://koji.fedoraproject.org/koji/taskinfo?taskID=21425550 (failed)
f27: https://koji.fedoraproject.org/koji/taskinfo?taskID=21425571 (completed)
f26: https://koji.fedoraproject.org/koji/taskinfo?taskID=21425582 (completed)

Comment 2 Miro Hrončok 2017-08-23 19:38:46 UTC
Package diff:
















I bet the change in glibc is relevant.

Comment 3 Charalampos Stratakis 2017-10-30 13:21:06 UTC
I can verify that the issue is with the latest glibc and more specifically with the 2.26.90 version.

My tests compared version 2.26-15.fc27.x86_64 to 2.26.90-3.fc28 [0].

The easiest way to reproduce it, is to initialize a F27 chroot or VM, run the test (it will pass on this step) and then upgrade the five glibc* packages that come with the base install to those from [0] and rerun the test (it will fail here).

dnf will complain during upgrading so it's better to wget the rpm's and then install them through 'rpm -Uvh <rpm's>'.

[0] https://koji.fedoraproject.org/koji/buildinfo?buildID=956358

Comment 4 Charalampos Stratakis 2017-10-30 18:14:43 UTC
A PR has been submitted upstream which fixes the issue [0]. Waiting on it to be merged before backporting it.

[0] https://github.com/python/cpython/pull/4174

Comment 5 Fedora Update System 2018-01-24 08:25:20 UTC
python3-3.6.4-6.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-ff1b193de3

Comment 6 Fedora Update System 2018-01-25 08:37:39 UTC
python3-3.6.4-6.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-ff1b193de3

Comment 7 Fedora Update System 2018-01-26 18:10:04 UTC
python3-3.6.4-6.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.