Description of problem: Since glibc-2.33.9000-13.fc35, Python (3.10, 3.9, 3.8, 3.7...) fails to build on Fedora Rawhide due to test failures: ====================================================================== ERROR: test_float_parsing (test.test__locale._LocaleTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/builddir/build/BUILD/Python-3.10.0b2/Lib/test/test__locale.py", line 188, in test_float_parsing if localeconv()['decimal_point'] != '.': UnicodeDecodeError: 'locale' codec can't decode byte 0xa0 in position 0: decoding error ====================================================================== ERROR: test_lc_numeric_basic (test.test__locale._LocaleTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/builddir/build/BUILD/Python-3.10.0b2/Lib/test/test__locale.py", line 155, in test_lc_numeric_basic li_radixchar = localeconv()[lc] UnicodeDecodeError: 'locale' codec can't decode byte 0xa0 in position 0: decoding error ====================================================================== ERROR: test_lc_numeric_localeconv (test.test__locale._LocaleTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/builddir/build/BUILD/Python-3.10.0b2/Lib/test/test__locale.py", line 134, in test_lc_numeric_localeconv formatting = localeconv() UnicodeDecodeError: 'locale' codec can't decode byte 0xa0 in position 0: decoding error ====================================================================== ERROR: test_lc_numeric_nl_langinfo (test.test__locale._LocaleTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/builddir/build/BUILD/Python-3.10.0b2/Lib/test/test__locale.py", line 120, in test_lc_numeric_nl_langinfo if self.numeric_tester('nl_langinfo', nl_langinfo(li), lc, loc): UnicodeDecodeError: 'locale' codec can't decode byte 0xa0 in position 0: decoding error ---------------------------------------------------------------------- Version-Release number of selected component (if applicable): glibc-2.33.9000-13.fc35 or newer, at least up to -18.fc35 Steps to Reproduce: Build python3.X package in rawhide buildroot. (I'll try to isolate a smaller reproducer later.) Actual results: %check fails with the errors above. Expected results: %check passes Additional info: This was discovered by Koschei, but I've bisected the build that started this further down: https://koschei.fedoraproject.org/package/python3.10?collection=f35 https://koschei.fedoraproject.org/package/python3.9?collection=f35 https://koschei.fedoraproject.org/package/python3.8?collection=f35 https://koschei.fedoraproject.org/package/python3.7?collection=f35 This blocks upgrade from Python 3.10.0b2 to Python 3.10.0b3.
Also reproduced with glibc-2.33.9000-13.fc35 (will edit the initial comment here to have that version).
Would you please check if adding BuildRequires: glibc-gconv-extra fixes this? Thanks.
(In reply to Florian Weimer from comment #2) > Would you please check if adding > > BuildRequires: glibc-gconv-extra > > fixes this? Thanks. $ git diff diff --git a/python3.9.spec b/python3.9.spec index 9abe196..91143c7 100644 --- a/python3.9.spec +++ b/python3.9.spec @@ -222,6 +222,7 @@ BuildRequires: gdbm-devel BuildRequires: git-core BuildRequires: glibc-all-langpacks BuildRequires: glibc-devel +BuildRequires: glibc-gconv-extra BuildRequires: gmp-devel BuildRequires: gnupg2 BuildRequires: libappstream-glib https://koji.fedoraproject.org/koji/taskinfo?taskID=70359053
Yes indeed. Should that be required by Python on runtime?
(In reply to Miro Hrončok from comment #4) > Yes indeed. Should that be required by Python on runtime? Great. Outside mock, we default to installing weak dependencies, so the package will be installed by default. A strong dependency from Python would make it basically impossible to deinstall it, so it entirely defeats the gconv split. We are considering adding a stronger dependency to glibc-all-langpacks or redhat-rpm-config to help along downstream. There is a strong desire to backport this to reduce minimal container size.
So, what should we do? If we add: BuildRequires: (glibc-gconv-extra if glibc > 2.33.9000-12) It will most likely fix the build but break Python for some users who install without recommends.
I think we should add Requires: (glibc-gconv-extra if redhat-rpm-config) to glibc-all-langpacks. This is based on the observation that BuildRequires: glibc-all-langpacks is used by many packages to indicate that they can't work with the minimal set of locales in the buildroot.
I agree. I still don't understand the impact of this change for the actual users, but at least that would unblock our test failures.
Also, maybe the split something to be communicated more explicitly in https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/7R4RBS2MD75S2MTY42L44IW3KOUT5K36/ ?
(In reply to Miro Hrončok from comment #9) > Also, maybe the split something to be communicated more explicitly in > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/ > thread/7R4RBS2MD75S2MTY42L44IW3KOUT5K36/ ? The split isn't really related to the snapshots, it was a case of parallel development. We simply forgot that mock defaults to not installing weak dependencies, we expected the change to be invisible to users by default.
On Rawhide (glibc-2.33.9000-18.fc35.x86_64), if I uninstall glibc-gconv-extra, mbstowcs() fails to decode non-ASCII bytes (0x80..0xFF) for the lv_LV locale, whereas nl_langinfo(CODESET) announces the ISO-8859-13 encoding. If I install the package, mbstowcs() decodes b"\xA0" byte as the U+00A0 character as expected (ISO-8859-13 encoding). Maybe setlocale(LC_CTYPE, "lv_LV") must fail if glibc-gconv-extra is not installed. Or at least, localeconv() and nl_langinfo() must only return byte strings which can be decoded with mbstowcs() (only return ASCII strings). Modifying the Python package to require glibc-gconv-extra works around the issue, but it would make the Python installation larger, even if it's not needed. I would suggest to only use a *strong* dependency to *test* Python (as we do for glibc-all-langpacks). Python doesn't stricly *require* all available and working locales to run. Only tests require them.
Python locale.nl_langinfo() function also has an old know encoding issue with complex locale configuration: https://bugs.python.org/issue25812 But it doesn't seem to matter here. Here it seems like it's a glibc change which causes Python test_locale and test__locale to fail.
(In reply to Victor Stinner from comment #11) > On Rawhide (glibc-2.33.9000-18.fc35.x86_64), if I uninstall > glibc-gconv-extra, mbstowcs() fails to decode non-ASCII bytes (0x80..0xFF) > for the lv_LV locale, whereas nl_langinfo(CODESET) announces the ISO-8859-13 > encoding. If I install the package, mbstowcs() decodes b"\xA0" byte as the > U+00A0 character as expected (ISO-8859-13 encoding). Thank you for pointing that out, I had not considered this aspect. I have filed this as an upstream bug: https://sourceware.org/bugzilla/show_bug.cgi?id=27996 We will fix this separate in Fedora (unless upstream does not want to). glibc-2.33.9000-23.fc35 is building in the f35-build-side-42875 side tag (with just the dependency change). I will try to rebuild python3.10 there and merge the side tag if that succeeds. Sorry for the inconvenience.
Just a note, python3.9 will build faster than python3.10 as it does not build/test the debug build.
Successfully waited 3:19 for glibc-2.33.9000-23.fc35 to appear in the f35-build-side-42875 repo Building python3.9-3.9.5-3.fc35 for f35-build-side-42875 Created task: 70367829 Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=70367829
The arm builds are still running but I think we can call it a success.
My builds passed as well, merging the side tag.
Thanks for the fast response.
FEDORA-2021-88b61c4e53 has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report.
> Status: MODIFIED → CLOSED Oh, that was quick! Thanks for the fix Florian.