Created attachment 1445326 [details] build.log from copr Description of problem: python-kitchen FTBFS on Python 3.7 due to test failures. Version-Release number of selected component (if applicable): 1.2.5-3.fc29 Logs attached.
Created attachment 1445327 [details] root.log from copr
Ralph?
Ack - thanks for the ping.
Wow, it is super broken.
Miro, do you have a recommended way to do development here? Do we have a rawhide base image around with py3.7?
Recommendation for "normal" Python projects (such as all dependencies are installable with pip): Use venv with python3.7 from the python37 package. https://fedoramagazine.org/python-3-7-now-available-in-fedora/ Recommendation for "weird" Python packages (such as if they need RPM isntalled Python dependencies (python3-rpm, etc.)): Use rawhide mock with this copr: https://copr.fedorainfracloud.org/coprs/g/python/python3.7/builds/ (Feel free to ping me if some package got updated in Fedora and needs a bump.)
FYI: 3.7.0c1 is expected in 3 days (2018-06-11, [1]). I'd like to start building stuff in a side tag soon after that. This blocks the "packager stack", hence a bit critical. [1] https://www.python.org/dev/peps/pep-0537/
OK, I got all py3.7 tests passing again: https://github.com/fedora-infra/kitchen/pull/24 And applied that as a patch to the rpm on the master branch: https://src.fedoraproject.org/rpms/python-kitchen/c/0a273349664f3b8613a17776f7e685a01b444c04?branch=master Can you trigger a rebuild in copr? I don't know how.
Didn't help much. https://copr.fedorainfracloud.org/coprs/g/python/python3.7/build/765081/
Created attachment 1449382 [details] build.log from copr
Created attachment 1449383 [details] root.log from copr
Weird. I won't why I can't replicate that locally. Toshio, any ideas?
*wonder
Any chance it's b4 vs b5? b5 is in testing and also in the copr repo.
There's two changes to upstream Python in 3.7, one of which I think is causing this. * https://www.python.org/dev/peps/pep-0538/ From the description, this is the first one I would have guessed was causing this. However, IIRC, Fedora carried this as a patch for 3.6.x so we should have started seeing this then. * https://www.python.org/dev/peps/pep-0540/ The text of the PEP says "This mode is disabled by default and enabled by the POSIX locale. Users can explicitly disable UTF-8 Mode with the command-line option -X utf8=0 or by setting the environment variable PYTHONUTF8=0." That's a little hard for me to understand but one meaning could be that if the locale is C, then it is automatically enabled which would lead to this error. In either case, the majority of these tests ("lngettext method in a C locale" and lgettext method in a C locale") are failing due to needing an ascii-only locale and python-3.7 using utf8 with surrogateescape error handler instead. PYTHONUTF8 may fix PEP 540 and PYTHONCOERCECLOCALE may fix PEP538 (in the test_i18n.py file): def check_lgettext(self, message, value, charset=None, locale='en_US.UTF-8'): os.environ['LC_ALL'] = locale + os.environ['PYTHONUTF8'] = 0 self.translations.set_output_charset(charset) tools.eq_(self.translations.lgettext(message), value, msg='lgettext(%s): trans: %s != val: %s (charset=%s, locale=%s)' % (repr(message), repr(self.translations.lgettext(message)), repr(value), charset, locale)) Comments: * The code for check_lngettext() will be similar. * FAIL: Test that the easy_gettext_setup function works * FAIL: Test when we have an invalid fallback that it does not raise. * And the four lgettext and lngettext tests which are not running in the C locale may be something wholly different. * If either of the PEP behaviours read the environment variables on interpreter startup and ignores it later these may still fail. * Solution to that: Consult Victor Stinner to see how to specify an ascii-only locale for our purposes. (Maybe LC_ALL=C.ascii?)
Victor, please help.
Hmm... Python-3.7 dev as used by travis (Running on Ubuntu14.04) works fine: https://travis-ci.org/fedora-infra/kitchen/jobs/390416387 It looks like that is Python 3.7.0a4+. Victor, was the code for PEP538 and 540 added before alpha 4? If so, this could be a different problem.... In order to get the tests to pass on Ubuntu, I had to make sure that the pt_BR and pt_BR.UTF8 locales were generated. I've never had to do that with Fedora but perhaps that's been changed recently? (Or it's different on a machine installed for user-work versus the build machines?)
Here's a PR implementing https://bugzilla.redhat.com/show_bug.cgi?id=1583600#c15 https://github.com/fedora-infra/kitchen/pull/28 However, since things are passing in travis (and that threebean was testing it locally with the Python-3.7 package... I missed that nuance as well), I am not sure whether that addresses the real reason that the build is failing in copr. (And as stated in my earlier comment... even if one of the PEPs is the reason, I'm not sure if this is setting the environment variables too late to have an effect.) You're welcome to try the code from that PR (It is based on latest develop so the code at the HEAD of the PR contains threebean's earlier update to the combining tables) but I'm not sure if it will help or not. If it does help, it would help us know we're on the right track and something similar is probably needed for the remaining unittests which were not implemented via check_lgettext and check_lngettext.
Testing in copr.
BTW "With locale coercion now only setting LC_CTYPE and LANG, it will have no effect if LC_ALL is also set." https://www.python.org/dev/peps/pep-0538/#skipping-locale-coercion-if-lc-all-is-set-in-the-current-environment
https://copr.fedorainfracloud.org/coprs/g/python/python3.7/build/765281/ fails in the same way
I'm testing: - running tests locally with LANG=C (fails in the same way as in 3.7 Copr!) - building the same srpm in python 3.6 copr - setting LANG to C.utf8 in %check in 3.7 copr
> - building the same srpm in python 3.6 copr builds ok. > - setting LANG to C.utf8 in %check in 3.7 copr builds ok. --- a/python-kitchen.spec +++ b/python-kitchen.spec %check +export LANG=C.utf8 That's it. Did we find some strange 3.7 bug?
How are you testing locally that is getting it to fail? I read your message and tried this but it succeeded: [pts/17@peru /srv/kitchen/kitchen/kitchen3]$ LANG=C PYTHONPATH=. nosetests *[ascii-locale-tests] (19:33:40) ........................................................................S................S..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S..................... ---------------------------------------------------------------------- Ran 590 tests in 8.037s OK (SKIP=3) [pts/17@peru /srv/kitchen/kitchen/kitchen3]$ which nosetests *[ascii-locale-tests] (19:33:50) /srv/kitchen/kitchen/testenv/bin/nosetests [pts/17@peru /srv/kitchen/kitchen/kitchen3]$ python --version *[ascii-locale-tests] (19:33:53) Python 3.7.0b4 Python-3.7.0b4 is installed (and the virtualenv created) from the fedoramagazine article posted above: https://fedoramagazine.org/python-3-7-now-available-in-fedora/ nosetests is installed via pip install nose inside of the virtualenv. (Test OS is Fedora 27)
[kitchen (develop)]$ python3.7 -m venv __venv__ [kitchen (develop)]$ . __venv__/bin/activate (__venv__) [kitchen (develop)]$ pip install nose ... (__venv__) [kitchen (develop)]$ cd kitchen3 (__venv__) [kitchen3 (develop)]$ LANG=C python -m nose # or LANG=C nosetests I have 3.7.0b5.
How do I get 3.7.0b5?
(In reply to Toshio Kuratomi from comment #26) > How do I get 3.7.0b5? https://bodhi.fedoraproject.org/updates/?packages=python37 (In reply to Miro Hrončok from comment #23) > > - setting LANG to C.utf8 in %check in 3.7 copr > > builds ok. > > --- a/python-kitchen.spec > +++ b/python-kitchen.spec > %check > +export LANG=C.utf8 > > > That's it. https://src.fedoraproject.org/rpms/python-kitchen/pull-request/2
Okay, we're still missing a piece of the puzzle... Python3.7 installed on Fedora 27 works for me: [pts/12@peru /srv/kitchen/kitchen]$ python3.7 -m venv __venv__ *[ascii-locale-tests] (06:57:06) [pts/12@peru /srv/kitchen/kitchen]$ . __venv__/bin/activate *[ascii-locale-tests] (06:57:10) [pts/12@peru /srv/kitchen/kitchen]$ pip install nose *[ascii-locale-tests] (06:57:15) Collecting nose Using cached https://files.pythonhosted.org/packages/15/d8/dd071918c040f50fa1cf80da16423af51ff8ce4a0f2399b7bf8de45ac3d9/nose-1.3.7-py3-none-any.whl Installing collected packages: nose Successfully installed nose-1.3.7 [pts/12@peru /srv/kitchen/kitchen]$ cd kitchen3 *[ascii-locale-tests] (06:57:21) [pts/12@peru /srv/kitchen/kitchen/kitchen3]$ LANG=C python -m nose # or LANG=C nosetests ...................................................................../srv/kitchen/kitchen/kitchen3/tests/base_classes.py:73: DeprecationWarning: invalid escape sequence \. repr_re = re.compile(b'^<[^ ]*\.([^.]+) object at .*>$') ...S................S..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S..................... ---------------------------------------------------------------------- Ran 590 tests in 10.977s OK (SKIP=3) [pts/12@peru /srv/kitchen/kitchen/kitchen3]$ python --version *[ascii-locale-tests] (06:57:53) Python 3.7.0b5 Could it be a change in Fedora 29?
*Python 3.7.0b5 installed on Fedora 27 works for me
For me, it's on Fedora 28. But let me test in F27 mock.
In F27, this fails as well: $ mock -r fedora-27-x86_64 --install git https://kojipkgs.fedoraproject.org//packages/python37/3.7.0/0.19.b5.fc27/x86_64/python37-3.7.0-0.19.b5.fc27.x86_64.rpm $ mock -r fedora-27-x86_64 --shell --enable-network <mock-chroot> sh-4.4# git clone https://github.com/fedora-infra/kitchen.git <mock-chroot> sh-4.4# cd kitchen/ <mock-chroot> sh-4.4# python3.7 -m venv __venv__ <mock-chroot> sh-4.4# . __venv__/bin/activate (__venv__) <mock-chroot> sh-4.4# pip install nose ... Successfully installed nose-1.3.7 (__venv__) <mock-chroot> sh-4.4# cd kitchen3 (__venv__) <mock-chroot> sh-4.4# LANG=C python -m nose ...FFFF....
Okay, I'm able to reproduce inside of the mock. So there's something different inside of there. I'll try to figure out what we can do about that when I have some time today.
Okay, it works with: LANG=C PYTHONUTF8=0 python -m nose So Victor, it is UTF8 mode (PEP-538) that is causing the problem in the test cases. If you have some input on how best to get an ascii locale from within a unittest for testing, that would be great.
Here's the options I think I see to fix this in the upstream tests: * Find a locale that we can use which will be ascii-encoded. * Pros: Will test that the code is robust in an ascii-only locale. (Running under older Python, running on the locale that we find). * Cons: I don't think I can see any * Give the unittests different versions of the output if we're on Python-3.7 instead of python < 3.7. * Pros: the output is "good" (not mangled) in more cases (2) these tests are concenred primarily with whether the code tracesback (it doesn't) and in some cases, making sure the output is not mangled (this change means 3.7 mangles in less cases). * Cons: Having conditionals in unittests is sucky as it makes the unittest itself more prone to bugs. Probably should code the alternatives as skip on Python-3.7 or skip on Python < 3.7. People who may have depended on particular output will now get something else (but this all appears to be gettext output so people are only depending on this if they're screenscraping or misuing gettext for something it wasn't meant for). There's a fair amount of work modifying all the test cases to have Python-3.7+ versions. Doesn't test all code paths on Python-3.7+ as we can't test that we do not traceback in an ascii locale if we never use an ascii locale. * Set PYTHONUTF8=0 when running the tests. * Pros: Easy. Tests the case of an ascii locale. * Cons: Doesn't test the way that most people will run under Python-3.7+. We can mitigate this by running the tests twice. Once with PYTHONUTF8="0" and once without. An ascii-only locale would be my preferred method of fixing this. So I'm going to wait for input from Victor... If we don't have that, I'm not sure which of the other two options we'd want. ralphbean, thoughts on which of those two is preferred (You can assume I'll do the work for either of them so it's a question of "rightness" rather than "do I have time for this")?
What strikes me a bid odd that this hasn't been a problem on our 3.6. https://fedoraproject.org/wiki/Changes/python3_c.utf-8_locale
Oops, I've been reversing my PEP numbers. PEP-538 is nick's PEP. PEP-540 is Victor's PEP. It is PEP-540 which is causing this problem. PYTHONUTF8=0 disables PEP-540 behaviour.
I am confused, though, as to why PEP-538 causes this to fail inside of my mock of Feodra-27 but not outside of it (also Fedora 27). The python37 package is the same in both places so I'm not sure what is causing that difference.
*sigh* PEP-540 causes this to fail inside of my mock....
FYI There are ~70 pkgs left in the chain before I'll need to rebuild kitchen. If no package FTBFS (unlikely), 70 packages is (very approximately) 12-15 hours. Let me know whether I should: 1) skip the tests 2) PYTHONUTF8=0 3) LANG=C.utf8 4) block the rebuild (I'd rather not) Note that I can do 1/2/3, rebuild and revert the change, so a proper fix can be done.
Toshio, to your original question, your option 1 makes the most sense: "Find a locale that we can use which will be ascii-encoded." To Miro's question, his option 1 (skip the tests) makes the most sense short-term if Toshio doesn't have time to find and use an appropriate locale by then.
Built with tests skipped.
Thanks Miro!
Note, I think (2) build with PYTHONUTF8=0 is probably the best option if we make no upstream changes (as it runs all of the tests, just runs them closer to how they'd be run under Python-3.6 than how they'd be run under Python-3.7 (by default). Skipping all tests leaves more potential for bugs to exist.
*** This bug has been marked as a duplicate of bug 1605741 ***