Bug 1691434

Summary: python3 update causes program to deadlock
Product: [Fedora] Fedora Reporter: D. Hugh Redelmeier <hugh>
Component: python3Assignee: Miro Hrončok <mhroncok>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 29CC: cstratak, dmalcolm, mcyprian, mhroncok, pviktori, pwouters, rkuska, shcherbina.iryna, tomspur, torsava, yaneti
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python3-3.7.4-1.fc30 python3-3.7.4-1.fc29 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-29 01:08:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description D. Hugh Redelmeier 2019-03-21 15:32:08 UTC
Description of problem:
cf-deadlock.py hangs after a few minutes when using latest python3, python3-libs

Version-Release number of selected component (if applicable):
python3-3.7.2-4.fc29.x86_64
python3-libs-3.7.2-4.fc29.x86_64

How reproducible:
Takes a few minutes.

Steps to Reproduce:
1. fetch https://bugs.python.org/file48090/cf-deadlock.py
2. run it
3. wait a few minutes and the output will stop

Actual results:
The stream of "." characters on the console freezes.

Expected results:
The stream of "." characters on the console continues.


Additional info:
The program runs correctly with python3-3.7.1-4.fc29.x86_64 and python3-libs-3.7.1-4.fc29.x86_64

The libreswan team experiences a similar problem running the libreswan test-suite.  But testing with that is much less convenient than using cj-deadlock.py.  We hope that any fix for cj-deadlock.py will solve our problem.

cj-deadlock.py comes from a Python.org bug report: https://bugs.python.org/issue35866?

Comment 1 Paul Wouters 2019-04-12 08:08:42 UTC
I talked to Miro and he pointed out that:

That is indeed the only change logged for 3.7.2-5, however there have been other releases between 3.7.1-4 and 3.7.2-5:

* Thu Mar 14 2019 Miro Hrončok <mhroncok> - 3.7.2-5
- Security fix for CVE-2019-9636 (#1688543, #1688546)

* Wed Jan 16 2019 Miro Hrončok <mhroncok> - 3.7.2-4
- Security fix for CVE-2019-5010 (#1666519, #1666522)

* Mon Jan 14 2019 Björn Esser <besser82> - 3.7.2-3
- Rebuilt for libcrypt.so.2 (#1666033)

* Fri Jan 04 2019 Miro Hrončok <mhroncok> - 3.7.2-2
- No longer revert upstream commit 3b699932e5ac3e7
- This was a dirty workaround for (#1644936)

* Tue Dec 25 2018 Miro Hrončok <mhroncok> - 3.7.2-1
- Update to 3.7.2

* Fri Dec 07 2018 Miro Hrončok <mhroncok> - 3.7.1-5
- Make sure we don't ship any exe files (not needed an prebuilt)


   I'm pretty sure that "No longer revert upstream commit 3b699932e5ac3e7" is here
   to blame. Should I provide you with a test build of current version with that
   commit reverted once more?

Miro, that would be great, as we would really like to get this resolved as it is having a large impact on our parallel testing we do with libreswan.

Comment 2 Paul Wouters 2019-04-12 08:21:41 UTC
See also https://github.com/pexpect/ptyprocess/issues/53

It's not the same but related for us in our use case

Comment 3 Miro Hrončok 2019-04-12 08:23:24 UTC
https://src.fedoraproject.org/rpms/python3/pull-request/105
https://koji.fedoraproject.org/koji/taskinfo?taskID=34124221

Once it builds, grab it fast, scratch builds are garbage collected.

Comment 4 Miro Hrončok 2019-04-12 08:25:59 UTC
To clarify: fast = couple days here, not seconds

Comment 5 Miro Hrončok 2019-04-12 10:15:35 UTC
A better (hopefully) scratchbuild: https://koji.fedoraproject.org/koji/taskinfo?taskID=34124496

Comment 6 Paul Wouters 2019-04-12 19:31:23 UTC
that one does not seem the have any rpms associated with it?

Comment 7 Miro Hrončok 2019-04-12 21:14:04 UTC
It's a bad link, copy paste error. Sorry about that.

https://koji.fedoraproject.org/koji/taskinfo?taskID=34124810

Comment 8 Paul Wouters 2019-04-15 10:33:41 UTC
so we can confirm that build fixes the deadlock/hanging issue for the libreswan testing infrastructure :/

Comment 9 Miro Hrončok 2019-04-16 13:50:11 UTC
This is the anaconda fix for reference: https://github.com/rhinstaller/anaconda/pull/1721

Comment 10 Miro Hrončok 2019-05-17 20:30:40 UTC
Should be fixed upstream. Let me know if you need a backport right now or a build to test with. https://bugs.python.org/issue36533#msg341768

Comment 11 Miro Hrončok 2019-05-28 09:38:53 UTC
Paul e-mailed me and kindly asked us to backport https://github.com/python/cpython/pull/13170

IMO We are good only doing this in 3.7, because:

* F28 has 3.6 but EOLS tomorrow, so the fix would need to be super rush and it still might not make it (also, AFAIK libreswan doesn't need this)
* F29 and further has 3.7
* once we switch to 3.8 in F31, the fix will already be included

Comment 12 Miro Hrončok 2019-05-28 10:08:19 UTC
Note that an upstream release of 3.7.4 is to be released in little less than a month.

https://www.python.org/dev/peps/pep-0537/#id4

3.7.4 candidate 1: 2019-06-10 (expected)
3.7.4 final: 2019-06-24 (expected)

So we would only carry the patch for a while.

Comment 13 Paul Wouters 2019-05-28 14:29:00 UTC
that would be great, as this bug is causing our 8+ hour jobs to sometimes completely fail.

Comment 14 Miro Hrončok 2019-06-25 13:44:43 UTC
First of all, we are sorry that we were not able to dedicate the time for this during June. 

The 3.7.4 release is already at rc and should be out next Friday, as the release has been postponed by a week. I'll do my best to update soon.

Comment 16 Fedora Update System 2019-07-09 17:26:39 UTC
FEDORA-2019-9bfb4a3e4b has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-9bfb4a3e4b

Comment 17 Fedora Update System 2019-07-09 17:29:52 UTC
FEDORA-2019-60a1defcd1 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-60a1defcd1

Comment 18 Fedora Update System 2019-07-10 00:37:40 UTC
python3-3.7.4-1.fc30, python3-docs-3.7.4-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-9bfb4a3e4b

Comment 19 Fedora Update System 2019-07-10 02:41:36 UTC
python3-3.7.4-1.fc29, python3-docs-3.7.4-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-60a1defcd1

Comment 20 Fedora Update System 2019-07-29 01:08:07 UTC
python3-3.7.4-1.fc30, python3-docs-3.7.4-1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 21 Fedora Update System 2019-08-05 01:41:01 UTC
python3-3.7.4-1.fc29, python3-docs-3.7.4-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.