Bug 1609298

Summary: koji: When running under Python 3, wait-* commands will not wait indefinitely
Product: [Fedora] Fedora Reporter: Florian Weimer <fweimer>
Component: kojiAssignee: Mike McLean <mikem>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: awilliam, dennis, hobbes1069, jkeating, mikem, projects.rg, pzhukov, rbarlow, rharwood, rjones, swt, thofmann, vondruch
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: koji-1.16.1-3.fc29 koji-1.16.1-3.fc28 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-13 02:31:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Florian Weimer 2018-07-27 13:32:43 UTC
Example:

$ koji watch-task 28644925
Watching tasks (this may be safely interrupted)...
28644925 build (f29-binutils, /rpms/mame.git:ed5e2ca7746bae9042ed03640e8704d44023deca): open (buildhw-07.phx2.fedoraproject.org)
  28645038 buildArch (mame-0.200-2.fc29.src.rpm, s390x): open (buildvm-s390x-03.s390.fedoraproject.org)
  28645039 buildArch (mame-0.200-2.fc29.src.rpm, aarch64): open (buildvm-aarch64-13.arm.fedoraproject.org)
  28645041 buildArch (mame-0.200-2.fc29.src.rpm, i686): open (buildvm-14.phx2.fedoraproject.org)
  28645031 buildArch (mame-0.200-2.fc29.src.rpm, armv7hl): open (buildvm-armv7-15.arm.fedoraproject.org)
  28645036 buildArch (mame-0.200-2.fc29.src.rpm, x86_64): open (buildvm-32.phx2.fedoraproject.org)
  28644926 buildSRPMFromSCM (/rpms/mame.git:ed5e2ca7746bae9042ed03640e8704d44023deca): closed
ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

This does not happen when running under Python 2.

Seen with:

koji-1.15.1-1.fc27.noarch
python3-requests-2.18.4-2.fc27.noarch
python3-urllib3-1.22-6.fc27.noarch
python3-libs-3.6.5-1.fc27.x86_64
python3-pyOpenSSL-17.2.0-2.fc27.noarch

Comment 1 Pavel Zhukov 2018-11-20 14:12:23 UTC
koji-1.16.1-2.fc30.noarch
python3-requests-2.20.0-1.fc30.noarch

Comment 2 Richard Shaw 2018-12-13 14:59:56 UTC
I'm seeing this on Fedora 29... Will new packages be built for it as well?

Comment 3 Raphael Groner 2018-12-17 15:58:01 UTC
Me too.

Comment 4 Randy Barlow 2018-12-19 02:39:50 UTC
https://github.com/fedora-infra/bodhi/issues/2870 may or may not be related, but Bodhi frequently gets an error from Koji under Python 3 that it did not get under Python 2. It's not the same error as this, however.

Comment 5 Randy Barlow 2018-12-20 16:36:18 UTC
We've been having a very difficult time getting Bodhi to authenticate with Koji today, and I observed this same error message in Bodhi's logs to Koji. Thus, I think https://github.com/fedora-infra/bodhi/issues/2870 is related.

Comment 6 Randy Barlow 2018-12-20 16:36:52 UTC
There is now an infrastructure ticket about this as well: https://pagure.io/fedora-infrastructure/issue/7458

Comment 7 Adam Williamson 2019-01-08 20:10:24 UTC
I believe https://pagure.io/koji/pull-request/1203 should fix this.

Comment 8 Mike McLean 2019-01-08 20:25:08 UTC
I wonder if this is some variation on the keepalive race [1]. Or perhaps Fedora's hub really is just shutting down the connection due to load or something. I'm not sure why this would be py3 specific, but if it is a race, then odd things might tickle it.

Have you adjusted your poll_interval config in koji.conf?

If anyone can reliably reproduce this with --debug on, I'd be interested to see the full traceback.

Comment 9 Mike McLean 2019-01-08 20:28:13 UTC
Possible workaround: enable anon_retry in your koji.conf

Comment 10 Adam Williamson 2019-01-08 20:39:11 UTC
Uh, Mike, did you see my pull request? I'm pretty sure I know what the problem is :)

Comment 11 Mike McLean 2019-01-08 20:40:20 UTC
Looking now. We had a mid-air collision

Comment 12 Fedora Update System 2019-01-09 17:47:44 UTC
koji-1.16.1-3.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-1fba27cd97

Comment 13 Fedora Update System 2019-01-09 17:47:48 UTC
koji-1.16.1-3.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2019-f1d84fda6e

Comment 14 Fedora Update System 2019-01-11 04:16:49 UTC
koji-1.16.1-3.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-1fba27cd97

Comment 15 Fedora Update System 2019-01-11 05:42:46 UTC
koji-1.16.1-3.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-f1d84fda6e

Comment 16 Fedora Update System 2019-01-13 02:31:11 UTC
koji-1.16.1-3.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Comment 17 Fedora Update System 2019-01-18 01:37:57 UTC
koji-1.16.1-3.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.