So stompest has been failing rebuild for Python 3.9; it looks to me like the build actually gets stuck during the %check phase, it gets here:

+ /usr/bin/python3 -m pytest -v --ignore stompest/twisted/tests/async_client_integration_test.py stompest/twisted/tests
============================= test session starts ==============================
platform linux -- Python 3.9.0b1, pytest-4.6.10, py-1.8.0, pluggy-0.13.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /builddir/build/BUILD/stompest-715f358b8503320ea42bd4de6682916339943edc/src/twisted
collecting ... collected 14 items

and then I think actually just gets stuck there until the build times out. Not sure why, yet.

It built before: https://copr.fedorainfracloud.org/coprs/g/python/python3.9/package/python-stompest/

Hello,

This is the first reminder (step 3 from https://docs.fedoraproject.org/en-US/fesco/Fails_to_build_from_source_Fails_to_install/#_package_removal_for_long_standing_ftbfs_and_fti_bugs).

If you know about this problem and are planning on fixing it, please acknowledge so by setting the bug status to ASSIGNED. If you don't have time to maintain this package, consider orphaning it, so maintainers of dependent packages realize the problem.

So I'm looking into this. It's gonna be much fun, as it involves async stuff. It reproduces easily in mock, which I guess is the good news. It seems like what happens is, pytest queues up test_connected_timeout() first (I'm not sure why it doesn't start with test_connection_timeout()). We actually run through the whole of that test, but then we just...get stuck eternally. I hacked the test with a bunch of print()s:

    @defer.inlineCallbacks
    def test_connected_timeout(self):
        print("XXX HERE 7!")
        port = self.connections[0].getHost().port
        print("XXX HERE 8!")
        config = StompConfig(uri='tcp://localhost:%d' % port)
        print("XXX HERE 9!")
        client = Stomp(config)
        print("XXX HERE 10!")
        try:
            print("XXX HERE 11!")
            ret = client.connect(connectedTimeout=self.TIMEOUT)
            print("XXX HERE 11.1!")
            print(ret)
            print("XXX HERE 11.2!")
            yield(ret)
        except StompCancelledError:
            print("XXX HERE 12 GOT STOMPCANCELLEDERROR!")
            pass
        else:
            print("XXX HERE 13!")
            raise Exception('Expected connected timeout, but connection was established.')

and this is the output:

<mock-chroot> sh-5.0# PYTHONPATH=../core python3 -m pytest -vvvvv -s --ignore stompest/twisted/tests/async_client_integration_test.py stompest/twisted/tests/debug.py 
================================================ test session starts =================================================
platform linux -- Python 3.9.0b1, pytest-5.4.3, py-1.8.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /builddir/build/BUILD/stompest-715f358b8503320ea42bd4de6682916339943edc/src/twisted
collected 2 items                                                                                                    

stompest/twisted/tests/debug.py::AsyncClientConnectTimeoutTestCase::test_connected_timeout INFO:twisted:--> stompest.twisted.tests.debug.AsyncClientConnectTimeoutTestCase.test_connected_timeout <--
XXX HERE 1!
XXX HERE 2!
XXX HERE 3!
XXX HERE 4!
INFO:twisted:Factory starting on 38051
INFO:twisted:Starting factory <twisted.internet.protocol.Factory object at 0x7f5836bfba90>
XXX HERE 5!
XXX HERE 6!
[<<class 'twisted.internet.tcp.Port'> of <class 'twisted.internet.protocol.Factory'> on 38051>]
XXX HERE 7!
XXX HERE 8!
XXX HERE 9!
XXX HERE 10!
XXX HERE 11!
INFO:stompest.twisted.protocol:Connecting to localhost:38051 ...
INFO:twisted:Starting factory <stompest.twisted.protocol.StompFactory object at 0x7f5836b8aa90>
XXX HERE 11.1!
<Deferred at 0x7f5836b8ae80>
XXX HERE 11.2!
DEBUG:stompest.twisted.tests.broker_simulator:Connection made
DEBUG:stompest.twisted.protocol:Sending CONNECT frame [version=1.0]
DEBUG:stompest.twisted.tests.broker_simulator:Received CONNECT frame [version=1.0]
ERROR:stompest.twisted.listener:Disconnect because of failure: STOMP broker did not answer on time [timeout=1.0]
INFO:stompest.twisted.listener:Disconnecting ... [reason=STOMP broker did not answer on time [timeout=1.0]]
INFO:stompest.twisted.listener:Disconnected: Connection was closed cleanly.
DEBUG:stompest.twisted.listener:Calling disconnected errback: STOMP broker did not answer on time [timeout=1.0]
XXX HERE 12 GOT STOMPCANCELLEDERROR!

and that's where it sticks. So we get to the point where test_connected_timeout() should have completed successfully - we caught the exception we wanted to catch - but now instead of continuing on, the test process just hangs indefinitely. I suspect it's waiting for some kind of callback or something from the `client.connect()` call.

Compared to the log from the successful run of the test Miro mentioned in #c2, there's a couple of obvious differences in the environment:

SUCCESSFUL RUN: platform linux -- Python 3.9.0a6, pytest-4.6.9, py-1.8.0, pluggy-0.13.0 -- /usr/bin/python3
NOW:            platform linux -- Python 3.9.0b1, pytest-5.4.3, py-1.8.0, pluggy-0.13.1 -- /usr/bin/python3

so we have Python 3.9 beta 1 rather than alpha 6, and we have pytest 5.4.3 rather than 4.6.9. The SUCCESSFUL run used python3-twisted-19.10.0-2.fc33.x86_64 , currently we have python3-twisted-19.10.0-3.fc33.x86_64 , so there isn't a significant difference there - it's most likely either Python or pytest, I guess.

I downgraded to pytest 4.6.10 and the hang still happens. Only one change was made between 4.6.9 and 4.6.10 and it doesn't look relevant, so Python 3.9 beta 1 vs. alpha 6 seems like the best suspect now.

I think the defer.inlineCallbacks wrapper is probably pretty relevant here, my best guess so far is that's not working as intended with the newer Python 3.9. It's part of twisted. I tried with twisted 20.3.0 but that still hangs.

(In reply to Adam Williamson from comment #5)
> I downgraded to pytest 4.6.10 and the hang still happens. Only one change
> was made between 4.6.9 and 4.6.10 and it doesn't look relevant, so Python
> 3.9 beta 1 vs. alpha 6 seems like the best suspect now.

That seems correct, the hang started to happen before we upgraded pytest to 5.

I'm currently trying to build a current-style Python 3.9a6 package to confirm (can't use the one in Koji because it's the old style that was designed to install alongside a 3.8 'python3'). jcline is also looking into it.

I did some poking, Python is stuck in an infinite loop in _PyErr_SetObject because the object's context == the object so "while ((context = PyException_GetContext(o)))" spins forever.

It looks like https://bugs.python.org/issue40696 and appears to be fixed by https://github.com/python/cpython/pull/20287.

That PR doesn't apply cleanly to the package. 3.9b2 appears to be scheduled to come out today:

https://www.python.org/dev/peps/pep-0596/#id5

so probably the simplest thing to do here is simply wait for b2 to be released and the package updated, then try the stompest build again.

Thanks for the investigation. We might delay actual update to b2 in Fedora due to the infra outage, but we'll test it in Copr.

Python 3.9.0b2 indeed fixes this: https://copr.fedorainfracloud.org/coprs/g/python/python3.9/package/python-stompest/

Cool! Someone said we shouldn't ship b2, though, because it has a big brown paper bag bug in it:

https://twitter.com/llanga/status/1270305329453170688
https://bugs.python.org/issue40924

Thanks, we are aware.

The b3 PR is at https://src.fedoraproject.org/rpms/python3.9/pull-request/12 (though I shall not post more comments there during the outage).

The copr build runs in https://copr.fedorainfracloud.org/coprs/g/python/python3.9/package/python3.9/