Hide Forgot
At the moment we're using a thread pool to perform requests against the JSON APIs. Instead we should investigate if we can get an asyncio aware requests (the library) wrapper to avoid doing this, since it is expensive and doesn't scale very well.
There is an asyncio-aware fork of requests here: https://github.com/rdbhost/yieldfromrequests but it doesn't seem to have been keeping up to date with upstream requests so I would wary of using it. This seems to be the most popular solution for doing HTTP in asyncio: https://github.com/KeepSafe/aiohttp
(In reply to Dan Callaghan from comment #1) > This seems to be the most popular solution for doing HTTP in asyncio: > > https://github.com/KeepSafe/aiohttp ... except it doesn't work on Trollius.
So I think we are at a bit of an impasse here. We are stuck using AMQP-0-9 for now (until Errata Tool starts publishing on the new "Unified Message Bus") which means we are stuck using old qpid-python, which means we are stuck on Python 2 and Trollius. So we can't use aiohttp. There do not appear to be any HTTP client libraries supporting Trollius at all (that is cited as one of the reasons for its deprecation).
I don't think we need to be stuck with Python 2. The current messaging situation is temporary and changing soon, so we should design with that in mind. It is easy enough to write a bridging service in python 2 that listens to an AMQP 0-9 broker and passes along to a python 3 service using proton. I've already written code like this before; I'll post a link as soon as I find it. This changes the design from: * Write in python 2 and use workarounds for the lack of asyncio because a small part temporarily needs qpid 0-9 bits to: * Write things properly using Python 3, asyncio and other needed concurrency mechanisms. Write a small throwaway service to handle the qpid/python 2 bits, which will be eliminated by the time serious production use is needed.
Jon, I think you're right. Continuing to use Trollius (instead of asyncio in Python 3) is going to just cost us more and more in terms of missing features and libraries. Errata Tool is not expected to publish on the new message bus until some time reasonably far in the future. But a small shim service on the side to relay messages from the qpid broker to a local Proton connection seems like a viable approach -- better than being stuck with the old qpid Python bindings.
Yeah I believe we have run into a few issues with trollius in general across the team, so I agree with the moving it to a shim service.
So the next problem with switching to Python 3 is the koji client library. It looks like koji will not have Python 3 support anytime in the near future, due to the requirement for the koji codebase to run back to RHEL5 (Python 2.4) for Itanium. https://bugzilla.redhat.com/show_bug.cgi?id=1024827 https://pagure.io/koji/issue/151 https://pagure.io/koji/pull-request/152
This patch switches to using Proton instead of qpid.messaging: https://gerrit.beaker-project.org/5474 (although the bridge from the old message bus to the new, as suggested by Jon in comment 4, still needs to be written.) This patch removes the dependency on Koji: https://gerrit.beaker-project.org/5485
https://gerrit.beaker-project.org/5523 use aiohttp instead of requests for downloading packages https://gerrit.beaker-project.org/5524 use aiohttp instead of requests for advisory API requests Only missing piece is the AMQP-0-9-1 -> AMQP-1.0 bridge now.
(In reply to Dan Callaghan from comment #10) > Only missing piece is the AMQP-0-9-1 -> AMQP-1.0 bridge now. https://gerrit.beaker-project.org/5529 Based on Jon's script from comment 4 (thanks Jon!).