Bug 1504089

CC Randy, the Bodhi maintainer. Randy, I believe that after last Bodhi update the error above starting happening. This prevents people from filing karma (it happens always, in my case). Is there anything that can be fixed on Bodhi side, or does f-e-k need to be adjusted, and how exactly do you recommend (how large queries to perform)? Thanks.

As a workaround the limit can be changed in the script from 1000 to 100, see:
https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/thread/NT4OE3W3PG5LAEVQGHW6XLTKBP4JIAUX/

Till and I have talked about the performance issues. The Bodhi API doesn't have an endpoint that is idea for f-e-k, so f-e-k uses one of its general endpoints. Unfortunately, with this endpoint Bodhi loads and serializes all related objects for each update which actually consumes a pretty large amount of memory (about 2-4 GB per request). Bodhi only has two web servers with 16 GB of memory each, so just a few concurrent users doing this can quickly exhaust the memory.

Till's workaround will help, but won't be perfect since the data can shift between pagination requests. Most of the time that probably will be fine though.

The better long term fix is probably to do one or both of these:

0) Give Bodhi an API endpoint tailored for f-e-k that just loads the data it needs.
1) Give Bodhi's existing API a way for the requestor to ask it not to respond with related objects.

fedora-easy-karma-0-0.34.20171019gitbc16e916.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-d4277c7b96

fedora-easy-karma-0-0.34.20171019gitbc16e916.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-d136b70561

fedora-easy-karma-0-0.34.20171019gitbc16e916.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-b1f9957279

FWIW: after manually download fedora-easy-karma-0-0.34.20171019gitbc16e916.fc26.noarch and give karma for the packages from updates-testing on my machine at least for 3 packages marekd as secruity-fix my karma was the missing one for push it to stable

hopefully you guys realize the importance of fedora-easy-karma for fedora as a whole and keep it working in the future - very likely the ongoing issues leaded to lose some updates-testing users completly and then left and right "too few testers" is cried

the "look at my power" reaction to https://bugzilla.redhat.com/show_bug.cgi?id=1501210 leaded for some critical security updates a complete week hanging in updates-testing and was *not* helpful for the project at all as well as the behavior of the FF maintainer which needs policy changes to replace common sense

in other words: i slowly have enough of the whole Fedora project after a decade of playing tester

fedora-easy-karma-0-0.34.20171019gitbc16e916.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-b1f9957279

fedora-easy-karma-0-0.34.20171019gitbc16e916.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-d4277c7b96

fedora-easy-karma-0-0.34.20171019gitbc16e916.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-d136b70561

fedora-easy-karma-0-0.34.20171019gitbc16e916.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

and it's back again

[root@testserver:~]$ rpm -q fedora-easy-karma
fedora-easy-karma-0-0.34.20171019gitbc16e916.fc26.noarch

Getting list of installed packages...
Waiting for Bodhi for a list of packages in updates-testing (F26)...
Fetching updates page 2 of 5
Error while querying Bodhi: HTTPSConnectionPool(host='bodhi.fedoraproject.org', port=443): Max retries exceeded with url: /updates/?status=testing&rows_per_page=100&releases=F26&page=2 (Caused by ResponseError('too many 504 error responses',))
Traceback (most recent call last):
  File "/usr/bin/fedora-easy-karma", line 848, in <module>
    fek = FedoraEasyKarma()
  File "/usr/bin/fedora-easy-karma", line 579, in __init__
    testing_updates = self.query_bodhi(bc, release, pending=False)
  File "/usr/bin/fedora-easy-karma", line 767, in query_bodhi
    raise e
requests.exceptions.RetryError: HTTPSConnectionPool(host='bodhi.fedoraproject.org', port=443): Max retries exceeded with url: /updates/?status=testing&rows_per_page=100&releases=F26&page=2 (Caused by ResponseError('too many 504 error responses',))

Hello Till,

I wonder if we should try a smaller default for --bodhi-request-limit, like 16 or 32?

(In reply to Randy Barlow from comment #13)
> Hello Till,
> 
> I wonder if we should try a smaller default for --bodhi-request-limit, like
> 16 or 32?

This is very small, but if it is required I can do this. The downside is that many small requests seem to be even slower than a few big ones (at least when I tested it a while ago).

fedora-easy-karma-0-0.34.20171019gitbc16e916.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Not fixed on F25...

$ rpm -q fedora-easy-karma
fedora-easy-karma-0-0.34.20171019gitbc16e916.fc25.noarch

$ fedora-easy-karma 
Getting list of installed packages...
Waiting for Bodhi for a list of packages in updates-testing (F25)...
Fetching updates page 2 of 4
Error while querying Bodhi: HTTPSConnectionPool(host='bodhi.fedoraproject.org', port=443): Max retries exceeded with url: /updates/?status=testing&rows_per_page=100&releases=F25&page=2 (Caused by ResponseError('too many 504 error responses',))
Traceback (most recent call last):
  File "/usr/bin/fedora-easy-karma", line 848, in <module>
    fek = FedoraEasyKarma()
  File "/usr/bin/fedora-easy-karma", line 579, in __init__
    testing_updates = self.query_bodhi(bc, release, pending=False)
  File "/usr/bin/fedora-easy-karma", line 767, in query_bodhi
    raise e
requests.exceptions.RetryError: HTTPSConnectionPool(host='bodhi.fedoraproject.org', port=443): Max retries exceeded with url: /updates/?status=testing&rows_per_page=100&releases=F25&page=2 (Caused by ResponseError('too many 504 error responses',))

@Graham White: not fixed at all, just workarounds which works sometimes or not

"Unfortunately, with this endpoint Bodhi loads and serializes all related objects for each update which actually consumes a pretty large amount of memory (about 2-4 GB per request)" is just broken by design

OK, thanks.  I've given up and provided karma on the web interface (which is ridiculously slow but does work).

the website is not a good option - just look how often people give karma for the wrong build, even for the wrong fedora version and in case negative karma that holds back updates and probably security relevant ones or fixing exactly the bugs the person pretended exists

frankly this needs to get higher priority and should had as bodhi was rewritten given that it worked for all that years in the past

fedora-easy-karma-0-0.34.20171019gitbc16e916.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

wow - now we have that nonsense for submit the "works for me" over the last hours

 inst. RPMS: kernel-core-4.14.13-200.fc26.x86_64 - The Linux kernel (installed 0 days ago)
           : kernel-devel-4.14.13-200.fc26.x86_64 - Development package for building kernel modules to match the kernel (installed 0 days ago)
           : kernel-headers-4.14.13-200.fc26.x86_64 - Header files for the Linux kernel for use by glibc (installed 0 days ago)
           : kernel-modules-4.14.13-200.fc26.x86_64 - kernel modules to match the core kernel (installed 0 days ago)

Comment? -1/0/1 -> karma, 'i' -> ignore, other -> skip> 1
Comment> works for me
Password:
Traceback (most recent call last):
  File "/usr/bin/fedora-easy-karma", line 848, in <module>
    fek = FedoraEasyKarma()
  File "/usr/bin/fedora-easy-karma", line 708, in __init__
    karma)
  File "/usr/bin/fedora-easy-karma", line 832, in send_comment
    res = bc.comment(update["title"], comment, karma=karma)
  File "/usr/lib/python2.7/site-packages/bodhi/client/bindings.py", line 75, in wrapper
    result = method(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/bodhi/client/bindings.py", line 309, in comment
    'csrf_token': self.csrf()})
  File "/usr/lib/python2.7/site-packages/fedora/client/openidbaseclient.py", line 244, in send_request
    output = func(method, **kwargs)
  File "/usr/lib/python2.7/site-packages/fedora/client/openidbaseclient.py", line 82, in _decorator
    output = func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/fedora/client/openidbaseclient.py", line 199, in _authed_post
    response = self._session.post(url, params=params, data=data, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 535, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 482, in send
    raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool(host='bodhi.fedoraproject.org', port=443): Max retries exceeded with url: /comments/ (Caused by ResponseError('too many 504 error responses',))