Bug 1504089 - easy-karma or bodhi broken: too many 500 error responses
Summary: easy-karma or bodhi broken: too many 500 error responses
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: fedora-easy-karma
Version: 26
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Till Maas
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-19 13:37 UTC by Harald Reindl
Modified: 2018-01-11 19:51 UTC (History)
6 users (show)

Fixed In Version: fedora-easy-karma-0-0.34.20171019gitbc16e916.fc26 fedora-easy-karma-0-0.34.20171019gitbc16e916.fc27 fedora-easy-karma-0-0.34.20171019gitbc16e916.fc25
Clone Of:
Environment:
Last Closed: 2017-10-25 23:14:39 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Harald Reindl 2017-10-19 13:37:20 UTC
what is wrong since bodhi2 and most of the time fedora-easy-karma or the bodhi side or both are broken? do you really expect testers to give karma on the webpage while you can see all the time that people give karma for packages they don't have installed at all because you need to be very careful about the build?

closing with "won't fix" and "If you would like to have a bug to track the technical problem. please create a new bug with only the technical description" don't magically fix problems

Getting list of installed packages...
Waiting for Bodhi for a list of packages in updates-testing (F26)...
Error while querying Bodhi: HTTPSConnectionPool(host='bodhi.fedoraproject.org', port=443): Max retries exceeded with url: /updates/?status=testing&rows_per_page=1000&releases=F26 (Caused by ResponseError('too many 500 error responses',))
Traceback (most recent call last):
  File "/usr/bin/fedora-easy-karma", line 839, in <module>
    fek = FedoraEasyKarma()
  File "/usr/bin/fedora-easy-karma", line 570, in __init__
    testing_updates = self.query_bodhi(bc, release, pending=False)
  File "/usr/bin/fedora-easy-karma", line 758, in query_bodhi
    raise e
requests.exceptions.RetryError: HTTPSConnectionPool(host='bodhi.fedoraproject.org', port=443): Max retries exceeded with url: /updates/?status=testing&rows_per_page=1000&releases=F26 (Caused by ResponseError('too many 500 error responses',))

Waiting for Bodhi for a list of packages in updates-testing (F26)...
Error while querying Bodhi: HTTPSConnectionPool(host='bodhi.fedoraproject.org', port=443): Max retries exceeded with url: /updates/?status=testing&rows_per_page=1000&releases=F26 (Caused by ResponseError('too many 504 error responses',))
Traceback (most recent call last):
  File "/usr/bin/fedora-easy-karma", line 839, in <module>
    fek = FedoraEasyKarma()
  File "/usr/bin/fedora-easy-karma", line 570, in __init__
    testing_updates = self.query_bodhi(bc, release, pending=False)
  File "/usr/bin/fedora-easy-karma", line 758, in query_bodhi
    raise e
requests.exceptions.RetryError: HTTPSConnectionPool(host='bodhi.fedoraproject.org', port=443): Max retries exceeded with url: /updates/?status=testing&rows_per_page=1000&releases=F26 (Caused by ResponseError('too many 504 error responses',))

Comment 1 Kamil Páral 2017-10-19 13:56:16 UTC
CC Randy, the Bodhi maintainer. Randy, I believe that after last Bodhi update the error above starting happening. This prevents people from filing karma (it happens always, in my case). Is there anything that can be fixed on Bodhi side, or does f-e-k need to be adjusted, and how exactly do you recommend (how large queries to perform)? Thanks.

Comment 2 Till Maas 2017-10-19 18:26:57 UTC
As a workaround the limit can be changed in the script from 1000 to 100, see:
https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org/thread/NT4OE3W3PG5LAEVQGHW6XLTKBP4JIAUX/

Comment 3 Randy Barlow 2017-10-19 20:40:09 UTC
Till and I have talked about the performance issues. The Bodhi API doesn't have an endpoint that is idea for f-e-k, so f-e-k uses one of its general endpoints. Unfortunately, with this endpoint Bodhi loads and serializes all related objects for each update which actually consumes a pretty large amount of memory (about 2-4 GB per request). Bodhi only has two web servers with 16 GB of memory each, so just a few concurrent users doing this can quickly exhaust the memory.

Till's workaround will help, but won't be perfect since the data can shift between pagination requests. Most of the time that probably will be fine though.

The better long term fix is probably to do one or both of these:

0) Give Bodhi an API endpoint tailored for f-e-k that just loads the data it needs.
1) Give Bodhi's existing API a way for the requestor to ask it not to respond with related objects.

Comment 4 Fedora Update System 2017-10-19 21:46:55 UTC
fedora-easy-karma-0-0.34.20171019gitbc16e916.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-d4277c7b96

Comment 5 Fedora Update System 2017-10-19 21:47:05 UTC
fedora-easy-karma-0-0.34.20171019gitbc16e916.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-d136b70561

Comment 6 Fedora Update System 2017-10-19 21:47:26 UTC
fedora-easy-karma-0-0.34.20171019gitbc16e916.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-b1f9957279

Comment 7 Harald Reindl 2017-10-20 09:34:54 UTC
FWIW: after manually download fedora-easy-karma-0-0.34.20171019gitbc16e916.fc26.noarch and give karma for the packages from updates-testing on my machine at least for 3 packages marekd as secruity-fix my karma was the missing one for push it to stable

hopefully you guys realize the importance of fedora-easy-karma for fedora as a whole and keep it working in the future - very likely the ongoing issues leaded to lose some updates-testing users completly and then left and right "too few testers" is cried

the "look at my power" reaction to https://bugzilla.redhat.com/show_bug.cgi?id=1501210 leaded for some critical security updates a complete week hanging in updates-testing and was *not* helpful for the project at all as well as the behavior of the FF maintainer which needs policy changes to replace common sense

in other words: i slowly have enough of the whole Fedora project after a decade of playing tester

Comment 8 Fedora Update System 2017-10-21 19:27:15 UTC
fedora-easy-karma-0-0.34.20171019gitbc16e916.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-b1f9957279

Comment 9 Fedora Update System 2017-10-22 02:23:51 UTC
fedora-easy-karma-0-0.34.20171019gitbc16e916.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-d4277c7b96

Comment 10 Fedora Update System 2017-10-22 03:23:10 UTC
fedora-easy-karma-0-0.34.20171019gitbc16e916.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-d136b70561

Comment 11 Fedora Update System 2017-10-25 23:14:39 UTC
fedora-easy-karma-0-0.34.20171019gitbc16e916.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 12 Harald Reindl 2017-10-31 14:17:39 UTC
and it's back again

[root@testserver:~]$ rpm -q fedora-easy-karma
fedora-easy-karma-0-0.34.20171019gitbc16e916.fc26.noarch

Getting list of installed packages...
Waiting for Bodhi for a list of packages in updates-testing (F26)...
Fetching updates page 2 of 5
Error while querying Bodhi: HTTPSConnectionPool(host='bodhi.fedoraproject.org', port=443): Max retries exceeded with url: /updates/?status=testing&rows_per_page=100&releases=F26&page=2 (Caused by ResponseError('too many 504 error responses',))
Traceback (most recent call last):
  File "/usr/bin/fedora-easy-karma", line 848, in <module>
    fek = FedoraEasyKarma()
  File "/usr/bin/fedora-easy-karma", line 579, in __init__
    testing_updates = self.query_bodhi(bc, release, pending=False)
  File "/usr/bin/fedora-easy-karma", line 767, in query_bodhi
    raise e
requests.exceptions.RetryError: HTTPSConnectionPool(host='bodhi.fedoraproject.org', port=443): Max retries exceeded with url: /updates/?status=testing&rows_per_page=100&releases=F26&page=2 (Caused by ResponseError('too many 504 error responses',))

Comment 13 Randy Barlow 2017-11-01 03:17:14 UTC
Hello Till,

I wonder if we should try a smaller default for --bodhi-request-limit, like 16 or 32?

Comment 14 Till Maas 2017-11-03 06:02:22 UTC
(In reply to Randy Barlow from comment #13)
> Hello Till,
> 
> I wonder if we should try a smaller default for --bodhi-request-limit, like
> 16 or 32?

This is very small, but if it is required I can do this. The downside is that many small requests seem to be even slower than a few big ones (at least when I tested it a while ago).

Comment 15 Fedora Update System 2017-11-11 02:58:49 UTC
fedora-easy-karma-0-0.34.20171019gitbc16e916.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Comment 16 Graham White 2017-11-14 14:48:29 UTC
Not fixed on F25...

$ rpm -q fedora-easy-karma
fedora-easy-karma-0-0.34.20171019gitbc16e916.fc25.noarch

$ fedora-easy-karma 
Getting list of installed packages...
Waiting for Bodhi for a list of packages in updates-testing (F25)...
Fetching updates page 2 of 4
Error while querying Bodhi: HTTPSConnectionPool(host='bodhi.fedoraproject.org', port=443): Max retries exceeded with url: /updates/?status=testing&rows_per_page=100&releases=F25&page=2 (Caused by ResponseError('too many 504 error responses',))
Traceback (most recent call last):
  File "/usr/bin/fedora-easy-karma", line 848, in <module>
    fek = FedoraEasyKarma()
  File "/usr/bin/fedora-easy-karma", line 579, in __init__
    testing_updates = self.query_bodhi(bc, release, pending=False)
  File "/usr/bin/fedora-easy-karma", line 767, in query_bodhi
    raise e
requests.exceptions.RetryError: HTTPSConnectionPool(host='bodhi.fedoraproject.org', port=443): Max retries exceeded with url: /updates/?status=testing&rows_per_page=100&releases=F25&page=2 (Caused by ResponseError('too many 504 error responses',))

Comment 17 Harald Reindl 2017-11-14 14:59:33 UTC
@Graham White: not fixed at all, just workarounds which works sometimes or not

"Unfortunately, with this endpoint Bodhi loads and serializes all related objects for each update which actually consumes a pretty large amount of memory (about 2-4 GB per request)" is just broken by design

Comment 18 Graham White 2017-11-14 15:13:14 UTC
OK, thanks.  I've given up and provided karma on the web interface (which is ridiculously slow but does work).

Comment 19 Harald Reindl 2017-11-14 15:18:13 UTC
the website is not a good option - just look how often people give karma for the wrong build, even for the wrong fedora version and in case negative karma that holds back updates and probably security relevant ones or fixing exactly the bugs the person pretended exists

frankly this needs to get higher priority and should had as bodhi was rewritten given that it worked for all that years in the past

Comment 20 Fedora Update System 2017-12-09 04:56:05 UTC
fedora-easy-karma-0-0.34.20171019gitbc16e916.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 21 Harald Reindl 2018-01-11 19:51:34 UTC
wow - now we have that nonsense for submit the "works for me" over the last hours

 inst. RPMS: kernel-core-4.14.13-200.fc26.x86_64 - The Linux kernel (installed 0 days ago)
           : kernel-devel-4.14.13-200.fc26.x86_64 - Development package for building kernel modules to match the kernel (installed 0 days ago)
           : kernel-headers-4.14.13-200.fc26.x86_64 - Header files for the Linux kernel for use by glibc (installed 0 days ago)
           : kernel-modules-4.14.13-200.fc26.x86_64 - kernel modules to match the core kernel (installed 0 days ago)

Comment? -1/0/1 -> karma, 'i' -> ignore, other -> skip> 1
Comment> works for me
Password:
Traceback (most recent call last):
  File "/usr/bin/fedora-easy-karma", line 848, in <module>
    fek = FedoraEasyKarma()
  File "/usr/bin/fedora-easy-karma", line 708, in __init__
    karma)
  File "/usr/bin/fedora-easy-karma", line 832, in send_comment
    res = bc.comment(update["title"], comment, karma=karma)
  File "/usr/lib/python2.7/site-packages/bodhi/client/bindings.py", line 75, in wrapper
    result = method(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/bodhi/client/bindings.py", line 309, in comment
    'csrf_token': self.csrf()})
  File "/usr/lib/python2.7/site-packages/fedora/client/openidbaseclient.py", line 244, in send_request
    output = func(method, **kwargs)
  File "/usr/lib/python2.7/site-packages/fedora/client/openidbaseclient.py", line 82, in _decorator
    output = func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/fedora/client/openidbaseclient.py", line 199, in _authed_post
    response = self._session.post(url, params=params, data=data, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 535, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 482, in send
    raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool(host='bodhi.fedoraproject.org', port=443): Max retries exceeded with url: /comments/ (Caused by ResponseError('too many 504 error responses',))


Note You need to log in before you can comment on or make changes to this bug.