Bug 916725 - running fedora-easy-karma returns a python error
Summary: running fedora-easy-karma returns a python error
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: python-requests
Version: 17
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Ralph Bean
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 921163 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-02-28 19:14 UTC by Nehal J Wani
Modified: 2013-11-15 18:57 UTC (History)
8 users (show)

Fixed In Version: python-urllib3-1.5-7.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-11-15 18:57:35 UTC


Attachments (Terms of Use)

Description Nehal J Wani 2013-02-28 19:14:22 UTC
Description of problem:
[root@Linux Wani]# fedora-easy-karma 
Getting list of installed packages...
Getting list of packages in updates-testing...
Traceback (most recent call last):
  File "/usr/bin/fedora-easy-karma", line 510, in <module>
    FedoraEasyKarma()
  File "/usr/bin/fedora-easy-karma", line 310, in __init__
    testing_updates = bc.query(release=release, status="testing", limit=1000)["updates"]
  File "/usr/lib/python2.7/site-packages/fedora/client/bodhi.py", line 147, in query
    return self.send_request('list', req_params=params, auth=auth)
  File "/usr/lib/python2.7/site-packages/fedora/client/baseclient.py", line 344, in send_request
    auth_params=auth_params, retries=retries)
  File "/usr/lib/python2.7/site-packages/fedora/client/proxyclient.py", line 394, in send_request
    if 'exc' in data:
TypeError: argument of type 'NoneType' is not iterable
[root@Linux Wani]# 


Version-Release number of selected component (if applicable):
  fedora-easy-karma.noarch 0:0-0.16.20110825git36efb338.fc17

Comment 1 Till Maas 2013-02-28 19:28:55 UTC
This happens because of a bug in python-fedora.

Comment 2 Nehal J Wani 2013-02-28 19:32:42 UTC
Is the error coming because I am behind a proxy server? I've tried setting the http_proxy variable but to no avail!(In reply to comment #1)
> This happens because of a bug in python-fedora.

Comment 3 Luke Macken 2013-02-28 19:45:15 UTC
What version of python-fedora do you have installed? Can you try it a few times to see if it's an intermittent issue?

Comment 4 Nehal J Wani 2013-02-28 19:50:17 UTC
My python-fedora package version is : python-fedora-0.3.32.3-1.fc17.noarch. I've tried it several times now and I still get the same error!(In reply to comment #3)
> What version of python-fedora do you have installed? Can you try it a few
> times to see if it's an intermittent issue?

Comment 5 Luke Macken 2013-02-28 20:08:11 UTC
Can you try running fedora-easy-karma with the --debug argument and paste the output?

Comment 6 Nehal J Wani 2013-02-28 20:14:42 UTC
[root@Linux Wani]# fedora-easy-karma --debug
DEBUG: fedora_cert_error - timedelta: 0:00:00.000064
DEBUG: fas_username NameError - timedelta: 0:00:00.000063
Getting list of installed packages...
DEBUG: starting yum query - timedelta: 0:00:00.022471
Getting list of packages in updates-testing...
DEBUG: starting bodhi query - timedelta: 0:00:00.883169
Traceback (most recent call last):
  File "/usr/bin/fedora-easy-karma", line 510, in <module>
    FedoraEasyKarma()
  File "/usr/bin/fedora-easy-karma", line 310, in __init__
    testing_updates = bc.query(release=release, status="testing", limit=1000)["updates"]
  File "/usr/lib/python2.7/site-packages/fedora/client/bodhi.py", line 147, in query
    return self.send_request('list', req_params=params, auth=auth)
  File "/usr/lib/python2.7/site-packages/fedora/client/baseclient.py", line 344, in send_request
    auth_params=auth_params, retries=retries)
  File "/usr/lib/python2.7/site-packages/fedora/client/proxyclient.py", line 394, in send_request
    if 'exc' in data:
TypeError: argument of type 'NoneType' is not iterable
[root@Linux Wani]#

Comment 7 Toshio Ernie Kuratomi 2013-03-01 00:51:34 UTC
Ralph, how does python-requests handle proxies?  Is it with the http_proxy environment variable?

Comment 8 Toshio Ernie Kuratomi 2013-03-01 01:09:10 UTC
According to this, http://docs.python-requests.org/en/latest/user/advanced/#proxies   http_proxy is honored.

Is your proxy using authentication?

Comment 9 Toshio Ernie Kuratomi 2013-03-01 01:09:52 UTC
What's the format of your http_proxy environment variable?

Comment 10 Toshio Ernie Kuratomi 2013-03-01 01:14:50 UTC
Okay -- and I found more thing I need to ask you about:  Do you have https_proxy set?

The servers that fedora-easy-karma is trying to access are https, not http.  For those, python-requests is looking for the value of the https_proxy environment variable.

Comment 11 Nehal J Wani 2013-03-01 05:05:32 UTC
My proxy server doesn't require authentication. Even if I set the https_proxy env variable, I get the same error.
[Wani@Linux ~]$ export https_proxy="10.4.3.204:8080"
[Wani@Linux ~]$ export http_proxy="10.4.3.204:8080"
[Wani@Linux ~]$ fedora-easy-karma 
Getting list of installed packages...
Getting list of packages in updates-testing...
Traceback (most recent call last):
  File "/usr/bin/fedora-easy-karma", line 510, in <module>
    FedoraEasyKarma()
  File "/usr/bin/fedora-easy-karma", line 310, in __init__
    testing_updates = bc.query(release=release, status="testing", limit=1000)["updates"]
  File "/usr/lib/python2.7/site-packages/fedora/client/bodhi.py", line 147, in query
    return self.send_request('list', req_params=params, auth=auth)
  File "/usr/lib/python2.7/site-packages/fedora/client/baseclient.py", line 344, in send_request
    auth_params=auth_params, retries=retries)
  File "/usr/lib/python2.7/site-packages/fedora/client/proxyclient.py", line 394, in send_request
    if 'exc' in data:
TypeError: argument of type 'NoneType' is not iterable
[Wani@Linux ~]$ 
(In reply to comment #10)
> Okay -- and I found more thing I need to ask you about:  Do you have
> https_proxy set?
> 
> The servers that fedora-easy-karma is trying to access are https, not http. 
> For those, python-requests is looking for the value of the https_proxy
> environment variable.

Comment 12 Till Maas 2013-03-01 08:01:26 UTC
(In reply to comment #11)
> My proxy server doesn't require authentication. Even if I set the
> https_proxy env variable, I get the same error.
> [Wani@Linux ~]$ export https_proxy="10.4.3.204:8080"

Probably it needs to be HTTPS_PROXY in all caps to work (see for example the curl man page).

Comment 13 Nehal J Wani 2013-03-01 08:26:24 UTC
[Wani@Linux ~]$ export http_proxy="http://10.4.3.204:8080"
[Wani@Linux ~]$ export https_proxy="https://10.4.3.204:8080"
[Wani@Linux ~]$ export HTTP_PROXY="http://10.4.3.204:8080"
[Wani@Linux ~]$ export HTTPS_PROXY="https://10.4.3.204:8080"
[Wani@Linux ~]$ fedora-easy-karma 
Getting list of installed packages...
Getting list of packages in updates-testing...
Traceback (most recent call last):
  File "/usr/bin/fedora-easy-karma", line 510, in <module>
    FedoraEasyKarma()
  File "/usr/bin/fedora-easy-karma", line 310, in __init__
    testing_updates = bc.query(release=release, status="testing", limit=1000)["updates"]
  File "/usr/lib/python2.7/site-packages/fedora/client/bodhi.py", line 147, in query
    return self.send_request('list', req_params=params, auth=auth)
  File "/usr/lib/python2.7/site-packages/fedora/client/baseclient.py", line 344, in send_request
    auth_params=auth_params, retries=retries)
  File "/usr/lib/python2.7/site-packages/fedora/client/proxyclient.py", line 351, in send_request
    verify=not self.insecure,
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 98, in post
    return request('post', url, data=data, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/safe_mode.py", line 39, in wrapped
    return function(method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 51, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 241, in request
    r.send(prefetch=prefetch)
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 639, in send
    raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
[Wani@Linux ~]$

(In reply to comment #12)
> (In reply to comment #11)
> > My proxy server doesn't require authentication. Even if I set the
> > https_proxy env variable, I get the same error.
> > [Wani@Linux ~]$ export https_proxy="10.4.3.204:8080"
> 
> Probably it needs to be HTTPS_PROXY in all caps to work (see for example the
> curl man page).

Comment 14 Toshio Ernie Kuratomi 2013-03-01 08:36:47 UTC
Bug is in python-requests.

I setup a squid proxy.

Set https_proxy=192.168.3.13:3128

Verified that the proxy was working with curl (pycurl [using old version of python-fedora] also worked):  curl https://admin.fedoraproject.org/accounts/

Then tried this test case in a python shell:

import requests
from getpass import getpass

username = 'toshio'
password = getpass()
headers = {'Accept': 'application/json', 'User agent': 'testing client'}
params = {'user_name': username, 'password': password, 'login': 'Login'}
response = requests.post(url, data=params, headers=headers, verify=True)

response.json => None
response.text => html of the web page

If I instead do:
response = requests.post(url+'?tg_format=json', data=complete_params, headers=headers, verify=True)

response.json => expected values

So the bug seems to be when python-requests uses an https proxy set via the https_proxy environment variable, it does not send the value of the headers parameter on to the server.

One enhancement for python-fedora: We should change the code to throw an error about being unable to decode json in this situation.  I don't see a way for response.json to do that in the current version (request-1.1.0 documents: "In case the JSON decoding fails, r.json raises an exception." so that might work out of the box with an upgraded python-requests [although, we need to know if what exception is thrown so we can catch and rethrow it properly])  In the current version we are probably best off checking

if response.json is None:
   raise ServerError('msg about being unable to decode json')

Comment 15 Toshio Ernie Kuratomi 2013-03-01 09:20:29 UTC
forgot one piece of the test case:

url = 'https://admin.fedoraproject.org/accounts/home'

Comment 16 Toshio Ernie Kuratomi 2013-03-01 10:13:33 UTC
In the bundled urllib3 poolmanager.py:ProxyManager()

    def _set_proxy_headers(self, headers=None):
        headers = headers or {}

        # Same headers are curl passes for --proxy1.0
        headers['Accept'] = '*/*'
        headers['Proxy-Connection'] = 'Keep-Alive'

        return headers


so... it's overwriting our Accept header when using a proxy.

Comment 17 Toshio Ernie Kuratomi 2013-03-01 10:30:21 UTC
And... here's the fix: https://github.com/shazow/urllib3/pull/93

That needs to be applied to python-urllib3 (the 1.5 release doesn't have it)  and if we haven't unbundled python-urllib3 from python-requests, it needs to be applied to the bundled python-urllib3 as well.

@threebean, let me know if you need any further help.

Comment 18 Ralph Bean 2013-03-01 15:16:55 UTC
Cool.  I created updates for the unbundling yesterday in python-requests-1.1.0-3

https://bugzilla.redhat.com/show_bug.cgi?id=904623

Patch to python-urllib3 on its way!  Thanks for figuring this out, Toshio.

Comment 19 Fedora Update System 2013-03-01 16:13:03 UTC
python-urllib3-1.5-4.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/python-urllib3-1.5-4.fc18

Comment 20 Fedora Update System 2013-03-01 16:13:48 UTC
python-urllib3-1.5-4.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/python-urllib3-1.5-4.fc17

Comment 21 Fedora Update System 2013-03-01 16:14:20 UTC
python-urllib3-1.5-4.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/python-urllib3-1.5-4.el6

Comment 22 Richard W.M. Jones 2013-03-13 16:09:50 UTC
*** Bug 921163 has been marked as a duplicate of this bug. ***

Comment 23 Ralph Bean 2013-04-09 03:51:15 UTC
Closing.  Should be fixed with python-urllib3-1.5-5 and python-requests-1.1.0-3

https://admin.fedoraproject.org/updates/python-urllib3
https://admin.fedoraproject.org/updates/python-requests

Comment 24 Fedora Update System 2013-06-11 17:00:17 UTC
python-urllib3-1.5-6.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/python-urllib3-1.5-6.fc18

Comment 25 Fedora Update System 2013-06-11 17:00:50 UTC
python-urllib3-1.5-6.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/python-urllib3-1.5-6.fc19

Comment 26 Fedora Update System 2013-06-11 17:01:33 UTC
python-urllib3-1.5-6.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/python-urllib3-1.5-6.fc17

Comment 27 Fedora Update System 2013-06-18 06:07:16 UTC
python-urllib3-1.5-6.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 28 Fedora Update System 2013-06-25 03:25:28 UTC
python-urllib3-1.5-6.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 29 Fedora Update System 2013-06-25 03:26:14 UTC
python-urllib3-1.5-6.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 30 Fedora Update System 2013-10-28 17:45:15 UTC
python-urllib3-1.5-7.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/python-urllib3-1.5-7.el6

Comment 31 Fedora Update System 2013-10-29 18:05:18 UTC
Package python-urllib3-1.5-7.el6:
* should fix your issue,
* was pushed to the Fedora EPEL 6 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=epel-testing python-urllib3-1.5-7.el6'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-EPEL-2013-11982/python-urllib3-1.5-7.el6
then log in and leave karma (feedback).

Comment 32 Fedora Update System 2013-11-15 18:57:35 UTC
python-urllib3-1.5-7.el6 has been pushed to the Fedora EPEL 6 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.