Bug 916725

This happens because of a bug in python-fedora.

Is the error coming because I am behind a proxy server? I've tried setting the http_proxy variable but to no avail!(In reply to comment #1)
> This happens because of a bug in python-fedora.

What version of python-fedora do you have installed? Can you try it a few times to see if it's an intermittent issue?

My python-fedora package version is : python-fedora-0.3.32.3-1.fc17.noarch. I've tried it several times now and I still get the same error!(In reply to comment #3)
> What version of python-fedora do you have installed? Can you try it a few
> times to see if it's an intermittent issue?

Can you try running fedora-easy-karma with the --debug argument and paste the output?

[root@Linux Wani]# fedora-easy-karma --debug
DEBUG: fedora_cert_error - timedelta: 0:00:00.000064
DEBUG: fas_username NameError - timedelta: 0:00:00.000063
Getting list of installed packages...
DEBUG: starting yum query - timedelta: 0:00:00.022471
Getting list of packages in updates-testing...
DEBUG: starting bodhi query - timedelta: 0:00:00.883169
Traceback (most recent call last):
  File "/usr/bin/fedora-easy-karma", line 510, in <module>
    FedoraEasyKarma()
  File "/usr/bin/fedora-easy-karma", line 310, in __init__
    testing_updates = bc.query(release=release, status="testing", limit=1000)["updates"]
  File "/usr/lib/python2.7/site-packages/fedora/client/bodhi.py", line 147, in query
    return self.send_request('list', req_params=params, auth=auth)
  File "/usr/lib/python2.7/site-packages/fedora/client/baseclient.py", line 344, in send_request
    auth_params=auth_params, retries=retries)
  File "/usr/lib/python2.7/site-packages/fedora/client/proxyclient.py", line 394, in send_request
    if 'exc' in data:
TypeError: argument of type 'NoneType' is not iterable
[root@Linux Wani]#

Ralph, how does python-requests handle proxies?  Is it with the http_proxy environment variable?

According to this, http://docs.python-requests.org/en/latest/user/advanced/#proxies   http_proxy is honored.

Is your proxy using authentication?

What's the format of your http_proxy environment variable?

Okay -- and I found more thing I need to ask you about:  Do you have https_proxy set?

The servers that fedora-easy-karma is trying to access are https, not http.  For those, python-requests is looking for the value of the https_proxy environment variable.

My proxy server doesn't require authentication. Even if I set the https_proxy env variable, I get the same error.
[Wani@Linux ~]$ export https_proxy="10.4.3.204:8080"
[Wani@Linux ~]$ export http_proxy="10.4.3.204:8080"
[Wani@Linux ~]$ fedora-easy-karma 
Getting list of installed packages...
Getting list of packages in updates-testing...
Traceback (most recent call last):
  File "/usr/bin/fedora-easy-karma", line 510, in <module>
    FedoraEasyKarma()
  File "/usr/bin/fedora-easy-karma", line 310, in __init__
    testing_updates = bc.query(release=release, status="testing", limit=1000)["updates"]
  File "/usr/lib/python2.7/site-packages/fedora/client/bodhi.py", line 147, in query
    return self.send_request('list', req_params=params, auth=auth)
  File "/usr/lib/python2.7/site-packages/fedora/client/baseclient.py", line 344, in send_request
    auth_params=auth_params, retries=retries)
  File "/usr/lib/python2.7/site-packages/fedora/client/proxyclient.py", line 394, in send_request
    if 'exc' in data:
TypeError: argument of type 'NoneType' is not iterable
[Wani@Linux ~]$ 
(In reply to comment #10)
> Okay -- and I found more thing I need to ask you about:  Do you have
> https_proxy set?
> 
> The servers that fedora-easy-karma is trying to access are https, not http. 
> For those, python-requests is looking for the value of the https_proxy
> environment variable.

(In reply to comment #11)
> My proxy server doesn't require authentication. Even if I set the
> https_proxy env variable, I get the same error.
> [Wani@Linux ~]$ export https_proxy="10.4.3.204:8080"

Probably it needs to be HTTPS_PROXY in all caps to work (see for example the curl man page).

[Wani@Linux ~]$ export http_proxy="http://10.4.3.204:8080"
[Wani@Linux ~]$ export https_proxy="https://10.4.3.204:8080"
[Wani@Linux ~]$ export HTTP_PROXY="http://10.4.3.204:8080"
[Wani@Linux ~]$ export HTTPS_PROXY="https://10.4.3.204:8080"
[Wani@Linux ~]$ fedora-easy-karma 
Getting list of installed packages...
Getting list of packages in updates-testing...
Traceback (most recent call last):
  File "/usr/bin/fedora-easy-karma", line 510, in <module>
    FedoraEasyKarma()
  File "/usr/bin/fedora-easy-karma", line 310, in __init__
    testing_updates = bc.query(release=release, status="testing", limit=1000)["updates"]
  File "/usr/lib/python2.7/site-packages/fedora/client/bodhi.py", line 147, in query
    return self.send_request('list', req_params=params, auth=auth)
  File "/usr/lib/python2.7/site-packages/fedora/client/baseclient.py", line 344, in send_request
    auth_params=auth_params, retries=retries)
  File "/usr/lib/python2.7/site-packages/fedora/client/proxyclient.py", line 351, in send_request
    verify=not self.insecure,
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 98, in post
    return request('post', url, data=data, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/safe_mode.py", line 39, in wrapped
    return function(method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 51, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 241, in request
    r.send(prefetch=prefetch)
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 639, in send
    raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
[Wani@Linux ~]$

(In reply to comment #12)
> (In reply to comment #11)
> > My proxy server doesn't require authentication. Even if I set the
> > https_proxy env variable, I get the same error.
> > [Wani@Linux ~]$ export https_proxy="10.4.3.204:8080"
> 
> Probably it needs to be HTTPS_PROXY in all caps to work (see for example the
> curl man page).

Bug is in python-requests.

I setup a squid proxy.

Set https_proxy=192.168.3.13:3128

Verified that the proxy was working with curl (pycurl [using old version of python-fedora] also worked):  curl https://admin.fedoraproject.org/accounts/

Then tried this test case in a python shell:

import requests
from getpass import getpass

username = 'toshio'
password = getpass()
headers = {'Accept': 'application/json', 'User agent': 'testing client'}
params = {'user_name': username, 'password': password, 'login': 'Login'}
response = requests.post(url, data=params, headers=headers, verify=True)

response.json => None
response.text => html of the web page

If I instead do:
response = requests.post(url+'?tg_format=json', data=complete_params, headers=headers, verify=True)

response.json => expected values

So the bug seems to be when python-requests uses an https proxy set via the https_proxy environment variable, it does not send the value of the headers parameter on to the server.

One enhancement for python-fedora: We should change the code to throw an error about being unable to decode json in this situation.  I don't see a way for response.json to do that in the current version (request-1.1.0 documents: "In case the JSON decoding fails, r.json raises an exception." so that might work out of the box with an upgraded python-requests [although, we need to know if what exception is thrown so we can catch and rethrow it properly])  In the current version we are probably best off checking

if response.json is None:
   raise ServerError('msg about being unable to decode json')

forgot one piece of the test case:

url = 'https://admin.fedoraproject.org/accounts/home'

In the bundled urllib3 poolmanager.py:ProxyManager()

    def _set_proxy_headers(self, headers=None):
        headers = headers or {}

        # Same headers are curl passes for --proxy1.0
        headers['Accept'] = '*/*'
        headers['Proxy-Connection'] = 'Keep-Alive'

        return headers


so... it's overwriting our Accept header when using a proxy.

And... here's the fix: https://github.com/shazow/urllib3/pull/93

That needs to be applied to python-urllib3 (the 1.5 release doesn't have it)  and if we haven't unbundled python-urllib3 from python-requests, it needs to be applied to the bundled python-urllib3 as well.

@threebean, let me know if you need any further help.

Cool.  I created updates for the unbundling yesterday in python-requests-1.1.0-3

https://bugzilla.redhat.com/show_bug.cgi?id=904623

Patch to python-urllib3 on its way!  Thanks for figuring this out, Toshio.

python-urllib3-1.5-4.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/python-urllib3-1.5-4.fc18

python-urllib3-1.5-4.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/python-urllib3-1.5-4.fc17

python-urllib3-1.5-4.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/python-urllib3-1.5-4.el6

*** Bug 921163 has been marked as a duplicate of this bug. ***

Closing.  Should be fixed with python-urllib3-1.5-5 and python-requests-1.1.0-3

https://admin.fedoraproject.org/updates/python-urllib3
https://admin.fedoraproject.org/updates/python-requests

python-urllib3-1.5-6.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/python-urllib3-1.5-6.fc18

python-urllib3-1.5-6.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/python-urllib3-1.5-6.fc19

python-urllib3-1.5-6.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/python-urllib3-1.5-6.fc17

python-urllib3-1.5-6.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

python-urllib3-1.5-6.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

python-urllib3-1.5-6.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.

python-urllib3-1.5-7.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/python-urllib3-1.5-7.el6

Package python-urllib3-1.5-7.el6:
* should fix your issue,
* was pushed to the Fedora EPEL 6 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=epel-testing python-urllib3-1.5-7.el6'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-EPEL-2013-11982/python-urllib3-1.5-7.el6
then log in and leave karma (feedback).

python-urllib3-1.5-7.el6 has been pushed to the Fedora EPEL 6 stable repository.  If problems still persist, please make note of it in this bug report.