Bug 825289 - [abrt] fedora-easy-karma-0-0.16.20110825git36efb338.fc17: proxyclient.py:400:send_request:error: (18, 'transfer closed with 656621 bytes remaining to read')
[abrt] fedora-easy-karma-0-0.16.20110825git36efb338.fc17: proxyclient.py:400:...
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: fedora-easy-karma (Show other bugs)
17
x86_64 Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Till Maas
Fedora Extras Quality Assurance
abrt_hash:b07aad186898d808e7e6b4ff2ab...
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-05-25 11:03 EDT by Alen Siljak
Modified: 2013-07-24 20:46 EDT (History)
5 users (show)

See Also:
Fixed In Version: fedora-easy-karma-0-0.19.20130707git121694f6.fc18
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-07-11 22:56:27 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Alen Siljak 2012-05-25 11:03:28 EDT
libreport version: 2.0.10
abrt_version:   2.0.10
cmdline:        /usr/bin/python /usr/bin/fedora-easy-karma
event_log:      2012-05-25-17:02:50> Smolt profile successfully saved
executable:     /usr/bin/fedora-easy-karma
kernel:         3.3.4-4.fc17.x86_64
time:           Wed 09 May 2012 16:47:30 CEST
uid:            1000
username:       alen

backtrace:
:proxyclient.py:400:send_request:error: (18, 'transfer closed with 656621 bytes remaining to read')
:
:Traceback (most recent call last):
:  File "/usr/bin/fedora-easy-karma", line 510, in <module>
:    FedoraEasyKarma()
:  File "/usr/bin/fedora-easy-karma", line 310, in __init__
:    testing_updates = bc.query(release=release, status="testing", limit=1000)["updates"]
:  File "/usr/lib/python2.7/site-packages/fedora/client/bodhi.py", line 147, in query
:    return self.send_request('list', req_params=params, auth=auth)
:  File "/usr/lib/python2.7/site-packages/fedora/client/baseclient.py", line 344, in send_request
:    auth_params=auth_params, retries=retries)
:  File "/usr/lib/python2.7/site-packages/fedora/client/proxyclient.py", line 400, in send_request
:    request.perform()
:error: (18, 'transfer closed with 656621 bytes remaining to read')
:
:Local variables in innermost frame:
:username: None
:retries: 0
:req_params: {'status': 'testing', 'release': 'F17', 'updates_tgp_limit': 1000}
:i: ('updates_tgp_limit', 1000)
:self: <fedora.client.bodhi.BodhiClient object at 0x17609d0>
:request: <pycurl.Curl object at 0x1cbe400>
:session_id: None
:req_data: 'status=testing&release=F17&updates_tgp_limit=1000'
:auth_params: {}
:headers: ['User-agent: Fedora Easy Karma/0-0.16.20110825git36efb338.fc17', 'Accept: application/json']
:num_tries: 0
:complete_params: {'status': 'testing', 'release': 'F17', 'updates_tgp_limit': 1000}
:url: 'https://admin.fedoraproject.org/updates/list'
:file_params: None
:method: 'list'
:password: None
:data: None
:response: <fedora.client.proxyclient._PyCurlData object at 0x2287890>

smolt_data:
:
:
:General
:=================================
:UUID: 8f03b668-83b5-41b9-a27b-bdcb641ea811
:OS: Fedora release 17 (Beefy Miracle)
:Default run level: Unknown
:Language: en_AU.UTF-8
:Platform: x86_64
:BogoMIPS: 6186.29
:CPU Vendor: GenuineIntel
:CPU Model: Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz
:CPU Stepping: 7
:CPU Family: 6
:CPU Model Num: 42
:Number of CPUs: 4
:CPU Speed: 3101
:System Memory: 7965
:System Swap: 10015
:Vendor: Unknown
:System:  
:Form factor: Desktop
:Kernel: 3.3.7-1.fc17.x86_64
:SELinux Enabled: 1
:SELinux Policy: targeted
:SELinux Enforce: Enforcing
:MythTV Remote: Unknown
:MythTV Role: Unknown
:MythTV Theme: Unknown
:MythTV Plugin: 
:MythTV Tuner: -1
:
:
:Devices
:=================================
:(4147:404:32902:8194) pci, xhci_hcd, USB, uPD720200 USB 3.0 Host Controller
:(32902:7202:32902:8194) pci, i801_smbus, SERIAL, 6 Series/C200 Series Chipset Family SMBus Controller
:(32902:7190:32902:8194) pci, pcieport, PCI/PCI, 6 Series/C200 Series Chipset Family PCI Express Root Port 4
:(32902:7184:32902:8194) pci, pcieport, PCI/PCI, 6 Series/C200 Series Chipset Family PCI Express Root Port 1
:(32902:7242:32902:8194) pci, None, PCI/ISA, H67 Express Chipset Family LPC Controller
:(4318:4608:6618:5733) pci, nvidia, VIDEO, GF110 [GeForce GTX 560 Ti]
:(32902:7200:32902:8194) pci, snd_hda_intel, MULTIMEDIA, 6 Series/C200 Series Chipset Family High Definition Audio Controller
:(32902:5379:32902:8194) pci, e1000e, ETHERNET, 82579V Gigabit Network Connection
:(4318:3596:6618:5733) pci, snd_hda_intel, MULTIMEDIA, N/A
:(4739:34962:32902:8194) pci, None, PCI/PCI, N/A
:(32902:7170:32902:8194) pci, ahci, STORAGE, 6 Series/C200 Series Chipset Family 6 port SATA AHCI Controller
:(32902:7213:32902:8194) pci, ehci_hcd, USB, 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2
:(32902:7206:32902:8194) pci, ehci_hcd, USB, 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1
:(32902:256:32902:8194) pci, None, HOST/PCI, 2nd Generation Core Processor Family DRAM Controller
:(32902:7226:32902:8194) pci, None, SIMPLE, 6 Series/C200 Series Chipset Family MEI Controller #1
:(32902:257:32902:8194) pci, pcieport, PCI/PCI, Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port
:
:
:Filesystem Information
:=================================
:device mtpt type bsize frsize blocks bfree bavail file ffree favail
:-------------------------------------------------------------------
:/dev/mapper/vg_thunder-lv_root / ext4 4096 4096 13092026 10344085 10213048 3276800 3017174 3017174
:/dev/mapper/vg_thunder-lv_home /home ext4 4096 4096 10185281 5939161 5428800 2555904 2484604 2484604
:/dev/sda2 WITHHELD fuseblk 4096 4096 218156925 148620270 148620270 594939832 594540558 594540558
:/dev/sda3 /boot ext4 1024 1024 508745 387335 361735 128016 127687 127687
:
Comment 1 Till Maas 2013-05-27 15:40:34 EDT
This is an exception that should be catched in python-fedora.
Comment 2 Toshio Ernie Kuratomi 2013-05-28 01:56:08 EDT
Couple things:

1) You're using an older version of python-fedora.  The current version uses python-requests rather than pycurl so you won't see this particular traceback with the current package in Fedora.  The latest version of python-fedora is python-fedora-0.3.32.3-1.fc17 -- be sure that you have an updated python-requests as well since earlier versions of python-requests were buggy.  The latest f17 version of that is: python-requests-1.1.0-3.fc17

2) This error should be caught at the fedora-easy-karma level, not the python-fedora level.  This is an error from the request to the server that python-fedora can't recover from therefore it's up to fedora-easy-karma to decide what to do with it.
Comment 3 Till Maas 2013-05-28 03:41:36 EDT
(In reply to Toshio Ernie Kuratomi from comment #2)
> Couple things:
> 
> 1) You're using an older version of python-fedora.  The current version uses
> python-requests rather than pycurl so you won't see this particular
> traceback with the current package in Fedora.  The latest version of
> python-fedora is python-fedora-0.3.32.3-1.fc17 -- be sure that you have an
> updated python-requests as well since earlier versions of python-requests
> were buggy.  The latest f17 version of that is: python-requests-1.1.0-3.fc17

The bug report is old, therefore an older version was used.

> 2) This error should be caught at the fedora-easy-karma level, not the
> python-fedora level.  This is an error from the request to the server that
> python-fedora can't recover from therefore it's up to fedora-easy-karma to
> decide what to do with it.

IMHO python-fedora should raise a BodhiClientException instead of an exception depending on the underlying request abstraction, then f-e-k can easily except it.
Comment 4 Toshio Ernie Kuratomi 2013-05-29 01:03:32 EDT
I used to think that as well but changed my mind after reading

http://plope.com/Members/chrism/now_not_to_write_python

and 

http://blog.ianbicking.org/2007/09/12/re-raising-exceptions/

and seeing the exceptions portion of this talk:

http://pyvideo.org/video/880/stop-writing-classes
Comment 5 Fedora End Of Life 2013-07-03 21:40:44 EDT
This message is a reminder that Fedora 17 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 17. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '17'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 17's end of life.

Bug Reporter:  Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 17 is end of life. If you 
would still like  to see this bug fixed and are able to reproduce it 
against a later version  of Fedora, you are encouraged  change the 
'version' to a later Fedora version prior to Fedora 17's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 6 Till Maas 2013-07-05 14:29:46 EDT
(In reply to Toshio Ernie Kuratomi from comment #4)
> I used to think that as well but changed my mind after reading
> 
> http://plope.com/Members/chrism/now_not_to_write_python
> 
> and 
> 
> http://blog.ianbicking.org/2007/09/12/re-raising-exceptions/
> 
> and seeing the exceptions portion of this talk:
> 
> http://pyvideo.org/video/880/stop-writing-classes

This does not really convince me, as the exception that was raised appears to come from pycurl and would not be raised anymore and there is no comment which exceptions BodhiClient.query() is supposed to raise, even if they come from other modules. Can you please give an example how you would use the method according to the ideas from the posted URLs?
Comment 7 Toshio Ernie Kuratomi 2013-07-05 15:35:58 EDT
It depends on what you want to do.  It sounds like you want to catch all errors that may be thrown from this function.  So:

try:
    testing_updates = bc.query(release=release, status="testing", limit=1000)["updates"]
except Exception as e:
    do_stuff()

Do you actually have a way to handle this specific exception and recover gracefully instead?
Comment 8 Till Maas 2013-07-05 15:45:35 EDT
If the exception indicates a transient network error, then I would just retry bc.query(), e.g. up to three times. If it is a permanent error, printing a descriptive error message and just quitting would be the way to go.
Comment 9 Toshio Ernie Kuratomi 2013-07-05 18:22:20 EDT
try:
    testing_updates = bc.query(release=release, status="testing", limit=1000, retries=3)["updates"]
except Exception as e:
    print_descriptive_error()

(Note: you could also set bc.retries=3 to make that the default for all bc actions if you want.)
Comment 10 Till Maas 2013-07-07 16:48:59 EDT
(In reply to Toshio Ernie Kuratomi from comment #9)
> try:
>     testing_updates = bc.query(release=release, status="testing",
> limit=1000, retries=3)["updates"]
> except Exception as e:
>     print_descriptive_error()
> 
> (Note: you could also set bc.retries=3 to make that the default for all bc
> actions if you want.)

It looks like bc.query() does not support a retries parameter on Fedora 19. Also I changed it to

try:
    testing_updates = bc.query(release=release, status="testing", limit=1000)
except Exception as e:
    print "Cannot query Bodhi: {0}".format(e)
    sys.exit(1)
else:
    testing_updates = testing_updates["updates"]

to at least not mask a potential KeyError for "updates", but it does not really feel clean, because e.g. this also masks errors that would result from a change to Bodhi's API and therefore making the client fail. Therefore ther should IMHO be an easy way to distinguish permanent from transient errors.
Comment 11 Fedora Update System 2013-07-07 17:56:20 EDT
fedora-easy-karma-0-0.19.20130707git121694f6.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/fedora-easy-karma-0-0.19.20130707git121694f6.fc19
Comment 12 Fedora Update System 2013-07-07 17:56:52 EDT
fedora-easy-karma-0-0.19.20130707git121694f6.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/fedora-easy-karma-0-0.19.20130707git121694f6.fc18
Comment 13 Toshio Ernie Kuratomi 2013-07-08 12:57:09 EDT
(In reply to Till Maas from comment #10)
> (In reply to Toshio Ernie Kuratomi from comment #9)
> > try:
> >     testing_updates = bc.query(release=release, status="testing",
> > limit=1000, retries=3)["updates"]
> > except Exception as e:
> >     print_descriptive_error()
> > 
> > (Note: you could also set bc.retries=3 to make that the default for all bc
> > actions if you want.)
> 
> It looks like bc.query() does not support a retries parameter on Fedora 19.

Yeah, i was thinking incorrectly.  You'll need to use bc.retries=3

send_request() which every call to the server is built on supports setting retries at the time of the call.  But individual methods that call the server have the option not to support every parameter that send_request() does.

> Also I changed it to
> 
> try:
>     testing_updates = bc.query(release=release, status="testing", limit=1000)
> except Exception as e:
>     print "Cannot query Bodhi: {0}".format(e)
>     sys.exit(1)
> else:
>     testing_updates = testing_updates["updates"]
>
Note:  There's no need for an else: there.  You can simply set testing_updates = testing_updates["updates"] after you exit the try: except:
 
> to at least not mask a potential KeyError for "updates", but it does not
> really feel clean, because e.g. this also masks errors that would result
> from a change to Bodhi's API and therefore making the client fail. Therefore
> ther should IMHO be an easy way to distinguish permanent from transient
> errors.

Distinguishing transient from permanent errors won't help you with that case (API changes are a type of permanent error...).  And transient errors that are repeatable X number of times (via retries) seem to fall into the same error handling as permanent errors in your use case....

I'm not sure that misuse of the API can be distinguished from other errors either... For some methods, especially generic methods ("query()" sounds like such a method although lmacken wrote it, not I)  the user is allowed to pass strings in that border on arbitrary.  These are passed on to the server which sanitizes them and then tries to do something sensible with them.  But the attempt may cause a traceback on the server.  This would be indistinguishable from a traceback caused by a bug in the server code.  (Equally, if the strings that were given, say as a simple enum replacement, were to change, sending bad enums might simply make the server return empty data sets rather than throwing an error).

If you're simply worried that bc.query might go away, you can check for a NameError exception before your catch-all exception.
Comment 14 Till Maas 2013-07-08 17:02:28 EDT
(In reply to Toshio Ernie Kuratomi from comment #13)
> (In reply to Till Maas from comment #10)
> > (In reply to Toshio Ernie Kuratomi from comment #9)

> > Also I changed it to
> > 
> > try:
> >     testing_updates = bc.query(release=release, status="testing", limit=1000)
> > except Exception as e:
> >     print "Cannot query Bodhi: {0}".format(e)
> >     sys.exit(1)
> > else:
> >     testing_updates = testing_updates["updates"]
> >
> Note:  There's no need for an else: there.  You can simply set
> testing_updates = testing_updates["updates"] after you exit the try: except:

Thank you, I changed it.

> > to at least not mask a potential KeyError for "updates", but it does not
> > really feel clean, because e.g. this also masks errors that would result
> > from a change to Bodhi's API and therefore making the client fail. Therefore
> > ther should IMHO be an easy way to distinguish permanent from transient
> > errors.
> 
> Distinguishing transient from permanent errors won't help you with that case
> (API changes are a type of permanent error...).  And transient errors that
> are repeatable X number of times (via retries) seem to fall into the same
> error handling as permanent errors in your use case....

For me, typical transient errors, that should not create a traceback are IOErrors, e.g. because the network connections is buggy, the server is down etc.
Permanent errors that should case a traceback would be code errors in python-bugzilla or if bc.query() would not accept the parameters I specified.

> I'm not sure that misuse of the API can be distinguished from other errors
> either... For some methods, especially generic methods ("query()" sounds
> like such a method although lmacken wrote it, not I)  the user is allowed to
> pass strings in that border on arbitrary.  These are passed on to the server
> which sanitizes them and then tries to do something sensible with them.  But
> the attempt may cause a traceback on the server.  This would be
> indistinguishable from a traceback caused by a bug in the server code. 

Whether or it it can be distinguished depends on how the server reports each kind of traceback.
Comment 15 Fedora Update System 2013-07-08 21:30:23 EDT
Package fedora-easy-karma-0-0.19.20130707git121694f6.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing fedora-easy-karma-0-0.19.20130707git121694f6.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-12590/fedora-easy-karma-0-0.19.20130707git121694f6.fc18
then log in and leave karma (feedback).
Comment 16 Fedora Update System 2013-07-11 22:56:27 EDT
fedora-easy-karma-0-0.19.20130707git121694f6.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 17 Fedora Update System 2013-07-24 20:46:09 EDT
fedora-easy-karma-0-0.19.20130707git121694f6.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.