Bug 1451042 - sdk curl debug fails with UnicodeDecodeError: 'utf8' codec can't decode
Summary: sdk curl debug fails with UnicodeDecodeError: 'utf8' codec can't decode
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine-sdk-python
Classification: oVirt
Component: Core
Version: 4.1.3
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ovirt-4.1.3
: 4.1.4
Assignee: Juan Hernández
QA Contact: Petr Matyáš
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-05-15 15:52 UTC by Fabrice Bacchella
Modified: 2017-07-06 13:25 UTC (History)
5 users (show)

Fixed In Version: python-ovirt-engine-sdk4-4.1.4
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-07-06 13:25:22 UTC
oVirt Team: Infra
Embargoed:
rule-engine: ovirt-4.1+
lsvaty: testing_ack+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 76863 0 master MERGED Exclude unknown cURL debug types 2017-05-16 12:00:42 UTC
oVirt gerrit 76864 0 master MERGED Add debug data prefixes to indicate direction 2017-05-16 12:01:15 UTC
oVirt gerrit 76880 0 sdk_4.1 MERGED Exclude unknown cURL debug types 2017-05-16 13:13:09 UTC
oVirt gerrit 76885 0 sdk_4.1 MERGED Add debug data prefixes to indicate direction 2017-05-16 13:17:14 UTC

Description Fabrice Bacchella 2017-05-15 15:52:54 UTC
I'm running the sdk  using a virtualenv installed with:

virtualenv-2.7 venv
. ./venv/bin/activate
pip install ovirt-engine-sdk-python ...

It install pycurl 7.43.0

A very simple out of the box installation. But when I try to run with debug activated, I get:

Traceback (most recent call last):
  File ".../venv/lib/python2.7/site-packages/ovirtsdk4/__init__.py", line 664, in _curl_debug
    text = data.decode('utf-8') if type(data) == bytes else data
  File ".../venv/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfc in position 3: invalid start byte

If I add a little dump line in _curl_debug:
print "%s %s %s" % (debug_type, type(data), data)

I get:
6 <type 'str'> lots of binary data.

Indeed type 6 is CURLINFO_SSL_DATA_OUT, ssl raw data that can't be of any use and for sure will not decode as clean utf-8.

And any way the debug_type information is not used any more. In sdk v3, >>> and <<< symbols where added for in/out. That was usefull.

Can we get it back ? Or at least, at:

        self._curl.setopt(pycurl.DEBUGFUNCTION, self._curl_debug)

can we provide our own debug function ?

Comment 1 Fabrice Bacchella 2017-05-15 17:00:03 UTC
For my application, I have done that:

https://github.com/fbacchella/ovirtcmd/blob/sdk4/ovlib/context.py#L15
https://github.com/fbacchella/ovirtcmd/blob/sdk4/ovlib/context.py#L116

It generate output like:
   TCP_NODELAY set
   Connected to ovirt.prod.exalead.com (10.83.16.34) port 443 (#0)
   ALPN, offering http/1.1
...
>  POST /ovirt-engine/sso/oauth/token-http-auth HTTP/1.1
...
>> scope=ovirt-app-api&grant_type=urn%3Aovirt%3Aparams%3Aoauth%3Agrant-type%3Ahttp
<  HTTP/1.1 401 Unauthorized
...
<< <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<< <html><head>

if self.filter is set to CurlDebugType.TEXT | CurlDebugType.HEADER | CurlDebugType.DATA

Comment 2 Juan Hernández 2017-05-16 09:10:59 UTC
We need to fix this, and we can add the < and > to indicate the direction of messages. But I think we should not add a debug function, at least not that uses the same contract that pycurl uses, as that is an internal implementation detail, and it may be impossible to implement if one day we replace libcurl or pycurl with a different library.

Comment 3 Petr Matyáš 2017-06-06 13:24:23 UTC
Verified on python sdk 4.1.4


Note You need to log in before you can comment on or make changes to this bug.