Bug 841596 - ValueError: invalid enum value: 4294967296
ValueError: invalid enum value: 4294967296
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: pygobject3 (Show other bugs)
17
ppc64 Linux
urgent Severity urgent
: ---
: ---
Assigned To: John (J5) Palmieri
Fedora Extras Quality Assurance
:
Depends On:
Blocks: F18Betappc
  Show dependency treegraph
 
Reported: 2012-07-19 10:09 EDT by Mark Hamzy
Modified: 2013-03-13 00:52 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-11-28 12:21:41 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Work in progress patch to fix enum issues on 64-bit bigendian archs (2.87 KB, patch)
2012-07-31 17:30 EDT, Dave Malcolm
no flags Details | Diff
Work in progress patch against upstream master to fix various issues on ppc64 (13.87 KB, patch)
2012-08-01 16:10 EDT, Dave Malcolm
no flags Details | Diff
Work in progress patch to fix various issues on ppc64 (18.32 KB, patch)
2012-08-08 15:03 EDT, Gustavo Luiz Duarte
no flags Details | Diff
Work in progress patch to fix various issues on ppc64 (23.48 KB, patch)
2012-08-09 19:27 EDT, Gustavo Luiz Duarte
no flags Details | Diff
Fix "test_interface3impl (test_gi.TestPythonGObject)" failure (5.93 KB, patch)
2012-08-10 16:54 EDT, Dave Malcolm
no flags Details | Diff

  None (edit)
Description Mark Hamzy 2012-07-19 10:09:06 EDT
[root@hamzy-fedora17-build ~]# uname -a
Linux hamzy-fedora17-build.localdomain 3.3.4-5.fc17.x86_64 #1 SMP Mon May 7 17:29:34 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
[root@hamzy-fedora17-build ~]# yum install -y pygobject3
[root@hamzy-fedora17-build ~]# cat << __EOF__ | python -
#!/usr/bin/python
from gi.repository import NMClient

if __name__ == "__main__":
        client = NMClient.Client.new()

        for con in client.get_active_connections():
                print con
                device = con.get_devices()[0]
                print device
                print device.get_device_type()
__EOF__
<ActiveConnection object at 0x7f5a611291e0 (NMActiveConnection at 0xe06010)>
<DeviceEthernet object at 0x7f5a61129230 (NMDeviceEthernet at 0xe0b000)>
<enum NM_DEVICE_TYPE_ETHERNET of type DeviceType>
[root@hamzy-fedora17-build ~]# nmcli con
NAME                      UUID                                   TYPE              TIMESTAMP-REAL
Boot Disk                 f4b02c17-5a5c-49aa-bc30-b4ad5926b8d3   802-3-ethernet    Thu 19 Jul 2012 08:56:12 AM CDT



[root@bluebill ~]# uname -a
Linux bluebill.rchland.ibm.com 3.3.4-5.fc17.ppc64 #1 SMP Mon May 14 10:18:37 MST 2012 ppc64 ppc64 ppc64 GNU/Linux
[root@bluebill ~]# yum install -y pygobject3
[root@bluebill ~]# cat << __EOF__ | python -
> #!/usr/bin/python
> from gi.repository import NMClient
>
> if __name__ == "__main__":
>         client = NMClient.Client.new()
>
>         for con in client.get_active_connections():
>                 print con
>                 device = con.get_devices()[0]
>                 print device
>                 print device.get_device_type()
> __EOF__
<ActiveConnection object at 0xfffa02a6190 (NMActiveConnection at 0x1003002c800)>
<DeviceEthernet object at 0xfffa02a61e0 (NMDeviceEthernet at 0x10030034000)>
Traceback (most recent call last):
  File "<stdin>", line 11, in <module>
  File "/usr/lib64/python2.7/site-packages/gi/types.py", line 43, in function
    return info.invoke(*args, **kwargs)
ValueError: invalid enum value: 4294967296
[root@bluebill ~]# nmcli con
NAME                      UUID                                   TYPE              TIMESTAMP-REAL
Wired connection 1        34edb1f4-82bc-4471-ab69-4d01893d3293   802-3-ethernet    Fri 13 Jul 2012 01:57:48 AM CDT
Boot Disk                 bd10e0ca-184a-45d2-aa2a-769f973d54b6   802-3-ethernet    Thu 19 Jul 2012 09:03:57 AM CDT
Comment 1 Karsten Hopp 2012-07-19 11:01:48 EDT
Might even be a NM bug as get_device_type is from NM, I'll CC: the maintainer

<DeviceEthernet object at 0x1001318a4b0 (NMDeviceEthernet at 0x10013256070)>
<gi.FunctionInfo object (get_device_type) at 0x0x100131c1c68>
get_device_type
NMClient
Traceback (most recent call last):
  File "<stdin>", line 10, in <module>
  File "/usr/lib64/python2.7/site-packages/gi/types.py", line 46, in function
    return info.invoke(*args, **kwargs)
ValueError: invalid enum value: 4294967296
Comment 3 Dan Horák 2012-07-25 08:48:12 EDT
looks like a big-endian or 32bit vs 64bit issue, 4294967296 = 0x100000000

a check of build logs for "implicit declaration" warnings can be useful
Comment 4 Mark Hamzy 2012-07-26 11:52:46 EDT
1) Many testcases for python3-pygopject3 fail.

(cd ~/rpmbuild/BUILD/python3-pygobject3-3.2.2-1.fc17/tests; make check)

2) gobject-introspection testcase seems incorrectly written.  Where/What is the official standard?

[root@sharpie ~]# rpm -qf /usr/share/gobject-introspection-1.0/tests/gimarshallingtests.c
gobject-introspection-devel-1.32.1-1.fc17.ppc64

I see stuff like this:

test_glist_int_none_in (test_gi.TestGList) ...
g_list_length(list) = 4
list = 0x1000606f620
g_list_nth_data(list, 0) = 0xffffffff70ddde1c
GPOINTER_TO_INT(g_list_nth_data(list, 0)) = 1893588508
g_list_nth_data(list, 1) = 0x70ddde1c
GPOINTER_TO_INT(g_list_nth_data(list, 1)) = 1893588508
g_list_nth_data(list, 2) = 0x170ddde1c
GPOINTER_TO_INT(g_list_nth_data(list, 2)) = 1893588508
g_list_nth_data(list, 3) = 0x270ddde1c
GPOINTER_TO_INT(g_list_nth_data(list, 3)) = 1893588508

it is looking for -1, 0, 1, 2, and 3.  But there is garbage in half of it.
Comment 5 Matthias Clasen 2012-07-27 04:21:53 EDT
Filed upstream: https://bugzilla.gnome.org/show_bug.cgi?id=680692
Comment 6 Dave Malcolm 2012-07-31 17:30:25 EDT
Created attachment 601594 [details]
Work in progress patch to fix enum issues on 64-bit bigendian archs

Enums appear to be thoroughly broken on ppc64, with all enum values coming out as 0; the upstream selftest suite shows many failures, coredumping before running to completion.

The root-cause for this appears to be this implicit cast to "long" within:
  pygobject/gi/pygi-marshal-from-py.c:_pygi_marshal_from_py_interface_enum
where the enum value is extracted as a C "long" (OK) but then written through to the "v_long" member of the union, thus truncating the value to the most-significant 32-bits on a 64-bit big-endian box, thus leaving most values as 0:
  1295  
>>1296      arg->v_long = PYGLIB_PyLong_AsLong (int_);
  1297      Py_DECREF (int_);
  1298  
I believe this will affect any big-endian box where sizeof(long) > 4, such as ppc64, where sizeof(long) == 8

gobject-introspection's girepository/gifieldinfo.c:g_field_info_get_field has some logic for writing an enum value into a GIArgument using an appropriate size (which appears to be computed in girepository/giroffsets.c), so I've written similar code (perhaps should be a new helper function in gobject-introspection).

I'm attaching the work-in-progress patch I have for this.

With this patch, the
  TESTNAMES=test_gi.TestEnum 
subset of the test suite actually completes (rather than coredumping), with only 1 failure:

======================================================================
ERROR: test_enum_out (test_gi.TestEnum)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dmalcolm/pygobject/tests/test_gi.py", line 1136, in test_enum_out
    enum = GIMarshallingTests.enum_out()
  File "/home/dmalcolm/pygobject/gi/types.py", line 47, in function
    return info.invoke(*args, **kwargs)
ValueError: invalid enum value: 180388626432

----------------------------------------------------------------------
Ran 6 tests in 68.107s

FAILED (errors=1)

This appears to be a similar issue in _pygi_marshal_to_py_interface_enum, but I need to head out now, so am posting what I've got (in case my dev box gets accidentally reprovisioned).
Comment 7 Dave Malcolm 2012-08-01 16:10:51 EDT
Created attachment 601820 [details]
Work in progress patch against upstream master to fix various issues on ppc64
Comment 8 Martin Pitt 2012-08-03 00:54:17 EDT
I fixed the marshalling from/to GList and GSList in trunk:

http://git.gnome.org/browse/pygobject/commit/?id=770e6abfd5bc5dad7d5f56a18f1ef63f9754ada9
Comment 9 Martin Pitt 2012-08-03 00:59:17 EDT
FYI, yesterday I committed 

  http://git.gnome.org/browse/gobject-introspection/commit/?id=d0ffee9bf

which makes the assertion failures a lot more useful.
Comment 10 Martin Pitt 2012-08-03 01:23:34 EDT
When running "xvfb-run make check" in the porter mock chroot, I also got two breakages because Pango threw a warning. Fixed in 

  http://git.gnome.org/browse/pygobject/commit/?id=dbc6df6aad

This shouldn't affect your porting on desktop machines, but if it does, you can grab the patch.

Now I only have one abort left:

test_constructor (test_overrides.TestGdk) ... 
(runtests.py:35764): Gdk-WARNING **: gdkwindow.c:1406cannot make windows of type 0
make[2]: *** [check-local] Trace/Breakpoint ausgelöst

This might be a bug in Gdk or PyGObject, I haven't investigated this yet. When I disable the test (test_overrides.py TestGdk.test_constructor), there are no C-level assertions any more, and I get

Ran 575 tests in 6.506s
FAILED (failures=9, errors=1, expected failures=1)

which still points out some serious errors, but at least there's some visible progress.

I need to stop looking at this now, I'm afraid. I'll be back from holidays on August 20, so I guess I'll let you continue working on the currently attached patch version for the remaining issues. You can hop into #python on GNOME IRC to find some people to review/commit patches if you need it in the next two weeks (in particular, tomeu and pbor).

Thanks!
Comment 11 Gustavo Luiz Duarte 2012-08-08 15:03:55 EDT
Created attachment 603090 [details]
Work in progress patch to fix various issues on ppc64

Attached patch is a revision of Dave's patch including the following fixes:
a) Fix enum/flags conversion to/from PyObject in gi/pygi-argument.c
b) Fix closure return value storage in gi/pygi-closure.c
    When using girffi you must be careful to use libffi conventions. It means that the closure function passed to g_callable_info_prepare_closure() must store integral return values into a storage of size sizeof(ffi_arg) even for smaller-sized return values.
Comment 12 Gustavo Luiz Duarte 2012-08-08 15:08:30 EDT
Note that I've been working on top of Fedora 17's code, I didn't try to apply this on upstream, it may need some rework (though previous patches were flawless portable between f17 and upstream).
Comment 13 Dave Malcolm 2012-08-08 22:31:29 EDT
I added an invocation of the upstream selftest suite (just for Python 2 for now) to pygobject.spec's %check in f18 and master (now f19) as:
  http://pkgs.fedoraproject.org/cgit/pygobject3.git/commit/?id=4b2a8915cf06909568537feb2bbd3cbac392ee18

This will fail the build if any unexpected selftest failures happen (I had to mark one of the tests as failing, as it fails on i686, though not on x86_64).

I successfully rebuilt on the primary archs into f18:
  Building pygobject3-3.3.4-4.fc18 for f18-candidate
  Task info: http://koji.fedoraproject.org/koji/taskinfo?taskID=4370760

(I'm not running it for the python3 build yet; this appears to require additional fixes, but it isn't necessary for getting anaconda running, hence I believe it's a lower priority than the python2 build).
Comment 14 Dave Malcolm 2012-08-08 23:13:13 EDT
(In reply to comment #11)
> Created attachment 603090 [details]
> Work in progress patch to fix various issues on ppc64
> 
> Attached patch is a revision of Dave's patch including the following fixes:
> a) Fix enum/flags conversion to/from PyObject in gi/pygi-argument.c
> b) Fix closure return value storage in gi/pygi-closure.c
>     When using girffi you must be careful to use libffi conventions. It
> means that the closure function passed to g_callable_info_prepare_closure()
> must store integral return values into a storage of size sizeof(ffi_arg)
> even for smaller-sized return values.

Patch looks reasonable.

I tried scratch rebuilding the f19 package with this on the primary architectures, but ran into this new error (on both archs):
======================================================================
FAIL: test_python_calls_sync (test_gdbus.TestGDBusClient)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/pygobject-3.3.4/tests/test_gdbus.py", line 140, in test_python_calls_sync
    self.assertTrue('Timeout' in str(e), str(e))
AssertionError: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Could not get PID of name '1': no such name
----------------------------------------------------------------------
Comment 15 Gustavo Luiz Duarte 2012-08-09 19:27:12 EDT
Created attachment 603367 [details]
Work in progress patch to fix various issues on ppc64

A new revision of previous patch including a fix for the conversion of GHashTable from/to PyObject.
I had to move the functions _pygi_hash_pointer_to_arg() and _pygi_arg_to_hash_pointer() to gi/pygi-argument.c
Comment 16 Dave Malcolm 2012-08-09 20:52:01 EDT
(In reply to comment #15)
> Created attachment 603367 [details]
> Work in progress patch to fix various issues on ppc64
> 
> A new revision of previous patch including a fix for the conversion of
> GHashTable from/to PyObject.
> I had to move the functions _pygi_hash_pointer_to_arg() and
> _pygi_arg_to_hash_pointer() to gi/pygi-argument.c

Thanks.  I've updated "endianness-fixes.patch" within f18 and master to attachment 603367 [details] as:
http://pkgs.fedoraproject.org/cgit/pygobject3.git/commit/?id=58e8a6c275c75dc06d3b2baef623932569c82240
and am rebuilding into f18-candidate as http://koji.fedoraproject.org/koji/taskinfo?taskID=4374917
(it successfully scratch built for both i686 and x86_64).

I attempted a scratch build of this change into f18-candidate for ppc-koji as:
  http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=662395
and there was a single remaining failure:
======================================================================
ERROR: test_interface3impl (test_gi.TestPythonGObject)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/pygobject-3.3.4/tests/test_gi.py", line 1752, in test_interface3impl
    self.assertEqual(iface3.variants[0].unpack(), 27)
IndexError: list index out of range
----------------------------------------------------------------------
Ran 568 tests in 6.311s
FAILED (errors=1, expected failures=1, unexpected successes=2)
Comment 17 Dave Malcolm 2012-08-09 23:53:06 EDT
Getting late here, but this failure seems to be due to this assignment within _pygi_argument_to_array within "case GI_ARRAY_TYPE_C":

769              /* FIXME: Take into account the type of the length argument */
770              length = args[length_arg_pos]->v_int;

(gdb) p *args[length_arg_pos]
$22 = {v_boolean = 0, v_int8 = 0 '\000', v_uint8 = 0 '\000', v_int16 = 0, v_uint16 = 0, v_int32 = 0, v_uint32 = 0, v_int64 = 2, v_uint64 = 2, v_float = 0, \
v_double = 9.8813129168249309e-324, v_short = 0, v_ushort = 0, v_int = 0, v_uint = 0, v_long = 2, v_ulong = 2, v_ssize = 2, v_size = 2, v_string = 0x2 <Add\
ress 0x2 out of bounds>, v_pointer = 0x2}

Hence the python list passed as "variants" to the do_test_variant_array_in closure is being constructed with length 0; do_test_variant_array_in is called with
  variants: []
  n_variants: 2L
and this causes the IndexError looking up iface3.variants[0]
Comment 18 Dave Malcolm 2012-08-10 16:54:04 EDT
Created attachment 603634 [details]
Fix "test_interface3impl (test_gi.TestPythonGObject)" failure

This applies on top of attachment 603367 [details] to fix the error described in comment #16 and comment #17
Comment 19 Dave Malcolm 2012-08-10 17:10:36 EDT
(In reply to comment #18)
> Created attachment 603634 [details]
> Fix "test_interface3impl (test_gi.TestPythonGObject)" failure
> 
> This applies on top of attachment 603367 [details] to fix the error
> described in comment #16 and comment #17
Committed to dist-git (f18 and master) as:
http://pkgs.fedoraproject.org/cgit/pygobject3.git/commit/?id=794d0789064aa6fb5bbfe111393037e3c0e623bb

Building pygobject3-3.3.4-8.fc18 for f18-candidate
Task info: http://koji.fedoraproject.org/koji/taskinfo?taskID=4378195

Built successfully on a test ppc64 box:
  Ran 576 tests in 30.793s
  (skipped=68, expected failures=1)
but I wasn't able to build into koji-ppc due to an unrelated build.log issue:
http://ppc.koji.fedoraproject.org/koji/getfile?taskID=663123&name=root.log
Comment 20 Dave Malcolm 2012-08-13 11:09:05 EDT
I successfully rebuilt pygobject3-3.3.4-8.fc18 into koji-ppc as:
http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=664109

On ppc:
  Ran 568 tests in 5.994s
  OK (expected failures=2, unexpected successes=1)

On ppc64:
  Ran 568 tests in 6.255s
  OK (expected failures=1, unexpected successes=2)

Are there any remaining endianness issues with pygobject3?

(still need to get the patches upstream, of course, but Martin Pitt is still on vacation for another week IIRC).
Comment 21 Dave Malcolm 2012-08-13 16:40:10 EDT
Backported to the 3.2.2 in f17 and built for ppc into f17-candidate as pygobject3-3.2.2-6.fc17:
http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=664311
Comment 22 Martin Pitt 2012-08-20 05:25:57 EDT
Thanks Malcolm, great work! I did some minor cleanup in the patch, and pushed it upstream:

http://git.gnome.org/browse/pygobject/commit/?id=1c5d497d3c354f4d02f1d4570df2c61d6f47300c

With this I only get one test failure on ppc64 (test_gi.TestPythonGObject.test_interface3impl) which is covered and fixed by bug 842880.
Comment 23 Mark Hamzy 2012-08-21 09:26:02 EDT
This is still reproducible on http://ppc.koji.fedoraproject.org/stage/f18-20120820.0/18/ppc64/iso/Fedora-18-ppc64-DVD.iso
Comment 24 David Aquilina 2012-08-21 11:36:39 EDT
(In reply to comment #23)
> This is still reproducible on
> http://ppc.koji.fedoraproject.org/stage/f18-20120820.0/18/ppc64/iso/Fedora-
> 18-ppc64-DVD.iso

Looks like a tagging mishap (perhaps due to koji-shadow) caused the wrong version of pygobject3 to be included. I'll make sure this is fixed for our next ISO spin.
Comment 25 Phil Knirsch 2012-11-28 12:21:41 EST
As we haven't encountered any more pygobject3 bugs in the last 3 months i'm closing this bug as fixed for now.

Should we hit any more we'll create a new bug.

Thanks everyone for fixing all those issues!

Regards, Phil

Note You need to log in before you can comment on or make changes to this bug.