Bug 2055249 - Test failures on s390x: suspected endianness bug
Summary: Test failures on s390x: suspected endianness bug
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: python-Rtree
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Ben Beasley
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-02-16 14:11 UTC by Ben Beasley
Modified: 2022-03-26 15:06 UTC (History)
4 users (show)

Fixed In Version: python-Rtree-0.9.7-4.fc37 python-Rtree-0.9.7-4.el8 python-Rtree-0.9.7-4.fc36
Clone Of:
Environment:
Last Closed: 2022-02-26 16:58:26 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Patch to resolve test failures on s390x (693 bytes, patch)
2022-02-17 00:21 UTC, Scott K Logan
no flags Details | Diff


Links
System ID Private Priority Status Summary Last Updated
Github Toblerity rtree issues 220 0 None open Apparent C API mismatch causes test failure on big-endian architecture (s390x) 2022-02-20 15:05:32 UTC

Description Ben Beasley 2022-02-16 14:11:08 UTC
Description of problem:

Two tests fail on s390x, a suspected endianness issue.

Version-Release number of selected component (if applicable): 0.9.7-3.fc37


How reproducible:


Steps to Reproduce:

1. Move “BuildArch:  noarch” from the base package (python-Rtree) to python3-Rtree. The result is that the package is “built” and tested on all architectures to test for arch-dependent issues, but still produces only a single noarch package.

2. Add “%global debug_package %{nil}”, since the package still doesn’t contain any compiled code.

3. Do “fedpkg scratch-build --srpm”

Actual results:

s390x build fails due to two failing tests: https://koji.fedoraproject.org/koji/taskinfo?taskID=82890544

Expected results:

Build succeeds on all architectures.

Additional info:

============================= test session starts ==============================
platform linux -- Python 3.10.2, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /builddir/build/BUILD/Rtree-0.9.7
collected 30 items
tests/test_index.py .............F............F..                        [ 96%]
tests/test_tpr.py .                                                      [100%]
=================================== FAILURES ===================================
_____________________ IndexSerialization.test_interleaving _____________________
self = <tests.test_index.IndexSerialization testMethod=test_interleaving>
    def test_interleaving(self):
        """Streaming against a persisted index without interleaving"""
        def data_gen(interleaved=True):
            for i, (minx, miny, maxx, maxy) in enumerate(self.boxes15):
                if interleaved:
                    yield (i, (minx, miny, maxx, maxy), 42)
                else:
                    yield (i, (minx, maxx, miny, maxy), 42)
        p = index.Property()
        tname = tempfile.mktemp()
        idx = index.Index(tname,
                          data_gen(interleaved=False),
                          properties=p,
                          interleaved=False)
        hits = sorted(list(idx.intersection((0, 60, 0, 60))))
        self.assertTrue(len(hits), 10)
        self.assertEqual(hits, [0, 4, 16, 27, 35, 40, 47, 50, 76, 80])
    
        leaves = idx.leaves()
        expected = [
            (0, [2, 92, 51, 55, 26, 95, 7, 81, 38, 22, 58, 89, 91, 83, 98, 37,
                 70, 31, 49, 34, 11, 6, 13, 3, 23, 57, 9, 96, 84, 36, 5, 45,
                 77, 78, 44, 12, 42, 73, 93, 41, 71, 17, 39, 54, 88, 72, 97,
                 60, 62, 48, 19, 25, 76, 59, 66, 64, 79, 94, 40, 32, 46, 47,
                 15, 68, 10, 0, 80, 56, 50, 30],
             [-186.673789279, -96.7177218184, 172.392784956, 45.4856075292]),
            (2, [61, 74, 29, 99, 16, 43, 35, 33, 27, 63, 18, 90, 8, 53, 82,
                 21, 65, 24, 4, 1, 75, 67, 86, 52, 28, 85, 87, 14, 69, 20],
             [-174.739939684, 32.6596016791, 184.761387556, 96.6043699778])]
    
        if PY3 and False:
            # TODO : this reliably fails on Python 2.7 and 3.5
            # go through the traversal and see if everything is close
            assert all(all(np.allclose(a, b) for a, b in zip(L, E))
                       for L, E in zip(leaves, expected))
    
        hits = sorted(list(idx.intersection((0, 60, 0, 60), objects=True)))
        self.assertTrue(len(hits), 10)
>       self.assertEqual(hits[0].object, 42)
E       AssertionError: None != 42
/builddir/build/BUILD/Rtree-0.9.7/tests/test_index.py:367: AssertionError
________________________ IndexStream.test_stream_input _________________________
self = <tests.test_index.IndexStream testMethod=test_stream_input>
    def test_stream_input(self):
        p = index.Property()
        sindex = index.Index(self.boxes15_stream(), properties=p)
        bounds = (0, 0, 60, 60)
        hits = sindex.intersection(bounds)
        self.assertEqual(sorted(hits), [0, 4, 16, 27, 35, 40, 47, 50, 76, 80])
        objects = list(sindex.intersection((0, 0, 60, 60), objects=True))
    
        self.assertEqual(len(objects), 10)
>       self.assertEqual(objects[0].object, 42)
E       AssertionError: None != 42
/builddir/build/BUILD/Rtree-0.9.7/tests/test_index.py:485: AssertionError
=============================== warnings summary ===============================
tests/conftest.py:12
  /builddir/build/BUILD/Rtree-0.9.7/tests/conftest.py:12: PytestDeprecationWarning: @pytest.yield_fixture is deprecated.
  Use @pytest.fixture instead; they are the same.
    @pytest.yield_fixture(autouse=True)
-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
FAILED tests/test_index.py::IndexSerialization::test_interleaving - Assertion...
FAILED tests/test_index.py::IndexStream::test_stream_input - AssertionError: ...
=================== 2 failed, 28 passed, 1 warning in 0.37s ====================

Comment 1 Scott K Logan 2022-02-17 00:21:28 UTC
Created attachment 1861592 [details]
Patch to resolve test failures on s390x

I think I found the bug here - the size_t is getting chopped in half on 64-bit platforms, and that matters more for big endian machines.

I think the prototypes are wrong for size_t values in other signatures, but this seems to be the only one causing test failures. Maybe it only matters if it's a size_t pointer - I'm not sure.

Comment 2 Scott K Logan 2022-02-17 00:39:25 UTC
From what I can tell, this was done intentionally to work around an issue in spatialindex itself, though I can't seem to justify that change myself: https://github.com/Toblerity/rtree/pull/77

Here's the signature in spatialindex, which appears to have always used size_t: https://github.com/libspatialindex/libspatialindex/blob/8ee223632f95c81f49f5eb2d547ad973475c4601/include/spatialindex/capi/sidx_api.h#L42-L44

Comment 3 Ben Beasley 2022-02-17 01:56:45 UTC
From a quick glance on my phone, your analysis seems to make sense.

The change was written by the author of libspatialindex, which makes it even weirder.

> I think the prototypes are wrong for size_t values in other signatures, but this seems to be the only one causing test failures. Maybe it only matters if it's a size_t pointer - I'm not sure.

One is more likely to get away with prototype mismatches in value parameters than in pointer ones, depending on the particular types involved and the platform’s calling conventions, but it’s certainly a nest of dragons.

I’ll take a look too as soon as I have a chance. We should probably consult with upstream at some point.

Comment 4 Ben Beasley 2022-02-19 14:53:27 UTC
I opened a work-in-progress PR with your patch: https://src.fedoraproject.org/rpms/python-Rtree/pull-request/5

Comment 5 Ben Beasley 2022-02-20 15:05:32 UTC
Reported upstream with a full audit of the ctypes wrappers: https://github.com/Toblerity/rtree/issues/220

Comment 6 Ben Beasley 2022-02-21 15:52:06 UTC
I now have a PR open upstream that fixes all of the C API mismatches I found: https://github.com/Toblerity/rtree/pull/222

I have updated the python-Rtree package PR (https://src.fedoraproject.org/rpms/python-Rtree/pull-request/5) to use a patch corresponding to the full PR sent upstream, rebased against the current 0.9.7 release.

Comment 7 Fedora Update System 2022-02-26 16:55:58 UTC
FEDORA-2022-5319d9ea7f has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-5319d9ea7f

Comment 8 Fedora Update System 2022-02-26 16:58:26 UTC
FEDORA-2022-5319d9ea7f has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 9 Fedora Update System 2022-03-01 22:49:47 UTC
FEDORA-2022-f760eaf2f6 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-f760eaf2f6

Comment 10 Fedora Update System 2022-03-01 23:11:58 UTC
FEDORA-EPEL-2022-f060866d30 has been submitted as an update to Fedora EPEL 8. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-f060866d30

Comment 11 Fedora Update System 2022-03-02 16:17:13 UTC
FEDORA-EPEL-2022-f060866d30 has been pushed to the Fedora EPEL 8 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-f060866d30

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 12 Fedora Update System 2022-03-02 19:43:08 UTC
FEDORA-2022-f760eaf2f6 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-f760eaf2f6`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-f760eaf2f6

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 13 Fedora Update System 2022-03-09 20:41:09 UTC
FEDORA-EPEL-2022-f060866d30 has been pushed to the Fedora EPEL 8 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 14 Fedora Update System 2022-03-26 15:06:24 UTC
FEDORA-2022-f760eaf2f6 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.