Bug 2064737 - Test failures on s390x: endianness bug
Summary: Test failures on s390x: endianness bug
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: python-pynetdicom
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Ben Beasley
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-03-16 13:03 UTC by Ben Beasley
Modified: 2022-03-17 14:59 UTC (History)
2 users (show)

Fixed In Version: python-pynetdicom-2.0.1-3.fc37
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-17 14:59:07 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Ben Beasley 2022-03-16 13:03:41 UTC
Description of problem:

Some tests fail on s390x. The output shows that there is a little-endian assumption somewhere.

Version-Release number of selected component (if applicable): 2.0.1-2


How reproducible:


Steps to Reproduce:
1. fedpkg scratch-build --arches s390x

Actual results:

=================================== FAILURES ===================================
_____________________ TestSetUID.test_bytes_decoding_error _____________________
self = <pynetdicom.tests.test_utils.TestSetUID object at 0x3ff94679990>
caplog = <_pytest.logging.LogCaptureFixture object at 0x3ff604eaa40>
    def test_bytes_decoding_error(self, caplog):
        """Test invalid bytes raises exception"""
        b = "1.2.3".encode("utf_32")
        assert isinstance(b, bytes)
        msg = (
            r"Unable to decode 'FF FE 00 00 31 00 00 00 2E 00 00 00 32 00 00 "
            r"00 2E 00 00 00 33 00 00 00' using the ascii codec\(s\)"
        )
        with caplog.at_level(logging.ERROR, logger="pynetdicom"):
            with pytest.raises(ValueError, match=msg):
>               set_uid(b, "foo")
pynetdicom/tests/test_utils.py:241: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
value = b'\x00\x00\xfe\xff\x00\x00\x001\x00\x00\x00.\x00\x00\x002\x00\x00\x00.\x00\x00\x003'
name = 'foo', allow_empty = True, allow_none = True, validate = True
    def set_uid(
        value: Union[None, str, bytes, UID],
        name: str,
        allow_empty: bool = True,
        allow_none: bool = True,
        validate: bool = True,
    ) -> Optional[UID]:
        """Convert `value` to a :class:`UID` and apply validation.
    
        Parameters
        ----------
        value : str, bytes, UID (and optionally None)
            The value to be converted.
        name : str
            The name of the parameter being converted.
        allow_empty : bool, optional
            If ``True`` then allow an empty UID (default).
        allow_none : bool, optional
            Allow the returned value to be ``None`` if `value` is ``None``
            (default ``True``).
        validate : bool, optional
            If ``True`` (default) perform validation of the UID using
            :func:`~pynetdicom.utils.validate_uid` and raise a
            :class:`ValueError` exception if the validation fails. If ``False``
            return the UID without performing and validation.
    
        Returns
        -------
        pydicom.uid.UID or None
            If ``allow_none`` is ``True`` then may return ``None``, otherwise
            only a UID will be returned.
        """
        if allow_none and value is None:
            return None
    
        if isinstance(value, bytes):
>           value = decode_bytes(value)
pynetdicom/utils.py:243: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
encoded_value = b'\x00\x00\xfe\xff\x00\x00\x001\x00\x00\x00.\x00\x00\x002\x00\x00\x00.\x00\x00\x003'
    def decode_bytes(encoded_value: bytes) -> str:
        """Return the decoded string from `encoded_value`.
    
        .. versionadded:: 2.0
    
        Parameters
        ----------
        encoded_value : bytes
            The encoded value to be decoded.
    
        Returns
        -------
        str
            The decoded ISO 646 (ASCII) string.
        """
        # Always try ASCII first
        try:
            return encoded_value.decode("ascii", errors="strict")
        except UnicodeDecodeError as exc:
            LOGGER.exception(exc)
    
        codecs: Sequence[str] = _config.CODECS
        codecs = [c for c in codecs if c not in ("ascii", "646", "us-ascii")]
    
        # If that fails then try the fallbacks and re-encode into ASCII
        for codec in codecs:
            try:
                value = encoded_value.decode(codec, errors="strict")
                encoded_value = value.encode("ascii", errors="ignore")
                return decode_bytes(encoded_value)
            except UnicodeError as exc:
                LOGGER.exception(exc)
    
        codecs.insert(0, "ascii")
        as_hex = " ".join([f"{b:02X}" for b in encoded_value])
>       raise ValueError(
            f"Unable to decode '{as_hex}' using the {', '.join(codecs)} codec(s)"
        )
E       ValueError: Unable to decode '00 00 FE FF 00 00 00 31 00 00 00 2E 00 00 00 32 00 00 00 2E 00 00 00 33' using the ascii codec(s)
pynetdicom/utils.py:68: ValueError
During handling of the above exception, another exception occurred:
self = <pynetdicom.tests.test_utils.TestSetUID object at 0x3ff94679990>
caplog = <_pytest.logging.LogCaptureFixture object at 0x3ff604eaa40>
    def test_bytes_decoding_error(self, caplog):
        """Test invalid bytes raises exception"""
        b = "1.2.3".encode("utf_32")
        assert isinstance(b, bytes)
        msg = (
            r"Unable to decode 'FF FE 00 00 31 00 00 00 2E 00 00 00 32 00 00 "
            r"00 2E 00 00 00 33 00 00 00' using the ascii codec\(s\)"
        )
        with caplog.at_level(logging.ERROR, logger="pynetdicom"):
>           with pytest.raises(ValueError, match=msg):
E           AssertionError: Regex pattern "Unable to decode 'FF FE 00 00 31 00 00 00 2E 00 00 00 32 00 00 00 2E 00 00 00 33 00 00 00' using the ascii codec\\(s\\)" does not match "Unable to decode '00 00 FE FF 00 00 00 31 00 00 00 2E 00 00 00 32 00 00 00 2E 00 00 00 33' using the ascii codec(s)".
pynetdicom/tests/test_utils.py:240: AssertionError
----------------------------- Captured stderr call -----------------------------
E: 'ascii' codec can't decode byte 0xfe in position 2: ordinal not in range(128)
Traceback (most recent call last):
  File "/builddir/build/BUILD/pynetdicom-2.0.1/pynetdicom/utils.py", line 50, in decode_bytes
    return encoded_value.decode("ascii", errors="strict")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfe in position 2: ordinal not in range(128)
------------------------------ Captured log call -------------------------------
ERROR    pynetdicom.utils:utils.py:52 'ascii' codec can't decode byte 0xfe in position 2: ordinal not in range(128)
Traceback (most recent call last):
  File "/builddir/build/BUILD/pynetdicom-2.0.1/pynetdicom/utils.py", line 50, in decode_bytes
    return encoded_value.decode("ascii", errors="strict")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfe in position 2: ordinal not in range(128)
_____________________ TestDecodeBytes.test_decoding_error ______________________
self = <pynetdicom.tests.test_utils.TestDecodeBytes object at 0x3ff9467a0b0>
caplog = <_pytest.logging.LogCaptureFixture object at 0x3ff60639cc0>
    def test_decoding_error(self, caplog):
        """Test decoding error raises and logs"""
        b = "1.2.3".encode("utf_32")
        with caplog.at_level(logging.ERROR, logger="pynetdicom"):
            msg = (
                r"Unable to decode 'FF FE 00 00 31 00 00 00 2E 00 00 00 32 "
                r"00 00 00 2E 00 00 00 33 00 00 00' using the ascii codec\(s\)"
            )
            with pytest.raises(ValueError, match=msg):
>               decode_bytes(b)
pynetdicom/tests/test_utils.py:322: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
encoded_value = b'\x00\x00\xfe\xff\x00\x00\x001\x00\x00\x00.\x00\x00\x002\x00\x00\x00.\x00\x00\x003'
    def decode_bytes(encoded_value: bytes) -> str:
        """Return the decoded string from `encoded_value`.
    
        .. versionadded:: 2.0
    
        Parameters
        ----------
        encoded_value : bytes
            The encoded value to be decoded.
    
        Returns
        -------
        str
            The decoded ISO 646 (ASCII) string.
        """
        # Always try ASCII first
        try:
            return encoded_value.decode("ascii", errors="strict")
        except UnicodeDecodeError as exc:
            LOGGER.exception(exc)
    
        codecs: Sequence[str] = _config.CODECS
        codecs = [c for c in codecs if c not in ("ascii", "646", "us-ascii")]
    
        # If that fails then try the fallbacks and re-encode into ASCII
        for codec in codecs:
            try:
                value = encoded_value.decode(codec, errors="strict")
                encoded_value = value.encode("ascii", errors="ignore")
                return decode_bytes(encoded_value)
            except UnicodeError as exc:
                LOGGER.exception(exc)
    
        codecs.insert(0, "ascii")
        as_hex = " ".join([f"{b:02X}" for b in encoded_value])
>       raise ValueError(
            f"Unable to decode '{as_hex}' using the {', '.join(codecs)} codec(s)"
        )
E       ValueError: Unable to decode '00 00 FE FF 00 00 00 31 00 00 00 2E 00 00 00 32 00 00 00 2E 00 00 00 33' using the ascii codec(s)
pynetdicom/utils.py:68: ValueError
During handling of the above exception, another exception occurred:
self = <pynetdicom.tests.test_utils.TestDecodeBytes object at 0x3ff9467a0b0>
caplog = <_pytest.logging.LogCaptureFixture object at 0x3ff60639cc0>
    def test_decoding_error(self, caplog):
        """Test decoding error raises and logs"""
        b = "1.2.3".encode("utf_32")
        with caplog.at_level(logging.ERROR, logger="pynetdicom"):
            msg = (
                r"Unable to decode 'FF FE 00 00 31 00 00 00 2E 00 00 00 32 "
                r"00 00 00 2E 00 00 00 33 00 00 00' using the ascii codec\(s\)"
            )
>           with pytest.raises(ValueError, match=msg):
E           AssertionError: Regex pattern "Unable to decode 'FF FE 00 00 31 00 00 00 2E 00 00 00 32 00 00 00 2E 00 00 00 33 00 00 00' using the ascii codec\\(s\\)" does not match "Unable to decode '00 00 FE FF 00 00 00 31 00 00 00 2E 00 00 00 32 00 00 00 2E 00 00 00 33' using the ascii codec(s)".
pynetdicom/tests/test_utils.py:321: AssertionError
----------------------------- Captured stderr call -----------------------------
E: 'ascii' codec can't decode byte 0xfe in position 2: ordinal not in range(128)
Traceback (most recent call last):
  File "/builddir/build/BUILD/pynetdicom-2.0.1/pynetdicom/utils.py", line 50, in decode_bytes
    return encoded_value.decode("ascii", errors="strict")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfe in position 2: ordinal not in range(128)
------------------------------ Captured log call -------------------------------
ERROR    pynetdicom.utils:utils.py:52 'ascii' codec can't decode byte 0xfe in position 2: ordinal not in range(128)
Traceback (most recent call last):
  File "/builddir/build/BUILD/pynetdicom-2.0.1/pynetdicom/utils.py", line 50, in decode_bytes
    return encoded_value.decode("ascii", errors="strict")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfe in position 2: ordinal not in range(128)
_________________ TestDecodeBytes.test_decoding_error_fallback _________________
self = <pynetdicom.tests.test_utils.TestDecodeBytes object at 0x3ff9467a230>
caplog = <_pytest.logging.LogCaptureFixture object at 0x3ff605037c0>
utf8 = None
    def test_decoding_error_fallback(self, caplog, utf8):
        """Test decoding error raises and logs"""
        b = "1.2.3".encode("utf_32")
        with caplog.at_level(logging.ERROR, logger="pynetdicom"):
            msg = (
                r"Unable to decode 'FF FE 00 00 31 00 00 00 2E 00 00 00 32 "
                r"00 00 00 2E 00 00 00 33 00 00 00' using the ascii, "
                r"utf8 codec\(s\)"
            )
            with pytest.raises(ValueError, match=msg):
>               decode_bytes(b)
pynetdicom/tests/test_utils.py:336: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
encoded_value = b'\x00\x00\xfe\xff\x00\x00\x001\x00\x00\x00.\x00\x00\x002\x00\x00\x00.\x00\x00\x003'
    def decode_bytes(encoded_value: bytes) -> str:
        """Return the decoded string from `encoded_value`.
    
        .. versionadded:: 2.0
    
        Parameters
        ----------
        encoded_value : bytes
            The encoded value to be decoded.
    
        Returns
        -------
        str
            The decoded ISO 646 (ASCII) string.
        """
        # Always try ASCII first
        try:
            return encoded_value.decode("ascii", errors="strict")
        except UnicodeDecodeError as exc:
            LOGGER.exception(exc)
    
        codecs: Sequence[str] = _config.CODECS
        codecs = [c for c in codecs if c not in ("ascii", "646", "us-ascii")]
    
        # If that fails then try the fallbacks and re-encode into ASCII
        for codec in codecs:
            try:
                value = encoded_value.decode(codec, errors="strict")
                encoded_value = value.encode("ascii", errors="ignore")
                return decode_bytes(encoded_value)
            except UnicodeError as exc:
                LOGGER.exception(exc)
    
        codecs.insert(0, "ascii")
        as_hex = " ".join([f"{b:02X}" for b in encoded_value])
>       raise ValueError(
            f"Unable to decode '{as_hex}' using the {', '.join(codecs)} codec(s)"
        )
E       ValueError: Unable to decode '00 00 FE FF 00 00 00 31 00 00 00 2E 00 00 00 32 00 00 00 2E 00 00 00 33' using the ascii, utf8 codec(s)
pynetdicom/utils.py:68: ValueError
During handling of the above exception, another exception occurred:
self = <pynetdicom.tests.test_utils.TestDecodeBytes object at 0x3ff9467a230>
caplog = <_pytest.logging.LogCaptureFixture object at 0x3ff605037c0>
utf8 = None
    def test_decoding_error_fallback(self, caplog, utf8):
        """Test decoding error raises and logs"""
        b = "1.2.3".encode("utf_32")
        with caplog.at_level(logging.ERROR, logger="pynetdicom"):
            msg = (
                r"Unable to decode 'FF FE 00 00 31 00 00 00 2E 00 00 00 32 "
                r"00 00 00 2E 00 00 00 33 00 00 00' using the ascii, "
                r"utf8 codec\(s\)"
            )
>           with pytest.raises(ValueError, match=msg):
E           AssertionError: Regex pattern "Unable to decode 'FF FE 00 00 31 00 00 00 2E 00 00 00 32 00 00 00 2E 00 00 00 33 00 00 00' using the ascii, utf8 codec\\(s\\)" does not match "Unable to decode '00 00 FE FF 00 00 00 31 00 00 00 2E 00 00 00 32 00 00 00 2E 00 00 00 33' using the ascii, utf8 codec(s)".
pynetdicom/tests/test_utils.py:335: AssertionError
----------------------------- Captured stderr call -----------------------------
E: 'ascii' codec can't decode byte 0xfe in position 2: ordinal not in range(128)
Traceback (most recent call last):
  File "/builddir/build/BUILD/pynetdicom-2.0.1/pynetdicom/utils.py", line 50, in decode_bytes
    return encoded_value.decode("ascii", errors="strict")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfe in position 2: ordinal not in range(128)
E: 'utf-8' codec can't decode byte 0xfe in position 2: invalid start byte
Traceback (most recent call last):
  File "/builddir/build/BUILD/pynetdicom-2.0.1/pynetdicom/utils.py", line 60, in decode_bytes
    value = encoded_value.decode(codec, errors="strict")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 2: invalid start byte
------------------------------ Captured log call -------------------------------
ERROR    pynetdicom.utils:utils.py:52 'ascii' codec can't decode byte 0xfe in position 2: ordinal not in range(128)
Traceback (most recent call last):
  File "/builddir/build/BUILD/pynetdicom-2.0.1/pynetdicom/utils.py", line 50, in decode_bytes
    return encoded_value.decode("ascii", errors="strict")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfe in position 2: ordinal not in range(128)
ERROR    pynetdicom.utils:utils.py:64 'utf-8' codec can't decode byte 0xfe in position 2: invalid start byte
Traceback (most recent call last):
  File "/builddir/build/BUILD/pynetdicom-2.0.1/pynetdicom/utils.py", line 60, in decode_bytes
    value = encoded_value.decode(codec, errors="strict")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 2: invalid start byte
=========================== short test summary info ============================
FAILED pynetdicom/tests/test_utils.py::TestSetUID::test_bytes_decoding_error
FAILED pynetdicom/tests/test_utils.py::TestDecodeBytes::test_decoding_error
FAILED pynetdicom/tests/test_utils.py::TestDecodeBytes::test_decoding_error_fallback
==== 3 failed, 2861 passed, 30 skipped, 444 deselected in 834.75s (0:13:54) ====

Expected results:

(successful build)

Additional info:

It seems like the problem is in the test rather than the library, and the message simply does not match the one in the assertion when the platform is big-endian. The best fix is probably replacing each occurrence of

> b = "1.2.3".encode("utf_32")

in pynetdicom/tests/test_utils.py with

> b = "1.2.3".encode("utf_32_le")

If I have a chance, I’ll verify this fix and submit a PR upstream.

Comment 1 Ben Beasley 2022-03-16 13:58:22 UTC
Working on an upstream bug report and PR.

Comment 2 Ben Beasley 2022-03-16 18:26:57 UTC
Reported upstream: https://github.com/pydicom/pynetdicom/issues/755

Upstream PR: https://github.com/pydicom/pynetdicom/pull/756

Downstream PR to follow.

Comment 4 Fedora Update System 2022-03-17 14:56:47 UTC
FEDORA-2022-ee1da274d1 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-ee1da274d1

Comment 5 Fedora Update System 2022-03-17 14:59:07 UTC
FEDORA-2022-ee1da274d1 has been pushed to the Fedora 37 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.