Bug 2220295

$ fedpkg mockbuild --enablerepo=local

+ /usr/bin/pytest --ignore=test/test_consumer_integration.py --ignore=test/record/test_util.py test
ImportError while loading conftest '/builddir/build/BUILD/kafka-python-2.0.2/test/conftest.py'.
test/__init__.py:7: in <module>
    from kafka.future import Future
kafka/__init__.py:23: in <module>
    from kafka.consumer import KafkaConsumer
kafka/consumer/__init__.py:3: in <module>
    from kafka.consumer.group import KafkaConsumer
kafka/consumer/group.py:13: in <module>
    from kafka.consumer.fetcher import Fetcher
kafka/consumer/fetcher.py:19: in <module>
    from kafka.record import MemoryRecords
kafka/record/__init__.py:1: in <module>
    from kafka.record.memory_records import MemoryRecords, MemoryRecordsBuilder
kafka/record/memory_records.py:27: in <module>
    from kafka.record.legacy_records import LegacyRecordBatch, LegacyRecordBatchBuilder
kafka/record/legacy_records.py:50: in <module>
    from kafka.codec import (
kafka/codec.py:9: in <module>
    from kafka.vendor.six.moves import range
E   ModuleNotFoundError: No module named 'kafka.vendor.six.moves'

If I fix the above error by unbundling six as follows,

diff --git a/python-kafka.spec b/python-kafka.spec
index a7d4a47..8d6444c 100644
--- a/python-kafka.spec
+++ b/python-kafka.spec
@@ -55,6 +55,10 @@ is also supported for message sets.}
 %package -n python3-%{mod_name}
 Summary:          %{summary}
 
+# Unbundled
+BuildRequires:    %{py3_dist six}
+Requires:         %{py3_dist six}
+
 %description -n python3-%{mod_name} %_description
 
 
@@ -75,6 +79,15 @@ Documentation for Pure Python client for Apache Kafka.
 %autosetup -p0 -n %{project_name}-%{version}
 install -m 644 %{SOURCE1} %{_builddir}/%{project_name}-%{version}/LICENSE_doc
 
+# Unbundle six
+rm -vf kafka/vendor/six.py
+# The find-then-modify pattern keeps us from discarding mtimes on any sources
+# that do not need modification.
+find . -type f -name '*.py' -exec \
+    gawk '/vendor.*six/ { print FILENAME; nextfile }' '{}' '+' |
+  xargs -r -t sed -r -i -e 's/from kafka\.vendor (import six)/\1/' \
+      -e 's/(from )kafka\.vendor\.(six.*import)/\1\2/'
+
 %generate_buildrequires
 %pyproject_buildrequires -r
 
then I encounter the following test failure:

=================================== FAILURES ===================================
__________________________________ test_send ___________________________________

cli = <kafka.client_async.KafkaClient object at 0x7fa5fe0b4440>
conn = <MagicMock name='BrokerConnection' id='140350929294336'>

    def test_send(cli, conn):
        # Send to unknown node => raises AssertionError
        try:
            cli.send(2, None)
            assert False, 'Exception not raised'
        except AssertionError:
            pass

        # Send to disconnected node => NodeNotReady
        conn.state = ConnectionStates.DISCONNECTED
        f = cli.send(0, None)
        assert f.failed()
        assert isinstance(f.exception, Errors.NodeNotReadyError)

        conn.state = ConnectionStates.CONNECTED
        cli._maybe_connect(0)
        # ProduceRequest w/ 0 required_acks -> no response
        request = ProduceRequest[0](0, 0, [])
        assert request.expect_response() is False
        ret = cli.send(0, request)
>       assert conn.send.called_with(request)

test/test_client_async.py:223:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <MagicMock name='BrokerConnection.send' id='140350923425936'>
name = 'called_with'

    def __getattr__(self, name):
        if name in {'_mock_methods', '_mock_unsafe'}:
            raise AttributeError(name)
        elif self._mock_methods is not None:
            if name not in self._mock_methods or name in _all_magics:
                raise AttributeError("Mock object has no attribute %r" % name)
        elif _is_magic(name):
            raise AttributeError(name)
        if not self._mock_unsafe and (not self._mock_methods or name not in self._mock_methods):
            if name.startswith(('assert', 'assret', 'asert', 'aseert', 'assrt')) or name in _ATTRIB_DENY_LIST:
>               raise AttributeError(
                    f"{name!r} is not a valid assertion. Use a spec "
                    f"for the mock if {name!r} is meant to be an attribute.")
E               AttributeError: 'called_with' is not a valid assertion. Use a spec for the mock if 'called_with' is meant to be an attribute.

/usr/lib64/python3.12/unittest/mock.py:663: AttributeError
---------------------------- Captured stderr setup -----------------------------
WARNING:kafka.client:<MagicMock name='BrokerConnection' id='140350929294336'> timed out after <MagicMock name='BrokerConnection.config.__getitem__()' id='140350923160224'> ms. Closing connection.
------------------------------ Captured log setup ------------------------------
1688648698.883586 client_async.py         MainThread <MagicMock name='BrokerConnection' id='140350929294336'> timed out after <MagicMock name='BrokerConnection.config.__getitem__()' id='140350923160224'> ms. Closing connection.

[...]

=========================== short test summary info ============================
FAILED test/test_client_async.py::test_send - AttributeError: 'called_with' i...
================== 1 failed, 1081 passed, 33 skipped in 8.30s ==================

I don’t know what should be done about that. It does look like unbundling six and skipping this test (since it is a test/mocking issue rather than a library issue) could be a valid approach.

Thanks for the comments! I can make a patch to work around this issue. The following issue may cause this problem.

Prevent prefix "called_" for methods on mock objects in safe mode #100690 
https://github.com/python/cpython/issues/100690

Hirotaka

Hello, scratch build is a success.
https://koji.fedoraproject.org/koji/taskinfo?taskID=103200273

Here are description on my patches. I will submit patches to upstream if no problems.

Patch#1. kafka_codec.py.patch, test_fixture.py.patch, test_codec.py.patch

In my environment, an error occurs when trying to import the `range` class from `kafka.vendor.six.moves` module.
```
$ python3
Python 3.12.0b3 (main, Jun 21 2023, 00:00:00) [GCC 13.1.1 20230614 (Red Hat 13.1.1-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('.')
>>> from kafka.client_async import KafkaClient, IdleConnectionManager
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/build/dev/fedora_packaging/python-kafka/tmp/kafka-python-2.0.2/kafka/__init__.py", line 23, in <module>
    from kafka.consumer import KafkaConsumer
  File "/home/build/dev/fedora_packaging/python-kafka/tmp/kafka-python-2.0.2/kafka/consumer/__init__.py", line 3, in <module>
    from kafka.consumer.group import KafkaConsumer
  File "/home/build/dev/fedora_packaging/python-kafka/tmp/kafka-python-2.0.2/kafka/consumer/group.py", line 13, in <module>
    from kafka.consumer.fetcher import Fetcher
  File "/home/build/dev/fedora_packaging/python-kafka/tmp/kafka-python-2.0.2/kafka/consumer/fetcher.py", line 19, in <module>
    from kafka.record import MemoryRecords
  File "/home/build/dev/fedora_packaging/python-kafka/tmp/kafka-python-2.0.2/kafka/record/__init__.py", line 1, in <module>
    from kafka.record.memory_records import MemoryRecords, MemoryRecordsBuilder
  File "/home/build/dev/fedora_packaging/python-kafka/tmp/kafka-python-2.0.2/kafka/record/memory_records.py", line 27, in <module>
    from kafka.record.legacy_records import LegacyRecordBatch, LegacyRecordBatchBuilder
  File "/home/build/dev/fedora_packaging/python-kafka/tmp/kafka-python-2.0.2/kafka/record/legacy_records.py", line 50, in <module>
    from kafka.codec import (
  File "/home/build/dev/fedora_packaging/python-kafka/tmp/kafka-python-2.0.2/kafka/codec.py", line 9, in <module>
    from kafka.vendor.six.moves import range
ModuleNotFoundError: No module named 'kafka.vendor.six.moves'
>>>
```

My workaround is to call the `range` class using `moves.range` after import `moves` module.
```
>>> from kafka.vendor.six import moves
>>> moves.range
<class 'range'>
```


Patch#2. test_client_async.py.patch

In my environment, an error occurs at the following line.
> assert conn.send.called_with(request)

Two assertions exists at the above line. The first assertion `called_with` is called and then `assert` is called. At the first assertion, we should seem to use `assert_called_with` instead of `called_with`.
https://github.com/python/cpython/issues/100690

Then, the arguments of `assert_called_with` should be `request, blocking=False` because `KafkaClient` internally calls `send` with them.
https://github.com/dpkp/kafka-python/blob/master/kafka/client_async.py#L539

The second assertion should be removed because `assert` raises an error when an assert statement fails and do nothing when the statement succeeds.
```
$ python3
Python 3.12.0b3 (main, Jun 21 2023, 00:00:00) [GCC 13.1.1 20230614 (Red Hat 13.1.1-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from unittest.mock import Mock,MagicMock
>>> req = Mock
>>> conn = Mock
>>> conn.send = MagicMock(return_value=True)
>>> conn.send(req)
True
>>> conn.send.assert_called_with(req)     <--- return nothing when assertion is a success.
>>> conn.send.assert_not_called()    <--- raise AssertionError only when assertion fails.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.12/unittest/mock.py", line 905, in assert_not_called
    raise AssertionError(msg)
AssertionError: Expected 'mock' to not have been called. Called 1 times.
Calls: [call(<class 'unittest.mock.Mock'>)].
```

Thanks in advance,
Hirotaka

Note to self: restore kafka instrumentation in python-opentelemetry-contrib after this is resolved.

Hello, I submitted the patches to the upstream and submitted the official build on rawhide branch.

https://github.com/dpkp/kafka-python/pull/2375
https://github.com/dpkp/kafka-python/pull/2376
https://koji.fedoraproject.org/koji/taskinfo?taskID=103301305