Description of problem: Package python-googleapis-common-protos fails to build from source in Fedora Rawhide. Version-Release number of selected component (if applicable): 1.53.0-13.fc36 Steps to Reproduce: koji build --scratch f36 python-googleapis-common-protos-1.53.0-13.fc36.src.rpm Additional info: This package is tracked by Koschei. See: https://koschei.fedoraproject.org/package/python-googleapis-common-protos The regression appears beginning with pyproject-rpm-macros-0-49.fc36. I don’t yet understand the root cause, so I can’t make a claim about whether pyproject-rpm-macros or python-googleapis-common-protos needs to be corrected. In the %py3_check_import “smoke tests,” packages and modules under the “google” namespace package (beginning with “google.api”) can no longer be imported. The same happens if I add “PYTHONPATH=%{buildroot}%{python3_sitelib} %{python3} -c 'import google.api'” to %check. If I skip the smoke tests and build the RPM with pyproject-rpm-macros-0-49.fc36 anyway, then install the resulting RPM into a clean mock chroot, “python3 -c 'import google.api'” *does* work, so I think the package is not actually broken. (After this issue is resolved, I intend to switch to the new %pyproject_check_import, which will clean up the spec file considerably, but which would currently suffer from the same issue.)
CC’ing Karolina Surma and Miro Hronçok, who both worked on this release of pyproject-rpm-macros. If one of you has a chance to take a look at this and help understand what change, I would appreciate it! Otherwise, I’ll come back to this issue when I have a little more dedicated time to read through recent commits in pyproject-rpm-macros.
Sorry, diacritic typo. That should have been “Miro Hrončok.” I don’t like messing up people’s names.
> The same happens if I add “PYTHONPATH=%{buildroot}%{python3_sitelib} %{python3} -c 'import google.api'” to %check. That is peculiarly weird because that's what the macro used to do. Note that the package was actually not tracked in koschei. I've enabled it and now trying to reproduce locally.
Before: Executing(%check): /bin/sh -e /var/tmp/rpm-tmp.DfLHlH + umask 022 + cd /builddir/build/BUILD + cd python-api-common-protos-1.53.0 + cd /builddir/build + PATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-13.fc36.x86_64/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin + PYTHONPATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-13.fc36.x86_64/usr/lib64/python3.10/site-packages:/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-13.fc36.x86_64/usr/lib/python3.10/site-packages + PYTHONDONTWRITEBYTECODE=1 + /usr/bin/python3 -c 'import google.api, google.api.annotations_pb2, google.api.auth_pb2, google.api.backend_pb2, google.api.billing_pb2, google.api.client_pb2, google.api.config_change_pb2, google.api.consumer_pb2, google.api.context_pb2, google.api.control_pb2, google.api.distribution_pb2, google.api.documentation_pb2, google.api.endpoint_pb2, google.api.field_behavior_pb2, google.api.httpbody_pb2, google.api.http_pb2, google.api.label_pb2, google.api.launch_stage_pb2, google.api.logging_pb2, google.api.log_pb2, google.api.metric_pb2, google.api.monitored_resource_pb2, google.api.monitoring_pb2, google.api.quota_pb2, google.api.resource_pb2, google.api.service_pb2, google.api.source_info_pb2, google.api.system_parameter_pb2, google.api.usage_pb2, google.gapic, google.gapic.metadata, google.gapic.metadata.gapic_metadata_pb2, google.logging.type, google.logging.type.http_request_pb2, google.logging.type.log_severity_pb2, google.longrunning, google.longrunning.operations_pb2, google.longrunning.operations_proto_pb2, google.longrunning.operations_proto, google.rpc, google.rpc.code_pb2, google.rpc.error_details_pb2, google.rpc.status_pb2, google.rpc.context, google.rpc.context.attribute_context_pb2, google.type, google.type.calendar_period_pb2, google.type.color_pb2, google.type.date_pb2, google.type.datetime_pb2, google.type.dayofweek_pb2, google.type.expr_pb2, google.type.fraction_pb2, google.type.latlng_pb2, google.type.money_pb2, google.type.month_pb2, google.type.postal_address_pb2, google.type.quaternion_pb2, google.type.timeofday_pb2' + cd /builddir/build + PATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-13.fc36.x86_64/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin + PYTHONPATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-13.fc36.x86_64/usr/lib64/python3.10/site-packages:/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-13.fc36.x86_64/usr/lib/python3.10/site-packages + PYTHONDONTWRITEBYTECODE=1 + /usr/bin/python3 -c 'import google.longrunning.operations_grpc_pb2, google.longrunning.operations_grpc, google.longrunning.operations_pb2_grpc' + RPM_EC=0 Now: Executing(%check): /bin/sh -e /var/tmp/rpm-tmp.fPY28S + umask 022 + cd /builddir/build/BUILD + cd python-api-common-protos-1.53.0 + PATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-13.fc35.noarch/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin + PYTHONPATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-13.fc35.noarch/usr/lib64/python3.10/site-packages:/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-13.fc35.noarch/usr/lib/python3.10/site-packages + PYTHONDONTWRITEBYTECODE=1 + /usr/bin/python3 -s /usr/lib/rpm/redhat/import_all_modules.py google.api Check import: google.api Traceback (most recent call last): File "/usr/lib/rpm/redhat/import_all_modules.py", line 152, in <module> main() File "/usr/lib/rpm/redhat/import_all_modules.py", line 148, in main import_modules(modules) File "/usr/lib/rpm/redhat/import_all_modules.py", line 94, in import_modules importlib.import_module(module) File "/usr/lib64/python3.10/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1050, in _gcd_import File "<frozen importlib._bootstrap>", line 1027, in _find_and_load File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked ModuleNotFoundError: No module named 'google.api' I see that passing arguments to the macro on multiple lines no longer works but it worked before (and we can fix this be pre-preocesing the arguments in the macro itself and replacing any whitespace with a normal space). But that is not causing the import failure. %check PYTHONPATH=%{buildroot}%{python3_sitelib} %{python3} -c 'import google.api' -> Executing(%check): /bin/sh -e /var/tmp/rpm-tmp.nmwq0T + umask 022 + cd /builddir/build/BUILD + cd python-api-common-protos-1.53.0 + PYTHONPATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages + /usr/bin/python3 -c 'import google.api' Traceback (most recent call last): File "<string>", line 1, in <module> ModuleNotFoundError: No module named 'google.api' %check (cd %_topdir && PYTHONPATH=%{buildroot}%{python3_sitelib} %{python3} -c 'import google.api') -> Executing(%check): /bin/sh -e /var/tmp/rpm-tmp.9sZsmu + umask 022 + cd /builddir/build/BUILD + cd python-api-common-protos-1.53.0 + cd /builddir/build + PYTHONPATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages + /usr/bin/python3 -c 'import google.api' + exit 0 The old version of the macro performed a cd to %_topdir before importing the modules and now it filters out $PWD from sys.path before attempting the imports. This is most likely this implementation difference has some negative impact on namespace packages. However, if I add a manual "cd %_topdir" before calling %py3_check_import google.api, it still doesn't work properly :( %check cd %_topdir %py3_check_import google.api -> Executing(%check): /bin/sh -e /var/tmp/rpm-tmp.taPr99 + umask 022 + cd /builddir/build/BUILD + cd python-api-common-protos-1.53.0 + cd /builddir/build + PATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin + PYTHONPATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib64/python3.10/site-packages:/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages + PYTHONDONTWRITEBYTECODE=1 + /usr/bin/python3 -s /usr/lib/rpm/redhat/import_all_modules.py google.api Check import: google.api Traceback (most recent call last): File "/usr/lib/rpm/redhat/import_all_modules.py", line 152, in <module> main() File "/usr/lib/rpm/redhat/import_all_modules.py", line 148, in main import_modules(modules) File "/usr/lib/rpm/redhat/import_all_modules.py", line 94, in import_modules importlib.import_module(module) File "/usr/lib64/python3.10/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1050, in _gcd_import File "<frozen importlib._bootstrap>", line 1027, in _find_and_load File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked ModuleNotFoundError: No module named 'google.api' I believe the culprit is not in calling python with -s. When I do: %check %global py3_shebang_flags u %py3_check_import google.api It still fails. :/
Apparently, this works: %check cd %_topdir PYTHONPATH=%{buildroot}%{python3_sitelib} %{python3} -sc 'import google.api' But this fails. %check cd %_topdir PYTHONPATH=%{buildroot}%{python3_sitelib} %{python3} -sc 'import sys; sys.path.remove(""); import google.api' I have no idea why removing the current directory from sys.path makes a difference when the current directory only contains: BUILD BUILDROOT RPMS SOURCES SPECS SRPMS originals Also: This works: %check cd %_topdir echo 'import sys' > /tmp/foo.py echo 'print(sys.path)' >> /tmp/foo.py echo 'import google.api' >> /tmp/foo.py PYTHONPATH=%{buildroot}%{python3_sitelib} %{python3} -s /tmp/foo.py But this fails: %check cd %_topdir echo 'import sys' > /tmp/foo.py echo 'sys.path.remove("/tmp")' >> /tmp/foo.py # <-- here is the only difference echo 'print(sys.path)' >> /tmp/foo.py echo 'import google.api' >> /tmp/foo.py PYTHONPATH=%{buildroot}%{python3_sitelib} %{python3} -s /tmp/foo.py This seems like dark magic to me at this point :/
> I see that passing arguments to the macro on multiple lines no longer works but it worked before (and we can fix this be pre-preocesing the arguments in the macro itself and replacing any whitespace with a normal space). But that is not causing the import failure. I have filed a separate bug, https://bugzilla.redhat.com/show_bug.cgi?id=2018809, to track this, since it will cause a larger number of packages that adopted %py3_check_import to FTBFS.
Funnily enough, if I call sys.path.remove("/tmp") I get: sys.path: ['/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages', '/usr/lib64/python310.zip', '/usr/lib64/python3.10', '/usr/lib64/python3.10/lib-dynload', '/usr/lib64/python3.10/site-packages', '/usr/lib/python3.10/site-packages'] Traceback (most recent call last): File "/tmp/foo.py", line 4, in <module> import google.api ModuleNotFoundError: No module named 'google.api' But if I call sys.path[0] = "/tmp/xxx" ("/tmp" was at index 0) I get: sys.path: ['/tmp/xxx', '/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages', '/usr/lib64/python310.zip', '/usr/lib64/python3.10', '/usr/lib64/python3.10/lib-dynload', '/usr/lib64/python3.10/site-packages', '/usr/lib/python3.10/site-packages'] And the import succeeds. I can even use an ellipsis (sys.path[0] = ...) and the import works. Even this works: import sys sys.path.remove("/tmp") sys.path.append("xxx") print(sys.path) import google.api I honestly have no idea why this is relevant at all. I cannot import google.api outside of %check section (e.g. be shelling into the mock chroot and executing the same commands from the same directories). A workaround to restore the previous behavior seems to be inserting a bogus value to sys.path, but I am reluctant to push that without understanding the reason. It feels like something is messing with me.
> I cannot import google.api outside of %check section (e.g. be shelling into the mock chroot and executing the same commands from the same directories). This was not true, I was just not testing it properly, as BUILDROOT was emptied by implicit %clean. <mock-chroot> sh-5.1$ PYTHONPATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages python3 -s Python 3.10.0 (default, Oct 5 2021, 00:00:00) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path.remove('') >>> import google.api Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'google.api' >>> sys.path.append('this is madness') >>> import google.api >>>
Paint me green and call me a cucumber: <mock-chroot> sh-5.1$ PYTHONPATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages python3 -s Python 3.10.0 (default, Oct 5 2021, 00:00:00) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path ['', '/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages', '/usr/lib64/python310.zip', '/usr/lib64/python3.10', '/usr/lib64/python3.10/lib-dynload', '/usr/lib64/python3.10/site-packages', '/usr/lib/python3.10/site-packages'] >>> sys.path.remove('') >>> import google.api Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'google.api' >>> sys.path.remove('/usr/lib64/python310.zip') >>> import google.api >>>
Observation: Removing /usr/lib/python3.10/site-packages/protobuf-3.18.1-py3.10-nspkg.pth makes the import work even when sys.path is ['/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages', '/usr/lib64/python310.zip', '/usr/lib64/python3.10', '/usr/lib64/python3.10/lib-dynload', '/usr/lib64/python3.10/site-packages', '/usr/lib/python3.10/site-packages']
Is this the same issue that Miro helped me with before? %if %{with tests} %check # Work around an usual pytest/PEP 420 issue where pytest can't import the # installed module. Thanks to mhroncok for the help! mv google{,_} %pytest --disable-warnings tests/unit mv google{_,} %endif https://src.fedoraproject.org/rpms/python-google-cloud-iam/blob/rawhide/f/python-google-cloud-iam.spec#_69
I don't think so, we are explicitly cd'ing out of $PWD.
Upon interpreter startup, we get: $ PYTHONPATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages python3 -s >>> import sys >>> sys.modules['google'].__path__ _NamespacePath(['/usr/lib/python3.10/site-packages/google']) >>> sys.modules['google'].__path__._last_parent_path ('/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages', '/usr/lib64/python310.zip', '/usr/lib64/python3.10', '/usr/lib64/python3.10/lib-dynload', '/usr/lib64/python3.10/site-packages', '/usr/lib/python3.10/site-packages') I believe this is because /usr/lib/python3.10/site-packages/protobuf-3.18.1-py3.10-nspkg.pth was executed prior prepending CWD / script's parent sys.path. If we call `import google.api` now, tuple(sys.path) != sys.modules['google'].__path__._last_parent_path and hence the _NamespacePath is reloaded and google.api is found. When we however remove sys.path[0], tuple(sys.path) == sys.modules['google'].__path__._last_parent_path and hence the _NamespacePath is not reloaded and google.api is not found. https://github.com/python/cpython/blob/v3.10.0/Lib/importlib/_bootstrap_external.py#L1235 By adding anything to sys.path or removing something, we invalidate this cache. Stuff I don't know yet: - why the _NamespacePath isn't _NamespacePath(['/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages'/google', '/usr/lib/python3.10/site-packages/google']) from the beginning - how to invalidate this cache without messign with actual sys.path values
> why the _NamespacePath isn't _NamespacePath(['/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages'/google', '/usr/lib/python3.10/site-packages/google']) from the beginning Apparently, the /usr/lib/python3.10/site-packages/protobuf-3.18.1-py3.10-nspkg.pth code does that explicitly.
I think I have figured this out. The import only works nicely if both /usr/lib/python3.10/site-packages/protobuf-3.18.1-py3.10-nspkg.pth and /builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages/googleapis_common_protos-1.53.0-py3.10-nspkg.pth are executed. But /usr/lib/python3.10/site-packages/ is a site directory (.pth files executed) and /builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages is not a site directory (.pth files not executed). There are several things we can do here. 1) acknowledge the problem and work around it by adding a bogus value to sys.path to invalidate the cache -- ugly but easy 2) additionally to adding %{buildroot}%{python3_site|arch} as PYTHONPATH, pass them as arguments to the script and call site.addsitedir() on them in main() -- more correct but can lead to unexpected side effects 3) disable the site module (and not execute any .pth files at all), construct the sys.path manually -- can lead to unexpected side effects I think number 2 is the best thing to do as it will resemble the installed environment the most. It will be harder to test, but not impossible.
2) could even be implemented as: for path in os.getenv('PYTHONPATH', '').split(':'): site.addsitedir(path) However, the macro accepts custom PYTHONPATH, so I wonder if this is the best approach. We could change the macro to do: PYTHONPATH="${PYTHONPATH:-%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}}"\\\ _PYTHONSITE="%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}"\\\ And then read _PYTHONSITE instead of PYTHONPATH env var.
(In reply to Miro Hrončok from comment #15) > I think I have figured this out. > > The import only works nicely if […]. > > But /usr/lib/python3.10/site-packages/ is a site directory […]. Thanks for taking the time to work through this issue. This makes some kind of sense—although it’s still a deeply arcane problem.
https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/122
Now included in https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/121
FEDORA-2021-70e7a5dae0 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2021-70e7a5dae0
FEDORA-2021-70e7a5dae0 has been pushed to the Fedora 36 stable repository. If problem still persists, please make note of it in this bug report.
Thanks! After confirming the fix, I switched this package to %pyproject_check_import, which also now works splendidly.