Bug 2018551 - python-googleapis-common-protos: FTBFS in Fedora Rawhide
Summary: python-googleapis-common-protos: FTBFS in Fedora Rawhide
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: python-rpm-macros
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Miro Hrončok
QA Contact: Fedora Extras Quality Assurance
URL: https://koschei.fedoraproject.org/pac...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-10-29 16:45 UTC by Ben Beasley
Modified: 2021-11-03 13:59 UTC (History)
10 users (show)

Fixed In Version: python-rpm-macros-3.10-13.fc36
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-11-03 11:11:45 UTC
Type: Bug


Attachments (Terms of Use)

Description Ben Beasley 2021-10-29 16:45:57 UTC
Description of problem:
Package python-googleapis-common-protos fails to build from source in Fedora Rawhide.

Version-Release number of selected component (if applicable):
1.53.0-13.fc36

Steps to Reproduce:
koji build --scratch f36 python-googleapis-common-protos-1.53.0-13.fc36.src.rpm

Additional info:
This package is tracked by Koschei. See:
https://koschei.fedoraproject.org/package/python-googleapis-common-protos

The regression appears beginning with pyproject-rpm-macros-0-49.fc36. I don’t yet understand the root cause, so I can’t make a claim about whether pyproject-rpm-macros or python-googleapis-common-protos needs to be corrected.

In the %py3_check_import “smoke tests,” packages and modules under the “google” namespace package (beginning with “google.api”) can no longer be imported.

The same happens if I add “PYTHONPATH=%{buildroot}%{python3_sitelib} %{python3} -c 'import google.api'” to %check.

If I skip the smoke tests and build the RPM with pyproject-rpm-macros-0-49.fc36 anyway, then install the resulting RPM into a clean mock chroot, “python3 -c 'import google.api'” *does* work, so I think the package is not actually broken.

(After this issue is resolved, I intend to switch to the new %pyproject_check_import, which will clean up the spec file considerably, but which would currently suffer from the same issue.)

Comment 1 Ben Beasley 2021-10-29 16:55:14 UTC
CC’ing Karolina Surma and Miro Hronçok, who both worked on this release of pyproject-rpm-macros.

If one of you has a chance to take a look at this and help understand what change, I would appreciate it!

Otherwise, I’ll come back to this issue when I have a little more dedicated time to read through recent commits in pyproject-rpm-macros.

Comment 2 Ben Beasley 2021-10-29 16:56:38 UTC
Sorry, diacritic typo. That should have been “Miro Hrončok.” I don’t like messing up people’s names.

Comment 3 Miro Hrončok 2021-10-29 18:19:48 UTC
> The same happens if I add “PYTHONPATH=%{buildroot}%{python3_sitelib} %{python3} -c 'import google.api'” to %check.

That is peculiarly weird because that's what the macro used to do.


Note that the package was actually not tracked in koschei. I've enabled it and now trying to reproduce locally.

Comment 4 Miro Hrončok 2021-10-29 18:49:27 UTC
Before:

Executing(%check): /bin/sh -e /var/tmp/rpm-tmp.DfLHlH
+ umask 022
+ cd /builddir/build/BUILD
+ cd python-api-common-protos-1.53.0
+ cd /builddir/build
+ PATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-13.fc36.x86_64/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin
+ PYTHONPATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-13.fc36.x86_64/usr/lib64/python3.10/site-packages:/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-13.fc36.x86_64/usr/lib/python3.10/site-packages
+ PYTHONDONTWRITEBYTECODE=1
+ /usr/bin/python3 -c 'import google.api, google.api.annotations_pb2, google.api.auth_pb2, google.api.backend_pb2, google.api.billing_pb2, google.api.client_pb2, google.api.config_change_pb2, google.api.consumer_pb2, google.api.context_pb2, google.api.control_pb2, google.api.distribution_pb2, google.api.documentation_pb2, google.api.endpoint_pb2, google.api.field_behavior_pb2, google.api.httpbody_pb2, google.api.http_pb2, google.api.label_pb2, google.api.launch_stage_pb2, google.api.logging_pb2, google.api.log_pb2, google.api.metric_pb2, google.api.monitored_resource_pb2, google.api.monitoring_pb2, google.api.quota_pb2, google.api.resource_pb2, google.api.service_pb2, google.api.source_info_pb2, google.api.system_parameter_pb2, google.api.usage_pb2, google.gapic, google.gapic.metadata, google.gapic.metadata.gapic_metadata_pb2, google.logging.type, google.logging.type.http_request_pb2, google.logging.type.log_severity_pb2, google.longrunning, google.longrunning.operations_pb2, google.longrunning.operations_proto_pb2, google.longrunning.operations_proto, google.rpc, google.rpc.code_pb2, google.rpc.error_details_pb2, google.rpc.status_pb2, google.rpc.context, google.rpc.context.attribute_context_pb2, google.type, google.type.calendar_period_pb2, google.type.color_pb2, google.type.date_pb2, google.type.datetime_pb2, google.type.dayofweek_pb2, google.type.expr_pb2, google.type.fraction_pb2, google.type.latlng_pb2, google.type.money_pb2, google.type.month_pb2, google.type.postal_address_pb2, google.type.quaternion_pb2, google.type.timeofday_pb2'
+ cd /builddir/build
+ PATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-13.fc36.x86_64/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin
+ PYTHONPATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-13.fc36.x86_64/usr/lib64/python3.10/site-packages:/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-13.fc36.x86_64/usr/lib/python3.10/site-packages
+ PYTHONDONTWRITEBYTECODE=1
+ /usr/bin/python3 -c 'import google.longrunning.operations_grpc_pb2, google.longrunning.operations_grpc, google.longrunning.operations_pb2_grpc'
+ RPM_EC=0



Now:

Executing(%check): /bin/sh -e /var/tmp/rpm-tmp.fPY28S
+ umask 022
+ cd /builddir/build/BUILD
+ cd python-api-common-protos-1.53.0
+ PATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-13.fc35.noarch/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin
+ PYTHONPATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-13.fc35.noarch/usr/lib64/python3.10/site-packages:/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-13.fc35.noarch/usr/lib/python3.10/site-packages
+ PYTHONDONTWRITEBYTECODE=1
+ /usr/bin/python3 -s /usr/lib/rpm/redhat/import_all_modules.py google.api
Check import: google.api
Traceback (most recent call last):
  File "/usr/lib/rpm/redhat/import_all_modules.py", line 152, in <module>
    main()
  File "/usr/lib/rpm/redhat/import_all_modules.py", line 148, in main
    import_modules(modules)
  File "/usr/lib/rpm/redhat/import_all_modules.py", line 94, in import_modules
    importlib.import_module(module)
  File "/usr/lib64/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'google.api'


I see that passing arguments to the macro on multiple lines no longer works but it worked before (and we can fix this be pre-preocesing the arguments in the macro itself and replacing any whitespace with a normal space). But that is not causing the import failure.



%check
PYTHONPATH=%{buildroot}%{python3_sitelib} %{python3} -c 'import google.api'

->

Executing(%check): /bin/sh -e /var/tmp/rpm-tmp.nmwq0T
+ umask 022
+ cd /builddir/build/BUILD
+ cd python-api-common-protos-1.53.0
+ PYTHONPATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages
+ /usr/bin/python3 -c 'import google.api'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'google.api'



%check
(cd %_topdir && PYTHONPATH=%{buildroot}%{python3_sitelib} %{python3} -c 'import google.api')

->

Executing(%check): /bin/sh -e /var/tmp/rpm-tmp.9sZsmu
+ umask 022
+ cd /builddir/build/BUILD
+ cd python-api-common-protos-1.53.0
+ cd /builddir/build
+ PYTHONPATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages
+ /usr/bin/python3 -c 'import google.api'
+ exit 0




The old version of the macro performed a cd to %_topdir before importing the modules and now it filters out $PWD from sys.path before attempting the imports.
This is most likely this implementation difference has some negative impact on namespace packages.

However, if I add a manual "cd %_topdir" before calling %py3_check_import google.api, it still doesn't work properly :(

%check
cd %_topdir
%py3_check_import google.api

->

Executing(%check): /bin/sh -e /var/tmp/rpm-tmp.taPr99
+ umask 022
+ cd /builddir/build/BUILD
+ cd python-api-common-protos-1.53.0
+ cd /builddir/build
+ PATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin
+ PYTHONPATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib64/python3.10/site-packages:/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages
+ PYTHONDONTWRITEBYTECODE=1
+ /usr/bin/python3 -s /usr/lib/rpm/redhat/import_all_modules.py google.api
Check import: google.api
Traceback (most recent call last):
  File "/usr/lib/rpm/redhat/import_all_modules.py", line 152, in <module>
    main()
  File "/usr/lib/rpm/redhat/import_all_modules.py", line 148, in main
    import_modules(modules)
  File "/usr/lib/rpm/redhat/import_all_modules.py", line 94, in import_modules
    importlib.import_module(module)
  File "/usr/lib64/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'google.api'




I believe the culprit is not in calling python with -s.

When I do:

%check
%global py3_shebang_flags u
%py3_check_import google.api

It still fails.


:/

Comment 5 Miro Hrončok 2021-10-29 23:16:33 UTC
Apparently, this works:

%check
cd %_topdir
PYTHONPATH=%{buildroot}%{python3_sitelib} %{python3} -sc 'import google.api'

But this fails.

%check
cd %_topdir
PYTHONPATH=%{buildroot}%{python3_sitelib} %{python3} -sc 'import sys; sys.path.remove(""); import google.api'


I have no idea why removing the current directory from sys.path makes a difference when the current directory only contains:

BUILD
BUILDROOT
RPMS
SOURCES
SPECS
SRPMS
originals


Also:

This works:

%check
cd %_topdir
echo 'import sys' > /tmp/foo.py
echo 'print(sys.path)' >> /tmp/foo.py
echo 'import google.api' >> /tmp/foo.py
PYTHONPATH=%{buildroot}%{python3_sitelib} %{python3} -s /tmp/foo.py

But this fails:

%check
cd %_topdir
echo 'import sys' > /tmp/foo.py
echo 'sys.path.remove("/tmp")' >> /tmp/foo.py   # <-- here is the only difference
echo 'print(sys.path)' >> /tmp/foo.py
echo 'import google.api' >> /tmp/foo.py
PYTHONPATH=%{buildroot}%{python3_sitelib} %{python3} -s /tmp/foo.py


This seems like dark magic to me at this point :/

Comment 6 Ben Beasley 2021-10-31 12:11:41 UTC
> I see that passing arguments to the macro on multiple lines no longer works but it worked before (and we can fix this be pre-preocesing the arguments in the macro itself and replacing any whitespace with a normal space). But that is not causing the import failure.

I have filed a separate bug, https://bugzilla.redhat.com/show_bug.cgi?id=2018809, to track this, since it will cause a larger number of packages that adopted %py3_check_import to FTBFS.

Comment 7 Miro Hrončok 2021-10-31 23:32:32 UTC
Funnily enough, if I call sys.path.remove("/tmp") I get:

sys.path: ['/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages', '/usr/lib64/python310.zip', '/usr/lib64/python3.10', '/usr/lib64/python3.10/lib-dynload', '/usr/lib64/python3.10/site-packages', '/usr/lib/python3.10/site-packages']

Traceback (most recent call last):
  File "/tmp/foo.py", line 4, in <module>
    import google.api
ModuleNotFoundError: No module named 'google.api'


But if I call sys.path[0] = "/tmp/xxx" ("/tmp" was at index 0) I get:

sys.path: ['/tmp/xxx', '/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages', '/usr/lib64/python310.zip', '/usr/lib64/python3.10', '/usr/lib64/python3.10/lib-dynload', '/usr/lib64/python3.10/site-packages', '/usr/lib/python3.10/site-packages']

And the import succeeds.


I can even use an ellipsis (sys.path[0] = ...) and the import works.

Even this works:

import sys
sys.path.remove("/tmp")
sys.path.append("xxx")
print(sys.path)
import google.api



I honestly have no idea why this is relevant at all. I cannot import google.api outside of %check section (e.g. be shelling into the mock chroot and executing the same commands from the same directories).



A workaround to restore the previous behavior seems to be inserting a bogus value to sys.path, but I am reluctant to push that without understanding the reason. It feels like something is messing with me.

Comment 8 Miro Hrončok 2021-10-31 23:50:12 UTC
> I cannot import google.api outside of %check section (e.g. be shelling into the mock chroot and executing the same commands from the same directories).

This was not true, I was just not testing it properly, as BUILDROOT was emptied by implicit %clean.


<mock-chroot> sh-5.1$ PYTHONPATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages python3 -s 
Python 3.10.0 (default, Oct  5 2021, 00:00:00) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.remove('')
>>> import google.api
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'google.api'
>>> sys.path.append('this is madness')
>>> import google.api
>>>

Comment 9 Miro Hrončok 2021-11-01 10:00:11 UTC
Paint me green and call me a cucumber:

<mock-chroot> sh-5.1$ PYTHONPATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages python3 -s 
Python 3.10.0 (default, Oct  5 2021, 00:00:00) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages', '/usr/lib64/python310.zip', '/usr/lib64/python3.10', '/usr/lib64/python3.10/lib-dynload', '/usr/lib64/python3.10/site-packages', '/usr/lib/python3.10/site-packages']
>>> sys.path.remove('')
>>> import google.api
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'google.api'
>>> sys.path.remove('/usr/lib64/python310.zip')
>>> import google.api
>>>

Comment 10 Miro Hrončok 2021-11-01 10:31:24 UTC
Observation: Removing /usr/lib/python3.10/site-packages/protobuf-3.18.1-py3.10-nspkg.pth makes the import work even when sys.path is ['/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages', '/usr/lib64/python310.zip', '/usr/lib64/python3.10', '/usr/lib64/python3.10/lib-dynload', '/usr/lib64/python3.10/site-packages', '/usr/lib/python3.10/site-packages']

Comment 11 Major Hayden 🤠 2021-11-01 10:52:52 UTC
Is this the same issue that Miro helped me with before?

%if %{with tests}
%check
# Work around an usual pytest/PEP 420 issue where pytest can't import the
# installed module. Thanks to mhroncok for the help!
mv google{,_}
%pytest --disable-warnings tests/unit
mv google{_,}
%endif

https://src.fedoraproject.org/rpms/python-google-cloud-iam/blob/rawhide/f/python-google-cloud-iam.spec#_69

Comment 12 Miro Hrončok 2021-11-01 11:11:03 UTC
I don't think so, we are explicitly cd'ing out of $PWD.

Comment 13 Miro Hrončok 2021-11-01 11:31:58 UTC
Upon interpreter startup, we get:

$ PYTHONPATH=/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages python3 -s 
>>> import sys
>>> sys.modules['google'].__path__
_NamespacePath(['/usr/lib/python3.10/site-packages/google'])
>>> sys.modules['google'].__path__._last_parent_path
('/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages', '/usr/lib64/python310.zip', '/usr/lib64/python3.10', '/usr/lib64/python3.10/lib-dynload', '/usr/lib64/python3.10/site-packages', '/usr/lib/python3.10/site-packages')


I believe this is because /usr/lib/python3.10/site-packages/protobuf-3.18.1-py3.10-nspkg.pth was executed prior prepending CWD / script's parent sys.path.

If we call `import google.api` now, tuple(sys.path) !=  sys.modules['google'].__path__._last_parent_path and hence the _NamespacePath is reloaded and google.api is found.

When we however remove sys.path[0], tuple(sys.path) ==  sys.modules['google'].__path__._last_parent_path and hence the _NamespacePath is not reloaded and google.api is not found.

https://github.com/python/cpython/blob/v3.10.0/Lib/importlib/_bootstrap_external.py#L1235


By adding anything to sys.path or removing something, we invalidate this cache.



Stuff I don't know yet:

 - why the _NamespacePath isn't _NamespacePath(['/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages'/google', '/usr/lib/python3.10/site-packages/google']) from the beginning
 - how to invalidate this cache without messign with actual sys.path values

Comment 14 Miro Hrončok 2021-11-01 11:40:46 UTC
> why the _NamespacePath isn't _NamespacePath(['/builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages'/google', '/usr/lib/python3.10/site-packages/google']) from the beginning

Apparently, the /usr/lib/python3.10/site-packages/protobuf-3.18.1-py3.10-nspkg.pth code does that explicitly.

Comment 15 Miro Hrončok 2021-11-01 12:10:48 UTC
I think I have figured this out.

The import only works nicely if both /usr/lib/python3.10/site-packages/protobuf-3.18.1-py3.10-nspkg.pth and /builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages/googleapis_common_protos-1.53.0-py3.10-nspkg.pth are executed.

But /usr/lib/python3.10/site-packages/ is a site directory (.pth files executed) and /builddir/build/BUILDROOT/python-googleapis-common-protos-1.53.0-14.fc36.x86_64/usr/lib/python3.10/site-packages is not a site directory (.pth files not executed).


There are several things we can do here.

 1) acknowledge the problem and work around it by adding a bogus value to sys.path to invalidate the cache -- ugly but easy

 2) additionally to adding %{buildroot}%{python3_site|arch} as PYTHONPATH, pass them as arguments to the script and call site.addsitedir() on them in main() -- more correct but can lead to unexpected side effects

 3) disable the site module (and not execute any .pth files at all), construct the sys.path manually -- can lead to unexpected side effects


I think number 2 is the best thing to do as it will resemble the installed environment the most. It will be harder to test, but not impossible.

Comment 16 Miro Hrončok 2021-11-01 12:39:28 UTC
2) could even be implemented as:

for path in os.getenv('PYTHONPATH', '').split(':'):
    site.addsitedir(path)

However, the macro accepts custom PYTHONPATH, so I wonder if this is the best approach. We could change the macro to do:

PYTHONPATH="${PYTHONPATH:-%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}}"\\\
_PYTHONSITE="%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}"\\\

And then read _PYTHONSITE instead of PYTHONPATH env var.

Comment 17 Ben Beasley 2021-11-01 13:38:42 UTC
(In reply to Miro Hrončok from comment #15)
> I think I have figured this out.
> 
> The import only works nicely if […].
> 
> But /usr/lib/python3.10/site-packages/ is a site directory […].

Thanks for taking the time to work through this issue. This makes some kind of sense—although it’s still a deeply arcane problem.

Comment 19 Miro Hrončok 2021-11-02 11:33:09 UTC
Now included in https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/121

Comment 20 Fedora Update System 2021-11-03 11:10:09 UTC
FEDORA-2021-70e7a5dae0 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2021-70e7a5dae0

Comment 21 Fedora Update System 2021-11-03 11:11:45 UTC
FEDORA-2021-70e7a5dae0 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 22 Ben Beasley 2021-11-03 13:59:49 UTC
Thanks! After confirming the fix, I switched this package to %pyproject_check_import, which also now works splendidly.


Note You need to log in before you can comment on or make changes to this bug.