Bug 1893533 - grpcio hangs connecting to remote
Summary: grpcio hangs connecting to remote
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: grpc
Version: 33
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Sergey Avseyev
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1886153 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-11-01 15:27 UTC by Seppo Yli-Olli
Modified: 2020-11-27 01:23 UTC (History)
6 users (show)

Fixed In Version: grpc-1.26.0-8.fc33
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-11-27 01:23:10 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Seppo Yli-Olli 2020-11-01 15:27:33 UTC
Description of problem:
grpcio in Fedora 33 is not usable, hangs when trying to connect over TLS, causing https://gitlab.com/BuildStream/buildstream/-/issues/1404

Version-Release number of selected component (if applicable):
1.26.0-6

How reproducible:
Always

Steps to Reproduce:
1. Install python3-grpcio
2. Run sample in additional info

Actual results:
"Hello world" outputted

Expected results:
Hang with 100% CPU usage

Additional info:
Minimal reproducing sample. This is using our CAS, feel free to adapt to your own for local testing. Seems to also repro with insecure_channel.

import grpc

if __name__ == "__main__":
    credentials = grpc.ssl_channel_credentials()
    channel = grpc.secure_channel("freedesktop-sdk-cache.codethink.co.uk:11001",
                                  credentials=credentials)
    print ("Hello world")

Comment 1 John F Sullivan 2020-11-09 02:00:33 UTC
Reproducible using the gRPC quickstart example:  git clone -b v1.26.0 https://github.com/grpc/grpc

   [vagrant@vanilla helloworld]$ rpm -qa | grep -e grpc -e protobuf | sort -u
   grpc-1.26.0-6.fc33.x86_64
   protobuf-3.12.4-1.fc33.x86_64
   protobuf-c-1.3.3-3.fc33.x86_64
   python3-grpcio-1.26.0-6.fc33.x86_64
   python3-protobuf-3.12.4-1.fc33.noarch

I modified the server example to use pytest's debugger to help figure out where the hang is occurring.  The server hangs with the invocation of cygrpc.CompletionQueue():


   (Pdb) bt
     /home/vagrant/Projects/grpc/examples/python/helloworld/greeter_server.py(44)<module>()
   -> serve()
     /home/vagrant/Projects/grpc/examples/python/helloworld/greeter_server.py(35)serve()
   -> server = grpc.server(tpe)
     /usr/lib64/python3.9/site-packages/grpc/__init__.py(1939)server()
   -> return _server.create_server(thread_pool, ()
     /usr/lib64/python3.9/site-packages/grpc/_server.py(985)create_server()
   -> return _Server(thread_pool, generic_rpc_handlers, interceptors, options,
   > /usr/lib64/python3.9/site-packages/grpc/_server.py(942)__init__()
   -> completion_queue = cygrpc.CompletionQueue()
   (Pdb) s

The client hangs with the invocation of cygrpc.Channel():

   (Pdb) bt
     /home/vagrant/Projects/grpc/examples/python/helloworld/greeter_client.py(39)<module>()
   -> run()
     /home/vagrant/Projects/grpc/examples/python/helloworld/greeter_client.py(31)run()
   -> with grpc.insecure_channel('localhost:50051') as channel:
     /usr/lib64/python3.9/site-packages/grpc/__init__.py(1855)insecure_channel()
   -> return _channel.Channel(target, ()
   > /usr/lib64/python3.9/site-packages/grpc/_channel.py(1345)__init__()
   -> self._channel = cygrpc.Channel(
   (Pdb) s

Note that neither hangs if the python3-grpcio RPM is removed, but the Python package is installed using pip:

   pip install grpcio==1.26.0

Comment 2 Filippe LeMarchand 2020-11-12 21:05:15 UTC
I think this might be caused by LTO, since building grpc package with `%define _lto_cflags %{nil}` removes the issue.
It worth noting that I tried updating grpc upstream release first, so the LTO case was tested on 1.33.2 (with LTO enabled, it gives the same issue as current 1.26.0).

Comment 3 Artem 2020-11-13 06:00:46 UTC
Pull request: https://src.fedoraproject.org/rpms/grpc/pull-request/2

Comment 4 John F Sullivan 2020-11-16 01:47:03 UTC
I tested the gRPC Python examples referenced in comment 1 using a grpc and python3-grpcio RPM built from the patch referenced in comment 3 and confirmed that with the changes, the greeter_server.py and greeter_client.py examples work as expected.  Thank you!

Comment 5 Artem 2020-11-19 12:25:25 UTC
*** Bug 1886153 has been marked as a duplicate of this bug. ***

Comment 6 Fedora Update System 2020-11-26 15:08:02 UTC
FEDORA-2020-5af900489d has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-5af900489d

Comment 7 Fedora Update System 2020-11-27 01:23:10 UTC
FEDORA-2020-5af900489d has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.