Bug 1109488

Summary: python-qpid-0.28 hangs when using with glance (eventlet)
Product: [Fedora] Fedora EPEL Reporter: Attila Fazekas <afazekas>
Component: python-qpidAssignee: Ted Ross <tross>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: epel7CC: afazekas, bmbouter, tross
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-qpid-1.37.0-1.el7 python-qpid-1.37.0-1.fc26 python-qpid-1.37.0-1.fc27 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-01-13 22:34:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
bt-g-api
none
all-glance-thread.txt none

Description Attila Fazekas 2014-06-14 12:41:16 UTC
Created attachment 908781 [details]
bt-g-api

Description of problem:
openstack-glance by using devstack hangs at installation time.
When devstack tries to upload an image.

devstack issues a similar command:
$ glance --os-auth-token $TONE --os-image-url http://172.16.32.18:9292 image-create --name cirros-0.3.2-x86_64-uec-kernel --is-public True --container-format aki --disk-format aki


Version-Release number of selected component (if applicable):
python-qpid-0.28-1.el7.noarch

How reproducible:
always

Steps to Reproduce:
1. create a stack user and /opt/stck/new directory owned by the stack user.
2. git clone https://github.com/openstack-dev/devstack.git
3. create local.conf in devstack directory like:
[[local|localrc]]
TEMPEST_ALLOW_TENANT_ISOLATION='True'
SERVICE_PASSWORD='secret'
API_RATE_LIMIT='False'
TERMINATE_TIMEOUT='60'
SYSLOG='False'
ENABLED_SERVICES='g-api,g-reg,key,n-api,n-crt,n-obj,n-cpu,n-sch,horizon,sysstat,tempest,s-proxy,s-account,s-container,s-object,cinder,c-api,c-vol,c-sch,c-bak,n-cond,heat,h-api,h-api-cfn,h-api-cw,h-eng,ceilometer-acompute,ceilometer-acentral,ceilometer-collector,ceilometer-api,trove,tr-api,tr-tmgr,tr-cond,qpid,mysql,quantum,q-svc,q-agt,q-dhcp,q-l3,q-meta,q-lbaas,q-vpn,q-metering,q-fwaas,neutron,sahara'
ADMIN_PASSWORD='secret'
SERVICE_HOST='172.16.32.18'
ROOTSLEEP='0'
RHEL6_RDO_REPO_ID='openstack-icehouse'
HEAT_CREATE_TEST_IMAGE='False'
FIXED_RANGE='10.1.0.0/24'
QPID_HOST='172.16.32.18'
TEMPEST_HTTP_IMAGE='http://google.com'
NOVNC_FROM_PACKAGE='False'
RHEL6_RDO_REPO_RPM='http://repos.fedorapeople.org/repos/openstack/openstack-icehouse/rdo-release-icehouse-2.noarch.rpm'
DEFAULT_IMAGE_NAME='cirros-0.3.2-x86_64-uec'
PIP_USE_MIRRORS='True'
LOGFILE='/opt/stack/new/devstacklog.txt'
SWIFT_REPLICAS='1'
MYSQL_PASSWORD='secret'
CEILOMETER_PIPELINE_INTERVAL='15'
DEST='/opt/stack/new'
CINDER_PERIODIC_INTERVAL='10'
FIXED_NETWORK_SIZE='256'
SWIFT_LOOPBACK_DISK_SIZE='6G'
BOOT_TIMEOUT='90'
VOLUME_BACKING_FILE_SIZE='16G'
FORCE='yes'
RABBIT_PASSWORD='secret'
export OS_NO_CACHE='True'
VERBOSE='True'
SERVICE_TOKEN='111222333444'
ACTIVE_TIMEOUT='90'
CINDER_SECURE_DELETE='False'
ASSOCIATE_TIMEOUT='60'
SWIFT_HASH='1234123412341234'
NETWORK_GATEWAY='10.1.0.1'
SCREEN_LOGDIR='/opt/stack/new/screen-logs'
DATABASE_PASSWORD='secret'
VIRT_DRIVER='libvirt'
LOG_COLOR='False'
FLOATING_RANGE='172.24.4.0/24'
4. execute the stack.sh as the 'stack' user.

Additional info:
Workaround is downgrading the python-qpid:
sudo pip install -U qpid-python==0.26

gdb after installing the python debuginfo is able to print stack traces including the python line number.
One glance api process bt attached as an example. May be it hanged at the glace registry (g-reg).

The used eventlet and greenlet version according to the pip list are:
eventlet (0.14.0)
greenlet (0.4.2)

Comment 1 Darryl L. Pierce 2014-06-16 12:47:43 UTC
Can you provide the stacktrace that includes the Python line numbers?

Comment 2 Attila Fazekas 2014-06-18 13:11:38 UTC
The already attached file includes:
'file /usr/lib/python2.7/site-packages/qpid/compat.py, line 80, in select (rlist=set([<...>]), wlist=set([]), xlist=(), timeout=None, fd_count=1, rset=set([<...>]), wset=set([]),'


Note: glance api does not uses the event let monkey patch on all modules.

Comment 3 Attila Fazekas 2014-06-18 13:12:55 UTC
Created attachment 910005 [details]
all-glance-thread.txt

Comment 4 Darryl L. Pierce 2014-06-20 20:11:26 UTC
Okay, I think I'm missing something. I'm not sure what's the expected vs. what's actually happening here. There's a lot of text in the two attachments to parse, so can you provide those two answers (they're not in the original description) to help me know what I should be looking for?

Comment 5 Attila Fazekas 2014-07-03 11:27:13 UTC
http://svn.apache.org/viewvc/qpid/trunk/qpid/python/qpid/compat.py?r1=1598586&r2=1598585&pathrev=1598586

This above change expected to fix the issue, and it is not in the python-qpid-0.28 package.

(The poll expected to work with eventlet, would be nice to know more about why it did not worked without that patch, BTW. the epoll is the preferred event handling way on Linux.)

svn update -r r1598353 # not working
r1598586 and r1607590 # worked

Please backport the above patch or update the package to an even newer version.

Comment 6 Attila Fazekas 2014-07-03 11:27:46 UTC
http://svn.apache.org/viewvc/qpid/trunk/qpid/python/qpid/compat.py?r1=1598586&r2=1598585&pathrev=1598586

This above change expected to fix the issue, and it is not in the python-qpid-0.28 package.

(The poll expected to work with eventlet, would be nice to know more about why it did not worked without that patch, BTW. the epoll is the preferred event handling way on Linux.)

svn update -r r1598353 # not working
r1598586 and r1607590 # worked

Please backport the above patch or update the package to an even newer version.

Comment 7 Fedora Update System 2014-07-07 16:00:45 UTC
python-qpid-0.28-3.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/python-qpid-0.28-3.fc20

Comment 8 Fedora Update System 2014-07-07 16:16:41 UTC
python-qpid-0.28-3.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/python-qpid-0.28-3.fc19

Comment 9 Fedora Update System 2014-07-08 00:59:54 UTC
Package python-qpid-0.28-3.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing python-qpid-0.28-3.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-8145/python-qpid-0.28-3.fc20
then log in and leave karma (feedback).

Comment 10 Fedora Update System 2014-11-13 18:49:36 UTC
python-qpid-0.30-2.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/python-qpid-0.30-2.fc20

Comment 12 Fedora Update System 2017-12-01 18:58:43 UTC
python-qpid-1.37.0-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-0233f4ea2e

Comment 13 Fedora Update System 2017-12-01 18:58:59 UTC
python-qpid-1.37.0-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-432cb6bb51

Comment 14 Fedora Update System 2017-12-01 18:59:08 UTC
python-qpid-1.37.0-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-81c89b85c0

Comment 15 Fedora Update System 2017-12-02 19:56:30 UTC
python-qpid-1.37.0-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-0233f4ea2e

Comment 16 Fedora Update System 2017-12-02 21:48:02 UTC
python-qpid-1.37.0-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-81c89b85c0

Comment 17 Fedora Update System 2017-12-02 22:40:21 UTC
python-qpid-1.37.0-1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-432cb6bb51

Comment 18 Fedora Update System 2018-01-13 22:34:12 UTC
python-qpid-1.37.0-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.

Comment 19 Fedora Update System 2018-01-13 23:04:17 UTC
python-qpid-1.37.0-1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 20 Fedora Update System 2018-01-13 23:18:26 UTC
python-qpid-1.37.0-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.