Bug 1612971 - Metadata service takes a long time to response
Summary: Metadata service takes a long time to response
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-eventlet
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z2
: 13.0 (Queens)
Assignee: Jon Schlueter
QA Contact: Eran Kuris
URL:
Whiteboard:
Depends On: 1504279 1628772
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-08-06 16:05 UTC by Daniel Alvarez Sanchez
Modified: 2021-12-10 17:10 UTC (History)
16 users (show)

Fixed In Version: python-eventlet-0.20.1-5.1.el7ost
Doc Type: Bug Fix
Doc Text:
Due to a bug in eventlet, systems that did not configure any nameservers (or unreachable) and relied only on hosts file for name resolution hit a delay when booting instances. This is because of an attempt at resolving the IPv6 entry if only an IPv4 host was specified. With this fix, eventlet returns immediately without attempting to use network resolution if at least one of the entries is present in the hosts file.
Clone Of: 1504279
Environment:
Last Closed: 2018-08-29 16:33:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github rdo-common/python-eventlet/commit/5c9aff6d5635fcc9cb6b96d4b9ccd881b4099952 0 None None None 2019-11-21 05:34:30 UTC
Launchpad 1696094 0 None None None 2018-08-06 16:05:22 UTC
Launchpad 1785615 0 None None None 2018-08-06 16:05:22 UTC
Red Hat Issue Tracker OSP-11555 0 None None None 2021-12-10 17:10:36 UTC
Red Hat Product Errata RHBA-2018:2573 0 None None None 2018-08-29 16:35:03 UTC

Comment 1 Alfredo Moralejo 2018-08-08 08:51:01 UTC
Fixed in RDO by included in https://github.com/rdo-common/python-eventlet/commit/5c9aff6d5635fcc9cb6b96d4b9ccd881b4099952 python-eventlet-0.20.1-3.el7

Comment 11 Joanne O'Flynn 2018-08-13 11:18:51 UTC
This bug is marked for inclusion in the errata but does not currently contain draft documentation text. To ensure the timely release of this advisory please provide draft documentation text for this bug as soon as possible.

If you do not think this bug requires errata documentation, set the requires_doc_text flag to "-".


To add draft documentation text:

* Select the documentation type from the "Doc Type" drop down field.

* A template will be provided in the "Doc Text" field based on the "Doc Type" value selected. Enter draft text in the "Doc Text" field.

Comment 14 Eran Kuris 2018-08-19 12:09:34 UTC
Fix verified:
OSP13 puddle 2018-08-16.1
rpm -qa | grep python-eventlet
python-eventlet-0.20.1-5.1.el7ost.noarch


I curl the metadata  I got it immediately, no delay.

Comment 15 Daniel Alvarez Sanchez 2018-08-20 07:57:42 UTC
Hi Eran,
Curl is not using eventlet so we're not exercising the fix in this bug.
The right way to do it is either through metadata service or through a python script importing eventlet and monkey patching. I attached a reproducer earlier:


--- Additional comment from Daniel Alvarez Sanchez on 2018-08-01 12:04:02 EDT ---


Here is a reproducer:

import eventlet
eventlet.monkey_patch()

import socket
print socket.getaddrinfo('overcloud.internalapi.localdomain', 80, 0, socket.SOCK_STREAM)



For actually verifying this bug you need to add overcloud.internalapi.localdomain to /etc/hosts and remove all nameservers from /etc/resolv.conf. Timing the above script without the fix used to take over 30 seconds. If the fix is applied it should be way faster (1-2 seconds).
Let me know if you need any help.

Thanks,
Daniel

Comment 16 Daniel Alvarez Sanchez 2018-08-20 10:06:43 UTC
Thanks Eran for giving me access to your setup.

Metadata agent container:

()[root@controller-0 /]# rpm -qa | grep eventlet
python-eventlet-0.20.1-5.1.el7ost.noarch


Empty resolv.conf:

()[root@controller-0 /]# cat /etc/resolv.conf
# Generated by NetworkManager


()[root@controller-0 /]# cat test.py
import eventlet
eventlet.monkey_patch()

import socket
print socket.getaddrinfo('overcloud.internalapi.localdomain', 80, 0, socket.SOCK_STREAM)



()[root@controller-0 /]# time python test.py
[(2, 1, 6, '', ('172.21.33.14', 80))]

real    0m0.166s
user    0m0.126s
sys     0m0.040s


We can leave this as VERIFIED

Comment 17 Eran Kuris 2018-08-20 10:12:36 UTC
After checked again with Daniel script it looks like the fix is working 
thanks for help Daniel.

()[root@controller-0 /]# time python test.py                                                                                                                                      │············
[(2, 1, 6, '', ('172.21.33.14', 80))]                                                                                                                                             │············
                                                                                                                                                                                  │············
real    0m0.166s                                                                                                                                                                  │············
user    0m0.126s                                                                                                                                                                  │············
sys     0m0.040s     
()[root@controller-0 /]# cat /etc/resolv.conf                                                                                                                                     │············
# Generated by NetworkManager                                                                                                                                                     │············
()[root@controller-0 /]# # empty resolv.conf, and python script using eventlet worked fine  

()[root@controller-0 /]# rpm -qa | grep eventlet                                                                                                                                  │············
python-eventlet-0.20.1-5.1.el7ost.noarch

Comment 19 errata-xmlrpc 2018-08-29 16:33:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2573


Note You need to log in before you can comment on or make changes to this bug.