Bug 1607967 - UnexpectedSource: got a response from ('2001:4860:4860:0:0:0:0:8888', 53, 0, 0) instead of ('2001:4860:4860::8888', 53, 0, 0)
Summary: UnexpectedSource: got a response from ('2001:4860:4860:0:0:0:0:8888', 53, 0, ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-eventlet
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: z2
: 13.0 (Queens)
Assignee: Lon Hohberger
QA Contact: Shai Revivo
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-24 15:54 UTC by David Vallee Delisle
Modified: 2021-12-10 16:50 UTC (History)
11 users (show)

Fixed In Version: python-eventlet-0.20.1-5.1.el7ost
Doc Type: Bug Fix
Doc Text:
There was an issue in python-eventlet UDP address handling that resulted in some IPv6 addresses being handled incorrectly in some cases. As a result, when receiving DNS responses via UDP, python-eventlet ignored the response and stalled for several seconds, severely impacting performance. This issue is now resolved.
Clone Of:
Environment:
Last Closed: 2018-08-29 16:33:16 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Backport (part 1) (2.44 KB, patch)
2018-07-31 18:58 UTC, Lon Hohberger
no flags Details | Diff
Backport (part 2) (1.99 KB, patch)
2018-07-31 18:59 UTC, Lon Hohberger
no flags Details | Diff
Unit tests (part 3) (2.77 KB, patch)
2018-07-31 18:59 UTC, Lon Hohberger
no flags Details | Diff


Links
System ID Private Priority Status Summary Last Updated
Github eventlet eventlet pull 510 0 None None None 2018-08-21 19:56:52 UTC
Github eventlet eventlet pull 510/commits/a2d40db2074ddb8039e414fa969c13aca006590e 0 None None None 2018-08-21 19:56:10 UTC
RDO 15293 0 None master: MERGED rdoinfo: Promote new eventlet with backport for IPv6 issue (I4b06d2a63d1a4563e812794380dd36412ba9bace) 2018-08-21 20:03:40 UTC
Red Hat Issue Tracker OSP-11482 0 None None None 2021-12-10 16:50:06 UTC
Red Hat Product Errata RHBA-2018:2573 0 None None None 2018-08-29 16:34:05 UTC

Description David Vallee Delisle 2018-07-24 15:54:18 UTC
Description of problem:

When using short notation in resolv.conf, dns resolution times out after 30 seconds

Version-Release number of selected component (if applicable):


How reproducible:
All the time

Steps to Reproduce:
1. Use short notation in resolv.conf
2. Snoop dns queries with tcpdump
3. Find a bunch of NXRECORD everytime a hostname is looked up, thus slowing down deployment and other functions
~~~
82	2018-07-06 14:17:03.637557	2001:4860:4860:0:0:0:0:8888	2001:4860:4860:0:0:0:0:8888	DNS	199	Standard query response 0x2c34 No such name AAAA blabla.com SOA blablabla.com
~~~

Actual results:
Everytime a hostname lookup is executed, it takes 30 seconds before falling back on ipv4 because the reply is ignored because the source is considered as being different, even though it's the right source.

Expected results:
We should accept results that that match the source, no matter if we use short or long notation in resolv.conf

Additional info:
I think we should probably develop a better validation than just == in eventlet/support/greendns.py:

~~~
627         while 1:
628             try:
629                 (wire, from_address) = s.recvfrom(65535)
630             except socket.timeout:
631                 # Q: Do we also need to catch coro.CoroutineSocketWake and pass?
632                 if expiration - time.time() <= 0.0:
633                     raise dns.exception.Timeout
634             if from_address == destination:
635                 break
636             if not ignore_unexpected:
637                 raise dns.query.UnexpectedSource(
638                     'got a response from %s instead of %s'
639                     % (from_address, destination))
~~~

Comment 2 David Vallee Delisle 2018-07-24 16:02:39 UTC
For information, our customer is having this error in the logs:

2018-07-24 12:39:48.515 55 ERROR dns.resolver [req-c444b6ff-a64c-41a8-bc81-801081c59832 - - - - -] : UnexpectedSource: got a response from ('2001:4860:4860:0:0:0:0:8888', 53, 0, 0) instead of ('2001:4860:4860::8888', 53, 0, 0)

Comment 5 David Vallee Delisle 2018-07-25 23:04:54 UTC
Something like this might be a better way to compare IPv6

~~~
Python 2.7.15 (default, May 15 2018, 15:37:31) 
[GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from socket import inet_pton, AF_INET6
>>> source = inet_pton(AF_INET6, "2001:4860:4860::8888")
>>> dest = inet_pton(AF_INET6, "2001:4860:4860:0:0:0:0:8888")
>>> if source == dest:
...     print("same")
... 
same
>>> print("Source %s Dest: %s", (source, dest))
('Source %s Dest: %s', (' \x01H`H`\x00\x00\x00\x00\x00\x00\x00\x00\x88\x88', ' \x01H`H`\x00\x00\x00\x00\x00\x00\x00\x00\x88\x88'))
~~~

Comment 10 Lon Hohberger 2018-07-31 18:25:04 UTC
Merged upstream: https://github.com/eventlet/eventlet/pull/510

Comment 11 Lon Hohberger 2018-07-31 18:58:51 UTC
Created attachment 1471908 [details]
Backport (part 1)

Comment 12 Lon Hohberger 2018-07-31 18:59:14 UTC
Created attachment 1471909 [details]
Backport (part 2)

Comment 13 Lon Hohberger 2018-07-31 18:59:41 UTC
Created attachment 1471910 [details]
Unit tests (part 3)

Comment 23 Joanne O'Flynn 2018-08-13 11:18:36 UTC
This bug is marked for inclusion in the errata but does not currently contain draft documentation text. To ensure the timely release of this advisory please provide draft documentation text for this bug as soon as possible.

If you do not think this bug requires errata documentation, set the requires_doc_text flag to "-".


To add draft documentation text:

* Select the documentation type from the "Doc Type" drop down field.

* A template will be provided in the "Doc Text" field based on the "Doc Type" value selected. Enter draft text in the "Doc Text" field.

Comment 31 Bernard Cafarelli 2018-08-27 15:11:04 UTC
eventlet upstream unit tests pass with the fix + no relevant regressions identified in OSP CI, so marking this VERIFIED

Comment 34 errata-xmlrpc 2018-08-29 16:33:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2573


Note You need to log in before you can comment on or make changes to this bug.