Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 1607967 - UnexpectedSource: got a response from ('2001:4860:4860:0:0:0:0:8888', 53, 0, 0) instead of ('2001:4860:4860::8888', 53, 0, 0)
UnexpectedSource: got a response from ('2001:4860:4860:0:0:0:0:8888', 53, 0, ...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-eventlet (Show other bugs)
13.0 (Queens)
Unspecified Unspecified
urgent Severity high
: z2
: 13.0 (Queens)
Assigned To: Lon Hohberger
Shai Revivo
: Triaged, ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2018-07-24 11:54 EDT by David Vallee Delisle
Modified: 2018-08-29 12:34 EDT (History)
11 users (show)

See Also:
Fixed In Version: python-eventlet-0.20.1-5.1.el7ost
Doc Type: Bug Fix
Doc Text:
There was an issue in python-eventlet UDP address handling that resulted in some IPv6 addresses being handled incorrectly in some cases. As a result, when receiving DNS responses via UDP, python-eventlet ignored the response and stalled for several seconds, severely impacting performance. This issue is now resolved.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2018-08-29 12:33:16 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Backport (part 1) (2.44 KB, patch)
2018-07-31 14:58 EDT, Lon Hohberger
no flags Details | Diff
Backport (part 2) (1.99 KB, patch)
2018-07-31 14:59 EDT, Lon Hohberger
no flags Details | Diff
Unit tests (part 3) (2.77 KB, patch)
2018-07-31 14:59 EDT, Lon Hohberger
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
RDO 15293 None master: MERGED rdoinfo: Promote new eventlet with backport for IPv6 issue (I4b06d2a63d1a4563e812794380dd36412ba9bace) 2018-08-21 16:03 EDT
Github eventlet/eventlet/pull/510 None None None 2018-08-21 15:56 EDT
Github eventlet/eventlet/pull/510/commits/a2d40db2074ddb8039e414fa969c13aca006590e None None None 2018-08-21 15:56 EDT
Red Hat Product Errata RHBA-2018:2573 None None None 2018-08-29 12:34 EDT

  None (edit)
Description David Vallee Delisle 2018-07-24 11:54:18 EDT
Description of problem:

When using short notation in resolv.conf, dns resolution times out after 30 seconds

Version-Release number of selected component (if applicable):


How reproducible:
All the time

Steps to Reproduce:
1. Use short notation in resolv.conf
2. Snoop dns queries with tcpdump
3. Find a bunch of NXRECORD everytime a hostname is looked up, thus slowing down deployment and other functions
~~~
82	2018-07-06 14:17:03.637557	2001:4860:4860:0:0:0:0:8888	2001:4860:4860:0:0:0:0:8888	DNS	199	Standard query response 0x2c34 No such name AAAA blabla.com SOA blablabla.com
~~~

Actual results:
Everytime a hostname lookup is executed, it takes 30 seconds before falling back on ipv4 because the reply is ignored because the source is considered as being different, even though it's the right source.

Expected results:
We should accept results that that match the source, no matter if we use short or long notation in resolv.conf

Additional info:
I think we should probably develop a better validation than just == in eventlet/support/greendns.py:

~~~
627         while 1:
628             try:
629                 (wire, from_address) = s.recvfrom(65535)
630             except socket.timeout:
631                 # Q: Do we also need to catch coro.CoroutineSocketWake and pass?
632                 if expiration - time.time() <= 0.0:
633                     raise dns.exception.Timeout
634             if from_address == destination:
635                 break
636             if not ignore_unexpected:
637                 raise dns.query.UnexpectedSource(
638                     'got a response from %s instead of %s'
639                     % (from_address, destination))
~~~
Comment 2 David Vallee Delisle 2018-07-24 12:02:39 EDT
For information, our customer is having this error in the logs:

2018-07-24 12:39:48.515 55 ERROR dns.resolver [req-c444b6ff-a64c-41a8-bc81-801081c59832 - - - - -] : UnexpectedSource: got a response from ('2001:4860:4860:0:0:0:0:8888', 53, 0, 0) instead of ('2001:4860:4860::8888', 53, 0, 0)
Comment 5 David Vallee Delisle 2018-07-25 19:04:54 EDT
Something like this might be a better way to compare IPv6

~~~
Python 2.7.15 (default, May 15 2018, 15:37:31) 
[GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from socket import inet_pton, AF_INET6
>>> source = inet_pton(AF_INET6, "2001:4860:4860::8888")
>>> dest = inet_pton(AF_INET6, "2001:4860:4860:0:0:0:0:8888")
>>> if source == dest:
...     print("same")
... 
same
>>> print("Source %s Dest: %s", (source, dest))
('Source %s Dest: %s', (' \x01H`H`\x00\x00\x00\x00\x00\x00\x00\x00\x88\x88', ' \x01H`H`\x00\x00\x00\x00\x00\x00\x00\x00\x88\x88'))
~~~
Comment 10 Lon Hohberger 2018-07-31 14:25:04 EDT
Merged upstream: https://github.com/eventlet/eventlet/pull/510
Comment 11 Lon Hohberger 2018-07-31 14:58 EDT
Created attachment 1471908 [details]
Backport (part 1)
Comment 12 Lon Hohberger 2018-07-31 14:59 EDT
Created attachment 1471909 [details]
Backport (part 2)
Comment 13 Lon Hohberger 2018-07-31 14:59 EDT
Created attachment 1471910 [details]
Unit tests (part 3)
Comment 23 Joanne O'Flynn 2018-08-13 07:18:36 EDT
This bug is marked for inclusion in the errata but does not currently contain draft documentation text. To ensure the timely release of this advisory please provide draft documentation text for this bug as soon as possible.

If you do not think this bug requires errata documentation, set the requires_doc_text flag to "-".


To add draft documentation text:

* Select the documentation type from the "Doc Type" drop down field.

* A template will be provided in the "Doc Text" field based on the "Doc Type" value selected. Enter draft text in the "Doc Text" field.
Comment 31 Bernard Cafarelli 2018-08-27 11:11:04 EDT
eventlet upstream unit tests pass with the fix + no relevant regressions identified in OSP CI, so marking this VERIFIED
Comment 34 errata-xmlrpc 2018-08-29 12:33:16 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2573

Note You need to log in before you can comment on or make changes to this bug.