Bug 1539608
| Summary: | [ovn]ipv6 dns server works abnormally | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | haidong li <haili> | ||||||||
| Component: | openvswitch | Assignee: | Mark Michelson <mmichels> | ||||||||
| Status: | CLOSED ERRATA | QA Contact: | haidong li <haili> | ||||||||
| Severity: | medium | Docs Contact: | |||||||||
| Priority: | high | ||||||||||
| Version: | 7.5 | CC: | atragler, fleitner, kzhang, mmichels, nusiddiq, pvauter | ||||||||
| Target Milestone: | rc | ||||||||||
| Target Release: | --- | ||||||||||
| Hardware: | x86_64 | ||||||||||
| OS: | Linux | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2018-03-19 10:22:13 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Bug Depends On: | |||||||||||
| Bug Blocks: | 1475436 | ||||||||||
| Attachments: |
|
||||||||||
|
Description
haidong li
2018-01-29 10:22:02 UTC
Hello, The thing that worries me right now are that in your IPv6 tests, I see "A?" in the dump. This implies that the DNS request is only querying for A records. If you try to `dig` or `nslookup` hv0_vm00_vnet2.org then do you see any AAAA records listed? (In reply to Mark Michelson from comment #2) > Hello, > > The thing that worries me right now are that in your IPv6 tests, I see "A?" > in the dump. This implies that the DNS request is only querying for A > records. If you try to `dig` or `nslookup` hv0_vm00_vnet2.org then do you > see any AAAA records listed? Hi Mark, Yes,there is AAAA if I use ping6 command.I tested it again with another domain name abc.org,and use nslookup here: [root@localhost ~]# cat /etc/resolv.conf ; generated by /usr/sbin/dhclient-script nameserver 2001:db8:102::254 [root@localhost ~]# ping6 abc.org 01:24:15.392604 IP6 2001:db8:102::11.58452 > 2001:db8:102::254.53: 52023+ AAAA? abc.org. (25) 01:24:15.393142 IP6 2001:db8:102::254.53 > 2001:6d2f:102::11.58452: 52023- 1/0/0 (25) 01:24:15.393169 IP6 2001:db8:102::11 > 2001:db8:102::254: ICMP6, destination unreachable, unreachable route 2001:6d2f:102::11, length 81 01:24:20.397669 IP6 2001:db8:102::11.58452 > 2001:db8:102::254.53: 52023+ AAAA? abc.org. (25) 01:24:20.398149 IP6 2001:db8:102::254.53 > 2001:6d2f:102::11.58452: 52023- 1/0/0 (25) 01:24:20.398168 IP6 2001:db8:102::11 > 2001:db8:102::254: ICMP6, destination unreachable, unreachable route 2001:6d2f:102::11, length 81 01:24:25.402862 IP6 2001:db8:102::11.56189 > 2001:db8:102::254.53: 73+ AAAA? abc.org.localdomain. (37) 01:24:30.407934 IP6 2001:db8:102::11.56189 > 2001:db8:102::254.53: 73+ AAAA? abc.org.localdomain. (37) ping: abc.org: Name or service not known [root@localhost ~]# [root@localhost ~]# cat /etc/resolv.conf ; generated by /usr/sbin/dhclient-script options inet6 nameserver 2001:db8:102::254 [root@localhost ~]# ping6 abc.org 01:25:21.666279 IP6 2001:db8:102::11.51854 > 2001:db8:102::254.53: 40144+ AAAA? abc.org. (25) 01:25:21.666768 IP6 2001:db8:102::254.53 > 2001:6d2f:102::11.51854: 40144- 1/0/0 (25) 01:25:21.666792 IP6 2001:db8:102::11 > 2001:db8:102::254: ICMP6, destination unreachable, unreachable route 2001:6d2f:102::11, length 81 01:25:26.671344 IP6 2001:db8:102::11.51854 > 2001:db8:102::254.53: 40144+ AAAA? abc.org. (25) 01:25:26.671792 IP6 2001:db8:102::254.53 > 2001:6d2f:102::11.51854: 40144- 1/0/0 (25) 01:25:26.671823 IP6 2001:db8:102::11 > 2001:db8:102::254: ICMP6, destination unreachable, unreachable route 2001:6d2f:102::11, length 81 01:25:26.675114 IP6 fe80::2de:adff:fe01:1 > 2001:db8:102::254: ICMP6, neighbor solicitation, who has 2001:db8:102::254, length 32 01:25:26.675524 IP6 2001:db8:102::254 > fe80::2de:adff:fe01:1: ICMP6, neighbor advertisement, tgt is 2001:db8:102::254, length 32 01:25:31.675179 IP6 2001:db8:102::11.35868 > 2001:db8:102::254.53: 6652+ AAAA? abc.org.localdomain. (37) 01:25:36.680291 IP6 2001:db8:102::11.35868 > 2001:db8:102::254.53: 6652+ AAAA? abc.org.localdomain. (37) ping: abc.org: Name or service not known [root@localhost ~]# [root@localhost ~]# nslookup abc.org 01:50:53.484008 IP6 2001:db8:102::11.51439 > 2001:db8:102::254.53: 53358+ A? abc.org. (25) 01:50:53.484575 IP6 2001:db8:102::254.53 > 2001:6d3b:102::11.51439: 53358- 1/0/0 (25) 01:50:53.484600 IP6 2001:db8:102::11 > 2001:db8:102::254: ICMP6, destination unreachable, unreachable route 2001:6d3b:102::11, length 81 01:50:58.483712 IP6 2001:db8:102::11.51439 > 2001:db8:102::254.53: 53358+ A? abc.org. (25) 01:50:58.484126 IP6 2001:db8:102::254.53 > 2001:6d3b:102::11.51439: 53358- 1/0/0 (25) 01:50:58.484140 IP6 2001:db8:102::11 > 2001:db8:102::254: ICMP6, destination unreachable, unreachable route 2001:6d3b:102::11, length 81 01:50:58.499117 IP6 fe80::2de:adff:fe01:1 > 2001:db8:102::254: ICMP6, neighbor solicitation, who has 2001:db8:102::254, length 32 01:50:58.499748 IP6 2001:db8:102::254 > fe80::2de:adff:fe01:1: ICMP6, neighbor advertisement, tgt is 2001:db8:102::254, length 32 01:51:03.483780 IP6 2001:db8:102::11.51439 > 2001:db8:102::254.53: 53358+ A? abc.org. (25) 01:51:03.484177 IP6 2001:db8:102::254.53 > 2001:6d3b:102::11.51439: 53358- 1/0/0 (25) 01:51:03.484190 IP6 2001:db8:102::11 > 2001:db8:102::254: ICMP6, destination unreachable, unreachable route 2001:6d3b:102::11, length 81 ;; connection timed out; trying next origin 01:51:08.483999 IP6 2001:db8:102::11.58139 > 2001:db8:102::254.53: 303+ A? abc.org. (25) 01:51:08.484428 IP6 2001:db8:102::254.53 > 2001:6d3b:102::11.58139: 303- 1/0/0 (25) 01:51:08.484444 IP6 2001:db8:102::11 > 2001:db8:102::254: ICMP6, destination unreachable, unreachable route 2001:6d3b:102::11, length 81 ;; connection timed out; no servers could be reached [root@localhost ~]# ping6 2001:db8:102::11 PING 2001:db8:102::11(2001:db8:102::11) 56 data bytes 64 bytes from 2001:db8:102::11: icmp_seq=1 ttl=64 time=0.064 ms 64 bytes from 2001:db8:102::11: icmp_seq=2 ttl=64 time=0.017 ms 64 bytes from 2001:db8:102::11: icmp_seq=3 ttl=64 time=0.017 ms --- 2001:db8:102::11 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2000ms rtt min/avg/max/mdev = 0.017/0.032/0.064/0.023 ms [root@localhost ~]# then I changed the server to ipv4,it works normal: [root@localhost ~]# cat /etc/resolv.conf options inet6 ; generated by /usr/sbin/dhclient-script nameserver 172.16.102.254 [root@localhost ~]# [root@localhost ~]# ping6 abc.org 01:31:32.463970 IP 172.16.102.11.41846 > 172.16.102.254.53: 305+ AAAA? abc.org. (25) 01:31:32.464504 IP 172.16.102.254.53 > 172.16.102.11.41846: 305- 1/0/0 AAAA 2001:db8:102::11 (60) 01:31:32.465281 IP 172.16.102.11.50106 > 172.16.102.254.53: 49245+ PTR? 1.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.1.0.8.b.d.0.1.0.0.2.ip6.arpa. (90) 01:31:37.466149 IP 172.16.102.11.50106 > 172.16.102.254.53: 49245+ PTR? 1.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.1.0.8.b.d.0.1.0.0.2.ip6.arpa. (90) PING abc.org(localhost.localdomain (2001:db8:102::11)) 56 data bytes 01:31:42.472563 IP 172.16.102.11.58291 > 172.16.102.254.53: 32121+ PTR? 1.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.1.0.8.b.d.0.1.0.0.2.ip6.arpa. (90) 01:31:47.477654 IP 172.16.102.11.58291 > 172.16.102.254.53: 32121+ PTR? 1.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.1.0.8.b.d.0.1.0.0.2.ip6.arpa. (90) 64 bytes from localhost.localdomain (2001:db8:102::11): icmp_seq=1 ttl=64 time=0.055 ms 64 bytes from localhost.localdomain (2001:db8:102::11): icmp_seq=2 ttl=64 time=0.038 ms 01:31:52.483078 ARP, Request who-has 172.16.102.254 tell 172.16.102.11, length 28 01:31:52.483311 ARP, Reply 172.16.102.254 is-at 00:de:ad:ff:01:02, length 28 64 bytes from localhost.localdomain (2001:db8:102::11): icmp_seq=3 ttl=64 time=0.019 ms --- abc.org ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 11009ms rtt min/avg/max/mdev = 0.019/0.037/0.055/0.015 ms [root@localhost ~]# nslookup abc.org 01:49:56.554857 IP 172.16.102.11.36968 > 172.16.102.254.53: 3326+ A? abc.org. (25) 01:49:56.555331 IP 172.16.102.254.53 > 172.16.102.11.36968: 3326- 1/0/0 A 172.16.102.11 (48) Server: 172.16.102.254 Address: 172.16.102.254#53 Non-authoritative answer: Name: abc.org Address: 172.16.102.11 [root@localhost ~] [root@dell-per730-49 ovn]# ovn-nbctl list DNS _uuid : 5d5193ba-61f8-4881-bccc-6c9d885fe8f8 external_ids : {} records : {abc.org="172.16.102.11 2001:db8:102::11"} [root@dell-per730-49 ovn]# Hello, In your tests, I see: [root@localhost ~]# ping6 abc.org 01:25:21.666279 IP6 2001:db8:102::11.51854 > 2001:db8:102::254.53: 40144+ AAAA? abc.org. (25) 01:25:21.666768 IP6 2001:db8:102::254.53 > 2001:6d2f:102::11.51854: 40144- 1/0/0 (25) 01:25:21.666792 IP6 2001:db8:102::11 > 2001:db8:102::254: ICMP6, destination unreachable, unreachable route 2001:6d2f:102::11, length 81 Notice the 2001:6d2f:102::11 address that the DNS answer is being sent to. Compare that to the IPv4 case: [root@localhost ~]# ping6 abc.org 01:31:32.463970 IP 172.16.102.11.41846 > 172.16.102.254.53: 305+ AAAA? abc.org. (25) 01:31:32.464504 IP 172.16.102.254.53 > 172.16.102.11.41846: 305- 1/0/0 AAAA 2001:db8:102::11 (60) Here, the DNS answer is being sent back to the same address that requested it. I *think* I might know what is causing this. I believe there is a bug in the DNS handling code that assumes that the incoming DNS request came in over IPv4. The code attempts to set the IPv4 header's length and checksum fields, but this ends up writing to other fields in an IPv6 header. Looking at the testsuite, I see that all of the DNS lookup tests use DNS over IPv4. It is likely that nobody ever tested sending DNS requests over IPv6. (In reply to Mark Michelson from comment #4) > Hello, > > In your tests, I see: > > [root@localhost ~]# ping6 abc.org > 01:25:21.666279 IP6 2001:db8:102::11.51854 > 2001:db8:102::254.53: 40144+ > AAAA? abc.org. (25) > 01:25:21.666768 IP6 2001:db8:102::254.53 > 2001:6d2f:102::11.51854: 40144- > 1/0/0 (25) > 01:25:21.666792 IP6 2001:db8:102::11 > 2001:db8:102::254: ICMP6, destination > unreachable, unreachable route 2001:6d2f:102::11, length 81 > > Notice the 2001:6d2f:102::11 address that the DNS answer is being sent to. > Compare that to the IPv4 case: > > [root@localhost ~]# ping6 abc.org > 01:31:32.463970 IP 172.16.102.11.41846 > 172.16.102.254.53: 305+ AAAA? > abc.org. (25) > 01:31:32.464504 IP 172.16.102.254.53 > 172.16.102.11.41846: 305- 1/0/0 AAAA > 2001:db8:102::11 (60) > > Here, the DNS answer is being sent back to the same address that requested > it. > > I *think* I might know what is causing this. I believe there is a bug in the > DNS handling code that assumes that the incoming DNS request came in over > IPv4. The code attempts to set the IPv4 header's length and checksum fields, > but this ends up writing to other fields in an IPv6 header. Looking at the > testsuite, I see that all of the DNS lookup tests use DNS over IPv4. It is > likely that nobody ever tested sending DNS requests over IPv6. That's right. When I worked on it and wrote the tests, I couldn't test over IPv6. I have created a fix for this and submitted it to OVS. The patch was merged upstream. I have requested for the change to be backported to the OVS 2.8 branch. I am moving this to the "POST" status, and once it is backported to OVS 2.8, I will change to "MODIFIED". This has now been backported to 2.8. Hi Mark, I have tried with the version 2.9, the dns server can reply packet to correct address now.But the guest in my environment still can't ping success.It seems the guest doesn't ping the ip address translated by dns server.Do you know what's the reason? Thanks! [root@localhost ~]# uname -a Linux localhost.localdomain 3.10.0-845.el7.x86_64 #1 SMP Mon Feb 5 07:43:47 EST 2018 x86_64 x86_64 x86_64 GNU/Linux [root@localhost ~]# [root@localhost ~]# cat /etc/resolv.conf ; generated by /usr/sbin/dhclient-script nameserver 2001:db8:102::254 [root@localhost ~]# [root@localhost ~]# ping -c10 abc.org 08:46:41.800846 IP6 2001:db8:102::12.50257 > 2001:db8:102::254.53: 34728+ A? abc.org. (25) 08:46:41.801712 IP6 2001:db8:102::254.53 > 2001:db8:102::12.50257: 34728- 1/0/0 A 172.16.102.11 (48) 08:46:46.806007 IP6 2001:db8:102::12.50257 > 2001:db8:102::254.53: 34728+ A? abc.org. (25) 08:46:46.806482 IP6 2001:db8:102::254.53 > 2001:db8:102::12.50257: 34728- 1/0/0 A 172.16.102.11 (48) 08:46:46.810917 IP6 fe80::2de:adff:fe01:101 > 2001:db8:102::254: ICMP6, neighbor solicitation, who has 2001:db8:102::254, length 32 08:46:46.811502 IP6 2001:db8:102::254 > fe80::2de:adff:fe01:101: ICMP6, neighbor advertisement, tgt is 2001:db8:102::254, length 32 08:46:51.811221 IP6 2001:db8:102::12.38529 > 2001:db8:102::254.53: 27956+ A? abc.org.localdomain. (37) 08:46:56.816346 IP6 2001:db8:102::12.38529 > 2001:db8:102::254.53: 27956+ A? abc.org.localdomain. (37) ping: abc.org: Name or service not known [root@localhost ~]# ping6 -c10 abc.org 08:48:39.284912 IP6 2001:db8:102::12.53408 > 2001:db8:102::254.53: 36782+ AAAA? abc.org. (25) 08:48:39.285432 IP6 2001:db8:102::254.53 > 2001:db8:102::12.53408: 36782- 1/0/0 AAAA 2001:db8:102::11 (60) 08:48:44.289996 IP6 2001:db8:102::12.53408 > 2001:db8:102::254.53: 36782+ AAAA? abc.org. (25) 08:48:44.290299 IP6 2001:db8:102::254.53 > 2001:db8:102::12.53408: 36782- 1/0/0 AAAA 2001:db8:102::11 (60) 08:48:44.298952 IP6 fe80::2de:adff:fe01:101 > 2001:db8:102::254: ICMP6, neighbor solicitation, who has 2001:db8:102::254, length 32 08:48:44.299438 IP6 2001:db8:102::254 > fe80::2de:adff:fe01:101: ICMP6, neighbor advertisement, tgt is 2001:db8:102::254, length 32 08:48:49.295221 IP6 2001:db8:102::12.45208 > 2001:db8:102::254.53: 10057+ AAAA? abc.org.localdomain. (37) 08:48:54.300322 IP6 2001:db8:102::12.45208 > 2001:db8:102::254.53: 10057+ AAAA? abc.org.localdomain. (37) ping: abc.org: Name or service not known [root@localhost ~]# Hello Haidong Li, Can you provide a pcap file for when you attempted this? I want to see what records the DNS server is returning. Thank you. Created attachment 1402202 [details]
pcap of ping4 packet
Created attachment 1402203 [details]
pcp of ping6 packet
Hi Mark,
I have attached the packets of ping4 and ping6,please check it,thanks.
Thanks for the pcaps. I thought I had put a comment on here last week, but I apparently never clicked the "save changes". Sorry about that. In the pcap, everything looks mostly normal, except at the very end. OVN receives a DNS request for abc.org, and then OVN sends an answer with the configured IP address. The client then sends a DNS request for "abc.org.localdomain". This DNS request does not get answered. I don't understand why the client is sending a request for "abc.org.localdomain", but I suspect that is what is causing the ping to fail. Created attachment 1405116 [details]
pcap of server_ipv4 packet
I have checked the packet I sent last time in wireshark,I found the packet responded by dns server is marked as error,it seems the udp checksum is illegal. Then I captured a packet responded by ipv4 dns server which can work correctly,and attach it here.The udp checksum is also 0x0000,but it is normal displayed in wireshark with not error.
Hi, You are correct about the UDP checksum. With IPv4, the UDP checksum is optional. With IPv6, the UDP checksum is required. I have submitted a patch to the OVS mailing list which should fix the issue: https://patchwork.ozlabs.org/patch/882655/ Since the original problem of incorrect replied address is resolved according to comment 11,change the bug to verified.I will open another bug to trace the checksum thing. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0550 |