Bug 713247
Summary: | virtual machines cannot speak to each other over IPv6 | ||||||
---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Jeff Bastian <jbastian> | ||||
Component: | libvirt | Assignee: | Laine Stump <laine> | ||||
Status: | CLOSED NOTABUG | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||
Severity: | medium | Docs Contact: | |||||
Priority: | medium | ||||||
Version: | 15 | CC: | aquini, berrange, clalance, crobinso, itamar, jforbes, laine, rdassen, veillard, virt-maint | ||||
Target Milestone: | --- | ||||||
Target Release: | --- | ||||||
Hardware: | Unspecified | ||||||
OS: | Linux | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2011-06-20 14:35:56 UTC | Type: | --- | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Attachments: |
|
Description
Jeff Bastian
2011-06-14 19:25:17 UTC
Also, ping6 to the host machine works: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [root@kirk ~]# ping6 -n -c 3 2001:d00:dad:1::1 PING 2001:d00:dad:1::1(2001:d00:dad:1::1) 56 data bytes 64 bytes from 2001:d00:dad:1::1: icmp_seq=1 ttl=64 time=0.228 ms 64 bytes from 2001:d00:dad:1::1: icmp_seq=2 ttl=64 time=0.263 ms 64 bytes from 2001:d00:dad:1::1: icmp_seq=3 ttl=64 time=0.335 ms --- 2001:d00:dad:1::1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1999ms rtt min/avg/max/mdev = 0.228/0.275/0.335/0.046 ms ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It's only to the other VM that it fails. Please attach the output of "ip6tables -S | grep ipv6test". [root@localhost ~]# ip6tables -S | grep ipv6test -A INPUT -i ipv6test -p udp -m udp --dport 53 -j ACCEPT -A INPUT -i ipv6test -p tcp -m tcp --dport 53 -j ACCEPT -A FORWARD -d 2001:d00:dad:1::/64 -o ipv6test -j ACCEPT -A FORWARD -s 2001:d00:dad:1::/64 -i ipv6test -j ACCEPT -A FORWARD -i ipv6test -o ipv6test -j ACCEPT -A FORWARD -o ipv6test -j REJECT --reject-with icmp6-port-unreachable -A FORWARD -i ipv6test -j REJECT --reject-with icmp6-port-unreachable Created attachment 504776 [details]
sysctl ipv6 settings
In case it's useful, attached is the output of
sysctl -a | grep ipv6
I believe your guest routes may not be correct. For each guest, please provide: # ip -6 addr # ip -6 route Your guest should have 3 routes, one for the link local address, one for the virtual network subnet, and a default route to the virtual network gateway # ip -6 addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 inet6 ::1/128 scope host 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000 inet6 2000:dead:beef:a:5054:ff:fee6:1f01/64 scope global dynamic inet6 fe80::5054:ff:fee6:1f01/64 scope link # ip -6 route 2000:dead:beef:a::/64 dev eth0 proto kernel metric 256 mtu 1500 advmss 1440 fe80::/64 dev eth0 proto kernel metric 256 mtu 1500 advmss 1440 default via fe80::211:22ff:fe33:440a dev eth0 proto kernel metric 1024 mtu 1500 advmss 1440 Based on your description of how you configured NetworkManager, I suspect you're missing the first route in my example above. This requires using the 'Routes' dialog box in the NM ipv6 settings page. Alternatively just make the guest use stateless autoconf, since the host runs radvd. The routes look okay to me, both systems have 2001:d00:dad:1::/64 dev eth0 in the route table. See below for full details. Is radvd supposed to be automatically launched by libvirt? It's not running on my system. I do have dnsmasq running though for the ipv6test network. I can try launching it manually and switch to autoconf and see what happens. ~~~ kirk ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [jbastian@kirk ~]$ ip -6 addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000 inet6 2001:d00:dad:1::10/64 scope global valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fe21:58dc/64 scope link valid_lft forever preferred_lft forever [jbastian@kirk ~]$ ip -6 route 2001:d00:dad:1::1 via 2001:d00:dad:1::1 dev eth0 metric 0 cache 2001:d00:dad:1::/64 dev eth0 proto kernel metric 256 unreachable fe80::/64 dev lo proto kernel metric 256 error -101 fe80::/64 dev eth0 proto kernel metric 256 default via 2001:d00:dad:1::1 dev eth0 proto static metric 1024 ~~~ spock ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [jbastian@spock ~]$ ip -6 addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000 inet6 2001:d00:dad:1::20/64 scope global valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fe25:a91/64 scope link valid_lft forever preferred_lft forever [jbastian@spock ~]$ ip -6 route 2001:d00:dad:1::1 via 2001:d00:dad:1::1 dev eth0 metric 0 cache 2001:d00:dad:1::/64 dev eth0 proto kernel metric 256 unreachable fe80::/64 dev lo proto kernel metric 256 error -101 fe80::/64 dev eth0 proto kernel metric 256 default via 2001:d00:dad:1::1 dev eth0 proto static metric 1024 I ran 'tcpdump icmp6' on both kirk and spock during a ping6 from kirk to spock. spock didn't see anything at all. kirk only showed neighbor solicitations: [jbastian@kirk ~]$ sudo tcpdump icmp6 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 18:32:03.497938 IP6 2001:d00:dad:1::10 > ff02::1:ff00:20: ICMP6, neighbor solicitation, who has 2001:d00:dad:1::20, length 32 18:32:04.499689 IP6 2001:d00:dad:1::10 > ff02::1:ff00:20: ICMP6, neighbor solicitation, who has 2001:d00:dad:1::20, length 32 18:32:05.501734 IP6 2001:d00:dad:1::10 > ff02::1:ff00:20: ICMP6, neighbor solicitation, who has 2001:d00:dad:1::20, length 32 ... If I ping6 the host system, the tcpdump is much more interesting: 18:40:21.687694 IP6 2001:d00:dad:1::10 > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has 2001:d00:dad:1::1, length 32 18:40:21.688028 IP6 2001:d00:dad:1::1 > 2001:d00:dad:1::10: ICMP6, neighbor advertisement, tgt is 2001:d00:dad:1::1, length 32 18:40:23.777705 IP6 2001:d00:dad:1::10 > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has 2001:d00:dad:1::1, length 32 18:40:23.778083 IP6 2001:d00:dad:1::1 > 2001:d00:dad:1::10: ICMP6, neighbor advertisement, tgt is 2001:d00:dad:1::1, length 32 18:40:39.122912 IP6 2001:d00:dad:1::10 > 2001:d00:dad:1::1: ICMP6, echo request, seq 1, length 64 18:40:39.123367 IP6 2001:d00:dad:1::1 > 2001:d00:dad:1::10: ICMP6, echo reply, seq 1, length 64 spock doesn't see any of this traffic (according to tcpdump). Hmmm, if I add an explicit route to the other virtual machine via the host machine, then I can get them to see each other. This shouldn't be necessary, though, since they're both on the same virtual subnet (unless I'm not understanding something). I did not have to add an explicit route for IPv4 ~~~ kirk ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [jbastian@kirk ~]$ sudo ip -6 route add 2001:d00:dad:1::20 via 2001:d00:dad:1::1 dev eth0 [jbastian@kirk ~]$ ip -6 route 2001:d00:dad:1::1 via 2001:d00:dad:1::1 dev eth0 metric 0 cache 2001:d00:dad:1::20 via 2001:d00:dad:1::1 dev eth0 metric 1024 2001:d00:dad:1::/64 dev eth0 proto kernel metric 256 unreachable fe80::/64 dev lo proto kernel metric 256 error -101 fe80::/64 dev eth0 proto kernel metric 256 default via 2001:d00:dad:1::1 dev eth0 proto static metric 1024 ~~~ spock ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [jbastian@spock ~]$ sudo ip -6 route add 2001:d00:dad:1::10 via 2001:d00:dad:1::1 dev eth0 [jbastian@spock ~]$ ip -6 route 2001:d00:dad:1::1 via 2001:d00:dad:1::1 dev eth0 metric 0 cache 2001:d00:dad:1::10 via 2001:d00:dad:1::1 dev eth0 metric 1024 2001:d00:dad:1::/64 dev eth0 proto kernel metric 256 unreachable fe80::/64 dev lo proto kernel metric 256 error -101 fe80::/64 dev eth0 proto kernel metric 256 default via 2001:d00:dad:1::1 dev eth0 proto static metric 1024 ~~~ kirk ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [jbastian@kirk ~]$ ping6 -n -c 3 2001:d00:dad:1::20 PING 2001:d00:dad:1::20(2001:d00:dad:1::20) 56 data bytes 64 bytes from 2001:d00:dad:1::20: icmp_seq=1 ttl=63 time=0.983 ms 64 bytes from 2001:d00:dad:1::20: icmp_seq=2 ttl=63 time=0.637 ms 64 bytes from 2001:d00:dad:1::20: icmp_seq=3 ttl=63 time=0.745 ms --- 2001:d00:dad:1::20 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.637/0.788/0.983/0.146 ms ~~~ spock ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [jbastian@spock ~]$ ping6 -n -c 3 2001:d00:dad:1::10 PING 2001:d00:dad:1::10(2001:d00:dad:1::10) 56 data bytes 64 bytes from 2001:d00:dad:1::10: icmp_seq=1 ttl=63 time=1.19 ms 64 bytes from 2001:d00:dad:1::10: icmp_seq=2 ttl=63 time=0.744 ms 64 bytes from 2001:d00:dad:1::10: icmp_seq=3 ttl=63 time=0.829 ms --- 2001:d00:dad:1::10 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.744/0.921/1.192/0.197 ms There was a new Fedora kernel today, kernel-2.6.38.8-32.fc15.x86_64, and when I updated the host machine to it (and rebooted), everything started working: the virtual guests could ping each other just fine! I rebooted back into kernel-2.6.38.7-30.fc15.x86_64 and it broke again. I wonder what changed. I did a few more reboots just to check my sanity. I made no configuration changes on the host or the guests. I only changed which kernel the host machine booted. kernel-2.6.38.6-27.fc15.x86_64 -- broken kernel-2.6.38.7-30.fc15.x86_64 -- broken kernel-2.6.38.8-32.fc15.x86_64 -- works! I guess this is/was a kernel problem, not libvirt! ~~~ localhost ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [jbastian@localhost ~]$ uname -r 2.6.38.8-32.fc15.x86_64 ~~~ kirk ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [jbastian@kirk ~]$ ping6 -n -c 3 2001:d00:dad:1::20 PING 2001:d00:dad:1::20(2001:d00:dad:1::20) 56 data bytes 64 bytes from 2001:d00:dad:1::20: icmp_seq=1 ttl=64 time=1.25 ms 64 bytes from 2001:d00:dad:1::20: icmp_seq=2 ttl=64 time=0.649 ms 64 bytes from 2001:d00:dad:1::20: icmp_seq=3 ttl=64 time=0.550 ms --- 2001:d00:dad:1::20 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 0.550/0.818/1.256/0.312 ms [jbastian@kirk ~]$ ip -6 route show dev eth0 2001:d00:dad:1::20 via 2001:d00:dad:1::20 metric 0 cache 2001:d00:dad:1::/64 proto kernel metric 256 fe80::/64 proto kernel metric 256 default via 2001:d00:dad:1::1 proto static metric 1024 ~~~ spock ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [jbastian@spock ~]$ ping6 -n -c 3 2001:d00:dad:1::10 PING 2001:d00:dad:1::10(2001:d00:dad:1::10) 56 data bytes 64 bytes from 2001:d00:dad:1::10: icmp_seq=1 ttl=64 time=0.641 ms 64 bytes from 2001:d00:dad:1::10: icmp_seq=2 ttl=64 time=0.579 ms 64 bytes from 2001:d00:dad:1::10: icmp_seq=3 ttl=64 time=0.606 ms --- 2001:d00:dad:1::10 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2000ms rtt min/avg/max/mdev = 0.579/0.608/0.641/0.038 ms [jbastian@spock ~]$ ip -6 route show dev eth0 2001:d00:dad:1::10 via 2001:d00:dad:1::10 metric 0 cache 2001:d00:dad:1::/64 proto kernel metric 256 fe80::/64 proto kernel metric 256 default via 2001:d00:dad:1::1 proto static metric 1024 (In reply to comment #6) > Is radvd supposed to be automatically launched by libvirt? It's not running > on my system. That was because I didn't have radvd installed. Oops. Should libvirt have radvd as an RPM Requires? I installed radvd and restarted my ipv6test network and it launched radvd automatically. [jbastian@localhost ~]$ sudo virsh net-destroy ipv6test Network ipv6test destroyed [jbastian@localhost ~]$ sudo virsh net-start ipv6test Network ipv6test started [jbastian@localhost ~]$ pgrep -lf radvd 3273 radvd --debug 1 --config /var/lib/libvirt/radvd/ipv6test-radvd.conf --pidfile /var/run/libvirt/network/ipv6test-radvd.pid-bin It looks like the Fedora version of the specfile didn't get updated for F14 and F15 to require radvd. There is both a Requires and a BuildRequires for radvd in the upstream specfile, and in rawhide's specfile. I opened bug 714169 for the libvirt specfile and "Requires: radvd" Should this bug be closed now since it appears to be a kernel problem? Can anyone duplicate this? A co-worker, Flavio Leitner, has also tested this and confirmed that it's a kernel issue (on the host) and appears to be a problem with the bridge. He also noted that there have been some recent patches to fix bridges and IPv6, for example: bridge: fix forwarding of IPv6 http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.38.y.git;a=commit;h=3b7219a8c7a581033008ffbf60d6529a24291718 The original bug seems to have been an issue in the kernel, not libvirt, and a separate bug has been filed the missing requires of radvd, so this one can be closed. |