Bug 713247

Also, ping6 to the host machine works:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[root@kirk ~]# ping6 -n -c 3 2001:d00:dad:1::1
PING 2001:d00:dad:1::1(2001:d00:dad:1::1) 56 data bytes
64 bytes from 2001:d00:dad:1::1: icmp_seq=1 ttl=64 time=0.228 ms
64 bytes from 2001:d00:dad:1::1: icmp_seq=2 ttl=64 time=0.263 ms
64 bytes from 2001:d00:dad:1::1: icmp_seq=3 ttl=64 time=0.335 ms

--- 2001:d00:dad:1::1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.228/0.275/0.335/0.046 ms
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

It's only to the other VM that it fails.

Please attach the output of "ip6tables -S | grep ipv6test".

[root@localhost ~]# ip6tables -S | grep ipv6test
-A INPUT -i ipv6test -p udp -m udp --dport 53 -j ACCEPT 
-A INPUT -i ipv6test -p tcp -m tcp --dport 53 -j ACCEPT 
-A FORWARD -d 2001:d00:dad:1::/64 -o ipv6test -j ACCEPT 
-A FORWARD -s 2001:d00:dad:1::/64 -i ipv6test -j ACCEPT 
-A FORWARD -i ipv6test -o ipv6test -j ACCEPT 
-A FORWARD -o ipv6test -j REJECT --reject-with icmp6-port-unreachable 
-A FORWARD -i ipv6test -j REJECT --reject-with icmp6-port-unreachable

Created attachment 504776 [details]
sysctl ipv6 settings

In case it's useful, attached is the output of
  sysctl -a | grep ipv6

I believe your guest routes may not be correct. For each guest, please provide:

  # ip -6 addr
  # ip -6 route

Your guest should have 3 routes, one for the link local address, one for the virtual network subnet, and a default route to the virtual network gateway

# ip -6 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436
inet6 ::1/128 scope host
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000
inet6 2000:dead:beef:a:5054:ff:fee6:1f01/64 scope global dynamic
inet6 fe80::5054:ff:fee6:1f01/64 scope link
# ip -6 route
2000:dead:beef:a::/64 dev eth0  proto kernel  metric 256 mtu 1500 advmss 1440
fe80::/64 dev eth0  proto kernel  metric 256  mtu 1500 advmss 1440
default via fe80::211:22ff:fe33:440a dev eth0  proto kernel  metric 1024 mtu 1500 advmss 1440


Based on your description of how you configured NetworkManager, I suspect you're missing the first route in my example above. This requires using the 'Routes' dialog box in the NM ipv6 settings page.

Alternatively just make the guest use stateless autoconf, since the host runs radvd.

The routes look okay to me, both systems have
  2001:d00:dad:1::/64 dev eth0
in the route table.  See below for full details.

Is radvd supposed to be automatically launched by libvirt?  It's not running on my system.  I do have dnsmasq running though for the ipv6test network.  I can try launching it manually and switch to autoconf and see what happens.


~~~ kirk ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[jbastian@kirk ~]$ ip -6 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000
    inet6 2001:d00:dad:1::10/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe21:58dc/64 scope link 
       valid_lft forever preferred_lft forever

[jbastian@kirk ~]$ ip -6 route
2001:d00:dad:1::1 via 2001:d00:dad:1::1 dev eth0  metric 0 
    cache 
2001:d00:dad:1::/64 dev eth0  proto kernel  metric 256 
unreachable fe80::/64 dev lo  proto kernel  metric 256  error -101
fe80::/64 dev eth0  proto kernel  metric 256 
default via 2001:d00:dad:1::1 dev eth0  proto static  metric 1024 

~~~ spock ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[jbastian@spock ~]$ ip -6 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000
    inet6 2001:d00:dad:1::20/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe25:a91/64 scope link 
       valid_lft forever preferred_lft forever

[jbastian@spock ~]$ ip -6 route
2001:d00:dad:1::1 via 2001:d00:dad:1::1 dev eth0  metric 0 
    cache 
2001:d00:dad:1::/64 dev eth0  proto kernel  metric 256 
unreachable fe80::/64 dev lo  proto kernel  metric 256  error -101
fe80::/64 dev eth0  proto kernel  metric 256 
default via 2001:d00:dad:1::1 dev eth0  proto static  metric 1024

I ran 'tcpdump icmp6' on both kirk and spock during a ping6 from kirk to spock.

spock didn't see anything at all.

kirk only showed neighbor solicitations:

[jbastian@kirk ~]$ sudo tcpdump icmp6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
18:32:03.497938 IP6 2001:d00:dad:1::10 > ff02::1:ff00:20: ICMP6, neighbor solicitation, who has 2001:d00:dad:1::20, length 32
18:32:04.499689 IP6 2001:d00:dad:1::10 > ff02::1:ff00:20: ICMP6, neighbor solicitation, who has 2001:d00:dad:1::20, length 32
18:32:05.501734 IP6 2001:d00:dad:1::10 > ff02::1:ff00:20: ICMP6, neighbor solicitation, who has 2001:d00:dad:1::20, length 32
...


If I ping6 the host system, the tcpdump is much more interesting:

18:40:21.687694 IP6 2001:d00:dad:1::10 > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has 2001:d00:dad:1::1, length 32
18:40:21.688028 IP6 2001:d00:dad:1::1 > 2001:d00:dad:1::10: ICMP6, neighbor advertisement, tgt is 2001:d00:dad:1::1, length 32
18:40:23.777705 IP6 2001:d00:dad:1::10 > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has 2001:d00:dad:1::1, length 32
18:40:23.778083 IP6 2001:d00:dad:1::1 > 2001:d00:dad:1::10: ICMP6, neighbor advertisement, tgt is 2001:d00:dad:1::1, length 32
18:40:39.122912 IP6 2001:d00:dad:1::10 > 2001:d00:dad:1::1: ICMP6, echo request, seq 1, length 64
18:40:39.123367 IP6 2001:d00:dad:1::1 > 2001:d00:dad:1::10: ICMP6, echo reply, seq 1, length 64


spock doesn't see any of this traffic (according to tcpdump).

Hmmm, if I add an explicit route to the other virtual machine via the host machine, then I can get them to see each other.  This shouldn't be necessary, though, since they're both on the same virtual subnet (unless I'm not understanding something).  I did not have to add an explicit route for IPv4

~~~ kirk ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[jbastian@kirk ~]$ sudo ip -6 route add 2001:d00:dad:1::20 via 2001:d00:dad:1::1 dev eth0

[jbastian@kirk ~]$ ip -6 route
2001:d00:dad:1::1 via 2001:d00:dad:1::1 dev eth0  metric 0
    cache
2001:d00:dad:1::20 via 2001:d00:dad:1::1 dev eth0  metric 1024
2001:d00:dad:1::/64 dev eth0  proto kernel  metric 256
unreachable fe80::/64 dev lo  proto kernel  metric 256  error -101
fe80::/64 dev eth0  proto kernel  metric 256
default via 2001:d00:dad:1::1 dev eth0  proto static  metric 1024


~~~ spock ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[jbastian@spock ~]$ sudo ip -6 route add 2001:d00:dad:1::10 via 2001:d00:dad:1::1 dev eth0

[jbastian@spock ~]$ ip -6 route
2001:d00:dad:1::1 via 2001:d00:dad:1::1 dev eth0  metric 0
    cache
2001:d00:dad:1::10 via 2001:d00:dad:1::1 dev eth0  metric 1024
2001:d00:dad:1::/64 dev eth0  proto kernel  metric 256
unreachable fe80::/64 dev lo  proto kernel  metric 256  error -101
fe80::/64 dev eth0  proto kernel  metric 256
default via 2001:d00:dad:1::1 dev eth0  proto static  metric 1024


~~~ kirk ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[jbastian@kirk ~]$ ping6 -n -c 3 2001:d00:dad:1::20
PING 2001:d00:dad:1::20(2001:d00:dad:1::20) 56 data bytes
64 bytes from 2001:d00:dad:1::20: icmp_seq=1 ttl=63 time=0.983 ms
64 bytes from 2001:d00:dad:1::20: icmp_seq=2 ttl=63 time=0.637 ms
64 bytes from 2001:d00:dad:1::20: icmp_seq=3 ttl=63 time=0.745 ms

--- 2001:d00:dad:1::20 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.637/0.788/0.983/0.146 ms

~~~ spock ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[jbastian@spock ~]$ ping6 -n -c 3 2001:d00:dad:1::10
PING 2001:d00:dad:1::10(2001:d00:dad:1::10) 56 data bytes
64 bytes from 2001:d00:dad:1::10: icmp_seq=1 ttl=63 time=1.19 ms
64 bytes from 2001:d00:dad:1::10: icmp_seq=2 ttl=63 time=0.744 ms
64 bytes from 2001:d00:dad:1::10: icmp_seq=3 ttl=63 time=0.829 ms

--- 2001:d00:dad:1::10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.744/0.921/1.192/0.197 ms

There was a new Fedora kernel today, kernel-2.6.38.8-32.fc15.x86_64, and when I updated the host machine to it (and rebooted), everything started working: the virtual guests could ping each other just fine!

I rebooted back into kernel-2.6.38.7-30.fc15.x86_64 and it broke again.

I wonder what changed.

I did a few more reboots just to check my sanity.  I made no configuration changes on the host or the guests.  I only changed which kernel the host machine booted.

kernel-2.6.38.6-27.fc15.x86_64 -- broken
kernel-2.6.38.7-30.fc15.x86_64 -- broken
kernel-2.6.38.8-32.fc15.x86_64 -- works!

I guess this is/was a kernel problem, not libvirt!


~~~ localhost ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[jbastian@localhost ~]$ uname -r
2.6.38.8-32.fc15.x86_64

~~~ kirk ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[jbastian@kirk ~]$ ping6 -n -c 3 2001:d00:dad:1::20
PING 2001:d00:dad:1::20(2001:d00:dad:1::20) 56 data bytes
64 bytes from 2001:d00:dad:1::20: icmp_seq=1 ttl=64 time=1.25 ms
64 bytes from 2001:d00:dad:1::20: icmp_seq=2 ttl=64 time=0.649 ms
64 bytes from 2001:d00:dad:1::20: icmp_seq=3 ttl=64 time=0.550 ms

--- 2001:d00:dad:1::20 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.550/0.818/1.256/0.312 ms

[jbastian@kirk ~]$ ip -6 route show dev eth0
2001:d00:dad:1::20 via 2001:d00:dad:1::20  metric 0 
    cache 
2001:d00:dad:1::/64  proto kernel  metric 256 
fe80::/64  proto kernel  metric 256 
default via 2001:d00:dad:1::1  proto static  metric 1024 

~~~ spock ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[jbastian@spock ~]$ ping6 -n -c 3 2001:d00:dad:1::10
PING 2001:d00:dad:1::10(2001:d00:dad:1::10) 56 data bytes
64 bytes from 2001:d00:dad:1::10: icmp_seq=1 ttl=64 time=0.641 ms
64 bytes from 2001:d00:dad:1::10: icmp_seq=2 ttl=64 time=0.579 ms
64 bytes from 2001:d00:dad:1::10: icmp_seq=3 ttl=64 time=0.606 ms

--- 2001:d00:dad:1::10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.579/0.608/0.641/0.038 ms

[jbastian@spock ~]$ ip -6 route show dev eth0                                   
2001:d00:dad:1::10 via 2001:d00:dad:1::10  metric 0                             
    cache                                                                       
2001:d00:dad:1::/64  proto kernel  metric 256                                   
fe80::/64  proto kernel  metric 256                                             
default via 2001:d00:dad:1::1  proto static  metric 1024

(In reply to comment #6)
> Is radvd supposed to be automatically launched by libvirt?  It's not running
> on my system.


That was because I didn't have radvd installed.  Oops.

Should libvirt have radvd as an RPM Requires?


I installed radvd and restarted my ipv6test network and it launched radvd automatically.

[jbastian@localhost ~]$ sudo virsh net-destroy ipv6test 
Network ipv6test destroyed

[jbastian@localhost ~]$ sudo virsh net-start ipv6test
Network ipv6test started

[jbastian@localhost ~]$ pgrep -lf radvd
3273 radvd --debug 1 --config /var/lib/libvirt/radvd/ipv6test-radvd.conf --pidfile /var/run/libvirt/network/ipv6test-radvd.pid-bin

It looks like the Fedora version of the specfile didn't get updated for F14 and F15 to require radvd. There is both a Requires and a BuildRequires for radvd in the upstream specfile, and in rawhide's specfile.

I opened bug 714169 for the libvirt specfile and "Requires: radvd"

Should this bug be closed now since it appears to be a kernel problem?  Can anyone duplicate this?

A co-worker, Flavio Leitner, has also tested this and confirmed that it's a kernel issue (on the host) and appears to be a problem with the bridge.  He also noted that there have been some recent patches to fix bridges and IPv6, for example:
  bridge: fix forwarding of IPv6
  http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.38.y.git;a=commit;h=3b7219a8c7a581033008ffbf60d6529a24291718

The original bug seems to have been an issue in the kernel, not libvirt, and a separate bug has been filed the missing requires of radvd, so this one can be closed.