Bug 886682

Summary: starting a ipv6-enabled libvirt network breaks host networking
Product: Red Hat Enterprise Linux 6 Reporter: Gerd Hoffmann <kraxel>
Component: libvirtAssignee: Michal Privoznik <mprivozn>
Status: CLOSED DUPLICATE QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.4CC: acathrow, berrange, dallan, dyasny, dyuan, eblake, laine, mzhan, rwu, thozza, whuang, ydu
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-12-20 20:05:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Gerd Hoffmann 2012-12-12 21:14:28 UTC
Description of problem:
$subject

Version-Release number of selected component (if applicable):
kernel-2.6.32-348.el6.x86_64
libvirt-0.10.2-12.el6.x86_64
dnsmasq-2.48-10.el6.x86_64
radvd-1.6-1.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Create a network with ipv6 support.  Both adding ipv6 addresses
   to the default network and creating a separate ipv6-only network
   trigger this bug.
2. Start the network.

Actual results:
Host networking stops working.  At least.
Can't ssh into the machine any more.
Machine has no serial line unfortunaly, which
makes investigation a bit difficuilt :(

Expected results:
Everything works fine.

Additional info:

Comment 3 Gerd Hoffmann 2012-12-13 07:37:41 UTC
Downgrading dnsmasq and libvirt gets things going again.  Known good versions:

dnsmasq-2.48-7.el6.x86_64
libvirt-0.10.2-10.el6.x86_64

Comment 5 Gerd Hoffmann 2012-12-13 07:44:50 UTC
Upgraded dnsmasq to dnsmasq-2.48-10.el6.x86_64 -- still working.

Comment 6 Gerd Hoffmann 2012-12-13 07:54:34 UTC
Upgraded libvirt to libvirt-0.10.2-11.el6.x86_64 -> FAIL.

Comment 7 Laine Stump 2012-12-13 09:37:49 UTC
Okay. Can you try this dnsmasq scratch build and see if it fixes things for you:

https://brewweb.devel.redhat.com/taskinfo?taskID=5185194

Comment 8 Gerd Hoffmann 2012-12-13 09:59:47 UTC
(In reply to comment #7)
> Okay. Can you try this dnsmasq scratch build and see if it fixes things for
> you:
> 
> https://brewweb.devel.redhat.com/taskinfo?taskID=5185194

It doesn't, still broken (when combined with latest libvirt).

Comment 9 Laine Stump 2012-12-13 18:20:14 UTC
Is it broken in the same way? Or differently? I've finally been able to test that scratch build myself and found that DHCP for the guests works, and everything about the host's networking remains fine, but just the DNS service provided by dnsmasq isn't responding to the guest's DNS requests (see Bug 884957)

If the IP addresses on your host interfaces are getting deleted (as your earlier ifconfig output indicates) you are apparently experiencing something much worse than either Eric or I. Can you think of anything out of the ordinary on that machine's config (note that my machine has both IPv6+IPv4 networks and IPv6-only networks, so that's apparently not the cause, or at least not the *only* cause).

Comment 10 yanbing du 2012-12-14 03:51:04 UTC
I'm trying to reproduce this bug, but can't.
# rpm -q libvirt
libvirt-0.10.2-12.el6.x86_64
# rpm -q dnsmasq
dnsmasq-2.48-10.el6.x86_64
# rpm -q radvd
radvd-1.6-1.el6.x86_64

1. Add a ip address to 'default' network and define&start a ipv6-only network
 
# virsh net-list --all
Name                 State      Autostart     Persistent
--------------------------------------------------
default              active     yes           yes
ipv6                 active     no            yes

# virsh net-dumpxml default
<network>
  <name>default</name>
  <uuid>efba4d2c-ba6b-4e4a-8cfe-03dd6bdb41d0</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0' />
  <mac address='52:54:00:61:41:C3'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254' />
    </dhcp>
  </ip>
  <ip family='ipv6' address='2002:db8:ac10:fe01::2' prefix='64'>
  </ip>
</network>

# virsh net-dumpxml ipv6
<network>
  <name>ipv6</name>
  <uuid>cbf4a523-152d-286c-56cf-a0e71b006686</uuid>
  <forward mode='nat'/>
  <bridge name='virbr6' stp='on' delay='0' />
  <mac address='52:54:00:E3:D6:A4'/>
  <ip family='ipv6' address='2002:db8:ac10:fe01::1' prefix='64'>
  </ip>
</network>

2. Another problem.
When start the 'default' network without ipv6 support, then it will start a dnsmasq process, but after start the ipv6 network, this process will disappear.

# virsh net-start default
Network default started

# virsh net-dumpxml default
<network>
  <name>default</name>
  <uuid>efba4d2c-ba6b-4e4a-8cfe-03dd6bdb41d0</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0' />
  <mac address='52:54:00:61:41:C3'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254' />
    </dhcp>
  </ip>
</network>

# ps aux|grep dnsmasq
nobody    4649  0.0  0.0  12888   588 ?        S    11:46   0:00 /usr/sbin/dnsmasq --strict-order --local=// --domain-needed --pid-file=/var/run/libvirt/network/default.pid --conf-file= --bind-dynamic --interface virbr0 --dhcp-range 192.168.122.2,192.168.122.254 --dhcp-leasefile=/var/lib/libvirt/dnsmasq/default.leases --dhcp-lease-max=253 --dhcp-no-override --dhcp-hostsfile=/var/lib/libvirt/dnsmasq/default.hostsfile --addn-hosts=/var/lib/libvirt/dnsmasq/default.addnhosts
root      4651  0.0  0.0 103244   808 pts/0    S+   11:46   0:00 grep dnsmasq

# virsh net-start ipv6
Network ipv6 started

# virsh net-dumpxml ipv6
<network>
  <name>ipv6</name>
  <uuid>cbf4a523-152d-286c-56cf-a0e71b006686</uuid>
  <forward mode='nat'/>
  <bridge name='virbr6' stp='on' delay='0' />
  <mac address='52:54:00:E3:D6:A4'/>
  <ip family='ipv6' address='2002:db8:ac10:fe01::1' prefix='64'>
  </ip>
</network>

# ps aux|grep dnsmasq
root      4742  0.0  0.0 103244   808 pts/0    S+   11:48   0:00 grep dnsmasq

Comment 11 Gerd Hoffmann 2012-12-14 07:17:19 UTC
(In reply to comment #9)
> Is it broken in the same way? Or differently?

Same way, host network seems to be seriously busted.

Meanwhile tried the same on a box with serial console.  There are no kernel messages, but a simple "ip link ls" hangs even when typed at the serial console, so it's not just the ssh connection which is broken.

> If the IP addresses on your host interfaces are getting deleted (as your
> earlier ifconfig output indicates) you are apparently experiencing something
> much worse than either Eric or I.

Oh, that is the output without libvirt networks started.  With libvirt networks started I can't list it (see above), at least not with a libvirt version installed which starts dnsmasq with the new --bind-dynamic switch.

> Can you think of anything out of the
> ordinary on that machine's config (note that my machine has both IPv6+IPv4
> networks and IPv6-only networks, so that's apparently not the cause, or at
> least not the *only* cause).

The machine has native ipv6 connectivity and is configured as ipv6 router.

Comment 16 Laine Stump 2012-12-20 20:05:21 UTC
This was apparently another bit of fallout from the fist attempted fix to CVE-2012-3411 - see Bug 884957

*** This bug has been marked as a duplicate of bug 884957 ***