libvirt already uses --bind-interfaces, and has for quite some time. As long as the system-wide dnsmasq also uses --bind-interfaces (or "bind-interfaces" if in dnsmasq.conf) and limits the list of interfaces it uses in the manner you suggest above, you should not run into problem (1) (To verify, I just tried starting a system-wide instance of dnsmasq on my Fedora 16 system both with and without "bind-interfaces", then tried starting up a libvirt network in both cases - behavior was as I state above).

The difference between what libvirt currently does and what you're suggesting is that libvirt uses --listen-address=x.x.x.x instead of --interface=xxx. Note that you would still encounter problem 1 (and require the change to the system-wide dnsmasq configuration) even if libvirt used "--interface xxx" (instead of the current "--listen-address x.x.x.x") - this is something you should take up with the dnsmasq maintainer for your Linux distro.

As far as problem 2, that's an interesting report, can you provide dnsmasq-generated logs (usually in /var/log/messages) from instances when that happens? (If you can manage it, a wireshark/tcpdump pcap file of both virbr0 and virbr1 would be very useful.) If it's getting confused about interfaces as you suggest, I could image a guest getting *incorrect* ip address / dns information, but not *missing*; that doesn't fit the MO you're suggesting...

BZ is a year old now, but seems still relevant.

(In reply to Laine Stump from comment #1)
> libvirt already uses --bind-interfaces, and has for quite some time. As long

Well, not on F19B, updated today.

Here is dnsmasq configuration file generated by libvirt:
##WARNING:  THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
##OVERWRITTEN AND LOST.  Changes to this configuration should be made using:
##    virsh net-edit isolated0
## or other application using the libvirt API.
##
## dnsmasq conf file created by libvirt
strict-order
domain-needed
domain=isolated0
expand-hosts
local=/isolated0/
pid-file=/var/run/libvirt/network/isolated0.pid
except-interface=lo
bind-dynamic
interface=virbr1
dhcp-option=3
no-resolv
dhcp-range=192.168.100.128,192.168.100.254
dhcp-no-override
dhcp-leasefile=/var/lib/libvirt/dnsmasq/isolated0.leases
dhcp-lease-max=127
dhcp-hostsfile=/var/lib/libvirt/dnsmasq/isolated0.hostsfile
addn-hosts=/var/lib/libvirt/dnsmasq/isolated0.addnhosts

And stracing it with the same command line the error dialog told me:
strace -f -o log.dnsmasq /sbin/dnsmasq --conf-file=./isolated0.conf

11649 setsockopt(4, SOL_IP, IP_MTU_DISCOVER, [0], 4) = 0
11649 setsockopt(4, SOL_IP, IP_TOS, [192], 4) = 0
11649 setsockopt(4, SOL_IP, IP_PKTINFO, [1], 4) = 0
11649 setsockopt(4, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
11649 setsockopt(4, SOL_SOCKET, SO_REUSEPORT, [1], 4) = 0
11649 bind(4, {sa_family=AF_INET, sin_port=htons(67), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EADDRINUSE (Address already in use)
11649 write(2, "\n", 1)                 = 1
11649 write(2, "dnsmasq: ", 9)          = 9
11649 write(2, "failed to bind DHCP server socke"..., 57) = 57

libvirt-1.0.5.1-1.fc19.x86_64
dnsmasq-2.66-6.fc19.x86_64

I run dhcpd on em1:
[root@localhost ~]# netstat -anp | grep :67
udp        0      0 192.168.254.3:67        0.0.0.0:*                           7975/dhcpd          
[root@localhost ~]#

If I stop my em1-dhcpd and start libvirt default network, I get:
[root@localhost ~]# netstat -anp | grep dnsmasq
tcp        0      0 192.168.122.1:53        0.0.0.0:*               OUÇA       11926/dnsmasq       
tcp        0      0 192.168.101.1:53        0.0.0.0:*               OUÇA       11862/dnsmasq       
tcp        0      0 192.168.100.1:53        0.0.0.0:*               OUÇA       11777/dnsmasq       
udp        0      0 192.168.122.1:53        0.0.0.0:*                           11926/dnsmasq       
udp        0      0 192.168.101.1:53        0.0.0.0:*                           11862/dnsmasq       
udp        0      0 192.168.100.1:53        0.0.0.0:*                           11777/dnsmasq       
udp        0      0 0.0.0.0:67              0.0.0.0:*                           11926/dnsmasq         <-----------
unix  2      [ ]         DGRAM                    86170    11862/dnsmasq        
unix  2      [ ]         DGRAM                    83951    11777/dnsmasq        
unix  2      [ ]         DGRAM                    87223    11926/dnsmasq        
[root@localhost ~]# 

And that's for virbr0, default. I asked libvirt to not start dhcpd on my isolated0.

From dhcpd strace, all alone:
12188 socket(PF_PACKET, SOCK_RAW, 768)  = 5
12188 ioctl(5, SIOCGIFINDEX, {ifr_name="em1", ifr_index=2}) = 0
12188 bind(5, {sa_family=AF_PACKET, proto=0x03, if2, pkttype=PACKET_HOST, addr(0)={0, }, 20) = 0
12188 setsockopt(5, SOL_PACKET, PACKET_AUXDATA, [1], 4) = 0
12188 setsockopt(5, SOL_SOCKET, SO_ATTACH_FILTER, "\v\0\0\0\0\0\0\0 !\304\265\367\177\0\0", 16) = 0
12188 sendto(3, "<190>Jun 11 20:09:28 dhcpd: List"..., 83, MSG_NOSIGNAL, NULL, 0) = 83
12188 sendto(3, "<190>Jun 11 20:09:28 dhcpd: Send"..., 83, MSG_NOSIGNAL, NULL, 0) = 83
12188 fcntl(5, F_SETFD, FD_CLOEXEC)     = 0
12188 socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP) = 7
12188 setsockopt(7, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
12188 bind(7, {sa_family=AF_INET, sin_port=htons(67), sin_addr=inet_addr("192.168.254.3")}, 16) = 0

Even though dhcpd sets REUSEADDR and bound to interface, dnsmasq is using REUSEPORT, a different setting, and not binding to the interface.

I'll move the bug to Fedora component so we get visibility.

(In reply to Marcelo Ricardo Leitner from comment #2)
> 
> (In reply to Laine Stump from comment #1)
> > libvirt already uses --bind-interfaces, and has for quite some time. As long
> 
> Well, not on F19B, updated today.

That comment was posted > 1 year ago. Since that time, libvirt has moved on from using

   --bind-interfaces --listen-address=x.x.x.x

to using:

   --bind-dynamic --interface=xxx

> except-interface=lo
> bind-dynamic
> interface=virbr1

There is a long involved discussion and many reasons behind this change, but the relationship between --bind-dynamic and having other instances of dnsmasq running is the same: the other instances of dnsmasq *must* be configured to not listen on all interfaces, or there will be a conflict.

The configuration required in other dnsmasq instances so that they won't conflict is to use one of the above two possible config snippets to limit the scope of the other dnsmasq. I'm not sure how that would translate to dhcpd as it's been > 10 years since I used dhcpd.

Perhaps Simon (the dnsmasq maintainer) knows something about what needs to be done to make dhcpd cooperate with dnsmasq?

(In reply to Laine Stump from comment #4)
> (In reply to Marcelo Ricardo Leitner from comment #2)
> > 
> > (In reply to Laine Stump from comment #1)
> > > libvirt already uses --bind-interfaces, and has for quite some time. As long
> > 
> > Well, not on F19B, updated today.
> 
> That comment was posted > 1 year ago. Since that time, libvirt has moved on
> from using

Yes.. I hesitated to re-use this BZ due to that, but well, it seems the same issue report still.

>    --bind-interfaces --listen-address=x.x.x.x
> 
> to using:
> 
>    --bind-dynamic --interface=xxx
> 
> > except-interface=lo
> > bind-dynamic
> > interface=virbr1
> 
> There is a long involved discussion and many reasons behind this change, but
> the relationship between --bind-dynamic and having other instances of
> dnsmasq running is the same: the other instances of dnsmasq *must* be
> configured to not listen on all interfaces, or there will be a conflict.
> 
> The configuration required in other dnsmasq instances so that they won't
> conflict is to use one of the above two possible config snippets to limit
> the scope of the other dnsmasq. I'm not sure how that would translate to
> dhcpd as it's been > 10 years since I used dhcpd.
> 
> Perhaps Simon (the dnsmasq maintainer) knows something about what needs to
> be done to make dhcpd cooperate with dnsmasq?

At the end of comment #2 I pasted:
I run dhcpd on em1:
[root@localhost ~]# netstat -anp | grep :67
udp        0      0 192.168.254.3:67        0.0.0.0:*                           7975/dhcpd      

This shows that dhcpd is listening only on that address, 192.168.254.3, port 67. It is not binding on wildcard address.

It's actually dnsmasq, invoke by libvirt, that is binding on all interfaces:
udp        0      0 0.0.0.0:67              0.0.0.0:*                           11926/dnsmasq 

dnsmasq is using SO_PORTREUSE to allow multiple runs of that type, but dhcpd doesn't know about that. dhcpd only binds to the given IP address/port, sets SO_REUSEADDR, and it thinks it's done.

We have nearly two ways out to make this compatible:
- Patch dhcpd to start using SO_PORTREUSE, but this would actually be more like a workaround. dhcpd is just using its own port, what's the deal?

- Patch libvirt/dnsmasq to start binding only on the given interfaces, even while using SO_PORTREUSE - that is, I don't know much about dnsmasq, there may be just a config for that.

Thanks,
Marcelo

Doing DHCP is slightly involved: the server has to be able to receive packets from unconfigured clients which are sent from 0.0.0.0 and to 255.255.255.255. There are essentially two strategies. The first, used by dhcpd, is not coercing the kernel IP stack at all, and use non-portable raw network access (via the packet filter, or similar) to do the tricky bits. The second, used by dnsmasq, does everything via the IP stack and a bit of clever ARP-table manipulation. Both strategies have advantages and drawbacks, but that's off-topic here.

To use the IP stack, it's necessary to bind 0.0.0.0/67, at least on some platforms, which is why dnsmasq does it, and it's not possible to configure dnsmasq otherwise. Since dhcpd is using the packet filter for reception of awkward packets, it doesn't need to bind the wildcard.

Note that SO_REUSEPORT is very new on Linux, the same effect used to achieved with SO_REUSEADDR. It may be that dhcpd isn't doing SO_REUSEPORT just because it's not been updated yet, and if you can coerce both dhcpd and dnsmasq to use SO_REUSEADDR or get both to use SO_REUSEPORT, then it will just work.

Cheers,

Simon.

(In reply to Simon Kelley from comment #6)

Fully agreed, thanks. But so, where do we move this ticket, dnsmasq or dhcp? :)
Or should I create a new one?

Best regards,
Marcelo

To summarize:

1) The new problem you describe is not the same as the original (which was complaining about inability to run multiple dnsmasq instances, and which itself was not even correct - the problem was with the configuration of the system instance of dnsmasq, not a bug in the code); it is instead about inability to run dnsmasq and dhcpd on the same system.

2) As far as I understand from Simon's explanation, it isn't possible for dnsmasq and dhcpd (in its current state) to coexist on the same machine. Is that correct?

Based on the above two facts, in my opinion this bz isn't the correct place to track this new problem. If this newly reported problem can be fixed, it requires one of:

a) a patch to the dhcp package to use SO_REUSEPORT

b) yet another option to dnsmasq (which would then need supporting in libvirt) to make it coexist with what dhcpd is doing (if that is possible)

It sounds possible, though, that due to the very different designs of dnsmasq and dhcpd, they simply can't coexist and there is nothing that can be done about it - if you use dhcpd you can't use dnsmasq, and vice versa.

In any of those cases, I think a new bug should be opened against the dhcp package, since it seems unlikely that option (b) is feasible at all (am I correct, Simon?)

------

In the meantime, since the originally reported problem of this BZ was a configuration error, and the only reason I left the BZ open earlier was for the secondary problem (guests sometimes not getting an address), *and* in the year since I requested additional information from the reporter, they haven't supplied that info (possibly once they fixed their configuration it worked properly? or possibly other changes in the meantime have eliminated it), I am closing this BZ as "WORKSFORME".

Assuming my conjecture that if both dnsmasq and dhcpd use the same SO_REUSE* option it will work, then that could be achieved by

1) patching dhcpd to use SO_REUSEPORT

2) patching dnsmasq to use SO_REUSEADDR 

3) Adding an option to dnsmasq to use SO_REUSEADDR, even when the kernel supports SO_REUSEPORT

4) waiting for dhcpd to catch up with the existance of SO_REUSEPORT and use it in preference to SO_REUSEADDR.


I'm not keen on 3 as it's adding a forever option to solve a (maybe) transient problem.

If my conjecture is false (I've not tested it) then the problem looks to be insoluble. 

Cheers,

Simon.

This bug was closed incorrectly.  The problem that was originally mentioned does not WORKSFORANYONE:


libvirt cannot start up a network that is configured for DHCP because it will try to bind to the DHCP port on 0.0.0.0 that is already used by the system daemon.

Please reopen.

Apologies, I misread "system daemon" as "dhcpd" -- libvirt's dnsmasq does conflict with that -- not as "another dnsmasq", as it seems the original poster was talking about.  The dhcpd conflict is in  https://bugzilla.redhat.com/show_bug.cgi?id=981973 .