Bug 981973

(In reply to Rolf Fokkens from comment #0)
> Description of problem: After upgrading to F19 dhcpd wasn't able to start
> during boot. It appeared that libvirtd was staring dnsmasq, which bound do
> "lo". Because dhcpd apparently tries to bind to 0.0.0.0:67 there clearly was
> a conflict.

dnsmasq does NOT bind to loopback interface when run by libvirt. It binds
to specified interface address for DNS and to INADDR_ANY for DHCP.

> AFAIK dnsmasq is started by libvirtd to actually bind to virbr0, to support
> NAT for VM's. dnsmasq intentionally always binds to lo as well.
> 
> Remarkable is the fact that dhcpd binds to 0.0.0.0:67 dispite the fact that
> a specific list of interfaces is provided on the commandline, and lo is not
> one of them.

dhcpd listens *only* on interfaces for which it finds subnet declaration in
dhcpd.conf.

> I'm not sure why this manifests itself after the upgrade to F19
> 
> Version-Release number of selected component (if applicable):
> dnsmasq-2.66-7.fc19.x86_64
> dhcp-4.2.5-15.fc19.x86_64
> libvirt-daemon-1.0.5.2-1.fc19.x86_64
> 
> How reproducible: 100% (given my upgrade history?)
> 
> Steps to Reproduce:
> 1. Install libvirt; enable it
> 2. Install dhcpd; enable it
> 3. reboot
> 4. Notice that dhcpd isn't able to start
> 
> Actual results:
> dhcpd not starting
> 
> Expected results:
> dhcpd starting
> 
> Additional info:

I tried to reproduce your issue, but with no luck so far.

Can you please attach your dhcpd.conf and dhcpd output from /var/log/messages?

Thanks!

All the details of my successful attempt to reproduce on another PC:

[root@home07 ~]# yum erase libvirt-daemon
Loaded plugins: fastestmirror, refresh-packagekit
...
Complete!
[root@home07 ~]# rm -Rf /etc/libvirt/
[root@home07 ~]# yum erase dnsmasq
Loaded plugins: fastestmirror, refresh-packagekit
...
Complete!
[root@home07 ~]# rm -Rf /etc/dnsmasq*
[root@home07 ~]# yum install libvirt-daemon-qemu
Loaded plugins: fastestmirror, refresh-packagekit
...
Dependencies Resolved
...
Install  1 Package (+9 Dependent packages)
Installed:
  libvirt-daemon-qemu.x86_64 0:1.0.5.2-1.fc19
Dependency Installed:
  dnsmasq.x86_64 0:2.66-7.fc19
  libvirt-daemon.x86_64 0:1.0.5.2-1.fc19
  libvirt-daemon-driver-interface.x86_64 0:1.0.5.2-1.fc19               
  libvirt-daemon-driver-network.x86_64 0:1.0.5.2-1.fc19
  libvirt-daemon-driver-nodedev.x86_64 0:1.0.5.2-1.fc19
  libvirt-daemon-driver-nwfilter.x86_64 0:1.0.5.2-1.fc19                
  libvirt-daemon-driver-qemu.x86_64 0:1.0.5.2-1.fc19
  libvirt-daemon-driver-secret.x86_64 0:1.0.5.2-1.fc19
  libvirt-daemon-driver-storage.x86_64 0:1.0.5.2-1.fc19                 
Complete!
[root@home07 ~]# netstat -anp | grep dnsmasq
[root@home07 ~]# systemctl start libvirtd
[root@home07 ~]# netstat -anp | grep dnsmasq
tcp   0  0 192.168.122.1:53 0.0.0.0:*  LISTEN 4072/dnsmasq
udp   0  0 192.168.122.1:53 0.0.0.0:*         4072/dnsmasq
udp   0  0 0.0.0.0:67       0.0.0.0:*         4072/dnsmasq
unix  2    [ ]         DGRAM         36016    4072/dnsmasq
[root@home07 ~]# yum install dhcp
Loaded plugins: fastestmirror, refresh-packagekit
...
Complete!
[root@home07 ~]# vi /etc/systemd/system/dhcpd.service
--- tekst:
[Unit]
Description=DHCPv4 Server Daemon
Documentation=man:dhcpd(8) man:dhcpd.conf(5)
After=network.target
After=time-sync.target

[Service]
ExecStart=/usr/sbin/dhcpd -f -cf /etc/dhcp/dhcpd.conf -user dhcpd -group dhcpd --no-pid br0

[Install]
WantedBy=multi-user.target
[root@home07 ~]# systemctl daemon-reload
[root@home07 ~]# vi /etc/dhcp/dhcpd.conf
...tekst:
#
# DHCP Server Configuration file.
#   see /usr/share/doc/dhcp*/dhcpd.conf.example
#   see dhcpd.conf(5) man page
#
subnet 192.168.254.0 netmask 255.255.255.0 {
  #
}
[root@home07 ~]# systemctl start dhcpd
[root@home07 ~]# systemctl status dhcpd
dhcpd.service - DHCPv4 Server Daemon
   Loaded: loaded (/usr/lib/systemd/system/dhcpd.service; enabled)
   Active: failed (Result: exit-code) since Mon 2013-07-08 22:49:21 CEST; 1min 43s ago
     Docs: man:dhcpd(8)
           man:dhcpd.conf(5)
  Process: 13921 ExecStart=/usr/sbin/dhcpd -f -cf /etc/dhcp/dhcpd.conf -user dhcpd -group dhcpd --no-pid (code=exited, status=1/FAILURE)
....
Jul 08 22:49:21 home07 dhcpd[13921]: Can't bind to dhcp address: Address already in use
Jul 08 22:49:21 home07 dhcpd[13921]: Please make sure there is no other dhcp server
Jul 08 22:49:21 home07 dhcpd[13921]: running and that there's no entry for dhcp or
Jul 08 22:49:21 home07 dhcpd[13921]: bootp in /etc/inetd.conf.   Also make sure you
Jul 08 22:49:21 home07 dhcpd[13921]: are not running HP JetAdmin software, which
Jul 08 22:49:21 home07 dhcpd[13921]: includes a bootp server.
....
Jul 08 22:49:21 home07 dhcpd[13921]: exiting.
....
[root@home07 ~]# killall dnsmasq
[root@home07 ~]# systemctl start dhcpd
[root@home07 ~]# systemctl status dhcpd
dhcpd.service - DHCPv4 Server Daemon
   Loaded: loaded (/usr/lib/systemd/system/dhcpd.service; enabled)
   Active: active (running) since Mon 2013-07-08 22:55:38 CEST; 3s ago
     Docs: man:dhcpd(8)
           man:dhcpd.conf(5)
 Main PID: 13945 (dhcpd)
   CGroup: name=systemd:/system/dhcpd.service
           └─13945 /usr/sbin/dhcpd -f -cf /etc/dhcp/dhcpd.conf -user dhcpd -g...
...
[root@home07 ~]# netstat  -anp | grep dhcp
udp  0 0 0.0.0.0:26595  0.0.0.0:*     13945/dhcpd         
udp  0 0 0.0.0.0:67     0.0.0.0:*     13945/dhcpd         
udp6 0 0 :::56812       :::*          13945/dhcpd         
raw  0 0 0.0.0.0:1      0.0.0.0:*  7  13945/dhcpd         
unix  3 [ ] STREAM CONNECTED  60820   13945/dhcpd          
unix  3 [ ] STREAM CONNECTED  60841   13945/dhcpd          
unix  2 [ ] DGRAM             60834   13945/dhcpd          
[root@home07 ~]# ps -efw | grep dhcpd
dhcpd    14062     1  0 23:06 ?        00:00:00 /usr/sbin/dhcpd -f -cf /etc/dhcp/dhcpd.conf -user dhcpd -group dhcpd --no-pid br0
root     14076  3724  0 23:07 pts/0    00:00:00 grep --color=auto dhcpd
[root@home07 ~]# systemctl stop libvirtd
[root@home07 ~]# systemctl stop dhcpd
[root@home07 ~]# systemctl start libvirtd
[root@home07 ~]# systemctl start dhcpd
[root@home07 ~]# service dhcpd status
Redirecting to /bin/systemctl status  dhcpd.service
dhcpd.service - DHCPv4 Server Daemon
   Loaded: loaded (/usr/lib/systemd/system/dhcpd.service; disabled)
   Active: failed (Result: exit-code) since Mon 2013-07-08 19:07:08 CEST; 15s ago
     Docs: man:dhcpd(8)
           man:dhcpd.conf(5)
  Process: 4707 ExecStart=/usr/sbin/dhcpd -f -cf /etc/dhcp/dhcpd.conf -user dhcpd -group dhcpd --no-pid (code=exited, status=1/FAILURE)

I can confirm this problem also exists on i686, so I'm updating the bug.

I hit this as well updating from F18 to F19 (not sure that's relevant, though).

Here's my libvirt default network.xml (as mentioned  https://www.redhat.com/archives/libvirt-users/2012-September/msg00070.html )

<network>
  <name>default</name>
  <uuid>fd0f89d6-5401-4af1-b8bd-eb2a22bf4cd4</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0' />
  <mac address='52:54:00:9c:7f:cc'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254' />
    </dhcp>
  </ip>
</network>

...and dnsmasq is definitely binding to all addresses, not just the address it needs to:

# lsof -P -n -i | grep dnsmasq
dnsmasq    5830   nobody    4u  IPv4 165094      0t0  UDP *:67 
dnsmasq    5830   nobody    6u  IPv4 165103      0t0  UDP 192.168.122.1:53 
dnsmasq    5830   nobody    7u  IPv4 165104      0t0  TCP 192.168.122.1:53 (LISTEN)

...which seems quite impolite.

It looks like in https://bugzilla.redhat.com/show_bug.cgi?id=804296 it was acknowledged that dnsmasq / libvirt is causing this problem (see https://bugzilla.redhat.com/show_bug.cgi?id=804296#c6 ).

Either in this bug or in that bug, it'd be nice to not have libvirt break dhcpd.

*** Bug 995863 has been marked as a duplicate of this bug. ***

I patched dnsmasq so that it uses SO_REUSEADDR instead of SO_REUSEPORT
as ISC DHCP does for DHCPv4 socket.

dnsmasq-2.66-9.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/dnsmasq-2.66-9.fc19

Package dnsmasq-2.66-10.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dnsmasq-2.66-10.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-14825/dnsmasq-2.66-10.fc19
then log in and leave karma (feedback).

dnsmasq-2.66-10.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

This effects Fedora 18 as well and should be pushed into the updates repo.

I was not able to reproduce it with the latest dnsmasq and dhcpd on F18 in
VM. Then I updated the kernel and the issue appeared.

I'll backport the fix into F18...

dnsmasq-2.65-8.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/dnsmasq-2.65-8.fc18

Package dnsmasq-2.65-8.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dnsmasq-2.65-8.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-18276/dnsmasq-2.65-8.fc18
then log in and leave karma (feedback).

dnsmasq-2.65-8.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.