Bug 981973 - dnsmasq conflicts withs dhcpd on loopback usage
dnsmasq conflicts withs dhcpd on loopback usage
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: dnsmasq (Show other bugs)
18
All Linux
unspecified Severity urgent
: ---
: ---
Assigned To: Tomáš Hozza
Fedora Extras Quality Assurance
: Reopened
: 995863 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-07 08:13 EDT by Rolf Fokkens
Modified: 2013-10-19 05:12 EDT (History)
11 users (show)

See Also:
Fixed In Version: dnsmasq-2.65-8.fc18
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-19 05:12:30 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Rolf Fokkens 2013-07-07 08:13:33 EDT
Description of problem: After upgrading to F19 dhcpd wasn't able to start during boot. It appeared that libvirtd was staring dnsmasq, which bound do "lo". Because dhcpd apparently tries to bind to 0.0.0.0:67 there clearly was a conflict.

AFAIK dnsmasq is started by libvirtd to actually bind to virbr0, to support NAT for VM's. dnsmasq intentionally always binds to lo as well.

Remarkable is the fact that dhcpd binds to 0.0.0.0:67 dispite the fact that a specific list of interfaces is provided on the commandline, and lo is not one of them.

I'm not sure why this manifests itself after the upgrade to F19

Version-Release number of selected component (if applicable):
dnsmasq-2.66-7.fc19.x86_64
dhcp-4.2.5-15.fc19.x86_64
libvirt-daemon-1.0.5.2-1.fc19.x86_64

How reproducible: 100% (given my upgrade history?)

Steps to Reproduce:
1. Install libvirt; enable it
2. Install dhcpd; enable it
3. reboot
4. Notice that dhcpd isn't able to start

Actual results:
dhcpd not starting

Expected results:
dhcpd starting

Additional info:
Comment 1 Tomáš Hozza 2013-07-08 06:48:30 EDT
(In reply to Rolf Fokkens from comment #0)
> Description of problem: After upgrading to F19 dhcpd wasn't able to start
> during boot. It appeared that libvirtd was staring dnsmasq, which bound do
> "lo". Because dhcpd apparently tries to bind to 0.0.0.0:67 there clearly was
> a conflict.

dnsmasq does NOT bind to loopback interface when run by libvirt. It binds
to specified interface address for DNS and to INADDR_ANY for DHCP.

> AFAIK dnsmasq is started by libvirtd to actually bind to virbr0, to support
> NAT for VM's. dnsmasq intentionally always binds to lo as well.
> 
> Remarkable is the fact that dhcpd binds to 0.0.0.0:67 dispite the fact that
> a specific list of interfaces is provided on the commandline, and lo is not
> one of them.

dhcpd listens *only* on interfaces for which it finds subnet declaration in
dhcpd.conf.

> I'm not sure why this manifests itself after the upgrade to F19
> 
> Version-Release number of selected component (if applicable):
> dnsmasq-2.66-7.fc19.x86_64
> dhcp-4.2.5-15.fc19.x86_64
> libvirt-daemon-1.0.5.2-1.fc19.x86_64
> 
> How reproducible: 100% (given my upgrade history?)
> 
> Steps to Reproduce:
> 1. Install libvirt; enable it
> 2. Install dhcpd; enable it
> 3. reboot
> 4. Notice that dhcpd isn't able to start
> 
> Actual results:
> dhcpd not starting
> 
> Expected results:
> dhcpd starting
> 
> Additional info:

I tried to reproduce your issue, but with no luck so far.

Can you please attach your dhcpd.conf and dhcpd output from /var/log/messages?

Thanks!
Comment 2 Rolf Fokkens 2013-07-08 17:22:16 EDT
All the details of my successful attempt to reproduce on another PC:

[root@home07 ~]# yum erase libvirt-daemon
Loaded plugins: fastestmirror, refresh-packagekit
...
Complete!
[root@home07 ~]# rm -Rf /etc/libvirt/
[root@home07 ~]# yum erase dnsmasq
Loaded plugins: fastestmirror, refresh-packagekit
...
Complete!
[root@home07 ~]# rm -Rf /etc/dnsmasq*
[root@home07 ~]# yum install libvirt-daemon-qemu
Loaded plugins: fastestmirror, refresh-packagekit
...
Dependencies Resolved
...
Install  1 Package (+9 Dependent packages)
Installed:
  libvirt-daemon-qemu.x86_64 0:1.0.5.2-1.fc19
Dependency Installed:
  dnsmasq.x86_64 0:2.66-7.fc19
  libvirt-daemon.x86_64 0:1.0.5.2-1.fc19
  libvirt-daemon-driver-interface.x86_64 0:1.0.5.2-1.fc19               
  libvirt-daemon-driver-network.x86_64 0:1.0.5.2-1.fc19
  libvirt-daemon-driver-nodedev.x86_64 0:1.0.5.2-1.fc19
  libvirt-daemon-driver-nwfilter.x86_64 0:1.0.5.2-1.fc19                
  libvirt-daemon-driver-qemu.x86_64 0:1.0.5.2-1.fc19
  libvirt-daemon-driver-secret.x86_64 0:1.0.5.2-1.fc19
  libvirt-daemon-driver-storage.x86_64 0:1.0.5.2-1.fc19                 
Complete!
[root@home07 ~]# netstat -anp | grep dnsmasq
[root@home07 ~]# systemctl start libvirtd
[root@home07 ~]# netstat -anp | grep dnsmasq
tcp   0  0 192.168.122.1:53 0.0.0.0:*  LISTEN 4072/dnsmasq
udp   0  0 192.168.122.1:53 0.0.0.0:*         4072/dnsmasq
udp   0  0 0.0.0.0:67       0.0.0.0:*         4072/dnsmasq
unix  2    [ ]         DGRAM         36016    4072/dnsmasq
[root@home07 ~]# yum install dhcp
Loaded plugins: fastestmirror, refresh-packagekit
...
Complete!
[root@home07 ~]# vi /etc/systemd/system/dhcpd.service
--- tekst:
[Unit]
Description=DHCPv4 Server Daemon
Documentation=man:dhcpd(8) man:dhcpd.conf(5)
After=network.target
After=time-sync.target

[Service]
ExecStart=/usr/sbin/dhcpd -f -cf /etc/dhcp/dhcpd.conf -user dhcpd -group dhcpd --no-pid br0

[Install]
WantedBy=multi-user.target
[root@home07 ~]# systemctl daemon-reload
[root@home07 ~]# vi /etc/dhcp/dhcpd.conf
...tekst:
#
# DHCP Server Configuration file.
#   see /usr/share/doc/dhcp*/dhcpd.conf.example
#   see dhcpd.conf(5) man page
#
subnet 192.168.254.0 netmask 255.255.255.0 {
  #
}
[root@home07 ~]# systemctl start dhcpd
[root@home07 ~]# systemctl status dhcpd
dhcpd.service - DHCPv4 Server Daemon
   Loaded: loaded (/usr/lib/systemd/system/dhcpd.service; enabled)
   Active: failed (Result: exit-code) since Mon 2013-07-08 22:49:21 CEST; 1min 43s ago
     Docs: man:dhcpd(8)
           man:dhcpd.conf(5)
  Process: 13921 ExecStart=/usr/sbin/dhcpd -f -cf /etc/dhcp/dhcpd.conf -user dhcpd -group dhcpd --no-pid (code=exited, status=1/FAILURE)
....
Jul 08 22:49:21 home07 dhcpd[13921]: Can't bind to dhcp address: Address already in use
Jul 08 22:49:21 home07 dhcpd[13921]: Please make sure there is no other dhcp server
Jul 08 22:49:21 home07 dhcpd[13921]: running and that there's no entry for dhcp or
Jul 08 22:49:21 home07 dhcpd[13921]: bootp in /etc/inetd.conf.   Also make sure you
Jul 08 22:49:21 home07 dhcpd[13921]: are not running HP JetAdmin software, which
Jul 08 22:49:21 home07 dhcpd[13921]: includes a bootp server.
....
Jul 08 22:49:21 home07 dhcpd[13921]: exiting.
....
[root@home07 ~]# killall dnsmasq
[root@home07 ~]# systemctl start dhcpd
[root@home07 ~]# systemctl status dhcpd
dhcpd.service - DHCPv4 Server Daemon
   Loaded: loaded (/usr/lib/systemd/system/dhcpd.service; enabled)
   Active: active (running) since Mon 2013-07-08 22:55:38 CEST; 3s ago
     Docs: man:dhcpd(8)
           man:dhcpd.conf(5)
 Main PID: 13945 (dhcpd)
   CGroup: name=systemd:/system/dhcpd.service
           └─13945 /usr/sbin/dhcpd -f -cf /etc/dhcp/dhcpd.conf -user dhcpd -g...
...
[root@home07 ~]# netstat  -anp | grep dhcp
udp  0 0 0.0.0.0:26595  0.0.0.0:*     13945/dhcpd         
udp  0 0 0.0.0.0:67     0.0.0.0:*     13945/dhcpd         
udp6 0 0 :::56812       :::*          13945/dhcpd         
raw  0 0 0.0.0.0:1      0.0.0.0:*  7  13945/dhcpd         
unix  3 [ ] STREAM CONNECTED  60820   13945/dhcpd          
unix  3 [ ] STREAM CONNECTED  60841   13945/dhcpd          
unix  2 [ ] DGRAM             60834   13945/dhcpd          
[root@home07 ~]# ps -efw | grep dhcpd
dhcpd    14062     1  0 23:06 ?        00:00:00 /usr/sbin/dhcpd -f -cf /etc/dhcp/dhcpd.conf -user dhcpd -group dhcpd --no-pid br0
root     14076  3724  0 23:07 pts/0    00:00:00 grep --color=auto dhcpd
[root@home07 ~]# systemctl stop libvirtd
[root@home07 ~]# systemctl stop dhcpd
[root@home07 ~]# systemctl start libvirtd
[root@home07 ~]# systemctl start dhcpd
[root@home07 ~]# service dhcpd status
Redirecting to /bin/systemctl status  dhcpd.service
dhcpd.service - DHCPv4 Server Daemon
   Loaded: loaded (/usr/lib/systemd/system/dhcpd.service; disabled)
   Active: failed (Result: exit-code) since Mon 2013-07-08 19:07:08 CEST; 15s ago
     Docs: man:dhcpd(8)
           man:dhcpd.conf(5)
  Process: 4707 ExecStart=/usr/sbin/dhcpd -f -cf /etc/dhcp/dhcpd.conf -user dhcpd -group dhcpd --no-pid (code=exited, status=1/FAILURE)
Comment 3 Ira Malinich 2013-08-05 16:04:56 EDT
I can confirm this problem also exists on i686, so I'm updating the bug.
Comment 4 Martin Dengler 2013-08-11 17:55:16 EDT
I hit this as well updating from F18 to F19 (not sure that's relevant, though).

Here's my libvirt default network.xml (as mentioned  https://www.redhat.com/archives/libvirt-users/2012-September/msg00070.html )

<network>
  <name>default</name>
  <uuid>fd0f89d6-5401-4af1-b8bd-eb2a22bf4cd4</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0' />
  <mac address='52:54:00:9c:7f:cc'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254' />
    </dhcp>
  </ip>
</network>

...and dnsmasq is definitely binding to all addresses, not just the address it needs to:

# lsof -P -n -i | grep dnsmasq
dnsmasq    5830   nobody    4u  IPv4 165094      0t0  UDP *:67 
dnsmasq    5830   nobody    6u  IPv4 165103      0t0  UDP 192.168.122.1:53 
dnsmasq    5830   nobody    7u  IPv4 165104      0t0  TCP 192.168.122.1:53 (LISTEN)

...which seems quite impolite.

It looks like in https://bugzilla.redhat.com/show_bug.cgi?id=804296 it was acknowledged that dnsmasq / libvirt is causing this problem (see https://bugzilla.redhat.com/show_bug.cgi?id=804296#c6 ).

Either in this bug or in that bug, it'd be nice to not have libvirt break dhcpd.
Comment 5 Tomáš Hozza 2013-08-12 05:00:38 EDT
*** Bug 995863 has been marked as a duplicate of this bug. ***
Comment 6 Tomáš Hozza 2013-08-12 09:19:44 EDT
I patched dnsmasq so that it uses SO_REUSEADDR instead of SO_REUSEPORT
as ISC DHCP does for DHCPv4 socket.
Comment 7 Fedora Update System 2013-08-12 09:55:41 EDT
dnsmasq-2.66-9.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/dnsmasq-2.66-9.fc19
Comment 8 Fedora Update System 2013-08-15 19:27:04 EDT
Package dnsmasq-2.66-10.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dnsmasq-2.66-10.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-14825/dnsmasq-2.66-10.fc19
then log in and leave karma (feedback).
Comment 9 Fedora Update System 2013-08-17 20:34:23 EDT
dnsmasq-2.66-10.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 10 Brian 2013-10-02 12:46:46 EDT
This effects Fedora 18 as well and should be pushed into the updates repo.
Comment 11 Tomáš Hozza 2013-10-03 05:52:51 EDT
I was not able to reproduce it with the latest dnsmasq and dhcpd on F18 in
VM. Then I updated the kernel and the issue appeared.

I'll backport the fix into F18...
Comment 12 Fedora Update System 2013-10-03 08:17:02 EDT
dnsmasq-2.65-8.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/dnsmasq-2.65-8.fc18
Comment 13 Fedora Update System 2013-10-03 22:03:21 EDT
Package dnsmasq-2.65-8.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dnsmasq-2.65-8.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-18276/dnsmasq-2.65-8.fc18
then log in and leave karma (feedback).
Comment 14 Fedora Update System 2013-10-19 05:12:30 EDT
dnsmasq-2.65-8.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.