Bug 873621

Summary: NetworkManager-0.9.6.4-1.fc17 breaks dnsmasq configurations
Product: [Fedora] Fedora Reporter: Peter Robinson <pbrobinson>
Component: NetworkManagerAssignee: Dan Williams <dcbw>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 17CC: danw, dcbw, ipilcher, james, jklimes, mishu, roland
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-11-22 03:58:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Peter Robinson 2012-11-06 10:42:50 UTC
I use dnsmasq to allow split DNS for access to VPN and local connected (see related RHBZ 842037) but it seems that 0.9.6.4-1.fc17 breaks this.

In my /etc/NetworkManager/NetworkManager.conf file I have:
[main]
plugins=ifcfg-rh
dns=dnsmasq

And with NetworkManager-0.9.4.0-9.git20120521.fc17 I have 127.0.0.1 in my resolv.conf file and I can query DNS on both the local network and the corporate network without issues. As soon as I upgrade to 0.9.6.4 I get the local DNS and when I connect to the VPN the VPN DNS and the local network breaks.

If I "yum downgrade NetworkManager*" it all starts working again.

Comment 1 Ian Pilcher 2012-11-08 16:32:32 UTC
I tried to git bisect this, but I wasn't able to get the dnsmasq plugin to work with 0.9.4.0 when I built it from source.  (It works just fine with 1:0.9.4.0-7.git20120403.fc17.)

I'd just like to emphasize what a huge issue this is.  I am completely reliant on split DNS to have access to resources on my employer's VPN and my local network when I'm working from home.  (This laptop is also a frequent traveler, so a static configuration is not an option.)  I'm sure that I'm not alone in this.

Comment 2 Roland Roberts 2012-11-09 01:37:59 UTC
Indeed, I just updated yesterday, much to my chagrin now since I also rely on split DNS to work on my employer's VPN. My primary workstation uses NFS and NIS so as soon as I bring up the VPN, I lose all access to filesystems on the local network.

I'm also downgrading which will hopefully get me back to a working configuration. Sorry I have nothing new to report except to confirm it is definitely 0.9.6.4-1.fc17 and it's definitely a big issue.

Comment 3 Jirka Klimes 2012-11-12 09:58:50 UTC
It appears the issue is SELinux-related. We've added a possibility to put custom dnsmasq configurations in /etc/NetworkManager/dnsmasq.d directory. However, we failed to inform SELinux guys that's OK for dnsmasq to read the directory.

You probably could see an AVC related to dnsmasq:
# ausearch -m avc | grep dnsmasq
or in /var/log/messages

Temporary workaround is to put SELinux into permissive mode, or create
custom policy allowing the access:
# grep dnsmasq /var/log/audit/audit.log | audit2allow -M mypol
# semodule -i mypol.pp

Bug 875648 has been filed to allow the access.

Anyway, would you attach?
$ cat /etc/resolv.conf
$ cat /var/run/nm-dns-dnsmasq.conf

Comment 4 Roland Roberts 2012-11-12 15:17:25 UTC
I was already running in permissive mode due to other selinux issues related to NFS home directories and colord. Note that I have no audit records related to dnsmasq:

[root@airy log]# ausearch -m avc | grep dnsmasq
[root@airy log]# 

Even searching older audit logs isn't turning up anything. Do you still want the other files for this case? When running 0.9.6.4, I don't think I was getting /var/run/nm-dns-dnsmasq.conf.

Comment 5 Peter Robinson 2012-11-13 06:30:48 UTC
I'm running in Enforcing and I'm getting the following deny statements on connect:

[306122.913920] type=1400 audit(1352787962.906:254): avc:  denied  { search } for  pid=8276 comm="dnsmasq" name="NetworkManager" dev="dm-2" ino=155049 scontext=system_u:system_r:dnsmasq_t:s0 tcontext=system_u:object_r:NetworkManager_etc_t:s0 tclass=dir
[306161.625524] type=1400 audit(1352788001.715:255): avc:  denied  { search } for  pid=8317 comm="dnsmasq" name="NetworkManager" dev="dm-2" ino=155049 scontext=system_u:system_r:dnsmasq_t:s0 tcontext=system_u:object_r:NetworkManager_etc_t:s0 tclass=dir

Setting to Permissive does in fact fix the problem for me. The question still remains why wasn't this tested in our default of Enforcing before it was pushed as an update?

Comment 6 Peter Robinson 2012-11-13 07:16:33 UTC
Actually setting it to Permissive doesn't completely fix the issue

Comment 7 Jirka Klimes 2012-11-13 10:08:59 UTC
(In reply to comment #4)
> I was already running in permissive mode due to other selinux issues related
> to NFS home directories and colord. Note that I have no audit records
> related to dnsmasq:
> 
> [root@airy log]# ausearch -m avc | grep dnsmasq
> [root@airy log]# 
> 
> Even searching older audit logs isn't turning up anything.
You probably don't have auditd.service running, so the log would go to /var/log/messages.

> Do you still want
> the other files for this case? When running 0.9.6.4, I don't think I was
> getting /var/run/nm-dns-dnsmasq.conf.

If you do
$ ps aux | grep dnsmasq
you should see 'dnsmasq' process with --conf-file=/var/run/nm-dns-dnsmasq.conf

The contents of the files would help, to verify that they were correctly created.


(In reply to comment #6)
> Actually setting it to Permissive doesn't completely fix the issue
Can you elaborate on this? Is it working or not?

Comment 8 Peter Robinson 2012-11-13 10:55:26 UTC
> (In reply to comment #6)
> > Actually setting it to Permissive doesn't completely fix the issue
> Can you elaborate on this? Is it working or not?

No, it's not. It looked to be at the initial quick test I did and then I discovered that it seemed to be getting further than it had previously but it was still not working in permissive mode. Local DNS were returning details with blank A records etc so I had to downgrade again so I could do some work.

Comment 9 Roland Roberts 2012-11-13 12:33:43 UTC
(In reply to comment #7)
[...]
> > Even searching older audit logs isn't turning up anything.
> You probably don't have auditd.service running, so the log would go to
> /var/log/messages.

auditd is running. The only avc messages in /var/log/messages are policyload messages.

I reupdated to 0.9.6.4. It would appear that the upgrade process to F17 did not create /etc/NetworkManager/dnsmasq.d

Nov 13 07:20:20 airy NetworkManager[917]: <info> DNS: loaded plugin dnsmasq
Nov 13 07:20:24 airy NetworkManager[917]: <info> DNS: starting dnsmasq...
Nov 13 07:20:24 airy dnsmasq[984]: cannot access directory /etc/NetworkManager/dnsmasq.d: No such file or directory
Nov 13 07:20:24 airy dnsmasq[984]: FAILED to start up
Nov 13 07:20:24 airy NetworkManager[917]: <warn> dnsmasq exited with error: Filesystem problem (missing file/directory; permissions; etc) (3)

Creating the directory gets dnsmasq to start, albeit with avc denials (permissive mode) for dnsmasq attempting to read from that directory.

It seems to be working for me now. Additionally, resolv.conf and /var/run/nm-dns-dnsmasq.conf are the same now as they were with 0.9.6.0.

Note that I did an update on everything which also picked up a new selinux, though since it's permissive mode that shouldn't matter:

Nov 13 06:57:56 Updated: 1:NetworkManager-glib-0.9.6.4-1.fc17.x86_64
Nov 13 06:57:59 Updated: 1:NetworkManager-0.9.6.4-1.fc17.x86_64
Nov 13 06:57:59 Updated: selinux-policy-3.10.0-159.fc17.noarch
Nov 13 06:58:00 Updated: 1:NetworkManager-gtk-0.9.6.4-1.fc17.x86_64
Nov 13 06:58:08 Updated: 1:NetworkManager-gnome-0.9.6.4-1.fc17.x86_64
Nov 13 06:58:19 Updated: selinux-policy-devel-3.10.0-159.fc17.noarch
Nov 13 06:58:53 Updated: selinux-policy-targeted-3.10.0-159.fc17.noarch
Nov 13 06:59:04 Updated: NetworkManager-openconnect-0.9.4.0-7.git20120612.fc17.x86_64

Comment 10 Jirka Klimes 2012-11-14 15:52:00 UTC
Roland, thanks a lot for providing the info. Indeed, the directory was not created by the package and that prevented dnsmasq from starting up. It will be fixed in the next update.

Comment 11 Fedora Update System 2012-11-14 15:52:45 UTC
NetworkManager-0.9.6.4-2.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/NetworkManager-0.9.6.4-2.fc17

Comment 12 Fedora Update System 2012-11-15 02:36:14 UTC
Package NetworkManager-0.9.6.4-2.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing NetworkManager-0.9.6.4-2.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-18229/NetworkManager-0.9.6.4-2.fc17
then log in and leave karma (feedback).

Comment 13 Peter Robinson 2012-11-15 08:17:10 UTC
This still isn't fixed with this update for me.

dnsmasq-2.63-1.fc17.x86_64
NetworkManager-0.9.6.4-2.fc17.x86_64
NetworkManager-vpnc-0.9.3.997-1.fc17.x86_64
NetworkManager-openvpn-0.9.3.997-1.fc17.x86_64
NetworkManager-gtk-0.9.6.4-2.fc17.x86_64
NetworkManager-devel-0.9.6.4-2.fc17.x86_64
NetworkManager-glib-0.9.6.4-2.fc17.x86_64
NetworkManager-gnome-0.9.6.4-2.fc17.x86_64
NetworkManager-pptp-0.9.3.997-1.fc17.x86_64

The /etc/resolv.conf gets updated to the VPN configuration rather than pointing to 127.0.0.1 like it did previously which means queries to the VPN work fine as do general internet bits but anything on the local network doesn't work.

Comment 14 Jirka Klimes 2012-11-15 10:29:45 UTC
(In reply to comment #13)
> The /etc/resolv.conf gets updated to the VPN configuration rather than
> pointing to 127.0.0.1 like it did previously which means queries to the VPN
> work fine as do general internet bits but anything on the local network
> doesn't work.

127.0.0.1 is put to resolv.conf only when a caching server is enabled.
So please make sure you still have "dns=dnsmasq" in [main] section of /etc/NetworkManager/NetworkManager.conf

When problems persist, please provide more information so it's possible to analyze your issue.

1) attach /var/log/messages
2) cat /etc/resolv.conf
3) cat /var/run/nm-dns-dnsmasq.conf
4) ps aux | grep dnsmasq
5) what VPN type do you use
6) any other relevant info ...

Comment 15 Peter Robinson 2012-11-15 12:42:10 UTC
(In reply to comment #14)
> (In reply to comment #13)
> > The /etc/resolv.conf gets updated to the VPN configuration rather than
> > pointing to 127.0.0.1 like it did previously which means queries to the VPN
> > work fine as do general internet bits but anything on the local network
> > doesn't work.
> 
> 127.0.0.1 is put to resolv.conf only when a caching server is enabled.
> So please make sure you still have "dns=dnsmasq" in [main] section of
> /etc/NetworkManager/NetworkManager.conf

Yes I did. I double check it every time. If it did change it would be due to the spec file not handling config files correctly :)

> When problems persist, please provide more information so it's possible to
> analyze your issue.
> 
> 1) attach /var/log/messages

The whole lot? There's a number of NM configuration details I can't add due to security reasons. Please be specific about the details you want to see.

> 2) cat /etc/resolv.conf

I do cat it. I can't attach it because of security reasons/

> 3) cat /var/run/nm-dns-dnsmasq.conf
> 4) ps aux | grep dnsmasq
> 5) what VPN type do you use

Cisco VPN using NetworkManager-vpnc connecting to RH corporate network.

> 6) any other relevant info ...

Basically the /etc/resolv.conf use to be updated to the following (some entries changed due to security). Post update I get the VPN configuration and hence no local DNS. So it ultimately doesn't matter a hoot whether dnsmasq is configured properly or not because if the /etc/resolv.conf isn't updated dnsmasq is never going to be used.

# cat /etc/resolv.conf 
# Generated by NetworkManager
domain redhat.com
search redhat.com blah.local
nameserver 127.0.0.1

# cat /etc/NetworkManager/NetworkManager.conf 
[main]
plugins=ifcfg-rh
dns=dnsmasq

Comment 16 Jirka Klimes 2012-11-15 14:20:36 UTC
Actually, it does matter whether dnsmasq can properly start. Because when NM detects that dnsmasq failed, it disables DNS caching and thus it puts regular DNS servers into resolv.conf instead of 127.0.0.1
And this is where the bugs were:
a) dnsmasq can be inhibited by SELinux due to /etc/NetworkManager/dnsmasq.d access - bug 875648
  please make sure you run in SELinux permissive mode
  or install selinux-policy-3.10.0-160.fc17 (it may be necessary to reboot or
  relabel the file, I guess)
b) The directory (/etc/NetworkManager/dnsmasq.d) might be missing which also causes dnsmasq failure (and resolv.conf populated by DNS servers) - this is addressed in NetworkManager-0.9.6.4-2.fc17

It's very hard to say what's wrong in your case without any data. In /var/log/messages we could see possible errors, like the dnsmasq starting error, AVCs, bunch of NM-DNS logs, or other relevant pointers.
So it may help when you attach the file as a private comment. Or if you are still reluctant to do that, please check for errors yourself.

Comment 17 Peter Robinson 2012-11-15 20:50:50 UTC
There's no mention what so ever that a new selinux-policy is required and while -160 is built for F-17 it's not been submitted as an update either separately or as part of the NM update that was submitted so there was no indication that a new selinux-policy was required to fix this. It works just fine with -159 using dnsmasq with NetworkManager-0.9.4.0-7.git20120403.fc17 so with just a NM update I'd kind of assumed the problem was fixed with the NM update.

I can submit data if I know exactly what needs to be submitted. I did previously submit AVCs above. I cannot submit any site specific data, even tagged private, so I can't submit blanket /var/log/messages.

Now that I know I need a new policy I'll try them both tomorrow. Why wasn't it submitted as part of the update?

Comment 18 Fedora Update System 2012-11-20 09:01:00 UTC
NetworkManager-0.9.6.4-3.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/NetworkManager-0.9.6.4-3.fc17

Comment 19 Fedora Update System 2012-11-20 09:53:27 UTC
NetworkManager-0.9.6.4-2.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/NetworkManager-0.9.6.4-2.fc16

Comment 20 Fedora Update System 2012-11-22 03:58:56 UTC
NetworkManager-0.9.6.4-3.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 21 Peter Robinson 2012-11-25 08:27:25 UTC
It's fixed now in F-17 the selinux-policy has been submitted as an update.

It's broken in F-18 it seems:
Nov 25 08:21:43 host dnsmasq[19728]: cannot access directory /etc/NetworkManager/dnsmasq.d: No such file or directory
Nov 25 08:21:43 host dnsmasq[19728]: FAILED to start up
Nov 25 08:21:43 host NetworkManager[805]: <warn> dnsmasq exited with error: Filesystem problem (missing file/directory; permissions; etc)

Comment 22 Fedora Update System 2012-11-27 13:56:51 UTC
NetworkManager-0.9.7.0-8.git20121004.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/NetworkManager-0.9.7.0-8.git20121004.fc18

Comment 23 Fedora Update System 2012-11-29 06:40:05 UTC
NetworkManager-0.9.7.0-8.git20121004.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 24 Fedora Update System 2013-01-23 16:10:54 UTC
NetworkManager-0.9.6.4-2.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.