Bug 1213194 - Firewalld changed processing of direct rules
Summary: Firewalld changed processing of direct rules
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: firewalld
Version: 22
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Thomas Woerner
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-04-20 01:10 UTC by Dan Mossor [danofsatx]
Modified: 2016-07-19 19:07 UTC (History)
11 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2016-07-19 19:07:04 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
firewalld direct rules (898 bytes, application/xml)
2015-04-20 09:38 UTC, Dan Mossor [danofsatx]
no flags Details
/etc/sysconfig/ipset.save (2.86 MB, text/plain)
2015-04-20 10:19 UTC, Dan Mossor [danofsatx]
no flags Details
output of iptables-save (11.63 KB, text/plain)
2015-04-20 10:20 UTC, Dan Mossor [danofsatx]
no flags Details

Description Dan Mossor [danofsatx] 2015-04-20 01:10:11 UTC
Description of problem:
Fedora 22 Beta has partial connectivity to the internet.

Version-Release number of selected component (if applicable):
NetworkManager-1.0.0-8.fc22.x86_64
NetworkManager-config-connectivity-fedora-1.0.0-8.fc22.x86_64
NetworkManager-wifi-1.0.0-8.fc22.x86_64
NetworkManager-libnm-1.0.0-8.fc22.x86_64
NetworkManager-glib-1.0.0-8.fc22.x86_64

How reproducible:
In my case, 100%, In others, varies from 50% to 100%.

Steps to Reproduce:
1. Install Fedora 22 Beta
2. Configure it to connect to a wired or wireless network.
3. Use the system for a few hours.

Actual results:
Connectivity begins to fail.

Expected results:
No noticeable issues.

Additional info:
This is a hard one to track down. Some sites work just fine - all Fedora Project (except one), Red Hat, GNOME, Freedesktop, Arch Linux and openSUSE sites work. id.fedoraproject.org, debian.org, ubuntu.com, and various sites from within the Stack Exchange family do not work. www.rpmfusion.org and download1.rpmfusion.org work, but mirrors.rpmfusion.org doesn't which breaks dnf because it can't read the mirror list. Google searches work fine (the entire Google infrastructure does), but only 2 or 3 links off of each page of search results is actually reachable from F22.

It is not my network - I have Android, ChromeOS and Windows machines on the same network using the same equipment and configurations that have ZERO issues with these same sites that F22 cannot reach. One of my affected systems was just updated to F22 this weekend from F21 and NetworkManager 0.9.8 - it exhibited zero issues before the update. I started testing the wireless, thinking it may be an Intel Driver issue, but when both systems' NICs (Realtek on one, Qualcomm Atheros on another) are plugged into an ethernet cable, the problems remain. Multiple reboots of both affected systems shows zero change.

The one common thread through all of this is NetworkManager 1.0.0-8.

Comment 1 Adam Williamson 2015-04-20 05:06:49 UTC
I haven't seen anything like this, and I've had 1.0.0-8 installed for a month.

Comment 2 Thomas Haller 2015-04-20 09:03:08 UTC
When certain sites don't work, does that mean that they are not resolvable via DNS? Can you ping mirrors.rpmfusion.org?

If you run traceroute to whose failed destination, does that give any useful hints?


NetworkManager only configures networking on your system, e.g. it configures addresses, DNS (resolv.conf), etc.
It does not do networking itself (still done by the kernel) or name-resolution (still done by libc/resolver).


If this is a NetworkManager issue, it would be helpful to understand what NM misconfigured to cause your issues. What is /etc/resolv.conf for you? Can you resolve those destinations? Iptables/firewalld?

You don't use a http proxy, do you?


This doesn't look like a NetworkManager issue...

Comment 3 Dan Mossor [danofsatx] 2015-04-20 09:36:15 UTC
I can resolve the sites just fine. My resolv.conf contains one entry, that of my gateway device and assigned via DHCP.

Traceroutes and pings to the affected sites do not work - traceroute gets exactly one return, the gateway device.

You may be on to something with the IPTables/firewalld idea - I've got a direct rule in place to use ipset lists, and the lists it is checking are GeoIP blocks.  For testing, I've removed the Great Britain and Germany IP blocks from my ipset tables for ubuntu,com and mirrors.rpmfusion.org respectively, but the problem remains. I'll attach my direct.xml for reference.

Comment 4 Dan Mossor [danofsatx] 2015-04-20 09:38:44 UTC
Created attachment 1016275 [details]
firewalld direct rules

/etc/firewalld/direct.xml - the direct rules that run all traffic through my GeoIP ipset tables.

Comment 5 Thomas Haller 2015-04-20 09:46:18 UTC
(In reply to Dan Mossor from comment #3)
> I can resolve the sites just fine. My resolv.conf contains one entry, that
> of my gateway device and assigned via DHCP.
> 
> Traceroutes and pings to the affected sites do not work - traceroute gets
> exactly one return, the gateway device.

You don't have any IPv4 routes that treat failing destinations any differently from non-failing ones? See `ip route show`, or `ip -6 route show` if applicable.

If traceroute indicates that the network traffic reaches the gateway but not any further, it seems that your gateway/provider causes the problem.

Comment 6 Dan Mossor [danofsatx] 2015-04-20 09:53:29 UTC
[dan@g55 .ssh]$ ip route show
default via 192.168.1.254 dev enp4s0  proto static  metric 100 
192.168.1.0/24 dev enp4s0  proto kernel  scope link  src 192.168.1.51 
192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1

I fail to see how it can be the gateway if other systems using the same gateway are unaffected, and one of the systems with issues had no issues before the update to F22.

I will reboot the gateway again, just to troubleshoot (and I'm the only one awake at this horrible hour, so the family won't be complaining to me).

Comment 7 Dan Mossor [danofsatx] 2015-04-20 10:14:00 UTC
After a reboot of the gateway, the problem remains. I just realized the reason I don't get any traceroutes past the gateway is because AT&T seems to have disabled ICMP traffic originating from within their residential customer networks. I can traceroute and ping my public IP from my VPS, but not vice-versa.

But, this laptop cannot reach certain sites. My phone can reach those sites.

I just ran another test disabling firewalld and those sites are accessible. So, it seems the error is in firewalld not in NetworkManager.

Comment 8 Dan Mossor [danofsatx] 2015-04-20 10:17:18 UTC
On F21, the rules in the attached direct.xml in comment 4 worked just fine. After the update to F22, something in the way those rules are processed changed, blocking outgoing access to IPs contained in those ipsets.

firewalld-0.3.13-5.fc22.noarch

Comment 9 Dan Mossor [danofsatx] 2015-04-20 10:19:27 UTC
Created attachment 1016300 [details]
/etc/sysconfig/ipset.save

The ipset lists that the direct.xml is reading

Comment 10 Dan Mossor [danofsatx] 2015-04-20 10:20:57 UTC
Created attachment 1016301 [details]
output of iptables-save

The iptables rules generated by firewalld.

Comment 11 Stephen Gallagher 2015-04-20 12:16:33 UTC
So, I've had similar issues with NetworkManager and/or firewalld in Fedora 22 (but not in F21) on my Lenovo T540p.

On my home network, if I am connected to all three of
 * My wired network
 * My wireless network
 * OpenVPN into work (untested with vpnc)

I experience a loss of connectivity like Dan describes after several minutes (never more than 30) except to those sites internal to the VPN. I have no special firewall rules in my environment; I'm using the Verizon FiOS router directly.

Just as Dan describes, if I traceroute, the connection appears to terminate at my Verizon router, however every other device in my network retains full connectivity. Only Fedora 22 is affected.

If I disable either the wired network or the wireless network, the problem never occurs. I haven't been able to spot any obvious differences in 'ip addr', 'ip route', etc., which is why I hadn't filed a bug yet.

Network Hardware:
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-LM (rev 04)
04:00.0 Network controller: Intel Corporation Wireless 7260 (rev 83)

Comment 12 Jiri Popelka 2015-04-20 12:53:40 UTC
(In reply to Dan Mossor from comment #8)
> On F21, the rules in the attached direct.xml in comment 4 worked just fine.
> After the update to F22, something in the way those rules are processed
> changed, blocking outgoing access to IPs contained in those ipsets.

We have the same versions of firewalld (0.3.13) and iptables (1.4.21) in both F21 & F22.
There's ipset-6.22 in F22, while F21 ships ipset-6.21 so you might try ipset-6.21
https://koji.fedoraproject.org/koji/buildinfo?buildID=558955
on the other hand Stephen doesn't AFAICT use ipsets so it's unlikely the culprit.

Comment 13 Adam Williamson 2015-04-20 14:23:03 UTC
Well, I see another potential significant difference between sgallagh's case and Dan's: Dan says "Configure it to connect to a wired *or* wireless network." (emphasis added), but Stephen says it only happens when he's connected to *both*.

Stephen, you also didn't actually say clearly that you still have connectivity to *some* sites, which is a significant feature of Dan's report - is that the case?

Comment 14 Stephen Gallagher 2015-04-20 14:35:17 UTC
(In reply to awilliam from comment #13)
> Well, I see another potential significant difference between sgallagh's case
> and Dan's: Dan says "Configure it to connect to a wired *or* wireless
> network." (emphasis added), but Stephen says it only happens when he's
> connected to *both*.
> 

Right; it happens very quickly when I am using both.


> Stephen, you also didn't actually say clearly that you still have
> connectivity to *some* sites, which is a significant feature of Dan's report
> - is that the case?

Sorry, I thought I had. I retain connectivity to any site that routes through the VPN, but I lose access to everything else on the internet. Interestingly, if the VPN isn't engaged, I will still experience the issue, but have zero access at all. I can't recall if my local subnet was accessible when this is happening (and I'm not at home today to test it; I'll do so tonight or tomorrow).

Comment 15 Dan Mossor [danofsatx] 2015-04-26 01:42:55 UTC
This appears to be a progressive issue. When I flushed my IPtables rules and restarted firewalld, all of the rules ad ipset address blocks were rebuilt and put into place. This worked fine for 4 days before outgoing connections to addresses contained in the ipset rules started getting blocked.

Is this an error in firewalld, or are my rules built incorrectly?

Comment 16 Dan Mossor [danofsatx] 2015-04-26 01:45:56 UTC
Oh, and sorry Jiri - I missed your note. I'll try downgrading ipset and see if that resolves it.

Comment 17 Fedora End Of Life 2016-07-19 19:07:04 UTC
Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.