Bug 1569744

Summary: Network via bridge device on host no longer works when firewalld is active
Product: [Fedora] Fedora Reporter: Adam Williamson <awilliam>
Component: firewalldAssignee: Eric Garver <egarver>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 28CC: agedosier, awilliam, berrange, clalancette, egarver, itamar, jforbes, jpopelka, laine, libvirt-maint, twoerner, veillard
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-30 18:13:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
filter
none
nat
none
raw
none
mangle none

Description Adam Williamson 2018-04-19 23:12:32 UTC
On my main test desktop, running F28, I have several VMs configured to access the network via a bridge device (br1) on the host system. In libvirt config this looks like:

    <interface type='bridge'>
      <mac address='52:54:00:ad:cd:65'/>
      <source bridge='br1'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

note I set these up via virt-manager , that's just the config virt-manager generated. These have worked fine for years, then in the last few weeks - sorry, I don't know exactly when this started - it doesn't work any more. The VMs (whatever OS they run) don't get network access. If I change the VM configuration to use NAT ("Virtual network 'default': NAT" in virt-manager) it works fine (but obviously not with the advantages of using a host bridge).

I don't see any obvious errors logged either to the journal or in /var/log/libvirt on the host. In the guest, if running Linux, 'ip addr' shows the device with no IPv4 address and only a 'link' scope IPv6 address; NetworkManager logs show DHCP timing out over and over again, basically.

This is with libvirt-4.1.0-2.fc28.x86_64 , currently. I tried downgrading to libvirt-4.0.0-2.fc28.x86_64 and that didn't seem to make any difference. Not sure how to debug this further.

Comment 1 Laine Stump 2018-04-20 14:47:19 UTC
Since the plumbing of a connection to the default virtual network is essentially the same as the plumbing for a connection to a bridge device, It seems more likely that there is a network configuration problem on your host (maybe iptables, maybe the config of the bridge), the DHCP server is no longer providing addresses to your guests, or that the port of your switch that you're connected to has been reconfigured to only allow the host's own MAC address.

I would try the following, and if they don't lead to the source of the problem, come join us on IRC in #virt to do more troubleshooting in real time:

* verify that "brctl show" lists both the guest's tap device and the host's physical ethernet as being connected to br1.

* try configuring a static IP address in the guest OS config, and see if a) the guest and host can communicate, and b) the guest and any outside address can communicate.

* change the config to be type='bridge', but with the source bridge='virbr0' (the bridge device of the default network) and see if everything works. If so, the the problem is definitely outside of libvirt.

*  run tcpdump or wireshark on the guest's tap device, on br1, and on the host's physical ethernet to see how far the dhcp request gets (or maybe it gets all the way out and a response is sent, but *it* doesn't make it all the way). That won't tell you what the problem is, but will at least let give you a better clue where to look.

Comment 2 Adam Williamson 2018-04-20 14:52:13 UTC
"It seems more likely that there is a network configuration problem on your host (maybe iptables, maybe the config of the bridge), the DHCP server is no longer providing addresses to your guests, or that the port of your switch that you're connected to has been reconfigured to only allow the host's own MAC address."

None of those seems *particularly* likely, as I haven't changed any of them. I don't have any custom firewall setup, it's just a desktop box; the firewall is just default firewalld. firewalld has had a couple of version bumps, I'll look into those. I haven't touched the configuration of the bridge either, and it still has the correct IP assigned, and I can still access the internet through it from the host system. I also run the DHCP server and haven't touched that at all, and it's handing out IP addresses fine to everything else; it's just a typical OpenWRT setup, it doesn't have any restrictions, it'll hand out an address to anything that shows up and asks for one. I also run the switch the machine is connected to and that's just a dumb consumer one bought off the shelf, haven't touched its configuration since it was hooked up.

I'll check some of those diagnostic steps, thanks.

Comment 3 Adam Williamson 2018-04-20 15:01:57 UTC
Aha. Long story short, it's firewalld on the host.

static IP in the guest makes ping work, but everything else fails with 'no route to host'; if I stop firewalld on the host, everything else suddenly starts working.

Re-assigning to firewalld, will try and pin down a version.

Comment 4 Adam Williamson 2018-04-21 00:35:37 UTC
So I actually have a ton of errors from firewalld every time I restart it:

Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table mangle --delete POSTROUTING --out-interface virbr0 --protocol udp --destination-port 68 --jump CHECKSUM --checksum-fill' failed: iptables: No chain/target/match by that name.
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 --destination 224.0.0.0/24 --jump RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -n -L DOCKER' failed: iptables: No chain/target/match by that name.
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 --destination 255.255.255.255/32 --jump RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 -p tcp ! --destination 192.168.122.0/24 --jump MASQUERADE --to-ports 1024-65535' failed: iptables: No chain/target/match by that name.
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -n -L DOCKER' failed: iptables: No chain/target/match by that name.
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 -p udp ! --destination 192.168.122.0/24 --jump MASQUERADE --to-ports 1024-65535' failed: iptables: No chain/target/match by that name.
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 ! --destination 192.168.122.0/24 --jump MASQUERADE' failed: iptables: No chain/target/match by that name.
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -n -L DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete FORWARD --destination 192.168.122.0/24 --out-interface virbr0 --match conntrack --ctstate ESTABLISHED,RELATED --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete FORWARD --source 192.168.122.0/24 --in-interface virbr0 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete FORWARD --in-interface virbr0 --out-interface virbr0 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete FORWARD --out-interface virbr0 --jump REJECT' failed: iptables: No chain/target/match by that name.
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete FORWARD --in-interface virbr0 --jump REJECT' failed: iptables: No chain/target/match by that name.
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE' failed: iptables: No chain/target/match by that name.
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete INPUT --in-interface virbr0 --protocol udp --destination-port 53 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete INPUT --in-interface virbr0 --protocol tcp --destination-port 53 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete OUTPUT --out-interface virbr0 --protocol udp --destination-port 68 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete INPUT --in-interface virbr0 --protocol udp --destination-port 67 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete INPUT --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed: iptables: No chain/target/match by that name.
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -j DOCKER' failed: iptables: No chain/target/match by that name.
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -j DOCKER-ISOLATION' failed: iptables: No chain/target/match by that name.
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -s 172.18.0.0/16 ! -o br-34679612069a -j MASQUERADE' failed: iptables: No chain/target/match by that name.
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -i br-34679612069a -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i br-34679612069a -o br-34679612069a -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i br-34679612069a -o br-34679612069a -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i br-34679612069a ! -o br-34679612069a -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o br-34679612069a -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o br-34679612069a -j DOCKER' failed: iptables: No chain/target/match by that name.
Apr 20 08:42:00 adam.happyassassin.net firewalld[9084]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).

downgrading firewalld doesn't help with this...but those errors don't happen on a clean install. :/ thinking back, I actually noticed this before, but didn't report it at the time for whatever reason. It may actually have started happening around the time I installed some docker-related packages. Will keep investigating and try to pin down what it is that triggers these errors...

Comment 5 Adam Williamson 2018-04-21 00:58:00 UTC
So there seems to be a set of errors that's triggered by having 'docker.service' enabled. Those are all the ones with 'docker' or 'DOCKER' in them, basically. I can recreate those in a clean VM by just doing this:

dnf install docker
systemctl enable docker
systemctl start docker
systemctl restart firewalld

seems like any time firewalld is (re)started while docker.service is running, this happens. https://github.com/moby/moby/issues/16137 seems like a long-running upstream discussion of that.

But there's a *second* set of errors which happens on my desktop even when docker.service is stopped, that's these ones:

Apr 20 17:44:27 adam.happyassassin.net firewalld[13261]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table mangle --delete POSTROUTING --out-interface virbr0 --protocol udp --destination-port 68 --jump CHECKSUM --checksum-fill' failed: iptables: No chain/target/match by that name.
Apr 20 17:44:27 adam.happyassassin.net firewalld[13261]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 --destination 224.0.0.0/24 --jump RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 17:44:27 adam.happyassassin.net firewalld[13261]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 --destination 255.255.255.255/32 --jump RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 17:44:27 adam.happyassassin.net firewalld[13261]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 -p tcp ! --destination 192.168.122.0/24 --jump MASQUERADE --to-ports 1024-65535' failed: iptables: No chain/target/match by that name.
Apr 20 17:44:27 adam.happyassassin.net firewalld[13261]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 -p udp ! --destination 192.168.122.0/24 --jump MASQUERADE --to-ports 1024-65535' failed: iptables: No chain/target/match by that name.
Apr 20 17:44:27 adam.happyassassin.net firewalld[13261]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table nat --delete POSTROUTING --source 192.168.122.0/24 ! --destination 192.168.122.0/24 --jump MASQUERADE' failed: iptables: No chain/target/match by that name.
Apr 20 17:44:27 adam.happyassassin.net firewalld[13261]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete FORWARD --destination 192.168.122.0/24 --out-interface virbr0 --match conntrack --ctstate ESTABLISHED,RELATED --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 17:44:27 adam.happyassassin.net firewalld[13261]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete FORWARD --source 192.168.122.0/24 --in-interface virbr0 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 17:44:27 adam.happyassassin.net firewalld[13261]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete FORWARD --in-interface virbr0 --out-interface virbr0 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 17:44:27 adam.happyassassin.net firewalld[13261]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete FORWARD --out-interface virbr0 --jump REJECT' failed: iptables: No chain/target/match by that name.
Apr 20 17:44:27 adam.happyassassin.net firewalld[13261]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete FORWARD --in-interface virbr0 --jump REJECT' failed: iptables: No chain/target/match by that name.
Apr 20 17:44:27 adam.happyassassin.net firewalld[13261]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete INPUT --in-interface virbr0 --protocol udp --destination-port 53 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 17:44:27 adam.happyassassin.net firewalld[13261]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete INPUT --in-interface virbr0 --protocol tcp --destination-port 53 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 17:44:27 adam.happyassassin.net firewalld[13261]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete OUTPUT --out-interface virbr0 --protocol udp --destination-port 68 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 17:44:27 adam.happyassassin.net firewalld[13261]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete INPUT --in-interface virbr0 --protocol udp --destination-port 67 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 20 17:44:27 adam.happyassassin.net firewalld[13261]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w --table filter --delete INPUT --in-interface virbr0 --protocol tcp --destination-port 67 --jump ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).

The trigger for *those* ones appears to be having libvirt-daemon-config-network installed and libvirtd started. I can reproduce those in the clean VM by doing this:

dnf install libvirt-daemon-config-network
systemctl enable libvirtd
systemctl restart libvirtd
systemctl restart firewalld

But just having libvirt-daemon installed and libvirtd running does *not* cause the errors, they really only happen if libvirt-daemon-config-network is installed.

I removed and cleaned out traces of libvirt-daemon-config-network from my desktop, disabled and stopped docker.service, restarted libvirtd and firewalld, and all the errors went away. Then I booted a VM with the networking set to the host bridge, and...network *still* doesn't work.

I am now officially annoyed. :P

Comment 6 Laine Stump 2018-04-22 22:14:56 UTC
Notice that all of those log messages are *warnings*, not errors. When libvirtd is told to reload all of its rules (triggered either by restarting libvirtd, or by firewalld  sending a message when it restarts itself), libvirt will first attempt to delete any existing rules that may be left over (so that there is a clean slate), and then add new identical rules. Usually this results in the old copy of a rule being deleted, then a new copy being added, but in some cases (e.g. when firewalld has restarted, which apparently deletes any direct/passthrough rules added by other software, even via firewalld) the old copy of the rule isn't there, so when we try to delete it we are returned a warning that the rule we want to delete doesn't exist.

That's fine with libvirt - we realize that sometimes the rules we are deleting as a safety precaution won't be there to begin with and we ignore the failure. But firewalld still insists on loging a message (it was previously an error log; we requested that they stop logging it completely since the messages were a red herring for libvirt users, but for some reason they don't want to stop logging these messages. There is a BZ about this [somewhere, too lazy to look right now]). Possibly there is a way to configure firewalld to not log warnings; don't know, never checked.

Anyway, all of the firewalld logs that warn about failure to delete rules including virbr0 in them are innocuous and can/should be ignored.

Comment 7 Adam Williamson 2018-07-10 22:23:03 UTC
Still having this problem, and honestly a bit stumped about how to debug it. Downgrading firewall and iptables doesn't seem to have helped. I can reproduce the problem at will, just booting a VM with bridged networking and disabling / enabling firewalld on the host flips the guest between being able to do stuff on the network and not being able to. But I can't really figure out what's blocking it specifically. iptables -L -v shows just a *ton* of rules, the purpose of each not being terribly clear from its name, and I'm not really sure where to start poking at it to find the cause.

Anyone got any ideas? Thanks.

Comment 8 Adam Williamson 2018-07-10 22:27:24 UTC
Note, one thing I did figure out: if I put br1 in the 'trusted' zone, things work. If I put it in 'home' or 'FedoraWorkstation' they do not. I guess I could just put it in 'trusted' permanently, but not sure if that's the correct fix...

Comment 9 Adam Williamson 2018-07-10 22:32:01 UTC
If I set 'Log Denied' in firewall-config to All, I get denial logs like this when I try to do `ping www.google.com` from the guest:

Jul 10 15:30:53 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=em1 PHYSOUT=vnet0 MAC=01:00:5e:00:00:fb:b8:ae:ed:72:91:f3:08:00 SRC=192.168.1.21 DST=224.0.0.251 LEN=161 TOS=0x00 PREC=0x00 TTL=255 ID=535 DF PROTO=UDP SPT=5353 DPT=5353 LEN=141 
Jul 10 15:30:54 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=em1 PHYSOUT=vnet0 MAC=01:00:5e:00:00:fb:b8:ae:ed:72:91:f3:08:00 SRC=192.168.1.21 DST=224.0.0.251 LEN=161 TOS=0x00 PREC=0x00 TTL=255 ID=544 DF PROTO=UDP SPT=5353 DPT=5353 LEN=141 
Jul 10 15:30:56 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=em1 PHYSOUT=vnet0 MAC=01:00:5e:00:00:fb:b8:ae:ed:72:91:f3:08:00 SRC=192.168.1.21 DST=224.0.0.251 LEN=161 TOS=0x00 PREC=0x00 TTL=255 ID=952 DF PROTO=UDP SPT=5353 DPT=5353 LEN=141

Comment 10 Adam Williamson 2018-07-10 22:33:42 UTC
The "PHYSOUT=vnet0" seems odd as I thought vnet0 was to do with libvirt's default NAT routing, and shouldn't be involved in bridged connections - perhaps that could be related?

Comment 11 Adam Williamson 2018-07-10 22:48:54 UTC
Well, I think that's actually bonjour stuff, but I also see these, when booting a VM. 192.168.1.191 is the VM, 192.168.1.1 is the router, 192.168.1.5 is the host system. br1 is the bridge interface on the host system, em1 is the underlying physical interface.

Jul 10 15:30:49 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=vnet0 PHYSOUT=em1 MAC=fc:f5:28:d3:5b:1a:52:54:00:ad:cd:65:08:00 SRC=192.168.1.191 DST=192.168.1.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=31987 DF PROTO=UDP SPT=38960 DPT=53 LEN=40 
Jul 10 15:30:49 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=vnet0 PHYSOUT=em1 MAC=fc:f5:28:d3:5b:1a:52:54:00:ad:cd:65:08:00 SRC=192.168.1.191 DST=192.168.1.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=31988 DF PROTO=UDP SPT=38960 DPT=53 LEN=40 
Jul 10 15:30:49 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=vnet0 PHYSOUT=em1 MAC=fc:f5:28:d3:5b:1a:52:54:00:ad:cd:65:08:00 SRC=192.168.1.191 DST=192.168.1.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=31989 DF PROTO=UDP SPT=53550 DPT=53 LEN=40 
Jul 10 15:30:49 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=vnet0 PHYSOUT=em1 MAC=fc:f5:28:d3:5b:1a:52:54:00:ad:cd:65:08:00 SRC=192.168.1.191 DST=192.168.1.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=31990 DF PROTO=UDP SPT=53550 DPT=53 LEN=40 
Jul 10 15:30:49 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=vnet0 PHYSOUT=em1 MAC=fc:f5:28:d3:5b:1a:52:54:00:ad:cd:65:08:00 SRC=192.168.1.191 DST=192.168.1.1 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=31991 DF PROTO=UDP SPT=40207 DPT=53 LEN=58 
Jul 10 15:30:49 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=vnet0 PHYSOUT=em1 MAC=fc:f5:28:d3:5b:1a:52:54:00:ad:cd:65:08:00 SRC=192.168.1.191 DST=192.168.1.1 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=31992 DF PROTO=UDP SPT=40207 DPT=53 LEN=58 
Jul 10 15:30:49 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=vnet0 PHYSOUT=em1 MAC=fc:f5:28:d3:5b:1a:52:54:00:ad:cd:65:08:00 SRC=192.168.1.191 DST=192.168.1.1 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=31993 DF PROTO=UDP SPT=53049 DPT=53 LEN=58 
Jul 10 15:30:49 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=vnet0 PHYSOUT=em1 MAC=fc:f5:28:d3:5b:1a:52:54:00:ad:cd:65:08:00 SRC=192.168.1.191 DST=192.168.1.1 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=31994 DF PROTO=UDP SPT=53049 DPT=53 LEN=58 
Jul 10 15:45:37 adam.happyassassin.net kernel: STATE_INVALID_DROP: IN=br1 OUT= PHYSIN=em1 MAC=f4:6d:04:9a:1d:45:fc:f5:28:d3:5b:1a:08:00 SRC=216.58.217.36 DST=192.168.1.5 LEN=40 TOS=0x00 PREC=0x00 TTL=59 ID=37200 PROTO=TCP SPT=443 DPT=45716 WINDOW=0 RES=0x00 RST URGP=0 
Jul 10 15:45:37 adam.happyassassin.net kernel: STATE_INVALID_DROP: IN=br1 OUT= PHYSIN=em1 MAC=f4:6d:04:9a:1d:45:fc:f5:28:d3:5b:1a:08:00 SRC=216.58.217.36 DST=192.168.1.5 LEN=40 TOS=0x00 PREC=0x00 TTL=59 ID=37201 PROTO=TCP SPT=443 DPT=45716 WINDOW=0 RES=0x00 RST URGP=0 
Jul 10 15:45:37 adam.happyassassin.net kernel: STATE_INVALID_DROP: IN=br1 OUT= PHYSIN=em1 MAC=f4:6d:04:9a:1d:45:fc:f5:28:d3:5b:1a:08:00 SRC=216.58.217.36 DST=192.168.1.5 LEN=40 TOS=0x00 PREC=0x00 TTL=59 ID=37205 PROTO=TCP SPT=443 DPT=45716 WINDOW=0 RES=0x00 RST URGP=0 
Jul 10 15:45:37 adam.happyassassin.net kernel: STATE_INVALID_DROP: IN=br1 OUT= PHYSIN=em1 MAC=f4:6d:04:9a:1d:45:fc:f5:28:d3:5b:1a:08:00 SRC=216.58.217.36 DST=192.168.1.5 LEN=40 TOS=0x00 PREC=0x00 TTL=59 ID=37206 PROTO=TCP SPT=443 DPT=45716 WINDOW=0 RES=0x00 RST URGP=0 
Jul 10 15:46:06 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=vnet0 PHYSOUT=em1 MAC=ff:ff:ff:ff:ff:ff:52:54:00:ad:cd:65:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=354 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=334 
Jul 10 15:46:06 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT= PHYSIN=vnet0 MAC=ff:ff:ff:ff:ff:ff:52:54:00:ad:cd:65:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=354 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=334 
Jul 10 15:46:11 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=vnet0 PHYSOUT=em1 MAC=ff:ff:ff:ff:ff:ff:52:54:00:ad:cd:65:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=354 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=334 
Jul 10 15:46:11 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT= PHYSIN=vnet0 MAC=ff:ff:ff:ff:ff:ff:52:54:00:ad:cd:65:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=354 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=334 
Jul 10 15:46:22 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=vnet0 PHYSOUT=em1 MAC=ff:ff:ff:ff:ff:ff:52:54:00:ad:cd:65:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=354 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=334 
Jul 10 15:46:22 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT= PHYSIN=vnet0 MAC=ff:ff:ff:ff:ff:ff:52:54:00:ad:cd:65:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=354 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=334 
Jul 10 15:46:32 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=vnet0 PHYSOUT=em1 MAC=ff:ff:ff:ff:ff:ff:52:54:00:ad:cd:65:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=354 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=334 
Jul 10 15:46:32 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT= PHYSIN=vnet0 MAC=ff:ff:ff:ff:ff:ff:52:54:00:ad:cd:65:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=354 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=334 
Jul 10 15:46:40 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=em1 PHYSOUT=vnet0 MAC=01:00:5e:00:00:01:fc:f5:28:d3:5b:1a:08:00 SRC=0.0.0.0 DST=224.0.0.1 LEN=32 TOS=0x00 PREC=0xC0 TTL=1 ID=0 DF PROTO=2 
Jul 10 15:46:40 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT= PHYSIN=em1 MAC=01:00:5e:00:00:01:fc:f5:28:d3:5b:1a:08:00 SRC=0.0.0.0 DST=224.0.0.1 LEN=32 TOS=0x00 PREC=0xC0 TTL=1 ID=0 DF PROTO=2 
Jul 10 15:46:43 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=vnet0 PHYSOUT=em1 MAC=ff:ff:ff:ff:ff:ff:52:54:00:ad:cd:65:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=354 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=334 
Jul 10 15:46:43 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT= PHYSIN=vnet0 MAC=ff:ff:ff:ff:ff:ff:52:54:00:ad:cd:65:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=354 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=334 
Jul 10 15:46:51 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=vnet0 PHYSOUT=em1 MAC=ff:ff:ff:ff:ff:ff:52:54:00:ad:cd:65:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=354 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=334 
Jul 10 15:46:51 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT= PHYSIN=vnet0 MAC=ff:ff:ff:ff:ff:ff:52:54:00:ad:cd:65:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=354 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=334 
Jul 10 15:46:57 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=vnet0 PHYSOUT=em1 MAC=ff:ff:ff:ff:ff:ff:52:54:00:ad:cd:65:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=354 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=334 
Jul 10 15:46:57 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT= PHYSIN=vnet0 MAC=ff:ff:ff:ff:ff:ff:52:54:00:ad:cd:65:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=354 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=334 
Jul 10 15:47:05 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=vnet0 PHYSOUT=em1 MAC=ff:ff:ff:ff:ff:ff:52:54:00:ad:cd:65:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=354 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=334 
Jul 10 15:47:05 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT= PHYSIN=vnet0 MAC=ff:ff:ff:ff:ff:ff:52:54:00:ad:cd:65:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=354 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=334 
Jul 10 15:47:18 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT=br1 PHYSIN=vnet0 PHYSOUT=em1 MAC=ff:ff:ff:ff:ff:ff:52:54:00:ad:cd:65:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=354 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=334 
Jul 10 15:47:18 adam.happyassassin.net kernel: FINAL_REJECT: IN=br1 OUT= PHYSIN=vnet0 MAC=ff:ff:ff:ff:ff:ff:52:54:00:ad:cd:65:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=354 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=334

Comment 12 Eric Garver 2018-07-11 17:40:23 UTC
Adam, Can you attach the output of the following after a reboot?
 
  - "iptables -nvL -t filter"
  - "iptables -nvL -t nat"
  - "iptables -nvL -t raw"
  - "iptables -nvL -t mangle"

Comment 13 Adam Williamson 2018-07-11 20:01:07 UTC
Created attachment 1458213 [details]
filter

Comment 14 Adam Williamson 2018-07-11 20:01:55 UTC
Created attachment 1458214 [details]
nat

Comment 15 Adam Williamson 2018-07-11 20:02:19 UTC
Created attachment 1458215 [details]
raw

Comment 16 Adam Williamson 2018-07-11 20:02:53 UTC
Created attachment 1458216 [details]
mangle

Comment 17 Eric Garver 2018-07-12 19:24:12 UTC
Adam, When you created the iptables output did you have a VM in issue state (i.e. not getting an IP via DHCP)?

Also can you show ebtables output? e.g. ebtables -t  -L

Comment 18 Adam Williamson 2018-07-12 19:29:32 UTC
"Adam, When you created the iptables output did you have a VM in issue state (i.e. not getting an IP via DHCP)?"

No - you asked for it "after a reboot", so I assumed you wanted it as clean as possible. Would you like the same output before and after reproducing the issue?

"Also can you show ebtables output? e.g. ebtables -t  -L"

will do.

Comment 19 Eric Garver 2018-07-13 13:52:02 UTC
(In reply to Adam Williamson from comment #18)
> "Adam, When you created the iptables output did you have a VM in issue state
> (i.e. not getting an IP via DHCP)?"
> 
> No - you asked for it "after a reboot", so I assumed you wanted it as clean
> as possible. Would you like the same output before and after reproducing the
> issue?

My fault - I wasn't clear. Please reboot, reproduce, then show the iptables/ebtables output.

Comment 20 Adam Williamson 2018-07-13 22:51:20 UTC
Well, crap, now it's gone and turned into a heisenbug...I just rebooted and it's *working* now, with firewalld enabled. Grr.

I'll give it a week or so and if I still can't break it again, guess I'll just close the bug...

Comment 21 Eric Garver 2018-07-16 13:21:57 UTC
(In reply to Adam Williamson from comment #20)
> Well, crap, now it's gone and turned into a heisenbug...I just rebooted and
> it's *working* now, with firewalld enabled. Grr.
> 
> I'll give it a week or so and if I still can't break it again, guess I'll
> just close the bug...

No problem. I'll leave this open and close it if I don't hear anything from you after a couple weeks.

Comment 22 Eric Garver 2018-07-30 18:13:38 UTC
(In reply to Eric Garver from comment #21)
> (In reply to Adam Williamson from comment #20)
> > Well, crap, now it's gone and turned into a heisenbug...I just rebooted and
> > it's *working* now, with firewalld enabled. Grr.
> > 
> > I'll give it a week or so and if I still can't break it again, guess I'll
> > just close the bug...
> 
> No problem. I'll leave this open and close it if I don't hear anything from
> you after a couple weeks.

It's been two weeks. Closing.

Comment 23 Adam Williamson 2019-11-13 02:17:25 UTC
For the record I don't think I did manage to reproduce this again. My main test VM is still using a bridge and firewalld and I haven't seen it lately for sure.