Bug 1829090

I am seeing the same issue, Testing seems to show some kind of an issue with NAT networks and nftables. Problems persist on attempting to create a new NAT network.


Error starting network 'default': error from service: changeZoneOfInterface: COMMAND_FAILED: 'python-nftables' failed: 
JSON blob:
{"nftables": [{"metainfo": {"json_schema_version": 1}}, {"add": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_IN_libvirt_allow", "expr": [{"match": {"left": {"payload": {"protocol": "udp", "field": "dport"}}, "op": "==", "right": 67}}, {"match": {"left": {"ct": {"key": "state"}}, "op": "in", "right": {"set": ["new", "untracked"]}}}, {"accept": null}]}}}, {"add": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_IN_libvirt_allow", "expr": [{"match": {"left": {"payload": {"protocol": "udp", "field": "dport"}}, "op": "==", "right": 547}}, {"match": {"left": {"ct": {"key": "state"}}, "op": "in", "right": {"set": ["new", "untracked"]}}}, {"accept": null}]}}}, {"add": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_IN_libvirt_allow", "expr": [{"match": {"left": {"payload": {"protocol": "tcp", "field": "dport"}}, "op": "==", "right": 53}}, {"match": {"left": {"ct": {"key": "state"}}

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 66, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/network.py", line 75, in start
    self._backend.create()
  File "/usr/lib64/python3.8/site-packages/libvirt.py", line 3174, in create
    if ret == -1: raise libvirtError ('virNetworkCreate() failed', net=self)
libvirt.libvirtError: error from service: changeZoneOfInterface: COMMAND_FAILED: 'python-nftables' failed: 
JSON blob:
{"nftables": [{"metainfo": {"json_schema_version": 1}}, {"add": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_IN_libvirt_allow", "expr": [{"match": {"left": {"payload": {"protocol": "udp", "field": "dport"}}, "op": "==", "right": 67}}, {"match": {"left": {"ct": {"key": "state"}}, "op": "in", "right": {"set": ["new", "untracked"]}}}, {"accept": null}]}}}, {"add": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_IN_libvirt_allow", "expr": [{"match": {"left": {"payload": {"protocol": "udp", "field": "dport"}}, "op": "==", "right": 547}}, {"match": {"left": {"ct": {"key": "state"}}, "op": "in", "right": {"set": ["new", "untracked"]}}}, {"accept": null}]}}}, {"add": {"rule": {"family": "inet", "table": "firewalld", "chain": "filter_IN_libvirt_allow", "expr": [{"match": {"left": {"payload": {"protocol": "tcp", "field": "dport"}}, "op": "==", "right": 53}}, {"match": {"left": {"ct": {"key": "state"}}

I get the same error when running a VM from virt-manager, and when trying to bring up the default network (virbr0), I get the following:

$: sudo virsh net-start default

error: Failed to start network default
error: Error desde el servicio: changeZoneOfInterface: COMMAND_FAILED: '{"%%ZONE_INTERFACE%%": null, "chain": "raw_PREROUTING_ZONES", "expr": [{"match": {"left": {"meta": {"key": "iifname"}}, "op": "==", "right": "virbr0"}}, {"goto": {"target": "raw_PRE_trusted"}}], "family": "inet", "table": "firewalld"}'

I started seeing this today.

After a lot of digging I noticed on boot an error in the journal:
"May 26 11:19:08 jmontleo.usersys.redhat.com firewalld[2850]: ERROR: ZONE_CONFLICT: 'docker0' already bound to a zone"

This ended up being because I used the workaround of adding docker0 to the trusted zone for:
https://bugzilla.redhat.com/show_bug.cgi?id=1817022

It looks like a new moby-engine package went to stable a few days ago which added a docker.xml zone file in /usr/lib/firewalld/zones.

I ended up having to delete /etc/firewalld/zones/trusted.xml to remove the workaround. One this was done and I rebooted libvirt (and my ethernet interface for that matter!) ended up in the correct zones.

It is not great that firewalld stops adding interfaces to zones and becomes pretty unusable when it hits a zone conflict like that. Might be better to move this to firewalld to see if they can do anything about that.

I have same problem as described.

I've tried to disect this issue and found out that removing docker (moby-engine / containerd) fixed this. One thing i noticed is that when docker is installed new br-... interface is created (ifconfig) and libvirt is unable to start any network.

This issue (with different errors messages) occurs on both iptables and nftables backends for firewalld. I've changed to iptables to fix containers access to external network and I've reversed to nftables for testing.

As suggested in comment #3 - deleting the "/etc/firewalld/zones/trusted.xml" file and restarting the OS fixed the issue of VMs not launching in virt-manager.

+1 to deleting /etc/firewalld/zones/trusted.xml for aforementioned reasons.

I did not need to reboot, just restart firewalld service, then restart libvirtd service.

Closing since evidence points towards the problem being firewalld error due to bad docker zone config.

It's not completely clear why the added trusted.xml is the cause here. Dut docker/moby is certainly doing something wrong. My libvirt networking broke when I updated fedora 32 - it was fine when I first installed, so breakage must be after the fedora 32 release.

Interestingly I see

# firewall-cmd --list-services
You're performing an operation over default zone ('FedoraWorkstation'),
but your connections/interfaces are in zone 'docker' (see --get-active-zones)
You most likely need to use --zone=docker option.

dhcpv6-client mdns postgresql samba-client ssh

# firewall-cmd --get-active-zones
docker
  interfaces: docker0

# firewall-cmd --get-zones
FedoraServer FedoraWorkstation block dmz docker drop external home internal libvirt nm-shared public trusted work

# firewall-cmd --get-zone-of-interface docker0
docker

# firewall-cmd --get-zone-of-interface lo
no zone

# firewall-cmd --list-interfaces --zone docker
docker0


So it seems like by putting the bridge in the trusted 'docker' zone, moby has inadvertently hijacked zoning for the system.

*** Bug 1857820 has been marked as a duplicate of this bug. ***

(In reply to Craig Ringer from comment #8)
> So it seems like by putting the bridge in the trusted 'docker' zone, moby
> has inadvertently hijacked zoning for the system.

See this upstream moby change: https://github.com/moby/libnetwork/pull/2548

(In reply to Tadas Giniotis from comment #5)
> As suggested in comment #3 - deleting the "/etc/firewalld/zones/trusted.xml"
> file and restarting the OS fixed the issue of VMs not launching in
> virt-manager.

I tried this - then I needed to manually start default -- `virsh net-start default
`