Bug 1857820

Summary: libvirt can't start virtual networks
Product: [Fedora] Fedora Reporter: Bill Nottingham <notting>
Component: firewalldAssignee: Eric Garver <egarver>
Status: CLOSED DUPLICATE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 32CC: agedosier, berrange, clalancette, egarver, itamar, jforbes, laine, libvirt-maint, psutter, veillard, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-16 17:53:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Bill Nottingham 2020-07-16 15:49:07 UTC
Description of problem:

libvirt fails to start virtual networks

Error starting network 'default': error from service: changeZoneOfInterface: COMMAND_FAILED: '/usr/sbin/ip6tables-restore -w -n' failed: ip6tables-restore: line 14 failed


Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 66, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/network.py", line 75, in start
    self._backend.create()
  File "/usr/lib64/python3.8/site-packages/libvirt.py", line 3174, in create
    if ret == -1: raise libvirtError ('virNetworkCreate() failed', net=self)
libvirt.libvirtError: error from service: changeZoneOfInterface: COMMAND_FAILED: '/usr/sbin/ip6tables-restore -w -n' failed: ip6tables-restore: line 14 failed


(the 'line' number changes from time to time).


Version-Release number of selected component (if applicable):

libvirt-daemon-6.1.0-4.fc32.x86_64
firewalld-0.8.3-1.fc32.noarch
iptables-1.8.4-8.fc32.x86_64


How reproducible:
100%

Steps to Reproduce:
1. attempt to start VMs
2. fails due to no `default` virtual network
3. attempt to start default virtual network

Additional info:

Broke with recent updates, but downgrading does not fix it.

Comment 1 Bill Nottingham 2020-07-16 16:08:57 UTC
2020-07-16 12:02:18 DEBUG1: zone.getZones()
2020-07-16 12:02:18 DEBUG1: zone.changeZoneOfInterface('libvirt', 'virbr0')
2020-07-16 12:02:18 DEBUG1: Setting zone of interface 'virbr0' to 'libvirt'
2020-07-16 12:02:18 DEBUG1: Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/firewall/core/fw_transaction.py", line 128, in execute
    self.fw.rules(backend_name, rules[backend_name])
  File "/usr/lib/python3.8/site-packages/firewall/core/fw.py", line 874, in rules
    backend.set_rules(_rules, self._log_denied)
  File "/usr/lib/python3.8/site-packages/firewall/core/ipXtables.py", line 505, in set_rules
    raise ValueError("'%s %s' failed: %s" % (self._restore_command,
ValueError: '/usr/sbin/iptables-restore -w -n' failed: iptables-restore: line 14 failed


2020-07-16 12:02:18 ERROR: '/usr/sbin/iptables-restore -w -n' failed: iptables-restore: line 14 failed

2020-07-16 12:02:18 DEBUG1: Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/firewall/core/fw_transaction.py", line 128, in execute
    self.fw.rules(backend_name, rules[backend_name])
  File "/usr/lib/python3.8/site-packages/firewall/core/fw.py", line 874, in rules
    backend.set_rules(_rules, self._log_denied)
  File "/usr/lib/python3.8/site-packages/firewall/core/ipXtables.py", line 505, in set_rules
    raise ValueError("'%s %s' failed: %s" % (self._restore_command,
ValueError: '/usr/sbin/ip6tables-restore -w -n' failed: ip6tables-restore: line 14 failed


2020-07-16 12:02:18 ERROR: '/usr/sbin/ip6tables-restore -w -n' failed: ip6tables-restore: line 14 failed

2020-07-16 12:02:18 DEBUG1: Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/firewall/server/decorators.py", line 68, in dbus_handle_exceptions
    return func(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/firewall/server/firewalld.py", line 1185, in changeZoneOfInterface
    _zone = self.fw.zone.change_zone_of_interface(zone, interface, sender)
  File "/usr/lib/python3.8/site-packages/firewall/core/fw_zone.py", line 450, in change_zone_of_interface
    _zone = self.add_interface(zone, interface, sender)
  File "/usr/lib/python3.8/site-packages/firewall/core/fw_zone.py", line 428, in add_interface
    transaction.execute(True)
  File "/usr/lib/python3.8/site-packages/firewall/core/fw_transaction.py", line 173, in execute
    raise FirewallError(errors.COMMAND_FAILED, errorMsg)
firewall.errors.FirewallError: COMMAND_FAILED: '/usr/sbin/ip6tables-restore -w -n' failed: ip6tables-restore: line 14 failed


2020-07-16 12:02:18 ERROR: COMMAND_FAILED: '/usr/sbin/ip6tables-restore -w -n' failed: ip6tables-restore: line 14 failed

Comment 2 Daniel Berrangé 2020-07-16 16:10:05 UTC
This error is coming from firewalld, when libvirt invokes the changeZoneOfInterface DBus call.

We've recently seen issues with failures due to historical workarounds for docker, so I'm guessing this is probably be a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1829090

Comment 3 Eric Garver 2020-07-16 16:33:45 UTC
Can you check the firewalld log for errors? It may help to enable debug via /etc/sysconfig/firewalld.

Also try enabling IndividualCalls=yes in /etc/firewalld/firewalld.conf to get a better pointer to the failed iptables command.

Comment 4 Bill Nottingham 2020-07-16 17:43:29 UTC
Example errors from a simple service restart:
2020-07-16 13:38:33 DEBUG1: Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/firewall/core/fw.py", line 862, in rules
    backend.set_rule(rule, self._log_denied)
  File "/usr/lib/python3.8/site-packages/firewall/core/ipXtables.py", line 536, in set_rule
    output = self.__run(rule)
  File "/usr/lib/python3.8/site-packages/firewall/core/ipXtables.py", line 197, in __run
    raise ValueError("'%s %s' failed: %s" % (self._command,
ValueError: '/usr/sbin/iptables -w10 -I PREROUTING_ZONES 1 -t raw -i virbr0 -g PRE_libvirt' failed: iptables: No chain/target/match by that name.


2020-07-16 13:38:33 ERROR: '/usr/sbin/iptables -w10 -I PREROUTING_ZONES 1 -t raw -i virbr0 -g PRE_libvirt' failed: iptables: No chain/target/match by that name.

2020-07-16 13:38:34 DEBUG1: Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/firewall/core/fw_transaction.py", line 128, in execute
    self.fw.rules(backend_name, rules[backend_name])
  File "/usr/lib/python3.8/site-packages/firewall/core/fw.py", line 872, in rules
    raise msg
  File "/usr/lib/python3.8/site-packages/firewall/core/fw.py", line 862, in rules
    backend.set_rule(rule, self._log_denied)
  File "/usr/lib/python3.8/site-packages/firewall/core/ipXtables.py", line 536, in set_rule
    output = self.__run(rule)
  File "/usr/lib/python3.8/site-packages/firewall/core/ipXtables.py", line 197, in __run
    raise ValueError("'%s %s' failed: %s" % (self._command,
ValueError: '/usr/sbin/iptables -w10 -I PREROUTING_ZONES 1 -t raw -i virbr0 -g PRE_libvirt' failed: iptables: No chain/target/match by that name.


2020-07-16 13:38:34 ERROR: '/usr/sbin/iptables -w10 -I PREROUTING_ZONES 1 -t raw -i virbr0 -g PRE_libvirt' failed: iptables: No chain/target/match by that name.


When libvirt tries to start the default network:

2020-07-16 13:38:33 DEBUG1: Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/firewall/core/fw.py", line 862, in rules
    backend.set_rule(rule, self._log_denied)
  File "/usr/lib/python3.8/site-packages/firewall/core/ipXtables.py", line 536, in set_rule
    output = self.__run(rule)
  File "/usr/lib/python3.8/site-packages/firewall/core/ipXtables.py", line 197, in __run
    raise ValueError("'%s %s' failed: %s" % (self._command,
ValueError: '/usr/sbin/iptables -w10 -I PREROUTING_ZONES 1 -t raw -i virbr0 -g PRE_libvirt' failed: iptables: No chain/target/match by that name.


2020-07-16 13:38:33 ERROR: '/usr/sbin/iptables -w10 -I PREROUTING_ZONES 1 -t raw -i virbr0 -g PRE_libvirt' failed: iptables: No chain/target/match by that name.

2020-07-16 13:38:34 DEBUG1: Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/firewall/core/fw_transaction.py", line 128, in execute
    self.fw.rules(backend_name, rules[backend_name])
  File "/usr/lib/python3.8/site-packages/firewall/core/fw.py", line 872, in rules
    raise msg
  File "/usr/lib/python3.8/site-packages/firewall/core/fw.py", line 862, in rules
    backend.set_rule(rule, self._log_denied)
  File "/usr/lib/python3.8/site-packages/firewall/core/ipXtables.py", line 536, in set_rule
    output = self.__run(rule)
  File "/usr/lib/python3.8/site-packages/firewall/core/ipXtables.py", line 197, in __run
    raise ValueError("'%s %s' failed: %s" % (self._command,
ValueError: '/usr/sbin/iptables -w10 -I PREROUTING_ZONES 1 -t raw -i virbr0 -g PRE_libvirt' failed: iptables: No chain/target/match by that name.


2020-07-16 13:38:34 ERROR: '/usr/sbin/iptables -w10 -I PREROUTING_ZONES 1 -t raw -i virbr0 -g PRE_libvirt' failed: iptables: No chain/target/match by that name.

Comment 5 Bill Nottingham 2020-07-16 17:44:39 UTC
Take 2: service restart even when libvirtd isn't running yields:

2020-07-16 13:41:58 DEBUG1: Setting zone of interface 'wlp4s0' to 'FedoraWorkstation'
2020-07-16 13:41:58 DEBUG1: Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/firewall/core/fw.py", line 862, in rules
    backend.set_rule(rule, self._log_denied)
  File "/usr/lib/python3.8/site-packages/firewall/core/ipXtables.py", line 536, in set_rule
    output = self.__run(rule)
  File "/usr/lib/python3.8/site-packages/firewall/core/ipXtables.py", line 197, in __run
    raise ValueError("'%s %s' failed: %s" % (self._command,
ValueError: '/usr/sbin/iptables -w10 -I PREROUTING_ZONES 1 -t raw -i wlp4s0 -g PRE_FedoraWorkstation' failed: iptables: No chain/target/match by that name.


2020-07-16 13:41:58 ERROR: '/usr/sbin/iptables -w10 -I PREROUTING_ZONES 1 -t raw -i wlp4s0 -g PRE_FedoraWorkstation' failed: iptables: No chain/target/match by that name.

2020-07-16 13:41:58 DEBUG1: Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/firewall/core/fw_transaction.py", line 128, in execute
    self.fw.rules(backend_name, rules[backend_name])
  File "/usr/lib/python3.8/site-packages/firewall/core/fw.py", line 872, in rules
    raise msg
  File "/usr/lib/python3.8/site-packages/firewall/core/fw.py", line 862, in rules
    backend.set_rule(rule, self._log_denied)
  File "/usr/lib/python3.8/site-packages/firewall/core/ipXtables.py", line 536, in set_rule
    output = self.__run(rule)
  File "/usr/lib/python3.8/site-packages/firewall/core/ipXtables.py", line 197, in __run
    raise ValueError("'%s %s' failed: %s" % (self._command,
ValueError: '/usr/sbin/iptables -w10 -I PREROUTING_ZONES 1 -t raw -i wlp4s0 -g PRE_FedoraWorkstation' failed: iptables: No chain/target/match by that name.

Comment 6 Bill Nottingham 2020-07-16 17:53:12 UTC
Yep, it's https://bugzilla.redhat.com/show_bug.cgi?id=1829090.

*** This bug has been marked as a duplicate of bug 1829090 ***

Comment 7 Bill Nottingham 2020-07-16 17:53:53 UTC
Since it's technically a firewalld config, might be worth changing the component on https://bugzilla.redhat.com/show_bug.cgi?id=1829090 so it comes up easier in a search.