Bug 1731921

Summary:	Re-installing libvirt-daemon-driver-network kills the network connection on s390x host
Product:	Red Hat Enterprise Linux 8	Reporter:	Thomas Huth <thuth>
Component:	libvirt	Assignee:	Laine Stump <laine>
Status:	CLOSED DUPLICATE	QA Contact:	Virtualization Bugs <virt-bugs>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	8.1	CC:	cohuck, dzheng, egarver, laine, ldoktor, rbalakri, smitterl, yalzhang
Target Milestone:	rc	Keywords:	AutomationBlocker, TestBlocker
Target Release:	8.1
Hardware:	s390x
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-08-12 14:41:53 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1738779

Description Thomas Huth 2019-07-22 12:03:47 UTC

Description of problem:
Re-installing the "libvirt-daemon-driver-network" package on s390x LPAR puts the firewall into an unusable state, so that the host machine does not have any network connectivity anymore.

Version-Release number of selected component (if applicable):
libvirt-daemon-driver-network-4.5.0-30.module+el8.1.0+3574+3a63752b.s390x
kernel-4.18.0-119.el8.s390x
firewalld-0.7.0-2.el8.noarch

How reproducible:
100%

Steps to Reproduce:
1. Install libvirt on a s390x LPAR.
2. Run:
   dnf reinstall -y libvirt-daemon-driver-network

Actual results:
Network connectivity break, it's no longer possible to ssh into the system or ping it.

Expected results:
Network connectivity should continue to work.

Additional info:
In the output of "journalctl", I can see the following error message afterwards in the serial console:

ERROR: '/usr/sbin/nft add rule inet firewalld raw_PREROUTING jump raw_PREROUTING_ZONES' failed: Error: Could not process rule: No such file or directory
add rule inet firewalld raw_PREROUTING jump raw_PREROUTING_ZONES
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR: '/usr/sbin/nft add rule inet firewalld raw_PREROUTING jump raw_PREROUTING_ZONES' failed: Error: Could not process rule: No such file or directory
add rule inet firewalld raw_PREROUTING jump raw_PREROUTING_ZONES
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR: '/usr/sbin/nft add rule inet firewalld filter_IN_public jump filter_IN_public_pre' failed: Error: Could not process rule: No such file or directory
add rule inet firewalld filter_IN_public jump filter_IN_public_pre
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR: '/usr/sbin/nft add rule inet firewalld filter_IN_public jump filter_IN_public_pre' failed: Error: Could not process rule: No such file or directory
add rule inet firewalld filter_IN_public jump filter_IN_public_pre
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR: COMMAND_FAILED: '/usr/sbin/nft add rule inet firewalld filter_IN_public jump filter_IN_public_pre' failed: Error: Could not process rule: No such file or directory
add rule inet firewalld filter_IN_public jump filter_IN_public_pre
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR: '/usr/sbin/nft add rule inet firewalld raw_PREROUTING jump raw_PREROUTING_ZONES' failed: Error: Could not process rule: No such file or directory
add rule inet firewalld raw_PREROUTING jump raw_PREROUTING_ZONES
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR: '/usr/sbin/nft add rule inet firewalld raw_PREROUTING jump raw_PREROUTING_ZONES' failed: Error: Could not process rule: No such file or directory
add rule inet firewalld raw_PREROUTING jump raw_PREROUTING_ZONES
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR: '/usr/sbin/nft add rule inet firewalld filter_IN_public jump filter_IN_public_pre' failed: Error: Could not process rule: No such file or directory
add rule inet firewalld filter_IN_public jump filter_IN_public_pre
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR: '/usr/sbin/nft add rule inet firewalld filter_IN_public jump filter_IN_public_pre' failed: Error: Could not process rule: No such file or directory
add rule inet firewalld filter_IN_public jump filter_IN_public_pre
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR: COMMAND_FAILED: '/usr/sbin/nft add rule inet firewalld filter_IN_public jump filter_IN_public_pre' failed: Error: Could not process rule: No such file or directory
add rule inet firewalld filter_IN_public jump filter_IN_public_pre
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Running "systemctl restart firewalld" on the serial console fixes the network connectivity again.

Comment 1 Lukáš Doktor 2019-07-24 16:13:02 UTC

Hello guys, it is probably worth mentioning that since the broken update, I am unable to get "virbr0" from libvirt, reboot won't help, which means my machine is blocked and no testing can be performed there. Do you have any estimates? Should I attempt to find a workaround?

Comment 3 Laine Stump 2019-08-12 14:33:01 UTC

The error message is of a failure to run an nft command related to two chains that are created/controlled by firewalld (the "raw_PRE_libvirt" chain is apparently related to the firewalld zone named "libvirt". I notice that your firewalld has been rebased to 0.7.0 - Eric do you have any idea about this failure?

Comment 4 Eric Garver 2019-08-12 14:41:53 UTC

Marking as a duplicate of bug 1740182 as that one is more accurately describes the underlying problem.

As for this bug, some possible workarounds to unblock your testing:

 - downgrade nftables to nftables-0.9.0-8.el8.s390x
 - use the iptables backend for firewalld (FirewallBackend=iptables in /etc/firewalld/firewalld.conf)

*** This bug has been marked as a duplicate of bug 1740182 ***