Bug 2073316

Summary:	default network doesn't start
Product:	[Fedora] Fedora	Reporter:	Dan Horák <dan>
Component:	iptables	Assignee:	Phil Sutter <psutter>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	36	CC:	agedosier, berrange, clalancette, crobinso, jforbes, kevin, laine, libvirt-maint, psutter, tstaudt, veillard, virt-maint
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-05-25 11:17:20 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	467765

Description Dan Horák 2022-04-08 08:14:01 UTC

Seems the default network can't start any more on my Fedora 35 s390x system with 5.18-rc kernels. After downgrading to kernel 5.17 the network starts correctly. Interestingly I don't see such problem on my Rawhide ppc64le system (with libvirt 8.2.0 and 5.18-rc1 kernel)

[sharkcz@rock-kvmlp-fedora ~]$ virsh net-info --network default 
Name:           default
UUID:           d0be7373-95c7-4eeb-80c2-67568480102d
Active:         no
Persistent:     yes
Autostart:      yes
Bridge:         virbr0

[sharkcz@rock-kvmlp-fedora ~]$ virsh net-start --network default 
error: Failed to start network default
error: internal error: Failed to apply firewall rules /usr/sbin/iptables -w --table nat --insert POSTROUTING --jump LIBVIRT_PRT: iptables v1.8.7 (nf_tables):  CHAIN_ADD failed (No such file or directory): chain POSTROUTING

[sharkcz@rock-kvmlp-fedora ~]$ sudo iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain LIBVIRT_PRT (0 references)
target     prot opt source               destination         


Version-Release number of selected component (if applicable):
libvirt-8.2.0-1.fc35.s390x - from virt-preview
kernel-5.18.0-0.rc1.18.fc37.s390x - from rawhide-nodebug
kernel-5.18.0-0.rc1.20220406git3e732ebf7316ac8.19.fc37.s390x - from rawhide


How reproducible:
100%

Comment 1 Laine Stump 2022-04-08 14:43:51 UTC

It's odd that the output of iptables -L would show the POSTROUTING chain, but yet that is what apparently fails. I tried the same command but using a non-existing chain for 1) the --insert option and 2) the --jump option, and in both cases the error was different from the one you've experienced. Possibly the iptables code that translates input into nftables is trying to autoadd the POSTROUTING chain and throws an error in spite of being successful? (that's purely a guess!)

Anyway, this must be either in iptables or nftables. I'm reassigning to iptables since that's next in line after libvirt, and it can be further triaged from there.

Comment 2 Dan Horák 2022-04-08 14:56:57 UTC

Thanks, Laine, for your review.

If needed I can provide access to the system where I observe this behaviour.

Comment 3 Dan Horák 2022-04-20 11:47:13 UTC

no change with kernel-5.18.0-0.rc3.27.fc37.s390x

Comment 4 IBM Bug Proxy 2022-04-21 10:00:44 UTC

------- Comment From WINTERA.com 2022-04-21 05:53 EDT-------
I agree with laine that this must be either iptabels or nftables...

Last year I helped debugging an issue with nftables and it turned out that
a bad nft_chain_nat_ipv4 module was causing the problems in that configuration.
(I have no further details, what was bad about the module)

Comment 5 Dan Horák 2022-05-25 11:17:20 UTC

Seems the problem went away after upgrading the system to F-36, but still keeping rawhide 5.18 kernel and rawhide virt stack.