Seems the default network can't start any more on my Fedora 35 s390x system with 5.18-rc kernels. After downgrading to kernel 5.17 the network starts correctly. Interestingly I don't see such problem on my Rawhide ppc64le system (with libvirt 8.2.0 and 5.18-rc1 kernel) [sharkcz@rock-kvmlp-fedora ~]$ virsh net-info --network default Name: default UUID: d0be7373-95c7-4eeb-80c2-67568480102d Active: no Persistent: yes Autostart: yes Bridge: virbr0 [sharkcz@rock-kvmlp-fedora ~]$ virsh net-start --network default error: Failed to start network default error: internal error: Failed to apply firewall rules /usr/sbin/iptables -w --table nat --insert POSTROUTING --jump LIBVIRT_PRT: iptables v1.8.7 (nf_tables): CHAIN_ADD failed (No such file or directory): chain POSTROUTING [sharkcz@rock-kvmlp-fedora ~]$ sudo iptables -L -t nat Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain INPUT (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination Chain LIBVIRT_PRT (0 references) target prot opt source destination Version-Release number of selected component (if applicable): libvirt-8.2.0-1.fc35.s390x - from virt-preview kernel-5.18.0-0.rc1.18.fc37.s390x - from rawhide-nodebug kernel-5.18.0-0.rc1.20220406git3e732ebf7316ac8.19.fc37.s390x - from rawhide How reproducible: 100%
It's odd that the output of iptables -L would show the POSTROUTING chain, but yet that is what apparently fails. I tried the same command but using a non-existing chain for 1) the --insert option and 2) the --jump option, and in both cases the error was different from the one you've experienced. Possibly the iptables code that translates input into nftables is trying to autoadd the POSTROUTING chain and throws an error in spite of being successful? (that's purely a guess!) Anyway, this must be either in iptables or nftables. I'm reassigning to iptables since that's next in line after libvirt, and it can be further triaged from there.
Thanks, Laine, for your review. If needed I can provide access to the system where I observe this behaviour.
no change with kernel-5.18.0-0.rc3.27.fc37.s390x
------- Comment From WINTERA.com 2022-04-21 05:53 EDT------- I agree with laine that this must be either iptabels or nftables... Last year I helped debugging an issue with nftables and it turned out that a bad nft_chain_nat_ipv4 module was causing the problems in that configuration. (I have no further details, what was bad about the module)
Seems the problem went away after upgrading the system to F-36, but still keeping rawhide 5.18 kernel and rawhide virt stack.