Bug 2052094

Summary: [rhel9-cnv] VM fails to start, virt-handler error msg: Couldn't configure ip nat rules
Product: Container Native Virtualization (CNV) Reporter: Kedar Bidarkar <kbidarka>
Component: NetworkingAssignee: Radim Hrazdil <rhrazdil>
Status: CLOSED ERRATA QA Contact: Yossi Segev <ysegev>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.10.0CC: cnv-qe-bugs, phoracek, rhrazdil, ysegev
Target Milestone: ---   
Target Release: 4.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-09-14 19:28:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kedar Bidarkar 2022-02-08 17:07:46 UTC
Description of problem:
[rhel9-cnv] VM fails to start with message: Couldn't configure ip nat rules

Version-Release number of selected component (if applicable):
v4.10.0-7 ( RHEL9) / registry-proxy.engineering.redhat.com/rh-osbs/iib:168613

How reproducible:
Create and start a VM using RHEL-9 CNV Builds

Steps to Reproduce:
1. Create either a  DV or cDisk Based VM
2. 
3.

Actual results:
DV based VM
------------
{"component":"virt-handler","level":"error","msg":"virt-launcher crashed due to a network error. Updating VMI vm-rhel84-ocs status to Failed","pos":"vm.go:1120","timestamp":"2022-02-08T16:45:41.058355Z"}
{"component":"virt-handler","level":"error","msg":"failed to create ipv4 nat rules for vm error: Couldn't configure ip nat rules","pos":"masquerade.go:127","reason":"Couldn't configure ip nat rules","timestamp":"2022-02-08T16:49:51.024375Z"}

containerDisk based VM
-----------------------
{"component":"virt-handler","level":"error","msg":"failed to prepare pod networking","pos":"podnic.go:223","reason":"Couldn't configure ip nat rules","timestamp":"2022-02-08T16:49:51.024436Z"}
{"component":"virt-handler","kind":"","level":"error","msg":"Synchronizing the VirtualMachineInstance failed.","name":"vm2-rhel84","namespace":"default","pos":"vm.go:1685","reason":"failed to configure vmi network: setup failed, err: failed plugging phase1 at nic 'eth0': Critical network error: Couldn't configure ip nat rules","timestamp":"2022-02-08T16:49:51.024532Z","uid":"8af0f412-008d-4975-af08-8abf116a8b76"}


Expected results:
VM starts based on RHEL9 CNV Builds

Additional info:

Comment 1 Kedar Bidarkar 2022-02-08 17:12:15 UTC
Important Note: This was encountered when running RHEL9 based CNV Builds on top of RHEL8 RHCOS OCP Worker Nodes.

Comment 2 Yossi Segev 2022-05-08 09:25:35 UTC
@rad

Comment 3 Yossi Segev 2022-05-08 09:25:52 UTC
@

Comment 4 Yossi Segev 2022-05-08 09:28:09 UTC
@rhrazdil @phoracek 
There's no "Fixed in Version" and no details, so I don't even know which component was fixed, let alone on which version.
Can you please add this info?
Thank you

Comment 5 Radim Hrazdil 2022-05-09 07:06:13 UTC
Hello Yossi,

The linked patch in gerrit fixes Dockerfile for virt-handler, hence the version of virt-handler itself isn't really relevant.
Since the fix was merged in February, any recent build is good for verification.

Comment 6 Yossi Segev 2022-05-09 13:45:35 UTC
Changing target release to 4.11, as RHEL9-based images will not be part of any CNV 4.10 release, and this bug must be verified on a RHEL9-based CNV image (this bug is a result of not cherry-picking a fix from RHEL8 to RHEL9).

Comment 7 Yossi Segev 2022-05-11 13:10:49 UTC
Verified on
 CNV v4.11.0.rhel9-372
 hyperconverged-cluster-operator-rhel9-container-v4.11.0-46
 virt-handler v4.11.0-58
by running the scenario given in the bug description.

Comment 12 errata-xmlrpc 2022-09-14 19:28:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Virtualization 4.11.0 Images security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:6526