Bug 2217541

Summary: IP address not set by cloud-init when using secondary NIC via NetworkAttachmentDefinition
Product: Container Native Virtualization (CNV) Reporter: Matthew Secaur <msecaur>
Component: NetworkingAssignee: Petr Horáček <phoracek>
Status: NEW --- QA Contact: Nir Rozen <nrozen>
Severity: high Docs Contact:
Priority: medium    
Version: 4.13.0CC: alkaplan, jeremy.coulombe, sgott
Target Milestone: ---   
Target Release: 4.14.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Test cases none

Description Matthew Secaur 2023-06-26 15:49:12 UTC
Created attachment 1972677 [details]
Test cases

Description of problem:
When using the Red Hat documentation to configure cloud-init to assign an IP address to an interface [1], the IP address is not assigned to the secondary NIC. If the IP Address information is added to the cloud-init via the OCP Console's "Add network data" option, the IP address is not set, nor is the customized cloud-user password set. As a result, I am unable to login to the VM to validate anything, but the OCP console does not show the specified IP address, nor does the VM ping, so I presume the IP is not being added.

If the IP address information is added directly to the YAML when creating a VM via the OCP Console (i.e. configure the VM as desired, then click on the "YAML" tab and paste the networkData directly into the YAML before clicking on Create VirtualMachine), then the networkData section is removed completely, so the IP address is obviously not set then, either.

[1] https://docs.openshift.com/container-platform/4.13/virt/virtual_machines/vm_networking/virt-configuring-ip-for-vms.html

Version-Release number of selected component (if applicable):
Customer has this issue with OCP-Virt 4.13, but I can also recreate the issue in my lab, where I am running 4.12.3.

How reproducible:
Scenario #1: Use the console to configure a VM based on the Red Hat-provided RHEL8 or RHEL9 template and add a secondary NIC via a NetworkAttachmentDefinition. Use the "Edit" link to configure the Cloud-init, select the check box "Add Network Data" and complete the form appropriately. Click "Create VirtualMachine" and observe that the IP address is not set, and likely you will not be able to login to the VM with as password passed in the Cloud-init section.

Scenario #2: Use the console to configure a VM based on the Red Hat-provided RHEL8 or RHEL9 template and add a secondary NIC via a NetworkAttachmentDefinition. Configure the VM as desired, then view the raw YAML for the VM. Replace the default "- cloudInitConfigDrive" section [2] with the format provided in the documentation [3]. Click the "Create VirtualMachine" button. Once the VM is up, the IP address is not set on the secondary NIC and in the VM YAML, you can see that the networkData section is no longer present [4]. Also, the "- cloudInitNoCloud" is replaced with "- cloudInitConfigDrive".

[2]
        - cloudInitConfigDrive:
            userData: |-
              #cloud-config
              user: cloud-user
              password: o53s-igku-yexg
              chpasswd: { expire: False }
          name: cloudinitdisk

[3]
        - cloudInitNoCloud:
            networkData: |
              version: 2
              ethernets:
                eth1: 
                  addresses:
                  - 172.16.0.185/24 
            userData: |
              #cloud-config
              user: cloud-user
              password: redhat
              chpasswd:
                expire: false
          name: cloudinitdisk

[4]
        - cloudInitConfigDrive:
            userData: |-
              #cloud-config
              user: cloud-user
              password: o53s-igku-yexg
              chpasswd: { expire: False }
          name: cloudinitdisk

Steps to Reproduce:
See Scenario #1 and #2 above.

Actual results:
Scenario #1: The IP address is not set and neither is the cloud-user password.
Scenario #2: The IP address is not set because it has been removed from the YAML once the VM is created.

Expected results:
In either scenario, the IP address defined in the networkData should be applied to the secondary NIC.

Additional info:
The same situation occurs if the default (pod) network interface is removed, so that the only interface on the VM is the NetworkAttachmentDefinition.
I am attaching four test cases I did in my lab with additional information.

Comment 1 sgott 2023-06-28 12:39:20 UTC
Reassigning component to Networking based on the description. Please feel free to reassign if this appears to be in error.

Comment 2 Matthew Secaur 2023-06-28 13:49:41 UTC
@sgott 
I'm actually not sure which component this is. OCP-Virt is mangling the cloud-init (or at least that's part of the issue). If the folks working on Networking handle that piece, then the change is great. If not, then maybe it needs to go somewhere else.

Comment 3 Petr Horáček 2023-06-28 14:05:36 UTC
I believe this is the same issue as the one reported and investigated here: https://issues.redhat.com/browse/CNV-28700

Comment 4 jeremy 2023-06-28 14:38:14 UTC
I think that there is a good chance that this is the same issue

Comment 5 Matthew Secaur 2023-07-06 17:59:40 UTC
Jira CNV-28700 is exactly the same for at least part of this BZ (Scenario #1 in the original description).

However, Scenario #2 isn't covered in that Jira. Having said that, I suppose if CNV-28700 is fixed, it would ultimately negate the need for Scenario #2 at all.

Comment 6 Petr Horáček 2023-07-13 09:32:54 UTC
Even after removing the redundant "network" object under "networkData" generated by Console, I still can't make this work. This should be investigated by the network team, to come up with a working example of IP configuration through networkData for VMs on https://console-openshift-console.apps.cnv2.engineering.redhat.com/k8s/cluster/projects:

1) Create a VM from a template
2) Edit the YAML, so it has networkData adjusting the config
3) See if it works

The network team will investigate this.

Comment 7 Petr Horáček 2023-07-24 09:48:20 UTC
Increasing the priority since providing a working example blocks the progress on the UI side of this, and because of the severity

Comment 8 Yossi Segev 2023-07-27 09:40:30 UTC
To Alona's request - I'm adding 2 examples of networkData definitions (from a VM spec) that end up with a valid VMI, including a functioning secondary NIC (note that the second example also includes networData for the default interface):

Example #1:
      - cloudInitNoCloud:
          networkData: |
            version: 2
            ethernets:
              eth1: 
                addresses:
                - 10.10.10.14/24

Example #2:
        networkData: |                                                                                                                                                       
          ethernets:
            eth0:
              addresses:
              - fd10:0:2::2/120
              dhcp4: true
              gateway6: fd10:0:2::1
            eth1:
              addresses:
              - 10.200.0.2/24
            eth2:
              addresses:
              - 10.200.1.2/24
            eth3:
              addresses:
              - 10.200.2.2/24
          version: 2

Comment 9 Petr Horáček 2023-08-04 08:01:33 UTC
Perhaps while we are at this, we can try to resolve a bigger issue around our current use of cloud init network configuration:

As highlighted by Dominik in https://issues.redhat.com/browse/OCPBUGS-13573?focusedId=22739386&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-22739386, the problem goes beyond faulty configuration - the networkData API is unusable when the user applies credentials API to inject SSH keys.

I favor his suggestion to convert our UI and docs to the network_data.json format, where network and SSH config can coexist. Before we make that step however, we'll need to come up with a VMI example using that API.