Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1954458

Summary: CDI importer fails if the selected migration network for the target provider is not in the kube-system namespace
Product: Migration Toolkit for Virtualization Reporter: Ilanit Stein <istein>
Component: GeneralAssignee: Fabien Dupont <fdupont>
Status: CLOSED ERRATA QA Contact: Ilanit Stein <istein>
Severity: high Docs Contact: Avital Pinnick <apinnick>
Priority: urgent    
Version: 2.0.0CC: apinnick, fdupont, istein, miguel, mlehrer, mturley, slucidi
Target Milestone: ---Keywords: TestBlocker
Target Release: 2.0.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-06-10 17:11:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ilanit Stein 2021-04-28 07:48:20 UTC
Description of problem:
OCP Cluster:
Bare metal, or PSI with an added "ovn-kubernetes1" multus network by running [1].
Network "ovn-kubernetes1" selected within a migration plan.
After starting the migration plan the importer pos did not start due to events errors [2].

[1]
# oc apply -f second_network.yaml

second_network.yaml:

apiVersion: "k8s.cni.cncf.io/v1"

kind: NetworkAttachmentDefinition

metadata:

  name: ovn-kubernetes1

spec:

  config: '{

      "cniVersion": "0.3.1",

      "type": "cnv-bridge",

      "bridge": "br1"

    }'

[2]
Pimporter-1a5b9de1-bcc6-4eed-960b-aa8df8ddfcf1-2000
NamespaceNSdefault
Apr 26, 6:42 am
Generated from multus
cannot find a network-attachment-definition (ovn-kubernetes1) in namespace (kube-system): network-attachment-definitions.k8s.cni.cncf.io "ovn-kubernetes1" not found
PodPimporter-1a5b9de1-bcc6-4eed-960b-aa8df8ddfcf1-2000
NamespaceNSdefault
Apr 26, 6:42 am
Generated from kubelet on mgn05-vxwx7-worker-0-q4f9r

Version-Release number of selected component (if applicable):
MTV-2.0.0-20

Comment 1 mlehrer 2021-04-28 08:30:01 UTC
Just expanding that we reproduce this bug on MTV-2.0.0-21 / CNV-2.6.2-25 on baremetal with using a host-device network.
An additional issue that impacts the severity is that after switching to non-default pod network you cannot re-use the default pod network as shown in this plan describe snippet as its marked as non valid transfer network.


root@f01-h14-000-r640:~$ oc describe plan/m-test-1 -n openshift-rhmtv
Name:         m-test-1
Namespace:    openshift-rhmtv
Labels:       <none>
Annotations:  <none>
API Version:  forklift.konveyor.io/v1alpha1
Kind:         Plan
Metadata:
  Creation Timestamp:  2021-04-27T15:54:43Z
  Generation:          1
  Managed Fields:
    API Version:  forklift.konveyor.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:map:
          .:
          f:network:
            .:
            f:name:
            f:namespace:
          f:storage:
            .:
            f:name:
            f:namespace:
        f:provider:
          .:
          f:destination:
            .:
            f:name:
            f:namespace:
          f:source:
            .:
            f:name:
            f:namespace:
        f:targetNamespace:
        f:transferNetwork:
        f:vms:
    Manager:      Mozilla
    Operation:    Update
    Time:         2021-04-27T15:54:43Z
    API Version:  forklift.konveyor.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:migration:
        f:observedGeneration:
    Manager:         manager
    Operation:       Update
    Time:            2021-04-27T15:54:43Z
  Resource Version:  99709106
  Self Link:         /apis/forklift.konveyor.io/v1alpha1/namespaces/openshift-rhmtv/plans/m-test-1
  UID:               970822d0-46d5-4647-9164-b15dc701d261
Spec:
  Description:  
  Map:
    Network:
      Name:       m-test-1-cbcff
      Namespace:  openshift-rhmtv
    Storage:
      Name:       m-test-1-pkvlx
      Namespace:  openshift-rhmtv
  Provider:
    Destination:
      Name:       host
      Namespace:  openshift-rhmtv
    Source:
      Name:          vmware65
      Namespace:     openshift-rhmtv
  Target Namespace:  openshift-rhmtv
  Transfer Network:  Pod network
  Vms:
    Id:  vm-1570
  Warm:  false
Status:
  Conditions:
    Category:              Critical
    Last Transition Time:  2021-04-27T15:54:43Z
    Message:               Transfer network is not valid.
    Reason:                NotFound
    Status:                True
    Type:                  TransferNetworkNotValid
  Migration:
  Observed Generation:  1
Events:
  Type     Reason                   Age   From  Message
  ----     ------                   ----  ----  -------
  Warning  TransferNetworkNotValid  10m   plan  Transfer network is not valid.
root@f01-h14-000-r640:~$

Comment 3 Fabien Dupont 2021-04-28 09:07:07 UTC
There are 2 issues here:

1. The transfer network is not respected
2. Reverting to pod network doesn't work

To better understand #1, could you please attach the VirtualMachineImport CR generated by MTV?

Comment 5 mlehrer 2021-04-28 13:07:21 UTC
We don't have an environment right now thats free that we can break to get the VMImport CR right now.
We'll try to get this or see if functional qe can provide this sooner.

FYI Sam - yesterday mentioned you had a hunch on this regarding the validation service and use of namespace/name vs name- is the VMMachineImport CR still needed or you think you have what you need to move forward on this?

Comment 6 Sam Lucidi 2021-04-28 14:18:44 UTC
Thanks Mordechai, after our conversation I was able to track down the root of this bug. One half of the issue is that the UI was accidentally setting an incorrect value when switching from a Multus network back to the pod network. The other half is that Multus looks for the network by name in the kube-system namespace, not in the pod namespace as I originally thought. I've merged a PR with a change to the spec to require specifying the network's namespace, and Mike has a corresponding PR to incorporate the API change and fix the pod network selection.

https://github.com/konveyor/forklift-controller/pull/240
https://github.com/konveyor/forklift-ui/pull/559
https://github.com/konveyor/forklift-ui/pull/560

Comment 7 Sam Lucidi 2021-04-28 18:37:33 UTC
All PRs have been merged, moving to MODIFIED.

Comment 8 Fabien Dupont 2021-05-03 12:08:55 UTC
The fix should be part of build mtv-operator-bundle-container-2.0.0-4 / iib:72115.

Comment 9 Ilanit Stein 2021-05-05 08:06:15 UTC
Verified on mtv-operator-bundle-container-2.0.0-4 using same flow as in bug description.
This error no longer appear:
"cannot find a network-attachment-definition (ovn-kubernetes1) in namespace (kube-system): network-attachment-definitions.k8s.cni.cncf.io "ovn-kubernetes1" not found" 


However I could not run migration using the ovn-kubernetes1 network:
I run migration using the ovn-kubernetes1 network - 
the importer pod events no longer contain the reported error,
however this pod remains in state "Podinitialiazing", and it has repeated events like this one:
 
PodPimporter-652bc02f-4ca9-48d4-8a86-3d530d3cf280-2000NamespaceNSdefault4 minutes agoGenerated from multusAdd eth0 [] from default/ovn-kubernetes1

I then changed the default OCP network for migration back to POD, and ran a new migration plan successfully, using POD.

My guess is that the importer could not use this ovn-kubernetes1 network for migration, 
since further network settings are required on the PSI based cluster to allow it.

Comment 12 errata-xmlrpc 2021-06-10 17:11:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (MTV 2.0.0 images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:2381