Bug 2224663

Summary: The VM clone cannot be used as a source for other clonings
Product: Container Native Virtualization (CNV) Reporter: Denys Shchedrivyi <dshchedr>
Component: VirtualizationAssignee: Ram Lavi <ralavi>
Status: CLOSED MIGRATED QA Contact: Kedar Bidarkar <kbidarka>
Severity: high Docs Contact:
Priority: high    
Version: 4.14.0CC: acardace, iholder
Target Milestone: ---Keywords: Reopened
Target Release: 4.15.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-12-14 16:14:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Denys Shchedrivyi 2023-07-21 21:19:13 UTC
Description of problem:
 
 VM Cloning stuck in RestoreInProgress phase when using recently created VM clone as the source:

> $ oc get vmclone
> NAME               PHASE               SOURCEVIRTUALMACHINE   TARGETVIRTUALMACHINE
> testclone          Succeeded           vm-fedora              vm-fedora-clone
> testclone-cloned   RestoreInProgress   vm-fedora-clone        vm-fedora-clone2


 There is an error in vmrestore events:
> $  oc describe vmrestore
> .
> Events:
>   Type     Reason                      Age                 From                Message
>  Warning  VirtualMachineRestoreError  49s                 restore-controller  VirtualMachineRestore encountered error Operation cannot be fulfilled on virtualmachines.kubevirt.io "vm-fedora-clone2": the object has been modified; please apply your changes to the latest version and try again
>  Warning  VirtualMachineRestoreError  45s (x24 over 51s)  restore-controller  VirtualMachineRestore encountered error error patching VM vm-fedora-clone2: failed to apply patch for VM [
> {"op": "replace", "path": "/spec/template/spec/domain/devices/interfaces/0/macAddress", "value": ""},
> {"op": "replace", "path": "/spec/template/spec/domain/firmware/serial", "value": "new-serial-new"},
> {"op": "remove", "path": "/metadata/annotations/kubevirt.io~1latest-observed-api-version"},
> {"op": "remove", "path": "/metadata/annotations/kubevirt.io~1storage-observed-api-version"},
> {"op": "remove", "path": "/metadata/annotations/restore.kubevirt.io~1lastRestoreUID"},
> {"op": "replace", "path": "/spec/template/spec/domain/firmware/uuid", "value": ""}
> ]: error in remove for path: '/metadata/annotations/kubevirt.io~1latest-observed-api-version': Unable to remove nonexistent key: kubevirt.io/latest-observed-api-version: missing value




Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. create VM
2. Clone VM
3. Clone again using the VM clone as a source

Actual results:
 VM cloning failed

Expected results:
 VM successfully cloned

Additional info:

Comment 1 Denys Shchedrivyi 2023-07-21 22:15:12 UTC
Wanted to add - see this issue only when used `annotationFilters` field in the VirtualMachineClone, e.g.:

apiVersion: clone.kubevirt.io/v1alpha1
kind: VirtualMachineClone
metadata:
  name: testclone-cloned
spec:
  source:
    apiGroup: kubevirt.io
    kind: VirtualMachine
    name: vm-fedora-clone
  target:
    apiGroup: kubevirt.io
    kind: VirtualMachine
    name: vm-fedora-clone2
  annotationFilters:
    - "somekey/*"



 Without `annotationFilters` VM succesfully cloned.

Comment 2 Itamar Holder 2023-09-04 09:26:37 UTC
Hey @dshchedr 

As can be seen in the user-guide [1] and the code itself [2] the only supported source type as of now are VM and a VM Snapshot.
Am I missing something here?

Also, when I try to do so, I get a nice clean error:
```
> k create -f clone-from-clone.yaml
selecting docker as container runtime
The request is invalid: 
* spec.Source.Kind: Source kind is not supported
```

Closing as not as bug, you're welcome to re-open if I miss anything.

[1] https://kubevirt.io/user-guide/operations/clone_api/
[2] https://github.com/kubevirt/kubevirt/blob/v1.0.0/pkg/virt-controller/watch/clone/clone.go#L113

Comment 3 Itamar Holder 2023-09-04 09:27:15 UTC
Maybe you intended to test a clone from a VMSnapshot source?

Comment 5 Denys Shchedrivyi 2023-09-04 15:21:40 UTC
 @iholder sorry, I guess it's kind of confusing naming: VM clone, VM clone job, cloned VM, etc...

 Let me try explain what I mean: I didn't set VMCLone job as a source, instead - I chose the target VM (from previous cloning), so the full steps:

1) create VM - let's name it _source-vm_
2) clone _source-vm_ to _target-vm_
3) try to clone _target-vm_ (create new vm cloning job which has target VM from previous job as a source)


and if I have this field in cloning job:

>  annotationFilters:
>    - "somekey/*"

I'm getting the error:

> ]: error in remove for path: '/metadata/annotations/kubevirt.io~1latest-observed-api-version': Unable to remove nonexistent key: kubevirt.io/latest-observed-api-version: missing value


 Need to add, I see the problem only for annotationFilters. LabelFilters working good.

Comment 6 Itamar Holder 2023-09-06 15:17:31 UTC
Thank you Denys! The bug you found is very real, sorry for the misunderstanding.

The fix is not out: https://github.com/kubevirt/kubevirt/pull/10393.

Comment 7 Itamar Holder 2023-09-06 15:17:51 UTC
now*

Comment 8 Denys Shchedrivyi 2023-11-10 22:44:40 UTC
Tried to verify on CNV-v4.15.0.rhel9-1638 but seems the problem was not fixed (or may be it is some another issue with cloning the clone):

# 1) created VM

> $ oc create -f vm-fedora.yml 
> virtualmachine.kubevirt.io/vm-fedora created

> $ oc get vm
> NAME        AGE    STATUS    READY
> vm-fedora   107s   Running   True

# 2) Succesfully cloned and target VM can be started:

> $ oc get vmclone
> NAME        PHASE       SOURCEVIRTUALMACHINE   TARGETVIRTUALMACHINE
> testclone   Succeeded   vm-fedora              vm-fedora-clone

> $ oc get vm
> NAME              AGE     STATUS    READY
> vm-fedora         2m55s   Running   True
> vm-fedora-clone   59s     Running   True

# 3) Trying to clone second time by using previous target VM as a source:

cloning job stuck in `RestoreInProgress` state
> $ oc get vmclone
> NAME               PHASE               SOURCEVIRTUALMACHINE   TARGETVIRTUALMACHINE
> testclone-cloned   RestoreInProgress   vm-fedora-clone        vm-fedora-clone2

vmrestore contains error message in events:
> $ oc describe vmrestore
>  Warning  VirtualMachineRestoreError  50s                restore-controller  VirtualMachineRestore encountered error virtualmachines.kubevirt.io "vm-fedora-clone2" already exists
>  Warning  VirtualMachineRestoreError  8s (x12 over 49s)  restore-controller  VirtualMachineRestore encountered error admission webhook "mutatevirtualmachines.kubemacpool.io" denied the request: Failed to update virtual machine allocation error: failed to allocate requested mac address

But I see that new VM `vm-fedora-clone2` created:
> $ oc get vm
> NAME               AGE     STATUS    READY
> vm-fedora          3m47s   Running   True
> vm-fedora-clone    111s    Running   True
> vm-fedora-clone2   17s     Stopped   False

 However, when I'm trying to run it - I see the error message:
> $ virtctl start vm-fedora-clone2
> Error starting VirtualMachine Internal error occurred: admission webhook "virtualmachine-validator.kubevirt.io" denied the request: Cannot start VM until restore "clone-testclone-cloned-restore-vm-fedora-clone-r2xqc" completes

Comment 9 Denys Shchedrivyi 2023-11-10 22:56:26 UTC
just wanted to add - all VMs have different mac addresses:

$ for i in $(oc get vm -o name);do echo $i; oc describe $i | grep "Mac "; done
virtualmachine.kubevirt.io/vm-fedora
            Mac Address:  02:8a:48:00:00:00

virtualmachine.kubevirt.io/vm-fedora-clone
            Mac Address:  02:8a:48:00:00:01

virtualmachine.kubevirt.io/vm-fedora-clone2
            Mac Address:  02:8a:48:00:00:08