Bug 1803107
| Summary: | [Customer0 data migration] VM can't be accessed after data migration | ||
|---|---|---|---|
| Product: | Container Native Virtualization (CNV) | Reporter: | Qixuan Wang <qixuan.wang> |
| Component: | Storage | Assignee: | Alexander Wels <awels> |
| Status: | CLOSED DUPLICATE | QA Contact: | Qixuan Wang <qixuan.wang> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 2.3.0 | CC: | alitke, cnv-qe-bugs, ncredi |
| Target Milestone: | --- | ||
| Target Release: | 2.3.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-03-05 14:24:30 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Qixuan Wang
2020-02-14 14:01:11 UTC
Alexander please take a look. Qixuan, please provide Alexander with environment details so he can look into this. Alexander has got the environment. Tested with the latest https://gitlab.cee.redhat.com/awels/hostpath-provisioner-upgrade. Still can't access the console, but with different reason. Check disk image on cnv1.4 node: [cloud-user@cnv-executor-cnv14-node-e6a2cb-2 hpvolumes]$ ls -ls pvc-0ca13e4b-5883-11ea-81d2-fa163ed49b51 total 28148 28148 -rw-r--r--. 1 107 107 104857600 Feb 26 05:32 disk.img [cloud-user@ocp-psi-executor hostpath-provisioner-upgrade-master]$ oc get pvc -n test-migration-3 NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE cirros-dv-3 Bound pvc-0ca13e4b-5883-11ea-81d2-fa163ed49b51 39Gi RWO hostpath-provisioner 21m fedora-dv-3 Bound pvc-f7df7271-5882-11ea-81d2-fa163ed49b51 39Gi RWO hostpath-provisioner 21m rhel-dv-3 Bound pvc-f81cc5a2-5882-11ea-81d2-fa163ed49b51 39Gi RWO hostpath-provisioner 21m Create a dir with the same PV name and copy disk.img to it [core@qwang-23-fbfr6-worker-hk6rd hpvolumes]$ ls -ls pvc-0ca13e4b-5883-11ea-81d2-fa163ed49b51 total 102400 102400 -rw-r--r--. 1 core core 104857600 Feb 28 11:00 disk.img Import data to cnv2.3 and start the VM: [cloud-user@ocp-psi-executor hostpath-provisioner-upgrade-master]$ oc get vm NAME AGE RUNNING VOLUME vm-cirros-dv-3 27m false vm-fedora-dv-3 27m false vm-rhel-dv-3 27m false [cloud-user@ocp-psi-executor hostpath-provisioner-upgrade-master]$ virtctl start vm-cirros-dv-3 VM vm-cirros-dv-3 was scheduled to start [cloud-user@ocp-psi-executor hostpath-provisioner-upgrade-master]$ oc get vmi NAME AGE PHASE IP NODENAME vm-cirros-dv-3 2m19s Running 10.131.0.34 qwang-23-fbfr6-worker-hk6rd [cloud-user@ocp-psi-executor hostpath-provisioner-upgrade-master]$ oc describe vmi vm-cirros-dv-3 Name: vm-cirros-dv-3 Namespace: test-migration-3 Labels: kubevirt.io/nodeName=qwang-23-fbfr6-worker-hk6rd kubevirt.io/vm=vm-datavolume Annotations: kubevirt.io/latest-observed-api-version: v1alpha3 kubevirt.io/storage-observed-api-version: v1alpha3 API Version: kubevirt.io/v1alpha3 Kind: VirtualMachineInstance Metadata: Creation Timestamp: 2020-02-28T10:37:52Z Finalizers: foregroundDeleteVirtualMachine Generate Name: vm-cirros-dv-3 Generation: 474 Owner References: API Version: kubevirt.io/v1alpha3 Block Owner Deletion: true Controller: true Kind: VirtualMachine Name: vm-cirros-dv-3 UID: 5130d9c7-fa81-4b1e-8727-78da67f12a3d Resource Version: 3451430 Self Link: /apis/kubevirt.io/v1alpha3/namespaces/test-migration-3/virtualmachineinstances/vm-cirros-dv-3 UID: 847f353e-d414-46ea-945e-57d30040ccdf Spec: Domain: Devices: Disks: Disk: Bus: virtio Name: datavolumevolume Interfaces: Bridge: Name: default Features: Acpi: Enabled: true Firmware: Uuid: ee510f58-6805-5d71-8465-f6f1b55198ef Machine: Type: q35 Resources: Requests: Cpu: 100m Memory: 64M Networks: Name: default Pod: Termination Grace Period Seconds: 0 Volumes: Data Volume: Name: cirros-dv-3 Name: datavolumevolume Status: Conditions: Last Probe Time: <nil> Last Transition Time: <nil> Message: cannot migrate VMI with non-shared PVCs Reason: DisksNotLiveMigratable Status: False Type: LiveMigratable Last Probe Time: <nil> Last Transition Time: <nil> Message: cannot migrate VMI which does not use masquerade to connect to the pod network Reason: InterfaceNotLiveMigratable Status: False Type: LiveMigratable Last Probe Time: <nil> Last Transition Time: 2020-02-28T10:37:59Z Status: True Type: Ready Guest OS Info: Interfaces: Ip Address: 10.131.0.34 Mac: 52:54:00:df:d6:e4 Name: default Migration Method: BlockMigration Node Name: qwang-23-fbfr6-worker-hk6rd Phase: Running Qos Class: Burstable Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 2m21s virtualmachine-controller Created virtual machine pod virt-launcher-vm-cirros-dv-3-dt6wp Warning SyncFailed 2m13s virt-handler, qwang-23-fbfr6-worker-hk6rd server error. command SyncVMI failed: "LibvirtError(Code=1, Domain=10, Message='internal error: qemu unexpectedly closed the monitor: 2020-02-28T10:37:59.971943Z qemu-kvm: -device virtio-blk-pci,scsi=off,bus=pci.3,addr=0x0,drive=drive-ua-datavolumevolume,id=ua-datavolumevolume,bootindex=1,write-cache=on: Could not reopen file: Permission denied')" Normal Started 2m13s virt-handler, qwang-23-fbfr6-worker-hk6rd VirtualMachineInstance started. Normal Created 2m11s (x23 over 2m13s) virt-handler, qwang-23-fbfr6-worker-hk6rd VirtualMachineInstance defined. [cloud-user@ocp-psi-executor hostpath-provisioner-upgrade-master]$ virtctl console vm-cirros-dv-3 Successfully connected to vm-cirros-dv-3 console. The escape sequence is ^] You were disconnected from the console. This has one of the following reasons: - another user connected to the console of the target vm - network issues http: websocket: close 1006 (abnormal closure): unexpected EOF Can you show me the yaml of the VM? The describe is complaining about live migration not working, and for hostpath provisioner storage live migration will not be an option. In particular the evictionStrategy of the VM should not be set to LiveMigrate. It's wired that I can't reproduce it with CNV 2.2 anymore.
[cnv-qe-jenkins@cnv-executor-qwang hostpath-provisioner-upgrade-master]$ virtctl console vm-cirros-dv-3 -n test-migration-3
Successfully connected to vm-cirros-dv-3 console. The escape sequence is ^]
login as 'cirros' user. default password: 'gocubsgo'. use 'sudo' for root.
cnv-executor-qwang-worker-0 login:
[cnv-qe-jenkins@cnv-executor-qwang hostpath-provisioner-upgrade-master]$ oc describe vmi vm-cirros-dv-3 -n test-migration-3
Name: vm-cirros-dv-3
Namespace: test-migration-3
Labels: kubevirt.io/nodeName=host-172-16-0-34
kubevirt.io/vm=vm-datavolume
Annotations: kubevirt.io/latest-observed-api-version: v1alpha3
kubevirt.io/storage-observed-api-version: v1alpha3
API Version: kubevirt.io/v1alpha3
Kind: VirtualMachineInstance
Metadata:
Creation Timestamp: 2020-03-03T10:31:41Z
Finalizers:
foregroundDeleteVirtualMachine
Generate Name: vm-cirros-dv-3
Generation: 8
Owner References:
API Version: kubevirt.io/v1alpha3
Block Owner Deletion: true
Controller: true
Kind: VirtualMachine
Name: vm-cirros-dv-3
UID: 87329303-0855-4d0b-892e-3685fda10a22
Resource Version: 5814580
Self Link: /apis/kubevirt.io/v1alpha3/namespaces/test-migration-3/virtualmachineinstances/vm-cirros-dv-3
UID: 5b7d79df-cec7-4214-a9bd-8cbae297cc8d
Spec:
Domain:
Devices:
Disks:
Disk:
Bus: virtio
Name: datavolumevolume
Interfaces:
Bridge:
Name: default
Features:
Acpi:
Enabled: true
Firmware:
Uuid: ee510f58-6805-5d71-8465-f6f1b55198ef
Machine:
Type: q35
Resources:
Requests:
Cpu: 100m
Memory: 64M
Networks:
Name: default
Pod:
Termination Grace Period Seconds: 0
Volumes:
Data Volume:
Name: cirros-dv-3
Name: datavolumevolume
Status:
Conditions:
Last Probe Time: <nil>
Last Transition Time: <nil>
Message: cannot migrate VMI with non-shared PVCs
Reason: DisksNotLiveMigratable
Status: False
Type: LiveMigratable
Last Probe Time: <nil>
Last Transition Time: <nil>
Message: cannot migrate VMI with a bridge interface connected to a pod network
Reason: InterfaceNotLiveMigratable
Status: False
Type: LiveMigratable
Last Probe Time: <nil>
Last Transition Time: 2020-03-03T10:31:54Z
Status: True
Type: Ready
Guest OS Info:
Interfaces:
Ip Address: 10.129.0.212
Mac: 0a:58:0a:81:00:d4
Name: default
Migration Method: BlockMigration
Node Name: host-172-16-0-34
Phase: Running
Qos Class: Burstable
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 3m31s virtualmachine-controller Created virtual machine pod virt-launcher-vm-cirros-dv-3-4cttm
Normal Created 3m19s (x3 over 3m19s) virt-handler, host-172-16-0-34 VirtualMachineInstance defined.
Normal Started 3m19s virt-handler, host-172-16-0-34 VirtualMachineInstance started.
Also can't reproduce it on CNV 2.3 IPI now. Perhaps there was something wrong when I copied PVC. I'm going to close the bug. https://bugzilla.redhat.com/show_bug.cgi?id=1805627 addressed the problem. *** This bug has been marked as a duplicate of bug 1805627 *** The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |