Bug 2099321 - Migration failure with "'IndexError' object is not iterable" error in ppc arch
Summary: Migration failure with "'IndexError' object is not iterable" error in ppc arch
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: vdsm
Classification: oVirt
Component: Core
Version: 4.50.1.1
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ovirt-4.5.2
: ---
Assignee: Tomáš Golembiovský
QA Contact: Polina
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-06-20 14:48 UTC by Polina
Modified: 2022-08-30 08:47 UTC (History)
5 users (show)

Fixed In Version: vdsm-4.50.2
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-08-30 08:47:42 UTC
oVirt Team: Virt
Embargoed:
pagranat: needinfo-
pagranat: needinfo-
pm-rhel: ovirt-4.5?


Attachments (Terms of Use)
logs (1.33 MB, application/gzip)
2022-06-20 14:48 UTC, Polina
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github oVirt vdsm pull 265 0 None open Fix migration of VM with dedicated CPU 2022-06-22 08:03:32 UTC
Red Hat Issue Tracker RHV-46454 0 None None None 2022-06-20 15:01:07 UTC

Description Polina 2022-06-20 14:48:33 UTC
Created attachment 1891317 [details]
logs

Description of problem: migrating failure in a specific environment (ppc) and in a specific direction .


Version-Release number of selected component (if applicable):
ovirt-engine-4.5.1.2-0.11.el8ev.noarch

How reproducible: 100%


Steps to Reproduce:

host1_source 

CPU(s):               192
On-line CPU(s) list:  0,8,16,24,32,40,48,56,64,72,80,88,96,104,112,120,128,136,144,152,160,168,176,184
Off-line CPU(s) list: 1-7,9-15,17-23,25-31,33-39,41-47,49-55,57-63,65-71,73-79,81-87,89-95,97-103,105-111,113-119,121-127,129-135,137-143,145-151,153-159,161-167,169-175,177-183,185-191
Thread(s) per core:   1
Core(s) per socket:   6
Socket(s):            4
NUMA node(s):         4

host2_ destination:

CPU(s):               128
On-line CPU(s) list:  0,8,16,24,32,40,48,56,64,72,80,88,96,104,112,120
Off-line CPU(s) list: 1-7,9-15,17-23,25-31,33-39,41-47,49-55,57-63,65-71,73-79,81-87,89-95,97-103,105-111,113-119,121-127
Thread(s) per core:   1
Core(s) per socket:   4
Socket(s):            4
NUMA node(s):         4


1. VM is configured with Isolated Threads , cpu topology 15(sockets):1:1, 4 numa nodes, no host pinning. It could be started on any of these hosts , but if it was started on host1_source , the migration to host2_destination fails (not vise versa)


Actual results:
2022-06-20 17:26:39,502+03 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-11) [5f5bfce4] FINISH, DestroyVDSCommand, return: , log id: 2ffa8b38
2022-06-20 17:26:39,502+03 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-11) [5f5bfce4] VM 'f144ca55-43ea-4900-ab06-20203b36d117'(golden_env_mixed_virtio_1_0) was unexpectedly detected as 'Down' on VDS 'be502bf8-d7b4-4ba3-87bc-0b7ae2aea4a7'(host_mixed_2) (expected on 'b1ad99aa-b75a-4e3b-a2b4-0028cc418d52')
2022-06-20 17:26:39,502+03 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-11) [5f5bfce4] Migration of VM 'golden_env_mixed_virtio_1_0' to host 'host_mixed_2' failed: 'IndexError' object is not iterable.
2022-06-20 17:26:46,300+03 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [] Fetched 0 VMs from VDS 'be502bf8-d7b4-4ba3-87bc-0b7ae2aea4a7'
2022-06-20 17:26:53,355+03 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-30) [] VM 'f144ca55-43ea-4900-ab06-20203b36d117' was reported as Down on VDS 'b1ad99aa-b75a-4e3b-a2b4-0028cc418d52'(host_mixed_1)
2022-06-20 17:26:53,357+03 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-30) [] START, DestroyVDSCommand(HostName = host_mixed_1, DestroyVmVDSCommandParameters:{hostId='b1ad99aa-b75a-4e3b-a2b4-0028cc418d52', vmId='f144ca55-43ea-4900-ab06-20203b36d117', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 62a9a440



Expected results: must migrate 


Additional info:

Comment 1 Polina 2022-07-21 07:19:32 UTC
Hi Tomas , I'm trying to verify this bug and face a new problem that prevents to verify.

Testing in the same environment (topology in the https://bugzilla.redhat.com/show_bug.cgi?id=2099321#c0)

The Isolated VM with 15 virtual sockets can't start now on no host in this environment. Though according to the topology it must fit.

Liran , could you please also look at this ? in case it is not related to the fix and we are introduced a new bug

Comment 2 Polina 2022-07-21 07:47:02 UTC
removing 'need info'. the problem is known https://bugzilla.redhat.com/show_bug.cgi?id=2104858

Comment 3 Polina 2022-07-21 08:03:14 UTC
verified on ovirt-engine-4.5.1.3-0.36.el8ev.noarch

Comment 4 Sandro Bonazzola 2022-08-30 08:47:42 UTC
This bugzilla is included in oVirt 4.5.2 release, published on August 10th 2022.
Since the problem described in this bug report should be resolved in oVirt 4.5.2 release, it has been closed with a resolution of CURRENT RELEASE.
If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.