Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1777325

Summary: Error is reported while migration between hosts supporting tsc frequency flag.
Product: [oVirt] ovirt-engine Reporter: Polina <pagranat>
Component: BLL.VirtAssignee: Michal Skrivanek <michal.skrivanek>
Status: CLOSED CURRENTRELEASE QA Contact: Polina <pagranat>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.4.0CC: ahadas, bugs, nkoenig, rbarry, smelamud, tbaransk
Target Milestone: ovirt-4.4.1Flags: pm-rhel: ovirt-4.4+
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-08 08:27:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
engine log none

Description Polina 2019-11-27 12:25:24 UTC
Created attachment 1640081 [details]
engine log

Description of problem:
VM migration between hosts supporting tsc frequency flag succeeds, but an error "VM '0f276f51-9077-4e96-9f31-a25ec6e5d938' managed non pluggable device was removed unexpectedly from libvirt" is reported. 

Version-Release number of selected component (if applicable):
http://bob.eng.lab.tlv.redhat.com/builds/4.4/rhv-4.4.0-4
vdsm-4.40.0-141.gitb9d2120.el8ev.x86_64

How reproducible:100%


Steps to Reproduce:
1. Start VM on a host supporting tsc flag
virsh -r capabilities | grep "counter name='tsc'"
<counter name='tsc' frequency='2099999000' scaling='yes'/>

2. Migrate to another host also supporting the tsc.

Actual results: The VM is migrating, but ERROR is reported to the engine

2019-11-19 08:17:29,238-05 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (ForkJoinPool-1-worker-5) [5726fb46] VM '0f276f51-9077-4e96-9f31-a25ec6e5d938' managed non pluggable device was removed unexpectedly from libvirt: 'VmDevice:{id='VmDeviceId:{deviceId='33d1f135-f1cc-443f-8245-d95e83c37340', vmId='0f276f51-9077-4e96-9f31-a25ec6e5d938'}', device='usb', type='CONTROLLER', specParams='[index=0, model=none]', address='', managed='true', plugged='false', readOnly='false', deviceAlias='', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}'
2019-11-19 08:17:32,496-05 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (EE-ManagedThreadFactory-engineScheduled-Thread-79) [] Fetched 0 VMs from VDS 'ce043440-63a6-4c97-98d6-ce90ed5e7091'

Expected results: no error


Additional info:

Comment 1 Ryan Barry 2019-11-28 03:17:45 UTC
Possibly another PCIe issue. But tsc is only intended for HP VMs with cpu -host. Was this the test case?

Holidays here, so I may not see logs until Monday

Comment 2 Tomasz Barański 2019-12-11 11:56:43 UTC
I don't think it's related to HP VM. Does the same error occur when you migrate a non-HP VM between the same hosts?

Comment 3 Polina 2019-12-11 13:47:59 UTC
This is not about HP VM. The described test is for Desktop VMs

Comment 4 Tomasz Barański 2019-12-12 10:50:17 UTC
Oh, you're right.

I reviewed the log. I don't think that has anything to do with TSC frequency.

The device in question is:

    <controller type="usb" model="none" index="0"/>

I don't know why this device is created without address, I don't know enough about the whole topic. Is this just a generic USB device?

However, later the device is reported as:

    <controller type='usb' index='0' model='none'>
      <alias name='usb'/>
    </controller>

Even though the device is there, the engine is not able to match it to what is in the DB. Also, adding ovirt aliases to USB devices has been disabled per bug 1552127. Devices are correlated by aliases only as per bug 1718852, though. This might be related.

@Polina, is this bug reproducible?

@Shmuel, does this ring any bells?

Comment 5 Polina 2019-12-18 14:34:52 UTC
Tomasz, 

I've tried to test this in the latest 4.4.0-9. 
Unfortunately impossible to test in this version because of the https://bugzilla.redhat.com/show_bug.cgi?id=1784867

Comment 6 Polina 2020-01-13 14:53:40 UTC
Hi, I tried to reproduce the issue on the last build http://bob-dr.lab.eng.brq.redhat.com/builds/4.4/rhv-4.4.0-14.
Manually installed two hosts with tsc support.(though manual installation I've enrolled the certificates just to be sure). 
There is an ssh communication between the hosts. 
Though the migration failed :

2020-01-13 09:28:09,753-0500 INFO  (migsrc/e7a4285e) [virt.vm] (vmId='e7a4285e-dd36-4d63-ad48-a67f50faa66f') starting migration to qemu+tls://ocelot01.qa.lab.tlv.redhat.com/system with miguri tcp://10.35.30.1 (migration:527)
2020-01-13 09:28:09,762-0500 ERROR (migsrc/e7a4285e) [virt.vm] (vmId='e7a4285e-dd36-4d63-ad48-a67f50faa66f') operation failed: Failed to connect to remote libvirt URI qemu+tls://ocelot01.qa.lab.tlv.redhat.com/system: unable to connect to server at 'ocelot01.qa.lab.tlv.redhat.com:16514': Connection refused (migration:280)
2020-01-13 09:28:10,077-0500 ERROR (migsrc/e7a4285e) [virt.vm] (vmId='e7a4285e-dd36-4d63-ad48-a67f50faa66f') Failed to migrate (migration:443)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 424, in _regular_run
    time.time(), machineParams
  File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 530, in _startUnderlyingMigration
    self._perform_with_conv_schedule(duri, muri)
  File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 611, in _perform_with_conv_schedule
    self._perform_migration(duri, muri)
  File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 547, in _perform_migration
    self._migration_flags)
  File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 101, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", line 94, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1838, in migrateToURI3
    if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self)
libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tls://ocelot01.qa.lab.tlv.redhat.com/system: unable to connect to server at 'ocelot01.qa.lab.tlv.redhat.com:16514': Connection refused

Could you please to look at the setup compute-ge-6.scl.lab.tlv.redhat.com . The hosts are ocelot01.qa.lab.tlv.redhat.com (host_mixed_4) and ocelot02.qa.lab.tlv.redhat.com (host_mixed_5).

Comment 7 Polina 2020-02-06 08:47:19 UTC
re-tested in 4.4.0-18
the error is still there, but I think it is actually related to the related to https://bugzilla.redhat.com/show_bug.cgi?id=1788018 and not to the special tsc scenario.

Comment 8 Ryan Barry 2020-02-06 11:03:19 UTC
Given that the other bug shows the VM starting ok (with an error in the vdsm log), I would question it being the cause. Tomasz, care to take a look at Polina's environment?

Comment 9 Tomasz Barański 2020-02-26 12:04:39 UTC
The problem might be related to #1779161. 

I'd like to postpone my investigation or a retest until that other bug is fixed.

Comment 10 Michal Skrivanek 2020-03-10 12:25:51 UTC
change SLA team to virt, we're not tracking SLA separately anymore

Comment 11 Arik 2020-06-11 20:58:33 UTC
Polina, per comment 9, could you please check if it's still reproducible?

Comment 12 Polina 2020-06-12 07:53:59 UTC
verified on ovirt-engine-4.4.1.1-0.5.el8ev.noarch . The problem fixed. No error on migration of VM between the hosts supporting tsc.

Comment 13 Arik 2020-06-12 08:47:28 UTC
Thanks, so per comment 12 updating the states

Comment 14 Sandro Bonazzola 2020-07-08 08:27:32 UTC
This bugzilla is included in oVirt 4.4.1 release, published on July 8th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.