Bug 1392746

Summary: Unable to migrate virtual machines with direct attached lun to hypervisor running vdsm-4.16.36-1 or higher
Product: Red Hat Enterprise Virtualization Manager Reporter: Steffen Froemer <sfroemer>
Component: vdsmAssignee: Dan Kenigsberg <danken>
Status: CLOSED CURRENTRELEASE QA Contact: Israel Pinto <ipinto>
Severity: urgent Docs Contact:
Priority: medium    
Version: 3.5.7CC: amureini, bazulay, lsurette, michal.skrivanek, nsoffer, pzhang, sfroemer, srevivo, tjelinek, ycui, ykaul
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-23 13:07:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1021480, 1291930    
Bug Blocks:    
Attachments:
Description Flags
hook-script 50_remove_serialtag_from_lun none

Description Steffen Froemer 2016-11-08 06:39:45 UTC 
Created attachment 1218421 [details]
hook-script 50_remove_serialtag_from_lun

Description of problem:
Customer is unable to migrate any virtual machine with direct attached lun away from their hypervisor, which is running vdsm-4.16.20-1.el7ev.x86_64. On trying to migrate the virtual machine, following error is shown in the vdsm.log

unsupported configuration: scsi-block 'lun' devices do not support the serial property

This issue is caused after upgrading their environment to RHEV-3.6 and also upgrading hypervisor to RHEL-7.2 with latest vdsm-version.


Version-Release number of selected component (if applicable):
vdsm-4.16.20-1.el7ev.x86_64 (source)
vdsm-4.16.36-1.el7ev.x86_64 or higher (destination)

How reproducible:
100%

Steps to Reproduce:
1. install one RHEL-7.1 and one RHEL-7.2
2. pin the version of vdsm on RHEL-7.1 Hypervisor to 4.16.20-1

# /etc/yum.conf
exclude=vdsm-infra*

# $: yum install yum-versionlock
# /etc/yum/plugins.conf/versionlock.list
vdsm-4.16.20-1.el7ev.x86_64
vdsm-cli-4.16.20-1.el7ev.noarch
vdsm-jsonrpc-4.16.20-1.el7ev.noarch
vdsm-python-4.16.20-1.el7ev.noarch
vdsm-python-zombiereaper-4.16.20-1.el7ev.noarch
vdsm-xmlrpc-4.16.20-1.el7ev.noarch
vdsm-yajsonrpc-4.16.20-1.el7ev.noarch
libvirt-client-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-config-nwfilter-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-interface-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-network-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-nodedev-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-nwfilter-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-qemu-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-secret-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-storage-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-kvm-1.2.8-16.el7_1.3.x86_64
libvirt-lock-sanlock-1.2.8-16.el7_1.3.x86_64
libvirt-python-1.2.8-7.el7_1.1.x86_64

3. install the RHEL-7.2 hypervisor to latest version
4. Attach both hypervisor to RHEV-M
5. Create a virtual machine with direct attached lun (virtoio-SCSI and SCSI-passthrough) and start it on RHEL-7.1 hypervisor
6. start a migration to the RHEL-7.2 hypervisor

Actual results:
Migration fail with error:

Nov 07 20:04:27 localhost vdsm[14043]: vdsm vm.Vm ERROR vmId=`9efbd206-ccf9-4bbc-8871-d16e6a274118`::unsupported configuration: scsi-block 'lun' devices do not support the serial property
Nov 07 20:04:27 localhost vdsm[14043]: vdsm vm.Vm ERROR vmId=`9efbd206-ccf9-4bbc-8871-d16e6a274118`::Failed to migrate
                                       Traceback (most recent call last):
                                         File "/usr/share/vdsm/virt/migration.py", line 246, in run
                                           self._startUnderlyingMigration(time.time())
                                         File "/usr/share/vdsm/virt/migration.py", line 335, in _startUnderlyingMigration
                                           None, maxBandwidth)
                                         File "/usr/share/vdsm/virt/vm.py", line 702, in f
                                           ret = attr(*args, **kwargs)
                                         File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 111, in wrapper
                                           ret = f(*args, **kwargs)
                                         File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1701, in migrateToURI2
                                           if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
                                       libvirtError: unsupported configuration: scsi-block 'lun' devices do not support the serial property

Expected results:
Migration should work.

Additional info:
This issue was affected by following BZ
- #1291930
- #1021480

I tried to workaround this issue by using a vdsm-hook. But the hook has no influence to the running domainxml.
Implemented the hook on following places:

source = RHEL-7.1 hypervisor
dest = RHEL-7.2 hypervisor

source:/usr/libexec/vdsm/hooks/
 - before_vm_migrate_source

dest:/usr/libexec/vdsm/hooks/
 - before_vm_migrate_destination
 - before_vm_start

> simple hook-script 50_remove_serialtag_from_lun is attached.
Comment 1 Allon Mureinik 2016-11-09 08:40:24 UTC 
Seems like the same issue as bug 1291930 which was solved in vdsm-4.17.18 and backported (via bug 1299235) to vdsm-4.16.36-1.

Nir - do we need the fix the SOURCE vdsm too, or is having the fixed version in the TARGET vdsm sufficient?
Comment 2 Michal Skrivanek 2016-11-09 08:49:22 UTC 
There is not much else to do. The regression was introduced by libvirt in RHEL 7.2 bug 1021480 and subsequently fixed on RHEV side in 3.5.8 (vdsm-4.16.36-1).

That fix only applies to VMs started on 3.5.8 and it is not fixing any existing VMs prior to that. You would either have to restart them on the new host and then they can migrate back and forth. But old running VMs originally started on <3.5.8 are unable to migrate to new hosts.

I propose a maintenance window to power cycle the VMs and start them on new hosts, then proceed with remaining hosts upgrades.

Since we do not plan further 3.5.z builds and the issue is fixed in 3.6 GA I propose to close this bug as CURRENTRELEASE
Comment 3 Steffen Froemer 2016-11-09 09:12:41 UTC 
I know the solution by shutting down the vm and powering up on another host. Is there a one-time workaround available, to migrate these vms and removing serial-tag on migration? It's hard for the customer to find a possible maintenance windows in the near future for shutting down all vms. If we would have more time, I would suggest this solution, but there is only one hypervisor left, which is not updated and full of vms, which needs to be migrated, but can't shut off in near future.
Currently there is no failover possible, as all hosts are upgraded.

I know, that we can't provide a fix for this solution, but a workaround or advice for migrating those vms would be very helpful.
Comment 4 Michal Skrivanek 2016-11-09 10:04:52 UTC 
the only hope is to make the hook work. why was it not working? Did it not remove the tag on migrating out from the old host?
Comment 5 Tomas Jelinek 2016-11-23 08:29:31 UTC 
ping, any news? lowering priority since there is no real action from our side.
Comment 6 Michal Skrivanek 2016-11-23 13:07:23 UTC 
seems that it was resolved