Description of problem:

Two areas exist:

1. if an UserDefinedVMProperties exist for current cluster level and it is assigned to VMs, cluster level cannot be updated to higher one because the UserDefinedVMProperty is present by default to higher cluster level. And the cluster level update info is not enough informative (BZ1504673).

An example:

$ grep UserDefinedVMProperties sos_commands/ovirt/engine-config_--all UserDefinedVMProperties: SerialConsole=^([a-z]|[0-9])+$ version: 3.6 UserDefinedVMProperties: version: 4.0 UserDefinedVMProperties: version: 4.1

101972018-04-04 10:37:08,109+02 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-39) [3d99451d] EVENT_ID: SYSTEM_UPDATE_VM(253), Correlation ID: 3d99451d, Call Stack: null, Custom ID: null, Cus\
     tom Event ID: -1, Message: VM infotainment1example.com configuration was updated by system.                                                                                                                                       
101982018-04-04 10:37:08,163+02 WARN  [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-39) [79998880] Validation of action 'UpdateVm' failed for user foobar@example.com. Reasons: VAR__ACTION__UPDATE,VAR__\
     TYPE__VM,ACTION_TYPE_FAILED_INVALID_CUSTOM_PROPERTIES_INVALID_KEYS,$MissingKeys SerialConsole        

115972018-04-04 10:37:16,462+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-39) [64bf3421] EVENT_ID: CLUSTER_CANNOT_UPDATE_VM_COMPATIBILITY_VERSION(12,005), Correlation ID: e0c99833-3e65-4\
     80e-9acd-a255d2faafd4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Cannot update compatibility version of Vm/Template: [webapp5vq_tb], Message: [No Message]    
Version-Release number of selected component (if applicable):

Thus the upgrade via engine-setup finished ok, cluster level cannot be upgraded.  Maybe we could either clone such __used__ UserDefinedVMProperties for higher cluster level automatically, or we could show a warning that the user should do manual action.

2. Issue I hit myself when using CustomDeviceProperties for mac spoofing individual vnic (vnic profiles).

# engine-config -g CustomDeviceProperties | grep 'macspoof.*4\.1'
CustomDeviceProperties: {type=interface;prop={SecurityGroups=^(?:(?:[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}, *)*[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}|)$;ifacemacspoof=^(true|false)$}} version: 4.1

Here after cluster level upgrade I saw that vnic profile which used to had 'ifacemacspoof' CustomDeviceProperty got it removed (no cluster level match) and this caused that 'Network Filter' got replaced with default value, ie. vdsm-no-mac-spoofing. This caused network communication disrupture for all VMs in this cluster which were fuctioning as virtual hosts (nestedvt).

Here we could do same action as above.

How reproducible:
100%

Steps to Reproduce:
1. create UserDefinedVMProperty for 4.1 and assign it to vms on 4.1 cluster
2. create vnic profile with custom CustomDeviceProperties like above (use macspoof vdsm hook on hosts)[1] and assing this vnic profile to vms
3. have the vms running
4. update to 4.2
5a. update cluster level to 4.2
5b. (if update passes) see what is network filter for the vnic profile

Actual results:
5a - upgrade fails as there are assiged userdefinedvmproperties to vms and those are not available for cluster level version which we try to upgrade to
5b - network filter changes to default

Expected results:


Additional info:
[1] https://github.com/oVirt/vdsm/tree/master/vdsm_hooks/macspoof