Bug 1497614

+++ This bug was initially created as a clone of Bug #1480238 +++

Description of problem:
Update DC from 4.0->4.1 fails ONLY with RESTAPI - Manually upgrading DC works.

REST response is:
'Cannot migrate MACs to another MAC pool, because that action would create duplicates in target MAC pool, which are not allowed. Problematic MACs are 	00:1a:4a:16:25:b2'

00:1a:4a:16:25:b2 is the MAC of the VM that is created on the new 4.0 DC/cluster

Many of my automated runs on storage_qcow2_v3 TP has the similar flow that fails when upgrading DC to 4.1 , so this is an automation blocker for me.

Version-Release number of selected component (if applicable):
Engine - > 4.2.0-0.0.master.20170809175855.git1082d9d.el7.centos
VDSM   -> 4.20.2-50.gitb718903

How reproducible:
100%

Steps to Reproduce:

Done Manually/RESTAPI:
1) Create 4.0 DC + cluster with a matching host
2) Create VM with NIC + a thin disk (1G)+ snapshot.
3) Upgrade cluster to 4.1 

Done ONLY VIA RESTAPI:
4) upgrade DC to 4.1 

Actual results:
REST request:
2017-08-10 15:34:17,016 - MainThread - datacenters - DEBUG - PUT request

url:/ovirt-engine/api/datacenters/ad59c7c8-05a6-4290-ad71-3b6103b1207c 

body:

<data_center>
    <version>
        <major>4</major>
        <minor>1</minor>
    </version>
</data_center>

Response is :
Cannot migrate MACs to another MAC pool, because that action would create duplicates in target MAC pool, which are not allowed. Problematic MACs are 	00:1a:4a:16:25:b2

This problematic MAC is the newly created VM MAC. (00:1a:4a:16:25:b2)


Expected results:
Upgrade succeeds.

Additional info:

Engine:
2017-08-10 15:34:17,132+03 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-5) [] Operation Failed: [Cannot migrate MACs to another MAC pool, because that action would create duplicates in target MAC pool, which are not allowed. Problematic MACs are 	00:1a:4a:16:25:b2]

REST REQUEST:
2017-08-10 15:34:17,016 - MainThread - datacenters - DEBUG - PUT request content is --  

url:/ovirt-engine/api/datacenters/ad59c7c8-05a6-4290-ad71-3b6103b1207c 


body:

<data_center>
    <version>
        <major>4</major>
        <minor>1</minor>
    </version>
</data_center>


REST RESPONSE:
2017-08-10 15:34:17,138 - MainThread - api_utils - ERROR - Failed to update element NOT as expected:
	Status: 409
	Reason: Conflict
	Detail: [Cannot migrate MACs to another MAC pool, because that action would create duplicates in target MAC pool, which are not allowed. Problematic MACs are 	00:1a:4a:16:25:b2]




-REST GET DC before Upgrade the get to the going to be upgraded DC  :

2017-08-10 15:34:17,013 - MainThread - datacenters - DEBUG - Response body for GET request is: 
<data_centers>
    <data_center href="/ovirt-engine/api/datacenters/ad59c7c8-05a6-4290-ad71-3b6103b1207c" id="ad59c7c8-05a6-4290-ad71-3b6103b1207c">
        <name>dc_upgrade_4_0_to_4_1</name>
        <link href="/ovirt-engine/api/datacenters/ad59c7c8-05a6-4290-ad71-3b6103b1207c/clusters" rel="clusters"/>
        <link href="/ovirt-engine/api/datacenters/ad59c7c8-05a6-4290-ad71-3b6103b1207c/networks" rel="networks"/>
        <link href="/ovirt-engine/api/datacenters/ad59c7c8-05a6-4290-ad71-3b6103b1207c/storagedomains" rel="storagedomains"/>
        <link href="/ovirt-engine/api/datacenters/ad59c7c8-05a6-4290-ad71-3b6103b1207c/permissions" rel="permissions"/>
        <link href="/ovirt-engine/api/datacenters/ad59c7c8-05a6-4290-ad71-3b6103b1207c/quotas" rel="quotas"/>
        <link href="/ovirt-engine/api/datacenters/ad59c7c8-05a6-4290-ad71-3b6103b1207c/qoss" rel="qoss"/>
        <link href="/ovirt-engine/api/datacenters/ad59c7c8-05a6-4290-ad71-3b6103b1207c/iscsibonds" rel="iscsibonds"/>
        <local>false</local>
        <quota_mode>disabled</quota_mode>
        <status>up</status>
        <storage_format>v3</storage_format>
        <supported_versions>
            <version>
                <major>4</major>
                <minor>1</minor>
            </version>
            <version>
                <major>4</major>
                <minor>0</minor>
            </version>
        </supported_versions>
        <version>
            <major>4</major>
            <minor>0</minor>
        </version>
        <mac_pool href="/ovirt-engine/api/macpools/58ca604b-017d-0374-0220-00000000014e" id="58ca604b-017d-0374-0220-00000000014e"/>

--- Additional comment from Dan Kenigsberg on 2017-08-10 10:07:08 EDT ---

Could you please check if you see the same bug in recent 4.1 builds?

--- Additional comment from Avihai on 2017-08-13 03:58:47 EDT ---

Hi Dan , Issue does not occur on latest 4.1 . 

Tested on ovirt-engine-4.1.5.2-0.1.el7.noarch

--- Additional comment from Martin Mucha on 2017-08-13 05:30:20 EDT ---

just to give some specific info:

error got into master: Aug 9 2:39 PM
error was fixed in master: Aug 13 11:10 AM

So it was (luckily) very short period of time, thanks to the reporter(s) of this bug.

--- Additional comment from Avihai on 2017-08-13 07:44:13 EDT ---

Verified at 4.2.0-0.0.master.20170811144920.gita423008.el7.centos

--- Additional comment from Avihai on 2017-08-13 10:41:53 EDT ---

My bad , running automation again I still see the same issue in latest master.
It appears I did not add a VM when I verified this issue so the upgrade worked.

The Same setup (as above) is your's for further debug.

please ping me when done.

Engine log:
2017-08-13 17:18:14,409+03 INFO  [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-8) [5a71d545] Running command: CreateUserSessionCommand internal: false.
2017-08-13 17:18:14,471+03 WARN  [org.ovirt.engine.core.bll.storage.pool.UpdateStoragePoolCommand] (default task-8) [cca7ce5f-8055-469b-b877-1b27402b5da2] Validation of action 'UpdateStoragePool' failed for user
 admin@internal-authz. Reasons: VAR__TYPE__STORAGE__POOL,VAR__ACTION__UPDATE,ACTION_TYPE_FAILED_CANNOT_MIGRATE_MACS_DUE_TO_DUPLICATES,$ACTION_TYPE_FAILED_CANNOT_MIGRATE_MACS_DUE_TO_DUPLICATES_LIST         00:1a:
4a:16:25:ad,$ACTION_TYPE_FAILED_CANNOT_MIGRATE_MACS_DUE_TO_DUPLICATES_LIST_COUNTER 1
2017-08-13 17:18:14,481+03 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-8) [] Operation Failed: [Cannot migrate MACs to another MAC pool, because that action would create d
uplicates in target MAC pool, which are not allowed. Problematic MACs are    00:1a:4a:16:25:ad]

--- Additional comment from Avihai on 2017-08-13 10:44:10 EDT ---

DC name = dc1
ID = 80ef202f-2454-483d-a1f0-beedb57e21c7

See print screen of the issue with postman app .

--- Additional comment from Avihai on 2017-08-13 10:45 EDT ---



--- Additional comment from Avihai on 2017-08-13 12:30:48 EDT ---

build was wrong , waiting for fixed build.

--- Additional comment from Avihai on 2017-08-14 10:50:01 EDT ---

verified on 4.2.0-0.0.master.20170813134654.gitaee967b.el7.centos

--- Additional comment from Martin Mucha on 2017-08-15 12:04:24 EDT ---