From the description it seems that 3.5.7-allocated MACs were completely ignored on upgrade. If this reproduces, its severity is much worse.

Can you reproduce this (allocate macs in 3.5.7, upgrade to 3.6.1, verify that preallocated macs are acknowledged as such)?

Hi Dan,

Unfortunately, i can't reproduce this report.

I'll try to add what I know about mac pools. I don't know what's "mixed QE upgrade environment" and how that can relate, so I'll talk only about production environment.

MacPools was and are in-memory objects only. MacPools content does not have it's own db storage. In 3.5, mac pool settings were stored in vdc_options table; 3 options were stored there: MacPoolRanges, MaxMacsCountInPool and AllowDuplicateMacAddresses (all constants taken from ConfigValues.java). During engine startup one global mac pool was created using these options. Then, as a part of global mac pool initialization, all VmNics were queried from db, and their mac addresses were registered in this pool. So if you (somehow) created VmNic record outside of engine, duplicity could be easily created.

3.5 and 3.6 brought some changes. Upgrade process takes three options mentioned above, and creates one record in mac_pool table, naming such pool 'Default', and this pool agregates 0..N ranges in mac_pool_ranges table, which were parsed from option MacPoolRanges. Rest remains the same. Pool is initialized on engine startup, and it's content in initialized using VmNics records.

some comments:
• There's *no* manipulation with VmNics / mac addresses during upgrade.
• even if duplication is allowed, when asking for *some* mac address, pool will never return already used MAC address.
• To get already used mac address from mac pool, client [of mac pool] must specifically ask for this address. When happens next, depends on several conditions. Already used MAC may be returned, if duplicates are allowed OR [in logical operator meaning] or method 'forceAddMac' of mac pool is used to register MAC address. This method does not check duplicity setting and does not care about duplicates. And this method (aside from other usages) is called from ImportVmCommandBase when getParameters().isImportAsNewEntity() is false. So if your college used this way to create VM it's ‘normal’ that there's duplicity.

elaborating some more: I've checked how are macs obtained when creating VM from pool and mentioned 'possibly problematic' method is not used. That means, that for mac to be used again pool must have thought, that this mac is not used. Since you created 2 vm in 'old' engine, then turned it off, and did upgrade, then no possible errors are that important, because pool got reinitialized on next engine start. And because you can see mac assigned in those 2 vms, there probably wasn't any error. All MACs initialized on engine start are obtained using: GetAllMacsByMacPoolId db stored procedure. So if they were not initialized on app startup following db query probably did not return them:

SELECT
  mac_addr
FROM  vm_interface
WHERE  
  EXISTS(
    SELECT 1
    FROM  
      vm_static
    JOIN  vds_groups ON vm_static.vds_group_id = vds_groups.vds_group_id
    WHERE 
      vds_groups.storage_pool_id IN (
        SELECT 
          sp.id
        FROM 
          storage_pool sp
        WHERE 
          sp.mac_pool_id = v_id
      )
      AND vm_static.vm_guid = vm_interface.vm_guid
  );

where 'v_id' is ID of some mac pool (there should be only one named 'Default' after upgrade).

———
So probable resolution:
a) default mac pool was not created during upgrade (see 03_06_0090_add_mac_pool_ranges_to_storage_pool.sql)

b) your datacenter wasn't updated to reference to default mac pool during upgrade (see 03_06_0090_add_mac_pool_ranges_to_storage_pool.sql)

c) all options above is invalid and given db query just did not return MAC addresses of those VMs, because of some error in that query I'm not aware of.

Can any of these make sense?

Hi Martin, i don't know, but my 2 VMs that were created in the 'old' 3.5.7 engine were ON(they didn't turned off) during the upgrade process.

Those VMs should be out of play, their state/content shouldn't make a difference. Mac pool works only using data in engine DB. If you still have access to engine exhibiting this error, please verify whether a) or b) isn't met. Query I posted seems ok to me, so if a) or b) isn't met and query is valid, I'm out of ideas (so far).

also — I know that you mentioned in initial description, that it did not happen, but are you really sure, that your college did not created another mac pool for example? If he would created another mac pool with same ranges (or overlapping another pool), this scenario can easily happen ...

(In reply to Martin Mucha from comment #6)
> Those VMs should be out of play, their state/content shouldn't make a
> difference. Mac pool works only using data in engine DB. If you still have
> access to engine exhibiting this error, please verify whether a) or b) isn't
> met. Query I posted seems ok to me, so if a) or b) isn't met and query is
> valid, I'm out of ideas (so far).

Hi Martin,

I don't have access to the engine that exhibiting this error and i couldn't reproduce this report. (maybe we have a snapshot of the engine before the upgrade, not sure. If we have i will let you know)

No one has created another mac pool range, only the default one(from 3.5) is the one that exists.

This smells serious, but it does not reproduce and we fail to understand where it could come from. Please reopen when another instance of the bug shows up.