Bug 1970718 - Engine hits NPE when importing template with disks on 2 storage domains
Summary: Engine hits NPE when importing template with disks on 2 storage domains
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 4.4.5
Hardware: x86_64
OS: Linux
urgent
high
Target Milestone: ovirt-4.4.7
: ---
Assignee: Benny Zlotnik
QA Contact: Evelina Shames
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-11 01:20 UTC by Germano Veit Michel
Modified: 2021-07-22 15:14 UTC (History)
7 users (show)

Fixed In Version: ovirt-engine-4.4.7.4
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-22 15:13:32 UTC
oVirt Team: Storage
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 6113881 0 None None None 2021-06-11 01:29:56 UTC
Red Hat Product Errata RHSA-2021:2865 0 None None None 2021-07-22 15:14:29 UTC
oVirt gerrit 115260 0 master MERGED core: use disk's sd_id when retrieving info 2021-06-17 10:50:10 UTC

Description Germano Veit Michel 2021-06-11 01:20:17 UTC
Description of problem:

Engine is unable to import templates with subversions that have disks that were moved to another SD. 
It looks for the image of the base version of the template on the SD that the other version of the template has that disk, and import fails.

Version-Release number of selected component (if applicable):
rhvm-4.4.5.11-0.1.el8ev.noarch

How reproducible:
Always

Steps to Reproduce:
1. Have 2 Storage Domains
2. Create a VM with raw 2 disks, one one each
3. Make a template of it, with each disk on a different SD
4. On the VM, move one disk to the other SD, so that both disks of the VM are now on the same SD
5. Create another template, as subversion of the first.

6. At this point you should have this, each template version has 2 disks, the base has 2 disks on 2 SDs and the subversion has both disks in 1 SD.

engine=# SELECT template_version_names,image_group_id,disk_alias,storage_id,storage_name FROM images_storage_domain_view WHERE vm_names ilike '%Test-Template%' order by template_version_names;
 template_version_names |            image_group_id            |  disk_alias   |              storage_id              | storage_name 
------------------------+--------------------------------------+---------------+--------------------------------------+--------------
 base version           | c0a2e69e-98a7-4338-bddb-9a40dba78490 | Test-VM_Disk1 | 83dfe34b-83cd-4450-8280-07d7b5e2e10d | NFS-1
 base version           | 0fc52816-db4e-4803-89b5-af0bab2032d3 | Test-VM_Disk2 | cdec5e42-4908-4441-85fc-ccbf9873afd0 | NFS-2
 Single-SD              | 1fdc5175-cf98-4ced-8481-6f1adf394310 | Test-VM_Disk1 | 83dfe34b-83cd-4450-8280-07d7b5e2e10d | NFS-1
 Single-SD              | ab9898e7-abad-4eba-88f6-b40b8d5126b9 | Test-VM_Disk2 | 83dfe34b-83cd-4450-8280-07d7b5e2e10d | NFS-1
(4 rows)

7. Maintenance and detach both SDs

8. Attach them again

9. Look at the OVFs imported into the DB again (unregistered_ovf_of_entities table), they match the table above, so all good until this point.

    <TemplateType>TEMPLATE</TemplateType>
    <BaseTemplateId>bfd16d67-9933-45fe-be12-e90278eedb7e</BaseTemplateId>
    <TemplateVersionNumber>2</TemplateVersionNumber>
    <TemplateVersionName>Single-SD</TemplateVersionName>

    Has 2 disks pointing to 83dfe34b-83cd-4450-8280-07d7b5e2e10d

      <Item>
        <rasd:Caption>Test-VM_Disk1</rasd:Caption>
        <rasd:InstanceId>856f55dd-7b59-448e-b0dd-54a20af7cae2</rasd:InstanceId>
        <rasd:ResourceType>17</rasd:ResourceType>
        <rasd:HostResource>1fdc5175-cf98-4ced-8481-6f1adf394310/856f55dd-7b59-448e-b0dd-54a20af7cae2</rasd:HostResource>
        <rasd:Parent>00000000-0000-0000-0000-000000000000</rasd:Parent>
        <rasd:Template>00000000-0000-0000-0000-000000000000</rasd:Template>
        <rasd:ApplicationList/>
        <rasd:StorageId>83dfe34b-83cd-4450-8280-07d7b5e2e10d</rasd:StorageId>
        <rasd:StoragePoolId>00e7c810-b844-11eb-b8bb-5254000000ff</rasd:StoragePoolId>
        <rasd:CreationDate>2021/06/11 00:33:17</rasd:CreationDate>
        <rasd:LastModified>2021/06/11 00:33:16</rasd:LastModified>
        <rasd:last_modified_date>2021/06/11 00:44:04</rasd:last_modified_date>
        <Type>disk</Type>
        <Device>disk</Device>
        <rasd:Address/>
        <BootOrder>1</BootOrder>
        <IsPlugged>true</IsPlugged>
        <IsReadOnly>false</IsReadOnly>
        <Alias/>
      </Item>
      <Item>
        <rasd:Caption>Test-VM_Disk2</rasd:Caption>
        <rasd:InstanceId>00e82209-dc63-4d98-bc90-7132acaa1f59</rasd:InstanceId>
        <rasd:ResourceType>17</rasd:ResourceType>
        <rasd:HostResource>ab9898e7-abad-4eba-88f6-b40b8d5126b9/00e82209-dc63-4d98-bc90-7132acaa1f59</rasd:HostResource>
        <rasd:Parent>00000000-0000-0000-0000-000000000000</rasd:Parent>
        <rasd:Template>00000000-0000-0000-0000-000000000000</rasd:Template>
        <rasd:ApplicationList/>
        <rasd:StorageId>83dfe34b-83cd-4450-8280-07d7b5e2e10d</rasd:StorageId>
        <rasd:StoragePoolId>00e7c810-b844-11eb-b8bb-5254000000ff</rasd:StoragePoolId>
        <rasd:CreationDate>2021/06/11 00:33:17</rasd:CreationDate>
        <rasd:LastModified>2021/06/11 00:33:16</rasd:LastModified>
        <rasd:last_modified_date>2021/06/11 00:44:04</rasd:last_modified_date>
        <Type>disk</Type>
        <Device>disk</Device>
        <rasd:Address/>
        <BootOrder>0</BootOrder>
        <IsPlugged>true</IsPlugged>
        <IsReadOnly>false</IsReadOnly>
        <Alias/>
      </Item>

    ----/----

    <TemplateType>TEMPLATE</TemplateType>
    <BaseTemplateId>bfd16d67-9933-45fe-be12-e90278eedb7e</BaseTemplateId>
    <TemplateVersionNumber>1</TemplateVersionNumber>
    <TemplateVersionName>base version</TemplateVersionName>
    
    And 2 disks pointing, each pointing to each SD.

      <Item>
        <rasd:Caption>Test-VM_Disk2</rasd:Caption>
        <rasd:InstanceId>ebba1150-d3ba-4af0-ac59-c81e5254e479</rasd:InstanceId>
        <rasd:ResourceType>17</rasd:ResourceType>
        <rasd:HostResource>0fc52816-db4e-4803-89b5-af0bab2032d3/ebba1150-d3ba-4af0-ac59-c81e5254e479</rasd:HostResource>
        <rasd:Parent>00000000-0000-0000-0000-000000000000</rasd:Parent>
        <rasd:Template>00000000-0000-0000-0000-000000000000</rasd:Template>
        <rasd:ApplicationList/>
        <rasd:StorageId>cdec5e42-4908-4441-85fc-ccbf9873afd0</rasd:StorageId>
        <rasd:StoragePoolId>00e7c810-b844-11eb-b8bb-5254000000ff</rasd:StoragePoolId>
        <rasd:CreationDate>2021/06/11 00:29:15</rasd:CreationDate>
        <rasd:LastModified>2021/06/11 00:29:14</rasd:LastModified>
        <rasd:last_modified_date>2021/06/11 00:44:04</rasd:last_modified_date>
        <Type>disk</Type>
        <Device>disk</Device>
        <rasd:Address/>
        <BootOrder>0</BootOrder>
        <IsPlugged>true</IsPlugged>
        <IsReadOnly>false</IsReadOnly>
        <Alias/>
      </Item>
      <Item>
        <rasd:Caption>Test-VM_Disk1</rasd:Caption>
        <rasd:InstanceId>a5de6da7-6a9a-44e3-9fc4-29d461e43281</rasd:InstanceId>
        <rasd:ResourceType>17</rasd:ResourceType>
        <rasd:HostResource>c0a2e69e-98a7-4338-bddb-9a40dba78490/a5de6da7-6a9a-44e3-9fc4-29d461e43281</rasd:HostResource>
        <rasd:Parent>00000000-0000-0000-0000-000000000000</rasd:Parent>
        <rasd:Template>00000000-0000-0000-0000-000000000000</rasd:Template>
        <rasd:ApplicationList/>
        <rasd:StorageId>83dfe34b-83cd-4450-8280-07d7b5e2e10d</rasd:StorageId>
        <rasd:StoragePoolId>00e7c810-b844-11eb-b8bb-5254000000ff</rasd:StoragePoolId>
        <rasd:CreationDate>2021/06/11 00:29:15</rasd:CreationDate>
        <rasd:LastModified>2021/06/11 00:29:14</rasd:LastModified>
        <rasd:last_modified_date>2021/06/11 00:44:04</rasd:last_modified_date>
        <Type>disk</Type>
        <Device>disk</Device>
        <rasd:Address/>
        <BootOrder>1</BootOrder>
        <IsPlugged>true</IsPlugged>
        <IsReadOnly>false</IsReadOnly>
        <Alias/>
      </Item>

10. Now try to import the template (base first), engine hits NPE on ImportVmTemplateFromConfigurationCommand because VDSM returned nothing on a prepareImage() command.

2021-06-11 10:56:16,358+10 INFO  [org.ovirt.engine.core.bll.exportimport.ImportVmTemplateFromConfigurationCommand] (EE-ManagedThreadFactory-engine-Thread-612) [0ff0f3e6-7ded-42a7-b033-932cf83990cb] Running command: ImportVmTemplateFromConfigurationCommand internal: false. Entities affected :  ID: 83dfe34b-83cd-4450-8280-07d7b5e2e10d Type: StorageAction group IMPORT_EXPORT_VM with role type ADMIN
2021-06-11 10:56:16,366+10 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (EE-ManagedThreadFactory-engine-Thread-612) [0ff0f3e6-7ded-42a7-b033-932cf83990cb] START, PrepareImageVDSCommand(HostName = host1.kvm.local, PrepareImageVDSCommandParameters:{hostId='dd996116-6a54-4b95-92fe-c497269f8e63'}), log id: 7fc67ee5
2021-06-11 10:56:16,395+10 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (EE-ManagedThreadFactory-engine-Thread-612) [0ff0f3e6-7ded-42a7-b033-932cf83990cb] Failed in 'PrepareImageVDS' method, for vds: 'host1.kvm.local'; host: 'host1.kvm.local': null
2021-06-11 10:56:16,395+10 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (EE-ManagedThreadFactory-engine-Thread-612) [0ff0f3e6-7ded-42a7-b033-932cf83990cb] Command 'PrepareImageVDSCommand(HostName = host1.kvm.local, PrepareImageVDSCommandParameters:{hostId='dd996116-6a54-4b95-92fe-c497269f8e63'})' execution failed: null
2021-06-11 10:56:16,395+10 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (EE-ManagedThreadFactory-engine-Thread-612) [0ff0f3e6-7ded-42a7-b033-932cf83990cb] FINISH, PrepareImageVDSCommand, return: , log id: 7fc67ee5
2021-06-11 10:56:16,396+10 INFO  [org.ovirt.engine.core.utils.transaction.TransactionSupport] (EE-ManagedThreadFactory-engine-Thread-612) [0ff0f3e6-7ded-42a7-b033-932cf83990cb] transaction rolled back
2021-06-11 10:56:16,396+10 ERROR [org.ovirt.engine.core.bll.exportimport.ImportVmTemplateFromConfigurationCommand] (EE-ManagedThreadFactory-engine-Thread-612) [0ff0f3e6-7ded-42a7-b033-932cf83990cb] Command 'org.ovirt.engine.core.bll.exportimport.ImportVmTemplateFromConfigurationCommand' failed: EngineException: java.lang.NullPointerException (Failed with error ENGINE and code 5001)
2021-06-11 10:56:16,415+10 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-612) [0ff0f3e6-7ded-42a7-b033-932cf83990cb] EVENT_ID: TEMPLATE_IMPORT_FROM_CONFIGURATION_FAILED(181), Failed to import Template Test-Template from the given configuration.

11. So PrepareImage() failed. Let's see the prepare image:

2021-06-11 10:56:16,371+1000 INFO  (jsonrpc/6) [vdsm.api] START prepareImage(sdUUID='83dfe34b-83cd-4450-8280-07d7b5e2e10d', spUUID='00e7c810-b844-11eb-b8bb-5254000000ff', imgUUID='0fc52816-db4e-4803-89b5-af0bab2032d3', leafUUID='ebba1150-d3ba-4af0-ac59-c81e5254e479', allowIllegal=True) from=::ffff:192.168.100.253,35864, flow_id=0ff0f3e6-7ded-42a7-b033-932cf83990cb, task_id=8a3c681c-c122-460d-bd8a-0bdd5a28eabf (api:48)
2021-06-11 10:56:16,384+1000 INFO  (jsonrpc/6) [vdsm.api] FINISH prepareImage error=Volume does not exist: ('ebba1150-d3ba-4af0-ac59-c81e5254e479',) from=::ffff:192.168.100.253,35864, flow_id=0ff0f3e6-7ded-42a7-b033-932cf83990cb, task_id=8a3c681c-c122-460d-bd8a-0bdd5a28eabf (api:52)
2021-06-11 10:56:16,384+1000 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='8a3c681c-c122-460d-bd8a-0bdd5a28eabf') Unexpected error (task:880)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 887, in _run
    return fn(*args, **kargs)
  File "<decorator-gen-169>", line 2, in prepareImage
  File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method
    ret = func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 3272, in prepareImage
    raise se.VolumeDoesNotExist(leafUUID)
vdsm.storage.exception.VolumeDoesNotExist: Volume does not exist: ('ebba1150-d3ba-4af0-ac59-c81e5254e479',)

What happened? engine asked for image 0fc52816-db4e-4803-89b5-af0bab2032d3 from SD 83dfe34b-83cd-4450-8280-07d7b5e2e10d. But its not there, is on the other SD.

Note the OVF above is correct, its saying 0fc52816-db4e-4803-89b5-af0bab2032d3 is on cdec5e42-4908-4441-85fc-ccbf9873afd0, just pasting it below again here.

     <Item>
        <rasd:Caption>Test-VM_Disk2</rasd:Caption>
        <rasd:InstanceId>ebba1150-d3ba-4af0-ac59-c81e5254e479</rasd:InstanceId>
        <rasd:ResourceType>17</rasd:ResourceType>
        <rasd:HostResource>0fc52816-db4e-4803-89b5-af0bab2032d3/ebba1150-d3ba-4af0-ac59-c81e5254e479</rasd:HostResource>    <-----------
        <rasd:Parent>00000000-0000-0000-0000-000000000000</rasd:Parent>
        <rasd:Template>00000000-0000-0000-0000-000000000000</rasd:Template>
        <rasd:ApplicationList/>
        <rasd:StorageId>cdec5e42-4908-4441-85fc-ccbf9873afd0</rasd:StorageId>             <--------------
        
So the engine is looking for the disk of the base version on the wrong SD.

Comment 1 Germano Veit Michel 2021-06-11 01:23:30 UTC
(In reply to Germano Veit Michel from comment #0)
> So the engine is looking for the disk of the base version on the wrong SD.

It seems to be looking at the SD that the import was triggered from instead of what is in the OVF, but not sure.

Comment 3 Germano Veit Michel 2021-06-11 05:11:50 UTC
This is actually a lot simpler, there is no need for subversions.

Steps to Reproduce:
1. Have 2 Storage Domains
2. Create a VM with raw 2 disks, one on each
3. Make a template of it, with each disk on a different SD
4. Maintenance and detach both SDs
5. Attach them again
6. Import

Unfortunately its not a corner case any more. Also tried on very latest, same issue.

Version-Release number of selected component (if applicable):
rhvm-4.4.6.8-0.1.el8ev.noarch

Comment 5 Arik 2021-06-13 08:07:26 UTC
(In reply to Germano Veit Michel from comment #1)
> (In reply to Germano Veit Michel from comment #0)
> > So the engine is looking for the disk of the base version on the wrong SD.
> 
> It seems to be looking at the SD that the import was triggered from instead
> of what is in the OVF, but not sure.

Yeah, that seems to be right.
We fixed a similar issue for importing from OVA (bz 1830762)
I think it would be better to do something similar, i.e., to fetch the storage domain that the disk resides in from the database, rather than taking it from the OVF (because that's the most accurate information, theoretically the disk can move between the storage domain before initiating the import).
This flow, however, is owned by the storage team so updating this bz accordingly.

Comment 8 Evelina Shames 2021-06-21 07:44:23 UTC
(In reply to Germano Veit Michel from comment #3)
> This is actually a lot simpler, there is no need for subversions.
> 
> Steps to Reproduce:
> 1. Have 2 Storage Domains
> 2. Create a VM with raw 2 disks, one on each
> 3. Make a template of it, with each disk on a different SD
> 4. Maintenance and detach both SDs
> 5. Attach them again
> 6. Import

Verified on ovirt-engine-4.4.7.4-0.9.el8ev with the above steps.
Template was imported successfully with no errors.

Moving to 'Verified'.

Comment 14 errata-xmlrpc 2021-07-22 15:13:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: RHV Manager (ovirt-engine) security update [ovirt-4.4.7]), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2865


Note You need to log in before you can comment on or make changes to this bug.