Bug 1935073 - Ansible ovirt_disk module can create disks with conflicting IDs that cannot be removed
Summary: Ansible ovirt_disk module can create disks with conflicting IDs that cannot b...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 4.4.4
Hardware: x86_64
OS: Unspecified
high
medium
Target Milestone: ovirt-4.4.6
: ---
Assignee: Ahmad Khiet
QA Contact: Amit Sharir
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-03-04 10:53 UTC by Juan Orti
Modified: 2021-06-01 13:23 UTC (History)
6 users (show)

Fixed In Version: ovirt-engine-4.4.6.6
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-06-01 13:22:12 UTC
oVirt Team: Storage
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2021:2179 0 None None None 2021-06-01 13:23:04 UTC
oVirt gerrit 114134 0 master MERGED core: prevent adding duplicate disk images 2021-04-25 10:07:42 UTC

Description Juan Orti 2021-03-04 10:53:55 UTC
Description of problem:
When creating a disk using the Ansible module ovirt_disk and then trying to attach to several VMs, I managed to create several disks with the same ID, throwing errors like the shown below and causing them to be in locked status.

~~~
2021-03-04 07:15:29,067Z ERROR [org.ovirt.engine.core.bll.storage.disk.AddDiskCommand] (default task-4) [c51e6b9c-b144-46d6-a3ae-8901370a6db9] Command 'org.ovirt.engine.core.bll.storage.disk.AddDiskCommand' failed: CallableStatementCallback; SQL [{call insertpermission(?, ?, ?, ?, ?)}ERROR: duplicate key value violates unique constraint "idx_combined_ad_role_object"
  Detail: Key (ad_element_id, role_id, object_id)=(102a6079-54b5-4629-bdbd-9c56fc962f2c, def0000a-0000-0000-0000-def00000000b, 490b2ace-cc16-49c7-8df9-c3d7ea2a1fd4) already exists.
  Where: SQL statement "INSERT INTO permissions (
        ad_element_id,
        id,
        role_id,
        object_id,
        object_type_id
        )
    VALUES (
        v_ad_element_id,
        v_id,
        v_role_id,
        v_object_id,
        v_object_type_id
        )"
PL/pgSQL function insertpermission(uuid,uuid,uuid,uuid,integer) line 3 at SQL statement; nested exception is org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "idx_combined_ad_role_object"
~~~

Version-Release number of selected component (if applicable):
ovirt-engine-4.4.4.7-0.2.el8ev.noarch

How reproducible:
Always

Steps to Reproduce:
1. Run Ansible playbook with tasks like these:

~~~
      - name: Create disk
        ovirt_disk:
          auth: "{{ ovirt_auth }}"
          state: present
          storage_domain: data
          size: "{{ he_disk_size }}"
          name: jorti-{{ rhv_env_name }}-HE_Disk01
          description: hosted_storage for jorti-{{ rhv_env_name }}
          format: raw 
          shareable: yes 
        register: he_disk

      - name: Attach disk to VMs
        ovirt_disk:
          auth: "{{ ovirt_auth }}"
          state: attached
          storage_domain: "{{ rhv_storage_domain }}"
          id: "{{ he_disk.id }}"
          size: "{{ he_disk_size }}"
          vm_name: "{{ item }}"
          interface: virtio_scsi
          format: raw 
          shareable: yes 
        loop: "{{ vm_names }}"
~~~


Actual results:
Several disks created in status locked. They cannot be removed from admin portal and I had to edit the database and remove the LVs manually.

After running the playbook 2 times, there were several volumes with the same image_group_id:

[root@dell-r430-03 ~]# lvs -o +tags --config 'global { use_lvmetad=0 locking_type=1 } devices { filter = [ "a|.*|" ] }' | egrep 'Attr|d41f63b5-9713-4951-b5a8-2774041564f0|490b2ace-cc16-49c7-8df9-c3d7ea2a1fd4'
  LV                                   VG                                   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert LV Tags                                                                               
  31a3c5c3-cec5-40c5-9316-413cd82da490 6814fac1-2e9a-4ab3-bbfe-02a14c2049fe -wi------- 100,00g                                                     IU_d41f63b5-9713-4951-b5a8-2774041564f0,MD_428,PU_00000000-0000-0000-0000-000000000000
  3398cdfa-2228-4c81-9b35-f99c6aa98201 6814fac1-2e9a-4ab3-bbfe-02a14c2049fe -wi------- 100,00g                                                     IU_490b2ace-cc16-49c7-8df9-c3d7ea2a1fd4,MD_447,PU_00000000-0000-0000-0000-000000000000
  49b36024-f794-4c9f-acdc-fbe74ded2199 6814fac1-2e9a-4ab3-bbfe-02a14c2049fe -wi------- 100,00g                                                     IU_490b2ace-cc16-49c7-8df9-c3d7ea2a1fd4,MD_445,PU_00000000-0000-0000-0000-000000000000
  8c82263a-986e-48bd-96f8-86ac835cd970 6814fac1-2e9a-4ab3-bbfe-02a14c2049fe -wi------- 100,00g                                                     IU_d41f63b5-9713-4951-b5a8-2774041564f0,MD_426,PU_00000000-0000-0000-0000-000000000000
  ae93654b-8c07-402a-b32b-8b9779313043 6814fac1-2e9a-4ab3-bbfe-02a14c2049fe -wi------- 100,00g                                                     IU_490b2ace-cc16-49c7-8df9-c3d7ea2a1fd4,MD_446,PU_00000000-0000-0000-0000-000000000000
  fb327b79-ba25-4733-b61b-ecc4d2b5c0fe 6814fac1-2e9a-4ab3-bbfe-02a14c2049fe -wi------- 100,00g                                                     IU_d41f63b5-9713-4951-b5a8-2774041564f0,MD_427,PU_00000000-0000-0000-0000-000000000000


engine=> select vm_names, disk_alias, image_guid, size, parentid, vm_snapshot_id, image_group_id, imagestatus, active, volume_type, volume_format, qcow_compat, volume_classification, storage_name from all_disks_including_snapshots_and_memory where image_group_id in ('490b2ace-cc16-49c7-8df9-c3d7ea2a1fd4', 'd41f63b5-9713-4951-b5a8-2774041564f0') order by image_group_id;
 vm_names |      disk_alias       |              image_guid              |     size     |               parentid               | vm_snapshot_id |            image_group_id            | imagestatus | active | volume_type | volume_format | qcow_compat | volume_classification | storage_name 
----------+-----------------------+--------------------------------------+--------------+--------------------------------------+----------------+--------------------------------------+-------------+--------+-------------+---------------+-------------+-----------------------+--------------
          | jorti-test2-HE_Disk01 | 3398cdfa-2228-4c81-9b35-f99c6aa98201 | 107374182400 | 00000000-0000-0000-0000-000000000000 |                | 490b2ace-cc16-49c7-8df9-c3d7ea2a1fd4 |           2 | t      |           1 |             5 |           0 |                     0 | FC01
          | jorti-test2-HE_Disk01 | 49b36024-f794-4c9f-acdc-fbe74ded2199 | 107374182400 | 00000000-0000-0000-0000-000000000000 |                | 490b2ace-cc16-49c7-8df9-c3d7ea2a1fd4 |           1 | t      |           1 |             5 |           0 |                     0 | FC01
          | jorti-test2-HE_Disk01 | ae93654b-8c07-402a-b32b-8b9779313043 | 107374182400 | 00000000-0000-0000-0000-000000000000 |                | 490b2ace-cc16-49c7-8df9-c3d7ea2a1fd4 |           2 | t      |           1 |             5 |           0 |                     0 | FC01
          | jorti-test-HE_Disk01  | 31a3c5c3-cec5-40c5-9316-413cd82da490 | 107374182400 | 00000000-0000-0000-0000-000000000000 |                | d41f63b5-9713-4951-b5a8-2774041564f0 |           2 | t      |           1 |             5 |           0 |                     0 | FC01
          | jorti-test-HE_Disk01  | 8c82263a-986e-48bd-96f8-86ac835cd970 | 107374182400 | 00000000-0000-0000-0000-000000000000 |                | d41f63b5-9713-4951-b5a8-2774041564f0 |           1 | t      |           1 |             5 |           0 |                     0 | FC01
          | jorti-test-HE_Disk01  | fb327b79-ba25-4733-b61b-ecc4d2b5c0fe | 107374182400 | 00000000-0000-0000-0000-000000000000 |                | d41f63b5-9713-4951-b5a8-2774041564f0 |           2 | t      |           1 |             5 |           0 |                     0 | FC01
(6 rows)

Expected results:
If there's a conflict of disk ID, the action should be refused and should not lead to this inconsistent state.

Additional info:

Comment 5 Ahmad Khiet 2021-04-01 14:36:04 UTC
Hi, 

I can't reproduce this bug using ovirt engine web admin, because if it is backend issue, it should also appear when using webadmin.
can you please verify if it is reproducing?
or try reproduce it using webadmin?


Thank you

Comment 6 Eyal Shenitzky 2021-04-04 05:21:57 UTC
(In reply to Ahmad Khiet from comment #5)
> Hi, 
> 
> I can't reproduce this bug using ovirt engine web admin, because if it is
> backend issue, it should also appear when using webadmin.
> can you please verify if it is reproducing?
> or try reproduce it using webadmin?
> 
> 
> Thank you

This bug cannot be reproduced via webadmin, only via REST-API.

Comment 7 Ahmad Khiet 2021-04-05 10:53:05 UTC
Hi, 

I did automation to simulate HTTP calls from ansible modules.
I did not face this error or any error creating a shareable disk in iscsi and nfs domains and attach them to 5 virtual machines.

can you please verify if it is still reproducing?

Comment 8 Eyal Shenitzky 2021-04-05 10:55:37 UTC
Ahmad, 
Please attach the requests you used for the disk creation.
Did you try to create 2 disks with the same ID?

Comment 9 Ahmad Khiet 2021-04-05 13:26:20 UTC
Hi,

finally reproduced. 
added the same id to the disks as Eyal mentioned in the comment.

Comment 10 Ahmad Khiet 2021-04-05 13:31:24 UTC
the error I see is diffrent 

[697edb29-50bd-4c05-bf26-d276ceb2c984] Command 'org.ovirt.engine.core.bll.storage.disk.image.AddImageFromScratchCommand' failed: CallableStatementCallback; SQL [{call insertimage(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: duplicate key value violates unique constraint "pk_images" 
Detail: Key (image_guid)=(b548366b-fb51-4b41-97be-733c887fe305) already exists.                                                                                                                │
│  Where: SQL statement "INSERT INTO images (                                                                                                                                                     │
│        creation_date,                                                                                                                                                                           │
│        image_guid,                                                                                                                                                                              │
│        it_guid,                                                                                                                                                                                 │
│        size,                                                                                                                                                                                    │
│        ParentId,                                                                                                                                                                                │
│        imageStatus,                                                                                                                                                                             │
│        lastModified,                                                                                                                                                                            │
│        vm_snapshot_id,                                                                                                                                                                          │
│        volume_type,                                                                                                                                                                             │
│        image_group_id,                                                                                                                                                                          │
│        volume_format,                                                                                                                                                                           │
│        active,                                                                                                                                                                                  │
│        volume_classification                                                                                                                                                                    │
│        )                                                                                                                                                                                        │
│    VALUES (                                                                                                                                                                                     │
│        v_creation_date,                                                                                                                                                                         │
│        v_image_guid,                                                                                                                                                                            │
│        v_it_guid,                                                                                                                                                                               │
│        v_size,                                                                                                                                                                                  │
│        v_ParentId,                                                                                                                                                                              │
│        v_imageStatus,                                                                                                                                                                           │
│        v_lastModified,                                                                                                                                                                          │
│        v_vm_snapshot_id,                                                                                                                                                                        │
│        v_volume_type,                                                                                                                                                                           │
│        v_image_group_id,                                                                                                                                                                        │
│        v_volume_format,                                                                                                                                                                         │
│        v_active,                                                                                                                                                                                │
│        v_volume_classification                                                                                                                                                                  │
│        )"                         
PL/pgSQL function insertimage(timestamp with time zone,uuid,uuid,bigint,uuid,integer,timestamp with time zone,uuid,integer,integer,uuid,boolean,smallint) line 3 at SQL statement; 
....

Comment 11 Ahmad Khiet 2021-04-05 13:35:54 UTC
Here's what I used to create the disks :

curl --location --request POST 'http://akhiet-vm:8080/ovirt-engine/api/disks' \
--header 'Authorization: Basic ABC' \
--header 'Content-Type: application/xml' \
--data-raw '<disk id="b7a4c6c5-443b-47c5-967f-6abc79675e8b">
    <image_id>b548366b-fb51-4b41-97be-733c887fe305</image_id>
  <storage_domains>
    <storage_domain id="d14754a2-92c3-4b2d-a2fb-74e20114cbd6"/>
  </storage_domains>
  <name>t9</name>
  <provisioned_size>1048576</provisioned_size>
  <format>raw</format>
  <sparse>false</sparse>
  <shareable>true</shareable>
</disk>'

Comment 12 Juan Orti 2021-04-06 13:25:48 UTC
Removing needinfo as you managed to reproduce. The issue happens when creating a disk specifying an existing ID.

Comment 16 Amit Sharir 2021-05-04 11:09:11 UTC
I verified the bug using REST-API calls like mentioned in comment #6.

Version for testing :

# rpm -q vdsm
vdsm-4.40.60.6-1.el8ev.x86_64

# rpm -q ovirt-engine
ovirt-engine-4.4.6.6-0.10.el8ev.noarch

# rpm -qa | grep release
rhv-release-4.4.6-7-001.noarch
redhat-release-8.4-0.6.el8.x86_64
redhat-release-eula-8.4-0.6.el8.x86_64


POST API call I used for verification (used it twice in a row with the same paremters): 

<disk id="[]">
    <image_id>"[]"</image_id>
  <storage_domains>
    <storage_domain id="[]"/>
  </storage_domains>
  <name>"[]"</name>
  <provisioned_size>"[]"</provisioned_size>
  <format>raw</format>
  <sparse>false</sparse>
  <shareable>true</shareable>
</disk>



Verification summary and conclusions -

After completing the flow mentioned - the expected log output matched the actual log output.
When trying to add a disk with the same id as an existing disk I got this error - "Cannot add Virtual Disk. One of the Template Images already exists" (as expected) - this message appeared both in API call response and in the engine.log.

Bug verified.

Comment 20 errata-xmlrpc 2021-06-01 13:22:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: RHV Manager security update (ovirt-engine) [ovirt-4.4.6]), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2179


Note You need to log in before you can comment on or make changes to this bug.