Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1940867

Summary: instance_system_metadata for an instance to be updated if the properties on the corresponding image is updated without rebuild of instance.
Product: Red Hat OpenStack Reporter: Rohini Diwakar <rdiwakar>
Component: openstack-novaAssignee: OSP DFG:Compute <osp-dfg-compute>
Status: CLOSED DUPLICATE QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 16.1 (Train)CC: alifshit, dasmith, eglynn, jhakimra, kchamart, ltamagno, sbauza, sgordon, smooney, srelf, vromanso
Target Milestone: ---Keywords: FutureFeature, Triaged
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-10-18 22:04:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Rohini Diwakar 2021-03-19 12:20:29 UTC
Description of problem:
When the properties on an image is updated, the existing VM does not reflect the image properties. This is expected behaviour and we have to rebuild the instance with the updated image in order for the VM to reflect the change. 

Please see the use case here: 
~~~
Before metadata
Create new image without metadata

openstack image show 81385203-ce46-4ad4-8487-002b9095885d
+------------------+----------------------------------------------------------------------------------------------------------+
| Field            | Value                                                                                                    |
+------------------+----------------------------------------------------------------------------------------------------------+
| checksum         | 443b7623e27ecf03dc9e01ee93f67afe                                                                         |
| container_format | bare                                                                                                     |
| created_at       | 2021-03-18T11:38:03Z                                                                                     |
| disk_format      | raw                                                                                                      |
| file             | /v2/images/81385203-ce46-4ad4-8487-002b9095885d/file                                                     |
| id               | 81385203-ce46-4ad4-8487-002b9095885d                                                                     |
| min_disk         | 0                                                                                                        |
| min_ram          | 0                                                                                                        |
| name             | cirros-test                                                                                              |
| owner            | 299a7232981440f4be9dfc6355008459                                                                         |
| properties       | direct_url='rbd://3a0b5464-47c0-42da-8a03-612b55199261/images/81385203-ce46-4ad4-8487-002b9095885d/snap' |
| protected        | False                                                                                                    |
| schema           | /v2/schemas/image                                                                                        |
| size             | 12716032                                                                                                 |
| status           | active                                                                                                   |
| tags             |                                                                                                          |
| updated_at       | 2021-03-18T11:38:05Z                                                                                     |
| virtual_size     | None                                                                                                     |
| visibility       | private                                                                                                  |
+------------------+----------------------------------------------------------------------------------------------------------+


Create instance off that image.
openstack server show c83fd8d5-aa01-4f00-8971-4c5dc2b25e33
+-------------------------------------+----------------------------------------------------------+
| Field                               | Value                                                    |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                   | AUTO                                                     |
| OS-EXT-AZ:availability_zone         | 00028-1                                                  |
| OS-EXT-SRV-ATTR:host                | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx                      |
| OS-EXT-SRV-ATTR:hypervisor_hostname | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx                      |
| OS-EXT-SRV-ATTR:instance_name       | instance-000017c6                                        |
| OS-EXT-STS:power_state              | Running                                                  |
| OS-EXT-STS:task_state               | None                                                     |
| OS-EXT-STS:vm_state                 | active                                                   |
| OS-SRV-USG:launched_at              | 2021-03-18T11:41:14.000000                               |
| OS-SRV-USG:terminated_at            | None                                                     |
| accessIPv4                          |                                                          |
| accessIPv6                          |                                                          |
| addresses                           | 10.10.20.0/24=10.10.20.9                                 |
| config_drive                        |                                                          |
| created                             | 2021-03-18T11:41:03Z                                     |
| flavor                              | t1.small (7df723bd-3e48-4c0a-9e89-954b2f01831e)          |
| hostId                              | 91ce13aec693a84057739ffce922d6887ea25e45a585e5edc40263b8 |
| id                                  | c83fd8d5-aa01-4f00-8971-4c5dc2b25e33                     |
| image                               | cirros-test (81385203-ce46-4ad4-8487-002b9095885d)       |
| key_name                            | None                                                     |
| name                                | instance-before-metadata-osp13                           |
| progress                            | 0                                                        |
| project_id                          | 299a7232981440f4be9dfc6355008459                         |
| properties                          |                                                          |
| security_groups                     | name='default'                                           |
| status                              | ACTIVE                                                   |
| updated                             | 2021-03-18T11:41:14Z                                     |
| user_id                             | 57dd04593ed34ba4890c358f633c5558                         |
| volumes_attached                    |                                                          |
+-------------------------------------+----------------------------------------------------------+

Update metadata on image
openstack image show 81385203-ce46-4ad4-8487-002b9095885d
+------------------+-----------------------------------------------------------------------------------------------------------------------------+
| Field            | Value                                                                                                                       |
+------------------+-----------------------------------------------------------------------------------------------------------------------------+
| checksum         | 443b7623e27ecf03dc9e01ee93f67afe                                                                                            |
| container_format | bare                                                                                                                        |
| created_at       | 2021-03-18T11:38:03Z                                                                                                        |
| disk_format      | raw                                                                                                                         |
| file             | /v2/images/81385203-ce46-4ad4-8487-002b9095885d/file                                                                        |
| id               | 81385203-ce46-4ad4-8487-002b9095885d                                                                                        |
| min_disk         | 0                                                                                                                           |
| min_ram          | 0                                                                                                                           |
| name             | cirros-test                                                                                                                 |
| owner            | 299a7232981440f4be9dfc6355008459                                                                                            |
| properties       | direct_url='rbd://3a0b5464-47c0-42da-8a03-612b55199261/images/81385203-ce46-4ad4-8487-002b9095885d/snap', os_type='windows' |
| protected        | False                                                                                                                       |
| schema           | /v2/schemas/image                                                                                                           |
| size             | 12716032                                                                                                                    |
| status           | active                                                                                                                      |
| tags             |                                                                                                                             |
| updated_at       | 2021-03-18T11:42:12Z                                                                                                        |
| virtual_size     | None                                                                                                                        |
| visibility       | private                                                                                                                     |
+------------------+-----------------------------------------------------------------------------------------------------------------------------+

Check to see if image metadata is reflecting (expected)
openstack server show c83fd8d5-aa01-4f00-8971-4c5dc2b25e33
+-------------------------------------+----------------------------------------------------------+
| Field                               | Value                                                    |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                   | AUTO                                                     |
| OS-EXT-AZ:availability_zone         | 00028-1                                                  |
| OS-EXT-SRV-ATTR:host                | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx                      |
| OS-EXT-SRV-ATTR:hypervisor_hostname | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx                      |
| OS-EXT-SRV-ATTR:instance_name       | instance-000017c6                                        |
| OS-EXT-STS:power_state              | Running                                                  |
| OS-EXT-STS:task_state               | None                                                     |
| OS-EXT-STS:vm_state                 | active                                                   |
| OS-SRV-USG:launched_at              | 2021-03-18T11:41:14.000000                               |
| OS-SRV-USG:terminated_at            | None                                                     |
| accessIPv4                          |                                                          |
| accessIPv6                          |                                                          |
| addresses                           | 10.10.20.0/24=10.10.20.9                                 |
| config_drive                        |                                                          |
| created                             | 2021-03-18T11:41:03Z                                     |
| flavor                              | t1.small (7df723bd-3e48-4c0a-9e89-954b2f01831e)          |
| hostId                              | 91ce13aec693a84057739ffce922d6887ea25e45a585e5edc40263b8 |
| id                                  | c83fd8d5-aa01-4f00-8971-4c5dc2b25e33                     |
| image                               | cirros-test (81385203-ce46-4ad4-8487-002b9095885d)       |
| key_name                            | None                                                     |
| name                                | instance-before-metadata-osp13                           |
| progress                            | 0                                                        |
| project_id                          | 299a7232981440f4be9dfc6355008459                         |
| properties                          |                                                          |
| security_groups                     | name='default'                                           |
| status                              | ACTIVE                                                   |
| updated                             | 2021-03-18T11:41:14Z                                     |
| user_id                             | 57dd04593ed34ba4890c358f633c5558                         |
| volumes_attached                    |                                                          |
+-------------------------------------+----------------------------------------------------------+

and from the database
select * from instance_system_metadata where instance_uuid = 'c83fd8d5-aa01-4f00-8971-4c5dc2b25e33';
+---------------------+------------+------------+-------+--------------------------------------+------------------------+--------------------------------------+---------+
| created_at          | updated_at | deleted_at | id    | instance_uuid                        | key                    | value                                | deleted |
+---------------------+------------+------------+-------+--------------------------------------+------------------------+--------------------------------------+---------+
| 2021-03-18 11:41:05 | NULL       | NULL       | 65546 | c83fd8d5-aa01-4f00-8971-4c5dc2b25e33 | image_min_disk         | 60                                   |       0 |
| 2021-03-18 11:41:05 | NULL       | NULL       | 65549 | c83fd8d5-aa01-4f00-8971-4c5dc2b25e33 | owner_user_name        | xxxxxxxxxxxxxxxxx                    |       0 |
| 2021-03-18 11:41:05 | NULL       | NULL       | 65552 | c83fd8d5-aa01-4f00-8971-4c5dc2b25e33 | owner_project_name     | xxxxxxxxxxxxxx                       |       0 |
| 2021-03-18 11:41:05 | NULL       | NULL       | 65555 | c83fd8d5-aa01-4f00-8971-4c5dc2b25e33 | image_container_format | bare                                 |       0 |
| 2021-03-18 11:41:05 | NULL       | NULL       | 65558 | c83fd8d5-aa01-4f00-8971-4c5dc2b25e33 | image_min_ram          | 0                                    |       0 |
| 2021-03-18 11:41:05 | NULL       | NULL       | 65561 | c83fd8d5-aa01-4f00-8971-4c5dc2b25e33 | image_disk_format      | raw                                  |       0 |
| 2021-03-18 11:41:05 | NULL       | NULL       | 65564 | c83fd8d5-aa01-4f00-8971-4c5dc2b25e33 | image_base_image_ref   | 81385203-ce46-4ad4-8487-002b9095885d |       0 |
| 2021-03-18 11:41:06 | NULL       | NULL       | 65567 | c83fd8d5-aa01-4f00-8971-4c5dc2b25e33 | boot_roles             | _member_                             |       0 |
+---------------------+------------+------------+-------+--------------------------------------+------------------------+--------------------------------------+---------+
8 rows in set (0.00 sec)

Create new instance 
openstack server show 4a53f6c4-81ef-4514-9bef-46cefd5161aa
+-------------------------------------+----------------------------------------------------------+
| Field                               | Value                                                    |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                   | AUTO                                                     |
| OS-EXT-AZ:availability_zone         | 00028-1                                                  |
| OS-EXT-SRV-ATTR:host                | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx                      |
| OS-EXT-SRV-ATTR:hypervisor_hostname | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx                      |
| OS-EXT-SRV-ATTR:instance_name       | instance-000017c9                                        |
| OS-EXT-STS:power_state              | Running                                                  |
| OS-EXT-STS:task_state               | None                                                     |
| OS-EXT-STS:vm_state                 | active                                                   |
| OS-SRV-USG:launched_at              | 2021-03-18T11:46:10.000000                               |
| OS-SRV-USG:terminated_at            | None                                                     |
| accessIPv4                          |                                                          |
| accessIPv6                          |                                                          |
| addresses                           | 10.10.20.0/24=10.10.20.90                                |
| config_drive                        |                                                          |
| created                             | 2021-03-18T11:45:59Z                                     |
| flavor                              | t1.small (7df723bd-3e48-4c0a-9e89-954b2f01831e)          |
| hostId                              | 91ce13aec693a84057739ffce922d6887ea25e45a585e5edc40263b8 |
| id                                  | 4a53f6c4-81ef-4514-9bef-46cefd5161aa                     |
| image                               | cirros-test (81385203-ce46-4ad4-8487-002b9095885d)       |
| key_name                            | None                                                     |
| name                                | instance-after-metadata-osp13                            |
| progress                            | 0                                                        |
| project_id                          | 299a7232981440f4be9dfc6355008459                         |
| properties                          |                                                          |
| security_groups                     | name='default'                                           |
| status                              | ACTIVE                                                   |
| updated                             | 2021-03-18T11:46:10Z                                     |
| user_id                             | 57dd04593ed34ba4890c358f633c5558                         |
| volumes_attached                    |                                                          |
+-------------------------------------+----------------------------------------------------------+

Then from the database
select * from instance_system_metadata where instance_uuid = '4a53f6c4-81ef-4514-9bef-46cefd5161aa';
+---------------------+------------+------------+-------+--------------------------------------+------------------------+--------------------------------------+---------+
| created_at          | updated_at | deleted_at | id    | instance_uuid                        | key                    | value                                | deleted |
+---------------------+------------+------------+-------+--------------------------------------+------------------------+--------------------------------------+---------+
| 2021-03-18 11:46:02 | NULL       | NULL       | 65570 | 4a53f6c4-81ef-4514-9bef-46cefd5161aa | image_os_type          | windows                              |       0 |
| 2021-03-18 11:46:02 | NULL       | NULL       | 65573 | 4a53f6c4-81ef-4514-9bef-46cefd5161aa | image_min_disk         | 60                                   |       0 |
| 2021-03-18 11:46:02 | NULL       | NULL       | 65576 | 4a53f6c4-81ef-4514-9bef-46cefd5161aa | owner_user_name        | xxxxxxxxxxxxxxxxx                    |       0 |
| 2021-03-18 11:46:02 | NULL       | NULL       | 65579 | 4a53f6c4-81ef-4514-9bef-46cefd5161aa | owner_project_name     | xxxxxxxxxxxxxxx                      |       0 |
| 2021-03-18 11:46:02 | NULL       | NULL       | 65582 | 4a53f6c4-81ef-4514-9bef-46cefd5161aa | image_container_format | bare                                 |       0 |
| 2021-03-18 11:46:02 | NULL       | NULL       | 65585 | 4a53f6c4-81ef-4514-9bef-46cefd5161aa | image_min_ram          | 0                                    |       0 |
| 2021-03-18 11:46:02 | NULL       | NULL       | 65588 | 4a53f6c4-81ef-4514-9bef-46cefd5161aa | image_disk_format      | raw                                  |       0 |
| 2021-03-18 11:46:02 | NULL       | NULL       | 65591 | 4a53f6c4-81ef-4514-9bef-46cefd5161aa | image_base_image_ref   | 81385203-ce46-4ad4-8487-002b9095885d |       0 |
| 2021-03-18 11:46:03 | NULL       | NULL       | 65594 | 4a53f6c4-81ef-4514-9bef-46cefd5161aa | boot_roles             | _member_                             |       0 |
+---------------------+------------+------------+-------+--------------------------------------+------------------------+--------------------------------------+---------+
9 rows in set (0.00 sec)
~~~

Version-Release number of selected component (if applicable):
RHOSP13

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 smooney 2021-03-30 15:43:15 UTC
we have a similar feature request in https://bugzilla.redhat.com/show_bug.cgi?id=1886885
to allow an atomic update of both the flavor and image.

we spoke about it upstrema at the last ptg.

"""
    sean-k-mooney): add a server recreate API

    Example:

    openstack server recreate --image <image-id> --flavor <flavor-id> <server>

    recreate is a data and port/resouce preserving move opertion by default

    image and flavor would be optional

    if neither image or flavor are passed then recreate consults the schduler for a new host that is compatiable with the latest state fo the image metadta and flavor extra specs instead of the embeded copy

    so this is a cold migrate  the new thing is updating the image metadata if that was updated since the boot

    if image is passed recreate is a rebuild that is allowed to change resouce useage/numa toplogy since it is a move operations and can/likely will change host.

    this is a new case

    if flavor is passed recreate is a resize that auto confrims. 

    this is negotiably but i think we shotheuld not have to manually confirm resizes or migration today so i would prefer not to intoduce that requirement to recreate

    i would prefer to eventully remove resize verify and explict confrim/revert form the other apis but that is out of scope. it is the reason why i would prefer this api to work like rebuild, live-migrate and shelve however rather then emulate resize.

    This can be closely emulated today by setting resize_confirm_window to a small positive number

    if both flavor and image are passed, this action recreates the server by reusign its volums and port etc but remiaging the root disk and updating the flavor as a singel atomic operation

    currently this requires stoping the vm, then rebuilding and then resizing before starting it again.

    this is useful for cloud instances that use rebuild for upgrades where the resouce requirements of the updated stateless application differ form n to n+1

    this is an edge case that customers have encountered and providing a single atomic operation would simplify there workflow.

    recreate could support shelved instances

    i think resize should support shelved instnace today as resize a shelved instance is jsut confirming its new flavor is compatiable with the image end the updating the flavor in the db.

    the new flaovr will take effect when the instance is unshleved.

    this allow upgrading fo shelved isntance to new flaovrs if operators want to discontinue an old flaovr without

    this could be defered to a follow up that adds supprot for shelved instance to resize and recreate.

    (melwitt): Any reason not to do this as 'openstack server rebuild --keep-ports-and-volumes'? Maybe it's just me but that would feel more intuitive.

    yes i want this to be able to change host so that the requirement in the image can change rebuild are always to the same host they are not move operations.

    also we want to be able to change the image and flavor at teh same time in one operation

    (melwitt): Gah, sorry, I meant resize. But noted on wanting to change image and flavor at the same time :/

    (johnthetubaguy):  Is this really resize with --rebuild <image-id>?

    It feels close to what we said about "no more orchestration" APIs +1 as far as I see a heat template can automate the stop + rebuild + resize + confirm +  start process+1

    New Resize API microversion idea:

    get rid of confirm resize / revert resize, or default to auto confirm resize+1

    -- flavor to the same flavor, allow that if extra specs have changed, or maybe allow always+1

    shelve offloaded, allow resize, in the new microversion, disaalow auto confirm resize

    -- image addition, so you can avoid copying the disk during resize-1 from me (bauzas)... if this work with volumes

    AGREED:

    new microversion 

    auto confirming resize by default

    allow resizing to the same flavor to update extra_spec

    allow resizing shelve offloaded instances

    wait for the rebuild of bvf solution before we do the --image part

"""

the new recreate api was provionally rejected in favor of enhanceing resize at least for the flavor extra spec update usecase with the image related parts defered for now.

we could consider this feature for osp 18 but we do not have a way to support this before then.
it is not backprotable in any way even to osp 17 assuming that will be based on wallaby.
the eairlise we coudl work on this upstream is xena.

i will raise this with the wider team in our triage call tomorrow

Comment 6 Sylvain Bauza 2021-04-07 15:57:58 UTC
We triaged this BZ by our team meeting and we said that while we will continue to accept this BZ for the OSP13 support exception approval question, we ask the customer to rather look at https://bugzilla.redhat.com/show_bug.cgi?id=1886885 for a new OSP release RFE as this is a duplicate one.

In case the support exception is not accepted, please tell us so we would mark this BZ as CLOSED/DUPLICATE.

Comment 9 smooney 2021-05-04 10:13:58 UTC
i cross checked this on our last tech call and we confrimed that only the second query

INSERT INTO nova.instance_system_metadata ( created_at, instance_uuid, `key`, value, deleted ) select now(), uuid, 'image_os_type', 'windows', 0 from nova.instances where deleted = 0 && image_ref IN (select image_id from glance.image_properties where name='os_type' && value='windows') && uuid NOT IN (select instance_uuid from nova.instance_system_metadata where `key` = 'image_os_type' && value = 'windows' && deleted = 0);

is required and teh os_type in the instances table is nolonger used

Comment 14 Artom Lifshitz 2022-07-26 17:43:14 UTC
We can keep this in our RFE backlog and close it if necessary when we do a proper backlog scrub.

Comment 15 Artom Lifshitz 2022-10-18 22:04:45 UTC

*** This bug has been marked as a duplicate of bug 1886885 ***