Description of problem: Editing of snapshot image from horizon dashboard is not allowed until following 2 properties are set or removed from metadata tree. os_hash_algo os_hash_value Version-Release number of selected component (if applicable): RHOSP16.1 How reproducible: Steps to Reproduce: 1.Create instance, "do not create volume", wait for instance to be ready. 2."create snapshot". Wait for snapshot-image to be created. 3. Editing the image via horizon enforces either manually specifying or removing os_hash_algo and os_hash_value Actual results: the update image dialog considers the form as incomplete and greys out the 'Update Image' button Expected results: These fields, if needed to be present as default image property, should have some default value like 'none' or any default to avoid forcing end user to update them or remove them Additional info: An upstream issue was reported in Rocky for same problem. https://bugs.launchpad.net/horizon/+bug/1830301
> creating an image using the nova/glance(?)-path (openstack server create image) does not fill os_hash_algo or os_hash_value. Is this the workflow you had in mind? $ openstack image create --file cirros-0.5.2-x86_64-disk.img foo-img ... $ openstack server create --image foo-img --flavor m1.tiny --network private foo-server ... $ openstack server image create --name foo-snapshot foo-server ... $ openstack image show foo-snapshot -c properties -f json | grep os_hash "os_hash_algo": "sha512", "os_hash_value": "2f81a63f92929f22400165eb82edee6676d245f307d49bc4d73c80be060796b560c9ae40c89679bfb5da02f4852af5de2d066a0f181128d6646ffbd2bb21fe56", On devstack@master, the snapshot seems to have correct os_hash_* properties.
I am retargeting this to Nova for now. To the Nova maintainers: In #c11 Radomir narrowed this down to an issue with Glance/Nova, when creating a snapshot, but I think he could not reproduce the issue. I cannot reproduce this on an up-to-date Devstack. I'm not sure if there is anything really "special" that Nova does when creating a snapshot. Does this issue ring a bell?
(In reply to Cyril Roelandt from comment #17) > I am retargeting this to Nova for now. > > To the Nova maintainers: > > In #c11 Radomir narrowed this down to an issue with Glance/Nova, when > creating a snapshot, but I think he could not reproduce the issue. I cannot > reproduce this on an up-to-date Devstack. I'm not sure if there is anything > really "special" that Nova does when creating a snapshot. > > Does this issue ring a bell? No, isn't this something that Glance is responsible for calculating? https://github.com/openstack/glance/blob/9cb3240b94dc4ca7fa288fc66eba1499fb99295f/glance/location.py#L445-L491 You should be able to trace an example request to g-api using Nova's request-id from the image create requests FWIW.
I talked to Dan Smith and he may have had an epiphany. 1) Can we confirm that the user actually uses Nova's "direct snapshot" feature? Which would mean that Ceph does the snapshotting (probably using Copy on Write) and passes a reference to Glance 2) Can we confirm that, when os_hash_algo and os_hash_value are NULL, the user can use the CLI to run the operations they need to perform? (I know they'd rather use Horizon, but it's just for debugging purposes)
FWIW I thought about the zero size snapshot case that I think you're talking about above and it worked for me with boot from volume, I didn't try ephemeral rbd and direct snapshots however.
Yes, I think a direct snapshot would result in having os_hash_{algo,value} set to NULL because in this case, Glance does not download the image and hash it. It would still be completely valid to not set these fields, and Horizon should not prevent the user from doing whatever they want to the image.
(In reply to Cyril Roelandt from comment #21) > Yes, I think a direct snapshot would result in having os_hash_{algo,value} > set to NULL because in this case, Glance does not download the image and > hash it. It would still be completely valid to not set these fields, and > Horizon should not prevent the user from doing whatever they want to the > image. As far as I understand, Horizon does not prevent the user from doing anything. They just have to do two more mouse clicks, to remove the two fields from the values to be saved in the form. I didn't close this because I was worried the missing hashes are a sign of something more serious.
> As far as I understand, Horizon does not prevent the user from doing anything. They just have to do two more mouse clicks, to remove the two fields from the values to be saved in the form. I didn't close this because I was worried the missing hashes are a sign of something more serious. OK. So, this is really not an issue when it comes to Glance. Maybe the user experience could be improved somehow by ignoring fields with NULL values, thus not forcing the user to remove them? I'll re-assign this bug to Horizon and let you decide whether it should be closed or whether some kind of enhancement could be added upstream. Thanks for helping me debug this!
I've finally worked out why we end up with the different behaviour between an ephemeral rbd and volume rbd snapshot when using Nova's imageCreate API. When creating a volume backed snapshot imageCreate will set the size of the image to 0 in the image_meta [1]. This in turn activates an odd hackaround within Nova's image code that asks glance to create the image with an empty string for data, instead of None [2][3]. This causes the Glance backend to actually create an image on the backend, write data and compute the associated hash properties. When creating an ephemeral snapshot that uses the underlying direct snapshot mechanism in Nova imageCreate doesn't do this, instead creating an image initially with an unset size [4][5] that results in nothing being created in Glance's backend. As we don't upload any further data to the image in this case [6] the hash properties are not recalculated and left empty. I can try to create this initial image with a size of 0 in this flow but I'm worried that Glance will not recalculate the hashes later for other backends when we actually upload data, only one way to find out I guess. FWIW I also this behaviour when shelving an ephemeral backed instance using rbd, with the call to create the image [7] not activating the image. Backup should also hit this but I haven't tested as yet [8]. [1] https://github.com/openstack/nova/blob/1c502ebaec29615f08d4af7dc6680f3141d70e67/nova/compute/api.py#L3307-L3310 [2] https://github.com/openstack/nova/blob/1c502ebaec29615f08d4af7dc6680f3141d70e67/nova/image/glance.py#L531-L536 [3] https://github.com/openstack/nova/blob/1c502ebaec29615f08d4af7dc6680f3141d70e67/nova/image/glance.py#L632-L644 [4] https://github.com/openstack/nova/blob/1c502ebaec29615f08d4af7dc6680f3141d70e67/nova/compute/api.py#L3247-L3249 [5] https://github.com/openstack/nova/blob/1c502ebaec29615f08d4af7dc6680f3141d70e67/nova/compute/utils.py#L1203-L1233 [6] https://github.com/openstack/nova/blob/1c502ebaec29615f08d4af7dc6680f3141d70e67/nova/virt/libvirt/driver.py#L2932-L2933 [7] https://github.com/openstack/nova/blob/1c502ebaec29615f08d4af7dc6680f3141d70e67/nova/compute/api.py#L4285-L4287 [8] https://github.com/openstack/nova/blob/1c502ebaec29615f08d4af7dc6680f3141d70e67/nova/compute/api.py#L3218-L3220
@Lee: but is this a bug? From a Glance point of view, not setting os_hash_{algo,value} to any value seems valid to me. @Radomir: Horizon does not treat those two parameters in any special way, but it should still not grey out the update button and require additional action from the user, should it?
> @Radomir: Horizon does not treat those two parameters in any special way, but it should still not grey out the update button and require additional action from the user, should it? I honestly have no idea. We never received any kind of specification about how this dialog is supposed to work exactly, so we just left the default behavior. If someone can point me to any documentation that specifies how exactly it should behave in this case, I will be happy to implement it for the future versions of Horizon.
(In reply to Cyril Roelandt from comment #28) > @Lee: but is this a bug? From a Glance point of view, not setting > os_hash_{algo,value} to any value seems valid to me. Yeah I think for empty images g-api not calculating os_hash_{algo,value} is entirely valid. I'd really like to remove the hackaround in the boot from volume snapshot flow tbh, I'll take a look at that now. > @Radomir: Horizon does not treat those two parameters in any special way, > but it should still not grey out the update button and require additional > action from the user, should it? Agreed, these should be considered optional by Horizon especially when the image reports a size of 0. (In reply to Radomir Dopieralski from comment #29) > > I honestly have no idea. We never received any kind of specification about > how this dialog is supposed to work exactly, so we just left the default > behavior. If someone can point me to any documentation that specifies how > exactly it should behave in this case, I will be happy to implement it for > the future versions of Horizon. Yeah I don't see the os_hash_{algo,value} properties documented anywhere but they should be: https://docs.openstack.org/glance/latest/admin/useful-image-properties.html (In reply to Lee Yarwood from comment #27) > > I can try to create this initial image with a size of 0 in this flow but I'm > worried that Glance will not recalculate the hashes later for other backends > when we actually upload data, only one way to find out I guess. FWIW I did try this and it caused more issues, specially duplicate rbd volumes in the images pool so it isn't the way forward IMHO. As above I think Horizon just needs to treat these properties as optional and allow them to be None for empty images. Moving back to Horizon.
@Shravan: so, is this a regression? Did it use to work? We are probably going to try and change this dialog's behaviour in a future version of Horizon. > I honestly have no idea. We never received any kind of specification about how this dialog is supposed to work exactly, so we just left the default behavior. If someone can point me to any documentation that specifies how exactly it should behave in this case, I will be happy to implement it for the future versions of Horizon. So, I'm not familiar with Horizon. Can you point me to an example of specification for a dialog box, just so I know exactly what you're expecting? > Yeah I don't see the os_hash_{algo,value} properties documented anywhere but they should be: > https://docs.openstack.org/glance/latest/admin/useful-image-properties.html I think we only document the properties that can be modified by users. os_hash_{algo,value} are read-only properties that the users cannot and should not be messing with. > FWIW I did try this and it caused more issues, specially duplicate rbd volumes in the images pool so it isn't the way forward IMHO. Thanks for trying!
(In reply to Lee Yarwood from comment #27) > I've finally worked out why we end up with the different behaviour between > an ephemeral rbd and volume rbd snapshot when using Nova's imageCreate API. > > When creating a volume backed snapshot imageCreate will set the size of the > image to 0 in the image_meta [1]. This in turn activates an odd hackaround > within Nova's image code that asks glance to create the image with an empty > string for data, instead of None [2][3]. This causes the Glance backend to > actually create an image on the backend, write data and compute the > associated hash properties. > > When creating an ephemeral snapshot that uses the underlying direct snapshot > mechanism in Nova imageCreate doesn't do this, instead creating an image > initially with an unset size [4][5] that results in nothing being created in > Glance's backend. As we don't upload any further data to the image in this > case [6] the hash properties are not recalculated and left empty. > > I can try to create this initial image with a size of 0 in this flow but I'm > worried that Glance will not recalculate the hashes later for other backends > when we actually upload data, only one way to find out I guess. > > FWIW I also this behaviour when shelving an ephemeral backed instance using > rbd, with the call to create the image [7] not activating the image. Backup > should also hit this but I haven't tested as yet [8]. > > [1] > https://github.com/openstack/nova/blob/ > 1c502ebaec29615f08d4af7dc6680f3141d70e67/nova/compute/api.py#L3307-L3310 > [2] > https://github.com/openstack/nova/blob/ > 1c502ebaec29615f08d4af7dc6680f3141d70e67/nova/image/glance.py#L531-L536 > [3] > https://github.com/openstack/nova/blob/ > 1c502ebaec29615f08d4af7dc6680f3141d70e67/nova/image/glance.py#L632-L644 > [4] > https://github.com/openstack/nova/blob/ > 1c502ebaec29615f08d4af7dc6680f3141d70e67/nova/compute/api.py#L3247-L3249 > [5] > https://github.com/openstack/nova/blob/ > 1c502ebaec29615f08d4af7dc6680f3141d70e67/nova/compute/utils.py#L1203-L1233 > [6] > https://github.com/openstack/nova/blob/ > 1c502ebaec29615f08d4af7dc6680f3141d70e67/nova/virt/libvirt/driver.py#L2932- > L2933 > [7] > https://github.com/openstack/nova/blob/ > 1c502ebaec29615f08d4af7dc6680f3141d70e67/nova/compute/api.py#L4285-L4287 > [8] > https://github.com/openstack/nova/blob/ > 1c502ebaec29615f08d4af7dc6680f3141d70e67/nova/compute/api.py#L3218-L3220 I've been asked to iterate why this needs to be fixed in Horizon. Nova's behaviour here while odd isn't entirely invalid. With volume snapshots we end up sending data to glance that it then computes the hashes for and adds these as properties to the image. With ephemeral direct snapshots with rbd we don't send data this data to glance so it doesn't. Both are valid behaviours in the eyes of the Glance API itself and so Horizon shouldn't expect the hash properties to be set. By making these properties optional in Horizon we avoid the entire issue. Moving forward we will however look into aligning our behaviour in both use cases to ensure _NO_ data is uploaded to Glance, ensuring these properties are never generated: [OSP 18] n-cpu uploads volume snapshot images with an empty string as data to Glance causing bogus hashes to be calculated https://bugzilla.redhat.com/show_bug.cgi?id=2011771
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 17.0.1 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:0271