Jeremy,

You mention that you can reproduce this deploying an OSP7 environment but the packages you referenced are for OSP6. Could you clarify this and update the version field accordingly?

Flavio, 

In the case the customer listed the RHOSP version as 7.0;maybe they made a mistake. How can we confirm which version this actually?

The customer says it is actually 6.0

Jeremy,

Please, check with the customer whether they booted Nova instances from that image, if they did what is the current status of these instances. We have somehow confirmed that this image is not used.

glance image-show  96fbfb00-913c-4dc4-bcb9-ffdd68597b8c
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| Property 'architecture'        | x86_64                               |
| Property 'hw_qemu_guest_agent' | yes                                  |
| Property 'hw_scsi_model'       | virtio-scsi                          |
| Property 'os_require_quiesce'  | yes                                  |
| checksum                       | dcf5c0995870b8bdd55b40f2e1d46888     |
| container_format               | bare                                 |
| created_at                     | 2016-03-31T17:14:25                  |
| deleted                        | False                                |
| disk_format                    | raw                                  |
| id                             | 96fbfb00-913c-4dc4-bcb9-ffdd68597b8c |
| is_public                      | True                                 |
| min_disk                       | 32                                   |
| min_ram                        | 1024                                 |
| name                           | Win Std 2012R2 1.0                   |
| owner                          | e77ab73efbcc4a538b452476b5ed0e59     |
| protected                      | False                                |
| size                           | 34359738368                          |
| status                         | active                               |
| updated_at                     | 2016-04-11T19:35:44                  |
+--------------------------------+--------------------------------------+

glance image-delete  96fbfb00-913c-4dc4-bcb9-ffdd68597b8c
HTTPInternalServerError (HTTP 500): Unable to delete image 96fbfb00-913c-4dc4-bcb9-ffdd68597b8c



 nova show 741a6c1b-4035-4854-9b39-714962a7a7a2
+--------------------------------------+------------------------------------------------------------------+
| Property                             | Value                                                            |
+--------------------------------------+------------------------------------------------------------------+
| Internal network                     | 10.10.10.33                                 |
| OS-DCF:diskConfig                    | AUTO                                                             |
| OS-EXT-AZ:availability_zone          | nova                                                             |
| OS-EXT-SRV-ATTR:host                 | ocd1c007.osinst.net                                              |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | ocd1c007.osinst.net                                              |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000614                                                |
| OS-EXT-STS:power_state               | 1                                                                |
| OS-EXT-STS:task_state                | -                                                                |
| OS-EXT-STS:vm_state                  | active                                                           |
| OS-SRV-USG:launched_at               | 2016-03-31T17:47:10.000000                                       |
| OS-SRV-USG:terminated_at             | -                                                                |
| accessIPv4                           |                                                                  |
| accessIPv6                           |                                                                  |
| config_drive                         |                                                                  |
| created                              | 2016-03-31T17:46:55Z                                             |
| flavor                               | w1.small (7d2cba2d-4e1c-4713-b259-a1ccb0750ca5)                  |
| hostId                               | f780bac8450eb35da83730ac631bef9bf0de33d8897d7005fd55e340         |
| id                                   | 741a6c1b-4035-4854-9b39-714962a7a7a2                             |
| image                                | Win Std 2012R2 1.0 (96fbfb00-913c-4dc4-bcb9-ffdd68597b8c)        |
| key_name                             | cp-prodkey                                                       |
| metadata                             | {}                                                               |
| name                                 | cpwintest1                                                       |
| os-extended-volumes:volumes_attached | []                                                               |
| progress                             | 0                                                                |
| security_groups                      | default                                                          |
| status                               | ACTIVE                                                           |
| tenant_id                            | 5d2be791d77f4cd19b0666ad831d4377                                 |
| updated                              | 2016-03-31T17:47:10Z                                             |
| user_id                              | def8b1fba74af3b8edb20229430bf82d0f8ba1a7962f36a12fc9c8a15820c45a |
+--------------------------------------+------------------------------------------------------------------+

any update on this?

Jeremy,

IIUC, comment #6, Nova instance 741a6c1b-4035-4854-9b39-714962a7a7a2 is booted from image 96fbfb00-913c-4dc4-bcb9-ffdd68597b8c which you trying to delete. Can you, please terminate this nova instance and then try to delete image again. What happens?

See comment #8.

Sergey, 
If they delete the instance then they can delete the image.

About comment #10, it is the expected behaviour. Now, we should probably log the error and print a nice message rather than leaving the user with a stacktrace. The way I see it, we would need to:

- patch glance_store so that the InUseByStore exception is propagated;
- patch glance so that safe_delete_from_backend catches the InUseByStore exception and logs it.

Would that be an acceptable solution? Also, the code has changed quite a bit since Juno, so I might try and fix this upstream, and then backport the patch all the way to Juno, but that might take some time (or even be impossible/unpratical). Any thought on that?

(In reply to Cyril Roelandt from comment #13)
> About comment #10, it is the expected behaviour. Now, we should probably log
> the error and print a nice message rather than leaving the user with a
> stacktrace. The way I see it, we would need to:
> 
> - patch glance_store so that the InUseByStore exception is propagated;
> - patch glance so that safe_delete_from_backend catches the InUseByStore
> exception and logs it.
> 
> Would that be an acceptable solution? Also, the code has changed quite a bit
> since Juno, so I might try and fix this upstream, and then backport the
> patch all the way to Juno, but that might take some time (or even be
> impossible/unpratical). Any thought on that?

IIUC, this Juno patch https://review.openstack.org/108289 already addresses the api logging issue.

This patch seems to exist in the Juno branch : https://github.com/openstack/glance/blob/juno-eol/glance/api/v1/images.py#L1101 .

OP's issue arise from somewhere else:

File "/usr/lib/python2.7/site-packages/glance/common/store_utils.py", line 58, in safe_delete_from_backend

I believe a second patch would be needed for this code path. Not sure of how well this would apply to master, though, I'll need to look at the code a bit more.

Cyril, 
Ok that's fine, we just need a fix for the customer as soon as possible.

So, I can reproduce this issue on master when using *raw* images. I can delete qcow2 images, though.

There is also a nice error message on master:

$ glance image-delete 2a0342c3-5f03-4c19-a960-cb9e229e266a
Unable to delete image '2a0342c3-5f03-4c19-a960-cb9e229e266a' because it is in use.

cyril,
That seems to be correct. The customer is using ceph so the images are in raw format. Just to clarify, the customer wants to be able to delete the raw images without deleting the instance. The logging seems to be fine.

Indeed. To be honest, I do not think this will be doable in Juno. This feature does not exist in the master branch, and will already be quite tricky to add, so there really is little hope for a backport.

That's understandable, what RHOS version will this feature available?

I can't say for sure. I brought up the subject at today's glance meeting:

http://eavesdrop.openstack.org/meetings/glance/2016/glance.2016-05-26-14.03.txt
http://eavesdrop.openstack.org/meetings/glance/2016/glance.2016-05-26-14.03.log.html

I'm not exactly sure of what we are going to be able to do about this, unfortunately.

Cyril, 
Any update on this?

I'm not sure exactly what should be done about the scrubber. adding Erno.

Erno, you said that the scrubber was a bit buggy at the moment, could you elaborate? How hard would it be to use it to delete images once they are no longer used by any VM?

@Cyril,

The scrubber has no knowledge of whether the image is being used or not. The scrubber is used to delete images asynchronously (known as "soft delete" in Glance). When soft delete is enabled, delete commands will only mark the image as pending_deletion and then the scrubber will do the actual deletion.

The InUser error seen in this report is an (unfortunate) side effect of how the interaction between Nova and Glance happens. If soft deletion is enabled, I'd assume the scrubber will try to delete this image on every run until it finally succeeds, but this should be tested

Hi Jeremy,

Did Flavio's comments help the customer? It sounds to the Glance dev team as though the service is working as configured in this case (and that the image will eventually be scrubbed). Please update us with the case's status when you can.

Thanks,
Elise

Elise, so you are saying the instance is immediately marked for soft delete but will eventually be really deleted? So the solution here is to just wait until it's really deleted ?

Customer is still seeing this issue in OSP8.0

Hello,
Any update on this BZ?

Honestly, this seems like the expected behaviour to me. Being able to delete images that are "in use" might be a nice feature, but this would have to be taken upstream and then integrated into RHOS. This is not something that can be fixed downstream, I'm afraid.