Hide Forgot
Description of problem: Customer has an issue Cinder not able to delete volumes They have the following issue when trying to delete a volume: 2016-08-17 11:16:04.562 24112 WARNING cinder.volume.drivers.rbd [req-ab25ce32-9375-4b7d-aa12-741145cb54bb admin 18e1133b35444a538043f4eb663be872 - - -] ImageBusy error raised while deleting rbd volume. This may have been caused by a connection from a client that has crashed and, if so, may be resolved by retrying the delete after 30 seconds has elapsed. 2016-08-17 11:16:04.565 24112 ERROR cinder.volume.manager [req-ab25ce32-9375-4b7d-aa12-741145cb54bb admin 18e1133b35444a538043f4eb663be872 - - -] Cannot delete volume 0f3b7fb1-ede0-4fb3-a988-45bffac6d63f: volume is busy Our backend is ceph, we have checked if the volume has snapshots but it has not: [root@vmboae00cephmp02:~]$ rbd snap ls volumes/volume-0f3b7fb1-ede0-4fb3-a988-45bffac6d63f [root@vmboae00cephmp02:~]$ It seems https://bugs.launchpad.net/cinder/+bug/1464259 fixed the nova part for liberty https://git.openstack.org/cgit/openstack/nova/commit/?id=4f2a46987cf705d5dea84e97ef2006342cc5d9c4 for Make sure bdm.volume_id is set after auto-creating volumes. (https://bugs.launchpad.net/cinder/+bug/1522036 is basically a duplicated for https://bugs.launchpad.net/cinder/+bug/1464259 On the Red Hat code, this is already solved for OSP 9 (based on Mitaka) that would public ally released soon. Additionally and for documenting correctly the case, the only affected volumes are coming from volume snapshot or your noticed any additional owws? Customer confirm that it reproduce using: 1. create a volume from an image 2. boot a server from that volume 3. create snapshot from the server (creates image and volume snapshots) 4. delete the server 5. create 2nd server from the image snapshot 6. cleanup If yes, The failure (race) is when deleting the volume snapshot, there is a backing volume that is dependent on the snapshot, and that's not deleted yet. That 2nd volume is attached to the 2nd server (nova does this b/c of the image metadata that has the bdm snapshot info in it). In that case sounds like there is a timing issue in the cinder rbd driver when deleting the dependent volume (for the snapshot) such that it's not gone when we try to delete the volume snapshot (it's considered busy), so that fails. If the backing volume delete from ceph takes longer than the default 196 seconds (that tempest waits for), then we might need to just increase the delete timeout in the ceph job (maybe to 300 seconds) We could also just look at setting rbd_flatten_volume_from_snapshot=True in cinder.conf for the ceph job. This was also corrected at https://review.openstack.org/#/c/205282/ and covered for the OSP 9 cycle. could we backport to OSP8 please
Effectively, the SNAP couldn't deleted, but the volume launched from the snap yes.. In our case cant delete the volumes.. This is the results of the tests: [root@vmboae00mgmtlp01 ~(keystone_admin)]$ nova volume-create --image-id 67f38463-825a-479d-9b8b-4ab6e9a0bdcd --display-name testVOL01 20 +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | BOAE-1 | | bootable | false | | created_at | 2016-08-19T11:02:10.853418 | | display_description | - | | display_name | testVOL01 | | encrypted | False | | id | c9a935c5-6ee5-4718-a187-5a237e989a89 | | image_id | 67f38463-825a-479d-9b8b-4ab6e9a0bdcd | | metadata | {} | | size | 20 | | snapshot_id | - | | source_volid | - | | status | creating | | volume_type | standard | +---------------------+--------------------------------------+ [root@vmboae00mgmtlp01 ~(keystone_admin)]$ nova boot --flavor A3 --key_name pruebas_iaas --availability-zone BOAE-1 --boot-volume c9a935c5-6ee5-4718-a187-5a237e989a89 --nic net-id=6edea878-ee4a-4724-b0ff-53f1a8653ad9 testINST01 +--------------------------------------+--------------------------------------------------+ | Property | Value | +--------------------------------------+--------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | BOAE-1 | | OS-EXT-SRV-ATTR:host | - | | OS-EXT-SRV-ATTR:hypervisor_hostname | - | | OS-EXT-SRV-ATTR:instance_name | inst-00004d09 | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | adminPass | zKAHLrYQ7Tzk | | config_drive | | | created | 2016-08-19T11:12:57Z | | flavor | A3 (1003) | | hostId | | | id | 4e25ed87-a326-492c-a25a-af6f1f24bfbd | | image | Attempt to boot from volume - no image supplied | | key_name | pruebas_iaas | | metadata | {} | | name | testINST01 | | os-extended-volumes:volumes_attached | [{"id": "c9a935c5-6ee5-4718-a187-5a237e989a89"}] | | progress | 0 | | security_groups | default | | status | BUILD | | tenant_id | 18e1133b35444a538043f4eb663be872 | | updated | 2016-08-19T11:12:58Z | | user_id | admin | +--------------------------------------+--------------------------------------------------+ [root@vmboae00mgmtlp01 ~(keystone_admin)]$ cinder snapshot-list +--------------------------------------+--------------------------------------+-----------+------------------------------------+------+ | ID | Volume ID | Status | Display Name | Size | +--------------------------------------+--------------------------------------+-----------+------------------------------------+------+ | 5d05952e-942a-4bcb-99a0-e220dbe960e4 | 21c2437e-af9c-4b14-a3d9-90680170115f | available | None | 60 | | 5f549246-c189-41a7-802d-7a9c11905694 | 9ac19c32-de7b-4248-9d8e-63661b445095 | available | test-snapshot-using-cinder-command | 60 | | 6e0d04de-e846-4617-b661-aefe91c0f0d9 | c9a935c5-6ee5-4718-a187-5a237e989a89 | available | testVOL1_SNAP | 20 | +--------------------------------------+--------------------------------------+-----------+------------------------------------+------+ [root@vmboae00mgmtlp01 ~(keystone_admin)]$ cinder create --snapshot-id 6e0d04de-e846-4617-b661-aefe91c0f0d9 --display-name testVOL1_SNAP 20 +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | BOAE-1 | | bootable | false | | created_at | 2016-08-19T11:43:37.512486 | | display_description | None | | display_name | testVOL1_SNAP | | encrypted | False | | id | 88f474fe-4711-4d5f-b6a0-dc4d4feb4f34 | | metadata | {} | | size | 20 | | snapshot_id | 6e0d04de-e846-4617-b661-aefe91c0f0d9 | | source_volid | None | | status | creating | | volume_type | standard | +---------------------+--------------------------------------+ [root@vmboae00mgmtlp01 ~(keystone_admin)]$ nova delete 4e25ed87-a326-492c-a25a-af6f1f24bfbd Request to delete server 4e25ed87-a326-492c-a25a-af6f1f24bfbd has been accepted. [root@vmboae00mgmtlp01 ~(keystone_admin)]$ [root@vmboae00mgmtlp01 ~(keystone_admin)]$ nova boot --flavor A3 --key_name pruebas_iaas --availability-zone BOAE-1 --boot-volume 88f474fe-4711-4d5f-b6a0-dc4d4feb4f34 --nic net-id=6edea878-ee4a-4724-b0ff-53f1a8653ad9 testINST02 +--------------------------------------+--------------------------------------------------+ | Property | Value | +--------------------------------------+--------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | BOAE-1 | | OS-EXT-SRV-ATTR:host | - | | OS-EXT-SRV-ATTR:hypervisor_hostname | - | | OS-EXT-SRV-ATTR:instance_name | inst-00004d0c | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | adminPass | F3nXkq3SQV6u | | config_drive | | | created | 2016-08-19T11:51:29Z | | flavor | A3 (1003) | | hostId | | | id | 2ae33ad8-3d31-4dc3-815a-dda816eebd60 | | image | Attempt to boot from volume - no image supplied | | key_name | pruebas_iaas | | metadata | {} | | name | testINST02 | | os-extended-volumes:volumes_attached | [{"id": "88f474fe-4711-4d5f-b6a0-dc4d4feb4f34"}] | | progress | 0 | | security_groups | default | | status | BUILD | | tenant_id | 18e1133b35444a538043f4eb663be872 | | updated | 2016-08-19T11:51:29Z | | user_id | admin | +--------------------------------------+--------------------------------------------------+ [root@vmboae00mgmtlp01 ~(keystone_admin)]$ nova delete 2ae33ad8-3d31-4dc3-815a-dda816eebd60 Request to delete server 2ae33ad8-3d31-4dc3-815a-dda816eebd60 has been accepted. [root@vmboae00mgmtlp01 ~(keystone_admin)]$ [root@vmboae00mgmtlp01 ~(keystone_admin)]$ nova show 2ae33ad8-3d31-4dc3-815a-dda816eebd60 ERROR (CommandError): No server with a name or ID of '2ae33ad8-3d31-4dc3-815a-dda816eebd60' exists. [root@vmboae00mgmtlp01 ~(keystone_admin)]$ cinder show 88f474fe-4711-4d5f-b6a0-dc4d4feb4f34 +---------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Property | Value | +---------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | attachments | [] | | availability_zone | BOAE-1 | | bootable | true | | created_at | 2016-08-19T11:43:37.000000 | | display_description | None | | display_name | testVOL1_SNAP | | encrypted | False | | id | 88f474fe-4711-4d5f-b6a0-dc4d4feb4f34 | | metadata | {u'readonly': u'False'} | | os-vol-host-attr:host | cinder-BOAE-1@standard#STANDARD | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 18e1133b35444a538043f4eb663be872 | | os-volume-replication:driver_data | None | | os-volume-replication:extended_status | None | | size | 20 | | snapshot_id | 6e0d04de-e846-4617-b661-aefe91c0f0d9 | | source_volid | None | | status | available | | volume_image_metadata | {u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'raw', u'image_name': u'Cirros', u'image_id': u'67f38463-825a-479d-9b8b-4ab6e9a0bdcd', u'checksum': u'0590d15336f919496ccc91b2c0f667bc', u'min_disk': u'0', u'size': u'41126400'} | | volume_type | standard | +---------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ [root@vmboae00mgmtlp01 ~(keystone_admin)]$ cinder delete 88f474fe-4711-4d5f-b6a0-dc4d4feb4f34 [root@vmboae00mgmtlp01 ~(keystone_admin)]$ [root@vmboae00mgmtlp01 ~(keystone_admin)]$ cinder snapshot-list +--------------------------------------+--------------------------------------+-----------+------------------------------------+------+ | ID | Volume ID | Status | Display Name | Size | +--------------------------------------+--------------------------------------+-----------+------------------------------------+------+ | 5d05952e-942a-4bcb-99a0-e220dbe960e4 | 21c2437e-af9c-4b14-a3d9-90680170115f | available | None | 60 | | 5f549246-c189-41a7-802d-7a9c11905694 | 9ac19c32-de7b-4248-9d8e-63661b445095 | available | test-snapshot-using-cinder-command | 60 | | 6e0d04de-e846-4617-b661-aefe91c0f0d9 | c9a935c5-6ee5-4718-a187-5a237e989a89 | available | testVOL1_SNAP | 20 | +--------------------------------------+--------------------------------------+-----------+------------------------------------+------+ [root@vmboae00mgmtlp01 ~(keystone_admin)]$ cinder snapshot-delete 6e0d04de-e846-4617-b661-aefe91c0f0d9 [root@vmboae00mgmtlp01 ~(keystone_admin)]$ [root@vmboae00mgmtlp01 ~(keystone_admin)]$ cinder snapshot-list +--------------------------------------+--------------------------------------+-----------+------------------------------------+------+ | ID | Volume ID | Status | Display Name | Size | +--------------------------------------+--------------------------------------+-----------+------------------------------------+------+ | 5d05952e-942a-4bcb-99a0-e220dbe960e4 | 21c2437e-af9c-4b14-a3d9-90680170115f | available | None | 60 | | 5f549246-c189-41a7-802d-7a9c11905694 | 9ac19c32-de7b-4248-9d8e-63661b445095 | available | test-snapshot-using-cinder-command | 60 | | 6e0d04de-e846-4617-b661-aefe91c0f0d9 | c9a935c5-6ee5-4718-a187-5a237e989a89 | available | testVOL1_SNAP | 20 | +--------------------------------------+--------------------------------------+-----------+------------------------------------+------+
OSP 8 resolves this issue. Will works for cinder volume created with OSP6? They are upgrading OSP 6 to OSP8. The bug resolved in OSP8 will has retroactive effect?
The customer has all information. Pending to check out. It is an operational issue from technicians.
closing based on comment 9