Bug 1862931 - Share got stuck on "error_deleting" status after restarting manila share service
Summary: Share got stuck on "error_deleting" status after restarting manila share service
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-manila
Version: 17.0 (Wallaby)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Tom Barron
QA Contact: lkuchlan
RHOS Documentation Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-08-03 09:04 UTC by lkuchlan
Modified: 2022-01-11 19:30 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-01-11 19:30:10 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Manila logs (726.57 KB, application/gzip)
2020-08-03 09:10 UTC, lkuchlan
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker OSP-702 0 None None None 2022-01-10 18:34:05 UTC

Description lkuchlan 2020-08-03 09:04:57 UTC
Description of problem:
When trying to delete a manila share after restart manila share service
the share got stuck on "error_deleting" status, although the manila share 
service was up in the delete operation.

Version-Release number of selected component (if applicable):
puppet-manila-15.4.1-0.20200403160104.e41b1b6.el8ost.noarch
python3-manilaclient-1.29.0-0.20200310223441.1b2cafb.el8ost.noarch

How reproducible:
100%

Steps to Reproduce:
(.venv) (overcloud) [stack@undercloud-0 tempest]$ manila create NFS 1
+---------------------------------------+--------------------------------------+
| Property                              | Value                                |
+---------------------------------------+--------------------------------------+
| id                                    | d895f7bf-df9a-46a9-a4e3-dde32da8d55e |
| size                                  | 1                                    |
| availability_zone                     | None                                 |
| created_at                            | 2020-08-03T08:48:27.000000           |
| status                                | creating                             |
| name                                  | None                                 |
| description                           | None                                 |
| project_id                            | c790bd8deb344a3397550acfc22ea4b7     |
| snapshot_id                           | None                                 |
| share_network_id                      | None                                 |
| share_proto                           | NFS                                  |
| metadata                              | {}                                   |
| share_type                            | c775bef9-7e5e-45b5-9234-9d7f81a21640 |
| is_public                             | False                                |
| snapshot_support                      | True                                 |
| task_state                            | None                                 |
| share_type_name                       | default                              |
| access_rules_status                   | active                               |
| replication_type                      | None                                 |
| has_replicas                          | False                                |
| user_id                               | 3548c355531b4eb195dcbcb8774529e8     |
| create_share_from_snapshot_support    | False                                |
| revert_to_snapshot_support            | False                                |
| share_group_id                        | None                                 |
| source_share_group_snapshot_member_id | None                                 |
| mount_snapshot_support                | False                                |
| share_server_id                       | None                                 |
| host                                  |                                      |
+---------------------------------------+--------------------------------------+

(.venv) (overcloud) [stack@undercloud-0 tempest]$ manila list
+--------------------------------------+------+------+-------------+----------------+-----------+-----------------+-------------------------+-------------------+
| ID                                   | Name | Size | Share Proto | Status         | Is Public | Share Type Name | Host                    | Availability Zone |
+--------------------------------------+------+------+-------------+----------------+-----------+-----------------+-------------------------+-------------------+
| d895f7bf-df9a-46a9-a4e3-dde32da8d55e | None | 1    | NFS         | available      | False     | default         | hostgroup@cephfs#cephfs | nova              |
+--------------------------------------+------+------+-------------+----------------+-----------+-----------------+-------------------------+-------------------+

# Restart manila share service.

[root@controller-0 ~]# docker restart openstack-manila-share-podman-0

# Wait until manila share service is up.
 
[root@controller-0 ~]# docker ps | grep openstack-manila-share-podman-0
c8d8cf5fc065  undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-manila-share:16.1_20200625.1                /bin/bash /usr/lo...  2 weeks ago  Up 5 minutes ago         openstack-manila-share-podman-0

(.venv) (overcloud) [stack@undercloud-0 tempest]$ manila delete d895f7bf-df9a-46a9-a4e3-dde32da8d55e

(.venv) (overcloud) [stack@undercloud-0 tempest]$ manila list
+--------------------------------------+------+------+-------------+----------------+-----------+-----------------+-------------------------+-------------------+
| ID                                   | Name | Size | Share Proto | Status         | Is Public | Share Type Name | Host                    | Availability Zone |
+--------------------------------------+------+------+-------------+----------------+-----------+-----------------+-------------------------+-------------------+
| d895f7bf-df9a-46a9-a4e3-dde32da8d55e | None | 1    | NFS         | error_deleting | False     | default         | hostgroup@cephfs#cephfs | nova              |
+--------------------------------------+------+------+-------------+----------------+-----------+-----------------+-------------------------+-------------------+

Actual results:
Share got stuck on "error_deleting" status

Expected results:
Share deletion should succeed

Comment 1 lkuchlan 2020-08-03 09:10:53 UTC
Created attachment 1703252 [details]
Manila logs

Comment 2 Victoria Martinez de la Cruz 2020-08-03 13:39:27 UTC
Have you checked pacemaker status? manila-shr is the only service for manila that is under pacemaker control, checking is status might provide it more info to us.

Comment 3 lkuchlan 2020-08-03 15:53:05 UTC
Pacemaker status was up.

Comment 8 Goutham Pacha Ravi 2022-01-11 19:30:10 UTC
(In reply to lkuchlan from comment #7)
> (In reply to Victoria Martinez de la Cruz from comment #6)
> > Liron, is this one still reproducible? Would you be able to check this?
> > Thanks!
> 
> I could not reproduce it.
> The share was deleted successfully after openstack-manila-share-podman-0
> recovery.

Awesome. "error_deleting" can only occur when the share driver fails during the deletion operation, not if the share manager service is unresponsive (stopped, dead, starting up or "not ready" yet). 

Since this isn't reproducible at the moment. I'm closing this bug. If this occurs, we should grab the logs to see what happened with the share driver and report a new bug.


Note You need to log in before you can comment on or make changes to this bug.