Bug 977865

Summary: cinder [Negative]: create volume is stuck in creating when nfs server is unavailable
Product: Red Hat OpenStack Reporter: Dafna Ron <dron>
Component: openstack-cinderAssignee: Eric Harney <eharney>
Status: CLOSED WONTFIX QA Contact: Dafna Ron <dron>
Severity: high Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: dron, eharney, sgotliv, srevivo
Target Milestone: ---Keywords: ZStream
Target Release: 6.0 (Juno)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-04-02 13:39:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1016224, 1035891, 1051605, 1066955    
Bug Blocks:    
Attachments:
Description Flags
logs none

Description Dafna Ron 2013-06-25 13:39:37 UTC
Created attachment 765094 [details]
logs

Description of problem:

when running cinder create --display_name <name> 5 when external nfs server is unavailable the volume is stuck in creating instead of moving to error

(snapshot-create for example will be in status error). 
 
Version-Release number of selected component (if applicable):

openstack-cinder-2013.1.2-3.el6ost.noarch
python-cinderclient-1.0.4-1.el6ost.noarch
python-cinder-2013.1.2-3.el6ost.noarch

How reproducible:

100%

Steps to Reproduce:
1. add an nfs server to cinder
2. block connectivity to the server using iptables 
3. create volume 

Actual results:

volume is stuck in creating status instead of moving to error

Expected results:

we should have consistency and move status to error

Additional info: logs

creating a volume is stuck in creating: 

[root@opens-vdsb tmp(keystone_admin)]# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| 0d1c1913-0e9d-4de4-87d1-2d051b6fcfc8 | available | orion-share3 |  5   |     None    |  false   |                                      |
| 6d7ff4c0-86f7-40b6-b59d-99e412bf6bbd |   in-use  | orion-share  |  5   |     None    |  false   | 8f107ee9-809f-45c3-b062-520968e53bb5 |
| 7325035d-01c7-4f8b-bb3e-7c2aac40fb1a |  creating | orion-share7 |  5   |     None    |  false   |                                      |
| b88a6220-e22d-4297-a072-516a146f9474 |  creating | orion-share6 |  5   |     None    |  false   |                                      |
| e243132e-3f70-4a02-b414-3cc450edc1f0 |  creating | orion-share5 |  5   |     None    |  false   |                                      |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+


creating snapshot results in error: 

[root@opens-vdsb ~(keystone_admin)]# cinder snapshot-create 0d1c1913-0e9d-4de4-87d1-2d051b6fcfc8
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|      created_at     |      2013-06-25T13:30:23.477040      |
| display_description |                 None                 |
|     display_name    |                 None                 |
|          id         | fc8862e0-f3cc-4cec-b3c9-931182c6e856 |
|       metadata      |                  {}                  |
|         size        |                  5                   |
|        status       |               creating               |
|      volume_id      | 0d1c1913-0e9d-4de4-87d1-2d051b6fcfc8 |
+---------------------+--------------------------------------+
[root@opens-vdsb ~(keystone_admin)]# cinder snapshot-list 
+--------------------------------------+--------------------------------------+--------+--------------+------+
|                  ID                  |              Volume ID               | Status | Display Name | Size |
+--------------------------------------+--------------------------------------+--------+--------------+------+
| fc8862e0-f3cc-4cec-b3c9-931182c6e856 | 0d1c1913-0e9d-4de4-87d1-2d051b6fcfc8 | error  |     None     |  5   |
+--------------------------------------+--------------------------------------+--------+--------------+------+

Comment 1 Eric Harney 2013-06-25 13:46:48 UTC
Just for my own testing, how did you block connectivity w/ iptables?

The snapshot-create operation is failing because the NFS driver doesn't support it, unrelated to the NFS server availability.

Comment 2 Dafna Ron 2013-06-25 13:53:54 UTC
iptables -A OUTPUT -d <nfs_server> -j DROP

Comment 3 Flavio Percoco 2014-05-02 15:20:39 UTC
The volume is never moved to error because no error is being raised. The reason is that the NFS mount is stalled and the commands (stat, du, ...) have *way too long* timeouts. Besides this, the nfs client never raises an error.

A workaround for this is setting the parameter below in `cinder.conf`:

nfs_mount_options=soft,timeo=20,retrans=1,retry=0

soft -> report an error back when an nfs operation fails
timeo -> timeout for each nfs action
retrans -> number of times an nfs request should be retried
retry -> number of minutes a `mount` operation should be retried.

I believe `soft` should be present by default and `timeo` should be set to 40 (4s) and retrans to `2`. These sound like a saner default values to me. The user can manually change them if necessary.