Bug 1370012

Summary: NetApp Cinder driver: cloning operations are unsuccessful
Product: Red Hat OpenStack Reporter: Faiaz Ahmed <fahmed>
Component: openstack-cinderAssignee: Eric Harney <eharney>
Status: CLOSED ERRATA QA Contact: lkuchlan <lkuchlan>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.0 (Juno)CC: egafford, eharney, fahmed, kan.itani, nalmond, ochalups, pgrist, srevivo
Target Milestone: asyncKeywords: Triaged, ZStream
Target Release: 6.0 (Juno)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-cinder-2014.2.4-11.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1386253 (view as bug list) Environment:
Last Closed: 2017-01-19 13:28:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1386253, 1386255, 1386263, 1386268    

Description Faiaz Ahmed 2016-08-25 04:13:01 UTC
Description of problem:
Cinder cloning operations are unsuccessful when using the cinder NetApp driver

Version-Release number of selected component (if applicable):
openstack-cinder-2014.2.4-6.el7ost.noarch

How reproducible:
Very often (10% of the time),  cinder cloning operations are unsuccessful

Steps to Reproduce:
1.
2.
3.

Actual results:
--
2016-08-24 17:32:43.698 12620 INFO cinder.volume.manager [-] Updating volume replication status.
2016-08-24 17:32:44.337 12620 INFO cinder.volume.drivers.netapp.ssc_utils [-] Successfully completed stale refresh job for server: 192.168.34.201 and vserver dc2svm06
2016-08-24 17:33:19.156 12620 WARNING cinder.volume.drivers.netapp.nfs [req-b9f9c82f-90da-43a5-bfb1-da7c3f7ff45f 5d56086ca2c7e3f69bcc41026758d9ef3deb23ad629c61db5dba4e82cd13f9fd 2add742a5a6f4b318ea28a728d574e10 - - -] Discover file retries exhausted.
2016-08-24 17:33:19.159 12620 INFO cinder.volume.drivers.netapp.nfs [req-b9f9c82f-90da-43a5-bfb1-da7c3f7ff45f 5d56086ca2c7e3f69bcc41026758d9ef3deb23ad629c61db5dba4e82cd13f9fd 2add742a5a6f4b318ea28a728d574e10 - - -] Image cloning unsuccessful for image 7eef9bdf-8386-4afd-bdce-f1cb1e5aa6ef. Message: NFS file could not be discovered.
2016-08-24 17:33:19.176 12620 INFO cinder.brick.remotefs.remotefs [req-b9f9c82f-90da-43a5-bfb1-da7c3f7ff45f 5d56086ca2c7e3f69bcc41026758d9ef3deb23ad629c61db5dba4e82cd13f9fd 2add742a5a6f4b318ea28a728d574e10 - - -] Already mounted: /var/lib/cinder/mnt/66e9ef8a8caf03e6456a414b229748c9
2016-08-24 17:33:19.177 12620 INFO cinder.volume.drivers.netapp.nfs [req-b9f9c82f-90da-43a5-bfb1-da7c3f7ff45f 5d56086ca2c7e3f69bcc41026758d9ef3deb23ad629c61db5dba4e82cd13f9fd 2add742a5a6f4b318ea28a728d574e10 - - -] casted to 192.168.34.201:/nfs_openstack_cinder_prod_vol01
2016-08-24 17:33:20.163 12620 INFO cinder.volume.drivers.netapp.nfs [req-b9f9c82f-90da-43a5-bfb1-da7c3f7ff45f 5d56086ca2c7e3f69bcc41026758d9ef3deb23ad629c61db5dba4e82cd13f9fd 2add742a5a6f4b318ea28a728d574e10 - - -] Performing post clone for volume-493d0240-8d92-4f99-bdbb-1ceb3c3655f5
2016-08-24 17:33:20.327 12620 INFO cinder.volume.drivers.netapp.nfs [req-b9f9c82f-90da-43a5-bfb1-da7c3f7ff45f 5d56086ca2c7e3f69bcc41026758d9ef3deb23ad629c61db5dba4e82cd13f9fd 2add742a5a6f4b318ea28a728d574e10 - - -] Resizing file to 30G
2016-08-24 17:33:20.441 12620 INFO cinder.volume.drivers.netapp.nfs [req-b9f9c82f-90da-43a5-bfb1-da7c3f7ff45f 5d56086ca2c7e3f69bcc41026758d9ef3deb23ad629c61db5dba4e82cd13f9fd 2add742a5a6f4b318ea28a728d574e10 - - -] Copied image 7eef9bdf-8386-4afd-bdce-f1cb1e5aa6ef to volume 493d0240-8d92-4f99-bdbb-1ceb3c3655f5 using copy offload workflow.
2016-08-24 17:33:20.926 12620 INFO cinder.volume.flows.manager.create_volume [req-b9f9c82f-90da-43a5-bfb1-da7c3f7ff45f 5d56086ca2c7e3f69bcc41026758d9ef3deb23ad629c61db5dba4e82cd13f9fd 2add742a5a6f4b318ea28a728d574e10 - - -] Volume volume-493d0240-8d92-4f99-bdbb-1ceb3c3655f5 (493d0240-8d92-4f99-bdbb-1ceb3c3655f5): created successfully
2016-08-24 17:33:43.438 12620 INFO cinder.volume.manager [-] Updating volume status
2016-08-24 17:33:43.459 12620 INFO cinder.brick.remotefs.remotefs [-] Already mounted: /var/lib/cinder/mnt/66e9ef8a8caf03e6456a414b229748c9
2016-08-24 17:33:43.462 12620 INFO cinder.volume.drivers.netapp.ssc_utils [-] Running stale ssc refresh job for server: 192.168.34.201 and vserver dc2svm06
2016-08-24 17:33:43.816 12620 INFO cinder.volume.manager [-] Updating volume replication status.

Expected results:


Additional info:

Comment 6 Faiaz Ahmed 2016-09-06 00:37:13 UTC
Customer feedback
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The NetApp support engineer is currently indicating it is an NFS client configuration problem.  NetApp are keen to hear Red Hat's opinion as they are aware we have opened a case also with Red Hat support for this issue.

NetApp would like us to apply the NFS "lookupcache=pos" mount option as detailed in the Launchpad bug.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Comment 10 Eric Harney 2016-10-13 15:45:31 UTC
I've submitted a new patch upstream which I think may be an easier workaround.  It will be waiting for review from NetApp engineers.

https://review.openstack.org/#/c/386041/

Comment 17 lkuchlan 2017-01-11 08:24:29 UTC
Tested using:
openstack-cinder-2014.2.4-11.el7ost.noarch

Verification flow:

[root@dhcp88-153 ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 4edaba62-7078-43e9-90fc-7bf7dfd0ebad | available |     None     |  1   |     None    |  false   |             |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+

[root@dhcp88-153 ~(keystone_admin)]# cinder show 4edaba62-7078-43e9-90fc-7bf7dfd0ebad
+---------------------------------------+--------------------------------------+
|                Property               |                Value                 |
+---------------------------------------+--------------------------------------+
|              attachments              |                  []                  |
|           availability_zone           |                 nova                 |
|                bootable               |                false                 |
|               created_at              |      2017-01-10T09:55:56.000000      |
|          display_description          |                 None                 |
|              display_name             |                 None                 |
|               encrypted               |                False                 |
|                   id                  | 4edaba62-7078-43e9-90fc-7bf7dfd0ebad |
|                metadata               |                  {}                  |
|         os-vol-host-attr:host         | hostgroup@tripleo_ceph#tripleo_ceph  |
|     os-vol-mig-status-attr:migstat    |                 None                 |
|     os-vol-mig-status-attr:name_id    |                 None                 |
|      os-vol-tenant-attr:tenant_id     |   fe2ed3123296428bb1e57999b6015df7   |
|   os-volume-replication:driver_data   |                 None                 |
| os-volume-replication:extended_status |                 None                 |
|                  size                 |                  1                   |
|              snapshot_id              |                 None                 |
|              source_volid             |                 None                 |
|                 status                |              available               |
|              volume_type              |                 None                 |
+---------------------------------------+--------------------------------------+

[root@dhcp88-153 ~(keystone_admin)]# cinder create 1 --source-volid 4edaba62-7078-43e9-90fc-7bf7dfd0ebad
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2017-01-11T08:17:58.232120      |
| display_description |                 None                 |
|     display_name    |                 None                 |
|      encrypted      |                False                 |
|          id         | e97111a4-0dc9-4640-aa2c-099da2a7615f |
|       metadata      |                  {}                  |
|         size        |                  1                   |
|     snapshot_id     |                 None                 |
|     source_volid    | 4edaba62-7078-43e9-90fc-7bf7dfd0ebad |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+

[root@dhcp88-153 ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 4edaba62-7078-43e9-90fc-7bf7dfd0ebad | available |     None     |  1   |     None    |  false   |             |
| e97111a4-0dc9-4640-aa2c-099da2a7615f | available |     None     |  1   |     None    |  false   |             |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+

[root@dhcp88-153 ~(keystone_admin)]# cinder create 1 --image-id f4b3f773-8594-4d0f-ae06-778c4035bc8e
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2017-01-11T08:19:41.944088      |
| display_description |                 None                 |
|     display_name    |                 None                 |
|      encrypted      |                False                 |
|          id         | 9319797e-5b83-49d2-9d4a-2c0e6aff7608 |
|       image_id      | f4b3f773-8594-4d0f-ae06-778c4035bc8e |
|       metadata      |                  {}                  |
|         size        |                  1                   |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+

[root@dhcp88-153 ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 4edaba62-7078-43e9-90fc-7bf7dfd0ebad | available |     None     |  1   |     None    |  false   |             |
| 9319797e-5b83-49d2-9d4a-2c0e6aff7608 | available |     None     |  1   |     None    |   true   |             |
| e97111a4-0dc9-4640-aa2c-099da2a7615f | available |     None     |  1   |     None    |  false   |             |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+

Comment 19 errata-xmlrpc 2017-01-19 13:28:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0156.html