Description of problem: Cinder cloning operations are unsuccessful when using the cinder NetApp driver Version-Release number of selected component (if applicable): openstack-cinder-2014.2.4-6.el7ost.noarch How reproducible: Very often (10% of the time), cinder cloning operations are unsuccessful Steps to Reproduce: 1. 2. 3. Actual results: -- 2016-08-24 17:32:43.698 12620 INFO cinder.volume.manager [-] Updating volume replication status. 2016-08-24 17:32:44.337 12620 INFO cinder.volume.drivers.netapp.ssc_utils [-] Successfully completed stale refresh job for server: 192.168.34.201 and vserver dc2svm06 2016-08-24 17:33:19.156 12620 WARNING cinder.volume.drivers.netapp.nfs [req-b9f9c82f-90da-43a5-bfb1-da7c3f7ff45f 5d56086ca2c7e3f69bcc41026758d9ef3deb23ad629c61db5dba4e82cd13f9fd 2add742a5a6f4b318ea28a728d574e10 - - -] Discover file retries exhausted. 2016-08-24 17:33:19.159 12620 INFO cinder.volume.drivers.netapp.nfs [req-b9f9c82f-90da-43a5-bfb1-da7c3f7ff45f 5d56086ca2c7e3f69bcc41026758d9ef3deb23ad629c61db5dba4e82cd13f9fd 2add742a5a6f4b318ea28a728d574e10 - - -] Image cloning unsuccessful for image 7eef9bdf-8386-4afd-bdce-f1cb1e5aa6ef. Message: NFS file could not be discovered. 2016-08-24 17:33:19.176 12620 INFO cinder.brick.remotefs.remotefs [req-b9f9c82f-90da-43a5-bfb1-da7c3f7ff45f 5d56086ca2c7e3f69bcc41026758d9ef3deb23ad629c61db5dba4e82cd13f9fd 2add742a5a6f4b318ea28a728d574e10 - - -] Already mounted: /var/lib/cinder/mnt/66e9ef8a8caf03e6456a414b229748c9 2016-08-24 17:33:19.177 12620 INFO cinder.volume.drivers.netapp.nfs [req-b9f9c82f-90da-43a5-bfb1-da7c3f7ff45f 5d56086ca2c7e3f69bcc41026758d9ef3deb23ad629c61db5dba4e82cd13f9fd 2add742a5a6f4b318ea28a728d574e10 - - -] casted to 192.168.34.201:/nfs_openstack_cinder_prod_vol01 2016-08-24 17:33:20.163 12620 INFO cinder.volume.drivers.netapp.nfs [req-b9f9c82f-90da-43a5-bfb1-da7c3f7ff45f 5d56086ca2c7e3f69bcc41026758d9ef3deb23ad629c61db5dba4e82cd13f9fd 2add742a5a6f4b318ea28a728d574e10 - - -] Performing post clone for volume-493d0240-8d92-4f99-bdbb-1ceb3c3655f5 2016-08-24 17:33:20.327 12620 INFO cinder.volume.drivers.netapp.nfs [req-b9f9c82f-90da-43a5-bfb1-da7c3f7ff45f 5d56086ca2c7e3f69bcc41026758d9ef3deb23ad629c61db5dba4e82cd13f9fd 2add742a5a6f4b318ea28a728d574e10 - - -] Resizing file to 30G 2016-08-24 17:33:20.441 12620 INFO cinder.volume.drivers.netapp.nfs [req-b9f9c82f-90da-43a5-bfb1-da7c3f7ff45f 5d56086ca2c7e3f69bcc41026758d9ef3deb23ad629c61db5dba4e82cd13f9fd 2add742a5a6f4b318ea28a728d574e10 - - -] Copied image 7eef9bdf-8386-4afd-bdce-f1cb1e5aa6ef to volume 493d0240-8d92-4f99-bdbb-1ceb3c3655f5 using copy offload workflow. 2016-08-24 17:33:20.926 12620 INFO cinder.volume.flows.manager.create_volume [req-b9f9c82f-90da-43a5-bfb1-da7c3f7ff45f 5d56086ca2c7e3f69bcc41026758d9ef3deb23ad629c61db5dba4e82cd13f9fd 2add742a5a6f4b318ea28a728d574e10 - - -] Volume volume-493d0240-8d92-4f99-bdbb-1ceb3c3655f5 (493d0240-8d92-4f99-bdbb-1ceb3c3655f5): created successfully 2016-08-24 17:33:43.438 12620 INFO cinder.volume.manager [-] Updating volume status 2016-08-24 17:33:43.459 12620 INFO cinder.brick.remotefs.remotefs [-] Already mounted: /var/lib/cinder/mnt/66e9ef8a8caf03e6456a414b229748c9 2016-08-24 17:33:43.462 12620 INFO cinder.volume.drivers.netapp.ssc_utils [-] Running stale ssc refresh job for server: 192.168.34.201 and vserver dc2svm06 2016-08-24 17:33:43.816 12620 INFO cinder.volume.manager [-] Updating volume replication status. Expected results: Additional info:
Customer feedback ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The NetApp support engineer is currently indicating it is an NFS client configuration problem. NetApp are keen to hear Red Hat's opinion as they are aware we have opened a case also with Red Hat support for this issue. NetApp would like us to apply the NFS "lookupcache=pos" mount option as detailed in the Launchpad bug. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I've submitted a new patch upstream which I think may be an easier workaround. It will be waiting for review from NetApp engineers. https://review.openstack.org/#/c/386041/
Tested using: openstack-cinder-2014.2.4-11.el7ost.noarch Verification flow: [root@dhcp88-153 ~(keystone_admin)]# cinder list +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | 4edaba62-7078-43e9-90fc-7bf7dfd0ebad | available | None | 1 | None | false | | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ [root@dhcp88-153 ~(keystone_admin)]# cinder show 4edaba62-7078-43e9-90fc-7bf7dfd0ebad +---------------------------------------+--------------------------------------+ | Property | Value | +---------------------------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | created_at | 2017-01-10T09:55:56.000000 | | display_description | None | | display_name | None | | encrypted | False | | id | 4edaba62-7078-43e9-90fc-7bf7dfd0ebad | | metadata | {} | | os-vol-host-attr:host | hostgroup@tripleo_ceph#tripleo_ceph | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | fe2ed3123296428bb1e57999b6015df7 | | os-volume-replication:driver_data | None | | os-volume-replication:extended_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | available | | volume_type | None | +---------------------------------------+--------------------------------------+ [root@dhcp88-153 ~(keystone_admin)]# cinder create 1 --source-volid 4edaba62-7078-43e9-90fc-7bf7dfd0ebad +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | created_at | 2017-01-11T08:17:58.232120 | | display_description | None | | display_name | None | | encrypted | False | | id | e97111a4-0dc9-4640-aa2c-099da2a7615f | | metadata | {} | | size | 1 | | snapshot_id | None | | source_volid | 4edaba62-7078-43e9-90fc-7bf7dfd0ebad | | status | creating | | volume_type | None | +---------------------+--------------------------------------+ [root@dhcp88-153 ~(keystone_admin)]# cinder list +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | 4edaba62-7078-43e9-90fc-7bf7dfd0ebad | available | None | 1 | None | false | | | e97111a4-0dc9-4640-aa2c-099da2a7615f | available | None | 1 | None | false | | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ [root@dhcp88-153 ~(keystone_admin)]# cinder create 1 --image-id f4b3f773-8594-4d0f-ae06-778c4035bc8e +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | created_at | 2017-01-11T08:19:41.944088 | | display_description | None | | display_name | None | | encrypted | False | | id | 9319797e-5b83-49d2-9d4a-2c0e6aff7608 | | image_id | f4b3f773-8594-4d0f-ae06-778c4035bc8e | | metadata | {} | | size | 1 | | snapshot_id | None | | source_volid | None | | status | creating | | volume_type | None | +---------------------+--------------------------------------+ [root@dhcp88-153 ~(keystone_admin)]# cinder list +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+ | 4edaba62-7078-43e9-90fc-7bf7dfd0ebad | available | None | 1 | None | false | | | 9319797e-5b83-49d2-9d4a-2c0e6aff7608 | available | None | 1 | None | true | | | e97111a4-0dc9-4640-aa2c-099da2a7615f | available | None | 1 | None | false | | +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2017-0156.html