Description of problem: Tempest upstream tests: tempest.api.volume.admin.test_group_snapshots.GroupSnapshotsTest openstack setup with two cinder backedns (Netapp and NFS) group snapshot works with netapp driver but fails with NFS. /var/log/containers/cinder/cinder-volume.log:108:2022-09-12 13:37:38.775 15 ERROR cinder.volume.manager [req-fe7972e1-a30f-42a3-b239-9b4b4036a2bc 3b068f0e972a42a4aea90fec3f18c8d7 d6e274e5b6174776a4e8635e93c5f32f - - -] Error occurred when creating group_snapshot 3cfb73b2-03ea-4c41-ac39-27d9f315fedf. /var/log/containers/cinder/cinder-volume.log:109:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server [req-fe7972e1-a30f-42a3-b239-9b4b4036a2bc 3b068f0e972a42a4aea90fec3f18c8d7 d6e274e5b6174776a4e8635e93c5f32f - - -] Exception during message handling: cinder.exception.VolumeDriverException: Volume driver reported an error: Error occurred when creating group_snapshot 3cfb73b2-03ea-4c41-ac39-27d9f315fedf. /var/log/containers/cinder/cinder-volume.log:110:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server Traceback (most recent call last): /var/log/containers/cinder/cinder-volume.log:111:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming /var/log/containers/cinder/cinder-volume.log:112:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) /var/log/containers/cinder/cinder-volume.log:113:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch /var/log/containers/cinder/cinder-volume.log:114:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) /var/log/containers/cinder/cinder-volume.log:115:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch /var/log/containers/cinder/cinder-volume.log:116:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) /var/log/containers/cinder/cinder-volume.log:117:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/volume/manager.py", line 4074, in create_group_snapshot /var/log/containers/cinder/cinder-volume.log:118:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server snapshot.save() /var/log/containers/cinder/cinder-volume.log:119:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ /var/log/containers/cinder/cinder-volume.log:120:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server self.force_reraise() /var/log/containers/cinder/cinder-volume.log:121:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise /var/log/containers/cinder/cinder-volume.log:122:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server raise self.value /var/log/containers/cinder/cinder-volume.log:123:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/volume/manager.py", line 4059, in create_group_snapshot /var/log/containers/cinder/cinder-volume.log:124:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server raise exception.VolumeDriverException(message=msg) /var/log/containers/cinder/cinder-volume.log:125:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server cinder.exception.VolumeDriverException: Volume driver reported an error: Error occurred when creating group_snapshot 3cfb73b2-03ea-4c41-ac39-27d9f315fedf. /var/log/containers/cinder/cinder-volume.log:126:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server
(In reply to bkopilov from comment #0) > Description of problem: > Tempest upstream tests: > tempest.api.volume.admin.test_group_snapshots.GroupSnapshotsTest > > openstack setup with two cinder backedns (Netapp and NFS) > > group snapshot works with netapp driver but fails with NFS. Just to clarify: the backends are iSCSI/Netapp and Generic NFS, and it fails with Generic NFS.
We need a c-vol log showing the source of the error from req-fe7972e1-a30f. The attached log info only shows where the manager received the failure response but not the details of why/where it was caused.
The only error I was able to find is the following cryptic message in cinder-volume.log: 2022-09-30 13:28:10.126 17 ERROR cinder.volume.manager [req-7a3a6ff8-9c63-4277-b8fb-a46c68fa4b01 f12a87c61aba40629663b263b5be08d1 c3c5dc8645104a99bec89171c55f4fb9 - - -] Error occurred whe$ creating group_snapshot c1a6e37f-05b9-4f17-9fa1-87cf3738a17c. 2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server [req-7a3a6ff8-9c63-4277-b8fb-a46c68fa4b01 f12a87c61aba40629663b263b5be08d1 c3c5dc8645104a99bec89171c55f4fb9 - - -] Exception duri$ g message handling: cinder.exception.VolumeDriverException: Volume driver reported an error: Error occurred when creating group_snapshot c1a6e37f-05b9-4f17-9fa1-87cf3738a17c. 2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/volume/manager.py", line 4074, in create_group_snapshot 2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server snapshot.save() 2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server self.force_reraise() 2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server raise self.value 2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/volume/manager.py", line 4059, in create_group_snapshot 2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server raise exception.VolumeDriverException(message=msg) 2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server cinder.exception.VolumeDriverException: Volume driver reported an error: Error occurred when creating group_snapshot c1a6e37f-05b$-4f17-9fa1-87cf3738a17c. 2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server
One data point -- this test passes on upstream master. See: https://review.opendev.org/c/openstack/devstack-plugin-nfs/+/859844/1/ https://zuul.opendev.org/t/openstack/build/182080caeb6441ef91b65b020f5d081c/logs
Removed the "Regression" flag given that the generic-NFS driver is unsupported, and hence not "really" in the product, and hence can't cause a regression. Not closing because we want to look more closely, but the fact that this works upstream makes me think this may be some kind of configuration problem.