Bug 2126136

Summary: [RHOS17] NFS Error occurred when creating group_snapshot
Product: Red Hat OpenStack Reporter: bkopilov <bkopilov>
Component: openstack-cinderAssignee: Eric Harney <eharney>
Status: NEW --- QA Contact: Evelina Shames <eshames>
Severity: high Docs Contact: Andy Stillman <astillma>
Priority: unspecified    
Version: 17.0 (Wallaby)CC: brian.rosmaita, eharney, eshames, ltoscano, pgrist
Target Milestone: ---Keywords: Triaged
Target Release: ---Flags: eharney: needinfo? (bkopilov)
lsvaty: needinfo? (eshames)
lsvaty: needinfo? (eharney)
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description bkopilov 2022-09-12 14:19:34 UTC
Description of problem:
Tempest upstream tests: tempest.api.volume.admin.test_group_snapshots.GroupSnapshotsTest

openstack setup with two cinder backedns (Netapp and NFS)

group snapshot works with netapp driver but fails with NFS.


/var/log/containers/cinder/cinder-volume.log:108:2022-09-12 13:37:38.775 15 ERROR cinder.volume.manager [req-fe7972e1-a30f-42a3-b239-9b4b4036a2bc 3b068f0e972a42a4aea90fec3f18c8d7 d6e274e5b6174776a4e8635e93c5f32f - - -] Error occurred when creating group_snapshot 3cfb73b2-03ea-4c41-ac39-27d9f315fedf.
/var/log/containers/cinder/cinder-volume.log:109:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server [req-fe7972e1-a30f-42a3-b239-9b4b4036a2bc 3b068f0e972a42a4aea90fec3f18c8d7 d6e274e5b6174776a4e8635e93c5f32f - - -] Exception during message handling: cinder.exception.VolumeDriverException: Volume driver reported an error: Error occurred when creating group_snapshot 3cfb73b2-03ea-4c41-ac39-27d9f315fedf.
/var/log/containers/cinder/cinder-volume.log:110:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
/var/log/containers/cinder/cinder-volume.log:111:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
/var/log/containers/cinder/cinder-volume.log:112:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
/var/log/containers/cinder/cinder-volume.log:113:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
/var/log/containers/cinder/cinder-volume.log:114:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
/var/log/containers/cinder/cinder-volume.log:115:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
/var/log/containers/cinder/cinder-volume.log:116:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
/var/log/containers/cinder/cinder-volume.log:117:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinder/volume/manager.py", line 4074, in create_group_snapshot
/var/log/containers/cinder/cinder-volume.log:118:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server     snapshot.save()
/var/log/containers/cinder/cinder-volume.log:119:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
/var/log/containers/cinder/cinder-volume.log:120:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server     self.force_reraise()
/var/log/containers/cinder/cinder-volume.log:121:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
/var/log/containers/cinder/cinder-volume.log:122:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server     raise self.value
/var/log/containers/cinder/cinder-volume.log:123:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinder/volume/manager.py", line 4059, in create_group_snapshot
/var/log/containers/cinder/cinder-volume.log:124:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server     raise exception.VolumeDriverException(message=msg)
/var/log/containers/cinder/cinder-volume.log:125:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server cinder.exception.VolumeDriverException: Volume driver reported an error: Error occurred when creating group_snapshot 3cfb73b2-03ea-4c41-ac39-27d9f315fedf.
/var/log/containers/cinder/cinder-volume.log:126:2022-09-12 13:37:38.790 15 ERROR oslo_messaging.rpc.server

Comment 2 Luigi Toscano 2022-09-12 14:28:15 UTC
(In reply to bkopilov from comment #0)
> Description of problem:
> Tempest upstream tests:
> tempest.api.volume.admin.test_group_snapshots.GroupSnapshotsTest
> 
> openstack setup with two cinder backedns (Netapp and NFS)
> 
> group snapshot works with netapp driver but fails with NFS.

Just to clarify: the backends are iSCSI/Netapp and Generic NFS, and it fails with Generic NFS.

Comment 3 Eric Harney 2022-09-22 15:05:53 UTC
We need a c-vol log showing the source of the error from req-fe7972e1-a30f.

The attached log info only shows where the manager received the failure response but not the details of why/where it was caused.

Comment 5 Luigi Toscano 2022-09-30 13:49:00 UTC
The only error I was able to find is the following cryptic message in cinder-volume.log:

2022-09-30 13:28:10.126 17 ERROR cinder.volume.manager [req-7a3a6ff8-9c63-4277-b8fb-a46c68fa4b01 f12a87c61aba40629663b263b5be08d1 c3c5dc8645104a99bec89171c55f4fb9 - - -] Error occurred whe$
 creating group_snapshot c1a6e37f-05b9-4f17-9fa1-87cf3738a17c.                                                                                                                               
2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server [req-7a3a6ff8-9c63-4277-b8fb-a46c68fa4b01 f12a87c61aba40629663b263b5be08d1 c3c5dc8645104a99bec89171c55f4fb9 - - -] Exception duri$
g message handling: cinder.exception.VolumeDriverException: Volume driver reported an error: Error occurred when creating group_snapshot c1a6e37f-05b9-4f17-9fa1-87cf3738a17c.               
2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server Traceback (most recent call last):                                                                                                
2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming                            
2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)                                                                                       
2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch                                 
2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)                                                                        
2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch                             
2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)                                                                                               
2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinder/volume/manager.py", line 4074, in create_group_snapshot                           2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server     snapshot.save()                                                                                                               
2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__                                           
2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server     self.force_reraise()                                                                                                          
2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise                                      
2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server     raise self.value                                                                                                              
2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinder/volume/manager.py", line 4059, in create_group_snapshot                           2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server     raise exception.VolumeDriverException(message=msg)                                                                            
2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server cinder.exception.VolumeDriverException: Volume driver reported an error: Error occurred when creating group_snapshot c1a6e37f-05b$-4f17-9fa1-87cf3738a17c.                                                                                                                                                                     
2022-09-30 13:28:10.139 17 ERROR oslo_messaging.rpc.server

Comment 10 Brian Rosmaita 2023-07-11 17:12:51 UTC
Removed the "Regression" flag given that the generic-NFS driver is unsupported, and hence not "really" in the product, and hence can't cause a regression.  Not closing because we want to look more closely, but the fact that this works upstream makes me think this may be some kind of configuration problem.