Bug 1889774

Summary: Veritas Netbackup leave nova db dirty after back-up performed on rhosp13 (API used)
Product: Red Hat OpenStack Reporter: Luigi Tamagnone <ltamagno>
Component: openstack-cinderAssignee: Cinder Bugs List <cinder-bugs>
Status: CLOSED NOTABUG QA Contact: Tzach Shefi <tshefi>
Severity: high Docs Contact: RHOS Documentation Team <rhos-docs>
Priority: high    
Version: 13.0 (Queens)CC: abishop, gkadam, jvisser, rdhasman, senrique
Target Milestone: ---Keywords: OtherQA, Triaged, ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-09-22 09:32:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Luigi Tamagnone 2020-10-20 14:30:44 UTC 
Description of problem:
It's impossible to use Veritas Netbackup to create a back-up. The first attempt finished successful but leave the nova DB dirty and next attempt fail every time

Version-Release number of selected component (if applicable):
[redhat-release] Red Hat Enterprise Linux Server release 7.7 (Maipo)
[rhosp-release] Red Hat OpenStack Platform release 13.0.10 (Queens)


How reproducible:
Almost every time a back-up from veritas product is launched

Steps to Reproduce:
Launch Veritas back-up. The steps for a successful backup are as follows:
1- snapshot creation
2- volume creation
3- attempt to delete snapshot (in this case fail)
4- attach volume to backup host 
5- agent host run back-up successfully
6- volume detach
7- volume deletion
8- deletion of the snapshot

Actual results:
The first back-up ended successfully but leave nova DB dirty and another back-up fail

Expected results:
All back-up ended successfully

Additional info:
- On case cinder-volume.log and cinder-api.log and backup-host.logs (from Veritas host)
- SOSreport from controllers and compute 03 

Instance where the volumes are attached for doing the Veritas Back-up:
openstack server show 67c0eafb-3a68-43df-9fb4-847233e196bd
| OS-EXT-SRV-ATTR:host                | xxx-cmpc03.xxx                                      |
| flavor                              | Bastion_ webgate_coll (319290c2-7b48-4d45-ae4c-07ba70cf5706)             |
| name                                | xxx-bckcf1.xxx                                                           |
| volumes_attached                    | id='3491404a-c9f3-438b-9be6-7e56436884f4'                                |
|                                     | id='8bd758a3-ab3c-481d-8b55-cd94268a810f'                                |
|                                     | id='3740a4b2-3c2d-4c36-acef-c6015f269b6f'                                |
|                                     | id='d40d5fac-98ce-4d96-b834-985f6095b14b'  

first backup attempt created volume (Oct 15 2020 around 3 p.m.):
openstack volume show 8bd758a3-ab3c-481d-8b55-cd94268a810f 
not present

second backup attempt created volume:
openstack volume show 3740a4b2-3c2d-4c36-acef-c6015f269b6f 
| created_at                     | 2020-10-15T13:01:10.000000  
| description                    | Temp volume created from snapshot: 173e4289-419b-445c-8be7-968b46fab601

thirt backup attempt created volume:
openstack volume show d40d5fac-98ce-4d96-b834-985f6095b14b
| created_at                     | 2020-10-15T13:41:29.000000 
| description                    | Temp volume created from snapshot: 277225fc-47a7-4b5f-b8b7-7e8541ce476b