Bug 1642811

Summary: Failed to detach Storage Domain hosted_storage_old
Product: [oVirt] ovirt-engine Reporter: Nikolai Sednev <nsednev>
Component: GeneralAssignee: Simone Tiraboschi <stirabos>
Status: CLOSED NOTABUG QA Contact: Elad <ebenahar>
Severity: low Docs Contact:
Priority: unspecified    
Version: 4.2.7CC: bugs, nsednev, tnisan
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-11-05 09:31:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1568841    
Bug Blocks:    
Attachments:
Description Flags
screencast
none
engine logs none

Description Nikolai Sednev 2018-10-25 06:38:08 UTC
Created attachment 1497330 [details]
screencast

Description of problem:
There is an issue with detaching and removing of an old NFS data storage domain, which was also hosted engine storage domain prior to restore of the hosted engine in node0, in case of an old hosted storage was manually wiped out using "rm" command.

Failed to detach Storage Domain hosted_storage_old_20181024T175247 from Data Center Default. (User: admin@internal-authz)
10/25/189:23:45 AM

VDSM command DetachStorageDomainVDS failed: Storage domain does not exist: (u'7d8dec75-ab36-40ee-a612-344dad55c0d8',)
10/25/189:27:28 AM

Version-Release number of selected component (if applicable):
ovirt-hosted-engine-setup-2.2.29-1.el7ev.noarch
ovirt-hosted-engine-ha-2.2.18-1.el7ev.noarch
Linux 3.10.0-957.el7.x86_64 #1 SMP Thu Oct 4 20:48:51 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.6 (Maipo)

How reproducible:
100%

Steps to Reproduce:
1.Deploy node 0 on pair of hosts over NFS and attach one more storage domain.
2.Make backup and restore of the hosted engine on first host.
3.During restore, use different NFS clean mount point and wipe out anything within the old share on old hosted-engine mount point using "rm command", i.e. "rm -rf /mnt/nsednev_he_1/*".
4.After hosted engine restore, try detaching and removing the old hosted-engine data storage domain.

Actual results:
An old inactive hosted engine data storage domain can only be set in to maintenance, but it can't be detached and removed.

Expected results:
Old hosted engine data storage domain have to be removed without errors.

Additional info:
Screencast and the logs attached.

Comment 1 Nikolai Sednev 2018-10-25 06:43:18 UTC
Created attachment 1497343 [details]
engine logs

Comment 2 Nikolai Sednev 2018-10-25 07:38:04 UTC
The only option is to use "destroy" on selected inactive, old, hosted-engine data storage domain.
Lowering severity to low.

Comment 3 Simone Tiraboschi 2018-10-25 07:39:20 UTC
Nikolai, is this HE specific?
Can you please try creating an NFS data storage domain, then emptying its directory with 'rm -rf /mnt/nsednev_he_1/*' as you did and then trying to remove it from the engine?

Comment 4 Nikolai Sednev 2018-10-25 07:48:05 UTC
(In reply to Simone Tiraboschi from comment #3)
> Nikolai, is this HE specific?
> Can you please try creating an NFS data storage domain, then emptying its
> directory with 'rm -rf /mnt/nsednev_he_1/*' as you did and then trying to
> remove it from the engine?

I think its not specific to hosted engine, although it was caused during restore, as any data storage domain can be wiped out and then could not get detached without destroying them.

Moving to ovirt-engine instead of hosted engine.

Comment 5 Tal Nisan 2018-11-05 09:31:06 UTC
This is an expected behavior, there is no storage domain that exists on the NFS share anymore as the metadata is deleted, in that case detach is not possible, use the "Force Remove" option instead.

Comment 6 Nikolai Sednev 2018-11-05 09:57:01 UTC
(In reply to Tal Nisan from comment #5)
> This is an expected behavior, there is no storage domain that exists on the
> NFS share anymore as the metadata is deleted, in that case detach is not
> possible, use the "Force Remove" option instead.

Its not force remove, its destroy and its hidden and not intuitive for customer's reach.