Bug 1463083 - RESTAPI - Attaching a storage domain to 4.0 DC fails due to discard after delete is now True by default but not supported
RESTAPI - Attaching a storage domain to 4.0 DC fails due to discard after del...
Status: VERIFIED
Product: ovirt-engine
Classification: oVirt
Component: BLL.Storage (Show other bugs)
4.2.0
Unspecified Unspecified
unspecified Severity high (vote)
: ovirt-4.2.0
: ---
Assigned To: Idan Shaby
Avihai
: Automation, AutomationBlocker
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-20 02:06 EDT by Avihai
Modified: 2017-07-11 03:40 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Release Note
Doc Text:
When adding a new data storage domain without stating the values of Discard After Delete (DAD) and storage format, their default values used to be calculated according to the data center version of the host that added the storage domain: DAD = true for dc >= 4.1, otherwise false. Storage format = V4 for dc >= 4.1, otherwise V3. This was a bad heuristic since the dc of the host that added the domain is a random dc and is not necessarily the dc that the domain will be later attached to. Therefore, the logic was changed to be as follows: - The default storage format for new data domains will be the latest format (now it is V4). For non data domains, nothing has changed here - V1 was and will remain the default value. - The default value of DAD will be calculated according to the storage format: true for >= V4 and otherwise false.
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
rule-engine: ovirt‑4.2+
amureini: devel_ack+


Attachments (Terms of Use)
engine & vdsm logs (892.81 KB, application/x-gzip)
2017-06-20 02:06 EDT, Avihai
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 78698 master MERGED backend: DAD's default value criteria 2017-07-06 03:16 EDT
oVirt gerrit 79042 None None None 2017-07-11 03:18 EDT

  None (edit)
Description Avihai 2017-06-20 02:06:27 EDT
Created attachment 1289400 [details]
engine & vdsm logs

Description of problem:
Attaching V3 storage domain to 4.0 DC due to discard after delete is now True by default ( was not so a 1-2 weeks ago - I tested on Jun9 build and this did not occur)

Severity is high as this blocks all of our upgrade testing requiring SD's to be attached to a 4.0 DC. 

Version-Release number of selected component (if applicable):
oVirt Engine Version: 4.2.0-0.0.master.20170618184152.gitb0f84a1.el7.centos
VDSM: 4.20.0-914

How reproducible:
100%

Steps to Reproduce:
1.Create DC & cluster in v4.0 
2.VIA REST Create SD ( I used ISCSI but it does not matter) - no issues.
3.VIA REST Attach SD 

Actual results:
Attaching SD fails with following response:
2017-06-20 07:09:22,016 - MainThread - api_utils - ERROR - Failed to create element NOT as expected:
	Status: 400
	Reason: Bad Request
	Detail: [Cannot attach Storage. Discard after delete is not supported by data center version 4.0.]



Expected results:


Additional info:
REST API that was done:

1) Create SD succeeds :

2017-06-20 07:09:07,878 - MainThread - storagedomains - DEBUG - CREATE request content is --  url:https://storage-ge-04.scl.lab.tlv.redhat.com/ovirt-engine/api/storagedomains

 body:

<storage_domain>
    <name>upgrade_4_0_to_4_1iscsi0</name>
    <storage>
        <logical_units>
            <logical_unit id="3514f0c5a51600268">
                <address>10.35.146.129</address>
                <port>3260</port>
                <target>iqn.2008-05.com.xtremio:xio00153500071-514f0c50023f6c00</target>
            </logical_unit>
        </logical_units>
        <override_luns>true</override_luns>
        <type>iscsi</type>
    </storage>
    <storage_format>v3</storage_format>
    <type>data</type>
    <host>
        <name>host_mixed_1</name>
    </host>
</storage_domain>


2017-06-20 07:09:07,879 - MainThread - storagedomains - INFO - Using Correlation-Id: storagedomains_create_b1309abe-e699-4e38
2017-06-20 07:09:20,864 - MainThread - core_api - DEBUG - Request POST response time: 0.037
2017-06-20 07:09:20,865 - MainThread - storagedomains - DEBUG - Cleaning Correlation-Id: storagedomains_create_b1309abe-e699-4e38
2017-06-20 07:09:20,866 - MainThread - storagedomains - DEBUG - Response code is valid: [200, 201, 202] 
2017-06-20 07:09:20,866 - MainThread - storagedomains - DEBUG - GET request content is --  url:https://storage-ge-04.scl.lab.tlv.redhat.com/ovirt-engine/api/storagedomains 

2) Attaching storage fails with following message : " Discard after delete is not supported by data center version 4.0 "

2017-06-20 07:09:21,780 - MainThread - storagedomains - DEBUG - CREATE request content is --  url:/ovirt-engine/api/datacenters/d47c5c14-9e4e-47ed-a55b-0ba6f3007abb/storagedomains

 body:
<storage_domain id="bde102af-38c3-4c13-bcd9-ec4b886405eb"/>
 
2017-06-20 07:09:21,781 - MainThread - storagedomains - INFO - Using Correlation-Id: storagedomains_create_b4ee9c8f-3311-465c
2017-06-20 07:09:22,015 - MainThread - core_api - DEBUG - Request POST response time: 0.043
2017-06-20 07:09:22,015 - MainThread - storagedomains - DEBUG - Cleaning Correlation-Id: storagedomains_create_b4ee9c8f-3311-465c
2017-06-20 07:09:22,016 - MainThread - api_utils - ERROR - Failed to create element NOT as expected:
	Status: 400
	Reason: Bad Request
	Detail: [Cannot attach Storage. Discard after delete is not supported by data center version 4.0.]

Engine log:
2017-06-20 08:57:58,088+03 INFO  [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-5) [0efc14db-9735-4afa-9361-b2338c3af1db] Lock Acquired to object 'EngineLock:{exclusiveLocks='[bde102af-38c3-4c13-
bcd9-ec4b886405eb=STORAGE]', sharedLocks=''}'
2017-06-20 08:57:58,093+03 WARN  [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-5) [0efc14db-9735-4afa-9361-b2338c3af1db] Validation of action 'AttachStorageDomainToPool' failed for user admin@internal-authz. Reasons: VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ATTACH,ACTION_TYPE_FAILED_STORAGE_DOMAIN_FORMAT_ILLEGAL,$storageFormat V4
2017-06-20 08:57:58,094+03 INFO  [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-5) [0efc14db-9735-4afa-9361-b2338c3af1db] Lock freed to object 'EngineLock:{exclusiveLocks='[bde102af-38c3-4c13-bcd9-ec4b886405eb=STORAGE]', sharedLocks=''}'
Comment 1 Avihai 2017-06-20 02:16:54 EDT
No issues with attaching SD to 4.1/4.2 DC , only with 4.0 this issue occurs.

Maybe a solution can be to leave discard after delete flag as False by default so we can still work with 4.0 DC's .

Why was this changed to True in the past 2 weeks?
Comment 2 Allon Mureinik 2017-06-20 04:10:24 EDT
This is the result of bug 1456414.

IMHO, if this is done in two steps - create succeeds, and then the attach fails, it's a user error (i.e., NOTABUG), provided that bug 1456414 is documented properly in the release notes.

Yaniv - your two cents?
Comment 3 Allon Mureinik 2017-06-21 01:57:22 EDT
(In reply to Allon Mureinik from comment #2)
> This is the result of bug 1456414.
"Of the fix to bug 1456414", of course - i.e., this behavior is intentional.
This is not a bug in the sense of "the system behaves in a way we didn't inted it to", but may still be a bug in the sense of "the intended behavior is a bad idea" - pending PM's input on this.
Comment 4 Avihai 2017-06-21 03:35:57 EDT
Yaniv , can you please update what is the decision here?

This blocks multiple automation TC's(in upgrade&qcow2_v3 TP's) runs on 4.2 that currently create 4.0 SD's without stating DAD to be disabled & thus fails. 

So if this is not a bug we(QA automation) need to update the create storage domain function & add a configurable DAD boolean optional variable & change the current TC's accordingly.
Comment 5 Yaniv Lavi (Dary) 2017-06-25 10:08:11 EDT
The issue here is that the REST API call was to create v3 SD that should not have a different default than before. 
I think we should make sure when we change a default for v4 SD it doesn't affect older SD versions.
Comment 6 Allon Mureinik 2017-06-25 15:04:10 EDT
Yaniv, Avihai - thanks for clarifying, I see the point now.

I debugged through the code and here's what I think is going on:
- If you supply a DAD version, the engine attempts to use it - this makes sense to me.
- If you don't supply a DAD value, but do supply a DC, the engine calculates the default DAD version from the DC level - still makes sense.
- If you don't supply a DAD or a DC, the DC is implicitly taken for the host (kinda-sorta makes sense), and the DAD value is calculated from that.

What happens here, if I'm following the case correctly, is an attempt to create a detached SD using a host from a 4.1 DC (which causes DAD=true), and then attempting to attach it to a 4.0 DC, which then fails, as this is not supported.

I don't think this is a particularly common case, but it's a bug nonetheless, and I agree with Yaniv's comment 5 - if no DC nor DAD is given, we should attempt to take the DAD value from the storage format, which kinda-sorta implies a potential DC level.

Idan - can you please handle this?
Comment 7 Avihai 2017-07-10 05:02:54 EDT
verified at 4.2.0-0.0.master.20170707124946.gitf15a6d9.el7.centos

Note You need to log in before you can comment on or make changes to this bug.