Bug 845310 - ovirt-engine-backend: StoragePool storage_pool_format_type is lost in update
ovirt-engine-backend: StoragePool storage_pool_format_type is lost in update
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine (Show other bugs)
3.1.0
x86_64 Linux
urgent Severity urgent
: ---
: 3.1.0
Assigned To: Asaf Shakarchi
Dafna Ron
storage
: Regression, TestBlocker
: 842149 846692 848401 (view as bug list)
Depends On:
Blocks: 851003
  Show dependency treegraph
 
Reported: 2012-08-02 11:45 EDT by Dafna Ron
Modified: 2016-02-10 11:35 EST (History)
15 users (show)

See Also:
Fixed In Version: SI15
Doc Type: Release Note
Doc Text:
Previously, it was not possible to change the properties of a data center after its initial creation. Doing this would result in the associated storage becoming unusable after the first Storage Pool Manager (SPM) reelection following the change being applied occurred. This issue has now been resolved, data center properties may now be edited safely.
Story Points: ---
Clone Of:
: 851003 (view as bug list)
Environment:
Last Closed: 2012-12-04 15:04:31 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
logs (363.78 KB, application/x-gzip)
2012-08-02 11:45 EDT, Dafna Ron
no flags Details
logs for vdsm restart after quota change (469.11 KB, application/x-gzip)
2012-08-12 08:16 EDT, Dafna Ron
no flags Details

  None (edit)
Description Dafna Ron 2012-08-02 11:45:37 EDT
Created attachment 601989 [details]
logs

Description of problem:

I updated my pool which caused the storagePoolFormatType to change to null. 
as part of the DestroyPoolCommand backend sends SpmStarts. 
this fails with the below: 

2012-08-02 17:40:09,446 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-4-thread-48) [5507c9fd] START, SpmStartVDSCommand(vdsId = 96cd1716-db0a-11e1-af67-001a4a169741, storagePoolId = 88577c5f-67b7-4728-aa5f-b
b3d8da58ac9, prevId=-1, prevLVER=0, storagePoolFormatType=null, recoveryMode=Manual, SCSIFencing=false), log id: 3539028a
2012-08-02 17:40:09,446 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (pool-4-thread-48) [5507c9fd] Failed in SpmStartVDS method, for vds: blond-vdsh; host: blond-vdsh.qa.lab.tlv.redhat.com
2012-08-02 17:40:09,446 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-4-thread-48) [5507c9fd] Command SpmStartVDS execution failed. Exception: NullPointerException: 

as a result the storage domain is stuck on image locked forever. 

Version-Release number of selected component (if applicable):

si12

How reproducible:

100%

Steps to Reproduce:
1. edit Data Center (make sure that the storagePoolFormatType value in db changes to null). 
2. remove Data Center (remove master domain)
3.
  
Actual results:

we cannot remove master domain
we have NPE in the log and the domain is stuck in image locked forever


Expected results:

1. we should not clear the storagePoolFormatType when editing DC
2. we should be able to rollback if value is null. 


Additional info: backend and vdsm logs
Comment 1 Dafna Ron 2012-08-02 11:48:14 EDT
I was not even able to force remove the DC since its "Active" 

2012-08-02 18:43:05,375 WARN  [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (ajp-/127.0.0.1:8009-3) [e61c995] CanDoAction of action RemoveStoragePool failed. Reasons:VAR__TYPE__STORAGE__POOL,VAR__ACTION__REMOVE,ERROR_CANNOT_REMOVE_POOL_WITH_ACTIVE_DOMAINS
Comment 2 Dafna Ron 2012-08-12 08:06:42 EDT
moving to urgent. 
every UpdateStoragePoolCommand will cause version to ve deleted which means that if I edit DC name or change between quota enforce/disabled will cause the next SpmStartVDSCommand to fail with NPE. 
example: 

I changed from quota enabled to enforce -> back to disabled -> restarted vdsm 
results: 

we cannot recover from vdsm restart. 

a second example is editing DC name: 

https://bugzilla.redhat.com/show_bug.cgi?id=846692

I'm attaching logs from my vdsm restart scenario.
Comment 3 Dafna Ron 2012-08-12 08:15:13 EDT
*** Bug 846692 has been marked as a duplicate of this bug. ***
Comment 4 Dafna Ron 2012-08-12 08:16:21 EDT
Created attachment 603773 [details]
logs for vdsm restart after quota change
Comment 6 Haim 2012-08-12 12:53:46 EDT
*** Bug 842149 has been marked as a duplicate of this bug. ***
Comment 7 Stephen Gordon 2012-08-14 12:32:42 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
During this beta you must not change the properties of a data center after its initial creation. Doing this may result in unusable storage after the first Storage Pool Manager (SPM) reelection event that occurs following the change being applied.
Comment 8 Stephen Gordon 2012-08-14 12:55:45 EDT
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-During this beta you must not change the properties of a data center after its initial creation. Doing this may result in unusable storage after the first Storage Pool Manager (SPM) reelection event that occurs following the change being applied.+During this beta you must not change the properties of a data center after its initial creation. Doing this may result in the associated storage becoming unusable after the first Storage Pool Manager (SPM) reelection that occurs following the change being applied.
Comment 9 Haim 2012-08-15 09:42:45 EDT
*** Bug 848401 has been marked as a duplicate of this bug. ***
Comment 10 Asaf Shakarchi 2012-08-16 10:40:47 EDT
(In reply to comment #0)
> Created attachment 601989 [details]
> logs
> 
> Description of problem:
> 
> I updated my pool which caused the storagePoolFormatType to change to null. 
> as part of the DestroyPoolCommand backend sends SpmStarts. 
> this fails with the below: 
> 
> 2012-08-02 17:40:09,446 INFO 
> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
> (pool-4-thread-48) [5507c9fd] START, SpmStartVDSCommand(vdsId =
> 96cd1716-db0a-11e1-af67-001a4a169741, storagePoolId =
> 88577c5f-67b7-4728-aa5f-b
> b3d8da58ac9, prevId=-1, prevLVER=0, storagePoolFormatType=null,
> recoveryMode=Manual, SCSIFencing=false), log id: 3539028a
> 2012-08-02 17:40:09,446 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
> (pool-4-thread-48) [5507c9fd] Failed in SpmStartVDS method, for vds:
> blond-vdsh; host: blond-vdsh.qa.lab.tlv.redhat.com
> 2012-08-02 17:40:09,446 ERROR
> [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-4-thread-48)
> [5507c9fd] Command SpmStartVDS execution failed. Exception:
> NullPointerException: 
> 
> as a result the storage domain is stuck on image locked forever. 
> 
> Version-Release number of selected component (if applicable):
> 
> si12
> 
> How reproducible:
> 
> 100%
> 
> Steps to Reproduce:
> 1. edit Data Center (make sure that the storagePoolFormatType value in db
> changes to null). 
> 2. remove Data Center (remove master domain)
> 3.
>   
> Actual results:
> 
> we cannot remove master domain
> we have NPE in the log and the domain is stuck in image locked forever
> 
> 
> Expected results:
> 
> 1. we should not clear the storagePoolFormatType when editing DC
This is fixed.
> 2. we should be able to rollback if value is null. 
This field should never be null thus this situation should no longer happen.
In general we don't perform validations to situations that not supposed to happen.
> 
> 
> Additional info: backend and vdsm logs

Changed title to describe the exact problem,
Patch submitted: http://gerrit.ovirt.org/#/c/7265/
Comment 13 Ayal Baron 2012-08-21 17:56:33 EDT
Asaf, we did not consider upgrade (edit did the same in previous versions so we could have storage_pools which are already messed up.
Comment 14 Ayal Baron 2012-08-23 01:07:31 EDT
Cloned to track the upgrade fix.
Moving to ON_QA
Comment 15 Haim 2012-08-23 07:14:29 EDT
verified on SI15. changed pool name while host was down, storage_pool_format_type was not lost and set to null, this scenario was a reproducer for the above, now it works.
Comment 16 Stephen Gordon 2012-10-09 15:51:34 EDT
Removing the release note flags as this was rectified.

Note You need to log in before you can comment on or make changes to this bug.