Bug 1213238

Summary: [New] - Snapshot creation succeeds from UI when some of the bricks in the volume are down.
Product: [Retired] oVirt Reporter: RamaKasturi <knarra>
Component: ovirt-engine-webadminAssignee: Shubhendu Tripathi <shtripat>
Status: CLOSED CURRENTRELEASE QA Contact: Stanislav Graf <sgraf>
Severity: high Docs Contact:
Priority: medium    
Version: 3.6CC: bugs, ecohen, gklein, knarra, lsurette, mgoldboi, rbalakri, sabose, sgraf, shtripat, yeylon
Target Milestone: ---   
Target Release: 3.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: gluster
Fixed In Version: ovirt-engine-3.6.0-0.0.master.20150514172356.git1f1cc88 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-04 11:48:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Gluster RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1187461    

Description RamaKasturi 2015-04-20 06:20:24 UTC
Description of problem:
As of now snapshot creation is successful from UI but there are no snapshots listed even when bricks in a volume are down.

Snapshot creation should not be succeeded when the bricks in a volume are down/glusterd is down / node in a cluster is down.

Version-Release number of selected component (if applicable):
ovirt-release-master-001-0.6.master.noarch

How reproducible:
Always

Steps to Reproduce:
1. Create a volume from UI.
2. Now bring down one of the brick in the volume.
3. Try creating snapshot from UI.


Actual results:
Snapshot creation succeeds from UI but does not list any snapshots.

Expected results:
Snapshot creation should not be succeeded from UI and UI should not be passing the force option during snapshot creation.

Additional info:

Comment 1 Shubhendu Tripathi 2015-04-22 07:08:09 UTC
@Kasturi, I tried to bring down one the brick processes from CLI to test out the scenario.
The UI shows brick as DOWN and volume as partially UP, and thus the New Snapshot option is disabled now in UI.
May be the issue happens before the brick status is refreshed in UI and snapshot creation is tried.

To be safer side, I feel a check can be introduced in engine to check the UP state of bricks for the volume, before creation of snapshot.

Comment 2 Stanislav Graf 2015-05-13 12:51:06 UTC
Successfully reproduced today on
ovirt-engine-3.6.0-0.0.master.20150512172421.git310f642.el6.noarch

Steps to reproduce:
===================
1. Create a volume from UI.
2. Login to one node and poweroff the node
(without further waiting to get updates into webui)
3. Create a snapshot using webui (console thinks everything is fine)
4. Event log says '2015-May-13, 14:45 Snapshot test created for volume volume1.'
5. Check 'Snapshots' -> nothing there

Here is my eventlog:
====================
~~~
2015-May-13, 14:46
Host HOST1 is non responsive.
	
2015-May-13, 14:45
Snapshot test created for volume volume1.
	
2015-May-13, 14:45
Host HOST1 is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued.
	
2015-May-13, 14:44
Detected change in status of brick HOST1:/rhs/brick1/brick1 of volume volume1 from UNKNOWN to UP.
	
2015-May-13, 14:39
Status of host HOST1 was set to Up.
~~~

--> ASSIGNED

Comment 3 Shubhendu Tripathi 2015-05-13 13:03:13 UTC
There is another issue with snapshot creation logic which would be fixed by patch https://gerrit.ovirt.org/#/c/40407/.
We will keep this BZ open till above patch is merged and both can be verified together.
Appologies for confusion.

Comment 4 Shubhendu Tripathi 2015-05-18 05:06:11 UTC
The patch https://gerrit.ovirt.org/#/c/40407/ is merged now. If some of the bricks are down for the volume, and still not refreshed in UI, the snapshot creation would fail in gluster side and same would be communicated back to UI.

Comment 5 Stanislav Graf 2015-05-18 16:43:03 UTC
Retested with
ovirt-engine-3.6.0-0.0.master.20150517172245.git089e126.el6.noarch

Now it fails properly with:
~~~
Operation Canceled
Error while executing action Create Gluster Volume Snapshot: Snapshot create failed
error: glusterds are not in quorum
return code: -1
~~~

Later when oVirt knows that the brick is down, it disables creating new snapshots.

--> VERIFIED

Comment 6 Sandro Bonazzola 2015-11-04 11:48:22 UTC
oVirt 3.6.0 has been released on November 4th, 2015 and should fix this issue.
If problems still persist, please open a new BZ and reference this one.