Bug 823283 - Gluster - Backend: Can't stop volume
Gluster - Backend: Can't stop volume
Status: CLOSED WORKSFORME
Product: oVirt
Classification: Community
Component: ovirt-engine-core (Show other bugs)
unspecified
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Kaushal
Haim
gluster
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-05-20 11:02 EDT by Daniel Paikov
Modified: 2014-12-29 12:09 EST (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-06-10 04:06:37 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Daniel Paikov 2012-05-20 11:02:28 EDT
Stop of volume always fails with the following exception:



2012-05-20 17:54:12,370 WARN  [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (pool-5-thread-49) The message key StopGlusterVolume is missing from bundles/ExecutionMessages
2012-05-20 17:54:12,480 INFO  [org.ovirt.engine.core.bll.gluster.StopGlusterVolumeCommand] (pool-5-thread-49) [70e2511d] Running command: StopGlusterVolumeCommand internal: false. Entities affected :  ID: 72e810f4-5a7f-4f1f-b52b-ced416b30732 Type: GlusterVolume
2012-05-20 17:54:12,484 INFO  [org.ovirt.engine.core.vdsbroker.gluster.StopGlusterVolumeVDSCommand] (pool-5-thread-49) [70e2511d] START, StopGlusterVolumeVDSCommand(vdsId = c0513964-a27a-11e1-a47d-001a4a1697b0), log id: 24f0e9b2
2012-05-20 17:54:12,586 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-49) [70e2511d] Failed in StopGlusterVolumeVDS method
2012-05-20 17:54:12,587 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-49) [70e2511d] Error code unexpected and error message VDSGenericException: VDSErrorException: Failed to StopGlusterVolumeVDS, error = Unexpected exception
2012-05-20 17:54:12,587 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-49) [70e2511d] Command org.ovirt.engine.core.vdsbroker.gluster.StopGlusterVolumeVDSCommand return value 
 Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc
mStatus                       Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc
mCode                         16
mMessage                      Unexpected exception


2012-05-20 17:54:12,587 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-49) [70e2511d] Vds: node1
2012-05-20 17:54:12,588 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-49) [70e2511d] Command StopGlusterVolumeVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to StopGlusterVolumeVDS, error = Unexpected exception
2012-05-20 17:54:12,588 INFO  [org.ovirt.engine.core.vdsbroker.gluster.StopGlusterVolumeVDSCommand] (pool-5-thread-49) [70e2511d] FINISH, StopGlusterVolumeVDSCommand, log id: 24f0e9b2
2012-05-20 17:54:12,588 ERROR [org.ovirt.engine.core.bll.gluster.StopGlusterVolumeCommand] (pool-5-thread-49) [70e2511d] Command org.ovirt.engine.core.bll.gluster.StopGlusterVolumeCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to StopGlusterVolumeVDS, error = Unexpected exception

On the host side, the request looks like this:
2012-05-20 17:47:51.460786] I [glusterd-handler.c:495:glusterd_handle_cluster_lock] 0-glusterd: Received LOCK from uuid: 6a206019-f8
b9-4be9-9644-81d578d83fe6
[2012-05-20 17:47:51.460846] I [glusterd-utils.c:282:glusterd_lock] 0-glusterd: Cluster lock held by 6a206019-f8b9-4be9-9644-81d578d8
3fe6
[2012-05-20 17:47:51.460927] I [glusterd-handler.c:1313:glusterd_op_lock_send_resp] 0-glusterd: Responded, ret: 0
[2012-05-20 17:47:51.461588] I [glusterd-handler.c:1357:glusterd_handle_cluster_unlock] 0-glusterd: Received UNLOCK from uuid: 6a2060
19-f8b9-4be9-9644-81d578d83fe6
[2012-05-20 17:47:51.461664] I [glusterd-handler.c:1333:glusterd_op_unlock_send_resp] 0-glusterd: Responded to unlock, ret: 0
Comment 1 Yaniv Kaul 2012-05-20 11:31:05 EDT
Please attach complete engine log.
Nothing on VDSM side?
Comment 2 Shireesh 2012-05-21 05:18:36 EDT
I suspect the Gluster CLI command for stopping the volume is failing on the host, and the vdsm log file will provide vital clues. Apart from the vdsm log file, can you please execute the gluster cli manually and see whether it throws any error? The command is:

gluster volume stop <vol_name>
Comment 3 Daniel Paikov 2012-05-21 06:04:50 EDT
Stop fails because volume doesn't exist. But it does appear in volume info.

[root@localhost ~]# gluster volume stop vol1
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
Volume vol1 does not exist


[root@localhost ~]# gluster volume info vol1
 
Volume Name: vol1
Type: Distribute
Volume ID: 72e810f4-5a7f-4f1f-b52b-ced416b30732
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 10.35.97.158:/fasdfdas
Brick2: 10.35.97.159:/cvcvc
Options Reconfigured:
auth.allow: *
nfs.disable: off
Comment 4 Shireesh 2012-05-21 07:30:13 EDT
Sounds like a bug in GlusterFS. Adding Vijay in CC.
Comment 5 Vijay Bellur 2012-05-21 07:56:29 EDT
(In reply to comment #4)
> Sounds like a bug in GlusterFS. Adding Vijay in CC.

Can you please provide details of the peers in the cluster and ensure that gluster volume info lists vol1 on all peers in the cluster?

If it does, please provide glusterd logs from all peers.

Thanks,
Vijay
Comment 6 Shireesh 2012-05-22 08:17:07 EDT
Waiting for response to Vijay's comments.
Comment 7 Daniel Paikov 2012-05-23 04:48:01 EDT
(In reply to comment #5)
> (In reply to comment #4)
> > Sounds like a bug in GlusterFS. Adding Vijay in CC.
> 
> Can you please provide details of the peers in the cluster and ensure that
> gluster volume info lists vol1 on all peers in the cluster?
> 
> If it does, please provide glusterd logs from all peers.
> 
> Thanks,
> Vijay

Yes, the problem must be that only 2 out of 3 hosts in the cluster can see this volume. It's possible that I added the 3rd host after the volume was already created. Do we support this flow?
Comment 8 Kaushal 2012-05-25 03:43:48 EDT
A newly added peer will be synced with the volumes already created. If everything goes well, then there shouldn't be any problem. In this case looks like the sync wasn't successful.
Can you provide the glusterd logs for all the 3 peers, so that we can take a look?
Comment 9 Shireesh 2012-06-02 10:36:55 EDT
Please provide details asked by Kaushal. In any case it looks like a combination of glusterfs environment related issue and 

https://bugzilla.redhat.com/823565 (which is fixed)
Comment 10 Daniel Paikov 2012-06-10 04:06:37 EDT
Haven't been able to reproduce in recent builds. Closing the bug.

Note You need to log in before you can comment on or make changes to this bug.