Stop of volume always fails with the following exception: 2012-05-20 17:54:12,370 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (pool-5-thread-49) The message key StopGlusterVolume is missing from bundles/ExecutionMessages 2012-05-20 17:54:12,480 INFO [org.ovirt.engine.core.bll.gluster.StopGlusterVolumeCommand] (pool-5-thread-49) [70e2511d] Running command: StopGlusterVolumeCommand internal: false. Entities affected : ID: 72e810f4-5a7f-4f1f-b52b-ced416b30732 Type: GlusterVolume 2012-05-20 17:54:12,484 INFO [org.ovirt.engine.core.vdsbroker.gluster.StopGlusterVolumeVDSCommand] (pool-5-thread-49) [70e2511d] START, StopGlusterVolumeVDSCommand(vdsId = c0513964-a27a-11e1-a47d-001a4a1697b0), log id: 24f0e9b2 2012-05-20 17:54:12,586 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-49) [70e2511d] Failed in StopGlusterVolumeVDS method 2012-05-20 17:54:12,587 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-49) [70e2511d] Error code unexpected and error message VDSGenericException: VDSErrorException: Failed to StopGlusterVolumeVDS, error = Unexpected exception 2012-05-20 17:54:12,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-49) [70e2511d] Command org.ovirt.engine.core.vdsbroker.gluster.StopGlusterVolumeVDSCommand return value Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc mCode 16 mMessage Unexpected exception 2012-05-20 17:54:12,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-49) [70e2511d] Vds: node1 2012-05-20 17:54:12,588 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-49) [70e2511d] Command StopGlusterVolumeVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to StopGlusterVolumeVDS, error = Unexpected exception 2012-05-20 17:54:12,588 INFO [org.ovirt.engine.core.vdsbroker.gluster.StopGlusterVolumeVDSCommand] (pool-5-thread-49) [70e2511d] FINISH, StopGlusterVolumeVDSCommand, log id: 24f0e9b2 2012-05-20 17:54:12,588 ERROR [org.ovirt.engine.core.bll.gluster.StopGlusterVolumeCommand] (pool-5-thread-49) [70e2511d] Command org.ovirt.engine.core.bll.gluster.StopGlusterVolumeCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to StopGlusterVolumeVDS, error = Unexpected exception On the host side, the request looks like this: 2012-05-20 17:47:51.460786] I [glusterd-handler.c:495:glusterd_handle_cluster_lock] 0-glusterd: Received LOCK from uuid: 6a206019-f8 b9-4be9-9644-81d578d83fe6 [2012-05-20 17:47:51.460846] I [glusterd-utils.c:282:glusterd_lock] 0-glusterd: Cluster lock held by 6a206019-f8b9-4be9-9644-81d578d8 3fe6 [2012-05-20 17:47:51.460927] I [glusterd-handler.c:1313:glusterd_op_lock_send_resp] 0-glusterd: Responded, ret: 0 [2012-05-20 17:47:51.461588] I [glusterd-handler.c:1357:glusterd_handle_cluster_unlock] 0-glusterd: Received UNLOCK from uuid: 6a2060 19-f8b9-4be9-9644-81d578d83fe6 [2012-05-20 17:47:51.461664] I [glusterd-handler.c:1333:glusterd_op_unlock_send_resp] 0-glusterd: Responded to unlock, ret: 0
Please attach complete engine log. Nothing on VDSM side?
I suspect the Gluster CLI command for stopping the volume is failing on the host, and the vdsm log file will provide vital clues. Apart from the vdsm log file, can you please execute the gluster cli manually and see whether it throws any error? The command is: gluster volume stop <vol_name>
Stop fails because volume doesn't exist. But it does appear in volume info. [root@localhost ~]# gluster volume stop vol1 Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y Volume vol1 does not exist [root@localhost ~]# gluster volume info vol1 Volume Name: vol1 Type: Distribute Volume ID: 72e810f4-5a7f-4f1f-b52b-ced416b30732 Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: 10.35.97.158:/fasdfdas Brick2: 10.35.97.159:/cvcvc Options Reconfigured: auth.allow: * nfs.disable: off
Sounds like a bug in GlusterFS. Adding Vijay in CC.
(In reply to comment #4) > Sounds like a bug in GlusterFS. Adding Vijay in CC. Can you please provide details of the peers in the cluster and ensure that gluster volume info lists vol1 on all peers in the cluster? If it does, please provide glusterd logs from all peers. Thanks, Vijay
Waiting for response to Vijay's comments.
(In reply to comment #5) > (In reply to comment #4) > > Sounds like a bug in GlusterFS. Adding Vijay in CC. > > Can you please provide details of the peers in the cluster and ensure that > gluster volume info lists vol1 on all peers in the cluster? > > If it does, please provide glusterd logs from all peers. > > Thanks, > Vijay Yes, the problem must be that only 2 out of 3 hosts in the cluster can see this volume. It's possible that I added the 3rd host after the volume was already created. Do we support this flow?
A newly added peer will be synced with the volumes already created. If everything goes well, then there shouldn't be any problem. In this case looks like the sync wasn't successful. Can you provide the glusterd logs for all the 3 peers, so that we can take a look?
Please provide details asked by Kaushal. In any case it looks like a combination of glusterfs environment related issue and https://bugzilla.redhat.com/823565 (which is fixed)
Haven't been able to reproduce in recent builds. Closing the bug.