Bug 1290997 - Handle missing glusterfs-cli package
Handle missing glusterfs-cli package
Product: vdsm
Classification: oVirt
Component: Core (Show other bugs)
Unspecified Unspecified
unspecified Severity medium (vote)
: ovirt-3.6.2
: 4.17.14
Assigned To: Ala Hino
Depends On:
  Show dependency treegraph
Reported: 2015-12-12 10:21 EST by Ala Hino
Modified: 2016-02-18 06:13 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Missing glusterfs-cli package Consequence: Not clear exception in vdsm log and no automatic mount for gluster backup servers Fix: Check whether glusterfs-cli exists Result: If glusterfs-cli doesn't exist, a we log a message indicating that the package is missing and recommend to install the package in order to enable automatic mount of gluster backup severs
Story Points: ---
Clone Of:
Last Closed: 2016-02-18 06:13:58 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
rule-engine: ovirt‑3.6.z+
ylavi: planning_ack+
rule-engine: devel_ack+
acanan: testing_ack+

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 50363 master MERGED gluster: Handle missing glusterfs-cli package 2015-12-21 02:22 EST
oVirt gerrit 50787 ovirt-3.6 MERGED gluster: Handle missing glusterfs-cli package 2015-12-21 08:19 EST

  None (edit)
Description Ala Hino 2015-12-12 10:21:37 EST
Description of problem:
In 3.6 we introduced a dependency to glusterfs-cli. This is required in order to get gluster volume info needed to get gluster backup servers and replica related logic.
During upgrade, if glusterfs-cli is not installed, there is a not-completely clear exception in the log:

jsonrpc.Executor/5::ERROR::2015-12-12 16:30:56,068::hsm::2476::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 2473, in connectStorageServer
  File "/usr/share/vdsm/storage/storageServer.py", line 222, in connect
  File "/usr/share/vdsm/storage/storageServer.py", line 337, in validate
    replicaCount = self.volinfo['replicaCount']
  File "/usr/share/vdsm/storage/storageServer.py", line 333, in volinfo
    self._volinfo = self._get_gluster_volinfo()
  File "/usr/share/vdsm/storage/storageServer.py", line 362, in _get_gluster_volinfo
  File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 51, in __call__
    return callMethod()
  File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 49, in <lambda>
  File "<string>", line 2, in glusterVolumeInfo
  File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
    raise convert_to_error(kind, result)
GlusterCmdExecFailedException: Command execution failed

In this case, we'd like to add a warning message in the log clearly indicating missing glustefs-cli package.

How reproducible:

Steps to Reproduce:
1. Install 3.5
2. Add gluster sd
3. Upgrade to 3.6

Actual results:
Cannot connect to gluster

Expected results:
Upgrade to successfully pass
Comment 1 Nir Soffer 2015-12-12 13:47:00 EST
The error seen here does not mean the glusterfs-cli package is missing, so we
cannot replace the error with a warning.

If we want to warn about missing package, we should check if the package is
available, and if not, warn about it and continue to work as we did in 
rhev 3.5 (no automatic backup servers).

But the real fix is to require the package on the platform which provide it.
Comment 2 Dan Kenigsberg 2015-12-12 16:04:35 EST
Vdsm reports (in Caps) whether glusterfs-cli is installed on the host. Engine should be smart enough not to allow a host without it into a 3.6 cluster with a gluster SD.
Comment 3 Nir Soffer 2015-12-12 16:52:12 EST
Dan, I think the issue is older engine that does not know anything about gluster
availability. In 3.6 would not let you add such host to a cluster or upgrade
the cluster if a host in the cluster does not have the required packages.

These packages are needed now because 3.6 try to provide more reliable mounting
by getting information about the gluster volume. Seems that we can try this
improved mounting only if gluster is available.
Comment 4 Ala Hino 2015-12-13 04:30:50 EST
Two different things:

1. In 3.6 the engine does check Vdsm Caps and alerts ifrequired gluster packages are missing

2. In this bug, we would like to warn the admin, mainly when upgrading from 3.5, that required gluster packages are missing.
Comment 5 Sandro Bonazzola 2015-12-23 08:41:13 EST
oVirt 3.6.2 RC1 has been released for testing, moving to ON_QA
Comment 6 Allon Mureinik 2016-01-14 09:36:56 EST
Ala, please handle the doctext.
Comment 7 Elad 2016-02-07 10:34:29 EST
After upgrading host cluster and DC from 3.5 to 3.6 while having an active Gluster domain and no gluster-cli package installed on the host, I got the following warning in vdsm.log:

jsonrpc.Executor/3::WARNING::2016-02-07 17:12:03,556::storageServer::338::Storage.StorageServer.MountConnection::(validate) Required glusterfs-cli package is missing on this host. Note that automatic detection of backup servers will be disabled! Please install the missing package in order to automatically mount gluster storage backup servers

Ala, can you please confirm so I'll move to VERIFIED? 


Checked using:
Comment 8 Ala Hino 2016-02-07 10:40:34 EST
The message is expected. 
Assuming the domain is still active after the upgrade, go ahead and move the bug to VERIFIED.
Comment 9 Elad 2016-02-07 12:06:46 EST
Indeed, the domain is active after the upgrade. 
Thanks, moving to VERIFIED.

Note You need to log in before you can comment on or make changes to this bug.