Bug 1410283 - gluster cli: Exception when brick resides on a btrfs subvolume
Summary: gluster cli: Exception when brick resides on a btrfs subvolume
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: vdsm
Classification: oVirt
Component: Gluster
Version: 4.18.15.2
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ovirt-4.2.2
: 4.20.18
Assignee: Gobinda Das
QA Contact: SATHEESARAN
URL: https://gerrit.ovirt.org/#/c/69668/
Whiteboard:
Depends On: 1406569
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-05 01:01 UTC by George Joseph
Modified: 2018-05-10 06:23 UTC (History)
5 users (show)

Fixed In Version: vdsm v4.20.18
Clone Of:
Environment:
Last Closed: 2018-05-10 06:23:24 UTC
oVirt Team: Gluster
Embargoed:
rule-engine: ovirt-4.2+
rule-engine: planning_ack+
rule-engine: devel_ack+
rule-engine: testing_ack+

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 69668 0 master MERGED gluster: Fix error when brick is on a btrfs subvolume 2018-01-31 14:14:48 UTC

Description George Joseph 2017-01-05 01:01:16 UTC 
Description of problem:

When a gluster brick resides on a btrfs subvolume, an exception is thrown when attempting to retrieve the volume status detail.

MainProcess|jsonrpc.Executor/7::ERROR::2017-01-04 17:50:18,828::supervdsmServer::96::SuperVdsm.ServerCallback::(wrapper) Error in volumeStatus
Traceback (most recent call last):
  File "/usr/share/vdsm/supervdsmServer", line 94, in wrapper
    res = func(*args, **kwargs)
  File "/usr/share/vdsm/gluster/cli.py", line 353, in volumeStatus
    return _parseVolumeStatusDetail(xmltree)
  File "/usr/share/vdsm/gluster/cli.py", line 217, in _parseVolumeStatusDetail
    'device': value['device'],
KeyError: 'device'

The exception happens because gluster doesn't report device (or blockSize, mntOptions or fsName) for bricks that reside on btrfs subvolumes but parseVolumeStatusDetail expects them.

Version-Release number of selected component (if applicable):

4.18.15.3 but the issue also occurs in the 'master' vdsm branch.

How reproducible:

Both periodically when the engine polls the bricks and on demand when a user selects "Advanced Details" from the Volumes tab in the UI.

Steps to Reproduce:
1.  Create a Gluster volume with bricks in btrfs subvolumes on any host in the cluster.
2.  In the web UI, navigate to Volumes and select the new volume.
3.  In the brick list, select a brick and press Advanced Details.

Actual results:

Error message: "Error in fetching the brick details, please try again."
In the supervdsm log on the host, you'll see the above exception.

Expected results:

The details dialog should be populated.

Additional info:

Patch forthcoming
Comment 1 Sahina Bose 2017-01-09 09:39:37 UTC 
Ramesh, can you check if this is related to vdsm issue that you fixed?
Comment 2 Ramesh N 2017-01-09 09:57:58 UTC 
(In reply to Sahina Bose from comment #1)
> Ramesh, can you check if this is related to vdsm issue that you fixed?

We saw similar issue from some community user earlier. 'device' field was missing for one arbiter brick in the arbiter volume. 


George Joseph: Can you tell the gluster version you are running?. Also it will be useful to get the gluster cli output for 'gluster volume status <vol-name> detail --xml'.
Comment 3 Ramesh N 2017-01-09 10:25:44 UTC 
This is side effect gluster bz#1406569 Please followup in the bz#1406569 for more details.
Comment 4 George Joseph 2017-01-14 17:59:26 UTC 
[root@vmhost1 ~]# gluster --version
glusterfs 3.7.18 built on Dec  8 2016 12:16:01
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.

[root@vmhost1 ~]# gluster volume status gvm1 detail --xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<cliOutput>
  <opRet>0</opRet>
  <opErrno>0</opErrno>
  <opErrstr/>
  <volStatus>
    <volumes>
      <volume>
        <volName>gvm1</volName>
        <nodeCount>4</nodeCount>
        <node>
          <hostname>vmhostg1</hostname>
          <path>/gfs1/gvm1-brick1/brick</path>
          <peerid>afc90e43-c1d3-4c88-b077-d8597a68abcd</peerid>
          <status>1</status>
          <port>49169</port>
          <ports>
            <tcp>49169</tcp>
            <rdma>N/A</rdma>
          </ports>
          <pid>2541</pid>
          <sizeTotal>750171193344</sizeTotal>
          <sizeFree>631472906240</sizeFree>
          <blockSize>4096</blockSize>
        </node>
        <node>
          <hostname>vmhostg2</hostname>
          <path>/gfs1/gvm1-brick1/brick</path>
          <peerid>386f553f-4902-4031-9b76-8ea98a63502d</peerid>
          <status>1</status>
          <port>49172</port>
          <ports>
            <tcp>49172</tcp>
            <rdma>N/A</rdma>
          </ports>
          <pid>3619</pid>
          <sizeTotal>750171193344</sizeTotal>
          <sizeFree>634974404608</sizeFree>
          <blockSize>4096</blockSize>
        </node>
        <node>
          <hostname>vmhostg3</hostname>
          <path>/gfs1/gvm1-brick1/brick</path>
          <peerid>af65f817-a655-403d-b5d1-f61ea1aab5fc</peerid>
          <status>1</status>
          <port>49172</port>
          <ports>
            <tcp>49172</tcp>
            <rdma>N/A</rdma>
          </ports>
          <pid>2324</pid>
          <sizeTotal>762173194240</sizeTotal>
          <sizeFree>682148352000</sizeFree>
          <blockSize>4096</blockSize>
        </node>
        <node>
          <hostname>vmhostg4</hostname>
          <path>/gfs1/gvm1-brick1/brick</path>
          <peerid>9811a9a7-1b25-470b-a18b-5e5a8178f395</peerid>
          <status>1</status>
          <port>49172</port>
          <ports>
            <tcp>49172</tcp>
            <rdma>N/A</rdma>
          </ports>
          <pid>3332</pid>
          <sizeTotal>762173194240</sizeTotal>
          <sizeFree>678508535808</sizeFree>
          <blockSize>4096</blockSize>
        </node>
      </volume>
    </volumes>
  </volStatus>
</cliOutput>
Comment 5 George Joseph 2017-01-14 18:06:04 UTC 
The referenced issue is specific to arbiter bricks. In my situation, there is no device on any brick.

Actually I have 2 volumes, the detail  above is from a 2x2 volume.  No device on any brick.

Here's the detail from a  4 x (2 + 1) = 12 volume

[root@vmhost1 ~]# gluster volume status gvm0 detail --xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<cliOutput>
  <opRet>0</opRet>
  <opErrno>0</opErrno>
  <opErrstr/>
  <volStatus>
    <volumes>
      <volume>
        <volName>gvm0</volName>
        <nodeCount>12</nodeCount>
        <node>
          <hostname>vmhostg1</hostname>
          <path>/gfs1/gvm0-brick1/brick</path>
          <peerid>afc90e43-c1d3-4c88-b077-d8597a68abcd</peerid>
          <status>1</status>
          <port>49160</port>
          <ports>
            <tcp>49160</tcp>
            <rdma>N/A</rdma>
          </ports>
          <pid>2478</pid>
          <sizeTotal>750171193344</sizeTotal>
          <sizeFree>631473377280</sizeFree>
          <blockSize>4096</blockSize>
        </node>
        <node>
          <hostname>vmhostg2</hostname>
          <path>/gfs1/gvm0-brick1/brick</path>
          <peerid>386f553f-4902-4031-9b76-8ea98a63502d</peerid>
          <status>1</status>
          <port>49163</port>
          <ports>
            <tcp>49163</tcp>
            <rdma>N/A</rdma>
          </ports>
          <pid>3599</pid>
          <sizeTotal>750171193344</sizeTotal>
          <sizeFree>634974969856</sizeFree>
          <blockSize>4096</blockSize>
        </node>
        <node>
          <hostname>vmhostg3</hostname>
          <path>/gfs1/gvm0-brick1/brick</path>
          <peerid>af65f817-a655-403d-b5d1-f61ea1aab5fc</peerid>
          <status>1</status>
          <port>49163</port>
          <ports>
            <tcp>49163</tcp>
            <rdma>N/A</rdma>
          </ports>
          <pid>2290</pid>
          <sizeTotal>762173194240</sizeTotal>
          <sizeFree>682126888960</sizeFree>
          <blockSize>4096</blockSize>
        </node>
        <node>
          <hostname>vmhostg4</hostname>
          <path>/gfs1/gvm0-brick1/brick</path>
          <peerid>9811a9a7-1b25-470b-a18b-5e5a8178f395</peerid>
          <status>1</status>
          <port>49163</port>
          <ports>
            <tcp>49163</tcp>
            <rdma>N/A</rdma>
          </ports>
          <pid>3282</pid>
          <sizeTotal>762173194240</sizeTotal>
          <sizeFree>678477139968</sizeFree>
          <blockSize>4096</blockSize>
        </node>
        <node>
          <hostname>vmhostg1</hostname>
          <path>/gfs1/gvm0-brick2/brick</path>
          <peerid>afc90e43-c1d3-4c88-b077-d8597a68abcd</peerid>
          <status>1</status>
          <port>49161</port>
          <ports>
            <tcp>49161</tcp>
            <rdma>N/A</rdma>
          </ports>
          <pid>2536</pid>
          <sizeTotal>750171193344</sizeTotal>
          <sizeFree>631473377280</sizeFree>
          <blockSize>4096</blockSize>
        </node>
        <node>
          <hostname>vmhostg2</hostname>
          <path>/gfs1/gvm0-brick2/brick</path>
          <peerid>386f553f-4902-4031-9b76-8ea98a63502d</peerid>
          <status>1</status>
          <port>49164</port>
          <ports>
            <tcp>49164</tcp>
            <rdma>N/A</rdma>
          </ports>
          <pid>3606</pid>
          <sizeTotal>750171193344</sizeTotal>
          <sizeFree>634974969856</sizeFree>
          <blockSize>4096</blockSize>
        </node>
        <node>
          <hostname>vmhostg3</hostname>
          <path>/gfs1/gvm0-brick2/brick</path>
          <peerid>af65f817-a655-403d-b5d1-f61ea1aab5fc</peerid>
          <status>1</status>
          <port>49164</port>
          <ports>
            <tcp>49164</tcp>
            <rdma>N/A</rdma>
          </ports>
          <pid>2297</pid>
          <sizeTotal>762173194240</sizeTotal>
          <sizeFree>682126888960</sizeFree>
          <blockSize>4096</blockSize>
        </node>
        <node>
          <hostname>vmhostg4</hostname>
          <path>/gfs1/gvm0-brick2/brick</path>
          <peerid>9811a9a7-1b25-470b-a18b-5e5a8178f395</peerid>
          <status>1</status>
          <port>49164</port>
          <ports>
            <tcp>49164</tcp>
            <rdma>N/A</rdma>
          </ports>
          <pid>3296</pid>
          <sizeTotal>762173194240</sizeTotal>
          <sizeFree>678477139968</sizeFree>
          <blockSize>4096</blockSize>
        </node>
        <node>
          <hostname>vmhostg1</hostname>
          <path>/gfs1/gvm0-brick3/brick</path>
          <peerid>afc90e43-c1d3-4c88-b077-d8597a68abcd</peerid>
          <status>1</status>
          <port>49162</port>
          <ports>
            <tcp>49162</tcp>
            <rdma>N/A</rdma>
          </ports>
          <pid>2527</pid>
          <sizeTotal>750171193344</sizeTotal>
          <sizeFree>631473377280</sizeFree>
          <blockSize>4096</blockSize>
        </node>
        <node>
          <hostname>vmhostg2</hostname>
          <path>/gfs1/gvm0-brick3/brick</path>
          <peerid>386f553f-4902-4031-9b76-8ea98a63502d</peerid>
          <status>1</status>
          <port>49165</port>
          <ports>
            <tcp>49165</tcp>
            <rdma>N/A</rdma>
          </ports>
          <pid>3612</pid>
          <sizeTotal>750171193344</sizeTotal>
          <sizeFree>634974969856</sizeFree>
          <blockSize>4096</blockSize>
        </node>
        <node>
          <hostname>vmhostg3</hostname>
          <path>/gfs1/gvm0-brick3/brick</path>
          <peerid>af65f817-a655-403d-b5d1-f61ea1aab5fc</peerid>
          <status>1</status>
          <port>49165</port>
          <ports>
            <tcp>49165</tcp>
            <rdma>N/A</rdma>
          </ports>
          <pid>2304</pid>
          <sizeTotal>762173194240</sizeTotal>
          <sizeFree>682126888960</sizeFree>
          <blockSize>4096</blockSize>
        </node>
        <node>
          <hostname>vmhostg4</hostname>
          <path>/gfs1/gvm0-brick3/brick</path>
          <peerid>9811a9a7-1b25-470b-a18b-5e5a8178f395</peerid>
          <status>1</status>
          <port>49165</port>
          <ports>
            <tcp>49165</tcp>
            <rdma>N/A</rdma>
          </ports>
          <pid>3308</pid>
          <sizeTotal>762173194240</sizeTotal>
          <sizeFree>678477139968</sizeFree>
          <blockSize>4096</blockSize>
        </node>
      </volume>
    </volumes>
  </volStatus>
</cliOutput>


Either way, vdsm shouldn't barf on a missing element.
Comment 6 Sahina Bose 2018-01-31 07:06:48 UTC 
Gobinda, can you follow up to make sure it gets merged? Thanks!
Comment 7 Gobinda Das 2018-02-05 05:26:18 UTC 
Sahina, It is merged.
Comment 8 SATHEESARAN 2018-05-10 02:30:19 UTC 
Tested with RHV 4.2.3, When using the btrfs FS backed gluster brick, 'Advanced details' from the bricks could be fetched
Comment 9 Sandro Bonazzola 2018-05-10 06:23:24 UTC 
This bugzilla is included in oVirt 4.2.2 release, published on March 28th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.2 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.
Note You need to log in before you can comment on or make changes to this bug.