Bug 1641680 - [RFE] Add ability to heketi server to output diagnostics information for OCS
Summary: [RFE] Add ability to heketi server to output diagnostics information for OCS
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: heketi
Version: ocs-3.11
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: OCS 3.11.1
Assignee: Raghavendra Talur
QA Contact: Nitin Goyal
URL:
Whiteboard:
: 1653158 (view as bug list)
Depends On:
Blocks: 1597222 OCS-3.11.1-devel-triage-done 1644143 1644171 1661529
TreeView+ depends on / blocked
 
Reported: 2018-10-22 13:49 UTC by Raghavendra Talur
Modified: 2019-02-08 07:31 UTC (History)
14 users (show)

Fixed In Version: heketi-8.0.0-7.el7rhgs
Doc Type: Enhancement
Doc Text:
This release introduces two diagnostics features in Heketi: * The new command `heketi-cli db check` and its offline server variant `heketi db check` check the consistency of heketi's database. * The new command `heketi-cli server state examine gluster` and its server counterpart `heketi offline state examine gluster` compare data from Gluster with that in heketi's database to detect out-of-sync situations.
Clone Of:
Environment:
Last Closed: 2019-02-07 03:41:00 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2019:0286 0 None None None 2019-02-07 03:41:26 UTC

Internal Links: 1661529

Description Raghavendra Talur 2018-10-22 13:49:22 UTC
Description of problem:
Heketi has permissions to execute commands on Gluster pods and also execute some queries on the OCP cluster. Using the ability it is possible for heketi to collect diagnostic information from OCS cluster and suggest remedies.

This bz is to be used as a tracker of such enhancements to be made in heketi.

Additional info:
1. Compare peer list from heketi and Gluster pool list
2. Compare the LogicalVolume from heketi brick entries and lvs output
3. Compare the volume count from heketi and gluster
4. Compare the volume info from heketi volume entry and gluster volume info output
5. Compare the blockvolume info from heketi and gluster-block



Optional:
Compare the from OCP PV list with the list of volumes from heketi and gluster

Comment 7 Raghavendra Talur 2018-11-16 22:38:58 UTC
This should also check the devices names those backing a vg are same as stored by heketi. A mismatch here is possible if the device naming in the host is not consistent.

Comment 9 John Mulligan 2018-11-27 22:41:40 UTC
*** Bug 1653158 has been marked as a duplicate of this bug. ***

Comment 19 krishnaram Karthick 2019-01-09 07:08:42 UTC
Talur,

I don't see an option 'state' available for heketi in 8.0.0-7. 

[root@dhcp46-173 ~]# heketi-cli --version
heketi-cli 8.0.0
[root@dhcp46-173 ~]# rpm -qa | grep 'heketi'
heketi-client-8.0.0-7.el7rhgs.x86_64
sh-4.2# rpm -qa | grep 'heketi'
heketi-client-8.0.0-7.el7rhgs.x86_64
heketi-8.0.0-7.el7rhgs.x86_64

[root@dhcp46-173 ~]# heketi-cli --help
Command line program for Heketi

Usage:
  heketi-cli [flags]
  heketi-cli [command]

Examples:
  $ heketi-cli volume list

Available Commands:
  blockvolume                    Heketi Volume Management
  cluster                        Heketi cluster management
  db                             Heketi Database Management
  device                         Heketi device management
  help                           Help about any command
  loglevel                       Heketi Log Level
  node                           Heketi Node Management
  server                         Heketi Server Management
  setup-openshift-heketi-storage Setup OpenShift/Kubernetes persistent storage for Heketi
  topology                       Heketi Topology Management
  volume                         Heketi Volume Management

Can you please confirm if the cli option is indeed made available? I'l move the bug back to assigned if the cli option needs to be added.

Comment 25 Nitin Goyal 2019-01-18 10:38:00 UTC
Hi Talur,

Can u see "pendingoperations" section in the output. It is not giving the count in "total", just giving the count in "ok".


# heketi-cli db check
{
    "clusters": {
        "total": 1,
        "pending": 0,
        "ok": 1,
        "notok": 0,
        "inconsistencies": null
    },
    "volumes": {
        "total": 9,
        "pending": 8,
        "ok": 8,
        "notok": 1,
        "inconsistencies": [
            "Volume 4963e5e6e0b617514935b7554e0bb958 mount hosts list([10.70.47.31 10.70.46.99 10.70.46.7]) is not same as list of all nodes of the cluster([48dac2c26112c558da055e2494101aa0 db67763501c3f6645d9a08a201f71fcc e3dd0582f004eada2f0017cf1138ea8f f19fdd1c652de2f1d6f4672829dffe02])"
        ]
    },
    "bricks": {
        "total": 27,
        "pending": 24,
        "ok": 27,
        "notok": 0,
        "inconsistencies": null
    },
    "nodes": {
        "total": 4,
        "pending": 0,
        "ok": 4,
        "notok": 0,
        "inconsistencies": null
    },
    "devices": {
        "total": 4,
        "pending": 0,
        "ok": 4,
        "notok": 0,
        "inconsistencies": null
    },
    "blockvolumes": {
        "total": 0,
        "pending": 0,
        "ok": 0,
        "notok": 0,
        "inconsistencies": null
    },
    "dbattributes": {
        "total": 0,
        "pending": 0,
        "ok": 0,
        "notok": 0,
        "inconsistencies": null
    },
    "pendingoperations": {
        "total": 0,
        "pending": 0,
        "ok": 8,
        "notok": 0,
        "inconsistencies": null
    },
    "totalinconsistencies": 1
}

Comment 28 Anjana KD 2019-01-21 10:30:28 UTC
Kindly provide the complete doc text.

Comment 33 errata-xmlrpc 2019-02-07 03:41:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0286


Note You need to log in before you can comment on or make changes to this bug.