Bug 1593852
| Summary: | IOPS chart on Disk Load of Brick Dashboard shows no data during brick read/write operation | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Martin Bukatovic <mbukatov> | ||||||||||
| Component: | web-admin-tendrl-monitoring-integration | Assignee: | Shubhendu Tripathi <shtripat> | ||||||||||
| Status: | CLOSED ERRATA | QA Contact: | Martin Bukatovic <mbukatov> | ||||||||||
| Severity: | unspecified | Docs Contact: | |||||||||||
| Priority: | unspecified | ||||||||||||
| Version: | rhgs-3.4 | CC: | amukherj, anbehl, mbukatov, nthomas, rhs-bugs, sankarshan, shtripat | ||||||||||
| Target Milestone: | --- | ||||||||||||
| Target Release: | RHGS 3.4.0 | ||||||||||||
| Hardware: | Unspecified | ||||||||||||
| OS: | Unspecified | ||||||||||||
| Whiteboard: | |||||||||||||
| Fixed In Version: | tendrl-monitoring-integration-1.6.3-6.el7rhgs | Doc Type: | If docs needed, set a value | ||||||||||
| Doc Text: | Story Points: | --- | |||||||||||
| Clone Of: | Environment: | ||||||||||||
| Last Closed: | 2018-09-04 07:07:57 UTC | Type: | Bug | ||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Embargoed: | |||||||||||||
| Bug Depends On: | |||||||||||||
| Bug Blocks: | 1503137 | ||||||||||||
| Attachments: |
|
||||||||||||
|
Description
Martin Bukatovic
2018-06-21 16:49:31 UTC
Linking related BZ about IOPS. Created attachment 1453564 [details]
screenshot 1
Created attachment 1453565 [details]
screenshot 2
Additional info =============== Inspecting timestamp from the screenshots, I can see that I waited just 16 in the step 6. <wild-guess> It's also possible that this is caused by some sync issue and bad time stamps somewhere. </wild-guess> Maybe important detail: I actually run out of the free space on the brick. Additional Info
===============
The IOPS chart in question has labels "vdd-read" and "vdd-write". So for reference, I checked
that this device really hosts the brick:
```
[root@mbukatov-usm1-gl1 ~]# lsblk /dev/vdd
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vdd 253:48 0 10G 0 disk
├─vg_beta_arbiter_3-pool_beta_arbiter_3_tmeta 252:6 0 52M 0 lvm
│ └─vg_beta_arbiter_3-pool_beta_arbiter_3-tpool 252:17 0 10G 0 lvm
│ ├─vg_beta_arbiter_3-pool_beta_arbiter_3 252:19 0 10G 0 lvm
│ └─vg_beta_arbiter_3-lv_beta_arbiter_3 252:23 0 10G 0 lvm /mnt/brick_beta_arbiter_3
└─vg_beta_arbiter_3-pool_beta_arbiter_3_tdata 252:9 0 10G 0 lvm
└─vg_beta_arbiter_3-pool_beta_arbiter_3-tpool 252:17 0 10G 0 lvm
├─vg_beta_arbiter_3-pool_beta_arbiter_3 252:19 0 10G 0 lvm
└─vg_beta_arbiter_3-lv_beta_arbiter_3 252:23 0 10G 0 lvm /mnt/brick_beta_arbiter_3
```
Created attachment 1454683 [details]
screenshot 3: with profiling disabled
I also noticed that this chart shown data even when profiling is disabled,
see screenshot 3.
(In reply to Martin Bukatovic from comment #9) > Created attachment 1454683 [details] > screenshot 3: with profiling disabled > > I also noticed that this chart shown data even when profiling is disabled, > see screenshot 3. Workload shown on the screenshot: extracting articles from enwiki-latest-pages-articles.xml.bz2 tarball into individual files for about 20 hours. Providing QE ack in a hope that patches linked to this BZ fixes the problem.
I will provide the `pstack {brick pid}` details during verification.
Now IOPS and disk data in grafana dashboards reflect at the same time from start. While there is a write happening on the bricks, the graphs reflect the same as expected. Created attachment 1476180 [details]
screenshot 4: verification
Testing with
============
[root@mbukatov-usm1-server ~]# rpm -qa | grep tendrl | sort
tendrl-ansible-1.6.3-6.el7rhgs.noarch
tendrl-api-1.6.3-5.el7rhgs.noarch
tendrl-api-httpd-1.6.3-5.el7rhgs.noarch
tendrl-commons-1.6.3-12.el7rhgs.noarch
tendrl-grafana-plugins-1.6.3-10.el7rhgs.noarch
tendrl-grafana-selinux-1.5.4-2.el7rhgs.noarch
tendrl-monitoring-integration-1.6.3-10.el7rhgs.noarch
tendrl-node-agent-1.6.3-10.el7rhgs.noarch
tendrl-notifier-1.6.3-4.el7rhgs.noarch
tendrl-selinux-1.5.4-2.el7rhgs.noarch
tendrl-ui-1.6.3-10.el7rhgs.noarch
[root@mbukatov-usm1-gl1 ~]# rpm -qa | grep tendrl | sort
tendrl-collectd-selinux-1.5.4-2.el7rhgs.noarch
tendrl-commons-1.6.3-12.el7rhgs.noarch
tendrl-gluster-integration-1.6.3-9.el7rhgs.noarch
tendrl-node-agent-1.6.3-10.el7rhgs.noarch
tendrl-selinux-1.5.4-2.el7rhgs.noarch
Results
=======
When I perform the steps to reproduce, IOPS chart of Disk Load section of
Brick dashboard now shows data immediately without any delay, which includes
both:
* zero or very small values (when no traffic from client is happening)
* iops data matching other charts on the dashboard during actual workload
Note: only single value (accounting both read and writes) is reported.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2616 |