Bug 1384311 - [Eventing]: 'gluster vol bitrot <volname> scrub ondemand' does not produce an event
Summary: [Eventing]: 'gluster vol bitrot <volname> scrub ondemand' does not produce an...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterfs
Version: rhgs-3.2
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: RHGS 3.2.0
Assignee: Kotresh HR
QA Contact: Sweta Anandpara
URL:
Whiteboard:
Depends On:
Blocks: 1351528 1387864 1387964
TreeView+ depends on / blocked
 
Reported: 2016-10-13 05:21 UTC by Sweta Anandpara
Modified: 2017-03-23 06:09 UTC (History)
3 users (show)

Fixed In Version: glusterfs-3.8.4-4
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1387864 (view as bug list)
Environment:
Last Closed: 2017-03-23 06:09:27 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:0486 normal SHIPPED_LIVE Moderate: Red Hat Gluster Storage 3.2.0 security, bug fix, and enhancement update 2017-03-23 09:18:45 UTC

Description Sweta Anandpara 2016-10-13 05:21:21 UTC
Description of problem:
========================
When we pause/resume the bitrot scrubber, it generates an event BITROT_SCRUB_OPTION, as there is a change in the scrub option. Similarly when we change the frequency to hourly/daily/weekly, it again generates an event BITROT_SCRUB_FREQ, as there is a change in the scrub frequency. Likewise, if we trigger the scrubber on demand, it necessitates a change in the system, thus producing an event. However, no event is generated/seen when the scrubber is triggered ondemand.


Version-Release number of selected component (if applicable):
============================================================
3.8.4-2


How reproducible:
=================
Always


Steps to Reproduce:
====================
1. Have a 4 node cluster, with a plain distribute volume. Enable eventing, and register a webhook, as a listener.
2. Create a volume 'vol1' and start it. The corresponding events are seen, as expected
3. Enable bitrot, change the scrub frequency and also try to pause/resume. Watch out for the related events BITROT_ENABLE, BITROT_SCRUB_FREQ, BITROT_SCRUB_OPTION. 
4. Trigger the scrubber on demand, and monitor if the event BITROT_SCRUB_OPTION is seen

Actual results:
==============
All expected events seen from step1 to 3. No event seen at step4

Expected results:
================
An event is expected at step4


Additional info:
=================

COMMANDS EXECUTED ON THE SERVER
--------------------------------
[root@dhcp35-100 ~]# gluster v list
dist
distrep
[root@dhcp35-100 ~]# 
[root@dhcp35-100 ~]# 
[root@dhcp35-100 ~]# 
[root@dhcp35-100 ~]# gluster v bitrot dist enable
volume bitrot: success
[root@dhcp35-100 ~]# gluster v info dist
 
Volume Name: dist
Type: Distribute
Volume ID: 9204b642-b380-4935-bb88-5aaaac1fe3fc
Status: Started
Snapshot Count: 0
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 10.70.35.115:/bricks/brick1/dist1
Brick2: 10.70.35.100:/bricks/brick1/dist2
Options Reconfigured:
features.scrub: Active
features.bitrot: on
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on
auto-delete: disable
[root@dhcp35-100 ~]# gluster v bitrot
Usage: volume bitrot <VOLNAME> {enable|disable} |
volume bitrot <volname> scrub-throttle {lazy|normal|aggressive} |
volume bitrot <volname> scrub-frequency {hourly|daily|weekly|biweekly|monthly} |
volume bitrot <volname> scrub {pause|resume|status|ondemand}
[root@dhcp35-100 ~]# gluster v bitrot dist scrub-throttle aggressive
volume bitrot: success
[root@dhcp35-100 ~]# 
[root@dhcp35-100 ~]# gluster  v bitrot 
Usage: volume bitrot <VOLNAME> {enable|disable} |
volume bitrot <volname> scrub-throttle {lazy|normal|aggressive} |
volume bitrot <volname> scrub-frequency {hourly|daily|weekly|biweekly|monthly} |
volume bitrot <volname> scrub {pause|resume|status|ondemand}
[root@dhcp35-100 ~]# gluster  v bitrot dist scrub-frequency minute
volume bitrot: success
[root@dhcp35-100 ~]# 
[root@dhcp35-100 ~]# 
[root@dhcp35-100 ~]# gluster v info dist
 
Volume Name: dist
Type: Distribute
Volume ID: 9204b642-b380-4935-bb88-5aaaac1fe3fc
Status: Started
Snapshot Count: 0
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 10.70.35.115:/bricks/brick1/dist1
Brick2: 10.70.35.100:/bricks/brick1/dist2
Options Reconfigured:
features.scrub-freq: minute
features.scrub-throttle: aggressive
features.scrub: Active
features.bitrot: on
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on
auto-delete: disable
[root@dhcp35-100 ~]# 
[root@dhcp35-100 ~]# gluster v bitrot 
Usage: volume bitrot <VOLNAME> {enable|disable} |
volume bitrot <volname> scrub-throttle {lazy|normal|aggressive} |
volume bitrot <volname> scrub-frequency {hourly|daily|weekly|biweekly|monthly} |
volume bitrot <volname> scrub {pause|resume|status|ondemand}
[root@dhcp35-100 ~]# 
[root@dhcp35-100 ~]# 
[root@dhcp35-100 ~]# gluster v bitrot dist scrub ondemand
volume bitrot: success
[root@dhcp35-100 ~]# gluster v bitrot dist scrub pause
volume bitrot: success
[root@dhcp35-100 ~]# gluster v bitrot dist scrub status

Volume name : dist

State of scrub: pause (Idle)

Scrub impact: aggressive

Scrub frequency: minute

Bitrot error log location: /var/log/glusterfs/bitd.log

Scrubber error log location: /var/log/glusterfs/scrub.log


=========================================================

Node: localhost

Number of Scrubbed files: 0

Number of Skipped files: 0

Last completed scrub time: 2016-10-13 05:01:51

Duration of last scrub (D:M:H:M:S): 0:0:0:0

Error count: 0


=========================================================

Node: dhcp35-115.lab.eng.blr.redhat.com

Number of Scrubbed files: 0

Number of Skipped files: 0

Last completed scrub time: 2016-10-13 05:01:52

Duration of last scrub (D:M:H:M:S): 0:0:0:0

Error count: 0

=========================================================

[root@dhcp35-100 ~]# gluster v bitrot dist scrub resume
volume bitrot: success
[root@dhcp35-100 ~]# gluster v bitrot dist scrub status

Volume name : dist

State of scrub: Active (Idle)

Scrub impact: aggressive

Scrub frequency: minute

Bitrot error log location: /var/log/glusterfs/bitd.log

Scrubber error log location: /var/log/glusterfs/scrub.log


=========================================================

Node: localhost

Number of Scrubbed files: 0

Number of Skipped files: 0

Last completed scrub time: 2016-10-13 05:01:51

Duration of last scrub (D:M:H:M:S): 0:0:0:0

Error count: 0


=========================================================

Node: dhcp35-115.lab.eng.blr.redhat.com

Number of Scrubbed files: 0

Number of Skipped files: 0

Last completed scrub time: 2016-10-13 05:01:52

Duration of last scrub (D:M:H:M:S): 0:0:0:0

Error count: 0

=========================================================

[root@dhcp35-100 ~]# 
[root@dhcp35-100 ~]# gluster v bitrot dist scrub ondemand
volume bitrot: success
[root@dhcp35-100 ~]# 

------------
EVENTS SEEN
------------
======================================================================================================================

{u'message': {u'name': u'dist'}, u'event': u'BITROT_ENABLE', u'ts': 1476334598, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'}

======================================================================================================================

{u'message': {u'svc_name': u'bitd'}, u'event': u'SVC_CONNECTED', u'ts': 1476334601, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'}

======================================================================================================================

{u'message': {u'svc_name': u'scrub'}, u'event': u'SVC_CONNECTED', u'ts': 1476334601, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'}

======================================================================================================================

{u'message': {u'svc_name': u'bitd'}, u'event': u'SVC_CONNECTED', u'ts': 1476334601, u'nodeid': u'6ac165c0-317f-42ad-8262-953995171dbb'}

======================================================================================================================

{u'message': {u'svc_name': u'scrub'}, u'event': u'SVC_CONNECTED', u'ts': 1476334601, u'nodeid': u'6ac165c0-317f-42ad-8262-953995171dbb'}

======================================================================================================================

{u'message': {u'name': u'dist', u'value': u'aggressive'}, u'event': u'BITROT_SCRUB_THROTTLE', u'ts': 1476334757, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'}

======================================================================================================================

{u'message': {u'name': u'dist', u'value': u'minute'}, u'event': u'BITROT_SCRUB_FREQ', u'ts': 1476334792, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'}

======================================================================================================================

{u'message': {u'name': u'dist', u'value': u'pause'}, u'event': u'BITROT_SCRUB_OPTION', u'ts': 1476334930, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'}

======================================================================================================================

{u'message': {u'name': u'dist', u'value': u'resume'}, u'event': u'BITROT_SCRUB_OPTION', u'ts': 1476334994, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'}

======================================================================================================================

Comment 2 Kotresh HR 2016-10-22 18:33:08 UTC
Upstream Patch:
http://review.gluster.org/15700    (master)

Comment 5 Atin Mukherjee 2016-11-07 14:31:54 UTC
upstream mainline : http://review.gluster.org/15700
upstream 3.9 : http://review.gluster.org/15705
downstream patch : https://code.engineering.redhat.com/gerrit/89264

Comment 7 Sweta Anandpara 2016-11-16 06:05:29 UTC
Tested and verified this on the build 3.8.4-5

When we execute 'gluster volume bitrot <volname> scrub ondemand' an event of name 'BITROT_SCRUB_ONDEMAND' is seen. Moving this BZ to verified in 3.2.

Detailed logs are pasted below:

[root@dhcp46-239 ~]# 
[root@dhcp46-239 ~]# rpm -qa | grep gluster
nfs-ganesha-gluster-2.3.1-8.el7rhgs.x86_64
glusterfs-api-3.8.4-5.el7rhgs.x86_64
python-gluster-3.8.4-5.el7rhgs.noarch
glusterfs-client-xlators-3.8.4-5.el7rhgs.x86_64
glusterfs-server-3.8.4-5.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-5.el7rhgs.x86_64
gluster-nagios-common-0.2.4-1.el7rhgs.noarch
glusterfs-devel-3.8.4-5.el7rhgs.x86_64
gluster-nagios-addons-0.2.8-1.el7rhgs.x86_64
glusterfs-libs-3.8.4-5.el7rhgs.x86_64
glusterfs-fuse-3.8.4-5.el7rhgs.x86_64
glusterfs-api-devel-3.8.4-5.el7rhgs.x86_64
glusterfs-rdma-3.8.4-5.el7rhgs.x86_64
glusterfs-3.8.4-5.el7rhgs.x86_64
glusterfs-cli-3.8.4-5.el7rhgs.x86_64
glusterfs-geo-replication-3.8.4-5.el7rhgs.x86_64
glusterfs-debuginfo-3.8.4-4.el7rhgs.x86_64
glusterfs-events-3.8.4-5.el7rhgs.x86_64
[root@dhcp46-239 ~]# 
[root@dhcp46-239 ~]# 
[root@dhcp46-239 ~]# gluster peer status
Number of Peers: 3

Hostname: 10.70.46.240
Uuid: 72c4f894-61f7-433e-a546-4ad2d7f0a176
State: Peer in Cluster (Connected)

Hostname: 10.70.46.242
Uuid: 1e8967ae-51b2-4c27-907e-a22a83107fd0
State: Peer in Cluster (Connected)

Hostname: 10.70.46.218
Uuid: 0dea52e0-8c32-4616-8ef8-16db16120eaa
State: Peer in Cluster (Connected)
[root@dhcp46-239 ~]# gluster v info
 
Volume Name: ozone
Type: Disperse
Volume ID: 376cdde0-194f-460a-b273-3904a704a7dd
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (4 + 2) = 6
Transport-type: tcp
Bricks:
Brick1: 10.70.46.239:/bricks/brick0/ozone0
Brick2: 10.70.46.240:/bricks/brick0/ozone2
Brick3: 10.70.46.242:/bricks/brick0/ozone2
Brick4: 10.70.46.239:/bricks/brick1/ozone3
Brick5: 10.70.46.240:/bricks/brick1/ozone4
Brick6: 10.70.46.242:/bricks/brick1/ozone5
Options Reconfigured:
features.expiry-time: 5
features.scrub-freq: daily
features.scrub: Active
features.bitrot: on
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on
cluster.enable-shared-storage: disable
 
Volume Name: rep3
Type: Distributed-Replicate
Volume ID: f29c2892-ac61-4b26-a146-8fa34038eb59
Status: Started
Snapshot Count: 0
Number of Bricks: 3 x 3 = 9
Transport-type: tcp
Bricks:
Brick1: 10.70.46.240:/bricks/brick0/rep30
Brick2: 10.70.46.242:/bricks/brick0/rep31
Brick3: 10.70.46.218:/bricks/brick0/rep32
Brick4: 10.70.46.240:/bricks/brick1/rep33
Brick5: 10.70.46.242:/bricks/brick1/rep34
Brick6: 10.70.46.218:/bricks/brick1/rep35
Brick7: 10.70.46.240:/bricks/brick2/rep36
Brick8: 10.70.46.242:/bricks/brick2/rep37
Brick9: 10.70.46.218:/bricks/brick2/rep38
Options Reconfigured:
features.expiry-time: 5
features.scrub-freq: hourly
features.scrub: Active
features.bitrot: on
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on
cluster.enable-shared-storage: disable
[root@dhcp46-239 ~]# 
[root@dhcp46-239 ~]# gluster v bitrot rep3 scrub status

Volume name : rep3

State of scrub: Active (Idle)

Scrub impact: lazy

Scrub frequency: hourly

Bitrot error log location: /var/log/glusterfs/bitd.log

Scrubber error log location: /var/log/glusterfs/scrub.log


=========================================================

Node: localhost

Number of Scrubbed files: 0

Number of Skipped files: 0

Last completed scrub time: Scrubber pending to complete.

Duration of last scrub (D:M:H:M:S): 0:0:0:0

Error count: 0


=========================================================

Node: 10.70.46.242

Number of Scrubbed files: 10

Number of Skipped files: 1

Last completed scrub time: 2016-11-16 06:01:45

Duration of last scrub (D:M:H:M:S): 0:0:0:5

Error count: 0


=========================================================

Node: 10.70.46.240

Number of Scrubbed files: 10

Number of Skipped files: 1

Last completed scrub time: 2016-11-16 06:01:45

Duration of last scrub (D:M:H:M:S): 0:0:0:5

Error count: 0


=========================================================

Node: 10.70.46.218

Number of Scrubbed files: 10

Number of Skipped files: 1

Last completed scrub time: 2016-11-16 06:01:45

Duration of last scrub (D:M:H:M:S): 0:0:0:5

Error count: 0

=========================================================

[root@dhcp46-239 ~]# gluster v bitrot rep3 scrub ondemand
volume bitrot: success
[root@dhcp46-239 ~]# 


EVENTS
-------
{u'message': {u'name': u'rep3'}, u'event': u'BITROT_SCRUB_ONDEMAND', u'ts': 1479276299, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}

Comment 9 errata-xmlrpc 2017-03-23 06:09:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0486.html


Note You need to log in before you can comment on or make changes to this bug.