Bug 1384311
| Summary: | [Eventing]: 'gluster vol bitrot <volname> scrub ondemand' does not produce an event | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Sweta Anandpara <sanandpa> | |
| Component: | glusterfs | Assignee: | Kotresh HR <khiremat> | |
| Status: | CLOSED ERRATA | QA Contact: | Sweta Anandpara <sanandpa> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | rhgs-3.2 | CC: | amukherj, rhinduja, vbellur | |
| Target Milestone: | --- | |||
| Target Release: | RHGS 3.2.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | glusterfs-3.8.4-4 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1387864 (view as bug list) | Environment: | ||
| Last Closed: | 2017-03-23 06:09:27 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1351528, 1387864, 1387964 | |||
Upstream Patch: http://review.gluster.org/15700 (master) upstream mainline : http://review.gluster.org/15700 upstream 3.9 : http://review.gluster.org/15705 downstream patch : https://code.engineering.redhat.com/gerrit/89264 Tested and verified this on the build 3.8.4-5
When we execute 'gluster volume bitrot <volname> scrub ondemand' an event of name 'BITROT_SCRUB_ONDEMAND' is seen. Moving this BZ to verified in 3.2.
Detailed logs are pasted below:
[root@dhcp46-239 ~]#
[root@dhcp46-239 ~]# rpm -qa | grep gluster
nfs-ganesha-gluster-2.3.1-8.el7rhgs.x86_64
glusterfs-api-3.8.4-5.el7rhgs.x86_64
python-gluster-3.8.4-5.el7rhgs.noarch
glusterfs-client-xlators-3.8.4-5.el7rhgs.x86_64
glusterfs-server-3.8.4-5.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-5.el7rhgs.x86_64
gluster-nagios-common-0.2.4-1.el7rhgs.noarch
glusterfs-devel-3.8.4-5.el7rhgs.x86_64
gluster-nagios-addons-0.2.8-1.el7rhgs.x86_64
glusterfs-libs-3.8.4-5.el7rhgs.x86_64
glusterfs-fuse-3.8.4-5.el7rhgs.x86_64
glusterfs-api-devel-3.8.4-5.el7rhgs.x86_64
glusterfs-rdma-3.8.4-5.el7rhgs.x86_64
glusterfs-3.8.4-5.el7rhgs.x86_64
glusterfs-cli-3.8.4-5.el7rhgs.x86_64
glusterfs-geo-replication-3.8.4-5.el7rhgs.x86_64
glusterfs-debuginfo-3.8.4-4.el7rhgs.x86_64
glusterfs-events-3.8.4-5.el7rhgs.x86_64
[root@dhcp46-239 ~]#
[root@dhcp46-239 ~]#
[root@dhcp46-239 ~]# gluster peer status
Number of Peers: 3
Hostname: 10.70.46.240
Uuid: 72c4f894-61f7-433e-a546-4ad2d7f0a176
State: Peer in Cluster (Connected)
Hostname: 10.70.46.242
Uuid: 1e8967ae-51b2-4c27-907e-a22a83107fd0
State: Peer in Cluster (Connected)
Hostname: 10.70.46.218
Uuid: 0dea52e0-8c32-4616-8ef8-16db16120eaa
State: Peer in Cluster (Connected)
[root@dhcp46-239 ~]# gluster v info
Volume Name: ozone
Type: Disperse
Volume ID: 376cdde0-194f-460a-b273-3904a704a7dd
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (4 + 2) = 6
Transport-type: tcp
Bricks:
Brick1: 10.70.46.239:/bricks/brick0/ozone0
Brick2: 10.70.46.240:/bricks/brick0/ozone2
Brick3: 10.70.46.242:/bricks/brick0/ozone2
Brick4: 10.70.46.239:/bricks/brick1/ozone3
Brick5: 10.70.46.240:/bricks/brick1/ozone4
Brick6: 10.70.46.242:/bricks/brick1/ozone5
Options Reconfigured:
features.expiry-time: 5
features.scrub-freq: daily
features.scrub: Active
features.bitrot: on
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on
cluster.enable-shared-storage: disable
Volume Name: rep3
Type: Distributed-Replicate
Volume ID: f29c2892-ac61-4b26-a146-8fa34038eb59
Status: Started
Snapshot Count: 0
Number of Bricks: 3 x 3 = 9
Transport-type: tcp
Bricks:
Brick1: 10.70.46.240:/bricks/brick0/rep30
Brick2: 10.70.46.242:/bricks/brick0/rep31
Brick3: 10.70.46.218:/bricks/brick0/rep32
Brick4: 10.70.46.240:/bricks/brick1/rep33
Brick5: 10.70.46.242:/bricks/brick1/rep34
Brick6: 10.70.46.218:/bricks/brick1/rep35
Brick7: 10.70.46.240:/bricks/brick2/rep36
Brick8: 10.70.46.242:/bricks/brick2/rep37
Brick9: 10.70.46.218:/bricks/brick2/rep38
Options Reconfigured:
features.expiry-time: 5
features.scrub-freq: hourly
features.scrub: Active
features.bitrot: on
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on
cluster.enable-shared-storage: disable
[root@dhcp46-239 ~]#
[root@dhcp46-239 ~]# gluster v bitrot rep3 scrub status
Volume name : rep3
State of scrub: Active (Idle)
Scrub impact: lazy
Scrub frequency: hourly
Bitrot error log location: /var/log/glusterfs/bitd.log
Scrubber error log location: /var/log/glusterfs/scrub.log
=========================================================
Node: localhost
Number of Scrubbed files: 0
Number of Skipped files: 0
Last completed scrub time: Scrubber pending to complete.
Duration of last scrub (D:M:H:M:S): 0:0:0:0
Error count: 0
=========================================================
Node: 10.70.46.242
Number of Scrubbed files: 10
Number of Skipped files: 1
Last completed scrub time: 2016-11-16 06:01:45
Duration of last scrub (D:M:H:M:S): 0:0:0:5
Error count: 0
=========================================================
Node: 10.70.46.240
Number of Scrubbed files: 10
Number of Skipped files: 1
Last completed scrub time: 2016-11-16 06:01:45
Duration of last scrub (D:M:H:M:S): 0:0:0:5
Error count: 0
=========================================================
Node: 10.70.46.218
Number of Scrubbed files: 10
Number of Skipped files: 1
Last completed scrub time: 2016-11-16 06:01:45
Duration of last scrub (D:M:H:M:S): 0:0:0:5
Error count: 0
=========================================================
[root@dhcp46-239 ~]# gluster v bitrot rep3 scrub ondemand
volume bitrot: success
[root@dhcp46-239 ~]#
EVENTS
-------
{u'message': {u'name': u'rep3'}, u'event': u'BITROT_SCRUB_ONDEMAND', u'ts': 1479276299, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2017-0486.html |
Description of problem: ======================== When we pause/resume the bitrot scrubber, it generates an event BITROT_SCRUB_OPTION, as there is a change in the scrub option. Similarly when we change the frequency to hourly/daily/weekly, it again generates an event BITROT_SCRUB_FREQ, as there is a change in the scrub frequency. Likewise, if we trigger the scrubber on demand, it necessitates a change in the system, thus producing an event. However, no event is generated/seen when the scrubber is triggered ondemand. Version-Release number of selected component (if applicable): ============================================================ 3.8.4-2 How reproducible: ================= Always Steps to Reproduce: ==================== 1. Have a 4 node cluster, with a plain distribute volume. Enable eventing, and register a webhook, as a listener. 2. Create a volume 'vol1' and start it. The corresponding events are seen, as expected 3. Enable bitrot, change the scrub frequency and also try to pause/resume. Watch out for the related events BITROT_ENABLE, BITROT_SCRUB_FREQ, BITROT_SCRUB_OPTION. 4. Trigger the scrubber on demand, and monitor if the event BITROT_SCRUB_OPTION is seen Actual results: ============== All expected events seen from step1 to 3. No event seen at step4 Expected results: ================ An event is expected at step4 Additional info: ================= COMMANDS EXECUTED ON THE SERVER -------------------------------- [root@dhcp35-100 ~]# gluster v list dist distrep [root@dhcp35-100 ~]# [root@dhcp35-100 ~]# [root@dhcp35-100 ~]# [root@dhcp35-100 ~]# gluster v bitrot dist enable volume bitrot: success [root@dhcp35-100 ~]# gluster v info dist Volume Name: dist Type: Distribute Volume ID: 9204b642-b380-4935-bb88-5aaaac1fe3fc Status: Started Snapshot Count: 0 Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: 10.70.35.115:/bricks/brick1/dist1 Brick2: 10.70.35.100:/bricks/brick1/dist2 Options Reconfigured: features.scrub: Active features.bitrot: on transport.address-family: inet performance.readdir-ahead: on nfs.disable: on auto-delete: disable [root@dhcp35-100 ~]# gluster v bitrot Usage: volume bitrot <VOLNAME> {enable|disable} | volume bitrot <volname> scrub-throttle {lazy|normal|aggressive} | volume bitrot <volname> scrub-frequency {hourly|daily|weekly|biweekly|monthly} | volume bitrot <volname> scrub {pause|resume|status|ondemand} [root@dhcp35-100 ~]# gluster v bitrot dist scrub-throttle aggressive volume bitrot: success [root@dhcp35-100 ~]# [root@dhcp35-100 ~]# gluster v bitrot Usage: volume bitrot <VOLNAME> {enable|disable} | volume bitrot <volname> scrub-throttle {lazy|normal|aggressive} | volume bitrot <volname> scrub-frequency {hourly|daily|weekly|biweekly|monthly} | volume bitrot <volname> scrub {pause|resume|status|ondemand} [root@dhcp35-100 ~]# gluster v bitrot dist scrub-frequency minute volume bitrot: success [root@dhcp35-100 ~]# [root@dhcp35-100 ~]# [root@dhcp35-100 ~]# gluster v info dist Volume Name: dist Type: Distribute Volume ID: 9204b642-b380-4935-bb88-5aaaac1fe3fc Status: Started Snapshot Count: 0 Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: 10.70.35.115:/bricks/brick1/dist1 Brick2: 10.70.35.100:/bricks/brick1/dist2 Options Reconfigured: features.scrub-freq: minute features.scrub-throttle: aggressive features.scrub: Active features.bitrot: on transport.address-family: inet performance.readdir-ahead: on nfs.disable: on auto-delete: disable [root@dhcp35-100 ~]# [root@dhcp35-100 ~]# gluster v bitrot Usage: volume bitrot <VOLNAME> {enable|disable} | volume bitrot <volname> scrub-throttle {lazy|normal|aggressive} | volume bitrot <volname> scrub-frequency {hourly|daily|weekly|biweekly|monthly} | volume bitrot <volname> scrub {pause|resume|status|ondemand} [root@dhcp35-100 ~]# [root@dhcp35-100 ~]# [root@dhcp35-100 ~]# gluster v bitrot dist scrub ondemand volume bitrot: success [root@dhcp35-100 ~]# gluster v bitrot dist scrub pause volume bitrot: success [root@dhcp35-100 ~]# gluster v bitrot dist scrub status Volume name : dist State of scrub: pause (Idle) Scrub impact: aggressive Scrub frequency: minute Bitrot error log location: /var/log/glusterfs/bitd.log Scrubber error log location: /var/log/glusterfs/scrub.log ========================================================= Node: localhost Number of Scrubbed files: 0 Number of Skipped files: 0 Last completed scrub time: 2016-10-13 05:01:51 Duration of last scrub (D:M:H:M:S): 0:0:0:0 Error count: 0 ========================================================= Node: dhcp35-115.lab.eng.blr.redhat.com Number of Scrubbed files: 0 Number of Skipped files: 0 Last completed scrub time: 2016-10-13 05:01:52 Duration of last scrub (D:M:H:M:S): 0:0:0:0 Error count: 0 ========================================================= [root@dhcp35-100 ~]# gluster v bitrot dist scrub resume volume bitrot: success [root@dhcp35-100 ~]# gluster v bitrot dist scrub status Volume name : dist State of scrub: Active (Idle) Scrub impact: aggressive Scrub frequency: minute Bitrot error log location: /var/log/glusterfs/bitd.log Scrubber error log location: /var/log/glusterfs/scrub.log ========================================================= Node: localhost Number of Scrubbed files: 0 Number of Skipped files: 0 Last completed scrub time: 2016-10-13 05:01:51 Duration of last scrub (D:M:H:M:S): 0:0:0:0 Error count: 0 ========================================================= Node: dhcp35-115.lab.eng.blr.redhat.com Number of Scrubbed files: 0 Number of Skipped files: 0 Last completed scrub time: 2016-10-13 05:01:52 Duration of last scrub (D:M:H:M:S): 0:0:0:0 Error count: 0 ========================================================= [root@dhcp35-100 ~]# [root@dhcp35-100 ~]# gluster v bitrot dist scrub ondemand volume bitrot: success [root@dhcp35-100 ~]# ------------ EVENTS SEEN ------------ ====================================================================================================================== {u'message': {u'name': u'dist'}, u'event': u'BITROT_ENABLE', u'ts': 1476334598, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'} ====================================================================================================================== {u'message': {u'svc_name': u'bitd'}, u'event': u'SVC_CONNECTED', u'ts': 1476334601, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'} ====================================================================================================================== {u'message': {u'svc_name': u'scrub'}, u'event': u'SVC_CONNECTED', u'ts': 1476334601, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'} ====================================================================================================================== {u'message': {u'svc_name': u'bitd'}, u'event': u'SVC_CONNECTED', u'ts': 1476334601, u'nodeid': u'6ac165c0-317f-42ad-8262-953995171dbb'} ====================================================================================================================== {u'message': {u'svc_name': u'scrub'}, u'event': u'SVC_CONNECTED', u'ts': 1476334601, u'nodeid': u'6ac165c0-317f-42ad-8262-953995171dbb'} ====================================================================================================================== {u'message': {u'name': u'dist', u'value': u'aggressive'}, u'event': u'BITROT_SCRUB_THROTTLE', u'ts': 1476334757, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'} ====================================================================================================================== {u'message': {u'name': u'dist', u'value': u'minute'}, u'event': u'BITROT_SCRUB_FREQ', u'ts': 1476334792, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'} ====================================================================================================================== {u'message': {u'name': u'dist', u'value': u'pause'}, u'event': u'BITROT_SCRUB_OPTION', u'ts': 1476334930, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'} ====================================================================================================================== {u'message': {u'name': u'dist', u'value': u'resume'}, u'event': u'BITROT_SCRUB_OPTION', u'ts': 1476334994, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'} ======================================================================================================================