Description of problem: ======================== When we pause/resume the bitrot scrubber, it generates an event BITROT_SCRUB_OPTION, as there is a change in the scrub option. Similarly when we change the frequency to hourly/daily/weekly, it again generates an event BITROT_SCRUB_FREQ, as there is a change in the scrub frequency. Likewise, if we trigger the scrubber on demand, it necessitates a change in the system, thus producing an event. However, no event is generated/seen when the scrubber is triggered ondemand. Version-Release number of selected component (if applicable): ============================================================ 3.8.4-2 How reproducible: ================= Always Steps to Reproduce: ==================== 1. Have a 4 node cluster, with a plain distribute volume. Enable eventing, and register a webhook, as a listener. 2. Create a volume 'vol1' and start it. The corresponding events are seen, as expected 3. Enable bitrot, change the scrub frequency and also try to pause/resume. Watch out for the related events BITROT_ENABLE, BITROT_SCRUB_FREQ, BITROT_SCRUB_OPTION. 4. Trigger the scrubber on demand, and monitor if the event BITROT_SCRUB_OPTION is seen Actual results: ============== All expected events seen from step1 to 3. No event seen at step4 Expected results: ================ An event is expected at step4 Additional info: ================= COMMANDS EXECUTED ON THE SERVER -------------------------------- [root@dhcp35-100 ~]# gluster v list dist distrep [root@dhcp35-100 ~]# [root@dhcp35-100 ~]# [root@dhcp35-100 ~]# [root@dhcp35-100 ~]# gluster v bitrot dist enable volume bitrot: success [root@dhcp35-100 ~]# gluster v info dist Volume Name: dist Type: Distribute Volume ID: 9204b642-b380-4935-bb88-5aaaac1fe3fc Status: Started Snapshot Count: 0 Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: 10.70.35.115:/bricks/brick1/dist1 Brick2: 10.70.35.100:/bricks/brick1/dist2 Options Reconfigured: features.scrub: Active features.bitrot: on transport.address-family: inet performance.readdir-ahead: on nfs.disable: on auto-delete: disable [root@dhcp35-100 ~]# gluster v bitrot Usage: volume bitrot <VOLNAME> {enable|disable} | volume bitrot <volname> scrub-throttle {lazy|normal|aggressive} | volume bitrot <volname> scrub-frequency {hourly|daily|weekly|biweekly|monthly} | volume bitrot <volname> scrub {pause|resume|status|ondemand} [root@dhcp35-100 ~]# gluster v bitrot dist scrub-throttle aggressive volume bitrot: success [root@dhcp35-100 ~]# [root@dhcp35-100 ~]# gluster v bitrot Usage: volume bitrot <VOLNAME> {enable|disable} | volume bitrot <volname> scrub-throttle {lazy|normal|aggressive} | volume bitrot <volname> scrub-frequency {hourly|daily|weekly|biweekly|monthly} | volume bitrot <volname> scrub {pause|resume|status|ondemand} [root@dhcp35-100 ~]# gluster v bitrot dist scrub-frequency minute volume bitrot: success [root@dhcp35-100 ~]# [root@dhcp35-100 ~]# [root@dhcp35-100 ~]# gluster v info dist Volume Name: dist Type: Distribute Volume ID: 9204b642-b380-4935-bb88-5aaaac1fe3fc Status: Started Snapshot Count: 0 Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: 10.70.35.115:/bricks/brick1/dist1 Brick2: 10.70.35.100:/bricks/brick1/dist2 Options Reconfigured: features.scrub-freq: minute features.scrub-throttle: aggressive features.scrub: Active features.bitrot: on transport.address-family: inet performance.readdir-ahead: on nfs.disable: on auto-delete: disable [root@dhcp35-100 ~]# [root@dhcp35-100 ~]# gluster v bitrot Usage: volume bitrot <VOLNAME> {enable|disable} | volume bitrot <volname> scrub-throttle {lazy|normal|aggressive} | volume bitrot <volname> scrub-frequency {hourly|daily|weekly|biweekly|monthly} | volume bitrot <volname> scrub {pause|resume|status|ondemand} [root@dhcp35-100 ~]# [root@dhcp35-100 ~]# [root@dhcp35-100 ~]# gluster v bitrot dist scrub ondemand volume bitrot: success [root@dhcp35-100 ~]# gluster v bitrot dist scrub pause volume bitrot: success [root@dhcp35-100 ~]# gluster v bitrot dist scrub status Volume name : dist State of scrub: pause (Idle) Scrub impact: aggressive Scrub frequency: minute Bitrot error log location: /var/log/glusterfs/bitd.log Scrubber error log location: /var/log/glusterfs/scrub.log ========================================================= Node: localhost Number of Scrubbed files: 0 Number of Skipped files: 0 Last completed scrub time: 2016-10-13 05:01:51 Duration of last scrub (D:M:H:M:S): 0:0:0:0 Error count: 0 ========================================================= Node: dhcp35-115.lab.eng.blr.redhat.com Number of Scrubbed files: 0 Number of Skipped files: 0 Last completed scrub time: 2016-10-13 05:01:52 Duration of last scrub (D:M:H:M:S): 0:0:0:0 Error count: 0 ========================================================= [root@dhcp35-100 ~]# gluster v bitrot dist scrub resume volume bitrot: success [root@dhcp35-100 ~]# gluster v bitrot dist scrub status Volume name : dist State of scrub: Active (Idle) Scrub impact: aggressive Scrub frequency: minute Bitrot error log location: /var/log/glusterfs/bitd.log Scrubber error log location: /var/log/glusterfs/scrub.log ========================================================= Node: localhost Number of Scrubbed files: 0 Number of Skipped files: 0 Last completed scrub time: 2016-10-13 05:01:51 Duration of last scrub (D:M:H:M:S): 0:0:0:0 Error count: 0 ========================================================= Node: dhcp35-115.lab.eng.blr.redhat.com Number of Scrubbed files: 0 Number of Skipped files: 0 Last completed scrub time: 2016-10-13 05:01:52 Duration of last scrub (D:M:H:M:S): 0:0:0:0 Error count: 0 ========================================================= [root@dhcp35-100 ~]# [root@dhcp35-100 ~]# gluster v bitrot dist scrub ondemand volume bitrot: success [root@dhcp35-100 ~]# ------------ EVENTS SEEN ------------ ====================================================================================================================== {u'message': {u'name': u'dist'}, u'event': u'BITROT_ENABLE', u'ts': 1476334598, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'} ====================================================================================================================== {u'message': {u'svc_name': u'bitd'}, u'event': u'SVC_CONNECTED', u'ts': 1476334601, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'} ====================================================================================================================== {u'message': {u'svc_name': u'scrub'}, u'event': u'SVC_CONNECTED', u'ts': 1476334601, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'} ====================================================================================================================== {u'message': {u'svc_name': u'bitd'}, u'event': u'SVC_CONNECTED', u'ts': 1476334601, u'nodeid': u'6ac165c0-317f-42ad-8262-953995171dbb'} ====================================================================================================================== {u'message': {u'svc_name': u'scrub'}, u'event': u'SVC_CONNECTED', u'ts': 1476334601, u'nodeid': u'6ac165c0-317f-42ad-8262-953995171dbb'} ====================================================================================================================== {u'message': {u'name': u'dist', u'value': u'aggressive'}, u'event': u'BITROT_SCRUB_THROTTLE', u'ts': 1476334757, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'} ====================================================================================================================== {u'message': {u'name': u'dist', u'value': u'minute'}, u'event': u'BITROT_SCRUB_FREQ', u'ts': 1476334792, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'} ====================================================================================================================== {u'message': {u'name': u'dist', u'value': u'pause'}, u'event': u'BITROT_SCRUB_OPTION', u'ts': 1476334930, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'} ====================================================================================================================== {u'message': {u'name': u'dist', u'value': u'resume'}, u'event': u'BITROT_SCRUB_OPTION', u'ts': 1476334994, u'nodeid': u'fcfacf2e-57fb-45ba-b1e1-e4ba640a4de5'} ======================================================================================================================
Upstream Patch: http://review.gluster.org/15700 (master)
upstream mainline : http://review.gluster.org/15700 upstream 3.9 : http://review.gluster.org/15705 downstream patch : https://code.engineering.redhat.com/gerrit/89264
Tested and verified this on the build 3.8.4-5 When we execute 'gluster volume bitrot <volname> scrub ondemand' an event of name 'BITROT_SCRUB_ONDEMAND' is seen. Moving this BZ to verified in 3.2. Detailed logs are pasted below: [root@dhcp46-239 ~]# [root@dhcp46-239 ~]# rpm -qa | grep gluster nfs-ganesha-gluster-2.3.1-8.el7rhgs.x86_64 glusterfs-api-3.8.4-5.el7rhgs.x86_64 python-gluster-3.8.4-5.el7rhgs.noarch glusterfs-client-xlators-3.8.4-5.el7rhgs.x86_64 glusterfs-server-3.8.4-5.el7rhgs.x86_64 glusterfs-ganesha-3.8.4-5.el7rhgs.x86_64 gluster-nagios-common-0.2.4-1.el7rhgs.noarch glusterfs-devel-3.8.4-5.el7rhgs.x86_64 gluster-nagios-addons-0.2.8-1.el7rhgs.x86_64 glusterfs-libs-3.8.4-5.el7rhgs.x86_64 glusterfs-fuse-3.8.4-5.el7rhgs.x86_64 glusterfs-api-devel-3.8.4-5.el7rhgs.x86_64 glusterfs-rdma-3.8.4-5.el7rhgs.x86_64 glusterfs-3.8.4-5.el7rhgs.x86_64 glusterfs-cli-3.8.4-5.el7rhgs.x86_64 glusterfs-geo-replication-3.8.4-5.el7rhgs.x86_64 glusterfs-debuginfo-3.8.4-4.el7rhgs.x86_64 glusterfs-events-3.8.4-5.el7rhgs.x86_64 [root@dhcp46-239 ~]# [root@dhcp46-239 ~]# [root@dhcp46-239 ~]# gluster peer status Number of Peers: 3 Hostname: 10.70.46.240 Uuid: 72c4f894-61f7-433e-a546-4ad2d7f0a176 State: Peer in Cluster (Connected) Hostname: 10.70.46.242 Uuid: 1e8967ae-51b2-4c27-907e-a22a83107fd0 State: Peer in Cluster (Connected) Hostname: 10.70.46.218 Uuid: 0dea52e0-8c32-4616-8ef8-16db16120eaa State: Peer in Cluster (Connected) [root@dhcp46-239 ~]# gluster v info Volume Name: ozone Type: Disperse Volume ID: 376cdde0-194f-460a-b273-3904a704a7dd Status: Started Snapshot Count: 0 Number of Bricks: 1 x (4 + 2) = 6 Transport-type: tcp Bricks: Brick1: 10.70.46.239:/bricks/brick0/ozone0 Brick2: 10.70.46.240:/bricks/brick0/ozone2 Brick3: 10.70.46.242:/bricks/brick0/ozone2 Brick4: 10.70.46.239:/bricks/brick1/ozone3 Brick5: 10.70.46.240:/bricks/brick1/ozone4 Brick6: 10.70.46.242:/bricks/brick1/ozone5 Options Reconfigured: features.expiry-time: 5 features.scrub-freq: daily features.scrub: Active features.bitrot: on transport.address-family: inet performance.readdir-ahead: on nfs.disable: on cluster.enable-shared-storage: disable Volume Name: rep3 Type: Distributed-Replicate Volume ID: f29c2892-ac61-4b26-a146-8fa34038eb59 Status: Started Snapshot Count: 0 Number of Bricks: 3 x 3 = 9 Transport-type: tcp Bricks: Brick1: 10.70.46.240:/bricks/brick0/rep30 Brick2: 10.70.46.242:/bricks/brick0/rep31 Brick3: 10.70.46.218:/bricks/brick0/rep32 Brick4: 10.70.46.240:/bricks/brick1/rep33 Brick5: 10.70.46.242:/bricks/brick1/rep34 Brick6: 10.70.46.218:/bricks/brick1/rep35 Brick7: 10.70.46.240:/bricks/brick2/rep36 Brick8: 10.70.46.242:/bricks/brick2/rep37 Brick9: 10.70.46.218:/bricks/brick2/rep38 Options Reconfigured: features.expiry-time: 5 features.scrub-freq: hourly features.scrub: Active features.bitrot: on transport.address-family: inet performance.readdir-ahead: on nfs.disable: on cluster.enable-shared-storage: disable [root@dhcp46-239 ~]# [root@dhcp46-239 ~]# gluster v bitrot rep3 scrub status Volume name : rep3 State of scrub: Active (Idle) Scrub impact: lazy Scrub frequency: hourly Bitrot error log location: /var/log/glusterfs/bitd.log Scrubber error log location: /var/log/glusterfs/scrub.log ========================================================= Node: localhost Number of Scrubbed files: 0 Number of Skipped files: 0 Last completed scrub time: Scrubber pending to complete. Duration of last scrub (D:M:H:M:S): 0:0:0:0 Error count: 0 ========================================================= Node: 10.70.46.242 Number of Scrubbed files: 10 Number of Skipped files: 1 Last completed scrub time: 2016-11-16 06:01:45 Duration of last scrub (D:M:H:M:S): 0:0:0:5 Error count: 0 ========================================================= Node: 10.70.46.240 Number of Scrubbed files: 10 Number of Skipped files: 1 Last completed scrub time: 2016-11-16 06:01:45 Duration of last scrub (D:M:H:M:S): 0:0:0:5 Error count: 0 ========================================================= Node: 10.70.46.218 Number of Scrubbed files: 10 Number of Skipped files: 1 Last completed scrub time: 2016-11-16 06:01:45 Duration of last scrub (D:M:H:M:S): 0:0:0:5 Error count: 0 ========================================================= [root@dhcp46-239 ~]# gluster v bitrot rep3 scrub ondemand volume bitrot: success [root@dhcp46-239 ~]# EVENTS ------- {u'message': {u'name': u'rep3'}, u'event': u'BITROT_SCRUB_ONDEMAND', u'ts': 1479276299, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'}
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2017-0486.html