Bug 1221938

Summary: SIGNING FAILURE Error messages are poping up in the bitd log
Product: [Community] GlusterFS Reporter: RajeshReddy <rmekala>
Component: bitrotAssignee: Venky Shankar <vshankar>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: unspecified Docs Contact: bugs <bugs>
Priority: unspecified    
Version: mainlineCC: bugs, mzywusko, rmekala, vshankar
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.8rc2 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1224165 1224650 (view as bug list) Environment:
Last Closed: 2016-06-16 13:02:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1224650    
Attachments:
Description Flags
logs none

Description RajeshReddy 2015-05-15 09:59:32 UTC 
Description of problem:
========================
SIGNING FAILURE  Error messages  are poping up in the bitd log

Version-Release number of selected component (if applicable):
==================================
glusterfs-server-3.7.0-2.el6rhs.x86_64

How reproducible:


Steps to Reproduce:
1.Create disribute volume with bricks from two nodes
2.Create direcotry and do some IO and observe the bitd log and able to see follwoing messages 

2015-05-15 05:21:13.362936] W [client-rpc-fops.c:1871:client3_3_fsetxattr_cbk] 0-vol2-client-1: remote operation failed: Invalid argument
[2015-05-15 05:21:13.363083] E [bit-rot.c:446:br_object_read_sign] 0-vol2-bit-rot-0: fsetxattr of signature to the object e2b130b3-8d40-491a-910f-9313effa0157 failed
[2015-05-15 05:21:13.363159] E [bit-rot.c:641:br_sign_object] 0-vol2-bit-rot-0: reading and signing of the object e2b130b3-8d40-491a-910f-9313effa0157 failed
[2015-05-15 05:21:13.363268] E [bit-rot.c:697:br_process_object] 0-vol2-bit-rot-0: SIGNING FAILURE [e2b130b3-8d40-491a-910f-9313effa0157]
[2015-05-15 05:22:35.450467] W [client-rpc-fops.c:1871:client3_3_fsetxattr_cbk] 0-vol2-client-1: remote operation failed: Invalid argument
[2015-05-15 05:22:35.450589] E [bit-rot.c:446:br_object_read_sign] 0-vol2-bit-rot-0: fsetxattr of signature to the object e2b130b3-8d40-491a-910f-9313effa0157 failed
[2015-05-15 05:22:35.450663] E [bit-rot.c:641:br_sign_object] 0-vol2-bit-rot-0: reading and signing of the object e2b130b3-8d40-491a-910f-9313effa0157 failed
[2015-05-15 05:22:35.450803] E [bit-rot.c:697:br_process_object] 0-vol2-bit-rot-0: SIGNING FAILURE [e2b130b3-8d40-491a-910f-9313effa0157]
[2015-05-15 05:23:57.539250] W [client-rpc-fops.c:1871:client3_3_fsetxattr_cbk] 0-vol2-client-1: remote operation failed: Invalid argument
[2015-05-15 05:23:57.539385] E [bit-rot.c:446:br_object_read_sign] 0-vol2-bit-rot-0: fsetxattr of signature to the object e2b130b3-8d40-491a-910f-9313effa0157 failed
[2015-05-15 05:23:57.539455] E [bit-rot.c:641:br_sign_object] 0-vol2-bit-rot-0: reading and signing of the object e2b130b3-8d40-491a-910f-9313effa0157 failed
[2015-05-15 05:23:57.539595] E [bit-rot.c:697:br_process_object] 0-vol2-bit-rot-0: SIGNING FAILURE [e2b130b3-8d40-491a-910f-9313effa0157]


Actual results:


Expected results:


Additional info:
Comment 1 Venky Shankar 2015-05-18 10:09:54 UTC 
Were there any unlinks/rmdirs in the file tree?
Comment 2 Venky Shankar 2015-05-18 10:17:03 UTC 
and please check (and paste/upload) brick logs.
Comment 3 RajeshReddy 2015-05-18 11:39:15 UTC 
Created attachment 1026668 [details]
logs

There were no unlinks/rmdirs and attaching the brick logs
Comment 4 Anand Avati 2015-05-19 16:07:08 UTC 
REVIEW: http://review.gluster.org/10832 (features/bitrot: serialize versioning) posted (#1) for review on master by Venky Shankar (vshankar)
Comment 5 Anand Avati 2015-05-29 04:49:47 UTC 
REVIEW: http://review.gluster.org/10832 (Current signing interface (fsetxattr()) had couple of issues:) posted (#3) for review on master by Venky Shankar (vshankar)
Comment 6 Anand Avati 2015-05-29 05:06:57 UTC 
REVIEW: http://review.gluster.org/10832 (features/bitrot: serialize versioning) posted (#4) for review on master by Venky Shankar (vshankar)
Comment 7 Anand Avati 2015-05-30 08:50:36 UTC 
REVIEW: http://review.gluster.org/10832 (features/bitrot: serialize versioning) posted (#5) for review on master by Venky Shankar (vshankar)
Comment 8 Anand Avati 2015-05-31 07:20:58 UTC 
COMMIT: http://review.gluster.org/10832 committed in master by Venky Shankar (vshankar) 
------
commit 457edddbaae62d6956c37b6d1fd45c75ae2c7cff
Author: Venky Shankar <vshankar>
Date:   Fri May 29 10:00:13 2015 +0530

    features/bitrot: serialize versioning
    
    Current signing interface (fsetxattr()) had couple of issues:
    
    One, a signing request (by bitrot daemon) is denied if the version
    against which an object is to be signed is unequal to the current
    version of the object (cases where another subsequent modification
    increments the version). Such request(s) are rejected with EINVAL
    sent back to the signer resulting in a bunch of errors (in logs)
    reported by bitrot daemon. Although, the object would be eventaully
    signed with the version matching the current version, the "lagging"
    request should be correctly handled.
    
    Two, more than one signing request could race against each other
    with the object getting signed with a version depending on which
    request ended up last in the race. Although harmless to some extent,
    such a case could end up marking the object's signature as stale
    for infinity (if the object is *never* touched) thereby resulting
    in scrubber skipping the object during verification.
    
    This patch fixes these issues by ordering signing request(s) and
    fixing version comparison checks at the time of signing.
    
    Change-Id: I9fa83dfa3be664ba4db61d7f2edc408f4bde77dd
    BUG: 1221938
    Signed-off-by: Venky Shankar <vshankar>
    Reviewed-on: http://review.gluster.org/10832
    Tested-by: Gluster Build System <jenkins.com>
Comment 9 Nagaprasad Sathyanarayana 2015-10-25 14:41:38 UTC 
Fix for this BZ is already present in a GlusterFS release. You can find clone of this BZ, fixed in a GlusterFS release and closed. Hence closing this mainline BZ as well.
Comment 10 Niels de Vos 2016-06-16 13:02:03 UTC 
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user