+++ This bug was initially created as a clone of Bug #1221938 +++ Description of problem: ======================== SIGNING FAILURE Error messages are poping up in the bitd log Version-Release number of selected component (if applicable): ================================== glusterfs-server-3.7.0-2.el6rhs.x86_64 How reproducible: Steps to Reproduce: 1.Create disribute volume with bricks from two nodes 2.Create direcotry and do some IO and observe the bitd log and able to see follwoing messages 2015-05-15 05:21:13.362936] W [client-rpc-fops.c:1871:client3_3_fsetxattr_cbk] 0-vol2-client-1: remote operation failed: Invalid argument [2015-05-15 05:21:13.363083] E [bit-rot.c:446:br_object_read_sign] 0-vol2-bit-rot-0: fsetxattr of signature to the object e2b130b3-8d40-491a-910f-9313effa0157 failed [2015-05-15 05:21:13.363159] E [bit-rot.c:641:br_sign_object] 0-vol2-bit-rot-0: reading and signing of the object e2b130b3-8d40-491a-910f-9313effa0157 failed [2015-05-15 05:21:13.363268] E [bit-rot.c:697:br_process_object] 0-vol2-bit-rot-0: SIGNING FAILURE [e2b130b3-8d40-491a-910f-9313effa0157] [2015-05-15 05:22:35.450467] W [client-rpc-fops.c:1871:client3_3_fsetxattr_cbk] 0-vol2-client-1: remote operation failed: Invalid argument [2015-05-15 05:22:35.450589] E [bit-rot.c:446:br_object_read_sign] 0-vol2-bit-rot-0: fsetxattr of signature to the object e2b130b3-8d40-491a-910f-9313effa0157 failed [2015-05-15 05:22:35.450663] E [bit-rot.c:641:br_sign_object] 0-vol2-bit-rot-0: reading and signing of the object e2b130b3-8d40-491a-910f-9313effa0157 failed [2015-05-15 05:22:35.450803] E [bit-rot.c:697:br_process_object] 0-vol2-bit-rot-0: SIGNING FAILURE [e2b130b3-8d40-491a-910f-9313effa0157] [2015-05-15 05:23:57.539250] W [client-rpc-fops.c:1871:client3_3_fsetxattr_cbk] 0-vol2-client-1: remote operation failed: Invalid argument [2015-05-15 05:23:57.539385] E [bit-rot.c:446:br_object_read_sign] 0-vol2-bit-rot-0: fsetxattr of signature to the object e2b130b3-8d40-491a-910f-9313effa0157 failed [2015-05-15 05:23:57.539455] E [bit-rot.c:641:br_sign_object] 0-vol2-bit-rot-0: reading and signing of the object e2b130b3-8d40-491a-910f-9313effa0157 failed [2015-05-15 05:23:57.539595] E [bit-rot.c:697:br_process_object] 0-vol2-bit-rot-0: SIGNING FAILURE [e2b130b3-8d40-491a-910f-9313effa0157] Actual results: Expected results: Additional info: --- Additional comment from Venky Shankar on 2015-05-18 06:09:54 EDT --- Were there any unlinks/rmdirs in the file tree? --- Additional comment from Venky Shankar on 2015-05-18 06:17:03 EDT --- and please check (and paste/upload) brick logs. --- Additional comment from RajeshReddy on 2015-05-18 07:39:15 EDT --- There were no unlinks/rmdirs and attaching the brick logs --- Additional comment from Anand Avati on 2015-05-19 12:07:08 EDT --- REVIEW: http://review.gluster.org/10832 (features/bitrot: serialize versioning) posted (#1) for review on master by Venky Shankar (vshankar)
REVIEW: http://review.gluster.org/10900 (features/bitrot: serialize versioning) posted (#2) for review on release-3.7 by Venky Shankar (vshankar)
REVIEW: http://review.gluster.org/10900 (features/bitrot: serialize versioning) posted (#3) for review on release-3.7 by Venky Shankar (vshankar)
COMMIT: http://review.gluster.org/10900 committed in release-3.7 by Venky Shankar (vshankar) ------ commit 06710aa1085a7c5f3259af6b63d23ac5f51bef18 Author: Venky Shankar <vshankar> Date: Fri May 29 10:00:13 2015 +0530 features/bitrot: serialize versioning Backport of http://review.gluster.org/10832 Current signing interface (fsetxattr()) had couple of issues: One, a signing request (by bitrot daemon) is denied if the version against which an object is to be signed is unequal to the current version of the object (cases where another subsequent modification increments the version). Such request(s) are rejected with EINVAL sent back to the signer resulting in a bunch of errors (in logs) reported by bitrot daemon. Although, the object would be eventaully signed with the version matching the current version, the "lagging" request should be correctly handled. Two, more than one signing request could race against each other with the object getting signed with a version depending on which request ended up last in the race. Although harmless to some extent, such a case could end up marking the object's signature as stale for infinity (if the object is *never* touched) thereby resulting in scrubber skipping the object during verification. This patch fixes these issues by ordering signing request(s) and fixing version comparison checks at the time of signing. Change-Id: I9fa83dfa3be664ba4db61d7f2edc408f4bde77dd BUG: 1224650 Signed-off-by: Venky Shankar <vshankar> Tested-by: Gluster Build System <jenkins.com> Reviewed-on: http://review.gluster.org/10900 Tested-by: NetBSD Build System <jenkins.org>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.1, please reopen this bug report. glusterfs-3.7.1 has been announced on the Gluster Packaging mailinglist [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.packaging/1 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user