Bug 866459
| Summary: | "Invalid Arguments" error messages seen in glustershd.log file when volume heal command is executed. | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | spandura |
| Component: | glusterfs | Assignee: | Pranith Kumar K <pkarampu> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | spandura |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | 2.0 | CC: | grajaiya, rhs-bugs, vbellur |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-08-10 07:45:36 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
spandura
2012-10-15 12:20:06 UTC
[2012-10-15 17:51:12.813975] E [afr-self-heal-algorithm.c:609:sh_diff_checksum_cbk] 0-replicate-replicate-0: checksum on <gfid:c8587f1f-7ff7-42b8-a001-d269ccf10611> failed on subvolume replicate-client-0 (Invalid argument) [2012-10-15 17:51:12.918598] W [client3_1-fops.c:2141:client3_1_rchecksum_cbk] 0-replicate-client-1: remote operation failed: Invalid argument [2012-10-15 17:51:12.918639] E [afr-self-heal-algorithm.c:609:sh_diff_checksum_cbk] 0-replicate-replicate-0: checksum on <gfid:91c4970e-f1d5-42bf-83d4-cae8766b061e> failed on subvolume replicate-client-1 (Invalid argument) [10/15/12 - 17:54:46 root@rhs-client7 ~]# getfattr -d -e hex -m . /disk0/59615823-6bad-406c-b9f5-f02cfc8d6103/images/f6f2f7d5-a272-47a4-b343-9bfbb65a5b05/9a4e65b2-66a9-43a2-bcb7-4ea7100604d4 getfattr: Removing leading '/' from absolute path names # file: disk0/59615823-6bad-406c-b9f5-f02cfc8d6103/images/f6f2f7d5-a272-47a4-b343-9bfbb65a5b05/9a4e65b2-66a9-43a2-bcb7-4ea7100604d4 trusted.afr.replicate-client-0=0x0004093e0000000000000000 trusted.afr.replicate-client-1=0x000000000000000000000000 trusted.gfid=0x91c4970ef1d542bf83d4cae8766b061e [10/15/12 - 17:55:50 root@rhs-client6 ~]# getfattr -d -e hex -m . /disk0/59615823-6bad-406c-b9f5-f02cfc8d6103/images/f6f2f7d5-a272-47a4-b343-9bfbb65a5b05/9a4e65b2-66a9-43a2-bcb7-4ea7100604d4 getfattr: Removing leading '/' from absolute path names # file: disk0/59615823-6bad-406c-b9f5-f02cfc8d6103/images/f6f2f7d5-a272-47a4-b343-9bfbb65a5b05/9a4e65b2-66a9-43a2-bcb7-4ea7100604d4 trusted.afr.replicate-client-0=0x000000010000000000000000 trusted.afr.replicate-client-1=0x000000010000000000000000 trusted.gfid=0x91c4970ef1d542bf83d4cae8766b061e does it depend on 827334 ? Amar, It should not. The issue does not happen always, most probably remote_fd must have gone bad and server must be the xl that is sending this EINVAL. I am waiting for the logs to confirm that. Pranith The invalid argument error is stemming from pread of posix_rchecksum. [2012-10-15 17:49:10.048155] W [posix.c:3901:posix_rchecksum] 0-replicate-posix: pread of 131072 bytes returned -1 (Invalid argument) [2012-10-15 17:49:10.048236] I [server3_1-fops.c:1498:server_rchecksum_cbk] 0-replicate-server: 338279: RCHECKSUM 3 (c8587f1f-7ff7-42b8-a001-d269ccf10611)==> -1 (Invalid argument) One of the reasons pread can give EINVAL is if the offset is -ve. But it is not the case here. So I need to figure out what else could be the reason. Breakpoint 3, posix_rchecksum (frame=0x7fd08a7c5cb8, this=0x10e3120, fd=0x1103a10, offset=<value optimized out>, len=131072, xdata=<value optimized out>)
at posix.c:3899
3899 gf_log (this->name, GF_LOG_WARNING,
Missing separate debuginfos, use: debuginfo-install libgcc-4.4.6-3.el6.x86_64
(gdb) bt
#0 posix_rchecksum (frame=0x7fd08a7c5cb8, this=0x10e3120, fd=0x1103a10, offset=<value optimized out>, len=131072, xdata=<value optimized out>) at posix.c:3899
#1 0x00000037a5a1bb9c in default_rchecksum (frame=<value optimized out>, this=0x10e4940, fd=0x1103a10, offset=131072, len=131072, xdata=0x0) at defaults.c:1182
#2 0x00000037a5a1bb9c in default_rchecksum (frame=<value optimized out>, this=0x10e5a40, fd=0x1103a10, offset=131072, len=131072, xdata=0x0) at defaults.c:1182
#3 0x00007fd086bbe15c in iot_rchecksum_wrapper (frame=<value optimized out>, this=0x10e6bc0, fd=0x1103a10, offset=131072, len=131072, xdata=0x0) at io-threads.c:2315
#4 0x00000037a5a2ed3c in call_resume_wind (stub=0x7fd08a48068c) at call-stub.c:2698
#5 call_resume (stub=0x7fd08a48068c) at call-stub.c:4151
#6 0x00007fd086bc6f1b in iot_worker (data=0x10f8de0) at io-threads.c:131
#7 0x000000322da077f1 in start_thread () from /lib64/libpthread.so.0
#8 0x000000322d2e5ccd in clone () from /lib64/libc.so.6
(gdb) info registers
rax 0xffffffffffffffff -1
rbx 0x7fd08a7c5cb8 140533653331128
rcx 0xffffffffffffffa8 -88
rdx 0x16 22
rsi 0x7fd068002910 140533074766096
rdi 0x0 0
rbp 0x10e3120 0x10e3120
rsp 0x7fd08413ac40 0x7fd08413ac40
r8 0xc 12
r9 0x9 9
r10 0x20000 131072
r11 0x0 0
r12 0x7fd068002910 140533074766096
r13 0x1103a10 17840656
r14 0xffffffff 4294967295
r15 0x20000 131072
rip 0x7fd08740aee8 0x7fd08740aee8 <posix_rchecksum+632>
eflags 0x286 [ PF SF IF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
Thanks to the tip from Jeff Darcy. Problem seems to be because of O_DIRECT and posix-aio. Here is the simpler test case that re-creates the problem.
Steps:
1) same volume configuration as above.
2) dd if=/dev/urandom of=a bs=1M count=1
3) kill one of the bricks in the replica pair
4) dd if=/dev/urandom of=a bs=1M count=1
5) bring back the brick. Self-heal will keep failing saying EINVAL in rchecksum.
I made the following change in posix_rchecksum before pread and it works fine.
pranithk @ pranithk-laptop ~/workspace/rhs-glusterfs
19:34:18 :) ⚡ git diff
diff --git a/xlators/storage/posix/src/posix.c b/xlators/storage/posix/src/posix.c
index 5260ff3..cf2b5fb 100644
--- a/xlators/storage/posix/src/posix.c
+++ b/xlators/storage/posix/src/posix.c
@@ -3871,6 +3871,7 @@ posix_rchecksum (call_frame_t *frame, xlator_t *this,
int ret = 0;
int32_t weak_checksum = 0;
unsigned char strong_checksum[MD5_DIGEST_LENGTH];
+ int flags = 0;
VALIDATE_OR_GOTO (frame, out);
VALIDATE_OR_GOTO (this, out);
@@ -3892,6 +3893,8 @@ posix_rchecksum (call_frame_t *frame, xlator_t *this,
goto out;
}
+ flags = fcntl (pfd->fd, F_GETFL);
+ ret = fcntl (pfd->fd, F_SETFL, (flags & (~O_DIRECT)));
_fd = pfd->fd;
ret = pread (_fd, buf, len, offset);
The problem happens because self-heal does rchecksum, readv, then writev in that order so before doing rchecksum necessary o-direct flag manipulation is not done, like the ones in posix-io readv/writev. So pread in rchecksum may give EINVAL if o-direct is set in the previous iteration of readv/writev.
While this confirms the theory, I don't think this is an elegant solution. I will have to discuss this with a few people before sending the patch.
CHANGE: http://review.gluster.org/4134 (storage/posix: Make rchecksum O_DIRECT friendly) merged in master by Vijay Bellur (vbellur) Verified the bug with glusterfs-3.3.0rhsvirt1-8.el6rhs.x86_64. |