Bug 1449169
Summary: | Multiple bricks WILL crash after TCP port probing | ||
---|---|---|---|
Product: | [Community] GlusterFS | Reporter: | Milind Changire <mchangir> |
Component: | rpc | Assignee: | bugs <bugs> |
Status: | CLOSED CURRENTRELEASE | QA Contact: | |
Severity: | high | Docs Contact: | |
Priority: | unspecified | ||
Version: | 3.10 | CC: | amukherj, bugs, kaushal, khiremat, nbalacha, ndevos, oleksandr, pkarampu, rcyriac, rgowdapp, rtalur, skoduri, skyler.vock |
Target Milestone: | --- | Keywords: | Triaged |
Target Release: | --- | ||
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | glusterfs-3.10.2 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | 1438966 | Environment: | |
Last Closed: | 2017-05-31 20:46:29 UTC | Type: | Bug |
Regression: | --- | Mount Type: | fuse |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1438966 | ||
Bug Blocks: | 1442603 |
Description
Milind Changire
2017-05-09 11:08:36 UTC
REVIEW: https://review.gluster.org/17217 (rpc: fix transport add/remove race on port probing) posted (#1) for review on release-3.10 by Milind Changire (mchangir) COMMIT: https://review.gluster.org/17217 committed in release-3.10 by Raghavendra Talur (rtalur) ------ commit eb7597b1a20c04a7fd017f7b0f620a5d11eb2769 Author: Milind Changire <mchangir> Date: Tue May 9 17:02:27 2017 +0530 rpc: fix transport add/remove race on port probing Problem: Spurious __gf_free() assertion failures seen all over the place with header->magic being overwritten when running port probing tests with 'nmap' Solution: Fix sequence of: 1. add accept()ed socket connection fd to epoll set 2. add newly created rpc_transport_t object in RPCSVC service list Correct sequence is #2 followed by #1. Reason: Adding new fd returned by accept() to epoll set causes an epoll_wait() to return immediately with a POLLIN event. This races ahead to a readv() which returms with errno:104 (Connection reset by peer) during port probing using 'nmap'. The error is then handled by POLLERR code to remove the new transport object from RPCSVC service list and later unref and destroy the rpc transport object. socket_server_event_handler() then catches up with registering the unref'd/destroyed rpc transport object. This is later manifest as assertion failures in __gf_free() with the header->magic field botched due to invalid address references. All this does not result in a Segmentation Fault since the address space continues to be mapped into the process and pages still being referenced elsewhere. As a further note: This race happens only in accept() codepath. Only in this codepath, the notify will be referring to two transports: 1, listener transport and 2. newly accepted transport All other notify refer to only one transport i.e., the transport/socket on which the event is received. Since epoll is ONE_SHOT another event won't arrive on the same socket till the current event is processed. However, in the accept() codepath, the current event - ACCEPT - and the new event - POLLIN/POLLER - arrive on two different sockets: 1. ACCEPT on listener socket and 2. POLLIN/POLLERR on newly registered socket. Also, note that these two events are handled different thread contexts. Cleanup: Critical section in socket_server_event_handler() has been removed. Instead, an additional ref on new_trans has been used to avoid ref/unref race when notifying RPCSVC. mainline: > BUG: 1438966 > Signed-off-by: Milind Changire <mchangir> > Reviewed-on: https://review.gluster.org/17139 > Smoke: Gluster Build System <jenkins.org> > NetBSD-regression: NetBSD Build System <jenkins.org> > CentOS-regression: Gluster Build System <jenkins.org> > Reviewed-by: Amar Tumballi <amarts> > Reviewed-by: Oleksandr Natalenko <oleksandr> > Reviewed-by: Jeff Darcy <jeff.us> (cherry picked from commit 4f7ef3020edcc75cdeb22d8da8a1484f9db77ac9) Change-Id: I4417924bc9e6277d24bd1a1c5bcb7445bcb226a3 BUG: 1449169 Signed-off-by: Milind Changire <mchangir> Reviewed-on: https://review.gluster.org/17217 Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Raghavendra G <rgowdapp> This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.2, please open a new bug report. |