Created attachment 933620 [details] python test.py IP PORT; The IP and port are which the glusterfsd process listens on Description of problem: We construct some rpc messages and send it to the IP and port which glusterfsd listens, the memory usage goes up quickly until exhausted Version-Release number of selected component (if applicable): 3.3.0, 3.4.1, 3.5.0 Steps to Reproduce: 1. Start glusterfs services, and get the IP and port that one glusterfsd process listens 2. Run the attachement python script, which connects the IP and port and send four bytes 00 00 00 00 to the glusterfsd process 3. Watch the memory usage of the glusterfsd process. It will grow up quickly Actual results: Memory of the glusterfsd process grows up quickly till exhausted Expected results: Glusterfsd just ignores the messages Additional info: The bug seems in __socket_proto_state_machine, which goes into an infinite loop to malloc memories when handle the special message. The special message is "multi fragments in a single record", and some values are not reset when handle next fragment. We tested below fix and it seems work: if (!RPC_LASTFRAG (in->fraghdr)) { + in->pending_vector = in->vector; + in->pending_vector->iov_base = &in->fraghdr; + in->pending_vector->iov_len = sizeof (in->fraghdr); in->record_state = SP_STATE_READING_FRAGHDR; break; }
More issues than imaging to handle the "multi fragments in a single record" message. The proposal is to refuse it: if (!RPC_LASTFRAG (in->fraghdr)) { gf_log (this->name, GF_LOG_ERROR, "multiple fragments per record not supported now"); ret = -1; goto out; }
It happens after 3.4; 3.3.1 reports error messages. It seems imported by the change id Icd9f256bb2fd8c6266a7abefdff16936b4f8922d to support SSL
REVIEW: http://review.gluster.org/8662 (socket: Fixed parsing RPC records containing multi fragments) posted (#1) for review on master by Gu Feng (flygoast)
http://review.gluster.org/8662 is for mainline bug 1139598, we can backport the change when it has been merged in the master branch.
REVIEW: http://review.gluster.org/8848 (socket: Fixed parsing RPC records containing multi fragments) posted (#1) for review on release-3.5 by Niels de Vos (ndevos)
COMMIT: http://review.gluster.org/8848 committed in release-3.5 by Niels de Vos (ndevos) ------ commit f206732faafa44b9224f21e6e959b6c3f54e5b34 Author: Niels de Vos <ndevos> Date: Thu Sep 25 11:40:08 2014 +0200 socket: Fixed parsing RPC records containing multi fragments In __socket_proto_state_machine(), when parsing RPC records containing multi fragments, just change the state of parsing process, had not processed the memory to coalesce the multi fragments. Cherry picked from commit fb6702b7f8ba19333b7ba4af543d908e3f5e1923: > Change-Id: I5583e578603bd7290814a5d26885b31759c73115 > BUG: 1139598 > Signed-off-by: Gu Feng <flygoast> > Reviewed-on: http://review.gluster.org/8662 > Tested-by: Gluster Build System <jenkins.com> > Reviewed-by: Niels de Vos <ndevos> > Reviewed-by: Raghavendra G <rgowdapp> > Tested-by: Raghavendra G <rgowdapp> Change-Id: I5583e578603bd7290814a5d26885b31759c73115 BUG: 1136221 Signed-off-by: Niels de Vos <ndevos> Reviewed-on: http://review.gluster.org/8848 Tested-by: Gluster Build System <jenkins.com>
The first (and last?) Beta for GlusterFS 3.5.3 has been released [1]. Please verify if the release solves this bug report for you. In case the glusterfs-3.5.3beta1 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED. Packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update (possibly an "updates-testing" repository) infrastructure for your distribution. [1] http://supercolony.gluster.org/pipermail/gluster-users/2014-October/018990.html [2] http://supercolony.gluster.org/pipermail/gluster-users/
The second Beta for GlusterFS 3.5.3 has been released [1]. Please verify if the release solves this bug report for you. In case the glusterfs-3.5.3beta2 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED. Packages for several distributions have been made available on [2] to make testing easier. [1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019359.html [2] http://download.gluster.org/pub/gluster/glusterfs/qa-releases/3.5.3beta2/
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.5.3, please reopen this bug report. glusterfs-3.5.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://supercolony.gluster.org/pipermail/announce/2014-November/000042.html [2] http://supercolony.gluster.org/pipermail/gluster-users/