Bug 1146470 - The memories are exhausted quickly when handle the message which has multi fragments in a single record
Summary: The memories are exhausted quickly when handle the message which has multi fr...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: rpc
Version: 3.4.5
Hardware: All
OS: Linux
unspecified
low
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact:
URL:
Whiteboard:
Depends On: 1139598
Blocks: glusterfs-3.4.6
TreeView+ depends on / blocked
 
Reported: 2014-09-25 10:03 UTC by Niels de Vos
Modified: 2015-04-13 07:04 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 1139598
Environment:
Last Closed: 2015-04-13 07:04:45 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Niels de Vos 2014-09-25 10:03:18 UTC
+++ This bug was initially created as a clone of Bug #1139598 +++
+++ This bug was initially created as a clone of Bug #1136221 +++
+++                                                           +++
+++ Use this bug to backport the change to release-3.4        +++

Description of problem:
    We construct some rpc messages and send it to the IP and port which glusterfsd listens, the memory usage goes up quickly until exhausted

Version-Release number of selected component (if applicable):
    3.3.0, 3.4.1, 3.5.0


Steps to Reproduce:
1. Start glusterfs services, and get the IP and port that one glusterfsd process listens

2. Run the attachement python script, which connects the IP and port and send four bytes 00 00 00 00 to the glusterfsd process

3. Watch the memory usage of the glusterfsd process. It will grow up quickly

Actual results:
   Memory of the glusterfsd process grows up quickly till exhausted

Expected results:
   Glusterfsd just ignores the messages


Additional info:
   The bug seems in __socket_proto_state_machine, which goes into an infinite loop to malloc memories when handle the special message. The special message is "multi fragments in a single record", and some values are not reset when handle next fragment.
  
   We tested below fix and it seems work:
          if (!RPC_LASTFRAG (in->fraghdr)) {
 
+             in->pending_vector = in->vector;
+             in->pending_vector->iov_base =  &in->fraghdr;
+             in->pending_vector->iov_len  = sizeof (in->fraghdr);
              in->record_state = SP_STATE_READING_FRAGHDR;
              break;
           }

--- Additional comment from jiangkai on 2014-09-04 06:35:44 EDT ---

More issues than imaging to handle the "multi fragments in a single record" message. The proposal is to refuse it:


 if (!RPC_LASTFRAG (in->fraghdr)) {
       gf_log (this->name, GF_LOG_ERROR, "multiple fragments per record not supported now");
       ret = -1;
       goto out;
 }

--- Additional comment from jiangkai on 2014-09-05 04:45:08 EDT ---

It happens after 3.4;  
3.3.1 reports error messages.

It seems imported by the change id Icd9f256bb2fd8c6266a7abefdff16936b4f8922d to support SSL

--- Additional comment from Anand Avati on 2014-09-09 04:30:12 EDT ---

REVIEW: http://review.gluster.org/8662 (socket: Fixed parsing RPC records containing multi fragments) posted (#1) for review on master by Gu Feng (flygoast)

--- Additional comment from Anand Avati on 2014-09-09 12:03:17 CEST ---

REVIEW: http://review.gluster.org/8662 (socket: Fixed parsing RPC records containing multi fragments) posted (#2) for review on master by Gu Feng (flygoast)

--- Additional comment from Anand Avati on 2014-09-13 18:27:24 CEST ---

REVIEW: http://review.gluster.org/8662 (socket: Fixed parsing RPC records containing multi fragments) posted (#3) for review on master by Gu Feng (flygoast)

--- Additional comment from Anand Avati on 2014-09-19 14:37:09 CEST ---

COMMIT: http://review.gluster.org/8662 committed in master by Raghavendra G (rgowdapp) 
------
commit fb6702b7f8ba19333b7ba4af543d908e3f5e1923
Author: Gu Feng <flygoast>
Date:   Tue Sep 9 18:00:22 2014 +0800

    socket: Fixed parsing RPC records containing multi fragments
    
    In __socket_proto_state_machine(), when parsing RPC records containing
    multi fragments, just change the state of parsing process, had not
    processed the memory to coalesce the multi fragments.
    
    Change-Id: I5583e578603bd7290814a5d26885b31759c73115
    BUG: 1139598
    Signed-off-by: Gu Feng <flygoast>
    Reviewed-on: http://review.gluster.org/8662
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Niels de Vos <ndevos>
    Reviewed-by: Raghavendra G <rgowdapp>
    Tested-by: Raghavendra G <rgowdapp>

Comment 1 Anand Avati 2014-09-25 10:09:52 UTC
REVIEW: http://review.gluster.org/8849 (socket: Fixed parsing RPC records containing multi fragments) posted (#1) for review on release-3.4 by Niels de Vos (ndevos)

Comment 2 Anand Avati 2014-09-25 10:57:00 UTC
COMMIT: http://review.gluster.org/8849 committed in release-3.4 by Kaleb KEITHLEY (kkeithle) 
------
commit ab0547eba39b155246f0d9f09e9a580665c6053f
Author: Niels de Vos <ndevos>
Date:   Thu Sep 25 12:08:47 2014 +0200

    socket: Fixed parsing RPC records containing multi fragments
    
    In __socket_proto_state_machine(), when parsing RPC records containing
    multi fragments, just change the state of parsing process, had not
    processed the memory to coalesce the multi fragments.
    
    Cherry picked from commit fb6702b7f8ba19333b7ba4af543d908e3f5e1923:
    > Change-Id: I5583e578603bd7290814a5d26885b31759c73115
    > BUG: 1139598
    > Signed-off-by: Gu Feng <flygoast>
    > Reviewed-on: http://review.gluster.org/8662
    > Tested-by: Gluster Build System <jenkins.com>
    > Reviewed-by: Niels de Vos <ndevos>
    > Reviewed-by: Raghavendra G <rgowdapp>
    > Tested-by: Raghavendra G <rgowdapp>
    
    Change-Id: I5583e578603bd7290814a5d26885b31759c73115
    BUG: 1146470
    Signed-off-by: Niels de Vos <ndevos>
    Reviewed-on: http://review.gluster.org/8849
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Kaleb KEITHLEY <kkeithle>


Note You need to log in before you can comment on or make changes to this bug.