Bug 1387976 - Continuous warning messages getting when one of the cluster node is down on SSL setup.
Summary: Continuous warning messages getting when one of the cluster node is down on S...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: glusterd
Version: 3.8
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
Assignee: Mohit Agrawal
QA Contact:
URL:
Whiteboard:
Depends On: 1385525
Blocks: 1386450 1387975
TreeView+ depends on / blocked
 
Reported: 2016-10-24 05:39 UTC by Mohit Agrawal
Modified: 2016-12-21 23:20 UTC (History)
5 users (show)

Fixed In Version: glusterfs-3.8.6
Clone Of: 1385525
Environment:
Last Closed: 2016-11-29 09:36:46 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Mohit Agrawal 2016-10-24 05:39:03 UTC
+++ This bug was initially created as a clone of Bug #1385525 +++

Description of problem:
=======================
Getting the below warning messages in glusterd and volume mount logs continuously on SSL SETUP when one of the cluster node glusterd is down/node is down.

Message frequency is very high and if cluster node is down for any known reason for days, it will consume lot of log storage.


[2016-10-17 08:15:55.008898] W [socket.c:590:__socket_rwv] 0-management: readv on 10.70.43.190:24007 failed (No data available)
[2016-10-17 08:15:55.009082] W [socket.c:590:__socket_rwv] 0-management: readv on 10.70.43.190:24007 failed (No data available)
[2016-10-17 08:15:55.009289] W [socket.c:590:__socket_rwv] 0-management: readv on 10.70.43.190:24007 failed (No data available)
[2016-10-17 08:15:55.009546] W [socket.c:590:__socket_rwv] 0-management: readv on 10.70.43.190:24007 failed (No data available)
[2016-10-17 08:15:55.009736] W [socket.c:590:__socket_rwv] 0-management: readv on 10.70.43.190:24007 failed (No data available) 





Version-Release number of selected component (if applicable):
=============================================================
glusterfs-3.8.4-2


How reproducible:
=================
Always


Steps to Reproduce:
===================
1. Have two node cluster 
2. Create a SSL configuration.
3. Create and start any volume type
4. fuse mount the volume.
5. Stop glusterd on vol file server node/ shutdown the node
6. Check the glusterd logs in the peer nodes and the volume mount log for the above mentioned warning messages  


Actual results:
===============
Continuous warning messages getting when one of the cluster node is down on SSL setup.


Expected results:
=================
There should be control on throwing the warning messages or it should not throw the warning messages.


Additional info:
================
I used backup vol file server while mounting the volume then also issue remains same in the volume mount log.

--- Additional comment from Red Hat Bugzilla Rules Engine on 2016-10-17 05:10:35 EDT ---

This bug is automatically being proposed for the current release of Red Hat Gluster Storage 3 under active development, by setting the release flag 'rhgs‑3.2.0' to '?'. 

If this bug should be proposed for a different release, please manually change the proposed release flag.

--- Additional comment from Byreddy on 2016-10-18 01:16:56 EDT ---

This issue is not there in the last GA build.

--- Additional comment from Mohit Agrawal on 2016-10-18 07:26:58 EDT ---

Hi,

  At the time of start glusterd process it returns a port to the other peer node on which it will communicate with glusterd(24007).After stop glusterd on a node means one end point of port is closed and other point is still used by socket(glusterd 24007),glusterd(socket_poller) on other node call's continuously socket_event_poll_in that calls __socket_rwv to read data from socket unless buffer is not clear.
Because other end point of socket is disconnected so it prints the message "0-management: readv on 10.65.7.252:24007 failed (No data available)".

After changed the condition to log the message issue will resolve.


Regards
Mohit Agrawal

Comment 1 Worker Ant 2016-10-24 05:43:19 UTC
REVIEW: http://review.gluster.org/15706 (rpc/socket.c : Modify socket_poller code in case of ENODATA error code.) posted (#1) for review on release-3.8 by MOHIT AGRAWAL (moagrawa)

Comment 2 Worker Ant 2016-10-24 09:14:40 UTC
REVIEW: http://review.gluster.org/15706 (rpc/socket.c : Modify socket_poller code in case of ENODATA error code.) posted (#2) for review on release-3.8 by MOHIT AGRAWAL (moagrawa)

Comment 3 Worker Ant 2016-11-23 11:50:03 UTC
COMMIT: http://review.gluster.org/15706 committed in release-3.8 by Niels de Vos (ndevos) 
------
commit 8cd077a05f5504a0e94a3565ce815bc4cec9972c
Author: Mohit Agrawal <moagrawa>
Date:   Wed Oct 19 07:27:40 2016 +0530

    rpc/socket.c : Modify socket_poller code in case of ENODATA error code.
    
    Problem:  Continuous warning message(ENODATA) are coming in socket_rwv
              while SSL is enabled.
    
    Solution: To avoid the warning message update one condition in
              socket_poller loop code before break from loop in case
              of error returned by poll functions.
    
    > BUG: 1386450
    > Change-Id: I19b3a92d4c3ba380738379f5679c1c354f0ab9b1
    > Signed-off-by: Mohit Agrawal <moagrawa>
    > Reviewed-on: http://review.gluster.org/15677
    > Smoke: Gluster Build System <jenkins.org>
    > NetBSD-regression: NetBSD Build System <jenkins.org>
    > CentOS-regression: Gluster Build System <jenkins.org>
    > Reviewed-by: Raghavendra G <rgowdapp>
    > (cherry picked from commit ec64ce2e1684003f4e7a20d4372e414bfbddb6fb)
    
    Change-Id: I70eaf8d454a1538e14b50c6fb1074f84dd10cdf5
    BUG: 1387976
    Signed-off-by: Mohit Agrawal <moagrawa>
    Reviewed-on: http://review.gluster.org/15706
    Smoke: Gluster Build System <jenkins.org>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Atin Mukherjee <amukherj>
    Reviewed-by: Niels de Vos <ndevos>

Comment 4 Niels de Vos 2016-11-29 09:36:46 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.6, please open a new bug report.

glusterfs-3.8.6 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/packaging/2016-November/000217.html
[2] https://www.gluster.org/pipermail/gluster-users/

Comment 5 Worker Ant 2016-12-21 23:20:50 UTC
REVIEW: http://review.gluster.org/16237 (rpc/socket.c : Modify socket_poller code in case of ENODATA error code.) posted (#1) for review on release-3.8-fb by Kevin Vigor (kvigor)


Note You need to log in before you can comment on or make changes to this bug.