Bug 1332129
Summary: | nfs-ganesha might fails to comeup with "Error binding to V6 interface" | |||
---|---|---|---|---|
Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Shashank Raj <sraj> | |
Component: | doc-Administration_Guide | Assignee: | Pratik Mulay <pmulay> | |
doc-Administration_Guide sub component: | 3.2 Release | QA Contact: | Jilju Joy <jijoy> | |
Status: | CLOSED CURRENTRELEASE | Docs Contact: | ||
Severity: | high | |||
Priority: | unspecified | CC: | abhishku, asriram, bkunal, jthottan, kkeithle, msaini, ndevos, nlevinki, pmulay, rhinduja, rhs-bugs, sheggodu, skoduri, sraj, storage-doc | |
Version: | rhgs-3.1 | Keywords: | ZStream | |
Target Milestone: | --- | |||
Target Release: | RHGS 3.4.z Batch Update 4 | |||
Hardware: | x86_64 | |||
OS: | Linux | |||
Whiteboard: | ||||
Fixed In Version: | Doc Type: | Bug Fix | ||
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 1477507 1477511 (view as bug list) | Environment: | ||
Last Closed: | 2019-06-03 05:11:45 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1369781, 1477507, 1477511, 1657798, 1672843 |
Description
Shashank Raj
2016-05-02 10:11:01 UTC
When this happens, make sure to have "netstat" (from the net-utils package) installed before generating the sosreport. With the details from netstat we might be able to see something like a pattern. from the TCP specs - https://tools.ietf.org/html/rfc793#page-22 TIME-WAIT - represents waiting for enough time to pass to be sure the remote TCP received the acknowledgment of its connection termination request. There might have been an issue with GlusterD that prevented it from communication/confirming back that the socket can be closed. It would be helpful to get the status+logs from the system that has its IP listed in the TIME_WAIT connection entry from "netstat". We are hitting this particular issue more frequently now where nfs-ganesha service fails to come up with below messages: 13/05/2016 23:57:50 : epoch c8c00000 : dhcp42-20.lab.eng.blr.redhat.com : ganesha.nfsd-16417[main] Bind_sockets_V6 :DISP :WARN :Cannot bind RQUOTA tcp6 socket, error 98 (Address already in use) 13/05/2016 23:57:50 : epoch c8c00000 : dhcp42-20.lab.eng.blr.redhat.com : ganesha.nfsd-16417[main] Bind_sockets :DISP :FATAL :Error binding to V6 interface. Cannot continue. and in one scenario we have observed that 875 port which we configure for rquota is being used by shd process [root@dhcp42-20 ~]# netstat -ntaun | grep 875 tcp 0 0 10.70.42.20:49212 10.70.43.175:875 ESTABLISHED [root@dhcp42-20 ~]# gluster v status | grep 49212 cks/brick5/nfsvol4_brick0 49212 0 Y 16249 [root@dhcp43-175 ~]# netstat -ntaunp | grep 875 tcp 0 0 10.70.43.175:875 10.70.42.20:49212 ESTABLISHED 7049/glusterfs [root@dhcp43-175 ~]# ps aux|grep glusterfs root 7049 0.0 0.8 1115812 65480 ? Ssl May13 0:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/971aa92e442920e8802d63fd4bd001a5.socket --xlator-option *replicate*.node-uuid=f63c454f-b24e-49f9-b65f-dee681762100 Since we have moved this bug to 3.2.0, we need to propose this for documentation for 3.1.3 @Niels Please let us know youe thoughts on the same. I thought that clients are not binding to privileged (< 1024) ports by default anymore. But if shd is often occupying port 875, this has been reverted maybe? (Older) Gluster clients starts to bind ports from 1024 and iterates downwards. There must have been quite some Gluster clients running when even port 875 is in use already. Can you see what the other ports between 875 - 1024 are used for? The option "client.bind-insecure" is expected to be enabled by default. SHD is expected to use a much higher port number. The changes that come in with glusterfs-3.8.x should prevent clients from using ports < 1024. Please re-test with the latest RHGS-3.2 builds. Thanks! I didn't hit this issue so far during downstream testing of below 3.2 build [root@dhcp43-110 exports]# rpm -qa|grep ganesha nfs-ganesha-debuginfo-2.4.0-2.el7rhgs.x86_64 nfs-ganesha-2.4.0-2.el7rhgs.x86_64 glusterfs-ganesha-3.8.4-2.el7rhgs.x86_64 nfs-ganesha-gluster-2.4.0-2.el7rhgs.x86_64 Will keep this bug open and update if i see it again. This is applicable for all 3.1.3,3.2 and 3.3 The change requested in this bug is already present in https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html-single/administration_guide/#ganesha_Troubleshooting_1 Marking this as verified. The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |