When using encrypted transport, the RPC clients in certain cases not reconnect. This happens mainly during the initial connection establishment phase. If a given hostname has multiple resolved addresses or when using multiple volfile servers, if the first address is down, the client will not attempt reconnects with the next addresses. This happens because in such situations the rpc layer tries to use epoll for the encrypted connections, which don't work together, instead of using own-threads.
REVIEW: http://review.gluster.org/14224 (socket: use own threads for all encrypted connections) posted (#1) for review on master by Kaushal M (kaushal)
REVIEW: http://review.gluster.org/14224 (socket: use own threads for all encrypted connections) posted (#2) for review on master by Kaushal M (kaushal)
REVIEW: http://review.gluster.org/14253 (glusterfsd: explicitly turn on encryption for volfile fetch) posted (#1) for review on master by Kaushal M (kaushal)
REVIEW: http://review.gluster.org/14254 (protocol/client: explicitly specify encryption for portmap) posted (#1) for review on master by Kaushal M (kaushal)
REVIEW: http://review.gluster.org/14224 (socket: use own threads for all encrypted connections) posted (#3) for review on master by Kaushal M (kaushal)
REVIEW: http://review.gluster.org/14254 (protocol/client: explicitly specify encryption for portmap) posted (#2) for review on master by Kaushal M (kaushal)
REVIEW: http://review.gluster.org/14253 (glusterfsd: explicitly turn on encryption for volfile fetch) posted (#2) for review on master by Kaushal M (kaushal)
COMMIT: http://review.gluster.org/14253 committed in master by Jeff Darcy (jdarcy) ------ commit 60d235515e582319474ba7231aad490d19240642 Author: Kaushal M <kaushal> Date: Thu May 5 14:19:55 2016 +0530 glusterfsd: explicitly turn on encryption for volfile fetch Change-Id: I58e1fe7f5edf0abb5732432291ff677e81429b79 BUG: 1333317 Signed-off-by: Kaushal M <kaushal> Reviewed-on: http://review.gluster.org/14253 Smoke: Gluster Build System <jenkins.com> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.com> Reviewed-by: Jeff Darcy <jdarcy>
REVIEW: http://review.gluster.org/14224 (socket: use own threads for all encrypted connections) posted (#4) for review on master by Kaushal M (kaushal)
REVIEW: http://review.gluster.org/14254 (protocol/client: explicitly specify encryption for portmap) posted (#3) for review on master by Kaushal M (kaushal)
(In reply to Kaushal from comment #0) > When using encrypted transport, the RPC clients in certain cases not > reconnect. > > This happens mainly during the initial connection establishment phase. If a > given hostname has multiple resolved addresses or when using multiple > volfile servers, if the first address is down, the client will not attempt > reconnects with the next addresses. > > This happens because in such situations the rpc layer tries to use epoll for > the encrypted connections, which don't work together, instead of using > own-threads. These reconnection failures are more visible when IPv6 addresses get resolved for a given hostname. IPv6 resolution was enabled in glusterfs-3.8, by having getaddrifo use AF_UNSPEC. Normal configuration for getaddrinfo() gives higher preference to IPv6 addresses in this case. So a rpc_clnt will try to connect to the IPv6 address first. But glusterfs rpcsvc listeners, listen on 0.0.0.0, which only listens on IPv4 addresses. This causes rpc_clnt connections to fail initially. But a reconnection should be triggered, which will use the next returned IPv4 address, and successfully connect. This reconnection happens for non-encrypted connections, but fails for encrypted connections in a similar manner to what was described before. This issue was reported by Michael Wyraz on the gluster-devel mailing list. [1] Fixing the reconnection issues, will solve this issue as well. This can also be solved by having rpcsvc listen on IPv6 addresses. A temporary workaround to this would be give a higher preference to IPv4 addresses by editing /etc/gai.conf and adding the line `precedence ::ffff:0:0/96 100`. Or the /etc/hosts file can be edited to remove the IPv6 address for localhost. [1]: https://www.gluster.org/pipermail/gluster-devel/2016-June/049833.html
REVIEW: http://review.gluster.org/14224 (socket: use own threads for all encrypted connections) posted (#5) for review on master by Kaushal M (kaushal)
REVIEW: http://review.gluster.org/14254 (protocol/client: explicitly specify encryption for portmap) posted (#4) for review on master by Kaushal M (kaushal)
REVIEW: http://review.gluster.org/14877 (socket: Remove own-thread references from the socket code and use ssl enable option to use socket_poller code.This commit makes all encrypted connections use socket_poller, and removes the ablitily to configure encrypted connections to use epoll. All references to own-thread has been removed from the code.The socket code now requires all users to explicitly set enable encryption on the RCP clients and servers. It no longer tries to guess if a connection needs to use encryption or not.) posted (#1) for review on master by MOHIT AGRAWAL (moagrawa)
REVIEW: http://review.gluster.org/14224 (socket: use own threads for all encrypted connections) posted (#6) for review on master by Kaushal M (kaushal)
REVIEW: http://review.gluster.org/14254 (protocol/client: explicitly specify encryption for portmap) posted (#5) for review on master by Kaushal M (kaushal)
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.9.0, please open a new bug report. glusterfs-3.9.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/gluster-users/2016-November/029281.html [2] https://www.gluster.org/pipermail/gluster-users/