Description of problem: When deploying with IPv6, the Swift proxy server cannot connect to memcached. Version-Release number of selected component (if applicable): OSP 7.3 puddle How reproducible: 100% Steps to Reproduce: 1. Deploy with IPv6 network isolation 2. 3. Actual results: swift-proxy-server.log:2016-02-05 14:06:02.974 28960 TRACE root ValueError: too many values to unpack swift-proxy-server.log:2016-02-05 14:06:02.974 28960 ERROR root [-] Error limiting server [fd00:fd00:fd00:2000::14]:11211 Expected results: Swift should be able to connect to memcached. Additional info: I didn't test Swift proxy for functionality, but I think it's probably bad that it can't connect to memcached. At the very least, I think that will prevent token caching, although there might be more severe consequences.
I found some more data on this issue. Looking more closely at the /var/log/ceilometer/swift-proxy-server.log, I see: 2016-02-06 11:06:03.621 23660 ERROR root [-] Error connecting to memcached: "[fd00:fd00:fd00:2000::15]:11211" 2016-02-06 11:06:03.621 23660 TRACE root Traceback (most recent call last): 2016-02-06 11:06:03.621 23660 TRACE root File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 239, in _get_conns 2016-02-06 11:06:03.621 23660 TRACE root fp, sock = self._client_cache[server].get() 2016-02-06 11:06:03.621 23660 TRACE root File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 135, in get 2016-02-06 11:06:03.621 23660 TRACE root fp, sock = self.create() 2016-02-06 11:06:03.621 23660 TRACE root File "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 121, in create 2016-02-06 11:06:03.621 23660 TRACE root host, port = self.server.split(':') 2016-02-06 11:06:03.621 23660 TRACE root ValueError: too many values to unpack So, looking at "/usr/lib/python2.7/site-packages/swift/common/memcached.py", line 121: 119 def create(self): 120 if ':' in self.server: 121 host, port = self.server.split(':') 122 else: 123 host = self.server 124 port = DEFAULT_MEMCACHED_PORT 125 sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 126 sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) 127 with Timeout(self._connect_timeout): 128 sock.connect((host, int(port))) 129 return (sock.makefile(), sock) Clearly, we can't just do a simple split on ":". This needs to be modified to properly handle the case of a list of bracketed IPv6 addresses. Something like this should work: 119 def create(self): 120 if ']' in self.server: 121 # deconstruct ipv6 address in brackets 122 server_uri = self.server.strip("[").split("]") 123 host = server_uri[0] 124 if len(server_uri) > 1: 125 port = server_uri[1].strip(":") 126 else: 127 port = DEFAULT_MEMCACHED_PORT 128 elif ':' in self.server: 129 host, port = self.server.split(':') 130 else: 131 host = self.server 132 port = DEFAULT_MEMCACHED_PORT 133 sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 134 sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) 135 with Timeout(self._connect_timeout): 136 sock.connect((host, int(port))) 137 return (sock.makefile(), sock)
I tested the above code sample, and though it does work for separating the host from port, the socket connection will have to be IPv6 capable as well.
(In reply to Dan Sneddon from comment #5) > I tested the above code sample, and though it does work for separating the > host from port, the socket connection will have to be IPv6 capable as well. OK, this seems to work (at least I am no longer seeing errors in /var/log/ceilometer/swift-proxy-server.log 119 def create(self): 120 if ']' in self.server: 121 # deconstruct ipv6 address in brackets 122 server_uri = self.server.strip("[").split("]") 123 host = server_uri[0] 124 if len(server_uri) > 1: 125 port = server_uri[1].strip(":") 126 else: 127 port = DEFAULT_MEMCACHED_PORT 128 elif ':' in self.server: 129 host, port = self.server.split(':') 130 else: 131 host = self.server 132 port = DEFAULT_MEMCACHED_PORT 133 if ":" in host: 134 sock = socket.socket(socket.AF_INET6, socket.SOCK_STREAM) 135 else: 136 sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 137 sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) 138 with Timeout(self._connect_timeout): 139 sock.connect((host, int(port))) 140 return (sock.makefile(), sock)
VERIFIED errors do NOT appear in the log files. [root@instack ~]# grep VERSION /etc/*release /etc/os-release:VERSION="7.2 (Maipo)" /etc/os-release:VERSION_ID="7.2" /etc/os-release:REDHAT_BUGZILLA_PRODUCT_VERSION=7.2 /etc/os-release:REDHAT_SUPPORT_PRODUCT_VERSION="7.2" [root@instack ~]# uname -a Linux instack.localdomain 3.10.0-327.4.5.el7.x86_64 #1 SMP Thu Jan 21 04:10:29 EST 2016 x86_64 x86_64 x86_64 GNU/Linux [root@instack ~]# rpm -qa openstack-swift openstack-swift-2.3.0-4.el7ost.noarch [root@instack ~]# see attachments.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0263.html