Description of problem:

nova_api fails to connect to the memcached server when using the dogpile.cache.pymemcache backend (the only one which supports TLS)


Version-Release number of selected component (if applicable):

python3-eventlet-0.25.1-6.1.el8ost.1.noarch


How reproducible:

Create a deployment of openstack via tripleo, enabling TLS with the following additional settings:

parameter_defaults:
    ExtraConfig:
        NetworkerExtraConfig:
            neutron::rpc_response_timeout: '150'
        ceilometer::cache_backend: dogpile.cache.pymemcace
        heat::cache::backend: dogpile.cache.pymemcache
        heat::rpc_response_timeout: '600'
        ironic::rpc_response_timeout: '600'
        keystone::cache::backend: dogpile.cache.pymemcache
        keystone::cache_enabled: true
        keystone::cache_memcache_servers:
        - controller-0.internalapi.redhat.local:11212
        - controller-1.internalapi.redhat.local:11212
        - controller-2.internalapi.redhat.local:11212
        keystone::tls_enabled: true
        memcached_authtoken_port: 11211
        memcached_node_ips:
        - controller-0.internalapi.redhat.local
        - controller-1.internalapi.redhat.local
        - controller-2.internalapi.redhat.local
        memcached_port: 11212
        neutron::rpc_response_timeout: '600'
        nova::cache::backend: dogpile.cache.pymemcache
        nova::rpc_response_timeout: '600'
        tripleo::haproxy::haproxy_default_timeout:
        - http-request 60s
        - queue 200s
        - connect 15s
        - client 200s
        - server 200s
        - check 15s
    MemcachedPort: 11212
    MemcachedTLS: true



Steps to Reproduce:
1. Enable TLS everywhere 
2. Start the overcloud deployment with the above environment
3. The deployment fails
4. Run "openstack compute service list"
5. Check the nova_api log on one of the controllers

Actual results:

The deployment fails with a timeout "Waiting for nova-compute service to register"

"openstack compute service list" returns empty, even though the services are registered in the compute_nodes db table


When the above is run, the nova_api logs contain a traceback:

 f285bbd2-60ee-4b91-ac1b-2f242d252160: ValueError: check_hostname needs server_hostname argument                                                                                                            
2021-05-27 17:16:01.766 12 ERROR nova.context Traceback (most recent call last):
2021-05-27 17:16:01.766 12 ERROR nova.context   File "/usr/lib/python3.6/site-packages/nova/context.py", line 422, in gather_result                                                                          
2021-05-27 17:16:01.766 12 ERROR nova.context     result = fn(*args, **kwargs)
2021-05-27 17:16:01.766 12 ERROR nova.context   File "/usr/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper                                                                  
2021-05-27 17:16:01.766 12 ERROR nova.context     result = fn(cls, context, *args, **kwargs)
2021-05-27 17:16:01.766 12 ERROR nova.context   File "/usr/lib/python3.6/site-packages/nova/objects/service.py", line 616, in get_all                                                                        
2021-05-27 17:16:01.766 12 ERROR nova.context     context, db_services)
2021-05-27 17:16:01.766 12 ERROR nova.context   File "/usr/lib/python3.6/site-packages/nova/availability_zones.py", line 88, in set_availability_zones                                                      
2021-05-27 17:16:01.766 12 ERROR nova.context     service['host'], az)
2021-05-27 17:16:01.766 12 ERROR nova.context   File "/usr/lib/python3.6/site-packages/nova/availability_zones.py", line 108, in update_host_availability_zone_cache                                        
2021-05-27 17:16:01.766 12 ERROR nova.context     cache.delete(cache_key)
2021-05-27 17:16:01.766 12 ERROR nova.context   File "/usr/lib/python3.6/site-packages/nova/cache_utils.py", line 122, in delete                                                                            
2021-05-27 17:16:01.766 12 ERROR nova.context     return self.region.delete(key)
2021-05-27 17:16:01.766 12 ERROR nova.context   File "/usr/lib/python3.6/site-packages/dogpile/cache/region.py", line 1367, in delete                                                                        
2021-05-27 17:16:01.766 12 ERROR nova.context     self.backend.delete(key)
2021-05-27 17:16:01.766 12 ERROR nova.context   File "/usr/lib/python3.6/site-packages/dogpile/cache/backends/memcached.py", line 212, in delete                                                            
2021-05-27 17:16:01.766 12 ERROR nova.context     self.client.delete(key)
2021-05-27 17:16:01.766 12 ERROR nova.context   File "/usr/lib/python3.6/site-packages/pymemcache/client/hash.py", line 446, in delete                                                                      
2021-05-27 17:16:01.766 12 ERROR nova.context     return self._run_cmd('delete', key, False, *args, **kwargs)                                                                                              
2021-05-27 17:16:01.766 12 ERROR nova.context   File "/usr/lib/python3.6/site-packages/pymemcache/client/hash.py", line 335, in _run_cmd                                                                    
2021-05-27 17:16:01.766 12 ERROR nova.context     client, func, default_val, *args, **kwargs
2021-05-27 17:16:01.766 12 ERROR nova.context   File "/usr/lib/python3.6/site-packages/pymemcache/client/hash.py", line 214, in _safely_run_func                                                            
2021-05-27 17:16:01.766 12 ERROR nova.context     result = func(*args, **kwargs)
2021-05-27 17:16:01.766 12 ERROR nova.context   File "/usr/lib/python3.6/site-packages/pymemcache/client/base.py", line 608, in delete                                                                      
2021-05-27 17:16:01.766 12 ERROR nova.context     results = self._misc_cmd([cmd], b'delete', noreply)
2021-05-27 17:16:01.766 12 ERROR nova.context   File "/usr/lib/python3.6/site-packages/pymemcache/client/base.py", line 999, in _misc_cmd                                                                    
2021-05-27 17:16:01.766 12 ERROR nova.context     self._connect()
2021-05-27 17:16:01.766 12 ERROR nova.context   File "/usr/lib/python3.6/site-packages/pymemcache/client/base.py", line 336, in _connect                                                                    
2021-05-27 17:16:01.766 12 ERROR nova.context     sock.connect(sockaddr)
2021-05-27 17:16:01.766 12 ERROR nova.context   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 374, in connect                                                                          
2021-05-27 17:16:01.766 12 ERROR nova.context     self.do_handshake()
2021-05-27 17:16:01.766 12 ERROR nova.context   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 301, in do_handshake                                                                    
2021-05-27 17:16:01.766 12 ERROR nova.context     super(GreenSSLSocket, self).do_handshake)
2021-05-27 17:16:01.766 12 ERROR nova.context   File "/usr/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining                                                              
2021-05-27 17:16:01.766 12 ERROR nova.context     return func(*a, **kw)
2021-05-27 17:16:01.766 12 ERROR nova.context   File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake                                                                                              
2021-05-27 17:16:01.766 12 ERROR nova.context     self._sslobj.do_handshake()
2021-05-27 17:16:01.766 12 ERROR nova.context   File "/usr/lib64/python3.6/ssl.py", line 651, in do_handshake                                                                                                
2021-05-27 17:16:01.766 12 ERROR nova.context     raise ValueError("check_hostname needs server_hostname "                                                                                                  
2021-05-27 17:16:01.766 12 ERROR nova.context ValueError: check_hostname needs server_hostname argument


Expected results:

The deployment succeeds, "openstack compute service list" returns non-empty


Additional info:

The issue is fixed by applying the following patch:

diff --git a/eventlet/green/ssl.py b/eventlet/green/ssl.py
index 53a0fa6db..3a5692d6d 100644
--- a/eventlet/green/ssl.py
+++ b/eventlet/green/ssl.py
@@ -364,7 +364,7 @@ def connect(self, addr):
                 sslobj = self._context._wrap_socket(self._sock, server_side, ssl_sock=self)
             else:
                 context = self.context if PY33 else self._context
-                sslobj = context._wrap_socket(self, server_side)
+                sslobj = context._wrap_socket(self, server_side, server_hostname=self.server_hostname)
         else:
             sslobj = sslwrap(self._sock, server_side, self.keyfile, self.certfile,
                              self.cert_reqs, self.ssl_version,

The above was only fixed on the master branch of eventlet (see issue links).