Description of problem: Memcached address in Swift object-expirer.conf gets set to localhost instead of the hosts where memcached is running: [root@overcloud-serviceapi-0 ~]# grep memcache_servers /etc/swift/object-expirer.conf memcache_servers = 127.0.0.1:11211 compared to proxy-server.conf: [root@overcloud-serviceapi-0 ~]# grep memcache_servers /etc/swift/proxy-server.conf memcache_servers = 10.0.0.13:11211,10.0.0.14:11211,10.0.0.16:11211 Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-5.0.0-1.7.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy overcloud with Swift services running on a different node than where memcached is running Actual results: memcache_servers in object-expirer.conf points to the lopback address Expected results: memcache_servers in object-expirer.conf points to the addresses where the memcached service is running Additional info:
Confirmed. The setting is the default shipped with the rpm; puppet-swift has no option yet to set this properly. Worst case is that expired objects are not deleted automatically. However, they will be unaccessible after they expired (as expected).
Quickly checked this; even with a wrong memcached server the object-expirer will delete the expired objects. However, it's slower because of the non-working cache. Setting this to medium prio. Nov 18 09:26:02 saio object-expirer: STDERR: ERROR:root:Error connecting to memcached: 127.0.0.1:11212#012Traceback (most recent call last):#012 File "/vagrant/swift/swift/common/memcached.py", line 214, in _get_conns#012 fp, sock = self._client_cache[server].get()#012 File "/vagrant/swift/swift/common/memcached.py", line 132, in get#012 fp, sock = self.create()#012 File "/vagrant/swift/swift/common/memcached.py", line 125, in create#012 sock.connect(sockaddr)#012 File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 242, in connect#012 socket_checkerr(fd)#012 File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012 raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx6a9d8df18e534d2184e5b-00582ec92a) Nov 18 09:26:02 saio object-expirer: STDERR: ERROR:root:Error limiting server 127.0.0.1:11212 (txn: tx6a9d8df18e534d2184e5b-00582ec92a) Nov 18 09:26:02 saio object-expirer: Run end (txn: tx939b68ecf13040569bc58-00582ec92a) Nov 18 09:26:02 saio object-expirer: Pass completed in 0s; 1 objects expired (txn: tx939b68ecf13040569bc58-00582ec92a) Nov 18 09:26:02 saio object-expirer: Exited (txn: tx939b68ecf13040569bc58-00582ec92a)
Required patches to fix this are proposed for review: https://review.openstack.org/377304 https://review.openstack.org/404598
TL;DR: that looks good to me. I just checked a default UC, afaict there is no process that uses expiring objects at all. Therefore the background process is basically doing nothing at all. Even if the process would be used, the setting is correct - the UC only consists of the localhost, and the memcached only listens on 127.0.0.1: memcach+ 13166 0.0 0.0 487612 9808 ? Ssl Mar07 0:25 /usr/bin/memcached -p 11211 -u memcached -m 6005 -c 8192 -l 127.0.0.1 -U 11211 -t 6 >> /var/log/memcached.log 2>&1 Therefore Swift must connect to 127.0.0.1. So both settings are fine IMO.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1245