Bug 1396290

Summary:	Memcached address in Swift object-expirer.conf gets set to localhost instead of the hosts where memcached is running
Product:	Red Hat OpenStack	Reporter:	Marius Cornea <mcornea>
Component:	puppet-swift	Assignee:	Christian Schwede (cschwede) <cschwede>
Status:	CLOSED ERRATA	QA Contact:	Mike Abrams <mabrams>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	10.0 (Newton)	CC:	cschwede, dbecker, ipilcher, jjoyce, jschluet, jslagle, mburns, morazi, pgrist, rhel-osp-director-maint, sacpatil, slinaber, tvignaud
Target Milestone:	Upstream M3
Target Release:	11.0 (Ocata)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	puppet-swift-10.0.0-0.20161209220524.9e6fff2.el7ost	Doc Type:	No Doc Update
Doc Text:	undefined	Story Points:	---
Clone Of:
Clones:	1500607 (view as bug list)		Environment:
Last Closed:	2017-05-17 19:47:08 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1500607

Description Marius Cornea 2016-11-17 21:44:28 UTC

Description of problem:
Memcached address in Swift object-expirer.conf gets set to localhost instead of the hosts where memcached is running:

[root@overcloud-serviceapi-0 ~]# grep memcache_servers /etc/swift/object-expirer.conf 
memcache_servers = 127.0.0.1:11211

compared to proxy-server.conf:

[root@overcloud-serviceapi-0 ~]# grep memcache_servers /etc/swift/proxy-server.conf 
memcache_servers = 10.0.0.13:11211,10.0.0.14:11211,10.0.0.16:11211


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-5.0.0-1.7.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy overcloud with Swift services running on a different node than where memcached is running

Actual results:
memcache_servers in object-expirer.conf points to the lopback address

Expected results:
memcache_servers in object-expirer.conf points to the addresses where the memcached service is running

Additional info:

Comment 1 Christian Schwede (cschwede) 2016-11-18 09:19:48 UTC

Confirmed. The setting is the default shipped with the rpm; puppet-swift has no option yet to set this properly.

Worst case is that expired objects are not deleted automatically. However, they will be unaccessible after they expired (as expected).

Comment 2 Christian Schwede (cschwede) 2016-11-18 09:35:48 UTC

Quickly checked this; even with a wrong memcached server the object-expirer will delete the expired objects. However, it's slower because of the non-working cache.

Setting this to medium prio.

Nov 18 09:26:02 saio object-expirer: STDERR: ERROR:root:Error connecting to memcached: 127.0.0.1:11212#012Traceback (most recent call last):#012  File "/vagrant/swift/swift/common/memcached.py", line 214, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/vagrant/swift/swift/common/memcached.py", line 132, in get#012    fp, sock = self.create()#012  File "/vagrant/swift/swift/common/memcached.py", line 125, in create#012    sock.connect(sockaddr)#012  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 242, in connect#012    socket_checkerr(fd)#012  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx6a9d8df18e534d2184e5b-00582ec92a)
Nov 18 09:26:02 saio object-expirer: STDERR: ERROR:root:Error limiting server 127.0.0.1:11212 (txn: tx6a9d8df18e534d2184e5b-00582ec92a)
Nov 18 09:26:02 saio object-expirer: Run end (txn: tx939b68ecf13040569bc58-00582ec92a)
Nov 18 09:26:02 saio object-expirer: Pass completed in 0s; 1 objects expired (txn: tx939b68ecf13040569bc58-00582ec92a)
Nov 18 09:26:02 saio object-expirer: Exited (txn: tx939b68ecf13040569bc58-00582ec92a)

Comment 3 Christian Schwede (cschwede) 2016-11-30 07:35:21 UTC

Required patches to fix this are proposed for review:

https://review.openstack.org/377304
https://review.openstack.org/404598

Comment 8 Christian Schwede (cschwede) 2017-03-07 14:28:57 UTC

TL;DR: that looks good to me.

I just checked a default UC, afaict there is no process that uses expiring objects at all. Therefore the background process is basically doing nothing at all.

Even if the process would be used, the setting is correct - the UC only consists of the localhost, and the memcached only listens on 127.0.0.1:

memcach+ 13166  0.0  0.0 487612  9808 ?        Ssl  Mar07   0:25 /usr/bin/memcached -p 11211 -u memcached -m 6005 -c 8192 -l 127.0.0.1 -U 11211 -t 6 >> /var/log/memcached.log 2>&1

Therefore Swift must connect to 127.0.0.1. So both settings are fine IMO.

Comment 10 errata-xmlrpc 2017-05-17 19:47:08 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245