Bug 1396290 - Memcached address in Swift object-expirer.conf gets set to localhost instead of the hosts where memcached is running
Summary: Memcached address in Swift object-expirer.conf gets set to localhost instead ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-swift
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: Upstream M3
: 11.0 (Ocata)
Assignee: Christian Schwede (cschwede)
QA Contact: Mike Abrams
URL:
Whiteboard:
Depends On:
Blocks: 1500607
TreeView+ depends on / blocked
 
Reported: 2016-11-17 21:44 UTC by Marius Cornea
Modified: 2018-03-30 10:58 UTC (History)
13 users (show)

Fixed In Version: puppet-swift-10.0.0-0.20161209220524.9e6fff2.el7ost
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
: 1500607 (view as bug list)
Environment:
Last Closed: 2017-05-17 19:47:08 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1245 normal SHIPPED_LIVE Red Hat OpenStack Platform 11.0 Bug Fix and Enhancement Advisory 2017-05-17 23:01:50 UTC
OpenStack gerrit 377304 None None None 2016-11-30 06:59:25 UTC
OpenStack gerrit 404598 None None None 2016-11-30 07:33:56 UTC
Launchpad 1627927 None None None 2016-11-29 16:13:16 UTC

Description Marius Cornea 2016-11-17 21:44:28 UTC
Description of problem:
Memcached address in Swift object-expirer.conf gets set to localhost instead of the hosts where memcached is running:

[root@overcloud-serviceapi-0 ~]# grep memcache_servers /etc/swift/object-expirer.conf 
memcache_servers = 127.0.0.1:11211

compared to proxy-server.conf:

[root@overcloud-serviceapi-0 ~]# grep memcache_servers /etc/swift/proxy-server.conf 
memcache_servers = 10.0.0.13:11211,10.0.0.14:11211,10.0.0.16:11211


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-5.0.0-1.7.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy overcloud with Swift services running on a different node than where memcached is running

Actual results:
memcache_servers in object-expirer.conf points to the lopback address

Expected results:
memcache_servers in object-expirer.conf points to the addresses where the memcached service is running

Additional info:

Comment 1 Christian Schwede (cschwede) 2016-11-18 09:19:48 UTC
Confirmed. The setting is the default shipped with the rpm; puppet-swift has no option yet to set this properly.

Worst case is that expired objects are not deleted automatically. However, they will be unaccessible after they expired (as expected).

Comment 2 Christian Schwede (cschwede) 2016-11-18 09:35:48 UTC
Quickly checked this; even with a wrong memcached server the object-expirer will delete the expired objects. However, it's slower because of the non-working cache.

Setting this to medium prio.

Nov 18 09:26:02 saio object-expirer: STDERR: ERROR:root:Error connecting to memcached: 127.0.0.1:11212#012Traceback (most recent call last):#012  File "/vagrant/swift/swift/common/memcached.py", line 214, in _get_conns#012    fp, sock = self._client_cache[server].get()#012  File "/vagrant/swift/swift/common/memcached.py", line 132, in get#012    fp, sock = self.create()#012  File "/vagrant/swift/swift/common/memcached.py", line 125, in create#012    sock.connect(sockaddr)#012  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 242, in connect#012    socket_checkerr(fd)#012  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 46, in socket_checkerr#012    raise socket.error(err, errno.errorcode[err])#012error: [Errno 111] ECONNREFUSED (txn: tx6a9d8df18e534d2184e5b-00582ec92a)
Nov 18 09:26:02 saio object-expirer: STDERR: ERROR:root:Error limiting server 127.0.0.1:11212 (txn: tx6a9d8df18e534d2184e5b-00582ec92a)
Nov 18 09:26:02 saio object-expirer: Run end (txn: tx939b68ecf13040569bc58-00582ec92a)
Nov 18 09:26:02 saio object-expirer: Pass completed in 0s; 1 objects expired (txn: tx939b68ecf13040569bc58-00582ec92a)
Nov 18 09:26:02 saio object-expirer: Exited (txn: tx939b68ecf13040569bc58-00582ec92a)

Comment 3 Christian Schwede (cschwede) 2016-11-30 07:35:21 UTC
Required patches to fix this are proposed for review:

https://review.openstack.org/377304
https://review.openstack.org/404598

Comment 8 Christian Schwede (cschwede) 2017-03-07 14:28:57 UTC
TL;DR: that looks good to me.

I just checked a default UC, afaict there is no process that uses expiring objects at all. Therefore the background process is basically doing nothing at all.

Even if the process would be used, the setting is correct - the UC only consists of the localhost, and the memcached only listens on 127.0.0.1:

memcach+ 13166  0.0  0.0 487612  9808 ?        Ssl  Mar07   0:25 /usr/bin/memcached -p 11211 -u memcached -m 6005 -c 8192 -l 127.0.0.1 -U 11211 -t 6 >> /var/log/memcached.log 2>&1

Therefore Swift must connect to 127.0.0.1. So both settings are fine IMO.

Comment 10 errata-xmlrpc 2017-05-17 19:47:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245


Note You need to log in before you can comment on or make changes to this bug.