Bug 1384031 - Openstack components uses in-process cache instead of memcache
Summary: Openstack components uses in-process cache instead of memcache
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-tripleo
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: rc
: 10.0 (Newton)
Assignee: Dan Prince
QA Contact: Omri Hochman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-10-12 11:58 UTC by Marian Krcmarik
Modified: 2016-12-14 16:16 UTC (History)
13 users (show)

Fixed In Version: puppet-tripleo-5.3.0-3.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-12-14 16:16:49 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:2948 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC
OpenStack gerrit 389176 None None None 2016-10-20 13:21:54 UTC
Launchpad 1633595 None None None 2016-10-14 19:55:43 UTC

Description Marian Krcmarik 2016-10-12 11:58:11 UTC
Description of problem:
Openstack components use in-process cache instead of memcache, in comparison with RHOS9 This may be considered as a regression since memcache cache was used in RHOS9, I noticed that because token authorization often failed for api calls and I can see in nova/neutron logs something like:
2016-10-12 00:13:03.324 5285 WARNING keystonemiddleware.auth_token [-] Using the in-process token cache is deprecated as of the 4.2.0 release and may be removed in the 5.0.0 release or the 'O' development cycle.
 The in-process cache causes inconsistent results and high memory usage. When the feature is removed the auth_token middleware will not cache tokens by default which may result in performance issues. It is recom
mended to use  memcache for the auth_token token cache by setting the memcached_servers option.

The memcache_servers option is not configured in keystone_authtoken section of config files, I added that option manually and memcache was in use for tone authorization. This option is configured in this config section in RHOS9.

I am not sure though why this changed in RHOS10

Version-Release number of selected component (if applicable):
openstack-ceilometer-common-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch
openstack-cinder-9.0.0-0.20160928223334.ab95181.el7ost.noarch
openstack-nova-novncproxy-14.0.0-0.20160929203854.59653c6.el7ost.noarch
openstack-ceilometer-compute-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch
openstack-zaqar-3.0.0-0.20160921221617.3ef0881.el7ost.noarch
openstack-neutron-openvswitch-9.0.0-0.20160929051647.71f2d2b.el7ost.noarch
python-openstack-mistral-3.0.0-0.20160929083341.c0a4501.el7ost.noarch
openstack-sahara-engine-5.0.0-0.20160926213141.cbd51fa.el7ost.noarch
openstack-neutron-bigswitch-lldp-7.0.5-0.20160927101710.06e6160.el7ost.noarch
openstack-gnocchi-carbonara-3.0.1-0.20160923180636.c6b2c51.el7ost.noarch
openstack-swift-plugin-swift3-1.11.1-0.20160929001717.e7a2b88.el7ost.noarch
openstack-nova-api-14.0.0-0.20160929203854.59653c6.el7ost.noarch
openstack-aodh-notifier-3.0.0-0.20160921151816.bb5103e.el7ost.noarch
openstack-ironic-conductor-6.2.1-0.20160930163405.3f54fec.el7ost.noarch
openstack-mistral-api-3.0.0-0.20160929083341.c0a4501.el7ost.noarch
openstack-manila-share-3.0.0-0.20160916162617.8f2fa31.el7ost.noarch
openstack-puppet-modules-9.0.0-0.20160915155755.8c758d6.el7ost.noarch
openstack-dashboard-10.0.0-0.20161002185148.3252153.1.el7ost.noarch
openstack-ceilometer-notification-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch
openstack-neutron-9.0.0-0.20160929051647.71f2d2b.el7ost.noarch
openstack-gnocchi-indexer-sqlalchemy-3.0.1-0.20160923180636.c6b2c51.el7ost.noarch
openstack-gnocchi-statsd-3.0.1-0.20160923180636.c6b2c51.el7ost.noarch
openstack-swift-account-2.10.1-0.20160929005314.3349016.el7ost.noarch
openstack-neutron-lbaas-9.0.0-0.20160921180958.6528738.el7ost.noarch
openstack-nova-console-14.0.0-0.20160929203854.59653c6.el7ost.noarch
openstack-glance-13.0.0-0.20160928121721.4404ae6.el7ost.noarch
openstack-ceilometer-api-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch
openstack-aodh-evaluator-3.0.0-0.20160921151816.bb5103e.el7ost.noarch
openstack-manila-ui-2.5.1-0.20160929180323.81c354a.el7ost.noarch
openstack-mistral-executor-3.0.0-0.20160929083341.c0a4501.el7ost.noarch
openstack-heat-api-cloudwatch-7.0.0-0.20160926200847.dd707bc.el7ost.noarch
openstack-manila-3.0.0-0.20160916162617.8f2fa31.el7ost.noarch
openstack-neutron-metering-agent-9.0.0-0.20160929051647.71f2d2b.el7ost.noarch
puppet-openstack_extras-9.4.0-1.el7ost.noarch
python-openstacksdk-0.9.5-0.20160912180601.d7ee3ad.el7ost.noarch
openstack-ironic-common-6.2.1-0.20160930163405.3f54fec.el7ost.noarch
openstack-dashboard-theme-10.0.0-0.20161002185148.3252153.1.el7ost.noarch
openstack-ceilometer-polling-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch
openstack-nova-common-14.0.0-0.20160929203854.59653c6.el7ost.noarch
openstack-gnocchi-common-3.0.1-0.20160923180636.c6b2c51.el7ost.noarch
openstack-gnocchi-metricd-3.0.1-0.20160923180636.c6b2c51.el7ost.noarch
openstack-swift-container-2.10.1-0.20160929005314.3349016.el7ost.noarch
openstack-swift-proxy-2.10.1-0.20160929005314.3349016.el7ost.noarch
openstack-nova-scheduler-14.0.0-0.20160929203854.59653c6.el7ost.noarch
openstack-nova-conductor-14.0.0-0.20160929203854.59653c6.el7ost.noarch
openstack-ceilometer-central-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch
openstack-aodh-listener-3.0.0-0.20160921151816.bb5103e.el7ost.noarch
openstack-sahara-ui-5.0.0-0.20160927074353.fdd3a75.el7ost.noarch
openstack-mistral-engine-3.0.0-0.20160929083341.c0a4501.el7ost.noarch
openstack-heat-api-cfn-7.0.0-0.20160926200847.dd707bc.el7ost.noarch
python-django-openstack-auth-2.4.1-0.20160927170809.55ebf6b.el7ost.noarch
openstack-neutron-sriov-nic-agent-9.0.0-0.20160929051647.71f2d2b.el7ost.noarch
python-openstackclient-3.2.0-0.20160914003636.8241f08.el7ost.noarch
openstack-sahara-common-5.0.0-0.20160926213141.cbd51fa.el7ost.noarch
openstack-neutron-bigswitch-agent-7.0.5-0.20160927101710.06e6160.el7ost.noarch
openstack-aodh-common-3.0.0-0.20160921151816.bb5103e.el7ost.noarch
openstack-gnocchi-api-3.0.1-0.20160923180636.c6b2c51.el7ost.noarch
openstack-swift-object-2.10.1-0.20160929005314.3349016.el7ost.noarch
openstack-nova-cert-14.0.0-0.20160929203854.59653c6.el7ost.noarch
openstack-aodh-api-3.0.0-0.20160921151816.bb5103e.el7ost.noarch
openstack-ironic-api-6.2.1-0.20160930163405.3f54fec.el7ost.noarch
openstack-heat-api-7.0.0-0.20160926200847.dd707bc.el7ost.noarch
puppet-openstacklib-9.4.0-0.20160929212001.0e58c86.el7ost.noarch
openstack-heat-common-7.0.0-0.20160926200847.dd707bc.el7ost.noarch
openstack-neutron-common-9.0.0-0.20160929051647.71f2d2b.el7ost.noarch
openstack-nova-compute-14.0.0-0.20160929203854.59653c6.el7ost.noarch
openstack-keystone-10.0.0-0.20160928144040.6520523.el7ost.noarch
openstack-ceilometer-collector-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch
openstack-sahara-5.0.0-0.20160926213141.cbd51fa.el7ost.noarch
openstack-heat-engine-7.0.0-0.20160926200847.dd707bc.el7ost.noarch
openstack-selinux-0.7.9-1.el7ost.noarch
openstack-neutron-ml2-9.0.0-0.20160929051647.71f2d2b.el7ost.noarch
openstack-mistral-common-3.0.0-0.20160929083341.c0a4501.el7ost.noarch
openstack-sahara-api-5.0.0-0.20160926213141.cbd51fa.el7ost.noarch


How reproducible:
Always

Steps to Reproduce:
1. Install openstack RHOS10
2. Pefroem some API calls

Actual results:
in-process cache used

Expected results:
memcache cache used

Additional info:

Comment 1 James Slagle 2016-10-14 15:56:39 UTC
dan, any chance you could take a look at this one? I wasn't sure if it might be related to some of the refactorings we've done in the templates around memcached.

Comment 2 Dan Prince 2016-10-14 19:07:25 UTC
I did some checking and I think I found the Nova regression. In http://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=0ef28cef1f1d97b89ad45c878c0c523c44a32f45 we dropped the ::nova::mechached_servers puppet setting so it no longer configures auth_token for nova. I will get an upstream patch pushed to resolve this for newton.

Marian: I was unable to find any record of setting auth_token/memcached settings for neutron in TripleO however. Could you confirm that OSP 9 did in fact set this for neutron as well?

Regardless, I can go ahead and push the fixes to Newton (OSP 10) to set memcached_servers for both Neutron and Nova to resolve this regression.

Comment 3 Marian Krcmarik 2016-10-14 20:05:42 UTC
(In reply to Dan Prince from comment #2)
> 
> Marian: I was unable to find any record of setting auth_token/memcached
> settings for neutron in TripleO however. Could you confirm that OSP 9 did in
> fact set this for neutron as well?
Hm, Actually I cannot confirm that, I cannot see it configured for neutron on OSP9, only for nova, but the log does complain about in-process cache to be deprecated and suggest to move to memcache so eventually we will have to change that. But the regression in behaviour is only for nova in comparison with RHOS9.

Comment 4 Dan Prince 2016-10-17 18:59:37 UTC
Testing an upstream patch for this here:

https://review.openstack.org/#/c/387624/

Comment 5 James Slagle 2016-10-20 12:42:21 UTC
dan, looks like the master patch merged, but it needs to be backported to stable/newton. It looks like it is merge conflicting during the backport from the gerrit ui, so can you have a look?

Comment 6 Dan Prince 2016-10-21 17:51:32 UTC
This was merged to stable/newton here:

https://review.openstack.org/#/c/389176/

Comment 11 errata-xmlrpc 2016-12-14 16:16:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html


Note You need to log in before you can comment on or make changes to this bug.