Bug 1384031

Summary: Openstack components uses in-process cache instead of memcache
Product: Red Hat OpenStack Reporter: Marian Krcmarik <mkrcmari>
Component: puppet-tripleoAssignee: Dan Prince <dprince>
Status: CLOSED ERRATA QA Contact: Omri Hochman <ohochman>
Severity: high Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: dbecker, dprince, jjoyce, jschluet, jslagle, mburns, mcornea, morazi, rhel-osp-director-maint, sclewis, slinaber, tvignaud
Target Milestone: rcKeywords: Regression, Triaged
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: puppet-tripleo-5.3.0-3.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-14 16:16:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marian Krcmarik 2016-10-12 11:58:11 UTC
Description of problem:
Openstack components use in-process cache instead of memcache, in comparison with RHOS9 This may be considered as a regression since memcache cache was used in RHOS9, I noticed that because token authorization often failed for api calls and I can see in nova/neutron logs something like:
2016-10-12 00:13:03.324 5285 WARNING keystonemiddleware.auth_token [-] Using the in-process token cache is deprecated as of the 4.2.0 release and may be removed in the 5.0.0 release or the 'O' development cycle.
 The in-process cache causes inconsistent results and high memory usage. When the feature is removed the auth_token middleware will not cache tokens by default which may result in performance issues. It is recom
mended to use  memcache for the auth_token token cache by setting the memcached_servers option.

The memcache_servers option is not configured in keystone_authtoken section of config files, I added that option manually and memcache was in use for tone authorization. This option is configured in this config section in RHOS9.

I am not sure though why this changed in RHOS10

Version-Release number of selected component (if applicable):
openstack-ceilometer-common-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch
openstack-cinder-9.0.0-0.20160928223334.ab95181.el7ost.noarch
openstack-nova-novncproxy-14.0.0-0.20160929203854.59653c6.el7ost.noarch
openstack-ceilometer-compute-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch
openstack-zaqar-3.0.0-0.20160921221617.3ef0881.el7ost.noarch
openstack-neutron-openvswitch-9.0.0-0.20160929051647.71f2d2b.el7ost.noarch
python-openstack-mistral-3.0.0-0.20160929083341.c0a4501.el7ost.noarch
openstack-sahara-engine-5.0.0-0.20160926213141.cbd51fa.el7ost.noarch
openstack-neutron-bigswitch-lldp-7.0.5-0.20160927101710.06e6160.el7ost.noarch
openstack-gnocchi-carbonara-3.0.1-0.20160923180636.c6b2c51.el7ost.noarch
openstack-swift-plugin-swift3-1.11.1-0.20160929001717.e7a2b88.el7ost.noarch
openstack-nova-api-14.0.0-0.20160929203854.59653c6.el7ost.noarch
openstack-aodh-notifier-3.0.0-0.20160921151816.bb5103e.el7ost.noarch
openstack-ironic-conductor-6.2.1-0.20160930163405.3f54fec.el7ost.noarch
openstack-mistral-api-3.0.0-0.20160929083341.c0a4501.el7ost.noarch
openstack-manila-share-3.0.0-0.20160916162617.8f2fa31.el7ost.noarch
openstack-puppet-modules-9.0.0-0.20160915155755.8c758d6.el7ost.noarch
openstack-dashboard-10.0.0-0.20161002185148.3252153.1.el7ost.noarch
openstack-ceilometer-notification-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch
openstack-neutron-9.0.0-0.20160929051647.71f2d2b.el7ost.noarch
openstack-gnocchi-indexer-sqlalchemy-3.0.1-0.20160923180636.c6b2c51.el7ost.noarch
openstack-gnocchi-statsd-3.0.1-0.20160923180636.c6b2c51.el7ost.noarch
openstack-swift-account-2.10.1-0.20160929005314.3349016.el7ost.noarch
openstack-neutron-lbaas-9.0.0-0.20160921180958.6528738.el7ost.noarch
openstack-nova-console-14.0.0-0.20160929203854.59653c6.el7ost.noarch
openstack-glance-13.0.0-0.20160928121721.4404ae6.el7ost.noarch
openstack-ceilometer-api-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch
openstack-aodh-evaluator-3.0.0-0.20160921151816.bb5103e.el7ost.noarch
openstack-manila-ui-2.5.1-0.20160929180323.81c354a.el7ost.noarch
openstack-mistral-executor-3.0.0-0.20160929083341.c0a4501.el7ost.noarch
openstack-heat-api-cloudwatch-7.0.0-0.20160926200847.dd707bc.el7ost.noarch
openstack-manila-3.0.0-0.20160916162617.8f2fa31.el7ost.noarch
openstack-neutron-metering-agent-9.0.0-0.20160929051647.71f2d2b.el7ost.noarch
puppet-openstack_extras-9.4.0-1.el7ost.noarch
python-openstacksdk-0.9.5-0.20160912180601.d7ee3ad.el7ost.noarch
openstack-ironic-common-6.2.1-0.20160930163405.3f54fec.el7ost.noarch
openstack-dashboard-theme-10.0.0-0.20161002185148.3252153.1.el7ost.noarch
openstack-ceilometer-polling-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch
openstack-nova-common-14.0.0-0.20160929203854.59653c6.el7ost.noarch
openstack-gnocchi-common-3.0.1-0.20160923180636.c6b2c51.el7ost.noarch
openstack-gnocchi-metricd-3.0.1-0.20160923180636.c6b2c51.el7ost.noarch
openstack-swift-container-2.10.1-0.20160929005314.3349016.el7ost.noarch
openstack-swift-proxy-2.10.1-0.20160929005314.3349016.el7ost.noarch
openstack-nova-scheduler-14.0.0-0.20160929203854.59653c6.el7ost.noarch
openstack-nova-conductor-14.0.0-0.20160929203854.59653c6.el7ost.noarch
openstack-ceilometer-central-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch
openstack-aodh-listener-3.0.0-0.20160921151816.bb5103e.el7ost.noarch
openstack-sahara-ui-5.0.0-0.20160927074353.fdd3a75.el7ost.noarch
openstack-mistral-engine-3.0.0-0.20160929083341.c0a4501.el7ost.noarch
openstack-heat-api-cfn-7.0.0-0.20160926200847.dd707bc.el7ost.noarch
python-django-openstack-auth-2.4.1-0.20160927170809.55ebf6b.el7ost.noarch
openstack-neutron-sriov-nic-agent-9.0.0-0.20160929051647.71f2d2b.el7ost.noarch
python-openstackclient-3.2.0-0.20160914003636.8241f08.el7ost.noarch
openstack-sahara-common-5.0.0-0.20160926213141.cbd51fa.el7ost.noarch
openstack-neutron-bigswitch-agent-7.0.5-0.20160927101710.06e6160.el7ost.noarch
openstack-aodh-common-3.0.0-0.20160921151816.bb5103e.el7ost.noarch
openstack-gnocchi-api-3.0.1-0.20160923180636.c6b2c51.el7ost.noarch
openstack-swift-object-2.10.1-0.20160929005314.3349016.el7ost.noarch
openstack-nova-cert-14.0.0-0.20160929203854.59653c6.el7ost.noarch
openstack-aodh-api-3.0.0-0.20160921151816.bb5103e.el7ost.noarch
openstack-ironic-api-6.2.1-0.20160930163405.3f54fec.el7ost.noarch
openstack-heat-api-7.0.0-0.20160926200847.dd707bc.el7ost.noarch
puppet-openstacklib-9.4.0-0.20160929212001.0e58c86.el7ost.noarch
openstack-heat-common-7.0.0-0.20160926200847.dd707bc.el7ost.noarch
openstack-neutron-common-9.0.0-0.20160929051647.71f2d2b.el7ost.noarch
openstack-nova-compute-14.0.0-0.20160929203854.59653c6.el7ost.noarch
openstack-keystone-10.0.0-0.20160928144040.6520523.el7ost.noarch
openstack-ceilometer-collector-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch
openstack-sahara-5.0.0-0.20160926213141.cbd51fa.el7ost.noarch
openstack-heat-engine-7.0.0-0.20160926200847.dd707bc.el7ost.noarch
openstack-selinux-0.7.9-1.el7ost.noarch
openstack-neutron-ml2-9.0.0-0.20160929051647.71f2d2b.el7ost.noarch
openstack-mistral-common-3.0.0-0.20160929083341.c0a4501.el7ost.noarch
openstack-sahara-api-5.0.0-0.20160926213141.cbd51fa.el7ost.noarch


How reproducible:
Always

Steps to Reproduce:
1. Install openstack RHOS10
2. Pefroem some API calls

Actual results:
in-process cache used

Expected results:
memcache cache used

Additional info:

Comment 1 James Slagle 2016-10-14 15:56:39 UTC
dan, any chance you could take a look at this one? I wasn't sure if it might be related to some of the refactorings we've done in the templates around memcached.

Comment 2 Dan Prince 2016-10-14 19:07:25 UTC
I did some checking and I think I found the Nova regression. In http://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=0ef28cef1f1d97b89ad45c878c0c523c44a32f45 we dropped the ::nova::mechached_servers puppet setting so it no longer configures auth_token for nova. I will get an upstream patch pushed to resolve this for newton.

Marian: I was unable to find any record of setting auth_token/memcached settings for neutron in TripleO however. Could you confirm that OSP 9 did in fact set this for neutron as well?

Regardless, I can go ahead and push the fixes to Newton (OSP 10) to set memcached_servers for both Neutron and Nova to resolve this regression.

Comment 3 Marian Krcmarik 2016-10-14 20:05:42 UTC
(In reply to Dan Prince from comment #2)
> 
> Marian: I was unable to find any record of setting auth_token/memcached
> settings for neutron in TripleO however. Could you confirm that OSP 9 did in
> fact set this for neutron as well?
Hm, Actually I cannot confirm that, I cannot see it configured for neutron on OSP9, only for nova, but the log does complain about in-process cache to be deprecated and suggest to move to memcache so eventually we will have to change that. But the regression in behaviour is only for nova in comparison with RHOS9.

Comment 4 Dan Prince 2016-10-17 18:59:37 UTC
Testing an upstream patch for this here:

https://review.openstack.org/#/c/387624/

Comment 5 James Slagle 2016-10-20 12:42:21 UTC
dan, looks like the master patch merged, but it needs to be backported to stable/newton. It looks like it is merge conflicting during the backport from the gerrit ui, so can you have a look?

Comment 6 Dan Prince 2016-10-21 17:51:32 UTC
This was merged to stable/newton here:

https://review.openstack.org/#/c/389176/

Comment 11 errata-xmlrpc 2016-12-14 16:16:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html