Description of problem: Openstack components use in-process cache instead of memcache, in comparison with RHOS9 This may be considered as a regression since memcache cache was used in RHOS9, I noticed that because token authorization often failed for api calls and I can see in nova/neutron logs something like: 2016-10-12 00:13:03.324 5285 WARNING keystonemiddleware.auth_token [-] Using the in-process token cache is deprecated as of the 4.2.0 release and may be removed in the 5.0.0 release or the 'O' development cycle. The in-process cache causes inconsistent results and high memory usage. When the feature is removed the auth_token middleware will not cache tokens by default which may result in performance issues. It is recom mended to use memcache for the auth_token token cache by setting the memcached_servers option. The memcache_servers option is not configured in keystone_authtoken section of config files, I added that option manually and memcache was in use for tone authorization. This option is configured in this config section in RHOS9. I am not sure though why this changed in RHOS10 Version-Release number of selected component (if applicable): openstack-ceilometer-common-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch openstack-cinder-9.0.0-0.20160928223334.ab95181.el7ost.noarch openstack-nova-novncproxy-14.0.0-0.20160929203854.59653c6.el7ost.noarch openstack-ceilometer-compute-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch openstack-zaqar-3.0.0-0.20160921221617.3ef0881.el7ost.noarch openstack-neutron-openvswitch-9.0.0-0.20160929051647.71f2d2b.el7ost.noarch python-openstack-mistral-3.0.0-0.20160929083341.c0a4501.el7ost.noarch openstack-sahara-engine-5.0.0-0.20160926213141.cbd51fa.el7ost.noarch openstack-neutron-bigswitch-lldp-7.0.5-0.20160927101710.06e6160.el7ost.noarch openstack-gnocchi-carbonara-3.0.1-0.20160923180636.c6b2c51.el7ost.noarch openstack-swift-plugin-swift3-1.11.1-0.20160929001717.e7a2b88.el7ost.noarch openstack-nova-api-14.0.0-0.20160929203854.59653c6.el7ost.noarch openstack-aodh-notifier-3.0.0-0.20160921151816.bb5103e.el7ost.noarch openstack-ironic-conductor-6.2.1-0.20160930163405.3f54fec.el7ost.noarch openstack-mistral-api-3.0.0-0.20160929083341.c0a4501.el7ost.noarch openstack-manila-share-3.0.0-0.20160916162617.8f2fa31.el7ost.noarch openstack-puppet-modules-9.0.0-0.20160915155755.8c758d6.el7ost.noarch openstack-dashboard-10.0.0-0.20161002185148.3252153.1.el7ost.noarch openstack-ceilometer-notification-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch openstack-neutron-9.0.0-0.20160929051647.71f2d2b.el7ost.noarch openstack-gnocchi-indexer-sqlalchemy-3.0.1-0.20160923180636.c6b2c51.el7ost.noarch openstack-gnocchi-statsd-3.0.1-0.20160923180636.c6b2c51.el7ost.noarch openstack-swift-account-2.10.1-0.20160929005314.3349016.el7ost.noarch openstack-neutron-lbaas-9.0.0-0.20160921180958.6528738.el7ost.noarch openstack-nova-console-14.0.0-0.20160929203854.59653c6.el7ost.noarch openstack-glance-13.0.0-0.20160928121721.4404ae6.el7ost.noarch openstack-ceilometer-api-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch openstack-aodh-evaluator-3.0.0-0.20160921151816.bb5103e.el7ost.noarch openstack-manila-ui-2.5.1-0.20160929180323.81c354a.el7ost.noarch openstack-mistral-executor-3.0.0-0.20160929083341.c0a4501.el7ost.noarch openstack-heat-api-cloudwatch-7.0.0-0.20160926200847.dd707bc.el7ost.noarch openstack-manila-3.0.0-0.20160916162617.8f2fa31.el7ost.noarch openstack-neutron-metering-agent-9.0.0-0.20160929051647.71f2d2b.el7ost.noarch puppet-openstack_extras-9.4.0-1.el7ost.noarch python-openstacksdk-0.9.5-0.20160912180601.d7ee3ad.el7ost.noarch openstack-ironic-common-6.2.1-0.20160930163405.3f54fec.el7ost.noarch openstack-dashboard-theme-10.0.0-0.20161002185148.3252153.1.el7ost.noarch openstack-ceilometer-polling-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch openstack-nova-common-14.0.0-0.20160929203854.59653c6.el7ost.noarch openstack-gnocchi-common-3.0.1-0.20160923180636.c6b2c51.el7ost.noarch openstack-gnocchi-metricd-3.0.1-0.20160923180636.c6b2c51.el7ost.noarch openstack-swift-container-2.10.1-0.20160929005314.3349016.el7ost.noarch openstack-swift-proxy-2.10.1-0.20160929005314.3349016.el7ost.noarch openstack-nova-scheduler-14.0.0-0.20160929203854.59653c6.el7ost.noarch openstack-nova-conductor-14.0.0-0.20160929203854.59653c6.el7ost.noarch openstack-ceilometer-central-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch openstack-aodh-listener-3.0.0-0.20160921151816.bb5103e.el7ost.noarch openstack-sahara-ui-5.0.0-0.20160927074353.fdd3a75.el7ost.noarch openstack-mistral-engine-3.0.0-0.20160929083341.c0a4501.el7ost.noarch openstack-heat-api-cfn-7.0.0-0.20160926200847.dd707bc.el7ost.noarch python-django-openstack-auth-2.4.1-0.20160927170809.55ebf6b.el7ost.noarch openstack-neutron-sriov-nic-agent-9.0.0-0.20160929051647.71f2d2b.el7ost.noarch python-openstackclient-3.2.0-0.20160914003636.8241f08.el7ost.noarch openstack-sahara-common-5.0.0-0.20160926213141.cbd51fa.el7ost.noarch openstack-neutron-bigswitch-agent-7.0.5-0.20160927101710.06e6160.el7ost.noarch openstack-aodh-common-3.0.0-0.20160921151816.bb5103e.el7ost.noarch openstack-gnocchi-api-3.0.1-0.20160923180636.c6b2c51.el7ost.noarch openstack-swift-object-2.10.1-0.20160929005314.3349016.el7ost.noarch openstack-nova-cert-14.0.0-0.20160929203854.59653c6.el7ost.noarch openstack-aodh-api-3.0.0-0.20160921151816.bb5103e.el7ost.noarch openstack-ironic-api-6.2.1-0.20160930163405.3f54fec.el7ost.noarch openstack-heat-api-7.0.0-0.20160926200847.dd707bc.el7ost.noarch puppet-openstacklib-9.4.0-0.20160929212001.0e58c86.el7ost.noarch openstack-heat-common-7.0.0-0.20160926200847.dd707bc.el7ost.noarch openstack-neutron-common-9.0.0-0.20160929051647.71f2d2b.el7ost.noarch openstack-nova-compute-14.0.0-0.20160929203854.59653c6.el7ost.noarch openstack-keystone-10.0.0-0.20160928144040.6520523.el7ost.noarch openstack-ceilometer-collector-7.0.0-0.20160928024313.67bbd3f.el7ost.noarch openstack-sahara-5.0.0-0.20160926213141.cbd51fa.el7ost.noarch openstack-heat-engine-7.0.0-0.20160926200847.dd707bc.el7ost.noarch openstack-selinux-0.7.9-1.el7ost.noarch openstack-neutron-ml2-9.0.0-0.20160929051647.71f2d2b.el7ost.noarch openstack-mistral-common-3.0.0-0.20160929083341.c0a4501.el7ost.noarch openstack-sahara-api-5.0.0-0.20160926213141.cbd51fa.el7ost.noarch How reproducible: Always Steps to Reproduce: 1. Install openstack RHOS10 2. Pefroem some API calls Actual results: in-process cache used Expected results: memcache cache used Additional info:
dan, any chance you could take a look at this one? I wasn't sure if it might be related to some of the refactorings we've done in the templates around memcached.
I did some checking and I think I found the Nova regression. In http://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=0ef28cef1f1d97b89ad45c878c0c523c44a32f45 we dropped the ::nova::mechached_servers puppet setting so it no longer configures auth_token for nova. I will get an upstream patch pushed to resolve this for newton. Marian: I was unable to find any record of setting auth_token/memcached settings for neutron in TripleO however. Could you confirm that OSP 9 did in fact set this for neutron as well? Regardless, I can go ahead and push the fixes to Newton (OSP 10) to set memcached_servers for both Neutron and Nova to resolve this regression.
(In reply to Dan Prince from comment #2) > > Marian: I was unable to find any record of setting auth_token/memcached > settings for neutron in TripleO however. Could you confirm that OSP 9 did in > fact set this for neutron as well? Hm, Actually I cannot confirm that, I cannot see it configured for neutron on OSP9, only for nova, but the log does complain about in-process cache to be deprecated and suggest to move to memcache so eventually we will have to change that. But the regression in behaviour is only for nova in comparison with RHOS9.
Testing an upstream patch for this here: https://review.openstack.org/#/c/387624/
dan, looks like the master patch merged, but it needs to be backported to stable/newton. It looks like it is merge conflicting during the backport from the gerrit ui, so can you have a look?
This was merged to stable/newton here: https://review.openstack.org/#/c/389176/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-2948.html