Description of problem: David Kranz executed a tempest test against a Nova Networking configurion I installed using foreman. After a bit (hours, a day), all command line action were extremely slugging. Top indicated the keystone and the database would peak during queries. David discussed with Adam Young and it appears that the revocation cache time is set really low by default. David update the service configtion file [keystone_auth] section with lines similar to the following, and saw performance inprovements: revocation_cache_time=300 signing_dir=/var/cache/nova/signing Why the preformance degrade over time may be an issue that Shak's team could explore. Version-Release number of selected component (if applicable): [root@rhos-foreman ~]# yum list installed | grep -e foreman -e puppet This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. foreman.noarch 1.3.0.2-1.el6sat @rhel-x86_64-server-6-ost-4 foreman-installer.noarch 1:1.3.0-1.el6sat @rhel-x86_64-server-6-ost-4 foreman-mysql.noarch 1.3.0.2-1.el6sat @rhel-x86_64-server-6-ost-4 foreman-mysql2.noarch 1.3.0.2-1.el6sat @rhel-x86_64-server-6-ost-4 foreman-proxy.noarch 1.3.0-3.el6sat @rhel-x86_64-server-6-ost-4 foreman-selinux.noarch 1.3.0-1.el6sat @rhel-x86_64-server-6-ost-4 openstack-foreman-installer.noarch packstack-modules-puppet.noarch 2013.2.1-0.22.dev956.el6ost puppet.noarch 3.2.4-3.el6_5 @rhel-x86_64-server-6-ost-4 puppet-server.noarch 3.2.4-3.el6_5 @rhel-x86_64-server-6-ost-4 ruby193-rubygem-foreman_openstack_simplify.noarch rubygem-foreman_api.noarch 0.1.6-1.el6sat @rhel-x86_64-server-6-ost-4 [root@rhos-foreman ~]# How reproducible: Not attempted as of yet Steps to Reproduce: 1. David needs to supply. 2. 3. Actual results: Expected results: Additional info:
Would this issue just be related to PKI tokens, or could it possibly affect UUID tokens as well?
This should be a quick fix with a declaration of keystone_config in the relevant foreman manifest.
Graeme, My understanding is this is a PKI issue only.
What we saw was that the cache directory was not created, meaning that validation was falling back to online calls to Keystone. Please run through the steps here. http://adam.younglogic.com/2013/07/troubleshooting-pki-middleware/
This recent havana fix may be relevant: https://review.openstack.org/#/c/69884/
According to the Packstack team, this isn't a problem of Packstack.
SPR is going to try to reproduce on A2 tonight, will revisit tomorrow.
Ran with A2. First run was 25 minutes. after ~12 hours upto 60 minutes. After 24 upto 66. After a weekend runs were about 74 mintues. [root@f20-tempest tempest]# grep -B 10 "~~~~" nn-0306.out ~~~~~~ -- Ran 663 tests in 1505.046s FAILED (failures=164, skipped=16) real 25m5.226s user 0m20.780s sys 0m2.639s ~~~~~~ -- Ran 663 tests in 1904.347s FAILED (failures=166, skipped=16) real 31m44.517s user 0m21.261s sys 0m2.764s ~~~~~~ -- Ran 657 tests in 1929.470s FAILED (failures=166, skipped=16) real 32m9.633s user 0m21.135s sys 0m2.562s ~~~~~~ -- Ran 657 tests in 2263.211s FAILED (failures=166, skipped=16) real 37m43.376s user 0m21.239s sys 0m2.557s ~~~~~~ -- Ran 657 tests in 2504.892s FAILED (failures=166, skipped=16) real 41m45.026s user 0m21.015s sys 0m2.675s ~~~~~~ -- Ran 657 tests in 2554.171s FAILED (failures=166, skipped=16) real 42m34.329s user 0m21.097s sys 0m2.636s ~~~~~~ -- Ran 657 tests in 2675.866s FAILED (failures=166, skipped=16) real 44m36.038s user 0m21.301s sys 0m2.698s ~~~~~~ -- Ran 657 tests in 2933.126s FAILED (failures=166, skipped=16) real 48m53.297s user 0m21.438s sys 0m2.640s ~~~~~~ -- Ran 657 tests in 2994.945s FAILED (failures=166, skipped=16) real 49m55.082s user 0m21.160s sys 0m2.620s ~~~~~~ -- Ran 657 tests in 3276.955s FAILED (failures=166, skipped=16) real 54m37.125s user 0m21.388s sys 0m2.691s ~~~~~~ -- Ran 657 tests in 3559.289s FAILED (failures=166, skipped=16) real 59m19.456s user 0m21.407s sys 0m2.758s ~~~~~~ -- Ran 657 tests in 3491.088s FAILED (failures=166, skipped=16) real 58m11.254s user 0m21.386s sys 0m2.739s ~~~~~~ -- Ran 657 tests in 3593.250s FAILED (failures=166, skipped=16) real 59m53.414s user 0m21.216s sys 0m2.711s ~~~~~~ -- Ran 657 tests in 3753.935s FAILED (failures=166, skipped=16) real 62m34.102s user 0m21.224s sys 0m2.759s ~~~~~~ -- Ran 657 tests in 3973.478s FAILED (failures=166, skipped=16) real 66m13.644s user 0m21.400s sys 0m2.692s ~~~~~~ -- Ran 657 tests in 4003.696s FAILED (failures=166, skipped=16) real 66m43.833s user 0m21.263s sys 0m2.763s ~~~~~~ -- Ran 657 tests in 4118.465s FAILED (failures=166, skipped=16) real 68m38.630s user 0m21.576s sys 0m2.783s ~~~~~~ -- Ran 657 tests in 4094.499s FAILED (failures=166, skipped=16) real 68m14.676s user 0m22.133s sys 0m2.873s ~~~~~~ -- Ran 657 tests in 4460.533s FAILED (failures=166, skipped=16) real 74m20.710s user 0m21.878s sys 0m2.802s ~~~~~~ -- Ran 657 tests in 4297.884s FAILED (failures=166, skipped=16) real 71m38.060s user 0m21.817s sys 0m2.706s ~~~~~~ -- Ran 657 tests in 4466.340s FAILED (failures=166, skipped=16) real 74m26.484s user 0m21.712s sys 0m2.739s ~~~~~~ -- Ran 657 tests in 4722.049s FAILED (failures=166, skipped=16) real 78m42.225s user 0m21.555s sys 0m2.749s ~~~~~~ -- Ran 657 tests in 4709.772s FAILED (failures=166, skipped=16) real 78m29.965s user 0m21.621s sys 0m2.778s ~~~~~~ -- Ran 657 tests in 4703.224s FAILED (failures=166, skipped=16) real 78m23.376s user 0m21.705s sys 0m2.775s ~~~~~~ -- Ran 657 tests in 4686.744s FAILED (failures=166, skipped=16) real 78m6.896s user 0m21.609s sys 0m2.714s ~~~~~~ -- Ran 657 tests in 4654.274s FAILED (failures=166, skipped=16) real 77m34.439s user 0m21.200s sys 0m2.633s ~~~~~~ -- Ran 657 tests in 4634.555s FAILED (failures=166, skipped=16) real 77m14.721s user 0m21.372s sys 0m2.629s ~~~~~~ -- Ran 657 tests in 4637.203s FAILED (failures=166, skipped=16) real 77m17.365s user 0m21.606s sys 0m2.798s ~~~~~~ -- Ran 657 tests in 4519.590s FAILED (failures=166, skipped=16) real 75m19.758s user 0m21.312s sys 0m2.774s ~~~~~~ -- Ran 657 tests in 4357.108s FAILED (failures=166, skipped=16) real 72m37.270s user 0m21.353s sys 0m2.692s ~~~~~~ -- Ran 657 tests in 4638.555s FAILED (failures=166, skipped=16) real 77m18.715s user 0m21.932s sys 0m2.686s ~~~~~~ -- Ran 657 tests in 4486.259s FAILED (failures=166, skipped=16) real 74m46.400s user 0m21.380s sys 0m2.747s ~~~~~~ -- Ran 657 tests in 4518.183s FAILED (failures=166, skipped=16) real 75m18.401s user 0m21.385s sys 0m2.687s ~~~~~~ -- Ran 657 tests in 4513.137s FAILED (failures=166, skipped=16) real 75m13.312s user 0m21.684s sys 0m2.788s ~~~~~~ -- Ran 657 tests in 4446.353s FAILED (failures=166, skipped=16) real 74m6.522s user 0m21.687s sys 0m2.746s ~~~~~~ -- Ran 657 tests in 4477.296s FAILED (failures=166, skipped=16) real 74m37.469s user 0m21.551s sys 0m2.716s ~~~~~~ -- Ran 657 tests in 4612.390s FAILED (failures=166, skipped=16) real 76m52.539s user 0m21.706s sys 0m2.812s ~~~~~~ -- Ran 657 tests in 4329.567s FAILED (failures=166, skipped=16) real 72m9.723s user 0m21.675s sys 0m2.746s ~~~~~~ -- Ran 657 tests in 4330.149s FAILED (failures=168, skipped=16) real 72m10.295s user 0m21.601s sys 0m2.738s ~~~~~~ -- Ran 657 tests in 4485.410s FAILED (failures=166, skipped=16) real 74m45.552s user 0m21.331s sys 0m2.649s ~~~~~~ -- Ran 657 tests in 4683.580s FAILED (failures=166, skipped=16) real 78m3.751s user 0m21.723s sys 0m2.675s ~~~~~~ -- Ran 657 tests in 4654.554s FAILED (failures=166, skipped=16) real 77m34.727s user 0m21.627s sys 0m2.747s ~~~~~~ -- Ran 657 tests in 4526.358s FAILED (failures=166, skipped=16) real 75m26.514s user 0m21.745s sys 0m2.744s ~~~~~~ -- Ran 657 tests in 4516.692s FAILED (failures=166, skipped=16) real 75m16.892s user 0m21.307s sys 0m2.785s ~~~~~~ -- Ran 657 tests in 3971.786s FAILED (failures=166, skipped=16) real 66m11.967s user 0m21.960s sys 0m2.695s ~~~~~~ -- Ran 657 tests in 3968.267s FAILED (failures=166, skipped=16) real 66m8.439s user 0m21.790s sys 0m2.783s ~~~~~~ -- Ran 657 tests in 3875.711s FAILED (failures=166, skipped=16) real 64m35.905s user 0m21.398s sys 0m2.741s ~~~~~~ -- Ran 657 tests in 4010.604s FAILED (failures=166, skipped=16) real 66m50.748s user 0m21.556s sys 0m2.717s ~~~~~~ -- Ran 657 tests in 3954.007s FAILED (failures=166, skipped=16) real 65m54.180s user 0m21.799s sys 0m2.699s ~~~~~~ -- Ran 657 tests in 4112.287s FAILED (failures=166, skipped=16) real 68m32.433s user 0m21.541s sys 0m2.679s ~~~~~~ -- Ran 657 tests in 3884.476s FAILED (failures=166, skipped=16) real 64m44.690s user 0m21.508s sys 0m2.693s ~~~~~~ -- Ran 657 tests in 3818.343s FAILED (failures=166, skipped=16) real 63m38.500s user 0m21.135s sys 0m2.657s ~~~~~~ -- Ran 657 tests in 3869.749s FAILED (failures=167, skipped=16) real 64m29.919s user 0m21.351s sys 0m2.701s ~~~~~~ -- Ran 657 tests in 4051.573s FAILED (failures=166, skipped=16) real 67m31.745s user 0m21.481s sys 0m2.742s ~~~~~~ -- Ran 657 tests in 3879.930s FAILED (failures=166, skipped=16) real 64m40.102s user 0m21.474s sys 0m2.718s ~~~~~~ -- Ran 657 tests in 3983.634s FAILED (failures=166, skipped=16) real 66m23.802s user 0m21.662s sys 0m2.735s ~~~~~~ -- Ran 657 tests in 3973.082s FAILED (failures=166, skipped=16) real 66m13.265s user 0m21.630s sys 0m2.777s ~~~~~~ -- Ran 657 tests in 3988.245s FAILED (failures=166, skipped=16) real 66m28.379s user 0m21.372s sys 0m2.567s ~~~~~~ -- Ran 657 tests in 4048.429s FAILED (failures=166, skipped=16) real 67m28.598s user 0m21.524s sys 0m2.635s ~~~~~~ -- Ran 657 tests in 4150.911s FAILED (failures=166, skipped=16) real 69m11.050s user 0m21.204s sys 0m2.670s ~~~~~~ -- Ran 657 tests in 4131.234s FAILED (failures=166, skipped=16) real 68m51.371s user 0m21.620s sys 0m2.786s ~~~~~~ -- Ran 657 tests in 4291.324s FAILED (failures=166, skipped=16) real 71m31.489s user 0m21.763s sys 0m2.747s ~~~~~~ -- Ran 657 tests in 4023.482s FAILED (failures=167, skipped=16) real 67m3.655s user 0m21.629s sys 0m2.704s ~~~~~~ -- Ran 657 tests in 4079.756s FAILED (failures=166, skipped=16) real 67m59.929s user 0m21.283s sys 0m2.645s ~~~~~~ -- Ran 657 tests in 4080.298s FAILED (failures=166, skipped=16) real 68m0.442s user 0m21.739s sys 0m2.667s ~~~~~~ -- Ran 657 tests in 4178.190s FAILED (failures=166, skipped=16) real 69m38.338s user 0m21.590s sys 0m2.710s ~~~~~~ -- Ran 657 tests in 4136.684s FAILED (failures=166, skipped=16) real 68m56.861s user 0m21.603s sys 0m2.638s ~~~~~~ -- Ran 657 tests in 4331.789s FAILED (failures=166, skipped=16) real 72m11.964s user 0m21.848s sys 0m2.649s ~~~~~~ -- Ran 657 tests in 4293.904s FAILED (failures=166, skipped=16) real 71m34.071s user 0m21.711s sys 0m2.735s ~~~~~~ -- Ran 657 tests in 4398.514s FAILED (failures=166, skipped=16) real 73m18.677s user 0m21.668s sys 0m2.760s ~~~~~~ -- Ran 657 tests in 4404.560s FAILED (failures=167, skipped=16) real 73m24.728s user 0m21.656s sys 0m2.624s ~~~~~~ -- Ran 657 tests in 4434.833s FAILED (failures=166, skipped=16) real 73m55.016s user 0m21.695s sys 0m2.777s ~~~~~~ [root@f20-tempest tempest]#