Description of problem: Comparing keystone performance in OSP 16 with OSP 13, we see that when running a scenario that creates a user and lists all users[1] for a total of 1000 times at a concurrency of 16(16 parallel threads of rally trying the scenario for a total of 1000 times) through browbeat[0], we see that the 95% response time in secondsis as follows Scenario/OSP version OSP 16 OSP 13 Create user 0.81 0.41 List users 1.47 2.22 This is a fair comparison in terms of caching as in both releases we have, #backend = dogpile.cache.null The one difference I could spot was that in OSP 13, we see 4 keystone-admin and 4 keystone-main processes (the controller server has 8 logical cores, so # of cores/2 according to Tripleo/puppet logic), however in the case of OSP 16, we only have one "keystone" process OSP 16 =============================================================================== [root@controller-0 heat-admin]# ps aux | grep keystone root 78671 0.0 0.0 142832 2420 ? Ssl 16:26 0:00 /usr/libexec/podman/conmon -s -c b757473d815a538bc62eb18a30e8b060a3ebfd2ef9aee1aaad169c0e0b84a0a6 -u b757473d815a538bc62eb18a30e8b060a3ebfd2ef9aee1aaad169c0e0b84a0a6 -n keystone -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/b757473d815a538bc62eb18a30e8b060a3ebfd2ef9aee1aaad169c0e0b84a0a6/userdata -p /var/run/containers/storage/overlay-containers/b757473d815a538bc62eb18a30e8b060a3ebfd2ef9aee1aaad169c0e0b84a0a6/userdata/pidfile --exit-dir /var/run/libpod/exits --conmon-pidfile /var/run/keystone.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg b757473d815a538bc62eb18a30e8b060a3ebfd2ef9aee1aaad169c0e0b84a0a6 --socket-dir-path /var/run/libpod/socket -l k8s-file:/var/log/containers/stdouts/keystone.log --log-level error 42425 108212 0.2 0.1 522640 102960 ? Sl 16:31 0:09 keystone -DFOREGROUND 42425 108213 0.2 0.1 522384 102800 ? Sl 16:31 0:09 keystone -DFOREGROUND 42425 108214 0.2 0.1 522128 102804 ? Sl 16:31 0:09 keystone -DFOREGROUND 42425 108215 0.2 0.1 522384 102908 ? Sl 16:31 0:09 keystone -DFOREGROUND root 437331 0.0 0.0 12108 1064 pts/0 S+ 17:38 0:00 grep --color=auto keystone OSP 13 =============================================================================== [root@controller-0 heat-admin]# vi /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf [root@controller-0 heat-admin]# ps aux | grep keystone 42425 124438 0.1 0.1 673968 95372 ? Sl 16:49 0:05 keystone-admin -DFOREGROUND 42425 124439 0.1 0.1 673200 93356 ? Sl 16:49 0:04 keystone-admin -DFOREGROUND 42425 124440 0.1 0.1 674224 95252 ? Sl 16:49 0:05 keystone-admin -DFOREGROUND 42425 124441 0.1 0.1 674224 95420 ? Sl 16:49 0:05 keystone-admin -DFOREGROUND 42425 124442 0.2 0.1 675800 94240 ? Sl 16:49 0:06 keystone-main -DFOREGROUND 42425 124443 0.2 0.1 675544 94096 ? Sl 16:49 0:06 keystone-main -DFOREGROUND 42425 124444 0.1 0.1 674776 92380 ? Sl 16:49 0:04 keystone-main -DFOREGROUND 42425 124445 0.2 0.1 676568 95300 ? Sl 16:49 0:06 keystone-main -DFOREGROUND root 465652 0.0 0.0 112716 972 pts/0 S+ 17:36 0:00 grep --color=auto keystone [0]- https://github.com/cloud-bulldozer/browbeat/blob/master/rally/keystonebasic/create_and_list_users-cc.yml [1]- https://github.com/openstack/rally-openstack/blob/master/rally_openstack/scenarios/keystone/basic.py#L149 Version-Release number of selected component (if applicable): 16 How reproducible: 100% Steps to Reproduce: 1. Deploy OSP 13 and OSP 16 2. Run browbeat+rally performance tests 3. Compare results Actual results: Performance in OSP 16 is significantly lower than in OSP 13 Expected results: Performance of keystone in OSP 16 should be equal to or greater than OSP 13 Additional info:
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2114