Bug 1647510 - Swift increased the number of workers from OSP13
Summary: Swift increased the number of workers from OSP13
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 14.0 (Rocky)
Assignee: Christian Schwede (cschwede)
QA Contact: Mike Abrams
URL:
Whiteboard: scale_lab
Depends On:
Blocks: 1635664
TreeView+ depends on / blocked
 
Reported: 2018-11-07 16:08 UTC by Joe Talerico
Modified: 2020-08-21 13:14 UTC (History)
8 users (show)

Fixed In Version: openstack-tripleo-heat-templates-9.2.1-0.20190119154863.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-03-18 13:03:13 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 618105 0 None MERGED Restrict number of Swift workers to 1 on the undercloud 2020-12-03 13:45:16 UTC
Red Hat Product Errata RHBA-2019:0446 0 None None None 2019-03-18 13:03:22 UTC

Description Joe Talerico 2018-11-07 16:08:38 UTC
Description of problem:

Swift memory usage has increased > 2x. 
http://norton.perf.lab.eng.rdu.redhat.com:3000/dashboard/snapshot/y669hKyGJKCNSN6h5L6TN0RB7IIe31E5?orgId=1 
Date: 10/22 is OSP14 Older bits
Date: 10/24 is OSP13z2 Bits
Date: Now() is OSP14  2018-10-25.3

Swift has also regressed to use # Of CPUs for swift-[account|container]-server (previously was the CPU worker calculation). 
http://norton.perf.lab.eng.rdu.redhat.com:3000/dashboard/snapshot/IbQoZq3n1jDrplm2uLtZmJScLC0fCnNx 


Version-Release number of selected component (if applicable):
OSP14

How reproducible:
100%

Comment 3 Christian Schwede (cschwede) 2018-11-16 13:02:17 UTC
Joe, can you please check the memory consumption again? I see quite different numbers on my undercloud. Is there another graph with real used memory? RSS and virtual might be misleading here, because of the high amount of shared memory.

I had a look with docker stats, and the memory consumption looks pretty okay to me:

[stack@undercloud ~]$ docker stats --format="{{.Name}} {{.MemUsage}}" --no-stream | grep swift
swift_proxy 132.8 MiB / 11.73 GiB
swift_container_server 106.6 MiB / 11.73 GiB
swift_object_updater 26.48 MiB / 11.73 GiB
swift_account_server 97.3 MiB / 11.73 GiB
swift_rsync 188 KiB / 11.73 GiB
swift_object_expirer 28.38 MiB / 11.73 GiB
swift_account_reaper 24.94 MiB / 11.73 GiB
swift_object_server 110.3 MiB / 11.73 GiB
swift_container_updater 25.39 MiB / 11.73 GiB

Comment 4 Christian Schwede (cschwede) 2018-11-16 13:06:15 UTC
For reference, adding my benchmark results posted in https://review.openstack.org/#/c/618105/ here:

I run some simple benchmarks, and here are my results (default auto setting on this machine with 24 cores results in 12 workers):

===================================== 12 WORKERS =====================================
swift-bench 2018-11-16 11:50:01,931 INFO 10000 PUTS **FINAL** [0 failures], 65.9/s
swift-bench 2018-11-16 11:50:45,672 INFO 10000 GETS **FINAL** [0 failures], 228.7/s
===================================== 10 WORKERS =====================================
swift-bench 2018-11-16 11:54:58,142 INFO 10000 PUTS **FINAL** [0 failures], 62.0/s
swift-bench 2018-11-16 11:55:42,562 INFO 10000 GETS **FINAL** [0 failures], 225.2/s
===================================== 8 WORKERS =====================================
swift-bench 2018-11-16 11:59:54,088 INFO 10000 PUTS **FINAL** [0 failures], 63.9/s
swift-bench 2018-11-16 12:00:38,464 INFO 10000 GETS **FINAL** [0 failures], 225.4/s
===================================== 6 WORKERS =====================================
swift-bench 2018-11-16 12:04:50,969 INFO 10000 PUTS **FINAL** [0 failures], 62.4/s
swift-bench 2018-11-16 12:05:34,058 INFO 10000 GETS **FINAL** [0 failures], 232.2/s
===================================== 4 WORKERS =====================================
swift-bench 2018-11-16 12:09:59,399 INFO 10000 PUTS **FINAL** [0 failures], 57.6/s
swift-bench 2018-11-16 12:10:51,284 INFO 10000 GETS **FINAL** [0 failures], 192.8/s
===================================== 2 WORKERS =====================================
swift-bench 2018-11-16 12:15:26,775 INFO 10000 PUTS **FINAL** [0 failures], 56.3/s
swift-bench 2018-11-16 12:16:16,462 INFO 10000 GETS **FINAL** [0 failures], 201.3/s
===================================== 1 WORKERS =====================================
swift-bench 2018-11-16 12:21:13,469 INFO 10000 PUTS **FINAL** [0 failures], 50.3/s
swift-bench 2018-11-16 12:22:34,887 INFO 10000 GETS **FINAL** [0 failures], 122.8/s


Concurrency was 20, object size 10KiB, 10.000 PUTS/GETS, no Keystone, direct access without haproxy. 

Swift doesn't benefit that much if there are many more workers, because there is just a single disk on the undercloud, and this one is the limiting factor. The performance with just a single worker should be still fine for the use cases on the undercloud, but I think the sweet spot is 2 workers actually - the performance difference between 2 and more workers is quite small, however the difference of the used memory for Swift is quite significant:

Workers	Memory (MiB)
--------------------
12	841,88
10	806,25
8	702,3
6	595,66
4	484,33
2	368,6
1	301,92

Comment 8 Joe Talerico 2018-11-19 15:51:40 UTC
(In reply to Christian Schwede (cschwede) from comment #3)
> Joe, can you please check the memory consumption again? I see quite
> different numbers on my undercloud. Is there another graph with real used
> memory? RSS and virtual might be misleading here, because of the high amount
> of shared memory.
> 
> I had a look with docker stats, and the memory consumption looks pretty okay
> to me:
> 
> [stack@undercloud ~]$ docker stats --format="{{.Name}} {{.MemUsage}}"
> --no-stream | grep swift
> swift_proxy 132.8 MiB / 11.73 GiB
> swift_container_server 106.6 MiB / 11.73 GiB
> swift_object_updater 26.48 MiB / 11.73 GiB
> swift_account_server 97.3 MiB / 11.73 GiB
> swift_rsync 188 KiB / 11.73 GiB
> swift_object_expirer 28.38 MiB / 11.73 GiB
> swift_account_reaper 24.94 MiB / 11.73 GiB
> swift_object_server 110.3 MiB / 11.73 GiB
> swift_container_updater 25.39 MiB / 11.73 GiB

Christian - What Puddle? I have the latest puddle?

I am seeing the same issue here : ( process/thread count ) http://norton.perf.lab.eng.rdu.redhat.com:3000/dashboard/snapshot/TolTSzYc32HorlULsSHme4qWQrsRQluP

We are seeing large consumptions of RSS still : (Memory usage)
http://norton.perf.lab.eng.rdu.redhat.com:3000/dashboard/snapshot/IMBl9vWwWG3pnjamMhC6t1UKfbJ8hLZq

Comment 9 Christian Schwede (cschwede) 2018-11-28 14:58:00 UTC
*** Bug 1651680 has been marked as a duplicate of this bug. ***

Comment 19 errata-xmlrpc 2019-03-18 13:03:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0446

Comment 20 Christian Schwede (cschwede) 2020-08-21 13:14:43 UTC
Clearing needinfo on closed BZ.


Note You need to log in before you can comment on or make changes to this bug.