Description of problem: Compared to pulpv1, repository creation takes much longer and degrades much worse as more repositories are created. For example, on my box the first repo to be created takes .60 seconds, while the 1600th repo takes ~6 seconds. Within pulpv1, we could create 1600 repositories within about 30 minutes. In pulpv2, this seems to take more like 90 minutes. Within katello our manifest import creates repositories for every available content, so 1600 repos is not abnormal. We are changing this very soon, so its not of high priority. Attached is a script i used to reproduce the issue, as well the output of all the timings, showing creation taking longer and longer.
Created attachment 702366 [details] script used to generate 1600 repos and record timings
Created attachment 702367 [details] resultts showing slowing of creation
Does the performance get back to normal afterwords? My gut reaction is that it's the tasking subsystem having more and more data to resolve against. What I want to see is if the tasks clear themselves out once they've completed and things go back to normal.
Nope, creating repos 1601-1605 still took around 6 seconds a piece, after waiting ~8 hours after the first 1600 were created.
Ok cool, thanks for checking.
Also to note, I re-ran the tests on a clean pulp server without the feed certs and the problem still occurred, although it was slightly better. After making 1600 repos, i waited ~90 minutes and created 5 more, each one took around 5.5 seconds.
We've done some investigation into this. The creation and updating of repos gets slower and slower the more repos that exist because they invoke configuration validation, which inspects the configuration of every other repo of the same type (i.e. same importer) for conflicts. In the reported use case, this is every repo, on every create and every update.
Pull requests have been submitted: https://github.com/pulp/pulp_rpm/pull/319 https://github.com/pulp/pulp_puppet/pull/66 https://github.com/pulp/pulp/pull/588
build: 2.3.0-0.10.alpha
Created attachment 808927 [details] Repo create result after the fix
verified [root@hp-dl360g7-01 ~]# rpm -qa pulp-server pulp-server-2.3.0-0.17.beta.el6.noarch [root@hp-dl360g7-01 ~]#
Pulp 2.3 released.