Red Hat Bugzilla – Bug 1184853
[RFE] Don't publish the same repositories over and over again
Last modified: 2018-03-05 10:30:53 EST
Publishing content views is slow. If I just want to add a single package to a single repo, the publishing process republishes the entire Content View. Would it be possible instead to only publish that one repo that had changes when creating the new Content View Version?
Since this issue was entered in Red Hat Bugzilla, the release flag has been set to ? to ensure that it is properly evaluated for this release.
Also this is the same for promotion -> if I'm promoting why isn't it just a single symlink?
The inodes of a node are getting quickly full. In our system 20Million Inodes are used by Sat6. # df -h /var/lib/pulp Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_li_lc_1017_app1-lv_app1 291G 239G 38G 87% /lvapp1 # df -i /var/lib/pulp Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/vg_li_lc_1017_app1-lv_app1 19333120 19332915 205 100% /lvapp1 Some calculation on the amount of symlinks (=inodes) Library: RedHat repos 4 releases (6.x,6Server,7.1,7Server) = 4 * 30000 = 120.000 Regular CVs using RedHat repos - Amount of CVs 6, at leat 1 for each RedHat release - Average has 5 versions - Additional lifecycles (including library) => 4 versions => 6 * 30000 * (5 + 4) =~ 1.600.000 Composite CVs - Amount of Composite 10, for each Application group - Inherits at least one RedHat Content View - Average has 2 versions - Additional lifecycles (including library) => 4 versions => 10 * (2 + 4) * 30000 =~ 1.800.000 As you can see it grows quickly, especially the fact the each (Composite) Content View has to include a full RedHat release that means 30000 symlinks. The use all inodes is even quicker reached as the pulp content is not deleted, see BZ1184442.
Checked that it is even worse, in the previous calculation the required directories were not included: For RPMs there are upto 4 directories created Example of the 6Server-Server repo in the standard Library: /var/lib/pulp/nodes/published/https/repos/Hilti-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_6_Server_RPMs_x86_64_6Server/content/rpm# time find . -type d | wc -l && find . -type l | wc -l 48714 14616 That means in total 63.000 inodes, for only the 6Server-Server repo in the Library.
Checked with Satellite 6.1. This problem still is available. We have a RedHat CDN update every day and the published YUM repositories are recreated every day again. That means 1. create tmp repo (xxxxx inodes actions) 2. delete live repo (xxxxx inodes actions) 3. rename tmp to live (single atomic operation) We have 70 of such repositories, with average 30.000 inodes each. 70 * 30.000 * 2 = 4.200.000 inodes operations per RedHat sync.
Moving to 6.2.0 since this is an RFE
Created redmine issue http://projects.theforeman.org/issues/15798 from this bug
Moving 6.2 bugs out to sat-backlog.
Is this maybe fixed in http://projects.theforeman.org/issues/18032 with https://github.com/Katello/katello/pull/6597 ?
partially, but I don't think fully. I think the heart of the issue that stephen is getting at is that adding even if you just want to add one package to a content view, we generate an entirely new version creating all new repos. This takes a large amount of time.
Hi Justin, The https://bugzilla.redhat.com/show_bug.cgi?id=1522912 looks similair and implements optimizations. For me it is ok to close this as duplicate. Peter
I agree! Thanks Peter for for pointing that out. Closing this as a duplicate. *** This bug has been marked as a duplicate of bug 1522912 ***