Description of problem: We have a script uploading rpms to pulp as they are generated, as a part of our CI system. Today I noticed that our /var/lib/pulp/uploads is 123 gigs. Our rpms themselves are also within the /var/lib/pulp/content directory, so this results in a vast amount of wasted space. Version-Release number of selected component (if applicable): # rpm -qa|grep pulp pulp-builtins-admin-extensions-2.1.3-1.el6.noarch python-pulp-rpm-extension-2.1.3-1.el6.noarch python-oauth2-1.5.170-3.pulp.el6.noarch python-pulp-common-2.1.3-1.el6.noarch python-pulp-puppet-common-2.1.3-1.el6.noarch m2crypto-0.21.1.pulp-8.el6.x86_64 python-pulp-client-lib-2.1.3-1.el6.noarch mod_wsgi-3.4-1.pulp.el6.x86_64 pulp-rpm-admin-extensions-2.1.3-1.el6.noarch python-isodate-0.5.0-1.pulp.el6.noarch python-rhsm-1.8.0-1.pulp.el6.x86_64 python-pulp-rpm-common-2.1.3-1.el6.noarch pulp-nodes-common-2.1.3-1.el6.noarch pulp-admin-client-2.1.3-1.el6.noarch pulp-nodes-admin-extensions-2.1.3-1.el6.noarch python-pulp-bindings-2.1.3-1.el6.noarch pulp-puppet-admin-extensions-2.1.3-1.el6.noarch pulp-server-2.1.3-1.el6.noarch pulp-rpm-plugins-2.1.3-1.el6.noarch pulp-nodes-parent-2.1.3-1.el6.noarch How reproducible: Immanently. Steps to Reproduce: 1. Upload RPMs 2. Wait to see if pulp cleans up its uploads directory Actual results: Directory fills up. Mine is 123 gigs right now.... Expected results: When pulp finishes with an rpm, it removes it from the uploads/ directory Additional info: For further details of my environment, please see https://bugzilla.redhat.com/show_bug.cgi?id=996302
Jim, can you give a sample of how your build script is calling the upload command? Is it using the pulp-admin command line or calling the rest API?
It is using the pulp-admin command. Here is our script: this is the upload script #!/bin/bash # Login to pulp /usr/bin/sudo pulp-admin login -u admin -p admin # Remove any existing copies of the uploaded RPMs if [ -e ./magic-$CI_GERRIT_CHANGE_NUMBER-$CI_GERRIT_PATCHSET_NUMBER.x86_64.rpm ]; then /usr/bin/sudo pulp-admin rpm repo remove rpm --repo-id=magic --match 'name=magic' --match 'version=$CI_GERRIT_CHANGE_NUMBER' --match 'release=$CI_GERRIT_PATCHSET_NUMBER' /usr/bin/sudo pulp-admin rpm repo publish run --repo-id magic fi if [ -e ./other_magic$CI_GERRIT_CHANGE_NUMBER-$CI_GERRIT_PATCHSET_NUMBER.noarch.rpm ]; then /usr/bin/sudo pulp-admin rpm repo remove rpm --repo-id=magic --match 'name=other_magic' --match 'version=$CI_GERRIT_CHANGE_NUMBER' --match 'release=$CI_GERRIT_PATCHSET_NUMBER' /usr/bin/sudo pulp-admin rpm repo publish run --repo-id magic fi /usr/bin/sudo pulp-admin orphan remove --all # Upload packages if [ -e ./magic-$CI_GERRIT_CHANGE_NUMBER-$CI_GERRIT_PATCHSET_NUMBER.x86_64.rpm ]; then /usr/bin/sudo /usr/bin/pulp-admin rpm repo uploads rpm --repo-id magic --file=./magic-$CI_GERRIT_CHANGE_NUMBER-$CI_GERRIT_PATCHSET_NUMBER.x86_64.rpm fi if [ -e ./other_magic$CI_GERRIT_CHANGE_NUMBER-$CI_GERRIT_PATCHSET_NUMBER.noarch.rpm ]; then /usr/bin/sudo /usr/bin/pulp-admin rpm repo uploads rpm --repo-id magic --file=./other_magic$CI_GERRIT_CHANGE_NUMBER-$CI_GERRIT_PATCHSET_NUMBER.noarch.rpm fi /usr/bin/sudo pulp-admin rpm repo publish run --repo-id magic # Remove extra files if [ -e ./magic-$CI_GERRIT_CHANGE_NUMBER-$CI_GERRIT_PATCHSET_NUMBER.x86_64.rpm ]; then /usr/bin/sudo /bin/rm ./magic-$CI_GERRIT_CHANGE_NUMBER-$CI_GERRIT_PATCHSET_NUMBER.x86_64.rpm fi if [ -e ./other_magic$CI_GERRIT_CHANGE_NUMBER-$CI_GERRIT_PATCHSET_NUMBER.noarch.rpm ]; then /usr/bin/sudo /bin/rm ./other_magic$CI_GERRIT_CHANGE_NUMBER-$CI_GERRIT_PATCHSET_NUMBER.noarch.rpm fi ---------------- here is the cron script on my client-servers (WTB rate limited nodes....) # cat /etc/cron.d/pulp_5min */5 * * * * root /opt/share/synch_pulp_5min.sh You have new mail in /var/spool/mail/root # cat /opt/share/synch_pulp_5min.sh #!/bin/bash /usr/bin/pulp-admin login -u admin -p admin /usr/bin/pulp-admin rpm repo sync run --repo-id=magic
Pull Request: https://github.com/pulp/pulp_rpm/pull/333
build: 2.3.0-0.10.alpha
[root@pulp-v2-server ~]# rpm -qa pulp-server pulp-server-2.3.0-0.14.alpha.el6.noarch [root@pulp-v2-server ~]# [root@pulp-v2-server ~]# ls -l /var/lib/pulp/uploads/total 0 -rw-r--r--. 1 apache apache 0 Sep 19 13:24 03d23a44-3a72-4648-9fca-ad4da499556c -rw-r--r--. 1 apache apache 0 Sep 19 13:24 1cdc715c-7f50-43eb-b7ad-8b04f7784079 -rw-r--r--. 1 apache apache 0 Sep 19 13:24 202c62fd-64e6-4cf1-8683-d45006735d00 -rw-r--r--. 1 apache apache 0 Sep 19 13:24 2617c668-3053-42e9-b0e2-2dc9b68de05b -rw-r--r--. 1 apache apache 0 Sep 19 13:24 2824d5c9-a6c5-4dab-be00-e12edf85fa0f -rw-r--r--. 1 apache apache 0 Sep 19 13:24 2b0fe4ae-ad5a-4f2d-9b39-c1c25ad941b9 -rw-r--r--. 1 apache apache 0 Sep 19 13:24 46f78957-031b-4d7d-850c-1788fb0811e5 -rw-r--r--. 1 apache apache 0 Sep 19 13:24 5a10aece-6b82-4223-9974-fdf119722ee5 -rw-r--r--. 1 apache apache 0 Sep 19 13:24 5aea1fa7-b605-4544-a059-905be0e60de9 -rw-r--r--. 1 apache apache 0 Sep 19 13:24 6b0b9b74-6235-4909-aa68-a5d1d3291915 -rw-r--r--. 1 apache apache 0 Sep 19 13:24 761f8982-f55e-4a64-aac1-ca026f253593 -rw-r--r--. 1 apache apache 0 Sep 19 13:24 7d601bf3-d111-44e9-ac0b-6265068ae5e2 -rw-r--r--. 1 apache apache 0 Sep 19 13:24 ab66c7f3-6714-4157-89bd-ce7e92a44dff -rw-r--r--. 1 apache apache 0 Sep 19 13:24 ae4a7f42-a3ce-4e1d-95c0-d9dfefacc1c6 -rw-r--r--. 1 apache apache 0 Sep 19 13:24 b0628fd4-595c-4dd4-a270-38c3d3d6279c -rw-r--r--. 1 apache apache 0 Sep 19 13:24 b1d6d5e7-7896-4f5b-ac75-6b5e5a04eb69 -rw-r--r--. 1 apache apache 0 Sep 19 13:24 b7e79d34-3da9-44c9-b702-3aaab31c60be -rw-r--r--. 1 apache apache 0 Sep 19 13:24 c0c0ad10-56b1-4ffe-98c8-57f79e2bef11 -rw-r--r--. 1 apache apache 0 Sep 19 13:24 d5ac3173-4a91-4d9a-ac7b-6397fa1151c7 -rw-r--r--. 1 apache apache 0 Sep 19 13:24 e5fc5994-9921-41b1-8462-c45114b269bd -rw-r--r--. 1 apache apache 0 Sep 19 13:24 e680441b-fe5f-4a56-98bf-0945c10f7fa0 -rw-r--r--. 1 apache apache 0 Sep 19 13:24 e6abccc8-a385-41e0-b013-b92a5b2a7d0f [root@pulp-v2-server ~]# pulp-admin rpm repo uploads list +----------------------------------------------------------------------+ Upload Requests +----------------------------------------------------------------------+ [ Paused ] Wanganui-6.8.3-1.noarch.rpm [ Paused ] acme-package-1.0.2-1.noarch.rpm [ Paused ] acme-package-1.1.2-1.noarch.rpm [ Paused ] thiefwise-10.8.0-1.noarch.rpm [ Paused ] Phitsanulok-10.0.6-1.noarch.rpm [ Paused ] intermeasure-3.10.6-1.noarch.rpm [ Paused ] jokeproof-10.2.0-1.noarch.rpm [ Paused ] newspaperwomen-9.6.1-1.noarch.rpm [ Paused ] weirdness-7.3.7-1.noarch.rpm [ Paused ] unheart-2.9.8-1.noarch.rpm [ Paused ] tiffany-10.7.10-1.noarch.rpm [ Paused ] Rhinobatus-10.8.8-1.noarch.rpm [ Paused ] worrywarts-6.1.1-1.noarch.rpm [ Paused ] wormseeds-2.7.9-1.noarch.rpm [ Paused ] paperhanger-8.4.7-1.noarch.rpm [ Paused ] acme-package-1.0.1-1.noarch.rpm [ Paused ] copus-10.5.1-1.noarch.rpm [ Paused ] succinoresinol-0.0.6-1.noarch.rpm [ Paused ] fled-10.7.4-1.noarch.rpm [ Paused ] shitted-6.10.8-1.noarch.rpm [ Paused ] posthospital-5.9.9-1.noarch.rpm [ Paused ] debacchate-0.7.6-1.noarch.rpm [root@pulp-v2-server ~]# pulp-admin rpm repo uploads resume +----------------------------------------------------------------------+ Upload Requests +----------------------------------------------------------------------+ Select one or more uploads to resume: - 1 : Wanganui-6.8.3-1.noarch.rpm - 2 : acme-package-1.0.2-1.noarch.rpm - 3 : acme-package-1.1.2-1.noarch.rpm - 4 : thiefwise-10.8.0-1.noarch.rpm - 5 : Phitsanulok-10.0.6-1.noarch.rpm - 6 : intermeasure-3.10.6-1.noarch.rpm - 7 : jokeproof-10.2.0-1.noarch.rpm - 8 : newspaperwomen-9.6.1-1.noarch.rpm - 9 : weirdness-7.3.7-1.noarch.rpm - 10: unheart-2.9.8-1.noarch.rpm - 11: tiffany-10.7.10-1.noarch.rpm - 12: Rhinobatus-10.8.8-1.noarch.rpm - 13: worrywarts-6.1.1-1.noarch.rpm - 14: wormseeds-2.7.9-1.noarch.rpm - 15: paperhanger-8.4.7-1.noarch.rpm - 16: acme-package-1.0.1-1.noarch.rpm - 17: copus-10.5.1-1.noarch.rpm - 18: succinoresinol-0.0.6-1.noarch.rpm - 19: fled-10.7.4-1.noarch.rpm - 20: shitted-6.10.8-1.noarch.rpm - 21: posthospital-5.9.9-1.noarch.rpm - 22: debacchate-0.7.6-1.noarch.rpm Enter value (1-22) to toggle selection, 'c' to confirm selections, or '?' for more commands: 1-22 Select one or more uploads to resume: x 1 : Wanganui-6.8.3-1.noarch.rpm x 2 : acme-package-1.0.2-1.noarch.rpm x 3 : acme-package-1.1.2-1.noarch.rpm x 4 : thiefwise-10.8.0-1.noarch.rpm x 5 : Phitsanulok-10.0.6-1.noarch.rpm x 6 : intermeasure-3.10.6-1.noarch.rpm x 7 : jokeproof-10.2.0-1.noarch.rpm x 8 : newspaperwomen-9.6.1-1.noarch.rpm x 9 : weirdness-7.3.7-1.noarch.rpm x 10: unheart-2.9.8-1.noarch.rpm x 11: tiffany-10.7.10-1.noarch.rpm x 12: Rhinobatus-10.8.8-1.noarch.rpm x 13: worrywarts-6.1.1-1.noarch.rpm x 14: wormseeds-2.7.9-1.noarch.rpm x 15: paperhanger-8.4.7-1.noarch.rpm x 16: acme-package-1.0.1-1.noarch.rpm x 17: copus-10.5.1-1.noarch.rpm x 18: succinoresinol-0.0.6-1.noarch.rpm x 19: fled-10.7.4-1.noarch.rpm x 20: shitted-6.10.8-1.noarch.rpm x 21: posthospital-5.9.9-1.noarch.rpm x 22: debacchate-0.7.6-1.noarch.rpm Enter value (1-22) to toggle selection, 'c' to confirm selections, or '?' for more commands: c Resuming upload for: Wanganui-6.8.3-1.noarch.rpm, acme-package-1.0.2-1.noarch.rpm, acme-package-1.1.2-1.noarch.rpm, thiefwise-10.8.0-1.noarch.rpm, Phitsanulok-10.0.6-1.noarch.rpm, intermeasure-3.10.6-1.noarch.rpm, jokeproof-10.2.0-1.noarch.rpm, newspaperwomen-9.6.1-1.noarch.rpm, weirdness-7.3.7-1.noarch.rpm, unheart-2.9.8-1.noarch.rpm, tiffany-10.7.10-1.noarch.rpm, Rhinobatus-10.8.8-1.noarch.rpm, worrywarts-6.1.1-1.noarch.rpm, wormseeds-2.7.9-1.noarch.rpm, paperhanger-8.4.7-1.noarch.rpm, acme-package-1.0.1-1.noarch.rpm, copus-10.5.1-1.noarch.rpm, succinoresinol-0.0.6-1.noarch.rpm, fled-10.7.4-1.noarch.rpm, shitted-6.10.8-1.noarch.rpm, posthospital-5.9.9-1.noarch.rpm, debacchate-0.7.6-1.noarch.rpm Starting upload of selected units. If this process is stopped through ctrl+c, the uploads will be paused and may be resumed later using the resume command or cancelled entirely using the cancel command. Uploading: Wanganui-6.8.3-1.noarch.rpm [==================================================] 100% 10928/10928 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: acme-package-1.0.2-1.noarch.rpm [==================================================] 100% 3790/3790 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: acme-package-1.1.2-1.noarch.rpm [==================================================] 100% 3790/3790 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: thiefwise-10.8.0-1.noarch.rpm [==================================================] 100% 69405/69405 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: Phitsanulok-10.0.6-1.noarch.rpm [==================================================] 100% 12022/12022 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: intermeasure-3.10.6-1.noarch.rpm [==================================================] 100% 28459/28459 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: jokeproof-10.2.0-1.noarch.rpm [==================================================] 100% 65282/65282 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: newspaperwomen-9.6.1-1.noarch.rpm [==================================================] 100% 31552/31552 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: weirdness-7.3.7-1.noarch.rpm [==================================================] 100% 11966/11966 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: unheart-2.9.8-1.noarch.rpm [==================================================] 100% 20193/20193 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: tiffany-10.7.10-1.noarch.rpm [==================================================] 100% 19132/19132 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: Rhinobatus-10.8.8-1.noarch.rpm [==================================================] 100% 8922/8922 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: worrywarts-6.1.1-1.noarch.rpm [==================================================] 100% 81680/81680 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: wormseeds-2.7.9-1.noarch.rpm [==================================================] 100% 46841/46841 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: paperhanger-8.4.7-1.noarch.rpm [==================================================] 100% 91950/91950 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: acme-package-1.0.1-1.noarch.rpm [==================================================] 100% 3789/3789 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: copus-10.5.1-1.noarch.rpm [==================================================] 100% 44736/44736 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: succinoresinol-0.0.6-1.noarch.rpm [==================================================] 100% 45907/45907 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: fled-10.7.4-1.noarch.rpm [==================================================] 100% 67248/67248 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: shitted-6.10.8-1.noarch.rpm [==================================================] 100% 78573/78573 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: posthospital-5.9.9-1.noarch.rpm [==================================================] 100% 71479/71479 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed Uploading: debacchate-0.7.6-1.noarch.rpm [==================================================] 100% 19166/19166 bytes ... completed Importing into the repository... ... completed Deleting the upload request... ... completed [root@pulp-v2-server ~]# ls -l /var/lib/pulp/uploads/ total 0 [root@pulp-v2-server ~]# pulp-admin repo list +----------------------------------------------------------------------+ Repositories +----------------------------------------------------------------------+ Id: rhel6-2 Display Name: rhel6-2 Description: None Content Unit Counts: Distribution: 1 Erratum: 1048 Package Category: 10 Package Group: 202 Yum Repo Metadata File: 1 Id: upload Display Name: upload Description: None Content Unit Counts: Rpm: 23 Id: test-upload Display Name: test-upload Description: None Content Unit Counts: Rpm: 23 [root@pulp-v2-server ~]#
Pulp 2.3 released.