Downloaded repodata is cached in /tmp/ in individual temp directories. After a RHUA has set for a little while, I'm seeing these leftover temp directories not getting cleaned up. There are no failed syncs in the grinder logs, so I'm not sure what's causing these directories to get left behind. Maybe we can just add a step as part of a try/finally handler to make sure these things get cleaned up.
[root@rhui2 ~]# ll /tmp/ total 36 -rw-r--r--. 1 root root 13 Sep 29 15:37 cloude-healthcheck srwxr-xr-x. 1 mongodb mongodb 0 Sep 22 17:18 mongodb-27017.sock srwxr-xr-x. 1 mongodb mongodb 0 Sep 22 17:18 mongodb-28017.sock drwx------. 3 apache apache 4096 Sep 27 20:04 tmpcqUl1N drwx------. 3 apache apache 4096 Sep 27 14:10 tmpfp8yhq drwx------. 3 apache apache 4096 Sep 27 19:58 tmpHyaFkr drwx------. 3 apache apache 4096 Sep 27 20:05 tmposNzLl drwx------. 3 apache apache 4096 Sep 27 14:09 tmpuhz0dn drwx------. 3 apache apache 4096 Sep 27 14:10 tmpUz1M5d drwx------. 3 apache apache 4096 Sep 27 14:10 tmpW4XNjF drwx------. 3 apache apache 4096 Sep 27 20:04 tmpxYWsHz This rhua is only configured to run 4 syncs at a time, so I should never see more than 4 of these directories at once. They're all repodata related: [root@rhui2 ~]# find /tmp/tmp* /tmp/tmpcqUl1N /tmp/tmpcqUl1N/repomd.xml /tmp/tmpcqUl1N/027ae7184275c41642d039bd67fc46bbd612b740-primary.xml.gz /tmp/tmpcqUl1N/8380ad947903da0951a1289558dbe9f696c37dac-filelists.xml.gz /tmp/tmpcqUl1N/packages /tmp/tmpcqUl1N/cachecookie /tmp/tmpfp8yhq /tmp/tmpfp8yhq/repomd.xml /tmp/tmpfp8yhq/0edbd92df665ca64c3e8ee66cd230044d696db4d-primary.xml.gz /tmp/tmpfp8yhq/d6c418b2d5f6cff6cea9f592c7d955f054bd7740-filelists.xml.gz /tmp/tmpfp8yhq/packages /tmp/tmpfp8yhq/cachecookie /tmp/tmpHyaFkr /tmp/tmpHyaFkr/repomd.xml /tmp/tmpHyaFkr/0edbd92df665ca64c3e8ee66cd230044d696db4d-primary.xml.gz /tmp/tmpHyaFkr/d6c418b2d5f6cff6cea9f592c7d955f054bd7740-filelists.xml.gz /tmp/tmpHyaFkr/packages /tmp/tmpHyaFkr/cachecookie /tmp/tmposNzLl /tmp/tmposNzLl/repomd.xml /tmp/tmposNzLl/027ae7184275c41642d039bd67fc46bbd612b740-primary.xml.gz /tmp/tmposNzLl/8380ad947903da0951a1289558dbe9f696c37dac-filelists.xml.gz /tmp/tmposNzLl/packages /tmp/tmposNzLl/cachecookie /tmp/tmpuhz0dn /tmp/tmpuhz0dn/repomd.xml /tmp/tmpuhz0dn/0edbd92df665ca64c3e8ee66cd230044d696db4d-primary.xml.gz /tmp/tmpuhz0dn/d6c418b2d5f6cff6cea9f592c7d955f054bd7740-filelists.xml.gz /tmp/tmpuhz0dn/packages /tmp/tmpuhz0dn/cachecookie /tmp/tmpUz1M5d /tmp/tmpUz1M5d/repomd.xml /tmp/tmpUz1M5d/027ae7184275c41642d039bd67fc46bbd612b740-primary.xml.gz /tmp/tmpUz1M5d/8380ad947903da0951a1289558dbe9f696c37dac-filelists.xml.gz /tmp/tmpUz1M5d/packages /tmp/tmpUz1M5d/cachecookie /tmp/tmpW4XNjF /tmp/tmpW4XNjF/repomd.xml /tmp/tmpW4XNjF/027ae7184275c41642d039bd67fc46bbd612b740-primary.xml.gz /tmp/tmpW4XNjF/8380ad947903da0951a1289558dbe9f696c37dac-filelists.xml.gz /tmp/tmpW4XNjF/packages /tmp/tmpW4XNjF/cachecookie /tmp/tmpxYWsHz /tmp/tmpxYWsHz/repomd.xml /tmp/tmpxYWsHz/0edbd92df665ca64c3e8ee66cd230044d696db4d-primary.xml.gz /tmp/tmpxYWsHz/d6c418b2d5f6cff6cea9f592c7d955f054bd7740-filelists.xml.gz /tmp/tmpxYWsHz/packages /tmp/tmpxYWsHz/cachecookie
Ok, now that I look at that output I realize that those are actually from failed syncs, so scratch my comment earlier about there not being any failed syncs. That being said, the fix is probably the same, we need to make sure we always delete these files even if the repo sync failed. That is likely to happen more often than one might expect.
Fixed. Always clean up the yum repo cache (tmp) cache directory. Even when an exception is raised during the sync. commit: b895db1c3b3752960b7e9a9b9a1910e5d6b50aa3
Fixed in grinder.
set tracker bug. 746803
I ran out of disk space 99% and had to reboot the rhua node, extended the disk space and started again. After that I see again these multiple /tmp/tmp* dir's. [root@dhcp201-133 tmp]# find /tmp/tmp* /tmp/tmp0qIQFF /tmp/tmp0sc9YD /tmp/tmp0sc9YD/cachecookie /tmp/tmp0sc9YD/e470e5ddbe858f7ae0c735350d5f32b53e3df5ac-primary.xml.gz /tmp/tmp0sc9YD/92c90a3f9ea3c5510c7e6b7d2779643f4ccefc14-comps-rhel-x86_64-server-6.xml /tmp/tmp0sc9YD/repomd.xml /tmp/tmp0sc9YD/packages /tmp/tmp0v6YVF /tmp/tmp0v6YVF/cachecookie /tmp/tmp0v6YVF/comps-rhel-x86_64-server-6.xml.gz /tmp/tmp0v6YVF/92c90a3f9ea3c5510c7e6b7d2779643f4ccefc14-comps-rhel-x86_64-server-6.xml /tmp/tmp0v6YVF/repomd.xml /tmp/tmp0v6YVF/503abe2074d4c9ebe88f7a0c58380e1e5373ce4b0b6667c331d790a33e299cb7-updateinfo.xml.gz /tmp/tmp0v6YVF/d21ac9839d5ae302e267d5ba224b0a6484bf780c-primary.xml.gz /tmp/tmp0v6YVF/packages /tmp/tmp0v6YVF/bf7a4f9943fa0e992510a481ce1fc76e455191df-filelists.xml.gz /tmp/tmp0v6YVF/13d7b37229bded6c78001549a91ad171d36d7f96-other.xml.gz /tmp/tmp0v6YVF/d21ac9839d5ae302e267d5ba224b0a6484bf780c-primary.xml.gz.sqlite /tmp/tmp3lSJrB /tmp/tmp3lSJrB/cachecookie /tmp/tmp3lSJrB/repomd.xml /tmp/tmp3lSJrB/packages /tmp/tmp3lSJrB/0a7529eb7d9c9d73b22871a9be026c7b5c7564ce-primary.xml.gz /tmp/tmp3lSJrB/96c2e1f89112992730389d5c82adf40b59f57c63-filelists.xml.gz /tmp/tmp3lSJrB/10fd60e546e2d38c12f27b49c787e2295244625d-comps-rhel-i386-server-5.xml /tmp/tmp9bGEkq /tmp/tmpcR2eKH /tmp/tmpg34tOh /tmp/tmpg34tOh/cachecookie /tmp/tmpg34tOh/repomd.xml /tmp/tmpg34tOh/productid.gz /tmp/tmpg34tOh/dce555efdebf77db39792ad2a2d39f772002e5fd-filelists.xml.gz /tmp/tmpg34tOh/2eeedd15377f34a36900d212b100f9d97e112144-primary.xml.gz.sqlite /tmp/tmpg34tOh/2eeedd15377f34a36900d212b100f9d97e112144-primary.xml.gz /tmp/tmpg34tOh/packages /tmp/tmpg34tOh/10fd60e546e2d38c12f27b49c787e2295244625d-comps-rhel-i386-server-5.xml /tmp/tmpg34tOh/b52acb405f84915e5b1c2b3850f4aa9a59fdbead-other.xml.gz /tmp/tmpg34tOh/e197d344126aa8b9e1d6235ce22775dd9c16864c-updateinfo.xml.gz /tmp/tmpg34tOh/comps-rhel-i386-server-5.xml.gz /tmp/tmpmmpK0N /tmp/tmpnXorxD /tmp/tmpOYLf7v /tmp/tmpOYLf7v/cachecookie /tmp/tmpOYLf7v/4e6a32936c23e093d2fe8de312732c84833cb394-comps-rhel-i386-server-6.xml /tmp/tmpOYLf7v/77d22557bf3fb1f7bcb03bada05904ffcbc8860e-other.xml.gz /tmp/tmpOYLf7v/b131e3e3a4429f5124886f5b42a5da5cd594691b1cbc19d8aaa07f73729656c6-updateinfo.xml.gz /tmp/tmpOYLf7v/repomd.xml /tmp/tmpOYLf7v/d550655e2d48e7ea14883a1b80fb0297e5f5db8c-filelists.xml.gz /tmp/tmpOYLf7v/2604b9bbba83bc689fac333317dcf2891bc5c052-primary.xml.gz /tmp/tmpOYLf7v/comps-rhel-i386-server-6.xml.gz /tmp/tmpOYLf7v/packages /tmp/tmpOYLf7v/2604b9bbba83bc689fac333317dcf2891bc5c052-primary.xml.gz.sqlite /tmp/tmptLRcOZ /tmp/tmptLRcOZ/cachecookie /tmp/tmptLRcOZ/df9c451fa1a3e3872d3453923e78c55403699ddd-other.xml.gz /tmp/tmptLRcOZ/comps-rhel-x86_64-server-5.xml.gz /tmp/tmptLRcOZ/87b78cae8caf76f232374d6a8f8392b8c7fb0239-comps-rhel-x86_64-server-5.xml /tmp/tmptLRcOZ/repomd.xml /tmp/tmptLRcOZ/productid.gz /tmp/tmptLRcOZ/b2a5fb8d7db9f54c7a4cfa1003357bbbb9d73fa6-primary.xml.gz /tmp/tmptLRcOZ/2eb500fe37d44c792f68191f884c65940225d0db-updateinfo.xml.gz /tmp/tmptLRcOZ/920b4b64087c4b9a3df7851ba05e81c2d938c765-filelists.xml.gz /tmp/tmptLRcOZ/packages /tmp/tmptLRcOZ/b2a5fb8d7db9f54c7a4cfa1003357bbbb9d73fa6-primary.xml.gz.sqlite /tmp/tmpvPADvg [root@dhcp201-133 tmp]# rpm -qav | grep -ie pulp -ie rhui pulp-client-0.0.214-3.el6.noarch rh-rhui-tools-2.0.44-1.el6.noarch pulp-common-0.0.214-3.el6.noarch pulp-0.0.214-3.el6.noarch
Few questions: Do you have /tmp configured to be purged on reboot? Did you do an: rm -rf /tmp/tmp* between tests?
Nothing is configured which will purge /tmp dir nor ever ran 'rm -rf /tmp/tmp* ' command between the tests. The comments2 and comments3 in this bug suggest that they should be deleted if syncs fail. But I am not sure, exactly when these need to be deleted. May be you can clarify when it needs to get deleted, meaning what stage/flow. so that we check more specifically this issue.
The original fix addressed scenarios where the sync failed. I believe the /tmp/tmp* directories are leaking when the process doing the sync is killed or dies. This may happen when httpd is restarted or kill (SIGHUP when logrotate) or machine reboot. I revised the tmpdir handling to clean up tmpdirs of dead processes. I also moved the tmpdirs to: /tmp/grinder/<pid>/<random> and write a .info file containing the repo label for traceability.
[root@dhcp201-133 tmpiXBzg9]# cd /tmp/grinder/ [root@dhcp201-133 grinder]# ls 9940 [root@dhcp201-133 grinder]# cd 9940 [root@dhcp201-133 9940]# ls [root@dhcp201-133 9940]# pwd /tmp/grinder/9940 [root@dhcp201-133 9940]# ls -a . .. I only see the /tmp/grinder/<pid> part and not the /tmp/grinder/<pid>/<random> Ok another thing is on restart of the pulp-server , the old directory gets deleted and a new dir with the new process-id is created.
This is all normal. After a sync completes, you will see /tmp/grinder/<pid> remain. This directory is reused between threads within the same process. After each sync is completed, the <random> directory which contains all the cached data for that sync is deleted. The /tmp/grinder/<pid> directories associated with dead processes are leaned up at the beginning of each sync to prevent leakage. When you did the httpd restart, the /tmp/grinder/<pid> directory associated with the old wsgi (pulp application) process was removed when the next sync was started (or resumed). This is all normal and does not represent a leak. However, if you saw /tmp/grinder/<pid>/<random> directory piling up - that would be a problem.
thanks for the confirmation. Moving to verified. Now I understand this. It removes the tmpdir immediately after its content is moved to repodata.new that is to its respective repo. [root@dhcp201-133 ~]# cd /tmp/grinder [root@dhcp201-133 grinder]# ls 10309 [root@dhcp201-133 grinder]# cd 10309 [root@dhcp201-133 10309]# ls FE00DB391 [root@dhcp201-133 10309]# cd FE00DB391/ [root@dhcp201-133 FE00DB391]# ls cachecookie e559cce9a453e50c6a26a5cbae3c120c4c2fb066-primary.xml.gz packages repomd.xml [root@dhcp201-133 FE00DB391]# ll total 1432 -rw-r--r--. 1 apache apache 0 Oct 26 19:57 cachecookie -rw-r--r--. 1 apache apache 1458176 Oct 26 19:58 e559cce9a453e50c6a26a5cbae3c120c4c2fb066-primary.xml.gz drwxr-xr-x. 2 apache apache 4096 Oct 26 19:57 packages -rw-r--r--. 1 apache apache 2432 Oct 26 14:40 repomd.xml [root@dhcp201-133 FE00DB391]# ll total 1632 -rw-r--r--. 1 apache apache 0 Oct 26 19:57 cachecookie -rw-r--r--. 1 apache apache 1662976 Oct 26 19:58 e559cce9a453e50c6a26a5cbae3c120c4c2fb066-primary.xml.gz drwxr-xr-x. 2 apache apache 4096 Oct 26 19:57 packages -rw-r--r--. 1 apache apache 2432 Oct 26 14:40 repomd.xml [root@dhcp201-133 ~]# cd - /var/lib/pulp/repos/content/dist/rhel/rhui/server/6/6Server/i386/os/repodata.new [root@dhcp201-133 repodata.new]# ls 27d27bfab36e4613a6da5b91833ea43773286f20-filelists.xml.gz 6c73a168dab681170d0dd7b0a17f58670473483234f416d82b2acd8e59b4728c-updateinfo.xml.gz productid.gz 4e6a32936c23e093d2fe8de312732c84833cb394-comps-rhel-i386-server-6.xml comps-rhel-i386-server-6.xml.gz repomd.xml 61c61ff923e37ee4bf1d8c731fac4e0cfe59465e-other.xml.gz e559cce9a453e50c6a26a5cbae3c120c4c2fb066-primary.xml.gz [root@dhcp201-133 repodata.new]# ll total 13700 -rw-r--r--. 1 apache apache 6816384 Oct 26 20:01 27d27bfab36e4613a6da5b91833ea43773286f20-filelists.xml.gz -rw-r--r--. 1 apache apache 1086916 Oct 26 19:59 4e6a32936c23e093d2fe8de312732c84833cb394-comps-rhel-i386-server-6.xml -rw-r--r--. 1 apache apache 1094806 Oct 26 20:01 61c61ff923e37ee4bf1d8c731fac4e0cfe59465e-other.xml.gz -rw-r--r--. 1 apache apache 454426 Oct 26 20:01 6c73a168dab681170d0dd7b0a17f58670473483234f416d82b2acd8e59b4728c-updateinfo.xml.gz -rw-r--r--. 1 apache apache 201067 Oct 26 20:01 comps-rhel-i386-server-6.xml.gz -rw-r--r--. 1 apache apache 4350070 Oct 26 20:01 e559cce9a453e50c6a26a5cbae3c120c4c2fb066-primary.xml.gz -rw-r--r--. 1 apache apache 1690 Oct 26 20:01 productid.gz -rw-r--r--. 1 apache apache 2432 Oct 26 20:01 repomd.xml [root@dhcp201-133 repodata.new]# date Wed Oct 26 20:02:55 IST 2011 [root@dhcp201-133 repodata.new]# cd /tmp/grinder [root@dhcp201-133 grinder]# ls 10309 [root@dhcp201-133 grinder]# cd 10309/ [root@dhcp201-133 10309]# ls
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: Do not document
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0367