repo sync's are staged in /tmp/ before being moved over to /var/lib/pulp. If there are multiple large syncs going on at once, this could easily run /tmp/ out of space since /tmp is often being used as tmpfs or the / partition on a system is small. The installation guide says to make /var/lib/pulp big enough to hold all of your repositories, but says nothing about /tmp/.
Created attachment 522696 [details] httpd error log log shows usage of /tmp and running out of space
After further investigation, I noticed that df and du on the system were reporting very different output. I googled around for why and found a helpful links: http://www.cyberciti.biz/tips/freebsd-why-command-df-and-du-reports-different-output.html grinder caches the repo metadata in /tmp. And it's possible it's keeping the file descriptors open even once the repo sync is completed.
Here's output from a system: [root@rhui2 /]# df -h Filesystem Size Used Avail Use% Mounted on /dev/xvde1 5.7G 3.5G 2.2G 62% / none 3.5G 0 3.5G 0% /dev/shm /dev/xvdl1 296G 99G 182G 36% /var/lib/pulp /dev/xvdm1 20G 1.2G 18G 7% /var/lib/mongodb /root/deploy/RHEL-6.1-RHUI-2.0-20110727.2-Server-x86_64-DVD1.iso 62M 62M 0 100% /mnt/rhuiso [root@rhui2 /]# du -h --max-depth 1 --exclude ./var/lib/pulp --exclude ./var/lib/mongodb 24K ./srv 0 ./sys 4.0K ./home 0 ./misc 27M ./etc 14M ./sbin 63M ./root 31M ./lib64 du: cannot access `./proc/20412/task/20412/fd/4': No such file or directory du: cannot access `./proc/20412/task/20412/fdinfo/4': No such file or directory du: cannot access `./proc/20412/fd/4': No such file or directory du: cannot access `./proc/20412/fdinfo/4': No such file or directory 0 ./proc 0 ./selinux 0 ./net 4.0K ./media 109M ./lib 61M ./mnt 8.9M ./bin 98M ./tmp 136K ./dev 1.6G ./usr 4.0K ./opt 365M ./var 16K ./lost+found 20M ./boot 2.4G . Notice df reports 3.5 GB used on /, but du only finds 2.4 GB used. Here's just a snippet of lsof output from the main wsgi process (pid 12548): httpd 12548 apache 166ur REG 202,65 8749056 35995 /tmp/tmpmPbbR2/f35bc58a174945d03a0fe9564cf378e7004381f3-primary.xml.gz.sqlite (deleted) httpd 12548 apache 170ur REG 202,65 8749056 35999 /tmp/tmp4K4fsI/f35bc58a174945d03a0fe9564cf378e7004381f3-primary.xml.gz.sqlite (deleted) httpd 12548 apache 171ur REG 202,65 6194176 36010 /tmp/tmpqZI0rT/98c3c5d7c133589e180d37bd847bd23756470a0c-primary.xml.gz.sqlite (deleted) httpd 12548 apache 172ur REG 202,65 1917952 36012 /tmp/tmp_9DsOx/3449a6b4a4c05bb6443eefbd0ef497feb4a156e5-primary.xml.gz.sqlite (deleted) httpd 12548 apache 173ur REG 202,65 1917952 36009 /tmp/temp_pulp_repov0CJHy/3449a6b4a4c05bb6443eefbd0ef497feb4a156e5-primary.xml.gz.sqlite (deleted) httpd 12548 apache 174ur REG 202,65 3040256 35984 /tmp/tmpMQJDMo/4a8801e3c4906b60a0e9493d38ea53600a493afa-primary.xml.gz.sqlite (deleted) Notice the files say deleted, and indeed the first one is not on the filesystem: [root@rhui2 /]# ls /tmp/tmpmPbbR2/f35bc58a174945d03a0fe9564cf378e7004381f3-primary.xml.gz.sqlite ls: cannot access /tmp/tmpmPbbR2/f35bc58a174945d03a0fe9564cf378e7004381f3-primary.xml.gz.sqlite: No such file or directory However, the process still has them open, and we can copy the full file out of /proc. cp /proc/12548/fd/166 /tmp/primary.xml.gz.sqlite It's the full sqlite db: [root@rhui2 /]# sqlite3 /tmp/primary.xml.gz.sqlite SQLite version 3.6.20 Enter ".help" for instructions Enter SQL statements terminated with a ";" sqlite> select count(*) from files; 3506 sqlite>
Looks like (in grinder) RepoFetch is creating a YumRepository object which contains a YumSqlitePackageSack object. Then, RepoFetch.getPackageList() is called which calls YumSqlitePackageSack.populate(). This opens the sqlite connection(s). The problem is that YumFetchGrinder calls RepoFetch.setupRepo() but never calls YumRepository.close(). I traced through YumRepository.close() in the debugger and verified it closes the sqlite DB connections. So in class YumFetchGrinder : def fetchYumRepo(...): try: self.yumFetch.setupRepo() except: self.fetchPkgs.stop() raise Needs to be: def fetchYumRepo(...): try: self.yumFetch.setupRepo() finally: self.fetchPkgs.stop() self.yumFetch.closeRepo() In class RepoFetch, added: def closeRepo(self): self.repo.close() Testing the fix now.
I tagged and rebuilt grinder this morning as grinder-0.0.113-1.el6.noarch. Installed that and restarted pulp-server. I'm still seeing the same issue, both disk space and memory usage continue to climb as it churns through the repo syncs. It's only the primary.xml.gz.sqlite file that is being kept open. The main process has 40 primary.xml.gz.sqlite files open, and that corresponds with exactly how many repos it has done so far. It's finished 36 so far, and is working on the next 4: [root@ec2-107-20-207-64 cron.d]# lsof -p 5379 | grep primary.xml.gz.sqlite | wc -l 40
Full lsof output: [root@ec2-107-20-207-64 cron.d]# lsof -p 5379 | grep primary.xml.gz.sqlite httpd 5379 apache 21ur REG 202,65 20480 35624 /tmp/temp_pulp_repoHqdtM4/primary.xml.gz.sqlite (deleted) httpd 5379 apache 27ur REG 202,65 20480 35643 /tmp/temp_pulp_repoU7a8ne/primary.xml.gz.sqlite (deleted) httpd 5379 apache 28ur REG 202,65 20480 35634 /tmp/temp_pulp_repodknXrh/primary.xml.gz.sqlite (deleted) httpd 5379 apache 31ur REG 202,65 20480 35607 /tmp/temp_pulp_repoqxAx7x/primary.xml.gz.sqlite (deleted) httpd 5379 apache 32ur REG 202,65 20480 35628 /tmp/temp_pulp_repoGfiWoU/primary.xml.gz.sqlite (deleted) httpd 5379 apache 33ur REG 202,65 20480 35625 /tmp/temp_pulp_repooGFqsB/primary.xml.gz.sqlite (deleted) httpd 5379 apache 34ur REG 202,65 44032 35686 /tmp/temp_pulp_repoRycazB/7dbd924b47c052ae4d83cb0a7cf7abb48779abc3-primary.xml.gz.sqlite (deleted) httpd 5379 apache 35ur REG 202,65 20480 35637 /tmp/temp_pulp_repowOCKrt/primary.xml.gz.sqlite (deleted) httpd 5379 apache 36ur REG 202,65 20480 35659 /tmp/temp_pulp_repouo3OZ6/primary.xml.gz.sqlite (deleted) httpd 5379 apache 37ur REG 202,65 20480 35644 /tmp/temp_pulp_repoWbfwiC/primary.xml.gz.sqlite (deleted) httpd 5379 apache 38ur REG 202,65 148480 35632 /tmp/temp_pulp_repoJK7qCt/c03ad509a5080a22213586d564cf0bc8418f12cd-primary.xml.gz.sqlite (deleted) httpd 5379 apache 39ur REG 202,65 20480 35641 /tmp/temp_pulp_repozzEjdO/primary.xml.gz.sqlite (deleted) httpd 5379 apache 40ur REG 202,65 47104 35649 /tmp/temp_pulp_repoeS0Xok/b1ff1e50ecba24a62c1ca57c6af8a73dfb881eb9-primary.xml.gz.sqlite (deleted) httpd 5379 apache 41ur REG 202,65 156672 35684 /tmp/temp_pulp_repo7B8WmF/b49877f6db9a96e55e8e3dad9cb90c06523a7562-primary.xml.gz.sqlite (deleted) httpd 5379 apache 42ur REG 202,65 148480 35658 /tmp/temp_pulp_repoR8cbuq/c03ad509a5080a22213586d564cf0bc8418f12cd-primary.xml.gz.sqlite (deleted) httpd 5379 apache 43ur REG 202,65 149504 35671 /tmp/temp_pulp_repogKF2Ia/26fb48415bb177260b2d48b53796648d3d1b251e-primary.xml.gz.sqlite (deleted) httpd 5379 apache 44ur REG 202,65 47104 35663 /tmp/temp_pulp_repo4cDtIX/854fa8bcd3c607836333117ddb5d7ede3967409e-primary.xml.gz.sqlite (deleted) httpd 5379 apache 45ur REG 202,65 20480 35670 /tmp/temp_pulp_repoKZdacp/primary.xml.gz.sqlite (deleted) httpd 5379 apache 46ur REG 202,65 149504 35655 /tmp/temp_pulp_repor0gyXQ/26fb48415bb177260b2d48b53796648d3d1b251e-primary.xml.gz.sqlite (deleted) httpd 5379 apache 47ur REG 202,65 20480 35653 /tmp/temp_pulp_repoKDBfor/primary.xml.gz.sqlite (deleted) httpd 5379 apache 48ur REG 202,65 20480 35646 /tmp/temp_pulp_repouo2AiZ/primary.xml.gz.sqlite (deleted) httpd 5379 apache 49ur REG 202,65 44032 35665 /tmp/temp_pulp_repoR1xmAy/7dbd924b47c052ae4d83cb0a7cf7abb48779abc3-primary.xml.gz.sqlite (deleted) httpd 5379 apache 50ur REG 202,65 44032 35662 /tmp/temp_pulp_repoqpP98k/75bc6ca6a3cee71c10e1c65ac2bd58d22b27c492-primary.xml.gz.sqlite (deleted) httpd 5379 apache 51ur REG 202,65 20480 35680 /tmp/temp_pulp_repougfMGq/primary.xml.gz.sqlite (deleted) httpd 5379 apache 52ur REG 202,65 20480 35681 /tmp/temp_pulp_repoK7mrOS/primary.xml.gz.sqlite (deleted) httpd 5379 apache 53ur REG 202,65 142336 35673 /tmp/temp_pulp_repojV95ov/637c9aa63a3903aac249cbf6bc8045ed2c592bc4-primary.xml.gz.sqlite (deleted) httpd 5379 apache 54ur REG 202,65 20480 35666 /tmp/temp_pulp_reponCzo5D/primary.xml.gz.sqlite (deleted) httpd 5379 apache 55ur REG 202,65 156672 35675 /tmp/temp_pulp_repoYR7WEw/b49877f6db9a96e55e8e3dad9cb90c06523a7562-primary.xml.gz.sqlite (deleted) httpd 5379 apache 56ur REG 202,65 142336 35695 /tmp/temp_pulp_repopWrMTH/637c9aa63a3903aac249cbf6bc8045ed2c592bc4-primary.xml.gz.sqlite (deleted) httpd 5379 apache 58ur REG 202,65 47104 35689 /tmp/temp_pulp_repoY8umBL/d61a3a33e19386a07ba85945f5df2eb91d707c14-primary.xml.gz.sqlite (deleted) httpd 5379 apache 60ur REG 202,65 48128 35688 /tmp/temp_pulp_repoThpyAp/3ba1ac9cef36ada33ea459a50e8200d1cbb3abfa-primary.xml.gz.sqlite (deleted) httpd 5379 apache 62ur REG 202,65 47104 35708 /tmp/temp_pulp_repo2NDboc/d61a3a33e19386a07ba85945f5df2eb91d707c14-primary.xml.gz.sqlite (deleted) httpd 5379 apache 63ur REG 202,65 48128 35701 /tmp/temp_pulp_repoYpcCrc/3ba1ac9cef36ada33ea459a50e8200d1cbb3abfa-primary.xml.gz.sqlite (deleted) httpd 5379 apache 75ur REG 202,65 47104 35651 /tmp/temp_pulp_repoQx9JT0/b1ff1e50ecba24a62c1ca57c6af8a73dfb881eb9-primary.xml.gz.sqlite (deleted) httpd 5379 apache 76ur REG 202,65 47104 35657 /tmp/temp_pulp_repoLhmPXH/854fa8bcd3c607836333117ddb5d7ede3967409e-primary.xml.gz.sqlite (deleted) httpd 5379 apache 80ur REG 202,65 32225280 35736 /tmp/tmpZZmMEs/4bf97c5be84e894005c81772ed963aba43d9cb5f-primary.xml.gz.sqlite httpd 5379 apache 81ur REG 202,65 44032 35706 /tmp/temp_pulp_repoXqOotZ/75bc6ca6a3cee71c10e1c65ac2bd58d22b27c492-primary.xml.gz.sqlite (deleted) httpd 5379 apache 85ur REG 202,65 31955968 35737 /tmp/tmp3IA_z_/d4ff61bef3ebbdbd27922056ab06e17aaf072993c7e85326839221450ee58389-primary.xml.gz.sqlite httpd 5379 apache 95ur REG 202,65 40440832 35738 /tmp/tmp9bZQz9/cd9e660b33f1bf845292e9e19e01ec21a9428ab5-primary.xml.gz.sqlite httpd 5379 apache 104ur REG 202,65 40366080 35739 /tmp/tmpTd9Rbc/eef9c6d19499820a6e01fd66d8645988ed316649fcab4af53375c7411f495314-primary.xml.gz.sqlite
The pulp/server/util.py get_repo_packages() is also not closing the YumRepository which leaks package sack objects. This fix is slightly more complicated because get_repo_packages() was returning actual yum package objects which has a reference to the repository. The fix is to have it return data objects instead and have get_repo_packages() close the repo after using it. Testing the fix now. note: Looks like server/api/depsolver.py is also not closing the YumRepository but does not affect the sync. This should also be fixed.
pulp fix (hash): ebf754347617ec5a87125a596c81719e48302ec9
I've cherry picked that commit into the rhui branch, and from there pulled that into the pulp-ec2 repository on axiom. Rebuilding pulp-ec2 in brew now and will start a full sync again to test the fix on the rhua.
Applied this update to my test RHUA and let the full sync run overnight. At 3:12 am, all the syncs died. It looks like apache was restarted with a SIGHUP at that time (probably by logrotate).
2011-09-14 09:33:55,439 15048:140719716620032: grinder.ParallelFetch:ERROR: ParallelFetch:337 Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/grinder/ParallelFetch.py", line 332, in run result = self.fetcher.fetchItem(itemInfo) File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 91, in __call__ return self.object(self, *args, **kwargs) File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 284, in __call__ return self.__call(method, args, kwargs) File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 260, in __call return self.__rmi(method.name, args, kwargs) File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 138, in __rmi packet = pickle.load(p.stdout) ValueError: I/O operation on closed file
set tracker bug. 746803
Verified this defect with following RHUI ISO: RHEL-6.1-RHUI-2.0.1-20111017.0-Server-x86_64-DVD1.iso I started a repo sync on RHUA for following repos: Red Hat Repositories Red Hat Update Infrastructure 2.0 (RPMs) (6.0-i386) Red Hat Update Infrastructure 2.0 (RPMs) (6.1-i386) Red Hat Update Infrastructure 2.0 (RPMs) (6Server-i386) Red Hat Update Infrastructure 2.0 (RPMs) (6.0-x86_64) Red Hat Update Infrastructure 2.0 (RPMs) (6Server-x86_64) Red Hat Update Infrastructure 2.0 (RPMs) (6.1-x86_64) Red Hat Enterprise Linux Server 6 (RPMs) (6.0-i386) Red Hat Enterprise Linux Server 6 (RPMs) (6.0-x86_64) Red Hat Enterprise Linux Server 6 (RPMs) (6Server-x86_64) Red Hat Enterprise Linux Server 6 (RPMs) (6.1-x86_64) Red Hat Enterprise Linux Server 6 (RPMs) (6Server-i386) Red Hat Enterprise Linux Server 6 (RPMs) (6.1-i386) Red Hat Enterprise Linux Server 5 (RPMs) (5.6-i386) Red Hat Enterprise Linux Server 5 (RPMs) (5.7-i386) Red Hat Enterprise Linux Server 5 (RPMs) (5.6-x86_64) Red Hat Enterprise Linux Server 5 (RPMs) (5.7-x86_64) Red Hat Enterprise Linux Server 6 Optional (RPMs) (6.0-i386) Red Hat Enterprise Linux Server 6 Optional (RPMs) (6.0-x86_64) Red Hat Enterprise Linux Server 6 Optional (RPMs) (6Server-i386) Red Hat Enterprise Linux Server 6 Optional (RPMs) (6.1-x86_64) Red Hat Enterprise Linux Server 6 Optional (RPMs) (6Server-x86_64) Red Hat Enterprise Linux Server 6 Optional (RPMs) (6.1-i386) Red Hat Enterprise Linux Server 5 (RPMs) (5Server-i386) Red Hat Enterprise Linux Server 5 (RPMs) (5Server-x86_64) tmp usage during sync: ======================== [root@dhcp201-152 /]# du -h --max-depth=1 112M ./lib 1.3G ./usr 0 ./net 16K ./lost+found 5.9G ./var 14M ./sbin 211M ./tmp 4.0K ./home 4.0K ./opt 0 ./selinux du: cannot access `./proc/23354/task/23354/fd/4': No such file or directory du: cannot access `./proc/23354/task/23354/fdinfo/4': No such file or directory du: cannot access `./proc/23354/fd/4': No such file or directory du: cannot access `./proc/23354/fdinfo/4': No such file or directory 0 ./proc 26M ./lib64 7.7M ./bin 24M ./etc 0 ./misc 4.0K ./cgroup 192K ./dev 46M ./root 0 ./sys 41M ./mnt 4.0K ./media 21M ./boot 20K ./srv 7.7G . # #5 sghai 2011-10-19 18:31:27 [root@dhcp201-152 /]# du -h --max-depth=1 112M ./lib 1.3G ./usr 0 ./net 16K ./lost+found 8.9G ./var 14M ./sbin 211M ./tmp 4.0K ./home 4.0K ./opt 0 ./selinux du: cannot access `./proc/32667/task/32667/fd/4': No such file or directory du: cannot access `./proc/32667/task/32667/fdinfo/4': No such file or directory du: cannot access `./proc/32667/fd/4': No such file or directory du: cannot access `./proc/32667/fdinfo/4': No such file or directory 0 ./proc 26M ./lib64 7.7M ./bin 24M ./etc 0 ./misc 4.0K ./cgroup 192K ./dev 46M ./root 0 ./sys 41M ./mnt 4.0K ./media 21M ./boot 20K ./srv 11G . [root@dhcp201-152 /]# All repos are synced successfully and no error so far. -= Repository Synchronization Status =- Last Refreshed: 10:19:29 (updated every 5 seconds, ctrl+c to exit) Next Sync Last Sync Last Result ------------------------------------------------------------------------------ Red Hat Enterprise Linux Server 5 (RPMs) (5.6-i386) 10-22-2011 11:15 10-22-2011 00:27 Success Red Hat Enterprise Linux Server 5 (RPMs) (5.6-x86_64) 10-22-2011 11:15 10-22-2011 00:05 Success Red Hat Enterprise Linux Server 5 (RPMs) (5.7-i386) 10-22-2011 11:15 10-22-2011 00:40 Success Red Hat Enterprise Linux Server 5 (RPMs) (5.7-x86_64) 10-22-2011 11:15 10-22-2011 01:09 Success Red Hat Enterprise Linux Server 5 (RPMs) (5Server-i386) 10-22-2011 11:15 10-21-2011 23:47 Success Red Hat Enterprise Linux Server 5 (RPMs) (5Server-x86_64) 10-22-2011 11:15 10-22-2011 09:04 Success Red Hat Enterprise Linux Server 6 (RPMs) (6.0-i386) 10-22-2011 10:53 10-22-2011 05:15 Success Red Hat Enterprise Linux Server 6 (RPMs) (6.0-x86_64) 10-22-2011 10:53 10-22-2011 05:06 Success Red Hat Enterprise Linux Server 6 (RPMs) (6.1-i386) 10-22-2011 10:53 10-22-2011 06:49 Success Red Hat Enterprise Linux Server 6 (RPMs) (6.1-x86_64) 10-22-2011 10:53 10-22-2011 08:33 Success Red Hat Enterprise Linux Server 6 (RPMs) (6Server-i386) 10-22-2011 10:53 10-22-2011 08:44 Success Red Hat Enterprise Linux Server 6 (RPMs) (6Server-x86_64) 10-22-2011 10:53 10-22-2011 05:06 Success Red Hat Enterprise Linux Server 6 Optional (RPMs) (6.0-i386) 10-22-2011 11:15 10-22-2011 00:58 Success Red Hat Enterprise Linux Server 6 Optional (RPMs) (6.0-x86_64) 10-22-2011 11:15 10-22-2011 01:08 Success Red Hat Enterprise Linux Server 6 Optional (RPMs) (6.1-i386) 10-22-2011 11:15 10-22-2011 00:53 Success Red Hat Enterprise Linux Server 6 Optional (RPMs) (6.1-x86_64) 10-22-2011 11:15 10-22-2011 01:09 Success Red Hat Enterprise Linux Server 6 Optional (RPMs) (6Server-i386) 10-22-2011 11:15 10-22-2011 01:02 Success Red Hat Enterprise Linux Server 6 Optional (RPMs) (6Server-x86_64) 10-22-2011 11:15 10-22-2011 00:56 Success Red Hat Update Infrastructure 2.0 (RPMs) (6.0-i386) 10-22-2011 15:38 10-22-2011 09:40 Success Red Hat Update Infrastructure 2.0 (RPMs) (6.0-x86_64) 10-22-2011 15:38 10-22-2011 09:43 Success Red Hat Update Infrastructure 2.0 (RPMs) (6.1-i386) 10-22-2011 15:38 10-22-2011 09:40 Success Red Hat Update Infrastructure 2.0 (RPMs) (6.1-x86_64) 10-22-2011 15:38 10-22-2011 09:45 Success Red Hat Update Infrastructure 2.0 (RPMs) (6Server-i386) 10-22-2011 15:38 10-22-2011 09:42 Success Red Hat Update Infrastructure 2.0 (RPMs) (6Server-x86_64) 10-22-2011 15:38 10-22-2011 09:45 Success Connected: dhcp201-152.englab.pnq.redhat.com ------------------------------------------------------------------------------ [root@dhcp201-152 ~]# lsof | grep tmp httpd 5766 apache DEL REG 252,3 394253 /tmp/ffikVZK9U httpd 5766 apache 22u REG 252,3 4096 394253 /tmp/ffikVZK9U (deleted) mongod 28121 mongodb 6u unix 0xffff88003be63080 0t0 11537743 /tmp/mongodb-27017.sock mongod 28121 mongodb 8u unix 0xffff8800378be380 0t0 11537747 /tmp/mongodb-28017.sock [root@dhcp201-152 ~]# /tmp space after sync completion: [root@dhcp201-152 /]# du -h --max-depth=1 112M ./lib 1.3G ./usr 0 ./net 16K ./lost+found 63G ./var 14M ./sbin 244M ./tmp 4.0K ./home 4.0K ./opt 0 ./selinux du: cannot access `./proc/6756/task/6756/fd/4': No such file or directory du: cannot access `./proc/6756/task/6756/fdinfo/4': No such file or directory du: cannot access `./proc/6756/fd/4': No such file or directory du: cannot access `./proc/6756/fdinfo/4': No such file or directory 0 ./proc 26M ./lib64 7.7M ./bin 24M ./etc 0 ./misc 4.0K ./cgroup 192K ./dev 46M ./root 0 ./sys 41M ./mnt 4.0K ./media 21M ./boot 20K ./srv 65G . [root@dhcp201-152 /]# ^Crhui (sync) => ------------------------------------------------------------------------------ -= Red Hat Update Infrastructure Management Tool =- -= CDS Synchronization Status =- Last Refreshed: 12:54:50 (updated every 5 seconds, ctrl+c to exit) cds171 ...................................................... [ UP ] cds207 ...................................................... [ UP ] Next Sync Last Sync Last Result ------------------------------------------------------------------------------ cds171 10-22-2011 17:13 10-22-2011 12:27 Success cds207 10-22-2011 17:11 10-22-2011 12:20 Success Connected: dhcp201-152.englab.pnq.redhat.com ------------------------------------------------------------------------------ All repos are synced successfully. I monitored the tmp usage and all logs files during and after repo sync. however the reported error is not reproducible. both CDS sre synced also and didn't see any error in gofer.log related to tmp. so moving this to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0367