I upgraded to RHUI 2.0.2. On the next round of scheduled repo syncs, the following tracebacks were in the log: 2012-03-19 23:18:09,370 8741:139971306452736: pulp.server.tasking.task:ERROR: task:474 Task failed: Task 58b92151-720c-11e1-8950-1231391025b1: _sync(rhel-server-6-6Server-i386, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7f4de1995190>, progress_callback=<bound method RepoSyncTask.progress_callback of <pulp.server.api.repo_sync_task.RepoSyncTask object at 0x7f4de1995050>>) Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 420, in run result = self.callable(*self.args, **self.kwargs) File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 272, in _sync progress_callback, synchronizer, max_speed, threads) File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 362, in fetch_content synchronizer.add_distribution_from_dir(repo_dir, repo_id, skip_dict) File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 344, in add_distribution_from_dir self._process_repo_images(dir, repo) File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 459, in _process_repo_images files = pulp.server.util.listdir(distro_path) or [] File "/usr/lib/python2.6/site-packages/pulp/server/util.py", line 373, in listdir raise Exception("Cannot read from directory %s" % directory) Exception: Cannot read from directory /var/lib/pulp/distributions/ks-Red Hat Enterprise Linux-Server-6.1-i386 2012-03-19 23:18:09,488 8741:139971306452736: pulp.server.tasking.task:WARNING: task:409 Task 58653b6e-720c-11e1-872a-1231391025b1: _sync(rhel-server-6-srpms-6Server-i386, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7f4de1664190>, progress_callback=<bound method RepoSyncTask.progress_callback of <pulp.server.api.repo_sync_task.RepoSyncTask object at 0x7f4de1664050>>) started 0:49:46.487607 after its scheduled start time 2012-03-19 23:18:09,495 8741:139971306452736: pulp.server.api.repo_sync:INFO: repo_sync:254 Sync invoked for repo <rhel-server-6-srpms-6Server-i386> 2012-03-19 23:18:09,566 8741:139971306452736: pulp.server.api.repo_sync:INFO: repo_sync:268 Sync of rhel-server-6-srpms-6Server-i386 starting, skip_dict = {} 2012-03-19 23:18:09,591 8741:139971306452736: pulp.server.api.synchronizers:INFO: synchronizers:726 cacert = </etc/pki/pulp/content/rhel-server-6-srpms-6Server-i386/feed-rhel-server-6-srpms-6Server-i386.ca>, cert = </etc/pki/pulp/content/rhel-server-6-srpms-6Server-i386/feed-rhel-server-6-srpms-6Server-i386.cert> 2012-03-19 23:18:09,592 8741:139971306452736: pulp.server.api.synchronizers:INFO: synchronizers:775 Fetching repo to </var/lib/pulp//repos/content/dist/rhel/rhui/server/6/6Server/i386/source/SRPMS> with verify_options <{'checksum': False, 'size': False}> 2012-03-19 23:18:37,959 8741:139971306452736: pulp.server.api.synchronizers:INFO: synchronizers:783 YumSynchronizer reported 2725 successes, 0 downloads, 0 errors 2012-03-19 23:22:36,831 8741:139971295962880: pulp.server.api.synchronizers:INFO: synchronizers:810 preserve metadata flag is set; skipping metadata update 2012-03-19 23:22:37,311 8741:139971295962880: pulp.server.tasking.task:ERROR: task:474 Task failed: Task 58e08030-720c-11e1-8a52-1231391025b1: _sync(rhel-server-6-6Server-x86_64, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7f4de133c190>, progress_callback=<bound method RepoSyncTask.progress_callback of <pulp.server.api.repo_sync_task.RepoSyncTask object at 0x7f4de133c050>>) Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 420, in run result = self.callable(*self.args, **self.kwargs) File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 272, in _sync progress_callback, synchronizer, max_speed, threads) File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 362, in fetch_content synchronizer.add_distribution_from_dir(repo_dir, repo_id, skip_dict) File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 344, in add_distribution_from_dir self._process_repo_images(dir, repo) File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 459, in _process_repo_images files = pulp.server.util.listdir(distro_path) or [] File "/usr/lib/python2.6/site-packages/pulp/server/util.py", line 373, in listdir raise Exception("Cannot read from directory %s" % directory) Exception: Cannot read from directory /var/lib/pulp/distributions/ks-Red Hat Enterprise Linux-Server-6.1-x86_64
Created attachment 571425 [details] pulp.log attaching full pulp log
From what I can tell in the attached pulp log the sync of rhel-server-6-6Server-x86_64 failed at 3-19 11:22 pm. The distribution was then created successfully by the sync of rhel-server-6-beta-6Server-x86_64 at 3-19 12:09 pm: 2012-03-20 00:09:10,983 8741:139971295962880: pulp.server.api.synchronizers:INFO: synchronizers:469 Created a distributionID ks-Red Hat Enterprise Linux-Server-6.1-x86_64 The next time the sync of rhel-server-6-6Server-x86_64 ran, it then was able to finish successfully. We need to find out why the sync of the beta repo passed and saw the need to create the directory, but the initial sync of the non-beta repo assumed the directory already existed. This is the same thing that happened for i386 too btw.
Created attachment 571430 [details] grinder.log.2 attaching relevent grinder.log portion as well
although there definitely seems to be some sort of bug here, no one knows what it is, so we're going to add code to skip distribution sync in a rhui environment. need to add a db migration to set this option on existing repos and update rhui tools to create new repos with this option set. committed to pulp rhui: fcea61d6eb46b1e916f88714f0c366a7e1d9911d
committed to cloude 1262374ec51f43110cfd0eb19bd92db2e7e8350f
QA: after updating to 2.0.3, or a fresh install of 2.0.3, you should never see this traceback. also, no distributions should be sync'd to the RHUI environment. You should no distribution files or images under /var/lib/pulp/distributions and should see log messages like the following in /var/log/pulp/pulp.log: 2012-03-23 06:27:09,083 26650:139688998483712: pulp.server.api.synchronizers:INFO: synchronizers:346 skipping distribution imports from sync process
Verified: - no 'cannot read' exceptions in pulp logs - 'skipping distribution' present in pulp logs - no distributions directory in /var/lib/pulp Version: RHEL-6.2-RHUI-2.0.3-20120322.0-Server-x86_64-DVD1.iso ### Screen log [root@dhcp-31-120 ~]# grep 'synchronizers:346\ skipping\ distribution' /var/log/pulp/pulp.log 2012-03-23 19:57:48,338 14236:140234557716224: pulp.server.api.synchronizers:INFO: synchronizers:346 skipping distribution imports from sync process 2012-03-23 19:58:56,691 14236:140234440308480: pulp.server.api.synchronizers:INFO: synchronizers:346 skipping distribution imports from sync process 2012-03-23 23:44:32,340 14561:139847872272128: pulp.server.api.synchronizers:INFO: synchronizers:346 skipping distribution imports from sync process 2012-03-24 01:58:02,526 14561:139847851292416: pulp.server.api.synchronizers:INFO: synchronizers:346 skipping distribution imports from sync process [root@dhcp-31-120 ~]# grep 'cannot\ read' /var/log/pulp/* [root@dhcp-31-120 ~]# echo $? 1 [root@dhcp-31-120 ~]# ls /var/lib/pulp/distributions ls: cannot access /var/lib/pulp/distributions: No such file or directory
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: C: ?? C: After upgrading to RHUI 2.0.2, the next round of scheduled repository synchronisations failed, and tracebacks were recorded in the pulp.log logfile. The following synchronisations occurred successfully without changes. F: a fix was created to skip distribution synchronisation in a RHUI environment. An option was added to allow users to toggle this behaviour. R: The synchronisation completed successfully without displaying a traceback.
Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1,4 +1,7 @@ C: ?? C: After upgrading to RHUI 2.0.2, the next round of scheduled repository synchronisations failed, and tracebacks were recorded in the pulp.log logfile. The following synchronisations occurred successfully without changes. F: a fix was created to skip distribution synchronisation in a RHUI environment. An option was added to allow users to toggle this behaviour. -R: The synchronisation completed successfully without displaying a traceback.+R: The synchronisation completed successfully without displaying a traceback. + +Final: +* After upgrading to RHUI 2.0.2, the next round of scheduled repository synchronisations would fail, and tracebacks were recorded in the pulp.log logfile. The following synchronisations would be successful without changes. A fix was created to skip distribution synchronisation in a RHUI environment. An option was added to allow users to toggle this behaviour. The synchronisation now completes successfully without displaying a traceback. (BZ#805046)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2012-0539.html