Bug 805046 - ks distribution directory traceback after upgrading to RHUI 2.0.2
Summary: ks distribution directory traceback after upgrading to RHUI 2.0.2
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Update Infrastructure for Cloud Providers
Classification: Red Hat
Component: RHUA
Version: 2.0.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: James Slagle
QA Contact: wes hayutin
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-03-20 14:12 UTC by James Slagle
Modified: 2012-05-01 18:37 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
C: ?? C: After upgrading to RHUI 2.0.2, the next round of scheduled repository synchronisations failed, and tracebacks were recorded in the pulp.log logfile. The following synchronisations occurred successfully without changes. F: a fix was created to skip distribution synchronisation in a RHUI environment. An option was added to allow users to toggle this behaviour. R: The synchronisation completed successfully without displaying a traceback. Final: * After upgrading to RHUI 2.0.2, the next round of scheduled repository synchronisations would fail, and tracebacks were recorded in the pulp.log logfile. The following synchronisations would be successful without changes. A fix was created to skip distribution synchronisation in a RHUI environment. An option was added to allow users to toggle this behaviour. The synchronisation now completes successfully without displaying a traceback. (BZ#805046)
Clone Of:
Environment:
Last Closed: 2012-05-01 18:37:15 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
pulp.log (5.80 MB, application/octet-stream)
2012-03-20 14:20 UTC, James Slagle
no flags Details
grinder.log.2 (9.54 MB, application/x-troff-man)
2012-03-20 14:28 UTC, James Slagle
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2012:0539 0 normal SHIPPED_LIVE Red Hat Update Infrastructure Bug Fix Update 2012-05-03 23:46:59 UTC

Description James Slagle 2012-03-20 14:12:39 UTC
I upgraded to RHUI 2.0.2.  On the next round of scheduled repo syncs, the following tracebacks were in the log:

2012-03-19 23:18:09,370 8741:139971306452736: pulp.server.tasking.task:ERROR: task:474 Task failed: Task 58b92151-720c-11e1-8950-1231391025b1: _sync(rhel-server-6-6Server-i386, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7f4de1995190>, progress_callback=<bound method RepoSyncTask.progress_callback of <pulp.server.api.repo_sync_task.RepoSyncTask object at 0x7f4de1995050>>)
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 420, in run
    result = self.callable(*self.args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 272, in _sync
    progress_callback, synchronizer, max_speed, threads)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 362, in fetch_content
    synchronizer.add_distribution_from_dir(repo_dir, repo_id, skip_dict)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 344, in add_distribution_from_dir
    self._process_repo_images(dir, repo)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 459, in _process_repo_images
    files = pulp.server.util.listdir(distro_path) or []
  File "/usr/lib/python2.6/site-packages/pulp/server/util.py", line 373, in listdir
    raise Exception("Cannot read from directory %s" % directory)
Exception: Cannot read from directory /var/lib/pulp/distributions/ks-Red Hat Enterprise Linux-Server-6.1-i386

2012-03-19 23:18:09,488 8741:139971306452736: pulp.server.tasking.task:WARNING: task:409 Task 58653b6e-720c-11e1-872a-1231391025b1: _sync(rhel-server-6-srpms-6Server-i386, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7f4de1664190>, progress_callback=<bound method RepoSyncTask.progress_callback of <pulp.server.api.repo_sync_task.RepoSyncTask object at 0x7f4de1664050>>)
started 0:49:46.487607 after its scheduled start time
2012-03-19 23:18:09,495 8741:139971306452736: pulp.server.api.repo_sync:INFO: repo_sync:254 Sync invoked for repo <rhel-server-6-srpms-6Server-i386>
2012-03-19 23:18:09,566 8741:139971306452736: pulp.server.api.repo_sync:INFO: repo_sync:268 Sync of rhel-server-6-srpms-6Server-i386 starting, skip_dict = {}
2012-03-19 23:18:09,591 8741:139971306452736: pulp.server.api.synchronizers:INFO: synchronizers:726 cacert = </etc/pki/pulp/content/rhel-server-6-srpms-6Server-i386/feed-rhel-server-6-srpms-6Server-i386.ca>, cert = </etc/pki/pulp/content/rhel-server-6-srpms-6Server-i386/feed-rhel-server-6-srpms-6Server-i386.cert>
2012-03-19 23:18:09,592 8741:139971306452736: pulp.server.api.synchronizers:INFO: synchronizers:775 Fetching repo to </var/lib/pulp//repos/content/dist/rhel/rhui/server/6/6Server/i386/source/SRPMS> with verify_options <{'checksum': False, 'size': False}>
2012-03-19 23:18:37,959 8741:139971306452736: pulp.server.api.synchronizers:INFO: synchronizers:783 YumSynchronizer reported 2725 successes, 0 downloads, 0 errors
2012-03-19 23:22:36,831 8741:139971295962880: pulp.server.api.synchronizers:INFO: synchronizers:810 preserve metadata flag is set; skipping metadata update
2012-03-19 23:22:37,311 8741:139971295962880: pulp.server.tasking.task:ERROR: task:474 Task failed: Task 58e08030-720c-11e1-8a52-1231391025b1: _sync(rhel-server-6-6Server-x86_64, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7f4de133c190>, progress_callback=<bound method RepoSyncTask.progress_callback of <pulp.server.api.repo_sync_task.RepoSyncTask object at 0x7f4de133c050>>)
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 420, in run
    result = self.callable(*self.args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 272, in _sync
    progress_callback, synchronizer, max_speed, threads)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 362, in fetch_content
    synchronizer.add_distribution_from_dir(repo_dir, repo_id, skip_dict)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 344, in add_distribution_from_dir
    self._process_repo_images(dir, repo)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 459, in _process_repo_images
    files = pulp.server.util.listdir(distro_path) or []
  File "/usr/lib/python2.6/site-packages/pulp/server/util.py", line 373, in listdir
    raise Exception("Cannot read from directory %s" % directory)
Exception: Cannot read from directory /var/lib/pulp/distributions/ks-Red Hat Enterprise Linux-Server-6.1-x86_64

Comment 1 James Slagle 2012-03-20 14:20:17 UTC
Created attachment 571425 [details]
pulp.log

attaching full pulp log

Comment 2 James Slagle 2012-03-20 14:26:01 UTC
From what I can tell in the attached pulp log the sync of rhel-server-6-6Server-x86_64 failed at 3-19 11:22 pm.  The distribution was then created successfully by the sync of rhel-server-6-beta-6Server-x86_64 at 3-19 12:09 pm:

2012-03-20 00:09:10,983 8741:139971295962880: pulp.server.api.synchronizers:INFO: synchronizers:469 Created a distributionID ks-Red Hat Enterprise Linux-Server-6.1-x86_64

The next time the sync of rhel-server-6-6Server-x86_64 ran, it then was able to finish successfully.

We need to find out why the sync of the beta repo passed and saw the need to create the directory, but the initial sync of the non-beta repo assumed the directory already existed.

This is the same thing that happened for i386 too btw.

Comment 3 James Slagle 2012-03-20 14:28:54 UTC
Created attachment 571430 [details]
grinder.log.2

attaching relevent grinder.log portion as well

Comment 4 James Slagle 2012-03-20 20:15:44 UTC
although there definitely seems to be some sort of bug here, no one knows what it is, so we're going to add code to skip distribution sync in a rhui environment.

need to add a db migration to set this option on existing repos and update rhui tools to create new repos with this option set.

committed to pulp rhui: fcea61d6eb46b1e916f88714f0c366a7e1d9911d

Comment 5 James Slagle 2012-03-20 20:23:01 UTC
committed to cloude 1262374ec51f43110cfd0eb19bd92db2e7e8350f

Comment 6 James Slagle 2012-03-23 13:23:18 UTC
QA:
after updating to 2.0.3, or a fresh install of 2.0.3, you should never see this traceback.

also, no distributions should be sync'd to the RHUI environment.  You should no distribution files or images under /var/lib/pulp/distributions and should see log messages like the following in /var/log/pulp/pulp.log:
2012-03-23 06:27:09,083 26650:139688998483712: pulp.server.api.synchronizers:INFO: synchronizers:346 skipping distribution imports from sync process

Comment 7 mkovacik 2012-03-26 10:30:31 UTC
Verified:
 - no 'cannot read' exceptions in pulp logs
 - 'skipping distribution' present in pulp logs
 - no distributions directory in /var/lib/pulp

Version: RHEL-6.2-RHUI-2.0.3-20120322.0-Server-x86_64-DVD1.iso


### Screen log
[root@dhcp-31-120 ~]# grep 'synchronizers:346\ skipping\ distribution' /var/log/pulp/pulp.log                                                                                                                                                                                                                           
2012-03-23 19:57:48,338 14236:140234557716224: pulp.server.api.synchronizers:INFO: synchronizers:346 skipping distribution imports from sync process
2012-03-23 19:58:56,691 14236:140234440308480: pulp.server.api.synchronizers:INFO: synchronizers:346 skipping distribution imports from sync process
2012-03-23 23:44:32,340 14561:139847872272128: pulp.server.api.synchronizers:INFO: synchronizers:346 skipping distribution imports from sync process
2012-03-24 01:58:02,526 14561:139847851292416: pulp.server.api.synchronizers:INFO: synchronizers:346 skipping distribution imports from sync process

[root@dhcp-31-120 ~]# grep 'cannot\ read' /var/log/pulp/*
[root@dhcp-31-120 ~]# echo $?
1

[root@dhcp-31-120 ~]# ls /var/lib/pulp/distributions
ls: cannot access /var/lib/pulp/distributions: No such file or directory

Comment 8 Lana Brindley 2012-04-20 00:58:04 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
C: ??
C: After upgrading to RHUI 2.0.2, the next round of scheduled repository synchronisations failed, and tracebacks were recorded in the pulp.log logfile. The following synchronisations occurred successfully without changes. 
F: a fix was created to skip distribution synchronisation in a RHUI environment. An option was added to allow users to toggle this behaviour.
R: The synchronisation completed successfully without displaying a traceback.

Comment 9 Lana Brindley 2012-04-20 04:22:33 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,4 +1,7 @@
 C: ??
 C: After upgrading to RHUI 2.0.2, the next round of scheduled repository synchronisations failed, and tracebacks were recorded in the pulp.log logfile. The following synchronisations occurred successfully without changes. 
 F: a fix was created to skip distribution synchronisation in a RHUI environment. An option was added to allow users to toggle this behaviour.
-R: The synchronisation completed successfully without displaying a traceback.+R: The synchronisation completed successfully without displaying a traceback.
+
+Final:
+* After upgrading to RHUI 2.0.2, the next round of scheduled repository synchronisations would fail, and tracebacks were recorded in the pulp.log logfile. The following synchronisations would be successful without changes. A fix was created to skip distribution synchronisation in a RHUI environment. An option was added to allow users to toggle this behaviour. The synchronisation now completes successfully without displaying a traceback. (BZ#805046)

Comment 10 errata-xmlrpc 2012-05-01 18:37:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2012-0539.html


Note You need to log in before you can comment on or make changes to this bug.