Bug 805046 - ks distribution directory traceback after upgrading to RHUI 2.0.2
ks distribution directory traceback after upgrading to RHUI 2.0.2
Status: CLOSED ERRATA
Product: Red Hat Update Infrastructure for Cloud Providers
Classification: Red Hat
Component: RHUA (Show other bugs)
2.0.3
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: James Slagle
wes hayutin
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-03-20 10:12 EDT by James Slagle
Modified: 2012-05-01 14:37 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
C: ?? C: After upgrading to RHUI 2.0.2, the next round of scheduled repository synchronisations failed, and tracebacks were recorded in the pulp.log logfile. The following synchronisations occurred successfully without changes. F: a fix was created to skip distribution synchronisation in a RHUI environment. An option was added to allow users to toggle this behaviour. R: The synchronisation completed successfully without displaying a traceback. Final: * After upgrading to RHUI 2.0.2, the next round of scheduled repository synchronisations would fail, and tracebacks were recorded in the pulp.log logfile. The following synchronisations would be successful without changes. A fix was created to skip distribution synchronisation in a RHUI environment. An option was added to allow users to toggle this behaviour. The synchronisation now completes successfully without displaying a traceback. (BZ#805046)
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-05-01 14:37:15 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
pulp.log (5.80 MB, application/octet-stream)
2012-03-20 10:20 EDT, James Slagle
no flags Details
grinder.log.2 (9.54 MB, application/x-troff-man)
2012-03-20 10:28 EDT, James Slagle
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2012:0539 normal SHIPPED_LIVE Red Hat Update Infrastructure Bug Fix Update 2012-05-03 19:46:59 EDT

  None (edit)
Description James Slagle 2012-03-20 10:12:39 EDT
I upgraded to RHUI 2.0.2.  On the next round of scheduled repo syncs, the following tracebacks were in the log:

2012-03-19 23:18:09,370 8741:139971306452736: pulp.server.tasking.task:ERROR: task:474 Task failed: Task 58b92151-720c-11e1-8950-1231391025b1: _sync(rhel-server-6-6Server-i386, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7f4de1995190>, progress_callback=<bound method RepoSyncTask.progress_callback of <pulp.server.api.repo_sync_task.RepoSyncTask object at 0x7f4de1995050>>)
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 420, in run
    result = self.callable(*self.args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 272, in _sync
    progress_callback, synchronizer, max_speed, threads)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 362, in fetch_content
    synchronizer.add_distribution_from_dir(repo_dir, repo_id, skip_dict)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 344, in add_distribution_from_dir
    self._process_repo_images(dir, repo)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 459, in _process_repo_images
    files = pulp.server.util.listdir(distro_path) or []
  File "/usr/lib/python2.6/site-packages/pulp/server/util.py", line 373, in listdir
    raise Exception("Cannot read from directory %s" % directory)
Exception: Cannot read from directory /var/lib/pulp/distributions/ks-Red Hat Enterprise Linux-Server-6.1-i386

2012-03-19 23:18:09,488 8741:139971306452736: pulp.server.tasking.task:WARNING: task:409 Task 58653b6e-720c-11e1-872a-1231391025b1: _sync(rhel-server-6-srpms-6Server-i386, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7f4de1664190>, progress_callback=<bound method RepoSyncTask.progress_callback of <pulp.server.api.repo_sync_task.RepoSyncTask object at 0x7f4de1664050>>)
started 0:49:46.487607 after its scheduled start time
2012-03-19 23:18:09,495 8741:139971306452736: pulp.server.api.repo_sync:INFO: repo_sync:254 Sync invoked for repo <rhel-server-6-srpms-6Server-i386>
2012-03-19 23:18:09,566 8741:139971306452736: pulp.server.api.repo_sync:INFO: repo_sync:268 Sync of rhel-server-6-srpms-6Server-i386 starting, skip_dict = {}
2012-03-19 23:18:09,591 8741:139971306452736: pulp.server.api.synchronizers:INFO: synchronizers:726 cacert = </etc/pki/pulp/content/rhel-server-6-srpms-6Server-i386/feed-rhel-server-6-srpms-6Server-i386.ca>, cert = </etc/pki/pulp/content/rhel-server-6-srpms-6Server-i386/feed-rhel-server-6-srpms-6Server-i386.cert>
2012-03-19 23:18:09,592 8741:139971306452736: pulp.server.api.synchronizers:INFO: synchronizers:775 Fetching repo to </var/lib/pulp//repos/content/dist/rhel/rhui/server/6/6Server/i386/source/SRPMS> with verify_options <{'checksum': False, 'size': False}>
2012-03-19 23:18:37,959 8741:139971306452736: pulp.server.api.synchronizers:INFO: synchronizers:783 YumSynchronizer reported 2725 successes, 0 downloads, 0 errors
2012-03-19 23:22:36,831 8741:139971295962880: pulp.server.api.synchronizers:INFO: synchronizers:810 preserve metadata flag is set; skipping metadata update
2012-03-19 23:22:37,311 8741:139971295962880: pulp.server.tasking.task:ERROR: task:474 Task failed: Task 58e08030-720c-11e1-8a52-1231391025b1: _sync(rhel-server-6-6Server-x86_64, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7f4de133c190>, progress_callback=<bound method RepoSyncTask.progress_callback of <pulp.server.api.repo_sync_task.RepoSyncTask object at 0x7f4de133c050>>)
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/tasking/task.py", line 420, in run
    result = self.callable(*self.args, **self.kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 272, in _sync
    progress_callback, synchronizer, max_speed, threads)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 362, in fetch_content
    synchronizer.add_distribution_from_dir(repo_dir, repo_id, skip_dict)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 344, in add_distribution_from_dir
    self._process_repo_images(dir, repo)
  File "/usr/lib/python2.6/site-packages/pulp/server/api/synchronizers.py", line 459, in _process_repo_images
    files = pulp.server.util.listdir(distro_path) or []
  File "/usr/lib/python2.6/site-packages/pulp/server/util.py", line 373, in listdir
    raise Exception("Cannot read from directory %s" % directory)
Exception: Cannot read from directory /var/lib/pulp/distributions/ks-Red Hat Enterprise Linux-Server-6.1-x86_64
Comment 1 James Slagle 2012-03-20 10:20:17 EDT
Created attachment 571425 [details]
pulp.log

attaching full pulp log
Comment 2 James Slagle 2012-03-20 10:26:01 EDT
From what I can tell in the attached pulp log the sync of rhel-server-6-6Server-x86_64 failed at 3-19 11:22 pm.  The distribution was then created successfully by the sync of rhel-server-6-beta-6Server-x86_64 at 3-19 12:09 pm:

2012-03-20 00:09:10,983 8741:139971295962880: pulp.server.api.synchronizers:INFO: synchronizers:469 Created a distributionID ks-Red Hat Enterprise Linux-Server-6.1-x86_64

The next time the sync of rhel-server-6-6Server-x86_64 ran, it then was able to finish successfully.

We need to find out why the sync of the beta repo passed and saw the need to create the directory, but the initial sync of the non-beta repo assumed the directory already existed.

This is the same thing that happened for i386 too btw.
Comment 3 James Slagle 2012-03-20 10:28:54 EDT
Created attachment 571430 [details]
grinder.log.2

attaching relevent grinder.log portion as well
Comment 4 James Slagle 2012-03-20 16:15:44 EDT
although there definitely seems to be some sort of bug here, no one knows what it is, so we're going to add code to skip distribution sync in a rhui environment.

need to add a db migration to set this option on existing repos and update rhui tools to create new repos with this option set.

committed to pulp rhui: fcea61d6eb46b1e916f88714f0c366a7e1d9911d
Comment 5 James Slagle 2012-03-20 16:23:01 EDT
committed to cloude 1262374ec51f43110cfd0eb19bd92db2e7e8350f
Comment 6 James Slagle 2012-03-23 09:23:18 EDT
QA:
after updating to 2.0.3, or a fresh install of 2.0.3, you should never see this traceback.

also, no distributions should be sync'd to the RHUI environment.  You should no distribution files or images under /var/lib/pulp/distributions and should see log messages like the following in /var/log/pulp/pulp.log:
2012-03-23 06:27:09,083 26650:139688998483712: pulp.server.api.synchronizers:INFO: synchronizers:346 skipping distribution imports from sync process
Comment 7 mkovacik 2012-03-26 06:30:31 EDT
Verified:
 - no 'cannot read' exceptions in pulp logs
 - 'skipping distribution' present in pulp logs
 - no distributions directory in /var/lib/pulp

Version: RHEL-6.2-RHUI-2.0.3-20120322.0-Server-x86_64-DVD1.iso


### Screen log
[root@dhcp-31-120 ~]# grep 'synchronizers:346\ skipping\ distribution' /var/log/pulp/pulp.log                                                                                                                                                                                                                           
2012-03-23 19:57:48,338 14236:140234557716224: pulp.server.api.synchronizers:INFO: synchronizers:346 skipping distribution imports from sync process
2012-03-23 19:58:56,691 14236:140234440308480: pulp.server.api.synchronizers:INFO: synchronizers:346 skipping distribution imports from sync process
2012-03-23 23:44:32,340 14561:139847872272128: pulp.server.api.synchronizers:INFO: synchronizers:346 skipping distribution imports from sync process
2012-03-24 01:58:02,526 14561:139847851292416: pulp.server.api.synchronizers:INFO: synchronizers:346 skipping distribution imports from sync process

[root@dhcp-31-120 ~]# grep 'cannot\ read' /var/log/pulp/*
[root@dhcp-31-120 ~]# echo $?
1

[root@dhcp-31-120 ~]# ls /var/lib/pulp/distributions
ls: cannot access /var/lib/pulp/distributions: No such file or directory
Comment 8 Lana Brindley 2012-04-19 20:58:04 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
C: ??
C: After upgrading to RHUI 2.0.2, the next round of scheduled repository synchronisations failed, and tracebacks were recorded in the pulp.log logfile. The following synchronisations occurred successfully without changes. 
F: a fix was created to skip distribution synchronisation in a RHUI environment. An option was added to allow users to toggle this behaviour.
R: The synchronisation completed successfully without displaying a traceback.
Comment 9 Lana Brindley 2012-04-20 00:22:33 EDT
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,4 +1,7 @@
 C: ??
 C: After upgrading to RHUI 2.0.2, the next round of scheduled repository synchronisations failed, and tracebacks were recorded in the pulp.log logfile. The following synchronisations occurred successfully without changes. 
 F: a fix was created to skip distribution synchronisation in a RHUI environment. An option was added to allow users to toggle this behaviour.
-R: The synchronisation completed successfully without displaying a traceback.+R: The synchronisation completed successfully without displaying a traceback.
+
+Final:
+* After upgrading to RHUI 2.0.2, the next round of scheduled repository synchronisations would fail, and tracebacks were recorded in the pulp.log logfile. The following synchronisations would be successful without changes. A fix was created to skip distribution synchronisation in a RHUI environment. An option was added to allow users to toggle this behaviour. The synchronisation now completes successfully without displaying a traceback. (BZ#805046)
Comment 10 errata-xmlrpc 2012-05-01 14:37:15 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2012-0539.html

Note You need to log in before you can comment on or make changes to this bug.