Bug 608145

Summary: Error at the end of RHN sync
Product: [Retired] Pulp Reporter: wes hayutin <whayutin>
Component: z_otherAssignee: Pradeep Kilambi <pkilambi>
Status: CLOSED CURRENTRELEASE QA Contact: wes hayutin <whayutin>
Severity: medium Docs Contact:
Priority: low    
Version: unspecifiedCC: jmatthew, pkilambi, pthomas
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-08-16 14:03:08 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 647488    

Description wes hayutin 2010-06-25 19:37:34 UTC 
[root@localhost pulp]# pulp repo sync --label=rhel-i386-server-5
Sync Status:: running

Sync Status:: error
Error : AttributeError: 'module' object has no attribute 'comps_util'

----------------------
log

2010-06-25 15:29:37,030  rpm_path: /var/lib/pulp/rhel-i386-server-5/libc-client-devel-2004g-2.2.1.i386.rpm
2010-06-25 15:29:37,058  rpm_path: /var/lib/pulp/rhel-i386-server-5/libIDL-0.8.7-1.fc6.i386.rpm
2010-06-25 15:29:37,073  Repo: rhel-i386-server-5 read [2439] packages took 69.1819229126 seconds
2010-06-25 15:29:37,073  compsfile = <open file u'/var/lib/pulp/rhel-i386-server-5/repodata/comps.xml', mode 'r' at 0xb22a7b10>, repo = rhel-i386-server-5

----------------------

Label              	rhel-i386-server-5       
Name               	rhel-i386-server-5       
Feed               	{'url': 'satellite.rhn.redhat.com/rhel-i386-server-5', 'type': 'rhn', 'supported_types': ['yum', 'local', 'rhn']}
Arch               	i386                     
Packages           	0                        
Package Categories 	[]                       
Package Groups     	0
Comment 1 wes hayutin 2010-07-22 17:05:04 UTC 
Got an error from the pulp cli at the end of a pulp repo sync. From the log it looks like the repo completed successfully though.

[root@localhost conf.d]# pulp repo sync --label=rhel-i386-server-5

Sync Status:: running
Sync Status:: running
Sync Status:: error
Error : KeyError: 'repo_defined'




--- Grinder Log -----

2010-07-22 12:53:01,152  Wait 1.  check again
2010-07-22 12:53:01,173  Package [kernel-2.6.18-194.8.1.el5.i686.rpm] is valid with checksum [5b0c26e99d011bce44ba2796432dc74b] and size [17717951]
2010-07-22 12:53:01,174  Successfully Fetched Package - [/var/lib/pulp/rhel-i386-server-5/kernel-2.6.18-194.8.1.el5.i686.rpm]
2010-07-22 12:53:01,174  Thread ending
2010-07-22 12:53:01,654  All threads have finished.
2010-07-22 12:53:01,688  ParallelFetch: 2396 items successfully processed, 88 downloaded, 0 items had errors
2010-07-22 12:53:01,688  Attempting to fetch comps.xml info from RHN
2010-07-22 12:53:11,412  Processed <rhel-i386-server-5> 2396 packages, 88 downloaded, 0 errors, completed in 218.091864109 seconds
2010-07-22 12:53:32,337  createrepo on /var/lib/pulp/rhel-i386-server-5/ finished in 20.9246790409 seconds
2010-07-22 12:53:33,038  updaterepo on /var/lib/pulp/rhel-i386-server-5/repodata/ finished in 0.700316905975 seconds
2010-07-22 12:53:42,284  createrepo on /var/lib/pulp/rhel-i386-server-5/ finished in 9.23037314415 seconds
2010-07-22 12:53:42,285  setParallel(10)
Comment 2 wes hayutin 2010-07-27 15:08:18 UTC 
Sync Status:: running
Sync Status:: running
Sync Status:: running
Sync Status:: running
Sync Status:: running
Sync Status:: error
Error : InvalidDocument: document too large - BSON documents are limited to 4 MB
Comment 3 wes hayutin 2010-07-27 15:09:44 UTC 
2010-07-27 11:08:36,057  Queue is empty, thread will end
2010-07-27 11:08:36,059  /var/lib/pulp/rhel-i386-server-5/libdbi-0.8.1-2.1.i386.rpm exists with correct size and md5sum, no need to fetch.
2010-07-27 11:08:36,060  /var/lib/pulp/rhel-i386-server-5/man-pages-ja-20060815-13.el5.noarch.rpm exists with correct size and md5sum, no need to fetch.
2010-07-27 11:08:36,060  Queue is empty, thread will end
2010-07-27 11:08:36,060  Queue is empty, thread will end
2010-07-27 11:08:36,060  Thread ending
2010-07-27 11:08:36,061  Thread ending
2010-07-27 11:08:36,061  Thread ending
2010-07-27 11:08:36,061  Thread ending
2010-07-27 11:08:36,061  Thread ending
2010-07-27 11:08:36,245  All threads have finished.
2010-07-27 11:08:36,264  ParallelFetch: 2396 items successfully processed, 0 downloaded, 0 items had errors
2010-07-27 11:08:36,264  Attempting to fetch comps.xml info from RHN
2010-07-27 11:08:40,610  Processed <rhel-i386-server-5> 2396 packages, 0 downloaded, 0 errors, completed in 65.9532880783 seconds
2010-07-27 11:08:48,374  createrepo on /var/lib/pulp/rhel-i386-server-5/ finished in 7.76347994804 seconds
2010-07-27 11:08:48,956  updaterepo on /var/lib/pulp/rhel-i386-server-5/repodata/ finished in 0.581706047058 seconds
2010-07-27 11:08:56,516  createrepo on /var/lib/pulp/rhel-i386-server-5/ finished in 7.55202913284 seconds
2010-07-27 11:08:56,516  setParallel(10)
Comment 4 Preethi Thomas 2010-09-10 14:55:01 UTC 
[root@preethi ~]# rpm -qa |grep grinder
grinder-0.0.60-1.fc13.noarch
[root@preethi ~]# rpm -qa |grep pulp
pulp-client-0.0.60-1.fc13.noarch
pulp-0.0.60-1.fc13.noarch
pulp-common-0.0.60-1.fc13.noarch


[root@preethi ~]#  pulp-admin -uadmin -padmin repo sync --id=rhel-i386-server-5
Task created with ID:: 5e50f15c-bcec-11df-a00b-002564a85a58
Sync Status:: running
Sync Status:: running
Sync Status:: running
Sync Status:: running
Sync Status:: running
Sync Status:: running
Sync Status:: running
Sync Status:: running
Sync Status:: running
Sync Status:: running
Sync Status:: running
Sync Status:: running
Sync Status:: running
Sync Status:: running
Sync Status:: running
Sync Status:: running
Sync Status:: running
Sync Status:: running
Sync Status:: error
Error : IOError: [Errno 2] No such file or directory: u'/var/lib/pulp/rhel-i386-server-5/comps.xml'

You have new mail in /var/spool/mail/root


<prad> yea looks like a valid bug to me
<prad> issue seems to be that its not fetching the initial data
<prad> so I dont see any repo info in /var/lib/pulp
<prad> so could be the account you're using is not able to fetch the data
<prad> either ways we should not look for comps.xml if its not there
<prad> and there are cases where comps wouldnt exist
Comment 5 John Matthews 2010-09-23 13:01:51 UTC 
Please re-test, I tried this with latest git code of grinder/pulp from 9/22/2010.  RHN Sync was successful.

There is a separate issue of some packages report an error during package import.  That is an issue with auditing not handling non-ascii characters in rpm description fields.
Comment 6 Preethi Thomas 2010-09-28 15:03:34 UTC 
Moving it to assigned

I don't see the above error anymore, but I see the other error with the auditing in the log.

2010-09-28 10:58:30,540 [ERROR][Thread-9] import_package() @ repo_sync.py:230 - error reading package checkpolicy-1.33.1-6.el5.i386.rpm
2010-09-28 10:58:30,540 [ERROR][Thread-9] import_package() @ repo_sync.py:231 - Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 219, in import_package
    hashtype, checksum, file_name)
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 202, in _audit
    _record_event()
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 175, in _record_event
    param_values_str))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 136: ordinal not in range(128)

2010-09-28 10:58:30,676 [ERROR][Thread-9] import_package() @ repo_sync.py:230 - error reading package cups-lpd-1.3.7-18.el5_5.7.i386.rpm
2010-09-28 10:58:30,677 [ERROR][Thread-9] import_package() @ repo_sync.py:231 - Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 219, in import_package
    hashtype, checksum, file_name)
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 202, in _audit
    _record_event()
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 175, in _record_event
    param_values_str))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 163: ordinal not in range(128)

2010-09-28 10:58:30,730 [ERROR][Thread-9] import_package() @ repo_sync.py:230 - error reading package libsemanage-1.9.1-4.4.el5.i386.rpm
2010-09-28 10:58:30,730 [ERROR][Thread-9] import_package() @ repo_sync.py:231 - Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 219, in import_package
    hashtype, checksum, file_name)
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 202, in _audit
    _record_event()
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 175, in _record_event
    param_values_str))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 137: ordinal not in range(128)

2010-09-28 10:58:31,183 [ERROR][Thread-9] import_package() @ repo_sync.py:230 - error reading package cups-libs-1.3.7-18.el5_5.7.i386.rpm
2010-09-28 10:58:31,183 [ERROR][Thread-9] import_package() @ repo_sync.py:231 - Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 219, in import_package
    hashtype, checksum, file_name)
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 202, in _audit
    _record_event()
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 175, in _record_event
    param_values_str))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 164: ordinal not in range(128)

2010-09-28 10:58:31,776 [ERROR][Thread-9] import_package() @ repo_sync.py:230 - error reading package policycoreutils-1.33.12-14.8.el5.i386.rpm
2010-09-28 10:58:31,776 [ERROR][Thread-9] import_package() @ repo_sync.py:231 - Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 219, in import_package
    hashtype, checksum, file_name)
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 202, in _audit
    _record_event()
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 175, in _record_event
    param_values_str))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 144: ordinal not in range(128)

2010-09-28 10:58:32,089 [ERROR][Thread-9] import_package() @ repo_sync.py:230 - error reading package libselinux-1.33.4-5.5.el5.i386.rpm
2010-09-28 10:58:32,090 [ERROR][Thread-9] import_package() @ repo_sync.py:231 - Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 219, in import_package
    hashtype, checksum, file_name)
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 202, in _audit
    _record_event()
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 175, in _record_event
    param_values_str))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 137: ordinal not in range(128)

2010-09-28 10:58:33,184 [ERROR][Thread-9] import_package() @ repo_sync.py:230 - error reading package libsepol-1.15.2-3.el5.i386.rpm
2010-09-28 10:58:33,184 [ERROR][Thread-9] import_package() @ repo_sync.py:231 - Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 219, in import_package
    hashtype, checksum, file_name)
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 202, in _audit
    _record_event()
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 175, in _record_event
    param_values_str))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 133: ordinal not in range(128)

2010-09-28 10:58:35,701 [ERROR][Thread-9] import_package() @ repo_sync.py:230 - error reading package mcstrans-0.2.11-3.el5.i386.rpm
2010-09-28 10:58:35,701 [ERROR][Thread-9] import_package() @ repo_sync.py:231 - Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 219, in import_package
    hashtype, checksum, file_name)
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 202, in _audit
    _record_event()
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 175, in _record_event
    param_values_str))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 133: ordinal not in range(128)

2010-09-28 10:58:36,720 [ERROR][Thread-9] import_package() @ repo_sync.py:230 - error reading package Deployment_Guide-zh-CN-5.2-11.noarch.rpm
2010-09-28 10:58:36,720 [ERROR][Thread-9] import_package() @ repo_sync.py:231 - Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 219, in import_package
    hashtype, checksum, file_name)
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 202, in _audit
    _record_event()
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 175, in _record_event
    param_values_str))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 94: ordinal not in range(128)

2010-09-28 10:58:37,454 [ERROR][Thread-9] import_package() @ repo_sync.py:230 - error reading package cups-devel-1.3.7-18.el5_5.7.i386.rpm
2010-09-28 10:58:37,454 [ERROR][Thread-9] import_package() @ repo_sync.py:231 - Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 219, in import_package
    hashtype, checksum, file_name)
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 202, in _audit
    _record_event()
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 175, in _record_event
    param_values_str))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 165: ordinal not in range(128)

2010-09-28 10:58:37,740 [ERROR][Thread-9] import_package() @ repo_sync.py:230 - error reading package cups-1.3.7-18.el5_5.7.i386.rpm
2010-09-28 10:58:37,741 [ERROR][Thread-9] import_package() @ repo_sync.py:231 - Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/api/repo_sync.py", line 219, in import_package
    hashtype, checksum, file_name)
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 202, in _audit
    _record_event()
  File "/usr/lib/python2.6/site-packages/pulp/server/auditing.py", line 175, in _record_event
    param_values_str))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 159: ordinal not in range(128)
Comment 7 Pradeep Kilambi 2011-01-05 17:40:17 UTC 
I just did a sync of vt and base rhel5 and see the properly encoded info in the event.log

files.  Rdjpgcom displays any text comments included in a JPEG file.
Wrjpgcom inserts text comments into a JPEG file., checksum_type: sha256, checksum: 82813f0b2e186515008fce4efe8c63adb56b1c0662a3771df8fac1ab052c1068, filename: libjpeg-6b-37.i386.rpm, repo_defined: True
[2011-01-04 15:34:39.137092] SYSTEM called PackageApi.create on name: classpathx-jaf-javadoc, epoch: 0, version: 1.0, release: 9jpp.1, arch: i386, description: Javadoc for classpathx-jaf., checksum_type: sha256, checksum: 63aeedb70120b15bdc44e349337e87fd3bb497e6415576b09e24f126fa048ce4, filename: classpathx-jaf-javadoc-1.0-9jpp.1.i386.rpm, repo_defined: True
[2011-01-04 15:34:39.140052] SYSTEM called PackageApi.create on name: perftest, epoch: 0, version: 1.2.3, release: 1.el5, arch: i386, description: gen2 uverbs microbenchmarks, checksum_type: sha256, checksum: ed26f61f772dabbee0cd3530a903d192bdaedbf82cdc52ea27e6f6c18c8378cd, filename: perftest-1.2.3-1.el5.i386.rpm, repo_defined: True
[2011-01-04 15:34:39.143340] SYSTEM called PackageApi.create on name: Deployment_Guide-te-IN, epoch: 0, version: 5.2, release: 11, arch: noarch, description: This Deployment Guide documents relevant information regarding the
deployment, configuration and administration of Red Hat Enterprise Linux
5.2., checksum_type: sha256, checksum: f793245fe452c2a70fca22ba84da2f77fa4673aede6b99c885d397e0ebcf9167, filename: Deployment_Guide-te-IN-5.2-11.noarch.rpm, repo_defined: True
[2011-01-04 15:34:39.146243] SYSTEM called PackageApi.create on name: xorg-x11-fonts-ISO8859-15-100dpi, epoch: 0, version: 7.1, release: 2.1.el5, arch: noarch, description: Contains a set of 100dpi fonts in the ISO8859-15 encoding which
provide Euro support., checksum_type: sha256, checksum: 16aaff2f0e833e6284864fb8e1aaa979a81ef5cc3a8c2eb4becd9fadf7d49b75, filename: xorg-x11-fonts-ISO8859-15-100dpi-7.1-2.1.el5.noarch.rpm, repo_defined: True
[2011-01-04 15:34:39.149175] SYSTEM called PackageApi.create on name: xorg-x11-drv-hyperpen, epoch: 0, version: 1.1.0, release: 2, arch: i386, description: X.Org X11 hyperpen input driver., checksum_type: sha256, checksum: 3692b92b2df6ffed6d4feca14d759f033fc2a8649856c8b6c30a3dacabbfdf1c, filename: xorg-x11-drv-hyperpen-1.1.0-2.i386.rpm, repo_defined: True
[2011-01-04 15:34:39.152485] SYSTEM called PackageApi.create on name: sblim-cmpi-dns-test, epoch: 1, version: 1.0, release: 40.el5, arch: i386, description: SBLIM WBEM-SMT DNS Provider Testcase Files for the SBLIM Testsuite, checksum_type: sha256, checksum: 51993a32db330e4067b28ae6fead891fa5a01ca1cea8fb1a6375c1536905aa6e, filename: sblim-cmpi-dns-test-1.0-40.el5.i386.rpm, repo_defined: True
[2011-01-04 15:34:39.155584] SYSTEM called PackageApi.create on name: gnome-python2-gnomevfs, epoch: 0, version: 2.16.0, release: 1.fc6, arch: i386, description: This module contains a wrapper that allows the use of gnome-vfs via python., checksum_type: sha256, checksum: 5fba24a00e12dd43f274362311bdab0091d55a6b1a87dcfa56bed207b96cba46, filename: gnome-python2-gnomevfs-2.16.0-1.fc6.i386.rpm, repo_defined: True
[2011-01-04 15:34:39.159778] SYSTEM called PackageApi.create on name: anthy-devel, epoch: 0, version: 7900, release: 4.el5, arch: i386, description: The anthy-devel package contains the development files which is needed to build
the programs which uses Anthy., checksum_type: sha256, checksum: 709e1e8849a4d07e7377548da308dd9ce2986596a1f2375c6ec1d8670bc4b36a, filename: anthy-devel-7900-4.el5.i386.rpm, repo_defined: True
[2011-01-04 15:34:39.162807] SYSTEM called PackageApi.create on name: iptables-ipv6, epoch: 0, version: 1.3.5, release: 5.3.el5_4.1, arch: i386, description: The iptables package contains IPv6 (the next version of the IP
protocol) support for iptables. Iptables controls the Linux kernel
network packet filtering code, allowing you to set up firewalls and IP
masquerading.

moving to on_qa for revalidation ..
Comment 8 Preethi Thomas 2011-01-07 16:13:03 UTC 
verified

[root@10 ~]# rpm -q pulp
pulp-0.0.121-1.fc13.noarch

I synced 
rhel-x86_64-client-vt-5
rhel-i386-client-vt-5 

And did not see any errors grinder or pulp logs.
Comment 9 Preethi Thomas 2011-08-16 14:03:08 UTC 
Closing with Community Release 15

pulp-0.0.223-4.