Bug 1303278 - EPEL5 import fails due to missing 'description' elements in updateinfo
EPEL5 import fails due to missing 'description' elements in updateinfo
Status: CLOSED ERRATA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Pulp (Show other bugs)
6.1.6
All Linux
medium Severity medium (vote)
: Beta
: --
Assigned To: satellite6-bugs
Lukas Pramuk
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-29 23:04 EST by Frank Hirtz
Modified: 2017-07-26 15:37 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-07-27 05:22:43 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Pulp Redmine 635 Normal CLOSED - CURRENTRELEASE Scientific Linux Errata Data does not get imported 2016-02-01 11:45 EST

  None (edit)
Description Frank Hirtz 2016-01-29 23:04:56 EST
Description of problem:

A client reported failures while attempting to import the EPEL5 repo into their satellite. I had a look and replicated the issue using pulp directly to simplify testing. The issue appears to be that the updateinfo errata metadata in the repo doesn't have "description" elements defined for all of the erratum. The pulp-rpm importer doesn't much like that and fails.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. pulp-admin rpm repo create --repo-id=test_epel5_pulp --display-name="RHEL5 EPEL" --feed=https://dl.fedoraproject.org/pub/epel/5/x86_64 --relative-url=epel5 --serve-http=true --serve-https=true
2. pulp-admin rpm repo sync run --repo-id=test_epel5_pulp
3. Fail.

Actual results:
[root@satellite updateinfotest]# pulp-admin rpm repo create --repo-id=test_epel5_pulp --display-name="RHEL5 EPEL" --feed=https://dl.fedoraproject.org/pub/epel/5/x86_64 --relative-url=epel5 --serve-http=true --serve-https=true
Successfully created repository [test_epel5_pulp]

[root@satellite updateinfotest]# pulp-admin rpm repo sync run --repo-id=test_epel5_pulp +----------------------------------------------------------------------+
               Synchronizing Repository [test_epel5_pulp]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[-]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... failed


Task Failed

Importer indicated a failed response

<log>
Jan 29 22:51:15 satellite pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[e5bdb13c-4135-4eee-9583-87a783940328]
Jan 29 22:51:15 satellite pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[0aa038fc-4885-49e2-965a-894a1a067f8b]
Jan 29 22:51:15 satellite pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from https://dl.fedoraproject.org/pub/epel/5/x86_64/.
Jan 29 22:51:15 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Jan 29 22:51:15 satellite pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[e5bdb13c-4135-4eee-9583-87a783940328] succeeded in 0.0381081299856s: None
Jan 29 22:51:16 satellite pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[af88ae8a-7aeb-47af-8d60-c50c22246ac6]
Jan 29 22:51:16 satellite pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Parsing metadata.
Jan 29 22:51:16 satellite pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata files.
Jan 29 22:51:17 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Jan 29 22:51:17 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Jan 29 22:51:17 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Jan 29 22:51:17 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Jan 29 22:51:17 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Jan 29 22:51:20 satellite pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Generating metadata databases.
Jan 29 22:52:00 satellite pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Determining which units need to be downloaded.
Jan 29 22:54:52 satellite pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading 0 RPMs.
Jan 29 22:55:11 satellite pulp: pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: The content container reported: {'downloads': {}, 'total_sources': 0} for base URL: https://dl.fedoraproject.org/pub/epel/5/x86_64/
Jan 29 22:55:11 satellite pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading additional units.
Jan 29 22:55:11 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Jan 29 22:55:12 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136) sync failed
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136) Traceback (most recent call last):
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 159, in run
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)     self.get_errata(metadata_files)
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)   File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)     self.gen.throw(type, value, traceback)
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 118, in update_state
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)     yield skip
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 159, in run
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)     self.get_errata(metadata_files)
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 522, in get_errata
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)     updateinfo.process_package_element, additive_type=True)
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 602, in save_fileless_units
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)     for model in package_info_generator:
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/packages.py", line 63, in package_list_generator
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/updateinfo.py", line 27, in process_package_element
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136) AttributeError: 'NoneType' object has no attribute 'text'
Jan 29 22:55:14 satellite pulp: pulp.server.event.http:INFO: (2977-19136) {'call_report': {u'exception': None, u'task_type': u'pulp.server.managers.repo.sync.sync', u'task_id': u'0aa038fc-4885-49e2-965a-894a1a067f8b', u'tags': [u'pulp:repository:test_epel5_pulp', u'pulp:action:sync'], u'finish_time': None, u'_ns': u'task_status', u'start_time': u'2016-01-30T03:51:15Z', u'traceback': None, u'spawned_tasks': [], u'progress_report': {u'yum_importer': {u'content': {u'size_total': 0, u'items_left': 0, u'items_total': 0, u'state': u'FINISHED', u'size_left': 0, u'details': {u'rpm_total': 0, u'rpm_done': 0, u'drpm_total': 0, u'drpm_done': 0}, u'error_details': []}, u'comps': {u'state': u'NOT_STARTED'}, u'distribution': {u'items_total': 0, u'state': u'FINISHED', u'error_details': [], u'items_left': 0}, u'errata': {u'state': u'FAILED', u'error': u"'NoneType' object has no attribute 'text'"}, u'metadata': {u'state': u'FINISHED'}}}, u'state': u'running', u'worker_name': u'reserved_resource_worker-0@satellite.maunet.us', u'result': None, u'error': None, u'_id': ObjectId('56ac3333295511b21ab954d7'), u'id': u'56ac3333c024450b6e4db11b'}, 'event_type': 'repo.sync.finish', 'payload': {'importer_id': u'yum_importer', 'exception': None, 'repo_id': u'test_epel5_pulp', 'traceback': None, 'started': '2016-01-30T03:51:15Z', '_ns': u'repo_sync_results', 'completed': '2016-01-30T03:55:14Z', 'importer_type_id': u'yum_importer', 'error_message': None, 'summary': {'content': {'state': 'FINISHED'}, 'comps': {'state': 'NOT_STARTED'}, 'distribution': {'state': 'FINISHED'}, 'errata': {'state': 'FAILED'}, 'metadata': {'state': 'FINISHED'}}, 'added_count': 0, 'result': 'failed', 'updated_count': 6781, 'details': {'content': {'size_total': 0, 'items_left': 0, 'items_total': 0, 'state': 'FINISHED', 'size_left': 0, 'details': {'rpm_total': 0, 'rpm_done': 0, 'drpm_total': 0, 'drpm_done': 0}, 'error_details': []}, 'comps': {'state': 'NOT_STARTED'}, 'distribution': {'items_total': 0, 'state': 'FINISHED', 'error_details': [], 'items_left': 0}, 'errata': {'state': 'FAILED',
Jan 29 22:55:14 satellite pulp: pulp.server.event.http:INFO: (2977-19136)  'error': "'NoneType' object has no attribute 'text'"}, 'metadata': {'state': 'FINISHED'}}, 'id': '56ac3422c024450ba1aa5bc6', 'removed_count': 0}}
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136) Task pulp.server.managers.repo.sync.sync[0aa038fc-4885-49e2-965a-894a1a067f8b] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136) Traceback (most recent call last):
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136)   File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136)     R = retval = fun(*args, **kwargs)
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136)   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 328, in __call__
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136)     return super(Task, self).__call__(*args, **kwargs)
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136)   File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136)     return self.run(*args, **kwargs)
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136)   File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/sync.py", line 114, in sync
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136)     raise PulpExecutionException(_('Importer indicated a failed response'))
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136) PulpExecutionException: Importer indicated a failed response
Jan 29 22:55:14 satellite pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[af88ae8a-7aeb-47af-8d60-c50c22246ac6] succeeded in 0.011311418988s: None
Jan 29 22:55:14 satellite pulp: pulp.server.event.http:WARNING: Error response from HTTP notifier: {"displayMessage":"Couldn't find repository 'test_epel5_pulp'","errors":["Couldn't find repository 'test_epel5_pulp'"]}
</log>

Expected results:
Works.

Additional info:
Having a look at the updateinfo for EPEL5:

<epel5>
[fmhirtz@x1 epel5]$ wget -q https://dl.fedoraproject.org/pub/epel/5/x86_64/repodata/c0f4c9f3e20d7aa48e98b37e9cdf713ed9060280-updateinfo.xml.gz[fmhirtz@x1 epel5]$ ls
c0f4c9f3e20d7aa48e98b37e9cdf713ed9060280-updateinfo.xml.gz
[fmhirtz@x1 epel5]$ zgrep '<id>' c0f4c9f3e20d7aa48e98b37e9cdf713ed9060280-updateinfo.xml.gz | wc -l
1877
[fmhirtz@x1 epel5]$ zgrep '<name>' c0f4c9f3e20d7aa48e98b37e9cdf713ed9060280-updateinfo.xml.gz | wc -l
1877
[fmhirtz@x1 epel5]$ zgrep '<description>' c0f4c9f3e20d7aa48e98b37e9cdf713ed9060280-updateinfo.xml.gz | wc -l
1652
[fmhirtz@x1 epel5]$ 
</epel5>

As a sanity check, EPEL6 doesn't have any such discrepancy:

<epel6 (works)>
[fmhirtz@x1 epel6]$ wget -q https://dl.fedoraproject.org/pub/epel/6/x86_64/repodata/7b7192d8e50b4613e2cca15a0a4875d6e985a8a96077151f7eebea7681831596-updateinfo.xml.bz2
[fmhirtz@x1 epel6]$ ls
7b7192d8e50b4613e2cca15a0a4875d6e985a8a96077151f7eebea7681831596-updateinfo.xml.bz2
[fmhirtz@x1 epel6]$ bzcat 7b7192d8e50b4613e2cca15a0a4875d6e985a8a96077151f7eebea7681831596-updateinfo.xml.bz2 | grep '<id>' | wc -l
3826
[fmhirtz@x1 epel6]$ bzcat 7b7192d8e50b4613e2cca15a0a4875d6e985a8a96077151f7eebea7681831596-updateinfo.xml.bz2 | grep '<name>' | wc -l
3826
[fmhirtz@x1 epel6]$ bzcat 7b7192d8e50b4613e2cca15a0a4875d6e985a8a96077151f7eebea7681831596-updateinfo.xml.bz2 | grep '<description>' | wc -l
3826
[fmhirtz@x1 epel6]$ 
</epel6>
Comment 2 Michael Hrivnak 2016-01-30 09:41:00 EST
This was fixed upstream in pulp 2.6.1 and can be cherry-picked on a sat 6.1 Z-stream if desired. I added the link to the upstream issue.
Comment 3 Bryan Kearney 2016-01-31 08:01:08 EST
based on comment 2, I am going to move this to ON_QA as part of 6.2
Comment 5 pulp-infra@redhat.com 2016-02-01 11:45:17 EST
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Comment 6 pulp-infra@redhat.com 2016-02-01 11:45:19 EST
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
Comment 10 errata-xmlrpc 2016-07-27 05:22:43 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501

Note You need to log in before you can comment on or make changes to this bug.