Bug 1303278

Description of problem:

A client reported failures while attempting to import the EPEL5 repo into their satellite. I had a look and replicated the issue using pulp directly to simplify testing. The issue appears to be that the updateinfo errata metadata in the repo doesn't have "description" elements defined for all of the erratum. The pulp-rpm importer doesn't much like that and fails.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. pulp-admin rpm repo create --repo-id=test_epel5_pulp --display-name="RHEL5 EPEL" --feed=https://dl.fedoraproject.org/pub/epel/5/x86_64 --relative-url=epel5 --serve-http=true --serve-https=true
2. pulp-admin rpm repo sync run --repo-id=test_epel5_pulp
3. Fail.

Actual results:
[root@satellite updateinfotest]# pulp-admin rpm repo create --repo-id=test_epel5_pulp --display-name="RHEL5 EPEL" --feed=https://dl.fedoraproject.org/pub/epel/5/x86_64 --relative-url=epel5 --serve-http=true --serve-https=true
Successfully created repository [test_epel5_pulp]

[root@satellite updateinfotest]# pulp-admin rpm repo sync run --repo-id=test_epel5_pulp +----------------------------------------------------------------------+
               Synchronizing Repository [test_epel5_pulp]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[-]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       0/0 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 0/0 items
... completed

Importing errata...
[-]
... failed


Task Failed

Importer indicated a failed response

<log>
Jan 29 22:51:15 satellite pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[e5bdb13c-4135-4eee-9583-87a783940328]
Jan 29 22:51:15 satellite pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[0aa038fc-4885-49e2-965a-894a1a067f8b]
Jan 29 22:51:15 satellite pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from https://dl.fedoraproject.org/pub/epel/5/x86_64/.
Jan 29 22:51:15 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Jan 29 22:51:15 satellite pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[e5bdb13c-4135-4eee-9583-87a783940328] succeeded in 0.0381081299856s: None
Jan 29 22:51:16 satellite pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[af88ae8a-7aeb-47af-8d60-c50c22246ac6]
Jan 29 22:51:16 satellite pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Parsing metadata.
Jan 29 22:51:16 satellite pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata files.
Jan 29 22:51:17 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Jan 29 22:51:17 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Jan 29 22:51:17 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Jan 29 22:51:17 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Jan 29 22:51:17 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Jan 29 22:51:20 satellite pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Generating metadata databases.
Jan 29 22:52:00 satellite pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Determining which units need to be downloaded.
Jan 29 22:54:52 satellite pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading 0 RPMs.
Jan 29 22:55:11 satellite pulp: pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: The content container reported: {'downloads': {}, 'total_sources': 0} for base URL: https://dl.fedoraproject.org/pub/epel/5/x86_64/
Jan 29 22:55:11 satellite pulp: pulp_rpm.plugins.importers.yum.sync:INFO: Downloading additional units.
Jan 29 22:55:11 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Jan 29 22:55:12 satellite pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): dl.fedoraproject.org
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136) sync failed
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136) Traceback (most recent call last):
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 159, in run
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)     self.get_errata(metadata_files)
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)   File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)     self.gen.throw(type, value, traceback)
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 118, in update_state
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)     yield skip
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 159, in run
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)     self.get_errata(metadata_files)
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 522, in get_errata
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)     updateinfo.process_package_element, additive_type=True)
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 602, in save_fileless_units
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)     for model in package_info_generator:
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/packages.py", line 63, in package_list_generator
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/repomd/updateinfo.py", line 27, in process_package_element
Jan 29 22:55:14 satellite pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (2977-19136) AttributeError: 'NoneType' object has no attribute 'text'
Jan 29 22:55:14 satellite pulp: pulp.server.event.http:INFO: (2977-19136) {'call_report': {u'exception': None, u'task_type': u'pulp.server.managers.repo.sync.sync', u'task_id': u'0aa038fc-4885-49e2-965a-894a1a067f8b', u'tags': [u'pulp:repository:test_epel5_pulp', u'pulp:action:sync'], u'finish_time': None, u'_ns': u'task_status', u'start_time': u'2016-01-30T03:51:15Z', u'traceback': None, u'spawned_tasks': [], u'progress_report': {u'yum_importer': {u'content': {u'size_total': 0, u'items_left': 0, u'items_total': 0, u'state': u'FINISHED', u'size_left': 0, u'details': {u'rpm_total': 0, u'rpm_done': 0, u'drpm_total': 0, u'drpm_done': 0}, u'error_details': []}, u'comps': {u'state': u'NOT_STARTED'}, u'distribution': {u'items_total': 0, u'state': u'FINISHED', u'error_details': [], u'items_left': 0}, u'errata': {u'state': u'FAILED', u'error': u"'NoneType' object has no attribute 'text'"}, u'metadata': {u'state': u'FINISHED'}}}, u'state': u'running', u'worker_name': u'reserved_resource_worker-0.us', u'result': None, u'error': None, u'_id': ObjectId('56ac3333295511b21ab954d7'), u'id': u'56ac3333c024450b6e4db11b'}, 'event_type': 'repo.sync.finish', 'payload': {'importer_id': u'yum_importer', 'exception': None, 'repo_id': u'test_epel5_pulp', 'traceback': None, 'started': '2016-01-30T03:51:15Z', '_ns': u'repo_sync_results', 'completed': '2016-01-30T03:55:14Z', 'importer_type_id': u'yum_importer', 'error_message': None, 'summary': {'content': {'state': 'FINISHED'}, 'comps': {'state': 'NOT_STARTED'}, 'distribution': {'state': 'FINISHED'}, 'errata': {'state': 'FAILED'}, 'metadata': {'state': 'FINISHED'}}, 'added_count': 0, 'result': 'failed', 'updated_count': 6781, 'details': {'content': {'size_total': 0, 'items_left': 0, 'items_total': 0, 'state': 'FINISHED', 'size_left': 0, 'details': {'rpm_total': 0, 'rpm_done': 0, 'drpm_total': 0, 'drpm_done': 0}, 'error_details': []}, 'comps': {'state': 'NOT_STARTED'}, 'distribution': {'items_total': 0, 'state': 'FINISHED', 'error_details': [], 'items_left': 0}, 'errata': {'state': 'FAILED',
Jan 29 22:55:14 satellite pulp: pulp.server.event.http:INFO: (2977-19136)  'error': "'NoneType' object has no attribute 'text'"}, 'metadata': {'state': 'FINISHED'}}, 'id': '56ac3422c024450ba1aa5bc6', 'removed_count': 0}}
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136) Task pulp.server.managers.repo.sync.sync[0aa038fc-4885-49e2-965a-894a1a067f8b] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136) Traceback (most recent call last):
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136)   File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136)     R = retval = fun(*args, **kwargs)
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136)   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 328, in __call__
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136)     return super(Task, self).__call__(*args, **kwargs)
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136)   File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136)     return self.run(*args, **kwargs)
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136)   File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/sync.py", line 114, in sync
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136)     raise PulpExecutionException(_('Importer indicated a failed response'))
Jan 29 22:55:14 satellite pulp: celery.worker.job:ERROR: (2941-19136) PulpExecutionException: Importer indicated a failed response
Jan 29 22:55:14 satellite pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[af88ae8a-7aeb-47af-8d60-c50c22246ac6] succeeded in 0.011311418988s: None
Jan 29 22:55:14 satellite pulp: pulp.server.event.http:WARNING: Error response from HTTP notifier: {"displayMessage":"Couldn't find repository 'test_epel5_pulp'","errors":["Couldn't find repository 'test_epel5_pulp'"]}
</log>

Expected results:
Works.

Additional info:
Having a look at the updateinfo for EPEL5:

<epel5>
[fmhirtz@x1 epel5]$ wget -q https://dl.fedoraproject.org/pub/epel/5/x86_64/repodata/c0f4c9f3e20d7aa48e98b37e9cdf713ed9060280-updateinfo.xml.gz[fmhirtz@x1 epel5]$ ls
c0f4c9f3e20d7aa48e98b37e9cdf713ed9060280-updateinfo.xml.gz
[fmhirtz@x1 epel5]$ zgrep '<id>' c0f4c9f3e20d7aa48e98b37e9cdf713ed9060280-updateinfo.xml.gz | wc -l
1877
[fmhirtz@x1 epel5]$ zgrep '<name>' c0f4c9f3e20d7aa48e98b37e9cdf713ed9060280-updateinfo.xml.gz | wc -l
1877
[fmhirtz@x1 epel5]$ zgrep '<description>' c0f4c9f3e20d7aa48e98b37e9cdf713ed9060280-updateinfo.xml.gz | wc -l
1652
[fmhirtz@x1 epel5]$ 
</epel5>

As a sanity check, EPEL6 doesn't have any such discrepancy:

<epel6 (works)>
[fmhirtz@x1 epel6]$ wget -q https://dl.fedoraproject.org/pub/epel/6/x86_64/repodata/7b7192d8e50b4613e2cca15a0a4875d6e985a8a96077151f7eebea7681831596-updateinfo.xml.bz2
[fmhirtz@x1 epel6]$ ls
7b7192d8e50b4613e2cca15a0a4875d6e985a8a96077151f7eebea7681831596-updateinfo.xml.bz2
[fmhirtz@x1 epel6]$ bzcat 7b7192d8e50b4613e2cca15a0a4875d6e985a8a96077151f7eebea7681831596-updateinfo.xml.bz2 | grep '<id>' | wc -l
3826
[fmhirtz@x1 epel6]$ bzcat 7b7192d8e50b4613e2cca15a0a4875d6e985a8a96077151f7eebea7681831596-updateinfo.xml.bz2 | grep '<name>' | wc -l
3826
[fmhirtz@x1 epel6]$ bzcat 7b7192d8e50b4613e2cca15a0a4875d6e985a8a96077151f7eebea7681831596-updateinfo.xml.bz2 | grep '<description>' | wc -l
3826
[fmhirtz@x1 epel6]$ 
</epel6>