Bug 1320471

Summary: "PulpExecutionException: Importer indicated a failed response" when syncing content for CFME
Product: Red Hat Quickstart Cloud Installer Reporter: Antonin Pagac <apagac>
Component: Installation - CloudFormsAssignee: John Matthews <jmatthew>
Status: CLOSED WORKSFORME QA Contact: Dave Johnson <dajohnso>
Severity: unspecified Docs Contact: Dan Macpherson <dmacpher>
Priority: unspecified    
Version: 1.0CC: bthurber
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-06-15 19:30:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
relevant part of production.log
none
errors tab from the ui none

Description Antonin Pagac 2016-03-23 10:02:43 UTC
Description of problem:
A subtask for CFME ISOs failed while syncing content in RHEV+CFME deployment with error message:

"PLP0000: Importer indicated a failed response"

/var/log/pulp is empty.

From messages:
"
Mar 23 05:27:57 localhost pulp: pulp.server.event.http:INFO: (18631-89024) {'call_report': {u'exception': None, u'task_type': u'pulp.server.managers.repo.sync.sync', u'task_id': u'53aaee8e-0607-48af-857f-e3ac52eb488a', u'tags': [u'pulp:repository:Default_Organization-Red_Hat_CloudForms-Red_Hat_CloudForms_Management_Engine_5_5_Files_x86_64', u'pulp:action:sync'], u'finish_time': None, u'_ns': u'task_status', u'start_time': u'2016-03-23T09:24:23Z', u'traceback': None, u'spawned_tasks': [], u'progress_report': {u'iso_importer': {u'error_message': None, u'traceback': None, u'finished_bytes': 7339439532L, u'num_isos': 8, u'state': u'isos_failed', u'iso_error_messages': [{u'name': u'cfme-vsphere-5.5.0.13-2.x86_64.vsphere.ova', u'error': {}}], u'state_times': {u'not_started': u'2016-03-23T09:24:23', u'manifest_in_progress': u'2016-03-23T09:24:23', u'isos_failed': u'2016-03-23T09:27:57', u'isos_in_progress': u'2016-03-23T09:24:24'}, u'num_isos_finished': 7, u'total_bytes': 7980279212L}}, u'state': u'running', u'worker_name': u'reserved_resource_worker-3.lab.eng.rdu2.redhat.com', u'result': None, u'error': None, u'_id': ObjectId('56f25f1c633fddd4fa8e29ba'), u'id': u'56f260c71191b048988befda'}, 'event_type': 'repo.sync.finish', 'payload': {'importer_id': u'iso_importer', 'exception': None, 'repo_id': u'Default_Organization-Red_Hat_CloudForms-Red_Hat_CloudForms_Management_Engine_5_5_Files_x86_64', 'traceback': None, 'started': '2016-03-23T09:24:23Z', '_ns': u'repo_sync_results', 'completed': '2016-03-23T09:27:57Z', 'importer_type_id': u'iso_importer', 'error_message': None, 'summary': {'traceback': None, 'error_message': None, 'finished_bytes': 7339439532, 'num_isos': 8, 'state': 'isos_failed', 'iso_error_messages': [{'name': 'cfme-vsphere-5.5.0.13-2.x86_64.vsphere.ova', 'error': {}}], 'total_bytes': 7980279212, 'num_isos_finished': 7, 'state_times': {'not_started': '2016-03-23T09:24:23', 'manifest_in_progress': '2016-03-23T09:24:23', 'isos_failed': '2016-03-23T09:27:57', 'isos_in_progress': '2016-03-23T09:24:24'}}, 'added_count':
Mar 23 05:27:57 localhost pulp: pulp.server.event.http:INFO: (18631-89024)  7, 'result': 'failed', 'updated_count': 0, 'details': None, 'id': '56f2619d1191b048c700e4e6', 'removed_count': 0}}
Mar 23 05:27:57 localhost pulp: celery.worker.job:ERROR: (18502-89024) Task pulp.server.managers.repo.sync.sync[53aaee8e-0607-48af-857f-e3ac52eb488a] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Mar 23 05:27:57 localhost pulp: celery.worker.job:ERROR: (18502-89024) Traceback (most recent call last):
Mar 23 05:27:57 localhost pulp: celery.worker.job:ERROR: (18502-89024)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
Mar 23 05:27:57 localhost pulp: celery.worker.job:ERROR: (18502-89024)     R = retval = fun(*args, **kwargs)
Mar 23 05:27:57 localhost pulp: celery.worker.job:ERROR: (18502-89024)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 328, in __call__
Mar 23 05:27:57 localhost pulp: celery.worker.job:ERROR: (18502-89024)     return super(Task, self).__call__(*args, **kwargs)
Mar 23 05:27:57 localhost pulp: celery.worker.job:ERROR: (18502-89024)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
Mar 23 05:27:57 localhost pulp: celery.worker.job:ERROR: (18502-89024)     return self.run(*args, **kwargs)
Mar 23 05:27:57 localhost pulp: celery.worker.job:ERROR: (18502-89024)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/sync.py", line 114, in sync
Mar 23 05:27:57 localhost pulp: celery.worker.job:ERROR: (18502-89024)     raise PulpExecutionException(_('Importer indicated a failed response'))
Mar 23 05:27:57 localhost pulp: celery.worker.job:ERROR: (18502-89024) PulpExecutionException: Importer indicated a failed response
"

Attaching excerpts from production.log and the error tab in the UI.

Version-Release number of selected component (if applicable):
TP3 RC2

How reproducible:
Happened to me once

Steps to Reproduce:
1. Start a RHEV+CFME deployment
2. Some time into the content sync the subtask for CFME ISOs fails
3.

Actual results:
Sync subtask for CFME ISOs fails

Expected results:
Sync completes successfully

Additional info:

Comment 1 Antonin Pagac 2016-03-23 10:03:35 UTC
Created attachment 1139448 [details]
relevant part of production.log

Comment 2 Antonin Pagac 2016-03-23 10:03:59 UTC
Created attachment 1139450 [details]
errors tab from the ui

Comment 3 Antonin Pagac 2016-03-23 12:48:28 UTC
The new Redeploy button worked and I'm now seeing completed content sync for CFME.

Comment 4 John Matthews 2016-03-30 14:22:00 UTC
I've seen this problem in the past when there was an issue with metadata on the CDN for file repos.

We hit a similar issue with OSE syncing when there was an issue with Akamai cache, rel-eng flushed the cache and it fixed the issue.

Comment 5 John Matthews 2016-06-15 19:30:11 UTC
The Content Sync resume was added to workaround these issues.