Bug 1013681 - Tooltip for failed sync has only one line from stacktrace
Tooltip for failed sync has only one line from stacktrace
Product: Red Hat Satellite 6
Classification: Red Hat
Component: WebUI (Show other bugs)
Unspecified Unspecified
unspecified Severity low (vote)
: Unspecified
: --
Assigned To: Eric Helms
Sachin Ghai
: Triaged
: 1010096 1025005 (view as bug list)
Depends On:
  Show dependency treegraph
Reported: 2013-09-30 11:23 EDT by Og Maciel
Modified: 2014-03-11 12:43 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2014-03-11 12:43:59 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Tooltip screenshot (1.55 MB, image/png)
2013-09-30 11:23 EDT, Og Maciel
no flags Details
no logs under notifications (63.30 KB, image/png)
2013-10-21 08:00 EDT, Sachin Ghai
no flags Details
tool_tip says 'Not Found' (87.48 KB, image/png)
2013-10-21 08:01 EDT, Sachin Ghai
no flags Details

  None (edit)
Description Og Maciel 2013-09-30 11:23:56 EDT
Created attachment 805257 [details]
Tooltip screenshot

Description of problem:

The attached screenshot shows that the tooltip for my failed sync process contains one single line of what could possibly be many lines of a stacktrace. Also, there's nothing shown on Notifications page.

Version-Release number of selected component (if applicable):

* apr-util-ldap-1.3.9-3.el6_0.1.x86_64
* candlepin-0.8.25-1.el6sam.noarch
* candlepin-scl-1-5.el6_4.noarch
* candlepin-scl-quartz-2.1.5-5.el6_4.noarch
* candlepin-scl-rhino-1.7R3-1.el6_4.noarch
* candlepin-scl-runtime-1-5.el6_4.noarch
* candlepin-selinux-0.8.25-1.el6sam.noarch
* candlepin-tomcat6-0.8.25-1.el6sam.noarch
* elasticsearch-0.19.9-8.el6sat.noarch
* foreman-1.3.0-6.el6sat.noarch
* foreman-compute-1.3.0-6.el6sat.noarch
* foreman-libvirt-1.3.0-6.el6sat.noarch
* foreman-postgresql-1.3.0-6.el6sat.noarch
* foreman-proxy-1.3.0-1.el6sat.noarch
* katello-1.4.6-14.el6sat.noarch
* katello-all-1.4.6-14.el6sat.noarch
* katello-candlepin-cert-key-pair-1.0-1.noarch
* katello-certs-tools-1.4.4-1.el6sat.noarch
* katello-cli-1.4.3-15.el6sat.noarch
* katello-cli-common-1.4.3-15.el6sat.noarch
* katello-common-1.4.6-14.el6sat.noarch
* katello-configure-1.4.5-8.el6sat.noarch
* katello-configure-foreman-1.4.5-8.el6sat.noarch
* katello-configure-foreman-proxy-1.4.5-8.el6sat.noarch
* katello-foreman-all-1.4.6-14.el6sat.noarch
* katello-glue-candlepin-1.4.6-14.el6sat.noarch
* katello-glue-elasticsearch-1.4.6-14.el6sat.noarch
* katello-glue-pulp-1.4.6-14.el6sat.noarch
* katello-qpid-broker-key-pair-1.0-1.noarch
* katello-qpid-client-key-pair-1.0-1.noarch
* katello-selinux-1.4.4-4.el6sat.noarch
* openldap-2.4.23-31.el6.x86_64
* pulp-katello-plugins-0.2-1.el6sat.noarch
* pulp-nodes-common-2.3.0-0.14.alpha.el6sat.noarch
* pulp-nodes-parent-2.3.0-0.14.alpha.el6sat.noarch
* pulp-puppet-plugins-2.3.0-0.14.alpha.el6sat.noarch
* pulp-rpm-plugins-2.3.0-0.14.alpha.el6sat.noarch
* pulp-selinux-2.3.0-0.14.alpha.el6sat.noarch
* pulp-server-2.3.0-0.14.alpha.el6sat.noarch
* python-ldap-2.3.10-1.el6.x86_64
* ruby193-rubygem-ldap_fluff-0.2.2-2.el6sat.noarch
* ruby193-rubygem-net-ldap-0.3.1-3.el6sat.noarch
* ruby193-rubygem-runcible-1.0.7-1.el6sat.noarch
* signo-0.0.20-3.el6sat.noarch
* signo-katello-0.0.20-3.el6sat.noarch

How reproducible:

Steps to Reproduce:
1. Try to make a repo sync fail (perhaps create a repo with an invalid url?). I had all of RHEL 5 and 6 for i386, ppc, s390 and x86_64 enabled and syncing fwiw

Actual results:

Tooltip shows only one line

Expected results:

Additional info:
Comment 1 Og Maciel 2013-09-30 11:28:44 EDT
Here's what the entire stacktrace shows:

2013-09-30 00:31:09,528 urllib3.connectionpool:INFO: Starting new HTTPS connection (2): cdn.redhat.com
2013-09-30 00:31:09,898 urllib3.connectionpool:INFO: Resetting dropped connection: cdn.redhat.com
2013-09-30 00:31:10,528 urllib3.connectionpool:WARNING: Retrying (0 attempts remain) after connection broken by 'BadStatusLine('',)': /content/dist/rhel/power/6/6Server/ppc64/os/Packages/python-decorator-3.0.1-3.1.el6.noarch.rpm
2013-09-30 00:31:10,532 nectar.downloaders.threaded:ERROR: HTTPSConnectionPool(host='cdn.redhat.com', port=443): Max retries exceeded with url: /content/dist/rhel/power/6/6Server/ppc64/os/Packages/python-decorator-3.0.1-3.1.el6.noarch.rpm
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nectar/downloaders/threaded.py", line 160, in _fetch
    response = session.get(request.url, headers=headers)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 310, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 279, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 374, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 209, in send
    raise ConnectionError(e)
ConnectionError: HTTPSConnectionPool(host='cdn.redhat.com', port=443): Max retries exceeded with url: /content/dist/rhel/power/6/6Server/ppc64/os/Packages/python-decorator-3.0.1-3.1.el6.noarch.rpm
2013-09-30 00:31:10,539 urllib3.connectionpool:INFO: Starting new HTTPS connection (2): cdn.redhat.com
2013-09-30 00:31:11,315 urllib3.connectionpool:INFO: Resetting dropped connection: cdn.redhat.com
2013-09-30 00:31:13,353 urllib3.connectionpool:INFO: Resetting dropped connection: cdn.redhat.com
2013-09-30 00:37:37,695 urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
2013-09-30 00:37:38,049 nectar.downloaders.threaded:ERROR: Download of https://cdn.redhat.com/content/dist/rhel/power/6/6Server/ppc64/os/.treeinfo failed with code 404: Not Found
Comment 3 Eric Helms 2013-10-02 16:11:09 EDT
Tracking upstream - https://github.com/Katello/katello/pull/3069
Comment 4 Eric Helms 2013-10-02 16:11:26 EDT
*** Bug 1010096 has been marked as a duplicate of this bug. ***
Comment 5 Eric Helms 2013-10-03 08:30:34 EDT
The fix is available as of ad5d5b72eb788c0d52da234c571bd1b4ca44d3a2

This fix updates the algorithm for parsing pulp sync status objects and updates the tooltip error output. Further, the tooltip is now a clickable link to the notifications page. 

This fix also re-enables the after-sync notification handling from Pulp. The user will once again be presented with a notification upon completion of a sync. And, if there are errors, a detailed view of those errors will be present on the notifications page.
Comment 8 Sachin Ghai 2013-10-21 08:00:04 EDT
Verified with snap6 compose2 (MDP2)


Sync failed and now tool tip just says 'Not Found' and when I moved to notices page by clicking on it, there were no error listed. 

no stack trace in production.log, it says:
[ERROR 2013-10-21 11:48:24 app  #5873] Task PulpSyncStatus (22) is in error state
[ERROR 2013-10-21 11:48:24 app  #5873] *** Sync error:
Comment 9 Sachin Ghai 2013-10-21 08:00:57 EDT
Created attachment 814554 [details]
no logs under notifications
Comment 10 Sachin Ghai 2013-10-21 08:01:52 EDT
Created attachment 814556 [details]
tool_tip says 'Not Found'
Comment 11 Sachin Ghai 2013-10-21 08:02:51 EDT
Pulp.log says the exact error:

pulp.log says:

2013-10-21 11:48:21,470 pulp_rpm.plugins.importers.yum.sync:ERROR: sync failed
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 109, in run
    metadata_files = self.get_metadata()
  File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 194, in get_metadata
    raise FailedException(str(e))
FailedException: Not Found
2013-10-21 11:48:21,472 pulp.server.event.http:INFO: {'call_report': {'task_group_id': '656d1c8d-bd27-498a-96ac-9003f6a4499a', 'exception': None, 'traceback': None, 'task_id': 'a2665234-6067-46b4-b8a1-447e31dc2767', 'call_request_tags': [u'pulp:repository:ACME_Corporation-product1-zoo', 'pulp:action:sync'], 'reasons': [], 'start_time': '2013-10-21T06:18:20Z', 'tags': [u'pulp:repository:ACME_Corporation-product1-zoo', 'pulp:action:sync'], 'state': 'running', 'finish_time': None, 'dependency_failures': {}, 'schedule_id': None, 'progress': {u'yum_importer': {'content': {'size_total': 0, 'items_left': 0, 'items_total': 0, 'state': 'NOT_STARTED', 'size_left': 0, 'details': {'rpm_total': 0, 'rpm_done': 0, 'drpm_total': 0, 'drpm_done': 0}, 'error_details': []}, 'comps': {'state': 'NOT_STARTED'}, 'distribution': {'items_total': 0, 'state': 'NOT_STARTED', 'error_details': [], 'items_left': 0}, 'errata': {'state': 'NOT_STARTED'}, 'metadata': {'state': 'FAILED', 'error': 'Not Found'}}}, 'call_request_group_id': '656d1c8d-bd27-498a-96ac-9003f6a4499a', 'call_request_id': 'a2665234-6067-46b4-b8a1-447e31dc2767', 'principal_login': u'admin', 'response': 'accepted', 'result': None}, 'event_type': 'repo.sync.finish', 'payload': {'importer_id': u'yum_importer', 'exception': None, 'repo_id': u'ACME_Corporation-product1-zoo', 'traceback': None, 'started': '2013-10-21T11:48:20+05:30', '_ns': u'repo_sync_results', 'completed': '2013-10-21T11:48:21+05:30', 'importer_type_id': u'yum_importer', 'error_message': None, 'summary': {'content': {'state': 'NOT_STARTED'}, 'comps': {'state': 'NOT_STARTED'}, 'distribution': {'state': 'NOT_STARTED'}, 'errata': {'state': 'NOT_STARTED'}, 'metadata': {'state': 'FAILED'}}, 'added_count': 0, 'result': 'failed', 'updated_count': 0, 'details': {'content': {'size_total': 0, 'items_left': 0, 'items_total': 0, 'state': 'NOT_STARTED', 'size_left': 0, 'details': {'rpm_total': 0, 'rpm_done': 0, 'drpm_total': 0, 'drpm_done': 0}, 'error_details': []}, 'comps': {'state': 'NOT_STARTED'}, 'distribution': {'items_total': 0, 'state': 'NOT_STARTED', 'error_details': [], 'items_left': 0}, 'errata': {'state': 'NOT_STARTED'}, 'metadata': {'state': 'FAILED', 'error': 'Not Found'}}, 'id': '5264c72dacdb986ee80004c5', 'removed_count': 0}}
2013-10-21 11:48:21,474 pulp.server.dispatch.task:ERROR: Importer indicated a failed response
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/dispatch/task.py", line 138, in _run
    result = call(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/managers/repo/sync.py", line 116, in sync
    raise PulpExecutionException(_('Importer indicated a failed response'))
PulpExecutionException: Importer indicated a failed response
2013-10-21 11:48:21,478 pulp.server.dispatch.task:INFO: FAILURE: Task a2665234-6067-46b4-b8a1-447e31dc2767: CallRequest: RepoSyncManager.sync(u'ACME_Corporation-product1-zoo', sync_config_override={'num_threads': 4})
2013-10-21 11:48:24,965 pulp.server.dispatch.task:INFO: SUCCESS: Task 7dc8ceaf-8e32-426e-9faf-cf7eb359d246: CallRequest: ApplicabilityRegenerationManager.regenerate_applicability_for_repos({'sort': None, 'skip': None, 'limit': None, 'filters': {'id': {'$in': ['ACME_Corporation-product1-zoo']}}, 'fields': ['id'], '_id': ObjectId('5264c730acdb986ee80004cc'), 'id': '5264c730acdb986ee80004cc'})
Comment 12 Sachin Ghai 2013-10-21 08:06:58 EDT
Hummn. so from comment11, I got to know that the repo url was incorrect. But still I think user should get the error on UI/notifications which is logged in pulp.log ?
Comment 13 Eric Helms 2013-10-21 23:23:39 EDT
We do not have access to what the pulp.log file contains, only that which Pulp returns to us in the task data. In this case, all we know is:

'metadata': {'state': 'FAILED', 'error': 'Not Found'}}}

And the best we can do right now, is parse that out from the output above and present it to the user. In my opinion, this is doing the best we can right now, and further improvements need to be made on the Pulp side with regards to error reporting in the future.
Comment 14 Mike McCune 2013-11-01 16:09:15 EDT
*** Bug 1025005 has been marked as a duplicate of this bug. ***
Comment 15 Bryan Kearney 2014-03-11 12:43:59 EDT
Closing old bugs which are not relevant based on new UI and CLI

Note You need to log in before you can comment on or make changes to this bug.