Bug 1367762

Summary:	[Insights] 500 Internal Server Error: Upload Failed
Product:	Red Hat Quickstart Cloud Installer	Reporter:	Thom Carlin <tcarlin>
Component:	Installation - RHEV	Assignee:	dgao
Status:	CLOSED ERRATA	QA Contact:	Tasos Papaioannou <tpapaioa>
Severity:	medium	Docs Contact:	Dan Macpherson <dmacpher>
Priority:	unspecified
Version:	1.0	CC:	bthurber, dgao, jesusr, jkim, jmatthew, joden, smallamp, tpapaioa, tsanders
Target Milestone:	---	Keywords:	Triaged
Target Release:	1.1
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1367867 (view as bug list)		Environment:
Last Closed:	2017-02-28 01:38:22 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Bug Depends On:
Bug Blocks:	1367867

Description Thom Carlin 2016-08-17 12:07:16 UTC

Description of problem:

During RHV deployment, insights upload ultimately causes RHV deployment failure

Version-Release number of selected component (if applicable):

QCI-1.0-RHEL-7-20160815.t.0

How reproducible:

Believe 100% for this configuration

Steps to Reproduce:
1. Install/configure QCI
2. Deploy RHV with Insights enabled and OpenShift
3. Monitor Installation Progress

Actual results:

RHV deployment error at 82.1%

Expected results:

Successful deployment

Additional info:

RHV deployment failed at Actions::Fusor::Deployment::Rhev::Deploy
Error was: "====== Puppet run for host <<engine_fqdn>> status reported as Error ======"

On engine system, /var/log/messages:
yum[12869]: Installed: redhat-access-insights-1.0.6-0.el7.noarch
puppet-agent[4587]: (/Stage[main]/Access_insights_client/Package[redhat-access-insights]/ensure) created
puppet-agent[4587]: (/Stage[main]/Access_insights_client/Exec[/usr/bin/redhat-access-insights --register]/returns) ERROR: Upload failed!
puppet-agent[4587]: (/Stage[main]/Access_insights_client/Exec[/usr/bin/redhat-access-insights --register]/returns) Debug Information:
puppet-agent[4587]: (/Stage[main]/Access_insights_client/Exec[/usr/bin/redhat-access-insights --register]/returns) HTTP Status Code: 500
puppet-agent[4587]: (/Stage[main]/Access_insights_client/Exec[/usr/bin/redhat-access-insights --register]/returns) HTTP Status Text: Internal Server Error
puppet-agent[4587]: /usr/bin/redhat-access-insights --register returned 1 instead of one of [0]
puppet-agent[4587]: (/Stage[main]/Access_insights_client/Exec[/usr/bin/redhat-access-insights --register]/returns) change from notrun to 0 failed: /usr/bin/redhat-access-insig
hts --register returned 1 instead of one of [0]

NOTE: This may be 2 issues conflated

Comment 3 Thom Carlin 2016-08-17 13:25:28 UTC

This seems to be due to a intermittent communication issue with Customer Portal  "RedhatAccess::Telemetry::PortalClient: Caught HTTP error when proxying call to tapi: Request Timeout" along with "ActionView::MissingTemplate"

After several hours research on my part, resuming the task led to succcessful completion of the deployment

Comment 5 John Matthews 2016-08-17 16:27:18 UTC

We are deferring this issue to post GA

Comment 7 John Matthews 2016-08-17 16:34:48 UTC

*** Bug 1366013 has been marked as a duplicate of this bug. ***

Comment 8 jkim 2016-08-24 14:29:59 UTC

Note: This occurred again in a libvirt environment, deploying RHV, self-hosted with 2 hypervisors in QCI-1.0-RHEL-7-20160823.t.1 ISO
Failed in the same manor as described above with similar trace.

Comment 9 James Olin Oden 2016-09-06 13:53:01 UTC

This occurred again in  a libvirt environment, deploying the basic two node RHV deployment (hypervisor + engine) in the QCI-1.0-RHEL-7-20160902.5 ISO.
Failed in the same manor as described above with similar trace.

Comment 10 dgao 2016-09-21 20:21:24 UTC

[root@mac5254007ae704 ~]# redhat-access-insights --register --verbose
<snip>
POST System: https://sat62fusor.example.com:443/redhat_access/r/insights/v1/systems
{"machine_id": "7e12eaf556664e0f9acb4d846377e14c", "remote_branch": "c78566e1-c6f7-43f9-b33c-b0613f767e9f", "hostname": "mac5254007ae704.example.com", "remote_leaf": "eb3d74b0-4b71-4e95-8796-2bb1ef8ff59d"}
POST System status: 500
ERROR: Upload failed!
Debug Information:
HTTP Status Code: 500
HTTP Status Text: Internal Server Error

This was found w/ access_insights plugin v1.0.11

After speaking with lphiri, issue is fixed w/ v1.0.13 found in:

https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=511791
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=511794

Comment 11 Tasos Papaioannou 2016-10-07 20:23:51 UTC

Verified on QCI-1.1-RHEL-7-20161006.t.0.

Comment 14 errata-xmlrpc 2017-02-28 01:38:22 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:0335