Bug 2126319

Summary: [Sat6/InisghtsSync/Bug] Insights Sync failing frequently on Sat6 instances
Product: Red Hat Satellite Reporter: Nikhil Gupta <ngupta>
Component: RH Cloud - InsightsAssignee: Shimon Shtein <sshtein>
Status: CLOSED DUPLICATE QA Contact: addubey
Severity: high Docs Contact:
Priority: high    
Version: 6.11.2CC: ahumbe, aruzicka, pdwyer, peter.vreman, sshtein
Target Milestone: UnspecifiedKeywords: Triaged, WorkAround
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-22 08:53:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nikhil Gupta 2022-09-13 07:15:32 UTC
Description of problem:
The Satellite 6 instances show too frequently failing Insights Sync failures resulting in Paused tasks. 

# (  zgrep InventoryScheduledSync /var/log/foreman/production.log-*.gz ; grep InventoryScheduledSync /var/log/foreman/production.log* ) | grep -v '\[D|' | grep result | grep -v success
/var/log/foreman/production.log-20220830.gz:2022-08-30T00:00:18 [I|bac|604517ea] Task {label: InventorySync::Async::InventoryScheduledSync, id: 4d769b54-0f7a-42d3-9ba1-c57dea1380c3, execution_plan_id: 1f20f6c4-4550-4f66-a417-3c7a9650fdbb} state changed: paused  result: error
/var/log/foreman/production.log-20220905.gz:2022-09-05T00:00:11 [I|bac|3d6bed41] Task {label: InventorySync::Async::InventoryScheduledSync, id: 95e656e8-3272-4456-8dbb-baaccace0606, execution_plan_id: 913f87d3-279a-4e8e-95d3-43e2c1465c68} state changed: paused  result: error


Version-Release number of selected component (if applicable):
Satellite 6.11.2

How reproducible:
Intermittent

Steps to Reproduce:
1. Schedule Insights inventory sync on the Satellite server


Actual results:
Insights Sync failures resulting in Paused tasks.

Expected results:
SaaS provided shall be reliable and 99.99% available. The measurable expectation is to fail max 1x per 3 months. Currently, it fails 1-3 times per week.

Additional info:
Manually restarting the tasks makes them work

Comment 4 Peter Vreman 2022-10-06 14:02:04 UTC
I see the problem with all my Sat6 instances, being 6.9, 6.10 and now also on all 6.11.
It is not proxy related because my Sat6.11 are directly connected without a Proxy

On 6.11 both 'Inventory scheduled sync' and 'Insights scheduled sync' are affected
The errors can be categorized in 2:
- '401 Unauthorized'
- '504 Gateway Timeout'

Can it be maybe that it is congestion because out-of-the-box every RedHat Satellite is configured to do the uplaod exactlu at 0:00am (in my case UTC).

Comment 7 Ashish Humbe 2022-11-22 08:53:37 UTC
Closing this bz as a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=2127180 

The engineering team is actively working on this issue

*** This bug has been marked as a duplicate of bug 2127180 ***