Bug 1965901

Summary: Unable to sync insights recommendations, task fails with "Unable to authenticate using rh_cloud_token setting".
Product: Red Hat Satellite Reporter: Jameer Pathan <jpathan>
Component: RH Cloud - InventoryAssignee: Shimon Shtein <sshtein>
Status: CLOSED ERRATA QA Contact: Jameer Pathan <jpathan>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.10.0CC: aruzicka, egolov, fgrosjea
Target Milestone: 6.10.0Keywords: Regression, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tfm-rubygem-foreman-tasks-4.1.2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-16 14:11:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jameer Pathan 2021-05-31 05:24:58 UTC
Description of problem:
Unable to sync insights recommendations, task fails with "Unable to authenticate using rh_cloud_token setting".

Version-Release number of selected component (if applicable):
- Satellite 6.10.0 snap 2
- tfm-rubygem-foreman_rh_cloud-3.0.20-1.el7sat.noarch

How reproducible:
- Always

Steps to Reproduce:
- Import manifest
- Go to Configure > Insights
- Add RH Cloud token
- Click on "Start Recommendation sync"
- Click on task link

Actual results:
- Task fails with following error message:
ERF50-1206 [Foreman::WrappedException]: Unable to authenticate using rh_cloud_token setting ([RestClient::BadRequest]: 400 Bad Request)ERF50-1206 [Foreman::WrappedException]: Unable to authenticate using rh_cloud_token setting ([RestClient::BadRequest]: 400 Bad Request)ERF50-1206 [Foreman::WrappedException]: Unable to authenticate using rh_cloud_token setting ([RestClient::BadRequest]: 400 Bad Request)ERF50-1206 [Foreman::WrappedException]: Unable to authenticate using rh_cloud_token setting ([RestClient::BadRequest]: 400 Bad Request)


Expected results:
- Successfully syncing insights recommendations.

Additional info:

Comment 2 Jameer Pathan 2021-05-31 12:51:16 UTC
This issue is not observed after I restart satellite services using 
foreman-maintain(even though everything is running already) and doing `systemctl daemon-reload`.

[root@dhcp-2-149 ~]# foreman-maintain service restart
Running Restart Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Restart applicable services: 

Stopping the following service(s):
rh-redis5-redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-resource-manager, rh-redis5-redis, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
/ stopping puppetserver                                                         
Warning: Stopping foreman.service, but it can still be activated by:
  foreman.socket
/ stopping rh-redis5-redis                                                      
Warning: postgresql.service changed on disk. Run 'systemctl daemon-reload' to reload units.

Warning: Stopping pulpcore-api.service, but it can still be activated by:
  pulpcore-api.socket

Warning: Stopping pulpcore-content.service, but it can still be activated by:
  pulpcore-content.socket
\ All services stopped                                                          

Starting the following service(s):
rh-redis5-redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-resource-manager, rh-redis5-redis, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
- starting rh-redis5-redis                                                      
Warning: postgresql.service changed on disk. Run 'systemctl daemon-reload' to reload units.
- starting puppetserver                                                         
Job for foreman.service failed because the control process exited with error code. See "systemctl status foreman.service" and "journalctl -xe" for details.
| All services started                                                [OK]      
--------------------------------------------------------------------------------

[root@dhcp-2-149 ~]# systemctl daemon-reload

Comment 3 Jameer Pathan 2021-05-31 12:53:28 UTC
After performin step mentioned in comment#2, task related to adding rh_token finishes successfully.
But even after that satellite doesn't show any insights recommendations.

Comment 4 Jameer Pathan 2021-06-01 09:12:04 UTC
*** Bug 1966418 has been marked as a duplicate of this bug. ***

Comment 7 Evgeni Golov 2021-06-01 10:07:23 UTC
As written in BZ#1966418:

I don't think the component (Foreman Maintain) is correct.

If this is a freshly deployed machine (or a fresh upgrade), the Installer should have kicked systemd accordingly after modifying the file.
If it didn't, that's an Installer bug.

At the same time, the attached traceback doesn't show anything about *why* things are not working.
So please provide a full one, or a reproducer machine.


However, this BZ has more logs attached, and they log that `foreman.service` could not be started properly.

Can we please see a reproducer machine?

Comment 9 Adam Ruzicka 2021-06-11 13:04:32 UTC
This bz was a symptom of underlying bug in foreman-tasks/dynflow. dynflow-sidekiq@* processes loaded settings on process start and used that cached view of settings throughout their entire life. The fix will land in foreman-tasks-4.1.2

Comment 10 Jameer Pathan 2021-06-28 12:51:15 UTC
Verified:

Verified with:
- Satellite 6.10.0 snap 6
- tfm-rubygem-foreman_rh_cloud-4.0.22-1.el7sat.noarch
- tfm-rubygem-foreman-tasks-4.1.2-1.el7sat.noarch

Test steps:
- Import manifest
- Register a satellite content host with insights.
- Go to Configure > Insights
- Add RH Cloud token
- Click on "Start Recommendation sync"
- Click on task link

Observation:
- Successfully synced insights recommendations.

Comment 13 errata-xmlrpc 2021-11-16 14:11:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4702