Bug 2222986

Summary: once subscription_connection_enabled is turned off, turning it back on is intermittent and unreliable
Product: Red Hat Satellite Reporter: Paul Dudley <pdudley>
Component: Subscription ManagementAssignee: satellite6-bugs <satellite6-bugs>
Status: NEW --- QA Contact: Satellite QE Team <sat-qe-bz-list>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.11.4CC: chrobert
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Paul Dudley 2023-07-14 18:23:14 UTC
In Satellite 6.11.z even after setting subscription_connection_enabled back to true from being false the setting is intermittent, sometimes setting itself back to false when loading the /subscriptions page.

--------------------------------|---------------------------------|-------|-----------------------------------------------------------                                                                                                                                                                                       
NAME                            | FULL NAME                       | VALUE | DESCRIPTION                                                                                                                                                                                                                                      
--------------------------------|---------------------------------|-------|-----------------------------------------------------------                                                                                                                                                                                       
subscription_connection_enabled | Subscription connection enabled | false | Can communicate with the Red Hat Portal for subscriptions.                                                                                                                                                                                       
--------------------------------|---------------------------------|-------|-----------------------------------------------------------                                                                                                                                                                                       
[root@sat611-rhel7 ~]# hammer settings list --search subscription_connection_enabled                                                                                                                                                                                                                                     
Warning: An error occured while loading module hammer_cli_foreman_puppet.                                                                                                                                                                                                                                                    
--------------------------------|---------------------------------|-------|-----------------------------------------------------------                                                                                                                                                                                       
NAME                            | FULL NAME                       | VALUE | DESCRIPTION                                                                                                                                                                                                                                      
--------------------------------|---------------------------------|-------|-----------------------------------------------------------                                                                                                                                                                                       
subscription_connection_enabled | Subscription connection enabled | true  | Can communicate with the Red Hat Portal for subscriptions.                                                                                                                                                                                       
--------------------------------|---------------------------------|-------|-----------------------------------------------------------                                                                                                                                                                                       
...
[root@sat611-rhel7 ~]# hammer settings list --search subscription_connection_enabled                                                                                                                                                                                                                                     
Warning: An error occured while loading module hammer_cli_foreman_puppet.                                                                                                                                                                                                                                                    
--------------------------------|---------------------------------|-------|-----------------------------------------------------------                                                                                                                                                                                       
NAME                            | FULL NAME                       | VALUE | DESCRIPTION                                                                                                                                                                                                                                      
--------------------------------|---------------------------------|-------|-----------------------------------------------------------                                                                                                                                                                                       
subscription_connection_enabled | Subscription connection enabled | true  | Can communicate with the Red Hat Portal for subscriptions.                                                                                                                                                                                       
--------------------------------|---------------------------------|-------|-----------------------------------------------------------                                                                                                                                                                                       
[root@sat611-rhel7 ~]# hammer settings list --search subscription_connection_enabled                                                                                                                                                                                                                                     
Warning: An error occured while loading module hammer_cli_foreman_puppet.                                                                                                                                                                                                                                                    
--------------------------------|---------------------------------|-------|-----------------------------------------------------------                                                                                                                                                                                       
NAME                            | FULL NAME                       | VALUE | DESCRIPTION                                                                                                                                                                                                                                      
--------------------------------|---------------------------------|-------|-----------------------------------------------------------                                                                                                                                                                                       
subscription_connection_enabled | Subscription connection enabled | false | Can communicate with the Red Hat Portal for subscriptions.                                                                                                                                                                                       
--------------------------------|---------------------------------|-------|-----------------------------------------------------------                                                                                                                                                                                       

Those are all from queries made on Jul 14 2023. But looking at the setting you can see the setting was last updated on the 13th;
[root@sat611-rhel7 ~]# su - postgres -c "psql -d foreman  -x -c \"select * from settings where name='subscription_connection_enabled'\""
-[ RECORD 1 ]-+-----------------------------------------------------------
id            | 79
name          | subscription_connection_enabled
value         | 
description   | Can communicate with the Red Hat Portal for subscriptions.
category      | Setting::Content
settings_type | boolean
default       | --- true                                                  +
              | ...
created_at    | 2022-12-22 23:26:25.819037
updated_at    | 2023-07-13 17:19:10.969237
full_name     | Subscription connection enabled
encrypted     | f

Comment 1 Paul Dudley 2023-07-14 19:16:42 UTC
You can see for this I've changed the setting many times on the 13th;
[root@sat611-rhel7 ~]# grep -ir connection /var/log/foreman/ | grep 'Started PUT'
/var/log/foreman/production.log-20230714:2023-07-13T13:09:18 [I|app|a3b7ffaf] Started PUT "/api/settings/subscription_connection_enabled" for x.x.x.x at 2023-07-13 13:09:18 -0400
/var/log/foreman/production.log-20230714:2023-07-13T13:12:20 [I|app|36d481fd] Started PUT "/api/settings/subscription_connection_enabled" for x.x.x.x at 2023-07-13 13:12:20 -0400
...
[root@sat611-rhel7 ~]# grep -ir connection /var/log/foreman/ | grep 'Started PUT' | wc
     16     176    2930
(lots of PUTs and GETs from troubleshooting)

There are no changes on the 14th, only GETs when querying the setting;
[root@sat611-rhel7 ~]# grep -i connection /var/log/foreman/production.log | grep 'Started GET'
2023-07-14T12:20:37 [I|app|4b3cb692] Started GET "/api/settings?search=subscription_connection_enabled&page=1&per_page=1000" for x.x.x.x at 2023-07-14 12:20:37 -0400
2023-07-14T12:20:41 [I|app|d2ccdc9e] Started GET "/api/settings?search=subscription_connection_enabled&page=1&per_page=1000" for x.x.x.x at 2023-07-14 12:20:41 -0400
...
[root@sat611-rhel7 ~]# grep -i connection /var/log/foreman/production.log | grep 'Started PUT'
[root@sat611-rhel7 ~]# 
(no PUTs, no changes to the setting even though we're still sometimes retrieving that the setting is false)

Still retrieving the error when loading the page;
[root@sat611-rhel7 ~]# grep -ir prohibited /var/log/foreman                                                                                                                                                                                                                                                              
/var/log/foreman/production.log:2023-07-14T12:22:57 [E|app|2ab99535] Katello::HttpErrors::BadRequest: Access to Red Hat Subscription Management is prohibited. If you would like to change this, please update the content setting 'Subscription connection enabled'.                                                        
/var/log/foreman/production.log:2023-07-14T12:52:05 [E|app|eb40db19] Katello::HttpErrors::BadRequest: Access to Red Hat Subscription Management is prohibited. If you would like to change this, please update the content setting 'Subscription connection enabled'.                                                        
/var/log/foreman/production.log:2023-07-14T13:06:02 [E|app|5499fb0e] Katello::HttpErrors::BadRequest: Access to Red Hat Subscription Management is prohibited. If you would like to change this, please update the content setting 'Subscription connection enabled'.                                                        
/var/log/foreman/production.log:2023-07-14T13:07:09 [E|app|733b6957] Katello::HttpErrors::BadRequest: Access to Red Hat Subscription Management is prohibited. If you would like to change this, please update the content setting 'Subscription connection enabled'.                                                        
/var/log/foreman/production.log:2023-07-14T15:01:59 [E|app|4ed9d3ad] Katello::HttpErrors::BadRequest: Access to Red Hat Subscription Management is prohibited. If you would like to change this, please update the content setting 'Subscription connection enabled'.

Comment 3 Paul Dudley 2023-07-14 19:48:17 UTC
The main issue here is that 
1) The manifest cannot be refreshed while this value is being retrieved as false, even fails on the command line as seen in;
~~~
[root@sat611-rhel7 ~]# hammer settings set --name subscription_connection_enabled --value true ; sleep 10 ; hammer subscription refresh-manifest --organization ACME
Setting [subscription_connection_enabled] updated to [true].
Manifest refresh failed:
  Access to Red Hat Subscription Management is prohibited. If you would like to change this, please update the content setting 'Subscription connection enabled'.
~~~

2) It's easy to believe that 1) is not the only problem since bz 2162425 is not yet implemented and an error is reported.