Bug 1632768 - foreman-maintain can't authenticate when 'hammer defaults' is pointing to organization that has no architecture assigned: upgrades and other commands fail
Summary: foreman-maintain can't authenticate when 'hammer defaults' is pointing to org...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Foreman Maintain
Version: 6.4
Hardware: x86_64
OS: Linux
high
high
Target Milestone: 6.4.3
Assignee: Anurag Patel
QA Contact: Jameer Pathan
URL:
Whiteboard:
: 1640230 1640462 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-09-25 13:57 UTC by Mike McCune
Modified: 2019-10-30 17:14 UTC (History)
21 users (show)

Fixed In Version: rubygem-foreman_maintain-0.2.12-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-04-29 18:12:24 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Foreman Issue Tracker 25032 Normal Closed foreman-maintain can't authenticate 'hammer defaults' are invalid - upgrades and other commands fail 2020-10-21 11:55:04 UTC
Red Hat Bugzilla 1640468 high CLOSED Add warning in upgrade section regarding /root/.hammer/defaults.yml 2020-10-14 00:28:05 UTC
Red Hat Bugzilla 1640625 unspecified CLOSED Invalid default of organization causes hammer not to be set up corretly, blocking upgrades and other foreman-maintain wo... 2020-10-14 00:28:05 UTC
Red Hat Knowledge Base (Solution) 3659041 None None None 2018-10-18 06:13:32 UTC
Red Hat Product Errata RHBA-2019:0904 None None None 2019-04-29 18:12:26 UTC

Internal Links: 1640468 1640625

Description Mike McCune 2018-09-25 13:57:52 UTC
If the user has specified invalid values in .hammer/defaults.yml (via the hammer defaults command) foreman-maintain will fail to authenticate and error on most commands.

This can be difficult as a user to determine why it is failing as the user only sees:

# foreman-maintain  health check
Running preparation steps required to run the next scenarios
================================================================================
Setup hammer: 
Configuring Hammer CLI...
Hammer admin password: 
                                                                      [FAIL]
Hammer configuration failed: Is the admin password correct? (it was stored in /etc/foreman-maintain/foreman-maintain-hammer.yml)Is the server down?

this makes it look like the password is incorrect or the server is down, when in fact, there may be a setting in ~./.hammer/defauts.yml that is invalid or conflicting with foreman-maintain's usage of hammer.

This results in failed upgrades with difficulty determining why:

# satellite-installer --upgrade
...
Upgrade Step: Running installer...
Installing             Done                                               [100%] [...........................................................................]
  The full log is at /var/log/foreman-installer/satellite.log
Upgrade Step: restart_services...
Redirecting to 'foreman-maintain service'
Running preparation steps required to run the next scenarios
=============================================================================
Setup hammer: Configuring Hammer CLI...
Hammer admin password: 


                                                                      [FAIL]

Comment 1 Martin Bacovsky 2018-09-25 14:21:52 UTC
Created redmine issue https://projects.theforeman.org/issues/25032 from this bug

Comment 2 Martin Bacovsky 2018-09-25 14:22:36 UTC
Checking the reproducer machine I found the cause of the failure is
the non-existing default org setup in hammer:

# hammer defaults list
----------------|-------------------
PARAMETER       | VALUE             
----------------|-------------------
organization_id | asdjfiou902354u901
----------------|-------------------

We run the following to test hammer connection from foreman-maintain:

# RUBYOPT='-W0' LANG=en_US.utf-8 hammer -c "/etc/foreman-maintain/foreman-maintain-hammer.yml" --interactive=no architecture list
Organization with id asdjfiou902354u901 not found

To fix the problem in this case would be to remove the non-existing default org.

To avoid this in the future we should:
- setup hammer only for unauthorized requests before using hammer-ping
- improve what is logged and printed for easier debugging

Comment 4 Mike McCune 2018-09-25 17:03:44 UTC
yeah, just need better error messaging to make this clearer. I think that would solve this.

Comment 5 Mike McCune 2018-09-25 18:22:38 UTC
NOTE: We may also be seeing this condition even when there are valid values in the defaults.yml file. If this is the case, the severity of this bug may increase and necessitate the inclusion in a 6.4 release.

Comment 6 Radovan Drazny 2018-10-04 12:23:34 UTC
(In reply to Martin Bacovsky from comment #2)
> Checking the reproducer machine I found the cause of the failure is
> the non-existing default org setup in hammer:
> 
> # hammer defaults list
> ----------------|-------------------
> PARAMETER       | VALUE             
> ----------------|-------------------
> organization_id | asdjfiou902354u901
> ----------------|-------------------
> 
> We run the following to test hammer connection from foreman-maintain:
> 
> # RUBYOPT='-W0' LANG=en_US.utf-8 hammer -c
> "/etc/foreman-maintain/foreman-maintain-hammer.yml" --interactive=no
> architecture list
> Organization with id asdjfiou902354u901 not found
> 
> To fix the problem in this case would be to remove the non-existing default
> org.
> 
> To avoid this in the future we should:
> - setup hammer only for unauthorized requests before using hammer-ping
> - improve what is logged and printed for easier debugging

I just encountered the same issue with an existing organization:

$ foreman-maintain service restart
Running Restart Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Restart applicable services:
Stopping the following service(s):

rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, httpd, puppetserver, foreman-proxy
- All services stopped
Starting the following service(s):

rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, httpd, puppetserver, foreman-proxy
/ All services started                                                [OK]      
--------------------------------------------------------------------------------

$ hammer organization list
---|----------------------|----------------------|-------------|----------------------|------------
ID | TITLE                | NAME                 | DESCRIPTION | LABEL                | DESCRIPTION
---|----------------------|----------------------|-------------|----------------------|------------
1  | Default Organization | Default Organization |             | Default_Organization |            
---|----------------------|----------------------|-------------|----------------------|------------

$ hammer defaults add --param-name organization_id --param-value 1
Added organization_id default-option with value 1.

$ foreman-maintain service restart
Running preparation steps required to run the next scenarios
================================================================================
Setup hammer: 
Configuring Hammer CLI...
Hammer admin password: 
                                                                      [FAIL]
Hammer configuration failed: Is the admin password correct? (it was stored in /etc/foreman-maintain/foreman-maintain-hammer.yml)Is the server down?
--------------------------------------------------------------------------------
Scenario [preparation steps required to run the next scenarios] failed.

The following steps ended up in failing state:

  [hammer-setup]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="hammer-setup"

Running manually the connection-checking command mentioned in the comment #2 gives the following output:

$ RUBYOPT='-W0' LANG=en_US.utf-8 hammer -c "/etc/foreman-maintain/foreman-maintain-hammer.yml" --interactive=no architecture list
Association not found for organization


Removing the default org and re-running the "architecture list" command works:

$ hammer defaults delete --param-name organization_id
organization_id was deleted successfully.

$ RUBYOPT='-W0' LANG=en_US.utf-8 hammer -c "/etc/foreman-maintain/foreman-maintain-hammer.yml" --interactive=no architecture list
---|-------
ID | NAME  
---|-------
1  | x86_64
2  | i386  
---|-------

Nevertheless, using the --organization-id option with the "architecture list" still fails:

$ RUBYOPT='-W0' LANG=en_US.utf-8 hammer -c "/etc/foreman-maintain/foreman-maintain-hammer.yml" --interactive=no architecture list --organization-id 1
Association not found for organization

When checking Hosts->Architectures tab in the WebUI with the Default Organization selected, architectures are listed correctly.

Maybe the actual problem lays with hammer not getting correct architecture list when an organization is specified/enabled as default?

Comment 7 Simon Reber 2018-10-18 06:15:32 UTC
*** Bug 1640230 has been marked as a duplicate of this bug. ***

Comment 9 Evgeni Golov 2018-10-18 07:09:15 UTC
*** Bug 1640462 has been marked as a duplicate of this bug. ***

Comment 10 Evgeni Golov 2018-10-18 07:12:09 UTC
As pointed out by Radovan (and my dupe in #1640462), this also happens with *valid* defaults when the architectures are not assigned to the Org that is in the defaults.

Comment 14 pm-sat@redhat.com 2018-10-18 10:04:38 UTC
Upstream bug assigned to apatel@redhat.com

Comment 15 Ivan Necas 2018-10-18 12:23:10 UTC
based on the recent discussion, suggesting for 6.4.z

Comment 16 Ivan Necas 2018-10-18 12:31:43 UTC
I'm renaming subject of this BZ to focus on the case where the organanization is not assigned to architecture. I will create new BZ to track the non-existing organization_id, as the fix would be probablye a bit more complex, and I don't want the fix for the valid organization_id to be blocked by the ultimate solution

Comment 18 Martin Bacovsky 2018-10-18 13:07:24 UTC
The causing API issue is tracked here: https://bugzilla.redhat.com/show_bug.cgi?id=1640617

Comment 19 Mike McCune 2019-03-01 22:16:34 UTC
This bug was cloned and is still going to be included in the 6.4.3 release. It no longer has the sat-6.4.z+ flag and 6.4.3 Target Milestone Set which are now on the 6.4.z cloned bug. Please see the Clones field to track the progress of this bug in the 6.4.3 release.

Comment 20 Jameer Pathan 2019-04-11 13:36:51 UTC
verified

@satellite 6.4.3 snap 1
@rubygem-foreman_maintain-0.2.12-1.el7sat.noarch

steps:
# hammer defaults add --param-name organization_id --param-value 1
Added organization_id default-option with value 1.
# hammer defaults list
----------------|------
PARAMETER       | VALUE
----------------|------
organization_id | 1    
----------------|------


# foreman-maintain health check
Running preparation steps required to run the next scenarios
================================================================================
Setup hammer: 
Configuring Hammer CLI...
Hammer admin password: 
                                                                      [OK]
--------------------------------------------------------------------------------


Running ForemanMaintain::Scenario::FilteredScenario
================================================================================
Check for verifying syntax for ISP DHCP configurations:               [SKIPPED]
DHCP feature is not enabled
--------------------------------------------------------------------------------
Check whether all services are running:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running using hammer ping:             [OK]
--------------------------------------------------------------------------------
Check for paused tasks:                                               [OK]
--------------------------------------------------------------------------------


observation:
- Foreman-maintain is able to authenticate 
when valid default organization is setup in hammer.
- Foreman-maintain is not able to authenticate 
when invalid default organization is setup in hammer,
there is separate bugzilla filed for this issue.[1]
- architecture list still fails when organization_id is provided,
there is separate bugzilla filed for this issue as well.[2]

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1640625
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1649011

Comment 22 errata-xmlrpc 2019-04-29 18:12:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0904


Note You need to log in before you can comment on or make changes to this bug.