If the user has specified invalid values in .hammer/defaults.yml (via the hammer defaults command) foreman-maintain will fail to authenticate and error on most commands. This can be difficult as a user to determine why it is failing as the user only sees: # foreman-maintain health check Running preparation steps required to run the next scenarios ================================================================================ Setup hammer: Configuring Hammer CLI... Hammer admin password: [FAIL] Hammer configuration failed: Is the admin password correct? (it was stored in /etc/foreman-maintain/foreman-maintain-hammer.yml)Is the server down? this makes it look like the password is incorrect or the server is down, when in fact, there may be a setting in ~./.hammer/defauts.yml that is invalid or conflicting with foreman-maintain's usage of hammer. This results in failed upgrades with difficulty determining why: # satellite-installer --upgrade ... Upgrade Step: Running installer... Installing Done [100%] [...........................................................................] The full log is at /var/log/foreman-installer/satellite.log Upgrade Step: restart_services... Redirecting to 'foreman-maintain service' Running preparation steps required to run the next scenarios ============================================================================= Setup hammer: Configuring Hammer CLI... Hammer admin password: [FAIL]
Created redmine issue https://projects.theforeman.org/issues/25032 from this bug
Checking the reproducer machine I found the cause of the failure is the non-existing default org setup in hammer: # hammer defaults list ----------------|------------------- PARAMETER | VALUE ----------------|------------------- organization_id | asdjfiou902354u901 ----------------|------------------- We run the following to test hammer connection from foreman-maintain: # RUBYOPT='-W0' LANG=en_US.utf-8 hammer -c "/etc/foreman-maintain/foreman-maintain-hammer.yml" --interactive=no architecture list Organization with id asdjfiou902354u901 not found To fix the problem in this case would be to remove the non-existing default org. To avoid this in the future we should: - setup hammer only for unauthorized requests before using hammer-ping - improve what is logged and printed for easier debugging
yeah, just need better error messaging to make this clearer. I think that would solve this.
NOTE: We may also be seeing this condition even when there are valid values in the defaults.yml file. If this is the case, the severity of this bug may increase and necessitate the inclusion in a 6.4 release.
(In reply to Martin Bacovsky from comment #2) > Checking the reproducer machine I found the cause of the failure is > the non-existing default org setup in hammer: > > # hammer defaults list > ----------------|------------------- > PARAMETER | VALUE > ----------------|------------------- > organization_id | asdjfiou902354u901 > ----------------|------------------- > > We run the following to test hammer connection from foreman-maintain: > > # RUBYOPT='-W0' LANG=en_US.utf-8 hammer -c > "/etc/foreman-maintain/foreman-maintain-hammer.yml" --interactive=no > architecture list > Organization with id asdjfiou902354u901 not found > > To fix the problem in this case would be to remove the non-existing default > org. > > To avoid this in the future we should: > - setup hammer only for unauthorized requests before using hammer-ping > - improve what is logged and printed for easier debugging I just encountered the same issue with an existing organization: $ foreman-maintain service restart Running Restart Services ================================================================================ Check if command is run as root user: [OK] -------------------------------------------------------------------------------- Restart applicable services: Stopping the following service(s): rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, httpd, puppetserver, foreman-proxy - All services stopped Starting the following service(s): rh-mongodb34-mongod, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflowd, httpd, puppetserver, foreman-proxy / All services started [OK] -------------------------------------------------------------------------------- $ hammer organization list ---|----------------------|----------------------|-------------|----------------------|------------ ID | TITLE | NAME | DESCRIPTION | LABEL | DESCRIPTION ---|----------------------|----------------------|-------------|----------------------|------------ 1 | Default Organization | Default Organization | | Default_Organization | ---|----------------------|----------------------|-------------|----------------------|------------ $ hammer defaults add --param-name organization_id --param-value 1 Added organization_id default-option with value 1. $ foreman-maintain service restart Running preparation steps required to run the next scenarios ================================================================================ Setup hammer: Configuring Hammer CLI... Hammer admin password: [FAIL] Hammer configuration failed: Is the admin password correct? (it was stored in /etc/foreman-maintain/foreman-maintain-hammer.yml)Is the server down? -------------------------------------------------------------------------------- Scenario [preparation steps required to run the next scenarios] failed. The following steps ended up in failing state: [hammer-setup] Resolve the failed steps and rerun the command. In case the failures are false positives, use --whitelist="hammer-setup" Running manually the connection-checking command mentioned in the comment #2 gives the following output: $ RUBYOPT='-W0' LANG=en_US.utf-8 hammer -c "/etc/foreman-maintain/foreman-maintain-hammer.yml" --interactive=no architecture list Association not found for organization Removing the default org and re-running the "architecture list" command works: $ hammer defaults delete --param-name organization_id organization_id was deleted successfully. $ RUBYOPT='-W0' LANG=en_US.utf-8 hammer -c "/etc/foreman-maintain/foreman-maintain-hammer.yml" --interactive=no architecture list ---|------- ID | NAME ---|------- 1 | x86_64 2 | i386 ---|------- Nevertheless, using the --organization-id option with the "architecture list" still fails: $ RUBYOPT='-W0' LANG=en_US.utf-8 hammer -c "/etc/foreman-maintain/foreman-maintain-hammer.yml" --interactive=no architecture list --organization-id 1 Association not found for organization When checking Hosts->Architectures tab in the WebUI with the Default Organization selected, architectures are listed correctly. Maybe the actual problem lays with hammer not getting correct architecture list when an organization is specified/enabled as default?
*** Bug 1640230 has been marked as a duplicate of this bug. ***
*** Bug 1640462 has been marked as a duplicate of this bug. ***
As pointed out by Radovan (and my dupe in #1640462), this also happens with *valid* defaults when the architectures are not assigned to the Org that is in the defaults.
Upstream bug assigned to apatel
based on the recent discussion, suggesting for 6.4.z
I'm renaming subject of this BZ to focus on the case where the organanization is not assigned to architecture. I will create new BZ to track the non-existing organization_id, as the fix would be probablye a bit more complex, and I don't want the fix for the valid organization_id to be blocked by the ultimate solution
The causing API issue is tracked here: https://bugzilla.redhat.com/show_bug.cgi?id=1640617
This bug was cloned and is still going to be included in the 6.4.3 release. It no longer has the sat-6.4.z+ flag and 6.4.3 Target Milestone Set which are now on the 6.4.z cloned bug. Please see the Clones field to track the progress of this bug in the 6.4.3 release.
verified @satellite 6.4.3 snap 1 @rubygem-foreman_maintain-0.2.12-1.el7sat.noarch steps: # hammer defaults add --param-name organization_id --param-value 1 Added organization_id default-option with value 1. # hammer defaults list ----------------|------ PARAMETER | VALUE ----------------|------ organization_id | 1 ----------------|------ # foreman-maintain health check Running preparation steps required to run the next scenarios ================================================================================ Setup hammer: Configuring Hammer CLI... Hammer admin password: [OK] -------------------------------------------------------------------------------- Running ForemanMaintain::Scenario::FilteredScenario ================================================================================ Check for verifying syntax for ISP DHCP configurations: [SKIPPED] DHCP feature is not enabled -------------------------------------------------------------------------------- Check whether all services are running: [OK] -------------------------------------------------------------------------------- Check whether all services are running using hammer ping: [OK] -------------------------------------------------------------------------------- Check for paused tasks: [OK] -------------------------------------------------------------------------------- observation: - Foreman-maintain is able to authenticate when valid default organization is setup in hammer. - Foreman-maintain is not able to authenticate when invalid default organization is setup in hammer, there is separate bugzilla filed for this issue.[1] - architecture list still fails when organization_id is provided, there is separate bugzilla filed for this issue as well.[2] [1] https://bugzilla.redhat.com/show_bug.cgi?id=1640625 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1649011
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0904