Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1622650

Summary: [RFE] foreman-maintain should check that Satellite Server is running, as first step in foreman-maintain upgrade check
Product: Red Hat Satellite Reporter: anerurka
Component: Satellite MaintainAssignee: Martin Bacovsky <mbacovsk>
Status: CLOSED DUPLICATE QA Contact: Katello QA List <katello-qa-list>
Severity: medium Docs Contact:
Priority: high    
Version: 6.4CC: apatel, bkearney, inecas, kgaikwad, mbacovsk, peter.vreman
Target Milestone: UnspecifiedKeywords: FutureFeature, Triaged
Target Release: Unused   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-11-08 21:53:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1122832, 1619394    

Description anerurka 2018-08-27 16:38:17 UTC
Description of problem:

[RFE] foreman-maintain should check that Satellite Server is running, as first step in foreman-maintain upgrade check

Version-Release number of selected component (if applicable):

satellite-installer-6.4.0.7-1.beta.el7sat.noarch
rubygem-foreman_maintain-0.2.6-1.el7sat.noarch

How reproducible: Always


Steps to Reproduce:

1. If the satellite server is down (or only httpd service is down)

2. Execute upgrade check using foreman-maintain 

# foreman-maintain upgrade check --target-version 6.4


Actual results:

- Foreman maintain fails trying to set-up hammer stating admin password is in-correct

Running preparation steps required to run the next scenarios
================================================================================
Setup hammer:
Hammer admin password:
                                                                      [FAIL]
Hammer configuration failed: Is the admin password correct? (it was stored in /etc/foreman-maintain/foreman-maintain-hammer.yml)Is the server down?
--------------------------------------------------------------------------------
Scenario [preparation steps required to run the next scenarios] failed.

The following steps ended up in failing state:

  [hammer-setup]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="hammer-setup"


Expected results:

Adding an additional first step to do the hammer ping (or alternative to start all services) would fix this issue and make the user aware that sat6 must be started.

Additional info:

>>> After testing few thing locally it seems that foreman-maintain does check for all applicable services status over Satellite Server .

- But adding this check as a first step would assist users in taking corrective actions.

If the httpd service is down over Satellite Server (which can happen if httpd service is stopped), connection over port 443 over Satellite Server is blocked. We can see the same in "hammer ping" output.

+++++
# hammer ping
Error: Failed to open TCP connection to satellite6.example.com:443 (Connection refused - connect(2) for "satellite6.example.com" port 443)
+++++

- It seems "Setup hammer" is performed if check_connection for hammer fails.

Comment 1 anerurka 2018-08-27 16:41:59 UTC
Additionally :

- If Satellite Server is up and any other katello service is down then foreman-maintain , tries to restart service.

# for i in pulp_resource_manager pulp_workers pulp_celerybeat; do service $i stop; done

Redirecting to /bin/systemctl stop pulp_resource_manager.service
Redirecting to /bin/systemctl stop pulp_workers.service
Redirecting to /bin/systemctl stop pulp_celerybeat.service

# systemctl start httpd

# hammer ping
candlepin:      
    Status:          ok
    Server Response: Duration: 320ms
candlepin_auth: 
    Status:          ok
    Server Response: Duration: 36ms
pulp:           
    Status:          FAIL
    Server Response:
pulp_auth:      
    Status: FAIL
foreman_tasks:  
    Status:          ok
    Server Response: Duration: 205ms


# foreman-maintain upgrade check --target-version 6.4

Running Checks before upgrading to Satellite 6.4
================================================================================
Check for verifying syntax for ISP DHCP configurations:               [SKIPPED]
DHCP feature is not enabled
--------------------------------------------------------------------------------
Check for paused tasks:                                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running using hammer ping:             [FAIL]
pulp, pulp_auth resource(s) are failing.                                                
--------------------------------------------------------------------------------
Continue with step [Restart applicable services]?, [y(yes), n(no), q(quit)]         <<<<<<<< Attempt to restart services in-case katello services are down

- From above we can see that foreman maintain indeed tries to start the katello services which are down. 

- However in-case of check_connection fails (i.e hammer ping fails for connection) then it  tries to set-up hammer and ask for admin password.

- - Now while setting up it asks for password for hammer admin user, which is not an existing password, but it tries to set-up a new hammer configuration.

- But the error is confusing as the output on the console primary hints to the password being incorrect. And only somewhere at the end 3 words with 'is server down?'.

- adding this check as a first step would assist users in taking corrective actions.

Comment 2 Satellite Program 2018-09-21 00:03:20 UTC
Upstream bug assigned to mbacovsk

Comment 3 Bryan Kearney 2018-11-08 21:53:43 UTC

*** This bug has been marked as a duplicate of bug 1625630 ***