Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1625169

Summary: foreman-maintain service restart failed with undefined method `[]' for nil:NilClass
Product: Red Hat Satellite Reporter: Nikhil Kathole <nkathole>
Component: Satellite MaintainAssignee: Martin Bacovsky <mbacovsk>
Status: CLOSED ERRATA QA Contact: Nikhil Kathole <nkathole>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.4CC: apatel, inecas, kgaikwad, mbacovsk, pcreech
Target Milestone: 6.4.0Keywords: Regression, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rubygem-foreman_maintain-0.2.10-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-16 15:16:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nikhil Kathole 2018-09-04 10:12:42 UTC
Description of problem: 
foreman-maintain service restart failed with undefined method `[]' for nil:NilClass on both satellite and capsule.

Version-Release number of selected component (if applicable):
# rpm -qa | grep maintain
rubygem-foreman_maintain-0.2.6-1.el7sat.noarch


How reproducible:always


Steps to Reproduce:
1. # foreman-maintain service restart
Running preparation steps required to run the next scenarios
================================================================================
Setup hammer:                                                         [FAIL]
undefined method `[]' for nil:NilClass
--------------------------------------------------------------------------------
Scenario [preparation steps required to run the next scenarios] failed.

The following steps ended up in failing state:

  [hammer-setup]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="hammer-setup"


Actual results: foreman-maintain service command failed

and log showed

I, [2018-09-04 06:11:58-0400 #119973]  INFO -- : Admin password was not found in hammer configs. Looking into installer answers
E, [2018-09-04 06:11:58-0400 #119973] ERROR -- : undefined method `[]' for nil:NilClass (NoMethodError)
/usr/share/gems/gems/foreman_maintain-0.2.6/definitions/features/hammer.rb:195:in `password_from_answers'
/usr/share/gems/gems/foreman_maintain-0.2.6/definitions/features/hammer.rb:128:in `on_missing_password'
/usr/share/gems/gems/foreman_maintain-0.2.6/definitions/features/hammer.rb:41:in `setup_admin_access'
/usr/share/gems/gems/foreman_maintain-0.2.6/definitions/procedures/hammer_setup.rb:8:in `run'
/usr/share/gems/gems/foreman_maintain-0.2.6/lib/foreman_maintain/executable.rb:139:in `__run__'
/usr/share/gems/gems/foreman_maintain-0.2.6/lib/foreman_maintain/runner/execution.rb:83:in `block (2 levels) in run'
/usr/share/gems/gems/foreman_maintain-0.2.6/lib/foreman_maintain/runner/execution.rb:107:in `capture_errors'
/usr/share/gems/gems/foreman_maintain-0.2.6/lib/foreman_maintain/runner/execution.rb:82:in `block in run'
/usr/share/gems/gems/foreman_maintain-0.2.6/lib/foreman_maintain/runner/execution.rb:101:in `with_metadata_calculation'
/usr/share/gems/gems/foreman_maintain-0.2.6/lib/foreman_maintain/runner/execution.rb:81:in `run'
/usr/share/gems/gems/foreman_maintain-0.2.6/lib/foreman_maintain/runner.rb:116:in `run_step'
/usr/share/gems/gems/foreman_maintain-0.2.6/lib/foreman_maintain/runner.rb:105:in `run_steps'
/usr/share/gems/gems/foreman_maintain-0.2.6/lib/foreman_maintain/runner.rb:98:in `execute_scenario_steps'
/usr/share/gems/gems/foreman_maintain-0.2.6/lib/foreman_maintain/runner.rb:51:in `run_scenario'
/usr/share/gems/gems/foreman_maintain-0.2.6/lib/foreman_maintain/runner.rb:94:in `block in execute_scenario_steps'
/usr/share/gems/gems/foreman_maintain-0.2.6/lib/foreman_maintain/runner.rb:94:in `each'
/usr/share/gems/gems/foreman_maintain-0.2.6/lib/foreman_maintain/runner.rb:94:in `execute_scenario_steps'
/usr/share/gems/gems/foreman_maintain-0.2.6/lib/foreman_maintain/runner.rb:51:in `run_scenario'
/usr/share/gems/gems/foreman_maintain-0.2.6/lib/foreman_maintain/runner.rb:32:in `block in run'
/usr/share/gems/gems/foreman_maintain-0.2.6/lib/foreman_maintain/runner.rb:31:in `each'
/usr/share/gems/gems/foreman_maintain-0.2.6/lib/foreman_maintain/runner.rb:31:in `run'
/usr/share/gems/gems/foreman_maintain-0.2.6/lib/foreman_maintain/cli/base.rb:47:in `run_scenario'
/usr/share/gems/gems/foreman_maintain-0.2.6/lib/foreman_maintain/cli/service_command.rb:48:in `execute'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in `execute'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in `execute'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
/usr/share/gems/gems/foreman_maintain-0.2.6/lib/foreman_maintain/cli.rb:27:in `run'
/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:132:in `run'
/usr/share/gems/gems/foreman_maintain-0.2.6/bin/foreman-maintain:12:in `<top (required)>'
/usr/bin/foreman-maintain:23:in `load'
/usr/bin/foreman-maintain:23:in `<main>'


Expected results: Services should restart.


Additional info:

Comment 3 Nikhil Kathole 2018-09-04 10:33:06 UTC
Related to only capsule (on satellite works fine)

Comment 4 Martin Bacovsky 2018-09-04 10:46:49 UTC
Thanks for the update Nikhil. I'm linking this with upstream issue adding more details and suggests possible solution.

Comment 5 Ivan Necas 2018-09-14 16:40:35 UTC
I believe this issue will be fixed as part of https://github.com/theforeman/foreman_maintain/pull/210/files

Comment 6 Martin Bacovsky 2018-09-18 22:59:18 UTC
This issue is primarily caused by hammer unable to check the connection due to httpd missing or being down. The PR #210  fixed it for capsule but on the server it is still present. It is possible to reproduce with following steps:
- rm /etc/foreman-maintain/foreman-maintain-hammer.yaml to make sure hammer was not setup
- systemctl stop httpd 
- foreman-maintain health check

Comment 7 Ivan Necas 2018-09-20 18:41:08 UTC
*** Bug 1626651 has been marked as a duplicate of this bug. ***

Comment 8 Martin Bacovsky 2018-09-20 23:22:13 UTC
Please ignore comment #6 where I stated wrong information about this issue. The real cause is that foreman-maintain is trying to fetch admin password from installer answers on a machine where no Foreman is installed and the options are missing in the answers file. The issue was fixed by PR linked in c#5

What was stated in c#6 pays for BZ#1626651 and is not related to this BZ.

Sorry for confusion. Links and statuses between upstream tracker, Github and BZ were corrected.

Comment 10 Patrick Creech 2018-09-24 14:49:07 UTC
snap 23, not 63

Comment 11 Nikhil Kathole 2018-09-25 09:52:25 UTC
VERIFIED

Version tested:
# rpm -qa | grep maintain
rubygem-foreman_maintain-0.2.10-1.el7sat.noarch

# foreman-maintain service restart
Running Restart Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Restart applicable services: 
Stopping the following service(s):

mongod, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, goferd, httpd, foreman-proxy
| stopping foreman-proxy                                                        
Redirecting to /bin/systemctl stop foreman-proxy.service
| stopping httpd                                                                
Redirecting to /bin/systemctl stop httpd.service
/ stopping goferd                                                               
Redirecting to /bin/systemctl stop goferd.service
/ stopping pulp_workers                                                         
Redirecting to /bin/systemctl stop pulp_workers.service
/ stopping pulp_streamer                                                        
Redirecting to /bin/systemctl stop pulp_streamer.service
- stopping pulp_resource_manager                                                
Redirecting to /bin/systemctl stop pulp_resource_manager.service
- stopping pulp_celerybeat                                                      
Redirecting to /bin/systemctl stop pulp_celerybeat.service
\ stopping squid                                                                
Redirecting to /bin/systemctl stop squid.service
\ stopping qpidd                                                                
Redirecting to /bin/systemctl stop qpidd.service
| stopping qdrouterd                                                            
Redirecting to /bin/systemctl stop qdrouterd.service
| stopping mongod                                                               
Redirecting to /bin/systemctl stop mongod.service
| All services stopped                                                          
Starting the following service(s):

mongod, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, goferd, httpd, foreman-proxy
| starting mongod                                                               
Redirecting to /bin/systemctl start mongod.service
/ starting qdrouterd                                                            
Redirecting to /bin/systemctl start qdrouterd.service
/ starting qpidd                                                                
Redirecting to /bin/systemctl start qpidd.service
- starting squid                                                                
Redirecting to /bin/systemctl start squid.service
- starting pulp_celerybeat                                                      
Redirecting to /bin/systemctl start pulp_celerybeat.service
\ starting pulp_resource_manager                                                
Redirecting to /bin/systemctl start pulp_resource_manager.service
\ starting pulp_streamer                                                        
Redirecting to /bin/systemctl start pulp_streamer.service
- starting pulp_workers                                                         
Redirecting to /bin/systemctl start pulp_workers.service
\ starting goferd                                                               
Redirecting to /bin/systemctl start goferd.service
\ starting httpd                                                                
Redirecting to /bin/systemctl start httpd.service
| starting foreman-proxy                                                        
Redirecting to /bin/systemctl start foreman-proxy.service
/ All services started                                                [OK]      
--------------------------------------------------------------------------------

Comment 13 errata-xmlrpc 2018-10-16 15:16:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2926