Bug 1248663

Summary: Mongod fails to start due to bad configuration of pid file
Product: Red Hat Satellite Reporter: Kameron <kameron.kenny>
Component: InstallationAssignee: Stephen Benjamin <stbenjam>
Status: CLOSED ERRATA QA Contact: Sachin Ghai <sghai>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.1.0CC: bbuckingham, bkearney, byount, cwelton, egolov, kameron.kenny, mdavis, mmccune, sghai, stbenjam
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-09-15 07:22:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kameron 2015-07-30 14:26:19 UTC
Description of problem:
Mongod fails to start due to bad configuration of pid file in init script after upgrade of Satellite Server from 6.0 to 6.1

Version-Release number of selected component (if applicable):
mongodb-2.4.9-3.el7sat.x86_64
python-pymongo-2.5.2-3.el7sat.x86_64
python-pymongo-gridfs-2.5.2-3.el7sat.x86_64
python-mongoengine-0.7.10-2.el7sat.noarch
mongodb-server-2.4.9-3.el7sat.x86_64


How reproducible:
Have only tested on one server.

Steps to Reproduce:
1. katello-installer --upgrade

Actual results:
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: migrate_pulp...
Upgrade Step: migrate_candlepin...
Upgrade Step: migrate_foreman...
Upgrade Step: Running installer...
 /Stage[main]/Mongodb::Server::Service/Service[mongodb]: Failed to call refresh: Could not restart Service[mongodb]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait restart mongod' returned 1: Redirecting to /bin/systemctl restart  mongod.service
 /Stage[main]/Mongodb::Server::Service/Service[mongodb]: Could not restart Service[mongodb]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait restart mongod' returned 1: Redirecting to /bin/systemctl restart  mongod.service
Installing             Debug: Unable to connect to mongodb server (127.0. [73%] [.............................................................................................                                  ]

Expected results:
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: migrate_pulp...
Upgrade Step: migrate_candlepin...
Upgrade Step: migrate_foreman...
Upgrade Step: Running installer...
Installing             Done                                               [100%] [..............................................................................................................................]
  The full log is at /var/log/katello-installer/katello-installer.log


Additional info:
I was able to correct the problem by digging in and recognizing that the pid file in /var/run/mongodb/ was different than what is specified in /etc/systemd/system/multi-user.target.wants/mongod.service after noticing that the service would start, then get signal 15 sent to the process after a timeout.  Further triage included correcting the line in /etc/systemd/system/multi-user.target.wants/mongod.service, then running `katello-installer --upgrade` again would overwrite it back to the way it was before, ultimately breaking it again.  I then went to /usr/share/katello-installer/modules/mongodb/manifests/params.pp to update line 52 from:
"        $pidfilepath             = '/var/run/mongodb/mongod.pid'" to:
"        $pidfilepath             = '/var/run/mongodb/mongodb.pid'".
After this, katello-installer --upgrade correctly generates /etc/systemd/system/multi-user.target.wants/mongod.service and completes successfully.

Comment 4 Stephen Benjamin 2015-08-13 16:55:15 UTC
*** Bug 1252974 has been marked as a duplicate of this bug. ***

Comment 6 Sachin Ghai 2015-09-02 09:00:08 UTC
Verified with sat6.1.2 snap1.


Before upgrade on sat6.0.8, the installed version of mongod package:

[root@cloud-qe-20 ~]# rpm -qa | grep mongo
mongodb-2.4.9-3.el7sat.x86_64
python-pymongo-2.5.2-3.el7sat.x86_64
mongodb-server-2.4.9-3.el7sat.x86_64


[root@cloud-qe-20 ~]# cat /usr/share/katello-installer/modules/mongodb/manifests/params.pp | grep pidfilepath
        $pidfilepath         = '/var/run/mongodb/mongodb.pid'

Comment 7 Sachin Ghai 2015-09-02 09:01:20 UTC
Upgrade started and break it by "ctrl + C" and restarted mongod.

[root@cloud-qe-20 yum.repos.d]# katello-installer --upgrade
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: migrate_pulp...
Upgrade Step: start_httpd...
^C/usr/share/katello-installer/hooks/boot/01-helpers.rb:24:in ``': Interrupt
	from /usr/share/katello-installer/hooks/boot/01-helpers.rb:24:in `block in execute'
	from /usr/share/katello-installer/hooks/boot/01-helpers.rb:23:in `each'
	from /usr/share/katello-installer/hooks/boot/01-helpers.rb:23:in `execute'
	from /usr/share/katello-installer/hooks/pre/30-upgrade.rb:13:in `start_httpd'
	from /usr/share/katello-installer/hooks/pre/30-upgrade.rb:39:in `upgrade_step'
	from /usr/share/katello-installer/hooks/pre/30-upgrade.rb:56:in `block (4 levels) in load'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:32:in `instance_eval'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:32:in `block (4 levels) in load'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hook_context.rb:13:in `instance_exec'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hook_context.rb:13:in `execute'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:49:in `block in execute'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:47:in `each'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:47:in `execute'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:301:in `run_installation'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:109:in `execute'
	from /usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
	from /usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:125:in `run'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:116:in `run'
	from /usr/sbin/katello-installer:46:in `<main>'

[root@cloud-qe-20 yum.repos.d]# service mongod restart
Redirecting to /bin/systemctl restart  mongod.service

Comment 8 Sachin Ghai 2015-09-02 09:01:50 UTC
Re-ran the upgrade and upgrade was completed successfully.

[root@cloud-qe-20 yum.repos.d]# katello-installer --upgrade
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: migrate_pulp...
Upgrade Step: start_httpd...
Upgrade Step: migrate_candlepin...
Upgrade Step: migrate_foreman...
Upgrade Step: Running installer...
Installing             Done                                               [100%] [..................................................................]
  The full log is at /var/log/katello-installer/katello-installer.log
Upgrade Step: restart_services...
Upgrade Step: db_seed...
Upgrade Step: errata_import (this may take a while) ...
Upgrade Step: update_gpg_urls (this may take a while) ...
Upgrade Step: update_repository_metadata (this may take a while) ...
Katello upgrade completed!

Comment 9 Sachin Ghai 2015-09-02 09:03:40 UTC
this file doesn't have pid file change under "migrate_pulp" section:

/usr/share/katello-installer/hooks/pre/30-upgrade.rb

<snippet>
def migrate_pulp
  # Start mongo if not running
  unless Kafo::Helpers.execute('pgrep mongod')
    Kafo::Helpers.execute('service-wait mongod start')
  end

Based on above comments, moving this to verified.

Comment 11 errata-xmlrpc 2015-09-15 07:22:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:1786