Bug 1248663 - Mongod fails to start due to bad configuration of pid file
Mongod fails to start due to bad configuration of pid file
Status: CLOSED ERRATA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Installer (Show other bugs)
6.1.0
x86_64 Linux
medium Severity medium (vote)
: 6.1.2
: --
Assigned To: Stephen Benjamin
Sachin Ghai
: Triaged
: 1252974 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-07-30 10:26 EDT by Kameron
Modified: 2017-02-23 14:48 EST (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-09-15 03:22:28 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Kameron 2015-07-30 10:26:19 EDT
Description of problem:
Mongod fails to start due to bad configuration of pid file in init script after upgrade of Satellite Server from 6.0 to 6.1

Version-Release number of selected component (if applicable):
mongodb-2.4.9-3.el7sat.x86_64
python-pymongo-2.5.2-3.el7sat.x86_64
python-pymongo-gridfs-2.5.2-3.el7sat.x86_64
python-mongoengine-0.7.10-2.el7sat.noarch
mongodb-server-2.4.9-3.el7sat.x86_64


How reproducible:
Have only tested on one server.

Steps to Reproduce:
1. katello-installer --upgrade

Actual results:
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: migrate_pulp...
Upgrade Step: migrate_candlepin...
Upgrade Step: migrate_foreman...
Upgrade Step: Running installer...
 /Stage[main]/Mongodb::Server::Service/Service[mongodb]: Failed to call refresh: Could not restart Service[mongodb]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait restart mongod' returned 1: Redirecting to /bin/systemctl restart  mongod.service
 /Stage[main]/Mongodb::Server::Service/Service[mongodb]: Could not restart Service[mongodb]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait restart mongod' returned 1: Redirecting to /bin/systemctl restart  mongod.service
Installing             Debug: Unable to connect to mongodb server (127.0. [73%] [.............................................................................................                                  ]

Expected results:
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: migrate_pulp...
Upgrade Step: migrate_candlepin...
Upgrade Step: migrate_foreman...
Upgrade Step: Running installer...
Installing             Done                                               [100%] [..............................................................................................................................]
  The full log is at /var/log/katello-installer/katello-installer.log


Additional info:
I was able to correct the problem by digging in and recognizing that the pid file in /var/run/mongodb/ was different than what is specified in /etc/systemd/system/multi-user.target.wants/mongod.service after noticing that the service would start, then get signal 15 sent to the process after a timeout.  Further triage included correcting the line in /etc/systemd/system/multi-user.target.wants/mongod.service, then running `katello-installer --upgrade` again would overwrite it back to the way it was before, ultimately breaking it again.  I then went to /usr/share/katello-installer/modules/mongodb/manifests/params.pp to update line 52 from:
"        $pidfilepath             = '/var/run/mongodb/mongod.pid'" to:
"        $pidfilepath             = '/var/run/mongodb/mongodb.pid'".
After this, katello-installer --upgrade correctly generates /etc/systemd/system/multi-user.target.wants/mongod.service and completes successfully.
Comment 4 Stephen Benjamin 2015-08-13 12:55:15 EDT
*** Bug 1252974 has been marked as a duplicate of this bug. ***
Comment 6 Sachin Ghai 2015-09-02 05:00:08 EDT
Verified with sat6.1.2 snap1.


Before upgrade on sat6.0.8, the installed version of mongod package:

[root@cloud-qe-20 ~]# rpm -qa | grep mongo
mongodb-2.4.9-3.el7sat.x86_64
python-pymongo-2.5.2-3.el7sat.x86_64
mongodb-server-2.4.9-3.el7sat.x86_64


[root@cloud-qe-20 ~]# cat /usr/share/katello-installer/modules/mongodb/manifests/params.pp | grep pidfilepath
        $pidfilepath         = '/var/run/mongodb/mongodb.pid'
Comment 7 Sachin Ghai 2015-09-02 05:01:20 EDT
Upgrade started and break it by "ctrl + C" and restarted mongod.

[root@cloud-qe-20 yum.repos.d]# katello-installer --upgrade
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: migrate_pulp...
Upgrade Step: start_httpd...
^C/usr/share/katello-installer/hooks/boot/01-helpers.rb:24:in ``': Interrupt
	from /usr/share/katello-installer/hooks/boot/01-helpers.rb:24:in `block in execute'
	from /usr/share/katello-installer/hooks/boot/01-helpers.rb:23:in `each'
	from /usr/share/katello-installer/hooks/boot/01-helpers.rb:23:in `execute'
	from /usr/share/katello-installer/hooks/pre/30-upgrade.rb:13:in `start_httpd'
	from /usr/share/katello-installer/hooks/pre/30-upgrade.rb:39:in `upgrade_step'
	from /usr/share/katello-installer/hooks/pre/30-upgrade.rb:56:in `block (4 levels) in load'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:32:in `instance_eval'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:32:in `block (4 levels) in load'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hook_context.rb:13:in `instance_exec'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hook_context.rb:13:in `execute'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:49:in `block in execute'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:47:in `each'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:47:in `execute'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:301:in `run_installation'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:109:in `execute'
	from /usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
	from /usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:125:in `run'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:116:in `run'
	from /usr/sbin/katello-installer:46:in `<main>'

[root@cloud-qe-20 yum.repos.d]# service mongod restart
Redirecting to /bin/systemctl restart  mongod.service
Comment 8 Sachin Ghai 2015-09-02 05:01:50 EDT
Re-ran the upgrade and upgrade was completed successfully.

[root@cloud-qe-20 yum.repos.d]# katello-installer --upgrade
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: migrate_pulp...
Upgrade Step: start_httpd...
Upgrade Step: migrate_candlepin...
Upgrade Step: migrate_foreman...
Upgrade Step: Running installer...
Installing             Done                                               [100%] [..................................................................]
  The full log is at /var/log/katello-installer/katello-installer.log
Upgrade Step: restart_services...
Upgrade Step: db_seed...
Upgrade Step: errata_import (this may take a while) ...
Upgrade Step: update_gpg_urls (this may take a while) ...
Upgrade Step: update_repository_metadata (this may take a while) ...
Katello upgrade completed!
Comment 9 Sachin Ghai 2015-09-02 05:03:40 EDT
this file doesn't have pid file change under "migrate_pulp" section:

/usr/share/katello-installer/hooks/pre/30-upgrade.rb

<snippet>
def migrate_pulp
  # Start mongo if not running
  unless Kafo::Helpers.execute('pgrep mongod')
    Kafo::Helpers.execute('service-wait mongod start')
  end

Based on above comments, moving this to verified.
Comment 11 errata-xmlrpc 2015-09-15 03:22:28 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:1786

Note You need to log in before you can comment on or make changes to this bug.