1248663 – Mongod fails to start due to bad configuration of pid file

Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1248663 - Mongod fails to start due to bad configuration of pid file

Summary: Mongod fails to start due to bad configuration of pid file

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Satellite
Classification:	Red Hat
Component:	Installation
Sub Component:
Version:	6.1.0
Hardware:	x86_64
OS:	Linux
Priority:	medium
Severity:	medium
Target Milestone:	Unspecified
Assignee:	Stephen Benjamin
QA Contact:	Sachin Ghai
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	1252974 (view as bug list)
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2015-07-30 14:26 UTC by Kameron
Modified:	2019-08-15 05:01 UTC (History)
CC List:	10 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2015-09-15 07:22:28 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2015:1786	0	normal	SHIPPED_LIVE	Satellite 6.1.2 bug fix update	2015-09-15 11:20:04 UTC

Description Kameron 2015-07-30 14:26:19 UTC

Description of problem:
Mongod fails to start due to bad configuration of pid file in init script after upgrade of Satellite Server from 6.0 to 6.1

Version-Release number of selected component (if applicable):
mongodb-2.4.9-3.el7sat.x86_64
python-pymongo-2.5.2-3.el7sat.x86_64
python-pymongo-gridfs-2.5.2-3.el7sat.x86_64
python-mongoengine-0.7.10-2.el7sat.noarch
mongodb-server-2.4.9-3.el7sat.x86_64


How reproducible:
Have only tested on one server.

Steps to Reproduce:
1. katello-installer --upgrade

Actual results:
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: migrate_pulp...
Upgrade Step: migrate_candlepin...
Upgrade Step: migrate_foreman...
Upgrade Step: Running installer...
 /Stage[main]/Mongodb::Server::Service/Service[mongodb]: Failed to call refresh: Could not restart Service[mongodb]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait restart mongod' returned 1: Redirecting to /bin/systemctl restart  mongod.service
 /Stage[main]/Mongodb::Server::Service/Service[mongodb]: Could not restart Service[mongodb]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait restart mongod' returned 1: Redirecting to /bin/systemctl restart  mongod.service
Installing             Debug: Unable to connect to mongodb server (127.0. [73%] [.............................................................................................                                  ]

Expected results:
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: migrate_pulp...
Upgrade Step: migrate_candlepin...
Upgrade Step: migrate_foreman...
Upgrade Step: Running installer...
Installing             Done                                               [100%] [..............................................................................................................................]
  The full log is at /var/log/katello-installer/katello-installer.log


Additional info:
I was able to correct the problem by digging in and recognizing that the pid file in /var/run/mongodb/ was different than what is specified in /etc/systemd/system/multi-user.target.wants/mongod.service after noticing that the service would start, then get signal 15 sent to the process after a timeout.  Further triage included correcting the line in /etc/systemd/system/multi-user.target.wants/mongod.service, then running `katello-installer --upgrade` again would overwrite it back to the way it was before, ultimately breaking it again.  I then went to /usr/share/katello-installer/modules/mongodb/manifests/params.pp to update line 52 from:
"        $pidfilepath             = '/var/run/mongodb/mongod.pid'" to:
"        $pidfilepath             = '/var/run/mongodb/mongodb.pid'".
After this, katello-installer --upgrade correctly generates /etc/systemd/system/multi-user.target.wants/mongod.service and completes successfully.

Comment 4 Stephen Benjamin 2015-08-13 16:55:15 UTC

*** Bug 1252974 has been marked as a duplicate of this bug. ***

Comment 6 Sachin Ghai 2015-09-02 09:00:08 UTC

Verified with sat6.1.2 snap1.


Before upgrade on sat6.0.8, the installed version of mongod package:

[root@cloud-qe-20 ~]# rpm -qa | grep mongo
mongodb-2.4.9-3.el7sat.x86_64
python-pymongo-2.5.2-3.el7sat.x86_64
mongodb-server-2.4.9-3.el7sat.x86_64


[root@cloud-qe-20 ~]# cat /usr/share/katello-installer/modules/mongodb/manifests/params.pp | grep pidfilepath
        $pidfilepath         = '/var/run/mongodb/mongodb.pid'

Comment 7 Sachin Ghai 2015-09-02 09:01:20 UTC

Upgrade started and break it by "ctrl + C" and restarted mongod.

[root@cloud-qe-20 yum.repos.d]# katello-installer --upgrade
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: migrate_pulp...
Upgrade Step: start_httpd...
^C/usr/share/katello-installer/hooks/boot/01-helpers.rb:24:in ``': Interrupt
	from /usr/share/katello-installer/hooks/boot/01-helpers.rb:24:in `block in execute'
	from /usr/share/katello-installer/hooks/boot/01-helpers.rb:23:in `each'
	from /usr/share/katello-installer/hooks/boot/01-helpers.rb:23:in `execute'
	from /usr/share/katello-installer/hooks/pre/30-upgrade.rb:13:in `start_httpd'
	from /usr/share/katello-installer/hooks/pre/30-upgrade.rb:39:in `upgrade_step'
	from /usr/share/katello-installer/hooks/pre/30-upgrade.rb:56:in `block (4 levels) in load'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:32:in `instance_eval'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:32:in `block (4 levels) in load'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hook_context.rb:13:in `instance_exec'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hook_context.rb:13:in `execute'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:49:in `block in execute'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:47:in `each'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:47:in `execute'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:301:in `run_installation'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:109:in `execute'
	from /usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
	from /usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:125:in `run'
	from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:116:in `run'
	from /usr/sbin/katello-installer:46:in `<main>'

[root@cloud-qe-20 yum.repos.d]# service mongod restart
Redirecting to /bin/systemctl restart  mongod.service

Comment 8 Sachin Ghai 2015-09-02 09:01:50 UTC

Re-ran the upgrade and upgrade was completed successfully.

[root@cloud-qe-20 yum.repos.d]# katello-installer --upgrade
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: migrate_pulp...
Upgrade Step: start_httpd...
Upgrade Step: migrate_candlepin...
Upgrade Step: migrate_foreman...
Upgrade Step: Running installer...
Installing             Done                                               [100%] [..................................................................]
  The full log is at /var/log/katello-installer/katello-installer.log
Upgrade Step: restart_services...
Upgrade Step: db_seed...
Upgrade Step: errata_import (this may take a while) ...
Upgrade Step: update_gpg_urls (this may take a while) ...
Upgrade Step: update_repository_metadata (this may take a while) ...
Katello upgrade completed!

Comment 9 Sachin Ghai 2015-09-02 09:03:40 UTC

this file doesn't have pid file change under "migrate_pulp" section:

/usr/share/katello-installer/hooks/pre/30-upgrade.rb

<snippet>
def migrate_pulp
  # Start mongo if not running
  unless Kafo::Helpers.execute('pgrep mongod')
    Kafo::Helpers.execute('service-wait mongod start')
  end

Based on above comments, moving this to verified.

Comment 11 errata-xmlrpc 2015-09-15 07:22:28 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:1786

Note You need to log in before you can comment on or make changes to this bug.