Bug 773616

Summary: mongod process is not started after upgrade packages and run migrate script.
Product: OKD Reporter: Johnny Liu <jialiu>
Component: ContainersAssignee: Mike McGrath <mmcgrath>
Status: CLOSED NOTABUG QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 1.xCC: mmcgrath, poelstra
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-01-16 18:22:16 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Johnny Liu 2012-01-12 12:30:42 UTC
Description of problem:
mongod process is not started after upgrade packages and run migrate script.

Version-Release number of selected component (if applicable):
rhc-node-0.84.23-1.el6_2.noarch
rhc-cartridge-rockmongo-1.1-1.2.6-1.el6_2.noarch
rhc-cartridge-mongodb-2.0-0.12.7-1.el6_2.noarch
rhc-devenv-0.84.11-1.el6_2.noarch

How reproducible:
Always

Steps to Reproduce:
1. Create apps, embed mongodb and rockmongo to it against old instance (stage_110)
2. Use rockmongo app access mongodb successfully.
3. Upgrade all the packages to the latest.
# yum update -y
3. Running migrate script for li git repo.
# ./migrate-2.0.3
4. Try to access mongodb using rockmongo app

  
Actual results:
mongodb can not be accessed with error message - "Transport endpoint
is not connected".

Log into app using rhcsh, found mongodb process is not listed in the output of "ps -ef".
Start mongodb via appname_mongodb_ctl.sh, rockmongo can access mongodb now.

Expected results:
After upgrade, mongodb should be always running.



Additional info:

!!!!!NOTE!!!!!
I am using a large instance (devenv_stage_110), before upgrade the disk usage about 79%; after upgrade the disk usage reached to 98%. I almost can not create more app. Or if there are already some more existing app before upgrade, after upgrade I think the disk usage must reach 100%.  This is very risky.

Check the size of /var/lib/mongodb/, found it occupied most of available disk usage.
# du -sm /var/lib/mongodb/*
1108	/var/lib/mongodb/journal
1	/var/lib/mongodb/mongod.lock

Found such a process is running after upgrade, compare with devenv_stage_112, this process should not be running.
# ps -ef|grep mongo
root     17257 14262  0 07:21 pts/0    00:00:00 grep mongo
mongodb  24613     1  0 06:05 ?        00:00:01 /usr/bin/mongod --quiet -f /etc/mongodb.conf run

Comment 1 Mike McGrath 2012-01-12 14:50:21 UTC
Just a note on /var/lib/mongodb/  that happens automatically when someone starts the system mongod package.  Which I believe we're disabling.  But if you're seeing that directory full, and you didn't start mongo manually, something's starting it, we'll have to figure that out.

Also, a running system mongod could also explain why your application's mongod didn't start.

Comment 2 Johnny Liu 2012-01-13 02:13:43 UTC
Here is some clue for your debug, when I "yum update", I found system mongod package is also been updated, so I guess maybe system mongod is started during pakcage update process.

Comment 3 John Poelstra 2012-01-16 18:22:16 UTC
Discussed with McGrath.  Could not reproduce.  Please re-open if you observe again.