Bug 859980 - Mongod crashed and then restarted leaving the app unavailable for a period of time
Summary: Mongod crashed and then restarted leaving the app unavailable for a period of...
Keywords:
Status: CLOSED DUPLICATE of bug 860013
Alias: None
Product: OKD
Classification: Red Hat
Component: Containers
Version: 2.x
Hardware: All
OS: All
high
high
Target Milestone: ---
: ---
Assignee: Rob Millner
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-09-24 14:36 UTC by Grant Shipley
Modified: 2015-05-14 22:59 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-09-27 18:30:31 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Grant Shipley 2012-09-24 14:36:15 UTC
Description of problem:
I noticed my app and mongod was running last night at 11:30 when I saw users inserting data into the mongodb instance.

This morning I noticed that mongod was not running on the machine.  Once I issued a mongo command via ssh, I was given the message it could not connect.  Did a ps -ef and then ran mongo again and it seemed to have restarted.  Full CLI commands below:

Actual results:
mongod was not running

Expected results:
mongod should have been running unless I stopped it

Additional info:

lappy:~ gshipley$ ssh be72eeab94be4b1c897c62dda82c673e.com

    *********************************************************************

    You are accessing a service that is for use only by authorized users.  
    If you do not have authorization, discontinue use at once. 
    Any use of the services is subject to the applicable terms of the 
    agreement which can be found at: 
    https://openshift.redhat.com/app/legal

    *********************************************************************

    Welcome to OpenShift shell

    This shell will assist you in managing OpenShift applications.

    !!! IMPORTANT !!! IMPORTANT !!! IMPORTANT !!!
    Shell access is quite powerful and it is possible for you to
    accidentally damage your application.  Proceed with care!
    If worse comes to worst, destroy your application with 'rhc app destroy'
    and recreate it
    !!! IMPORTANT !!! IMPORTANT !!! IMPORTANT !!!

    Type "help" for more info.

[mongotwil-onpaas.rhcloud.com ~]\> mongo
MongoDB shell version: 2.0.7
connecting to: 127.5.52.129:27017/admin
Mon Sep 24 10:13:50 Error: couldn't connect to server 127.5.52.129:27017 shell/mongo.js:86
exception: connect failed
[mongotwil-onpaas.rhcloud.com ~]\> mongo
MongoDB shell version: 2.0.7
connecting to: 127.5.52.129:27017/admin
Mon Sep 24 10:13:55 Error: couldn't connect to server 127.5.52.129:27017 shell/mongo.js:86
exception: connect failed
[mongotwil-onpaas.rhcloud.com ~]\> ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
2665      6341  6315  0 10:13 ?        00:00:00 sshd: be72eeab94be4b1c897c62dda82c673e@pts/0
2665      6342  6341  0 10:13 ?        00:00:00 /bin/bash --init-file /usr/bin/rhcsh -i
2665      6816  6342  0 10:14 ?        00:00:00 ps -ef
[mongotwil-onpaas.rhcloud.com ~]\> mongo
MongoDB shell version: 2.0.7
connecting to: 127.5.52.129:27017/admin
> exit
bye
[mongotwil-onpaas.rhcloud.com ~]\> ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
2665      6341  6315  0 10:13 ?        00:00:00 sshd: be72eeab94be4b1c897c62dda82c673e@pts/0
2665      6342  6341  0 10:13 ?        00:00:00 /bin/bash --init-file /usr/bin/rhcsh -i
2665      6995     1  0 10:14 ?        00:00:02 /usr/bin/mongod --auth --nojournal --smallfiles --quiet -f /var/lib/stickshift/be72eeab94be4b1c897c62dda82c673e//mongodb-2.0
2665      7114     1  0 10:14 ?        00:00:00 /usr/sbin/httpd -C Include /var/lib/stickshift/be72eeab94be4b1c897c62dda82c673e//php-5.3/conf.d/*.conf -f /usr/libexec/stick
2665      7117  7114  0 10:14 ?        00:00:00 /usr/sbin/rotatelogs /var/lib/stickshift/be72eeab94be4b1c897c62dda82c673e/php-5.3/logs/error_log-%Y%m%d-%H%M%S-%Z 86400
2665      7118  7114  0 10:14 ?        00:00:00 /usr/sbin/rotatelogs /var/lib/stickshift/be72eeab94be4b1c897c62dda82c673e/php-5.3/logs/access_log-%Y%m%d-%H%M%S-%Z 86400
2665      7138  7114  0 10:14 ?        00:00:00 /usr/sbin/httpd -C Include /var/lib/stickshift/be72eeab94be4b1c897c62dda82c673e//php-5.3/conf.d/*.conf -f /usr/libexec/stick
2665      7183  7114  0 10:14 ?        00:00:00 /usr/sbin/httpd -C Include /var/lib/stickshift/be72eeab94be4b1c897c62dda82c673e//php-5.3/conf.d/*.conf -f /usr/libexec/stick
2665     10843  6342  3 10:35 ?        00:00:00 ps -ef
[mongotwil-onpaas.rhcloud.com ~]\> 


The gear should not have been idle as I had requests coming in a few hours before.

Comment 1 Rob Millner 2012-09-27 00:34:16 UTC
We recently had a problem where one of the system management scripts killed all mongod processes rather than targeting a specific one.

Would you mind e-mailing me the mongodb log?  It should be in mongodb-2.0/log/mongodb.log 

Thanks!

Comment 2 Grant Shipley 2012-09-27 14:26:12 UTC
The logs are no longer available for this.  I was talking with Mike and Thomas and they figured out that my app went into idle state when it shouldn't have.  I think mike may have filed a duplicate bug for this but not sure.

Comment 3 Rob Millner 2012-09-27 18:30:31 UTC

*** This bug has been marked as a duplicate of bug 859990 ***

Comment 4 Rob Millner 2012-09-27 18:33:05 UTC

*** This bug has been marked as a duplicate of bug 860013 ***


Note You need to log in before you can comment on or make changes to this bug.