Bug 857344

Summary: Old existing mongodb is stopped after STAGE is upgraded
Product: OKD Reporter: Jianwei Hou <jhou>
Component: ContainersAssignee: Rob Millner <rmillner>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: medium    
Version: 2.xCC: mfisher, xtian
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-09-17 21:29:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
mongodb.log none

Description Jianwei Hou 2012-09-14 07:33:36 UTC
Created attachment 612748 [details]
mongodb.log

Description of problem:
After an upgrade on STG, the embedded mongodb cartridge is stopped.

Version-Release number of selected component (if applicable):
on STG

How reproducible:
Always

Steps to Reproduce:
1. On STG, create app and embed mongodb, app and mongodb are both running
2. After STG has been upgraded, check apps and mongodb.

  
Actual results:
App is running, but mongodb is stopped.
 
[hjw@hjwlaptop ruby19]$ rhc app cartridge status -a ruby19s -c mongodb-2.0
Password: ******

RESULT:
MongoDB is stopped

Here are some information in log file, full log is attached:
Thu Sep 13 07:51:00 [PeriodicTask::Runner] task: DBConnectionPool-cleaner took: 4ms
Thu Sep 13 10:51:16 got kill or ctrl c or hup signal 15 (Terminated), will terminate after current cmd ends
Thu Sep 13 10:51:16 [interruptThread] now exiting
Thu Sep 13 10:51:16 dbexit:
Thu Sep 13 10:51:16 [interruptThread] shutdown: going to close listening sockets...
Thu Sep 13 10:51:16 [interruptThread] closing listening socket: 9
Thu Sep 13 10:51:16 [interruptThread] shutdown: going to flush diaglog...
Thu Sep 13 10:51:16 [interruptThread] shutdown: going to close sockets...
Thu Sep 13 10:51:16 [interruptThread] shutdown: waiting for fs preallocator...
Thu Sep 13 10:51:16 [interruptThread] shutdown: closing all files...


Expected results:
mongodb should be running.

Additional info:
I have one non-scaling perl app and a scaling ruby-1.9 app, they are both embedded with mongodb, which stopped for both of them.
Have to start mongodb from CLI to get it get back to work

Comment 1 Rob Millner 2012-09-14 19:26:20 UTC
Upgrading the mongodb kills all instances of mongodb running in gears.  Adding a restart to the migration script brings it back up.

Li commit 5f87da6.

Comment 2 Rob Millner 2012-09-14 19:35:19 UTC
Master pull request:
https://github.com/openshift/li/pull/373

Stage pull request:
https://github.com/openshift/li/pull/375

Comment 3 Rob Millner 2012-09-14 19:36:16 UTC
Reset severity high based on this will wipe out all instances of mongodb on all gears in prod when we upgrade.

Comment 4 Rob Millner 2012-09-14 23:08:46 UTC
Merged into the stage and master branches.

Comment 5 Adam Miller 2012-09-14 23:36:02 UTC
Rob's patch is a part of devenv-stage_237 (ami-f58b3b9c)

Comment 6 Jianwei Hou 2012-09-17 06:01:39 UTC
Verified on STAGE

Steps:
1. Create apps and embed mongodb before upgrade on STAGE
2. After STAGE has been upgraded check app and embedded mongodb

Result:
App and mongodb are both running.