Bug 794885

Summary: configure mongodb for numa machine
Product: [Retired] CloudForms Cloud Engine Reporter: Dave Johnson <dajohnso>
Component: aeolus-configureAssignee: Mike Orazi <morazi>
Status: CLOSED WONTFIX QA Contact: wes hayutin <whayutin>
Severity: medium Docs Contact:
Priority: medium    
Version: 1.0.0CC: akarol, brad, deltacloud-maint, meyering, ssachdev
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-01-16 18:26:57 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Dave Johnson 2012-02-17 20:44:06 UTC 
Description of problem:
===============================
Not sure how big of a deal this is.  Basically I deployed a application earlier today and came back to the server hours later to deploy again and found mongodb/warehouse were stopped.

Looking through the logs, I found that mongo seemed to have crashed.  Not sure if that is related to it being a numa machine but if mongo needs a special configuration for numa, we probably need to detect and do the recommended configuration.

Copying warehouse guys for input...


Additional info:
=======================

numa = Non-Uniform Memory Access

snippet from mongodb.log
----------------------------- 
Fri Feb 17 12:18:45 [initandlisten] MongoDB starting : pid=3083 port=27017 dbpath=/var/lib/mongodb 64-bit
 
** WARNING: You are running on a NUMA machine.
**          We suggest launching mongod like this to avoid performance problems:
**              numactl --interleave=all mongod [other options]
 
Fri Feb 17 12:18:45 [initandlisten] db version v1.8.2, pdfile version 4.5
Fri Feb 17 12:18:45 [initandlisten] git version: nogitversion
Fri Feb 17 12:18:45 [initandlisten] build sys info: Linux hs20-bc2-5.build.redhat.com 2.6.18-274.17.1.el5 #1 SMP Wed Jan 4 22:45:44 EST 2012 x86_64 BOOST_LIB_VERSION=1_41
**************
old lock file: /var/lib/mongodb/mongod.lock.  probably means unclean shutdown
recommend removing file and running --repair
see: http://dochub.mongodb.org/core/repair for more information
*************
Fri Feb 17 12:18:45 [initandlisten] exception in initAndListen std::exception: old lock file, terminating
Fri Feb 17 12:18:45 dbexit:
Fri Feb 17 12:18:45 [initandlisten] shutdown: going to close listening sockets...
Fri Feb 17 12:18:45 [initandlisten] shutdown: going to flush diaglog...
Fri Feb 17 12:18:45 [initandlisten] shutdown: going to close sockets...
Fri Feb 17 12:18:45 [initandlisten] shutdown: waiting for fs preallocator...
Fri Feb 17 12:18:45 [initandlisten] shutdown: closing all files...
Fri Feb 17 12:18:45 closeAllFiles() finished
Fri Feb 17 12:18:45 dbexit: really exiting now
Comment 1 Dave Johnson 2012-02-17 20:52:12 UTC 
Mongo is now in a bad state and not able to restart (aeolus-restart-services) without manually removing old lock file (/var/lib/mongodb/mongod.lock)
Comment 2 Mike Orazi 2012-02-21 15:34:04 UTC 
Deploying to numa isn't on scope for 1.0, but we'll look at it in the future.
Comment 4 Mike Orazi 2013-01-16 18:26:57 UTC 
New version of imagefactory does not require iwhd, which resolves the dep on mongodb.