Hide Forgot
Description of problem: =============================== Not sure how big of a deal this is. Basically I deployed a application earlier today and came back to the server hours later to deploy again and found mongodb/warehouse were stopped. Looking through the logs, I found that mongo seemed to have crashed. Not sure if that is related to it being a numa machine but if mongo needs a special configuration for numa, we probably need to detect and do the recommended configuration. Copying warehouse guys for input... Additional info: ======================= numa = Non-Uniform Memory Access snippet from mongodb.log ----------------------------- Fri Feb 17 12:18:45 [initandlisten] MongoDB starting : pid=3083 port=27017 dbpath=/var/lib/mongodb 64-bit ** WARNING: You are running on a NUMA machine. ** We suggest launching mongod like this to avoid performance problems: ** numactl --interleave=all mongod [other options] Fri Feb 17 12:18:45 [initandlisten] db version v1.8.2, pdfile version 4.5 Fri Feb 17 12:18:45 [initandlisten] git version: nogitversion Fri Feb 17 12:18:45 [initandlisten] build sys info: Linux hs20-bc2-5.build.redhat.com 2.6.18-274.17.1.el5 #1 SMP Wed Jan 4 22:45:44 EST 2012 x86_64 BOOST_LIB_VERSION=1_41 ************** old lock file: /var/lib/mongodb/mongod.lock. probably means unclean shutdown recommend removing file and running --repair see: http://dochub.mongodb.org/core/repair for more information ************* Fri Feb 17 12:18:45 [initandlisten] exception in initAndListen std::exception: old lock file, terminating Fri Feb 17 12:18:45 dbexit: Fri Feb 17 12:18:45 [initandlisten] shutdown: going to close listening sockets... Fri Feb 17 12:18:45 [initandlisten] shutdown: going to flush diaglog... Fri Feb 17 12:18:45 [initandlisten] shutdown: going to close sockets... Fri Feb 17 12:18:45 [initandlisten] shutdown: waiting for fs preallocator... Fri Feb 17 12:18:45 [initandlisten] shutdown: closing all files... Fri Feb 17 12:18:45 closeAllFiles() finished Fri Feb 17 12:18:45 dbexit: really exiting now
Mongo is now in a bad state and not able to restart (aeolus-restart-services) without manually removing old lock file (/var/lib/mongodb/mongod.lock)
Deploying to numa isn't on scope for 1.0, but we'll look at it in the future.
New version of imagefactory does not require iwhd, which resolves the dep on mongodb.