Bug 1733538

Summary: mongodb fails to start due to stale lock file
Product: Red Hat OpenStack Reporter: David Hill <dhill>
Component: openstack-tripleo-heat-templatesAssignee: Matthias Runge <mrunge>
Status: CLOSED WONTFIX QA Contact: Nataf Sharabi <nsharabi>
Severity: high Docs Contact:
Priority: high    
Version: 10.0 (Newton)CC: emacchi, lnatapov, mburns, mrunge
Target Milestone: asyncKeywords: Triaged, ZStream
Target Release: 10.0 (Newton)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-09 21:01:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description David Hill 2019-07-26 12:29:52 UTC
Description of problem:
mongodb fails to start due to stale lock file:
2019-07-25T11:00:03.434+0000 ***** SERVER RESTARTED *****
2019-07-25T11:00:03.439+0000 [initandlisten] MongoDB starting : pid=55775 port=27017 dbpath=/var/lib/mongodb 64-bit host=pxipod001-controller-1
2019-07-25T11:00:03.440+0000 [initandlisten] 
2019-07-25T11:00:03.440+0000 [initandlisten] ** WARNING: You are running on a NUMA machine.
2019-07-25T11:00:03.440+0000 [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
2019-07-25T11:00:03.440+0000 [initandlisten] **              numactl --interleave=all mongod [other options]
2019-07-25T11:00:03.440+0000 [initandlisten] 
2019-07-25T11:00:03.440+0000 [initandlisten] db version v2.6.11
2019-07-25T11:00:03.440+0000 [initandlisten] git version: nogitversion
2019-07-25T11:00:03.440+0000 [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2019-07-25T11:00:03.440+0000 [initandlisten] build info: Linux x86-021.build.eng.bos.redhat.com 2.6.32-573.12.1.el6.x86_64 #1 SMP Mon Nov 23 12:55:32 EST 2015 x86_64 BOOST_LIB_VERSION=1_53
2019-07-25T11:00:03.440+0000 [initandlisten] allocator: tcmalloc
2019-07-25T11:00:03.440+0000 [initandlisten] options: { command: [ "run" ], config: "/etc/mongod.conf", net: { bindIp: "192.168.5.17" }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, replication: { replSet: "tripleo" }, security: { authorization: "disabled" }, storage: { dbPath: "/var/lib/mongodb" }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongodb.log", quiet: true } }
2019-07-25T11:00:03.441+0000 [initandlisten] 
2019-07-25T11:00:03.441+0000 [initandlisten] ** WARNING: Readahead for /var/lib/mongodb is set to 4096KB
2019-07-25T11:00:03.441+0000 [initandlisten] **          We suggest setting it to 256KB (512 sectors) or less
2019-07-25T11:00:03.441+0000 [initandlisten] **          http://dochub.mongodb.org/core/readahead
2019-07-25T11:00:03.441+0000 [initandlisten] exception in initAndListen: 12596 old lock file, terminating
2019-07-25T11:00:03.441+0000 [initandlisten] dbexit: 
2019-07-25T11:00:03.441+0000 [initandlisten] shutdown: going to close listening sockets...
2019-07-25T11:00:03.441+0000 [initandlisten] shutdown: going to flush diaglog...
2019-07-25T11:00:03.441+0000 [initandlisten] shutdown: going to close sockets...
2019-07-25T11:00:03.441+0000 [initandlisten] shutdown: waiting for fs preallocator...
2019-07-25T11:00:03.441+0000 [initandlisten] shutdown: lock for final commit...
2019-07-25T11:00:03.441+0000 [initandlisten] shutdown: final commit...
2019-07-25T11:00:03.441+0000 [initandlisten] shutdown: closing all files...
2019-07-25T11:00:03.441+0000 [initandlisten] closeAllFiles() finished
2019-07-25T11:00:03.441+0000 [initandlisten] dbexit: really exiting now


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-5.3.0-6.el7ost.noarch

How reproducible:
Random

Steps to Reproduce:
1. Do a minor update
2.
3.

Actual results:
Fails

Expected results:
Succeeds

Additional info:
second case I see with this behavior.