Hide Forgot
Description of problem: When creating application using mongodb-persistent-template.json or postgresql-persistent-template.json, pods keep restarting with errors Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Create an application using mongodb-persistent template 2. Create a persistent volume claim for the application bounded to a persistent volume 3. The pod gets created however it keeps on restarting. Logs show below errors : => Waiting for container IP address ... 10.211.2.30:27017 => Waiting for MongoDB service startup ... note: noprealloc may hurt performance in many applications 2016-03-09T08:52:51.774-0500 [initandlisten] MongoDB starting : pid=26 port=27017 dbpath=/var/lib/mongodb/data 64-bit host=embed-mongodb-1-nhpcb 2016-03-09T08:52:51.775-0500 [initandlisten] db version v2.6.9 2016-03-09T08:52:51.775-0500 [initandlisten] git version: nogitversion 2016-03-09T08:52:51.775-0500 [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013 2016-03-09T08:52:51.775-0500 [initandlisten] build info: Linux x86-036.build.eng.bos.redhat.com 2.6.32-504.16.2.el6.x86_64 #1 SMP Tue Mar 10 17:01:00 EDT 2015 x86_64 BOOST_LIB_VERSION=1_53 2016-03-09T08:52:51.775-0500 [initandlisten] allocator: tcmalloc 2016-03-09T08:52:51.775-0500 [initandlisten] options: { config: "/etc/mongod.conf", net: { http: { enabled: false }, port: 27017 }, processManagement: { pidFilePath: "/var/lib/mongodb/mongodb.pid" }, replication: { oplogSizeMB: 64 }, storage: { dbPath: "/var/lib/mongodb/data", preallocDataFiles: false, smallFiles: true }, systemLog: { quiet: true } } 2016-03-09T08:52:51.783-0500 [initandlisten] journal dir=/var/lib/mongodb/data/journal 2016-03-09T08:52:51.785-0500 [initandlisten] recover : no journal files present, no recovery needed 2016-03-09T08:52:51.788-0500 [initandlisten] info preallocateIsFaster couldn't run due to: couldn't open file /var/lib/mongodb/data/journal/tempLatencyTest for writing errno:1 Operation not permitted; returning false 2016-03-09T08:52:51.791-0500 [initandlisten] exception in initAndListen: 13516 couldn't open file /var/lib/mongodb/data/journal/j._0 for writing errno:1 Operation not permitted, terminating 2016-03-09T08:52:51.791-0500 [initandlisten] dbexit: 2016-03-09T08:52:51.791-0500 [initandlisten] shutdown: going to close listening sockets... 2016-03-09T08:52:51.791-0500 [initandlisten] shutdown: going to flush diaglog... 2016-03-09T08:52:51.791-0500 [initandlisten] shutdown: going to close sockets... 2016-03-09T08:52:51.791-0500 [initandlisten] shutdown: waiting for fs preallocator... 2016-03-09T08:52:51.791-0500 [initandlisten] shutdown: lock for final commit... 2016-03-09T08:52:51.791-0500 [initandlisten] shutdown: final commit... 2016-03-09T08:52:51.791-0500 [initandlisten] shutdown: closing all files... 2016-03-09T08:52:51.791-0500 [initandlisten] closeAllFiles() finished 2016-03-09T08:52:51.791-0500 [initandlisten] journalCleanup... 2016-03-09T08:52:51.791-0500 [initandlisten] removeJournalFiles 2016-03-09T08:52:51.795-0500 [initandlisten] shutdown: removing fs lock... 2016-03-09T08:52:51.796-0500 [initandlisten] dbexit: really exiting now => Waiting for MongoDB service startup ... => Waiting for MongoDB service startup ... Actual results: Mongodb fails to start every time using default Mongodb-persistent template Expected results: It should be started without any issues using default template. Additional info:
sounds like a storage configuration issue: 2016-03-09T08:52:51.791-0500 [initandlisten] exception in initAndListen: 13516 couldn't open file /var/lib/mongodb/data/journal/j._0 for writing errno:1 Operation not permitted, terminating
What is a the PV and PVC like? The message points Operation Not Permitted error. The underlying storage looks the issue.
Please provide the jaml for the PV the PVC, and the pod. Also what is the the backing storage ?
Note that our knowledge base suggests to fiddle with service accounts to work around this: https://access.redhat.com/solutions/2194761
So as Jan mentioned in comment#5 this: https://access.redhat.com/solutions/2194761 There are two more options: 1) Use a block device as the bases for storage backend (Ceph, iSCSI, AWS EBC, GCE PD, not NFS) and enable automatic fsGroup assignment - to enable automatic fsGroup assignment: oc get -o json pod | grep scc # get scc name oc edit <scc name> #set fsGroup type to MustRunAs instead of RunAsAny more information about fsGroup: https://docs.openshift.org/latest/install_config/persistent_storage/pod_security_context.html#fsgroup 2) Use NFS as a back end but change the group ownership of the target directory to some group (eg 12345) then add that group as a supplemental group to the pod either by editing the template file or editing the dc. we are working on a feature to enable adding it to the PV: https://github.com/kubernetes/kubernetes/pull/20490
closing this. Jaspreet please reopen if you feel you issues have not been addressed.