Bug 1274192 - Cannot connect to mongodb for ephemeral storage - mongodb-26-rhel7
Cannot connect to mongodb for ephemeral storage - mongodb-26-rhel7
Status: CLOSED ERRATA
Product: Red Hat Software Collections
Classification: Red Hat
Component: rh-mongodb26-docker (Show other bugs)
devassist09
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Marek Skalický
wewang
:
Depends On:
Blocks: 1220712
  Show dependency treegraph
 
Reported: 2015-10-22 04:41 EDT by wewang
Modified: 2015-11-16 10:56 EST (History)
7 users (show)

See Also:
Fixed In Version: rhscl/mongodb-26-rhel7:2.6-2
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-16 10:56:34 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description wewang 2015-10-22 04:41:23 EDT
Version-Release number of selected component (if applicable):
rhscl/mongodb-26-rhel7         fead03cf7fc4

Description of problem:
Cannot connect to  mongodb after using mongodb-ephemeral-template.json to create mongo app

How reproducible:
Always

Steps to Reproduce:
1. Create a project and  imagestream:mongo 

2. Create mongodb with mongodb-ephemeral-template.json
$ oc new-app -f mongodb-ephemeral-template.json

3. Check the pod status
# oc get pods
NAME              READY     STATUS    RESTARTS   AGE
mongodb-1-pcwkv   1/1       Running   0          2m
[root@dhcp-128-91 v3test]# oc env pod  mongodb-1-pcwkv --list
# pods mongodb-1-pcwkv, container mongodb
MONGODB_USER=userVIJ
MONGODB_PASSWORD=jOhwsOesdYvwI7Aw
MONGODB_DATABASE=sampledb
MONGODB_ADMIN_PASSWORD=Mf87qeYCtYQv0TeR
[root@dhcp-128-91 v3test]# oc rsh mongodb-1-pcwkv
bash-4.2$ mongo sampledb -u userVIJ -pMf87qeYCtYQv0TeR
MongoDB shell version: 2.6.9
connecting to: sampledb
2015-10-22T03:35:10.056-0400 Error: 18 { ok: 0.0, errmsg: "auth failed", code: 18 } at src/mongo/shell/db.js:1260
exception: login failed
Comment 1 wewang 2015-10-22 04:44:32 EDT
The same problem happened in mongodb-24-rhel7:
openshift3/mongodb-24-rhel7 d17602c1d664

[root@dhcp-128-91 v3test]# oc get pod
NAME              READY     STATUS    RESTARTS   AGE
mongodb-1-n96k0   1/1       Running   0          42s
[root@dhcp-128-91 v3test]# oc env pod mongodb-1-n96k0 --list
# pods mongodb-1-n96k0, container mongodb
MONGODB_USER=userSWU
MONGODB_PASSWORD=8OuLqRfG4LQChImE
MONGODB_DATABASE=sampledb
MONGODB_ADMIN_PASSWORD=LlXTbvLxEsmEfTrY
[root@dhcp-128-91 v3test]# oc rsh mongodb-1-n96k0
bash-4.2$ mongo sampledb -u userSWU -pLlXTbvLxEsmEfTrY
MongoDB shell version: 2.4.9
connecting to: sampledb
Thu Oct 22 04:34:45.187 Error: 18 { code: 18, ok: 0.0, errmsg: "auth fails" } at src/mongo/shell/db.js:228
exception: login failed
Comment 2 wewang 2015-10-22 04:50:34 EDT
reproduced  in fedora_2517
rhscl/mongodb-26-rhel7   7a98ef0dfe31

[root@dhcp-128-91 v3test]# oc get pods
NAME               READY     STATUS        RESTARTS   AGE
mongodb-1-brzun    1/1       Running       0          43s
[root@dhcp-128-91 v3test]# oc env pod mongodb-1-brzun --list
# pods mongodb-1-brzun, container mongodb
MONGODB_USER=userJ05
MONGODB_PASSWORD=fjLqxPpap83Xluhu
MONGODB_DATABASE=sampledb
MONGODB_ADMIN_PASSWORD=d2aEHhAD82aMHTAX
[root@dhcp-128-91 v3test]# oc rsh mongodb-1-brzun
bash-4.2$ mongo sampledb -u userJ05 -pd2aEHhAD82aMHTAX
MongoDB shell version: 2.6.9
connecting to: sampledb
2015-10-22T04:04:16.076-0400 Error: 18 { ok: 0.0, errmsg: "auth failed", code: 18 } at src/mongo/shell/db.js:1260
exception: login failed
Comment 3 Marek Skalický 2015-10-22 10:46:50 EDT
Everything is working as expected.

You are trying to auth with $MONGODB_USER and $MONGODB_ADMIN_PASSWORD. It is possible to auth only by ($MONGODB_USER and $MONGODB_PASSWORD) or (admin and $MONGODB_ADMIN_PASSWORD).
Or is there something else?
Comment 4 wewang 2015-10-23 04:54:07 EDT
Sorry for pasting wrong password,using MONGODB_PASSWORD doesn't work
[root@dhcp-128-91 v3test]# oc env pod mongodb-1-kdhrd --list
# pods mongodb-1-kdhrd, container mongodb
MONGODB_USER=userWLQ
MONGODB_PASSWORD=PEhlnW66N8SRiImR
MONGODB_DATABASE=sampledb
MONGODB_ADMIN_PASSWORD=kWxmibs8q0W21pDC
[root@dhcp-128-91 v3test]# oc rsh mongodb-1-kdhrd
bash-4.2$ mongo sampledb -u userWLQ -pPEhlnW66N8SRiImR
MongoDB shell version: 2.6.9
connecting to: sampledb
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
	http://docs.mongodb.org/
Questions? Try the support group
	http://groups.google.com/group/mongodb-user
2015-10-23T04:45:04.675-0400 In File::open(), ::open for '//.mongorc.js' failed with errno:13 Permission denied
Comment 5 Marek Skalický 2015-10-23 08:22:05 EDT
wewang, isn't this only effect of this bug #1273842 ?

Does your login really fail? Or you only see the error message of mentioned bug?
I can't reproduce it. I normally can login...
What image are you using?
Comment 8 wewang 2015-11-04 03:24:28 EST
verified in
ose3.1
openshift v3.0.2.905
kubernetes v1.2.0-alpha.1-1107-g4c8e6f4
rhscl/mongodb-26-rhel7   c2edfb2c2498

[root@dhcp-128-91 templates]# oc rsh mongodb-1-q5ehd
bash-4.2$ oc 
bash: oc: command not found
bash-4.2$ 
bash-4.2$  mongo sampledb -u userY8K -paHctmTGt5mMOMoef
MongoDB shell version: 2.6.9
connecting to: sampledb
> show collections;
> doc={"a":"test1"};
{ "a" : "test1" }

--------------------------

I have a question ,when I type "show dbs", the result as follow is correct? because user:userY8K has no permission ? 

> > show dbs;
2015-11-04T02:21:17.642-0500 listDatabases failed:{
	"ok" : 0,
	"errmsg" : "not authorized on admin to execute command { listDatabases: 1.0 }",
	"code" : 13
} at src/mongo/shell/mongo.js:47

-------------------
and I met some other problems:
Steps to Reproduce:
1. Create project 
2. Create mongodb app
$oc process -f https://raw.githubusercontent.com/wewang58/v3-testfiles/master/image/db-templates/mongodb-clustered.json | oc create -f -
--> Deploying template mongodb-replica-example for "mongodb-clustered.json"
     With parameters:
      MONGODB_USER=SPD # generated
      MONGODB_PASSWORD=l5A0TjULhe1O # generated
      MONGODB_DATABASE=userdb
      MONGODB_ADMIN_PASSWORD=qMSakW0ASGMX # generated
      MONGODB_REPLICA_NAME=rs0
      MONGODB_SERVICE_NAME=mongodb
      MONGODB_KEYFILE_VALUE=WgpsHG0LpNmFjFaTpRy5D6mGDAEVSuV5RRvy3vnRsmEhGv821JaMgReJHKGOgJSv2NLAG14bYMHx6J3MnuepYcjsFL7Qalf5t0qReFXcWeKAVmJPETeVUAgsdLdISgSXGIFc0twKsYViSs4yEUx8NjfeyPdvWWgFhPitqgH3sxw1OtG1dY52u7XXTvmBnmTUkvoHgphYRTHGho4TbpUFHH5U0wE74r8dscqdDwQ1aOysvqtJUOCHroqs8dhVe6R # generated
--> Creating resources ...
    Service "mongodb" created
    DeploymentConfig "mongodb" created
--> Success
    Run 'oc status' to view your app.

3. Access mongodb and create data
[root@dhcp-128-91 templates]# oc get pods
NAME              READY     STATUS    RESTARTS   AGE
mongodb-1-k8b3a   1/1       Running   0          4m
mongodb-1-q8a7k   1/1       Running   0          4m
mongodb-1-rbg4k   1/1       Running   0          4m
[root@dhcp-128-91 templates]# oc env pod mongodb-1-k8b3a --list
# pods mongodb-1-k8b3a, container mongodb
MONGODB_USER=SPD
MONGODB_PASSWORD=l5A0TjULhe1O
MONGODB_DATABASE=userdb
MONGODB_ADMIN_PASSWORD=qMSakW0ASGMX
MONGODB_REPLICA_NAME=rs0
MONGODB_SERVICE_NAME=mongodb
MONGODB_KEYFILE_VALUE=WgpsHG0LpNmFjFaTpRy5D6mGDAEVSuV5RRvy3vnRsmEhGv821JaMgReJHKGOgJSv2NLAG14bYMHx6J3MnuepYcjsFL7Qalf5t0qReFXcWeKAVmJPETeVUAgsdLdISgSXGIFc0twKsYViSs4yEUx8NjfeyPdvWWgFhPitqgH3sxw1OtG1dY52u7XXTvmBnmTUkvoHgphYRTHGho4TbpUFHH5U0wE74r8dscqdDwQ1aOysvqtJUOCHroqs8dhVe6R
[root@dhcp-128-91 templates]# oc rsh mongodb-1-k8b3a
bash-4.2$ mongo userdb -u SPD -pl5A0TjULhe1O
MongoDB shell version: 2.6.9
connecting to: userdb
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
	http://docs.mongodb.org/
Questions? Try the support group
	http://groups.google.com/group/mongodb-user
rs0:PRIMARY> show collections;
rs0:PRIMARY> db.createCollection("test_db", { size: 104857600 } );
{ "ok" : 1 }
rs0:PRIMARY> doc={'a':1, 'b':2}
{ "a" : 1, "b" : 2 }
rs0:PRIMARY> db.test_db.save(doc)
WriteResult({ "nInserted" : 1 })
rs0:PRIMARY>  db.test_db.find();
{ "_id" : ObjectId("5639b9de4984a4d1e701a046"), "a" : 1, "b" : 2 }
rs0:PRIMARY> exit

5. Change the mongodb password

$oc env dc/mongodb  MONGODB_PASSWORD=kdGnriXeR12y MONGODB_ADMIN_PASSWORD=nmNRhdtQ0q0k
6. Try to connect mongodb with new password 
[root@dhcp-128-91 templates]# oc get pods
NAME              READY     STATUS    RESTARTS   AGE
mongodb-2-0ca9r   1/1       Running   0          2m
mongodb-2-60sjm   1/1       Running   0          2m
mongodb-2-b6n5x   1/1       Running   0          2m
[
[root@dhcp-128-91 templates]# oc env pod mongodb-2-0ca9r --list
# pods mongodb-2-0ca9r, container mongodb
MONGODB_USER=SPD
MONGODB_PASSWORD=kdGnriXeR12y
MONGODB_DATABASE=userdb
MONGODB_ADMIN_PASSWORD=nmNRhdtQ0q0k
MONGODB_REPLICA_NAME=rs0
MONGODB_SERVICE_NAME=mongodb
MONGODB_KEYFILE_VALUE=WgpsHG0LpNmFjFaTpRy5D6mGDAEVSuV5RRvy3vnRsmEhGv821JaMgReJHKGOgJSv2NLAG14bYMHx6J3MnuepYcjsFL7Qalf5t0qReFXcWeKAVmJPETeVUAgsdLdISgSXGIFc0twKsYViSs4yEUx8NjfeyPdvWWgFhPitqgH3sxw1OtG1dY52u7XXTvmBnmTUkvoHgphYRTHGho4TbpUFHH5U0wE74r8dscqdDwQ1aOysvqtJUOCHroqs8dhVe6R
[root@dhcp-128-91 templates]# oc rsh mongodb-2-0ca9r
bash-4.2$ mongo userdb -u SPD -pkdGnriXeR12y
MongoDB shell version: 2.6.9
connecting to: userdb
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
	http://docs.mongodb.org/
Questions? Try the support group
	http://groups.google.com/group/mongodb-user
rs0:SECONDARY> show collections;
2015-11-04T03:01:49.353-0500 error: { "$err" : "not master and slaveOk=false", "code" : 13435 } at src/mongo/shell/query.js:131
rs0:SECONDARY> db.test_db.find();
error: { "$err" : "not master and slaveOk=false", "code" : 13435 }
rs0:SECONDARY> exit
bye
Comment 9 Marek Skalický 2015-11-04 05:25:50 EST
(In reply to wewang from comment #8)
> 
> I have a question ,when I type "show dbs", the result as follow is correct?
> because user:userY8K has no permission ? 
> 
> > > show dbs;
> 2015-11-04T02:21:17.642-0500 listDatabases failed:{
> 	"ok" : 0,
> 	"errmsg" : "not authorized on admin to execute command { listDatabases: 1.0
> }",
> 	"code" : 13
> } at src/mongo/shell/mongo.js:47

Yes, this is correct. MONGODB_USER have only permissions regarding MONGODB_DATABASE, so he doesn't have permission to show all database (stored in admin database).

"The listDatabases must run against the admin database" (https://docs.mongodb.org/manual/reference/command/listDatabases/), so MONGODB_USER can't run it.



> -------------------
> and I met some other problems:
> Steps to Reproduce:
> 1. Create project 
> 2. Create mongodb app
> $oc process -f
> https://raw.githubusercontent.com/wewang58/v3-testfiles/master/image/db-
> templates/mongodb-clustered.json | oc create -f -
> --> Deploying template mongodb-replica-example for "mongodb-clustered.json"
>      With parameters:
>       MONGODB_USER=SPD # generated
>       MONGODB_PASSWORD=l5A0TjULhe1O # generated
>       MONGODB_DATABASE=userdb
>       MONGODB_ADMIN_PASSWORD=qMSakW0ASGMX # generated
>       MONGODB_REPLICA_NAME=rs0
>       MONGODB_SERVICE_NAME=mongodb
>      
> MONGODB_KEYFILE_VALUE=WgpsHG0LpNmFjFaTpRy5D6mGDAEVSuV5RRvy3vnRsmEhGv821JaMgRe
> JHKGOgJSv2NLAG14bYMHx6J3MnuepYcjsFL7Qalf5t0qReFXcWeKAVmJPETeVUAgsdLdISgSXGIFc
> 0twKsYViSs4yEUx8NjfeyPdvWWgFhPitqgH3sxw1OtG1dY52u7XXTvmBnmTUkvoHgphYRTHGho4Tb
> pUFHH5U0wE74r8dscqdDwQ1aOysvqtJUOCHroqs8dhVe6R # generated
> --> Creating resources ...
>     Service "mongodb" created
>     DeploymentConfig "mongodb" created
> --> Success
>     Run 'oc status' to view your app.
> 
> 3. Access mongodb and create data
> [root@dhcp-128-91 templates]# oc get pods
> NAME              READY     STATUS    RESTARTS   AGE
> mongodb-1-k8b3a   1/1       Running   0          4m
> mongodb-1-q8a7k   1/1       Running   0          4m
> mongodb-1-rbg4k   1/1       Running   0          4m
> [root@dhcp-128-91 templates]# oc env pod mongodb-1-k8b3a --list
> # pods mongodb-1-k8b3a, container mongodb
> MONGODB_USER=SPD
> MONGODB_PASSWORD=l5A0TjULhe1O
> MONGODB_DATABASE=userdb
> MONGODB_ADMIN_PASSWORD=qMSakW0ASGMX
> MONGODB_REPLICA_NAME=rs0
> MONGODB_SERVICE_NAME=mongodb
> MONGODB_KEYFILE_VALUE=WgpsHG0LpNmFjFaTpRy5D6mGDAEVSuV5RRvy3vnRsmEhGv821JaMgRe
> JHKGOgJSv2NLAG14bYMHx6J3MnuepYcjsFL7Qalf5t0qReFXcWeKAVmJPETeVUAgsdLdISgSXGIFc
> 0twKsYViSs4yEUx8NjfeyPdvWWgFhPitqgH3sxw1OtG1dY52u7XXTvmBnmTUkvoHgphYRTHGho4Tb
> pUFHH5U0wE74r8dscqdDwQ1aOysvqtJUOCHroqs8dhVe6R
> [root@dhcp-128-91 templates]# oc rsh mongodb-1-k8b3a
> bash-4.2$ mongo userdb -u SPD -pl5A0TjULhe1O
> MongoDB shell version: 2.6.9
> connecting to: userdb
> Welcome to the MongoDB shell.
> For interactive help, type "help".
> For more comprehensive documentation, see
> 	http://docs.mongodb.org/
> Questions? Try the support group
> 	http://groups.google.com/group/mongodb-user
> rs0:PRIMARY> show collections;
> rs0:PRIMARY> db.createCollection("test_db", { size: 104857600 } );
> { "ok" : 1 }
> rs0:PRIMARY> doc={'a':1, 'b':2}
> { "a" : 1, "b" : 2 }
> rs0:PRIMARY> db.test_db.save(doc)
> WriteResult({ "nInserted" : 1 })
> rs0:PRIMARY>  db.test_db.find();
> { "_id" : ObjectId("5639b9de4984a4d1e701a046"), "a" : 1, "b" : 2 }
> rs0:PRIMARY> exit
> 
> 5. Change the mongodb password
> 
> $oc env dc/mongodb  MONGODB_PASSWORD=kdGnriXeR12y
> MONGODB_ADMIN_PASSWORD=nmNRhdtQ0q0k
> 6. Try to connect mongodb with new password 
> [root@dhcp-128-91 templates]# oc get pods
> NAME              READY     STATUS    RESTARTS   AGE
> mongodb-2-0ca9r   1/1       Running   0          2m
> mongodb-2-60sjm   1/1       Running   0          2m
> mongodb-2-b6n5x   1/1       Running   0          2m
> [
> [root@dhcp-128-91 templates]# oc env pod mongodb-2-0ca9r --list
> # pods mongodb-2-0ca9r, container mongodb
> MONGODB_USER=SPD
> MONGODB_PASSWORD=kdGnriXeR12y
> MONGODB_DATABASE=userdb
> MONGODB_ADMIN_PASSWORD=nmNRhdtQ0q0k
> MONGODB_REPLICA_NAME=rs0
> MONGODB_SERVICE_NAME=mongodb
> MONGODB_KEYFILE_VALUE=WgpsHG0LpNmFjFaTpRy5D6mGDAEVSuV5RRvy3vnRsmEhGv821JaMgRe
> JHKGOgJSv2NLAG14bYMHx6J3MnuepYcjsFL7Qalf5t0qReFXcWeKAVmJPETeVUAgsdLdISgSXGIFc
> 0twKsYViSs4yEUx8NjfeyPdvWWgFhPitqgH3sxw1OtG1dY52u7XXTvmBnmTUkvoHgphYRTHGho4Tb
> pUFHH5U0wE74r8dscqdDwQ1aOysvqtJUOCHroqs8dhVe6R
> [root@dhcp-128-91 templates]# oc rsh mongodb-2-0ca9r
> bash-4.2$ mongo userdb -u SPD -pkdGnriXeR12y
> MongoDB shell version: 2.6.9
> connecting to: userdb
> Welcome to the MongoDB shell.
> For interactive help, type "help".
> For more comprehensive documentation, see
> 	http://docs.mongodb.org/
> Questions? Try the support group
> 	http://groups.google.com/group/mongodb-user
> rs0:SECONDARY> show collections;
> 2015-11-04T03:01:49.353-0500 error: { "$err" : "not master and
> slaveOk=false", "code" : 13435 } at src/mongo/shell/query.js:131
> rs0:SECONDARY> db.test_db.find();
> error: { "$err" : "not master and slaveOk=false", "code" : 13435 }
> rs0:SECONDARY> exit
> bye


This is normal situation. After redeployment you have successfully connected to secondary member of replica set (rs0:SECONDARY prompt). However, by default it is not allowed to run commands on secondary. (you have to run https://docs.mongodb.org/manual/reference/method/rs.slaveOk/ on the secondary first)
Comment 10 wewang 2015-11-05 05:46:28 EST
@Marek Skalický , Really thanks for answer my questions, now I add "db.getMongo().setSlaveOk()" in secondary member of replica set , it works:

NAME              READY     STATUS    RESTARTS   AGE       NODE
mongodb-2-p3k8l   1/1       Running   0          5m        openshift-137.lab.sjc.redhat.com
mongodb-2-ra274   1/1       Running   0          5m        openshift-143.lab.sjc.redhat.com
mongodb-2-xnmg7   1/1       Running   0          5m        openshift-137.lab.sjc.redhat.com
[root@dhcp-128-91 1103]# oc rsh mongodb-2-ra274
bash-4.2$ mongo userdb -u Gvt -pkdGnriXeR12y
MongoDB shell version: 2.6.9
connecting to: userdb
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
	http://docs.mongodb.org/
Questions? Try the support group
	http://groups.google.com/group/mongodb-user
rs0:SECONDARY> db.getMongo().setSlaveOk()
rs0:SECONDARY> db.test_db.find();
rs0:SECONDARY>
Comment 12 wewang 2015-11-12 00:18:20 EST
I just confirm something ,before we can get /var/lib/mongodb/mongodb.conf in mongodb container, but now I just see 
bash-4.2$ cd /var/lib/mongodb
bash-4.2$ ls
data  mongodb.pid
Comment 13 wewang 2015-11-12 00:23:23 EST
I got it 
 bash-4.2$ more /opt/rh/mongodb24/root/etc/mongodb.conf
##
### Basic Defaults
##
bind_ip = 127.0.0.1
port = 27017
fork = true
pidfilepath = /opt/rh/mongodb24/root/var/run/mongodb/mongodb.pid
unixSocketPrefix = /opt/rh/mongodb24/root/var/run/mongodb
logpath = /var/log/mongodb24-mongodb/mongodb.log
dbpath =/opt/rh/mongodb24/root/var/lib/mongodb
journal = true

# Enables periodic logging of CPU utilization and I/O wait
#cpu = true

# Turn on/off security.  Off is currently the default
#noauth = true
#auth = true

# Verbose logging output.
#verbose = true

# Inspect all client data for validity on receipt (useful for
Comment 14 Marek Skalický 2015-11-12 04:29:41 EST
(In reply to wewang from comment #12)
> I just confirm something ,before we can get /var/lib/mongodb/mongodb.conf in
> mongodb container, but now I just see 
> bash-4.2$ cd /var/lib/mongodb
> bash-4.2$ ls
> data  mongodb.pid

In this PR https://github.com/openshift/mongodb/pull/100 we changed the layout in the container to more reflect FHS.
So mongod is now using /etc/mongod.conf configuration file.

(In reply to wewang from comment #13)
> I got it 
>  bash-4.2$ more /opt/rh/mongodb24/root/etc/mongodb.conf

This file is preconfigured from RPM package. For examle rh-mongodb26 install different file.
Also running mongod in container needs different configuration, so new configuration is stored in /etc/mongod.conf (same location as used in upstream and Fedora)
Comment 15 errata-xmlrpc 2015-11-16 10:56:34 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2056.html

Note You need to log in before you can comment on or make changes to this bug.