Bug 1182335

This problem is related to the introduction of mongoengine. The connection takes a username and password so you would think it would be used. However the get_db() is where the authentication actually occurs and that doesn't seem to be getting called.

PR available:  https://github.com/pulp/pulp/pull/1528

QA, you should configure mongoDB to require username and password authentication and put those same settings in the [database] section of server.conf.

Then verify two things:
1. Verify that pulp start up normally with the correct username and password.
2. Verify that pulp will not start correctly with an incorrect username or password.

Merged 2.6-testing -> 2.6-dev -> master

pulp 2.6.0 beta 5

Hi all,

I've manually applied this fix to my affected test server and it has resolved the issue.

Thanks!
Ben Roberts

Created attachment 987117 [details]
possible list and create repos, but not delete when database pass is wrong on localhost

checked on hostname and localhost..  
on hostname (1) evrth is fine, but smth weird is with localhost (2). 
On localhost when the database pass is wrong in server.conf it returns connection Fail. However, it's possible to list or create repos, but not to delete them and not to migrate a database

>> rpm -qa pulp-server
pulp-server-2.6.0-0.5.beta.el6.noarch

************************
(1) On HostName, for a wrong pass in [database] server.conf
************************

>>  for s in {qpidd,pulp_celerybeat,pulp_resource_manager,pulp_workers,httpd}; do sudo service $s restart; done;
Stopping Qpid AMQP daemon:                                 [  OK  ]
Starting Qpid AMQP daemon:                                 [  OK  ]
celery init v10.0.
Using configuration: /etc/default/pulp_workers, /etc/default/pulp_celerybeat
Restarting celery periodic task scheduler
Stopping pulp_celerybeat... OK
Starting pulp_celerybeat...
celery init v10.0.
Using config script: /etc/default/pulp_resource_manager
celery multi v3.1.11 (Cipater)
> Stopping nodes...
	> resource_manager@host: QUIT -> 30013
> Waiting for 1 node -> 30013.....
	> resource_manager@host: OK
> Restarting node resource_manager@host: No handlers could be found for logger "pulp.server.db.connection"
Traceback (most recent call last):
  File "/usr/lib64/python2.6/runpy.py", line 122, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.6/runpy.py", line 34, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.6/site-packages/celery/__main__.py", line 54, in <module>
    main()
  File "/usr/lib/python2.6/site-packages/celery/__main__.py", line 30, in main
    main()
  File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 81, in main
    cmd.execute_from_commandline(argv)
  File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 769, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/usr/lib/python2.6/site-packages/celery/bin/base.py", line 304, in execute_from_commandline
    argv = self.setup_app_from_commandline(argv)
  File "/usr/lib/python2.6/site-packages/celery/bin/base.py", line 464, in setup_app_from_commandline
    self.app = self.find_app(app)
  File "/usr/lib/python2.6/site-packages/celery/bin/base.py", line 484, in find_app
    return find_app(app, symbol_by_name=self.symbol_by_name)
  File "/usr/lib/python2.6/site-packages/celery/app/utils.py", line 225, in find_app
    sym = imp(app)
  File "/usr/lib/python2.6/site-packages/celery/utils/imports.py", line 101, in import_from_cwd
    return imp(module, package=package)
  File "/usr/lib/python2.6/site-packages/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/usr/lib/python2.6/site-packages/pulp/server/async/app.py", line 5, in <module>
    from pulp.server import initialization
  File "/usr/lib/python2.6/site-packages/pulp/server/initialization.py", line 9, in <module>
    db_connection.initialize()
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", line 117, in initialize
    raise RuntimeError(msg)
RuntimeError: Authentication to MongoDB with username and password failed.
* Child terminated with errorcode 255
FAILED

celery init v10.0.
Using config script: /etc/default/pulp_workers
celery multi v3.1.11 (Cipater)
> Stopping nodes...
	> reserved_resource_worker-0@host: QUIT -> 30209
> Waiting for 1 node -> 30209.....
	> reserved_resource_worker-0@host: OK
> Restarting node reserved_resource_worker-0@host: No handlers could be found for logger "pulp.server.db.connection"
Traceback (most recent call last):
  File "/usr/lib64/python2.6/runpy.py", line 122, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.6/runpy.py", line 34, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.6/site-packages/celery/__main__.py", line 54, in <module>
    main()
  File "/usr/lib/python2.6/site-packages/celery/__main__.py", line 30, in main
    main()
  File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 81, in main
    cmd.execute_from_commandline(argv)
  File "/usr/lib/python2.6/site-packages/celery/bin/celery.py", line 769, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/usr/lib/python2.6/site-packages/celery/bin/base.py", line 304, in execute_from_commandline
    argv = self.setup_app_from_commandline(argv)
  File "/usr/lib/python2.6/site-packages/celery/bin/base.py", line 464, in setup_app_from_commandline
    self.app = self.find_app(app)
  File "/usr/lib/python2.6/site-packages/celery/bin/base.py", line 484, in find_app
    return find_app(app, symbol_by_name=self.symbol_by_name)
  File "/usr/lib/python2.6/site-packages/celery/app/utils.py", line 225, in find_app
    sym = imp(app)
  File "/usr/lib/python2.6/site-packages/celery/utils/imports.py", line 101, in import_from_cwd
    return imp(module, package=package)
  File "/usr/lib/python2.6/site-packages/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/usr/lib/python2.6/site-packages/pulp/server/async/app.py", line 5, in <module>
    from pulp.server import initialization
  File "/usr/lib/python2.6/site-packages/pulp/server/initialization.py", line 9, in <module>
    db_connection.initialize()
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", line 117, in initialize
    raise RuntimeError(msg)
RuntimeError: Authentication to MongoDB with username and password failed.
* Child terminated with errorcode 255
FAILED

Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]

>> pulp-admin -u admin -p admin rpm repo create --repo-id rybka
There was an internal server error while trying to access the Pulp application.
One possible cause is that the database needs to be migrated to the latest
version. If this is the case, run pulp-manage-db and restart the services. More
information may be found in Apache's log.

>> pulp-admin -u admin -p admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

There was an internal server error while trying to access the Pulp application.
One possible cause is that the database needs to be migrated to the latest
version. If this is the case, run pulp-manage-db and restart the services. More
information may be found in Apache's log.

>> sudo -u apache pulp-manage-db 
Database initialization failed: Authentication to MongoDB with username and password failed.
Authentication to MongoDB with username and password failed.
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/db/manage.py", line 124, in main
    connection.initialize(max_timeout=1)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", line 117, in initialize
    raise RuntimeError(msg)
RuntimeError: Authentication to MongoDB with username and password failed.


************************
(2) On LocalHost, for a wrong pass in [database] server.conf
************************

When the database pass is wrong in server.conf, it's possible to list and create repos, but not delete them

In additional it can't migrate database, but why is it possible to create repos??
>> sudo -u apache pulp-manage-db 
Retruns 


Database initialization failed: Authentication to MongoDB with username and password failed.
Authentication to MongoDB with username and password failed.
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/db/manage.py", line 124, in main
    connection.initialize(max_timeout=1)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", line 117, in initialize
    raise RuntimeError(msg)
RuntimeError: Authentication to MongoDB with username and password failed.


>> 

Please see attachment. 

Brian, please let me know what to do about that?

Mongodb has some strange behaviors where even though auth is required in the config localhost connection get to bypass auth. This sounds like what is occurring here because a non-localhost connection behaves as expected and a localhost behaves strange.

My recommendation is to investigate the mongodb configuraiton independant of Pulp. Use the mongo CLI client to connect using localhost and non-localhost. Also turn up the logging on mongo. Only if you do that will you see the bypass statements if they are occurring. Send back some posts with database user configs in mongo along with the output of mongo CLI commands. When you do connect with the CLI, listing the collections in that database and doing a find() on the documents of a random collection are good things to do to demonstrate you have the necessary priviledges.

Try to verify that a correct username and password allows access on both localhost and a remote host. Also verify that an incorrect username and password does NOT allow access from both localhost and a remote host.

Brian, here is the investigation of the mongodb configuration independent of Pulp:

>> grep bind_ip /etc/mongodb.conf 
bind_ip = ip-XXXXXX.internal
#bind_ip = localhost

>> mongo --host `hostname` pulp_database -u cheburashka -p 
MongoDB shell version: 2.4.12
Enter password: 
connecting to: XXXX:27017/pulp_database

> db.repos.find()
{ "_id" : ObjectId("54c110f2542c8e720b16cc63"), "_ns" : "repos", "content_unit_counts" : { "rpm" : 1 }, "description" : null, "display_name" : "unicode_problem", "id" : "unicode_problem", "last_unit_added" : ISODate("2015-01-22T15:02:51.813Z"), "last_unit_removed" : null, "notes" : { "_repo-type" : "rpm-repo" }, "scratchpad" : {  } }
{ "_id" : ObjectId("54c7b75b542c8e1094d4fe43"), "scratchpad" : {  }, "display_name" : "test-repo", "description" : null, "last_unit_added" : null, "notes" : { "_repo-type" : "rpm-repo" }, "last_unit_removed" : null, "content_unit_counts" : {  }, "_ns" : "repos", "id" : "test-repo" }
{ "_id" : ObjectId("54c7bc82542c8e140acfa47a"), "scratchpad" : {  }, "display_name" : "test-repo2", "description" : null, "last_unit_added" : null, "notes" : { "_repo-type" : "rpm-repo" }, "last_unit_removed" : null, "content_unit_counts" : {  }, "_ns" : "repos", "id" : "test-repo2" }
{ "_id" : ObjectId("54cf7fa5542c8e6f24210181"), "scratchpad" : {  }, "display_name" : "rybka", "description" : null, "last_unit_added" : null, "notes" : { "_repo-type" : "rpm-repo" }, "last_unit_removed" : null, "content_unit_counts" : {  }, "_ns" : "repos", "id" : "rybka" }
{ "_id" : ObjectId("54cf802c542c8e70865d2b7e"), "scratchpad" : {  }, "display_name" : "rybka2", "description" : null, "last_unit_added" : null, "notes" : { "_repo-type" : "rpm-repo" }, "last_unit_removed" : null, "content_unit_counts" : {  }, "_ns" : "repos", "id" : "rybka2" }
{ "_id" : ObjectId("54cfba90542c8e05fb96c38d"), "_ns" : "repos", "content_unit_counts" : { "erratum" : 4, "package_category" : 1, "package_group" : 2, "rpm" : 32 }, "description" : null, "display_name" : "zoo", "id" : "zoo", "last_unit_added" : ISODate("2015-02-02T17:58:02.562Z"), "last_unit_removed" : null, "notes" : { "_repo-type" : "rpm-repo" }, "scratchpad" : { "checksum_type" : "sha256" } }
{ "_id" : ObjectId("54cfbb0f542c8e05fdb138eb"), "scratchpad" : {  }, "display_name" : "lion", "description" : null, "last_unit_added" : null, "notes" : { "_repo-type" : "rpm-repo" }, "last_unit_removed" : null, "content_unit_counts" : {  }, "_ns" : "repos", "id" : "lion" }

>> mongo --host `hostname` pulp_database -u cheburashka -p wrong_pass
MongoDB shell version: 2.4.12
connecting to: XXXX:27017/pulp_database
Wed Feb  4 06:56:34.113 Error: 18 { code: 18, ok: 0.0, errmsg: "auth fails" } at src/mongo/shell/db.js:228
exception: login failed


>> grep bind_ip /etc/mongodb.conf 
#bind_ip = ip-XXXXXX.internal
bind_ip = localhost

>> service mongod restart
Stopping mongod:                                           [  OK  ]
Starting mongod:                                           [  OK  ]

>> mongo --host localhost  pulp_database -u cheburashka -p 
MongoDB shell version: 2.4.12
Enter password: 
connecting to: localhost:27017/pulp_database

> db.repos.find()
{ "_id" : ObjectId("54c110f2542c8e720b16cc63"), "_ns" : "repos", "content_unit_counts" : { "rpm" : 1 }, "description" : null, "display_name" : "unicode_problem", "id" : "unicode_problem", "last_unit_added" : ISODate("2015-01-22T15:02:51.813Z"), "last_unit_removed" : null, "notes" : { "_repo-type" : "rpm-repo" }, "scratchpad" : {  } }
{ "_id" : ObjectId("54c7b75b542c8e1094d4fe43"), "scratchpad" : {  }, "display_name" : "test-repo", "description" : null, "last_unit_added" : null, "notes" : { "_repo-type" : "rpm-repo" }, "last_unit_removed" : null, "content_unit_counts" : {  }, "_ns" : "repos", "id" : "test-repo" }
{ "_id" : ObjectId("54c7bc82542c8e140acfa47a"), "scratchpad" : {  }, "display_name" : "test-repo2", "description" : null, "last_unit_added" : null, "notes" : { "_repo-type" : "rpm-repo" }, "last_unit_removed" : null, "content_unit_counts" : {  }, "_ns" : "repos", "id" : "test-repo2" }
{ "_id" : ObjectId("54cf7fa5542c8e6f24210181"), "scratchpad" : {  }, "display_name" : "rybka", "description" : null, "last_unit_added" : null, "notes" : { "_repo-type" : "rpm-repo" }, "last_unit_removed" : null, "content_unit_counts" : {  }, "_ns" : "repos", "id" : "rybka" }
{ "_id" : ObjectId("54cf802c542c8e70865d2b7e"), "scratchpad" : {  }, "display_name" : "rybka2", "description" : null, "last_unit_added" : null, "notes" : { "_repo-type" : "rpm-repo" }, "last_unit_removed" : null, "content_unit_counts" : {  }, "_ns" : "repos", "id" : "rybka2" }
{ "_id" : ObjectId("54cfba90542c8e05fb96c38d"), "_ns" : "repos", "content_unit_counts" : { "erratum" : 4, "package_category" : 1, "package_group" : 2, "rpm" : 32 }, "description" : null, "display_name" : "zoo", "id" : "zoo", "last_unit_added" : ISODate("2015-02-02T17:58:02.562Z"), "last_unit_removed" : null, "notes" : { "_repo-type" : "rpm-repo" }, "scratchpad" : { "checksum_type" : "sha256" } }
{ "_id" : ObjectId("54cfbb0f542c8e05fdb138eb"), "scratchpad" : {  }, "display_name" : "lion", "description" : null, "last_unit_added" : null, "notes" : { "_repo-type" : "rpm-repo" }, "last_unit_removed" : null, "content_unit_counts" : {  }, "_ns" : "repos", "id" : "lion" }

> mongo --host localhost  pulp_database -u cheburashka -p wrong_pass
MongoDB shell version: 2.4.12
connecting to: localhost:27017/pulp_database
Wed Feb  4 07:01:04.457 Error: 18 { code: 18, ok: 0.0, errmsg: "auth fails" } at src/mongo/shell/db.js:228
exception: login failed


Then setting localhost and wrong pass in conf file, and rebooting (to ensure that all services were restarted) I can list and create repos, but not delete or for example publish them:

>> pulp-admin -u admin -p admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                  unicode_problem
Display Name:        unicode_problem
Description:         None
Content Unit Counts: 
  Rpm: 1

Id:                  test-repo
Display Name:        test-repo
Description:         None
Content Unit Counts: 

Id:                  test-repo2
Display Name:        test-repo2
Description:         None
Content Unit Counts: 

Id:                  rybka
Display Name:        rybka
Description:         None
Content Unit Counts: 

Id:                  rybka2
Display Name:        rybka2
Description:         None
Content Unit Counts: 

Id:                  zoo
Display Name:        zoo
Description:         None
Content Unit Counts: 
  Erratum:          4
  Package Category: 1
  Package Group:    2
  Rpm:              32

Id:                  lion
Display Name:        lion
Description:         None
Content Unit Counts: 

>> pulp-admin login -u admin -p admin 
Successfully logged in. Session certificate will expire at Feb 11 12:19:46 2015
GMT.

>>  pulp-admin rpm repo create --repo-id krevetka
Successfully created repository [krevetka]

>> pulp-admin rpm repo delete --repo-id krevetka
This command may be exited via ctrl+c without affecting the request.

[-]
Running...
[-]
Waiting to begin...
^C 

coz it runs forever

Publish also doesn't work:

>> pulp-admin rpm repo publish run --repo-id krevetka
+----------------------------------------------------------------------+
                    Publishing Repository [krevetka]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


[|]
Waiting to begin...
^C[

Runs forever.

I agree it seems that authentication is being required, but I suspect that mongodb is bypassing authentication for Pulp. If mongodb were requiring authentication for Pulp then there is no way it could read anything without having proper credentials.

Can you list the users in the 'pulp_database' collection? Also please list the users in the 'admin' collection?

Three other differences from what I tried and what you'e done. (1) I didn't test 'mongo' with --host. (2) I don't set bind_ip in the mongodb settings file. (3) I require authentication with `auth=true` in the mongodb settings file.

(In reply to bbouters from comment #11)
> I agree it seems that authentication is being required, but I suspect that
> mongodb is bypassing authentication for Pulp. If mongodb were requiring
> authentication for Pulp then there is no way it could read anything without
> having proper credentials.
> 
> Can you list the users in the 'pulp_database' collection? 

notice, I don't use --host in the following command and in settings mongodb.conf I have bind_ip on localhost

$ mongo pulp_database -u cheburashka -p
MongoDB shell version: 2.4.12
Enter password: 
connecting to: pulp_database
> show users
{
	"_id" : ObjectId("54c758c2bdc32f83e3d4ea83"),
	"pwd" : "XXX",
	"roles" : [
		"readWrite",
		"dbAdmin"
	],
	"user" : "cheburashka"
}
{
	"_id" : ObjectId("54cf6edc01e17bd54572d44e"),
	"user" : "admin",
	"pwd" : "XXX",
	"roles" : [
		"dbAdmin",
		"dbOwnder",
		"userAdmin"
	]
}



so there are 2 users, but the issue I have with localhost was also there when there was only one user "cheburashka". 


Also please list
> the users in the 'admin' collection?
> 

$ use admin
switched to db admin
> show users
> db.getCollectionNames()
[]






> Three other differences from what I tried and what you'e done. (1) I didn't
> test 'mongo' with --host. 

so as you can see above mongo works with no --host option and good pass and bad pass is here: 

$ mongo pulp_database -u cheburashka -p wrong_pass
MongoDB shell version: 2.4.12
connecting to: pulp_database
Wed Feb  4 13:31:25.824 

> (2) I don't set bind_ip in the mongodb settings

Do you use fancy 0.0.0.0 IP?

> file. (3) I require authentication with `auth=true` in the mongodb settings
> file.


$ sudo cat /etc/mongodb.conf | grep  auth
#noauth = true
auth = true

When you switch to the db named 'admin' and show users you see no users; that is a problem. I ran into this same problem where even though I configured auth when Pulp would connect to it via localhost it would be bypassed. I'm not a mongoDB expert and the mongodb 2.4 docs were not very helpful in this area so I had to go to #mongodb on freenode to learn how to solve this mongodb problem. They told me that localhost connections bypass authentication without a user present in the 'admin' database. This is almost 100% completely undocumented in mongodb which is scary. This is almost certainly what is allowing Pulp to bypass authentication.

If Pulp lacks proper mongodb credentials, Pulp cannot do anything to bypass mongoDB authentication and authorization. Therefore any unexpected read or write access is due to mongodb allowing connections through without auth. Because of this, the resolution to this strange behavior is in configuring mongodb differently.

My server.conf is almost 100% default except having `auth = true` added. I just looked and it has 'bind_ip = 127.0.0.1'.

Brian, right, adding user to db named 'admin' solved the issue. Thank you.

Counting all previous debugging around this BZ I would move it to verified. Just one more question. Do you consider it's worth to document the required presence of users in 'admin' db to prevent unexpected access to Pulp database? Well, even if it's (not) documented in mongo docs, shouldn't we secure that from Pulp side? 

>> mongo
MongoDB shell version: 2.4.12
connecting to: test
> use admin
switched to db admin
> db.addUser({ user: "test-user", pwd: "pass", roles: ["readWrite", "dbAdmin"]})
{
	"user" : "test-user",
	"pwd" : "crypted_pass",
	"roles" : [
		"readWrite",
		"dbAdmin"
	],
	"_id" : ObjectId("54d32fa9b3c04aaa534143ae")
}

restart all services
wrong pass, localhost:

>> pulp-admin -u admin -p admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

There was an internal server error while trying to access the Pulp application.
One possible cause is that the database needs to be migrated to the latest
version. If this is the case, run pulp-manage-db and restart the services. More
information may be found in Apache's log.

>> pulp-admin login -u admin -p admin
There was an internal server error while trying to access the Pulp application.
One possible cause is that the database needs to be migrated to the latest
version. If this is the case, run pulp-manage-db and restart the services. More
information may be found in Apache's log.

>> sudo -u apache pulp-manage-db
Database initialization failed: Authentication to MongoDB with username and password failed.
Authentication to MongoDB with username and password failed.
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/db/manage.py", line 124, in main
    connection.initialize(max_timeout=1)
  File "/usr/lib/python2.6/site-packages/pulp/server/db/connection.py", line 117, in initialize
    raise RuntimeError(msg)
RuntimeError: Authentication to MongoDB with username and password failed.

To add: I checked that with several users in 'admin' db, like the same in 'pulp_database', with the same pass, and wrong pass, and with user named admin.. evrth was fine.

Thanks for taking so much care in verifying this. I'm glad it produced the correct behavior.

I do not feel Pulp should make recommendations on the configuration of mongodb administration. We do document Pulp's required mongodb configuration for the Pulp database, but configuring authorization for administering mongodb itself is not an area where Pulp can provide a recommendation.

I think the examples here provide a good record of the issue and the successful config to have our teams avoid doing mongoDB-auth-config-gymnastics again.

A documentation bug could be filed against mongodb for this issue with their 2.4 docs. I do think this behavior is not well documented and that is a mongodb community problem.

Moved to https://pulp.plan.io/issues/660