Bug 1743706 - MongoDB initialization failed when fips is enabled with External MongoDB
Summary: MongoDB initialization failed when fips is enabled with External MongoDB
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.5.0
Hardware: x86_64
OS: Linux
high
medium vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Kersom
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-20 13:57 UTC by Kenny Tordeurs
Modified: 2020-03-05 18:48 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
Satellite Server operating in FIPS mode cannot connect to an external MongoDB server. To use FIPS mode and MongoDB, use the internal MongoDB.
Clone Of:
Environment:
Last Closed: 2019-10-18 14:29:10 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Pulp Redmine 5512 Normal CLOSED - CURRENTRELEASE Can't connect to remote mongodb when FIPS-enabled 2020-03-05 18:47:59 UTC

Description Kenny Tordeurs 2019-08-20 13:57:44 UTC
Description of problem:
When trying to run the installer when FIPS is enabled it fails on:
~~~
Database initialization failed: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
~~~

Version-Release number of selected component (if applicable):
satellite-6.5.2-1.el7sat.noarch
Red Hat Enterprise Linux Server release 7.7 (Maipo)

# cat /proc/sys/crypto/fips_enabled
~~~
1
~~~

How reproducible:
100%

Steps to Reproduce:
1. Enable FIPS
2. Try to run installer
3.

Actual results:

# satellite-installer --scenario satellite   --foreman-db-host sat65-postgres.sysmgmt.lan   --foreman-db-password Foreman_Password   --foreman-db-database foreman   --katello-candlepin-db-host sat65-postgres.sysmgmt.lan   --katello-candlepin-db-name candlepin   --katello-candlepin-db-password Candlepin_Password   --katello-candlepin-manage-db false   --katello-pulp-db-username pulp   --katello-pulp-db-password pulp_password   --katello-pulp-db-seeds sat65-mongodb.sysmgmt.lan:27017   --katello-pulp-db-name pulp_database --disable-system-checks 
~~~
Resetting puppet server version param...
 'pulp-manage-db && touch /var/lib/pulp/init.flag' returned 70 instead of one of [0]
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/errors.rb:157:in `fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/exec.rb:164:in `sync'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:239:in `sync'
...
 /Stage[main]/Pulp::Database/Exec[migrate_pulp_db]/returns: change from 'notrun' to ['0'] failed: 'pulp-manage-db && touch /var/lib/pulp/init.flag' returned 70 instead of one of [0]
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[sat65.sysmgmt.lan]: Could not evaluate: Exception Failed to open TCP connection to sat65.sysmgmt.lan:443 (Connection refused - connect(2) for "sat65.sysmgmt.lan" port 443) in get request to: https://sat65.sysmgmt.lan/api/v2/smart_proxies?search=name=%22sat65.sysmgmt.lan%22
...
~~~

# sudo -u apache pulp-manage-db
~~~
Attempting to connect to sat65-mongodb.sysmgmt.lan:27017
Attempting to connect to sat65-mongodb.sysmgmt.lan:27017
Write concern for Mongo connection: {}
Database initialization failed: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 207, in main
    connection.initialize(max_timeout=1)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/connection.py", line 160, in initialize
    _DATABASE = mongoengine.connection.get_db()
  File "/usr/lib/python2.7/site-packages/mongoengine/connection.py", line 152, in get_db
    source=conn_settings['authentication_source'])
  File "/usr/lib64/python2.7/site-packages/pymongo/database.py", line 1023, in authenticate
    connect=True)
  File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 423, in _cache_credentials
    sock_info.authenticate(credentials)
  File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 314, in authenticate
    auth.authenticate(credentials, self)
  File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 436, in authenticate
    auth_func(credentials, sock_info)
  File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 416, in _authenticate_default
    return _authenticate_scram_sha1(credentials, sock_info)
  File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 199, in _authenticate_scram_sha1
    salted_pass = _hi(_password_digest(username, password).encode("utf-8"),
  File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 245, in _password_digest
    md5hash = md5()
ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
~~~


Expected results:
No errors

Additional info:
After disabling FIPS the installation finishes without errors.

Comment 5 Evgeni Golov 2019-08-26 08:27:14 UTC
I *think* this is not 7.7 related, but the fact that a remote DB is used, testing now.

Comment 6 Kenny Tordeurs 2019-08-26 08:34:32 UTC
Testing local db:

[root@sat65 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.7 (Maipo)

[root@sat65 ~]# cat /proc/sys/crypto/fips_enabled
1

#  satellite-installer --scenario satellite --foreman-initial-organization "initial_organization_name" --foreman-initial-location "initial_location_name" --foreman-admin-username admin_user_name --foreman-admin-password admin_password --disable-system-checks
~~~
Resetting puppet server version param...
 'pulp-manage-db && touch /var/lib/pulp/init.flag' returned 70 instead of one of [0]
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/errors.rb:157:in `fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/exec.rb:164:in `sync'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:239:in `sync'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:134:in `sync_if_needed'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `block in perform_changes'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:87:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:87:in `perform_changes'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:21:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:259:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:279:in `eval_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:183:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:183:in `block (2 levels) in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:519:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:518:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:183:in `block in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:121:in `traverse'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:173:in `evaluate'
/usr/share/gems/gems/kafo-2.1.0/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:30:in `evaluate_with_trigger'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block (2 levels) in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:519:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:518:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `block
...
~~~


[root@sat65 ~]# sudo -u apache pulp-manage-db
~~~
Attempting to connect to sat65-mongodb.sysmgmt.lan:27017
Attempting to connect to sat65-mongodb.sysmgmt.lan:27017
Write concern for Mongo connection: {}
Database initialization failed: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 207, in main
    connection.initialize(max_timeout=1)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/connection.py", line 160, in initialize
    _DATABASE = mongoengine.connection.get_db()
  File "/usr/lib/python2.7/site-packages/mongoengine/connection.py", line 152, in get_db
    source=conn_settings['authentication_source'])
  File "/usr/lib64/python2.7/site-packages/pymongo/database.py", line 1023, in authenticate
    connect=True)
  File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 423, in _cache_credentials
    sock_info.authenticate(credentials)
  File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 314, in authenticate
    auth.authenticate(credentials, self)
  File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 436, in authenticate
    auth_func(credentials, sock_info)
  File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 416, in _authenticate_default
    return _authenticate_scram_sha1(credentials, sock_info)
  File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 199, in _authenticate_scram_sha1
    salted_pass = _hi(_password_digest(username, password).encode("utf-8"),
  File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 245, in _password_digest
    md5hash = md5()
ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
~~~

Comment 7 Evgeni Golov 2019-08-26 09:27:02 UTC
Kennys output still used a remote DB because he forgot to wipe the answers.

I tried this on 7.7 with 6.6.0 Snap 17 and it worked fine for a local DB. Waiting on confirmation from Kenny, but pretty sure about this.

Now let's look at the actual error:

  File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 436, in authenticate
    auth_func(credentials, sock_info)
  File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 416, in _authenticate_default
    return _authenticate_scram_sha1(credentials, sock_info)
  File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 199, in _authenticate_scram_sha1
    salted_pass = _hi(_password_digest(username, password).encode("utf-8"),
  File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 245, in _password_digest
    md5hash = md5()

Pulp uses mongoengine, which in turn uses pymongo to talk to MongoDB. We use pymongo 3.2 currently, which defaults to use SCRAM-SHA-1 when talking to a 3.x MongoDB (https://api.mongodb.com/python/current/api/pymongo/database.html#pymongo.database.Database.authenticate, https://github.com/mongodb/mongo-python-driver/blob/v3.3/pymongo/auth.py). However, even this method uses MD5 to hash the users password (https://github.com/mongodb/mongo-python-driver/blob/v3.3/pymongo/auth.py#L211) and that will fail in FIPS mode.

The only solution I see is to use SCRAM-SHA-256 that was implemented in pymongo 3.7 as that does not use MD5 at all.

However, as far as I can see, there is no way in pulp to configure a non-default authentication mechanism today (https://github.com/pulp/pulp/blob/2-master/server/etc/pulp/server.conf#L48-L60).

So, TL;DR:
1. we need to make Pulp allow passing the auth mech to mongoengine/pymongo
2. we need to update pymongo (and maybe mongoengine) to a version that supports SCRAM-SHA-256
3. we need installer support for the auth mech in Katello/Satellite

Given the installer is only step 3, re-assigning to Pulp and moving back to NEW.

Comment 8 Kenny Tordeurs 2019-08-26 09:46:29 UTC
Hi,

Please ignore comment 6 as the answers file still pointed to the external databases.
I did the test once more after wiping all the answers.

So when using local DB's with FIPS it works.


[root@sat65 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)

[root@sat65 ~]# cat /proc/sys/crypto/fips_enabled
1


[root@sat65 ~]# satellite-installer --scenario satellite --foreman-initial-organization "initial_organization_name" --foreman-initial-location "initial_location_name" --foreman-admin-username admin_user_name --foreman-admin-password admin_password --disable-system-checks
Resetting puppet server version param...
Installing             Done                                               [100%] [................................................................................................................................]
  Success!
  * Satellite is running at https://sat65.sysmgmt.lan
      Initial credentials are admin_user_name / admin_password

  * To install an additional Capsule on separate machine continue by running:

      capsule-certs-generate --foreman-proxy-fqdn "$CAPSULE" --certs-tar "/root/$CAPSULE-certs.tar"

  * To upgrade an existing 6.4 Capsule to 6.5:
      Please see official documentation for steps and parameters to use when upgrading a 6.4 Capsule to 6.5.

  The full log is at /var/log/foreman-installer/satellite.log
yum install -y -q rh-mongodb34-syspaths finished successfully!

Comment 9 Evgeni Golov 2019-08-26 10:46:37 UTC
there is now a doc bz open for that too: https://bugzilla.redhat.com/show_bug.cgi?id=1745511

Comment 10 Grant Gainey 2019-08-28 18:52:14 UTC
(In reply to Evgeni Golov from comment #7)
> Kennys output still used a remote DB because he forgot to wipe the answers.
> 
> I tried this on 7.7 with 6.6.0 Snap 17 and it worked fine for a local DB.
> Waiting on confirmation from Kenny, but pretty sure about this.
> 
> Now let's look at the actual error:
> 
>   File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 436, in
> authenticate
>     auth_func(credentials, sock_info)
>   File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 416, in
> _authenticate_default
>     return _authenticate_scram_sha1(credentials, sock_info)
>   File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 199, in
> _authenticate_scram_sha1
>     salted_pass = _hi(_password_digest(username, password).encode("utf-8"),
>   File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 245, in
> _password_digest
>     md5hash = md5()
> 
> Pulp uses mongoengine, which in turn uses pymongo to talk to MongoDB. We use
> pymongo 3.2 currently, which defaults to use SCRAM-SHA-1 when talking to a
> 3.x MongoDB
> (https://api.mongodb.com/python/current/api/pymongo/database.html#pymongo.
> database.Database.authenticate,
> https://github.com/mongodb/mongo-python-driver/blob/v3.3/pymongo/auth.py).
> However, even this method uses MD5 to hash the users password
> (https://github.com/mongodb/mongo-python-driver/blob/v3.3/pymongo/auth.
> py#L211) and that will fail in FIPS mode.

If 'hashing using MD5' causes FIPS to fail (which, yeah, it should), how is it working in the local-mongo case? I think I'm missing something here.

Comment 11 Grant Gainey 2019-08-28 19:34:26 UTC
(In reply to Evgeni Golov from comment #7)
> 
> The only solution I see is to use SCRAM-SHA-256 that was implemented in
> pymongo 3.7 as that does not use MD5 at all.

Sat6 is currently on Mongo3.4, with 3.6 at least available in SCL. Looking at the docs, I don't think Mongo itself supports SCRAM-SHA-256 prior to 4.x:

https://docs.mongodb.com/v3.6/reference/parameters/#param.authenticationMechanisms
https://docs.mongodb.com/v4.0/reference/parameters/#authentication-parameters

Moving to 4.0 is probably a non-trivial exercise.

Thoughts, anyone?

Comment 12 Robin Chan 2019-09-03 14:42:31 UTC
(In reply to Grant Gainey from comment #11)
> (In reply to Evgeni Golov from comment #7)
> > 
> > The only solution I see is to use SCRAM-SHA-256 that was implemented in
> > pymongo 3.7 as that does not use MD5 at all.
> 
> Sat6 is currently on Mongo3.4, with 3.6 at least available in SCL. Looking
> at the docs, I don't think Mongo itself supports SCRAM-SHA-256 prior to 4.x:
> 
> https://docs.mongodb.com/v3.6/reference/parameters/#param.
> authenticationMechanisms
> https://docs.mongodb.com/v4.0/reference/parameters/#authentication-parameters
> 
> Moving to 4.0 is probably a non-trivial exercise.
> 
> Thoughts, anyone?

Another wrinkle is that Sat would have to move to a version of Mongo 4.x released prior to the license changes since the SCL takes care of this for us for the Mongo 3.4 & 3.6 versions.

Comment 13 Evgeni Golov 2019-09-06 09:36:31 UTC
(In reply to Grant Gainey from comment #10)
> (In reply to Evgeni Golov from comment #7)
> > Kennys output still used a remote DB because he forgot to wipe the answers.
> > 
> > I tried this on 7.7 with 6.6.0 Snap 17 and it worked fine for a local DB.
> > Waiting on confirmation from Kenny, but pretty sure about this.
> > 
> > Now let's look at the actual error:
> > 
> >   File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 436, in
> > authenticate
> >     auth_func(credentials, sock_info)
> >   File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 416, in
> > _authenticate_default
> >     return _authenticate_scram_sha1(credentials, sock_info)
> >   File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 199, in
> > _authenticate_scram_sha1
> >     salted_pass = _hi(_password_digest(username, password).encode("utf-8"),
> >   File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 245, in
> > _password_digest
> >     md5hash = md5()
> > 
> > Pulp uses mongoengine, which in turn uses pymongo to talk to MongoDB. We use
> > pymongo 3.2 currently, which defaults to use SCRAM-SHA-1 when talking to a
> > 3.x MongoDB
> > (https://api.mongodb.com/python/current/api/pymongo/database.html#pymongo.
> > database.Database.authenticate,
> > https://github.com/mongodb/mongo-python-driver/blob/v3.3/pymongo/auth.py).
> > However, even this method uses MD5 to hash the users password
> > (https://github.com/mongodb/mongo-python-driver/blob/v3.3/pymongo/auth.
> > py#L211) and that will fail in FIPS mode.
> 
> If 'hashing using MD5' causes FIPS to fail (which, yeah, it should), how is
> it working in the local-mongo case? I think I'm missing something here.

We don't do auth against local mongodb at all, so no auth is more secure than md5-based auth if you ask FIPS ;-)

Comment 14 pulp-infra@redhat.com 2019-09-27 17:01:52 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 15 pulp-infra@redhat.com 2019-09-27 17:01:54 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 16 pulp-infra@redhat.com 2019-09-27 18:01:44 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 17 pulp-infra@redhat.com 2019-10-15 18:31:52 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 19 Mike McCune 2019-10-18 14:29:10 UTC
Satellite 6 does not have plans to support FIPS mode for disconnected MongoDB configuration. This requires extensive changes in our installer and offers limited performance benefits.

Future versions of Satellite will not be using MongoDB so the lifespan for external Mongo databases is limited and we recommend embedded for the majority of deployments.

If the closing of this bug is problematic, feel free to re-open with justification around the needs of running Satellite 6 with an external MongoDB with FIPS.

Comment 20 pulp-infra@redhat.com 2020-01-07 12:31:59 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 21 pulp-infra@redhat.com 2020-02-27 17:02:38 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 22 pulp-infra@redhat.com 2020-03-05 18:48:00 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.


Note You need to log in before you can comment on or make changes to this bug.