Description of problem: When trying to run the installer when FIPS is enabled it fails on: ~~~ Database initialization failed: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips ~~~ Version-Release number of selected component (if applicable): satellite-6.5.2-1.el7sat.noarch Red Hat Enterprise Linux Server release 7.7 (Maipo) # cat /proc/sys/crypto/fips_enabled ~~~ 1 ~~~ How reproducible: 100% Steps to Reproduce: 1. Enable FIPS 2. Try to run installer 3. Actual results: # satellite-installer --scenario satellite --foreman-db-host sat65-postgres.sysmgmt.lan --foreman-db-password Foreman_Password --foreman-db-database foreman --katello-candlepin-db-host sat65-postgres.sysmgmt.lan --katello-candlepin-db-name candlepin --katello-candlepin-db-password Candlepin_Password --katello-candlepin-manage-db false --katello-pulp-db-username pulp --katello-pulp-db-password pulp_password --katello-pulp-db-seeds sat65-mongodb.sysmgmt.lan:27017 --katello-pulp-db-name pulp_database --disable-system-checks ~~~ Resetting puppet server version param... 'pulp-manage-db && touch /var/lib/pulp/init.flag' returned 70 instead of one of [0] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/errors.rb:157:in `fail' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/exec.rb:164:in `sync' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:239:in `sync' ... /Stage[main]/Pulp::Database/Exec[migrate_pulp_db]/returns: change from 'notrun' to ['0'] failed: 'pulp-manage-db && touch /var/lib/pulp/init.flag' returned 70 instead of one of [0] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[sat65.sysmgmt.lan]: Could not evaluate: Exception Failed to open TCP connection to sat65.sysmgmt.lan:443 (Connection refused - connect(2) for "sat65.sysmgmt.lan" port 443) in get request to: https://sat65.sysmgmt.lan/api/v2/smart_proxies?search=name=%22sat65.sysmgmt.lan%22 ... ~~~ # sudo -u apache pulp-manage-db ~~~ Attempting to connect to sat65-mongodb.sysmgmt.lan:27017 Attempting to connect to sat65-mongodb.sysmgmt.lan:27017 Write concern for Mongo connection: {} Database initialization failed: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 207, in main connection.initialize(max_timeout=1) File "/usr/lib/python2.7/site-packages/pulp/server/db/connection.py", line 160, in initialize _DATABASE = mongoengine.connection.get_db() File "/usr/lib/python2.7/site-packages/mongoengine/connection.py", line 152, in get_db source=conn_settings['authentication_source']) File "/usr/lib64/python2.7/site-packages/pymongo/database.py", line 1023, in authenticate connect=True) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 423, in _cache_credentials sock_info.authenticate(credentials) File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 314, in authenticate auth.authenticate(credentials, self) File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 436, in authenticate auth_func(credentials, sock_info) File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 416, in _authenticate_default return _authenticate_scram_sha1(credentials, sock_info) File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 199, in _authenticate_scram_sha1 salted_pass = _hi(_password_digest(username, password).encode("utf-8"), File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 245, in _password_digest md5hash = md5() ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips ~~~ Expected results: No errors Additional info: After disabling FIPS the installation finishes without errors.
I *think* this is not 7.7 related, but the fact that a remote DB is used, testing now.
Testing local db: [root@sat65 ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.7 (Maipo) [root@sat65 ~]# cat /proc/sys/crypto/fips_enabled 1 # satellite-installer --scenario satellite --foreman-initial-organization "initial_organization_name" --foreman-initial-location "initial_location_name" --foreman-admin-username admin_user_name --foreman-admin-password admin_password --disable-system-checks ~~~ Resetting puppet server version param... 'pulp-manage-db && touch /var/lib/pulp/init.flag' returned 70 instead of one of [0] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/errors.rb:157:in `fail' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/exec.rb:164:in `sync' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:239:in `sync' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:134:in `sync_if_needed' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `block in perform_changes' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:87:in `each' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:87:in `perform_changes' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:21:in `evaluate' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:259:in `apply' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:279:in `eval_resource' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:183:in `call' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:183:in `block (2 levels) in evaluate' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:519:in `block in thinmark' /opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:518:in `thinmark' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:183:in `block in evaluate' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:121:in `traverse' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:173:in `evaluate' /usr/share/gems/gems/kafo-2.1.0/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:30:in `evaluate_with_trigger' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block (2 levels) in apply' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:519:in `block in thinmark' /opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:518:in `thinmark' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `block ... ~~~ [root@sat65 ~]# sudo -u apache pulp-manage-db ~~~ Attempting to connect to sat65-mongodb.sysmgmt.lan:27017 Attempting to connect to sat65-mongodb.sysmgmt.lan:27017 Write concern for Mongo connection: {} Database initialization failed: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 207, in main connection.initialize(max_timeout=1) File "/usr/lib/python2.7/site-packages/pulp/server/db/connection.py", line 160, in initialize _DATABASE = mongoengine.connection.get_db() File "/usr/lib/python2.7/site-packages/mongoengine/connection.py", line 152, in get_db source=conn_settings['authentication_source']) File "/usr/lib64/python2.7/site-packages/pymongo/database.py", line 1023, in authenticate connect=True) File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 423, in _cache_credentials sock_info.authenticate(credentials) File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 314, in authenticate auth.authenticate(credentials, self) File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 436, in authenticate auth_func(credentials, sock_info) File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 416, in _authenticate_default return _authenticate_scram_sha1(credentials, sock_info) File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 199, in _authenticate_scram_sha1 salted_pass = _hi(_password_digest(username, password).encode("utf-8"), File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 245, in _password_digest md5hash = md5() ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips ~~~
Kennys output still used a remote DB because he forgot to wipe the answers. I tried this on 7.7 with 6.6.0 Snap 17 and it worked fine for a local DB. Waiting on confirmation from Kenny, but pretty sure about this. Now let's look at the actual error: File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 436, in authenticate auth_func(credentials, sock_info) File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 416, in _authenticate_default return _authenticate_scram_sha1(credentials, sock_info) File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 199, in _authenticate_scram_sha1 salted_pass = _hi(_password_digest(username, password).encode("utf-8"), File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 245, in _password_digest md5hash = md5() Pulp uses mongoengine, which in turn uses pymongo to talk to MongoDB. We use pymongo 3.2 currently, which defaults to use SCRAM-SHA-1 when talking to a 3.x MongoDB (https://api.mongodb.com/python/current/api/pymongo/database.html#pymongo.database.Database.authenticate, https://github.com/mongodb/mongo-python-driver/blob/v3.3/pymongo/auth.py). However, even this method uses MD5 to hash the users password (https://github.com/mongodb/mongo-python-driver/blob/v3.3/pymongo/auth.py#L211) and that will fail in FIPS mode. The only solution I see is to use SCRAM-SHA-256 that was implemented in pymongo 3.7 as that does not use MD5 at all. However, as far as I can see, there is no way in pulp to configure a non-default authentication mechanism today (https://github.com/pulp/pulp/blob/2-master/server/etc/pulp/server.conf#L48-L60). So, TL;DR: 1. we need to make Pulp allow passing the auth mech to mongoengine/pymongo 2. we need to update pymongo (and maybe mongoengine) to a version that supports SCRAM-SHA-256 3. we need installer support for the auth mech in Katello/Satellite Given the installer is only step 3, re-assigning to Pulp and moving back to NEW.
Hi, Please ignore comment 6 as the answers file still pointed to the external databases. I did the test once more after wiping all the answers. So when using local DB's with FIPS it works. [root@sat65 ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.7 (Maipo) [root@sat65 ~]# cat /proc/sys/crypto/fips_enabled 1 [root@sat65 ~]# satellite-installer --scenario satellite --foreman-initial-organization "initial_organization_name" --foreman-initial-location "initial_location_name" --foreman-admin-username admin_user_name --foreman-admin-password admin_password --disable-system-checks Resetting puppet server version param... Installing Done [100%] [................................................................................................................................] Success! * Satellite is running at https://sat65.sysmgmt.lan Initial credentials are admin_user_name / admin_password * To install an additional Capsule on separate machine continue by running: capsule-certs-generate --foreman-proxy-fqdn "$CAPSULE" --certs-tar "/root/$CAPSULE-certs.tar" * To upgrade an existing 6.4 Capsule to 6.5: Please see official documentation for steps and parameters to use when upgrading a 6.4 Capsule to 6.5. The full log is at /var/log/foreman-installer/satellite.log yum install -y -q rh-mongodb34-syspaths finished successfully!
there is now a doc bz open for that too: https://bugzilla.redhat.com/show_bug.cgi?id=1745511
(In reply to Evgeni Golov from comment #7) > Kennys output still used a remote DB because he forgot to wipe the answers. > > I tried this on 7.7 with 6.6.0 Snap 17 and it worked fine for a local DB. > Waiting on confirmation from Kenny, but pretty sure about this. > > Now let's look at the actual error: > > File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 436, in > authenticate > auth_func(credentials, sock_info) > File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 416, in > _authenticate_default > return _authenticate_scram_sha1(credentials, sock_info) > File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 199, in > _authenticate_scram_sha1 > salted_pass = _hi(_password_digest(username, password).encode("utf-8"), > File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 245, in > _password_digest > md5hash = md5() > > Pulp uses mongoengine, which in turn uses pymongo to talk to MongoDB. We use > pymongo 3.2 currently, which defaults to use SCRAM-SHA-1 when talking to a > 3.x MongoDB > (https://api.mongodb.com/python/current/api/pymongo/database.html#pymongo. > database.Database.authenticate, > https://github.com/mongodb/mongo-python-driver/blob/v3.3/pymongo/auth.py). > However, even this method uses MD5 to hash the users password > (https://github.com/mongodb/mongo-python-driver/blob/v3.3/pymongo/auth. > py#L211) and that will fail in FIPS mode. If 'hashing using MD5' causes FIPS to fail (which, yeah, it should), how is it working in the local-mongo case? I think I'm missing something here.
(In reply to Evgeni Golov from comment #7) > > The only solution I see is to use SCRAM-SHA-256 that was implemented in > pymongo 3.7 as that does not use MD5 at all. Sat6 is currently on Mongo3.4, with 3.6 at least available in SCL. Looking at the docs, I don't think Mongo itself supports SCRAM-SHA-256 prior to 4.x: https://docs.mongodb.com/v3.6/reference/parameters/#param.authenticationMechanisms https://docs.mongodb.com/v4.0/reference/parameters/#authentication-parameters Moving to 4.0 is probably a non-trivial exercise. Thoughts, anyone?
(In reply to Grant Gainey from comment #11) > (In reply to Evgeni Golov from comment #7) > > > > The only solution I see is to use SCRAM-SHA-256 that was implemented in > > pymongo 3.7 as that does not use MD5 at all. > > Sat6 is currently on Mongo3.4, with 3.6 at least available in SCL. Looking > at the docs, I don't think Mongo itself supports SCRAM-SHA-256 prior to 4.x: > > https://docs.mongodb.com/v3.6/reference/parameters/#param. > authenticationMechanisms > https://docs.mongodb.com/v4.0/reference/parameters/#authentication-parameters > > Moving to 4.0 is probably a non-trivial exercise. > > Thoughts, anyone? Another wrinkle is that Sat would have to move to a version of Mongo 4.x released prior to the license changes since the SCL takes care of this for us for the Mongo 3.4 & 3.6 versions.
(In reply to Grant Gainey from comment #10) > (In reply to Evgeni Golov from comment #7) > > Kennys output still used a remote DB because he forgot to wipe the answers. > > > > I tried this on 7.7 with 6.6.0 Snap 17 and it worked fine for a local DB. > > Waiting on confirmation from Kenny, but pretty sure about this. > > > > Now let's look at the actual error: > > > > File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 436, in > > authenticate > > auth_func(credentials, sock_info) > > File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 416, in > > _authenticate_default > > return _authenticate_scram_sha1(credentials, sock_info) > > File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 199, in > > _authenticate_scram_sha1 > > salted_pass = _hi(_password_digest(username, password).encode("utf-8"), > > File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 245, in > > _password_digest > > md5hash = md5() > > > > Pulp uses mongoengine, which in turn uses pymongo to talk to MongoDB. We use > > pymongo 3.2 currently, which defaults to use SCRAM-SHA-1 when talking to a > > 3.x MongoDB > > (https://api.mongodb.com/python/current/api/pymongo/database.html#pymongo. > > database.Database.authenticate, > > https://github.com/mongodb/mongo-python-driver/blob/v3.3/pymongo/auth.py). > > However, even this method uses MD5 to hash the users password > > (https://github.com/mongodb/mongo-python-driver/blob/v3.3/pymongo/auth. > > py#L211) and that will fail in FIPS mode. > > If 'hashing using MD5' causes FIPS to fail (which, yeah, it should), how is > it working in the local-mongo case? I think I'm missing something here. We don't do auth against local mongodb at all, so no auth is more secure than md5-based auth if you ask FIPS ;-)
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Satellite 6 does not have plans to support FIPS mode for disconnected MongoDB configuration. This requires extensive changes in our installer and offers limited performance benefits. Future versions of Satellite will not be using MongoDB so the lifespan for external Mongo databases is limited and we recommend embedded for the majority of deployments. If the closing of this bug is problematic, feel free to re-open with justification around the needs of running Satellite 6 with an external MongoDB with FIPS.
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.