Description of problem: Government site requiring FIPS supported in Ovirt-engine (specifically engine-setup) and certificates. Version-Release number of selected component (if applicable): Will add attachment for logcollector.
(In reply to Jason from comment #0) > Description of problem: > > Government site requiring FIPS supported in Ovirt-engine (specifically > engine-setup) and certificates. > > Version-Release number of selected component (if applicable): > > > Will add attachment for logcollector. Can you please explicit the list of changes requested to engine-setup in order to be FIPS compliant?
A first review of the logs shows: 2017-06-22 08:39:11 DEBUG otopi.context context._executeMethod:142 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-common/base/core/uninstall.py", line 174, in _cleanup otopicons.CoreEnv.MODIFIED_FILES File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-common/base/core/uninstall.py", line 131, in _addFiles self._digestFile(name), File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-common/base/core/uninstall.py", line 80, in _digestFile md5 = hashlib.new('md5') File "/usr/lib64/python2.7/hashlib.py", line 111, in __hash_new return _hashlib.new(name, string, usedforsecurity) ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
(In reply to Sandro Bonazzola from comment #2) > A first review of the logs shows: > > 2017-06-22 08:39:11 DEBUG otopi.context context._executeMethod:142 method > exception > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in > _executeMethod > method['method']() > File > "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-common/base/core/ > uninstall.py", line 174, in _cleanup > otopicons.CoreEnv.MODIFIED_FILES > File > "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-common/base/core/ > uninstall.py", line 131, in _addFiles > self._digestFile(name), > File > "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-common/base/core/ > uninstall.py", line 80, in _digestFile > md5 = hashlib.new('md5') > File "/usr/lib64/python2.7/hashlib.py", line 111, in __hash_new > return _hashlib.new(name, string, usedforsecurity) > ValueError: error:060800A3:digital envelope > routines:EVP_DigestInit_ex:disabled for fips I have a feeling this won't be the only issue, so perhaps we need a tracker bug and another per specific issue. Re above specific issue: One of the jobs of engine-setup is to write/change various configuration files. When it finishes, it writes to /etc/ovirt-engine/uninstall.d a file the the names and the md5 hashes of files it wrote. When a user runs engine-cleanup, it checks this file, creates a list of files that were (presumably manually) modified since written by engine-setup, and prompts the user asking whether to remove them anyway. Not sure this is a security issue, but it might be that fips compliance still prevents using md5. So to fix this, we can use some other hash function (say sha1 or sha256). That's easy for new setups. Need to think what we want to do in upgrades. Should it be allowed to upgrade a non-fips-compliant setup of oVirt/RHV to a compliant one? If yes, engine-cleanup won't be able to verify files that we only kept md5 sums for. An identical test is done also by ovirt-engine-rename. Also, this isn't the only place we use md5. Some other places, in ovirt-engine/packaging: - engine-backup keeps the md5 of the stuff if backs up. Easy to fix for new setups, need to think what to do if we restore a backup that used md5. - db schema scripts keep inside the database a list of scripts already processed together with their md5, to prevent applying again a script already processed (e.g. during upgrade to a previous version). Again, main issue is upgrades. - When auto-provisioning postgresql, we configure it to use 'md5' authentication scheme. Need to see what postgresql supports that's fips compliant, and also how to handle upgrades.
Hi Sandro, As discussed with Ryan Barry, the issue is when you have fips=1 and try to run engine-setup you get errors. Secondly Ryan confirmed that fips=1 is not presently tested/confirmed with QE and that using it may not work right with the certificates. Customer provided the following steps to re-create: 1. Install RHEL 7.3 with fips=1 2. Subscribe to the RHV 4.1 repositories 3. Install the rhevm package 4. Run engine-setup No logcollector available but will upload the setup log and sosreport from the system in question.
(In reply to Yedidyah Bar David from comment #3) > (In reply to Sandro Bonazzola from comment #2) > > A first review of the logs shows: > > > > 2017-06-22 08:39:11 DEBUG otopi.context context._executeMethod:142 method > > exception > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in > > _executeMethod > > method['method']() > > File > > "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-common/base/core/ > > uninstall.py", line 174, in _cleanup > > otopicons.CoreEnv.MODIFIED_FILES > > File > > "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-common/base/core/ > > uninstall.py", line 131, in _addFiles > > self._digestFile(name), > > File > > "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-common/base/core/ > > uninstall.py", line 80, in _digestFile > > md5 = hashlib.new('md5') > > File "/usr/lib64/python2.7/hashlib.py", line 111, in __hash_new > > return _hashlib.new(name, string, usedforsecurity) > > ValueError: error:060800A3:digital envelope > > routines:EVP_DigestInit_ex:disabled for fips > > I have a feeling this won't be the only issue, so perhaps we need a tracker > bug and another per specific issue. > > Re above specific issue: > > One of the jobs of engine-setup is to write/change various configuration > files. > When it finishes, it writes to /etc/ovirt-engine/uninstall.d a file the the > names and the md5 hashes of files it wrote. When a user runs engine-cleanup, > it checks this file, creates a list of files that were (presumably manually) > modified since written by engine-setup, and prompts the user asking whether > to remove them anyway. > > Not sure this is a security issue, but it might be that fips compliance > still prevents using md5. > > So to fix this, we can use some other hash function (say sha1 or sha256). > That's easy for new setups. Need to think what we want to do in upgrades. > Should it be allowed to upgrade a non-fips-compliant setup of oVirt/RHV to a > compliant one? If yes, engine-cleanup won't be able to verify files that we > only kept md5 sums for. We're probably safer using sha256, since SHA1 will probably become non-compliant at some point. I would imagine that upgrades from non-FIPS to FIPS will not be supported, and that users will need a clean reinstall, but we may be able to document a workaround involving some conversion of the backup before FIPS is enabled with verifies the backups using md5sums then rewrites it with SHA. > > An identical test is done also by ovirt-engine-rename. > > Also, this isn't the only place we use md5. Some other places, in > ovirt-engine/packaging: > > - engine-backup keeps the md5 of the stuff if backs up. Easy to fix for new > setups, need to think what to do if we restore a backup that used md5. > > - db schema scripts keep inside the database a list of scripts already > processed together with their md5, to prevent applying again a script > already processed (e.g. during upgrade to a previous version). Again, main > issue is upgrades. That's much harder. Does the engine also keep any other metadata about the script? mtime? > > - When auto-provisioning postgresql, we configure it to use 'md5' > authentication scheme. Need to see what postgresql supports that's fips > compliant, and also how to handle upgrades. Probably certificate auth instead of local GSSAPI/LDAP. trust and ident are clearly bad. It will definitely be interesting on upgrades, but we already have a CA for engine, and postgresql allows multiple authentication methods to be provided, I think, so it may be a pretty seamless conversion for auth. Also, FIPS allows MD5 in conjunction with TLS, so it may be possible to reuse the CA and keep some of the md5 pieces in the db.
I briefly discussed this with Yuval, and it seems most of our problems our going to be around upgrade. A possible plan can be: 1. Change the system to generate both old (MD5) and new (SHA-256) hashes on new setups. 2. Change the system to convert, perhaps during engine-setup but perhaps that's not enough, existing old hashes to new hashes 3. Change the system to drop use/generation of old hashes. Each step should probably be in its own version. Perhaps (1.)+(2.) can be in same. (3.) definitely needs to be in a later one. Once we finish (1.) we can start supporting fips for new setups. Systems upgraded from previous versions can be supported once they are upgraded to a version that includes (3.). Each of 1/2/3 is not a single bz but a collection. See comment 3 and comment 4 for some, but definitely not all, cases that need to be handled, probably each with its own bug.
It seems like enabling fips [1] does not disable or prevent use of md5 in general. md5sum(1) continues to work, as well as the function 'md5' of postgresql. The failure described in comment 2 is in a call to python's hashlib. It seem that RHEL (and derivatives, probably) implements [2], so running this: python -c "import hashlib; md5 = hashlib.new('md5')" works in non-fips mode but fails in fips mode with: Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib64/python2.7/hashlib.py", line 111, in __hash_new return _hashlib.new(name, string, usedforsecurity) ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips But running this: python -c "import hashlib; md5 = hashlib.new('md5', usedforsecurity=False)" works in both modes. The actual error seems to be due to a fallback of python's hashlib from internal implementation, which is disabled in fips mode, to openssl's, and openssl's implementation is also disabled, and emits the above error. Also the disabling of md5 (and related) algos in openssl seems to be done in a RHEL- (and derivatives) specific way, checking existence of /etc/system-fips and content of /proc/sys/crypto/fips_enabled , which are set by following [1]. Using this, it enables fips mode [3] of openssl. [1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Security_Guide/chap-Federal_Standards_and_Regulations.html#sec-Enabling-FIPS-Mode [2] https://bugs.python.org/issue9216 [3] https://www.openssl.org/docs/fips.html
Didn't test yet if the patch works, whether previous comment 16 is enough, what's the next failure, etc., but I think now is a good time to discuss what we want, and who can approve it. Is it enough to "make things work" in fips mode, in whatever way, or do we want/need to get a review from a security expert that knows FIPS to tell us what's actually allowed or not? One can claim that the patch is a cheat - that the affected code _does_ use md5 for security, and that it should be changed to use an approved algorithm. I don't mind analyzing (here or elsewhere) what I think about this, but I am not sure my word counts. Yaniv, what do you think?
BTW, I reproduced the error in comment 2, and it's not fatal, as it happens in the "cleanup" stage of engine-setup (a very late one). The engine does start and I can login to the web admin ui. Didn't try anything else yet. The failure mainly affects a future engine-cleanup (and ovirt-engine-rename), and should be fixed (with the trivial patch I pushed, or by changing the algo, which is harder if we want to support upgrades from md5-using setups).
Moving to NEW for now, because the patch is not a real fix, and to stop the bot that's annoying me daily about ACKs.
It is possible that this topic is related https://bugzilla.redhat.com/show_bug.cgi?id=1444449
(In reply to Yedidyah Bar David from comment #17) > Didn't test yet if the patch works, whether previous comment 16 is enough, > what's the next failure, etc., but I think now is a good time to discuss > what we want, and who can approve it. > > Is it enough to "make things work" in fips mode, in whatever way, or do we > want/need to get a review from a security expert that knows FIPS to tell us > what's actually allowed or not? One can claim that the patch is a cheat - > that the affected code _does_ use md5 for security, and that it should be > changed to use an approved algorithm. I don't mind analyzing (here or > elsewhere) what I think about this, but I am not sure my word counts. > > Yaniv, what do you think? Kurt, what do you think?
(In reply to Ariel O. Barria from comment #20) > It is possible that this topic is related > https://bugzilla.redhat.com/show_bug.cgi?id=1444449 Indeed, thanks for pointing out. Keeping open for now, as current is on RHV and the other is on oVirt.
(In reply to Yedidyah Bar David from comment #21) > (In reply to Yedidyah Bar David from comment #17) > > Didn't test yet if the patch works, whether previous comment 16 is enough, > > what's the next failure, etc., but I think now is a good time to discuss > > what we want, and who can approve it. > > > > Is it enough to "make things work" in fips mode, in whatever way, or do we > > want/need to get a review from a security expert that knows FIPS to tell us > > what's actually allowed or not? One can claim that the patch is a cheat - > > that the affected code _does_ use md5 for security, and that it should be > > changed to use an approved algorithm. I don't mind analyzing (here or > > elsewhere) what I think about this, but I am not sure my word counts. > > > > Yaniv, what do you think? > > Kurt, what do you think? Honestly I know very little about the FIPS side of things, but shawn does, ping, can you take a look at this and comment? thanks!
Is this still considered in scope for 4.3? Urgent?
(In reply to Yedidyah Bar David from comment #29) > Is this still considered in scope for 4.3? I sincerely hope so, why? > > Urgent? That's our strange method to denote it is a P1 item.
comment from onsite RH employee: The customer POC still is experiencing problems related to this and is very interested in a fix. I have good familiarity with FIPS from my most recent site as well. If you need any more information or need testing done onsite let me know. Thanks!
From an RH Architect: Any movement on this? Are we still looking at 4.3 for a fix. If you need context around the federal requirement for fips please reach out. I have multiple clients that are requiring this. Thanks...
re-targeting to 4.3.1 since this BZ has not been proposed as blocker for 4.3.0. If you think this bug should block 4.3.0 please re-target and set blocker flag.
Moving to 4.3.2 not being identified as blocker for 4.3.1.
Verified. version: 4.3.3.2-0.1.el7 Engine and hosts were sucessfully installed on fips=1 enabled system. Did sanity check on this environment. for deploying hosted-engine, there is a new option with default option set to no: Do you want to apply a default OpenSCAP security profile (Yes, No) [No] which also enable FIPS on appliance For hardening and ensuring security, please see bug 1392051 which covers also FIPS
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:1085