Bug 2011309 - sshd service fails to start after rebooting to apply FIPS settings
Summary: sshd service fails to start after rebooting to apply FIPS settings
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: rhvm-appliance
Version: 4.4.8
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ovirt-4.5.0-1
: 4.5.0
Assignee: Asaf Rachmani
QA Contact: Qin Yuan
URL:
Whiteboard:
: 2023206 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-10-06 11:44 UTC by Juan Orti
Modified: 2022-08-08 19:57 UTC (History)
8 users (show)

Fixed In Version: scap-security-guide-0.1.59-1.el8, rhvm-appliance-4.5-20220318.0.el8ev
Doc Type: Bug Fix
Doc Text:
Previously, Self-Hosted Engine deployment failed when applying an OpenSCAP security profile. As a result, the premissions for the SSH key file were changed to 0640, which is not secure enough. In this release, the permissions are not changed, and Self-Hosted Engine deployment succeeds when applying an OpenSCAP security profile.
Clone Of:
Environment:
Last Closed: 2022-06-07 15:22:53 UTC
oVirt Team: Integration
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHV-43766 0 None None None 2021-10-06 11:45:10 UTC
Red Hat Knowledge Base (Solution) 6388441 0 None None None 2021-10-06 11:44:30 UTC
Red Hat Product Errata RHSA-2022:4931 0 None None None 2022-06-07 15:23:11 UTC

Description Juan Orti 2021-10-06 11:44:30 UTC
Description of problem:
If FIPS is enabled during the deployment of hosted-engine, the sshd service of the engine fails to start after a reboot and the deployment fails.

Version-Release number of selected component (if applicable):
ovirt-hosted-engine-setup-2.5.3-1.el8ev.noarch
openssh-8.0p1-6.el8_4.2.x86_64
crypto-policies-20210209-1.gitbfb6bed.el8_3.noarch

How reproducible:
Always

Steps to Reproduce:
1. In a RHEL 8 host fully updated, deploy hosted-engine

# hosted-engine --deploy

2. Answer yes to this question:

Do you want to apply a default OpenSCAP security profile? (Yes, No) [No]: Yes

Actual results:
The deploy fails after this task:

[ INFO  ] TASK [ovirt.ovirt.hosted_engine_setup : Reboot the engine VM to ensure that FIPS is enabled]

The sshd service in the engine fails to start after the reboot:

Oct 06 11:24:19 systemd[1]: Starting OpenSSH server daemon...
Oct 06 11:24:19 sshd[1458]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Oct 06 11:24:19 sshd[1458]: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Oct 06 11:24:19 sshd[1458]: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Oct 06 11:24:19 sshd[1458]: Permissions 0640 for '/etc/ssh/ssh_host_rsa_key' are too open.
Oct 06 11:24:19 sshd[1458]: It is required that your private key files are NOT accessible by others.
Oct 06 11:24:19 sshd[1458]: This private key will be ignored.
Oct 06 11:24:19 sshd[1458]: Unable to load host key "/etc/ssh/ssh_host_rsa_key": bad permissions
Oct 06 11:24:19 sshd[1458]: Unable to load host key: /etc/ssh/ssh_host_rsa_key

Expected results:
No errors

Additional info:
The engine key files have 0640 permissions:

# ls -la /etc/ssh/ssh*key
-rw-r-----. 1 root root  570 Oct  6 11:18 /etc/ssh/ssh_host_ecdsa_key
-rw-r-----. 1 root root  476 Oct  6 11:18 /etc/ssh/ssh_host_ed25519_key
-rw-r-----. 1 root root 2667 Oct  6 11:18 /etc/ssh/ssh_host_rsa_key

Comment 3 Juan Orti 2021-10-06 14:44:13 UTC
The problem is that the key files have group root. If I change the group to ssh_keys, which is the default, the SSH service starts fine independently if the system is in FIPS mode or not.

In the OpenSSH function 'sshkey_perm_ok' I see that only the UID of the file is checked, so I'm not sure why this check is no triggered when the file is owned by root:ssh_keys

https://anongit.mindrot.org/openssh.git/tree/authfile.c#n90

~~~
int
sshkey_perm_ok(int fd, const char *filename)
{
	struct stat st;

	if (fstat(fd, &st) == -1)
		return SSH_ERR_SYSTEM_ERROR;
	/*
	 * if a key owned by the user is accessed, then we check the
	 * permissions of the file. if the key owned by a different user,
	 * then we don't care.
	 */
#ifdef HAVE_CYGWIN
	if (check_ntsec(filename))
#endif
	if ((st.st_uid == getuid()) && (st.st_mode & 077) != 0) {  <------------------------------
		error("@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@");
		error("@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @");
		error("@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@");
		error("Permissions 0%3.3o for '%s' are too open.",
		    (u_int)st.st_mode & 0777, filename);
		error("It is required that your private key files are NOT accessible by others.");
		error("This private key will be ignored.");
		return SSH_ERR_KEY_BAD_PERMISSIONS;
	}
	return 0;
}
~~~

Comment 4 Juan Orti 2021-10-06 15:27:49 UTC
Ok, I'm going to answer my own question. We are applying the patch openssh-6.6p1-keyperm.patch in the RPM to allow the ssh_keys group. This way the /usr/libexec/openssh/ssh-keysign helper used for host-based authentication can work. 

So the remaining question is, why the keys were created as root:root 0640?

Comment 5 Asaf Rachmani 2021-10-18 14:01:48 UTC
We can reproduce this issue.
I checked the openscap-report and the following rule under Services -> SSH Server, marked as "fixed":
   Verify Permissions on SSH Server Private *_key Key Files
   To properly set the permissions of /etc/ssh/*_key, run the command:
   $ sudo chmod 0640 /etc/ssh/*_key

Once changing the /etc/ssh/*_key permissions back to be 0600, ssh works fine.

Watson,
Do you know if this is ok?

Comment 7 Watson Yuuma Sato 2021-10-19 08:45:42 UTC
Hello, 

In https://github.com/ComplianceAsCode/content/pull/7742 the rules have been fixed and they do not change the the permission to 0640 any more, they ensure the permissions are 0600.

Comment 9 Asaf Rachmani 2021-11-15 11:54:59 UTC
*** Bug 2023206 has been marked as a duplicate of this bug. ***

Comment 13 Qin Yuan 2022-04-18 13:33:20 UTC
Verified with:
rhvm-appliance-4.5-20220412.0.el8ev
scap-security-guide-0.1.60-7.el8.noarch
ovirt-hosted-engine-ha-2.5.0-1.el8ev.noarch
ovirt-hosted-engine-setup-2.6.3-1.el8ev.noarch

Steps:
1. In a RHEL 8.6 host fully updated, deploy hosted-engine

# hosted-engine --deploy

2. Answer yes to this question:

Do you want to apply a default OpenSCAP security profile? (Yes, No) [No]: Yes

Results:
1. hosted-engine deployment succeeds when applying OpenSCAP security profile.
2. FIPS mode is enabled in engine.
3. The permissions of ssh keys in engine are 0600, sshd service is up.

Comment 15 errata-xmlrpc 2022-06-07 15:22:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: RHV Appliance (rhvm-appliance) security update [ovirt-4.5.0]), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:4931

Comment 16 meital avital 2022-08-08 19:57:49 UTC
Due to QE capacity, we are not going to cover this issue in our automation


Note You need to log in before you can comment on or make changes to this bug.