Bug 1894852 - [security] Local VM deploys failed when deploying hosted engine with STIG profile, due to ssh unreachable.
Summary: [security] Local VM deploys failed when deploying hosted engine with STIG pro...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-ansible-collection
Classification: oVirt
Component: hosted-engine-setup
Version: 1.2.1
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ovirt-4.4.6
: ---
Assignee: Asaf Rachmani
QA Contact: Wei Wang
URL:
Whiteboard:
Depends On: 1900662 1902646 1909956 2025906
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-11-05 09:21 UTC by Wei Wang
Modified: 2021-11-23 11:02 UTC (History)
11 users (show)

Fixed In Version: ovirt-engine-4.4.5.5, rhvm-appliance-4.4-20210401.0.el8ev.x86_64
Clone Of:
Environment:
Last Closed: 2021-11-15 08:28:40 UTC
oVirt Team: Integration
Embargoed:
weiwang: testing_ack+


Attachments (Terms of Use)
issue log files (2.14 MB, application/gzip)
2020-11-05 09:21 UTC, Wei Wang
no flags Details
stig picture (112.32 KB, image/png)
2020-11-05 09:22 UTC, Wei Wang
no flags Details
VPP issue log files (2.21 MB, application/gzip)
2020-11-05 09:28 UTC, Wei Wang
no flags Details

Description Wei Wang 2020-11-05 09:21:14 UTC
Created attachment 1726818 [details]
issue log files

Description of problem:
Selecting STIG security profile during install RHVH with anaconda, deploy hosted engine via cockpit, it is stucked at TASK [ovirt.ovirt.hosted_engine_setup : Wait for the local VM], then displays error message

[ ERROR ] fatal: [localhost -> rhevh-hostedengine-vm-05.lab.eng.pek2.redhat.com]: FAILED! => {"changed": false, "elapsed": 3605, "msg": "timed out waiting for ping module test success: [Errno 24] Too many open files"}
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Sync on engine machine]
[ ERROR ] fatal: [localhost]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Bad owner or permissions on /etc/crypto-policies/back-ends/openssh.config", "unreachable": true}


ovirt-hosted-engine-setup-ansible-bootstrap_local_vm-20201105105140-atd9y8.log
-------------------------------------------------------------------------------
2020-11-05 11:59:17,088+0800 ERROR ansible failed {
    "ansible_host": "localhost",
    "ansible_playbook": "/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml",
    "ansible_result": {
        "_ansible_delegated_vars": {
            "ansible_host": "rhevh-hostedengine-vm-05.lab.eng.pek2.**FILTERED**.com",
            "ansible_port": null,
            "ansible_user": "root"
        },
        "_ansible_no_log": false,
        "changed": false,
        "elapsed": 3605,
        "msg": "timed out waiting for ping module test success: [Errno 24] Too many open files"
    },
    "ansible_task": "Wait for the local VM",
    "ansible_type": "task",
    "status": "FAILED",
    "task_duration": 3606
}

2020-11-05 11:59:18,204+0800 ERROR ansible unreachable {'status': 'UNREACHABLE', 'ansible_type': 'task', 'ansible_playbook': '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', 'ansible_host': 'localhost', 'ansible_task': 'Sync on engine machine', 'ansible_result': "type: <class 'dict'>\nstr: {'unreachable': True, 'msg': 'Failed to connect to the host via ssh: Bad owner or permissions on /etc/crypto-policies/back-ends/openssh.config', 'changed': False}", 'task_duration': 1}
2020-11-05 11:59:18,205+0800 DEBUG ansible on_any args <ansible.executor.task_result.TaskResult object at 0x7f8924631b00>  kwargs
2020-11-05 11:59:18,207+0800 INFO ansible stats {
    "ansible_playbook": "/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml",
    "ansible_playbook_duration": "07:32 Minutes",
    "ansible_result": "type: <class 'dict'>\nstr: {'localhost': {'ok': 173, 'failures': 0, 'unreachable': 1, 'changed': 52, 'skipped': 61, 'rescued': 1, 'ignored': 0}}",
    "ansible_type": "finish",
    "status": "FAILED"
}


Version-Release number of selected component (if applicable):
RHVH-4.4-20201029.0-RHVH-x86_64-dvd1.iso
cockpit-bridge-224.2-1.el8.x86_64
cockpit-ws-224.2-1.el8.x86_64
cockpit-storaged-224.2-1.el8.noarch
subscription-manager-cockpit-1.27.16-1.el8.noarch
cockpit-dashboard-224.2-1.el8.noarch
cockpit-224.2-1.el8.x86_64
cockpit-ovirt-dashboard-0.14.12-1.el8ev.noarch
cockpit-system-224.2-1.el8.noarch
ovirt-hosted-engine-setup-2.4.8-1.el8ev.noarch
ovirt-hosted-engine-ha-2.4.5-1.el8ev.noarch
rhvm-appliance-4.4-20200915.0.el8ev.x86_64


How reproducible:
100%

Steps to Reproduce:
1.Select STIG profile during installation
2.Deploy hosted engine via cockpit
3.

Actual results:
Local VM deploys failed when deploying hosted engine with , due to ssh unreachable.

Expected results:
Hosted engine deploy successful with STIG profile.

Additional info:

Comment 1 Wei Wang 2020-11-05 09:22:58 UTC
Created attachment 1726819 [details]
stig picture

Comment 2 Wei Wang 2020-11-05 09:27:26 UTC
The same issue will occur when selecting VPP security profile during installation RHVH.

Comment 3 Wei Wang 2020-11-05 09:28:21 UTC
Created attachment 1726820 [details]
VPP issue log files

Comment 4 Asaf Rachmani 2020-11-22 16:54:29 UTC
Seems that openssh.config file has incorrect permissions, 777 instead of 644.

Trying to connect to the localVM:
# ssh root.222.176
Bad owner or permissions on /etc/crypto-policies/back-ends/openssh.config

# ll /etc/crypto-policies/back-ends/openssh.config
-rwxrwxrwx. 1 root root 480 Nov 20 14:56 /etc/crypto-policies/back-ends/openssh.config

Comment 5 Asaf Rachmani 2020-11-23 12:26:04 UTC
The openssh.config file permissions have been changed in el8.3.

crypto-policies for 8.3:
$ rpm -qp --dump crypto-policies-20200713-1.git51d1222.el8.noarch.rpm | grep openssh.config
/etc/crypto-policies/back-ends/openssh.config 46 1594656270 0000000000000000000000000000000000000000000000000000000000000000 0120777 root root 1 0 0 /usr/share/crypto-policies/DEFAULT/openssh.txt
/usr/share/crypto-policies/back-ends/DEFAULT/openssh.config 1257 1594656269 3d3c6acdc4f04733dc586be2b3ac59d695c9d81232b9a77ac0f4f5db1715b2b6 0100644 root root 0 0 0 X
/usr/share/crypto-policies/back-ends/FIPS/openssh.config 854 1594656269 1c9b17757243c929f310d96a9e2290060aa7eef033f2e8313ae4e2fd9622d3f7 0100644 root root 0 0 0 X
/usr/share/crypto-policies/back-ends/FUTURE/openssh.config 986 1594656269 b83dad7da4e110ca351fa5ea43040138f2a4e1043ac2f0406accddd3d3632fdb 0100644 root root 0 0 0 X
/usr/share/crypto-policies/back-ends/LEGACY/openssh.config 1355 1594656269 8e230fadfa6ef25bb3f732cbb15fa4b38503e9f4c3ba9a700e6fec5e8540987d 0100644 root root 0 0 0 X


crypto-policies for 8.2:
$ rpm -qp --dump crypto-policies-20191128-2.git23e1bf1.el8.noarch.rpm | grep openssh.config
/etc/crypto-policies/back-ends/openssh.config 0 1576519808 0000000000000000000000000000000000000000000000000000000000000000 0100000 root root 0 0 0 X
/usr/share/crypto-policies/back-ends/DEFAULT/openssh.config 1173 1576519805 1f6ad778c1b4f3c2ee4c3300a2a829ada209c0f4daa211bde3159a46ad45a14b 0100644 root root 0 0 0 X
/usr/share/crypto-policies/back-ends/FIPS/openssh.config 854 1576519805 d140ff8ee38d517fae026cda89037192693a049286889c05fdee060467599ca2 0100644 root root 0 0 0 X
/usr/share/crypto-policies/back-ends/FUTURE/openssh.config 923 1576519805 c0c2c69bea40231791f5c93f61b9f13644dd8cf0c799e550606083e0875e2727 0100644 root root 0 0 0 X
/usr/share/crypto-policies/back-ends/LEGACY/openssh.config 1271 1576519805 76fe9070b172ee9baf59c3cba81814c203580a979bb275f699cd3f5d5efed30b 0100644 root root 0 0 0 X

Comment 6 Asaf Rachmani 2021-01-25 16:40:45 UTC
The issue described in comment 0 has been fixed in bug 1902646,
Adding depends on bug 1909956 in order to verify HE deployment on a host with a stig profile.

Comment 8 Wei Wang 2021-04-06 01:06:23 UTC
Test with RHVH-4.4-20210331.0-RHVH-x86_64-dvd1.iso and rhvm-appliance-4.4-20210401.0.el8ev.x86_64

Bug is fixed, move it to "VERIFIED"

Comment 9 Sandro Bonazzola 2021-05-05 05:36:03 UTC
This bugzilla is included in oVirt 4.4.6 release, published on May 4th 2021.

Since the problem described in this bug report should be resolved in oVirt 4.4.6 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.

Comment 10 Richard Evans 2021-08-10 01:04:36 UTC
Still hitting this issue. Loaded Virt STIG profile on 4.4.6 install and said yes to OpenSCAP profile for engine. Engine gets stuck on "Waiting for VM Status" or something like that and then dies with the same ssh error.

Tried loading on Virt STIG'd host profile and answered no to OpenSCAP profile for engine. Still stuck at "Waiting for VM Status" then time out.

Loaded no STIG profile on host with "Yes" to OpenSCAP profile for engine. Stuck at "Waiting for VM Status" then timeout.

Loaded no STIG profile on host with "No" to OpenSCAP profile for engine. This allowed the engine to install.

Comment 11 cshao 2021-11-12 11:27:46 UTC
Reopen this bug due to can reproduce this issue with OpenSCAP enabled.

Test version:
RHVH ISO: RHVH-4.4-20210903.0-RHVH-x86_64-dvd1.iso
rhvm-appliance-4.4-20210827.0.el8ev.x86_64.rhevm.ova

Test Steps:
1. Install RHVH ISO host with VPP profile.
2. Upload current ova file for rhvm onto the RHVH.
3. Attempt to install SHE (Enable OpenSCAP)

Test result:
Failed to deploy HE RHVM SHE when OpenSCAP enabled.

weiwang will try to reproduce this issue with the latest RHVH build.

Comment 14 Michal Skrivanek 2021-11-12 16:25:07 UTC
failed starting ssh while booting the temporary HE VM. From journal:

 Starting OpenSSH server daemon...
 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
 @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
 Permissions 0640 for '/etc/ssh/ssh_host_rsa_key' are too open.
 It is required that your private key files are NOT accessible by others.
 This private key will be ignored.
 Unable to load host key "/etc/ssh/ssh_host_rsa_key": bad permissions
 Unable to load host key: /etc/ssh/ssh_host_rsa_key
 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
 @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
 Permissions 0640 for '/etc/ssh/ssh_host_ecdsa_key' are too open.
 It is required that your private key files are NOT accessible by others.
 This private key will be ignored.
 Unable to load host key "/etc/ssh/ssh_host_ecdsa_key": bad permissions
 Unable to load host key: /etc/ssh/ssh_host_ecdsa_key
 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
 @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
 Permissions 0640 for '/etc/ssh/ssh_host_ed25519_key' are too open.
 It is required that your private key files are NOT accessible by others.
 This private key will be ignored.
 Unable to load host key "/etc/ssh/ssh_host_ed25519_key": bad permissions
 Unable to load host key: /etc/ssh/ssh_host_ed25519_key
 sshd: no hostkeys available -- exiting.
 sshd.service: Main process exited, code=exited, status=1/FAILURE
 sshd.service: Failed with result 'exit-code'.
 Failed to start OpenSSH server daemon.

Comment 15 Sandro Bonazzola 2021-11-15 08:18:21 UTC
(In reply to Sandro Bonazzola from comment #9)
> This bugzilla is included in oVirt 4.4.6 release, published on May 4th 2021.
> 
> Since the problem described in this bug report should be resolved in oVirt
> 4.4.6 release, it has been closed with a resolution of CURRENT RELEASE.
> 
> If the solution does not work for you, please open a new bug report.

This bug was fixed in 4.4.6 and verified fixed in 4.4.6 using RHV-H iso.
Any reason for reopening this 4.4.6 bug with a reproduced issue in 4.4.8 instead of opening a new bug?

The issue in 4.4.6 was "Bad owner or permissions on /etc/crypto-policies/back-ends/openssh.config""

while here according to comment #14 it seems the issue comes from /etc/ssh/ which belongs to different set of packages.

Can you please open a new bug and close this bug again as current release?

Comment 16 cshao 2021-11-15 08:28:40 UTC
(In reply to Sandro Bonazzola from comment #15)
> (In reply to Sandro Bonazzola from comment #9)
> > This bugzilla is included in oVirt 4.4.6 release, published on May 4th 2021.
> > 
> > Since the problem described in this bug report should be resolved in oVirt
> > 4.4.6 release, it has been closed with a resolution of CURRENT RELEASE.
> > 
> > If the solution does not work for you, please open a new bug report.
> 
> This bug was fixed in 4.4.6 and verified fixed in 4.4.6 using RHV-H iso.
> Any reason for reopening this 4.4.6 bug with a reproduced issue in 4.4.8
> instead of opening a new bug?
> 
> The issue in 4.4.6 was "Bad owner or permissions on
> /etc/crypto-policies/back-ends/openssh.config""
> 
> while here according to comment #14 it seems the issue comes from /etc/ssh/
> which belongs to different set of packages.
> 
> Can you please open a new bug and close this bug again as current release?

Reported new Bug 2023206 - Failed starting ssh while booting the temporary HE VM with OpenSCAP enabled. 

Thanks.


Note You need to log in before you can comment on or make changes to this bug.