Description of problem:

This sounds similar to https://bugzilla.redhat.com/show_bug.cgi?id=1791007 however this is with RHHI/RHV (4.3.8)

When trying to connect to a VM via ovirt-vmconsole (serial console) an error is observed:

~~~
[root@rhvm ~]# ssh -i ~/.ssh/id_rsa -p 2222 -t ovirt-vmconsole.com
Available Serial Consoles:
00 HostedEngine[b421251f-73f4-4ea6-a598-50175efbe63c]
01 ocp4-helper[3407335a-a2cf-44bb-a511-cf0cdd9d4cf3]
02 ocp4-master0[00a2a19a-3bcd-4968-b990-dfb0c7c17700]
03 ocp4-master1[800c06c9-3e40-460b-9a83-876f20b35ddc]
04 ocp4-master2[b6d5ff40-ed3a-4ba1-9650-9eb67bb7b72c]
05 ocp4-worker0[d4b02a5b-2798-429b-a2c2-5141ab952665]
06 ocp4-worker1[80d7d43b-5e42-4d06-a7e9-96e9ad661e26]
07 asdf-adsf-helper[cc0deb16-46c8-4666-83dd-25278ec57937]

Please, enter the id of the Serial Console you want to connect to.
To disconnect from a Serial Console, enter the sequence: <Enter><~><.>
SELECT> 1
key_cert_check_authority: invalid certificate
Certificate invalid: not a host certificate
Host key verification failed.
Connection to rhvm.example.com closed.
~~~

"Enroll Certificates" was performed from RHV-M, however the issue persists.

When does the behavior occur? Frequency? Repeatedly? At certain times?
Repeatedly

Version-Release number of selected component (if applicable):

RHV-M 4.3.8:

ovirt-engine-vmconsole-proxy-helper-4.3.8.2-0.4.el7.noarch  Tue Jan 28 15:57:22 2020
ovirt-vmconsole-1.0.7-3.el7ev.noarch                        Tue Jan 28 15:53:10 2020
ovirt-vmconsole-proxy-1.0.7-3.el7ev.noarch                  Tue Jan 28 15:53:38 2020

RHV-H:

ovirt-vmconsole-1.0.7-3.el7ev.noarch                        Sun Jan 26 06:12:22 2020
ovirt-vmconsole-host-1.0.7-3.el7ev.noarch                   Sun Jan 26 06:12:36 2020
redhat-release-virtualization-host-4.3.8-1.el7ev.x86_64     Sun Jan 26 06:00:39 2020


How reproducible:
It was reproduced in two environments.
The first was a newly deployed 4.3 environment. From the sosreport, it appears the host was on 4.3.7 and upgraded to 4.3.8, and RHV-M was 4.3.8 from the start. From the start the vmconsole certificate error was present and re-enrolling certificate did not resolve it.

The same issue was also observed in a lab environment that been upgraded to RHV-M 4.3.8 and RHV-H 4.3.8, however when adding a new RHV-H 4.3.8 host to this environment, the newly installed host worked OK. There was also a RHEL host in the same environment that did not exhibit the issue. 

Steps to Reproduce:
When the issue with certificate key is observed, try to re-enroll certificate from RHV-M.

Actual results:
The vmconsole certificate should be renewed.

Expected results:
The vmconsole certificate is not renewed.

Additional info:

The workaround is to reinstall the host from RHV-M, this appears to re-enroll the vmconsole certificate. Also see screenshot that will be attached showing that the vmconsole certificate is enrolled during host (re)install but not when "Enroll Certificates" is invoked.

On the host I do see this during enroll certificates, however the error is still seen until the host is "reinstalled", after which the issue is resolved.

~~~
Mar 18 19:43:58 host1 python: ansible-getent Invoked with fail_key=True key=ovirt-vmconsole split=None service=None database=passwd
Mar 18 19:44:02 host1 python: ansible-tempfile Invoked with path=None prefix=ansible. suffix=vmconsole state=file
Mar 18 19:44:06 host1 python: ansible-command Invoked with creates=None executable=None _uses_shell=False strip_empty_ends=True _raw_params='/usr/bin/openssl'#012'req'#012'-new'#012'-newkey'#012'rsa:2048'#012'-n
odes'#012'-subj'#012'/'#012'-keyout'#012'/tmp/ansible.WfkgBivmconsole'#012 removes=None argv=None warn=True chdir=None stdin_add_newline=True stdin=None
Mar 18 19:44:12 host1 python: ansible-stat Invoked with checksum_algorithm=sha1 get_checksum=True follow=False path=vmconsole get_md5=False get_mime=True get_attributes=True
Mar 18 19:44:14 host1 python: ansible-file Invoked with directory_mode=None force=False remote_src=None _original_basename=tmp4mBLyb path=vmconsole owner=None follow=True group=None unsafe_writes=None setype=Non
e content=NOT_LOGGING_PARAMETER serole=None selevel=None state=file dest=vmconsole access_time=None access_time_format=%Y%m%d%H%M.%S modification_time=None regexp=None src=None seuser=None recurse=False _diff_peek
=None delimiter=None mode=None modification_time_format=%Y%m%d%H%M.%S attributes=None backup=None
Mar 18 19:44:18 host1 python: ansible-stat Invoked with checksum_algorithm=sha1 get_checksum=True follow=False path=/etc/pki/ovirt-vmconsole/host-ssh_host_rsa-cert.pub get_md5=False get_mime=True get_attributes=
True
Mar 18 19:44:21 host1 python: ansible-copy Invoked with directory_mode=None force=True remote_src=False _original_basename=host1.example.com-ssh-cert.pub owner=None follow=False local_follow=None g
roup=None unsafe_writes=None setype=None content=NOT_LOGGING_PARAMETER serole=None dest=/etc/pki/ovirt-vmconsole/host-ssh_host_rsa-cert.pub selevel=None regexp=None validate=None src=/root/.ansible/tmp/ansible-tmp
-1584575054.92-133316649710949/source checksum=ce1785c1b2e5630ac66db2b4287bb1a1253866f0 seuser=None delimiter=None mode=None attributes=None backup=False
Mar 18 19:44:25 host1 python: ansible-copy Invoked with directory_mode=None force=True remote_src=True _original_basename=None owner=ovirt-vmconsole follow=False local_follow=None group=ovirt-vmconsole unsafe_wr
ites=None setype=None content=NOT_LOGGING_PARAMETER serole=None dest=/etc/pki/ovirt-vmconsole/host-ssh_host_rsa selevel=None regexp=None validate=None src=/tmp/ansible.WfkgBivmconsole checksum=None seuser=None del
imiter=None mode=256 attributes=None backup=False
Mar 18 19:44:29 host1 python: ansible-file Invoked with directory_mode=None force=False remote_src=None _original_basename=None path=/tmp/ansible.WfkgBivmconsole owner=None follow=True group=None unsafe_writes=N
one state=absent content=NOT_LOGGING_PARAMETER serole=None selevel=None setype=None access_time=None access_time_format=%Y%m%d%H%M.%S modification_time=None regexp=None src=None seuser=None recurse=False _diff_pee
k=None delimiter=None mode=None modification_time_format=%Y%m%d%H%M.%S attributes=None backup=None
~~~