Bug 1563737 - Unable to connect via serial console to HE. /bin/sh: Permission denied.
Summary: Unable to connect via serial console to HE. /bin/sh: Permission denied.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: rhevm-appliance
Version: 4.2.2
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ovirt-4.2.3
: ---
Assignee: Yuval Turgeman
QA Contact: Vitalii Yerys
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-04 15:01 UTC by Vitalii Yerys
Modified: 2019-05-16 13:09 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-05-15 19:00:29 UTC
oVirt Team: Node


Attachments (Terms of Use)
logs (860.61 KB, application/x-gzip)
2018-04-04 15:01 UTC, Vitalii Yerys
no flags Details
connection_attempt.log (11.82 KB, text/plain)
2018-04-04 15:02 UTC, Vitalii Yerys
no flags Details
packages_list (3.77 KB, text/plain)
2018-04-04 15:02 UTC, Vitalii Yerys
no flags Details
full_logs (6.38 MB, application/x-gzip)
2018-04-05 12:30 UTC, Vitalii Yerys
no flags Details
anaconda packaging.log (246.22 KB, text/plain)
2018-04-22 11:50 UTC, Yuval Turgeman
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:1525 None None None 2018-05-15 19:00:37 UTC
Red Hat Bugzilla 1570829 None CLOSED ovirt-vmconsole must require selinux-policy-targeted in post section 2019-07-07 11:36:19 UTC
Red Hat Bugzilla 1570831 None CLOSED rh-postgresql95 must require selinux-policy-targeted in post section 2019-07-07 11:36:19 UTC
Red Hat Bugzilla 1570833 None CLOSED cockpit-dashboard must require selinux-policy-targeted in post section 2019-07-07 11:36:19 UTC

Internal Links: 1570829 1570831 1570833

Description Vitalii Yerys 2018-04-04 15:01:51 UTC
Created attachment 1417298 [details]
logs

Description of problem:
SELinux prevents serial console connection. I have tried to get list of vms (or any other action) via serial console for ovirt 4.2 and it fails. If I set "setenforce 0" serial console works fine, but with "setenforce 1" it fails with error /bin/sh: Permission denied. This is reproducible only on HE env. 

from /var/log/messages:

Apr  4 16:56:39 hosted-engine-02 sshd[9760]: ssh_selinux_change_context: setcon system_u:system_r:sshd_net_t:s0 from system_u:system_r:unconfined_service_t:s0 failed with Permission denied [preauth]

Version-Release number of selected component (if applicable):

ovirt-engine-4.2.2.6-0.1.el7.noarch
rhv-release-4.2.2-9-001.noarch

Red Hat Enterprise Linux Server release 7.5 (Maipo)

How reproducible:
100% on HE env.

Steps to Reproduce:

1. Generate a set of public/private ssh keys via ssh-keygen
2. Copy the key from *.pub, e.g. "cat /root/.ssh/sc_id_rsa.pub"
3. Log in to the web ui, on the right top corner select an session icon (little man icon) and press "Options".
4. Paste the content of public key into "User's Public Key" field.
5. Try to get the list of vms via serial console using following command:
"ssh -v -i {private_key_path} -p 2222 ovirt-vmconsole@{host_engine_fqdn} list"

Actual results:
Permission denied when getenforce == 1

Expected results:
Serial console should work when getenforce == 1

Additional info:
Packages info:

packages_list.log in the attachments.

Output on serial console connection attempt.

connection_attempt.log in the attachments.

Comment 1 Vitalii Yerys 2018-04-04 15:02:35 UTC
Created attachment 1417299 [details]
connection_attempt.log

Comment 2 Vitalii Yerys 2018-04-04 15:02:59 UTC
Created attachment 1417301 [details]
packages_list

Comment 3 Michal Skrivanek 2018-04-05 05:15:11 UTC
You’re not supposed to connect to the console from engine itself, but it should behave the same even when you ssh externally anyway

Seems the vmconsole proxy deployment is not correct. Can you get install logs for the HE? And please get current selinux context settings(ls -Z) for ovirt-vmconsole-* files please

Comment 4 Vitalii Yerys 2018-04-05 12:28:39 UTC
Hi,

(In reply to Michal Skrivanek from comment #3)
> You’re not supposed to connect to the console from engine itself, but it
> should behave the same even when you ssh externally anyway

Actually I was connecting from my local machine (root@FENNEL:~#), it can be seen in connection_attempt.log, and in the log I have pasted "enforce" enable/disable console screens from the engine. (sorry if it created confusion)

> Seems the vmconsole proxy deployment is not correct. Can you get install
> logs for the HE? And please get current selinux context settings(ls -Z) for
> ovirt-vmconsole-* files please

Unfortunately I do not have HE env for personal usage, and using the one used for regression purposes, thus it is redeployed quite often. I was able to reproduce the issue on similar env (same packages/etc). Hope it will bring some light on the issue. 

Context settings:

[root@hosted-engine-02 ~]# ls -Z /etc/ovirt-vmconsole*
drwxr-xr-x. root root system_u:object_r:etc_t:s0       ovirt-vmconsole-proxy
[root@hosted-engine-02 ~]# ls -Z /etc/ovirt-vmconsole/ovirt-vmconsole-proxy*
drwxr-xr-x. root root system_u:object_r:etc_t:s0       conf.d
[root@hosted-engine-02 ~]# ls -Z /etc/ovirt-vmconsole/ovirt-vmconsole-proxy/conf.d/*
-rw-r--r--. root root system_u:object_r:etc_t:s0       /etc/ovirt-vmconsole/ovirt-vmconsole-proxy/conf.d/20-ovirt-vmconsole-proxy-helper.conf
-rw-r--r--. root root system_u:object_r:etc_t:s0       /etc/ovirt-vmconsole/ovirt-vmconsole-proxy/conf.d/NOTICE

Comment 5 Vitalii Yerys 2018-04-05 12:30:01 UTC
Created attachment 1417669 [details]
full_logs

Comment 6 Kobi Hakimi 2018-04-08 09:56:44 UTC
we set the appliance to enforce as part of the hosted engine deployment since the following bug resolved:
https://bugzilla.redhat.com/show_bug.cgi?id=1418579

Comment 7 Michal Skrivanek 2018-04-16 12:57:06 UTC
(In reply to Kobi Hakimi from comment #6)
> we set the appliance to enforce as part of the hosted engine deployment
> since the following bug resolved:
> https://bugzilla.redhat.com/show_bug.cgi?id=1418579

right, but it seems it's not correctly applied to vmconsole. It adds it's own policy and IIRC on RHVH you basically have to relabel afterwards. Since this issue doesn't exist for "host deploy" in normal deployments I believe this is specific to appliance build process and needs to be fixed there

Comment 8 Sandro Bonazzola 2018-04-20 14:23:02 UTC
Yuval can you please investigate on this?

Comment 10 Sandro Bonazzola 2018-04-20 14:26:32 UTC
Moved downstream since it implies RHEL 7.5 and CentOS 7.5 has not been released yet. If this reproduces also on oVirt Engine Appliance 4.2.2 based on CentOS 7.4 we'll move back to oVirt.

Comment 11 Yuval Turgeman 2018-04-22 08:35:24 UTC
Few problems:

1. The ovirt_vmconsole module for selinux is not loaded

[root@engine system]# semodule -l|grep vmcons
[root@engine system]# 

2. The context for /usr/libexec/ovirt-vmconsole-proxy-sshd is wrong:

[root@engine ~]# ls -lZ /usr/libexec/ovirt-vmconsole-proxy-sshd
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       /usr/libexec/ovirt-vmconsole-proxy-sshd


To fix this, I had to do the following:
semodule -i "/usr/share/selinux/packages/ovirt-vmconsole/ovirt_vmconsole.pp"
fixfiles -R ovirt-vmconsole-proxy restore
systemctl restart ovirt-vmconsole-proxy-sshd.service

Those actions should get executed in %post for ovirt-vmconsole and ovirt-vmconsole-proxy (if %{_sbindir}/selinuxenabled), but for some reason, they don't.  I'll check in the appliance build

Comment 12 Yuval Turgeman 2018-04-22 09:05:43 UTC
Looks like selinux-policy-targeted gets installed late in the build:

Apr 10 10:54:36 localhost.localdomain packaging[1494]: Installing ovirt-vmconsole (821/996)
Apr 10 10:55:56 localhost.localdomain packaging[1494]: Installing selinux-policy-targeted (966/996)

So https://gerrit.ovirt.org/#/c/79842/ should help, but it looks like this issue affects other rpms as well, see the packaging.log for tracebacks.

Comment 13 Yuval Turgeman 2018-04-22 11:50:55 UTC
Created attachment 1425309 [details]
anaconda packaging.log

Comment 15 Michal Skrivanek 2018-04-25 13:28:35 UTC
Yuval, package has been fixed, please verify with ovirt-vmconsole-1.0.5-4

Comment 16 Yuval Turgeman 2018-04-26 07:55:40 UTC
Thanks Michal, I actually verified this already by moving the installation of the RHV bits to %post in the kickstart, since the same issue happens in rh-postgresql95-postgresql-server

Comment 17 Sandro Bonazzola 2018-04-26 19:24:46 UTC
Moving bugs from depends to see also, since we don't need their fixes having introduced a workaround in appliance build process.

Comment 19 Nikolai Sednev 2018-05-08 14:06:04 UTC
Tested on these components:
ovirt-vmconsole-1.0.5-4.el7ev.noarch
ovirt-engine-4.2.3.3-0.1.el7.noarch
ovirt-hosted-engine-setup-2.2.20-1.el7ev.noarch
ovirt-hosted-engine-ha-2.2.11-1.el7ev.noarch
rhvm-appliance-4.2-20180427.0.el7.noarch
Linux 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.5 (Maipo)

alma04 ~]# getenforce 
Enforcing


# ssh -t -p 2222 ovirt-vmconsole@nsednev-he-2.qa.lab.tlv.redhat.com list 
c2c15782-895f-4bd3-976f-c5029852e310    HostedEngine
Connection to nsednev-he-2.qa.lab.tlv.redhat.com closed.
[root@nsednev ~]# ssh -v -i /root/.ssh/id_rsa -p 2222 ovirt-vmconsole@nsednev-he-2.qa.lab.tlv.redhat.com list
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Connecting to nsednev-he-2.qa.lab.tlv.redhat.com [10.35.92.52] port 2222.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to nsednev-he-2.qa.lab.tlv.redhat.com:2222 as 'ovirt-vmconsole'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-rsa-cert-v01@openssh.com
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host certificate: ssh-rsa-cert-v01@openssh.com SHA256:qe8FY+8uOkjjstP8FEI3DdZgC/KlULLwKZEkEDXsG4w, serial 0 ID "vmconsole-proxy-host" CA ssh-rsa SHA256:mEDiDbOurRn/XL1+CBxuwd5sDRaqMjVHHzR4zWg0uto valid from 2018-05-03T17:55:10 to 2023-04-07T18:55:10
debug1: checking without port identifier
debug1: No matching CA found. Retry with plain key
debug1: No matching CA found. Retry with plain key
debug1: Host '[nsednev-he-2.qa.lab.tlv.redhat.com]:2222' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:2
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
debug1: Authentication succeeded (publickey).
Authenticated to nsednev-he-2.qa.lab.tlv.redhat.com ([10.35.92.52]:2222).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Remote: Forced command.
debug1: Remote: Agent forwarding disabled.
debug1: Remote: Port forwarding disabled.
debug1: Remote: User rc execution disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Remote: Forced command.
debug1: Remote: Agent forwarding disabled.
debug1: Remote: Port forwarding disabled.
debug1: Remote: User rc execution disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending env LANGUAGE = 
debug1: Sending command: list
c2c15782-895f-4bd3-976f-c5029852e310    HostedEngine
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 2932, received 4096 bytes, in 0.8 seconds
Bytes per second: sent 3609.9, received 5043.0
debug1: Exit status 0



The SHE-VM itself won't be listed due to:
https://bugzilla.redhat.com/show_bug.cgi?id=1561964

Comment 20 Vitalii Yerys 2018-05-08 14:38:52 UTC
Verified with:
ovirt-vmconsole-1.0.5-4.el7ev.noarch
ovirt-engine-4.2.3.5-0.1.el7.noarch

Comment 21 Nikolai Sednev 2018-05-08 14:39:34 UTC
Correction to my previous comment, HE-VM is listed just fine as:
debug1: Sending command: list
c2c15782-895f-4bd3-976f-c5029852e310    HostedEngine

But will not be available for logging due to https://bugzilla.redhat.com/show_bug.cgi?id=1561964.

Comment 25 errata-xmlrpc 2018-05-15 19:00:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1525

Comment 26 Franta Kust 2019-05-16 13:09:03 UTC
BZ<2>Jira Resync


Note You need to log in before you can comment on or make changes to this bug.