Bug 1262003

Summary: [selinux][rhel-6.7] /bin/sh: Permission denied
Product: [oVirt] ovirt-vmconsole Reporter: Nikolai Sednev <nsednev>
Component: selinuxAssignee: Alon Bar-Lev <alonbl>
Status: CLOSED CURRENTRELEASE QA Contact: Nikolai Sednev <nsednev>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: masterCC: alonbl, bazulay, bugs, ecohen, fromani, gklein, iheim, istein, lsurette, oourfali, rbalakri, Rhev-m-bugs, yeylon
Target Milestone: ovirt-3.6.0-rcKeywords: Triaged
Target Release: 1.0.0Flags: rule-engine: ovirt-3.6.0+
rule-engine: blocker+
ylavi: planning_ack+
rule-engine: devel_ack+
rule-engine: testing_ack+
Hardware: x86_64   
OS: Linux   
Whiteboard: virt
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-27 07:55:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1223671, 1261519    
Attachments:
Description Flags
engine log
none
messages
none
engine log
none
messages none

Description Nikolai Sednev 2015-09-10 15:00:09 UTC 
Description of problem:
SELinux prevents serial console connections on RHEVM3.6.
I've tried to connect over serial console to the 3.6 engine and found that SELinux actively prevents this. To check if it's SELinux, I've disabled it and serial console session went through.

Version-Release number of selected component (if applicable):
On 3.6engine:
rhevm-setup-plugin-vmconsole-proxy-helper-3.6.0-0.13.master.el6.noarch
ovirt-vmconsole-proxy-1.0.0-0.0.1.master.el6ev.noarch
rhevm-vmconsole-proxy-helper-3.6.0-0.13.master.el6.noarch
jboss-as-console-2.5.6-2.Final_redhat_2.1.ep6.el6.noarch
ovirt-vmconsole-1.0.0-0.0.1.master.el6ev.noarch
rhevm-3.6.0-0.13.master.el6.noarch
Linux version 2.6.32-573.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Wed Jul 1 18:23:37 EDT 2015

How reproducible:
100%

Steps to Reproduce:
1.Install RHEVM3.6 with ovirt-vmconsole-proxy.
2.Add one clean RHEL7.2 host to the engine.
3.Create VM with serial console enabled on it and start the VM.
4.Create on your client pc/laptop on RHEL6.6 public and secret keys using "ssh-keygen", you should get created the $HOME/.ssh/id_rsa and $HOME/.ssh/id_rsa.pub
5.Copy the key from id_rsa.pub, e.g. "cat $HOME/.ssh/id_rsa.pub"
6.Log in to the WEBUI of the engine and then go to Virtual machines tab->on the top just righter than "Export" you will find symbol of "Play"; click on it and select "Set Serial Console Key", then paste the public key there.
7.Try to connect to serial console from you PC using " ssh -v -t -i $HOME/.ssh/id_rsa -p 2222 ovirt-vmconsole@<IP_OF_YOUR_ENGINE> connect".


Actual results:
Permission denied (publickey).

Expected results:
SSH session should be opened to the engine successfully.
Comment 1 Nikolai Sednev 2015-09-10 15:03:23 UTC 
Created attachment 1072244 [details]
engine log
Comment 2 Alon Bar-Lev 2015-09-10 15:39:48 UTC 
where do you see that this is selinux? from the description it is unclear how this conclusion is derived.

please enable debug for the ovirt-vmconsole-proxy.

/etc/ovirt-vmconsole/ovirt-vmconsole-proxy/conf.d/debug.conf
---
[proxy]
debug = true
---

debug messages should appear at /var/log/messages

retry and provide the log.

thanks.
Comment 3 Alon Bar-Lev 2015-09-10 15:48:02 UTC 
Also, please please please, before you attach logs, stop all services, clear all logs, start services, reproduce, then collect logs.

abusing bugzilla with 3M log with no relevant data is not required.
Comment 4 Nikolai Sednev 2015-09-10 15:55:32 UTC 
(In reply to Alon Bar-Lev from comment #3)
> Also, please please please, before you attach logs, stop all services, clear
> all logs, start services, reproduce, then collect logs.
> 
> abusing bugzilla with 3M log with no relevant data is not required.

I tested this together with fromani and we disabled the SELINUX to get this working, once it was shut off, we got through and at least saw the VM to get connected to.
Comment 5 Alon Bar-Lev 2015-09-10 15:58:40 UTC 
(In reply to Nikolai Sednev from comment #4)
> (In reply to Alon Bar-Lev from comment #3)
> > Also, please please please, before you attach logs, stop all services, clear
> > all logs, start services, reproduce, then collect logs.
> > 
> > abusing bugzilla with 3M log with no relevant data is not required.
> 
> I tested this together with fromani and we disabled the SELINUX to get this
> working, once it was shut off, we got through and at least saw the VM to get
> connected to.

activity in relevant logs? for example /var/log/audit/audit.log once connecting.
and once again, without proxy/host log no root cause can be done.
where have you disabled selinux? proxy or host (or both).
what differ from this environment and other environments in which it is working?
this is all data you can collect.
Comment 6 Nikolai Sednev 2015-09-10 16:13:08 UTC 
As a matter of fact, I setenforce 1 and tried to connect to serial console:
# ssh -v -t -i $HOME/.ssh/id_rsa -p 2222 ovirt-vmconsole.160.204 connect
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013                                               
debug1: Reading configuration data /etc/ssh/ssh_config                                       
debug1: /etc/ssh/ssh_config line 56: Applying options for *                                  
debug1: Connecting to 10.35.160.204 [10.35.160.204] port 2222.                               
debug1: Connection established.                                                              
debug1: permanently_set_uid: 0/0                                                             
debug1: identity file /root/.ssh/id_rsa type 1                                               
debug1: identity file /root/.ssh/id_rsa-cert type -1                                         
debug1: Enabling compatibility mode for protocol 2.0                                         
debug1: Local version string SSH-2.0-OpenSSH_6.6.1                                           
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3                     
debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000                                  
debug1: SSH2_MSG_KEXINIT sent                                                                
debug1: SSH2_MSG_KEXINIT received                                                            
debug1: kex: server->client aes128-ctr hmac-md5 none                                         
debug1: kex: client->server aes128-ctr hmac-md5 none                                         
debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16                         
debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16                         
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<3072<8192) sent                                     
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP                                                  
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent                                                        
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY                                                  
debug1: ssh_rsa_verify: signature correct                                                    
debug1: Server host key: RSA-CERT 04:52:9f:4b:cf:11:67:8c:8d:ee:1b:48:84:2f:dc:b7            
debug1: checking without port identifier                                                     
debug1: No matching CA found. Retry with plain key                                           
debug1: No matching CA found. Retry with plain key                                           
debug1: Host '[10.35.160.204]:2222' is known and matches the RSA host key.                   
debug1: Found key in /root/.ssh/known_hosts:2                                                
debug1: ssh_rsa_verify: signature correct                                                    
debug1: SSH2_MSG_NEWKEYS sent                                                                
debug1: expecting SSH2_MSG_NEWKEYS                                                           
debug1: SSH2_MSG_NEWKEYS received                                                            
debug1: Roaming not allowed by server                                                        
debug1: SSH2_MSG_SERVICE_REQUEST sent                                                        
debug1: SSH2_MSG_SERVICE_ACCEPT received                                                     
debug1: Authentications that can continue: publickey                                         
debug1: Next authentication method: publickey                                                
debug1: Offering RSA public key: /root/.ssh/id_rsa                                           
debug1: Remote: Forced command: exec "/usr/libexec/ovirt-vmconsole-proxy-shell"  accept --entityid="00000018-0018-0018-0018-0000000000c0" --entity="user-id"                                                                    
debug1: Remote: Agent forwarding disabled.                                                                      
debug1: Remote: Port forwarding disabled.                                                                       
debug1: Remote: User rc file execution disabled.                                                                
debug1: Remote: X11 forwarding disabled.                                                                        
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: Remote: Forced command: exec "/usr/libexec/ovirt-vmconsole-proxy-shell"  accept --entityid="00000018-0018-0018-0018-0000000000c0" --entity="user-id"
debug1: Remote: Agent forwarding disabled.
debug1: Remote: Port forwarding disabled.
debug1: Remote: User rc file execution disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Authentication succeeded (publickey).
Authenticated to 10.35.160.204 ([10.35.160.204]:2222).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env XMODIFIERS = @im=none
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending env LANGUAGE =
debug1: Sending command: connect
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow reply 0
/bin/sh: Permission denied
debug1: channel 0: free: client-session, nchannels 1
Connection to 10.35.160.204 closed.
Transferred: sent 4056, received 4112 bytes, in 0.2 seconds
Bytes per second: sent 19110.7, received 19374.6
debug1: Exit status 1



Now I setenforce 0 on engine and tried again to connect via serial console:
# ssh -v -t -i $HOME/.ssh/id_rsa -p 2222 ovirt-vmconsole.160.204 connect
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013                                               
debug1: Reading configuration data /etc/ssh/ssh_config                                       
debug1: /etc/ssh/ssh_config line 56: Applying options for *                                  
debug1: Connecting to 10.35.160.204 [10.35.160.204] port 2222.                               
debug1: Connection established.                                                              
debug1: permanently_set_uid: 0/0                                                             
debug1: identity file /root/.ssh/id_rsa type 1                                               
debug1: identity file /root/.ssh/id_rsa-cert type -1                                         
debug1: Enabling compatibility mode for protocol 2.0                                         
debug1: Local version string SSH-2.0-OpenSSH_6.6.1                                           
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3                     
debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000                                  
debug1: SSH2_MSG_KEXINIT sent                                                                
debug1: SSH2_MSG_KEXINIT received                                                            
debug1: kex: server->client aes128-ctr hmac-md5 none                                         
debug1: kex: client->server aes128-ctr hmac-md5 none                                         
debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16                         
debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16                         
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<3072<8192) sent                                     
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP                                                  
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent                                                        
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY                                                  
debug1: ssh_rsa_verify: signature correct                                                    
debug1: Server host key: RSA-CERT 04:52:9f:4b:cf:11:67:8c:8d:ee:1b:48:84:2f:dc:b7            
debug1: checking without port identifier                                                     
debug1: No matching CA found. Retry with plain key                                           
debug1: No matching CA found. Retry with plain key                                           
debug1: Host '[10.35.160.204]:2222' is known and matches the RSA host key.                   
debug1: Found key in /root/.ssh/known_hosts:2                                                
debug1: ssh_rsa_verify: signature correct                                                    
debug1: SSH2_MSG_NEWKEYS sent                                                                
debug1: expecting SSH2_MSG_NEWKEYS                                                           
debug1: SSH2_MSG_NEWKEYS received                                                            
debug1: Roaming not allowed by server                                                        
debug1: SSH2_MSG_SERVICE_REQUEST sent                                                        
debug1: SSH2_MSG_SERVICE_ACCEPT received                                                     
debug1: Authentications that can continue: publickey                                         
debug1: Next authentication method: publickey                                                
debug1: Offering RSA public key: /root/.ssh/id_rsa                                           
debug1: Remote: Forced command: exec "/usr/libexec/ovirt-vmconsole-proxy-shell"  accept --entityid="00000018-0018-0018-0018-0000000000c0" --entity="user-id"
debug1: Remote: Agent forwarding disabled.
debug1: Remote: Port forwarding disabled.
debug1: Remote: User rc file execution disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: Remote: Forced command: exec "/usr/libexec/ovirt-vmconsole-proxy-shell"  accept --entityid="00000018-0018-0018-0018-0000000000c0" --entity="user-id"
debug1: Remote: Agent forwarding disabled.
debug1: Remote: Port forwarding disabled.
debug1: Remote: User rc file execution disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Authentication succeeded (publickey).
Authenticated to 10.35.160.204 ([10.35.160.204]:2222).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env XMODIFIERS = @im=none
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending env LANGUAGE =
debug1: Sending command: connect
Available Serial Consoles:
00 test[8ab54a0a-7eb1-4112-9456-0499e654ed9a]
SELECT>



/var/log/messages from the engine attached.
Comment 7 Nikolai Sednev 2015-09-10 16:13:59 UTC 
Created attachment 1072271 [details]
messages
Comment 8 Alon Bar-Lev 2015-09-10 16:15:40 UTC 
Comment on attachment 1072271 [details]
messages

again, no need for  1.74 MB file, only for the duration of reproduction. please do not ignore my comments.
Comment 9 Alon Bar-Lev 2015-09-10 16:21:55 UTC 
(In reply to Alon Bar-Lev from comment #8)
> Comment on attachment 1072271 [details]
> messages
> 
> again, no need for  1.74 MB file, only for the duration of reproduction.
> please do not ignore my comments.

also debug is not enabled per requested in comment#5.
and what the log contains? how can I know if it the attempt before you disabled selinux or after?

ilanit... very hard to work this way... too much noise, I need your help.
Comment 10 Nikolai Sednev 2015-09-10 16:25:59 UTC 
Created attachment 1072274 [details]
engine log
Comment 11 Nikolai Sednev 2015-09-10 16:26:36 UTC 
Created attachment 1072275 [details]
messages
Comment 12 Alon Bar-Lev 2015-09-10 16:32:19 UTC 
(In reply to Nikolai Sednev from comment #11)
> Created attachment 1072275 [details]
> messages

once again, this is complete log instead of per reproduction / action. not sure why you keep ignoring my instructions. you bloat bugzilla database, you send irrelevant data to us, it is impossible to work this way.
Comment 13 Alon Bar-Lev 2015-09-10 16:36:02 UTC 
this should not be, probably something else is wrong:
---
Sep 10 19:23:09 nsednev-rhevm-smoketests-3-5 2015-09-10 19:23:09,336 ovirt-engine: ERROR run:532 Error: File '/var/log/ovirt-engine/engine.log' cannot be accessed for writing
---

also this suggest for invalid configuration/installation:
---
Sep 10 19:23:55 nsednev-rhevm-smoketests-3-5 2015-09-10 19:23:55,632 ovirt-vmconsole-list: ERROR main:167 Error: HTTP Error 404: Not Found
Sep 10 19:23:55 nsednev-rhevm-smoketests-3-5 ovirt-vmconsole-proxy-keys[56650]: ERROR Key list execution failed rc=1
---

these are not selinux issues.

please prepare an environment that reproduce this, and send me credentials I will take it from there and reduce ping pong. what sad is that next time we repeat this sequence again.
Comment 14 Nikolai Sednev 2015-09-10 18:54:46 UTC 
(In reply to Alon Bar-Lev from comment #13)
> this should not be, probably something else is wrong:
> ---
> Sep 10 19:23:09 nsednev-rhevm-smoketests-3-5 2015-09-10 19:23:09,336
> ovirt-engine: ERROR run:532 Error: File '/var/log/ovirt-engine/engine.log'
> cannot be accessed for writing
> ---
> 
> also this suggest for invalid configuration/installation:
> ---
> Sep 10 19:23:55 nsednev-rhevm-smoketests-3-5 2015-09-10 19:23:55,632
> ovirt-vmconsole-list: ERROR main:167 Error: HTTP Error 404: Not Found
> Sep 10 19:23:55 nsednev-rhevm-smoketests-3-5
> ovirt-vmconsole-proxy-keys[56650]: ERROR Key list execution failed rc=1
> ---
> 
> these are not selinux issues.
> 
> please prepare an environment that reproduce this, and send me credentials I
> will take it from there and reduce ping pong. what sad is that next time we
> repeat this sequence again.

Done, environment prepared and details sent via email.
Comment 15 Alon Bar-Lev 2015-09-10 19:01:54 UTC 
/var/log/audit/audit.log
---
type=AVC msg=audit(1441911617.306:218): avc:  denied  { create } for  pid=55047 comm="sshd" scontext=unconfined_u:system_r:ovirt_vmconsole_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:prelink_mask_t:s0-s0:c0.c1023 tclass=key
type=SYSCALL msg=audit(1441911617.306:218): arch=c000003e syscall=1 success=no exit=-13 a0=4 a1=7f61292c9910 a2=34 a3=726379656b2f7274 items=0 ppid=54997 pid=55047 auid=0 uid=248 gid=214 euid=248 suid=248 fsuid=248 egid=214 sgid=214 fsgid=214 tty=(none) ses=11 comm="sshd" exe="/usr/sbin/sshd" subj=unconfined_u:system_r:ovirt_vmconsole_t:s0-s0:c0.c1023 key=(null)
type=AVC msg=audit(1441911617.312:219): avc:  denied  { dyntransition } for  pid=55060 comm="sshd" scontext=unconfined_u:system_r:ovirt_vmconsole_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:prelink_mask_t:s0-s0:c0.c1023 tclass=process
type=SYSCALL msg=audit(1441911617.312:219): arch=c000003e syscall=1 success=no exit=-13 a0=5 a1=7f61292c9890 a2=34 a3=65727275632f7274 items=0 ppid=55047 pid=55060 auid=0 uid=248 gid=214 euid=248 suid=248 fsuid=248 egid=214 sgid=214 fsgid=214 tty=(none) ses=11 comm="sshd" exe="/usr/sbin/sshd" subj=unconfined_u:system_r:ovirt_vmconsole_t:s0-s0:c0.c1023 key=(null)
type=AVC msg=audit(1441911618.291:220): avc:  denied  { entrypoint } for  pid=55061 comm="sshd" path="/bin/bash" dev=dm-0 ino=2078 scontext=unconfined_u:system_r:prelink_mask_t:s0-s0:c0.c1023 tcontext=system_u:object_r:shell_exec_t:s0 tclass=file
type=SYSCALL msg=audit(1441911618.291:220): arch=c000003e syscall=59 success=no exit=-13 a0=7f61292ca5e0 a1=7ffca7fe8e80 a2=7f61292ca9d0 a3=0 items=0 ppid=55060 pid=55061 auid=0 uid=248 gid=214 euid=248 suid=248 fsuid=248 egid=214 sgid=214 fsgid=214 tty=pts2 ses=11 comm="sshd" exe="/usr/sbin/sshd" subj=unconfined_u:system_r:ovirt_vmconsole_t:s0-s0:c0.c1023 key=(null)
---
Comment 16 Alon Bar-Lev 2015-09-10 22:53:42 UTC 
There were changes in selinux policy of rhel.
Comment 17 Alon Bar-Lev 2015-09-10 23:47:31 UTC 
openssh is using nss which uses a library for fips which is prelinked, so our type need this as well.
Comment 18 Nikolai Sednev 2015-11-09 08:56:44 UTC 
Works for me on these components:
Engine:
Red Hat Enterprise Virtualization Manager Version: 3.6.0.3-0.1.el6 
rhevm-3.6.0.3-0.1.el6.noarch
virt-vmconsole-proxy-1.0.0-1.el6ev.noarch
ovirt-engine-extension-aaa-jdbc-1.0.1-1.el6ev.noarch
ovirt-host-deploy-1.4.0-1.el6ev.noarch
ovirt-host-deploy-java-1.4.0-1.el6ev.noarch
ovirt-vmconsole-1.0.0-1.el6ev.noarch
Linux version 2.6.32-573.7.1.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Thu Sep 10 13:42:16 EDT 2015

Host:
ovirt-vmconsole-1.0.1-0.0.master.20151105234454.git3e5d52e.el7.noarch
ovirt-release36-snapshot-001-2.noarch
ovirt-vmconsole-host-1.0.1-0.0.master.20151105234454.git3e5d52e.el7.noarch
ovirt-release36-001-2.noarch
vdsm-4.17.10.1-0.el7.centos.noarch
libvirt-client-1.2.17-13.el7.x86_64
mom-0.5.1-2.el7.noarch
qemu-kvm-rhev-2.3.0-31.el7.x86_64
sanlock-3.2.4-1.el7.x86_64
Linux version 3.10.0-322.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Mon Oct 5 21:41:10 EDT 2015


debug1: Sending command: connect
Available Serial Consoles:
00 RHEL7_2_VM_1[ea857677-f6d2-4d16-a40a-a44222670482]
01 RHEL7_2_VM_1[ea857677-f6d2-4d16-a40a-a44222670482]
SELECT> 00


Red Hat Enterprise Linux Server 7.2 (Maipo)
Kernel 3.10.0-327.el7.x86_64 on an x86_64

localhost login: root
Password: 
Login incorrect

localhost login: root
Password: 
Last failed login: Mon Nov  9 10:51:08 IST 2015 on hvc0
There was 1 failed login attempt since the last successful login.
Last login: Sun Nov  8 21:45:11 on hvc0
[root@localhost ~]# 



[root@nsednev-rhevm-smoketests-3-5 ~]# getenforce 
Enforcing
[root@alma02 ~]# getenforce 
Enforcing
[root@black-vdsb ~]# getenforce 
Enforcing
Comment 19 Sandro Bonazzola 2015-11-27 07:55:51 UTC 
Since oVirt 3.6.0 has been released, moving from verified to closed current release.