Bug 1419829

Summary: avc denied errors (sshd_t) in audit.log after upgrade
Product: [oVirt] ovirt-node Reporter: Huijuan Zhao <huzhao>
Component: Installation & UpdateAssignee: Ryan Barry <rbarry>
Status: CLOSED CURRENTRELEASE QA Contact: Huijuan Zhao <huzhao>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.1CC: bugs, cshao, dguo, dougsland, fromani, huzhao, jiawu, leiwang, lvrabec, mgoldboi, michal.skrivanek, qiyuan, rbarry, sbonazzo, stirabos, weiwang, yaniwang, ycui, yzhao
Target Milestone: ovirt-4.1.3Flags: rule-engine: ovirt-4.1+
mgoldboi: blocker-
rule-engine: planning_ack+
sbonazzo: devel_ack+
cshao: testing_ack+
Target Release: 4.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: imgbased-0.9.28-0.1.el7ev Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-07-06 14:04:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1457670    
Bug Blocks:    
Attachments:
Description Flags
All logs and sosreport
none
Comment 36: sosreport and all logs in /var/log, /tmp none

Description Huijuan Zhao 2017-02-07 08:04:08 UTC
Created attachment 1248307 [details]
All logs and sosreport

Description of problem:
avc denied errors (sshd_t) in /var/log/audit/audit.log after upgrade.

# grep "avc:  denied" /var/log/audit/audit.log
type=AVC msg=audit(1486452770.275:69): avc:  denied  { name_bind } for  pid=2909 comm="sshd" src=2223 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:ovirt_vmconsole_host_port_t:s0 tclass=tcp_socket

# imgbase layout
rhvh-4.0-0.20160817.0
 +- rhvh-4.0-0.20160817.0+1
rhvh-4.0-0.20170201.0
 +- rhvh-4.0-0.20170201.0+1

Version-Release number of selected component (if applicable):
1. Before upgrade:
redhat-virtualization-host-4.0-20160817.0
2. After upgrade:
redhat-virtualization-host-4.0-20170201.0
imgbased-0.8.11-0.1.el7ev.noarch
kernel-3.10.0-514.2.2.el7.x86_64

How reproducible:
Tested twice with above builds, both can reproduce.

Steps to Reproduce:
1. Install RHVH 4.0 GA build redhat-virtualization-host-4.0-20160817.0 via interactive anaconda.
2. Login RHVH, add RHVH to RHVM 4.0(4.0.6.3-0.1.el7ev), and add NFS storage to host.
3. Login RHVH, download update rpm and upgrade RHVH:
   # yum install redhat-virtualization-host-image-update-4.0-20170201.0.el7_3.noarch.rpm
4. Reboot and login the new build RHVH-4.0-20170201
5. # grep "avc:  denied" /var/log/audit/audit.log


Actual results:
After step5, avc denied errors (sshd_t) in audit.log after upgrade:
# grep "avc:  denied" /var/log/audit/audit.log
type=AVC msg=audit(1486452770.275:69): avc:  denied  { name_bind } for  pid=2909 comm="sshd" src=2223 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:ovirt_vmconsole_host_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1486452770.276:70): avc:  denied  { name_bind } for  pid=2909 comm="sshd" src=2223 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:ovirt_vmconsole_host_port_t:s0 tclass=tcp_socket

Excepted results:
After step5, there should be no avc error.

Additional info:
No such issue on clean RHVH(no update) 4.0.6 build redhat-virtualization-host-image-update-4.0-20170201.0

Comment 1 Sandro Bonazzola 2017-02-07 09:04:29 UTC
Francesco, is this a vmconsole bug?

Comment 2 Francesco Romani 2017-02-07 09:59:35 UTC
Well, it's a bug in the sense that we had a selinux denial and the vmconsole doesn't start, as result, but AFAIK vmconsole works OK on RHEL 7.3, so I don't think it's just a vmconsole issue.

Do we have evidence of vmconsole issues on plain 7.3?

Comment 3 Douglas Schilling Landgraf 2017-02-07 16:07:30 UTC
let me try to investigate.

Comment 4 Douglas Schilling Landgraf 2017-02-08 04:33:07 UTC
The AVC looks real but I couldn't trigger it in pure rhel 7.3 yet. 

If we can reproduce it, I would suggest add:

allow sshd_t ovirt_vmconsole_host_port_t:tcp_socket name_bind;

to:

https://github.com/oVirt/ovirt-vmconsole/blob/master/selinux-ovirt_vmconsole/ovirt_vmconsole.te

I see the sshd_t type defined in ovirt_vmconsole.te but never used as the allow command showed above.

To make the error stop for now, need to build and install selinux policy:

# audit2allow -M myerroutput < /var/log/audit/audit.log 

# cat myerroutput.te
module my 1.0;

require {
	type ovirt_vmconsole_host_port_t;
	type sshd_t;
	class tcp_socket name_bind;
}

# semodule -i myerroutput.pp
# echo > /var/log/audit/audit.log
# reboot

After reboot, no more denied.

Huijuan Zhao, could you please give a new try in the reproducer maybe upgrading from 7.2 to 7.3 in pure rhel? Upgrading from redhat-virtualization-host-4.0-20160817.0.x86_64.liveimg.squashfs to redhat-virtualization-host-image-update-4.0-20170201.0.el7_3.noarch.rpm is such scenario.

Comment 5 Francesco Romani 2017-02-08 08:30:36 UTC
(In reply to Douglas Schilling Landgraf from comment #4)
> The AVC looks real but I couldn't trigger it in pure rhel 7.3 yet. 
> 
> If we can reproduce it, I would suggest add:
> 
> allow sshd_t ovirt_vmconsole_host_port_t:tcp_socket name_bind;
> 
> to:
> 
> https://github.com/oVirt/ovirt-vmconsole/blob/master/selinux-ovirt_vmconsole/
> ovirt_vmconsole.te
> 
> I see the sshd_t type defined in ovirt_vmconsole.te but never used as the
> allow command showed above.

Hi Douglas!

I can add this easily but I'm not sure I understand all the implications (aka if there is any hidden interaction), so I'd like to have another review from another selinux expert. I will add you to the patch, please suggest additional reviewers.

Comment 6 Huijuan Zhao 2017-02-08 08:58:13 UTC
(In reply to Douglas Schilling Landgraf from comment #4) 
> Huijuan Zhao, could you please give a new try in the reproducer maybe
> upgrading from 7.2 to 7.3 in pure rhel? Upgrading from
> redhat-virtualization-host-4.0-20160817.0.x86_64.liveimg.squashfs to
> redhat-virtualization-host-image-update-4.0-20170201.0.el7_3.noarch.rpm is
> such scenario.

Douglas, I tried in the reproducer upgrade from rhel 7.2(kernel-3.10.0-327.el7.x86_64) to rhel 7.3(kernel-3.10.0-514.el7.x86_64), but can not reproduce this issue, did not encounter avc denied errors after upgrade in pure rhel.

And according to comment 4 steps to run audit2allow in the reproducer(upgrade from redhat-virtualization-host-4.0-20160817.0.x86_64.liveimg.squashfs to redhat-virtualization-host-image-update-4.0-20170201.0.el7_3.noarch.rpm), after reboot, no more avc denied error, same as your test results.

Comment 7 Douglas Schilling Landgraf 2017-02-08 19:41:51 UTC
Hi,

I couldn't find any difference in the systems and I believe the AVC is something real and can be handle by this patch: https://gerrit.ovirt.org/#/c/71919/

Lukas, could you please share your thoughts on this patch? 

I have tested in both systems rebuilding ovirt-vmconsole and it worked and removed the avc as well.

# yum install selinux-policy-devel
# make -f /usr/share/selinux/devel/Makefile
Compiling targeted ovirt_vmconsole module
/usr/bin/checkmodule:  loading policy configuration from tmp/ovirt_vmconsole.tmp
/usr/bin/checkmodule:  policy configuration loaded
/usr/bin/checkmodule:  writing binary representation (version 17) to tmp/ovirt_vmconsole.mod
Creating targeted ovirt_vmconsole.pp policy package
rm tmp/ovirt_vmconsole.mod.fc tmp/ovirt_vmconsole.mod

# semodule -i ovirt_vmconsole.pp 
# 


From both machines with the change loaded:

(Pure RHEL 7.3)
# sesearch -A -s sshd_t -c tcp_socket -p name_bind
Found 11 semantic av rules:
   allow sshd_t ssh_port_t : tcp_socket name_bind ; 
   allow sshd_t xserver_port_t : tcp_socket name_bind ; 
   allow sshd_t vnc_port_t : tcp_socket name_bind ; 
   allow sshd_t ovirt_vmconsole_host_port_t : tcp_socket name_bind ; 
   allow nsswitch_domain port_t : tcp_socket name_bind ; 
   allow sshd_t rpc_port_type : tcp_socket name_bind ; 
   allow sshd_t unreserved_port_t : tcp_socket name_bind ; 
   allow sshd_t port_t : tcp_socket name_bind ; 
   allow nsswitch_domain ephemeral_port_t : tcp_socket name_bind ; 
   allow nsswitch_domain unreserved_port_t : tcp_socket name_bind ; 
   allow sshd_t ephemeral_port_t : tcp_socket name_bind ; 


# sesearch -A -s sshd_t -c tcp_socket -p name_bind
Found 11 semantic av rules:
   allow sshd_t ssh_port_t : tcp_socket name_bind ; 
   allow sshd_t xserver_port_t : tcp_socket name_bind ; 
   allow sshd_t vnc_port_t : tcp_socket name_bind ; 
   allow sshd_t ovirt_vmconsole_host_port_t : tcp_socket name_bind ; 
   allow nsswitch_domain port_t : tcp_socket name_bind ; 
   allow sshd_t rpc_port_type : tcp_socket name_bind ; 
   allow sshd_t unreserved_port_t : tcp_socket name_bind ; 
   allow sshd_t port_t : tcp_socket name_bind ; 
   allow nsswitch_domain ephemeral_port_t : tcp_socket name_bind ; 
   allow nsswitch_domain unreserved_port_t : tcp_socket name_bind ; 
   allow sshd_t ephemeral_port_t : tcp_socket name_bind ; 

I also checked in EL6.8 contains such requirements:

 # sesearch -A -s sshd_t -c tcp_socket -p name_bind | grep -i ssh_port_t
 allow sshd_t ssh_port_t : tcp_socket name_bind ; 


(Pure RHEL 7.3)
# rpm -qa | grep -i selinux | sort
libselinux-2.5-6.el7.x86_64
libselinux-python-2.5-6.el7.x86_64
libselinux-utils-2.5-6.el7.x86_64
selinux-policy-3.13.1-102.el7_3.13.noarch
selinux-policy-devel-3.13.1-102.el7_3.13.noarch
selinux-policy-targeted-3.13.1-102.el7_3.13.noarch


RHVH:
# rpm -qa | grep -i selinux| sort
libselinux-2.5-6.el7.x86_64
libselinux-python-2.5-6.el7.x86_64
libselinux-utils-2.5-6.el7.x86_64
selinux-policy-3.13.1-102.el7_3.13.noarch
selinux-policy-devel-3.13.1-102.el7_3.13.noarch
selinux-policy-targeted-3.13.1-102.el7_3.13.noarch

Both systems:
# rpm -qa | grep -i ovirt-vm | sort
ovirt-vmconsole-1.0.4-1.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch


# /bin/systemctl  status ovirt-vmconsole-host-sshd.service 
● ovirt-vmconsole-host-sshd.service - oVirt VM Console SSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/ovirt-vmconsole-host-sshd.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2017-02-09 02:09:02 CST; 1h 31min ago
 Main PID: 1488 (sshd)
   CGroup: /system.slice/ovirt-vmconsole-host-sshd.service
           └─1488 /usr/sbin/sshd -f /usr/share/ovirt-vmconsole/ovirt-vmconsole-host/ovirt-vmconsole-host-sshd/sshd_config -D

Feb 09 02:09:02 localhost systemd[1]: Started oVirt VM Console SSH server daemon.
Feb 09 02:09:02 localhost systemd[1]: Starting oVirt VM Console SSH server daemon...
Feb 09 02:09:03 localhost sshd[1488]: Server listening on 0.0.0.0 port 2223.
Feb 09 02:09:03 localhost sshd[1488]: Server listening on :: port 2223.

Thanks!

Comment 8 Sandro Bonazzola 2017-02-14 09:10:29 UTC
Francesco, can you please review?

Comment 9 Lukas Vrabec 2017-02-14 09:27:45 UTC
Hi, 

Patch looks fine from SELinux POV. If you need sshd binding on port 2223, this is way how to fix it. 

Thanks,
Lukas.

Comment 10 Francesco Romani 2017-02-14 13:41:31 UTC
merging the patch is cheap, but I still have a couple of questions I can't help but ask
1. why does it work with plain RHEL 7.3?
2. why did it broke now? what changed?

Comment 11 Francesco Romani 2017-02-14 13:48:55 UTC
On a CentOS 7.3 (same selinux conf of plain RHEL7.3) I see this

$ ps -auxwZ | grep ssh

system_u:system_r:ovirt_vmconsole_t:s0-s0:c0.c1023 ovirt-v+ 1427 0.0  0.0 82468 3548 ? Ss Feb13   0:00 /usr/sbin/sshd -f /usr/share/ovirt-vmconsole/ovirt-vmconsole-host/ovirt-vmconsole-host-sshd/sshd_config -D
system_u:system_r:sshd_t:s0-s0:c0.c1023 root 1438 0.0  0.0 82468 1332 ?        Ss   Feb13   0:00 /usr/sbin/sshd
system_u:system_r:sshd_t:s0-s0:c0.c1023 root 3282 0.2  0.0 145056 5144 ?       Ss   14:42   0:00 sshd: fromani [priv]
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 fromani 3287 0.0  0.0 145056 2248 ? S 14:42   0:00 sshd: fromani@pts/0
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 fromani 3382 0.0  0.0 112648 964 pts/0 S+ 14:43   0:00 grep --color=auto ssh

So, the system sshd runs with sshd_t context, while the ovirt-vmconsole sshd runs in ovirt_vmconsole_t context

Which makes sense.

I'm under the impression that somehow on RHV-N the ovirt-vmconsole sshd process is started under the wrong context.

So the proposed patch fixes the issue at hand, but not sure it fixes the root cause.

The service file

$ cat /usr/lib/systemd/system/ovirt-vmconsole-host-sshd.service
[Unit]
Description=oVirt VM Console SSH server daemon
After=syslog.target network.target auditd.service

[Service]
EnvironmentFile=-/etc/sysconfig/ovirt-vmconsole-host-sshd
User=ovirt-vmconsole
Group=ovirt-vmconsole
ExecStart=/usr/libexec/ovirt-vmconsole-host-sshd -f /usr/share/ovirt-vmconsole/ovirt-vmconsole-host/ovirt-vmconsole-host-sshd/sshd_config -D $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process

[Install]
WantedBy=multi-user.target

Its context: 
$ ls -lhZ /usr/libexec/ovirt-vmconsole-host-sshd 
-rwxr-xr-x. root root system_u:object_r:ovirt_vmconsole_exec_t:s0 /usr/libexec/ovirt-vmconsole-host-sshd

Does this match what we have on RHV-N?

Comment 13 Douglas Schilling Landgraf 2017-02-15 22:04:21 UTC
Before upgrade:
===================

#  ps -auxwZ | grep ssh
system_u:system_r:ovirt_vmconsole_t:s0-s0:c0.c1023 ovirt-v+ 1360 0.0  0.0 82468 3856 ? Ss Feb07   0:00 /usr/sbin/sshd -f /usr/share/ovirt-vmconsole/ovirt-vmconsole-host/ovirt-vmconsole-host-sshd/sshd_config -D
system_u:system_r:sshd_t:s0-s0:c0.c1023 root 1419 0.0  0.0 82468 1348 ?        Ss   Feb07   0:00 /usr/sbin/sshd
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 29484 2.0  0.0 145032 5420 ? Ss 05:36   0:00 sshd: root@pts/0
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 29527 0.0  0.0 112652 960 pts/0 S+ 05:36   0:00 grep --color=auto ssh

# ls -lhZ /usr/libexec/ovirt-vmconsole-host-sshd 
-rwxr-xr-x. root root system_u:object_r:ovirt_vmconsole_exec_t:s0 /usr/libexec/ovirt-vmconsole-host-sshd

After upgrade
===============

[root@localhost ~]#  ps -auxwZ | grep ssh
system_u:system_r:ovirt_vmconsole_t:s0-s0:c0.c1023 ovirt-v+ 1476 0.0  0.0 82468 3856 ? Ss 05:57   0:00 /usr/sbin/sshd -f /usr/share/ovirt-vmconsole/ovirt-vmconsole-host/ovirt-vmconsole-host-sshd/sshd_config -D
system_u:system_r:sshd_t:s0-s0:c0.c1023 root 1546 0.0  0.0 82468 1272 ?        Ss   05:57   0:00 /usr/sbin/sshd
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 23717 1.9  0.0 145016 5424 ? Ss 06:00   0:00 sshd: root@pts/0
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 23760 0.0  0.0 112652 960 pts/0 S+ 06:00   0:00 grep --color=auto ssh

[root@localhost ~]# ls -lhZ /usr/libexec/ovirt-vmconsole-host-sshd 
-rwxr-xr-x. root root system_u:object_r:ovirt_vmconsole_exec_t:s0 /usr/libexec/ovirt-vmconsole-host-sshd

Comment 16 Sandro Bonazzola 2017-02-16 08:35:03 UTC
Francesco, anythiing preventing the patch to be merged?

Comment 18 Francesco Romani 2017-02-16 10:27:35 UTC
(In reply to Sandro Bonazzola from comment #16)
> Francesco, anythiing preventing the patch to be merged?

Sandro, I believe that the patch is not the best fix.
It works, but does not address the root cause.

I had a chance to peek into a faulty environment, and discovered what follows:

In the faulty environement, here's the SELinux labels of the launcher scripts
# ls -lhZ /usr/libexec/ovirt-vmconsole-host-sshd
-rwxr-xr-x. root root system_u:object_r:unlabeled_t:s0 /usr/libexec/ovirt-vmconsole-host-sshd

Please note this is a dumb helper, mostly needed by the SELinux policy to do its stuff:

$ cat /usr/libexec/ovirt-vmconsole-host-sshd
#!/bin/sh
exec "/usr/sbin/sshd" "$@"

let's compare the SElinux labels to a working environment (CentOS 7.3):
$ ls -lhZ /usr/libexec/ovirt-vmconsole-host-sshd
-rwxr-xr-x. root root system_u:object_r:ovirt_vmconsole_exec_t:s0 /usr/libexec/ovirt-vmconsole-host-sshd


So I fixed the labels:
# chcon -t ovirt_vmconsole_exec_t /usr/libexec/ovirt-vmconsole-host-sshd
You have new mail in /var/spool/mail/root

It seems enough:
# ls -lhZ /usr/libexec/ovirt-vmconsole-host-sshd
-rwxr-xr-x. root root system_u:object_r:ovirt_vmconsole_exec_t:s0 /usr/libexec/ovirt-vmconsole-host-sshd
# systemctl start ovirt-vmconsole-host-sshd
# systemctl status ovirt-vmconsole-host-sshd
● ovirt-vmconsole-host-sshd.service - oVirt VM Console SSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/ovirt-vmconsole-host-sshd.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2017-02-16 10:21:07 GMT; 9s ago
 Main PID: 16789 (sshd)
   CGroup: /system.slice/ovirt-vmconsole-host-sshd.service
           └─16789 /usr/sbin/sshd -f /usr/share/ovirt-vmconsole/ovirt-vmconsole-host/ovirt-vmconsole-host-sshd/sshd_config -D

I believe somehow the SElinux label is lost in RHV-H. I don't have any clue why and how this happened, but it is not correct.
The best fix is IMO to restore the original labeling, and merge the patch only if this is not possible (but I need a good rationale for that, since we are allowing more system interactions, increasing the attack surface).

Comment 19 Francesco Romani 2017-02-16 10:28:08 UTC
Douglas, please check https://bugzilla.redhat.com/show_bug.cgi?id=1419829#c18

Comment 20 Douglas Schilling Landgraf 2017-02-16 21:20:00 UTC
(In reply to Francesco Romani from comment #19)
> Douglas, please check https://bugzilla.redhat.com/show_bug.cgi?id=1419829#c18

Thanks for your findings. I am speaking with the Selinux policy maint to see if we can find a reason for the difference between rhvh4 (based on rhel7.3) and pure rhel7.3. However, I believe the allow command still missing in ovirt-vmconsole as explicit requirement, even if the issue doesn't show in rhel7.3. Basically, sshd daemon is trying to bind on port labeled as ovirt_vmconsole_host_port_t instead of ssh_t/sshd_port_t.

Comment 21 Francesco Romani 2017-02-17 08:09:15 UTC
(In reply to Douglas Schilling Landgraf from comment #20)
> (In reply to Francesco Romani from comment #19)
> > Douglas, please check https://bugzilla.redhat.com/show_bug.cgi?id=1419829#c18
> 
> Thanks for your findings. I am speaking with the Selinux policy maint to see
> if we can find a reason for the difference between rhvh4 (based on rhel7.3)
> and pure rhel7.3. However, I believe the allow command still missing in
> ovirt-vmconsole as explicit requirement, even if the issue doesn't show in
> rhel7.3. Basically, sshd daemon is trying to bind on port labeled as
> ovirt_vmconsole_host_port_t instead of ssh_t/sshd_port_t.

Your proposed patch (https://gerrit.ovirt.org/#/c/71919/1) *is* correct in the case the istance of sshd used by ovirt-vmconsole runs with the sshd_t selinux type. In *this* case, yes, we will need to add the additional permission granted in the patch to make it work.

However, the selinux policy of ovirt-vmconsole was designed in a way to use a different selinux type for its custom sshd instance:

1. system sshd uses sshd_t
2. ovirt-vmconsole sshd uses ovirt_vmconsole_t

Please note that permission just works if the ovirt-vmconsole sshd instance has the right selinux type ovirt_vmconsole_t.
So the question is just how to make the ovirt-vmconsole sshd gain the selinux type ovirt_vmconsole_t.

In general and in RHEL 7.3, the launch script "/usr/libexec/ovirt-vmconsole-host-sshd" must have the ovirt_vmconsole_exec_t type to be able to change its type to ovirt_vmconsole_t when executed.

In RHV-H, the /usr/libexec/ovirt-vmconsole-host-sshd changed its selinux type  to unlabeled_t, instead of ovirt_vmconsole_exec_t; I don't see any reason for that change, which smells like a bug.

Because of this change (which is the root cause), the ovirt-vmconsole instance uses the sshd_t selinux type, *thus* requiring additional grants as per your patch 71919.

This is a policy change, not just a fix. So, for patch https://gerrit.ovirt.org/#/c/71919/1 we *also* need a clear rationale about why we need to change the label of /usr/libexec/ovirt-vmconsole-host-sshd.

Comment 22 Sandro Bonazzola 2017-02-17 13:59:09 UTC
Looking at comment #18 I think the error is due to:

%post
if %{_sbindir}/selinuxenabled ; then
	semodule -i "%{_datadir}/selinux/packages/%{name}/ovirt_vmconsole.pp"
	fixfiles -R %{name} restore
fi

On standard RHEL the rpm is installed on a selinux enabled system.
I can only guess that during the build of RHV-H / oVirt Node this *may* not be true.

Any way to check or workaround this?

Comment 23 Francesco Romani 2017-02-17 14:05:51 UTC
(In reply to Sandro Bonazzola from comment #22)
> Looking at comment #18 I think the error is due to:
> 
> %post
> if %{_sbindir}/selinuxenabled ; then
> 	semodule -i "%{_datadir}/selinux/packages/%{name}/ovirt_vmconsole.pp"
> 	fixfiles -R %{name} restore
> fi
> 
> On standard RHEL the rpm is installed on a selinux enabled system.
> I can only guess that during the build of RHV-H / oVirt Node this *may* not
> be true.
> 
> Any way to check or workaround this?

We did this change between 1.0.3/1.0.4 (https://gerrit.ovirt.org/#/c/59111/). Before, we did the selinux system changes unconditionally; so yes, this may well be part or the root cause itself.

If that could help, I'm fine adding one workaround for RHV-H/Node to do the selinux steps also if selinux disabled, something like


%post
if %{_sbindir}/selinuxenabled || somehow_detect_installing_on_RHVH; then
	semodule -i "%{_datadir}/selinux/packages/%{name}/ovirt_vmconsole.pp"
 	fixfiles -R %{name} restore
fi

(we will need this change in a couple more places, but no big deal)

I'll need a reliable way to detect RHVH/node, however.

Comment 24 Douglas Schilling Landgraf 2017-02-17 20:43:12 UTC


(In reply to Francesco Romani from comment #23)
> (In reply to Sandro Bonazzola from comment #22)
> > Looking at comment #18 I think the error is due to:
> > 
> > %post
> > if %{_sbindir}/selinuxenabled ; then
> > 	semodule -i "%{_datadir}/selinux/packages/%{name}/ovirt_vmconsole.pp"
> > 	fixfiles -R %{name} restore
> > fi
> > 
> > On standard RHEL the rpm is installed on a selinux enabled system.
> > I can only guess that during the build of RHV-H / oVirt Node this *may* not
> > be true.
> > 
> > Any way to check or workaround this?
> 
> We did this change between 1.0.3/1.0.4
> (https://gerrit.ovirt.org/#/c/59111/). Before, we did the selinux system
> changes unconditionally; so yes, this may well be part or the root cause
> itself.
> 
> If that could help, I'm fine adding one workaround for RHV-H/Node to do the
> selinux steps also if selinux disabled, something like
> 
> 
> %post
> if %{_sbindir}/selinuxenabled || somehow_detect_installing_on_RHVH; then
> 	semodule -i "%{_datadir}/selinux/packages/%{name}/ovirt_vmconsole.pp"
>  	fixfiles -R %{name} restore
> fi
> 
> (we will need this change in a couple more places, but no big deal)
> 
> I'll need a reliable way to detect RHVH/node, however.

I would go with /etc/os-release or try to catch the rpm redhat-release-virtualization-host from the database of pkgs.

Comment 25 Michal Skrivanek 2017-02-20 10:18:26 UTC
I believe you should either build the image with selinux enabled; or you need to manually run the extra policy and fixfiles later on, but without that the selinux contexts are all wrong, not only for ovirt-vmconsole, but in general.
I believe we then runfixfiles on whole filesystem on first boot or something like that...which fixes a lot of things but it doesn't compile the ovirt-vmconsole policy so the labeling fails.
IIRC when I was looking at that a long time ago indeed there were quite a few labeling problems on the first boot. I do not recall much details though, sorry.

Comment 26 Douglas Schilling Landgraf 2017-02-21 04:07:55 UTC
Looks like the change will be in the ovirt-vmconsole spec, moving the assign.

Comment 27 Michal Skrivanek 2017-02-21 09:36:32 UTC
why in spec?
To me this seems as an issue while building node image. That should be fixed in the build process - you're building and installing packages with SELinux disabled and then enable it on boot - for that to work things need to be reconfigured. There's nothing wrong with the spec and I'm not sure it can even be hacked like it was proposed in comment #23 - as the reason for adding the condition was that the compilation of policies do not work when SELinux is disabled.

Comment 29 Ryan Barry 2017-02-28 18:24:02 UTC
This is also a terrible way to do things, but instead of relying on /usr/sbin/selinuxenabled (which can fail in a variety of cases, including image building), checking the value of SELINUX in /etc/sysconfig may be a good workaround here.

Comment 30 Francesco Romani 2017-03-08 15:19:19 UTC
indeed we can't solve this in the spec using https://gerrit.ovirt.org/#/c/73370/

Comment 32 Michal Skrivanek 2017-03-08 15:43:16 UTC
(In reply to Ryan Barry from comment #29)
> This is also a terrible way to do things, but instead of relying on
> /usr/sbin/selinuxenabled (which can fail in a variety of cases, including
> image building), checking the value of SELINUX in /etc/sysconfig may be a
> good workaround here.

Francesco's patch doesn't fix the issue unfortunately.
We didn't find any good way how to do that during node build. It seems we need to run a script on the actual node run and fix the selinux contexts. This may need to also run semodule -i with the ovirt-vmconsole policy file
Is that doable?

Comment 33 Ryan Barry 2017-03-08 17:01:13 UTC
That is doable. However, unless this denial actually has a functional impact, I'd rather push this out and work on getting selinux working inside brew.

If we were to do this on first boot, I'd like to go all the rpm scripts and do this agnostically, rather than just ovirt-vmconsole

Comment 34 Michal Skrivanek 2017-03-11 07:26:03 UTC
I believe is breaks the serial console since the sshd's bind is denied. Not sure how we can best identify which policies be compiled

Comment 35 Nikolai Sednev 2017-05-04 07:56:23 UTC
Moving back forth to team Node, which was set by Sandro Bonazzola 2017-02-14 04:09:44 EST.

Comment 36 Huijuan Zhao 2017-05-09 07:36:04 UTC
Still encountered this issue in imgbased-0.9.24-0.1.el7ev.noarch.

Test version:
1. Before upgrade:
redhat-virtualization-host-4.0-20160817.0
2. After upgrade:
redhat-virtualization-host-4.1-20170506.0
imgbased-0.9.24-0.1.el7ev.noarch
kernel-3.10.0-514.16.1.el7.x86_64

# imgbase layout
rhvh-4.0-0.20160817.0
 +- rhvh-4.0-0.20160817.0+1
rhvh-4.1-0.20170506.0
 +- rhvh-4.1-0.20170506.0+1

Test steps:
Same as Comment 0

Actual results:
After step5, avc denied errors (sshd_t) in audit.log after upgrade:
type=AVC msg=audit(1494313582.560:69): avc:  denied  { name_bind } for  pid=2917 comm="sshd" src=2223 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:ovirt_vmconsole_host_port_t:s0 tclass=tcp_socket
type=AVC msg=audit(1494313582.560:70): avc:  denied  { name_bind } for  pid=2917 comm="sshd" src=2223 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:ovirt_vmconsole_host_port_t:s0 tclass=tcp_socket


Excepted results:
After step5, there should be no avc error.


Additional info:
1. No such issue when upgrade from rhvh-4.0-0.20170308.0 to rhvh-4.1-0.20170506.0
2. No such issue when upgrade from rhvh-4.1-0.20170202.0 to rhvh-4.1-0.20170506.0


So I will change the status to ASSIGNED.

Comment 37 Red Hat Bugzilla Rules Engine 2017-05-09 07:36:13 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 38 Huijuan Zhao 2017-05-09 07:39:35 UTC
Created attachment 1277277 [details]
Comment 36: sosreport and all logs in /var/log, /tmp

Comment 39 Moran Goldboim 2017-05-11 10:08:59 UTC
removing blocker flag, since this issue exists from 4.0 release, and looks like it has no RHV functional impact.

Comment 40 Ryan Barry 2017-05-12 18:11:52 UTC
imgbased was built before blocker+ was removed, so this is actually in 4.1.2

Comment 43 Huijuan Zhao 2017-05-16 06:24:53 UTC
Still encounter avc denied issue in imgbased-0.9.25-0.1.el7ev.noarch.


Test version:
From: rhvh-4.1-0.20170202.0
To:   rhvh-4.1-0.20170512.0
      imgbased-0.9.25-0.1.el7ev.noarch


[root@ibm-x3650m5-04 ~]# imgbase layout
rhvh-4.1-0.20170202.0
 +- rhvh-4.1-0.20170202.0+1
rhvh-4.1-0.20170512.0
 +- rhvh-4.1-0.20170512.0+1


Test steps:
Similar with comment 0


Actual results:
After step 5, there is "avc:  denied" info:

[root@ibm-x3650m5-04 ~]# grep "avc:  denied" /var/log/audit/audit.log
type=AVC msg=audit(1494915220.085:138): avc:  denied  { execute } for  pid=30214 comm="cockpit-ws" name="cockpit-ssh" dev="dm-7" ino=9307935 scontext=system_u:system_r:cockpit_ws_t:s0 tcontext=system_u:object_r:bin_t:s0 tclass=file

Comment 44 Red Hat Bugzilla Rules Engine 2017-05-16 15:17:35 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 45 Huijuan Zhao 2017-05-18 04:18:20 UTC
1. Tested according to comment 0 with rhvh-4.1-0.20170512.0, no such issue any more. But encountered another bug 1451974.

2. Tested 4 times according to comment 43 on the same machine and several times with other machines, can not reproduce it again. So maybe we can ignore it currently.



Below are detailed testing info according to comment 0:

Test version:
From: rhvh-4.0-0.20160817.0
To:   rhvh-4.1-0.20170512.0
      imgbased-0.9.25-0.1.el7ev.noarch

# imgbase layout
rhvh-4.0-0.20160817.0
 +- rhvh-4.0-0.20160817.0+1
rhvh-4.1-0.20170512.0
 +- rhvh-4.1-0.20170512.0+1

Test steps:
Same as comment 0.

Test steps:
There is no "avc:  denied (sshd)" error.
But encountered another bug 1451974.


According to above testing results, I will verify this bug after it is changed to ON_QA status.

And the new issue can be traced by bug 1451974.

Comment 46 Huijuan Zhao 2017-06-02 05:59:46 UTC
According to Comment 45, this issue was fixed in imgbased-0.9.25-0.1.el7ev.noarch.

But now the "Fixed In Version" changes to 4.1.3 build imgbased-0.9.28-0.1.el7ev, and there is blocker Bug 1457111 in imgbased-0.9.28-0.1.el7ev which block verify this bug.

For another 4.1.3 build imgbased-0.9.30-0.1.el7ev, there is blocker bug 1457670 which block this bug verification.

So can not verify this bug in the current 4.1.3 builds, QE will verify this bug once receive new available 4.1.3 build.

Change the status to MODIFIED.

Thanks!

Comment 47 Huijuan Zhao 2017-06-21 08:29:21 UTC
Test version:
From: rhvh-4.0-0.20160817.0
To:   rhvh-4.1-0.20170609.0
      imgbased-0.9.31-0.1.el7ev.noarch

Test steps:
Same as comment 0

Test results:
There is no "avc:  denied (sshd)" error.


So this issue is fixed in imgbased-0.9.31-0.1.el7ev.noarch, change the status to VERIFIED.