Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1158213

Summary: rubygem-staypuft: Unable to get to the VNC console of the launched instance - get: Failed to connect to server (code: 1006)
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-selinuxAssignee: Ryan Hallisey <rhallise>
Status: CLOSED ERRATA QA Contact: Alexander Chuzhoy <sasha>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.0 (RHEL 7)CC: ajeain, cwolfe, ddomingo, jguiditt, lhh, mburns, mgrepl, oblaut, sclewis, sputhenp, yeylon
Target Milestone: z3Keywords: ZStream
Target Release: 5.0 (RHEL 7)Flags: ddomingo: needinfo+
rhallise: needinfo+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-selinux-0.5.20-1.el7ost Doc Type: Release Note
Doc Text:
In High Availability environments deployed through the Red Hat Enterprise Linux OpenStack Platform Installer, SELinux no longer needs to be Permissive on the HA controller to allow VNC access. A bug that required this in previous releases has since been fixed.
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-12-02 15:24:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
logs and nova.conf files none

Description Alexander Chuzhoy 2014-10-28 20:27:04 UTC
rubygem-staypuft: Unable to get to the VNC console of the launched instance - get: Failed to connect to server (code: 1006)

Environment:
rhel-osp-installer-0.4.5-2.el6ost.noarch
ruby193-rubygem-staypuft-0.4.10-1.el6ost.noarch
ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el6ost.noarch
openstack-foreman-installer-2.0.31-1.el6ost.noarch
openstack-puppet-modules-2014.1-24.el6ost.noarch



Steps to reproduce:
1. Install rhel-osp-installer.
2. Configure HAneutron deployment + GRE.
3. The following networks roles should be separated:
   tenant 
   external
   public api
   management + admin api
   Provisioning/PXE + Cluster Management + Storage + Storage Clustering
4. After the deployment completes successfully, launch an instance.
5. Attempt to access the instance via console.

Result:
Get error: Failed to connect to server (code: 1006) 

Expected result:
Should get the login prompt via the console.

Comment 1 Alexander Chuzhoy 2014-10-28 20:48:56 UTC
Created attachment 951546 [details]
logs and nova.conf files

Comment 2 Crag Wolfe 2014-10-31 00:37:01 UTC
In a working setup, one of the HA controllers has lines like from /var/log/nova/nova-consoleauth.log:

2014-10-30 10:09:24.281 10368 AUDIT nova.consoleauth.manager [req-e19e73d6-372b-45dc-bdf2-2a1ebdfabcd9 None None] Checking Token: 62287dff-8b1e-4d01-a407-e76e47d22d53, True

The not-working setup (not currently available) had:

AUDIT nova.consoleauth.manager [req-6de704f4-0aca-43b7-8aeb-00d6cecf5d42 None None] Checking Token: 3576b648-203a-4c97-b51f-aaeb9d2e3fd2, False

Checking memcached the next time around would be the next thing to check.

Comment 3 Alexander Chuzhoy 2014-10-31 13:35:40 UTC
The VNC console works with no issues on nonHA Nova deployment.

Comment 4 Crag Wolfe 2014-10-31 23:22:38 UTC
Shut down memcached on 2 of the 3 HA controllers, then did the same with openstack-nova-consoleauth, but issue still continued.  Will need to set debug = true and verbose = true in nova.conf to continue debugging when hosts are available again.

Comment 6 Crag Wolfe 2014-11-03 23:58:57 UTC
It looks selinux related -- the vnc console works after setting selinux to Permissive.  Here is a the relevant AVC:

type=AVC msg=audit(1415056227.164:5191): avc:  denied  { name_connect } for  pid=28842 comm="nova-consoleaut" dest=11211 scontext=system_u:system_r:nova_console_t:s0 tcontext=system_u:object_r:memcache_port_t:s0 tclass=tcp_socket

See the uploaded 192.168.0.4.audit.log for more details, or all of the logs for all controllers included in logs.tgz.

Comment 9 Mike Burns 2014-11-04 01:22:35 UTC
based on comment 6, moving to openstack-selinux.

Comment 10 Ofer Blaut 2014-11-04 10:52:12 UTC
I have checked VNC with selinux permissive on controllers and it works
rhel-osp-installer-0.4.7-1.el6ost.noarch

Comment 13 Alexander Chuzhoy 2014-11-04 19:01:01 UTC
Edited the "Kickstart RHEL default" provision template and replaced
selinux --enforcing
with
selinux --permissive

After that deployed and the VNC worked with no issues in HA deployment.

Comment 14 Miroslav Grepl 2014-11-05 07:15:14 UTC
What does

ausearch -m avc, user_avc -ts recent

after re-testing in permissive mode?

Comment 15 Alexander Chuzhoy 2014-11-05 14:11:16 UTC
[root@macf04da2732fb1 ~]# ausearch -m avc, user_avc -ts recent     
user_avc is an unsupported option     


                             
[root@macf04da2732fb1 ~]# ausearch -m avc -ts recent     
----                                                     
time->Wed Nov  5 16:05:34 2014                           
type=SYSCALL msg=audit(1415196334.346:7113): arch=c000003e syscall=42 success=no exit=-13 a0=8 a1=7fff04a0d3b0 a2=10 a3=4 items=0 ppid=1 pid=32072 auid=4294967295 uid=162 gid=162 euid=162 suid=162 fsuid=162 egid=162 sgid=162 fsgid=162 tty=(none) ses=4294967295 comm="nova-consoleaut" exe="/usr/bin/python2.7" subj=system_u:system_r:nova_console_t:s0 key=(null)                                                                  
type=AVC msg=audit(1415196334.346:7113): avc:  denied  { name_connect } for  pid=32072 comm="nova-consoleaut" dest=11211 scontext=system_u:system_r:nova_console_t:s0 tcontext=system_u:object_r:memcache_port_t:s0 tclass=tcp_socket                                                                                                                                                                                                     
----                                                                                                                                                                                                                 
time->Wed Nov  5 16:05:34 2014                                                                                                                                                                                       
type=SYSCALL msg=audit(1415196334.346:7114): arch=c000003e syscall=42 success=no exit=-13 a0=8 a1=7fff04a0d3b0 a2=10 a3=2 items=0 ppid=1 pid=32072 auid=4294967295 uid=162 gid=162 euid=162 suid=162 fsuid=162 egid=162 sgid=162 fsgid=162 tty=(none) ses=4294967295 comm="nova-consoleaut" exe="/usr/bin/python2.7" subj=system_u:system_r:nova_console_t:s0 key=(null)                                                                  
type=AVC msg=audit(1415196334.346:7114): avc:  denied  { name_connect } for  pid=32072 comm="nova-consoleaut" dest=11211 scontext=system_u:system_r:nova_console_t:s0 tcontext=system_u:object_r:memcache_port_t:s0 tclass=tcp_socket                                                                                                                                                                                                     
----                                                                                                                                                                                                                 
time->Wed Nov  5 16:05:34 2014                                                                                                                                                                                       
type=SYSCALL msg=audit(1415196334.346:7115): arch=c000003e syscall=42 success=no exit=-13 a0=8 a1=7fff04a0d3b0 a2=10 a3=3 items=0 ppid=1 pid=32072 auid=4294967295 uid=162 gid=162 euid=162 suid=162 fsuid=162 egid=162 sgid=162 fsgid=162 tty=(none) ses=4294967295 comm="nova-consoleaut" exe="/usr/bin/python2.7" subj=system_u:system_r:nova_console_t:s0 key=(null)                                                                  
type=AVC msg=audit(1415196334.346:7115): avc:  denied  { name_connect } for  pid=32072 comm="nova-consoleaut" dest=11211 scontext=system_u:system_r:nova_console_t:s0 tcontext=system_u:object_r:memcache_port_t:s0 tclass=tcp_socket                                                                                                                                                                                                     
----                                                                                                                                                                                                                 
time->Wed Nov  5 16:06:32 2014                                                                                                                                                                                       
type=SYSCALL msg=audit(1415196392.960:7123): arch=c000003e syscall=42 success=no exit=-13 a0=9 a1=7fff04a0d450 a2=10 a3=4 items=0 ppid=1 pid=32072 auid=4294967295 uid=162 gid=162 euid=162 suid=162 fsuid=162 egid=162 sgid=162 fsgid=162 tty=(none) ses=4294967295 comm="nova-consoleaut" exe="/usr/bin/python2.7" subj=system_u:system_r:nova_console_t:s0 key=(null)                                                                  
type=AVC msg=audit(1415196392.960:7123): avc:  denied  { name_connect } for  pid=32072 comm="nova-consoleaut" dest=11211 scontext=system_u:system_r:nova_console_t:s0 tcontext=system_u:object_r:memcache_port_t:s0 tclass=tcp_socket                                                                                                                                                                                                     
----                                                                                                                                                                                                                 
time->Wed Nov  5 16:06:32 2014
type=SYSCALL msg=audit(1415196392.960:7124): arch=c000003e syscall=42 success=no exit=-13 a0=9 a1=7fff04a0d450 a2=10 a3=2 items=0 ppid=1 pid=32072 auid=4294967295 uid=162 gid=162 euid=162 suid=162 fsuid=162 egid=162 sgid=162 fsgid=162 tty=(none) ses=4294967295 comm="nova-consoleaut" exe="/usr/bin/python2.7" subj=system_u:system_r:nova_console_t:s0 key=(null)
type=AVC msg=audit(1415196392.960:7124): avc:  denied  { name_connect } for  pid=32072 comm="nova-consoleaut" dest=11211 scontext=system_u:system_r:nova_console_t:s0 tcontext=system_u:object_r:memcache_port_t:s0 tclass=tcp_socket
----
time->Wed Nov  5 16:06:32 2014
type=SYSCALL msg=audit(1415196392.960:7125): arch=c000003e syscall=42 success=no exit=-13 a0=9 a1=7fff04a0d450 a2=10 a3=3 items=0 ppid=1 pid=32072 auid=4294967295 uid=162 gid=162 euid=162 suid=162 fsuid=162 egid=162 sgid=162 fsgid=162 tty=(none) ses=4294967295 comm="nova-consoleaut" exe="/usr/bin/python2.7" subj=system_u:system_r:nova_console_t:s0 key=(null)
type=AVC msg=audit(1415196392.960:7125): avc:  denied  { name_connect } for  pid=32072 comm="nova-consoleaut" dest=11211 scontext=system_u:system_r:nova_console_t:s0 tcontext=system_u:object_r:memcache_port_t:s0 tclass=tcp_socket
----
time->Wed Nov  5 16:08:53 2014
type=SYSCALL msg=audit(1415196533.017:7159): arch=c000003e syscall=42 success=no exit=-115 a0=9 a1=7fff04a0d3b0 a2=10 a3=4 items=0 ppid=1 pid=32072 auid=4294967295 uid=162 gid=162 euid=162 suid=162 fsuid=162 egid=162 sgid=162 fsgid=162 tty=(none) ses=4294967295 comm="nova-consoleaut" exe="/usr/bin/python2.7" subj=system_u:system_r:nova_console_t:s0 key=(null)
type=AVC msg=audit(1415196533.017:7159): avc:  denied  { name_connect } for  pid=32072 comm="nova-consoleaut" dest=11211 scontext=system_u:system_r:nova_console_t:s0 tcontext=system_u:object_r:memcache_port_t:s0 tclass=tcp_socket



[root@macf04da2732fb1 ~]# ausearch -m user_avc -ts recent
----
time->Wed Nov  5 16:08:44 2014
type=USER_AVC msg=audit(1415196524.097:7158): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc:  received setenforce notice (enforcing=0)  exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
----
time->Wed Nov  5 16:09:32 2014
type=USER_AVC msg=audit(1415196572.048:7188): pid=1335 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_return dest=:1.309 spid=1334 tpid=79138 scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:system_r:puppetagent_t:s0 tclass=dbus  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'
[root@macf04da2732fb1 ~]#

Comment 16 Ryan Hallisey 2014-11-05 15:14:12 UTC
corenet_tcp_connect_memcache_port(nova_console_t)

Comment 17 Ryan Hallisey 2014-11-05 15:55:50 UTC
The policy exists in rawhide, but not in rhel7 yet.  We can add this to openstack-selinux.

Comment 20 Miroslav Grepl 2014-11-07 10:13:45 UTC
(In reply to Ryan Hallisey from comment #17)
> The policy exists in rawhide, but not in rhel7 yet.  We can add this to
> openstack-selinux.

We should have it in RHEL7.1.

If you need it for 7.0, you will need to add


require{
type nova_console_t;
}
corenet_tcp_connect_memcache_port(nova_console_t)


Any chance to test it with this rule to see if it works?

Comment 21 Ryan Hallisey 2014-11-18 15:03:16 UTC
Fixes already built in 6.0 just need acks to build for 5.0

Comment 24 Alexander Chuzhoy 2014-11-20 20:13:10 UTC
Verified:
openstack-puppet-modules-2014.1.1-1.el6ost.noarch
rhel-osp-installer-0.4.7-1.el6ost.noarch
ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el6ost.noarch
openstack-foreman-installer-2.0.32-1.el6ost.noarch
ruby193-rubygem-staypuft-0.4.14-1.el6ost.noarch


The reported issue doesn't reproduce, able to get to the console with no issues.

Comment 25 Alexander Chuzhoy 2014-11-20 20:14:43 UTC
The openstack-selinux version is:
openstack-selinux-0.5.20-1.el7ost.noarch

Comment 27 errata-xmlrpc 2014-12-02 15:24:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2014-1935.html