Bug 1266976 - RFE; virt-who needs visibility into all networks in the environment in which it runs.
RFE; virt-who needs visibility into all networks in the environment in which ...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: virt-who (Show other bugs)
7.1
All Linux
unspecified Severity medium
: rc
: ---
Assigned To: Radek Novacek
Eko
: FutureFeature
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-28 13:29 EDT by Greg Scott
Modified: 2016-11-30 19:36 EST (History)
8 users (show)

See Also:
Fixed In Version: virt-who-0.17-1.el7
Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-11-04 01:06:08 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Greg Scott 2015-09-28 13:29:35 EDT
Description of problem:

Customers who buy SKU RH00002, RHEL for virtual datacenters, purchase the right to unlimited supported VMs under one hypervisor.  To register in an environment such as VMware that uses third party hypervisors, customers must configure, install, and run a special program named virt-who.  Virt-who finds all the third party hypervisors and VMs in the environment and registers them with Red Hat.  To make this work, virt-who needs the ability to interact with all hypervisors and VMs in the environment.  Unfortunately, this is not possible in cases where some VMs are in a DMZ network and prohibited by design from interacting with systems in the internal network.  

Version-Release number of selected component (if applicable): 6.n, 7.n, and all 5.n versions that support the new RHSM method of registration.


How reproducible:

At will

Steps to Reproduce:
1. Build a virtual environment using VMware or Hyper-V with some VMs in a DMZ network.
2. Set up virt-who on a RHEL VM in this environment.  Give the virt-who VM visibility in both in the DMZ and internal network.
3. The separation between the DMZ and internal network is now weakened.

Actual results:

Fail a security audit because the DMZ network is no longer secure.

Expected results:

We need an RHSM registration solution that does not compromise the separation between the DMZ network and internal network.


Additional info:

See support case number 01511955.  Also see:  https://access.redhat.com/solutions/1757833 for a workaround to the problem.  Unfortunately, even with the virt-who server locked down, it still opens a potential attack vector between the DMZ and internal network.
Comment 2 Radek Novacek 2015-09-29 05:51:25 EDT
I don't see any solution to this problem. virt-who requires connection to the hypervisors to obtain host-guest association.

Only possible workaround currently is to use fake mode of virt-who. In this mode virt-who reads host/guest association from file and reports that. But customer would have to produce the association file somehow.

BTW: virt-who doesn't have to run on virtual machine on the hypervisor. It can run anywhere, for example on some other host in DMZ network (given it is running RHEL).
Comment 3 Greg Scott 2015-09-30 15:15:35 EDT
Thanks. How does a customer use fake mode of virt-who, and is there a suggested set of steps for the customer to produce the association file?
Comment 4 Radek Novacek 2015-10-06 04:37:40 EDT
In order to use fake mode, one needs to add this configuration file into /etc/virt-who.d/ containing:

[config-name]
type=fake
file=/path/to/json
is_hypervisor=True


The file should be in JSON format with following structure:
{
    "hypervisors": [
        {
            "uuid": "7e98b6ea-0af1-4afa-b846-919549bb0fe2",
            "guests": [
                {
                    "guestId": "8ae19f08-2605-b476-d42e-4bd5a39f466c"
                },
                ...
            ]
        },
        ...
    ]
}

virt-who command-line can be used to generate this file (the '-p' option). But if the customer can't use virt-who, he needs to generate the file himself somehow.
Comment 5 Greg Scott 2015-10-07 17:00:34 EDT
OK.  So in order for the customer to use virt-who, he need a system with visibility into both the
Comment 6 Greg Scott 2015-10-07 17:29:55 EDT
Woops - just get rid of comment 5.  Sorry about that. 

So we have a chicken and egg challenge.  In order to use virt-who as documented, the virt-who server needs access into everything, including DMZ or other protected networks.  If a security policy prohibits the virt-who server from accessing everything, it must run in another mode called fake mode.

Since the customer has to use virt-who in fake mode, does the customer build the above file by hand? Or just partially by hand, entering data about VMs the virt-who server is unable to see?  

It looks like that file associates VMs with the hypervisors hosting them.  How does a customer find the GUID of every hypervisor and every VM?  Is there any documentation for how to do this with products such as VMware and Hyper-V?  Or does virt-who create its own GUIDs - and if so, how does a customer generate them?

Let's say we have an environment with, say, 1000 VMs.  VMs can migrate from host to host at any time.  Sometimes the migration is automatic, sometimes manual.  How does a customer keep this file updated by hand with which VM is running on what host?

Let's say a customer is maintaining this file by hand.  In the event that a customer mistypes a GUID for a VM or hypervisor, what happens to that VM's ability to download updates?

Unless I'm missing something, expecting a customer to maintain this file by hand is unnecessarily burdensome with a few VMs and realistically impossible with dozens or hundreds of VMs that migrate among hosts all the time. 

What can we do to ease the burden?

Would Engineering and Product Management be willing to support a solution using a virt-who server with a specially crafted and documented iptables script to lock it down?  See the next comment for a suggested script that could be a basis for something realistically supportable.
Comment 7 Greg Scott 2015-10-07 17:35:00 EDT
Instead of forcing the customer to maintain a large and complex configuration file by hand to track system registrations, what about providing an officially supported method to lock down the virt-who server.  Here is a sample script to use as a starting point.

#!/bin/sh
#
# virt-who.firewall - sample script to run on a virt-who server virtual machine at
# boot time.
#
# This script is tailored for a single, special-purpose system running only the
# virt-who tool.  It is not intended to run on a general purpose system and is
# provided as-is with no support from Red Hat.
#
# Red Hat uses the virt-who daemon to register virtual systems and hypervisors in cases
# with unlimited guest virtual machine subscriptions.  Run virt-who on one RHEL system
# in the environment.  It will find the other hypervisors and virtual machines in the 
# environment and take care of registration. 
# 
# Use this script to lock down the virt-who server in cases where some VMs are in the
# internal network, while others are in a DMZ or other public facing network.  In such
# cases, the virt-who server needs to have a presence in both networks and should be
# locked down.
#
# This script presents 3 arrays of IP Addresses/ranges of systems to trust.  Fill in
# TRUSTED_IP,  DMZ_IP, and INET_IP appropriately below.  These are systems in the
# trusted, DMZ, and public networks to trust.  Anything originating from anywhere else
# will be logged and dropped.   Note that normally, nothing in the public network would
# be trusted.  That is why references to INET_IP are commented out.
#
# This is not intended to implement a full stateful packet filtering system; only to
# lock down a single virt-who server.  Assumes single public, DMZ, and internal
# networks, each with an array of IP Addresses/ranges to trust.
#
#	First draft Sept. 24, 2015
#
#	Red Hat Corporation
#	accepts no responsibility for any issues this script may create.
#
#	A few helpful commands:
#
#	/sbin/iptables -L -v -n		to get a verbose numeric listing
#					of all the rules and counters
#
#	/sbin/iptables -F		Flush the default table
#	/sbin/iptables -t nat -F	Flush the nat table
#	/sbin/iptables -t mangle -F	Flush the mangle table
#	/sbin/iptables -Z		Zero the counters of the default table
#

#
# List of IP Addresses to trust.  Broken into groups.  Modify as appropriate.
# Add the IP Addresses or ranges this server should trust on each virtual NIC.
# Modify the values for TRUSTED_IFACE, DMZ_IFACE, and INET_IFACE
# appropriately.
#

TRUSTED_IP[0]="192.168.0.0/24"
# Array of IP Addresses in the internal network to trust.  Each element can be 
# a range or an individual IP Address.  
TRUSTED_IFACE="eth0"

DMZ_IP[0]="10.10.10.77"
DMZ_IP[1]="10.10.10.78"
# Array of IP Addresses in the DMZ network to trust.  Each element can be 
# a range or an individual IP Address.  
DMZ_IFACE="eth1"

#INET_IP[0]="1.2.3.4"
# Array of public IP Addresses to trust.  
# Commented out to exclude everything in the public Internet.
INET_IFACE="eth99"
# It would be unusual to connect this VM to anything public facing.   So most
# of the time, this will just be a placeholder.

#
#  Done setting up parameters.
#

LOOPBACK_IFACE="lo"
LOCALHOST_IP="127.0.0.1"

IPTABLES="/sbin/iptables"

DATEFILE="/firewall-scripts/fwdate.txt"

echo "Recording the date and time in " $DATEFILE
echo "The virt-who.firewall script was last launched on $(date)" > $DATEFILE

#
# Start with a clean slate.
#

echo "Flushing and zeroing all iptables"

$IPTABLES -F
$IPTABLES -X
$IPTABLES -t nat -F
$IPTABLES -t nat -X
$IPTABLES -t mangle -F
$IPTABLES -t mangle -X

#
# set default policies for the INPUT, FORWARD and OUTPUT chains
#

echo "Setting up temporary policies to ACCEPT everything."
# So we can run this remotely hopefully without getting blown away
# if something goes wrong.

$IPTABLES -P INPUT ACCEPT
$IPTABLES -P OUTPUT ACCEPT
$IPTABLES -P FORWARD ACCEPT

#
# FORWARD chain
# This server should not do any forwarding.
#

echo "Setting up basic FORWARD chain rules."

$IPTABLES -A FORWARD -j LOG
# Debug - Log anything and everything that gets this far

$IPTABLES -A FORWARD -j DROP

#
# INPUT chain
#
# Establish the basic INPUT chain and filter the packets onto the correct
# chains.  (The INPUT chain contols packets coming from the outside bound
# for this box.)  Most of the action happens here.
#

echo "Setting up the basic INPUT chain rules"

$IPTABLES -A INPUT -p ALL -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
# Ok, a little bit of statefulness. 

for ADR in "${INET_IP[@]}"; do
    $IPTABLES -A INPUT -i $INET_IFACE -s $ADR -j ACCEPT
done

for ADR in "${TRUSTED_IP[@]}"; do
    $IPTABLES -A INPUT -i $TRUSTED_IFACE -s $ADR -j ACCEPT
done

for ADR in "${DMZ_IP[@]}"; do
    $IPTABLES -A INPUT -i $DMZ_IFACE -s $ADR -j ACCEPT
done

$IPTABLES -A INPUT -p ALL -i $LOOPBACK_IFACE -j ACCEPT
$IPTABLES -A INPUT -p ALL -d $LOCALHOST_IP -j ACCEPT
$IPTABLES -A INPUT -p ALL -s $LOCALHOST_IP -j ACCEPT

$IPTABLES -A INPUT -j LOG
# Debug - Log anything and everything that gets this far

$IPTABLES -A INPUT -j DROP

#
# OUTPUT chain
#
# Establish the basic OUTPUT chain and filter them onto the correct chain.
# The OUTPUT chain controls packets starting in this box with destination
# outside.
#

echo "OUTPUT chain rules (for packets starting on this box)"

$IPTABLES -A OUTPUT -j ACCEPT

$IPTABLES -A OUTPUT -j LOG
# Debug - Log anything and everything that gets this far

$IPTABLES -A OUTPUT -j DROP

echo "Setting up default policies to DROP all unless a rule says otherwise"

$IPTABLES -P INPUT DROP
$IPTABLES -P OUTPUT DROP
$IPTABLES -P FORWARD DROP

#
# CRITICAL:  Disable IP forwarding here. A virt-who server should not
# act as a router.
#

echo "Turning off IP forwarding"
echo "0" > /proc/sys/net/ipv4/ip_forward

exit
Comment 8 Radek Novacek 2015-10-08 02:13:12 EDT
Yes, creating this mapping by hand is impossible in larger environment, that the reason why virt-who exists in the first place.

Locking down is possible but I'm not sure we should provide official way how to do it. It's responsibility of the maintainer of that system. We can't fit all the policies that all our customers have. I think it would be better just to document what needs to be open for virt-who and let the customer to do the lock down according to his policies.
Comment 9 Greg Scott 2015-10-08 08:24:51 EDT
So here is the situation as it exists today:

- The customer needs virt-who to run on a system that can access everything.
- This violates a security policy with DMZ networks.
- Red Hat currently provides no guidance on how to lock down the system with virt-who installed.
- This forces the customer into an impossible position.

Like it or not, that's the situation.

If we can't provide any formal guidance for how to lock down that virt-who server, how about informal guidance with a knowledge base solution or article? See: https://access.redhat.com/solutions/1757833 not yet published.
Comment 10 Greg Scott 2015-10-08 11:48:21 EDT
Also - what ports does virt-who use?  All TCP?  Any UDP or other protocols?

Virt-who needs access to all the hypervisors.  Does it also need access to all the VMs in the environment?
Comment 11 Radek Novacek 2015-10-13 03:21:57 EDT
Greg,

virt-who doesn't need access to any VM. It just needs to talk to the hypervisor via its API. The required ports also depends on what hypervisor is used.

This is a list of ports that needs to be open for different hypervisors (note that only ports for used hypervisor needs to be open, the rest can remain closed):

* VMWare ESX/vCenter: 443/tcp
* Hyper-V: 5985/tcp
* RHEV-M: 443/tcp or 8443/tcp (depending on version)

virt-who also needs to have access to Subscription Asset Manager or Satellite (5 or 6). Default port is 443.

These are default configuration values, all the ports can be changed by administrator of that system.
Comment 12 Radek Novacek 2016-04-21 08:15:30 EDT
Greg,

would it be sufficient to put a section about securing virt-who server to its manual page?

I think it should be enough if we state what ports virt-who uses for what hypervisors, that it doesn't connect to guests, and it doesn't listen on any port. Anything else?
Comment 13 Greg Scott 2016-04-21 08:57:57 EDT
Yes, documenting how to secure virt-who in its man page would do the job nicely. As I think about it, even in situations with DMZ networks that need to be isolated, somehow, some way, all those ESXi/HyperV/Whatever hosts that run those VMs must be somehow visible to their management server.  So in addition to documenting what ports need to be open, maybe suggest a tactic of running virt-who in a special VM that has visibility into the host management networks so it can see all the hosts, but none of the other production VM networks.  

I put together a KCS article on this at:
https://access.redhat.com/solutions/1757833


- Greg
Comment 14 Radek Novacek 2016-04-21 11:34:42 EDT
Security section added to virt-who manual page:

https://github.com/virt-who/virt-who/commit/352feebdc9c63522bd12971fb1898c73574ebd31


Improvement suggestions welcomed.
Comment 15 Greg Scott 2016-04-21 12:15:05 EDT
I don't have an account on github.  The text looks good except for a grammar problem:

Change:

> virt-who doesn't not listen to any network ports. 

to

> virt-who does not listen to any network ports. 

Although is this true? How does virt-who know when new hypervisors come online?  Maybe a couple sentences about that works could help with security concerns.

- Greg
Comment 16 Radek Novacek 2016-04-26 07:54:51 EDT
Well spotted, fixed: 

https://github.com/virt-who/virt-who/commit/a8f2ce5

What I meant by the "virt-who does not listen to any network ports" is, that virt-who always figures as a client in the HTTP connection. It doesn't start any network server. Do you think I should clarify this in the man page? 

Do you have other suggestions?
Comment 17 Greg Scott 2016-04-26 10:46:36 EDT
I'll paste in the writeup here:

> +In certain environments, virt-who might conflict with security requirements. To minimize potential risks, virt-who should be in a small virtual system that is locked down from any unnecessary network access. virt-who does not listen to any network ports. Only outgoing network access is to the hypervisor running virtual machines you want to register and subscription manager server. virt-who doesn't need access to any virtual machine. 

So virt-who doesn't start any network server.  It's always a client, which means it launches periodically and polls for new hypervisors, right?  If so, I'd word it like this:

****************

Virt-who may present security concerns in some scenarios because it needs access to every hypervisor in the environment. To minimize security risk, virt-who is a network client, not a server. It only does outbound connections to find and register new hypervisors and does not need access to any virtual machines. To further reduce risk, deploy virt-who in a small virtual machine with a minimal installation and lock it down from any unsolicited inbound network connections.
Comment 18 Radek Novacek 2016-05-03 04:48:17 EDT
Thanks, Greg, I've used your wording for the security section of man page.

https://github.com/virt-who/virt-who/blob/master/virt-who.8#L272
Comment 19 Radek Novacek 2016-05-17 09:33:37 EDT
Fixed in virt-who-0.17-1.el7.
Comment 21 Liushihui 2016-06-02 04:07:42 EDT
Verified it on virt-who-0.17-2.el7.noarch since the security info has added to virt-who man page. therefore, verify it.

[root@dell-per920-02 ~]# man virt-who
SECURITY
       Virt-who may present security concerns in some scenarios because it needs access to every hypervisor in the environment. To mini‐
       mize security risk, virt-who is a network client, not a server. It only does outbound connections to find and register new hyper‐
       visors  and does not need access to any virtual machines. To further reduce risk, deploy virt-who in a small virtual machine with
       a minimal installation and lock it down from any unsolicited inbound network connections.

       Here is a list of ports that need to be open for different hypervisors:

           VMWare ESX/vCenter: 443/tcp
           Hyper-V: 5985/tcp
           RHEV-M: 443/tcp or 8443/tcp (depending on version)
           XenServer: 443/tcp
           libvirt: depending on transport type, default (for remote connections) is qemu over ssh on port 22
           local libvirt and vdsm use local connections and don't need open ports

       virt-who also needs to have access to Subscription Asset Manager, Satellite 5, or Satellite 6. Default port is 443/tcp.  All  the
       ports might be changed by system administrators.

       Using  the  same  network for machine running virt-who as for hypervisor management software instead of production VM networks is
       suggested.
Comment 23 errata-xmlrpc 2016-11-04 01:06:08 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2387.html

Note You need to log in before you can comment on or make changes to this bug.