Bug 1278637 - Support for encrypted Hyper-V connection in virt-who
Summary: Support for encrypted Hyper-V connection in virt-who
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: virt-who   
(Show other bugs)
Version: 7.1
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: rc
: 7.1
Assignee: Radek Novacek
QA Contact: Eko
Yehuda Zimmerman
URL:
Whiteboard:
Keywords:
: 1167283 (view as bug list)
Depends On: 1291737
Blocks: 1172231 1203710 1272873 1298337 1313485
TreeView+ depends on / blocked
 
Reported: 2015-11-06 04:02 UTC by Anand Vaddarapu
Modified: 2019-04-16 14:46 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Encrypted Hyper-V connections supported in _virt-who_ Previously, _virt-who_ used unencrypted Hyper-V connections. All data was sent in plain text. This had security implications and needed special configuration on Hyper-V servers to be allowed. With this update, _virt-who_ now uses Windows NT LAN Manager (NTLM) sealing and signing to protect communication with Hyper-V servers.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-11-04 05:06:50 UTC
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Disallowed auth with GPO.png (74.70 KB, image/png)
2016-02-02 03:57 UTC, Liushihui
no flags Details
Disallowed auth without GPO.png (71.76 KB, image/png)
2016-02-02 03:59 UTC, Liushihui
no flags Details
Allow auth with GPO.png (70.36 KB, image/png)
2016-02-02 03:59 UTC, Liushihui
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:2387 normal SHIPPED_LIVE virt-who bug fix and enhancement update 2016-11-03 13:53:39 UTC
Red Hat Knowledge Base (Solution) 1261263 None None None 2018-12-09 19:14 UTC

Description Anand Vaddarapu 2015-11-06 04:02:13 UTC
Description of problem:

virt-who configs:

#cat /etc/virt-who.d/hyperv_config
[hypervname]
type=hyperv
#server=hyperV-FQDN
server=http://10.0.0.0
username=username
encrypted_password=<encrypted using virt-who-password>
owner=OWNER
env=GA-Static-Patching-RHEL-7
hypervisor_id=hostname

>winrm get winrm/config/service
Service
    RootSDDL = XXXXX
    MaxConcurrentOperations = 4294967295
    MaxConcurrentOperationsPerUser = 1500
    EnumerationTimeoutms = 240000
    MaxConnections = 300
    MaxPacketRetrievalTimeSeconds = 120
    AllowUnencrypted = true [Source="GPO"]
    Auth
        Basic = false [Source="GPO"]
        Kerberos = true
        Negotiate = true [Source="GPO"]
        Certificate = false
        CredSSP = true
        CbtHardeningLevel = Relaxed
    DefaultPorts
        HTTP = 5985
        HTTPS = 5986
    IPv4Filter = * [Source="GPO"]
    IPv6Filter = * [Source="GPO"]
    EnableCompatibilityHttpListener = true
    EnableCompatibilityHttpsListener = true
    CertificateThumbprint
    AllowRemoteAccess = true [Source="GPO"]

virt-who -o -d
ERROR:root:Configuration file katello-ca-consumer-latest.noarch.rpm contains no section headers
2015-11-05 15:33:57,762 DEBUG: Using config named 'hyperv-hostname1'
2015-11-05 15:33:57,762 DEBUG: Using config named 'hyperv-hostname'
2015-11-05 15:33:57,763 INFO: Using configuration "hyperv-hostname1" ("hyperv" mode)
2015-11-05 15:33:57,763 INFO: Using configuration "hyperv-hostname2" ("hyperv" mode)
2015-11-05 15:33:57,829 DEBUG: Hyper-V url: http://10.0.0.0:5985/wsman
2015-11-05 15:33:57,833 DEBUG: Hyper-V url: http://10.0.0.0:5985/wsman
2015-11-05 15:33:57,841 DEBUG: Using NTLM authentication
2015-11-05 15:33:57,843 DEBUG: Using NTLM authentication
2015-11-05 15:33:57,887 ERROR: Virt backend 'hyperv-hostname1' fails with error: NTLM negotiation failed
2015-11-05 15:33:57,890 ERROR: Virt backend 'hyperv-hostname2' fails with error: NTLM negotiation failed

Version-Release number of selected component (if applicable):


How reproducible:
1. Install laste virt-who virt-who-0.14-4.el7.noarch.rpm
2. Configure virt-who with above configs
3. run virt-who -o -d to check autentication

Actual results:
ERROR: Virt backend 'hyperv-hostname2' fails with error: NTLM negotiation failed

Expected results:
Authentication should succeed.

Additional info:
Hyper-V are part of GPO (Microsoft Group Policy objects). When customer removes servers from GPO, virt-who can authenticate successfully.

Comment 1 Liushihui 2015-11-06 04:38:16 UTC
Please notice that: when Hyper-V is not part of GPO policy, virt-who can authenticate successfully only after set "AllowUnencrypted = false", However, if set "AllowUnencrypted = true",it still show " NTLM negotiation failed"

Comment 3 Radek Novacek 2015-11-24 07:34:30 UTC
Please, can you tell the customer to check if following setting in the Group Policy is set to "Disabled"

Computer > Policies > Administrative Templates > Windows Components > Windows Remote Management > WinRM Service: Disallow Negotiate Authentication.

Thanks.

Comment 7 Radek Novacek 2015-12-02 08:07:02 UTC
Ok, thanks. Did the customer try suggestion from comment #3?

Comment 10 Radek Novacek 2015-12-08 07:09:58 UTC
Jay,

looking at that screenshot, there is "Allow unencrypted traffic" option right above "Disallow Negotiate Authentication" and it is set to Disabled. virt-who is using unencrypted communication to talk to Hyper-V. Can the customer try to allow it?

Comment 13 Radek Novacek 2015-12-10 07:40:11 UTC
Anand,

can you please clarify what config options need to be changed in order to make virt-who work? I can work on finding some workaround but I need to know precisely what options in their GPO policy are not compatible with current virt-who requirements.

Comment 14 Radek Novacek 2015-12-10 07:47:17 UTC
Could the "AllowUnencrypted" option be the problem? virt-who needs this to be set to "true". In the bug description, this options is "true". In comment #4, there is "AllowUnencrypted = true [Source="GPO"]", but in comment #11, there is "AllowUnencrypted = false [Source="GPO"]"

Which one is customers original GPO setting?

Comment 16 Eko 2015-12-10 09:51:28 UTC
Hi Radek, I have set up an environment to reproduce this issue, my test result as following:
if set AllowUnencrypted = false [Source="GPO"], virt-who Failed;
if set AllowUnencrypted = true [Source="GPO"], virt-who pass;
if set AllowUnencrypted = true, virt-who pass;

so maybe it's not a GPO issue, the problem is how to make virt-who can work normally with AllowUnencrypted = false?

Comment 17 Radek Novacek 2015-12-10 11:54:43 UTC
Thanks, I'll take a look what would it need to support encrypted Hyper-V connection in virt-who.

Comment 21 Radek Novacek 2015-12-22 07:06:28 UTC
Anand, 

the workaround it to enable unencrypted connections. I'll give it a try to implement it in January. If I succeed, we can do a z-stream fix for this issue.

Comment 30 Liushihui 2016-02-02 03:56:11 UTC
In the latest In the latest virt-who-0.16-2.el6.noarch. test with the following three GPS settings, virt-who can get through NTLM authentication and send host/guest mapping to server successfully.

1. set AllowUnencrypted = false [Source="GPO"], virt-who pass;see the screenshot "Disallowed auth with GPO.png"
 "AllowUnencrypted = false [Source="GPO"]
    Auth
        Basic = false [Source="GPO"]
        Negotiate = true [Source="GPO"]
        IPv4Filter = * [Source="GPO"]
        IPv6Filter = * [Source="GPO"]
        AllowRemoteAccess = true [Source="GPO"] "

2. set AllowUnencrypted = false, without GPO, virt-who pass;see the screenshot "Disallowed auth without GPO.png"
 "AllowUnencrypted = false
    Auth
        Basic = false [Source="GPO"]
        Negotiate = true
        IPv4Filter = * [Source="GPO"]
        IPv6Filter = * [Source="GPO"]
        AllowRemoteAccess = true [Source="GPO"] "

3. set AllowUnencrypted = true [Source="GPO"], virt-who pass; see the screenshot "allow auth with GPO.png"
AllowUnencrypted = true [Source="GPO"]
    Auth
        Basic = false [Source="GPO"]
        Kerberos = true
        Negotiate = true [Source="GPO"]
    IPv4Filter = * [Source="GPO"]
    IPv6Filter = * [Source="GPO"]
    AllowRemoteAccess = true [Source="GPO"]

Comment 31 Liushihui 2016-02-02 03:57 UTC
Created attachment 1120307 [details]
Disallowed auth with GPO.png

Comment 32 Liushihui 2016-02-02 03:59 UTC
Created attachment 1120308 [details]
Disallowed auth without GPO.png

Comment 33 Liushihui 2016-02-02 03:59 UTC
Created attachment 1120309 [details]
Allow auth with GPO.png

Comment 37 Radek Novacek 2016-02-23 08:19:46 UTC
Anand, we can't just release it. We have to follow the process. That means to follow a z-stream process. But I don't think this bug qualifies for z-stream. This is clearly a new feature request. z-stream is meant to be for critical bug fixes only.

There is a risk that this change will break deployments for existing customers because it substantially changes how virt-who connects to hyper-v. I think we should wait for RHEL 7.3 GA.

If the customer really needs to have this ASAP, feel free to investigate options we have (z-stream, hotfix, async, etc.).

Comment 42 Radek Novacek 2016-03-08 06:50:15 UTC
The encrypted connection is already implemented upstream and will be resolved by rebase in 7.3. I will do the hotfix process ASAP.

Comment 44 Radek Novacek 2016-03-08 14:03:06 UTC
Hotfix build done: https://brewweb.devel.redhat.com/taskinfo?taskID=10620147

Comment 46 Radek Novacek 2016-03-09 07:44:57 UTC
Yes, although I would prefer QE team to test it first.

Comment 47 Liushihui 2016-03-10 04:38:07 UTC
Verified it on virt-who-0.14-9.el7.0.0.hotfix.1.bz1278637.noarch since virt-who can get through NTLM authentication and send host/guest mapping to server successfully when set "AllowUnencrypted = false" or "AllowUnencrypted = true".Meanwhile, guest can subscribe bonus pool successfully after hypervisor subscribe physical pool.

Checked version:
virt-who-0.14-9.el7.0.0.hotfix.1.bz1278637.noarch
subscription-manager-1.15.9-15.el7.x86_64
python-rhsm-1.15.4-5.el7.x86_64

Checked process:
1. Update virt-who version to the hostfix version
[root@hp-xl220agen8v2-01 ~]# rpm -q virt-who
virt-who-0.14-9.el7.noarch
[root@hp-xl220agen8v2-01 ~]# yum install -y python-requests
[root@hp-xl220agen8v2-01 ~]# rpm -Uvh virt-who-0.14-9.el7.0.0.hotfix.1.bz1278637.noarch.rpm 
Preparing...                          ################################# [100%]
Updating / installing...
   1:virt-who-0.14-9.el7.0.0.hotfix.1.################################# [ 50%]
Cleaning up / removing...
   2:virt-who-0.14-9.el7              ################################# [100%]
[root@hp-xl220agen8v2-01 ~]# rpm -q virt-who
virt-who-0.14-9.el7.0.0.hotfix.1.bz1278637.noarch
2. Register system to satellite6.1
3. In hyperv, configure local group policy as the following three conditions.
Config1: set AllowUnencrypted = false [Source="GPO"], virt-who pass;see the screenshot "Disallowed auth with GPO.png"
 "AllowUnencrypted = false [Source="GPO"]
    Auth
        Basic = false [Source="GPO"]
        Negotiate = true [Source="GPO"]
        IPv4Filter = * [Source="GPO"]
        IPv6Filter = * [Source="GPO"]
        AllowRemoteAccess = true [Source="GPO"] "
Config2. set AllowUnencrypted = false, without GPO, virt-who pass;see the screenshot "Disallowed auth without GPO.png"
 "AllowUnencrypted = false
    Auth
        Basic = false [Source="GPO"]
        Negotiate = true
        IPv4Filter = * [Source="GPO"]
        IPv6Filter = * [Source="GPO"]
        AllowRemoteAccess = true [Source="GPO"] "
Config3. set AllowUnencrypted = true [Source="GPO"], virt-who pass; see the screenshot "allow auth with GPO.png"
AllowUnencrypted = true [Source="GPO"]
    Auth
        Basic = false [Source="GPO"]
        Kerberos = true
        Negotiate = true [Source="GPO"]
    IPv4Filter = * [Source="GPO"]
    IPv6Filter = * [Source="GPO"]
    AllowRemoteAccess = true [Source="GPO"]
4. Configure virt-who run at hyperv mode ,restart virt-who and check virt-who's log
[root@hp-xl220agen8v2-01 ~]# cat /etc/virt-who.d/virt 
[test-hyperv1]
type=hyperv
server=10.73.5.227
username=administrator
password=Welcome1
owner=ACME_Corporation
env=Library 
[root@hp-xl220agen8v2-01 ~]# service virt-who restart && tail -f /var/log/rhsm/rhsm.log
2016-03-09 23:32:37,320 [INFO]  @virtwho.py:697 - Using configuration "test-hyperv1" ("hyperv" mode)
2016-03-09 23:32:37,320 [DEBUG]  @virtwho.py:216 - Starting infinite loop with 5 seconds interval
2016-03-09 23:32:37,358 [DEBUG]  @hyperv.py:477 - Hyper-V url: http://10.73.5.227:5985/wsman
2016-03-09 23:32:38,660 [DEBUG]  @hyperv.py:71 - Using NTLM authentication
2016-03-09 23:32:39,979 [DEBUG]  @hyperv.py:84 - Sending NTLM authentication data
2016-03-09 23:32:40,585 [DEBUG]  @hyperv.py:107 - NTLM authentication successful
2016-03-09 23:32:40,590 [DEBUG]  @hyperv.py:511 - Unable to enumerate using root/virtualization namespace, trying root/virtualization/v2 namespace
2016-03-09 23:32:44,461 [DEBUG]  @virt.py:343 - Getting the host/guests association took too long, interval waiting is skipped
2016-03-09 23:32:44,463 [DEBUG]  @subscriptionmanager.py:112 - Authenticating with certificate: /etc/pki/consumer/cert.pem
2016-03-09 23:32:44,589 [DEBUG]  @subscriptionmanager.py:146 - Checking if server has capability 'hypervisor_async'
2016-03-09 23:32:44,709 [DEBUG]  @subscriptionmanager.py:158 - Server does not have 'hypervisors_async' capability
2016-03-09 23:32:44,710 [INFO]  @subscriptionmanager.py:165 - Sending update in hosts-to-guests mapping: {
    "hyperv_01": [
        {
            "guestId": "32710A7E-94A9-A445-944E-16C01BFA63B3", 
            "state": 1, 
            "attributes": {
                "active": 1, 
                "virtWhoType": "hyperv", 
                "hypervisorType": "hyperv"
            }
        }, 
        {
            "guestId": "0E32F0E5-05CA-014A-BD59-F63D75843D5D", 
            "state": 1, 
            "attributes": {
                "active": 1, 
                "virtWhoType": "hyperv", 
                "hypervisorType": "hyperv"
            }
        }
    ]
}

5. In satellite webUI, Go to "content host" --> choose the [hyperv_hostnae]-->"Subscriptions" --> "Add", choose physical pool which can generate bonus pool on hypervisor ,then subscribe it.
6. In the Guest, list the bonus pool and subscribe the bonus pool

Result:
Virt-who send correct host/guest mapping info to satellite, guest can subscribe bonus pool successfully.

Comment 49 Radek Novacek 2016-03-10 06:41:22 UTC
Anand, yes, you can provide the package to the customer.

Comment 50 Eko 2016-04-27 05:40:35 UTC
*** Bug 1167283 has been marked as a duplicate of this bug. ***

Comment 51 Eko 2016-04-27 05:42:49 UTC
Verified it on virt-who-0.14-9.el7.0.0.hotfix.1.bz1278637.noarch,
and can't reproduce it in in virt-who-0.16-8.el6.noarch

Comment 53 Radek Novacek 2016-09-29 14:36:22 UTC
The updated Doc Text is fine. Thanks.

Comment 55 errata-xmlrpc 2016-11-04 05:06:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2387.html


Note You need to log in before you can comment on or make changes to this bug.